Belief Elicitation / Binarized Scoring Rule

A method for eliciting incentive-compatible probabilistic beliefs about discrete outcomes, using a binarized scoring rule that makes truthful probability reporting the payoff-maximising strategy regardless of the respondent's risk preferences.

What it is

Belief elicitation asks respondents to report the probability they assign to a specific event — whether a policy will be implemented, whether a crop will succeed, whether a loan will be repaid. The challenge is that simply asking for a probability provides no incentive to be truthful: a respondent may anchor on a round number, report a socially desirable belief, or exert no cognitive effort.

Proper scoring rules create incentive compatibility: the expected payoff from reporting a probability p is maximised only when p is the respondent’s true belief. The binarized scoring rule (BSR), introduced by Hossain and Okui (2013), extends proper scoring to subjects who may be risk-averse — a critical adaptation for field settings where standard quadratic scoring rules produce biased reports when respondents do not have linear utility.

The BSR works by converting the scored outcome into a binary lottery rather than a deterministic payment, which breaks the link between the scoring rule and the respondent’s utility function. Under the BSR, truthful reporting is optimal regardless of whether the respondent is risk-neutral, risk-averse, or risk-seeking.

When to use it

Belief elicitation is appropriate when first-order beliefs are a primary outcome — the study is specifically trying to measure what respondents think will happen, not just what they prefer or how they behave. It is distinct from subjective expectations elicitation, which focuses on beliefs about future states of the world. Belief elicitation under a scoring rule is used when beliefs about a specific event with a verifiable outcome are being measured and real incentives are feasible.

Applications in development economics include: eliciting managers’ or workers’ beliefs about their own or others’ performance before revealing actual outputs; measuring farmers’ beliefs about the success probability of new technologies before adoption; and measuring social beliefs — what proportion of one’s peers engage in a behaviour — where the verifiable outcome is the true population rate.

Schotter and Trevino (2014) review belief elicitation methods in experiments and document the gains in accuracy from incentivised over unincentivised protocols. Gächter and Renner (2010) show that incentivised belief elicitation in public goods experiments reveals that players are more conditional cooperators than unincentivised protocols suggest — demonstrating that the incentive structure matters for what beliefs are measured.

How it works

The binarized scoring rule converts a probabilistic report into a lottery as follows:

Step 1 — Belief report. The respondent reports a probability p (from 0 to 1, or 0 to 10 in integer steps) for an event E.

Step 2 — Random draw. An independent random number q is drawn uniformly from 0 to 1.

Step 3 — Payment rule:

If the event E occurs: the respondent receives the prize with probability p² (or 1, depending on formulation).
If the event E does not occur: the respondent receives the prize with probability (1−p)².

Equivalently: if E occurs, the respondent wins a lottery with winning probability p; if E does not occur, they win a lottery with winning probability (1−p). The respondent maximises their expected utility by reporting the true probability — overstating or understating p in either direction reduces the probability of winning the prize.

The implementation requires a two-stage randomisation: one draw to determine whether E occurred, and one draw to determine whether the respondent wins given their report and the outcome. In field settings, both draws can be done with a spinner or random draw device.

Key decisions

Integer scale. Asking for probabilities on a 0–10 integer scale (where 7 means 70%) rather than a continuous 0–1 scale is standard in field settings. This reduces cognitive burden while preserving sufficient resolution. The scoring rule is adjusted accordingly — probabilities are expressed as k/10 for integer reports k.

Outcome verification timing. The BSR is only incentive-compatible if the outcome is verifiable and will be revealed before payment. The timing of outcome revelation must be credible — if respondents believe they can “wait and see” what the researcher claims the outcome was, incentive compatibility breaks down. Short resolution windows (within the same session, or within a month) are preferred over long ones.

Pairing with other tasks. Belief elicitation is frequently paired with choice tasks to measure the relationship between beliefs and decisions. If beliefs about a peer’s behaviour are elicited alongside a cooperation decision, the belief can be used as a control in the analysis of cooperative choice. The order of elicitation (beliefs before or after the relevant decision) should be specified in advance, as it affects both the belief report and the decision.

Using a “natural” outcome vs. a constructed one. Some belief elicitation designs create outcomes specifically for the experiment (e.g., the researcher will randomly audit some transactions). Others tie belief elicitation to naturally occurring outcomes (the harvest season, an election result). Natural outcomes have greater ecological validity; constructed outcomes allow tighter control over timing and verifiability.

Communicating the scoring rule. The BSR is more complex than a simple prediction task. Respondents need to understand that they should not try to “game” the system and that truthful reporting maximises their expected payoff. A worked example with small numbers — “if you report 7 out of 10, and the event happens, you win a prize with probability 7 out of 10; if the event does not happen, you win with probability 3 out of 10” — helps. Comprehension checks are important.

Caveats & common mistakes

Risk preferences still matter for effort, not direction. The BSR makes truthful reporting optimal regardless of risk preferences, but the precision with which respondents search for their true probability (how much effort they put into introspection) may still be affected by the prize size. Higher prizes increase the value of reporting accurately and increase cognitive effort. Prizes that are too small may produce noisy reports even with a proper scoring rule.

Anchoring and round numbers. Even with incentives, respondents anchor on round numbers — reporting 0.5, 0.6, 0.7 more frequently than surrounding values. This is less severe with visual aids (a probability wheel) than with verbal reports, but it does not disappear entirely. Analysing the distribution of reports and testing for excess mass at focal values is a standard quality check.

Multiple beliefs and portfolio effects. If respondents report multiple beliefs in sequence — for different outcomes or different future periods — they may reason about their overall portfolio of scored outcomes rather than treating each independently. This is a general problem with incentivised multiple-elicitation designs and can be reduced by randomly selecting one belief report for actual payment, making each belief independent.

Complexity failure. The BSR requires understanding a probability-of-a-probability structure. In low-numeracy populations, comprehension failure is a genuine risk. An unincentivised simple probability question that is understood is more useful than an incentivised BSR that is not. Comprehension checks should include a worked example where the respondent calculates a payment, not just confirms they understand the instructions.

Analysis Guide

# belief_report: stated probability (0-10 integer scale)
# event_occurred: 1 if event happened, 0 if not
# q_draw: uniform random draw (0-1) for BSR payment determination
import pandas as pd
import numpy as np
import statsmodels.formula.api as smf
from sklearn.metrics import brier_score_loss

# 1. Convert integer report to probability — the 0-10 scale reduces cognitive burden
#    in the field; dividing by 10 gives the probability value used in all downstream
#    BSR payment calculations
df["prob_report"] = df["belief_report"] / 10

# 2. Compute the BSR win probability for each respondent — the formula differs by
#    whether the event occurred; this is what makes truthful reporting optimal
#    regardless of risk preferences: overstating or understating p reduces win_prob
#    in both event outcomes. Win prob if event = prob_report^2; if not = (1-prob_report)^2
df["win_prob"] = np.where(df["event_occurred"] == 1,
                        df["prob_report"]**2,
                        (1 - df["prob_report"])**2)
df["won_bsr"]  = (df["q_draw"] <= df["win_prob"]).astype(int)

# 3. Inspect the distribution of belief reports — excess mass at 0, 5, and 10
#    indicates anchoring on round numbers rather than genuine probability estimation;
#    above 30% of reports at exactly 5 suggests respondents defaulted to the midpoint
print(df["belief_report"].value_counts().sort_index())

# 4. Compute the Brier score and compare average beliefs to the realised event rate —
#    Brier score is the mean squared error between stated probability and outcome;
#    lower is better (0 = perfect, 0.25 = chance for a binary outcome at p=0.5);
#    a large gap between mean prob_report and mean event_occurred signals bias
print("Brier:", brier_score_loss(df["event_occurred"], df["prob_report"]))
print(df["prob_report"].mean(), df["event_occurred"].mean())

# 5. Formal calibration test — regress the event indicator on reported probabilities;
#    a coefficient near 1 with intercept near 0 means beliefs are well-calibrated;
#    a slope below 1 indicates overconfidence; above 1 indicates underconfidence
fit_cal = smf.ols("event_occurred ~ prob_report", data=df).fit(cov_type="HC1")
print(fit_cal.summary())

# 6. Use beliefs as a predictor of related decisions — a positive coefficient means
#    higher stated probability is associated with more of the related behaviour,
#    testing whether beliefs predict choices in the expected direction
fit = smf.ols("decision_variable ~ belief_report + age + C(female) + log_hh_expenditure",
            data=df).fit(cov_type="HC1")
print(fit.summary())

# 1. Convert integer report to probability — the 0–10 scale reduces field burden;
#    dividing by 10 gives the probability value used in BSR payment calculation
df$prob_report <- df$belief_report / 10

# 2. Compute BSR win probability — depends on whether the event occurred;
#    this structure makes truthful reporting the payoff-maximising strategy regardless
#    of whether the respondent is risk-averse, risk-neutral, or risk-seeking
df$win_prob <- ifelse(df$event_occurred == 1,
                    df$prob_report^2,
                    (1 - df$prob_report)^2)
df$won_bsr  <- as.integer(df$q_draw <= df$win_prob)

# 3. Inspect report distribution — excess mass at 0, 5, and 10 signals anchoring;
#    above 30% at exactly 5 suggests respondents defaulted to the midpoint rather
#    than engaging with the probability estimation
table(df$belief_report)

# 4. Calibration test — slope near 1 and intercept near 0 means stated probabilities
#    match realised event rates on average; slope below 1 indicates overconfidence
library(estimatr)
fit_cal <- lm_robust(event_occurred ~ prob_report, data = df, se_type = "HC2")
summary(fit_cal)   # intercept near 0, slope near 1 = well calibrated

# 5. Use beliefs as a predictor of related decisions — tests whether respondents who
#    assigned higher probability to the event were more likely to act on it;
#    a positive coefficient is the expected direction if beliefs drive behaviour
fit <- lm_robust(decision_variable ~ prob_report + age + female + log_hh_expenditure,
               data = df, se_type = "HC2")
summary(fit)

XLSForm / SurveyCTO

Use an integer field for the belief report (constrained 0–10). Store a pre-generated random number q for each respondent in the external dataset and load it with pulldata(), or generate it during the survey with once(random()). Calculate the win probability and outcome payment in calculate fields. Display the winning determination at the end of the session — or at a follow-up session if the outcome is not yet known. For the randomised payment device, use a spinner or numbered cards to make the probability draw tangible and credible to the respondent.

Reading the output

In tab belief_report, excess mass at 0, 5, and 10 signals anchoring. If more than 30% of reports fall exactly at 5, respondents may be defaulting to the midpoint rather than engaging with the probability estimation. A more even distribution across values is a sign of genuine engagement.
In the calibration regression (event_occurred ~ prob_report), a slope near 1 and intercept near 0 means beliefs are well-calibrated on average — stated probabilities match realised event rates. A slope below 1 indicates overconfidence (respondents overstated certainty in both directions); a slope above 1 indicates underconfidence.
won_bsr is the individual payment indicator. Verify that the fraction winning (mean of won_bsr) is close to what the BSR formula predicts for the distribution of reports and the event realisation rate — this serves as a check on the payment calculation.
A positive coefficient on prob_report in the decision regression means respondents who assigned higher probability to the event were more likely to take a related action. This is the standard test of whether beliefs predict behaviour.

References

Gächter, S., & Renner, E. (2010). The effects of (incentivized) belief elicitation in public goods experiments. Experimental Economics, 13(3), 364–377. https://doi.org/10.1007/s10683-010-9246-4

Hossain, T., & Okui, R. (2013). The binarized scoring rule. Review of Economic Studies, 80(3), 984–1001. https://doi.org/10.1093/restud/rds044

Schotter, A., & Trevino, I. (2014). Belief elicitation in the laboratory. Annual Review of Economics, 6, 103–128. https://doi.org/10.1146/annurev-economics-080213-041008

Selten, R. (1998). Axiomatic characterization of the quadratic scoring rule. Experimental Economics, 1(1), 43–62. https://doi.org/10.1007/BF01426214