What it is
Item response theory (IRT) is a family of measurement models that characterises each item in a test or scale by its statistical properties — difficulty, discrimination, and (for multiple-choice items) guessing — and each respondent by their position on the underlying latent trait. The most widely used IRT model is the Rasch model, which is the simplest case: it assumes all items have equal discrimination and characterises each item solely by its difficulty (the point on the latent trait where a respondent has a 50% probability of a correct response).
The key advantage over classical test theory (which underlies Cronbach’s alpha and sum scores) is that IRT parameter estimates are theoretically invariant: the difficulty of an item does not depend on which respondents happened to take the test, and a respondent’s ability estimate does not depend on which items they happened to answer. This invariance property enables fair comparison of groups who answered different items, calibration of item banks, and targeted test design.
When to use it
IRT is appropriate when: a test or scale is being developed for a specific population and items need to be calibrated and selected; groups are being compared on a latent trait measured by different test forms; or the precision of measurement needs to be evaluated at specific points on the ability distribution.
In development economics field research, IRT is increasingly used for learning assessments — ASER, EGRA, UWEZO — where the goal is to place children at different points on a reading or numeracy scale using items of varying difficulty. Sandefur (2018) uses IRT to place PISA results on a common scale across countries with different tests, enabling cross-national learning comparisons. Doss et al. (2019) use Rasch models to evaluate the psychometric properties of a phone-based learning assessment in Sierra Leone.
IRT is less appropriate when the sample is too small (stable IRT parameter estimates typically require 200+ respondents for the Rasch model and 500+ for more complex IRT models), when the scale has fewer than 5–10 items, or when the goal is simply a composite score rather than item calibration.
How it works
The Rasch model. The probability that respondent i with ability θi correctly answers item j with difficulty bj is:
P(correct) = exp(θi − bj) / (1 + exp(θi − bj))
This is a logistic function of the difference between person ability and item difficulty. Items are harder when bj is large; respondents have more ability when θi is large. Both are on the same logit scale, so a person whose ability equals an item’s difficulty has a 50% chance of answering correctly.
Parameter estimation. Person and item parameters are estimated jointly using maximum likelihood. In the Rasch model, the total number of correct responses is a sufficient statistic for person ability — all other information in the response pattern is irrelevant to ability estimation. More complex IRT models (2PL, 3PL) add discrimination and guessing parameters, providing better fit at the cost of more parameters and larger sample requirements.
Item fit statistics. After estimation, each item is assessed for fit to the model. Items with poor fit — where the observed proportion correct does not match the model-predicted proportion at each ability level — may be poorly constructed, multi-dimensional, or differentially functioning for subgroups. Infit and outfit statistics (mean-square residuals) are the standard fit diagnostics; values between 0.7 and 1.3 are typically acceptable.
Person-item maps. A Wright map (person-item map) plots the distribution of person abilities alongside the difficulty distribution of items on the same logit scale. This reveals whether items cover the full range of abilities in the population — items that are all easy or all hard relative to the ability distribution fail to differentiate respondents in the region where differentiation matters most.
Key decisions
Rasch vs. 2PL vs. 3PL. The Rasch model imposes equal discrimination across items and is more parsimonious. It is the appropriate choice when sample sizes are modest, items are purposefully designed to have similar formats, or theoretical reasons exist to expect parallel item-response functions. The 2-parameter logistic (2PL) model adds item-specific discrimination parameters, allowing items to vary in how strongly they differentiate between ability levels. The 3PL model adds a guessing parameter for multiple-choice items. More complex models require larger samples and are harder to interpret.
Polytomous items. For Likert-scale items with ordered response categories (1–5), the Rasch model generalises to the partial credit model or the rating scale model. These model the probability of each category response as a function of person ability and item threshold parameters. These models are used for attitude scales, non-cognitive skill batteries, and wellbeing measures.
Linking test forms. When different groups of respondents answer different item sets (as in adaptive testing or multi-form assessments), their ability estimates can still be placed on a common scale if the forms share a set of common anchor items. The anchor items are used to link the two forms — estimating the transformation needed to express scores from both forms on the same scale. This is standard in large-scale learning assessments.
Caveats & common mistakes
Unidimensionality assumption. IRT models assume that the items measure a single latent dimension. If items tap multiple dimensions — reading fluency, reading comprehension, and vocabulary in a “reading” test — the Rasch model parameters are biased and ability estimates conflate the dimensions. A confirmatory factor analysis checking the one-factor assumption before IRT calibration is a standard preliminary step.
Local independence. IRT also assumes conditional independence: given a respondent’s ability, responses to different items are uncorrelated. Violations occur when items share content (e.g., multiple questions about the same passage) or when a correct response to one item gives away the answer to another. Residual correlations between item pairs — estimated after removing the common factor — identify local dependence.
Invariance is conditional on model fit. The theoretical invariance properties of IRT hold only if the model fits the data. Items with poor fit violate the assumptions that produce invariance. Reporting item fit statistics and removing or revising poorly fitting items is necessary for the invariance claims to hold in practice.
Small sample limitations. Stable item difficulty estimates from the Rasch model require at least 100–200 respondents. With smaller samples, parameters are noisy and item calibration is unreliable. For pilot studies or small field trials, classical test theory approaches are more appropriate.
Analysis Guide
# item_1 to item_20: binary correct/incorrect items (0/1)
import pandas as pd
import numpy as np
from girth import rasch_jml, twopl_mml, ability_mle
item_cols = [f"item_{i}" for i in range(1, 21)]
# girth expects items as rows, persons as columns
item_data = df[item_cols].to_numpy().T
# 1. Fit Rasch (1PL) via joint maximum likelihood — estimates item difficulty parameters (b) and (implicitly) person abilities on the same logit scale; Rasch assumes equal discrimination across items, so difficulty is the only item parameter
rasch_fit = rasch_jml(item_data)
print("Difficulties:", rasch_fit["Difficulty"])
# 2. Inspect item difficulties — on the logit scale, an item with difficulty 0 is answered correctly by 50% of respondents at average ability; items far below the ability distribution are too easy, items far above are too hard
difficulties = pd.Series(rasch_fit["Difficulty"], index=item_cols)
print(difficulties.sort_values())
# 3. Person ability estimates — maximum likelihood theta is each respondent's position on the latent ability scale, used as the scale score in subsequent analysis
theta = ability_mle(item_data, rasch_fit["Difficulty"], np.ones_like(rasch_fit["Difficulty"]))
df["ability"] = theta
# 4. Fit 2PL — adds item-specific discrimination parameters; useful when items vary in how strongly they differentiate ability levels; requires larger samples than Rasch
twopl_fit = twopl_mml(item_data)
print("Discrimination:", twopl_fit["Discrimination"])
print("Difficulty: ", twopl_fit["Difficulty"])
# 5. Item characteristic curves — compute P(correct | theta) for each item across the ability grid; plot to verify the S-shape and that the 50% point sits within the population's ability range
theta_grid = np.linspace(-3, 3, 100)
icc = 1 / (1 + np.exp(-(theta_grid[:, None] - rasch_fit["Difficulty"][None, :])))
# import matplotlib.pyplot as plt; plt.plot(theta_grid, icc)
# Note: girth covers Rasch/2PL/3PL well. For partial credit, graded response, or rating scale
# models on polytomous Likert items, infit/outfit MSQ diagnostics, and Wright maps, the R mirt
# package is more mature - see the R tab.Reading the output
- Item infit and outfit mean-square (MSQ) statistics should fall between 0.7 and 1.3 for acceptable model fit. Values above 1.3 indicate an item is too noisy (responses do not conform to the expected pattern given ability); values below 0.7 indicate an item is over-predictable (possibly locally dependent with another item).
- Item difficulty parameters (xsi / b) are on the logit scale: an item with difficulty 0 is answered correctly by 50% of respondents at average ability. Items with difficulty far below the ability distribution are too easy and do not differentiate; items far above are too hard and contribute little information for most respondents.
- The Wright map visualises fit between person abilities and item difficulties: items should span the full range of the ability distribution. Large gaps in item coverage indicate regions where measurement precision is low.
- Person ability estimates (theta) are on the same logit scale as item difficulties. A person with theta = 1.0 has a higher-than-50% probability of correctly answering an item of difficulty 1.0.
- For the Partial Credit Model on polytomous items, inspect the threshold parameters — ordered thresholds confirm that response categories function as intended; reversed thresholds indicate a category that respondents skip over.
References
Bond, T., & Fox, C. (2015). Applying the Rasch Model: Fundamental Measurement in the Human Sciences (3rd ed.). Routledge.
Doss, C., Fahle, E. M., Loeb, S., & York, B. N. (2019). More than one way to assess: Comparing the reliability, validity, and equity of phone-based surveys with in-person assessments. Journal of Research on Educational Effectiveness, 12(4), 522–547. https://doi.org/10.1080/19345747.2019.1600730
Embretson, S. E., & Reise, S. P. (2000). Item Response Theory for Psychologists. Erlbaum.
Sandefur, J. (2018). Internationally comparable mathematics scores for fourteen African countries. Economics of Education Review, 62, 267–286. https://doi.org/10.1016/j.econedurev.2017.11.010