What it is
Test-retest reliability measures whether a survey instrument produces consistent results across repeated administrations under unchanged conditions. If the same respondent is asked the same questions two weeks apart and their true status has not changed, a reliable instrument should produce the same (or very similar) responses both times. Low test-retest reliability means the instrument is noisy — it measures something, but partly measures transient states, question interpretation variability, or random error rather than the stable underlying construct.
Test-retest reliability is distinct from internal consistency (Cronbach’s alpha, which measures consistency across items at a single point in time). A scale can have high internal consistency and low test-retest reliability if respondents interpret the same items differently on different occasions.
When to use it
Test-retest reliability assessment is most important during instrument development, before a large-scale baseline survey, and when adapting a validated instrument to a new context or language. It is also informative for understanding how much observed change in a panel study is real (treatment-induced) versus due to measurement noise — the measurement error component of observed change inflates the variance of difference estimators.
A test-retest interval of 2–4 weeks is standard for most survey constructs in development research. Shorter intervals risk memory contamination (respondents remember and repeat their first answers). Longer intervals risk genuine change in the construct of interest.
How it works
Intraclass correlation coefficient (ICC). For continuous or ordinal measures, the ICC(2,1) or ICC(3,1) from a two-way mixed effects model is the standard test-retest reliability statistic. ICC values above 0.75 indicate good reliability; values between 0.50 and 0.75 indicate moderate reliability; values below 0.50 indicate poor reliability. The appropriate ICC variant depends on whether the raters (or administrations) are considered fixed or random effects.
Pearson or Spearman correlation. Simpler than ICC and more commonly reported in the development economics literature. The correlation between Time 1 and Time 2 scores measures agreement in rank ordering. Note that this does not capture systematic bias (a measure that is always 10 points higher at Time 2 will have a high correlation but poor agreement).
Bland-Altman plot. A graphical method that plots the difference between Time 1 and Time 2 scores against their mean, across all respondents. Systematic bias (the mean difference is far from zero) and heteroscedastic error (differences are larger at high values) are immediately visible. This plot is standard in clinical measurement but underused in development economics.
Kappa for categorical items. For binary or nominal items, Cohen’s kappa measures agreement beyond chance. Kappa > 0.6 is moderate agreement; kappa > 0.8 is strong agreement. Weighted kappa extends this to ordinal items, giving partial credit for near-agreement.
Key decisions
Selecting respondents for the test-retest sample. Test-retest studies require a purposive subsample of 50–150 respondents who: (a) are representative of the full survey population; (b) have stable characteristics during the test-retest interval; and (c) can be re-contacted reliably. Selecting from the most accessible respondents (those in easily reached areas) can bias reliability estimates if these respondents differ systematically from the full sample.
Separating stability from reliability. Low test-retest reliability can reflect genuine instability in the construct (attitudes and recall can change week to week) or unreliability in the instrument (the same true attitude is measured inconsistently). These cannot be fully separated without an external criterion. For inherently volatile constructs (daily mood, recent expenditure), low test-retest correlation should not be interpreted as poor instrument quality.
Using reliability to adjust power calculations. If a primary outcome is measured with test-retest reliability r, the effective signal-to-noise ratio is reduced relative to a perfectly reliable measure. An effect size of δ measured with reliability r requires a sample size larger by a factor of approximately 1/r to achieve the same power. Building reliability estimates into power calculations produces more realistic sample sizes.
Caveats & common mistakes
Memory contamination. A test-retest interval that is too short allows respondents to remember and repeat their first answers rather than independently responding. This inflates reliability estimates. An interval of 10–14 days is usually sufficient to reduce memory effects while still being short enough to limit genuine construct change.
Confusing reliability with validity. A highly reliable measure can still be invalid — consistently measuring the wrong thing. Test-retest reliability is a necessary but not sufficient condition for validity. A thermometer that consistently reads 2°C too high has perfect test-retest reliability and poor validity.
Not reporting confidence intervals. Test-retest ICC estimates are statistics with sampling variability that can be substantial with small samples (n < 100). Confidence intervals — calculated from the F-distribution for ICC or bootstrapped — should be reported alongside point estimates.
Analysis Guide
import pandas as pd
import numpy as np
import pingouin as pg
from scipy import stats
from sklearn.metrics import cohen_kappa_score
import matplotlib.pyplot as plt
# 1. Pearson correlation between waves — a first-pass reliability check that
# measures rank-order consistency but cannot detect systematic mean shift
r, p = stats.pearsonr(df["score_t1"].dropna(), df["score_t2"].dropna())
print(f"Pearson r = {r:.3f}, p = {p:.4f}")
# 2. ICC via long-format reshape: long format is required because pingouin
# treats each rater/wave as a within-subject factor; ICC(2,1) corresponds
# to type "ICC2" (two-way random, single rater, absolute agreement)
long = df.melt(id_vars="respondent_id",
value_vars=["score_t1", "score_t2"],
var_name="wave", value_name="score")
icc = pg.intraclass_corr(data=long, targets="respondent_id",
raters="wave", ratings="score")
print(icc[icc["Type"] == "ICC2"])
# 3. Cohen's kappa for categorical items — unweighted for nominal codes,
# linear or quadratic weights for ordinal scales where near-misses matter less
k_unwt = cohen_kappa_score(df["item_t1"], df["item_t2"])
k_wt = cohen_kappa_score(df["item_t1"], df["item_t2"], weights="quadratic")
print(f"Kappa unweighted = {k_unwt:.3f} | quadratic-weighted = {k_wt:.3f}")
# 4. Bland-Altman plot: difference vs. mean exposes systematic bias (mean far
# from zero) and heteroscedastic error (fan-out at high values); 95% limits
# are mean diff +/- 1.96 SD and bound where individual differences should fall
diff = df["score_t2"] - df["score_t1"]
mean = (df["score_t1"] + df["score_t2"]) / 2
plt.scatter(mean, diff)
plt.axhline(diff.mean()); plt.axhline(diff.mean() + 1.96 * diff.std(), ls="--")
plt.axhline(diff.mean() - 1.96 * diff.std(), ls="--")
plt.xlabel("Mean (T1, T2)"); plt.ylabel("T2 - T1"); plt.show()Reading the output
- ICC above 0.75 indicates good test-retest reliability; 0.50–0.75 is moderate; below 0.50 is poor — an instrument with ICC < 0.50 will substantially attenuate treatment effect estimates in panel designs.
- Pearson correlation above 0.80 is a reasonable rule of thumb for continuous scales, but correlation does not detect systematic bias (a measure that is always 10 points higher at T2 can still correlate at r = 1.0).
- Bland-Altman: the mean difference (bias) should be close to zero; limits of agreement (mean ± 1.96 SD of differences) define the range within which 95% of individual differences fall — wide limits indicate high noise.
- Kappa above 0.60 is acceptable for most research purposes; kappa below 0.40 means the categorical coding is too unreliable to use as an outcome or covariate.
- Always report 95% confidence intervals around ICC and kappa — with n < 100, intervals are wide enough to matter.
References
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 327(8476), 307–310. https://doi.org/10.1016/S0140-6736(86)90837-8
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric Theory (3rd ed.). McGraw-Hill.