Attrition Diagnostics

A diagnostic for identifying attrition bias in panel or follow-up surveys — testing whether respondents lost to follow-up differ systematically from those retained, and whether attrition rates differ between treatment and control groups in ways that threaten causal inference.

What it is

Attrition occurs when respondents who were surveyed in an earlier wave cannot be located, refuse to participate, or are otherwise unavailable at follow-up. In an RCT, attrition is a threat to internal validity when it is differential — when the rate or composition of dropout differs between treatment and control groups. Even modest differential attrition can undo the balance achieved by randomisation, because the follow-up samples in treatment and control may no longer be comparable.

Attrition diagnostics address two questions: (1) what is the overall attrition rate and is it balanced across arms? and (2) do attriters and non-attriters differ in ways that could bias estimated treatment effects? These questions are related but distinct. High attrition that is completely random produces inefficiency (smaller effective sample) but not bias. Low attrition that is correlated with treatment response is a more serious problem.

When to use it

Attrition diagnostics should be run as a standard pre-analysis step in any panel or follow-up design. They are most critical in randomised evaluations where the follow-up sample will be used to estimate treatment effects. They should be reported transparently in the analysis even when attrition rates are low, since reviewers and readers will expect to see them.

Lee (2009) provides bounds on treatment effects that are robust to selective attrition — the Lee bounds approach does not eliminate bias but quantifies the range of effects consistent with the data. Glewwe and Kassouf (2012) report attrition diagnostic tables as a standard component of their experimental analysis.

How it works

Attrition rate by group. Compute the share of baseline respondents who were not interviewed at follow-up, separately for treatment and control. A chi-square test or t-test on the difference in attrition rates across arms provides a statistical test of differential attrition. Attrition rates above 20% warrant careful investigation; differential attrition of more than 5 percentage points between arms is a red flag regardless of the overall rate.

Baseline balance among attriters. Test whether attriters and non-attriters had similar baseline characteristics. Run a regression of attrition status (1 = attrited, 0 = retained) on baseline covariates, separately for treatment and control. If different covariates predict attrition in the two arms, the follow-up samples are no longer balanced on those dimensions.

Interaction test for differential attrition. Run a regression of attrition status on treatment assignment, baseline covariates, and interactions between treatment and each covariate. Significant interactions indicate that the composition of dropout differs across arms in ways that could bias treatment effect estimates.

Lee bounds. If differential attrition is confirmed, compute Lee (2009) bounds on the treatment effect. The bounds trim the treatment group by removing the proportion of respondents equal to the excess attrition differential, first from the top and then from the bottom of the outcome distribution. The resulting interval is the range of causal effects consistent with the data under any pattern of selective attrition. Narrow bounds support causal interpretation; wide bounds suggest attrition is a serious threat.

Key decisions

Treatment of one-sided vs. two-sided attrition. Attrition is most problematic when the treatment group loses more respondents than the control (or vice versa). When attrition rates are balanced but selection into retention differs, the problem is subtler. Both dimensions should be examined.

Tracking effort documentation. Attrition rates are partly a function of how hard the team tried to locate respondents. Documenting the number of contact attempts, the use of locator forms, and the reasons for non-contact (refusal vs. could not locate vs. deceased) is important for interpreting attrition rates. High-effort tracking that still fails for 15% of respondents is a different situation from low-effort tracking with the same rate.

Inverse probability weighting (IPW). If attrition is missing at random (MAR) — dropout depends on observed covariates but not on the unobserved outcome — IPW can recover unbiased estimates. Weights are the inverse of the predicted probability of retention given baseline characteristics. IPW requires the MAR assumption, which is untestable but can be made more plausible by including rich baseline covariates in the attrition model.

Caveats & common mistakes

Treating balanced attrition as harmless. Equal attrition rates across arms does not mean attrition is non-differential. If treatment causes some high-outcome respondents to migrate and be lost to follow-up while control causes some low-outcome respondents to drop out, attrition rates can be equal but the composition of the retained samples is biased in opposite directions.

Not reporting attrition tables. Omitting attrition diagnostics from reported results — or burying them in appendices — is a transparency failure that undermines the credibility of the analysis. Attrition tables (rates by arm, balance among attriters, Lee bounds if needed) should be standard.

Confusing tracking failure with survey non-response. A respondent who was located but refused to participate is categorically different from one who could not be located at all. Refusal may be correlated with treatment (e.g., treated respondents are more likely to refuse if treatment had negative effects). Separating these two components of attrition is informative.

Analysis Guide

import pandas as pd
import numpy as np
from scipy import stats
import statsmodels.formula.api as smf

# attrited: 1 = not interviewed at follow-up, 0 = interviewed
# treatment: 1 = treatment, 0 = control

# 1. Attrition rate by arm — the starting point for any attrition analysis;
#    differential rates between arms mean treatment caused some respondents
#    to drop out or made them harder to locate, compromising follow-up
#    comparability even if randomisation was successful at baseline
tab = pd.crosstab(df['attrited'], df['treatment'])
print(tab, df.groupby('treatment')['attrited'].mean())
chi2, p, _, _ = stats.chi2_contingency(tab)
print('chi2=', chi2, 'p=', p)

# 2. Baseline balance among attriters by arm — running separately reveals
#    whether different baseline characteristics predict dropout in each arm;
#    if age predicts attrition in control but not treatment, the follow-up
#    samples have different age distributions even if rates are balanced
fit_ctrl = smf.ols('attrited ~ age + C(female) + hh_size + log_expenditure',
                 data=df[df.treatment == 0]).fit(cov_type='HC1')
fit_trt  = smf.ols('attrited ~ age + C(female) + hh_size + log_expenditure',
                 data=df[df.treatment == 1]).fit(cov_type='HC1')
print(fit_ctrl.summary(), fit_trt.summary())

# 3. Interaction model — directly tests whether attrition composition differs
#    across arms; significant treatment by covariate interactions are the
#    key signal that follow-up samples are no longer comparable on those
#    dimensions
fit_int = smf.ols(
  'attrited ~ treatment * (age + C(female) + hh_size + log_expenditure)',
  data=df).fit(cov_type='HC1')
inter_terms = [t for t in fit_int.model.exog_names if ':' in t]
print(fit_int.f_test(inter_terms))

# 4. Lee bounds — if differential attrition is confirmed, trim the lower-
#    attrition arm by the excess retention share from the top and bottom of
#    the outcome distribution; the resulting interval is the range of effects
#    consistent with any pattern of selective dropout. Python has no canonical
#    implementation; the function below implements the primitive for a binary
#    treatment, mirroring R's leebounds package
def lee_bounds(y, t, attrited):
  obs = (~attrited).astype(bool)
  p1 = obs[t == 1].mean(); p0 = obs[t == 0].mean()
  q  = (p1 - p0) / p1 if p1 > p0 else (p0 - p1) / p0
  high_arm, low_arm = (1, 0) if p1 > p0 else (0, 1)
  y_high = np.sort(y[(t == high_arm) & obs])
  n_trim = int(round(q * len(y_high)))
  lower = y_high[n_trim:].mean()  - y[(t == low_arm) & obs].mean()
  upper = y_high[:-n_trim].mean() - y[(t == low_arm) & obs].mean()
  return lower, upper
print(lee_bounds(df['outcome'].values, df['treatment'].values,
               df['attrited'].values))

library(dplyr)
library(estimatr)

# 1. Attrition rate by arm — establishes whether treatment is associated with
#    differential dropout; the first number to check before any other diagnostic
df %>%
group_by(treatment) %>%
summarise(attrition_rate = mean(attrited, na.rm = TRUE), n = n())

# 2. Chi-square test of differential attrition — formal test of whether the
#    attrition rate difference between arms is larger than chance; a significant
#    result (p < 0.05) with a difference above 5 pp is the threshold for concern
chisq.test(table(df$attrited, df$treatment))

# 3. Baseline balance among attriters, by arm — running these separately reveals
#    whether different baseline characteristics predict dropout in each arm;
#    if age predicts attrition in control but not treatment, the follow-up
#    samples have different age distributions even if rates are balanced
fit_ctrl <- lm_robust(attrited ~ age + female + hh_size + log_expenditure,
                    data = filter(df, treatment == 0), se_type = "HC2")
fit_trt  <- lm_robust(attrited ~ age + female + hh_size + log_expenditure,
                    data = filter(df, treatment == 1), se_type = "HC2")
summary(fit_ctrl); summary(fit_trt)

# 4. Interaction model — directly tests whether attrition composition differs
#    across arms; significant treatment×covariate interactions are the key
#    signal that follow-up samples are no longer comparable on those dimensions
fit_int <- lm_robust(attrited ~ treatment * (age + female + hh_size + log_expenditure),
                   data = df, se_type = "HC2")
summary(fit_int)

Reading the output

Overall attrition above 20% warrants careful investigation; differential attrition of more than 5 percentage points between arms is a red flag.
A significant chi-square test (p < 0.05) on attrited × treatment confirms differential attrition rates.
Significant interaction terms in the joint model indicate that the composition of dropout differs across arms — the retained samples may no longer be comparable on those dimensions.
Lee bounds: a narrow interval (width < 0.1 SD of the outcome) supports causal interpretation; a wide interval signals that attrition is a serious threat to validity.

References

Lee, D. S. (2009). Training, wages, and sample selection: Estimating sharp bounds on treatment effects. Review of Economic Studies, 76(3), 1071–1102. https://doi.org/10.1111/j.1467-937X.2009.00536.x

Glewwe, P., & Kassouf, A. L. (2012). The impact of the Bolsa Escola/Familia conditional cash transfer program on enrollment, dropout rates and grade promotion in Brazil. Journal of Development Economics, 97(2), 505–517. https://doi.org/10.1016/j.jdeveco.2011.05.008

Duflo, E., Glennerster, R., & Kremer, M. (2007). Using randomization in development economics research: A toolkit. In T. P. Schultz & J. Strauss (Eds.), Handbook of Development Economics, Vol. 4 (pp. 3895–3962). Elsevier.

Angrist, J. D., & Pischke, J.-S. (2009). Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton University Press.