What it is
Enumerator fixed effects (EFE) analysis tests whether the responses collected by a specific enumerator are systematically different from those collected by other enumerators, after controlling for observable differences in respondent characteristics. If one enumerator consistently records higher income levels, shorter interviews, or more “yes” responses than colleagues interviewing similar respondents, the difference is likely attributable to enumerator behaviour: leading respondents, filling in responses without asking, rounding to convenient values, or outright fabrication.
Enumerator effects are one of the most consequential and least discussed sources of measurement error in field surveys. They are systematic (not random), can be correlated with outcomes of interest, and can invalidate inference if enumerators are not randomly assigned to respondents. The EFE check is a first-pass diagnostic that flags potential problems for further investigation.
When to use it
EFE checks should be run routinely as part of data cleaning and monitoring — not only when problems are suspected. They are most valuable during data collection (when enumerator behaviour can still be corrected) but are also informative post-collection for understanding the reliability of estimated effects.
EFE analysis is particularly important when: enumerators were not randomly assigned to respondents (leaving open the possibility that respondent-enumerator matching is confounded with outcomes); when the survey involves sensitive questions where enumerators may suppress or recode responses; or when previous waves have shown evidence of enumerator-level heterogeneity.
Caeyers, Chalmers and De Weerdt (2012) document fabrication and back-coding by enumerators in a Tanzanian panel survey.
How it works
The basic test runs an ANOVA or F-test of whether enumerator identity explains significant variation in a key outcome variable, after controlling for observable respondent characteristics. If the F-statistic is significant, some enumerators are producing systematically different values.
The regression approach includes enumerator fixed effects (one indicator per enumerator) as controls in a regression of the outcome on respondent characteristics. The F-test on the joint significance of the enumerator indicators tests whether enumerators differ. The size and sign of individual enumerator coefficients identify which enumerators are high or low outliers.
What to check. The most informative variables for EFE analysis are:
- Continuous outcomes with scope for rounding or guessing (income, expenditure, quantity measures).
- Sensitive questions where social desirability may lead enumerators to recode (violence, income, asset values).
- Survey duration — systematic outliers in interview length are a common fraud signal.
- Item non-response rates — enumerators who skip many items may be cutting corners.
- Internal consistency checks — if certain enumerators have higher rates of logically inconsistent responses, this is a red flag.
Key decisions
Random vs. non-random assignment. If enumerators were randomly assigned to respondents (a strong field protocol practice), enumerator fixed effects can be included as controls in outcome regressions to reduce noise without introducing bias. If assignment was non-random — enumerators were assigned to geographic clusters that may correlate with outcomes — including EFE as controls risks controlling away real variation. The assignment protocol must be understood before the analysis is interpreted.
Joint F-test vs. individual outlier detection. The F-test of joint significance identifies whether any systematic enumerator variation exists. Individual coefficient inspection identifies which enumerators are problematic. Both are informative: a significant F-test should be followed by inspection of individual coefficients, comparison of response distributions, and review of the specific enumerator’s forms.
Threshold for flagging. There is no universal threshold. Practical guidance: flag enumerators whose residualised outcomes are more than 2–2.5 standard deviations from the group mean; flag enumerators with the shortest or longest average interview durations; flag enumerators with non-response rates more than double the median. All flagged enumerators should be investigated rather than automatically excluded.
Caveats & common mistakes
Enumerator effects vs. area effects. If enumerators are assigned to geographic areas, apparent enumerator effects may reflect genuine area-level differences rather than enumerator behaviour. Disentangling these requires enumerators to interview respondents from multiple areas (cross-assignment), which is the best field protocol for this reason. Without cross-assignment, area effects and enumerator effects are confounded.
Small enumerator workloads. EFE estimates are noisy for enumerators who completed only 10–20 interviews. Differences from the group mean may reflect sampling variation rather than systematic behaviour. Flagging should be based on both the size of the deviation and the number of interviews completed.
Not excluding automatically. Finding that a specific enumerator’s data is flagged does not automatically mean their data should be excluded. Exclusion must be based on verified evidence of misconduct or error, not statistical suspicion alone. Excluding enumerator data post-hoc based on statistical outliers introduces selection bias. The appropriate response to a flag is investigation: review paper forms, conduct back-checks with a sample of that enumerator’s respondents, or re-interview a subset.
Analysis Guide
import pandas as pd
import numpy as np
import statsmodels.formula.api as smf
# income: continuous outcome; enum_id: enumerator identifier
# 1. Joint F-test — fit a model with respondent characteristics and enumerator
# indicators, then jointly test the enumerator dummies; a significant F
# means enumerators explain outcome variation beyond what respondent
# characteristics predict, pointing to enumerator-level bias
fit = smf.ols('income ~ age + C(female) + hh_size + C(region) + C(enum_id)',
data=df).fit(cov_type='HC1')
enum_terms = [t for t in fit.model.exog_names if t.startswith('C(enum_id)')]
print(fit.f_test(enum_terms))
# 2. Outlier enumerators via mean residuals — fit the model WITHOUT enumerator
# FE so residuals isolate the component of each response unexplained by
# respondent characteristics; averaging by enumerator reveals systematic
# directional bias from a specific collector
fit_no = smf.ols('income ~ age + C(female) + hh_size + C(region)',
data=df).fit(cov_type='HC1')
df['resid'] = fit_no.resid
enum_summary = df.groupby('enum_id').agg(mean_resid=('resid', 'mean'),
n_interviews=('resid', 'size'))
enum_summary['flag'] = enum_summary['mean_resid'].abs() > 2 * df['resid'].std()
print(enum_summary[enum_summary['flag']])
# 3. Survey duration check — short interviews relative to peers are the most
# sensitive fabrication signal because filling in answers without asking
# is almost always faster than genuine enumeration; 60 percent of the
# enumerator group median is a practical lower bound for honest completion
group_median = df['survey_duration'].median()
df['short'] = df['survey_duration'] < 0.6 * group_median
dur_summary = df.groupby('enum_id').agg(short_rate=('short', 'mean'),
n=('short', 'size'))
print(dur_summary[dur_summary['short_rate'] > 0.2])Reading the output
- A significant F-test on
testparm i.enum_id(p < 0.05) indicates that enumerators are producing systematically different values after controlling for respondent characteristics — the variation is attributable to the enumerators themselves, not the respondents they interviewed. - Enumerators with mean residuals more than 2 SDs from zero are outliers: they are consistently collecting values that are higher or lower than expected given their respondents’ observable characteristics.
- An enumerator with more than 20% of interviews flagged as short (below 60% of the enumerator group’s median duration) warrants manual review of their forms.
- The number of interviews per enumerator matters: a large mean residual based on only 10–15 interviews may reflect sampling variation. Prioritise investigating enumerators with both a large mean residual and a substantial workload (30+ interviews).
- Enumerator effects that cannot be explained by area or respondent composition should be followed up with back-checks on a sample of that enumerator’s interviews before any data exclusion decisions are made.
References
Caeyers, B., Chalmers, N., & De Weerdt, J. (2012). Improving consumption measurement and other survey data through CAPI: Evidence from a randomized experiment. Journal of Development Economics, 98(1), 19–33. https://doi.org/10.1016/j.jdeveco.2011.09.004