Back-Check / Audit Comparator

A data quality audit that re-interviews a random sample of respondents shortly after the original interview, comparing responses to identify enumerator fabrication, systematic errors, and genuine measurement instability in survey variables.

What it is

A back-check (also called an audit, re-interview, or verification survey) re-contacts a random sample of respondents a few days after their original interview to ask a subset of the same questions. Comparing original and back-check responses reveals three types of problems: enumerator fabrication (the original interview was never conducted, or responses were filled in without asking), systematic enumerator errors (the enumerator misunderstands a question and consistently records it incorrectly), and genuine measurement instability (the variable is inherently noisy across repeated measurements, which is useful to know for analysis).

Back-checks are a standard quality assurance tool in large-scale surveys and a widely expected component of rigorous field data collection. They are most powerful when the back-checker is independent of the original enumerator, when the respondent has not been informed of the back-check, and when the questionnaire is designed to separate stable facts from opinions that might legitimately change.

When to use it

Back-checks should be built into the field protocol for any survey with more than 15–20 enumerators, where the risk of fabrication or systematic error is non-trivial. Typically 5–10% of interviews are back-checked, stratified by enumerator to ensure each enumerator is checked at least twice. Back-checks are most valuable during active data collection (when findings can inform retraining or dismissal) but also serve as post-hoc documentation of data quality for research outputs.

The back-check should be conducted within 3–7 days of the original interview for factual items (demographic characteristics, asset ownership, employment status) and within 1–3 days for recall-based items (expenditure in the last 7 days) to minimise genuine recall change.

How it works

Questionnaire design. The back-check questionnaire includes: (1) identification verifiers — questions that confirm the respondent was actually interviewed (name, household composition, landmarks); (2) stable factual items — characteristics unlikely to change (date of birth, education level, asset ownership of durable goods); and (3) a subset of key survey items whose back-check discrepancy rate will be used as a quality measure.

Discrepancy calculation. For each back-checked item and respondent, code a discrepancy indicator: 1 if the original and back-check responses differ beyond a tolerance, 0 if they agree. The tolerance should be defined in advance: for age, ±2 years is a common tolerance; for binary items, any disagreement is a discrepancy; for continuous items (income, land area), a percentage difference threshold (e.g., >10%) is appropriate.

Enumerator-level analysis. Compute the discrepancy rate by enumerator — the proportion of back-checked items that differ between original and back-check, across all back-checked respondents for that enumerator. Enumerators with discrepancy rates above 2–3 times the median warrant investigation. Very high discrepancy rates on identification items (e.g., the respondent’s name does not match) indicate fabrication.

Population-level discrepancy rates. Aggregate discrepancy rates across the full back-check sample identify which survey items are unstable — potentially due to poor question wording, recall problems, or legitimate change over the interview interval. High item-level discrepancy rates are a signal to examine the question design, not necessarily to flag enumerators.

Key decisions

Sampling strategy. Pure random sampling of interviews for back-checking is unbiased but may miss problems concentrated among specific enumerators with small workloads. Stratifying by enumerator — back-checking a fixed share (e.g., 2 interviews per enumerator) plus an additional random sample — ensures coverage of all enumerators while maintaining representativeness.

Who conducts back-checks. Back-checks conducted by supervisors (who oversee the original enumerators) are cheaper but may be subject to social pressure not to report problems. Independent back-check teams that report to a different chain are more credible. In large studies, a small dedicated back-check team should be considered.

Response to high discrepancy. When an enumerator’s back-check discrepancy rate is high, the appropriate response is: (1) review the specific discrepant items to assess whether they could be due to legitimate change; (2) manually inspect that enumerator’s full set of questionnaires for patterns; (3) consider whether the original interviews need to be replaced. Automatic exclusion of all data from high-discrepancy enumerators without investigation wastes data and may introduce selection bias.

Caveats & common mistakes

Asking about legitimately changeable items. Back-checks on opinions, intentions, or recent experience (food consumption in the last 7 days) will show high discrepancy rates even when the original interview was conducted perfectly. The back-check questionnaire should focus on stable facts. If recall variables are included, interpret discrepancy rates with awareness that genuine change contributes.

Confusing discrepancy with error. A discrepancy between original and back-check does not tell you which version is correct. The back-check may be the erroneous version, especially for items that depend on recall within a specific reference period (the back-check may be conducted outside that window). Adjudication rules — which response to use in the final dataset — should be specified in advance.

Not using back-checks in real time. Back-check data collected during data collection and only reviewed at the end of the field period miss the opportunity to retrain or remove problematic enumerators. Back-check results should be reviewed weekly during active collection.

Analysis Guide

import pandas as pd
import numpy as np
from sklearn.metrics import cohen_kappa_score

# 1. Merge original and back-check on respondent ID — inner merge keeps only
#    respondents present in both datasets; suffix labels distinguish the two
#    versions of each variable
df = original.merge(backcheck, on='respondent_id', how='inner',
                  suffixes=('_orig', '_bc'))

# 2. Discrepancy indicators per item — encode domain-appropriate tolerances:
#    plus or minus 2 years for age (inexact knowledge is common), zero
#    tolerance for gender (should never change), 10 percent for income
#    (recall-based values vary legitimately); pre-specifying tolerances
#    prevents post-hoc adjustment to hide problems
df['disc_age']    = (df['age_orig'] - df['age_bc']).abs() > 2
df['disc_gender'] = df['gender_orig'] != df['gender_bc']
df['disc_income'] = ((df['income_orig'] - df['income_bc']).abs()
                   / df['income_orig'].replace(0, np.nan) > 0.1)

# 3. Cohen's kappa on the binary gender item — agreement beyond chance is
#    the standard measure of categorical reliability; values below 0.6
#    indicate poor reliability and above 0.8 indicate substantial agreement
kappa = cohen_kappa_score(df['gender_orig'], df['gender_bc'])
print('Cohen kappa (gender):', round(kappa, 3))

# 4. Respondent-level discrepancy rate — averaging across items gives a
#    summary quality score per back-checked interview; high rates point to
#    a specific enumerator's workload for review
df['disc_rate'] = df[['disc_age', 'disc_gender', 'disc_income']].mean(axis=1)

# 5. Enumerator-level summary — enumerators whose average discrepancy rate
#    is more than twice the median are producing data that does not survive
#    independent verification; high disc_rate plus high discrepancy on
#    identification items (name, gender) indicates fabrication
enum_summary = (df.groupby('enum_id')
                .agg(mean_disc=('disc_rate', 'mean'),
                     n_checked=('disc_rate', 'size'))
                .sort_values('mean_disc', ascending=False))
median_disc = enum_summary['mean_disc'].median()
print(enum_summary[enum_summary['mean_disc'] > 2 * median_disc])

# 6. Item-level discrepancy rates — identifies which specific items are
#    unstable across the full back-check sample; high rates on stable facts
#    suggest question wording or translation problems, not enumerator error
print(df[['disc_age', 'disc_gender', 'disc_income']].mean())

library(dplyr)

# 1. Merge original and back-check — inner join ensures only respondents
#    present in both datasets are analysed; suffix labels keep the two
#    versions of each variable distinguishable
df_merged <- original %>%
inner_join(backcheck, by = "respondent_id", suffix = c("_orig", "_bc")) %>%
mutate(
  # 2. Discrepancy indicators with domain-appropriate tolerances — age
  #    allows ±2 years (common recall imprecision), gender is binary
  #    (any mismatch is suspicious), income uses a 10% relative threshold
  disc_age    = abs(age_orig - age_bc) > 2,
  disc_gender = gender_orig != gender_bc,
  disc_income = !is.na(income_orig) & income_orig > 0 &
                abs(income_orig - income_bc) / income_orig > 0.1,
  # 3. Respondent-level discrepancy rate — average across items; this
  #    summarises overall interview quality for the back-check sample
  disc_rate   = rowMeans(cbind(disc_age, disc_gender, disc_income), na.rm = TRUE)
)

# 4. Enumerator-level discrepancy — sort by mean discrepancy rate to
#    surface the enumerators whose data deviates most from verification;
#    prioritise those with both high discrepancy and sufficient sample (n ≥ 3)
enum_summary <- df_merged %>%
group_by(enum_id) %>%
summarise(mean_disc = mean(disc_rate, na.rm = TRUE), n_checked = n()) %>%
arrange(desc(mean_disc))
head(enum_summary, 10)

# 5. Item-level discrepancy rates — identifies structurally unstable items
#    that should be reviewed for question wording or translation problems;
#    distinguish from enumerator-driven discrepancy by comparing across items
colMeans(df_merged[, c("disc_age", "disc_gender", "disc_income")], na.rm = TRUE)

Reading the output

An enumerator’s discrepancy rate above 2–3 times the median across enumerators warrants manual review of their questionnaires.
Item-level discrepancy rates above 10% on stable factual items (date of birth, asset ownership of durables) suggest question wording or translation problems, not necessarily enumerator error.
Discrepancy on identification verifiers (respondent name, household size) above 5% is a strong fabrication signal — prioritise these cases for follow-up.
Very high discrepancy rates (>30%) on recall-based items (expenditure last 7 days) may reflect genuine change over the back-check interval rather than enumerator error; interpret with awareness of the timing gap.

References

Caeyers, B., Chalmers, N., & De Weerdt, J. (2012). Improving consumption measurement and other survey data through CAPI: Evidence from a randomized experiment. Journal of Development Economics, 98(1), 19–33. https://doi.org/10.1016/j.jdeveco.2011.09.004

Duflo, E., Glennerster, R., & Kremer, M. (2007). Using randomization in development economics research: A toolkit. In T. P. Schultz & J. Strauss (Eds.), Handbook of Development Economics, Vol. 4 (pp. 3895–3962). Elsevier.

Finn, A., & Ranchhod, V. (2017). Genuine fakes: The prevalence and implications of data fabrication in a large South African survey. World Bank Economic Review, 31(1), 1–16. https://doi.org/10.1093/wber/lhw024

Grosh, M. E., & Glewwe, P. (Eds.). (2000). Designing Household Survey Questionnaires for Developing Countries: Lessons from 15 Years of the Living Standards Measurement Study. World Bank.