Survey Duration Outlier Detector

A diagnostic for detecting survey interviews that are too short to have been completed honestly or too long to represent normal enumeration — using paradata (timestamps, duration logs) to flag potentially problematic interviews for review.

What it is

Survey duration paradata — timestamps and elapsed time recorded by CAPI systems like SurveyCTO — provide a direct window into interview quality that respondent data cannot. An interview completed in 8 minutes that was designed to take 45 minutes almost certainly contains fabricated responses, skipped modules, or careless rushing. An interview lasting 3 hours for a questionnaire designed for 90 minutes may indicate genuine complexity, but may also reflect an enumerator who paused the tablet, left the respondent unattended, or re-administered sections after losing focus.

Duration outlier detection uses this paradata to flag problematic interviews before they contaminate analysis. It is most powerful when combined with enumerator fixed effects checks and backcheck comparisons, as duration outliers often cluster by enumerator.

When to use it

Duration analysis should be part of standard data monitoring during active data collection — preferably reviewed daily or weekly so that problems can be addressed before too many compromised interviews accumulate. Post-collection, duration data provide a basis for sensitivity analysis: does excluding very short interviews change the estimated treatment effect?

The shift to CAPI (tablet-based) data collection has made this diagnostic standard. SurveyCTO, ODK, and Kobo all record start and end times for the survey and, depending on configuration, for individual sections. Field managers who lack experience with paradata analysis often miss this information entirely, which is one reason enumerator fabrication persists even in well-funded evaluations.

How it works

Session-level duration analysis:

Compute duration as end time minus start time (in minutes).
Calculate the median duration and interquartile range across all completed surveys.
Flag surveys with duration below a threshold (typically the 5th percentile or a design-based minimum — the shortest plausible honest completion time, based on piloting).
Flag surveys above an upper threshold (typically the 95th percentile or a design-based maximum).

Design-based thresholds. The most defensible thresholds come from piloting: record duration in cognitive interviews and field pilots, and set the minimum threshold at approximately 60–70% of the pilot median. An interview substantially shorter than any pilot observation almost certainly contains skipped or pre-filled responses.

Section-level analysis. Many CAPI systems record timestamps at the module level. Section-level duration checks are more precise: an interview may have a normal total duration but spend only 30 seconds on a complex 10-item battery, indicating that the enumerator may have filled it quickly without asking the questions.

Enumerator-level patterns. Average duration by enumerator, and the share of interviews flagged as too short by enumerator, are diagnostic of systemic behaviour. An enumerator with 30% of surveys in the short-duration tail is a different problem than one where 2% of surveys are short and the remainder are normal.

Key decisions

Single-threshold vs. distribution-based flagging. A design-based minimum (e.g., 20 minutes for a 45-minute survey) is simpler and more transparent. A statistical threshold based on the observed distribution (e.g., 2 SDs below the mean) is more adaptive but will flag some percentage of surveys in any dataset regardless of true quality. Combining both — flagging surveys below both the design minimum and 2 SDs below the group mean — reduces false positives.

Pause time. Some CAPI systems record active screen time (excluding periods when the device was paused or locked) and total elapsed time separately. Active screen time is a better proxy for genuine completion time than total elapsed time, which inflates for interviews interrupted by a break, a phone call, or the enumerator temporarily leaving. SurveyCTO provides both measures; use active time if available.

Response to flags. Short-duration surveys should be reviewed manually before any action is taken. The supervisor should compare the flagged survey with a typical interview from the same questionnaire section by section. If a 12-minute interview shows complete, internally consistent data on all sections, it may be genuine (an unusually articulate respondent). If it shows identical responses to a previous interview from the same enumerator, it is fabricated. Manual review of flagged surveys is labour-intensive but necessary for accurate adjudication.

Caveats & common mistakes

Duration as a noisy signal. Duration measures interview speed, not interview quality. Some enumerators are faster without being less thorough. Some questionnaires can be completed more quickly by certain respondent types. Duration outliers are flags, not verdicts — they require follow-up investigation.

Not checking section duration. Reviewing only overall duration misses cases where the total is normal but one section was rushed. Configuring the CAPI instrument to log timestamps at key section breaks is a cheap design decision with high diagnostic value. If a survey was designed without section timestamps, this cannot be recovered post-collection.

Threshold drift. As a study progresses, enumerators often speed up as they become familiar with the questionnaire. Duration distributions shift downward over time. Computing thresholds separately by survey wave or survey month — rather than applying a single threshold across the full dataset — reduces false positives due to learning effects.

Analysis Guide

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# duration: survey duration in minutes; enum_id: enumerator identifier

# 1. Describe the duration distribution — before flagging, understand the
#    central tendency and spread; the 5th and 95th percentiles become the
#    statistical thresholds for short and long outliers
print(df['duration'].describe(percentiles=[0.05, 0.5, 0.95]))
p5, p95 = df['duration'].quantile([0.05, 0.95])

# 2. Flag short and long interviews by distribution — relative flags that
#    identify the tails regardless of design; useful when no pilot minimum
#    is available, but will always flag some fixed share of the sample
df['short_survey'] = df['duration'] < p5
df['long_survey']  = df['duration'] > p95

# 3. Design-based minimum — a survey cannot be completed honestly in less
#    time than it takes to read and respond to every item; set this from
#    piloting and flag anything shorter as almost certainly fabricated or
#    severely rushed
design_min = 20   # 20 minutes minimum for this survey
df['too_short'] = df['duration'] < design_min

# 4. Enumerator-level duration summary — distinguishes one-off short interviews
#    (random variation) from a pattern of short surveys (systematic behaviour);
#    only meaningful for enumerators with 10 or more interviews
enum_dur = (df.groupby('enum_id')
            .agg(median_dur=('duration', 'median'),
                 pct_short=('too_short', 'mean'),
                 n_completed=('duration', 'size'))
            .sort_values('pct_short', ascending=False))

# 5. Flag enumerators with high share of short surveys — more than 15 percent
#    of interviews below the design minimum across at least 10 completed
#    interviews is almost certainly rushing or fabricating, not chance
enum_dur['flag'] = (enum_dur['pct_short'] > 0.15) & (enum_dur['n_completed'] >= 10)
print(enum_dur[enum_dur['flag']])

# 6. Duration trend over time — a downward trend is expected as enumerators
#    learn the questionnaire; a trend that drops more than 30 to 40 percent
#    from the first week to mid-study suggests shortcuts, not efficiency gains
df['survey_date'] = pd.to_datetime(df['survey_date'])
daily_med = df.groupby('survey_date')['duration'].median()
daily_med.plot(title='Median interview duration over time'); plt.show()

# duration: interview duration in minutes

# 1. Describe the duration distribution — establishes the baseline before
#    flagging; histogram reveals whether the distribution is unimodal or
#    has a suspicious spike at very short durations
summary(df$duration)
hist(df$duration, breaks = 30, main = "Survey duration distribution", xlab = "Minutes")

# 2. Set thresholds — use the stricter of the statistical 5th percentile and
#    the design-based minimum; the design minimum is preferable when piloting
#    established a plausible lower bound for honest completion
p5         <- quantile(df$duration, 0.05, na.rm = TRUE)
p95        <- quantile(df$duration, 0.95, na.rm = TRUE)
design_min <- 20   # adjust to actual survey design

df$too_short <- df$duration < max(p5, design_min)
df$too_long  <- df$duration > p95
cat("Short surveys:", sum(df$too_short, na.rm = TRUE),
  " | Long surveys:", sum(df$too_long, na.rm = TRUE), "
")

# 3. By enumerator — identifies whether short interviews are random across
#    the team or concentrated in specific enumerators; concentration is a
#    systemic signal, dispersion suggests occasional legitimate fast responses
library(dplyr)
enum_dur <- df %>%
group_by(enum_id) %>%
summarise(median_duration = median(duration, na.rm = TRUE),
          pct_short = mean(too_short, na.rm = TRUE),
          n = n()) %>%
arrange(desc(pct_short))
head(enum_dur, 10)

# 4. Duration trend over time — declining medians are expected due to learning;
#    plot this to distinguish learning-curve compression from progressive
#    shortcutting, which would show a steeper or ongoing decline
df$survey_date <- as.Date(df$survey_date)
daily_med <- df %>% group_by(survey_date) %>%
summarise(med_dur = median(duration, na.rm = TRUE))
plot(daily_med$survey_date, daily_med$med_dur, type = "l",
   main = "Median interview duration over time")

Reading the output

Surveys below the design-based minimum (e.g., shorter than 60–70% of the pilot median) are the highest-priority flags; a survey cannot be completed honestly in that time. The 5th percentile cutoff is a useful secondary check when no design minimum was established.
An enumerator with more than 15% of interviews flagged as short, across at least 10 completed interviews, is a systemic concern — the short durations are unlikely to be coincidental.
A downward trend in median duration over the survey period is expected as enumerators gain familiarity with the questionnaire. A trend that drops more than 30–40% from the first week to mid-study suggests shortcuts are being taken, not just efficiency gains.
Very long surveys (above the 95th percentile) may reflect genuine complexity, pauses in data collection (tablet locked mid-interview), or a respondent who required extra attention — review the section-level timestamps if available to determine which module was slow.
Duration flags alone are not grounds for exclusion; they are signals for manual review. If a short-duration interview shows internally consistent data with no duplicate response patterns, it may be genuine.

References

Couper, M. P. (2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64(4), 464–494. https://doi.org/10.1086/318641

Höhne, J. K., & Lenzner, T. (2018). New perspective on the cognitive burden of survey questions: Insights from response times. Sociological Methods & Research, 47(4), 850–880. https://doi.org/10.1177/0049124116643678

Olson, K., & Parkhurst, B. (2013). Collecting paradata for measurement error evaluations. In F. Kreuter (Ed.), Improving Surveys with Paradata: Analytic Uses of Process Information (pp. 43–72). Wiley.