Logical Consistency Checker

A validation routine that identifies internally inconsistent responses — answers that cannot simultaneously be true — by encoding the logical relationships between survey items and checking each observation against them.

What it is

Logical consistency checks (also called hard edits or constraint violations) test whether responses within the same questionnaire are mutually compatible. Inconsistent responses arise from enumerator transcription errors, skip logic failures, respondent confusion, fabrication, or data entry mistakes. Common examples: a respondent’s reported age of birth is inconsistent with their stated age; a household lists fewer adults than the number of adult household members employed; a female respondent reports being asked about prostate health; a household with no land reports cultivating crops.

Unlike statistical outliers (which are extreme but possibly genuine), logical inconsistencies cannot all be true simultaneously — at least one response in the inconsistent pair must be wrong. Flagging them during data collection enables correction before the data leaves the field; detecting them post-collection requires adjudication rules to decide which response to trust.

When to use it

Logical consistency checks should be implemented at two stages. The first and preferred stage is in the CAPI instrument itself: SurveyCTO, ODK, and Kobo all support constraint syntax that rejects logically impossible entries at the point of data collection. The second stage is during data cleaning, where inconsistencies not caught by instrument constraints are identified programmatically.

Every survey should have a pre-specified list of hard constraints before data collection begins. These are not optional quality checks — they define the logical space of valid responses for the instrument.

How it works

Constraint categories. Logical constraints fall into several types:

Range constraints: values must fall within a plausible range (age 0–120, household size 1–30).
Cross-item constraints: item A implies a restriction on item B (if gender = male, skip female-specific modules).
Derived variable constraints: a computed value must equal the stated value (sum of individual incomes ≈ household income; number of children born minus deaths ≤ living children).
Temporal constraints: reported dates must be internally consistent (date of birth before date of interview; date of marriage after age 15).
Skip logic constraints: items that should not be answered (due to skip logic) must be missing or coded as inapplicable.

Implementation. For each constraint, create a binary indicator (1 = inconsistent, 0 = consistent). Sum across all constraints per observation to produce a total inconsistency count. Report the inconsistency rate per constraint (identifying which checks fail most often) and per observation (identifying respondents or enumerators with many errors).

Adjudication rules. When inconsistencies are found post-collection, the team must decide which response to trust. Common rules: (1) prefer the response that was asked earlier in the questionnaire; (2) prefer demographic roster responses over later module responses; (3) prefer the response that is more specific (an exact value over a range). Rules should be documented in the data cleaning protocol.

Key decisions

Soft vs. hard constraints. Hard constraints should produce errors that block form submission (impossible values: age = −5). Soft constraints should produce warnings that the enumerator can override after confirmation (unusual but possible: household size = 25, which is large but plausible in extended family compounds). Calibrating hard vs. soft thresholds requires field knowledge about the population.

Tolerance ranges for derived constraints. Derived variable checks often need a tolerance margin. Household expenditure derived from categories will rarely exactly equal total self-reported expenditure due to recall and categorisation differences. A 10% tolerance band is a reasonable default; tighter or looser tolerances should be set based on the instrument’s precision.

Documenting adjudication. Every inconsistency that is corrected during cleaning should be logged with: the observation ID, the inconsistent items, the rule applied, and the final value used. This log is part of the data provenance record and should be archived with the dataset.

Caveats & common mistakes

Skip logic failures as inconsistencies. Many apparent logical inconsistencies are actually skip logic failures: an item that should have been skipped was answered, or an item that should have been answered was skipped. These should be diagnosed separately from genuine content inconsistencies, as they often reflect instrument programming errors rather than enumerator or respondent mistakes.

Not building checks into the instrument. Post-collection logical checks are a fallback, not a substitute for CAPI constraint syntax. An enumerator who cannot advance past a constraint violation will correct the error in the field, when the respondent is still present. Post-collection adjudication based on a data cleaning rule is a much weaker fix.

Treating all inconsistencies as equally serious. An age-year of birth inconsistency of 1 year in a setting where exact birth dates are unknown is very different from an age inconsistency of 20 years. Severity coding — distinguishing minor discrepancies that fall within measurement tolerance from major logical impossibilities — makes the cleaning log more useful.

Analysis Guide

import pandas as pd
import numpy as np

# 1. Range constraints: flag values outside biologically/operationally plausible
#    bounds — hard logical errors that cannot be true regardless of context
df["flag_age"]    = ~df["age"].between(0, 120)
df["flag_hhsize"] = ~df["hh_size"].between(1, 50)

# 2. Cross-item check: age derived from birth year should match self-reported age
#    within a tolerance; a 2-year gap allows for reporting on different reference dates
df["calc_age"]     = pd.to_datetime(df["interview_date"]).dt.year - df["birth_year"]
df["flag_age_dob"] = (df[["age", "birth_year"]].notna().all(axis=1)
                    & (df["age"] - df["calc_age"]).abs().gt(2))

# 3. Derived constraint: living children cannot exceed children ever born —
#    a definitional impossibility, so any violation must be a coding or entry error
df["flag_children"] = (df[["living_children", "children_born"]].notna().all(axis=1)
                     & (df["living_children"] > df["children_born"]))

# 4. Skip logic: female-only module answered by a male respondent indicates
#    either a gender coding error or an instrument routing failure
df["flag_skip"] = df["female_module_item"].notna() & df["gender"].eq(1)

# 5. Composite count per observation: aggregating flags surfaces multi-error
#    forms; tabulating reveals the distribution of error burden across the sample
flag_cols = ["flag_age", "flag_hhsize", "flag_age_dob", "flag_children", "flag_skip"]
df["n_flags"] = df[flag_cols].sum(axis=1)
print(df["n_flags"].value_counts().sort_index())

# 6. By enumerator: averaging flags per enumerator surfaces individuals whose
#    forms are systematically failing checks — a signal to retrain or audit
enum_summary = df.groupby("enum_id")["n_flags"].agg(["mean", "size"])
print(enum_summary[enum_summary["mean"] > 0.5])

library(dplyr)
library(lubridate)

df <- df %>%
mutate(
  # Range constraints
  flag_age    = !between(age, 0, 120),
  flag_hhsize = !between(hh_size, 1, 50),

  # Cross-item: age vs. birth year
  calc_age     = year(interview_date) - birth_year,
  flag_age_dob = !is.na(age) & !is.na(birth_year) & abs(age - calc_age) > 2,

  # Derived: living children
  flag_children = !is.na(living_children) & !is.na(children_born) &
                  living_children > children_born,

  # Skip logic
  flag_skip = !is.na(female_module_item) & gender == 1,

  # Composite
  n_flags = rowSums(cbind(flag_age, flag_hhsize, flag_age_dob,
                          flag_children, flag_skip), na.rm = TRUE)
)

# Inconsistency rates per check
flag_cols <- c("flag_age", "flag_hhsize", "flag_age_dob", "flag_children", "flag_skip")
colMeans(df[, flag_cols], na.rm = TRUE)

# By enumerator
df %>%
group_by(enum_id) %>%
summarise(mean_flags = mean(n_flags, na.rm = TRUE), n = n()) %>%
arrange(desc(mean_flags))

Reading the output

Each flag_* column is a binary indicator; the item-level rate is the proportion of observations that fail that specific constraint — rates above 1–2% on hard logical constraints (living children > born, age outside 0–120) suggest instrument, training, or data entry problems.
n_flags > 0 across the full sample should be tracked over the data collection period; a rising rate signals a systematic problem that needs intervention.
An enumerator with mean_flags above 0.5 (on average at least one inconsistency per interview) has elevated error rates and warrants closer supervision or review of their questionnaires.
Distinguish skip logic failures (flag_skip) from content inconsistencies — skip logic failures often point to instrument programming errors, which may affect many enumerators uniformly rather than clustering on specific individuals.

References

Grosh, M. E., & Glewwe, P. (Eds.). (2000). Designing Household Survey Questionnaires for Developing Countries: Lessons from 15 Years of the Living Standards Measurement Study. World Bank.

Thissen, D., & Wainer, H. (Eds.). (2001). Test Scoring. Lawrence Erlbaum Associates.

SurveyCTO. (2024). Constraints and Relevance in SurveyCTO. SurveyCTO Documentation. https://docs.surveycto.com

Hammer, J., & Jack, W. (2015). The design of incentives for health and development workers: Insights from India. Journal of Development Economics, 118, 159–170.