What it is
Focus group discussions (FGDs) for survey validation convene small groups of 6–10 respondents drawn from the target population to work through a draft survey instrument together. Unlike cognitive interviews (which probe one person’s thought process), FGDs use group dynamics — discussion, disagreement, and clarification — to surface ambiguity. When respondents disagree about what a question means, or when they offer a response that the survey does not accommodate, the instrument has a problem.
FGDs in survey validation are not opinion research about the topic — they are a diagnostic tool for the instrument itself. The facilitator uses structured prompts to check whether each item is being interpreted as the researcher intended, whether the response categories are exhaustive, and whether the question order creates unintended effects.
When to use it
FGD-based validation should occur before the survey pilot, not as a substitute for it. The sequence is: instrument draft → FGD validation → revision → cognitive interviews (individual) → pilot → final revision → data collection. FGDs are most valuable when: the target population differs culturally or linguistically from the instrument’s origins; when adapting an instrument from another context; or when developing a new module without prior validation evidence.
How it works
Participant selection. Recruit 2–3 groups of 6–10 participants who resemble the full survey population on key dimensions (gender, age, wealth, education level). Same-gender groups are usually preferable for sensitive topics. Participants should not include individuals with prior survey experience from the same study.
Facilitation protocol. The facilitator reads each survey question aloud and asks the group:
- “What is this question asking?” (probes comprehension)
- “How would you answer this? Why?” (probes response strategy)
- “Is there any answer you might want to give that isn’t listed?” (probes completeness)
- “Does anything about this question feel strange or uncomfortable?” (probes cultural fit)
A note-taker records not only what respondents say but how — pauses, confusion, laughter, and visible disagreement are all data.
Systematic documentation. For each item, record: the intended interpretation, the group’s interpretation(s), whether interpretations matched, specific language or concepts that caused confusion, and the suggested revision. Items with discrepancies between intended and actual interpretation must be revised.
Red flags. Items that trigger: (1) multiple different interpretations across respondents; (2) consistent misinterpretation across groups; (3) no acceptable response within the provided categories; or (4) visible discomfort or refusal to engage — should be flagged for major revision, not minor wording tweaks.
Key decisions
Module prioritisation. Running FGDs across a 60-item questionnaire in its entirety is unwieldy. Prioritise modules with: newly developed items (no prior validation); sensitive topics (income, sexual behaviour, domestic violence); complex recall tasks (expenditure in the last 30 days); and any module adapted from a different language or context.
Language. FGDs should be conducted in the respondent’s first language, not the language of the questionnaire. If the final instrument will be administered in a translated version, the FGD should use the translated version, not the source language.
Recording vs. note-taking. Audio or video recording allows full review of the session but may reduce candour on sensitive topics. Trained note-takers with a structured template (item-by-item) are an acceptable alternative. Detailed notes taken during the session are far preferable to reconstruction from memory afterwards.
Caveats & common mistakes
Treating FGD agreement as validation. A group that unanimously agrees on a question’s interpretation does not mean the interpretation is the intended one. Always compare the group’s stated interpretation against the researcher’s intended interpretation. Consensus on the wrong meaning is not validation.
Skipping validation for “standard” instruments. Validated instruments (PHQ-9, WHODAS, USAID wealth indices) have been validated in specific contexts, often high-income settings or specific countries. Re-validation in a new context — especially a different language or socioeconomic setting — is not optional. It is frequently omitted and frequently produces problems.
Using the same FGD participants for the survey pilot. Participants who have discussed the instrument in an FGD have been primed to interpret questions in specific ways. They should not participate in the pilot or final survey. Maintain a separate pool for each phase.
Analysis Guide
There is no statistical code for FGD facilitation. The output of FGD validation is a structured revision log — a table with columns: item number, intended interpretation, observed interpretation(s), discrepancy (yes/no), proposed revision. This table drives instrument revision.
A minimal SurveyCTO/ODK form to track validation observations:
survey
type | name | label
-----------+------------------+------
text | item_number | Survey item number
text | intended_meaning | Intended interpretation (pre-filled)
text | observed_meaning | What respondents said this question means
select_one yes_no | discrepancy | Was there a discrepancy?
text | proposed_revision| Suggested revision
text | notes | Facilitator notesThe following snippet counts discrepancies by module once the validation log is in a tabular format:
import pandas as pd
# validation_log: DataFrame with one row per survey item
# vars: item_number, module, discrepancy (0/1), observed_meaning, proposed_revision
# 1. Discrepancy rate by module: row percentages show what proportion of each
# module's items had interpretation mismatches — modules above 20% discrepancy
# need substantive revision before piloting, not just wording tweaks
pd.crosstab(validation_log['module'], validation_log['discrepancy'], normalize='index') * 100
# 2. List items flagged for revision: reviewing these item-by-item (not just counting)
# is the actual work of FGD analysis — each discrepant item needs a specific revision
# based on the observed interpretation recorded in the validation log
validation_log.loc[validation_log['discrepancy'] == 1,
['item_number', 'module', 'observed_meaning', 'proposed_revision']]Reading the output
- A discrepancy rate above 20% within any module indicates the module needs substantial revision before piloting, not minor wording tweaks.
- Items where all groups produced the same wrong interpretation (unanimous misunderstanding) require complete rewrites — not just synonym substitution.
- Items with a discrepancy rate of 0% across all groups can proceed to cognitive interview testing; do not skip that step even for items that appear unambiguous.
- If a module has zero discrepancies but low response engagement (many “I don’t know” or blank answers in the tracker), recheck whether the facilitation prompts were applied to those items.
References
Krueger, R. A., & Casey, M. A. (2015). Focus Groups: A Practical Guide for Applied Research (5th ed.). Sage Publications.
Willis, G. B. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Sage Publications.
Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71(2), 287–311. https://doi.org/10.1093/poq/nfm006
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The Psychology of Survey Response. Cambridge University Press.