What it is
Qualitative coding for instrument development is the process of systematically analysing qualitative data (interview transcripts, FGD notes, ethnographic observations) to identify the concepts, dimensions, and language that respondents use to describe the phenomena the survey will measure. Rather than imposing researcher-defined constructs on respondents, this approach derives items from the ground up — grounding the survey in the conceptual vocabulary of the target population.
The method is particularly important for constructs that are culturally specific (well-being, trust, respect, dignity) or that vary substantially across contexts (household decision-making, financial management). A scale developed in one country may use concepts that have no equivalent in another. Qualitative coding surfaces these local constructs before the instrument is locked.
When to use it
Qualitative coding for item development is appropriate when: developing a new survey module for a context where validated instruments do not exist; adapting a validated instrument to a substantially different cultural context; or trying to understand why pilot data from an existing instrument show low variance or unexpected patterns. It precedes the formal survey validation process (cognitive interviews, FGDs for validation testing, pilot testing).
How it works
Data collection. Conduct 8–15 open-ended interviews or 3–5 FGDs with target population members. Interviews should use grand-tour questions (“Tell me about how households in this village think about managing money”) and probes that follow the respondent’s own language and frames, rather than imposing the researcher’s categories.
Transcript preparation. Transcribe or translate recordings into text. Translation of qualitative data requires bilingual research staff who can retain nuance — direct word-for-word translation misses idiom and metaphor. Decisions about back-translation of direct quotes used in publications should be documented.
Inductive coding. Read transcripts without a pre-existing code list. Mark passages that relate to the construct of interest. Assign a short label to each marked passage (the code). As you work through transcripts, new codes are added and existing codes are refined. Initial coding is open and exploratory.
Code consolidation. After all transcripts are coded, group similar codes into higher-order themes. This produces a codebook — a structured list of themes, sub-themes, and their definitions. The codebook organises the conceptual space of the construct as respondents understand it.
Item generation. For each theme and sub-theme in the codebook, draft one or more survey items using respondent language. Check that: (1) the items cover all major dimensions of the construct; (2) the response categories accommodate the full range of responses observed in the qualitative data; and (3) the question wording uses terms respondents used, not researcher jargon.
Saturation check. Qualitative coding should continue until new transcripts produce no new codes (conceptual saturation). In practice, saturation is often reached within 10–15 interviews for a focused construct in a relatively homogeneous population.
Key decisions
Deductive vs. inductive coding. Inductive coding (no prior code list) is appropriate for new constructs or contexts. Deductive coding (applying a pre-specified code list from theory or prior research) is appropriate when adapting a well-theorised instrument to a new context. Mixed approaches — a starter code list from theory, extended with inductive codes from the data — are most common.
Unit of analysis. Codes can be applied at the sentence, paragraph, or thematic passage level. Finer-grained coding produces more data but requires more judgment calls and is harder to achieve consistent inter-rater reliability. For item development purposes, passage-level coding is usually sufficient.
Number of coders. Single-coder analysis is common in qualitative research but produces no check on reliability or bias. Using two coders and computing inter-rater reliability on a subset of transcripts (Cohen’s kappa > 0.6 is a common standard) is best practice for research that will be published.
Caveats & common mistakes
Treating qualitative coding as pre-analysis and then ignoring it. Qualitative coding that generates a codebook that then gets filed away — and items are developed without reference to it — defeats the purpose. The codebook should directly drive item writing, with explicit links between each item and the qualitative theme it operationalises.
Conflating local language with universal constructs. Respondents may use a local term that appears to map onto a universal construct but carries distinct cultural meaning. Coding that simply matches local terms to pre-existing constructs without examining the local meaning structure produces superficially grounded items that miss the point.
Not documenting the item derivation. Publication of a survey instrument should include a methods appendix tracing each scale item to the qualitative coding from which it was derived. Reviewers and future users cannot evaluate construct validity without this chain of evidence.
Analysis Guide
Qualitative coding is typically done in software such as ATLAS.ti, NVivo, or Dedoose. Free alternatives include MAXQDA (free version for students) and manual coding with a structured spreadsheet.
A minimal structured coding spreadsheet:
Columns:
- transcript_id
- participant_id
- passage (verbatim text)
- first_level_code
- second_level_code (theme)
- memo (analytical note about why this code was applied)
- survey_item_candidate (draft item derived from this passage)Once the coding log is in a tabular format, use the following to summarise code frequency and check conceptual saturation:
import pandas as pd
from sklearn.metrics import cohen_kappa_score
# code_df: DataFrame with one row per coded passage
# vars: transcript_id, participant_id, first_level_code, second_level_code
# 1. Code frequency by theme — sorted descending shows dominant themes;
# themes with fewer than 3 passages are peripheral and weak item candidates
code_df['second_level_code'].value_counts()
# 2. Coverage per participant — checks whether some respondents are over- or
# under-represented in the coded corpus, which biases item derivation
code_df.groupby('participant_id').agg(
n_passages=('second_level_code', 'size'),
n_themes=('second_level_code', 'nunique'))
# 3. New codes introduced per transcript (saturation check) — once new codes
# drop to 0-1 per transcript, conceptual saturation has been reached
seen = set()
new_per_transcript = {}
for tid, codes in code_df.groupby('transcript_id')['second_level_code']:
new_per_transcript[tid] = len(set(codes) - seen)
seen.update(codes)
pd.Series(new_per_transcript, name='new_codes')
# 4. Inter-coder reliability on a subset double-coded by two coders
# (coder_a, coder_b columns); kappa below 0.60 flags ambiguous definitions
cohen_kappa_score(code_df['coder_a'], code_df['coder_b'])Reading the output
- Themes with fewer than 3 coded passages across all transcripts are likely peripheral; do not generate survey items from them unless domain theory requires it.
- If the number of new codes introduced per transcript drops to 0–1 after transcript 10, conceptual saturation has been reached and additional interviews are unlikely to add new dimensions.
- A participant with zero codes for one or more major themes may indicate the interviewer did not probe that domain sufficiently with that respondent — not that the respondent had nothing to say.
- Cohen’s kappa below 0.60 on the inter-rater reliability check signals that code definitions are too ambiguous; revise inclusion and exclusion criteria before proceeding.
References
Graneheim, U. H., & Lundman, B. (2004). Qualitative content analysis in nursing research: Concepts, procedures and measures to achieve trustworthiness. Nurse Education Today, 24(2), 105–112. https://doi.org/10.1016/j.nedt.2003.10.001
Strauss, A., & Corbin, J. (1998). Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (2nd ed.). Sage Publications.
Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), 59–82. https://doi.org/10.1177/1525822X05279903
Willis, G. B. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Sage Publications.