What it is
Natural language processing (NLP) applies computational methods to text data. Applied to open-ended survey responses, NLP can: classify responses into pre-defined categories (reproducing what human coders do, at scale); identify topics or themes inductively (topic modelling); extract specific information such as prices, quantities, or place names (entity extraction); and score sentiment, complexity, or specificity. These capabilities turn verbatim text responses — which are expensive to code manually — into structured quantitative variables that can be used in regression analysis.
In field surveys, NLP is most useful for: “other (specify)” fields that capture responses outside the closed-ended categories; cognitive test items requiring written answers; post-survey feedback; and pilot surveys used to develop future closed-ended instruments.
When to use it
NLP for survey coding is appropriate when: the number of open-ended responses exceeds what a human coding team can manually code within time and budget constraints (roughly, more than 2,000–3,000 responses per coder per week); when high inter-rater reliability has been established for a human coding scheme that can then be replicated by a model; or when the goal is inductive discovery of themes rather than applying a pre-specified scheme.
For most field surveys in development research, the primary NLP application is classification of “specify” fields — using a trained classifier to assign these responses to the original closed-ended categories, recovering responses that enumerators could not match in the field.
How it works
Pre-processing. Clean raw text: standardise encoding, remove special characters, normalise case, handle code-switching (responses mixing two languages), and transliterate non-Latin scripts if needed. For responses in local languages, language-specific tokenisation and stopword removal are necessary.
Text classification (supervised). Train a classifier on a labelled sample (human-coded responses). Represent text as features — TF-IDF vectors for simple models; contextual embeddings from pre-trained language models (BERT, multilingual mBERT, XLM-R) for higher accuracy. Fine-tune on the labelled training set and evaluate on a held-out validation set. Report precision, recall, and F1 per class.
Topic modelling (unsupervised). When no coding scheme exists, Latent Dirichlet Allocation (LDA) or BERTopic identify recurring topics across a corpus of responses without supervision. Each document is assigned a probability distribution across topics; topics are characterised by their most frequent words. Topics are inspected and labelled by the researcher. This approach is inductive — it generates themes from the data rather than testing a pre-specified scheme.
Validation against human codes. Any NLP-derived categorical variable used in analysis should be validated against a human-coded holdout set. The appropriate validation statistic depends on the use case: accuracy and kappa for classification; coherence scores for topic models. Report validation statistics alongside any analysis using NLP-derived variables.
Key decisions
Language model selection. For English-language responses, BERT and its variants (RoBERTa, DeBERTa) are strong fine-tuning baselines. For multilingual responses, mBERT or XLM-R handle 100+ languages. For low-resource languages common in development research (Amharic, Swahili, Bengali, Hindi), dedicated models (AfriXLMR, IndicBERT) outperform general multilingual models and should be used when available.
Training data size. Supervised classifiers require at least 50–100 labelled examples per class for reliable classification, and more when classes are nuanced or responses are short. If fewer labelled examples exist, few-shot prompting with a large language model (see the LLM-Assisted Qualitative Coding guide) may outperform fine-tuning.
Handling code-switching. Survey responses in multilingual settings often mix languages within a single response (“We don’t have enough paisa to send children to school”). Standard NLP pipelines that assume a single language fail on code-switched text. Specialised code-switching models or language-agnostic embeddings (LaBSE) are needed.
Caveats & common mistakes
Not validating on in-distribution data. A classifier trained on early survey responses from urban areas may not generalise to rural areas or later waves where language use differs. Validation on a hold-out set drawn from the target deployment population — not just the training population — is necessary.
Using NLP as a substitute for human coding on sensitive topics. NLP classifiers trained on majority-language data frequently perform poorly on responses involving sensitive, culturally specific, or idiomatic content. For outcomes involving domestic violence, mental health, or social norms, NLP-derived categories should be validated by domain experts and community members, not just measured against a generic labelled dataset.
Not reporting NLP-derived variable uncertainty. NLP classification produces a probability for each class, not a certain assignment. Binary cutoff variables discard this uncertainty. If analysis results are sensitive to the classification threshold, report sensitivity analyses over a range of thresholds.
Analysis Guide
# Fine-tune a multilingual classifier on labelled open-ended responses
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import TrainingArguments, Trainer
from datasets import Dataset
import numpy as np
from sklearn.metrics import classification_report, cohen_kappa_score
# responses_df: pandas DataFrame with columns 'text' and 'label'
import pandas as pd
responses_df = pd.read_csv("labelled_responses.csv")
tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-base")
model = AutoModelForSequenceClassification.from_pretrained(
"xlm-roberta-base",
num_labels=responses_df["label"].nunique())
def tokenize(batch):
return tokenizer(batch["text"], truncation=True, padding=True, max_length=128)
ds = Dataset.from_pandas(responses_df).train_test_split(test_size=0.2)
ds = ds.map(tokenize, batched=True)
args = TrainingArguments(output_dir="nlp_model", num_train_epochs=4,
per_device_train_batch_size=16, evaluation_strategy="epoch")
trainer = Trainer(model=model, args=args,
train_dataset=ds["train"], eval_dataset=ds["test"])
trainer.train()
# Evaluate on held-out set
preds = trainer.predict(ds["test"])
y_pred = np.argmax(preds.predictions, axis=1)
y_true = np.array(ds["test"]["label"])
print(classification_report(y_true, y_pred))
print("Cohen kappa:", round(cohen_kappa_score(y_true, y_pred), 3))Reading the output
- Cohen’s kappa below 0.60 between model predictions and human labels on the validation set indicates the classifier is not reliable enough for use in analysis; revise the code scheme or increase the training sample before deploying on the full corpus.
- F1 score per class below 0.65 for any individual category means that category will be systematically under- or over-assigned; treat results for that category with caution and report the class-level F1 in any publication using NLP-derived variables.
- If validation performance differs by more than 0.10 in kappa between early and late survey waves, the classifier is drifting — retrain on a balanced sample from both periods.
- For topic models, a coherence score (c_v) below 0.40 indicates topics are not semantically coherent; reduce the number of topics or increase minimum document frequency thresholds.
- Run the classifier at two probability thresholds (0.50 and 0.70) and report whether substantive conclusions change; if they do, the threshold choice is a material analytical decision that must be reported.
Getting started
-
Hugging Face NLP Course — Free, hands-on course covering text classification, token classification, and fine-tuning transformer models. Chapters 1–3 cover everything needed to build a survey response classifier. Includes multilingual examples.
-
BERTopic documentation — Quickstart guide for topic modelling using contextual embeddings. Covers fitting a model, inspecting topics, and visualising clusters. Works out of the box on multilingual text with the
language="multilingual"option. -
Hugging Face Text Classification tutorial — Official walkthrough for fine-tuning a pre-trained model on a labelled dataset. Includes evaluation and inference steps. Can be adapted for multilingual survey responses by swapping in
xlm-roberta-base. -
spaCy 101 — Introduction to spaCy for text preprocessing — tokenisation, entity recognition, and linguistic annotation. Useful for cleaning and structuring open-ended responses before classification or topic modelling.
-
Scikit-learn text analytics tutorial — Simpler alternative using TF-IDF features and logistic regression. A good starting point if transformer-based approaches feel too complex or if the labelled dataset is small.
References
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT 2019. https://doi.org/10.18653/v1/N19-1423
Grootendorst, M. (2022). BERTopic: Neural topic modelling with a class-based TF-IDF procedure. arXiv preprint. https://arxiv.org/abs/2203.05794
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. Proceedings of ACL 2020. https://doi.org/10.18653/v1/2020.acl-main.747
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.