LLM-Assisted Qualitative Coding

A workflow that uses large language models (LLMs) to apply researcher-defined codes to qualitative text — interview transcripts, FGD notes, open-ended responses — reducing the labour cost of systematic coding while preserving the transparency and inter-rater reliability standards of conventional qualitative analysis.

What it is

Large language models (LLMs) such as Claude, GPT-4, and Gemini can apply a researcher-specified codebook to qualitative text with reasonable accuracy on many coding tasks — faster and cheaper than human coders, but not reliably or autonomously. LLM-assisted qualitative coding uses LLMs as a first-pass coding tool: the LLM applies codes to a large corpus of text, and a human researcher validates a random sample of the LLM’s coding decisions to estimate reliability and identify systematic errors.

The appropriate frame is “LLM as junior research assistant under supervision,” not “LLM as autonomous coder.” The LLM reduces the labour required to process large qualitative datasets but does not replace the judgment required to develop the codebook, interpret ambiguous passages, or assess whether the coding is valid for the research question.

When to use it

LLM-assisted coding is appropriate when: the corpus is too large for full manual coding within time and budget constraints; the codebook is well-specified with clear inclusion/exclusion criteria and examples; and the coding task is at the passage or utterance level rather than requiring holistic interpretation of a full document. It is best suited to deductive coding (applying a pre-specified codebook) rather than inductive grounded theory development.

A minimum viable workflow requires: a codebook with at least 3–5 examples per code; a calibration set of 50–100 passages human-coded by at least two coders to establish human-level IRR; and a validation set of 50–100 passages to estimate LLM-human agreement.

How it works

Codebook specification. Define each code with: a name, a one-paragraph definition, inclusion criteria (what the code covers), exclusion criteria (what it does not cover), and 3–5 illustrative examples. Ambiguous codes with fuzzy boundaries perform poorly with LLMs. Codes that require broad contextual interpretation across a long document perform particularly poorly — LLMs are better at local passage classification.

Prompt engineering. Structure the prompt to provide: (1) the codebook definition for the relevant code(s); (2) the text passage to classify; (3) a structured output format (JSON or a simple categorical response). Few-shot prompting — including 2–3 labelled examples in the prompt — substantially improves accuracy relative to zero-shot prompting.

Batch processing. For large corpora, use the LLM’s API (not a chat interface) to process passages in batches. Log every input and output for auditability. Most LLM APIs support structured output (JSON mode) to enforce a consistent response format.

Validation. After LLM coding, human coders re-code a random sample (10–20% of passages, or a minimum of 100 passages). Compute Cohen’s kappa between LLM codes and human codes. If kappa exceeds the threshold acceptable for the research purpose (commonly 0.60–0.70), the LLM-coded data can be used. If kappa is below threshold, diagnose the failure modes: which codes fail? which text types fail? Revise the codebook and prompt, then re-validate.

Uncertainty quantification. LLMs with logprob access (or explicit confidence prompting) can report a confidence level for each coding decision. Low-confidence passages should be flagged for human review. This creates a tiered workflow: high-confidence LLM decisions are accepted; low-confidence ones are manually reviewed.

Key decisions

LLM selection. Larger, more capable models (Claude Sonnet/Opus, GPT-4, Gemini Pro) produce higher coding accuracy but cost more per API call. For simple binary classification of short passages, smaller models (GPT-4o-mini, Claude Haiku) are often sufficient. Benchmark on a calibration set before committing to a model for full-corpus processing.

Context window. Some coding tasks require reading a long document before classifying a passage (the passage meaning depends on earlier context). If so, the full document must fit within the LLM’s context window with the passage and codebook. For very long documents, sliding window approaches or summarisation of prior context may be needed.

Disclosure in publications. Research using LLM-assisted coding should disclose: which LLM was used and its version, the prompt structure, the LLM-human kappa on the validation set, how disagreements were resolved, and what share of the corpus was human-reviewed. Journals are developing reporting standards; transparency above the minimum is advisable.

Caveats & common mistakes

Using LLMs without measuring reliability. LLMs can produce confident but wrong codes — they rarely say “I don’t know.” Deploying LLM codes without measuring LLM-human agreement treats the LLM as infallible. This is the most common and serious error. Always validate.

Treating LLM codes as equivalent to human codes without evidence. Even when LLM-human kappa is acceptable on the validation set, the LLM may be systematically wrong on specific sub-populations (shorter passages, code-switched text, passages about sensitive topics). Report validation statistics disaggregated by relevant subgroups.

Not versioning the model and prompt. LLM outputs are not reproducible across model versions — the same prompt with GPT-4-0613 and GPT-4-turbo may produce different outputs. Document the model version and retain the full prompt used for coding. Rerunning the analysis after a model update may change results.

Analysis Guide

# Apply a codebook code to passages using the Claude API, then validate
import anthropic
import pandas as pd
from sklearn.metrics import cohen_kappa_score

client = anthropic.Anthropic()  # uses ANTHROPIC_API_KEY env var

CODEBOOK = """
Code: ECONOMIC_STRESS
Definition: The respondent describes difficulty meeting household expenses,
inability to purchase necessities, or anxiety about money.
Inclusion: Direct statements of financial hardship, inability to buy food/medicine.
Exclusion: General complaints about prices without personal hardship mentioned.
Examples:
- "We couldn't afford to send the children to school this term." → YES
- "Prices have gone up this year." → NO
"""

def code_passage(passage: str) -> dict:
  msg = client.messages.create(
      model="claude-sonnet-4-6",
      max_tokens=64,
      messages=[{
          "role": "user",
          "content": f"{CODEBOOK}

Passage: {passage}

"
                     "Does this passage contain ECONOMIC_STRESS? "
                     "Reply with JSON: {{"code": true/false, "confidence": 0-1}}"
      }]
  )
  import json
  return json.loads(msg.content[0].text)

# Process passages
passages_df = pd.read_csv("passages.csv")   # cols: passage_id, text, human_code
results = [code_passage(row.text) for _, row in passages_df.iterrows()]
passages_df["llm_code"]       = [r["code"] for r in results]
passages_df["llm_confidence"] = [r["confidence"] for r in results]

# Validate on human-coded subset
val = passages_df.dropna(subset=["human_code"])
kappa = cohen_kappa_score(val["human_code"].astype(bool), val["llm_code"])
print(f"LLM-human kappa: {kappa:.3f}")

library(httr2)
library(jsonlite)
library(irr)

# Apply LLM coding via Claude API
code_passage <- function(passage, api_key = Sys.getenv("ANTHROPIC_API_KEY")) {
codebook <- "Code: ECONOMIC_STRESS — respondent describes difficulty meeting
household expenses or inability to purchase necessities."

body <- list(
  model      = "claude-sonnet-4-6",
  max_tokens = 64L,
  messages   = list(list(
    role    = "user",
    content = paste0(codebook, "

Passage: ", passage,
                     "

Reply with JSON: {"code": true/false, "confidence": 0-1}")
  ))
)

resp <- request("https://api.anthropic.com/v1/messages") %>%
  req_headers("x-api-key" = api_key, "anthropic-version" = "2023-06-01",
              "content-type" = "application/json") %>%
  req_body_json(body) %>%
  req_perform()

fromJSON(resp_body_string(resp))$content[[1]]$text %>% fromJSON()
}

passages_df <- read.csv("passages.csv")   # cols: passage_id, text, human_code
results <- lapply(passages_df$text, code_passage)
passages_df$llm_code       <- sapply(results, (r) r$code)
passages_df$llm_confidence <- sapply(results, (r) r$confidence)

# Compute kappa on human-coded validation subset
val <- subset(passages_df, !is.na(human_code))
kappa2(data.frame(val$human_code, val$llm_code))$value

Reading the output

Cohen’s kappa below 0.60 between LLM and human codes on the validation set means the LLM should not be used as the primary coder; diagnose which codes fail and revise the codebook definitions before re-validating.
Kappa between 0.60 and 0.70 is acceptable for exploratory analysis; kappa above 0.70 meets the threshold commonly used for publication-quality coding.
Review passages where LLM confidence is below 0.65 — these are the cases where human review will add the most value; prioritise them for manual verification.
If kappa differs by more than 0.10 between short passages (under 50 words) and long passages (over 150 words), adjust the batching strategy so each passage contains sufficient context for reliable classification.
Log the model version used (claude-sonnet-4-6 in this example); re-running the same prompt on a different model version may produce different kappa values, so the model version must be fixed for reproducibility.

Getting started

Anthropic API documentation — Official documentation for the Claude API, including authentication, message formatting, and structured output. The Messages API is the primary interface for building LLM coding pipelines.
Anthropic Cookbook — Collection of worked examples for common Claude API use cases, including classification, structured extraction, and batch processing. The classification notebooks are directly applicable to qualitative coding workflows.
OpenAI Cookbook — Text classification — Practical guide to using GPT models for text classification, including prompt design, few-shot examples, and evaluation against human labels. Patterns transfer directly to Claude.
Prompt Engineering Guide — Explanation of few-shot prompting with worked examples across classification and extraction tasks. The few-shot section is the most directly relevant for codebook-based qualitative coding.
LLM-as-annotator validation template (Pangakis et al., 2023) — The paper includes a reproducible validation workflow for measuring LLM-human agreement using Cohen’s kappa. Use it as a template for the validation step of any LLM coding pipeline.

References

Gilardi, F., Alizadeh, M., & Kubli, M. (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30), e2305016120. https://doi.org/10.1073/pnas.2305016120

Pangakis, N., Wolken, S., & Fasching, N. (2023). Automated annotation with generative AI requires validation. arXiv preprint. https://arxiv.org/abs/2306.00176

Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology (2nd ed.). Sage Publications.

Anthropic. (2024). Claude API documentation. https://docs.anthropic.com