Automated Transcription Quality Assessment

A validation workflow for audio or video transcripts produced by automated speech recognition (ASR) systems — checking word error rate, speaker attribution, completeness, and handling of code-switching and local vocabulary before transcripts are used for coding, analysis, or publication.

What it is

Automated speech recognition (ASR) tools — Whisper, AssemblyAI, Google Speech-to-Text, AWS Transcribe — can convert audio recordings of interviews, FGDs, and community meetings into text at a fraction of the cost and time of manual transcription. But ASR systems produce transcripts with errors: mishearing words, incorrectly attributing speech to speakers, dropping or garbling sections, and failing on unfamiliar vocabulary (local names, technical terms, code-switching between languages). Using ASR transcripts in analysis without assessing their quality is a reliability threat.

Automated transcription quality assessment quantifies these errors before the transcripts are used — identifying which recordings produced poor transcripts that need manual correction before coding.

When to use it

ASR quality assessment is required whenever automated transcripts will be used as data for qualitative coding, NLP analysis, or direct quotation in publications. It is especially important when: audio quality is variable (field recordings with background noise, low-quality microphones, distance from the speaker); when respondents use local languages, dialects, or code-switched speech that ASR systems are not optimised for; or when the transcript will be used for LLM-assisted coding (see the LLM-Assisted Qualitative Coding guide), where errors propagate into the coding output.

How it works

Word Error Rate (WER). The standard ASR accuracy metric. WER = (substitutions + deletions + insertions) / total words in reference. Computed by aligning the ASR transcript against a manually verified reference transcript for a sample of recordings. WER below 10% is generally acceptable for research use; WER above 20% indicates the transcript requires significant manual correction. WER is calculated on a sample — manually transcribe 5–10% of recordings to estimate the distribution of WER across the corpus.

Sentence Error Rate (SER). A stricter metric: the proportion of sentences in which any error occurred. Useful when the analysis unit is the sentence or utterance (as in FGD coding) rather than individual words.

Speaker diarisation accuracy. In multi-speaker recordings (FGDs, key informant interviews with multiple respondents), ASR systems must attribute each speech segment to a speaker. Diarisation errors — wrong speaker attribution, merged speakers — corrupt FGD analysis. Validate speaker attribution on a sample by comparing ASR-attributed segments against manually annotated speaker turns.

Coverage check. Identify gaps in the transcript — time segments where ASR produced no text despite audio being present (common in noisy recordings or when speech is very quiet). Gaps can be identified by comparing transcript length against recording duration, or by checking the timestamps of ASR output against audio length.

Vocabulary coverage. ASR systems trained primarily on formal speech have high error rates on: local place names, personal names, agricultural or medical terminology specific to the study context, and code-switched utterances. Extract the vocabulary from the transcripts and check what proportion of words are recognised (appear in a reference dictionary or word frequency list for the language). High proportions of out-of-vocabulary words predict high WER.

Key decisions

ASR model selection. OpenAI’s Whisper (particularly the large-v3 variant) is the most accurate open-source ASR model across a broad range of languages as of 2024, and supports 99 languages. Commercial APIs (AssemblyAI, Rev AI) offer higher accuracy for English and Spanish with speaker diarisation, at a cost. For low-resource languages, test multiple models on a calibration set before committing.

Manual correction prioritisation. When WER is above threshold for some recordings, prioritise manual correction based on the recording’s importance (primary data source vs. supplementary) and usage (direct quotation vs. thematic coding). Full manual re-transcription is rarely feasible for large corpora; targeted correction of high-error passages is more practical.

Consent and data storage. Audio recordings of research participants require explicit consent and careful data management. ASR processing using commercial APIs sends audio data to third-party servers — check whether this is consistent with the IRB protocol and data sharing agreements. On-premise processing using open-source models (Whisper) avoids this issue.

Caveats & common mistakes

Assuming ASR accuracy is uniform. ASR accuracy varies substantially across recordings based on audio quality, speaker accent, speech rate, and background noise. A corpus-level mean WER of 8% may conceal 30% WER on 20% of recordings. Always examine the distribution of WER across recordings, not just the mean.

Using WER computed on clean audio as a proxy for field audio. WER benchmarks published for ASR models are typically computed on clean studio recordings. Field recordings with ambient noise, fan noise, multiple speakers, and informal speech styles will have substantially higher WER than the benchmark suggests. Calibrate on your own data.

Not documenting transcript corrections. When manual corrections are made to ASR transcripts, the corrected version should be distinguished from the raw ASR output in the data management system. Corrections should be logged (who corrected, what was changed) for auditability. Presenting corrected transcripts as raw ASR output misrepresents the data provenance.

Analysis Guide

# Compute WER for a sample of ASR transcripts using jiwer
import whisper
import jiwer
import pandas as pd

# Transcribe a sample of audio files with Whisper large-v3
model = whisper.load_model("large-v3")

sample_files = ["rec_001.mp3", "rec_002.mp3", "rec_003.mp3"]  # calibration sample

results = []
for f in sample_files:
  result = model.transcribe(f, language="hi")   # set language code as needed
  results.append({"file": f, "asr_text": result["text"]})

transcripts_df = pd.DataFrame(results)
transcripts_df.to_csv("asr_transcripts.csv", index=False)

# Compute WER against manually verified reference transcripts
ref_df = pd.read_csv("reference_transcripts.csv")  # cols: file, reference_text
merged = transcripts_df.merge(ref_df, on="file")

transform = jiwer.Compose([jiwer.ToLowerCase(), jiwer.RemovePunctuation(),
                          jiwer.Strip(), jiwer.ReduceToListOfListOfWords()])

for _, row in merged.iterrows():
  measures = jiwer.compute_measures(row["reference_text"], row["asr_text"],
                                    truth_transform=transform,
                                    hypothesis_transform=transform)
  print(f"{row['file']}: WER={measures['wer']:.2f}, SER={measures['mer']:.2f}")

library(reticulate)   # call jiwer from R via Python
library(dplyr)

# Install jiwer in the Python environment reticulate uses:
# py_install("jiwer")

jiwer <- import("jiwer")

# ref_df: data frame with cols file, reference_text, asr_text
ref_df <- read.csv("validation_transcripts.csv")

compute_wer <- function(reference, hypothesis) {
measures <- jiwer$compute_measures(reference, hypothesis)
list(wer = measures$wer, mer = measures$mer)
}

wer_results <- mapply(compute_wer,
                    ref_df$reference_text,
                    ref_df$asr_text,
                    SIMPLIFY = FALSE)

ref_df$wer <- sapply(wer_results, (x) x$wer)
ref_df$ser <- sapply(wer_results, (x) x$mer)

# Summary
summary(ref_df[, c("wer", "ser")])

# Flag recordings needing manual correction
filter(ref_df, wer > 0.20) %>%
select(file, wer, ser) %>%
arrange(desc(wer))

Reading the output

WER below 0.10 (10%) is generally acceptable for research use in thematic coding and NLP analysis; recordings above this threshold should be manually reviewed before coding.
WER above 0.20 indicates the transcript requires significant manual correction before it can be used; at this error rate, automated coding will propagate errors into analytic outputs.
Sentence Error Rate (SER / MER) above 0.30 means more than 30% of sentences contain at least one error — particularly problematic if the analysis unit is the utterance or sentence.
If more than 20% of recordings in the sample exceed WER 0.15, the ASR model is not well-suited to the audio conditions; test an alternative model or language-specific fine-tuned version on the same calibration sample.
Check whether WER is systematically higher for specific enumerators, locations, or interview types — heterogeneous WER by subgroup can introduce differential measurement error that biases comparisons.

Getting started

OpenAI Whisper GitHub — Repository for the open-source Whisper model with installation instructions, supported languages, and usage examples. The README covers transcribing a single file in three lines of Python. Use large-v3 for best accuracy on field recordings.
WhisperX — Extension of Whisper that adds accurate word-level timestamps and multi-speaker diarisation. The recommended choice when speaker attribution matters (FGDs, multi-respondent interviews).
jiwer documentation — Python library for computing WER, CER, and related metrics. The quickstart shows how to compare a reference and hypothesis transcript and get a detailed breakdown of substitution, deletion, and insertion errors.
AssemblyAI documentation — Commercial ASR API with built-in speaker diarisation, sentiment analysis, and topic detection. Higher accuracy than Whisper for English and Spanish; paid per audio hour. Useful when offline processing is not feasible.
Kaldi speech recognition toolkit — Advanced open-source ASR framework used in research settings. Steep learning curve but supports training custom acoustic models for languages where Whisper performs poorly. Relevant only for projects requiring very high accuracy on a specific low-resource language.

References

Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. Proceedings of ICML 2023. https://arxiv.org/abs/2212.04356

Woodrich, M., & Frauchiger, J. (2023). Whisper for African Languages: Evaluation on low-resource languages. AfricaNLP Workshop, ACL 2023.

Morris, A. C., Maier, V., & Green, P. (2004). From WER and RIL to MER and WIL: Improved evaluation measures for connected speech recognition. Proceedings of Interspeech 2004.