What it is
Inter-rater reliability (IRR) measures how consistently two or more independent raters assign the same code, score, or classification to the same observation. It is relevant whenever human judgment is required to convert raw data — open-ended responses, observed behaviours, fieldwork notes, audio recordings, photographs — into quantitative variables. Low IRR means the data contain substantial rater-specific noise: what one coder calls “high engagement” another codes as “moderate,” and the resulting variable reflects coder differences as much as the underlying construct.
IRR is a prerequisite for interpreting coded data. A study reporting that 40% of classroom interactions were “student-initiated” is only interpretable if two independent coders agreed on what counts as student-initiated. If they disagreed on 30% of interactions, the category is poorly defined and the data are unreliable.
When to use it
IRR assessment is required whenever coded categorical or ordinal data are a primary outcome or key covariate. It is particularly important for: open-ended survey responses coded into categories; direct observation protocols (classroom observation, household observation, health facility audits); qualitative data coded alongside quantitative surveys; and any study using trained coders whose agreement must be documented for research transparency.
The standard practice is to have a subset of material (10–20% of the full sample, or a randomly selected calibration set) double-coded by two independent raters and to compute IRR statistics before one rater proceeds with the remainder of the coding task.
How it works
Percentage agreement. The simplest measure: the proportion of observations where the two raters agree. Straightforward to compute but has a critical flaw — it does not account for chance agreement. Two raters randomly assigning binary codes would agree 50% of the time by chance. Percentage agreement inflates apparent reliability.
Cohen’s kappa. The standard IRR statistic for nominal categories. Kappa adjusts percentage agreement for expected agreement by chance:
κ = (P_o − P_e) / (1 − P_e)
where P_o is observed agreement and P_e is expected agreement by chance. Kappa = 0 means agreement at the level of chance; kappa = 1 means perfect agreement. Conventional benchmarks: kappa < 0.4 = poor; 0.4–0.6 = moderate; 0.6–0.8 = substantial; > 0.8 = almost perfect (Landis & Koch, 1977).
Weighted kappa. For ordinal ratings, gives partial credit for near-disagreements (a rating of 3 vs. 4 is treated as less serious than 3 vs. 5). Linear or quadratic weighting schemes reflect different assumptions about the cost of disagreement across rating levels.
Intraclass correlation coefficient (ICC). For continuous ratings, the ICC measures agreement analogously to kappa, using a variance components model. ICC(2,1) for absolute agreement between raters is the appropriate choice when raters are considered a random sample from the population of possible raters.
Krippendorff’s alpha. A generalised agreement measure that handles any level of measurement (nominal, ordinal, interval, ratio) and any number of raters, including incomplete data (not all raters rate all items). Preferred when raters do not all rate every item or when the scale is continuous.
Key decisions
Number of items to double-code. Double-coding all observations is ideal but expensive. A minimum of 50–100 items double-coded by both raters is needed to estimate IRR with reasonable precision. Selecting these items randomly from the full pool ensures the IRR estimate is representative.
Calibration before independent coding. Before independent coding begins, raters should complete a joint calibration session in which they code the same items and discuss disagreements, resolving them through discussion. Calibration builds a shared understanding of the coding scheme. Only after calibration do raters independently code the remainder of the sample.
Disagreement resolution protocol. For items that both raters code and disagree, a resolution protocol must be specified: adjudication by a senior coder, majority vote (if three raters), or assigning the modal response. Pre-specifying this prevents post-hoc cherry-picking of the higher-reliability resolution rule.
Caveats & common mistakes
Using percentage agreement as the sole metric. Percentage agreement without kappa is routinely over-interpreted. For categories with very unequal base rates (e.g., 90% of observations are coded “absent”), two raters who always code “absent” would agree 90% of the time with kappa = 0. Always report kappa alongside percentage agreement.
Assessing IRR on an unrepresentative subset. If the double-coded calibration set is selected from the “easy” observations (clear, unambiguous cases), IRR will be inflated relative to the full data. The calibration set should be a random sample, not a hand-picked set.
Not reporting IRR in publications. IRR statistics should be reported in the methods section of any study using coded data. Omitting them makes it impossible for readers to evaluate the reliability of the coding, and reviewers in top journals will request them.
Analysis Guide
import pandas as pd
import numpy as np
import pingouin as pg
from sklearn.metrics import cohen_kappa_score
from statsmodels.stats.inter_rater import fleiss_kappa, aggregate_raters
# 1. Cohen's kappa for two raters on nominal categories — adjusts raw agreement
# for chance agreement, which inflates with unbalanced category base rates
k_nominal = cohen_kappa_score(df["rater1"], df["rater2"])
print(f"Cohen kappa (nominal) = {k_nominal:.3f}")
# 2. Weighted kappa for ordinal ratings — quadratic weights penalise large
# disagreements more heavily, treating a 1-vs-2 gap as less serious than 1-vs-5
k_weighted = cohen_kappa_score(df["rater1"], df["rater2"], weights="quadratic")
print(f"Cohen kappa (quadratic-weighted) = {k_weighted:.3f}")
# 3. Percentage agreement: simple but uncorrected for chance — always report
# alongside kappa; a high raw agreement with low kappa flags base-rate dominance
pct_agree = (df["rater1"] == df["rater2"]).mean() * 100
print(f"Percentage agreement: {pct_agree:.1f}%")
# 4. Fleiss kappa for multiple raters — aggregate_raters produces the rater-count
# matrix per item that fleiss_kappa expects; use this when more than two raters
# code the same items independently
rater_cols = ["rater1", "rater2", "rater3"]
table, _ = aggregate_raters(df[rater_cols].values)
print(f"Fleiss kappa = {fleiss_kappa(table):.3f}")
# 5. ICC for continuous ratings — reshape to long so each row is one rating;
# ICC2 (two-way random, absolute agreement, single rater) is the right variant
# when raters are drawn from a larger pool and absolute scores matter
long = df.melt(id_vars="obs_id", value_vars=rater_cols,
var_name="rater", value_name="score")
icc = pg.intraclass_corr(data=long, targets="obs_id",
raters="rater", ratings="score")
print(icc[icc["Type"] == "ICC2"])Reading the output
- Kappa benchmarks (Landis & Koch): below 0.40 = poor; 0.40–0.60 = moderate; 0.60–0.80 = substantial; above 0.80 = almost perfect. For primary outcomes, aim for kappa > 0.70 before proceeding to full coding.
- Percentage agreement without kappa is misleading when one category dominates — a 90% agreement rate with kappa = 0.10 means coders are mostly agreeing by chance.
- Weighted kappa is always higher than unweighted for ordinal scales; use linear weights for scales where equal spacing is plausible, quadratic weights when large disagreements are disproportionately costly.
- Krippendorff’s alpha is preferred when raters do not code all items (missing data) or when more than two raters are involved.
- ICC above 0.75 is acceptable for continuous ratings; ICC below 0.50 means the coded variable is dominated by rater noise and should not be used as a primary outcome without re-training coders.
References
Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology (2nd ed.). Sage Publications.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420