Multiple Hypothesis Testing Correction

A statistical correction for the inflated false positive rate that results from testing multiple hypotheses in the same study — using family-wise error rate (FWER) or false discovery rate (FDR) methods to maintain the credibility of significance claims across a battery of outcomes.

What it is

When a study tests many outcomes, the probability of finding at least one spuriously significant result grows rapidly — even if the true treatment effect is zero for all outcomes. With 20 independent outcomes tested at α = 0.05, the expected number of false positives is 1. This problem — the multiple comparisons problem — is especially acute in development economics evaluations, which routinely test effects on 10–50 outcomes across health, education, economic, and behavioural domains.

Multiple hypothesis testing (MHT) corrections maintain the credibility of statistical claims by adjusting p-values or critical values to account for the number of hypotheses being tested. They come in two families: family-wise error rate (FWER) methods, which control the probability of any false positive in the family; and false discovery rate (FDR) methods, which control the expected proportion of false positives among rejected hypotheses. The choice between them depends on the cost of false positives relative to false negatives in the specific context.

When to use it

MHT corrections should be applied whenever a study reports treatment effects on multiple outcomes in the same family — outcomes that conceptually belong together and are jointly subject to the multiple comparisons problem. The key paper for development economics is Anderson (2008), which introduces the inverse covariance-weighted index approach and demonstrates how MHT inflation can produce the appearance of gender-differentiated treatment effects that are purely artefactual.

Pre-registration of outcome families and the correction method is the strongest protection against MHT inflation, because it prevents ex-post selection of which comparisons to correct.

How it works

Bonferroni correction. The simplest FWER method divides the significance threshold by the number of hypotheses: α* = α/m, where m is the number of tests. A family of 10 tests at α = 0.05 requires each individual p-value to be below 0.005. Bonferroni is conservative (underpowered) when tests are positively correlated, which is typical for related outcomes.

Holm-Bonferroni. A sequential version of Bonferroni that is uniformly more powerful. Sort p-values from smallest to largest. The smallest p-value must satisfy p < α/m; the second must satisfy p < α/(m−1); and so on. Once any p-value fails to meet its threshold, all remaining hypotheses are not rejected.

Benjamini-Hochberg (BH) FDR correction. Controls the expected proportion of false discoveries among rejected hypotheses. Sort p-values from smallest to largest. Find the largest k such that p_(k) ≤ kα/m. Reject all hypotheses 1 through k. BH is less conservative than FWER methods and is appropriate when some false discoveries are tolerable and power matters more than avoiding all false positives.

Anderson (2008) sharpened q-values. A FDR-controlling method adapted for development economics contexts. Produces a q-value for each outcome representing the minimum false discovery rate at which that outcome would be rejected. Q-values can be reported alongside p-values in standard results tables.

Randomisation inference (permutation tests). Re-randomise treatment assignment many times and compute the test statistic under the null of no effect. The empirical distribution of the maximum test statistic across outcomes provides a joint FWER correction that accounts for the correlation structure among outcomes without assuming a specific distribution.

Key decisions

Defining outcome families. The choice of which outcomes to group into a family is consequential — grouping all 50 outcomes together produces more conservative corrections than grouping by domain. Pre-registered families based on conceptual groupings (e.g., “primary outcomes,” “economic outcomes,” “health outcomes”) are more credible than post-hoc families constructed to produce significant results.

FWER vs. FDR. FWER is appropriate when any false positive is costly — confirmatory trials, regulatory submissions, policy decisions based on a single finding. FDR is appropriate for exploratory analyses where false positives are tolerable and the priority is identifying a set of promising outcomes for follow-up investigation.

Index construction as alternative. Anderson (2008) recommends constructing a single summary index from related outcomes before testing, rather than testing each outcome and correcting. The index approach reduces dimensionality without requiring a correction and is pre-specifiable. Both approaches are legitimate; the index approach has the advantage of being pre-committed to a single test.

Caveats & common mistakes

Applying correction to all outcomes in the paper. MHT corrections should be applied within families of conceptually related outcomes, not across all analyses in a paper. Correcting for every regression in a paper — including robustness checks, heterogeneity analyses, and placebo tests — is over-conservative and not standard practice. The correction scope should be pre-specified.

Not correcting within subgroup analyses. Heterogeneity analyses that test treatment effects for many subgroups face the same multiple comparisons problem. A significant subgroup effect found after testing 20 subgroups without correction has an inflated false positive rate. Pre-specifying the subgroups of interest and correcting within the subgroup family is important.

Reporting only corrected p-values. Corrected p-values should supplement, not replace, uncorrected p-values and effect sizes in results tables. Readers need both to evaluate the evidence — the correction tells them about statistical credibility; the effect size tells them about practical significance.

Analysis Guide

import numpy as np
import pandas as pd
from statsmodels.stats.multitest import multipletests

# p-values for a pre-specified family of outcomes
p_vals = np.array([0.03, 0.01, 0.24, 0.06, 0.18, 0.04, 0.52, 0.09, 0.11, 0.33])
outcome_names = [f'outcome_{i}' for i in range(1, 11)]

# 1. Holm-Bonferroni — controls the family-wise error rate (probability of any
#    false positive); more powerful than Bonferroni because it adjusts the
#    threshold sequentially as hypotheses are rejected; appropriate when any
#    false positive is costly (confirmatory analyses, policy decisions)
_, p_holm, _, _ = multipletests(p_vals, alpha=0.05, method='holm')

# 2. Benjamini-Hochberg FDR — controls the expected share of false positives
#    among rejected hypotheses rather than preventing all false positives;
#    less conservative than Holm and preferred for exploratory analyses where
#    some false discoveries are tolerable
_, p_bh, _, _ = multipletests(p_vals, alpha=0.05, method='fdr_bh')

# 3. Bonferroni — included as a reference; the simplest and most conservative
#    FWER method, equivalent to multiplying each p-value by the number of tests
_, p_bonf, _, _ = multipletests(p_vals, alpha=0.05, method='bonferroni')

# 4. Summary table — compare raw, Holm, BH, and Bonferroni columns side by
#    side; outcomes that survive Holm are the most credible; outcomes significant
#    only in the raw column are likely false positives if many outcomes were tested
results = pd.DataFrame({
  'outcome':  outcome_names,
  'p_raw':    p_vals,
  'p_holm':   p_holm,
  'p_bh':     p_bh,
  'p_bonf':   p_bonf,
  'sig_raw':  p_vals  < 0.05,
  'sig_holm': p_holm  < 0.05,
  'sig_bh':   p_bh    < 0.05,
})
print(results)

# 5. Romano-Wolf step-down — no canonical Python implementation; controls FWER
#    while accounting for the correlation structure across tests via bootstrap
#    re-sampling of the test statistics. Use R's rwolf package, or implement
#    by resampling residuals under the null and tracking the empirical
#    distribution of the maximum t-statistic across outcomes

# p-values for a pre-specified family of outcomes
p_vals        <- c(0.03, 0.01, 0.24, 0.06, 0.18, 0.04, 0.52, 0.09, 0.11, 0.33)
outcome_names <- paste0("outcome_", 1:10)

# 1. Holm-Bonferroni — controls the family-wise error rate (probability of any
#    false positive); more powerful than Bonferroni because it adjusts the
#    threshold sequentially as hypotheses are rejected; appropriate when any
#    false positive is costly (confirmatory analyses, policy decisions)
p_holm <- p.adjust(p_vals, method = "holm")

# 2. Benjamini-Hochberg FDR — controls the expected share of false positives
#    among rejected hypotheses rather than preventing all false positives;
#    less conservative than Holm and preferred for exploratory analyses where
#    some false discoveries are tolerable
p_bh <- p.adjust(p_vals, method = "BH")

# 3. Summary table — compare raw, Holm, and BH columns side by side;
#    outcomes that survive Holm are the most credible; outcomes significant
#    only in the raw column are likely false positives if many outcomes were tested
results <- data.frame(
outcome  = outcome_names,
p_raw    = p_vals,
p_holm   = p_holm,
p_bh     = p_bh,
sig_raw  = p_vals < 0.05,
sig_holm = p_holm < 0.05,
sig_bh   = p_bh < 0.05
)
print(results)

Reading the output

Compare sig_raw vs. sig_holm vs. sig_bh: findings that survive Holm correction are the most credible; findings that appear only in the raw column are likely false positives if many outcomes were tested.
Anderson q-values below 0.05 mean the outcome would be rejected at a 5% false discovery rate — report these alongside raw p-values in results tables.
BH correction is less conservative than Holm: outcomes significant under BH but not Holm sit in a grey zone and should be described as “suggestive” rather than conclusive.
If all corrected p-values are non-significant but several raw p-values cluster just below 0.05, this pattern itself is a signal of p-hacking or MHT inflation.

References

Anderson, M. L. (2008). Multiple inference and gender differences in the effects of early intervention: A reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects. Journal of the American Statistical Association, 103(484), 1481–1495. https://doi.org/10.1198/016214508000000841

Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70.

Kling, J. R., Liebman, J. B., & Katz, L. F. (2007). Experimental analysis of neighborhood effects. Econometrica, 75(1), 83–119. https://doi.org/10.1111/j.1468-0262.2007.00733.x