Measurement Invariance Testing

A sequence of confirmatory factor analysis tests that establish whether a multi-item scale has the same factor structure, loadings, and intercepts across groups — a necessary condition for valid treatment-control or male-female comparisons of scale scores.

What it is

Measurement invariance (also called measurement equivalence) is the property of a scale that guarantees it measures the same construct in the same way across different groups or time points. If scale scores are compared across treatment and control groups, across men and women, or across waves in a panel — as they routinely are in programme evaluations — the comparison only makes sense if the scale is measuring the same thing in both groups.

Without invariance, differences in scale scores between groups could reflect genuine differences in the underlying construct, or they could reflect differences in how respondents interpret the items, use the response scale, or relate to the construct. Measurement invariance testing distinguishes these explanations by testing a sequence of increasingly constrained factor models across groups.

When to use it

Measurement invariance tests are appropriate when: a multi-item scale is being compared across treatment and control groups as the primary outcome; the same scale is being compared across demographic groups (gender, caste, region) as a heterogeneous effects analysis; or the scale is measured at multiple time points and the interpretation of change over time depends on the scale being stable.

In development economics, invariance testing is most relevant for psychological scales (self-efficacy, mental health, empowerment) used in impact evaluations where gender or treatment group comparisons are planned. Haushofer and Shapiro (2016) compare PHQ-9 depression scores across treated and control groups in Kenya — a valid comparison only if PHQ-9 items function equivalently in both groups. Programmes that target women specifically and compare outcomes between treatment and control women have strong reasons to test whether scales elicited under different conditions are psychometrically equivalent.

The method is less important when individual-level outcomes are single observed variables (height, income, test score), which are not subject to factor analytic measurement issues, or when the comparison groups are expected to be drawn from the same population by random assignment (in which case measurement equivalence should hold by construction, though checking is still good practice).

How it works

Invariance testing follows a hierarchical sequence of increasingly constrained multigroup CFA models (Vandenberg & Lance, 2000):

Step 1 — Configural invariance. The same factor structure (same pattern of zero and non-zero loadings) is tested in both groups, with all parameters freely estimated within each group. If this model fits, the same items measure the same factors in both groups — a minimal equivalence requirement.

Step 2 — Metric invariance. Factor loadings are constrained to be equal across groups; intercepts and residuals are still freely estimated. If metric invariance holds, the factor affects items with the same strength in both groups, and latent variable variances and covariances can be meaningfully compared. A chi-square difference test (or ΔCFI ≤ 0.01) between the metric and configural models tests whether the constraints are supported.

Step 3 — Scalar invariance. Item intercepts are additionally constrained to be equal across groups. Scalar invariance means that the expected item response for a person with the same latent trait level is the same in both groups. This is the level required to compare latent mean scores across groups — the most common comparison in impact evaluations.

Step 4 — Strict invariance. Residual variances are additionally constrained to be equal. This is rarely required for mean comparisons and is frequently not supported. Failure of strict invariance while scalar invariance holds does not prevent valid latent mean comparisons.

Partial invariance. When scalar invariance fails for some items but not all, partial scalar invariance can be established by freeing the non-invariant intercepts. Partial invariance still allows latent mean comparisons if at least two items per factor have invariant loadings and intercepts, and the non-invariant items are acknowledged and reported.

Key decisions

Which groups to test. The groups should match the planned comparisons in the substantive analysis. If treatment-control comparisons are planned, test treatment vs. control. If heterogeneous effects by gender are planned, test male vs. female. Pre-specifying the invariance tests in the pre-analysis plan ensures they are not post-hoc adjustments to explain unexpected findings.

Response to invariance failure. If scalar invariance fails, the options are: (a) use partial invariance with the non-invariant items acknowledged; (b) remove the non-invariant items and recompute the scale; (c) report the limitation and interpret group comparisons with appropriate caution; or (d) investigate whether the non-invariant items have content-related explanations (translation issues, culturally specific interpretation). Option (d) is the most informative response and often reveals actionable scale improvements.

Sample size requirements. Multigroup CFA requires adequate sample sizes in each group — at minimum 150–200 per group. Tests of invariance with small groups produce unstable estimates and inflated rejection of invariance. With very small groups, approximate methods or Bayesian approaches to invariance testing provide more stable results.

Longitudinal invariance. For panel data, the same sequence of tests applies across time points. Metric invariance across waves allows comparison of change in factor variances and covariances; scalar invariance allows comparison of latent means over time. Longitudinal data may show scale drift — items shifting in their interpretive meaning over time — which violates invariance and complicates the interpretation of treatment effects on change scores.

Caveats & common mistakes

Skipping invariance testing entirely. The most common mistake. Researchers who compare group means on sum scores without testing invariance implicitly assume full invariance, which may not hold. Reporting invariance test results — even null results — adds credibility to group comparisons.

Using sum scores instead of latent means. Comparing sum scores or mean composite scores directly across groups conflates latent trait differences with measurement artefacts. Latent mean comparisons within a multigroup CFA framework are more appropriate when the scale is multi-item and invariance cannot be assumed.

Interpreting partial invariance as failure. Partial invariance — where most items are invariant but one or two are not — still supports meaningful group comparisons with appropriate caveats. The non-invariant items should be reported and discussed. Researchers sometimes misinterpret any deviation from full invariance as invalidating the comparison.

Ignoring scalar non-invariance in means comparisons. Metric invariance without scalar invariance allows comparison of factor variances and correlations but not latent means. Comparing latent means when only metric invariance has been established is incorrect. The level of invariance must match the intended comparison.

Analysis Guide

# Multigroup CFA tests whether the same factor structure holds across treatment and control groups
import pandas as pd
import numpy as np
from scipy import stats
from semopy import Model, calc_stats

model_desc = """
factor1 =~ item_1 + item_2 + item_3 + item_4 + item_5
"""

# 1. Configural invariance — fit the same factor structure separately within each group with no cross-group constraints; if both groups achieve adequate fit (CFI > 0.90, RMSEA < 0.08), the scale has the same basic structure across groups; poor fit here means items do not measure the same construct in both groups at all
m0 = Model(model_desc); m0.fit(df[df["treatment"] == 0])
m1 = Model(model_desc); m1.fit(df[df["treatment"] == 1])
print(calc_stats(m0)[["CFI", "RMSEA", "SRMR"]])
print(calc_stats(m1)[["CFI", "RMSEA", "SRMR"]])
chi2_config = m0.calc_chi2()[0] + m1.calc_chi2()[0]

# 2. Metric invariance — constrain factor loadings to be equal across groups; semopy supports this via the group= argument and shared parameter labels; metric invariance allows comparison of factor variances and correlations across groups
# Combined fit with cross-group equality constraints on loadings (use multi-group API or relabel loadings)
model_metric = Model(model_desc)
model_metric.fit(df, group="treatment", obj="MLW")   # default: loadings equal
chi2_metric = model_metric.calc_chi2()[0]

# 3. Scalar invariance — additionally constrain item intercepts to be equal across groups; if this holds, the expected response at a given latent level is the same in both groups; required before comparing latent means, which is the comparison relevant to treatment effect estimation
# semopy: extend group constraints to include intercepts
model_scalar = Model(model_desc + "\nitem_1 ~ 1\nitem_2 ~ 1\nitem_3 ~ 1\nitem_4 ~ 1\nitem_5 ~ 1\n")
model_scalar.fit(df, group="treatment", obj="MLW")
chi2_scalar = model_scalar.calc_chi2()[0]

# 4. Chi-square difference tests — a non-significant result (p > 0.05) means the additional constraints do not significantly worsen fit, supporting the higher invariance level; also compute Delta CFI: <= 0.010 supports invariance even when chi-square is significant in large samples
def lr_test(chi_restricted, chi_full, df_diff):
  return 1 - stats.chi2.cdf(chi_restricted - chi_full, df_diff)
# Pass the appropriate degree-of-freedom differences from the model summaries.

# Note: semopy's multigroup API is functional but less mature than lavaan's. For production
# invariance testing - especially partial invariance with selectively freed intercepts - the
# R lavaan workflow in the R tab is more reliable.

library(lavaan)

model <- '
factor1 =~ item_1 + item_2 + item_3 + item_4 + item_5
'

# 1. Configural invariance — same factor structure in both groups, all parameters freely estimated per group; this is the baseline; adequate fit (CFI > 0.90, RMSEA < 0.08) confirms the items measure the same construct in both groups
fit_config <- cfa(model, data = df, group = "treatment", group.equal = "")

# 2. Metric invariance — factor loadings constrained equal across groups; tests whether each item's relationship to the latent factor has the same strength in both groups; required for comparing factor variances and correlations
fit_metric <- cfa(model, data = df, group = "treatment", group.equal = "loadings")

# 3. Scalar invariance — loadings and item intercepts constrained equal across groups; tests whether the expected response at a given latent level is the same in both groups; required for comparing latent means, which is the comparison relevant to treatment effect estimation
fit_scalar <- cfa(model, data = df, group = "treatment",
                group.equal = c("loadings", "intercepts"))

# 4. Chi-square difference tests — a non-significant result (p > 0.05) or ΔCFI ≤ 0.010 means the constraints are supported and the higher invariance level holds
lavTestLRT(fit_config, fit_metric)   # metric vs. configural
lavTestLRT(fit_metric, fit_scalar)   # scalar vs. metric

# 5. If scalar fails: identify non-invariant intercepts via modification indices, then free only those intercepts — partial scalar invariance (at least two invariant intercepts per factor) still allows valid latent mean comparisons with appropriate caveats
modificationIndices(fit_metric, sort. = TRUE)[1:10, ]
fit_partial <- cfa(model, data = df, group = "treatment",
                 group.equal = c("loadings", "intercepts"),
                 group.partial = c("item_3~1", "item_4~1"))

# 6. Latent mean comparison — in the scalar model, the reference group latent mean is fixed at 0; the estimated mean for the comparison group is the latent mean difference on the construct's logit scale
summary(fit_scalar)   # group 2 latent mean relative to group 1 (fixed at 0)

Reading the output

Configural invariance holds if the same-factor-structure model fits adequately in both groups (CFI > 0.90, RMSEA < 0.08 per group). Failure here means the items do not measure the same construct in both groups at all.
Metric invariance holds if the chi-square difference test between metric and configural models is non-significant (p > 0.05) or ΔCFI ≤ 0.010. This allows comparison of factor variances and correlations, but not latent means.
Scalar invariance holds if the chi-square difference test between scalar and metric models is non-significant (p > 0.05) or ΔCFI ≤ 0.010. This is required before comparing latent mean scores across groups.
If scalar invariance fails for individual items but not all, partial scalar invariance (at least two invariant items per factor) still allows valid latent mean comparisons — the non-invariant items must be acknowledged.
The latent mean estimate for the non-reference group in the scalar model is the difference in latent means on the construct’s logit scale; a positive value means the non-reference group scores higher on the latent trait.

References

Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71–90. https://doi.org/10.1016/j.dr.2016.06.004

Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4–70. https://doi.org/10.1177/109442810031002

Widaman, K. F., & Reise, S. P. (1997). Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. In K. J. Bryant, M. Windle, & S. G. West (Eds.), The Science of Prevention (pp. 281–324). American Psychological Association.