Construct & Convergent Validity Diagnostics

A set of checks for validating that a new or adapted scale measures the construct it claims to measure — using correlations with conceptually related measures (convergent validity) and unrelated measures (discriminant validity) to establish the scale's meaning in a specific study context.

What it is

Construct validity is the degree to which a measurement instrument actually measures the theoretical construct it is intended to measure. A self-efficacy scale that captures self-efficacy — rather than, say, optimism or social desirability — has construct validity. Establishing construct validity requires evidence from multiple sources; it is never a single test but a pattern of evidence built over time.

The two core components are convergent validity — the scale should correlate positively and substantially with other measures of the same or closely related constructs — and discriminant validity — the scale should not correlate (or correlate only weakly) with measures of theoretically unrelated constructs. Together, they establish the nomological network of the construct: a map of which other constructs should be related to it and how.

When to use it

Construct validity diagnostics are appropriate whenever a new scale is developed or an existing scale is adapted to a new context — a new language, a new population, a new cultural setting. They are also important when a validated scale is being used as a primary outcome in an impact evaluation, since the interpretation of the treatment effect depends on the scale measuring what it claims to measure.

In development economics, construct validity is most often assessed for psychological scales (empowerment, self-efficacy, mental health) and multidimensional poverty indices. Measuring construct validity is particularly important for indices developed in one context (Western psychological scales, for example) and applied in another. Malhotra, Schuler and Boender (2002) assess convergent and discriminant validity for women’s empowerment scales in South Asia by testing correlations with theoretically related behaviours and attitudinal measures. The MIDP framework for measuring women’s empowerment in development research (Alkire et al., 2013) includes construct validity as a central criterion.

How it works

The standard validity assessment framework draws on Campbell and Fiske’s (1959) multi-trait, multi-method (MTMM) approach, adapted for practical field settings:

Convergent validity checks:

Correlate the new scale with existing validated measures of the same construct (if available in the same dataset). A self-efficacy scale should correlate positively with another validated self-efficacy measure.
Correlate with conceptually adjacent measures. Self-efficacy should correlate positively with locus of control (internal), with aspirations, and with initiative-taking behaviour.
Correlate with behavioural outcomes the construct is theorised to predict. A risk preference scale should predict investment behaviour; a trust scale should predict lending to community members.

Discriminant validity checks:

Correlate with theoretically unrelated constructs. An economic risk preference measure should not correlate strongly with verbal IQ or numeracy — if it does, it may be measuring cognitive ability rather than preferences.
Compare the correlation of the scale with a conceptually similar measure (convergent) against the correlation with a dissimilar measure (discriminant). The former should be substantially larger than the latter.

Average variance extracted (AVE). In the context of CFA, AVE measures what fraction of the variance in item responses is attributable to the latent factor rather than measurement error. An AVE > 0.5 indicates that the construct accounts for more variance than the combined measurement error. Convergent validity is supported if AVE is > 0.5.

Maximum shared squared variance (MSV). Discriminant validity is supported if the AVE for each construct exceeds the squared correlation between that construct and all other constructs. This is the Fornell-Larcker criterion.

Key decisions

What to correlate with. Identifying appropriate convergent and discriminant comparators requires knowledge of the nomological network — the theoretical relationships between the target construct and others. This should be specified before data collection. Post-hoc selection of correlates that happen to confirm validity is not credible evidence.

Strength thresholds. There are no universal thresholds for convergent and discriminant correlations. A convergent correlation of r = 0.4 with a related construct may be “good” if the constructs are only loosely related, or “poor” if they should measure the same thing. The appropriate standard depends on the theoretical relationship being tested. Reporting the full correlation matrix with a discussion of expected vs. observed correlations is more informative than applying arbitrary cutoffs.

Behavioural vs. self-report comparators. The strongest evidence for construct validity comes from correlations with observed behaviour — not with other self-report measures that may share method variance (the tendency of self-reports on similar-looking scales to correlate simply because they are both self-reported). Including at least one behavioural outcome or an independently validated scale as a comparator strengthens the validity evidence.

Cross-group validity. If the scale will be used to compare groups (treatment vs. control, men vs. women), construct validity should be assessed within each group separately. A scale that has construct validity for one group but not another is not a fair comparator across groups. Testing measurement invariance is the formal test of this.

Caveats & common mistakes

Confusing reliability with validity. A scale can be highly reliable (internally consistent, stable over time) and still lack construct validity — it measures something consistently, but not the intended construct. Reliability is a necessary but not sufficient condition for validity. The two require different evidence.

Single-study construct validity. Construct validity is established over multiple studies, populations, and methods — not in a single dataset. Evidence from a pilot validation study provides initial support; replications in new samples and new contexts strengthen the evidence base. Claiming “construct validity” on the basis of one internally consistent administration in one population is an overstatement.

Method variance. When all construct validity evidence comes from self-report measures administered in the same survey, correlations among measures may be inflated by common method variance — the tendency of all items in the same self-administered questionnaire to correlate due to shared response style, order effects, or mood. Including at least one non-self-report measure in the validity assessment mitigates this.

Nomological network violations. If a scale shows unexpectedly high correlations with constructs it should be unrelated to, or unexpectedly low correlations with constructs it should be strongly related to, this is diagnostic of poor construct validity rather than an interesting finding. It should prompt revision of the scale, not a reinterpretation of the theory.

Analysis Guide

# scale_new: the scale being validated
# related_1/2: theoretically related measures (convergent comparators - same construct, different measure)
# unrelated_1/2: theoretically unrelated measures (discriminant comparators - different construct entirely)
import pandas as pd
import numpy as np
from scipy import stats
import statsmodels.formula.api as smf

# 1. Full correlation matrix covering convergent and discriminant comparators — the pattern of high convergent and low discriminant correlations is the central evidence for construct validity
validity_vars = ["scale_new", "related_1", "related_2", "unrelated_1", "unrelated_2"]
cor_matrix = df[validity_vars].corr()
print(cor_matrix.round(2))

# 2. Fisher z-test for difference between two correlations — formally tests whether the convergent correlation (scale_new vs. related_1) is significantly larger than the discriminant correlation (scale_new vs. unrelated_1); atanh transforms r to z; dividing by the SE of the difference gives a standard normal test statistic
r_conv  = cor_matrix.loc["scale_new", "related_1"]
r_discr = cor_matrix.loc["scale_new", "unrelated_1"]
n = len(df)
z = (np.arctanh(r_conv) - np.arctanh(r_discr)) / np.sqrt(2 / (n - 3))
p = 2 * stats.norm.sf(abs(z))
print(f"z = {z:.2f}, p = {p:.3f}")

# 3. AVE (Average Variance Extracted) from standardised CFA loadings — AVE > 0.50 means the latent factor explains more than half of total item variance, supporting convergent validity; requires standardised loadings from a fitted semopy CFA (see the CFA guide)
# std_loadings = model.inspect(std_est=True).query("op == '~' and lval == 'factor1'")["Est. Std"]
# ave = (std_loadings**2).mean()
# Fornell-Larcker: AVE > squared factor correlation supports discriminant validity

# 4. Predictive validity regression — the new scale should predict a behavioural outcome that the construct is theorised to cause; a significant coefficient confirms the scale carries substantive predictive information beyond demographics
fit = smf.ols("behaviour ~ scale_new + age + C(female) + log_hh_expenditure", data=df).fit(cov_type="HC1")
print(fit.summary())

# 1. Compute full correlation matrix covering convergent and discriminant comparators — the pattern of high convergent, low discriminant correlations is the central evidence for construct validity
validity_vars <- c("scale_new", "related_1", "related_2", "unrelated_1", "unrelated_2")
cor_matrix <- cor(df[, validity_vars], use = "pairwise.complete.obs")
round(cor_matrix, 2)

library(psych)
corr.test(df[, validity_vars], use = "pairwise")   # adds significance tests to the correlation matrix

# 2. Fisher z-test — formally tests whether the convergent correlation (scale_new vs. related_1) is significantly larger than the discriminant correlation (scale_new vs. unrelated_1); atanh transforms r to z; dividing by the SE of the difference gives a standard normal test statistic
r_conv  <- cor_matrix["scale_new", "related_1"]
r_discr <- cor_matrix["scale_new", "unrelated_1"]
n <- nrow(df)
z <- (atanh(r_conv) - atanh(r_discr)) / sqrt(2 / (n - 3))
cat("z =", round(z, 2), "  p =", round(2 * pnorm(-abs(z)), 3))

# 3. AVE and Fornell-Larcker criterion — AVE is the mean squared standardised loading; AVE > 0.50 supports convergent validity; the Fornell-Larcker criterion (AVE > squared factor correlation) supports discriminant validity — requires a fitted lavaan CFA object from the CFA guide
library(lavaan)
std_loadings <- standardizedSolution(fit_ml)[standardizedSolution(fit_ml)$op == "=~", "est.std"]
ave         <- mean(std_loadings^2)
factor_cor  <- lavInspect(fit_ml, "cor.lv")
max_sq_cor  <- max(factor_cor[lower.tri(factor_cor)]^2)
cat("AVE:", round(ave, 3), " | Max sq. factor correlation:", round(max_sq_cor, 3))
cat(" Discriminant validity supported:", ave > max_sq_cor)

Reading the output

Convergent validity is supported when the correlation between the new scale and related measures is substantially positive (typically r ≥ 0.30–0.50, depending on how closely the constructs are theorised to overlap). A correlation near zero with a construct that should be highly related is evidence against convergent validity.
Discriminant validity is supported when the correlation with theoretically unrelated measures is substantially lower than the convergent correlations. There is no universal threshold, but a convergent correlation more than twice the discriminant correlation is a useful rule of thumb.
AVE > 0.50 means the factor explains more variance in its items than the combined measurement error — supporting convergent validity.
The Fornell-Larcker criterion is met when AVE exceeds the squared correlation between any pair of constructs; this is the formal discriminant validity threshold in CFA-based assessments.
A statistically significant Fisher z-test (z > 1.96) confirms that the convergent correlation is significantly larger than the discriminant correlation.

References

Alkire, S., Malapit, H., Meinzen-Dick, R., Peterman, A., Quisumbing, A., Seymour, G., & Vaz, A. (2013). Instructional guide on the Women’s Empowerment in Agriculture Index. International Food Policy Research Institute.

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. https://doi.org/10.1037/h0046016

Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50. https://doi.org/10.2307/3151312

Malhotra, A., Schuler, S. R., & Boender, C. (2002). Measuring women’s empowerment as a variable in international development. Background paper prepared for the World Bank Workshop on Poverty and Gender. World Bank.