What it is
Machine learning (ML) models learn flexible, non-linear relationships from data without requiring the researcher to pre-specify a functional form. In survey research, this flexibility is useful for two specific problems: imputing missing values (filling in unobserved responses using a model trained on observed responses and covariates) and predicting outcomes (estimating an outcome for observations or populations where the outcome was not directly measured, using a model trained where both predictors and outcome are available).
These are fundamentally prediction problems — the goal is minimising out-of-sample prediction error, not identifying causal effects. ML methods are well-suited to prediction tasks but should not be used to draw causal inferences without additional assumptions.
When to use it
Imputation: ML imputation is most valuable when: the proportion of missing values is large (>10%); the relationship between observed variables and the missing variable is non-linear or involves complex interactions; and the missing-at-random (MAR) assumption is plausible. Standard multiple imputation by chained equations (MICE) with linear models is often sufficient; ML versions (MICE with random forest models) outperform linear MICE when predictor-outcome relationships are complex.
Outcome prediction: ML prediction from proxies is appropriate when: a “gold standard” outcome (e.g., detailed consumption expenditure) is expensive to collect and can be predicted from cheaper variables (e.g., asset ownership, housing quality, access to services); when the researcher wants to predict the outcome for a larger comparison group than was directly measured; or when predicting future values of an outcome for targeting purposes.
How it works
ML imputation. Missingness Forest (MissForest) and MICE with random forests are the most commonly used ML imputation approaches:
- For each variable with missing values, train a random forest predicting that variable from all other variables, using only the observations where both are observed.
- Use the trained model to impute the missing values.
- Iterate across all variables with missing values until convergence. ML imputation preserves non-linear relationships and interactions without requiring the researcher to specify them. Multiple imputation (generating m ≥ 5 imputed datasets and combining estimates using Rubin’s rules) properly propagates imputation uncertainty to standard errors.
Proxy prediction (small area estimation and poverty mapping). Train an ML model on a survey sample where both the predictor variables (cheap to collect) and the outcome variable (expensive to collect) are observed. Apply the trained model to predict the outcome for a much larger dataset where only the predictor variables are available (a census, a registry, or an administrative dataset). This is the foundation of poverty mapping — predicting household consumption from census-available variables (Elbers, Lanjouw & Lanjouw, 2003).
Cross-validation. Always use k-fold cross-validation to estimate out-of-sample prediction error and to tune hyperparameters. Evaluate using appropriate metrics: RMSE and R² for continuous outcomes; AUC, precision, and recall for binary outcomes. Report cross-validated performance, not in-sample performance.
Key decisions
Model selection. Gradient boosted trees (XGBoost, LightGBM) consistently outperform random forests and linear models for structured tabular survey data. They handle missing predictors natively, require minimal preprocessing, and produce calibrated probability estimates. For imputation, random forests (via MissForest or MICE-RF) are more common and easier to implement with standard packages.
Uncertainty quantification. Point predictions from ML models do not come with standard errors. For imputation, multiple imputation with Rubin’s rules properly propagates uncertainty. For prediction, bootstrapped confidence intervals or conformal prediction intervals provide uncertainty estimates without distributional assumptions. Report uncertainty alongside predictions; point estimates without uncertainty bounds are not sufficient for research use.
Avoiding leakage. Feature leakage — using variables in the prediction model that are causally downstream of the outcome or would not be available at prediction time — produces spuriously optimistic validation performance that does not generalise. For poverty prediction, using variables that are themselves caused by poverty (not just correlated with it) is a form of leakage. Feature selection must be guided by causal logic, not just correlation.
Caveats & common mistakes
Using ML predictions as outcomes in causal analysis. Predicted outcomes from ML models absorb variance from the training features into the prediction. Using ML-predicted outcomes as the dependent variable in a treatment effect regression creates a bias: if the predictors include variables correlated with treatment (even if not causally affected by it), the treatment effect estimate picks up confounding through the prediction model. ML predictions are valid as proxies for targeting; they are not valid as outcomes in impact evaluations.
Not checking calibration. A model with good discrimination (high AUC) may still produce poorly calibrated probability estimates — predicting 80% probability for events that occur 50% of the time. Calibration plots (predicted probability vs. observed frequency across deciles) should be reported alongside discrimination metrics.
Overfitting through iterative model selection. Repeated model selection on the full training set — trying many models and selecting the best performer — produces overfitting even when cross-validation is used. Pre-register the model class and hyperparameter tuning procedure before running the analysis to prevent this.
Analysis Guide
# ML imputation using IterativeImputer (MissForest equivalent) + XGBoost prediction
import pandas as pd
import numpy as np
from sklearn.experimental import enable_iterative_imputer # noqa
from sklearn.impute import IterativeImputer
from sklearn.ensemble import RandomForestRegressor
from xgboost import XGBRegressor
from sklearn.model_selection import cross_val_score
from sklearn.metrics import mean_squared_error
# --- ML Imputation ---
df = pd.read_csv("survey_data.csv")
imputer = IterativeImputer(estimator=RandomForestRegressor(n_estimators=100,
random_state=42),
max_iter=10, random_state=42)
df_imputed = pd.DataFrame(imputer.fit_transform(df), columns=df.columns)
df_imputed.to_csv("imputed_data.csv", index=False)
# --- Proxy Prediction (poverty mapping) ---
# gold_df: sample where both outcome and predictors are observed
gold_df = pd.read_csv("gold_standard_sample.csv")
features = ["asset_index", "hh_size", "roof_material", "wall_material",
"dist_market", "land_area"]
X = gold_df[features]
y = gold_df["consumption_pcpd"]
# Cross-validated RMSE and R²
model = XGBRegressor(n_estimators=300, max_depth=5, learning_rate=0.05,
subsample=0.8, random_state=42)
cv_rmse = -cross_val_score(model, X, y, cv=5,
scoring="neg_root_mean_squared_error").mean()
cv_r2 = cross_val_score(model, X, y, cv=5, scoring="r2").mean()
print(f"CV RMSE: {cv_rmse:.2f} CV R²: {cv_r2:.3f}")
# Predict for full dataset
model.fit(X, y)
full_df = pd.read_csv("full_dataset.csv")
full_df["pred_consumption"] = model.predict(full_df[features])
full_df.to_csv("predicted_outcomes.csv", index=False)Reading the output
- For imputation: MissForest out-of-bag NRMSE above 0.30 indicates that the imputed values for continuous variables are unreliable; missing-at-random assumptions may be violated or the predictor set is too weak.
- For proxy prediction: cross-validated R² below 0.50 means the proxy variables explain less than half the variance in the gold-standard outcome — predicted values will have large uncertainty and should not be used as primary outcomes in impact evaluation.
- Cross-validated R² above 0.70 is the commonly used threshold for poverty-mapping applications where predicted consumption is used for programme targeting; below this, targeting errors will be substantial.
- Always check calibration: plot predicted vs. observed values across deciles; a well-calibrated model should fall close to the 45-degree line across the full range, not just in the middle.
- Never use ML-predicted outcomes as dependent variables in treatment effect regressions; the prediction model absorbs variation from covariates correlated with treatment, biasing the effect estimate.
Getting started
-
Scikit-learn imputation guide — Official documentation covering simple imputation (mean, median, mode) through to iterative imputation using random forest predictors. The
IterativeImputersection is the direct equivalent of MissForest in Python. -
XGBoost tutorials — Official tutorials for the XGBoost library, covering installation, basic usage, cross-validation, and feature importance. The introduction tutorial is sufficient to build a proxy prediction model for poverty mapping.
-
missForest R package (CRAN) — Vignette for the missForest R package, the standard tool for random forest-based imputation in R. Covers usage, parameter tuning, and out-of-bag error estimation for assessing imputation quality.
-
SWIFT poverty targeting methodology — Documentation for the Survey of Well-being via Instant and Frequent Tracking (SWIFT) methodology, which uses ML prediction from short proxy surveys to estimate consumption poverty. A concrete applied example of the proxy prediction approach.
-
Kaggle — Tabular ML competition notebooks — Practical reference for real-world tabular ML workflows. Winning notebooks from structured data competitions demonstrate cross-validation, feature engineering, and hyperparameter tuning patterns directly applicable to survey prediction tasks.
References
Elbers, C., Lanjouw, J. O., & Lanjouw, P. (2003). Micro–level estimation of poverty and inequality. Econometrica, 71(1), 355–364. https://doi.org/10.1111/1468-0262.00399
Stekhoven, D. J., & Bühlmann, P. (2012). MissForest — Non-parametric missing value imputation for mixed-type data. Bioinformatics, 28(1), 112–118. https://doi.org/10.1093/bioinformatics/btr597
Jean, N., Burke, M., Xie, M., Davis, W. M., Lobell, D. B., & Ermon, S. (2016). Combining satellite imagery and machine learning to predict poverty. Science, 353(6301), 790–794. https://doi.org/10.1126/science.aaf7894
van Buuren, S. (2018). Flexible Imputation of Missing Data (2nd ed.). CRC Press. https://stefvanbuuren.name/fimd/