What it is
Computer vision applies machine learning models to image and video data to perform tasks that humans can do visually — classifying objects, detecting and counting items, reading text, estimating area, or identifying change over time. In development research, computer vision has been used to: measure asset wealth from household photographs (Engstrom et al., 2017); estimate agricultural productivity from satellite imagery; count attendance at community meetings from video; assess infrastructure quality from street-level images; and identify child malnutrition from photographs of arm circumference measurements.
The advantage is scale: a model trained on a few thousand labelled images can classify hundreds of thousands of new images at near-zero marginal cost, enabling panel measurement or geographic coverage that would be prohibitively expensive with field enumerators.
When to use it
Computer vision is appropriate when: the outcome of interest is visually observable; collecting the outcome through conventional surveys is expensive, slow, or subject to interviewer effects; and sufficient labelled training data can be obtained (either from existing datasets or through human labelling of a training set).
Satellite-based computer vision (for agricultural outcomes, infrastructure, and night lights) is particularly mature. Household-level photo collection in CAPI surveys is increasingly feasible and has been validated in several settings for wealth measurement.
How it works
Task definition. Computer vision tasks fall into several categories: image classification (is this image class A or class B?), object detection (where are the X items in this image and how many?), semantic segmentation (classify each pixel), and optical character recognition (read text). The appropriate task depends on the outcome.
Training data. Supervised learning requires labelled training images — examples where a human has already coded the outcome. Labels might be: “household in the top/bottom wealth quartile,” “crop present/absent in this field,” “child shows signs of wasting/does not.” Labelling is expensive (requiring expert or local knowledge) but is a one-time fixed cost amortised over all images classified.
Model selection. For most field research applications, pre-trained models fine-tuned on domain-specific data outperform models trained from scratch. Options range from: (1) commercial APIs (Google Cloud Vision, AWS Rekognition, Azure Computer Vision) that require no model training but may not support all classification tasks; to (2) fine-tuned foundation models (Vision Transformers, ResNet, EfficientNet) using transfer learning; to (3) custom models trained from scratch, which require the most data and expertise.
Validation. The model’s classification accuracy must be validated on a held-out test set — images the model did not see during training — before using it for outcome measurement. Report precision, recall, and F1 score for each class. For outcomes used in regression analysis, validate that the predicted class is sufficiently correlated with the ground truth to support inference.
Measurement error implications. Computer vision produces outcome variables with classification error (the model is wrong on some images). Classification errors that are random with respect to treatment assignment produce attenuation bias in treatment effect estimates — exactly as classical measurement error does. Differential classification error (the model is more accurate for treated households) produces bias of unknown direction and should be diagnosed.
Key decisions
Build vs. buy. Commercial vision APIs are fast to deploy and require no ML expertise but charge per API call and may not provide models for specialized domains (identifying crop diseases, assessing sanitation infrastructure). Custom fine-tuning requires ML expertise and GPU compute but provides domain-specific accuracy and lower long-run cost. For research with a small number of images (<50,000), APIs are usually more cost-effective.
Image collection protocol. If images are collected specifically for computer vision analysis (rather than using satellite imagery), a consistent protocol is critical: standard distance, angle, lighting conditions, and image resolution. Variation in collection protocol introduces heteroscedastic classification error. Training enumerators on image collection is as important as training them on questionnaire administration.
Active learning. When labelled training data are scarce, active learning strategies — where the model identifies the images it is most uncertain about for human labelling — reduce the labelling burden by a factor of 3–10× relative to random labelling, while achieving comparable model accuracy.
Caveats & common mistakes
Overfitting to training data. A model with 99% accuracy on its training set may have 70% accuracy on new images from a different field season, lighting condition, or geographic area. Always validate on held-out data from the target deployment context, not just the training context.
Not disclosing image collection to respondents. Collecting photographs of households or individuals without informed consent is an ethical violation. Image collection must be disclosed in the consent process, and data storage and access protocols must comply with applicable privacy regulations.
Using computer vision as a black box. Models that cannot be inspected — where the researcher cannot explain what visual features drive classification — are difficult to validate and may capture confounding variables rather than the intended outcome. Use model interpretability tools (SHAP, Grad-CAM for visual models) to verify that the model is using the intended visual signals.
Analysis Guide
# Fine-tune a pre-trained image classifier on labelled field photos
from transformers import AutoFeatureExtractor, AutoModelForImageClassification
from transformers import TrainingArguments, Trainer
from datasets import load_dataset
import numpy as np
from sklearn.metrics import f1_score, classification_report
# Load labelled image dataset (ImageFolder format)
dataset = load_dataset("imagefolder", data_dir="labelled_photos/",
split={"train": "train", "test": "test"})
extractor = AutoFeatureExtractor.from_pretrained("microsoft/resnet-50")
model = AutoModelForImageClassification.from_pretrained(
"microsoft/resnet-50",
num_labels=len(dataset["train"].features["label"].names),
ignore_mismatched_sizes=True)
def preprocess(batch):
return extractor(batch["image"], return_tensors="pt")
dataset = dataset.map(preprocess, batched=True)
# Train
args = TrainingArguments(output_dir="cv_model", num_train_epochs=5,
per_device_train_batch_size=16, evaluation_strategy="epoch")
trainer = Trainer(model=model, args=args,
train_dataset=dataset["train"], eval_dataset=dataset["test"])
trainer.train()
# Evaluate on held-out test set
preds = trainer.predict(dataset["test"])
y_pred = np.argmax(preds.predictions, axis=1)
y_true = preds.label_ids
print(classification_report(y_true, y_pred,
target_names=dataset["train"].features["label"].names))Reading the output
- F1 score below 0.70 on the held-out test set indicates the model is not reliable enough for use as an outcome variable; collect more labelled images or simplify the classification task before proceeding.
- Precision and recall should be reported per class: a class with recall below 0.60 will systematically undercount that outcome, introducing attenuation bias in treatment effect estimates.
- If test-set accuracy differs by more than 10 percentage points from training accuracy, the model is overfitting; apply dropout regularisation or reduce model depth.
- Validate classification accuracy separately for treated and control households; differential accuracy above 5 percentage points by treatment status indicates the model may produce biased treatment effect estimates.
- For regression analysis using computer-vision-derived outcomes, compute intraclass correlation between model predictions and independent ground-truth labels on a 5–10% subsample; correlations below 0.70 indicate excessive measurement error.
Getting started
-
fast.ai Practical Deep Learning for Coders — Free course covering image classification from scratch using PyTorch. Lesson 1 alone is enough to build a working classifier on labelled field photos. No prior ML background required.
-
Hugging Face Image Classification tutorial — Step-by-step guide to fine-tuning a pre-trained vision model on a custom dataset using the Transformers library. Covers training, evaluation, and inference.
-
Roboflow — Browser-based tool for labelling images, augmenting training data, and exporting datasets in formats compatible with PyTorch, TensorFlow, and YOLO. Free tier available. Reduces the labelling workflow from days to hours.
-
Google Cloud Vision API quickstart — Documentation for the commercial API that requires no model training. Covers label detection, object localisation, and text recognition. Pay-per-call pricing; free tier for low volumes.
-
PyTorch Transfer Learning tutorial — Official tutorial on fine-tuning ResNet on a small labelled dataset. Covers the exact workflow for adapting a pre-trained model to a new classification task.
References
Engstrom, R., Hersh, J., & Newhouse, D. (2017). Poverty from Space: Using High-Resolution Satellite Imagery for Estimating Economic Well-Being. World Bank Policy Research Working Paper 8284. https://doi.org/10.1596/1813-9450-8284
Jean, N., Burke, M., Xie, M., Davis, W. M., Lobell, D. B., & Ermon, S. (2016). Combining satellite imagery and machine learning to predict poverty. Science, 353(6301), 790–794. https://doi.org/10.1126/science.aaf7894
Xie, M., Jean, N., Burke, M., Lobell, D., & Ermon, S. (2016). Transfer learning from deep features for remote sensing and poverty mapping. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1).