Factorial & Multi-Arm Design

A design that assigns experimental units to multiple treatment arms — either distinct interventions or factorial combinations of independent components — enabling estimation of main effects, interaction effects, and the returns to combining treatments within a single experiment.

What it is

Multi-arm designs assign experimental units to more than one treatment condition plus a control, allowing direct comparison of alternative interventions. A factorial design is a specific multi-arm structure where units are assigned to all combinations of two or more treatment factors — enabling estimation of both the main effect of each factor independently and the interaction effect between them (whether the combined treatment produces more or less than the sum of the parts).

A simple 2×2 factorial design with factors A and B produces four arms: control (neither A nor B), A only, B only, and A+B. This design answers three questions with the same sample: what is the effect of A alone? What is the effect of B alone? What is the interaction (does A enhance or diminish the effect of B)?

When to use it

Multi-arm and factorial designs are appropriate when: the study wants to compare multiple distinct interventions; the study wants to understand whether components of a bundled intervention work independently; or the study aims to disentangle mechanisms. They are more efficient than running separate experiments for each comparison when sample size is the binding constraint.

The efficiency gain from factorial designs relative to running separate trials is substantial: a 2×2 factorial with N total observations estimates both main effects with effective sample size N/2 each (because each main effect contrast uses all observations), whereas two separate trials would need N/2 observations per trial to achieve the same precision per effect.

How it works

Arm construction. In a 2×2 factorial design, randomly assign each unit to one of four arms with equal (or specified unequal) probability. Label arms: 00 (pure control), 10 (A only), 01 (B only), 11 (A+B).

Main effects. The main effect of A is the average of (A only vs. control) and (A+B vs. B only): the average treatment effect of A, averaging over the level of B. This contrast uses all observations, giving the main effect full-sample power.

Interaction effect. The interaction is: (A+B effect) − (A effect) − (B effect) = the additional effect of combining A and B beyond what each achieves alone. In regression terms, the interaction is the coefficient on the A×B product term. Interaction effects are typically estimated with lower precision than main effects, requiring larger samples to detect.

Regression analysis. For a 2×2 factorial: outcome = β₀ + β₁·A + β₂·B + β₃·(A×B) + ε

β₁ = main effect of A; β₂ = main effect of B; β₃ = interaction (synergy if positive, substitution if negative).

Unequal arm sizes. In multi-arm designs, the control arm is often larger than treatment arms to maximise power across all comparisons (since the control is the common comparison group). The optimal allocation puts the control arm at n₀ = n₁ × sqrt(k), where k is the number of treatment arms. This is known as the optimal allocation for k+1 groups.

Key decisions

Number of arms vs. sample size. Each additional treatment arm reduces the per-arm sample size for a given total N. With 500 observations and 5 arms (including control), each arm has 100 observations — often insufficient for detecting modest effects. Before adding arms, check the MDE for the smallest arm-level comparison.

Interaction as primary vs. secondary analysis. If the interaction is the primary question, the study should be powered to detect it. Interaction effects require approximately 4× the sample size of a main effect of the same magnitude (because the interaction contrast variance is larger). Studies powered for main effects often cannot detect interactions of the same size.

Partial factorial designs. When the full factorial (all combinations) is infeasible due to operational constraints, partial factorial designs test a subset of combinations. This sacrifices estimation of some interactions but preserves estimation of main effects. Fractional factorial designs from the statistical DOE (design of experiments) literature provide systematic approaches to selecting subsets.

Caveats & common mistakes

Reporting only arm-vs-control comparisons. Multi-arm studies that report only “arm A vs. control” and “arm B vs. control” without estimating interactions miss the point of the factorial design. The key advantage is the ability to estimate interactions; if interactions are not estimated, a multi-arm design provides little advantage over separate trials.

Unequal attrition across arms. When attrition rates differ across arms, the resulting sample in each arm is a non-random subset of the randomized sample. This problem is more complex in multi-arm designs because attrition may be correlated with the specific combination of treatments received.

Spillovers across arms. If individuals in different arms interact — neighbours with different treatment statuses influence each other — comparisons between arms are not clean. Cluster randomization, with all units in a cluster assigned to the same arm, prevents this if clusters are sufficiently isolated.

Analysis Guide

import pandas as pd
import numpy as np
import statsmodels.formula.api as smf

# 1. Assign units to 4 arms with balanced sizes — shuffling a repeated 0..3 array
#    ensures each arm receives equal counts without relying on quantile cuts that
#    can produce uneven splits when ties exist in the random draw
np.random.seed(42)
arms = np.repeat([0, 1, 2, 3], int(np.ceil(len(df) / 4)))[: len(df)]
df["arm"] = np.random.permutation(arms)

# 2. Create binary indicators for the two factors — treat_A and treat_B are
#    orthogonal by construction and make the regression formula unambiguous;
#    A is "on" in arms 1 (A only) and 3 (A+B), B is on in arms 2 and 3
df["treat_A"] = df["arm"].isin([1, 3]).astype(int)
df["treat_B"] = df["arm"].isin([2, 3]).astype(int)

# 3. Factorial regression with interaction — the * operator expands to both main
#    effects plus the interaction; the interaction coefficient tests whether A and B
#    reinforce (positive) or substitute (negative) each other beyond their sum
fit = smf.ols("outcome ~ treat_A * treat_B", data=df).fit(cov_type="HC1")
print(fit.summary().tables[1])

# 4. Main effect of A averaged over both levels of B — t_test computes the linear
#    combination beta_A + 0.5 * beta_AB, which is the full-sample summary effect
#    of A and the estimand that delivers the factorial design's power advantage
lc = fit.t_test("treat_A + 0.5 * treat_A:treat_B = 0")
print(lc)

# 5. F-test on the interaction term — failing to test this wastes the main
#    analytical advantage of the factorial structure; a significant F-test means
#    the combined effect differs from the sum of individual effects
print(fit.f_test("treat_A:treat_B = 0"))

# 6. Arm-level means alongside coefficients — always present raw means so the
#    direction and magnitude of the interaction are interpretable without
#    reconstructing fitted values from regression coefficients
print(df.groupby(["treat_A", "treat_B"])["outcome"].agg(["mean", "size"]))

set.seed(42)
library(dplyr)
library(estimatr)

# 1. Assign all units to one of four arms with equal probability — rep(0:3,...) then
#    sample shuffles the arm sequence, ensuring balanced assignment without xtile
df <- df %>%
mutate(
  arm     = sample(rep(0:3, length.out = n())),
  treat_A = arm %in% c(1, 3),
  treat_B = arm %in% c(2, 3)
)

# 2. Factorial regression: treat_A * treat_B expands to main effects + interaction;
#    the interaction coefficient is the key estimand — it tests whether combining
#    A and B produces a different effect than the sum of their separate effects
fit <- lm_robust(outcome ~ treat_A * treat_B, data = df, se_type = "HC2")
summary(fit)

# 3. Average main effect of A — average the A-vs-control contrast and the A+B-vs-B
#    contrast; this uses all N observations and is the full-power summary of A's effect
coef(fit)["treat_ATRUE"] + 0.5 * coef(fit)["treat_ATRUE:treat_BTRUE"]

# 4. Arm-level means table — always report raw means alongside regression coefficients;
#    this makes the direction and magnitude of the interaction immediately interpretable
#    without requiring the reader to reconstruct fitted values from coefficients
df %>%
group_by(treat_A, treat_B) %>%
summarise(mean_outcome = mean(outcome, na.rm = TRUE), n = n(), .groups = "drop")

Reading the output

The coefficient on treat_A is the main effect of A when B = 0 (A-only arm vs. control); the treat_A × treat_B interaction coefficient is the additional effect of combining A and B beyond what each achieves alone — a positive interaction means synergy, negative means substitution.
The lincom result (or R equivalent) gives the main effect of A averaged over both levels of B — this is the estimand that uses full-sample power and is the preferred summary of A’s effect.
Arm sizes should be approximately equal; large deviations (>10%) suggest a randomization or coding error.
Detecting an interaction of the same magnitude as a main effect requires roughly 4× the sample — if the study is powered for main effects only (80% power at the MDE), the same MDE for the interaction will have ~20% power.
Always report all four arm means in a table alongside the regression coefficients — this makes the interaction pattern immediately interpretable.

References

Duflo, E., Dupas, P., & Kremer, M. (2011). Peer effects, teacher incentives, and the impact of tracking: Evidence from a randomized evaluation in Kenya. American Economic Review, 101(5), 1739–1774. https://doi.org/10.1257/aer.101.5.1739

Muralidharan, K., & Sundararaman, V. (2011). Teacher performance pay: Experimental evidence from India. Journal of Political Economy, 119(1), 39–77. https://doi.org/10.1086/659655

List, J. A., Sadoff, S., & Wagner, M. (2011). So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design. Experimental Economics, 14(4), 439–457. https://doi.org/10.1007/s10683-011-9275-7

Nair, V. N. (Ed.). (1992). Taguchi’s parameter design: A panel discussion. Technometrics, 34(2), 127–161.