Best-Worst Scaling — Case 1 (Object Case)

What it is

Best-worst scaling (BWS) elicits preferences over a set of items by asking respondents to make two judgments per task: which item in the displayed subset is best and which is worst. Compared to rating scales (which suffer from response-style heterogeneity) or full ranking tasks (which become cognitively demanding beyond five or six items), BWS asks for only two picks per task and produces clean preference data.

There are two ways to summarise the results. The simple best-minus-worst count score is an ordinal ranking — it tells you which items are preferred to which, but not by how much. The MaxDiff model (Marley & Louviere, 2005) is a random-utility model fit by conditional logit; its coefficients are interval-scale utilities, and after exponentiating and normalising they become shares of preference on a ratio scale (an item with share 0.20 is twice as likely to be picked as best as one with share 0.10, holding the comparison set fixed).

The method was developed by Finn and Louviere (1992). It has three variants. Case 1 (object case) evaluates a fixed list of items — policy priorities, health conditions, brand attributes, organisational values, agricultural practices — by having respondents pick best and worst from rotating subsets. This guide covers Case 1. Case 2 (profile case) and Case 3 (multi-profile case) are closer to conjoint analysis and use different design and estimation logic; the R packages support.BWS2 and support.BWS3 cover them.

When to use it

BWS is appropriate when the research question is to rank or prioritise a list of items. The lower edge is roughly six items — below that, direct ranking is fine. There is no hard upper limit, but past 15–20 items the design becomes long enough to fatigue respondents. Typical applications:

• Community prioritisation of health conditions, infrastructure needs, or public services
• Ranking attributes of an agricultural technology, a job, or a product
• Policy preference ordering across a menu of programme options
• Brand or product attribute importance in market research
• Organisational values prioritisation
• Symptom severity weights in clinical research

It works well in low-literacy and oral interview settings because the task — “which of these is most important to you, and which is least?” — is intuitive and requires no numerical rating. Flynn et al. (2007) document this use in health economics. Note that BWS is not the same as conjoint: BWS Case 1 ranks single-attribute items, while conjoint estimates how respondents trade off the levels of multiple attributes described in each profile. Choose BWS when the question is “which of these matters most”; choose conjoint when it is “how do you weigh quality against price against waiting time”.

BWS is less suitable when the items are not meaningfully comparable to respondents, or when individual-level heterogeneity is the primary target — in that case, plan to use hierarchical Bayes MaxDiff (HB-MaxDiff) rather than aggregate count scores.

How it works

The researcher defines a list of J items. In each task, a subset of k items is shown and the respondent picks best and worst. The sequence of tasks is constructed as a balanced incomplete block design (BIBD), so that every item appears the same number of times (r) across the design and every pair of items co-appears the same number of times (λ). Balance matters: an unbalanced design lets exposure rates contaminate preference rankings.

A BIBD’s parameters must satisfy two identities — r(k − 1) = λ(J − 1) and bk = Jr, where b is the total number of tasks — and a valid design only exists for specific (J, k) combinations. A short lookup table for common configurations:

For other (J, k) combinations, R’s crossdes::find.BIB() searches for a valid design and crossdes::isGYD() verifies it.

The count score and the MaxDiff utility — two different estimators. The best-minus-worst count score for each item is the number of times it was chosen best minus the number of times it was chosen worst, divided by the number of appearances. It is an ordinal ranking — a score of 0.30 versus 0.15 tells you the first item is preferred, not “twice as preferred.” Under balanced designs, the count score rank correlates above 0.95 with MaxDiff utilities (Marley & Louviere, 2005; Lipovetsky & Conklin, 2014), which is why for many practical purposes it is sufficient.

The MaxDiff model (Marley & Louviere, 2005) is a random-utility extension. Best and worst choices are modelled as the outcome of a latent preference process and the parameters are estimated by conditional logit. The raw coefficients are interval-scale utilities (identified up to a location shift, with one item held as the reference at 0). After exponentiating and normalising to sum to 1, the resulting shares of preference are on a ratio scale: an item with share 0.20 is twice as likely to be picked as best as one with share 0.10, holding the comparison set fixed.

Identification assumptions for MaxDiff. Five conditions are required for the MaxDiff utility to mean what the analyst thinks it means: respondents are choosing to maximise utility within each task; the unobserved error terms are i.i.d. Gumbel (the conditional-logit assumption); the independence of irrelevant alternatives (IIA) holds — the relative odds of picking one item over another do not depend on what other items are in the same task; preferences are homogeneous across respondents (relaxed by mixed logit or HB-MaxDiff); and respondents do not learn or fatigue across tasks. IIA is the most consequential and the most often violated; the Hausman-McFadden test (mlogit::hmftest()) is the standard diagnostic.

The R package support.BWS (Aizaki) handles the design, dataset construction, count score, and prepares data for conditional-logit fitting via survival::clogit(). The bwsTools package (White, 2021) is a more modern alternative with individual-level scoring built in.

Key decisions

Number of items (J). Eight to fifteen is the most common range. Industry MaxDiff (Sawtooth, marketing applications) routinely runs 15–30 items. The constraint is the BIBD: as J grows, the number of tasks (b) needed to keep the design balanced grows faster. Check feasibility against the lookup table in “How it works” before committing.

Subset size (k). Four or five items per task is standard. Fewer than four reduces discrimination per choice; more than six increases cognitive load.

Number of tasks per respondent. Most BIBDs land in the 10–20 task range. Partial designs reduce this but cost balance. Pilot to confirm respondents can complete the full sequence without degraded data quality on later tasks.

Sample size. For stable aggregate count scores and MaxDiff utilities, 300–500 respondents is the standard rule of thumb in the Sawtooth and marketing literature (Cohen, 2003; Orme, 2009). For individual-level estimation via HB-MaxDiff, plan for at least 15 task-observations per item per respondent — meaning either a long task sequence per respondent or fewer items per design.

BW count score vs. MaxDiff. The count score is sufficient when the research question is “rank these items by preference, by group”. MaxDiff (or HB-MaxDiff for individual-level scores) is the better tool when the question requires ratio-scale shares of preference, covariate-adjusted comparisons, or simulations of choice probabilities under hypothetical scenarios. Most field applications can stop at the count score; switch to MaxDiff when the additional structure earns its keep.

Item construction. Items should be comparable in abstraction level and scope. Mixing very different scales of importance (e.g., “access to clean water” against “access to mobile coverage”) can produce floor effects where the less important items end up almost always worst. Pilot the spread of BW scores; rewrite or drop items that saturate at the extremes (|score| above ~0.8 on the standardised rate).

Caveats & common mistakes

Position effects. Items shown in certain positions within a task — first, last, or centre — can be chosen more often than their true rank warrants. The direction and magnitude depend on display modality (paper ballot, tablet, oral interview) and are not universal. Randomise item display order within each task. After the data come in, check for position effects ex post by regressing best/worst selection on display position.

Statistical uncertainty on the count score. A point ranking with no error bars hides whether adjacent items are actually distinguishable. The count score’s standard error is not straightforward (best and worst counts are correlated, and respondents contribute multiple correlated task observations). The cleanest fix is a cluster bootstrap by respondent — resample respondents with replacement, recompute scores, take 2.5/97.5 percentiles per item. Items whose 95% CIs overlap should not be reported as differently ranked.

BIBD generation. Building a valid BIBD is not trivial. Use crossdes::find.BIB() (R) to search for a valid design and crossdes::isGYD() to verify it before fieldwork. Verify both parameter identities — r(k − 1) = λ(J − 1) and bk = Jr — hold.

Aggregation masks heterogeneity. The aggregate count score and the MaxDiff utilities both average over respondents. When there are subgroups with systematically different priorities — by gender, wealth, region, treatment arm — report subgroup-level scores alongside the aggregate. If individual-level scores are the primary target, use HB-MaxDiff (Sawtooth, bwsTools::e_score(), or cmdstanpy/PyMC).

Testing IIA. The MaxDiff model assumes independence of irrelevant alternatives — adding or removing an item from a task does not change the relative odds of choosing between any two remaining items. Items that are complements or substitutes can violate this. Run the Hausman-McFadden test via mlogit::hmftest(), or compare the MaxDiff fit to a mixed-logit specification and check whether the IIA-relaxed coefficients move. A material difference is evidence against IIA, and HB-MaxDiff or mixed logit is the appropriate response.

Scope of this guide. This is Case 1 (object case). Case 2 (profile case, single profile rated for best and worst attribute) and Case 3 (multi-profile case, multiple full profiles compared) have different design and estimation logic. Use support.BWS2 or support.BWS3 in R for those.

Analysis Guide

import pandas as pd
import numpy as np

# Data in long format: one row per displayed item, per task, per respondent.
# Items not shown in a task have no row. Required columns: id (respondent),
# task, item_id, best (1/0), worst (1/0).

# 1. Aggregate counts per item across the full sample — best and worst sums tell
#    you how often each item was picked at each extreme; appearances is the total
#    rows for that item, which under a balanced BIBD equals r times n_respondents
agg = df.groupby('item_id').agg(
  times_best=('best', 'sum'),
  times_worst=('worst', 'sum'),
  appearances=('best', 'count')).reset_index()

# 2. Standardised BW count score — best minus worst, divided by appearances;
#    this puts every item on the same -1 to +1 scale regardless of how often it
#    appeared in the design
agg['bw_score_rate'] = (agg['times_best'] - agg['times_worst']) / agg['appearances']
ranking = agg.sort_values('bw_score_rate', ascending=False)

# 3. Cluster bootstrap by respondent for 95% CIs — point rankings are uninformative
#    without uncertainty; resample respondents with replacement, recompute scores,
#    take the 2.5/97.5 percentiles; items whose CIs overlap are not distinguishable
def boot_scores(df, B=1000):
  ids = df['id'].unique()
  out = []
  for _ in range(B):
      sample_ids = np.random.choice(ids, size=len(ids), replace=True)
      s = pd.concat([df[df['id'] == i] for i in sample_ids])
      a = s.groupby('item_id').agg(b=('best', 'sum'), w=('worst', 'sum'), n=('best', 'count'))
      out.append(((a['b'] - a['w']) / a['n']).rename(_))
  return pd.concat(out, axis=1)

boots = boot_scores(df, B=1000)
ci = boots.quantile([0.025, 0.975], axis=1).T
ranking = ranking.set_index('item_id').join(ci.rename(columns={0.025: 'ci_lo', 0.975: 'ci_hi'}))
print(ranking[['bw_score_rate', 'ci_lo', 'ci_hi']])

# 4. MaxDiff in Python — no widely-used dedicated package. For aggregate conditional
#    logit, use xlogit. For hierarchical Bayes MaxDiff (individual-level utilities),
#    use cmdstanpy or PyMC. R is the mature path — see the R tab.

library(crossdes)        # find.BIB, isGYD — BIBD construction and verification
library(support.BWS)     # Aizaki, CRAN — Case 1 BWS workflow
library(survival)        # clogit — conditional logit for MaxDiff

# 1. Build the BIBD before fieldwork — find.BIB searches for a valid design
#    given J items, b tasks, k items per task; isGYD verifies the balance
#    properties (each item appears r times, every pair co-appears lambda times)
J <- 10; b <- 15; k <- 4
des <- find.BIB(trt = J, b = b, k = k)
stopifnot(isGYD(des))

# 2. Build the questionnaire from the design — bws.questionnaire prints task-by-task
#    item lists ready to copy into a survey instrument; respondent IDs are then
#    used to merge their responses back to this design after field collection
item.names <- paste0("item", 1:J)
bws.questionnaire(choice.sets = des, design.type = 2, item.names = item.names)

# 3. After fieldwork, reshape respondent data into the long format support.BWS
#    expects — bws.dataset constructs (RES, ITEM*, STR) for the conditional-logit
#    fit; response.type = 1 means best/worst are recorded as row numbers in the
#    choice set; model = "maxdiff" prepares the data for MaxDiff (vs marginal or
#    sequential parameterisations)
ds <- bws.dataset(
data = respondent_wide,          # one row per respondent, B1, W1, B2, W2, ...
response.type = 1,
choice.sets = des,
design.type = 2,
item.names = item.names,
id = "id",
response = c(paste0("B", 1:b), paste0("W", 1:b)),
model = "maxdiff")

# 4. BW count score — ordinal aggregate ranking, sufficient for most field uses;
#    rank-correlates above 0.95 with the MaxDiff utilities under balanced designs
scores <- bws.count(ds)
print(scores)

# 5. MaxDiff via conditional logit with respondent clustering — clogit's cluster()
#    accounts for the within-respondent correlation across tasks (the conditional-
#    logit residuals are correlated within respondent); one item is held at 0
#    as the reference, so coefficients are interval-scale utilities relative to it
fit <- clogit(RES ~ ITEM1 + ITEM2 + ITEM3 + ITEM4 + ITEM5 + ITEM6 + ITEM7 +
            ITEM8 + ITEM9 + strata(STR) + cluster(ID), data = ds)
summary(fit)

# 6. Convert utilities to ratio-scale shares of preference — exponentiate and
#    normalise to sum to 1 across items; share 0.20 is twice as likely to be picked
#    as best as share 0.10, holding the comparison set fixed
util  <- c(0, coef(fit))    # reference item utility = 0
shares <- exp(util) / sum(exp(util))
data.frame(item = item.names, utility = util, share = shares)

# Alternative: bwsTools (White, 2021) provides cleaner ergonomics, individual-
# level scoring via e_score(), and an aggregate path that does not require
# manual long-format reshaping. install.packages("bwsTools")

SurveyCTO / XLSForm

Generate the BIBD before fieldwork (in R using crossdes::find.BIB or in Python by reading a pre-built design file). Encode each task as a fixed set of items with a select_one question for best and a second select_one for worst.

For best/worst mutual exclusion, use choice_filter rather than constraint. choice_filter removes the previously-selected item from the choice list shown for the second question, which is the correct UX — constraint fires on field exit and produces an error message after the fact. In the choices sheet, add a column (e.g., filter) holding the choice name, and on the worst question set choice_filter to filter != ${best_choice}.

Randomise item display order within each task using randomize(choice_list_name, ${random_seed}) in the choice_filter column. (The Sawtooth-style jr:choice-name-map is not a randomisation primitive in SurveyCTO/ODK.)

Store the items-displayed-per-task mapping in the submission. Either (a) build a calculate field per task with the items concatenated (concat(${item1}, ',', ${item2}, ...)), or (b) load the design from a pulldata server file keyed by task ID. Without this, the analyst cannot reconstruct which items appeared together.

Reading the output

• bw_score_rate (best minus worst, divided by appearances) is the primary ranking metric on a −1 to +1 scale. Higher values = stronger preference. Negative values = items respondents actively avoid.
• The count score is ordinal. A score of 0.30 is preferred to 0.15 but not “twice as preferred”. For ratio-scale claims use the MaxDiff shares.
• |bw_score_rate| > 0.8: the item is effectively saturated. Inspect whether it is substantively dominant or whether labelling or position effects are driving the pattern; consider rewriting in a future design.
• Items whose 95% bootstrap CIs overlap should not be reported as differently ranked. Adjacent rankings without CI separation are not statistically distinguishable.
• MaxDiff shares of preference (exponentiated normalised utilities) are on a ratio scale: share 0.20 is twice as likely to be picked as best as share 0.10, holding the comparison set fixed. The raw utility coefficients are interval-scale and identified up to the location of the reference item — they are not directly comparable across studies without re-normalisation.
• Items with bw_score_rate near zero are in the middle of the preference distribution, not necessarily unimportant.
• If one or two items dominate the ranking, check whether the result is substantive or whether saturation or position effects are responsible.

References

Aizaki, H. (2022). support.BWS: Tools for Case 1 Best-Worst Scaling [R package]. CRAN. https://CRAN.R-project.org/package=support.BWS

Auger, P., Devinney, T. M., & Louviere, J. J. (2007). Using best–worst scaling methodology to investigate consumer ethical beliefs across countries. Journal of Business Ethics, 70(3), 299–326. https://doi.org/10.1007/s10551-006-9112-7

Cohen, S. H. (2003). Maximum difference scaling: Improved measures of importance and preference for segmentation. Sawtooth Software Research Paper Series. https://sawtoothsoftware.com/resources/technical-papers

Finn, A., & Louviere, J. J. (1992). Determining the appropriate response to evidence of public concern: The case of food safety. Journal of Public Policy & Marketing, 11(2), 12–25. https://doi.org/10.1177/074391569201100202

Flynn, T. N., Louviere, J. J., Peters, T. J., & Coast, J. (2007). Best–worst scaling: What it can do for health care research and how to do it. Journal of Health Economics, 26(1), 171–189. https://doi.org/10.1016/j.jhealeco.2006.04.002

Hess, S., & Palma, D. (2019). Apollo: A flexible, powerful and customisable freeware package for choice model estimation and application. Journal of Choice Modelling, 32, 100170. https://doi.org/10.1016/j.jocm.2019.100170

Lipovetsky, S., & Conklin, M. (2014). Best-Worst Scaling in analytical closed-form solution. Journal of Choice Modelling, 10, 60–68. https://doi.org/10.1016/j.jocm.2014.02.001

Lipovetsky, S., & Conklin, M. (2015). MaxDiff priority estimations with and without HB-MNL. Advances in Adaptive Data Analysis, 7(1–2). https://doi.org/10.1142/S1793536915500028

Louviere, J. J., Flynn, T. N., & Marley, A. A. J. (2015). Best-worst scaling: Theory, methods and applications. Cambridge University Press. https://doi.org/10.1017/CBO9781107337855

Marley, A. A. J., & Louviere, J. J. (2005). Some probabilistic models of best, worst, and best–worst choices. Journal of Mathematical Psychology, 49(6), 464–480. https://doi.org/10.1016/j.jmp.2005.05.003

Mühlbacher, A. C., Kaczynski, A., Zweifel, P., & Johnson, F. R. (2016). Experimental measurement of preferences in health and healthcare using best-worst scaling: An overview. Health Economics Review, 6, 2. https://doi.org/10.1186/s13561-015-0079-x

Orme, B. (2009). MaxDiff analysis: Simple counting, individual-level logit, and HB. Sawtooth Software Research Paper Series. https://sawtoothsoftware.com/resources/technical-papers

White, M. H. II. (2021). bwsTools: An R package for case 1 best-worst scaling. Journal of Choice Modelling, 39, 100289. https://doi.org/10.1016/j.jocm.2021.100289