How should you evaluate synthetic choice-based conjoint results?
By Lewsearch
Evaluate whether the system predicts reserved human choices well enough for your decision, including how preferences change when alternatives enter or leave the choice set. Require a designed tradeoff study, realistic competitors, an appropriate None option, and acceptance rules agreed before results are revealed. A synthetic model's fit to its own generated answers cannot establish agreement with people.
Establish what the supplier is delivering
In choice-based conjoint (CBC), people choose among deliberately designed sets of alternatives. Attributes are features such as price, storage, or support. Levels are the values offered for each feature, such as 100 GB or 500 GB of storage. Varying these combinations lets a fitted choice model estimate utilities: preference weights used to predict choices. A simulator then compares specified alternatives using those weights. Sawtooth's CBC introduction explains this design-and-choice foundation.
Ask where synthetic data enters:
- Does the system generate choice rows?
- Does it estimate the choice model?
- Does it operate the simulator?
Identify whether estimated weights came from human choices, generated choices, or a mixture. Request the design, generated rows, estimation method, and simulator settings relevant to the claim. Good fit at one stage does not establish accuracy at the next. A believable explanation of a choice adds no independent human evidence.
Reserve human evidence before running the test
Freeze the attributes and levels, alternative sets, audience, outcome targets, and acceptance rules before unblinding. Record and lock the generator and choice-model versions, estimation method, and all generation, estimation and simulator settings. Lock the synthetic predictions before revealing the human evaluation outcomes. Keep those outcomes out of generator tuning, choice-model estimation, run selection, and later model selection. Earlier calibration data may be allowed if documented and separate.
If someone uses the evaluation answers to improve or choose the system, the revised version needs fresh untouched evidence. This applies the separation principle in scikit-learn's guidance on data leakage. For public human results, document known or unresolved prior model exposure; withholding the link cannot establish separation.
Ask exactly what held out means:
- Questions excluded from estimation within the same human survey test internal reliability of the fitted preferences.
- Untouched human respondents and outcomes support an evaluation of generalization to those people and conditions.
- Synthetic-only holdouts test performance against generated choices.
- Observed real-world behavior addresses a different claim about choices outside the survey.
Sawtooth's glossary distinguishes internal reliability from out-of-sample respondent evaluation and real-world validity. Human survey holdouts alone do not establish sales or market-share forecasting validity.
Test changes across alternatives
Include scenarios that add or replace alternatives. Check whether the system captures changes across the whole choice set, including None when declining every offer is realistic. If the purchase context requires a choice, explain that constraint.
Use plausible combinations and bounded price and feature levels. Separate conclusions within the tested range from extrapolation. Ask which substitution assumptions the choice model and simulator impose, including whether similar products can take preference disproportionately from each other. Examine logit's proportional-substitution assumption when close substitutes matter.
A simulator produces conditional shares of preference. Distribution, awareness, availability, and other factors can separate those results from purchases, as Sawtooth's simulator guidance explains.
A baseline match can hide a consequential miss
Consider a hypothetical cloud-storage study. Every profile and number below is invented. This is an arithmetic demonstration, not a fitted conjoint model, an actual study, a client result, or Lewsearch performance.
- A: $12/month, 100 GB, email support.
- B: $20/month, 500 GB, live support.
- C: $16/month, 500 GB, email support.
- Neither: decline all listed plans.
| Scenario and source | A | B | C | Neither |
|---|---|---|---|---|
| Baseline: invented human reference | 45% | 35% | Not offered | 20% |
| Baseline: invented synthetic output | 45% | 35% | Not offered | 20% |
| Add C: invented human reference | 25% | 15% | 40% | 20% |
| Add C: invented synthetic output | 35% | 30% | 15% | 20% |
Each row totals 100%. The baseline matches exactly. With C added, the human reference puts C first, while the synthetic output keeps A first and understates C by 25 percentage points.
A baseline-only review would miss the failure that matters for this portfolio decision. The second scenario gives a reason to withhold approval for decisions about adding C. Investigate the design, generated choices, estimated model, and substitution settings before narrowing the use case or arranging another human evaluation.
These aggregate shares reveal no individual switching paths. They also provide no estimate of sales or market share.
Set an acceptance boundary you can defend
Agree which errors would change your decision: alternative-level share gaps, ranking reversals, responses to price changes, None errors, or priority-segment misses. Choose tolerances from the decision's consequences. There is no universal passing score, human sample size, or task count.
Report uncertainty in the human reference, recruitment, weighting, and segment denominators. Count human respondents, human choice rows, and synthetic rows separately. Repeated tasks from one person remain observations from that person; they do not create independent people. Small segments may leave the comparison inconclusive.
The methodological sources above explain evaluation principles. They do not establish synthetic efficacy.
Before buying, request a locked design, untouched human comparison, complete scenario-level results, uncertainty, and documented settings. Approve only the audience, ranges, alternative changes, and decision types the evidence supports. If the result misses your rule, retain human research for that decision.