How do you check whether a synthetic concept ranking is stable?

Ask what the winning concept actually won

A synthetic screen can help a team decide what to investigate next. But a first-place label leaves important questions unanswered: was the concept always listed first, did it receive a richer description, and would the recommendation change under another reasonable version of the questionnaire?

Pew Research Center explains that wording, response choices and question order can affect human survey answers. Its guidance also distinguishes unordered choices, which can be randomized, from ordered rating scales, whose sequence carries meaning. That is human-survey guidance, not evidence that a particular synthetic panel reproduces the same effects. Pew questionnaire guidance.

In a 2024 study published in Transactions of the Association for Computational Linguistics, Lindia Tjuatja and colleagues tested nine language models on modified survey questions. They found sensitivity to changes that prior human research had not found significant effects for. Their comparison used documented human response patterns rather than newly recruited people answering every modified question. The study did not evaluate Lewsearch. Tjuatja and colleagues’ study.

The procedure below is our proposed buyer diagnostic. Agree its scope and cost before commissioning the screen; it is not an established certification or a universal passing standard.

Separate presentation checks from changes to the offer

First write a short specification for each concept: audience, benefit, price, conditions, channel and intended use. Review the stimuli together so one concept does not quietly gain an endorsement, an extra benefit or more explanatory detail. Preserve meaningful differences between the actual offers.

Classify each planned variation before looking at results. That prevents an inconvenient ranking change from being dismissed afterward as a different study.

• Presentation order: rotate or randomize unordered concept positions and neutral labels. Keep a record mapping each displayed label back to the actual concept. Keep ordered rating categories in a meaningful sequence.
• Equivalent wording: have a researcher review alternate instructions for the same meaning. Keep the offer, time frame, question, response options and scoring intact. Treat apparent equivalence as something to check with people, not something a model can certify for itself.
• A different proposition: changing the price, adding a guarantee, revealing a brand or removing a condition changes what is being evaluated. Report that as a separate concept or research question rather than a pure stability check.
• A different task: rating one concept in isolation and choosing between concepts side by side answer different questions. Either can fit a research goal, but switching between them is not merely reordering the same test.

Lock a small comparison plan before the run

Choose the primary outcome first: for example, stated preference between two offers. Keep comprehension and objections as separate outcomes instead of choosing whichever measure makes a favored concept look strongest.

Specify the audience composition, weighting, questionnaire, scoring, exclusions and generation settings that will stay fixed. Record the model or service version where available. Ask whether each condition starts without memory of the other conditions; otherwise an earlier exposure may become an additional input.

For a paired choice, balance which concept appears first. For a wording check, vary only the reviewed instruction while retaining the same concepts and answer categories. If possible, reuse the same simulated profile set so changing audience composition is not mixed into the comparison. Document any differences that cannot be held fixed.

Include planned repeats of an unchanged setup if you need to assess variation between runs. A repeated run and a changed questionnaire address different sources of variation. Keep both visible. More generated runs do not create more independently observed human evidence.

Decide in advance what result would alter the shortlist, and what you will do if planned checks disagree. There is no universal number of repeats or percentage-point threshold that makes a synthetic ranking safe to use.

A reversal changes the next research step

Illustrative example: a retailer is comparing a delivery offer with a pickup offer. These invented results describe a forced choice between exactly two concepts, with the same profiles, offers and scoring. There are no other answer categories. They are not client results or measured Lewsearch performance.

When delivery is displayed first, its synthetic choice share is 54% and pickup receives 46%. When pickup is displayed first, delivery receives 47% and pickup receives 53%. Delivery’s margin changes from plus 8 to minus 6 percentage points; its own share changes by 7 points.

The proposed shortlist rule was to advance a sole winner only if the same concept led in both planned orders. Under that rule, both concepts advance to a human comparison. The team reports the reversal and the two distributions. It does not select the favorable order or average the conditions and call delivery a reliable winner.

The example alone cannot identify whether position, generation variability or an implementation problem caused the reversal. Check label mapping and the run records, then use an appropriate comparison to investigate. Even a ranking that stays unchanged across every planned check still needs human validation for the intended use.

Ask for these items in the delivery

AAPOR’s disclosure standards call for the wording and presentation of measurement tools and preceding context that could influence reported responses. They also cover processing and study limitations. This synthetic comparison plan is our proposed extension for this buying decision, not an AAPOR approval of a method. AAPOR disclosure standards.

• The exact stimuli, instructions, answer options, presentation schedule and concept-label mapping for every condition.
• The complete planned result set, including failed runs, exclusions and retries. Separate exploration after the initial results from the pre-agreed evaluation.
• Answer shares and the margin between finalists for each condition, plus any priority-segment results agreed in advance. Explain the denominator and weighting.
• A record of what stayed fixed and what changed, including audience selection, model or service version, generation settings and prior exposure to other concepts.
• A distinction between variation across generated runs and uncertainty about human preferences. Require the provider to explain any interval or effective-sample-size claim.
• The recommendation under the original decision rule, with any unresolved disagreement and the next human or behavioral evidence needed.

Match the follow-up evidence to the decision

Use human pretesting to check whether the supposedly equivalent descriptions mean the same thing to the target audience. Use a human concept study to evaluate stated preference among the finalists, with its own order handling and uncertainty. Use an appropriate live experiment when the decision requires observed clicks, purchases or another behavior. These are separate outcomes.

A model-generated explanation of why a ranking changed can suggest a question to investigate. It does not establish the cause of the change or show how real customers interpreted the offer.

For a Lewsearch pilot, agree which presentation checks and human comparisons are included before the run. Book a demo and use the blind-validation guide to define the wider evaluation. The concept-testing overview explains the distinction between a stated-preference screen and a sales forecast.