How to check subgroup accuracy in synthetic research
By Lewsearch
To check subgroup accuracy in synthetic research, compare synthetic and human answers within the segments that drive your decision. Choose those segments before the test, show how many human respondents support each estimate, and score both answer shares and differences between groups. A matching overall result can conceal large segment errors.
How a matching topline can hide the wrong answer
Suppose you are choosing between messages A and B for current customers and prospects. Everyone must pick one of the two messages. Customers and prospects each receive half the weight in both sets of results.
Illustrative example only. Every number below is invented. This is not a Lewsearch result or a real human study.
| Audience | Human reference choosing A | Synthetic result choosing A |
|---|---|---|
| Current customers | 70% | 40% |
| Prospects | 30% | 60% |
| Overall, with equal group weights | 50% | 50% |
The topline matches exactly: (70% + 30%) / 2 and (40% + 60%) / 2 both equal 50%. Yet each segment misses by 30 percentage points. The human reference favors A among customers and B among prospects; the synthetic result reverses both choices.
The customer-minus-prospect gap also tells a different story. It is +40 percentage points in the human reference and -20 in the synthetic result. The absolute difference between those gaps is 60 points, with a reversal in direction.
The overall match would give you little help with a segment-specific message decision.
Choose the segments your decision depends on
Write down the groups you need to understand before the vendor produces its predictions. They might be current customers and prospects, frequent and occasional buyers, or two markets where you plan to use different messages.
Define each group the same way in both datasets. Check screening questions, category boundaries, geography and field dates. Ask how the synthetic respondents were assigned to behavior-based groups such as current customers. Matching a group's label or share of the audience does not establish that its answers are accurate.
Pew Research Center's September 2026 experiment assessed overall and subgroup errors, including distortions in differences between groups. It tested a digital-twin approach on public-opinion surveys and did not evaluate Lewsearch. Our recommendation is to score the groups behind the headline result before you buy.
Keep the human results hidden until predictions are locked. If earlier studies are used for calibration, reserve separate human evidence for the test. The blind-validation guide covers that setup in detail.
Ask for three views of the result
Answer shares within each priority group. Compare every agreed response option, including not-sure or none-of-these options when offered. If you use mean absolute error (MAE), state how option errors are averaged within a question and how questions are combined. Keep each group's score visible beside the topline score. Do not rely on a single average across all groups.
The differences between groups. Choose the contrasts that matter, such as the customer-minus-prospect gap for message A. Show the signed gap in each dataset and the absolute difference between the gaps. This catches reversals or exaggerated differences. Keep the share checks too: a correct gap can hide errors if both groups' estimates are shifted equally in the same direction.
The decision within each group. If the purchase is meant to support message screening, agree on the shortlisting rule before the test. Then check whether both datasets produce the same shortlist for each priority audience. Agreement on a winner and agreement on its share are separate checks. A close human result may leave the winner uncertain; do not force a pass or fail from an unstable ranking.
Report all agreed groups, including weak results. If you explore additional cells afterward, label those findings exploratory rather than quietly adding only the useful-looking ones to the scorecard.
Check the human base before treating a miss as settled
The reference survey has uncertainty too. Ask for the unweighted number of human respondents behind each subgroup estimate, how participants were recruited, how answers were weighted, and an uncertainty assessment appropriate to that design. Check whether each question went to the whole sample or only a subset.
A human study with a large total sample can still leave a priority segment poorly measured. The precision needed depends on the decision and study design, so this guide sets no universal minimum subgroup size. If the human evidence is too weak to judge the agreed tolerance, mark the comparison inconclusive.
AAPOR's disclosure standards include making unweighted subgroup sample sizes available on request. Its Transparency Initiative guidance also covers weighting, precision measures and AI use. These are useful questions for a provider; disclosure alone does not establish accuracy.
Keep human responses and simulated rows separately labeled. Generating more synthetic responses does not increase the number of people observed in the human reference. A synthetic row count alone is not a basis for attaching a human survey's sampling margin of error to the synthetic result.
Make subgroup performance part of the buying decision
Set acceptance rules before seeing the comparison. Choose tolerances for the intended use, including which segment failures would block it. There is no universal passing error rate in this guide.
If the overall result passes but a priority segment fails, do not approve the panel for decisions about that segment on the strength of the topline. If the human reference is too uncertain to judge the segment, collect better evidence before relying on it. Passing one test supports only the audiences, questions and conditions it covered.
Put this in the pilot brief
- Which decision will each priority subgroup result support?
- Are group definitions, question wording, answer options and routing aligned in both runs, with differences documented?
- What human base, weighting details and uncertainty assessment will accompany each cell?
- Will the report show every agreed group's shares, errors and signed gaps, with exclusions explained?
- Which segment-level outcomes count as pass, fail or inconclusive?
For a Lewsearch evaluation, read the Blind Challenge scope and agree on the questions, audience and scoring before the run. Keep your human results until the predictions arrive.
FAQ
Does a low topline MAE prove subgroup accuracy?
No. Opposing subgroup errors can cancel when answers are combined into a topline before MAE is calculated. Absolute errors do not cancel inside MAE itself. Score each priority group and the differences between groups separately.
Does demographic matching validate a synthetic crosstab?
No. It matches selected audience characteristics. You still need a separate human comparison, held out from calibration, to test answers within those groups.
What if a subgroup has too few human responses?
Treat that comparison as inconclusive when the human evidence cannot support the required precision. More simulated rows do not repair a weak human reference.