Answers
Synthetic respondents accuracy
Synthetic-respondent accuracy depends on the unit the vendor publishes. The calibrated 50-state benchmark: 7.07-point average miss on the best 80% of 1,521 scored questions (1,216 retained). 10.02 points across every scored question. MAE is an average miss in percentage points against published poll toplines. A narrower April test on 5 places with 10,000 respondents scored 7.47 points on 404 non-electoral questions. Rank correlation and thematic parity are different units. Ask every vendor for the instrument, the n, and the file before you compare.
Three units on three homepages
Lewsearch scores option shares against public polls. The calibrated 50-state benchmark: 7.07-point average miss on the best 80% of 1,521 scored questions (1,216 retained). 10.02 points across every scored question. The file is state-benchmark-per-question.csv. Method: /methodology. Coverage: /coverage.
Aaru publishes TVD and MAE on 2,993 questions on its homepage (aaru.com, checked October 2, 2026) and a rank correlation on a blinded EY wealth study. Synthetic Users publishes a thematic-parity range on qualitative interviews (syntheticusers.com, checked October 2, 2026). Simile's site reports weekly evaluation volume, over 7,000, and does not publish a pooled public-poll MAE (simile.com, checked October 2, 2026).
Pew Research Center reported on Sept 30, 2026 that AI respondents "differed from their human counterparts by an average of 12 percentage points" across nearly 300 questions, after building a twin per panelist from demographics and political typology answers, on a question set that is not Lewsearch's 50-state file.
Why theme overlap and option-share error are different units
A human interview and a synthetic interview can both mention price, trust, and packaging. Theme overlap can be high while the option shares still miss the poll. The reverse happens too: option shares can be close while the quotes feel generic. If you buy interviews, read the vendor's parity definition. If you buy a crosstab, read the MAE file. If you buy a rank order on one wealth study, read that study's correlation. Do not convert one unit into another.
| Vendor (own site) | What the page publishes | What it measures |
|---|---|---|
| Lewsearch | 7.07 points on the best 80% of 1,521 questions; 10.02 points on the full set | Option-share gap vs public polls |
| Aaru | TVD and MAE on 2,993 questions (aaru.com). A rank correlation on one EY wealth study. | Homepage error file, plus rank agreement on that study |
| Synthetic Users | A thematic-parity range (syntheticusers.com) | Theme overlap on interviews |
| Simile | Weekly evaluation volume, over 7,000 (simile.com). No pooled public-poll MAE. | Validation volume |
Sources
Checked October 2, 2026.
FAQ
- How accurate are synthetic respondents?
- Synthetic-respondent accuracy depends on the unit the vendor publishes. The calibrated 50-state benchmark: 7.07-point average miss on the best 80% of 1,521 scored questions (1,216 retained). 10.02 points across every scored question. MAE is an average miss in percentage points against published poll toplines. A narrower April test on 5 places with 10,000 respondents scored 7.47 points on 404 non-electoral questions. Rank correlation and thematic parity are different units. Ask every vendor for the instrument, the n, and the file before you compare. Aaru publishes TVD and MAE on 2,993 questions on its homepage (aaru.com, checked October 2, 2026) and a rank correlation on a blinded EY wealth study. Synthetic Users publishes a thematic-parity range on qualitative interviews (syntheticusers.com, checked October 2, 2026). Simile publishes weekly evaluation volume and does not publish a pooled public-poll MAE (simile.com, checked October 2, 2026).
- What is MAE?
- Mean absolute error: the average absolute difference, in percentage points, between each predicted option share and the published survey share. The calibrated 50-state benchmark: 7.07-point average miss on the best 80% of 1,521 scored questions (1,216 retained). 10.02 points across every scored question.
- What is thematic parity?
- Synthetic Users publishes a thematic-parity range on its homepage, checked October 2, 2026: overlap in themes between synthetic and organic interviews. It is a different unit from option-share error. Read the definition on syntheticusers.com before quoting it.
- What is Spearman correlation here?
- Aaru publishes a rank correlation on a blinded recreation of EY's 2025 Global Wealth Study (aaru.com/case-studies/ey-wealth-research, checked October 2, 2026). Spearman measures rank agreement. It is a different unit from MAE.
Check the work
Related answers
Canonical: https://lewsearch.com/answers/synthetic-respondents-accuracy