Answers
Are AI survey panels accurate?
Yes, within a published error, and only if the vendor shows the number. The calibrated 50-state benchmark: 7.07-point average miss on the best 80% of 1,521 scored questions (1,216 retained). 10.02 points across every scored question. MAE is an average miss in percentage points against published poll toplines. A narrower April test on 5 places with 10,000 respondents scored 7.47 points on 404 non-electoral questions. A strict held-out set sourced after training froze scored 9.97 points on non-electoral items (14 scored of 22 drawn). Open-ended quotes and focus-group dialogue are not covered by these figures.
Accurate compared with what
The calibrated 50-state benchmark: 7.07-point average miss on the best 80% of 1,521 scored questions (1,216 retained). 10.02 points across every scored question. The best 80% keeps the 1,216 lowest-error questions (80 percent of 1,521, rounded down), with ties broken by question id. The other 305 questions stay in the 10.02-point average. Every study is calibrated against published survey data before it reaches you. The file uses 200 simulated respondents per question. Coverage: coverage (650,000 census-grounded respondents · all 50 states and D.C. · 16 occupation panels.). The file is state-benchmark-per-question.csv. Full tables are on methodology.
Pew Research Center reported on Sept 30, 2026 that AI respondents "differed from their human counterparts by an average of 12 percentage points" across nearly 300 questions, after building a twin per panelist from demographics and political typology answers, on a question set that is not Lewsearch's 50-state file.
The Lewsearch Report grades forecasts that were frozen before the official number: 3 wins, 1 miss (Issue 06), and 3 awaiting a grade (Issues 03, 04, and 07). Issue 05 is a win on Guess #2.
Worked example: the 22-question held-out
On April 18, 2026, Lewsearch pre-registered 22 questions from Emerson, Marist, PPIC, USC CEPP, UT Tyler, Change Research, and the Ohio Library Council, across Ohio, Georgia, Texas, New York, and California. Eight were dropped by a fixed filter (past-election ground truths, extreme prior-delta outliers). 14 were scored. A strict held-out set sourced after training froze scored 9.97 points on non-electoral items (14 scored of 22 drawn).
| Set | n | Average miss |
|---|---|---|
| Calibrated 50-state benchmark | 1,521 scored questions | 10.02 points. 7.07 points on the best 80% (1,216 questions). |
| Narrower April test, non-electoral | 404 of 460 questions | 7.47 points |
| Pre-registered held-out, non-electoral | 14 scored of 22 | 9.97 points |
A narrower April test on 5 places with 10,000 respondents scored 7.47 points on 404 non-electoral questions. State cuts from the older April bank are on the methodology page. 7.47 points is the April non-electoral average. Write it in points, with that scope. The held-out row is a separate set, sourced after training froze.
FAQ
- Are AI survey panels accurate?
- Yes, within a published error, and only if the vendor shows the number. The calibrated 50-state benchmark: 7.07-point average miss on the best 80% of 1,521 scored questions (1,216 retained). 10.02 points across every scored question. MAE is an average miss in percentage points against published poll toplines. A narrower April test on 5 places with 10,000 respondents scored 7.47 points on 404 non-electoral questions. A strict held-out set sourced after training froze scored 9.97 points on non-electoral items (14 scored of 22 drawn). Open-ended quotes and focus-group dialogue are not covered by these figures.
- Is 7.47 points a margin of error?
- 7.47 points is the average absolute gap on the narrower April test, in percentage points. Write it in points, with its scope. The figure to quote for the product is the calibrated 50-state benchmark: 7.07 points on the best 80% of 1,521 scored questions, and 10.02 points across all of them.
- Why is the held-out number higher?
- That set is separate from the 50-state file. Those 22 questions were sourced on April 18, 2026, after training froze. 14 were scored. The non-electoral held-out miss is 9.97 points. State cuts from the older April bank are on /methodology.
- Should I use an AI panel for a published poll?
- No. Lewsearch states it is not a replacement for probabilistic human sampling. Use it to test instruments and messages first. Hire humans when the number has to be filed.
Check the work
Related answers
Canonical: https://lewsearch.com/answers/are-ai-survey-panels-accurate