Answers
Are AI survey panels accurate?
Yes, within a published error, and only if the vendor shows the number. Lewsearch's production figure is 7.47% mean absolute error on 404 ex-electoral questions of a 460-question benchmark versus real polls. The pre-registered held-out set is 9.97% ex-electoral. MAE is an average miss, not a sampling margin.
Accurate compared with what
7.47% mean absolute error on 404 ex-electoral questions of a 460-question benchmark vs real polls (Pew, Gallup, UT/Texas Politics Project, PPIC). Every published Lewsearch MAE is 5-fold cross-validated. The in-sample fit of the same calibrator is 5.79%. The site quotes the cross-validated number. Texas is 4.88%. California is 7.77%. Coverage: coverage (27 live panels · U.S. national · 4 census regions (32 audiences)). Full tables are on methodology.
Worked example: the 22-question held-out
On April 18, 2026, Lewsearch pre-registered 22 questions from Emerson, Marist, PPIC, USC CEPP, UT Tyler, Change Research, and the Ohio Library Council, across Ohio, Georgia, Texas, New York, and California. Eight were dropped by a fixed filter (past-election ground truths, extreme prior-delta outliers). Fourteen were scored. Calibrated MAE: 10.68% overall, 9.97% ex-electoral, 7.24% on political approval. That is the honest number for novel items, and it is worse than the Texas in-panel figure on purpose.
| Set | n | Calibrated MAE |
|---|---|---|
| Pooled, ex-electoral | 404 of 460 | 7.47% |
| Texas (UT/TPP), ex-electoral | 97 questions | 4.88% |
| California (PPIC), ex-electoral | 148 scored | 7.77% |
| Pre-registered held-out, ex-electoral | 14 scored of 22 | 9.97% |
| Texas political approval | 18 questions | 3.43% |
FAQ
- Are AI survey panels accurate?
- They can be, within a published error that is not a sampling margin. Lewsearch's production figure is 7.47% MAE on 404 ex-electoral questions of a 460-question benchmark. The stricter held-out set is 9.97% ex-electoral.
- Is 7.47% the same as plus or minus 7.47?
- No. MAE is the average absolute gap between predicted option shares and the published poll, in percentage points. It is not a confidence interval and must not be written with a plus-minus sign.
- Why is the held-out number worse?
- Those 22 questions were sourced on April 18, 2026, after training and calibration were frozen. 14 were scored. The gap between 4.88% on the Texas panel and 9.97% on the held-out set is the spread between familiar items and noisier edge cases.
- Should I use an AI panel for a published poll?
- No. Lewsearch states it is not a replacement for probabilistic human sampling. Use it to test instruments and messages first.
Check the work
Related
Canonical: https://lewsearch.com/answers/are-ai-survey-panels-accurate