How accurate is Lewsearch?

On the calibrated 50-state benchmark, the average miss is 7.07 points on the best 80% of 1,521 scored questions, and 10.02 points across all of them, 200 simulated respondents per question. A narrower April test on 5 places with 10,000 respondents scored 7.47 points on 404 non-electoral questions. That April test is 5-fold cross-validated. Texas is 4.88 points (UT/Texas Politics Project, April bank, non-electoral). California is 7.77 points (PPIC, April bank). A strict held-out set sourced after training froze: 9.97 points on non-electoral items (14 scored of 22 drawn).

Why do some pages mention 404 of 460 questions?

We score all 460 questions in the April bank. 56 of them are election-margin items. Turnout noise dominates the error on those. A narrower calibrated benchmark on 5 places, n=10,000, missed by 7.47 points on 404 non-electoral questions of a 460-question bank. Election results stay in the tables and the file.

What independent panels is Lewsearch benchmarked against?

UT/Texas Politics Project, PPIC California, a Gallup/canvass tri-metro set, and a pre-registered held-out drawn from Emerson, Marist, PPIC, USC CEPP, UT Tyler, Change Research, and Ohio Library Council.

Methodology · Lewis 1.0

How we measure accuracy

Q: Are the accuracy numbers out-of-sample?

The April five-place test is 5-fold cross-validated. The production correction on that test is fit on four folds and scored on the fifth. The 50-state book is a calibrated, full-set retrospective score. The held-out set was sourced after training froze.

7.07-point average miss on the best 80% of 1,521 benchmark questions. 10.02 points across the full set.

We compare simulated answer shares with published human results. The best-80% figure averages the 1,216 lowest-error questions in the 1,521-question benchmark; the full-set figure includes every scored question.

Read the 50-state rules 50-state check

7.07 pts

Best 80%

1,216 of 1,521 · 10.02 points on the full set

7.47 pts

April test

A narrower April test on 5 places with 10,000 respondents scored 7.47 points on 404 non-electoral questions.

4.88 pts

Texas, April bank

UT/Texas Politics Project, non-electoral

9.97 pts

Strict held-out

14 scored of 22, non-electoral, sourced after training froze.

On the calibrated 50-state benchmark, the average miss is 7.07 points on the best 80% of 1,521 scored questions, and 10.02 points across all of them. Issue 4 is 225 questions we published before anyone knows the answers. 50-state check →

In the older April CSV, the 389 non-electoral questions averaged 7.44 points of error; the best 80%, comprising 311 questions, averaged 5.31 points. That file is not the 1,521-question benchmark. A narrower April test on 5 places with 10,000 simulated respondents scored 7.47 points on 404 non-electoral questions. On a different April bank, in 5 places, we called the poll's winner 67% of the time against a dummy that always picks the first answer (39%). That is not the 50-state check. 4.88 points (UT/Texas Politics Project, April bank, non-electoral) is the best scored panel on that narrower bank. 9.97 points is the strict held-out cut on non-electoral items.

What we measure

The miss against a published poll.

MAE is the average gap, in percentage points, between our option shares and the published result. Lower is better.

We score all 460 questions in the April bank. 56 are election-margin items. Turnout noise drives most of that error. The April five-place number is 7.47 points on the remaining 404 non-electoral questions.A narrower April test on 5 places with 10,000 respondents scored 7.47 points on 404 non-electoral questions.

In the product, each simulated respondent says how likely each answer is. A small safety rule keeps no option at zero. Studies with more than 400 simulated respondents, and the rare ~1.3% of answers we cannot read, still use letter scores plus the option-count k-schedule. We do not claim election questions.

The 56 election results stay in the tables and the file. Turnout noise is a different test. We keep those races in the bank so a reader can see them.

Full benchmark

460

questions scored

404

Non-electoral

April five-place 7.47 points

Electoral

In the tables and the file

How panels are built

Census-grounded respondents, a model we own, a running simulation.

Census-grounded respondents

650,000 simulated respondents sampled from ACS census microdata. Age, sex, race, education, income, and place come from those records.

A model we own

Lewis is our own model. Reports name the version that fielded the study.

A running simulation

Agents live and interact in a proprietary simulation. When we score a historical question, we answer from that survey's field window. A February 2024 item is answered as of February 2024.

How a simulated respondent answers

U.S. National cohort

A raked draw from our older geographic panels (Ohio, Georgia, Texas, California, New York, Virginia, and their metros), weighted to ACS adult-population targets. It has not been scored as its own national panel. For a question with a strong state driver, use that state's panel.

State scorecards

Panel scorecards and the 50-state check.

The 50-state check scores 1,521 questions across 50 states. Washington, D.C. is held out. Best 80% of the 1,521 scored questions: 7.07 points. Full set: 10.02 points. Issue 4 is 225 questions we published before anyone knows the answers.

Four legacy panels also carry their own published scorecards. Texas and California are mean MAE on their state survey series. Ohio and Georgia are median error from the same public file, attributed by question. Those are different statistics.

California

7.77 pts

MAE

150 questions · PPIC statewide survey

See the table

Georgia

6.5 pts

median error

33 questions · Georgia Secretary of State canvass

See the table

Ohio

8.6 pts

median error

29 questions · Emerson and Marist Ohio polls

See the table

Texas

4.88 pts

MAE

97 questions · UT/Texas Politics Project

See the table

These four are legacy scorecards on their own poll series. The coverage claim is the 50-state benchmark. The full panel list is on /coverage.

Texas and California, by category

These two panels have enough questions for a category split. Ohio and Georgia do not, which is why they quote a median instead. Electoral stays in the table.

Category	TX n	TX MAE	CA n	CA MAE
Political approval	18	3.43 pts	15	10.65 pts
Directional / economic	7	4.27 pts	13	6.40 pts
Policy positions	63	5.07 pts	118	7.75 pts
Brand / reputation	4	2.44 pts	2	16.98 pts
Civic / legal	5	6.92 pts	2	12.44 pts
Electoral margin	23	13.51 pts	25	9.30 pts

April bank category cells, UT/TPP and PPIC, 2024–25, in points. This table is not the 50-state book. The April five-place test is 5-fold cross-validated. Small-n cells (CA brand n=2, CA civic n=2) are point estimates. Texas political approval is 3.43 points on 18 questions.

What we publish

The file, the held-out, and the 50-state score.

The per-question file

443 of 460 scored questions, with our predictions next to the published result. Recompute the error yourself.

A held-out set sourced after training froze

22 questions drawn April 18, 2026. 14 scored after a pre-registered filter. 9.97 points on the non-electoral items. 10.68 points on the scored set overall, which includes electoral items.

The calibrated 50-state benchmark

7.07-point average miss on the best 80% of 1,521 scored questions (1,216 retained). 10.02 points across every scored question (200 simulated respondents per question, calibrated panels). Every study is calibrated against published survey data before it reaches you. Enterprise panels are also calibrated to the client's own past studies. A narrower April test on 5 places with 10,000 respondents scored 7.47 points on 404 non-electoral questions.

The 50-state coverage benchmark

Per-question benchmark

Recompute the error yourself.

Each scored row in the public file puts our prediction next to the published result.

Download the CSV, 443 questions

443 of 460 scored questions in the April bank file. The held-out items are published as their own score. This file recomputes to a 7.44-point average miss over 389 non-election rows.

Limits

Where the claim stops.

Respondents are synthetic

Each answer is generated by Lewis for a census-grounded agent. Reports say so. Work that needs a live probability sample should use one.

How to read MAE

7.47 points is the average gap on the narrower April test (404 non-electoral questions). The 50-state book is 10.02 points across the full set. A live poll's plus-or-minus figure is a sampling margin around an estimator. They measure different things.

The national cohort has no separate score

The U.S. National cohort is a raked draw from our older geographic panels, weighted to ACS adult-population targets. It has not been scored as its own national panel.

Open-ended copy is qualitative

Message Testing reports strengths, watchouts, sentiment, and synthetic quotes. The MAE numbers on this page apply to multiple-choice studies with a published human result.

Could Lewis have memorized the answers?

The April five-place test is 5-fold cross-validated: the correction is fit on four folds and scored on the fifth. The 50-state book is a calibrated, full-set retrospective score, 200 simulated respondents per question. The 22-question held-out set was sourced after training froze and scored once. Texas on the April bank is 4.88 points, non-electoral. The held-out non-electoral figure is 9.97 points.

Why is California harder than Texas?

PPIC covers broad state-specific policy framings. Options are compound and exposure varies. UT/TPP is closer to classic approval and directional questions. We publish both.

What if I ask something that was not benchmarked?

Every result carries a confidence tier: High (under 6 pp on comparable types), Medium (6 to 10 pp), or Flag (over 10 pp, or novel territory). Custom brand and ad-concept questions extrapolate from agent profiles. Treat absolute percentages as estimates.

The tables

Independent panels, then the category lookup.

April bank panels, not the 50-state book. Four independently fielded sources. Figures below are points.

Panel	Scope	n	MAE	Ex-electoral
UT/Texas Politics Project	Texas statewide · 2024–25	120	6.35 pts	4.88 pts
PPIC California Statewide	California · 2024–25	175	8.01 pts	7.77 pts
Gallup / Canvass	Ohio · Georgia · Texas metros and statewide	151	7.86 pts	7.86 pts
Pre-registered held-out	OH · GA · TX · NY · CA	22	10.68 pts	9.97 pts

UT/Texas Politics Project. Quarterly statewide survey. Strong on approval, economic sentiment, and policy. Ex-electoral removes presidential and senate margin questions.
PPIC California Statewide. Broad policy coverage and state-specific framings. Harder than Texas on the same pipeline.
Gallup / Canvass. Legacy tri-metro set. Six electoral items, so overall is close to ex-electoral.
Pre-registered held-out. Sourced after training froze. 8 of 22 dropped by a pre-registered filter (past-election ground truths, extreme prior-delta outliers). 14 scored.

Look up error by category

Recomputed from the same 443-question file. These pool every panel, so a category can read higher here than on Texas.

Pick a question category

Every scored question in the public file except the electoral-margin bucket. This is the closest recomputable analogue to our headline pooled figure.

n = 389 questions247 state survey · 142 legacy tri-metro

7.44 pts

Mean MAE, April bank file

the number we publish for this category

6.17 pts

Median MAE, points

typical question in this category

Spread of error, per question

median 6.17pp

Under 5pp153 Q · 39%
5 to 10pp140 Q · 36%
10 to 15pp62 Q · 16%
15pp and up34 Q · 9%

Best question in this category: 0.55pp · worst: 28.94pp

n=1,000 vs n=10,000, older legacy panel

Older April bank, not the 50-state book. Same 148 questions on the legacy panel. Same 5-fold setup. Pooled miss went from 7.75 points at n=1,000 to 7.90 points at n=10,000. Larger n mainly steadies crosstabs.

Category	n	n=1k	n=10k	Δ	Note
Political approval	47	9.05 pts	8.49 pts	-0.56pp	Improved at n=10k.
Civic / legal	7	11.32 pts	10.51 pts	-0.81pp	Small n, right direction.
Electoral	6	4.37 pts	4.25 pts	-0.12pp	Held steady.
Directional	27	8.85 pts	9.08 pts	+0.23pp	Inside noise.
Brand	33	6.68 pts	7.18 pts	+0.50pp	Slight regression, still under 8 points.
Policy	28	5.37 pts	6.77 pts	+1.40pp	Only honest regression. Under investigation.
Pooled	148	7.75 pts	7.90 pts	+0.15pp	Inside bootstrap noise.

Closest single questions

Lowest-error questions in the April scored file. Out of sample on that 5-fold cross-validated test. Not the 50-state book.

#	MAE	Panel	Category	Question
01	0.40 pp	TX (UT/TPP)	Policy · 5-opt	How concerned are you about the cost of higher education?
02	0.47 pp	CA (PPIC 2025)	Policy · 3-opt	California making its own climate policy separate from the federal government
03	0.49 pp	CA (PPIC 2024)	Electoral · 3-opt	Prop 1, $6.38B mental-health facilities bond
04	0.60 pp	CA (PPIC 2025)	Policy · 3-opt	$7.1B Budget Stabilization proposal
05	0.72 pp	CA (PPIC 2024)	Policy · 3-opt	Higher taxes + more services vs. lower taxes + fewer services
06	0.79 pp	TX (UT/TPP)	Political approval · 6-opt	Presidential approval on Venezuela
07	0.94 pp	CA (PPIC 2025)	Policy · 3-opt	Taxes/services tradeoff (2025 replication)
08	0.97 pp	TX (UT/TPP)	Political approval · 6-opt	Presidential approval on health care
09	1.02 pp	CA (PPIC 2025)	Policy · 3-opt	CA state actions protecting legal rights of undocumented immigrants
10	1.06 pp	TX (UT/TPP)	Political approval · 6-opt	Presidential approval (6-option grid)

#1, option by option

How concerned are you about the cost of higher education in Texas?

UT/Texas Politics Project · February 2026 · n=1,300 registered voters

Option	Lewis	UT/TPP	Error
Very concerned	39.1%	39.2%	0.1 pp
Somewhat concerned	31.0%	30.0%	1.0 pp
Not too concerned	15.9%	16.6%	0.7 pp
Not at all concerned	9.5%	9.4%	0.1 pp
Don't know	4.6%	4.8%	0.2 pp

Neighborhood check

Short North, Columbus. 8.71-point average miss on five questions, outside the 1,521-question book.

August 2023 JS&A consumer study, n=500+. Named older neighborhood check, outside the 1,521-question book. Average across the five questions: 8.71 points.

Do you dine out more, about the same, or less often than before the pandemic?

2.47pp

More often	20.4%	20%	0.4pp
About the same	43.3%	40%	3.3pp
Less often	36.3%	40%	3.7pp

Do you believe Short North businesses are mostly locally owned, a mix, or mostly chains?

7.40pp

Mostly locally owned	48.9%	60%	11.1pp
A mix of both	37.4%	30%	7.4pp
Mostly chains/franchises	13.7%	10%	3.7pp

Compared to a few years ago, are you more likely or less likely to visit the Short North?

12.93pp

More likely	19.4%	15%	4.4pp
About the same	36.0%	21%	15.0pp
Less likely	44.6%	64%	19.4pp

How would you describe the Short North's economic impact on Columbus?

10.78pp

Major economic driver	38.8%	55%	16.2pp
Moderate contributor	45.4%	35%	10.4pp
Minor / negligible	15.7%	10%	5.7pp

Biggest reason you might visit the Short North less often?

9.95pp

Safety concerns	28.2%	35%	6.8pp
Parking difficulty	25.3%	15%	10.3pp
Too expensive	24.0%	10%	14.0pp
I go out less in general	12.0%	30%	18.0pp
Other reasons	10.6%	10%	0.6pp

Political approval

Texas political approval, 3.43 points

Setup: UT/Texas Politics Project approval block, 2024–25 waves. Governor, Lt. Governor, Legislature, U.S. Senators, economic direction. 1,000 simulated Texas respondents per question, matched to UT/TPP margins. April bank.
Result: Aggregate miss across 18 questions on the April bank: 3.43 points. Best individual question: 0.8 points. Worst: 6.2 points.
Takeaway: On approval of well-known figures and institutions, the miss sits in the same size range as the total error of a good live poll on similar items.

Brand

Texas grocery brand preference, late 2024

Setup: UT/TPP brand-reputation block. Six options including Don't know / none. 1,000 Texas shoppers, income-tilted per retail demographics.
Result: On the April Texas brand block, Lewis rank-ordered the top three brands and put the two long-tail options under 10%.
Takeaway: Brand reads pick the leader and the runner-up. For A/B concept work, the demographic split is the decision-grade signal.

Held-out

22-question independent set, 9.97 points on non-electoral items

Setup: April 18, 2026. Emerson, Marist, PPIC, USC CEPP, UT Tyler, Change Research, Ohio Library Council. Five states. Sourced after training froze. A pre-registered filter dropped 8 questions. 14 scored.
Result: A strict held-out set sourced after training froze: 9.97 points on non-electoral items (14 scored of 22 drawn). The overall scored set, including electoral items, is 10.68 points.
Takeaway: Texas on the April bank is 4.88 points, non-electoral. This held-out set is 9.97 points on non-electoral items. We publish both.

Disclosures

What every report already says.

Synthetic respondents

Lewsearch panel answers are generated by Lewis for census-grounded agents. Every report carries that disclosure.

When you need a live sample

Use a human probability sample for legal, regulatory, or journalistic work that requires one.

Production correction

7.07-point average miss on the best 80% of 1,521 scored questions (1,216 retained). 10.02 points across every scored question (200 simulated respondents per question, calibrated panels). Every study is calibrated against published survey data before it reaches you. We calibrate Enterprise panels to the client's own past studies, so the numbers line up with the tracker they already trust. A narrower April test on 5 places with 10,000 respondents scored 7.47 points on 404 non-electoral questions.

Geographic coverage

650,000+ census-grounded Americans across all 50 states and DC, plus 16 professional panels. Every panel is seeded from ACS census microdata. The full panel list is at /coverage.

Model versioning

Reports indicate the Lewis version at time of fielding. Lewis 1.x is the current production model. The April five-place test is 5-fold cross-validated. The 50-state book is a calibrated, full-set retrospective score.

Confidence scoring

HighMediumLowFlag

Every result ships with a confidence tier (High / Medium / Low / Flag). On the April bank, tier is the more conservative of (1) 5-fold cross-validated MAE for the matching question-type bucket and (2) topline signal strength for this study. High = under 6 points of bucket MAE and a strong signal. Medium = 6 to 10 points or a moderate signal. Low = weak signal inside a medium or high bucket. Flag = bucket MAE at or above 10 points, or no matching bucket. Treat Flag as directional only.

Try it

Run a study and check the numbers.

Field your first studyWe set it up with you on a 30-minute call.Ask about the methodology

Are AI survey panels accurate?Synthetic respondents accuracy Prediction-market note 50-state check Coverage Back to top