What is silicon sampling?
Silicon sampling is asking a language model to answer a survey as stand-in respondents. On September 30, 2026, Pew Research Center tested AI twins of its own panelists and found their answers differed from the real ones by 12 percentage points on average across nearly 300 questions. How a silicon sample is built and scored decides whether it is useful, so ask any vendor for its scored file. The 500-question benchmark showcase records mean error of 3.72 points per answer in the September 1, 2026 historical evaluation.
Definition
A silicon sample is a set of answers produced by a model that has been prompted, or otherwise conditioned, to respond as if it were a person or a population. Researchers use it when they want a fast, inexpensive read before or beside a human fielding. The name comes from the methods literature.
What Pew found in 2026
On September 30, 2026, Pew Research Center published “Can AI Stand In for Human Survey-Takers? Not Really”. Each member of its American Trends Panel got an AI twin built from demographics, political typology answers, and a model-written reflection on that person. The twin then took real Pew surveys one question at a time, in the same order and wording as the human.
- Across nearly 300 questions, the twins' estimates differed from the human results by 12 percentage points on average.
- Errors were larger for some groups than for the population as a whole.
- Humans were about four times as likely as the twins to say they were not sure.
- On many questions, at least one answer option was chosen by no AI respondent at all.
- Results moved with the choice of model, and the misses were often hard to predict in advance.
Pew's conclusion: “AI polling is not a replacement for rigorously surveying real humans.” We agree with it. The study is a careful public test of person-level twins, and every vendor in the category should read it.
Twins and distribution forecasts are different tasks
Pew asked a model to be one person at a time. A day later, Expected Parrot asked frontier models for each question's answer distribution in a single prompt and reported average errors of roughly 4 to 6 points on the same national questions. That is a vendor lab note, and it scores national toplines only. Both results are useful, and neither tells you how a method performs on your category, your audience, or your wording.
A national topline is one part of a research job. Teams also need crosstabs that add up to the total, the same respondents in the next wave, and an audience that looks like their customers. That is what a respondent-level panel is for, and it is also where it has to be checked.
Grounding and elicitation
Two design choices dominate results. Grounding is who the model is asked to be: a one-line persona, a census microdata record, or a persistent agent with memory. Elicitation is how the question is asked: a single prompt, a verbalized multiple-choice draw, or a tabulated panel run. Lewsearch grounds each respondent in a census microdata record with its own beliefs and memory, and elicits a full panel through Lewis, its own survey model. Changing either choice can move the number, which is why the number has to be published.
Known failure modes
Silicon samples are sensitive to wording, option order, and the profile attached. They tend to under-disperse: simulated respondents agree with each other more than people do, which shows up as options nobody picks and too few “not sure” answers. Subgroup errors are usually larger than topline errors. If a vendor shows only a confident mean, ask for the spread, the subgroup error, and the share of options with no simulated picks.
Validation against human data
A silicon sample is only as useful as its human comparison. Lewsearch scores 500 questions from published polls across 50 states, with 200 simulated respondents per question on calibrated panels, and publishes a public summary. It also freezes forecasts before official numbers are published and grades them in The Lewsearch Report (2 wins, 2 partials, 1 miss, 5 awaiting a grade so far). For your own use case, the Blind Challenge replays a study you already fielded, before we see your results. Census grounding is not the same as interviewing a human probability sample.
Buyer checklist
- Is the unit MAE, correlation, thematic overlap, or something else?
- How many questions were scored, and how many respondents per question?
- How large are the subgroup errors compared with the topline?
- Was the test retrospective, held-out, or frozen before the answer existed?
- Can you inspect the scored materials?
- What uses does the vendor refuse: legal sample, conversion forecast, willingness to pay?
| Idea | Silicon sample in general | Lewsearch panel |
|---|---|---|
| What it is | Model answers standing in for people | Census-grounded simulated respondents, one instrument, tabulated shares |
| Typical use | Exploratory screen | Panel study, message test, focus group, digital twins |
| Respondent-level records | Depends on the design | Yes, so crosstabs add up and waves can repeat |
| Published miss | Varies by paper and prompt | The 500-question benchmark showcase records mean error of 3.72 points per answer in the September 1, 2026 historical evaluation. |
| Frozen forecasts | Rare | 10 published in The Lewsearch Report, graded in public |
FAQ
- What did Pew find about silicon samples?
- In a study published September 30, 2026, Pew Research Center built an AI twin for each member of its American Trends Panel from demographics and political typology answers, then had a model answer real Pew surveys one question at a time. Across nearly 300 questions the twins differed from the humans by 12 percentage points on average. Errors were larger for some groups, the twins rarely said they were unsure, and on many questions some answer options were picked by no AI respondent at all. Pew concluded that AI polling is not a replacement for rigorously surveying real humans.
- Did Pew test Lewsearch?
- No. Pew tested its own twin design on its own panel. NIM's 2024 marketing-research note on silicon samples did not test Lewsearch either. Both are sources for the category. Lewsearch's own error is published separately, question by question.
- Is silicon sampling the same as a synthetic panel?
- They are related. Silicon sampling is the general method of using model-generated answers as stand-ins for people. A synthetic panel adds a fixed file of simulated respondents, one shared instrument, and tabulated option shares, so results can be cut into crosstabs and the same respondents can be asked again in a later wave.
- When is silicon sampling useful?
- For exploratory screens before and beside human fieldwork: pretesting an instrument, ranking messages, shortlisting concepts, and exploring segments. It is not a probability sample. Use human fieldwork when the sample itself is the deliverable, such as a legal, regulatory, or journalistic number.
- What should a buyer ask any vendor?
- What was measured, on how many questions, at what n, against which human source, how large the subgroup errors are, and whether the scored materials can be inspected. Ask whether any forecasts were frozen before the answer existed. If those cells are empty, treat the method as unvalidated for your use.
Related answers
Canonical: https://lewsearch.com/answers/what-is-silicon-sampling