How should synthetic surveys handle 'don't know' answers?

Check whether the simulation preserves uncertainty

In a Pew Research Center study published on September 30, 2026, human panelists selected 'not sure' 16% of the time across opinion questions that explicitly offered it. Pew's synthetic respondents selected it 4% of the time, despite instructions allowing uncertainty. These figures come from Pew's digital-twin setup using its own panelist profiles and Claude Opus 4.6. Pew did not evaluate Lewsearch.

Our implication for buyers: a panel can produce a more decisive-looking distribution while missing how often people are unsure. If your purchase supports an awareness, familiarity, or opinion study, ask whether the evaluation includes those answers. Adding permission to be unsure to a prompt is a method choice that still needs testing.

The checklist below is our proposed evaluation approach. It sets no universal target for the share of unsure answers and no universal passing error rate.

Agree on the answer categories before the run

For a replication, preserve the human instrument's wording, response options, instructions, and routing. Record whether an unsure answer was displayed or could only be volunteered. Pew's questionnaire guidance explains how wording and response choices can affect results. Silently adding or removing an option changes the comparison.

Use a codebook that keeps the following distinctions visible where the instrument or collection process makes them:

• Don't know or not sure. Preserve the original label. An answer alone does not tell you whether it reflects unfamiliarity, indecision, or another reason.
• Neutral or neither. Keep a stated midpoint separate from an unsure answer when both are offered. Do not infer that someone selecting the midpoint lacked an opinion.
• Not applicable. Preserve it as a response category when the questionnaire offers it.
• Not asked. Record eligibility and routing rules. A respondent routed past an item did not select an answer to it.
• Skipped or declined. Preserve item nonresponse and any explicit 'prefer not to answer' option according to the human study's coding rules.
• Generation failure. Record a timeout, blocked generation, or unparseable output separately. A system failure does not establish that a simulated respondent chose 'not sure' or declined the question.

Some published datasets combine categories. Keep the original codes where available and document any mapping used for comparison. If the human reference only reports a combined category, the comparison cannot establish accuracy for each component separately.

A denominator can hide the disagreement

Illustrative example. These invented counts describe two unweighted, single-choice datasets with 100 eligible responses each. Every response is valid; there are no skips or generation failures. They are not a customer result or a Lewsearch performance claim.

Among responses selecting support or oppose, support is 50% in both datasets: 40 divided by 80 for humans and 50 divided by 100 for the simulation. That conditional comparison hides a 20-percentage-point shortfall in 'not sure' answers. On the full base, synthetic support is ten points higher than human support.

If a report includes a result among those expressing a view, label that base and show the full distribution beside it. Use the same inclusion rules on both sides. In weighted studies, disclose the weighting and the unweighted bases as well; weighted percentages need not equal raw counts divided by the raw total.

Request this audit before approving the purchase

Ask for a delivery that lets your team answer these questions without reconstructing undocumented exclusions:

• Was the same question asked? Keep the original wording, offered options, routing, and treatment of volunteered answers beside the synthetic specification. Disclose changes needed to reproduce the instrument.
• What happened to every eligible record? Reconcile eligible cases with valid responses, skips or refusals, and generation failures. Report retries and replacements separately so discarded outputs remain visible in the audit.
• Which base produced each percentage? State who was eligible, who was included, which codes were excluded, and how weights were applied. Supply all response categories, including zero-share categories.
• How large is the uncertainty-response error? For each matched question and base, report the synthetic unsure share minus the human unsure share in percentage points. Show the full answer distribution too. Check priority audiences separately rather than relying on one average across all questions.
• Were handling rules fixed before comparison? Agree on coding, exclusions, retries, any imputation, and scoring before the evaluation. Keep questions used to tune those choices separate from the human outcomes reserved for evaluation.
• Could the human comparison resolve the difference? Report human recruitment, field dates, unweighted bases, weighting, and uncertainty appropriate to the study design. A small or unstable human reference may leave the result inconclusive.

AAPOR's disclosure standards cover the instrument, sampling, collection, weighting, processing, exclusions, and limitations of publicly reported survey research. The synthetic delivery checks above are our proposed additions for a buyer evaluation. A disclosure checklist does not certify a panel's accuracy.

Choose a buying rule that fits the decision

Set the acceptable error and the consequences of failure before seeing the comparison. For a study about public familiarity or readiness to choose, a large miss on unsure answers can matter even when the balance among expressed preferences matches. If the intended use cannot tolerate that miss, limit the approved use or collect stronger human evidence.

Matching the human unsure share is one check. It does not establish that the other answers, the priority subgroups, or the resulting business decision are accurate. Use the synthetic-panel validation guide for the broader evaluation and the subgroup accuracy guide for audience-level checks.

For a Lewsearch evaluation, review the Blind Challenge scope and agree which of these checks the evaluation will include before the run.

FAQ

Should every synthetic survey add a 'don't know' option?

For a replication, match the human questionnaire and its handling of volunteered or skipped answers. Adding an option creates a different instrument. For a new study, decide which options fit the research question and test the instrument with humans.

Can I compare only respondents who expressed a view?

Yes, if that conditional comparison fits the intended question. Label the included categories and denominator, use matching rules for human and synthetic results, and show the full distribution alongside it so excluded unsure answers remain visible.

Is a model's confidence score the same as a survey 'not sure' answer?

No. A confidence score describes a model output under a particular method. A 'not sure' category is an answer to the survey question. Report them separately; one does not establish the human share that would select the other.