Can synthetic respondents help pretest a survey questionnaire?

Start with what people must understand

Before reviewing the questionnaire, record the intended audience, language, survey mode and meaning of each important measure. What should count as using a product, making a purchase or holding a particular role? Which people should be able to answer?

AAPOR recommends pretesting with people similar to the intended survey respondents. It distinguishes questionnaire pretesting from a pilot that checks the wider process, including recruitment, administration and data cleaning. [1]

That gives a synthetic review a limited brief: suggest questions for the research team to investigate before human testing. It does not measure how common a misunderstanding is.

Keep a versioned issue log

Save the exact questionnaire being reviewed: question IDs, wording, answer choices, codes, instructions, stimuli and routing. Give every revision a new version rather than quietly replacing the original.

For each suspected problem, record:

• The question ID and questionnaire version.
• How it was found: researcher review, synthetic response, programmed test or human observation.
• The original wording and the specific concern.
• Any supporting record, labeled by its source.
• A proposed change and the human or mechanical check still needed.
• The decision, owner and resolution status.

If synthetic responses are part of the review, retain the supplied profile context, prompts and relevant model/settings information. Label a generated explanation as a synthetic suggestion. It is not an interview with a person.

A hypothetical questionnaire review

Suppose a draft question asks:

“In the past month, how often did you use a paid fitness app?”

Its answer choices are Never / 1–2 times / 2–5 times / 6 or more times.

The question and issue log below are invented. These are authored review suggestions, not findings from a human study or an actual synthetic run.

These concerns justify review. They do not establish that a particular percentage of people would be confused. Nor should the team adopt every suggested rewrite: changing the definition of “paid” could change the business question.

If the “Never” answer leads to a satisfaction question that assumes app use, add that routing concern to the log. A programmed test can check the specified branch. People still need to try the survey to show how they encounter and interpret it.

Check rules separately from human understanding

For the programmed questionnaire, use explicit test cases for eligible and ineligible respondents, every branch, skipped items, missing answers and permitted combinations. Check that exports retain the intended codes.

Sending questionnaire text to a model does not test the actual survey interface. It cannot establish whether a phone user sees all the options, understands an error message or can navigate back without losing answers.

The Census Bureau’s A2 standard treats questionnaire pretesting and verification of instrument functioning as parts of development. Its respondent-pretesting requirements address issues including understanding, question sequence and navigation. That standard governs Census Bureau work; it is a useful reference here, not a certification rule for private studies. [2]

Take unresolved questions to people

In human cognitive interviews, ask intended respondents to explain how they understood important terms and arrived at their answers. Start without teaching them the meaning the researcher hoped to convey.

CDC describes cognitive interviewing as a way to examine comprehension, recall, judgment and response. Its purpose is to identify questionnaire problems, rather than estimate population quantities. [3]

Useful probes for the hypothetical app question include asking someone to describe the time period they considered and what they counted as app use. Record what they actually say. Do not replace their explanation with a synthetic reconstruction or merge the two as if they were equivalent observations.

Then run a real pilot of the intended survey process. Inspect routing, completion difficulties, missing answers and data handling. If completion time or dropout matters, measure it with people in the actual implementation rather than treating simulated behavior as observed behavior.

Revise the questionnaire where needed, document the reasons, and recheck the affected wording and branches before wider fieldwork.

What should a buyer ask to receive?

Ask the provider or research team for:

1. The reviewed questionnaire and revision history.
2. An issue log separating synthetic suggestions, researcher judgments, mechanical checks and human findings.
3. The human pretest and pilot plan, with unresolved concerns made explicit.
4. Evidence that the agreed routing and exports were checked in the actual implementation.
5. A final instrument, with any remaining limitations documented.

A long synthetic transcript is not enough to close those items. The useful output is a traceable set of suspected problems, decisions and checks before money is spent on fieldwork.

For handling uncertain response categories, see how synthetic surveys should handle “don’t know” answers. Evaluating the resulting synthetic estimates is a separate task: use the synthetic panel validation guide.

Sources

1. AAPOR: Best Practices for Survey Research, section “What kinds of testing should I do before fielding a survey?” Checked October 24, 2026.
2. U.S. Census Bureau: Statistical Quality Standard A2, requirements A2-2, A2-3.3, A2-3.4 and A2-4. Checked October 24, 2026.
3. CDC/NCHS: Cognitive Interviewing, April 16, 2024. Checked October 24, 2026.