Can you use an old survey to validate synthetic respondents?

Start with the files you have

A slide saying “customers preferred offer A” is a finding, not enough material to repeat the test. Look for the final questionnaire, the offers people saw, the results tables and the methods notes. A draft questionnaire may differ from the version that actually ran.

The AAPOR disclosure standards identify useful records: exact questions and context, population and recruitment, collection dates, sample sizes, weighting and quality checks. Use them to understand the human study before treating it as a reference.

Ask the original research team whether the archive contains the final cleaned results. Establish which respondents were removed, why, and whether known recruitment or response-quality problems remain. A close synthetic match cannot repair an unreliable human comparison.

Find out what the percentages mean

A base is the group used to calculate a percentage. “Half were interested” might mean half of everyone interviewed, half of current customers or half of the people shown one offer. Recover that definition for every result you plan to score.

Weights give some human responses more influence than others. Identify whether the archived figures are weighted, and preserve the original calculation for the historical comparison. Applying today’s customer mix creates a different estimate; label it separately.

You may not need individual response records for a limited comparison of published answer shares. Complete tables with clear bases and methods can support that task. But tables cannot support every new analysis. If the archive lacks the subgroup you need, you cannot recover its answers from the overall average. Ask the original analyst whether the required results can be produced; otherwise narrow the test.

Keep actual respondent counts beside weighted percentages. A subgroup drawn from very few people may be too uncertain to settle a close buying decision. Have the research owner assess that uncertainty using the original study design.

Recover the question people actually answered

Preserve the answer choices, instructions, images and rules that determined who saw each question. Check what came before it. A question about an offer shown after a long product description differs from the same question asked without that description.

Pew Research Center’s questionnaire guidance explains that wording, answer options and question order can affect responses. It advises keeping wording and context comparable when measuring change. For an archived comparison, this means recovering the original experience as far as the proposed test requires.

If an image is missing or the routing cannot be established, do not quietly invent a replacement and score it as the original study. Recover the material, exclude the affected questions before the run, or choose another study. Record the narrower scope so a partial test does not become a claim about the whole survey.

Choose between a historical test and a current question

Write down the date and audience the synthetic result is meant to represent. Last year’s customer survey might be useful for testing last year’s offer choice. It is a weaker basis for deciding whether new prospects will buy a revised offer today.

List the changes that matter: price, product, competitors, customer eligibility or buying conditions. If your decision depends on those changes, get current human evidence. There is no useful expiry date that applies to every survey; the issue is whether the old study still answers your question.

For a historical test, agree on the information supplied to the vendor and the date it should describe. Merely putting last year’s date in a prompt does not establish that a model lacks later knowledge. For repeated measurements, see the separate guide to tracking changes over time.

Check whether the answers are already known

Ask whether the vendor has used the study’s findings to build, adjust or select its approach. If so, reserve another comparison for the blind test. An old study can remain useful for development without being an independent test.

The scikit-learn guidance on data leakage explains why using test information to build a model can make performance look better than it is. Our buyer application is simple: keep the target findings out of setup and tuning until predictions are locked. Publicly available results bring an additional uncertainty about prior exposure. Follow the panel-validation guide for arranging the test and scoring it.

Fictional example: a missing offer changes the plan

Imagine a subscription business with last year’s survey of existing customers. Its slide deck names the preferred annual plan. The team wants to use that result to evaluate a synthetic provider before testing a new plan for prospects.

The archive has the questionnaire and tables, but the offer screenshots are missing. The analyst finds them and discovers that the winning plan included a loyalty discount. The new offer has no discount, and prospects were never surveyed.

The buyer can now arrange a historical comparison using the recovered customer offer, provided the findings have stayed out of tuning. That test may inform whether to continue evaluating the provider. The prospect offer still needs its own evidence. Without the screenshots, the buyer would have been testing a guessed version of last year’s study.

Make the reuse decision before the pilot

• Reuse the study when its materials, human results and methods support the intended historical comparison.
• Narrow the test when only some questions or audiences can be reconstructed. Agree on those limits before seeing synthetic results.
• Choose another study when essential materials or credible human results cannot be recovered.
• Collect current human evidence when today’s audience, offer or buying conditions drive the decision.

To discuss a possible Lewsearch pilot, book a demo with a short description of the study, its date and the materials you have.