What files should a synthetic research vendor deliver?

Start with the result your team needs to check

Choose one headline table and one important subgroup result that your team should be able to rebuild. Put that acceptance check in the project brief before work starts.

Agree on the data, instructions and access needed for that check. A small workbook may be enough for a simple study. A larger delivery may need separate data files and a calculation script. The format should work with the tools your team can use.

Agree on redactions and restricted access before delivery. If an omitted step prevents a required check, record that limit and decide whether a narrower engagement still meets your need.

Request a package that explains the reported results

Our suggested handoff checklist can fit in a workbook or a small folder. The important point is that the pieces refer to the same delivered version.

• A file guide. List each file, its purpose, version and date. Identify which file replaces an earlier one if the delivery is corrected.
• The final research materials. Include the questions, answer choices, messages or concepts shown, and the rules determining which questions appear. Keep question IDs consistent across the materials and data.
• Preserved outputs. Label generated records clearly and explain what each row represents. Include the response codes and group labels needed for the agreed tables. If only summaries can be shared, state which checks they support.
• Calculation instructions. Specify which records count in each table, exclusions, missing-answer treatment, weights and rounding. Supply the spreadsheet formulas, code or explicit steps used to calculate the result, plus any required software versions.
• A run record. Give the run date and identifier, model or service version where available, relevant settings, and a description of the prompts and context supplied. Identify unavailable details and explain the gaps.
• A review and correction record. Summarize checks performed, who reviewed the work, known issues and corrections. State which results changed and which files now contain the corrected version.

AAPOR’s Code of Professional Ethics and Practices supports documenting research instruments, populations, processing and AI use so work can be independently reviewed. Our file checklist is a proposed purchasing requirement, rather than a claim that every item is a mandatory public disclosure.

AAPOR’s AI task force report discusses recording versions, context and settings while acknowledging that proprietary details may be unavailable. It allows descriptions where exact prompts cannot be shared. State what is known, what is missing and how that limits the check.

The UK Government Analysis Function’s reproducible analysis guidance supports documented steps, version history, checks and software dependency records. These principles help an analyst follow a calculation; they do not establish that synthetic answers represent people accurately.

A hypothetical delivery with an undocumented weight

All numbers in this example are invented to illustrate a delivery check. They are not Lewsearch results, and the weights are not a recommendation.

Imagine receiving 100 generated records from a message test. All 100 have valid answers to the same question, and none are excluded. In Group 1, 30 of 40 records choose message A. In Group 2, 18 of 60 choose A. Counting every record equally gives 48 choices for A out of 100, or 48%.

The report shows 52.5%. Its calculation gives each Group 1 record a weight of 1.5 and each Group 2 record a weight of 1.0:

• The weighted total choosing A is (30 × 1.5) + (18 × 1.0) = 63.
• The total weight is (40 × 1.5) + (60 × 1.0) = 120.
• The weighted share choosing A is 63 ÷ 120 = 52.5%.

An analyst cannot recreate the reported figure without the weight field or an equivalent calculation rule. Once that rule is supplied, the same fixed file is enough to check the arithmetic. There is no need to generate new answers.

The weight total of 120 is a calculation denominator, not 120 people or proof of 120 independent responses. Reconstructing 52.5% also says nothing about whether those weights or generated choices are appropriate for a real audience.

Check the delivery before accepting it

Ask someone who did not prepare the report to use the delivered files and instructions for the agreed acceptance check:

• Recreate the selected headline result and subgroup result from the preserved outputs.
• Confirm the question, included records, missing-answer rule and any weights match the report.
• Record any mismatch, including whether it comes from rounding, an unclear rule or a different file version.
• Have corrections returned as a clearly identified version, then repeat the affected check.

Agree in advance what counts as a match, including rounding. Keep a record of anything the package does not let you verify. An acceptance check that passes shows that the selected result can be reconstructed; it does not establish that every other table has been checked.

Separate rebuilding tables from running the model again

Rebuilding a table starts with the outputs already delivered. Generating answers again starts a new run. A supplier may be able to preserve the former without promising identical answers from the latter.

If repeat generation matters to your project, agree separately on what should be preserved, what may change and how a new run would be compared. Ask the supplier to document any limits rather than implying that a file export guarantees an identical rerun.

For evidence about accuracy, use a separate synthetic-panel validation plan. For clear labeling of different evidence types, see reporting human and synthetic research together. If questionnaire logic is central to your checks, the survey routing guide explains what to preserve.

FAQ

Is a PDF report enough?

It depends on what you are buying. A summary may meet a reading-only need. If your team must rebuild tables or apply new filters, agree on the additional outputs and calculation rules needed for that work before accepting the delivery.

Do I need the vendor’s exact prompts?

Ask for the detail needed to understand the study and perform your agreed checks. Where exact prompts cannot be shared, request a useful description and a clear account of what remains unavailable. Do not treat a partial description as proof that you can regenerate the run.

Does a complete file package prove the findings are accurate?

No. A package can let you trace a calculation and still contain predictions that do not match people. The buying decision should consider both whether you can inspect the work and whether relevant evidence supports its intended use.