Can synthetic survey data support driver analysis?

Why a topline score is not enough

A topline describes one question at a time. Driver analysis asks how an outcome, such as overall satisfaction or renewal intention, relates to other answers. That requires evidence about which answers occur together within respondent records.

In a 2024 Political Analysis paper, James Bisbee and colleagues compared persona-prompted ChatGPT responses with American National Election Studies data. Synthetic averages were close to the human averages, but variation was lower and regression coefficients often differed significantly. The research used 2023 ChatGPT outputs and did not evaluate Lewsearch or establish the performance of every current synthetic method. It demonstrates why matching averages is an incomplete validation target. Read the study.

The buyer procedure below is our proposed evaluation checklist, not a certification standard or a claim that Lewsearch has validated a driver-analysis product.

Identical toplines can conceal opposite relationships

Illustrative example only. All counts are invented. These are two hypothetical files of 100 equally weighted records, with complete answers to two binary questions: a high or low service rating, and an intention to renew or not renew. Neither file is a real study or measured Lewsearch result.

The hypothetical human-reference file contains:

• High service rating and intends to renew: 40 records.
• High service rating and does not intend to renew: 10 records.
• Low service rating and intends to renew: 10 records.
• Low service rating and does not intend to renew: 40 records.

The hypothetical synthetic file contains:

• High service rating and intends to renew: 10 records.
• High service rating and does not intend to renew: 40 records.
• Low service rating and intends to renew: 40 records.
• Low service rating and does not intend to renew: 10 records.

Both files report exactly 50% giving a high rating and 50% intending to renew. A check of these two toplines would find no error.

Yet 80% of the high-rating group intends to renew in the human-reference file, compared with 20% in the synthetic file. Among the low-rating group, those shares are 20% and 80%, respectively. The relationship reverses even though both individual question distributions match.

This is a mathematical illustration of a validation gap, not an estimate of how often a synthetic method will make that error. With several predictors, also check relationships among predictors and compare the same full model in both datasets. Matching separate two-way associations does not establish that a multivariable analysis is accurate.

Specify what the buyer means by a driver

Before requesting a pilot, name the outcome, candidate predictors and intended decision. Asking respondents which feature matters most is a direct importance question. Estimating which ratings are associated with satisfaction is a different analysis. Predicting stated renewal intention on new human cases is another target. Predicting actual renewal behavior requires observed behavioral outcomes, not just intention answers.

Agree on the statistical method, variable coding, scales, weights and treatment of missing or inapplicable answers. If several predictors overlap, ask the analyst how that affects interpretation and the stability of the ranking. Do not compare coefficients produced by different coding or scaling choices as though they were the same quantity.

A statistical association also does not establish that improving a feature will cause actual renewal to rise. NIST's definition of correlation explicitly separates association from cause. A budget decision that depends on an intervention's effect needs an appropriate causal design, such as a suitable randomized experiment. NIST on correlation.

Reserve human evidence for the analysis being claimed

For a standalone synthetic panel, keep the human comparison study separate from calibration and run selection. For a method fitted to an existing survey, reserve evaluation records before fitting, calibration or model selection. Document the inputs the provider may use and keep evaluation outcomes out of those inputs. If the intended task concerns a new question or audience, a respondent holdout on familiar questions is not enough to establish that wider scope.

The separation principle follows standard model evaluation: scikit-learn's guidance explains that using information unavailable at prediction time can make measured performance too optimistic. Apply that principle to the whole workflow, including preprocessing, tuning and selecting the most favorable synthetic run. Guidance on data leakage.

Have an evaluator run the agreed analysis on the reserved human evidence after the synthetic outputs and analysis specification are locked. Keep the reference's human sample size, recruitment, weights and uncertainty visible. Small or unstable human estimates cannot supply a precise verdict merely because the generated file is large.

Compare the outputs behind the recommendation

Choose the checks that match the proposed use before unblinding:

• Relationships between answers. Inspect the important cross-tabulations or associations, including their direction and size. Agreement on each question separately is insufficient.
• The specified driver model. Compare the agreed coefficients or importance measures and the resulting priority list. Record material sign reversals, omitted relationships and unresolved near-ties, rather than reporting only a ranking correlation.
• Human prediction, if claimed. Evaluate a model trained using the proposed synthetic workflow on untouched human cases. Compare it with a pre-agreed human-only baseline using the same permitted inputs and suitable scoring. Good prediction does not, by itself, validate an explanation of what causes the outcome.
• The buying decision. State whether discrepancies would change the recommendation. A matching first-place driver can still conceal errors elsewhere that matter to the proposed action.

Keep human-only, synthetic-only and any blended analyses distinguishable. Additional generated records are not additional observed human responses. Require an explanation and validation of any claimed uncertainty or effective sample size rather than inserting the enlarged row count into ordinary survey formulas.

There is no universal passing threshold here. Choose tolerances from the decision, and use a qualified analyst to assess whether the human evidence can resolve the differences. If the evidence is insufficient, narrow the claim or collect more suitable human data. Tuning after unblinding needs a fresh evaluation.

Bring the analysis question to the pilot discussion

Before commissioning a Lewsearch pilot, agree whether the scope includes the relationships and analysis your decision needs. Read the methodology and limits, use the blind-validation guide to plan separation and scoring, and discuss the proposed study. Do not infer driver-analysis validity from a published answer-share benchmark.

FAQ

Does matching every topline prove that synthetic driver results are accurate?

No. The same individual question distributions can coexist with different relationships between answers. Validate the associations and the specified analysis, not only the marginals.

Can synthetic rows make a driver model's confidence intervals narrower?

Software may produce narrower intervals if it treats generated records as independent observations. That output is not evidence that uncertainty about human relationships has fallen. Ask how the interval accounts for the synthetic method and how it was validated.

Does a driver ranking tell me what to change to increase sales or retention?

Not by itself. An association or predictive ranking does not establish an intervention's causal effect. Use evidence designed for the action and outcome you intend to claim.

Related answers

• How do you choose a synthetic sample-boosting provider?
• How to check subgroup accuracy in synthetic research
• How should you report a study with human and synthetic responses?