Can synthetic message tests predict advertising lift?

Define the effect you want to predict

Start the buying brief with one sentence: “We want to estimate the effect of showing this message, compared with this alternative, on this outcome, for this audience and time period.” Comparing creative B with creative A answers a different question from comparing a campaign with no campaign.

For example, Google Ads describes Conversion Lift as a controlled comparison between an audience that receives ads and an audience held back from seeing them. This illustrates the comparison needed to estimate a campaign’s effect. It provides no evidence about the accuracy of a synthetic forecast.

Specify whether the outcome is a survey rating, stated purchase intent, a click, a completed signup or a purchase within a defined period. A well-designed human survey experiment can estimate the effect on the answer it measures. A forecast of purchases needs validation against purchases. Keep those outcomes separate rather than converting a preference score into a conversion rate.

Test the forecast against results from people

Randomly assigning simulated respondents to messages can reveal differences produced by that simulation. Repeating the test can show whether those differences are consistent. To find out whether they predict responses from people, compare the forecast with relevant human experiments.

Those experiments need a credible comparison with what would have happened under the alternative condition. The UK government’s impact-evaluation guidance explains how random allocation can help establish that comparison, and why other designs depend on their own assumptions. An ordinary before-and-after sales comparison can also reflect seasonality, price changes or other marketing.

We recommend evaluating a forecast on experiments that were not used to develop or tune it and that resemble your intended use. Look for comparable audiences, message formats, exposure conditions, outcomes and alternatives. NIST’s AI Risk Management Framework, section 3.1 supports the general principle of validation for an intended use under realistic conditions. The evidence for a particular synthetic forecast still has to come from testing that forecast.

Check absolute changes as well as relative lift

The following example is entirely fictional. It is not customer data or a measure of Lewsearch performance. Suppose 10,000 people are randomly assigned to creative A and another 10,000 to creative B. Purchases are counted over the same period. A hypothetical synthetic system predicts the purchase rate for each group.

• Fictional human experiment. A has 200 purchases out of 10,000 people (2%). B has 300 out of 10,000 (3%). The difference is 1 percentage point. Relative to A, that is a 50% increase: (3% - 2%) / 2%.
• Hypothetical synthetic prediction. A is predicted to have a 4% purchase rate and B a 6% rate. The predicted difference is 2 percentage points. Relative to A, that is also a 50% increase: (6% - 4%) / 4%.

Both comparisons report 50% relative lift, yet the system predicts twice the purchase rate in each group and twice the absolute difference. Applied to 10,000 comparable people, its forecast implies 200 additional purchases from using B rather than A. The fictional experiment implies 100. That gap could change whether paying more to produce B is worthwhile.

This comparison estimates B’s advantage over A. Measuring either creative’s effect against no advertising requires a no-ad comparison. In a real study, a buyer would also need to consider uncertainty around the estimates and whether the experiment was designed and carried out well. The arithmetic alone does not settle the spending decision.

Agree on the evaluation before you buy

We recommend agreeing on the following before a supplier runs the evaluation.

• Define the target. Record what is being compared, the audience, how exposure will work, the outcome and the measurement period. Specify whether people, households or geographic areas are assigned to each group. Keep stated intentions and measured behavior as separate outcomes.
• Use a credible human comparison. Prefer relevant randomized human or field experiments when feasible. Record assignment, actual ad delivery, missing outcomes and the planned analysis. If households or regions are assigned together, account for that grouping in the analysis. Comparing only the people who viewed or clicked can lose the protection of random assignment.
• Keep evaluation results out of development. Use separate experiments to develop the system and test it. Record predictions, the service version and scoring rules before evaluation outcomes are revealed. Ask whether published historical results might already have been seen during development. Testing against a future experiment makes the timing easier to verify.
• Score the size of the effect. Compare predicted and measured differences in the same units. Show the result for each group, plus absolute and relative differences where meaningful. Include uncertainty in the human estimate and results for the audiences that matter to your decision. Naming the winning creative leaves the size of its advantage untested.
• Compare with a simple forecast. Choose a simple reference forecast in advance, such as predicting no difference between creatives. Score it on the same experiments to see whether the synthetic forecast improves on it.
• Include all agreed experiments. Keep small, negative and inconclusive human effects in the evaluation. Report exclusions and failed tests. Choose experiments independently of the synthetic predictions, or explain how selection limits the conclusions. A few successful campaign examples cannot establish accuracy across other campaigns.
• Tie the tolerance to the decision. Agree how much forecast error would change the spending or launch decision, and when a new human test is needed. An uncertain human result may leave the evaluation inconclusive. Reassess when the audience, outcome or service changes.

Keeping evaluation outcomes separate applies the scikit-learn guidance on data leakage: using information unavailable at prediction time during development can make test results too optimistic. If revealed outcomes guide revisions or the choice of a favorable run, ask for fresh, untouched experiments to evaluate the revised claim.

Match the purchase to the evidence

For an exploratory message test, use the results to identify questions, objections and candidates for human testing. Keep the report labeled as simulated. The message-testing guide explains that use, and the ranking-stability guide covers sensitivity to presentation.

For a quantified lift forecast that will guide spending, request the experimental comparisons and error measures above. If the evidence does not cover your audience or outcome, arrange an appropriate human or field evaluation before relying on the forecast. A general accuracy score cannot establish accuracy for an untested use.

To discuss your evaluation brief, bring the decision, audience and proposed comparison to a Lewsearch demo. The synthetic-panel validation guide can help you plan the wider assessment.

FAQ

Does a stable synthetic ranking establish advertising lift?

No. A stable ranking shows consistency under the tested procedure. A lift forecast needs evidence that its predicted differences match human results for the specified outcome and comparison.

Does creative B beating creative A prove the campaign works?

No. A well-designed human experiment can estimate B’s effect relative to A. Establishing either creative’s effect relative to no advertising requires that separate comparison.

Is there one accuracy threshold every synthetic lift forecast should pass?

The acceptable error depends on the decision, the units being measured and uncertainty in the human results. Set the tolerance before evaluation, then record which uses the evidence supports and which remain untested.