Can synthetic research transfer across languages and countries?

Buy a defined set of market and language combinations

Start with a coverage table in the brief. Each row should name the country or market, questionnaire language, target population, product or message, outcome, and intended decision. Include the human reference and the evidence available for that exact combination. A country with two research languages may need two rows; two countries using the same language still need separate evidence about their populations.

Ask the provider to distinguish supported operation from demonstrated accuracy. Being able to accept a translated questionnaire is an operational capability. Evidence about local people is a measurement claim. Agree which combinations the deliverable covers before commissioning estimates or comparing markets.

This guide proposes a buying procedure. It does not establish that a particular provider supports any language or country, or that every market requires the same validation budget.

Check meaning before comparing percentages

Pew Research Center describes a multi-step translation process involving professional linguists, local research organizations, review and approval before fieldwork. Its goal is comparable meaning across languages rather than literal word substitution.

Apply that distinction to the complete instrument: question wording, answer options, instructions, routing and the stimulus people are asked to judge. Ask a local reviewer to flag unfamiliar category terms, assumptions about product availability, and response options that do not fit the market. Keep the underlying concept consistent, and record any adaptation that changes what is being measured.

The European Social Survey’s Round 10 Translation Guidelines, section 3 describe a team process that includes translation, review, adjudication, pretesting and documentation. The process brings translation and survey expertise together. Section 4 also addresses coordination between countries sharing a language. Ask how the final version was checked in the target language, including with human participants.

These are methods for human surveys. Our recommendation is to use them to check the reference instrument and the questions supplied to a synthetic system. Good translation makes the intended comparison clearer; it does not validate the generated answers.

Separate language sensitivity from local accuracy

A useful diagnostic asks whether a synthetic estimate changes when equivalent versions of a question are supplied in different languages while the intended audience stays fixed. Record the audience definition, version and run procedure. If the output changes enough to alter the decision, investigate before choosing the more convenient result.

That diagnostic still does not tell you which version matches people. Compare synthetic outputs with relevant human evidence collected in the intended language and market. A check using bilingual volunteers can help investigate comprehension; do not treat it as a representative estimate for everyone who uses either language.

Keep the reference population, eligibility rules, timing, response options and outcome aligned. If one reference covers existing customers and another covers all adults, a difference between their results is not a clean country comparison. Document such mismatches instead of explaining every difference as culture.

NIST’s AI Risk Management Framework, section 3.1 connects accuracy evaluation with realistic test sets representative of expected use and documented test methods. Applying that principle here means evaluating the combinations you intend to buy. NIST does not certify synthetic survey panels or prescribe this checklist.

Worked example: a new language is not a new market test

Consider a fictional procurement brief. A team has evaluated a synthetic message study among English-speaking customers in Market A. It now wants to buy a Spanish-language study for customers in Market B. The vendor demonstrates fluent Spanish responses using the original Market A audience profiles. No actual provider or research result is described here.

• Market A / English: the earlier evaluation is evidence only for its tested audience, instrument and outcome.
• Market A / Spanish: the demonstration shows that Spanish output can be produced. A controlled language-switch diagnostic could investigate consistency, but neither establishes accuracy for Spanish-speaking customers.
• Market B / Spanish: local questionnaire review and relevant human comparison remain missing. The demonstration did not test this combination.

The buying decision is to reserve Market B approval until those gaps are addressed. Keep the audience definition fixed for the language diagnostic, then test the intended Market B population separately. Do not change both language and country and assume the source of any difference is known. If the budget covers only the demonstration, label its deliverable exploratory rather than a validated expansion.

Put five requirements in the buying brief

• A versioned instrument. Request the source questionnaire, approved translations, response codes, routing and adaptation record. Identify who approved local meaning and what the human pretest changed.
• Evidence for each intended use. Request relevant human comparisons for the market, language, population and outcome. Mark missing combinations as untested. Evidence in one combination can inform a test plan without certifying the others.
• Protected evaluation outcomes. Agree which human results may inform development and which remain untouched for evaluation. Freeze the synthetic predictions and scoring before revealing the reserved outcomes. Disclose possible prior exposure to public reference results.
• Separate acceptance decisions. Before results arrive, set tolerances around the business decision and human-reference uncertainty. Score local response shares and the cross-market differences you will actually use. Specify how inconclusive comparisons and failed runs will be handled.
• A delivery and change record. Label simulated outputs, human references and untested combinations separately. Record questionnaire and service versions. Decide when a new language, market, instrument adaptation or service change requires another check.

These are proposed procurement requirements, not a universal certification standard. A buyer making a small exploratory screen may choose a different evidence threshold from a team allocating a major launch budget. Put the permitted use in writing so a limited pilot does not quietly become a global forecast.

Choose the next step market by market

If translation review is incomplete, resolve the questionnaire before interpreting differences. If the instrument is sound but local accuracy evidence is missing, keep outputs exploratory and commission a suitable human comparison before using them for the consequential decision. If evidence supports only some combinations, scope the engagement to those combinations and list the remaining ones as untested.

For the wider evaluation plan, use the synthetic-panel validation guide. The subgroup guide covers accuracy within decision-relevant segments, and the reporting guide explains how to label different evidence sources.

For a Lewsearch discussion, bring the proposed market-and-language table to a demo. Agree on feasible scope and the evidence each intended use would need before commissioning the study.

FAQ

Does support for multiple languages establish local research accuracy?

No. Language support describes operation. Local accuracy requires evidence matched to the questionnaire, population, language, market and outcome used for the decision.

Can the same language version be reused in two countries?

It can be a starting point for review. Check local meaning, product context, eligibility and answer options in each country, and evaluate estimates for each intended population. Shared language alone does not establish comparability or accuracy.

Does a stable result across translations replace human validation?

No. Stability is a diagnostic of consistency under the tested language changes. Both versions could be consistently wrong about people. Use relevant human comparisons to evaluate that separate claim.