Can synthetic respondents replace a usability test?

Define the decision before choosing a substitute

An early review of button labels, a test of whether someone can recover from an error and a claim about completion time ask different questions. Write down the users, task, product version, device and context the decision concerns. Then specify the evidence you need: possible problems to inspect, observed barriers, successful task completion or a defensible comparison between designs.

The GOV.UK guide to moderated usability testing describes watching participants attempt tasks with a service. It recommends recruiting actual or likely users and designing believable tasks with clear goals that do not reveal the answer. That is a different evidence source from generated opinions about a screenshot.

This guide offers a buyer checklist, not a finding that every synthetic method fails or that a particular vendor can replace human usability research. Suitability needs evidence for the task and conclusion you intend to use.

Ask what interacted with the interface

Request a precise account of the input and execution. A model may receive a written description, a static image, a clickable prototype or access to a working service through an automated agent. These arrangements expose different information. A screenshot does not expose the complete behavior of a form, an error state or a keyboard interaction.

• Generated critique: which materials did the model receive, and which comments are hypotheses for inspection?
• Automated execution: which actions actually ran, on which version and environment, and what do the traces establish?
• Human observation: who attempted the task, what happened and what assistance did they receive?

An automated agent can execute a sequence of actions. Its success is evidence about that run under its configuration. Treating it as evidence of human ease, understanding or satisfaction requires a separate justification. Similarly, a functional check that a button submits a form does not measure whether the intended users can find it or understand the outcome.

Ask for the artifact behind each finding: the supplied stimulus, execution log or human session record. A generated first-person statement such as “I found the settings easily” should not be reported as a participant observation when no person attempted the task.

Agree what a usability claim must show

For a human task study, agree the starting point, success condition, allowed assistance and stopping rule before running sessions. Record partial completion, errors and interventions rather than forcing every attempt into a favorable result. If time is reported, define when timing starts and stops and how interruptions or think-aloud instructions affect its interpretation.

For a proposed synthetic substitute, ask for a comparison against relevant human task evidence that was not used to tune the result. Check whether it detects the problems that matter for this product, misses important barriers or raises issues that human evaluation does not support. Include unsuccessful tasks and consequential errors, not only examples where the outputs agree.

A demonstration on an unrelated website or agreement on interface preferences does not establish human task completion for your service. Define the intended use and acceptance decision before seeing the comparison. The synthetic-panel validation guide explains the broader discipline of protecting evaluation evidence and testing the proposed use.

Small human studies can help identify problems in the tasks examined. A claim about how often a wider population succeeds requires a suitable study design and uncertainty statement. Neither a handful of human sessions nor a large volume of generated cases automatically provides that population estimate.

Illustrative example: changing a delivery address

Imagine a fictional subscription service reviewing a new account screen. A synthetic critique describes the “Edit address” label as clear. An automated check opens the form, enters a valid address and saves it. Both are useful records of what was examined; neither records a customer recovering from an invalid postcode on a phone.

The team writes a separate task: change the delivery address before the next shipment. It agrees what counts as success and tests the relevant flow with intended users. Observers record whether users find the control, understand when the change takes effect and recover from validation errors. If the researcher explains the error message, that assistance is recorded.

The team can now compare the earlier hypotheses with what it observed. It should retain any uncovered problem even if the generated review sounded confident. The example is a proposed evaluation sequence, not a Lewsearch result or a claim that a particular design has been validated.

Evaluate accessibility with the appropriate evidence

The W3C WAI guidance on involving users in accessibility evaluation recommends combining evaluation with disabled users and assessment against accessibility standards. It cautions against generalizing from a few participants to everyone with similar or different disabilities.

Apply that distinction when buying synthetic feedback. A simulated disability profile does not document how a person used their assistive technology. Automated checks can be part of the evidence, but a generated account of a keyboard or screen-reader experience should not be labeled as an observed human session.

Ask which users, assistive technologies, browsers and tasks were included, which standards checks were performed and which gaps remain. Keep accessibility conformance claims separate from the findings of a limited usability exercise. This checklist is not an accessibility certification.

Commission a report that preserves the evidence type

Before commissioning work, request a report structure with these fields:

• Decision and scope: intended users, tasks, context, interface version and excluded uses.
• Inputs and execution: descriptions, images, prototypes or working service; human or automated actors; configuration and date.
• Finding and evidence: generated hypothesis, automated trace or human observation, with assistance and failures recorded.
• Comparison and limits: relevant human reference, protected evaluation material, acceptance criteria and unresolved gaps.
• Next action: inspect a proposed issue, revise the interface, run a human task study or gather evidence for a broader claim.

Keep generated comments distinct from participant quotations. Preserve the difference between a problem worth investigating and a measured usability outcome. The human and synthetic reporting guide covers those source labels; the questionnaire pretesting guide addresses the separate question of survey wording and routing.

For a Lewsearch discussion, bring the research decision and the interface materials you expect to evaluate. Agree on an exploratory scope and the human evidence required before relying on a usability claim. This guide does not establish that Lewsearch has validated usability prediction for your product.

FAQ

Can a synthetic respondent tell me whether a screen is confusing?

It can produce a critique to inspect. Record what the model saw and treat the comments as possible issues until the relevant interaction and user evidence support a stronger conclusion.

Does an AI agent completing a task prove that customers can complete it?

It establishes that the agent completed that run under its configuration. A claim about customers needs relevant human evidence and a suitable evaluation of the proposed inference.

Can generated feedback establish an accessibility result?

Generated feedback alone does not document disabled users interacting with the service or establish conformance to accessibility standards. Plan the relevant standards evaluation and user involvement, and report their scope separately.