What changes should trigger revalidation of a synthetic workflow?
By Lewsearch
Revisit the evidence when a change could affect the audience, measurement, output or decision covered by a workflow's previous acceptance. Review more than the model version: instruments, supplied context, response processing and intended use can matter too. Decide which uses may continue, which need narrower limits and which must wait for further evidence. The necessary checks should follow the change and its consequences, rather than one universal retesting rule.
Reopen the accepted use, not only the benchmark score
Start from the existing acceptance record. What audience, question types, stimuli, analysis and business decision did the evidence support? Which limitations were agreed? An earlier pilot does not automatically authorize a new application simply because the product name is unchanged.
The NIST AI Risk Management Framework connects evaluation to deployment conditions in MEASURE 2.3 and calls for production monitoring in MEASURE 2.4. GOVERN 1.5 addresses planned ongoing monitoring, periodic review and defined responsibilities. It is a voluntary framework, not certification of a synthetic research provider. The change-review procedure below is our proposed buyer application of those principles.
Ask what the change could invalidate. A new customer population may leave the original audience-specific evidence insufficient. A changed answer-coding rule may alter the reported base. An export-label correction may call for an output check without requiring the same work as a new population study. Document the reasoning instead of automatically demanding, or automatically skipping, a full evaluation.
Map each change to the permission and evidence it affects
Use a short change record before the revised workflow is relied on for a consequential decision. Describe what changed, when it took effect and which accepted uses might be affected. Ask for enough information to plan the review; this does not require a supplier's proprietary implementation.
| Change to review | Potentially affected use | Evidence to plan |
|---|---|---|
| New audience or business decision | Reliance on acceptance established for a different audience or task. | Use-relevant human evidence and scoring where the new claim requires it; an explicit statement of what remains unsupported. |
| Question, stimulus or response-processing change | Interpretation of answer shares, eligible bases or the resulting recommendation. | Instrument, coding and output checks; assess whether the altered measurement claim needs a human comparison. |
| Model, calibration, prompt or supplied-context change | Uses whose prior acceptance depended on the earlier setup. | Matched regression checks and a review of whether unexposed, use-relevant human evaluation is needed. |
| Display or export-only change | Readers' interpretation of previously assessed results. | Verify labels, values, bases and export fidelity. Do not assume every cosmetic edit needs a complete new human study. |
| Unexpected disagreement with human evidence | Reliance on the affected use, even if no version change was announced. | Investigate the instrument, reference and method; reconsider acceptance while consequential uncertainty remains unresolved. |
A release containing several changes needs an assessment of their combined effect. A series of individually small edits can still change the workflow that was originally assessed. If the change is poorly described, record that uncertainty and decide whether the affected use can responsibly continue; silence is not evidence that nothing important changed.
Keep known regression checks separate from unexposed evaluation
An archived human comparison can help detect a regression: does the revised workflow still meet an established check on material the team already knows? Keep it, with its original instrument, outputs and scoring rule. Also record whether its findings have been disclosed or used for calibration, prompt changes or run selection. An exposed comparison is not an untouched test merely because it is run again.
The scikit-learn cross-validation documentation explains that adjusting settings to improve performance on a test set can leak knowledge of that set into model selection, so its score no longer reports generalization performance. Applying that principle here means distinguishing maintenance checks on familiar material from evidence the supplier has not used to improve or select the workflow.
For a broader or consequential new claim, assess whether an additional relevant human comparison needs to remain unexposed until the synthetic delivery is locked. Use the blind-validation guide for that separation. A fresh comparison does not mean a universal requirement to commission a new human study for every minor edit; the research owner should justify the evidence needed for the affected claim.
Fictional example: an update fails a known release check
Imagine a buyer maintaining a two-option message screen. An archived human comparison has an option-A share of 58%. The original synthetic output was 56%; the revised output is 65%. All values are invented to explain the review, not a Lewsearch result, customer study or estimate of typical performance.
The absolute differences from the reference are 2 and 7 percentage points. Suppose the buyer had agreed, solely for this fictional check, to flag a release when that difference exceeds 5 points. The revised result is flagged. Five points is an invented example rule, not a recommended acceptance threshold, confidence interval or margin of error.
Assume the reference answer was disclosed after the original pilot. This rerun is therefore a check against a known comparison, not fresh evidence of generalization. The flag gives the owner a reason to hold the affected use and investigate. It does not identify the cause of the difference or establish how the revised method performs elsewhere. Check coding and run variability as appropriate before attributing the change to the model.
Passing this one item would not settle the release either. Review all agreed checks and priority audiences. If the proposed release also extends the screen to a different audience or decision, decide what additional evidence that extension needs rather than borrowing permission from the archived item.
Write a continue, limit or hold decision
The useful deliverable is a dated decision about a defined use. Ask the research owner to record:
- The prior accepted use, limitations and evidence record.
- The change, its effective date and the uses it could affect.
- The comparisons and compatibility checks completed, including failures and unresolved items.
- Which evaluation findings were already known or used in tuning, and which remained unexposed.
- The decision: continue the specified use, narrow it to an evidence-supported scope, or hold it pending further work.
- The responsible owner, any conditions or human-research requirements, and the next review date or event.
Do not silently carry an old acceptance label onto the revised workflow. Where the change leaves a use unsupported, withhold permission for that use until its evidence requirement is resolved. Preserve historical outputs under their original method and date; a new review does not rewrite what the previous study reported. Continued use elsewhere needs its own documented justification.
The AAPOR Code of Professional Ethics and Practices calls for disclosure of methods, processing and AI use. A buyer's maintenance record can extend that transparency into a handoff between versions. It is not an AAPOR approval process, and a completed form does not establish that a workflow is accurate.
Review the workflow even when there is no announced upgrade
A new decision, a different population or unexpected human evidence can reopen the question without a software release. Agree who receives supplier change notices, who monitors relevant performance and when the evidence will be reviewed. Set the review frequency around the use and its consequences; this guide proposes no universal monthly or quarterly schedule.
For a time-series study, the tracking guide adds checks for joining results across method changes. This page addresses the separate decision about whether a changed workflow may keep its accepted use. Bring that use, the change description and the decision to a Lewsearch demo, and agree on the evaluation scope before relying on an updated process.