Record configuration changes separately from intended behavior
Even one line of a prompt can affect search, tool arguments, and deferral. Using the same model with a different index, embedding, tool schema, policy, or grader changes the system being evaluated. Record both configurations with their model identifiers, prompt templates and settings, retrieval versions, tools, policies, evaluation data, and scoring.
Verify which configuration produced the previous baseline. Traffic can reach different models through regions, accounts, feature flags, or error fallbacks. Retain the actual model ID returned by the API where possible and note where the version behind an alias is unavailable. Responses need not be identical for the team to document the decision criteria and conditions.
If a new policy ships alongside the candidate, distinguish promises that must carry over from new requirements. Compare both configurations under the old policy and evaluate the new policy separately. See risk-based regression test scope to assess business impact beyond the size of the code change.
Choose what to rerun based on the impact
One change can affect multiple groups of tests. A shorter prompt may omit a deadline or exception. Record the difference between cases not run and cases where existing evidence was reviewed and accepted.
| Change | Possible impact | Re-evaluation and evidence |
|---|---|---|
| Model, inference settings, or output limit | Classification, omitted facts, structured output, tool choice | Same inputs and arguments; repeat exceptions and deferrals; check latency and cost |
| Prompt or template | Instruction priority, language, format, and representations of approval | Cases touched by the revised language; security and customer status checks |
| Search index, chunking, or ranking | Finding valid sources, versions, exceptions, and citations | Required evidence, excluded restricted sources, links to model context and citations |
| Tool schema, policy, or access | Allowed actions and arguments, retries, actual outcomes | Denials, payments or messages, side effects and recovery |
| Grader or expected answer | Meaning and comparability of scores | Rescore the same output; check agreement with a person; record criteria changes |
| Provider retirement or routing change | Availability of prior model and behavior of replacement | Confirm support and rehearse alternative or manual processing |
If the change involves search or tools, run the baseline on equivalent test data. Reproduce representative documents and permissions using approved sources if the production index is unavailable. Use a test tenant and isolated tools to verify side effects without sending money or messages. An expected recipient and message in the test record give stronger evidence than the model saying it sent mail. Anthropic — Demystifying evals for AI agents[1]
Maintenance that keeps results trustworthy — Run with stable selectors, classify failures, repair their causes and verify the result again.
Compare paired cases
Use A for the current configuration and B for the candidate. Supply the same question, role, order status, timestamp, tool response, and expected outcome. Identify improvements and regressions with the same denominator. Keep cases for newly required behavior separate from existing promises. Comparing different samples confounds configuration with input difficulty.
Use development cases to improve the system. Keep a separate set for the final decision. Merely rephrasing the same order and conditions can leak the case across both sets. Group similar cases by data source or business pattern when checking the split. Preserve important production failures in regression coverage. Google’s guide to model and system safety evaluations[2] describes development and separate assurance evaluations.
Repeat important cases when responses vary, under comparable repetition rules. Keep case count separate from runs. If all 20 cases run once and four receive three additional runs, you have 20 cases and 32 runs. Preserve a severe failure even if it occurred on only one attempt. Report not-run cases and timeouts separately so that successful responses are not the only denominator.
When model, prompt, and retrieval ship together, you can test the full configuration first. To isolate a cause, try the current model with the candidate prompt or the candidate model with the current prompt. Some combinations may be impossible, such as when the candidate requires a different schema. Document which comparisons are unavailable and what remains uncertain.
A fictional case where accuracy rises and the release waits
A fictional support team tests issue classification and deferral when no policy evidence is available. The owner asks for improved classification with no unsupported refund or delivery commitment. The trial uses the same cases for both configurations and keeps denominators separate.
| Measure | Baseline A | Candidate B | Change |
|---|---|---|---|
| Successful classification out of 100 | 88 | 94 | Two A successes lost and eight A failures corrected; net gain of six |
| Correct deferral out of 20 unsupported cases | 20 | 16 | Four A successes lost; down from 100% to 80% |
| Unsupported commitments out of the same 20 | 0 | 4 | Newly introduced failures; release criterion not met |
Inspect which classifications improved while holding the rollout for four unsupported commitments. Review the source and policy input in each case. If a failed call was interpreted as approval, fix that contract. You can test the candidate only for classification and verify that evidence checks and customer commitments still use the baseline path.
The six-case gain describes the sample. Review representativeness, cases sharing the same customers and orders, and rare high-impact failures. When the difference is small or reviewers disagree, add coverage and bring in the owner. A small change can still contain a consequential new failure.
Shadow testing and limited canaries provide different evidence
In shadow testing, the candidate receives production-like input without sending its response to customers or performing live writes, transfers, or outbound messages. Record intended tools and arguments and direct execution to isolated substitutes. Confirm authority to use and retain the input. Useful evidence includes formats, routing and latency; live customer and provider outcomes are still unavailable.
A limited canary exposes a defined subset of live traffic. Decide what stays together, such as a conversation, so an ongoing case does not switch. Begin with low-impact cases and record live volume, latency, user outcomes, and human handoffs. Assign a rollout owner and stop conditions. Rare failures may be absent from a small sample, so retain deliberate negative tests. Google SRE — Canarying Releases[3]
A provider can retire a model or move an alias to a different version. Check published notices and test actual recovery. If an old version is unavailable, plan for another validated configuration, paused automatic decisions, or manual processing. Reconcile actions already executed as part of rollback-ready deployment. One example of provider notices is OpenAI API Deprecations[4].
Comparison record for the release decision
| Field | What to record |
|---|---|
| Goal and controls | Target improvement, candidate changes, shared inputs and policy, unavoidable differences |
| Identifiers and criteria | Versions of model, prompt, retrieval, tools, permissions, dataset, and grader |
| Coverage and denominators | Case and run counts, successes, failures, not run, and ungradable |
| Improvements and regressions | Paired IDs and the action or hold reason |
| Operating evidence and rollout | Shadow or canary scope, observed outcomes, owners, and stop conditions |
| Recovery or alternative | Validated configurations and verification time, plus owner of in-flight work |
Scores can move after release because traffic, documents, or criteria change. Link the serving version to the observed inputs and periods. Turn approved production failures into regression cases. Record changes to source quality and evaluation criteria alongside model changes. Google Cloud’s guide to deploying and operating generative AI applications[5] covers continuous evaluation and monitoring.
