What is the denominator of 95%?
What the evaluation covers
- Separate criteria for answers, data access, tool actions, and completed work
- Test sets that include denials and handoffs, with grading based on authoritative state
- Conditions for holding, stopping, or limiting a release
This guide is for teams shipping support assistants, internal search, document processors, and order or CRM agents. Its subject is the behavior of the deployed feature.
A hypothetical support feature passes 190 of 200 requests. The dashboard reports 95%.
| Request | Tested | Passed | What a failure means |
|---|---|---|---|
| Public product information | 120 | 120 | A product condition was misstated |
| Own order status | 50 | 48 | Order status or lookup was wrong |
| Other customer or tenant | 20 | 17 | Restricted information was disclosed |
| Approved order update and duplicate requests | 10 | 5 | An approval, update, or duplicate control failed |
| Total | 200 | 190 | Different failures share a single total |
The access group has three failures and the state-change group has five. Add 800 easy product questions that all pass, and the score rises to 990/1,000, or 99%. The eight risk events are unchanged.
This is an illustration of how the test mixture changes an aggregate, rather than an estimate of production error rates. Show the result for each work type and risk category.
NIST's Generative AI Profile advises against extending narrow and anecdotal evaluations to broader contexts. Benchmarks and demos can inform plans. Acceptance needs evidence for the organization's users, data, and policies.[1]
Which combination are we evaluating?
The same model can behave differently when its retrieval corpus or tool permissions change. Record the model identifier, system instructions, retrieval settings and document versions, tool definitions, authorization policy, and application revision.
A clear returns answer can cite last month's policy. Correctly formed arguments can lead to an unauthorized lookup if the server accepts another tenant's order. A generated “done” message does not create a state change by itself.
Define the feature under test from the user's screen through APIs, business state, and the handoff to operations.
What evidence does conventional QA provide?
Login, session expiry, permissions, calculations, persistence, and duplicate prevention have rules that can be asserted directly. Sufficiency of an explanation, relevance of a follow-up, and whether a cited source supports a claim need explicit evaluation criteria.
Use API, browser, and regression tests for system contracts. Evaluate how the AI interprets their results as a separate judgment. The difference points to the right owner when something fails.
Verification that supports a release decision — Verify critical journeys and connect observed results and remaining conditions to the release decision.
Which seven questions define a pass?
Use this example to find gaps in acceptance criteria for a business workflow.
| Question | What to examine | Evidence | Example failure |
|---|---|---|---|
| Is the answer correct for this task? | Facts, calculations and rules | Expected result, policy and grading reason | Returns described without an eligibility condition |
| Does the evidence support it? | Retrieved text, citation and effective date | Document ID, version and passage | Unrelated citation used to imply support |
| May this person see the information? | Tenant, role, and object | Request principal and server decision | Another buyer's order is summarized |
| Were actions within scope? | Tool, parameters and approval | Execution inputs, approved payload, server outcome | A lookup authorization allows cancellation |
| Did the work finish? | Database and external system | Before-and-after state and operation ID | Chat says done with no ticket in the system |
| Does it stop or hand off safely? | Retries, unknown outcomes, manual handling | Reason, handoff payload, stopped state | Unknown payment is repeated |
| Are time and cost acceptable? | Latency, rework and total cost | Elapsed completion time, calls and review | Fast response with a growing review backlog |
Google Cloud's evaluation guidance treats response quality, adherence to instructions, grounding, and tool use as distinct dimensions.[2] Use separate criteria where the failures have separate consequences.
Rules and authoritative records can grade a refund amount or an authorization decision. An LLM grader has a different role.
Which tests should expect a refusal?
Start with one workflow. For a public product assistant, first test ordinary questions, boundaries, and handoffs. If order changes are out of scope, test that such requests receive useful guidance and create no change.
The following groups cover a hypothetical support feature.
| Group | Input | Expected behavior |
|---|---|---|
| Routine | Return window under the current policy | Cite the effective policy |
| Missing key fields | Refund request with no identifiable order | Ask for required details and hold |
| Missing or conflicting evidence | Contract terms differ from public policy | Resolve precedence or hand off |
| Access boundary | Request an old tenant's order after switching tenant | Enforce current entitlements and withhold details |
| Changed state | Customer changes quantity or recipient after approving | Present new parameters and obtain the applicable approval |
| Unknown outcome | Connection drops during an external operation | Reconcile the outcome before trying again |
| Instructions in external content | Document asks for an unrelated action | Treat document text as content and enforce execution policy |
Production cases can improve coverage. Before using them, review processing purpose, access and retention. Remove unnecessary details and assess whether order numbers or unusual events could still identify a person or organization.
How are development cases separated from final evaluation?
Developers use familiar examples to debug and check recurring problems. When the same inputs guide prompt edits and determine acceptance, improvement on unseen requests is hard to establish.
Reserve a set for release decisions, record exposure, and refresh it when it becomes familiar. Control who can see it and who owns it. Google's safety guidance distinguishes development evaluation from separate assurance evaluation.[3]
What should grade each outcome?
Anthropic describes code-based, model-based and human graders for agent evaluations. Choose the evidence that resolves the question.[4]
Use program assertions and server records for amounts, order state, duplicate actions and access. Compare open-ended language on its meaning with an appropriate rubric.
A model can flag missing explanation or weak evidence. Record its rubric, sources and version, then compare a sample against human judgments. Track unrun, blocked, and inconclusive tests separately from passes. Report both results among gradable runs and coverage of the original plan.
Multiple graders may share a bad policy or bias. Published work describes effects of answer ordering and verbosity. Swap answer order and inspect disagreement with human reviewers.[5]
Who owns the expected results?
If engineers and support staff disagree, resolve which policy applies and when. Track who approved the expectation and changes to it so future score changes remain interpretable.
Some cases remain legitimately disputed. Restrict automation or require manual judgment for those cases and record the decision.
What state remains after the conversation?
Consider a hypothetical order-update test.
- A customer asks to change the shipping address on their own order.
- The AI shows the destination and order and requests approval.
- The customer changes their mind and asks to change only the recipient. No operation should be dispatched at this step.
- The AI presents the new payload and receives the required approval. Inject a lost response on the downstream request.
- The customer asks whether the change is complete. The feature must verify the outcome before confirming completion.
The first reply alone cannot settle this test. Inspect recipient, address, approved parameters, number of actions, and the exact completion status shown.
Derive expectations from policy. The address stays fixed without its approval. If changing the recipient requires verification, the tool must enforce it. An ambiguous external result requires reconciliation or handoff.
Allow valid alternative tool sequences when they produce the correct result. Require identity verification, amount limits and approval wherever the policy applies.
Which rates describe the actual work?
An assistant can improve response rate by guessing or performing requests it should escalate. Sending every request to a person also fails to deliver the intended automation.
Keep distinct denominators.
| Metric | Denominator | Question |
|---|---|---|
| Success on answerable requests | Requests marked answerable in advance | Does it handle eligible questions? |
| Correct handoff rate | Requests that require human review or escalation | Does it stop when required? |
| Unnecessary handoff rate | Requests eligible for automatic handling | How much avoidable work reaches a person? |
| Execution completion rate | Requests with the required permissions and preconditions | Did authoritative state change correctly? |
| Prohibited action rate | Valid runs intended to exercise that risk | Did an unauthorized read or action occur? |
Record unique cases separately from trials. Ten cases repeated three times mean 10 cases and 30 runs. Report execution failures and inconclusive runs alongside scores and explain their causes.
If the denominator is unclear, settle the scope and policy before publishing a metric. Specify which requests the system accepts and what it promises to do with them.
Zero observed events describes the tested inputs, permissions, and runs. It cannot characterize untested requests or guarantee safety in production.
How do repeated evaluations stay comparable?
Run important scenarios more than once. Record successes, failures and failure types for all runs.
Reset unintended state from old orders, caches, transcripts or temporary files. A second run against an already completed order can fail for a different reason from a model regression.
Use the same cases and rubric where possible. If the corpus and judge also changed, record that confounding factor. A stub proves behavior under its simulated responses. Check real latency, authorization and duplication in an isolated integration environment.
Repeating tests helps a team avoid using a lucky trial as release evidence.[4]
What should the release hold and stop?
Suppose support is preparing product answers, order lookup and refunds. Public answers satisfy the criteria. Order lookup still has an authorization issue and refund retries are unverified.
Release public information only if the unready routes are disabled at the tool and server boundaries. Test the restrictions by sending requests for refunds and prohibited order lookups.
Examples of release holds include the following.
- A request can return another customer's data.
- An external message or mutation can happen without authorization.
- An operation with an unknown result can be repeated.
- An important claim is presented as certain without evidence.
- The evaluated and deployed configurations cannot be reconciled.
- A stop leaves outstanding tasks with no owner.
Record the approved scope, residual limitations, and monitoring owners. The general release decision criteria cover how to record a launch decision and the risk it accepts.
What changes invalidate release evidence?
Even with unchanged code, a model, prompt, source corpus, policy or connected tool can change the behavior. Identify the evidence to refresh with the change.
A wording update can omit a mandatory notice or handoff. A new document changes grounding and effective dates. A new tool adds an execution path. A lower-cost model needs the same business and exception checks.
Turn a live failure into evidence for the layer that caused it. Fix incorrect policy, expand retrieval tests for missing documents, and verify authoritative state for execution errors. Turning production failures into regression tests connects fixes to verification.
Record configuration and evaluation versions, results by failure type, uncovered scope, release bounds, stop ownership, and when to reassess. Keep a record the next reviewer can use.
How can we scope one evaluation?
Choose one feature and a representative user journey. Record which data it reads, which state it can change, and who handles failures. Assign evidence to each of the seven questions before collecting a combined score.
