Define the right evidence with the policy owner
Retrieval-augmented generation (RAG) builds answers from retrieved material. A long result list cannot settle an ambiguous or undocumented policy. A test asking for a shipping rule should say whether the general rule alone is sufficient or an exception is required. Have the owner verify the applicable document, effective date, clause, exception, and source. See requirements and test traceability for versioning requirements and evidence.
Record the expected outcome together with the question and source version. The system may be able to answer, ask for missing conditions, defer because evidence is absent, or escalate a conflict. For “Will it arrive tomorrow even though it’s urgent?”, it may need the region and product type before answering. If the question is specific and the owner cannot verify an applicable rule, mark a source or evaluation defect before inventing an expected answer.
A questioner’s role and the policy date matter. If a source is not available to a given role, use authorized information or direct the user to an appropriate channel. Keep the identity context and applicable version in the case. A replay might call for today’s policy or the policy that applied to an old order.
Follow the evidence through the answer pipeline
Use a run ID to connect authorized source material, retrieved results, the assembled model input, and what the user saw. A source in the result list may never reach the model. Preserve document version and passage boundaries. Use fictional data in tests and authorized references in production rather than retaining source text everywhere. See AI agent observability for connecting the run’s stages.
| Failure stage | Evidence to check | Next step |
|---|---|---|
| Source or expected answer | Approved version, effective date, exceptions, reference answer | Resolve missing or conflicting policy and check that the system can defer |
| Index and retrieval | Index version, rewritten query, filters, candidate IDs, ranks, passages | Distinguish indexing, filtering, and ranking defects |
| Context assembly | Candidates before and after reranking, model input, truncation and exclusion | Adjust passage grouping, reranking, or input budget |
| Generation and evaluation | Actual input, claims, citations, and deferral behavior | If evidence was sufficient, improve how the answer applies it |
| Presentation | Generated citations and actual document versions opened from the UI | Correct citation mapping and rendering |
Treat the table as a starting point for finding the earliest failure. Fixing retrieval may still leave a generated answer that overcommits on incomplete information. Add both failures to regression coverage. RAGAS research[1] and Google Cloud’s metrics guide[2] describe separate assessment of retrieved context and answer faithfulness. Define what those measures cover in your own business rules.
Verify the paths beyond the happy path — Reconstruct requirements from behavior and exercise exceptions in input, permissions and external calls.
Reproduce the same shipping error two ways
In a fictional retailer, standard items ship the same business day if ordered before 2 p.m. Items requiring installation instead follow an installation schedule. A customer asks whether an installation item ordered at 11 a.m. will ship today. In two runs the system says “Orders before 2 p.m. ship today.” Each run calls for a different investigation.
In the first run, only the general policy was retrieved. The installation exception was indexed but ranked too low. Check whether indexing included the approved passage, and whether rewriting “will my installation item ship today” as “same-day shipping rules” lost the product condition. Use fields or applicable product filters where appropriate and verify the general rule is still retrieved for standard items. A change to answer wording cannot reliably recover evidence the model never received.
In the second run the installation exception was among the top candidates. Confirm that its full text reached the prompt. If trimming removed the exception, make assembly preserve the general rule together with applicable exceptions and retest crowded candidate lists. If both passages were present, examine how the model chose between them. Score whether it avoided guaranteeing shipment and asked for the installation schedule rather than simply looking for the word “installation.”
Run a narrow comparison keeping the generator and retrieval settings fixed. Compare the production context against a complete approved context. If the correct answer appears only with complete context, improve evidence delivery. If both are wrong, inspect how the generator applies policy and handles uncertainty. A correct answer on incomplete input can still be a guess. Check whether its actual citations support the conclusion.
Reproduce missing and conflicting evidence too
Remove an applicable source or create two conflicting versions. The system may identify an installation item but lack access to its scheduled date. If conflicting policies cannot be resolved by effective dates or authority, it should summarize the conflict and direct it to an owner. Distinguish unavailable retrieval from a successful search with no evidence. A generic deferral can conceal an outage unless you check the run.
Tie metrics to questions and denominators
Retrieval relevance asks whether results match the question. Passage recall@5 asks what proportion of known relevant passages appear in the top five. If two of three annotated passages appear, recall is 2/3, or about 66.7% for that question. The denominator only covers passages known to the evaluation. Cases without any relevant passage need a separate measure for deferral.
For tasks needing a rule and an exception, also measure the proportion of cases with all required evidence present. A higher mean recall can still miss a critical exception. Raising the limit from five to twenty can bring in extra evidence and unrelated material, leaving important text vulnerable to truncation. Track retrieved and delivered evidence.
Faithfulness asks whether the answer’s claims follow from the provided context. Business correctness asks whether they meet the approved policy for the question. Quoting an obsolete rule accurately can be faithful and still wrong. A sound answer with a link to the wrong clause also fails citation verification. Check amounts and state against authoritative data and use owners or validated evaluations for interpretation. Anthropic — Demystifying evals for AI agents[3]
| Comparison | Denominator or criterion | Failure to record |
|---|---|---|
| Retrieval of required evidence | Same cases with approved evidence | Missing relevant passage or version |
| Evidence delivered to model | Cases where retrieval found what was needed | Reranking, truncation, or metadata loss |
| Business answer and citation | Answerable cases and required claims | Missing exceptions, false promises, wrong citation |
| Deferral and handoff | Cases lacking evidence, with conflicts, or with restrictions | Unsupported conclusion or incomplete handoff |
| Execution | All planned cases or intake | Not run, error, or ungradable |
Changing rank, model, and prompt together makes it hard to attribute an improvement. Document your hypothesis and replay questions against the same sources and permissions. After assembly changes, verify that required clauses reached the prompt. After generation changes, fix the input and compare judgments. Repeat important cases where runs vary and log changes to expected answers or missing evidence. NIST AI 600-1[4]
