Inspect the reference first
LLM judges can help assess qualities such as instruction following and clarity at scale. Check the actual system state for completed work, and have domain experts resolve unsettled policy before grading it. Verify which failures the score detects.
A usable reference explains what should happen under the user's conditions at the time of the case. Identify and version the applicable policy, API contract, or approved decision. Record effective dates and exceptions. A reference can include required elements, permissible variants, prohibited claims or actions, and when the system should defer. Include dates in the scenario if both current and historical answers can be valid.
Suppose a fictional exchange policy says seven days on the reference sheet. An approved provision gives a particular membership tier fourteen days for eligible items. Two models quote the general rule and answer seven days, and the grader passes them. Have a policy owner verify the effective version, tier, and product type. Correct the reference, trace other affected results, and regrade both baseline and candidate with the same corrected standard.
Keep a case indeterminate when evidence conflicts or an exception remains undecided. If policy already requires escalation when evidence is missing, an answer that acknowledges uncertainty and escalates can pass. Record missing ground truth separately from an appropriate refusal to guess. See requirements-to-test traceability for connecting the evidence to changes.
Assign each judgment to a suitable grader
An attractive response can mask a failed transaction. For actual refunds, check the approval, amount, and payment ledger alongside response quality. Use deterministic checks for conditions observable in the database or test harness. Model judges can assess clarity or completeness against approved evidence. A high-impact violation can fail independently of stylistic scores. Anthropic describes different roles for code, model and human graders and the need to calibrate model graders with people. Anthropic agent evaluation guide[1]
Record judge model, prompt and rubric versions, reference versions, parameters, parser, and execution time. Capture item-level verdicts and supporting evidence. Freeze documents retrieved during grading and record the excerpts provided. Separate call and parsing errors from product errors, and link reruns to the original. Selectively retaining successful grading runs can change the comparison.
Keep the evaluated response structurally distinct from grading instructions and test what happens when response content looks like an instruction. Give the judge only required tools and data. When examining a judgment, compare its explanation against actual answer evidence. See reviewing AI-generated test cases for validating expected outcomes before grading.
Blind the response identity and swap its order
Hide supplier names, model IDs and other unnecessary metadata when comparing responses. Use the same task and rubric and then evaluate with the answer order reversed. Keep response identifiers in the audit record. Track ties and inconsistent results separately to investigate differences. Zheng et al. tested position and verbosity effects under MT-Bench conditions and used order swapping. Their limited sample did not establish self-preference conclusively. Zheng et al., original study[2]
Consider two fictional replies that state the same approved exception. One uses a single paragraph and the other repeats the conditions four times. If scores differ, compare verbosity and required facts separately. If swapping positions makes the first reply win regardless of content, suspend the preference judgment. Also investigate a consistent preference if an expert finds that it misses an applicable rule. Order and repetition tests measure grading stability. Correctness still needs evidence.
A controlled experiment can match formatting to isolate its effect. If edits change meaning, the reason for a score change becomes unclear. Compare original responses for the product test, and give content-preserving variants their own IDs for bias testing. To check suspected self-preference, blind the generator, balance responses from different origins, and have domain experts grade to the same standard.
Verify the paths beyond the happy path — Reconstruct requirements from behavior and exercise exceptions in input, permissions and external calls.
Evaluation audit table
Use this table to request evidence from the evaluation owner. Review the score alongside sample sizes, indeterminate verdicts, and high-impact errors the judge passed.
| Audit area | Question | Evidence and follow-up |
|---|---|---|
| Reference | Do approved policy and state apply to the case? | Reference ID, effective date, owner, list of cases using a flawed reference |
| Rubric | Can it detect required conditions separately from good prose? | Passing and failing samples by criterion, standalone high-impact failures |
| Reproducibility | Did the same model and prompt apply? | Model, prompt and parser versions, outputs, rationale and call failures |
| Order, length and origin | Can identical content win on placement or verbosity? | Swapped orders, length variants, blinding procedure, and disagreements |
| Human agreement | Does the judge miss important errors? | Domain-specific calibration set, adjudicated outcomes, and costly false passes |
| Data independence | Were tuning examples or data used in both sets? | Access events, dataset versions, exact and near-duplicate review, replacement decisions |
| Interpretation | Can someone reproduce the judgment and measurement? | Denominators for pass, fail and indeterminate, exclusions, owners and open risks |
In a fictional set of 100 calibration cases, experts rule that 20 fail. The judge marks 15 of those as failures and passes five. Among the 80 cases passed by experts, it passes 78 and fails two. Agreement is 93 of 100, or 93%. The judge missed five of the 20 expert failures, a 25% false-pass rate on that subset. If a missed failure includes an unauthorized payment or permission error, withhold approval in that area regardless of aggregate agreement.
People can disagree as well. Ask two domain experts to grade independently and record the evidence behind their verdicts. Assign an owner to resolve rubric or reference ambiguities and retain the original disagreement. Budget ongoing human review for cases where automation remains weak. Multiple judges can share errors through common data, a vague rubric, or a bad reference. Compare any ensemble with human outcomes.
Disagreement analysis form
Track each unresolved case by ID and direct observations. Then establish whether to change the grader, reference, dataset, or product and locate the affected results.
| Field | Example or recording rule |
|---|---|
| Case ID, workflow, evidence version | Exchange E-104, membership tier, policy 3.2, relevant effective date |
| Model outputs and business state | Original output, source, actual approval, human intervention |
| Initial human judgments and resolution | Roles, evidence for disagreement, adjudicator and date |
| Judge verdicts | Both orders, criterion values and reasons, whether each response was valid |
| Cause and corrective action | Product defect, reference error, vague rubric, order or length effect, evaluator error, unresolved |
| Retest scope | Affected sample, rule and version, owner, baseline and candidate rerun conditions |
Separate development, final evaluation, and regression sets
Development cases help tune prompts and retrieval. Frequent access to answers changes what their scores can tell you about new work. Keep a final set held back for release decisions. If repeated exposure becomes part of tuning, mark those cases as development data and assemble new holdouts. Regression cases check whether known failures return and can remain visible to developers. Google's responsible AI evaluation guide[3] and Anthropic's agent evaluation guide[1] describe distinct development, independent, and regression evaluation roles.
Two different requests can be derived from the same support ticket. Split by source, customer, document, or scenario group. Use normalized text or hashes to find exact duplicates, and similarity or representative review for paraphrases. Keep lineage for synthetic cases. Research on public benchmarks has investigated potential overlap with training data. If a provider's full training set is unavailable, record which paths you checked and supplement public tests with independent tasks. Deng et al. on benchmark contamination[4], NIST generative AI profile[5]
If the new model scores better, check whether both runs used the same grader and evidence. If the judge or reference changed, replay the fixed responses under the previous and current measurement conditions to isolate the cause. Correct results affected by data exposure or duplication and use replacement cases. Report which failure types improved and which still need people, with the supporting sample.
