Define the unit behind each claim
Proposal numbers are a starting point. A correct answer, a completed business transaction, and an unedited response tell you different things. Define which evidence measures each outcome so a new reviewer can apply the same criteria. The NIST generative AI profile recommends supplier due diligence, ongoing monitoring, and empirically validated methods for testing capability claims before deployment.
A statement like “95% accuracy” leaves the unit unspecified. A workflow may extract five fields per inquiry, and a single incorrect field can send the entire case to a human. High field accuracy can coexist with a lower rate of automated business completion. For an intake form, routing accuracy, notification, and updates to the underlying status may need individual checks.
First agree on the boundaries of one task. Consider answering a contract question from approved material and recording the outcome in a support ledger. Check the answer, its association with the correct record, and whether a required human handoff actually appears in the system. Handoff can be the successful safety outcome. For automation rate, count handed-off work separately. Anthropic's guide to agent evaluations[1] distinguishes the transcript from the final state observed in the environment.
Set the denominator from eligible intake. Record policy exclusions separately from eligible cases that were incomplete, handed off, or timed out. If a vendor declines to support inputs found in the common set, retain that difference as a scope limitation. Use case IDs to trace each result marked successful, failed, not run, or handled by a person.
Compare two fictional proposals on one table
Vendor A claims 95% accuracy. It reports 76 adequate answers out of 80 scored cases. Of 100 incoming cases, 20 complex inquiries were transferred to staff and omitted from scoring. Staff made material edits or reentered information before sending 12 of the 76 answers. Vendor B claims 70% automation, with 70 of 100 incoming cases completed without staff. B's number does not establish the correctness of responses or business records. These figures illustrate the comparison method.
| Proposal claim | Denominator and observed result | Evidence still needed for comparison |
|---|---|---|
| A has 95% accuracy | 76 adequate answers among 80 scored cases, from 100 received | Type of the 20 excluded cases, results before and after editing, final ledger state |
| A gives correct answers without edits | 76 adequate answers less 12 edited gives 64 out of 100 received, or 64% | Whether the system checked the actual outcome for those 64 and handled required handoffs correctly |
| B completes 70% automatically | 70 of 100 cases completed without staff | Correctness and permissions for the 70, erroneous automatic actions, and outcomes of the other 30 |
A direct ranking of A's 64% against B's 70% would still mix measures. A counts answer quality, while B counts completion without intervention. Give both vendors the same 100 cases and count verified completion, material edits, handoffs, errors and blocked actions on the same basis. Separate cosmetic editing from a new amount, recipient or conclusion, so the figures reflect the actual risk and labor involved.
A vendor-selected demo can explain a feature. For selection, include representative work, edge cases, and failure paths chosen by the buyer. Check overlap with demo cases. When trying the same input repeatedly, report the number of distinct tasks and actual runs. Ten repetitions establish what happened on one case under repeated execution.
Evidence connected from requirement to approval — Link requirements, tests, defect actions and approval records so teams judge quality consistently.
Twelve questions to send to every vendor
Procurement can issue a common form, the business owner can sign off on expected outcomes, and engineering can check evidence and reproducibility. If an answer depends on a future test, record the owner, evidence due, and date. Use observations for the acceptance decision.
- What counts as one successful task? Show the start, final business state, user response, and evidence for a sample case.
- What enters the published metric? Reconcile intake, eligible cases, runs, failures, handoffs, and exclusions by case ID.
- Where did people intervene? Include input fixes, draft review, retries, approvals, tool use, and vendor-side post-processing.
- Who established expected outcomes? Give the governing documents, versions, and policy owner the buyer can consult.
- How were cases used for development and final evaluation? Explain what was shared for tuning, the held-out acceptance set, and how you detect exposure or duplicates.
- Which configuration reproduces the result? Provide identifiers for app build, model and parameters, prompt, data and index, tools, permissions, and grading rules.
- How are high-risk exceptions blocked? Show actual tool and business outcomes for missing permissions, missing or stale evidence, external timeouts, and unsafe requests.
- What is included in the cost per task? Cover model, search, storage, integrations, retries and staff review under the same usage assumptions.
- Who reports and approves changes to models or search? Show impact assessment, retesting, and recovery for planned and unannounced dependency changes.
- Who owns incident actions through resolution? Assign initial response, feature suspension, vendor escalation, reconciliation, customer communication and verified resumption.
- What can we take over or export? Show data, evaluation evidence, configuration, document state and access mappings, including limitations and evidence of restore or deletion.
- What is unverified or different between demo and operation? List environment differences, missing evidence, unrun tests, known defects, operating limits, owners and decision dates.
Compare an ordinary day and a peak day. Operations can provide demand, review capacity, and deadlines by workflow. The vendor can explain how it limits retrieval or tool retries and which steps the quoted price covers. Failed cases may still generate usage charges and review time. Provide intake and verified-completion counts so procurement can compare the cost of the whole workflow.
Acceptance tests must survive a change of people or tools
Define input channels, types of work, user roles, effective dates for documents and permissions, and sources of outcome evidence. A representative set measures common flows. A separate set checks conditions where the system must stop, so critical failures are visible alongside normal performance. Use approved synthetic or anonymized data that the buyer can operate. Keep tuning examples out of the final evaluation. When engineering and business interpretations differ, record when and how the expected result was agreed.
| Proposal claim | Evidence to request before acceptance | Example acceptance test |
|---|---|---|
| Answers follow approved product policy | Policy ID, version, effective date, answer and citation, behavior when evidence is missing | Explain the rule from the effective version and hold or escalate a question without support |
| Only approved changes execute | Access mappings, tool and approval events, state before and after | Check approver, target, amount and recipient, and verify no change for denied requests |
| Customer work completes automatically | Case-level input, edits, retries, and final outcomes | Reproduce the rate using agreed exclusions and human-intervention rules and check final state |
| Quality holds after changes | Configuration inventory, notifications and approval, regression set, return criteria | Change model or documents and identify which previous evidence needs fresh runs |
| Cost and latency stay within limits | Count, input size, retry and review assumptions, actual usage | Check cost, latency and backlog by workflow and the behavior at budget thresholds |
| The internal team can take over | Evaluation set and references, export, configuration details, account handover | An internal operator uses the export to run agreed checks in a new environment |
For workflows governed by changing rules, record more than the model name. Link a run to the model identifier the provider exposes, prompt digest, document snapshot, retrieval settings, tool parameters, access policy, and grading rules. If the identifier can change without notice, record the observed configuration and date, the vendor's notification coverage, and the expected retests. Google Cloud's generative AI operations guide[2] covers input drift and continuous evaluation of live outputs.
An evaluation set should carry a case ID, source, owner, authorized usage, governing policy, and reference outcome version. Separate permissions for tuning examples and restricted acceptance material. Ask for the means and data to rerun tests for the items the internal team must maintain. Where export is restricted, agree whether equivalent customer-controlled data and checks can reproduce the necessary evidence.
Exercise both model replacement and document refresh
Change the effective version of a fictional policy in the test environment. The vendor demonstrates index refresh, and the business owner verifies that the new rule and still-valid exceptions appear where expected. Change a model or tool setting next and run the same case to see which earlier evidence still applies. Having an internal operator actually change a configuration item checks the documentation, access, and allocation of responsibilities together.
If operations receives alerts but only the vendor can access execution, mitigation and reconciliation can stall. Distinguish first response time from mitigation and confirmed recovery in service commitments. The guide to MSP incident responsibility shows how to assign work from intake through change and verification. Use the software handover acceptance guide for general build, deployment, access, and backup checks.
Take the shared evaluation results linked to their configuration, operating artifacts, exceptions, and untested items into the acceptance review. Restrict or hold a feature if its critical workflow was not exercised or a dangerous action could not be shown to stop. For conditional operation, name the roles, enabled features, owner, deadline, and retest and exit criteria. After a vendor changes the configuration, retest the evidence affected by that change.
