Classify the work before choosing models
Routing classifies an incoming request and sends it to a model or workflow. It can lower costs for frequent simple requests. An unusual request sent to the cheap path can miss a critical exception or trigger calls to both models, so test the whole routing policy. Anthropic's agent design guide describes routing for distinguishable categories that can be classified accurately.
Fixed-category classification, extraction from policies or contracts, and decisions about compensation carry different consequences. A team may be able to transfer a misaimed inquiry. Extracted fields need their source, identifier, and units to prevent downstream errors. Compensation requires permissions, business approval, and evidence of the actual external result.
Document impact, authorized data and evidence, latency, human capacity, and failure actions. An easy task may still be restricted to one account. A complex task may tolerate a limited model if the system checks results in a sandbox and shows a human only a draft. Before comparing models, apply the constraints, preferences, and unknowns in the cloud workload classification guide to data handling by task.
Use signals aligned with policy. Server-verified roles, channel, order reference, and document classification can guide routing directly. For overlapping intents, unverified sources, or uncertain action risk, clarify or hand off. Decide allowed data and tools before choosing models, then enforce authorization at execution.
Evaluate candidates and routing policy on the same set
Assume model A has a lower call cost and B has a higher one. Give both models the same task context, evidence, and tool contracts. Record any model-specific prompts, and assess the same expected outcome and operating environment. Test individual models and the combined policy through the final business state.
| Task class | Example acceptance criteria | Where A or B can be used |
|---|---|---|
| Standard inquiry classification | Correct label, queue and urgency, urgent exceptions do not reach the general queue | Use A if it meets data and classification requirements. Route ambiguous or urgent inputs separately |
| Document field extraction | Accurate ID, amount, unit and effective date with correct source location. Hold invalid or missing fields | Accept A's fields after evidence and schema checks. Send conflicting evidence to B or a person |
| Exceptions and compensation | Advice within policy, sensitive data restricted, no action without approval, handoff confirmed | B may draft within limits. Verify decisions and execution through approval |
| Missing evidence or uncertain category | State why handling cannot proceed, clarify or escalate | Use a separate clarification path and hold automated action while evidence is missing |
Use code and reference answers for fixed categories. Check multi-part conditions item by item, and observe the customer response and actual handoff separately. The Google Cloud evaluation metrics guide[1] shows reference-based and task-specific measures. Anthropic's agent evaluation guide[2] distinguishes the result an agent claims from the state observed in the environment.
Include different documents, missing data, urgent exceptions buried late in a request, and requests for another organization's data. Record representative traffic and cases designed to exercise safety boundaries separately. When repeating runs, record distinct cases and execution count so repeat-heavy classes do not dominate interpretation.
Calibrate confidence on real tasks
A classifier's reported confidence of 0.9 needs verification. Check whether approximately 90 of 100 comparable requests at that level are actually correct. Verbal confidence and token probabilities also differ. Tian et al.'s 2023 study compared confidence elicitation methods on selected models and short-answer benchmarks. Its results do not estimate performance in this workflow. Tian et al. on confidence calibration[3]
Before using confidence to route, check observed outcomes by score band, task, and model version. Record the count and quality of predictions above 0.9, high-risk misroutes, and required reviews. Use other evidence for sparse bands and retest after changing models or prompts. Let the execution service verify permission, evidence, and payment limits regardless of confidence.
Count misroutes on their own denominator
Routing failures can hide behind model results. If ten of 100 urgent compensation cases go to the general low-cost path, the rate for that risk class is 10%. Reporting only 1% across 1,000 total requests obscures the weakness. Also count routine cases routed to higher-cost paths to estimate avoidable calls and latency.
Record the correct class, chosen route, and any later reassignment in a confusion matrix or associated event records. Verify whether downstream controls intercepted the misroute, whether customers saw the response, and whether staff completed the task. Log both the model's suggested class and the enforced outcome.
From cost drivers to ongoing control — Break down spending by workload and resource, assess each change and carry budget guardrails into operations.
Recheck what lower monthly call costs mean
Compare a fictional cohort of 1,000 tasks, with 700 routine inquiries, 200 extractions, and 100 exception cases. A costs KRW 40 per call, B costs KRW 120, routing costs KRW 5 per task, and staff review costs KRW 300. The calculation covers charges incurred in processing the cohort. Count only policy-compliant and verified completions in the measurement window. Common infrastructure and routine upkeep are identical and excluded. These are illustrative assumptions.
| Measure | B-only configuration | A/B routing |
|---|---|---|
| Routing evaluation | KRW 0 | 1,000 × KRW 5 = KRW 5,000 |
| Initial A calls | KRW 0 | 900 × KRW 40 = KRW 36,000 |
| Initial B calls | 1,000 × KRW 120 = KRW 120,000 | 100 × KRW 120 = KRW 12,000 |
| Additional B calls | KRW 0 | 120 escalations × KRW 120 = KRW 14,400 |
| Staff review | 100 × KRW 300 = KRW 30,000 | 120 × KRW 300 = KRW 36,000 |
| Total in scope | KRW 150,000 | KRW 103,400 |
| Verified successes | 970 | 960 |
| Cost per success | About KRW 154.64 | About KRW 107.71 |
Assume ten exception cases entered A and ten routine inquiries entered B in their place. The 120 escalations include all ten exceptions detected downstream. Check whether they were caught before the cutoff and whether any external execution took place. If a mandatory approval was bypassed, withhold approval of this routing policy regardless of cost. Evaluate the 960 verified successes against quality criteria by task.
Total in-scope costs fall by KRW 46,600 and verified successes fall by ten. If B escalations increase from 120 to 520, holding other items constant, the extra KRW 48,000 raises total cost to KRW 151,400. At the same completion count the successful-task cost increases as well. Run different scenarios for escalation and review rates. The FinOps Foundation's Unit Economics guidance[4] recommends linking spend to outcome units and documenting assumptions and costs. See calculating cloud cost per business unit for allocating costs.
Link call counts, input and output sizes, retrieval and storage charges, delays, and human intervention to the task ID. Set the time window and number of permitted model calls. One task that uses two models and staff incurs all three costs while entering the success denominator once. Explain changes to observation and cost scope when the workflow changes.
Reassess routing when production or dependencies change
Sending all work to B during an A outage can exceed B's quota or staff capacity. Both models may share an outage if they have the same provider. Define which tasks may escalate automatically, how long they may wait, when staff take over, and which actions stop. Verify data classes, regions, approvals and tool limits before routing to another model. A model change alone should not replay an older pending external action.
Release records should include model and prompt versions, routing rules and thresholds, and evaluation data versions. Start by recording what the new router would select without changing external effects, then run a limited live rollout on low-risk work. Check latency, correctness, escalation, handoff and cost by task class. Look for a shift in common cases and new exceptions. Use input drift and continuous evaluation practices from Google Cloud's operations guide[5] at the routing stage.
Document the scope of enabled tasks. Unsafe classification, use outside policy, and surges of unknown task types can justify suspending routing independently of cost. If restoring earlier rules, determine which work already went down a different path and what it did externally. For the next change, report task-specific boundaries, misroutes, recovered exceptions and cost per verified success alongside totals.
