Connect the invoice to the end of the job
If the goal is to resolve a customer’s delivery inquiry, producing a model response is only one step. Check the order and shipping status, whether the response promised an unconfirmed date, whether the customer received the answer, and whether work finished on time. A single inquiry can generate several calls.
Before calculating, business and finance owners define eligible, attempted, successful, and handed off. Eligible jobs meet the intake policy within the reporting period. Attempted jobs are unique eligible requests that entered the automated path. Successful jobs meet the selected completion criteria. Handed-off jobs transfer to a person and can later succeed or fail. The attempt and handoff counts can overlap. Record the route separately from the outcome. Google PAIR on user needs and defining success[1]
An overall business metric can count work a person finishes on time. An automation metric counts completions without human approval, review, or execution separately. Label both measures and their denominators. Use the same reporting period and handoff data to compare them. Anthropic’s guide to agent evaluation[2] explains how execution traces and observed outcomes can diverge.
Agree on the conditions for each count
Write these definitions before comparing configurations so a change in difficulty does not masquerade as an improvement. Report exclusions separately.
| Measure | Example definition | Companion evidence |
|---|---|---|
| Eligible work | Unique inquiries in a defined delivery category and service window | Failure cases, regions, inquiry mix, and exclusions |
| Automation attempts | Unique inquiries that started the automated route at least once | Unique jobs and model calls separately |
| Automated completions | On-time resolution using correct evidence, without human takeover | Share of eligible work and quality sampling |
| Handoffs | Inquiries assigned to a human operator | Handoff rate, later completions, and cost |
| On-time success | Unique jobs meeting accuracy, authorization, and deadline criteria | Automated and manual outcomes with reopenings deduplicated |
| Late, failed, or unscored | Each tracked under the declared deadline and judgment rules | Cumulative cost, rework, and unresolved outcomes |
If customers reopen the same problem within a defined window, link it to the original inquiry. Distinguish unresolved requests at period close from ones that exceeded a fixed deadline. When a new system processed the work and a legacy system recorded success, join both by job ID and count it once. Add timestamps to failures and unknown results so the next report can explain changes.
Record costs by call, job, and operating period
Include the first model call, escalation, retries, and regeneration after failure. Attribute retrieval, tools, and storage to the job where possible. For long-term logs, evaluation management, security reviews, and standby costs, record the reporting-period cost and allocation basis. The FinOps Foundation connects technology cost to outcomes and considers throughput, risk reduction, and service levels as proxies when revenue is hard to attribute. FinOps Foundation’s Unit Economics[3]
Measure the time spent on approvals, human handoffs, corrections, and incident investigation using the same labor rate. Avoid counting the same review in both case handling and operating overhead. Include recurring evaluation costs in operations and show one-time setup separately. During a pilot, compare the ongoing cost alongside total cost including transition. IXC’s guide to cloud migration TCO
Restricting the cost numerator to successful jobs favors configurations that fail often. Include failures, handoffs, and retries, with success in the denominator only. Choose an allocation key for fixed costs based on the operating model, such as eligible jobs or measured storage and computing use. Record when the basis changes. IXC’s guide to cloud unit economics
From cost drivers to ongoing control — Break down spending by workload and resource, assess each change and carry budget guardrails into operations.
Calculate two fictional configurations
Each configuration receives 1,000 inquiries in a month, with a similar difficulty mix. An inquiry resolved on time using accurate evidence counts as success even after a handoff. Both configurations have timely successes and overdue or unsuccessful work. Prices and quantities are illustrative assumptions.
| Monthly cost or job measure | Configuration A | Configuration B |
|---|---|---|
| Eligible inquiries | 1,000 | 1,000 |
| Automation attempted | 950 | 950 |
| Automated completions | 600 | 820 |
| Handoff or direct human handling | 400 | 180 |
| Total on-time successes | 900 | 960 |
| Initial model calls | KRW 18,000 | KRW 48,000 |
| Retry and fallback calls | KRW 16,000 | KRW 9,000 |
| Retrieval, storage, and monitoring | KRW 34,000 | KRW 41,000 |
| Allocated fixed evaluation and operations | KRW 80,000 | KRW 80,000 |
| Human handling and review time | KRW 420,000 | KRW 200,000 |
| Total operating cost | KRW 568,000 | KRW 378,000 |
| Cost per successful job | About KRW 631 | About KRW 394 |
A’s 900 successes comprise 600 automated and 300 human completions. B totals 820 automated and 140 human completions. The human route includes handoffs from automation as well as 50 requests handled directly. Adding the attempt count and human route does not give the unique inquiry count. In this example every inquiry outside automated completion reached the human route.
Human time uses an assumed internal rate of KRW 30,000 per hour. A requires 14 hours and B requires 6 hours and 40 minutes. A’s costs add to KRW 568,000 and B’s to KRW 378,000. Dividing by 900 and 960 respectively gives KRW 631.11 and KRW 393.75. Initial model cost per attempt and overall cost per timely success measure different things.
Which variable could change the conclusion?
Hold B’s 960 successes and non-labor cost of KRW 178,000 constant. Its cost per success would reach A’s at about KRW 427,867 of human time. Compute it as KRW 568,000 divided by 900, multiplied by 960, less KRW 178,000. At KRW 30,000 per hour, that is about 14.26 hours. If limiting review time changes the success count, recalculate the denominator.
Check whether the fixed cost allocation changes with volume. If monthly review remains constant and fewer jobs arrive, fixed cost per job rises. If a seasonal peak costs more in total, separate volume changes from per-job changes. Adjust billing to the same period and distinguish consumption from contract price movements.
Where the lower-cost model is sufficient
In a separate fictional test, 196 of 200 straightforward classification requests complete on time in each configuration, and the error mix and review time meet the same conditions. With KRW 60,000 of common operating cost and either KRW 12,000 or KRW 22,000 of model-related costs, cost per success is about KRW 367 or KRW 418. The cheaper configuration works for this group. The test does not cover long exception cases or special approvals.
Check comparable quality and risk before interpreting an overall average. Returning unauthorized customer details can fail a release criterion even with favorable costs and completion counts. Report missing evidence and disagreement between automated and human judgments. Align the population and criteria before treating measurements as equivalent service.
Report time savings and spending changes separately
If payroll stays the same, the move from 14 hours to 6 hours and 40 minutes frees 7 hours and 20 minutes. Pricing the time shows capacity for other work. Cash savings depend on evidence such as reduced contractor hours or an invoice for less external review. If staffing needs change only during a peak period, show that period and the contract terms.
Before taking the comparison to a decision meeting, capture the following.
- Record the period, eligible and excluded work, completion, handoffs, and reopenings.
- Link calls, retries, and final outcomes to the job ID.
- Identify direct charges, allocations, and labor valuations, with owners.
- Report time, success rate, unscored requests, and severe failures alongside cost per success.
- Vary demand, rework, and hourly rates and record what changes the decision.
