Separate intake, generation, external execution, and completion
At a fictional online retailer, a customer asks to change a shipping address. Intake creates a case, but the LLM API does not respond. If the screen asks the customer to start over, both the AI and an operator might end up assigned to duplicate cases. Once the request is stored, show its reference and the next update time, and make it available for a person to continue.
Intake durably stores the request and gives the customer a reference. Generation retrieves evidence and drafts an answer or action. Execution applies effects such as approval, sending, or a status change. Completion requires evidence that the job meets its criteria. A generated answer does not close a job that still requires an external change.
The same AI error screen can hide a failed save or failed generation after a successful save. The customer should know whether to submit again or use the reference already issued. Give users a route back to the job. Google PAIR on errors and paths forward[1]
Decide which work can proceed during the outage
Assess deadlines promised to customers, delay impact, required information, and effects of execution. A public policy question can use an approved static link. A personal order status requires authorization and a lookup. An address change needs a warehouse cutoff and a decision in the authorized order system. If a model call blocks internal lookup today, redesign the workflows that can safely continue without it.
Plan for the actual capacity of human support. Funneling every AI request to a person only works with adequate coverage. Prioritize approaching deadlines and account risk. Use validated material for routine guidance. An operator must be able to find and claim queued cases.
Decide whether to wait, retry, switch, or hand off
These decisions are illustrative. Check actual errors and retry behavior against the API contract. For externally visible changes, use the provider’s lookup or reconciliation if an HTTP response does not establish execution.
| Failure or signal | Immediate action | Criteria for waiting, retrying, switching, or handing off |
|---|---|---|
| Connection delay or read timeout | Separate stages reached and preserve intake | Retry generation within the deadline. Look up any tool action already sent |
| 429 or provider limit | Check limit scope, reset signal, and remaining time | Wait for a short retryable limit. Reduce scope or hand off a longer constraint |
| Temporary 5xx or repeated errors | Identify affected accounts, regions, and routes | Limit new calls and use an approved fallback or human queue |
| Authentication, authorization, or input error | Record impact and inspect settings and data | Stop identical retries and route to an owner or guide the user |
| Partial stream or incomplete tool arguments | Mark output incomplete and check execution conditions | Wait for finalization and validation before acting or making confirmed claims |
| External tool outcome unknown | Preserve case and external operation IDs | Reconcile before another model retries. Escalate as needed |
| Missing evidence or context | Record the missing check | Ask the customer or hand off when required data is unavailable |
RFC 6585 defines HTTP 429 for too many requests over some period and permits an indication of when to retry. RFC 9110 describes the format of Retry-After. Check the provider’s limit scope and error body for rate limits, exhausted quota, and billing configuration issues. Avoid scheduling retries beyond the job’s expiry. IETF RFC 6585[2] · IETF RFC 9110[3]
Limit retries and make queue age visible
Set a number of attempts, a timeout per attempt, and an overall expiry. If an application, SDK, proxy, and worker all retry, upstream calls can multiply. Two layers each allowing three total attempts can generate up to nine calls. Assign the retry to a layer and spread out attempts as the interval increases. AWS on timeouts, backoff, and jitter[4]
Watch queue depth and the age of its oldest eligible job, cases nearing deadlines, scheduled retries, and cases without an owner. Guide new customers to wait or return when intake exceeds capacity. Keep their state separate from requests already accepted. When a worker claims a case, reread the current order, approval, and expiry. An address change that was valid at intake may no longer be possible after shipping.
Operations connected from observation to improvement — Respond using service indicators and alerts, then connect change records and incident reviews to improvements.
Test fallback quality and data handling
Define allowed data classes, regions, retention, contracts, and tool permissions in advance. A primary model outage cannot authorize sending full support transcripts to a different processor. Check each feature’s actual handling of inputs, logs, and files. An identity provider or infrastructure dependency shared by the two services can also fail.
Test representative outcomes. A fallback can produce valid JSON while dropping a shipping-date qualification, mishandling protected orders, or failing to escalate. If retrieval or prompts change with the model, record the verified combination. Google Cloud’s generative AI operations guide[5] describes monitoring system health and evaluating live output.
With limited validation, restrict the fallback to approved work such as drafting or summarizing public guidance. For cases handed to a person, distinguish proposed actions from confirmed effects. A fallback replay can duplicate a write the primary model already made. Use the tool’s idempotency and observed state to decide whether a write can repeat. AWS on idempotent API design[6]
Transfer the case number and ownership to the operator
In the fictional retailer’s example, the customer receives case CS-014. If the AI cannot answer by the promised time, the same case goes to an operator. Transfer the request scope, verified order status, last generated or executed action, and remaining deadline. The operator checks the warehouse cutoff and follows the appropriate workflow.
| Handoff field | Fictional delivery example | What the operator checks |
|---|---|---|
| Case ID and owner group | CS-014, delivery changes | Duplicate cases and ownership acceptance |
| Customer request and access | Address change with authentication status recorded | Data the operator may expose and need for reverification |
| Verified fact and evidence | Last time a pre-shipment status was confirmed | Whether a fresh lookup is needed |
| AI proposal and actual effects | Incomplete draft and no address-change call | Proof of external action or unknown state |
| Deadline and next update | Warehouse cutoff and customer update time | Completion owner and delay or rejection message |
When model workers return, check ownership and completion before sending a new response to a case a person took over. In a test, deliver the model result just before and just after the operator claims the case. Verify that one route finishes the visible or external action and that approval for an address change remains provable.
Reconcile the backlog after the provider recovers
A green provider status does not close your own cases. Review intake, generation, and execution records. Prioritize pending cases by risk and deadline. Link cases settled by a person to their final outcome and prevent replay. For external results still unknown, record evidence of success or rejection or assign an owner. Run representative requests and error scenarios before returning to the prior model.
Verify available capacity before draining the backlog in stages. Keep room for incoming cases. Report accepted jobs, on-time completion, maximum queue age, handoff completions, unresolved external actions, and duplicates as well as outage length. See IXC’s SLO and error budget policy guide for measurement windows and denominators.
See the incident response runbook for shared processes and MSP incident ownership for roles across providers and product teams. In an LLM outage exercise, prove who can resume from the customer’s case number and what shows the final outcome.
