Define the policy shared by every path
A stop indicator on a screen does not establish whether sending or refunds have stopped. Workers may have taken a job from a queue or scheduled their next attempt. A stop is complete when new execution in scope is blocked and operations can track every job that already crossed the boundary to its final outcome.
Disabling messaging does not have to close customer support. Decide which tenant, feature, and tool are in scope. A problem with one customer's SMS automation can justify stopping that tenant's SMS sends while leaving lookups and drafting available. If a shared API key prevents separation or you cannot establish the affected paths, define a broader operating stop. The execution service must also determine which manual paths remain available.
Record the environment, scope such as tenant, user, tool or feature, issuing operator, effective time, policy version, reason, and incident reference. A workflow subject to overlapping stops can remain blocked until every applicable restriction is released. Expiry of a tenant-level pause should not override a global sending stop. Define release authority and evidence alongside the issuance process.
There can be a delay between updating the policy and all workers seeing it. For high-impact actions, withhold execution when the authoritative policy cannot be read. The screen, model, and cache can surface the current state, but the final check belongs at the component that can send the external request. OWASP recommends separating approval and execution, verifying authority and approval at execution, and failing closed on a policy lookup error. OWASP AI Agent Security Cheat Sheet[1]
Put the last policy check next to execution authority
Suppose intake checks the stop policy before queuing work. A worker checks it again immediately before the external call, even for requests already admitted. Associate the job with the policy version and approval ID so outdated decisions can be rejected. If stale credentials or cached permissions allow a worker to call the tool directly, policy changes can be bypassed. Limit production credentials to the execution service and route administrative actions, batches, and subsidiary agents through the same boundary.
A race can exist between final authorization and acceptance by the external system. Consistent handling of policy changes and internal permits can prevent a new internal permit after the effective stop. It cannot undo a call that has already started. Keep the times of internal authorization, send attempt, and provider acceptance to identify jobs that crossed during this interval. Test any cancellation and status APIs that the provider offers ahead of time.
Verify the stop for each job state
Adapt this table to your job ledger and identify what to block and what still needs handling. A single queue metric can miss work already taken by a worker.
| Job state | Action at stop | Work that can remain | Evidence |
|---|---|---|---|
| Before intake | Reject new automated work at the API and job creation paths | Existing manual intake or permitted read-only uses | Denial after the effective stop and no new job ID |
| Approval pending or granted | Freeze automated progression and check stop state, approval, and expiry at execution | Drafts, approval history, jobs needing review | Approval ID, policy version and expiry, execution denial |
| Queued | Mark paused or canceled in the job ledger and deny execution on delivery | Message payload, redelivery, jobs to classify | Job state linked to queue ID and denial on redelivery |
| In progress, before external call | Request cooperative cancellation and enforce denial at execution | Temporary files and internal changes to clean up | Worker step history and evidence of no downstream request |
| Sent, external outcome unknown | Suspend retries and request provider status | Acceptance or delivery outcome and follow-up | Internal attempt ID, provider correlation key, status response |
| Accepted externally | Request cancellation if available and block follow-on actions | Pending cancellation or completed effects | Provider acceptance, cancellation response, final outcome |
| Scheduled or retry delayed | Freeze new schedules and recheck policy when they fire | Paused schedules and late messages | Schedule ID, retry time, denial on replay or redelivery |
| Already completed | Verify the external result and customer impact, then approve any compensating action separately | Corrections or ledger reconciliation | Final state, business impact, and approved follow-up |
Deleting a queue can erase other tenants' work and identifiers needed for investigation. Keep a separate job history and reconcile it against messages and worker activity. Amazon SQS uses a visibility timeout to temporarily hide messages held by consumers and acknowledges possible duplicate deliveries. If your queue works this way, test both messages already taken and jobs that can appear again after the count reaches zero. Verify against the behavior of the queue you use. Amazon SQS visibility timeout[2]
Reconcile timeouts and external acceptance
On cancellation a local worker may stop waiting for a response. The provider may already have accepted the request. Mark the outcome as unknown, then query with the same correlation key or wait for the agreed callback. If the tool has no status API, agree on the account, time, and business identifiers an operator can use to reconcile with the provider. Consider excluding that path from automatic retries.
Idempotency limits duplicate effects when the same intended request is transmitted again. It depends on the provider's key, parameters, scope, and retention. A valid key does not authorize execution during a stop. After its validity window or a tool change, do not assume the same identifier carries the same intent. AWS Builders' Library on idempotent APIs[3]
Retain the internal job ID, external send time, approval version, provider request ID, acceptance or cancellation status, and status observation time. See AI agent observability for linking execution policy and actual tool effects, and the incident response runbook for incident handoff.
Operations connected from observation to improvement — Respond using service indicators and alerts, then connect change records and incident reviews to improvements.
Six cases for a simulated messaging exercise
Suppose a support system has to stop automated SMS for one tenant. Direct messages to a sandbox and prepare to match job IDs against provider acceptance IDs. Set up work in queued, running, approval, and scheduled states before the drill. Record separately that issuing the stop succeeded and which paths were blocked.
| Test | Injected condition | Expected result | Evidence of failure |
|---|---|---|---|
| Intake after stop | Request SMS for the affected tenant | New automated send denied, draft lookup still works | Send job created or draft lookup disabled |
| Worker already has job | Worker cached the old policy and reaches the external call | Execution boundary applies the new stop, no external request | Worker sends using cached policy |
| Late cancellation | Stop immediately after provider acceptance, delay acceptance response | Record unknown outcome, reconcile ID and verify final provider state | Declare completion from an empty queue or send twice |
| Redelivery and schedule | Replay queue delivery, trigger a schedule, replay failed work manually | Each path denies the affected tool call | Scheduled or manual path bypasses the stop |
| Approval validity | Old approval expires during the stop, job appears after resumption | Hold replay and verify current work, permission, and recipient | Use expired approval or changed message parameters |
| Scope control | Authorized send from an unaffected tenant | Expected authorized send completes for the unaffected tenant | Unrelated work is blocked or the stopped tenant is affected |
Use controllable pauses in the test to reproduce the sequence of policy updates and outgoing requests. Pause a worker just before the check, or have the sandbox accept the request and delay its response. Record the injected condition and the actual number of external calls separately. Successful sandbox cancellation still leaves the actual provider's cancellation semantics to verify.
An accepted job should not be sent again from the same draft. Have an owner verify acceptance time, recipient, content, and next action. If the provider confirms delivery, the business owner can decide whether to send a correction separately from reopening the tool. If new traffic must resume while old jobs are being reconciled, limit the resumed scope and keep unknown-outcome jobs from replaying.
Reassess previous approvals before resuming
Define separate conditions for new traffic, requeuing paused work, and reconciling external outcomes. Verify policy and execution in a sandbox first, then enable a limited path in which fresh work can be tested safely. Associate each recovered path with the current policy version and owner. NIST incident response guidance[4] integrates response and recovery into risk management. OWASP Securing Agentic Applications V2[5] provides guidance for runtime controls and assurance tests.
For old queues, verify whether each job ran, whether the intent still applies, whether the recipient or parameters changed, and whether current permission and restrictions are satisfied. Expired or changed requests need fresh approval. Escalate conflicting results and block replay until the outcome is established. If a code change caused the incident, use rollback-ready deployment to recover the version and evaluate paused work on its own evidence.
The final record should identify who could stop and resume, the scope and effective times, verified acceptance and denial paths, the reconciled provider outcomes, and any jobs still under review. Link compensating actions to their original state, reason, approver, and outcome. Evidence should connect the click that issued the stop to the last business outcome the system established.
