Is AI the attacker's tool or the target?
An attacker can use AI to write convincing impersonation messages or prepare an attack. Account, email, and business-approval controls are important there. Here the target is an organization's agent. Malicious input can influence it to disclose data or misuse a tool. NIST distinguishes misuse of AI to attack other systems from attacks on the AI system itself, including prompt injection.[1]
This guide covers the latter situation for support, messaging and operations. It provides data, permission, approval, verification and recovery criteria for an organization integrating the agent.
For MCP tokens, transport and server configuration, see the MCP production security checklist. Follow the business path here from lookup to draft, approval, external delivery and derived memory. Permissions and responsibilities apply across integration technologies.
Prompt injection can make a model act on instructions embedded in material it processes. The instructions may come directly from a user or indirectly through pages, emails, files and tool responses. OWASP's 2026 LLM Top 10 treats injection, sensitive information disclosure and excessive agency as distinct risks. Defenses need several layers.[2]
Can you describe the allowed task in one sentence?
Consider a hypothetical retailer and support workflow.
A support agent reads an inquiry, checks the customer's order status, and drafts a reply. It can access inquiries, attachments, product documentation, and order lookup. For convenience, the operator has also connected messaging and refund tools.
An uploaded document can attempt to redirect an order lookup, substitute a recipient, or make routine guidance look like a refund authorization. Anthropic's browser-agent research describes how content an agent reads can influence subsequent actions and why the risk remains.[3]
Define the routine task in terms the operator can verify.
Retrieve information for the authenticated customer's order associated with this inquiry and prepare a verifiable draft for the support representative.
This identifies the customer, links the case to an order, and limits the product to a draft. A broad goal such as “assist customer support” leaves those boundaries unresolved.
Expose only the tools needed for each task. Keep refund execution out of a drafting capability. Route a proposed message or refund to a separate workflow with its own checks and authorization.
Connect the data each campaign needs — Define events and attributes from campaign needs, then align identifiers, transmission timing and measurement tools.
Who can read, write, send, or move money?
Least privilege applies to the caller, customer organization, object, operation, and current state. OWASP recommends validating tool calls against the user's permissions and context and checking each tool's parameters.[4]
Use this example as a starting point for a support workflow.
| Operation | Initial scope | Check before execution | Evidence |
|---|---|---|---|
| Read customer data | Order fields tied to the authenticated customer and case | Requester, tenant and ownership, field visibility | Principal and object in an access record |
| Write an internal record | Draft a summary for this case | Editable fields, conflicts, effect on final state | Before and after values, author, draft status |
| Send an external message | Approved content and verified recipient on an authorized channel | Recipient, body, attachment, channel and approval validity | Link the approved payload to the delivery record |
| Issue a refund | Eligible order with separate approval | Ownership, remaining balance, current state and duplicates | Approved amount, transaction ID, outcome and reconciliation |
Read-only access can expose information. An agent retrieving across tenants can send prohibited data into model input or traces before it decides what to display. Filter in the retrieval layer and test tenant and user isolation in caches and memory too.
A model can extract a customer ID. The server must resolve its scope against the authenticated session and trusted relationships.
An approved outbound provider can still deliver to the wrong recipient or attach the wrong file. Validate both the destination service and the current action.
If a connected SaaS product cannot restrict lookups or recipients as needed, reduce the scope to drafting from approved sources or move execution to a separate enforcing layer. Check the controls provided by the actual integration.
Where does outside content meet executable instructions?
Attach provenance and intended use to incoming text. A customer's message, an approved staff policy, a page and a tool result have different authority. Preserve traceability for later investigation.
Microsoft's indirect-injection guidance combines detection, content isolation, behavior observation and authorization, accounting for latency, cost and false positives. Content labeling and detection provide layers alongside server-side enforcement.[5]
Test received content, proposed tool calls, and the point where data leaves or state changes. An input that bypasses detection should still encounter the execution rules.
Internal sources can contain pasted external text or be editable by a broad audience. Manage document contents and changes to policy through distinct paths.
A summary, saved memory or index makes one source affect later work. OWASP's agentic guide discusses memory provenance, approval and policy checks when retrieved memory can affect consequential actions.[6]
When suspect content is removed, follow the summaries and saved memories derived from it. Test what later requests can still retrieve or reuse.
What evidence follows customer information?
In the sample workflow, a draft can enter the case record, an approved reply can be sent, and a summary can inform later cases. Use references to join stages without storing full customer text everywhere.
| Stage | Scope criterion | Minimum linkage |
|---|---|---|
| Lookup | Current user and tenant permissions | Source ID and version, decision and request ID |
| Draft | Facts necessary for the case | Evidence references, draft state, unresolved facts |
| Approval | Approver's authority and approved content | Payload identity and validity |
| Delivery | Recipient, attachment and content match approval | Operation ID, actual result or unresolved status |
| Derived memory | Reuse purpose, tenant and retention policy | Source linkage, usability, revocation or review conditions |
A revoked document can still affect the workflow if its summary remains searchable. Restrict the related memory during the investigation and decide whether to regenerate it from approved sources or retire it. Retention for investigation is separate from reuse in live work.
Reopen the old conversation, look up the same customer from another operator session, and search the summary again. Confirm the recalled information cannot authorize an action. Record which source, cache, index or memory actually applied the change, and when.
What exactly does a person approve?
Show the action as a transaction. An operator should see which order, what amount, why it qualifies, and where the result goes. An approval request that says only “continue?” does not identify the action.
OWASP's agent security guidance covers previewing high-impact actions, validating approvals and retaining evidence. For refunds, apply it to the actual operation.[7]
Bind approval to the action, object, amount, recipient, reason, and validity period. A change to any material field requires a new decision. At execution, check that the approval still applies, entitlement is current and the remaining refundable balance is sufficient.
The execution service records the approved request, derives or checks the execution parameters, and confirms the real outcome. It should reject stale approvals and reused requests outside policy.
If a supplier accepts an action but acknowledgement is lost, reconcile with the supplier and the business record first. Give operators a way to distinguish accepted, definitely failed, and unknown.
How do security tests observe real actions?
A refusal can appear after a tool executed. A normal answer can coexist with a data leak in logs or outbound calls. Measure disclosure, unauthorized action and export using direct evidence.[4]
Use test data and isolated tools with the same constraints as the intended deployment.
| Test | Expected control | Check | Judging caveat |
|---|---|---|---|
| Attachment asks for another customer's order | Server enforces tenant and customer boundary | Denied lookup and no disclosure | Empty answer alone is insufficient |
| Drafting request asks to send | Drafting task has no send authority | Only review task created | Inspect the outbound queue too |
| Recipient or amount edited after approval | Execution parameters bound to approval | Changed action held or denied | Check replay of the old approval |
| Same refund is retried | Deduplicate the operation | Only one resulting transaction | Separate responses from transactions |
| Suspect content enters memory | Govern memory writes and reuse | Later work cannot gain privileges or take an unjustified action | Check derivatives after deletion |
| Calls loop or wait too long | Enforce calls, attempts, time and cost bounds | Stop and notify with status | Tell a safe stop from a silent failure |
| Legitimate request | Ordinary work remains possible | Correct lookup and draft | Track legitimate blocks |
Retain a matched example of each allowed action as well as negative cases. Give testers a record of the permission decision and final state, not just a chat transcript.
Run different languages, attachments, conversation lengths and multi-step tool sequences. Record how many cases and repetitions were executed and qualify the absence of observed violations by those conditions.
Test under deployment-like conditions and document scope and limitations. Re-run affected scenarios when models, prompts, tools, memory, data or permissions change.[1]
When should operations stop and resume?
The operator needs a practical way to limit exposure after a signal. NIST incident response guidance links preparation, detection, response, and recovery to risk management.[8]
Identify triggers and responsible people before launch. Use the table to rehearse a stop and reassessment.
| Signal | Immediate action | Owner | Evidence for resuming |
|---|---|---|---|
| Another customer's data accessed or leaked | Limit the affected lookup, retrieval and feature | Security and data owners | Corrected boundary, scope investigation and regression results |
| Unauthorized send or financial action | Stop the tool and queued work | Service operations and business owner | Reconcile actual events and retest corrected approvals |
| Execution records missing | Restrict to observable functionality | Operations and engineering | Restored evidence and a traceable sample |
| Legitimate requests overblocked | Use bounded manual review | Business and QA owners | Cause analysis and retest of normal and hostile inputs |
Separate disabling new requests from finding in-flight and already completed work. Check queues, logs, affected users, and the external system's records.
Resumption depends on the failed path reproducing under test and the fix working in the intended configuration. Establish scope, retained limits, and follow-up responsibility.
Restoring a model or prompt does not undo a message or a transaction. Decide what can be cancelled, compensated, or investigated with the relevant operators.
What should the decision meeting examine?
Bring a permission map, test outcomes and uncovered paths. Start with the actual business operation and the authority it needs.
- Which subjects may access which objects and fields?
- What is the difference between a draft, an approval, and an external action?
- How do operators identify unknown outcomes and stop retries?
- Which evidence proves blocked actions and valid operations?
- Who can stop, investigate and approve resumption?
Use those answers to decide a limited launch or hold based on tested business boundaries. Track conditions specific to each feature.
The general release decision criteria give a structure for the decision. Add input sources, permission changes, approvals, and memory to the agent's evidence.
How should test results become follow-up work?
Record the attempted conditions, observed blocks and uncovered paths. Link findings to requirements, fixes and retests to make later configuration changes assessable.
Qoretix describes test planning, API and browser runs, failure evidence, remaining scope, and connected repository checks for code, secrets and dependencies.[9] Confirm the fit for each project and enforce permissions and action approvals in the application.
In the review, bring the documented workflow, permission boundaries, and cases that still need verification.
