AI agent security depends on controlling what the agent can read, decide, call, and execute. Prompt filters help, but they cannot make an autonomous system safe by themselves. The durable design treats every model output as untrusted, checks each tool call against policy, uses task-scoped credentials, and runs risky work inside an isolated, disposable sandbox.
This guide explains the three controls that reduce the most risk: prompt-injection containment, least-privilege tool permissions, and sandboxed execution. It also gives platform and DevSecOps teams a practical architecture, approval matrix, test plan, and incident checklist.
The AI agent threat model
An agent combines a probabilistic model with data, memory, tools, credentials, and a loop that can take repeated actions. That creates a different risk profile from a chatbot that only returns text. A malicious instruction in a webpage, ticket, repository, email, tool response, or saved memory can influence the next tool call.
The main failure paths are:
- Goal hijacking: untrusted content redirects the agent away from the user’s task.
- Tool misuse: the agent selects an allowed tool for an unintended or destructive purpose.
- Privilege abuse: broad credentials let one compromised task reach unrelated data or systems.
- Data exfiltration: sensitive context leaves through a tool call, URL, message, log, or rendered output.
- Unsafe code execution: generated commands escape their intended workspace or contact unapproved services.
- Memory poisoning: hostile content persists and alters future sessions.
- Runaway behavior: retries, loops, parallel calls, or recursive delegation exhaust budget or amplify damage.
The OWASP AI Agent Security Cheat Sheet groups these risks around prompt injection, tool abuse, privilege escalation, data leakage, memory poisoning, and high-impact actions. Use that list to build a threat model for each workflow, not as a one-time compliance checklist.
If your team is still defining agent components and boundaries, begin with AI Agents Explained. The security design should map directly to the planner, memory, retrieval, tools, and execution layers described there.
Prompt injection: assume some attacks will succeed
Direct prompt injection arrives through the user’s request. Indirect prompt injection hides inside material the agent retrieves, such as a webpage that tells the model to reveal secrets or a code comment that tells it to change a deployment. Indirect attacks are especially dangerous because the content may look like ordinary task data.
No single detector catches every phrasing or obfuscation. OpenAI’s current agent safety guidance recommends keeping untrusted variables out of higher-priority instructions, using structured outputs between workflow stages, retaining tool approvals, applying input guardrails, and evaluating traces. Those measures reduce risk, but authorization still has to limit the impact when the model makes the wrong decision.
Separate instructions from data
Label retrieved text as untrusted data and keep it out of system or developer instructions. Do not concatenate a ticket, webpage, or tool result into a privileged prompt template. Use a dedicated extraction step to convert external content into a narrow schema such as identifiers, dates, severity, and evidence. Reject unexpected fields and length.
Break dangerous chains
Do not let content from one untrusted source freely determine the destination and payload of a second tool. For example, a webpage should not be able to supply both the database query and the external URL that receives its results. Bind destinations to trusted configuration, and require a separate authorization decision before data crosses a boundary.
Keep memory clean
Do not write raw model output or retrieved content into long-term memory by default. Validate the record, attach its source and tenant, set an expiry, and make it removable. Separate user memory, task state, and operational logs so a poisoned session cannot silently become global policy.
Render output safely
Escape HTML, URLs, Markdown, terminal control sequences, and file paths before displaying or executing them. Treat model-generated code and commands exactly like untrusted user input. A valid response format is not proof that its meaning is safe.
Tool permissions: make authority explicit
The model may propose a tool call, but a deterministic policy layer should decide whether it runs. Check the initiating user, agent identity, tool, action, resource, arguments, data classification, environment, time window, and approval state for every call.
NIST’s current agent identity and authorization project focuses on identification, authentication, authorization, auditability, delegation, and binding agent actions back to human authority. That is a useful operating model: the agent needs its own workload identity, but it must not erase the accountability or access limits of the person who started the task.
Design narrow tools
A tool named run_shell(command) is difficult to secure. Prefer capabilities such as get_deployment_status(service), restart_staging_service(service), or open_pull_request(repository, branch, patch). Constrain parameters with typed schemas, allowlists, maximum sizes, and valid state transitions. Make writes idempotent when possible so a retry does not duplicate a payment, message, or deployment.
Use task-scoped credentials
Issue short-lived credentials after authorization, restrict them to the exact repository, namespace, table, or account required, and revoke them when the task ends. Keep long-lived secrets outside model context and the execution environment. The tool gateway should inject a credential only after policy approval and should return a minimal result rather than the secret.
| Action class | Examples | Default control |
|---|---|---|
| Read, low sensitivity | Public documentation, service health, non-sensitive logs | Allowlisted source, rate limit, audit record |
| Read, sensitive | Customer records, private repositories, incident evidence | User and agent authorization, field filtering, no cross-tenant access |
| Reversible write | Create a branch, draft a ticket, update a staging resource | Argument validation, bounded scope, rollback path, notification |
| High-impact write | Production deployment, account change, external message | Independent policy check and explicit human approval |
| Destructive or irreversible | Delete data, rotate ownership, transfer funds | Deny by default or require a separate tightly governed workflow |
OpenAI’s guardrails and human-review guidance makes the same boundary concrete: validation belongs next to the tool that causes the side effect, and ambiguous or high-risk actions should pause before execution. Approval screens should show the exact action, target, changed fields, data leaving the boundary, and rollback plan. A generic “Allow?” prompt encourages rubber-stamping.
Sandboxing: contain execution, not just the model
A sandbox limits the files, processes, devices, credentials, and network destinations available to agent-generated code. It does not decide whether the business action is authorized. Use sandboxing together with tool policy and approvals.
For high-risk workloads, prefer a disposable virtual machine, microVM, or hardened sandbox runtime over a long-lived shared container. Create a fresh environment per task or trust boundary. Mount only required files, make source data read-only where possible, and export only explicitly approved artifacts.
Minimum sandbox controls
- Run as an unprivileged user with no host socket, privileged mode, or unnecessary Linux capabilities.
- Set CPU, memory, process, storage, token, tool-call, and wall-clock limits.
- Use default-deny egress and allow only task-specific destinations and protocols.
- Keep cloud metadata endpoints, control planes, internal admin services, and other tenants unreachable.
- Supply credentials through a broker at request time instead of storing them in the sandbox.
- Destroy the environment after the task and wipe ephemeral storage.
- Record the image digest, policy version, identity, network decisions, tool calls, and exported artifacts.
OpenAI’s sandbox security documentation emphasizes workload isolation, outbound allowlists, separate credentials, and brokering third-party secrets outside agent-generated code. The same principles apply whether the execution environment is hosted or self-managed.
For a broader deployment view, see our self-hosted AI agents architecture guide. It explains how to separate inference, orchestration, context, the tool gateway, and the control plane.
A defense-in-depth architecture
| Boundary | Enforcement | Failure it contains |
|---|---|---|
| Input and retrieval | Source trust labels, content limits, extraction, malware scanning | Direct and indirect prompt injection entering privileged context |
| Agent runtime | Step limits, structured state, memory validation, model and prompt versioning | Goal drift, poisoned memory, runaway loops |
| Policy gateway | Identity-aware authorization, typed arguments, approvals, data-loss rules | Tool misuse, privilege escalation, unsafe data transfer |
| Execution sandbox | Filesystem, process, device, resource, and network isolation | Code execution escape and lateral movement |
| Observability | Trace IDs, immutable decisions, redacted logs, anomaly alerts | Undetected abuse, weak forensics, repeated failures |
Google Cloud’s MCP security guidance explicitly warns about prompt injection, insecure tool chaining, and irreversible actions. It recommends a distinct agent identity and least-privilege permissions. Protocol compatibility makes tools easier to connect; it does not make those tools trustworthy. Review server ownership, code, updates, permissions, data destinations, and revocation before connecting any MCP server. Our MCP guide covers the integration model in more detail.
Test the controls, not only the answers
A security evaluation should exercise complete traces from input to tool result. Include direct and indirect injection, obfuscated instructions, malicious tool output, cross-tenant requests, poisoned memory, approval fatigue, replayed calls, oversized payloads, denied network destinations, secret-access attempts, sandbox escape attempts, loops, and unavailable policy services.
For each test, assert the expected tool, target, arguments, credential scope, approval state, network destination, artifact set, and final outcome. A fluent refusal is not a passing result if a background tool already leaked data.
Production signals to monitor
- Denied tool calls and the policy rule that denied them
- New tools, destinations, identities, or permission combinations
- Approval frequency, rejection rate, and repeated approval prompts
- Unexpected data volume crossing a tool or network boundary
- Step count, retry loops, concurrent workers, and budget exhaustion
- Sandbox violations, blocked egress, process anomalies, and exported artifacts
- Changes in injection-test pass rate across model, prompt, tool, and policy versions
Connect these events with a durable run ID and preserve the policy decision separately from model reasoning. The AI agent observability guide shows how to correlate model calls, tools, approvals, deployments, and alerts without logging sensitive content by default.
Prepare for an agent security incident
- Stop execution. Disable the affected workflow, revoke task and agent credentials, and block suspect destinations.
- Preserve evidence. Save trace IDs, policy decisions, tool inputs and outputs, sandbox image digests, network events, and exported artifacts.
- Bound the exposure. Identify the initiating user, affected tenants, data accessed, actions completed, and downstream systems reached.
- Remove persistence. Delete poisoned memory, rotate exposed secrets, rebuild sandboxes from trusted images, and invalidate unsafe caches.
- Fix the deterministic control. Tighten authorization, tool schemas, egress policy, approval rules, or isolation before changing prompts.
- Add a regression test. Reproduce the failure safely and prevent the same path across supported models and tools.
- Restore gradually. Start in read-only or shadow mode, then re-enable bounded actions with heightened monitoring.
For delivery workflows, apply these gates before the agent can merge, release, deploy, or roll back. Our AI agents for CI/CD guide maps approval and evidence boundaries across the software delivery lifecycle.
Frequently asked questions
Can prompt injection be completely prevented?
No reliable control can guarantee that every direct or indirect injection will be detected. Reduce exposure with trusted-source boundaries, structured extraction, input and output checks, and adversarial tests. Limit impact with independent tool authorization, scoped credentials, approvals, and sandboxing.
Should an AI agent inherit the user’s permissions?
Not wholesale. Evaluate both the user’s authority and the agent workflow’s allowed scope. The effective permission should be the narrow intersection needed for the current task, delivered through a short-lived credential and recorded for audit.
Is a container enough to sandbox an AI agent?
A standard container may be adequate for low-risk, trusted code, but it is not the strongest boundary for hostile or agent-generated execution. Higher-risk workloads often need a hardened runtime, microVM, or VM plus unprivileged execution, resource limits, default-deny networking, and per-task isolation.
Which agent actions require human approval?
Require approval for high-impact, externally visible, sensitive, destructive, or difficult-to-reverse actions. Examples include production changes, account or permission changes, external communications, sensitive-data exports, financial transactions, and deletion. Keep the approval specific and reviewable.
What is the safest first agent use case?
Choose a bounded read-only task with non-sensitive data, measurable output, and no ability to change systems. Run it in shadow mode, record traces, test adversarial inputs, and add narrow reversible actions only after the controls consistently hold.
Bottom line: secure agents by assuming prompts and model outputs can be manipulated. Keep authority in deterministic policy, credentials in a broker, execution in a disposable sandbox, and high-impact decisions behind clear approval gates.
