Self-hosted AI agents run the model, agent runtime, memory, and tool connections inside infrastructure you control. That can improve data residency, customization, and unit economics at steady utilization—but only if you treat the agent as a privileged production service, not a chatbot installed on a GPU.
The practical architecture separates inference from orchestration and puts an authenticated policy gateway between the agent and every tool. Start with one bounded workflow, benchmark it against a hosted API, and expand only after quality, security, and total cost meet explicit targets.
What “self-hosted AI agent” actually means
A self-hosted agent is more than a locally served language model. A useful production system has at least five layers:
- Inference: the model server, model weights, tokenizer, GPU or CPU capacity, and request scheduler.
- Orchestration: the loop that plans, calls tools, evaluates results, retries, and stops.
- Context and memory: prompts, retrieved documents, conversation state, and durable task records.
- Tool gateway: controlled access to repositories, CI/CD, tickets, cloud APIs, databases, browsers, and shells.
- Control plane: identity, authorization, secrets, policy, audit logs, evaluations, monitoring, and incident response.
Hosting only the model while sending prompts, traces, or tool outputs to third-party services is a hybrid design—not fully self-hosted. Draw the data boundary before selecting a model. The boundary should cover embeddings, vector stores, logs, backups, evaluation datasets, and external tool calls as well as inference.
If you are still defining the operational layer around models, see our LLMOps guide. For systems that ground answers in internal knowledge, the RAG explainer covers the retrieval path separately.
A reference architecture for self-hosted agents
The safest default is a layered service with explicit trust boundaries. A client authenticates through an API gateway. The orchestrator receives the user identity and task, retrieves only permitted context, and sends an inference request to a private model endpoint. When the model proposes a tool call, a separate policy service validates the user, agent, action, resource, and requested scope. Only then does a short-lived worker invoke the tool.
| Layer | Primary job | Production requirement |
|---|---|---|
| Edge gateway | Authenticate callers, rate-limit, and route | TLS, identity propagation, request limits, and deny-by-default exposure |
| Agent orchestrator | Manage state, planning, retries, and stopping | Deterministic limits on steps, time, tokens, and spend |
| Inference service | Serve model responses and tool-call proposals | Version-pinned weights, batching, health checks, and capacity metrics |
| Context service | Retrieve authorized documents and memory | Tenant filtering, provenance, retention limits, and deletion controls |
| Policy and tool gateway | Authorize and execute actions | Allowlisted tools, scoped credentials, approvals, sandboxing, and audit logs |
| Evaluation and observability | Measure quality, latency, safety, and cost | Trace correlation without leaking secrets or sensitive prompt content |
Keep inference and actions separate
A model should propose an action; it should not hold the credentials that execute it. This separation lets you replace models without redesigning authorization, and it gives security teams one place to enforce policy. The tool gateway should validate typed arguments, limit output size, attach the end-user identity, and reject commands outside the approved resource set.
Tool standards can simplify integrations, but they do not replace authorization. Our Model Context Protocol guide explains the protocol boundary and the controls still needed around it.
Make workers disposable
Run code execution, browser automation, and repository operations in short-lived sandboxes. Give each task a fresh filesystem, a restricted network policy, explicit CPU and memory limits, and credentials that expire when the task ends. Persist only approved artifacts and audit records. A long-running shared shell turns one poisoned task into a cross-task risk.
Deployment options compared
| Option | Best for | Main trade-off |
|---|---|---|
| Single workstation with Ollama | Developer experiments, offline prototypes, and low-concurrency internal tools | Simple and inexpensive, but limited isolation, availability, and centralized controls |
| Dedicated GPU server | One team with predictable demand and a stable model | Good control and utilization, but maintenance and failover are your responsibility |
| Kubernetes plus vLLM | Shared production inference with multiple workloads and stronger platform controls | Scalable and observable, but requires GPU scheduling, networking, and cluster expertise |
| Hybrid routing | Teams that need private local processing plus hosted overflow or frontier models | Flexible economics, but the data policy and routing logic become more complex |
Ollama’s official API documentation exposes local endpoints and OpenAI-compatible access, making it convenient for prototypes and small internal deployments. Local requests do not require an authorization header, so do not expose the service directly to an untrusted network; place an authenticated reverse proxy or application gateway in front of it.
For higher-throughput serving, vLLM provides an OpenAI-compatible server and a Prometheus metrics endpoint. Its documentation also warns that the built-in API key does not cover every endpoint, which is another reason to enforce authentication and network policy outside the model server.
Security controls that matter most
Self-hosting reduces reliance on an external inference provider, but it transfers more security work to your team. The highest-risk surface is usually not the model weights; it is the agent’s ability to act on systems.
1. Give every agent a verifiable identity
Use a workload identity for the agent service and preserve the initiating user’s identity through the entire request. Authorize both: the agent may be allowed to read deployment status, while only a specific on-call role may approve a rollback. NIST’s work on agent identity and authorization highlights identification, authorization, auditing, non-repudiation, and prompt-injection controls as core concerns.
2. Apply least privilege per tool and resource
Do not give an agent a broad cloud administrator token because one workflow may need to restart a service. Prefer a narrow operation such as “restart this deployment in this namespace,” with a maximum frequency and approval rule. Separate read tools from write tools, and require human approval for destructive, irreversible, or high-blast-radius actions.
3. Treat all retrieved content as untrusted
Tickets, webpages, logs, code comments, and documents can contain instructions designed to redirect the agent. Keep system policy outside retrieved text, label external content, prevent retrieved content from changing permissions, and validate every tool call independently. Prompt filtering helps, but authorization must hold even when the model follows a malicious instruction.
4. Isolate execution and restrict egress
Default-deny network access for execution workers, then allow only the destinations required for the task. Mount files read-only when possible, block privileged containers, and cap process, CPU, memory, storage, and runtime. Kubernetes documents controls including NetworkPolicy, Pod Security Standards, RBAC, quotas, and admission policy in its security guidance.
5. Keep secrets out of prompts and logs
Retrieve credentials only at execution time, scope them to one action, and rotate them automatically. Redact prompts, tool arguments, outputs, traces, and error messages before storage. Encryption at rest does not help if an over-privileged service or user can read every secret; access controls and audit alerts are equally important.
6. Make stopping a first-class control
Set hard ceilings for agent steps, wall-clock time, tokens, queued jobs, tool invocations, and cost. Add a global kill switch and a per-workflow circuit breaker. If evaluation scores fall, tool denials spike, or error loops begin, the system should fail closed rather than continue autonomously.
How to estimate self-hosted agent cost
Do not compare only GPU rental price with API token price. Compare the cost per successful, policy-compliant task. A cheaper model that needs more retries or human correction can be more expensive overall.
A useful monthly model is:
Total cost = compute + idle capacity + storage + networking + platform labor + security and compliance + evaluation and incident overhead.
Then divide by successful completed tasks, not raw requests. Track these inputs:
- Prompt, output, and cached tokens per task
- GPU-hours, utilization, queue time, and energy or cloud cost
- Model-loading time and memory required by context length and concurrency
- Retries, tool failures, timeouts, and human-review minutes
- Storage for models, embeddings, traces, logs, and backups
- Engineering time for upgrades, security reviews, on-call, and capacity planning
Self-hosting tends to become attractive when demand is predictable, GPUs remain well utilized, the workload can use a right-sized open model, and privacy or customization has material value. Hosted APIs often win for bursty demand, rapidly changing model requirements, and teams without inference operations expertise. A hybrid router can keep sensitive or routine tasks local while sending permitted complex tasks to a hosted model.
Measure inference separately from the rest of the agent. vLLM’s production metrics include queue time, time to first token, end-to-end latency, token counts, request success, and KV-cache use. Add business-level measures such as task success, approval rate, rollback rate, and cost per accepted outcome.
A practical rollout plan
- Choose one bounded task. Start with a reversible, high-volume workflow such as log summarization or draft pull-request analysis—not production administration.
- Define a benchmark. Use representative tasks with expected outcomes, prohibited actions, sensitive-data cases, and adversarial instructions.
- Set the data boundary. Document where prompts, documents, embeddings, outputs, traces, and backups may travel and how long they remain.
- Compare two model sizes and a hosted baseline. Measure task success, latency, throughput, memory, and total cost under realistic concurrency.
- Build the tool gateway. Enforce schemas, per-resource authorization, short-lived credentials, output limits, approvals, and immutable audit events.
- Run in shadow mode. Let the agent recommend actions without executing them. Compare its decisions with human operators.
- Enable reversible actions. Begin with low-blast-radius writes, automatic rollback, and human approval. Expand scopes only from evidence.
- Operate it like a service. Version models and prompts, use staged releases, publish SLOs, rehearse the kill switch, and add the workflow to incident response.
Agent changes should follow the same promotion discipline as application changes. Our CI/CD tools comparison can help teams choose the delivery control plane for evaluation gates and staged rollouts. For the human-facing distinction between autonomous workflows and chat interfaces, see AI agents vs assistants vs chatbots.
Frequently asked questions
Are self-hosted AI agents fully private?
Only if every component stays inside the approved boundary. Check inference, embeddings, vector databases, telemetry, crash reporting, model downloads, tool APIs, backups, and remote administration. A local model can still send sensitive data through an external tool or logging service.
Is Ollama suitable for production agents?
Ollama is well suited to local development, prototypes, and some controlled internal services. Production suitability depends on your availability, authentication, concurrency, monitoring, isolation, and support requirements. Put it behind an authenticated gateway and test failure behavior before using it for important workflows.
When should a team use vLLM?
Consider vLLM when you need an OpenAI-compatible serving API, batching, GPU-efficient throughput, and Prometheus metrics. It is an inference server, not a complete agent platform; you still need identity, policy enforcement, tool isolation, durable state, evaluations, and operations.
Can self-hosting eliminate AI API costs?
It can eliminate or reduce per-token vendor charges, but it replaces them with compute, idle capacity, storage, networking, engineering, security, and on-call costs. Calculate cost per successful task at realistic utilization before deciding.
What is the minimum safe production setup?
At minimum: an authenticated gateway, private inference endpoint, version-pinned model, separate policy-enforced tool gateway, scoped short-lived credentials, sandboxed workers, egress restrictions, audit logs, quality and security evaluations, resource ceilings, human approval for high-impact actions, and a tested kill switch.
Bottom line: self-host an agent when control, privacy, predictable utilization, or customization justifies owning the operational burden. The durable design is not “model plus tools”; it is a governed system in which inference can suggest, policy decides, and isolated workers act.
