Sebastian Maniak

One Front Door for Every Tool: MCP Multiplexing Through agentgateway

Tue, 18 Aug 2026 00:00:00 +0000

Here is a problem you don’t have with one MCP server, and can’t avoid with six.

You wire up an MCP server for your firewall so an agent can read policies. Great. Then one for your load balancer. Then your network fabric, your cloud bill, your ticketing system. Each one is useful. But now every developer who wants the firewall tools in their IDE is pasting a URL and a credential into a config file. Multiply that by six systems and a dozen laptops and you’ve built a credential sprawl machine: secrets for production infrastructure scattered across editors, each a place they can leak from, none of them centrally revocable.

The fix isn’t fewer tools. It’s one door. This article walks a real one running on Viper — a single agentgateway endpoint that multiplexes seven MCP servers into one URL, so clients get every tool through one governed front door and never hold a device password at all.

The sprawl, drawn

The thing we’re replacing looks like this — every client speaking every backend’s protocol, holding every backend’s secret:

flowchart LR l1["Laptop / IDE"] l2["Laptop / IDE"] l3["CI job"] f["FortiGate"] b["F5 BIG-IP"] a["Arista cEOS"] aw["AWS"] l1 --> f l1 --> b l1 --> a l2 --> f l2 --> aw l3 --> b l3 --> a style l1 fill:#FFF7D6,stroke:#17181C,color:#17181C style l2 fill:#FFF7D6,stroke:#17181C,color:#17181C style l3 fill:#FFF7D6,stroke:#17181C,color:#17181C

Every one of those arrows is a place a credential lives. The picture we want collapses all of that to a single hop into one proxy, with the backends hidden behind it:

flowchart LR client["Grok Bot / Cursor / Inspector"] gw["agentgateway-proxy :30100 /mcp (viper-mcp backend)"] subgraph kagent["in-cluster · ClusterIP only"] f["fortigate-mcp"] b["f5-bigip-mcp"] a["arista-ceos-mcp"] aw["aws-budget-mcp"] s["servicenow-mcp"] g["gcp-budget-mcp"] k["kagent-tools"] end vault["Vault"] client -->|"Streamable HTTP"| gw gw --> f & b & a & aw & s & g & k vault -.->|"ExternalSecret"| kagent style client fill:#FFF7D6,stroke:#17181C,color:#17181C style gw fill:#FFFFFF,stroke:#E5341F,color:#17181C style vault fill:#FFFFFF,stroke:#17181C,color:#17181C

One URL for the client. Seven backends it never sees. Every secret in Vault. That’s the whole idea — the rest is how it’s wired and why it’s more secure, not less.

The catalog behind the door

The single endpoint is http://172.16.10.135:30100/mcp — the same agentgateway that already fronts the OpenAI and Spark routes on this lab, just with one more backend attached. Behind it sit seven MCP servers, each a Deployment + ClusterIP in the kagent namespace, each speaking Streamable HTTP on :8084/mcp:

Tool prefix	MCP server	Fronts	Tools
`fortigate_`	`fortigate-mcp`	FortiGate 80F firewall	22
`f5-bigip_`	`f5-bigip-mcp`	F5 BIG-IP	6
`arista-ceos_`	`arista-ceos-mcp`	Arista cEOS fabric (eAPI)	6
`aws-budget_`	`aws-budget-mcp`	AWS us-east-2 billing/capacity	11
`servicenow_`	`servicenow-mcp`	ServiceNow tickets	8
`gcp-budget_`	`gcp-budget-mcp`	GCP us-east1 billing/capacity	8
`kagent-tools_`	`kagent-tools`	in-cluster Kubernetes reads	124

A live tools/list through the gateway on 2026-08-18 returned 185 tools across those seven targets. From the client’s side it’s one server; from the operator’s side it’s the entire lab. If those backend names look familiar, they should — several are the very sandbox agents covered in earlier posts. The kagent SandboxAgents still talk to their MCP servers directly over ClusterIP; the gateway is an additional front door for interactive MCP clients.

How it’s wired: one backend, two behaviors

The multiplex is a single AgentgatewayBackend named viper-mcp. Here it is live on the cluster — accepted, with all seven targets:

Live kubectl on k3s-viper, 2026-08-18 — the real CR, Accepted: True. Seven static targets, each *.kagent.svc.cluster.local:8084/mcp over StreamableHTTP.

Two fields in that spec do the interesting work:

failureMode: FailOpen. When a client opens an MCP session, the gateway aggregates the tool lists of all seven backends. If one is down — say the FortiGate MCP pod is restarting — FailOpen means the session still comes up with the other six, instead of the whole aggregate failing because one target was unreachable. One flaky backend degrades gracefully rather than taking down every tool.

prefixMode: Conditional. Seven servers means name collisions are inevitable — more than one backend could plausibly expose a health or a summary. Conditional prefixing namespaces each tool by its target, so arista-ceos’s BGP tool arrives at the client as arista-ceos_bgp_summary and AWS’s as aws-budget_cost_month. “Conditional” because the prefix is applied where it’s needed to disambiguate a large, multi-target catalog — which is exactly the situation with 185 tools.

An HTTPRoute named viper-mcp attaches that backend to the agentgateway-proxy Gateway on the path prefix /mcp. You can see it living alongside the lab’s other routes:

The viper-mcp backend and route sit next to the existing openai and desktop routes — one Gateway, many front doors.

The whole thing is a handful of YAML in git (platform/agentgateway-ai/backend-viper-mcp.yaml and httproute-viper-mcp.yaml). No new gateway, no new ingress — just one more backend on a proxy that was already there. And it’s the same proxy that fronts the models: /v1 (gpt-5.5), /spark (Qwen), and /desktop all live on this listener too. MCP is one more path on it, not a second box.

What a tool call actually does

The two behaviors above are easiest to understand by following a single request from initialize to answer. Here’s a client asking for FortiGate policies:

sequenceDiagram autonumber participant C as Grok Bot / Cursor participant G as agentgateway :30100/mcp participant P as fortigate-mcp participant T as FortiGate 172.16.10.1 C->>G: initialize (Streamable HTTP) G-->>C: serverInfo — agentgateway 1.4.1 C->>G: tools/list G->>P: list (and the other six, FailOpen) P-->>G: fg_list_policies … G-->>C: fortigate_fg_list_policies … C->>G: tools/call fortigate_fg_list_policies G->>P: strip prefix → fg_list_policies Note over P: Vault token is already in the pod P->>T: FortiOS REST T-->>P: policies P-->>G: result G-->>C: result

Two moments on that diagram carry the whole design:

On the way in (step 7), the prefix is stripped. The client calls fortigate_fg_list_policies; the gateway routes on the fortigate_ prefix, removes it, and hands the backend the plain fg_list_policies it actually implements. Namespacing is a gateway concern, invisible to the MCP server.
The credential never moves. The FortiOS token was synced into the fortigate-mcp pod by ExternalSecret long before this request. The gateway doesn’t inject it, doesn’t see it, and the client never had it. The secret’s blast radius is one pod.

One more nuance worth stating: the kagent UI uses this same Gateway for the model (/v1 → gpt-5.5), but its SandboxAgents reach their tools over ClusterIP directly — they skip the gateway for tool calls. The /mcp front door exists for interactive clients like Grok Bot and Cursor, not for the in-cluster agents.

Why one door is safer than many

It’s tempting to read “central endpoint” as “central risk.” It’s the opposite, and the reasons are worth being explicit about:

The MCP servers never leave the cluster. Every backend is a ClusterIP Service. The only thing published on the node is the gateway’s :30100, and it’s LAN-only. There is no path from the internet — or even from a random VLAN — to the FortiGate tools.
Device passwords and cloud keys stay in Vault. Each MCP server pulls its own credential from Vault via an ExternalSecret (secret/platform/fortigate, .../f5-bigip, and so on). The credential lives in the pod that needs it and nowhere else. A client calling through /mcp never holds — never sees — a device password.
One client identity instead of a dozen. Instead of every laptop authenticating to FortiOS, iControl, and eAPI in its own way, a single agentic client speaks to the gateway. One identity to reason about, one place to revoke.
The public surface stays documentation-only. The lab’s public site and GitHub Pages carry docs; the tool plane (:30100) and the kagent UI (:30500) are never published there.

Contrast that with the sprawl diagram: six protocols implemented in a dozen editors, each holding long-lived secrets. Centralizing the tool plane behind one governed proxy shrinks the credential footprint from “everywhere” to “one namespace, backed by Vault.”

What it does — and what it doesn’t, yet

It’s worth being precise about which of these properties are wired today versus which are the gateway’s potential. On this lab, right now, the gateway is doing exactly six jobs:

Job	How it shows up here
Single front door	One `agentgateway-proxy` for models, MCP, and desktop — not a second proxy
MCP multiplex	Seven Streamable HTTP targets behind one `/mcp` URL
Name isolation	`prefixMode: Conditional` → `fortigate_fg_…`, `arista-ceos_…`
Partial failure	`failureMode: FailOpen` — a dead budget MCP doesn’t hide FortiGate tools
Hide ClusterIP	Clients never need `fortigate-mcp.kagent:8084`, only `:30100`
Keep secrets in Vault	The gateway injects no device or cloud keys; the MCP pods already hold ExternalSecrets

And, just as importantly, what is not wired yet — so nobody mistakes this lab for a hardened deployment: there is no client auth on /mcp (it’s LAN, no bearer, same posture as /spark), no tool-level allow lists, and no public-internet exposure. Anything on the LAN that can reach :30100 can call these tools. That’s an acceptable trade for a LAN lab; it would not be for production, and the honest move is to name it rather than imply a security boundary that isn’t there.

Using it: Grok Bot as the governed client

The intended way to actually use the multiplex is not to wire it into your editor — it’s to ask Grok Bot, the k8s-viper agent, in plain language. It already has a jump onto Viper and calls http://127.0.0.1:30100/mcp from the host, so you never paste a token and never publish /mcp:

“What is the BGP summary on spine1?”

“List FortiGate policies that mention YouTube.”

“AWS MTD spend in us-east-2.”

One sentence in; the agent picks the right namespaced tool (arista-ceos_bgp_summary, fortigate_*, aws-budget_cost_month), calls it through the gateway, and answers. You’re driving seven systems’ worth of read-only tooling through a single chat, and the credentials for all of them stayed in Vault the entire time.

If you do want raw tools in a client, the same URL works from anything on the LAN — a Cursor mcpServers entry pointed at http://172.16.10.135:30100/mcp, or MCP Inspector over Streamable HTTP — with the standing rule that it never goes on a public machine. (A quick sanity check: GET / on :30100 returns 404 route not found. That’s expected — the tools live at /mcp, not the root.)

The rules that keep it honest

The doc this is drawn from is blunt about its own guardrails, and they’re worth repeating because they’re what make “one door” defensible:

LAN only. Do not publish :30100 or the kagent UI :30500 on the public site.
Vault path names only in git. Never commit a secret value; never paste one into chat.
Don’t bump the pins to “fix” a missing tool. kagent 0.10.0-rc2 and Substrate 0.0.9 are pinned on purpose.
FailOpen is a feature, not a mask. A missing tool usually means a backend pod is down — check the Deployment, don’t assume the gateway is broken.

The takeaway

MCP is going to give every system in your environment an agent-friendly interface. That’s the good news and the scaling problem in one sentence: the moment you have several, “how does each client reach each server, and who holds the keys?” becomes the real design question.

Multiplexing answers it by inverting the topology. Instead of N clients × M servers each carrying credentials, you get one front door, one client identity, and M servers that never leave the cluster with their secrets sealed in Vault. The client experience gets simpler — one URL, every tool — and the security posture gets tighter, not looser. On Viper that’s 185 tools across seven systems, reachable by asking Grok Bot a question in English, with not a single device password ever leaving the cluster.

One door. Every tool. Keys stay home.

The multiplex config, the full server catalog, and the live captures in this article are documented in k8s-viper / docs/mcp-servers, with the individual MCP servers in sebbycorp/kagent-agent-substrate-demos.

The After-Hours Kill Switch: Daytime Claude, Nighttime Grok, One CEL Expression

Tue, 18 Aug 2026 00:00:00 +0000

Here’s a decision every team using premium models eventually faces, usually after reading a bill: the expensive model is worth it during the workday, and it very much is not worth it at 2 a.m. for a batch job nobody is watching. Claude’s quality earns its price when a human is in the loop. Overnight, a cheaper, faster model is fine — and the difference, multiplied across every off-hours request, is real money.

The naive fix is an if statement in every client: check the hour, pick a model. Now that logic lives in a dozen codebases, each with its own idea of “after hours,” each needing a redeploy to change. The better fix is to make the client dumb and the gateway smart: let every app always ask for the same model by name, and let one policy decide where that request actually goes.

This is a walk through a small, complete agentgateway demo — 12-after-hours-kill-switch — that does exactly that. Clients always send "model": "claude". During daytime Toronto hours the gateway routes to Anthropic’s Claude Sonnet. After 7pm it silently rewrites every one of those requests to xAI’s Grok, until 8am. Same URL, same model name, no SDK change, no cron, no client-side clock check. The switch is a single CEL expression in config.

The idea: the client asks, the gateway decides

The whole demo turns on one inversion of control. The client doesn’t name a provider — it names an intent: claude. That’s a public virtual model. What it resolves to is the gateway’s decision, re-evaluated on every single request.

flowchart TB c["Client POST /v1/chat/completions model: claude"] g["agentgateway :4000 virtual model 'claude'"] cond{"daytime in Toronto? (and not forced off)"} cloud["Anthropic claude-sonnet-4-6"] grok["xAI grok-4.6"] c --> g --> cond cond -->|"yes — daytime quality"| cloud cond -->|"no — after-hours kill switch"| grok style g fill:#FFFFFF,stroke:#E5341F,color:#17181C style cond fill:#FFF7D6,stroke:#E5341F,color:#17181C style cloud fill:#F1EFE9,stroke:#17181C,color:#17181C style grok fill:#F1EFE9,stroke:#17181C,color:#17181C

The application code never changes between noon and midnight. It always sends claude. The gateway is the only thing that knows there are two backends behind that name — and which one is in force right now.

You can see the shape of it in the standalone admin UI’s overview: LLM enabled, one virtual model in front of two backend models, backed by two shared providers.

The standalone admin UI at :15000/ui/ — one virtual model, two backends, two providers. MCP isn’t part of this demo; it’s pure LLM routing.

How it’s built: virtual model + conditional routing + CEL

Three pieces of config.yaml compose the switch.

Providers hold the credentials, once — just the two this demo needs:

providers:
 - name: anthropic
 provider: anthropic
 params: { apiKey: "$ANTHROPIC_API_KEY" }
 - name: xai
 provider: xai
 params: { apiKey: "$XAI_API_KEY" }

Concrete models are the real upstreams — and, crucially, both are visibility: internal:

models:
 - name: anthropic-claude
 visibility: internal
 provider: { reference: anthropic }
 params: { model: claude-sonnet-4-6 }
 - name: xai-grok
 visibility: internal
 provider: { reference: xai }
 params: { model: grok-4.6 }

The virtual model claude is the only public name, and it routes conditionally. Its routing.conditional.targets are a list of CEL when expressions, evaluated top to bottom — first match wins:

virtualModels:
 - name: claude
 routing:
 conditional:
 targets:
 - model: anthropic-claude
 when: |
 default(request.headers["x-force-after-hours"], "") != "true"
 && timestamp(request.startTime).getHours() >= 12
 && timestamp(request.startTime).getHours() < 23 
 - model: xai-grok
 when: "true" # the fallback — always matches

Read it as a policy sentence: use Anthropic only if nobody forced the switch and it’s inside the daytime UTC window; otherwise fall through to Grok. That final when: "true" is the safety net — it always matches, so no request ever fails to route somewhere.

The admin UI renders the same structure: two backend models with no policy, and the claude virtual model marked conditional with 2 rules.

claude is a Virtual model with a Conditional policy (2 rules). The two concrete backends carry no policy of their own — the routing lives entirely on the virtual name.

flowchart LR pub["Public name claude"] t1{"target 1 when: daytime & not forced"} t2["target 2 when: true"] m1["anthropic-claude (internal)"] m2["xai-grok (internal)"] pub --> t1 t1 -->|match| m1 t1 -->|no match| t2 --> m2 style pub fill:#FFF7D6,stroke:#17181C,color:#17181C style t1 fill:#FFFFFF,stroke:#E5341F,color:#17181C style m1 fill:#F1EFE9,stroke:#17181C,color:#17181C style m2 fill:#F1EFE9,stroke:#17181C,color:#17181C

Why the switch can’t be dodged

Here’s the detail that turns a routing trick into a control: both concrete models are visibility: internal. A client cannot send "model": "anthropic-claude" to force Claude at 3 a.m., and cannot name xai-grok either — internal models aren’t addressable from outside. The only door into the gateway is the public name claude, and it always runs the CEL gauntlet first.

That’s the difference between a convenience and a guardrail. If clients could name the real upstream, “after-hours kill switch” would be a polite suggestion. Because they can’t, it’s enforced.

The timezone footgun (read this before you copy the CEL)

The one thing that trips everyone up: timestamp(request.startTime).getHours() returns the hour in UTC, not your local time. The demo’s business hours are America/Toronto, which in August 2026 is EDT (UTC−4), so the config translates the intended local window into UTC:

Local (America/Toronto)	UTC `getHours()`	Served
Daytime `08:00`–`18:59`	`12`–`22`	Anthropic `claude-sonnet-4-6`
After-hours `19:00`–`07:59`	`23`–`11`	xAI `grok-4.6`

That’s why the CEL reads >= 12 && < 23 rather than >= 8 && < 19. If you lift this pattern, convert your own local window to UTC — and remember it drifts by an hour across DST, so the boundaries you hardcode in August aren’t the ones you want in January.

Proof: a 7:11pm request served Grok

This is the moment the switch fires. A client asked for claude with no extra headers at 7:11pm Toronto — past the 7pm cutoff — and the gateway served grok-4.6:

Live, 2026-08-18 at 19:11 Toronto. Client sent claude; the JSON model field came back grok-4.6. The rewrite is invisible to the caller — only the response body reveals it.

There’s a subtlety worth calling out, visible in the admin UI’s Chat Playground: the playground labels the request with the public name claude even on a call that Grok actually served. The rewrite isn’t in the label — it’s in the response’s model field.

The Playground always shows the public name claude. To see which backend served a call, read the model field in the response JSON (claude-sonnet-4-6 = Anthropic, grok-4.6 = xAI) — or gen_ai.provider.name in the gateway log.

To demonstrate the night path without waiting until 7pm, the config honors one break-glass header — x-force-after-hours: true — which forces Grok at any hour:

curl -s http://localhost:4000/v1/chat/completions \
 -H "Content-Type: application/json" \
 -H "x-force-after-hours: true" \
 -d '{"model":"claude","messages":[{"role":"user","content":"Reply with exactly: ok"}],"max_tokens":16}' \
 | jq '{model, content: .choices[0].message.content}'

At noon that returns grok-4.6 instead of claude-sonnet-4-6 — the same rewrite the clock would trigger at night, on demand.

Why this is worth doing

Step back from the specifics and the pattern is a small piece of platform governance with an outsized payoff:

One switch, whole fleet. Every client that speaks OpenAI-compatible HTTP is governed by this one config. Changing the after-hours destination — or the window — is a config edit, not a fleet-wide redeploy.
Cost control on a clock. Premium models cost real money per token. Confining Claude to business hours and defaulting off-hours traffic to a cheaper model is a lever you can pull without asking a dozen teams to touch code.
Provider-agnostic clients. Application code names an intent (claude), not a vendor. Swapping the upstream, adding a failover, or retiring a model id happens in the gateway — the demo even pins grok-4.6 because xAI retired the older grok-2-latest the docs still show. Clients never noticed.
Enforced, not advisory. Internal-visibility upstreams mean the policy is a wall, not a naming convention. There’s no client-side flag to disrespect.

The honest scope: this is a standalone lab demo of routing governance. The /v1 listener here has no client auth (it’s a local demo), and the “kill switch” governs which model a request reaches, not whether the caller is allowed — that’s a separate policy layer. What it demonstrates cleanly is the principle: the decision of where model traffic goes belongs in the gateway, expressed as policy, re-evaluated per request.

The takeaway

A kill switch is only useful if it lives in one place and someone can actually reach it. Scattering model-selection logic across every client gives you neither. Pulling it into agentgateway as one virtual model and one CEL expression gives you both: a single, enforced, per-request decision about where your LLM traffic goes — flippable by the clock or one break-glass header, with not a single client redeploy.

Business hours, Claude. After hours, Grok. Same claude every time — and the clients never have to know which one answered.

Full config, the curl cases, the run scripts, and the live screenshots are in sebbycorp/agentgateway-demos / 12-after-hours-kill-switch. Background on virtual models and conditional routing is in the agentgateway standalone docs.

The Art of the Possible: Governed Sandbox Agents, and an Arista Operator That Can't Go Rogue

Tue, 18 Aug 2026 00:00:00 +0000

Ask a network engineer if they’d let an AI run commands on their production fabric and you’ll get a very short answer. The instinct is correct: a model that can configure, clear ip bgp, or shutdown an interface is a model that can take down a data center between two tokens. For years that instinct has kept agents out of the systems where they’d be most useful.

But the instinct is aimed at the wrong thing. The danger was never “an AI touches the network.” The danger is an AI with an ungoverned reach into the network. Remove the reach — cage it, scope it, watch it — and the same agent becomes something you’d actually want on call: a tireless read-only operator that can answer “is the fabric healthy?” at 3 a.m. without ever being able to break it.

This article is about that cage, and the art of what becomes possible once you have one. The vehicle is a real kagent demo — a SandboxAgent that operates a live 3-node Arista cEOS fabric — and the point it proves generalizes far past networking: with a governed sandbox substrate, you can safely put an agent in front of almost anything.

(If you want the foundations of the substrate itself — gVisor per-session isolation, snapshot/restore — start with Why Secure Sandbox Substrates Are the Future of AI Agents. This piece builds on it and focuses on governance: skills, tool catalogs, and blast-radius control.)

This time, the agent is the box

Most agent demos talk to a system that already exists — a cloud API, a SaaS backend. This one is different: the thing being operated is a live network fabric stood up just for the agent. Three Arista cEOS switches — spine1, leaf1, leaf2 — wired into a small Clos-ish underlay with Containerlab, running real EOS and real eBGP.

Live on Viper, 2026-08-17. Three ceos:4.33.9M nodes, running, on the arista-ceos management network.

Real EOS, not a mock: spine1 running 4.33.9M on x86_64.

The topology is deliberately boring — boring is checkable. eBGP on point-to-point /31s, one AS per switch, a loopback each. No EVPN, no MPLS, nothing an agent could misread.

flowchart TB subgraph clab["Containerlab fabric: arista-ceos"] spine1["spine1 AS 65000 Lo0 10.255.0.1/32"] leaf1["leaf1 AS 65101 Lo0 10.255.0.11/32"] leaf2["leaf2 AS 65102 Lo0 10.255.0.12/32"] spine1 -->|"Ethernet1 10.0.1.0/31"| leaf1 spine1 -->|"Ethernet2 10.0.2.0/31"| leaf2 end style spine1 fill:#FFF7D6,stroke:#E5341F,color:#17181C style leaf1 fill:#F1EFE9,stroke:#17181C,color:#17181C style leaf2 fill:#F1EFE9,stroke:#17181C,color:#17181C

Now the interesting part: a kagent SandboxAgent named arista-ceos sits in front of this fabric as a read-only network operator. It runs each conversation inside a gVisor actor, reaches the switches through a FastMCP server that speaks the EOS eAPI, and answers questions in plain English. And — this is the whole game — it is governed on three independent layers so that “operator” can never quietly become “attacker.”

Governance layer 1 — skills that encode how to behave

An agent’s tools tell it what it can do. Its skills tell it how it should act. In kagent, skills are markdown that gets folded into the agent’s systemMessage (kagent 0.10.0-rc2 deliberately rejects a spec.skills field on sandbox agents, so the instructions ride in the prompt). That markdown is where operational judgment — and hard guardrails — live.

The Arista agent ships three skills — fabric, routing, and executive-brief — and every one of them opens with the same standing rules. They read like a runbook written by someone who’s been burned:

Read-only. No configure, no write, no neighbor shutdown, no image upgrade.

Never invent BGP state, prefixes, LLDP neighbors, or ping results. If eAPI fails or returns empty, say so.

Never print eAPI passwords, basic-auth headers, or Vault tokens.

No generic CLI dump of the full running-config — it can include the lab AAA line. Prefer the named tools.

Notice what these rules are doing. “Never invent BGP state” turns the model’s most dangerous failure mode — a confident hallucination — into a contract: if the tool didn’t return it, don’t say it. “Never print passwords” is a data-exfiltration guard written in English. “Prefer the named tools over a raw config dump” keeps secrets out of the transcript by construction. The skill even dictates the shape of a good answer — lead with one sentence of fabric health, then a short table — so the agent is useful and predictable.

Skills are governance you can read, diff, and review in a pull request. That’s a very different thing from hoping a general-purpose model behaves.

Governance layer 2 — a curated, read-only tool catalog

Skills are the soft boundary. The tool catalog is the hard one. The agent’s entire vocabulary for touching the fabric is six eAPI wrappers, and every one of them is a show:

Tool	EOS command	Scope
`arista_inventory`	`show version`	all nodes
`arista_bgp_summary`	`show ip bgp summary`	one or all nodes
`arista_interfaces`	`show interfaces`	single node
`arista_lldp_neighbors`	`show lldp neighbors`	one or all nodes
`arista_routes`	`show ip route [prefix]`	single node, optional filter
`arista_health`	composed summary	all nodes, concise

There is no configure, no clear, no reload, no generic “run any EOS command” escape hatch. This matters more than any prompt rule, because it’s not advisory — it’s the shape of the API. Even a perfectly jailbroken model, convinced by some injected instruction that it must reset a BGP session, has no verb to do it. The catalog simply doesn’t contain destruction.

This is the same design decision behind every agent in the family: expose Describe / Get / View / Show, and nothing that mutates. You lose nothing an operator needs for triage, and you remove the entire class of “the agent changed something” incidents.

Governance layer 3 — the sandbox, the secrets, and the allowlist

The third wall is the runtime itself.

flowchart LR chat["Operator chat"] ui["kagent UI :30500"] actor["gVisor actor (the session)"] rmcp["RemoteMCPServer arista-ceos-mcp"] mcp["arista-ceos-mcp :8084/mcp"] fabric["cEOS eAPI spine1 · leaf1 · leaf2"] vault["Vault secret/platform/arista-ceos"] eso["External Secrets Operator"] chat --> ui --> actor --> rmcp --> mcp --> fabric vault --> eso --> mcp style actor fill:#FFF7D6,stroke:#E5341F,color:#17181C style mcp fill:#F1EFE9,stroke:#E5341F,color:#17181C style vault fill:#FFFFFF,stroke:#17181C,color:#17181C

Three things are happening on that diagram, each a control:

gVisor isolation. The model session runs in a gVisor actor — a user-space kernel between the agent and the host. Untrusted tool output (and any injection riding in it) never gets to talk to the node’s real kernel.
Secrets the session never holds. The eAPI username and password live in Vault at secret/platform/arista-ceos and are synced by External Secrets Operator into the MCP pod only — as username, password, and a hosts_json map of allowlisted node → eAPI URL. The gVisor actor calls a tool; the tool holds the credential; the session never sees it.
An explicit node allowlist. The MCP server is pinned to ARISTA_ALLOWED_NODES=spine1,leaf1,leaf2. Ask it about a switch that isn’t on the list and there’s nowhere for the request to go. (TLS verification is disabled for this lab only — a documented lab shortcut, not a production posture.)

Stack the three layers and you get genuine defense in depth. A prompt injection has to get past the skill rules, then find a destructive tool that doesn’t exist, then escape a gVisor sandbox, then reach a credential that isn’t in the session, to touch a node that isn’t on the allowlist. Each wall is boring on its own. Together they turn “an AI on the network” from reckless into routine.

Watch it actually work

Here’s the agent answering a real operator question — “What is the BGP summary on spine1?” — with exactly one tool call to arista_bgp_summary:

Live kagent UI. The agent’s whole toolset is visible on the right — six arista_* reads, nothing else. It reports both eBGP sessions Established, one prefix each, and closes with an operator-grade one-liner.

The number that matters isn’t the answer — it’s that the answer is true. Here is the raw show ip bgp summary from spine1 itself:

Ground truth from the switch: 10.0.1.1 (AS 65101) and 10.0.2.1 (AS 65102), both Estab, PfxRcd 1.

Line them up. Router ID 10.255.0.1, AS 65000, peers 65101 and 65102, both Established, one prefix each — the chat matches the CLI exactly. That’s the skill rule “never invent BGP state” paying off: the agent read the real device and reported it, nothing more. An answer you can trust is worth more than a clever one.

Why this is the art of the possible

Step back from Arista and the pattern is the real story. The Arista operator is one of a family of sibling agents in the same repo, each built the identical way — gVisor sandbox, Vault-backed secrets, read-only tools, skills in the system message — and each pointed at a different corner of the enterprise:

SandboxAgent	Operates	Read-only reach
`aws-budget`	AWS account	Cost Explorer, EC2/RDS/quota describes
`fortigate`	A FortiGate firewall	Policy and status reads
`f5-bigip`	An F5 BIG-IP	LTM/virtual-server state
`service-now`	ServiceNow	Ticket and CMDB reads
`gcp`	A GCP project	Billing and inventory
`arista-ceos`	A live Arista fabric	eAPI `show` commands

The governance is uniform; only the tools change. That’s the art of the possible: once you have a governed substrate, adding a safe operator for a new system is not a security research project — it’s writing a handful of read-only tool wrappers and a page of skill rules. The hard part (isolation, secret handling, blast-radius control) is solved once, in the substrate, and inherited by everything.

flowchart TB sub["Governed Sandbox Substrate (gVisor · Vault · read-only tools · skills)"] a["aws-budget"] f["fortigate"] b["f5-bigip"] s["service-now"] ar["arista-ceos"] sub --> a sub --> f sub --> b sub --> s sub --> ar style sub fill:#FFF7D6,stroke:#E5341F,color:#17181C style ar fill:#F1EFE9,stroke:#E5341F,color:#17181C style a fill:#FFFFFF,stroke:#17181C,color:#17181C style f fill:#FFFFFF,stroke:#17181C,color:#17181C style b fill:#FFFFFF,stroke:#17181C,color:#17181C style s fill:#FFFFFF,stroke:#17181C,color:#17181C

Being honest about the edges

This is v1, and the demo says so plainly:

Read-only by design. There are no write tools, on purpose. The day a configure-class tool is added, it belongs behind human-in-the-loop approval, not in the same catalog as the reads.
eBGP only. No EVPN, VXLAN, or MPLS in the fabric yet — the underlay is kept boring so the agent’s reads are checkable.
Lab shortcuts are labeled. eAPI TLS verification is off for the lab; the cEOS image is Arista-licensed and never vendored into git; the kagent UI is LAN-only. None of these are production claims.
The first boot failed honestly. cEOS died on inotify exhaustion (Too many open files) until the host fs.inotify limits were raised — a real operational wrinkle, recorded rather than airbrushed.

The takeaway: governed autonomy

The lesson of the Arista agent isn’t that the model is smart. It’s that the model is contained — and containment is what makes the smartness usable. Skills say how to behave. A curated catalog removes the verbs for harm. A gVisor sandbox, Vault-held secrets, and a node allowlist mean even a fully hijacked session runs out of road before it reaches anything it could break.

That combination is what lets you do the thing the network engineer’s instinct said you never could: put an agent in front of live infrastructure. Not because you stopped worrying about what an AI might do — but because you built the cage first, and the cage is the product.

Governed autonomy is the unlock. The art of the possible is everything you can safely point it at next.

The Arista fabric, skills, MCP tool map, and the live captures in this article live in sebbycorp/kagent-agent-substrate-demos / arista-ceos-sandbox-agent, with the SandboxAgent wiring documented in k8s-viper / docs/arista-ceos-agent.md.

How To: Build an AWS Budget SandboxAgent on kagent + Agent Substrate

Mon, 17 Aug 2026 09:00:00 -0400

I already wrote the why — why secure sandbox substrates are the future. This one is the how. Same agent, but from the build side: every object you apply, where the AWS key lives, and the one version pin that will waste your afternoon if you get it wrong.

The goal is a single question, answered honestly:

What’s our us-east-2 spend this month, and are we over capacity?

Not “roughly.” Not an estimate. Real ce:GetCostAndUsage numbers, or an explicit denied.

Live kagent UI, 2026-08-16. Three SandboxAgent cards, not plain Agents. aws-budget is described as “Executive AWS budget and capacity assistant for us-east-2 (gVisor).”

Why a SandboxAgent and not a plain Agent

A normal kagent Agent is a Kubernetes Deployment. Always on, container isolation, same as any other pod. That is completely fine for a cluster helper that reads pod logs.

This agent talks to your AWS bill. The model gets a filesystem, memory, and a live network for the whole conversation, and it chews on tool output it did not author. Substrate puts that session in a gVisor actor — a user-space kernel between the session and your k3s host.

	Plain kagent `Agent`	`SandboxAgent` (Substrate)
Kubernetes shape	Deployment, always-on pod	Actor on WorkerPool `kagent-default`, booted per session
Isolation	Container, shared host kernel	gVisor user-space kernel
Idle cost	A pod per conversation	Snapshot (zstd), worker freed
Resume	N/A	Restore the exact session

Honest rule of thumb: if all you need is a Python container with boto3 and no snapshot lifecycle, use a Deployment. The moment the session reads your bill, take the wall.

Architecture

flowchart LR chat["Executive chat"] ui["kagent UI :30500"] a2a["A2A sandboxes /api/a2a-sandboxes"] actor["gVisor actor ateom-gvisor:v0.0.9"] rmcp["RemoteMCPServer aws-budget-mcp"] mcp["aws-budget-mcp :8084/mcp"] aws["AWS us-east-2"] vault["Vault secret/platform/aws-budget"] eso["External Secrets Operator"] chat --> ui --> a2a --> actor --> rmcp --> mcp --> aws vault --> eso --> mcp style actor fill:#FFF7D6,stroke:#E5341F,color:#17181C style mcp fill:#F1EFE9,stroke:#E5341F,color:#17181C

Read that bottom row carefully. Vault and ESO feed the MCP pod, not the actor. The gVisor session — the part running the model — never holds an AWS access key. It calls a tool; the tool makes the API call on the far side of the wall.

The pins that matter (do not bump)

This is the part I would tattoo on the runbook.

Piece	Value
kagent OSS Helm + CRDs	`0.10.0-rc2`
Agent Substrate Helm + CRDs	`0.0.9`
Worker image	`ghcr.io/kagent-dev/substrate/ateom-gvisor:v0.0.9`
Pattern	Go declarative `SandboxAgent` + FastMCP + `RemoteMCPServer` + `ExternalSecret`
Model	`default-model-config` (my lab: gpt-5.5 via agentgateway)
Region	us-east-2 only

kagent rc2 always writes an ActorTemplate with spec.pauseImage and env[].valueFrom.secretKeyRef (for KAGENT_CONFIG_JSON, KAGENT_AGENT_CARD_JSON, KAGENT_SRT_SETTINGS_JSON, OPENAI_API_KEY).

Substrate 0.0.9 CRDs accept that shape. Substrate 0.0.12 removed valueFrom and moved the pause image into SandboxConfig. Point rc2 at 0.0.12 and the apiserver rejects the object — you get Ready=False, ActorTemplateNotFound, or spec.containers[0].env[…].value: Required value.

The trap: the first time your agent is not Ready, your instinct is to upgrade. Don’t. A pin mismatch is a CRD problem, not a staleness problem. And don’t “fix” it by flattening the env into literal values either.

Snapshots today: rustfs, and omit `snapshotsConfig`

Substrate checkpoints actor RAM and filesystem to object storage. On my Viper lab that storage is in-cluster rustfs, not Google Cloud:

	Live today
atelet 0.0.9	`ATE_STORAGE_BACKEND=s3` → `http://rustfs.ate-system.svc:9000`
Bucket	`ate-snapshots` (1Gi PVC)
SandboxAgent	omits `snapshotsConfig`
ActorTemplate location	`gs://ate-snapshots/kagent/aws-budget`

The gs:// scheme there is a URI prefix only. The bytes are on rustfs. I have a real GCS bucket (gs://viper-kagent-ate-snapshots, project viper-kagent, org maniak.io) reserved for a later cluster-wide atelet cutover, and I deliberately did not point this agent at it — atelet would go looking for that bucket on rustfs, not find it, and the golden snapshot would fail. No ignoreDifferences. Stay on rustfs.

Step 1 — a dedicated IAM identity

AWS console, region us-east-2 → IAM → create user aws-budget-agent, no console password → attach a customer-managed policy → create an access key (“application running outside AWS”).

The policy is AwsBudgetAgentReadOnly: ce:Get*, budgets:View*, ec2:Describe* and siblings, conditioned to us-east-2 wherever IAM allows a region condition. No iam:Create*. No ec2:Terminate*. No budgets:Delete*. No s3:* on your data.

Why a dedicated identity: the agent is read-mostly and it must not borrow a human admin key. If someone talks the model into something clever, the ceiling is “read one region’s billing metadata.”

Two housekeeping notes:

Enable Cost Explorer in the billing console if the account has never used it. Otherwise aws_cost_* fails honestly — and an honest failure is success, not a reason to fake $0.
Compute Optimizer / CE rightsizing enrollment is optional. Skip it and aws_rightsizing_hints must say “not available.”

Step 2 — keys into Vault, never git

docker exec -it k3s-viper kubectl -n vault exec -i vault-0 -- \
 vault kv put secret/platform/aws-budget \
 access_key_id='<paste>' \
 secret_access_key='<paste>' \
 region='us-east-2'

Then let ESO do the sync:

docker exec k3s-viper kubectl -n kagent get externalsecret aws-budget-mcp

You want STATUS: SecretSynced. Git only ever holds the ExternalSecret mapping — path names and key names. Never values. And please don’t kubectl get secret -o yaml and paste the result into Slack.

Step 3 — build and import the MCP image

Dockerized k3s cannot see host Docker images, and there is no registry pull for aws-budget-mcp:dev, so you import it into containerd:

./aws-sandbox-agent/scripts/02-build-import-mcp.sh

That builds the image and runs ctr images import. The Deployment uses imagePullPolicy: IfNotPresent on purpose. ImagePullBackOff means the import didn’t land or your tag doesn’t match.

Step 4 — apply the agent

Note the shape of this command. Kustomize runs on the host, and the output is piped into k3s — not apply -k against a host path the container can’t see:

kubectl kustomize aws-sandbox-agent/k8s | docker exec -i k3s-viper kubectl apply -f -

externalsecret.external-secrets.io/aws-budget-mcp created
deployment.apps/aws-budget-mcp created
service/aws-budget-mcp created
configmap/aws-budget-skills created
remotemcpserver.kagent.dev/aws-budget-mcp created
sandboxagent.kagent.dev/aws-budget created

One wrinkle worth knowing: kagent 0.10.0-rc2 rejects spec.skills on a SandboxAgent, and there is no skill-mount into the gVisor session. So the skill text lives in ConfigMap aws-budget-skills and gets inlined into systemMessage via declarative.promptTemplate. Keep skills/ and k8s/skills-configmap.yaml in sync.

Step 5 — what “Ready” actually means

kagent does not start a Deployment of the LLM runtime. It creates an ActorTemplate owned by the SandboxAgent. Substrate boots a golden actor once, checkpoints it, and stores that checkpoint as status.goldenSnapshot. New chats restore from that image.

You see	Meaning
No `ActorTemplate`	Wrong CRD pin (0.0.12) or the controller isn’t reconciling
Template, no `goldenSnapshot`	First checkpoint running (60–90s is normal), or gVisor/atelet can’t write storage
`Ready=True` + `goldenSnapshot: gs://…`	You can talk to it

docker exec k3s-viper kubectl -n kagent get sandboxagents,remotemcpservers
docker exec k3s-viper kubectl -n kagent get actortemplates

The tool catalog — no generic shell

Eleven tools, all Describe/Get/View/List. This is the single most important design decision in the demo, and it’s a design decision, not a config flag:

Tool	AWS API (typical)
`aws_whoami`	`sts:GetCallerIdentity`
`aws_cost_month`	`ce:GetCostAndUsage`
`aws_cost_by_service`	`ce:GetCostAndUsage` grouped by service
`aws_budgets`	`budgets:ViewBudget`
`aws_ec2_capacity`	`ec2:DescribeInstances`
`aws_asg`	`autoscaling:Describe*`
`aws_rds`	`rds:Describe*`
`aws_ebs_summary`	`ec2:DescribeVolumes`
`aws_service_quotas`	`servicequotas:GetServiceQuota`
`aws_rightsizing_hints`	CE rightsizing / Compute Optimizer
`aws_executive_brief`	composes the above

There is no “run any aws cli” tool. The agent has no verb for destruction. That boundary is enforced three independent times — in the tool code, in the IAM policy, and in the sandbox — so getting past one layer still lands you on the next.

Small API detail that trips people up: Cost Explorer and Budgets live in us-east-1. The tools still filter results to us-east-2.

The live run

2026-08-16 on Viper, through /api/a2a-sandboxes/kagent/aws-budget:


Month-to-date spend	$0.67
Budget	$4.13 / $100 (4.13% used)
EC2 / ASG / RDS / EBS	0 / 0 / 0 / 0
Identity	`aws-budget-agent`, account 616973157416
Tool calls	10 / 10

Live kagent UI, 2026-08-16. Ten of ten tool calls, every number sourced from a real AWS API.

The line I care about most is not the dollar figure. It’s this: the agent reported “Cost Explorer rightsizing API denied; Compute Optimizer not enrolled” instead of inventing a recommendation. An agent that fabricates a helpful-sounding number to fill a gap is an agent you cannot put in front of a budget.

If yours does invent spend, the tools didn’t run. Tell it “call the tools; do not estimate,” then check the MCP logs:

docker exec k3s-viper kubectl -n kagent logs deploy/aws-budget-mcp

That process must never print the secret key.

Proof, with nothing sensitive on screen

Live capture, 2026-08-16. Ready=True, MCP Accepted with 11 tools, pod 1/1 Running — and deliberately no Vault tokens or AWS keys in frame.

And the whole turn as a short reel:

One tell that these are genuinely sandboxed sessions: the classic /api/a2a/kagent/aws-budget endpoint 404s, because there is no Agent CR at all. The UI talks to /api/a2a-sandboxes/kagent/aws-budget, and the card wears a Sandbox: Agent Substrate badge.

Honest limits

The image must be imported on the k3s node (ctr images import) before the MCP pod starts.
The Vault path must exist first, or the ExternalSecret sits unsynced forever.
Nested gVisor on dockerized k3s can still hit runsc / seccomp / /dev/kvm problems. That’s a worker-environment issue — not a reason to downgrade the agent to an unsandboxed Deployment.
Snapshots are local rustfs. Cluster-wide object storage is future work.
Never commit AWS or GCP secret values. Manifests carry paths and key names only.

The rest of the series

Five demos, one runtime — same pins, same Vault/ESO shape, same gVisor wall. Different blast radius each time.

GCP budget — us-east1 capacity, and why the billing half honestly reports unavailable.
ServiceNow triage — 25 incidents into a manager briefing, and where write tools belong.
FortiGate 80F — my actual home firewall, 22 tools, and 3,007,844 hits on one policy.
F5 BIG-IP — 19 VIPs, 18 tool calls in one turn, and zero write tools.
Why secure sandbox substrates are the future — the argument behind all five.

Full demo — manifests, the FastMCP server, IAM policy, runbooks, and the live report this draws on: sebbycorp/kagent-agent-substrate-demos / aws-sandbox-agent.

A GCP Budget SandboxAgent on kagent — and the Value of an Agent That Says "Unavailable"

Mon, 17 Aug 2026 08:45:00 -0400

This is the GCP sibling of my AWS budget SandboxAgent: a gVisor-isolated agent that answers

What’s our us-east1 spend this month, and are we over capacity?

for org maniak.io, with real Cloud Billing and Compute Engine numbers and a service-account JSON that never touches git.

Here’s the thing: on the live run, half of it didn’t work. And that turned out to be the most useful part of the demo, so I’m publishing it that way rather than re-shooting a clean take.

Live kagent UI, 2026-08-16. Isolated sandboxes, not plain Agents — kagent/gcp-budget is on the grid.

Why a SandboxAgent and not a plain Agent

A normal kagent Agent is a Deployment: always on, container isolation, same as any other pod. Fine for a cluster helper.

This one reads the GCP bill and the Compute Engine inventory. The model gets a filesystem, memory, and a network for the whole chat. Substrate puts that session in a gVisor actor on WorkerPool kagent-default:

Isolated sandbox. gVisor’s user-space kernel is the wall between the model session and my Viper/k3s host. Tools reach GCP through the MCP pod; the service-account JSON stays in Vault, not in the actor.
Idle chats snapshot (zstd) and free the worker. The next message restores that same session instead of cold-booting a container.
No always-on pod per executive conversation.
A golden snapshot you can resume deterministically.

If you only needed a Python container with the google-cloud-* clients and no snapshot lifecycle, a Deployment would do. That’s not this.

Tradeoff, stated plainly: nested gVisor on dockerized k3s is finicky, and snapshots land on in-cluster rustfs today — the gs:// you’ll see is a URI prefix, not live GCS.

Architecture

flowchart LR chat["Chat"] ui["kagent UI :30500"] a2a["A2A sandboxes"] actor["gVisor actor ateom-gvisor:v0.0.9"] rmcp["RemoteMCPServer gcp-budget-mcp"] mcp["gcp-budget-mcp :8084/mcp"] gcp["GCP us-east1"] vault["Vault secret/platform/gcp-budget"] eso["ESO"] chat --> ui --> a2a --> actor --> rmcp --> mcp --> gcp vault --> eso --> mcp style actor fill:#FFF7D6,stroke:#E5341F,color:#17181C style mcp fill:#F1EFE9,stroke:#E5341F,color:#17181C

The service-account JSON goes Vault → External Secrets Operator → the MCP pod. Not the actor. The session that’s running the model and parsing untrusted tool output never holds a Google credential.

Pins (do not bump)

Piece	Value
kagent OSS Helm + CRDs	`0.10.0-rc2`
Agent Substrate Helm + CRDs	`0.0.9`
Worker image	`ghcr.io/kagent-dev/substrate/ateom-gvisor:v0.0.9`
Model	`default-model-config` (gpt-5.5 via agentgateway)
Region	us-east1 only
Org	maniak.io
Projects (names only)	viper-kagent, maniak-io, qr-maniak-io

rc2 writes ActorTemplate with spec.pauseImage and env[].valueFrom.secretKeyRef. Substrate 0.0.9 accepts that. 0.0.12 removed valueFrom and moved the pause image to SandboxConfig, so the apiserver rejects rc2’s object and you get Ready=False / ActorTemplateNotFound.

Do not “upgrade to fix” a Ready=False agent. That’s how I already burned an afternoon on this lab. A pin mismatch is a CRD problem.

Build it

1. Service account. GCP console, org maniak.io → IAM → create gcp-budget-agent → attach read-mostly permissions → download the JSON key once, and keep that file outside the repo. Enable Cloud Billing, Cloud Billing Budget, Compute Engine, and Cloud Resource Manager APIs on the projects it will read.

2. Vault. Path secret/platform/gcp-budget, keys credentials_json, billing_account, project, region. ESO syncs it into a Kubernetes Secret consumed only by the MCP pod. Git holds the ExternalSecret mapping — names, never values.

3. Image. Build gcp-budget-mcp:dev on the node and ctr images import it, because dockerized k3s can’t see host Docker images and there’s no registry pull for a :dev tag.

4. Apply. Kustomize on the host, piped into the cluster:

kubectl kustomize gcp-sandbox-agent/k8s | docker exec -i k3s-viper kubectl apply -f -

5. Verify. Expect this, and nothing sensitive in it:

NAME READY ACCEPTED
sandboxagent.kagent.dev/gcp-budget True True

NAME PROTOCOL URL ACCEPTED
remotemcpserver.kagent.dev/gcp-budget-mcp STREAMABLE_HTTP http://gcp-budget-mcp.kagent:8084/mcp True

NAME READY STATUS RESTARTS AGE
pod/gcp-budget-mcp-5664dfb8f7-pwlwv 1/1 Running 0 4m41s

NAME CLASS
actortemplate.ate.dev/gcp-budget-82cc62c613737d64 gvisor
location: gs://ate-snapshots/kagent/gcp-budget
goldenSnapshot: gs://ate-snapshots/kagent/gcp-budget/5edafe3c-.../2026-08-16T17:55:34Z-XIKENZRG7IHYGCONXJZFBBFY2R
phase: Ready

Ready=True plus a goldenSnapshot means Substrate booted a golden actor, checkpointed it, and new chats will restore from that image.

The tools

Eight, all read-only:

Tool	What it reads
`gcp_whoami`	The service-account identity actually in use
`gcp_cost_month`	Month-to-date attempt via Cloud Billing
`gcp_cost_by_service`	Per-service breakdown
`gcp_budgets`	Cloud Billing Budgets
`gcp_projects`	Resource Manager project list
`gcp_compute_capacity`	Instances and disks, zone-filtered to us-east1
`gcp_quotas`	Compute quotas vs. usage
`gcp_executive_brief`	Composes the above

No generic “run any gcloud” tool. No project-delete, no IAM-create.

The live run, including what broke

Live kagent UI, 2026-08-16. Five tool calls on Q1, three on Q2. Billing unavailable with the error quoted verbatim; capacity answered cleanly.

Q1 — “What’s our GCP budget status and which projects are on the billing account?” (~18s, tools: gcp_whoami, gcp_cost_month, gcp_budgets, gcp_cost_by_service, gcp_projects)

Billing account 011C38-867461-BE95B1, linked projects, budgets, and MTD spend all came back unavailable. The runtime error, quoted in the answer:

ImportError: cannot import name 'billing_budgets_v1' from 'google.cloud'

The agent then did something I want to highlight. It separately checked Resource Manager, listed viper-kagent, maniak-io, qr-maniak-io — and explicitly noted that Resource Manager visibility is not the same thing as billing-account linkage. It didn’t quietly present one as the other. My known lab budget (trail budget, $1) was not returned, and it said so.

After the rebuild. The import was wrong: google-cloud-billing-budgets==1.21.0 was pinned and installed, but the old alias from google.cloud import billing_budgets_v1 doesn’t exist. The working path is:

from google.cloud.billing import budgets_v1
client = budgets_v1.BudgetServiceClient()

Rebuilt as gcp-budget-mcp:dev, new pod, re-ran Q1. No more ImportError — and Cloud Billing then returned Unauthenticated for billing.accounts.get, billingbudgets.budgets.list, and cost-by-service. Resource Manager still listed the three projects. The $1 trail budget still didn’t appear. Month-to-date spend still unavailable.

Two failures in a row, and at no point did a dollar amount get invented.

Q2 — “Any compute running in us-east1, and are we near quota?” (~15s, tools: gcp_projects, gcp_compute_capacity, gcp_quotas)

This half worked perfectly:

Quota	Limit	Status
CPUs	200	OK
Instances	24	OK
Total disk GB	4096	OK
SSD total GB	500	OK
In-use addresses	8	OK

Zero instances running, zero stopped, zero disks. Not near quota.

The Cloud Billing gap is real, not a bug in my code

Worth separating two different problems here.

The ImportError was my bug and I fixed it. The Unauthenticated is a permissions gap on that service account. But underneath both sits a structural fact: Cloud Billing Accounts, Budgets, and Catalog do not expose month-to-date spend. There is no “what have I spent so far this month” call in that surface the way ce:GetCostAndUsage gives you on AWS. Real MTD on GCP means BigQuery billing export.

So the tools are written to say unavailable rather than approximate. That’s the design position, and it’s why the AWS agent can report $0.67 while this one can’t report anything.

Why “unavailable” is the feature

It would have been easy to make this demo look better. Return $0.00 on a failed billing call. Present the Resource Manager project list as “projects on the billing account.” Round something plausible. Every one of those makes a nicer screenshot and a worse agent.

An agent wired to your finances has exactly one job when a tool fails: say the tool failed. A confident wrong number is worse than a blank, because a blank gets escalated and a wrong number gets acted on. The whole point of putting keys in Vault, tools behind MCP, and the session behind gVisor is to earn trust in what the agent reports — and you throw that away the first time it papers over a gap to be helpful.

Honest limits

Import the image on the k3s node before the MCP pod starts.
The Vault path must exist or the ExternalSecret stays unsynced.
Cloud Billing does not return MTD spend. The tools say unavailable.
Billing APIs returned Unauthenticated on the live run. That’s a permissions gap I haven’t closed, not a fixed result.
No generic gcloud tool, no project-delete, no IAM-create.
Snapshots are in-cluster rustfs. gs:// is a prefix only.
Never commit the service-account JSON or Vault secret values.

The rest of the series

Five demos, one runtime — same pins, same Vault/ESO shape, same gVisor wall. Different blast radius each time.

AWS budget — Cost Explorer, least-privilege IAM, and the full build from pins to golden snapshot.
ServiceNow triage — 25 incidents into a manager briefing, and where write tools belong.
FortiGate 80F — my actual home firewall, 22 tools, and 3,007,844 hits on one policy.
F5 BIG-IP — 19 VIPs, 18 tool calls in one turn, and zero write tools.
Why secure sandbox substrates are the future — the argument behind all five.

Full demo — manifests, the FastMCP server, security notes, runbooks, and the live report: sebbycorp/kagent-agent-substrate-demos / gcp-sandbox-agent.

A ServiceNow Triage SandboxAgent on kagent — Where Write Tools Belong

Mon, 17 Aug 2026 08:30:00 -0400

Every ITSM demo I’ve ever sat through answers “how many tickets are open.” That’s a COUNT(*). The actual manager question is harder:

What IT tickets are open right now, and how should we organize them?

That needs priority buckets, someone to notice the unassigned P1, and a list you can act on in a standup. So I built it as a gVisor SandboxAgent on kagent + Agent Substrate, pointed at a real ServiceNow personal developer instance, and — unlike my AWS and GCP budget agents — I gave this one write tools. Two of them. That changes the design conversation.

Live kagent UI, 2026-08-16. Isolated sandboxes, not plain Agents — kagent/servicenow on the grid.

Why a SandboxAgent and not a plain Agent

A normal kagent Agent is a Deployment: always on, container isolation. Fine for a cluster helper.

This one talks to a ticketing system full of text other people wrote. That matters more than it sounds — incident descriptions, work notes, and short descriptions are untrusted input flowing straight into a model that holds tools. Prompt injection in a ticket body is not a hypothetical attack; it’s the obvious one.

Substrate puts the session in a gVisor actor on WorkerPool kagent-default:

Isolated sandbox. gVisor’s user-space kernel sits between the model session and my Viper/k3s host. Tools call ServiceNow through the MCP pod; username and password stay in Vault, not in the actor.
Idle chats snapshot (zstd) and free the worker; the next message restores the same session.
No always-on pod per manager conversation.
A golden snapshot you can resume.

Tradeoff on this lab: nested gVisor on dockerized k3s, and snapshots are in-cluster rustfs today (gs:// is a URI prefix only), not GCS.

Architecture

flowchart LR chat["Manager chat"] ui["kagent UI :30500"] a2a["A2A sandboxes"] actor["gVisor actor ateom-gvisor:v0.0.9"] rmcp["RemoteMCPServer servicenow-mcp"] mcp["servicenow-mcp :8084/mcp"] sn["ServiceNow Table API"] vault["Vault secret/platform/servicenow"] eso["ESO"] chat --> ui --> a2a --> actor --> rmcp --> mcp --> sn vault --> eso --> mcp style actor fill:#FFF7D6,stroke:#E5341F,color:#17181C style mcp fill:#F1EFE9,stroke:#E5341F,color:#17181C

The ServiceNow password goes Vault → ESO → the MCP pod. The actor calls tools; it never sees a credential.

Pins (do not bump)

Piece	Value
kagent OSS Helm + CRDs	`0.10.0-rc2`
Agent Substrate Helm + CRDs	`0.0.9`
Worker image	`ghcr.io/kagent-dev/substrate/ateom-gvisor:v0.0.9`
Pattern	Go declarative `SandboxAgent` + FastMCP + `RemoteMCPServer` + `ExternalSecret`
Model	`default-model-config` (gpt-5.5 via agentgateway)
Host (name only)	`https://dev203166.service-now.com`

rc2 always writes ActorTemplate with spec.pauseImage and env[].valueFrom.secretKeyRef. Substrate 0.0.9 accepts that shape; 0.0.12 does not — it dropped valueFrom and moved the pause image into SandboxConfig, so the apiserver rejects rc2’s object. Don’t “upgrade to fix” a Ready=False agent.

Build it

1. Vault. Path secret/platform/servicenow, keys host, username, password. Only the host name is safe to commit. ESO syncs into a Kubernetes Secret the MCP pod consumes.

2. Image. Build servicenow-mcp:dev and ctr images import it onto the k3s node — dockerized k3s can’t see host Docker images, and imagePullPolicy: IfNotPresent is deliberate.

3. Apply. Kustomize on the host, piped in:

kubectl kustomize service-now-sandbox-agent/k8s | docker exec -i k3s-viper kubectl apply -f -

You get the skills ConfigMap, the MCP Service + Deployment, the ExternalSecret, the RemoteMCPServer, and the SandboxAgent.

4. Verify. What I had live on 2026-08-16:

Object	Status
SandboxAgent `servicenow`	`Ready=True`, `Accepted=True`
RemoteMCPServer `servicenow-mcp`	Accepted, 8 tools, `STREAMABLE_HTTP` `http://servicenow-mcp.kagent:8084/mcp`
MCP pod	`servicenow-mcp-7c6c455c65-kvnrq` `1/1` Running, 0 restarts
ActorTemplate	`servicenow-f5f2dec1f2a81a41`, gvisor, Ready
Snapshot prefix	`gs://ate-snapshots/kagent/servicenow` (rustfs)
Golden snapshot	`2026-08-16T15:45:24Z-HYSFX5R3DHWLRWYVP7YXOAF4SN`

Live CLI, 2026-08-16. Nothing sensitive in frame.

Six read tools, two write tools

Tool	Kind	What it does
`sn_whoami`	read	Which ServiceNow user the agent is acting as
`sn_list_incidents`	read	Active incidents, compact rows
`sn_get_incident`	read	One incident by number or sys_id
`sn_search_incidents`	read	Encoded-query search
`sn_incident_summary`	read	Aggregated counts by field
`sn_list_requested_items`	read	Service catalog requested items
`sn_add_work_note`	write	Appends a work note to an incident
`sn_assign_incident`	write	Sets an assignee

There is no generic shell and no HTTP passthrough — you cannot ask this agent to hit an arbitrary Table API path. And critically: no create, no close, no delete. An incident cannot be conjured or disappeared.

Where write tools belong

This is the part worth arguing about, so let me be direct about the position.

Read-only agents are easy to trust and frequently useless. “Here are your 25 open tickets” is genuinely helpful, but the manager’s next sentence is always “okay, assign the payroll one to someone and note that we’re on it.” If the agent can’t do that, you alt-tab into ServiceNow and the agent was a fancy report.

So the two writes exist. What makes them defensible is not that they’re small — it’s that they’re specific and reversible:

sn_add_work_note is append-only. It cannot overwrite history. Worst case, someone gets a spurious note on a ticket, and the note is signed by the agent’s identity.
sn_assign_incident sets one field. Wrong assignee is an annoyance, fixed by setting it again.

Compare that to what I deliberately did not build. A sn_close_incident destroys the audit story — a closed ticket stops getting looked at, which is exactly the outcome a prompt injection would want. A generic sn_table_patch is a shell with extra steps: any field, any table, including sys_user roles.

On top of that, the agent’s instructions require it to ask before writing. That’s a soft control — a model can be talked out of a soft control, which is precisely why it’s the third layer and not the only one. The hard controls are the tool catalog (there is no verb for “close”) and the ServiceNow account’s own role grants. The prompt-level “ask first” is there for the ordinary case where the manager and the agent are just working through a list, not for the adversarial one.

The general rule I’d apply to any agent touching a system of record: give it writes that a human can undo in one click, and nothing else.

The live run

Two questions through /api/a2a-sandboxes/kagent/servicenow.

Live kagent UI, 2026-08-16. Three tool calls: sn_whoami, sn_incident_summary, sn_list_incidents.

Q1 — “What IT tickets are open right now? Give me a manager summary by priority.”

Acting as System Administrator (admin) on dev203166.service-now.com. 25 active incidents on the current page:

Priority	Count	Manager view
P1 – Critical	13	Payroll server, DNS, wireless/network, VPN, Exchange, SAP apps
P2 – High	4	Network storage, SAP login, hotfix removal, VM launch
P3 – Moderate	5	Desk phone, email, service request, website JavaScript, SFA access
P5 – Planning	3	Weather report, replacement iPhone, Oregon mail server

Then the section that makes it a briefing rather than a dump — needs eyes / unassigned:

INC0007001 — P1 Critical, New, unassigned: employee payroll application server is down
INC0000046 — P3 Moderate, New, unassigned: can’t access SFA software
INC0000039 — P5 Planning, New, unassigned: trouble getting to Oregon mail server

An unassigned P1 on a payroll server is exactly the thing that falls through a queue view, and it’s the first thing the agent surfaced.

The honest bit: sn_incident_summary — the aggregate-count tool — failed on this run. The agent said so in the answer (“the formal summary count tool failed, so priority totals below are from the returned incident list only”) and derived the buckets from the rows it actually had. It didn’t present a computed total as an API total. Same principle as the GCP agent saying unavailable: tell me where the number came from.

Q2 — “What about the critical VPN and DNS ones?”

Incident	Short description	State	Assignee
INC0000015	VPN client won’t launch since the last software update	In Progress	Don Goodliffe
INC0000016	Rain is leaking on the main DNS server	In Progress	ITIL User

Rain leaking on the DNS server is stock ServiceNow demo data, and I love it too much to swap for something plausible.

Here’s the same session as a scroll:

One tell that this is a real sandboxed session: /api/a2a/kagent/servicenow 404s — there is no Agent CR — and the UI posts to /api/a2a-sandboxes/kagent/servicenow. The card wears a Sandbox: Agent Substrate badge.

Honest limits

Import the MCP image on the k3s node before the pod starts.
The Vault path must exist or the ExternalSecret stays unsynced.
No generic shell, no HTTP passthrough, no incident create / close / delete.
The two write tools exist and the agent must ask first — that’s a prompt-level control backed by a tool-level one, not a substitute for it.
sn_incident_summary failed on the live run. Priority totals came from the returned rows.
This is a personal developer instance with stock demo data, not production volume.
Never commit the ServiceNow username or password values.

The rest of the series

Five demos, one runtime — same pins, same Vault/ESO shape, same gVisor wall. Different blast radius each time.

AWS budget — Cost Explorer, least-privilege IAM, and the full build from pins to golden snapshot.
GCP budget — us-east1 capacity, and why the billing half honestly reports unavailable.
FortiGate 80F — my actual home firewall, 22 tools, and 3,007,844 hits on one policy.
F5 BIG-IP — 19 VIPs, 18 tool calls in one turn, and zero write tools.
Why secure sandbox substrates are the future — the argument behind all five.

Full demo — manifests, the FastMCP server, security notes, runbooks, and the live report: sebbycorp/kagent-agent-substrate-demos / service-now-sandbox-agent.

Asking My Home FortiGate Questions: A gVisor SandboxAgent on kagent

Mon, 17 Aug 2026 08:15:00 -0400

My AWS and GCP budget agents read cloud bills. This one reads something with considerably more consequence in my house: the firewall my family’s internet goes through.

fw-maniak-hq is a FortiGate 80F at 172.16.10.1. Real traffic, real policies, and — importantly — policies I wrote months ago and no longer remember the details of. That last part is what makes a home firewall a better test subject than a pristine lab box. The interesting questions aren’t “is it up,” they’re archaeological:

What is fw-maniak-hq running, and which WAN is up?

What’s the YouTube policy?

Live kagent UI, 2026-08-16. Isolated sandboxes, not plain Agents — kagent/fortigate on the grid.

Why a SandboxAgent and not a plain Agent

A normal kagent Agent is a Deployment: always on, container isolation, same as every other pod. Fine for a cluster helper.

This one holds a REST token for the device that gates my entire household’s network. The model gets a filesystem, memory, and a live network for the whole chat. Substrate puts that session in a gVisor actor on WorkerPool kagent-default:

Isolated sandbox. gVisor’s user-space kernel is the wall between the model session and my Viper/k3s host. Tools call FortiOS through the MCP pod; the REST token stays in Vault, not in the actor.
Idle chats snapshot (zstd) and free the worker. The next message restores the same session instead of booting a new container.
No always-on pod per conversation.
A golden snapshot you can resume.

Tradeoff on this lab: nested gVisor on dockerized k3s, and snapshots are in-cluster rustfs today (gs:// is a URI prefix only), not GCS.

Architecture

flowchart LR chat["Chat"] ui["kagent UI :30500"] a2a["A2A sandboxes"] actor["gVisor actor ateom-gvisor:v0.0.9"] rmcp["RemoteMCPServer fortigate-mcp"] mcp["fortigate-mcp :8084/mcp"] fg["FortiGate 80F 172.16.10.1"] vault["Vault secret/platform/fortigate"] eso["ESO"] chat --> ui --> a2a --> actor --> rmcp --> mcp --> fg vault --> eso --> mcp style actor fill:#FFF7D6,stroke:#E5341F,color:#17181C style mcp fill:#F1EFE9,stroke:#E5341F,color:#17181C

The REST token goes Vault → External Secrets Operator → the MCP pod. The actor calls tools over MCP and never holds the token. If a session gets confused by something it read, the credential is still on the far side of the sandbox wall.

Pins (do not bump)

Piece	Value
kagent OSS Helm + CRDs	`0.10.0-rc2`
Agent Substrate Helm + CRDs	`0.0.9`
Worker image	`ghcr.io/kagent-dev/substrate/ateom-gvisor:v0.0.9`
Pattern	Go declarative `SandboxAgent` + FastMCP + `RemoteMCPServer` + `ExternalSecret`
Model	`default-model-config` (gpt-5.5 via agentgateway)
Box	FortiGate 80F · `fw-maniak-hq` · `172.16.10.1`
FortiOS	v7.4.11 build 2878 · VDOM `root`

rc2 always writes ActorTemplate with spec.pauseImage and env[].valueFrom.secretKeyRef. Substrate 0.0.9 accepts that shape; 0.0.12 does not. Do not “upgrade to fix” a Ready=False agent — a pin mismatch is a CRD problem.

Build it

GitOps for this one lives in sebbycorp/k8s-viper (platform/kagent-ai/fortigate-*.yaml, images/fortigate-mcp/, docs/fortigate-agent.md) — the demo folder in the substrate repo is the live-run record. The shape is the same as the others:

FortiGate side. Create a REST API admin with a read-mostly profile, restricted to the trusted host that the MCP pod egresses from. Generate the token once.
Vault. Path secret/platform/fortigate. ESO syncs it into a Kubernetes Secret the MCP pod consumes. Git holds the mapping only.
Image. Build fortigate-mcp:dev, ctr images import onto the k3s node. Dockerized k3s can’t see host Docker images; imagePullPolicy: IfNotPresent is deliberate.
Apply the SandboxAgent, RemoteMCPServer, ExternalSecret, and skills ConfigMap.

Verified live on 2026-08-16, and this is the whole proof surface:

NAME READY ACCEPTED
sandboxagent.kagent.dev/fortigate True True

NAME PROTOCOL URL ACCEPTED
remotemcpserver.kagent.dev/fortigate-mcp STREAMABLE_HTTP http://fortigate-mcp.kagent:8084/mcp True

NAME READY STATUS RESTARTS AGE
pod/fortigate-mcp-745d4c9ff5-bxjvc 1/1 Running 0 19h

NAME CLASS
actortemplate.ate.dev/fortigate-b8bc65944f9bc4df gvisor
location: gs://ate-snapshots/kagent/fortigate
goldenSnapshot: gs://ate-snapshots/kagent/fortigate/2bcc7a8b-.../2026-08-16T02:45:18Z-R7ZXMSV6CEC2D4NVN4XEX5UDO3
phase: Ready

Ready=True with a goldenSnapshot means Substrate booted a golden actor, checkpointed it, and new chats restore from that image rather than cold-starting.

The tool catalog

This agent is the widest of the five demos — 22 tools, because a firewall has a lot of surfaces worth reading:

Reads. fg_system_status, fg_resource_usage, fg_list_interfaces, fg_interface_stats, fg_list_policies, fg_get_policy, fg_policy_stats, fg_list_addresses, fg_list_addrgrp, fg_list_services, fg_list_routes, fg_list_static_routes, fg_vpn_status, fg_dhcp_leases, fg_list_vips, fg_log_state, fg_current_admins.

Writes. fg_create_address, fg_update_address, fg_create_policy, fg_set_policy_status, fg_update_policy_comment.

And, more importantly, what is absent: there is no generic “run any FortiOS CLI” tool, no tmsh-equivalent, no config backup, no firmware operation, and no delete of anything. The write tools that exist are additive or single-field — create an address object, flip a policy’s enable flag, edit a comment. The agent has to ask before using any of them.

That’s the same position I took with the ServiceNow agent’s two write tools: give it writes a human can undo in one click, and nothing else. On a firewall the stakes are higher, so the “nothing else” list is longer — a fg_delete_policy on a home gateway is a self-inflicted outage waiting for a bad prompt.

The live run

Two questions through /api/a2a-sandboxes/kagent/fortigate, 2026-08-16 evening.

Live kagent UI, 2026-08-16. Three tool calls on Q1, four on Q2. The right-hand pane shows the full 22-tool catalog.

Q1 — “What is fw-maniak-hq running, and which WAN is up?” (~6.6s, tools: fg_system_status, fg_list_interfaces, fg_interface_stats)

FortiGate 80F / FGT80F, serial FGT80FTK22061709
FortiOS v7.4.11 build 2878, VDOM root
wan1: down, DHCP, IP 0.0.0.0
wan2: up, DHCP, IP 24.141.221.254/20

Six and a half seconds for hardware, firmware, and WAN failover state. Worth noting it correctly read 0.0.0.0 as down rather than reporting an interface that exists as an interface that works.

Q2 — “What’s the YouTube policy?” (~12.0s, tools: fg_list_policies, fg_get_policy ×2, fg_policy_stats)

ID	Name	Source	Schedule	Hits	Active sessions
8	`Allow-YouTube-Whitelist`	`Grp-YouTube-Allowed`	always	3,007,844	154
7	`Allow-YouTube-MasterBR-Night`	`YT-AppleTV-Master-122`	`Allow-YT-MasterBR-8pm-2am`	18,902	0

Both from corp → BELL_35 / wan2, action accept, NAT on, logging all. Policy 8 even carried its own documentation in the comment field: “Whitelist: these devices may use YouTube (and everything). Add devices to Grp-YouTube-Allowed.”

Then the answer I actually needed: there is no separate YouTube block policy in the returned list. The allow-list is doing the work by being narrow, not by pairing with an explicit deny — and the agent said that rather than assuming a deny must exist somewhere because the setup “looks like” a whitelist pattern.

What this demo is actually good at

Two things stand out after using it.

It reads intent, not just config. fg_policy_stats turning “policy 8” into 3,007,844 hits, 154 sessions right now is the difference between reading a rule and knowing whether the rule matters. Policy 7 has 18,902 lifetime hits and zero active sessions — its schedule window (8pm-2am) wasn’t open. Neither of those facts is in the policy definition.

It’s honest about what it can see. Twice in one session it declined to fill a gap: no YouTube deny policy in the compact list, and wan1 reported as down rather than glossed. Same principle as the GCP agent saying unavailable rather than inventing spend. On a firewall, a confidently hallucinated policy is worse than no answer — you’d go make a change based on a rule that doesn’t exist.

Honest limits

Import the MCP image on the k3s node before the pod starts.
The Vault path secret/platform/fortigate must exist or the ExternalSecret stays unsynced.
The kagent UI at http://172.16.10.135:30500/ is LAN-only. This isn’t published.
No generic FortiOS CLI tool. No delete, no config backup, no firmware operation.
Writes exist (fg_create_policy, fg_set_policy_status, …) and the agent must ask first — a prompt-level control backed by a tool-level one, not a replacement for it.
Snapshots are in-cluster rustfs. gs:// is a prefix only.
Never commit the FortiGate REST token or Vault secret values.

The rest of the series

Five demos, one runtime — same pins, same Vault/ESO shape, same gVisor wall. Different blast radius each time.

AWS budget — Cost Explorer, least-privilege IAM, and the full build from pins to golden snapshot.
GCP budget — us-east1 capacity, and why the billing half honestly reports unavailable.
ServiceNow triage — 25 incidents into a manager briefing, and where write tools belong.
F5 BIG-IP — 19 VIPs, 18 tool calls in one turn, and zero write tools.
Why secure sandbox substrates are the future — the argument behind all five.

Live-run record: sebbycorp/kagent-agent-substrate-demos / fortigate-sandbox-agent. Manifests and the MCP image are in sebbycorp/k8s-viper — see docs/fortigate-agent.md.

An F5 BIG-IP SandboxAgent on kagent: 19 VIPs, 2 Up, and 18 Tool Calls to Find Out Why

Mon, 17 Aug 2026 08:00:00 -0400

The other four agents in this series answer what is the state of things — AWS spend, GCP quota, open tickets, firewall policy. This one answers something different, and it’s the question I care about most on a load balancer:

Which VIPs are down, and why?

“Down” is a lookup. “Why” is a fan-out — you have to walk from each offline virtual server to its pool, then to that pool’s members, and read the reason string. On my lab BIG-IP that turned into 18 tool calls in a single turn, and the agent did the walk itself.

Live kagent UI, 2026-08-17. Isolated sandboxes, not plain Agents — kagent/f5-bigip on the grid.

Why a SandboxAgent and not a plain Agent

A normal kagent Agent is a Deployment: always on, container isolation. Fine for a cluster helper.

This one holds credentials for the box that fronts every service in my lab. The model gets a filesystem, memory, and a network for the whole chat. Substrate puts that session in a gVisor actor on WorkerPool kagent-default:

Isolated sandbox. gVisor’s user-space kernel sits between the model session and my Viper/k3s host. Tools call iControl REST through the MCP pod; the BIG-IP password stays in Vault, not in the actor.
Idle chats snapshot (zstd) and free the worker. The next message restores that session.
No always-on pod per conversation.
A golden snapshot you can resume.

Tradeoff on this lab: nested gVisor on dockerized k3s, and snapshots are in-cluster rustfs today (gs:// is a URI prefix only), not GCS.

Architecture

flowchart LR chat["Chat"] ui["kagent UI :30500"] a2a["A2A sandboxes"] actor["gVisor actor ateom-gvisor:v0.0.9"] rmcp["RemoteMCPServer f5-bigip-mcp"] mcp["f5-bigip-mcp :8084/mcp"] f5["BIG-IP 172.16.10.10"] vault["Vault secret/platform/f5-bigip"] eso["ESO"] chat --> ui --> a2a --> actor --> rmcp --> mcp --> f5 vault --> eso --> mcp style actor fill:#FFF7D6,stroke:#E5341F,color:#17181C style mcp fill:#F1EFE9,stroke:#E5341F,color:#17181C

Vault → ESO → the MCP pod. Not the actor. Same shape as the other four demos, and the reason is the same: the part of the system running the model and parsing untrusted output should never be the part holding the credential.

Pins (do not bump)

Piece	Value
kagent OSS Helm + CRDs	`0.10.0-rc2`
Agent Substrate Helm + CRDs	`0.0.9`
Worker image	`ghcr.io/kagent-dev/substrate/ateom-gvisor:v0.0.9`
Pattern	Go declarative `SandboxAgent` + FastMCP + `RemoteMCPServer` + `ExternalSecret`
Model	`default-model-config` (gpt-5.5 via agentgateway)
Box	F5 BIG-IP · `https://172.16.10.10` · LAN-only, self-signed

rc2 always writes ActorTemplate with spec.pauseImage and env[].valueFrom.secretKeyRef. Substrate 0.0.9 accepts that shape; 0.0.12 does not. A Ready=False agent after a version bump is a CRD pin problem, not a reason to bump further.

Build it

GitOps lives in sebbycorp/k8s-viper (platform/kagent-ai/f5-bigip-*.yaml, images/f5-bigip-mcp/, docs/f5-bigip-agent.md); the demo folder is the live-run record.

BIG-IP side. A dedicated account with a read-only role. No tmsh shell access.
Vault. Path secret/platform/f5-bigip, keys host, username, password. ESO syncs it to the MCP pod; git holds the mapping only.
Image. Build f5-bigip-mcp:dev and ctr images import it onto the k3s node.
Apply the SandboxAgent, RemoteMCPServer, and ExternalSecret.

Live on 2026-08-17 — the whole proof surface, with nothing sensitive in it:

NAME READY ACCEPTED
sandboxagent.kagent.dev/f5-bigip True True

NAME PROTOCOL URL ACCEPTED
remotemcpserver.kagent.dev/f5-bigip-mcp STREAMABLE_HTTP http://f5-bigip-mcp.kagent:8084/mcp True

NAME STORETYPE STORE STATUS READY
externalsecret.../f5-bigip-mcp ClusterSecretStore vault-backend SecretSynced True

NAME READY STATUS RESTARTS AGE
pod/f5-bigip-mcp-7f75b47b78-mdblb 1/1 Running 0 17m
image: f5-bigip-mcp:dev

NAME CLASS
actortemplate.ate.dev/f5-bigip-3adfcbf7c448a873 gvisor
location: gs://ate-snapshots/kagent/f5-bigip
goldenSnapshot: gs://ate-snapshots/kagent/f5-bigip/2b9f5b6a-.../2026-08-17T14:32:42Z-23PD7HXZ5JL7OO7RNHUWZ5YOWS
phase: Ready

Six tools, zero writes

Tool	What it reads
`f5_system`	Product, version, build
`f5_list_vips`	Virtual servers
`f5_vip_status`	One virtual server’s availability and enabled state
`f5_list_pools`	Pools
`f5_pool_status`	One pool’s availability, reason string, and member states
`f5_vip_brief`	Composed VIP → destination → pool → availability rollup

That’s it. No write tools exist at all — not create, not delete, not disable, not “modify this monitor.” There is also no generic “call any iControl path” tool, which is the important one: a passthrough would let anyone with chat access reach every endpoint the account can, and read-only-on-the-account would be the only remaining control.

This is a different call than I made for FortiGate and ServiceNow, where a few narrow, reversible writes earn their place. A load balancer is the wrong place to be casual: disabling the wrong VIP is an instant outage for every service behind it, and there’s no cheap undo when the thing you broke is the path your monitoring travels over. So this agent gets to observe and nothing else.

The live run

Two questions through /api/a2a-sandboxes/kagent/f5-bigip.

Live kagent UI, 2026-08-17. Two tool calls on Q1, 18 on Q2. The 17-row down table continues below the fold.

Q1 — “What is this BIG-IP running, and which VIPs are up?” (~12.6s, tools: f5_system, f5_vip_brief)

f5_system reached https://172.16.10.10 and returned ok: true — but the identity fields came back empty. Product null. Version null. Build null.

The agent reported exactly that. It did not substitute a TMOS version from anywhere, and it didn’t decide the box was unreachable either — it separated “the API answered” from “the answer was empty,” which are genuinely different failure modes on iControl.

Then f5_vip_brief returned 19 virtual servers, of which two were available:

VIP	Destination	Pool	Availability	Enabled
`k8s_iceman_argocd_vs`	`/Common/172.16.20.60:443`	`/Common/k8s_iceman_argocd_pool`	`available`	`true`
`k8s_iceman_kagent_vs`	`/Common/172.16.20.62:8080`	`/Common/k8s_iceman_kagent_pool`	`available`	`true`

The other 17: offline.

Q2 — “Which VIPs are down, and why (pool members)?” (~36.4s, tools: f5_system, f5_vip_brief, f5_pool_status ×16)

This is where a single-shot tool wouldn’t have helped. The agent walked each offline VIP to its pool and pulled the status individually — sixteen f5_pool_status calls — and every one came back the same way:

Pool availability offline, pool state enabled
Reason: “The children pool member(s) are down”
Members state: down, session: monitor-enabled

And the conclusion that actually matters: all 17 offline VIPs were still enabled. Nobody administratively disabled anything. The VIPs are fine; the backends are gone. On my lab that’s the expected story — those pools point at Talos and k3s node ports across clusters I’d shut down — but “config is fine, backends are dead” versus “someone disabled the VIP” is the entire first branch of a load balancer triage tree, and the agent got there on its own.

A sample of the 17, with the down members it named:

VIP	Destination	Pool	Down members
`agentgateway-oss`	`/Common/172.16.20.30:8080`	`/Common/agentgetway-oss`	`172.16.10.144:30344`, `.144:30513`, `.148:30344`, `.148:30513`
`k8s_iceman_vault_vs`	`/Common/172.16.20.61:8200`	`/Common/k8s_iceman_vault_pool`	`talos-cp:30820`, `talos-worker:30820`
`kagent-oss`	`/Common/172.16.20.36:80`	`/Common/kagent-oss`	`172.16.10.144:31438`, `.144:32002`, `.148:31438`, `.148:32002`
`vs_mcp_gateway`	`/Common/172.16.20.123:8090`	`/Common/pool_mcp_gateway`	`172.16.10.130:30168`, `.132:30168`, `.133:30168`, `.136:30168`
`webui-https`	`/Common/172.16.20.31:443`	`/Common/webui-oss`	`172.16.10.144:30694`, `172.16.10.148:30694`

Final tally: 2 available, 17 offline, 19 total.

(Yes, one of my pool names is agentgetway-oss. The agent reported the name as configured rather than tidying it up, which is the correct behavior and also mildly embarrassing.)

Why the fan-out is the interesting part

Everything above could have been a script. I want to be clear about that — for vip in $(list); do pool_status $vip; done is not hard to write.

What the agent added is that nobody decided in advance how many calls to make. The question “why are they down” doesn’t specify a depth. Two VIPs up and 17 down produced sixteen pool lookups; a different day produces a different number. The agent read the shape of the first answer and sized the second turn to fit, then collapsed 16 identical reason strings into one finding instead of pasting sixteen JSON blobs at me.

That’s the actual value proposition for a read-only diagnostic agent, and it’s why I’m comfortable with this one having no write tools whatsoever. The scarce skill in an outage isn’t changing things — it’s asking the next question. This agent asks the next question and stops.

Honest limits

Import the MCP image on the k3s node before the pod starts.
The Vault path secret/platform/f5-bigip must exist or the ExternalSecret stays unsynced.
The kagent UI at http://172.16.10.135:30500/ is LAN-only.
No generic “call any iControl path” tool, no tmsh, and no write tools — the agent cannot create, delete, disable, or change virtuals, pools, or monitors.
f5_system returned null product/version/build on this run. Don’t read a TMOS version into that gap; I didn’t.
The BIG-IP is LAN-only with a self-signed certificate.
Snapshots are in-cluster rustfs. gs:// is a prefix only.
Never commit the F5 password or Vault secret values.

The rest of the series

Five demos, one runtime — same pins, same Vault/ESO shape, same gVisor wall. Different blast radius each time.

AWS budget — Cost Explorer, least-privilege IAM, and the full build from pins to golden snapshot.
GCP budget — us-east1 capacity, and why the billing half honestly reports unavailable.
ServiceNow triage — 25 incidents into a manager briefing, and where write tools belong.
FortiGate 80F — my actual home firewall, 22 tools, and 3,007,844 hits on one policy.
Why secure sandbox substrates are the future — the argument behind all five.

Live-run record: sebbycorp/kagent-agent-substrate-demos / f5-bigip-sandbox-agent. Manifests and the MCP image are in sebbycorp/k8s-viper — see docs/f5-bigip-agent.md.

Why Secure Sandbox Substrates Are the Future of AI Agents

Sun, 16 Aug 2026 00:00:00 +0000

For a couple of years, “AI safety” in production mostly meant content safety: don’t let the model say the wrong thing, leak a prompt, or hallucinate a number. That framing is now dangerously incomplete. Modern agents don’t just emit text — they act. They get a filesystem, a chunk of memory, and a live network connection for the entire length of a conversation, and they use tools that reach into your cloud accounts, your firewalls, and your data.

The moment an agent can do things, the security question changes. It’s no longer “what can the model say?” It’s “what can the model’s session touch — and what happens when it (or a tool, or a prompt injection) misbehaves?”

This article argues that the answer is a secure sandbox substrate: a runtime where the unit of isolation is the agent session itself, not a long-lived pod. To keep it concrete, we’ll walk through a real one — an AWS budget assistant built on kagent Agent Substrate, running each executive conversation inside a gVisor sandbox, that answers a genuinely useful question with real numbers:

What’s our us-east-2 spend this month, and are we over capacity?

Live kagent UI, 2026-08-16. Three SandboxAgent cards — not plain Agents. Each conversation runs in its own gVisor actor.

The problem: agents are processes with hands

A traditional chatbot is a pure function — text in, text out. You can reason about its safety by reasoning about its words.

An agent is a process with hands. Give it an MCP server and it can call AWS Cost Explorer, describe your EC2 fleet, read your firewall config, or worse. To do that work it needs an execution environment: somewhere to run the model loop, hold conversation state, parse tool results, and keep working memory. That environment is real compute with real reach.

So now you have to worry about the things you’d worry about for any untrusted workload:

A prompt injection buried in a tool result that tries to make the agent do something you never asked for.
A compromised or buggy tool that the agent trusts.
Secret sprawl — API keys sitting in the same process as a model that’s actively being manipulated by untrusted input.
Blast radius — if the session breaks out, what else on the host does it reach?

The usual answer is “run the agent in a Kubernetes pod.” That helps, but it quietly makes two assumptions that don’t hold at agent scale: that a container boundary is a strong enough wall against untrusted code, and that an always-on pod per conversation is an acceptable way to spend compute. Both deserve a second look.

The idea: make the session the unit of isolation

A secure sandbox substrate flips the model. Instead of “one long-running deployment that handles many conversations,” you get “one strongly isolated, ephemeral sandbox per session, that can be checkpointed and restored on demand.”

In kagent’s Agent Substrate, that sandbox is a gVisor actor. gVisor is a user-space kernel: it intercepts the sandboxed workload’s syscalls and services them itself, so the agent session never talks directly to the host kernel. That’s a materially stronger wall than a stock container — exactly the wall you want between “a model being fed untrusted tool output” and “the node your cluster runs on.”

kagent expresses this as a first-class resource: a SandboxAgent, scheduled onto a WorkerPool, running a gVisor worker image. The difference from a normal agent is not cosmetic — it’s a different execution contract.

graph TB subgraph plain["Plain Agent — a Deployment"] P1["Always-on pod"] P2["Container isolation (shared host kernel)"] P3["One long-lived process serves every chat"] end subgraph sub["SandboxAgent — a Substrate actor"] S1["Ephemeral gVisor actor per session"] S2["User-space kernel wall between session & host"] S3["Idle → snapshot (zstd) → worker freed Next message → restore"] end style plain fill:#FFF7D6,stroke:#E5341F,color:#17181C style sub fill:#F1EFE9,stroke:#E5341F,color:#17181C style P1 fill:#FFFFFF,stroke:#17181C,color:#17181C style P2 fill:#FFFFFF,stroke:#17181C,color:#17181C style P3 fill:#FFFFFF,stroke:#17181C,color:#17181C style S1 fill:#FFFFFF,stroke:#17181C,color:#17181C style S2 fill:#FFFFFF,stroke:#17181C,color:#17181C style S3 fill:#FFFFFF,stroke:#17181C,color:#17181C

	Plain kagent `Agent`	`SandboxAgent` (Substrate)
Kubernetes shape	A Deployment — always-on pod	An actor on a `WorkerPool`, booted per session
Isolation	Container (shared host kernel)	gVisor user-space kernel
Idle cost	A pod sits running per conversation	Session snapshots and frees the worker
Resume	N/A — it never left	Restore from snapshot — same session, new message
Right fit for	Trusted cluster helpers	Sessions that touch money, secrets, or untrusted input

The rule of thumb from the demo’s own docs says it well: if you only need a Python container with boto3 and no snapshot lifecycle, a plain Deployment is fine. The instant the session talks to your AWS bill, you want the wall and the lifecycle.

The demo: an AWS budget agent that can’t go rogue

The aws-budget SandboxAgent is deliberately mundane in what it does and deliberately strict in what it can do. It answers executive finance-and-capacity questions for a single region (us-east-2) — and that’s the entire point. A boring capability, locked down properly, is exactly the shape most real enterprise agents should take.

Here it is answering the question live, with 10 of 10 tool calls succeeding and every number sourced from a real AWS API:

Live kagent UI, 2026-08-16. MTD $0.67, budget $4.13 / $100 (4.13% used), 0 running EC2 / ASG / RDS / EBS. The agent even reports “Cost Explorer rightsizing API denied; Compute Optimizer not enrolled” instead of inventing a recommendation.

Notice what the agent does when a permission is missing: it says the API was denied. It does not paper over the gap with a plausible-sounding guess. That honesty is a security property too — an agent that fabricates a “$0 spend” to be helpful is an agent you can’t trust with a budget.

The tools themselves are a curated, read-mostly catalog — eleven named tools, all Describe/Get/View/List. There is no generic “run any AWS CLI” tool, the single most important design decision in the whole demo:

Tool	AWS API (typical)
`aws_whoami`	`sts:GetCallerIdentity`
`aws_cost_month`	`ce:GetCostAndUsage`
`aws_cost_by_service`	`ce:GetCostAndUsage` grouped by service
`aws_budgets`	`budgets:ViewBudget`
`aws_ec2_capacity`	`ec2:DescribeInstances`
`aws_service_quotas`	`servicequotas:GetServiceQuota`
`aws_executive_brief`	composes the above

There is no iam:Create*, no ec2:Terminate*, no budgets:Delete*, no s3:* on your data. The capability boundary is enforced in three independent places — the tool code, the IAM policy, and the sandbox — so a jailbreak of any one layer still hits the next.

The anatomy: where the trust boundaries are

Here’s the full request path. Read it as a sequence of trust boundaries, because that’s what it is.

flowchart LR exec["Executive browser"] ui["kagent UI :30500"] sa["SandboxAgent aws-budget"] pool["WorkerPool kagent-default"] actor["gVisor actor (the session)"] rmcp["RemoteMCPServer aws-budget-mcp"] mcp["aws-budget-mcp :8084/mcp"] aws["AWS APIs us-east-2"] exec --> ui --> sa --> pool --> actor --> rmcp --> mcp --> aws style actor fill:#FFF7D6,stroke:#E5341F,color:#17181C style mcp fill:#F1EFE9,stroke:#E5341F,color:#17181C

The critical detail is where the AWS keys live — and where they don’t:

flowchart TB vault["Vault secret/platform/aws-budget"] eso["External Secrets Operator"] mcp["aws-budget-mcp pod non-root · read-only rootfs dropped caps · no SA token"] actor["gVisor actor (the model session)"] aws["AWS us-east-2"] vault --> eso --> mcp mcp -->|"STS / Cost Explorer / EC2"| aws actor -.->|"calls tools over MCP never sees the keys"| mcp style actor fill:#FFF7D6,stroke:#17181C,color:#17181C style mcp fill:#F1EFE9,stroke:#E5341F,color:#17181C style vault fill:#FFFFFF,stroke:#17181C,color:#17181C

Trace the secret. The AWS access key lives in Vault. External Secrets Operator syncs it into a Kubernetes Secret consumed only by the MCP pod — which is non-root, has a read-only root filesystem, drops Linux capabilities, and doesn’t even mount a service-account token. The gVisor actor — the part running the model and chewing on untrusted tool output — never holds the key at all. It calls a tool over MCP; the tool makes the AWS call; the credential stays on the far side of the sandbox wall.

Layer that against the isolation and you get genuine defense in depth:

gVisor keeps a manipulated session off the host kernel.
Vault + ESO keep the secret out of the session (and out of git — the repo has only the mapping, never the values).
Least-privilege IAM, region-conditioned to us-east-2, means even a leaked key can only read, and only in one region.
Read-mostly tools mean the agent has no verb for “destroy.”

No single control is novel. Composing all four around the session is what a substrate makes routine instead of heroic.

The economics: snapshot, free, restore

Security is only half the argument. The other half is that “always-on pod per conversation” simply doesn’t scale to a world where every employee has a dozen agents.

Substrate’s answer is a lifecycle borrowed from serverless, applied to stateful agent sessions:

sequenceDiagram participant U as User participant A as gVisor actor (session) participant S as Snapshot store U->>A: First message — boot from golden snapshot A-->>U: Answer (tools run inside the sandbox) Note over A: Chat goes idle… A->>S: Checkpoint session (zstd) → free the worker Note over A,S: No pod is running for this conversation now. U->>A: Next message S->>A: Restore the exact session state A-->>U: Continues as if it never left

When a conversation goes quiet, the actor’s memory and filesystem are checkpointed (compressed with zstd) to object storage and the worker is freed. The next message restores that exact session instead of cold-booting a fresh container and replaying context. You get the continuity of a long-lived process with the cost profile of something that only exists while it’s being used — plus a golden snapshot you can resume deterministically.

That’s the “substrate” in secure sandbox substrate: a fabric that boots, isolates, checkpoints, and restores agent sessions as a first-class operation.

Proof it’s real, not a slide

Everything above is running, not aspirational. The control plane reports the SandboxAgent Ready, the RemoteMCPServer Accepted, and the MCP pod 1/1 Running — captured live, with no secrets on screen:

Live capture, 2026-08-16 on Viper (k3s). Ready=True, MCP Accepted, pod 1/1 Running — and deliberately nothing sensitive on screen.

Here’s the same live A2A turn as a short reel — question in, ten tool calls, executive-shaped answer out:

One tell that these really are sandboxed sessions and not plain Agents: the classic /api/a2a/<ns>/<name> endpoint 404s (there is no Agent CR), and the UI talks to /api/a2a-sandboxes/kagent/aws-budget instead. The card even wears a Sandbox: Agent Substrate badge.

Being honest about the tradeoffs

Substrates are early, and the demo is refreshingly candid about it:

Nested gVisor is finicky. Running gVisor actors on dockerized k3s can hit runsc/seccomp//dev/kvm issues. That’s a worker-environment problem — not a reason to downgrade the agent to an unsandboxed Deployment.
Snapshots are local here. In this lab they land on in-cluster rustfs; the gs:// you see is a URI prefix, not live Google Cloud storage. Cluster-wide object storage is future work.
Version pinning matters. The demo pins kagent 0.10.0-rc2 with Substrate 0.0.9 on purpose — a mismatched CRD pairing (e.g. 0.0.12) makes the agent go Ready=False. Substrates are moving fast, so pin deliberately.

None of these undercut the thesis. They’re the normal roughness of a capability that’s ahead of its tooling — the same place containers were a decade ago.

Why this is the future

Zoom out. Three trends are converging:

Agents are proliferating. Not one assistant per company — dozens per team, each with tools that reach real systems.
Tools mean reach. Every MCP server an agent gains is new blast radius, and much of what an agent reads (tool output, fetched pages, tickets) is untrusted input that can carry an injection.
Sessions are stateful and bursty. People talk to agents in spikes, then walk away. Paying for an always-on pod per idle conversation is indefensible at scale.

A secure sandbox substrate is the architecture that answers all three at once. It makes strong per-session isolation the default instead of a special project. It makes secrets-outside-the-session the natural shape rather than a retrofit. And it makes ephemeral, resumable sessions an economic reality through snapshot and restore.

The aws-budget agent is a small, honest proof of the pattern: a genuinely useful assistant that reads your cloud bill, holds no keys, has no verb for destruction, runs behind a user-space kernel wall, and disappears when the conversation ends — ready to resume the instant you come back.

That combination — useful, contained, ephemeral — is what production AI agents will need to be. The substrate is how you get there without hand-building the isolation for every single one.

The full demo — manifests, the FastMCP server, IAM policy, runbooks, and the live report this article draws on — lives in sebbycorp/kagent-agent-substrate-demos / aws-sandbox-agent.

How To: Point DeepSeek Harness at standalone agentgateway

Sat, 15 Aug 2026 00:00:00 +0000

First, what is DeepSeek Harness?

Two days ago — August 13, 2026 — DeepSeek open-sourced DeepSeek Harness, and the README’s pitch is one line: everything is a plugin.

Not marketing-everything. Actually everything. Models, tools, skills, sessions, sandboxes, storage, the agent loop, scheduling, even the UI are all swappable from configuration. It’s MIT licensed, it’s an honest developer preview — the package reads 0.1.0-rc.5 and the README promises breaking changes — and you can be running it in one command:

npx @deepseek-ai/dsh web

What comes up is a local agent workspace: chat on one side, the agent’s trajectory on the other, sessions in a sidebar, a model picker at the bottom. It runs on your laptop, with filesystem and shell access, and it’s good.

The plugin that matters for this post is the one deciding where the tokens come from. Harness will talk to DeepSeek’s own models, or to any OpenAI-compatible endpoint you point it at. It asks for exactly one thing in return, on the Settings → Models screen: an API key.

That’s the moment worth pausing on. A two-day-old agent framework, running with shell access, asking for a provider key it will write to a file in my home directory — and once it has it, every call it makes is between those two parties. No one else sees the traffic. Nothing counts it. Nothing constrains it.

And what is agentgateway?

agentgateway is an open source connectivity data plane built specifically for agent traffic — LLM calls, MCP tool calls, and agent-to-agent. I’ve written about why that needs to be its own thing rather than a reverse proxy with extra steps.

For this post the relevant part is simple: it’s a single binary that speaks the OpenAI API dialect. Anything that can call OpenAI can call it instead — including Harness.

What we’re actually here to do

Put the gateway between Harness and the provider, and that one hop becomes the place where five things live that the harness has no opinion about:

Govern — one place that decides what this client is allowed to ask for. Virtual keys, prompt guards, rate limits, spend caps.
Secure — the real OPENAI_API_KEY stays inside the gateway process. Harness gets a token I invented, and it works just as well.
Route — Harness asks for a model by name; the gateway decides which provider and which exact model version actually serves it.
Log — every call, its status, and the model it resolved to, on one page. Without a proxy, this traffic is invisible.
Cost — tokens turned into dollars with a cost catalog, attributed per model and per user.

I’ll build it in that order, and the order matters: the first four are what make the thing work, and governance is what makes it safe to leave running. Secure, route, log, and cost come first, because you want to see traffic before you start refusing it. Then we turn govern on at the end and watch the front door get pickier.

By the end of it I asked Harness two deliberately boring questions, and the gateway handed me a receipt: 39 tokens, 2 calls, $0.0000072. A meaningless amount of money, and exactly the point — that number simply does not exist when an agent talks to a provider directly.

Here’s the whole build, including the two places I broke it.

👉 Everything below is in the repo: sebbycorp/deepseek-agw · Never run the gateway binary before? Start with agentgateway standalone locally

The shape of it

Three processes on one laptop, and the only interesting part is which one holds the secret.

flowchart LR dsh["dsh web :3080"] -->|"dummy token /v1"| agw["agentgateway :4002"] agw -->|"real OPENAI_API_KEY"| openai[OpenAI] agw --> admin["admin UI :14010"] agw --> jaeger["Jaeger :16686"]

Harness thinks it’s talking to OpenAI. It isn’t — it’s talking to a gateway on 127.0.0.1:4002 that speaks the same OpenAI-compatible dialect, accepts my made-up token, and quietly swaps in the real key on the way out. Because every request passes through that one process, it’s also the natural place to count tokens, add up dollars, and emit traces.

The rule I care about: the key is not in GitHub, not in Harness’s config directory, and not in the Harness process. It lives in one file on disk with mode 600, and it gets loaded into exactly one process.

Standing up the gateway

Node first, and this one is worth stating plainly because it cost me a confusing minute: current dsh does not run on Node 20. Use 24.

nvm install 24
nvm use 24
node -v

Then the gateway itself, pinned to 1.4.1 — I’m on the 1.4 OSS line:

curl -sL https://agentgateway.dev/install | bash -s -- --version v1.4.1
agentgateway --version

Next, the piece people skip. The gateway can count tokens on its own, but tokens aren’t money. Import the cost catalog once and it can turn those counts into dollars:

mkdir -p costs
agctl costs import --source models.dev --providers openai --out ./costs/catalog.json

If you want traces too, Jaeger all-in-one is one command. Skip it if you don’t care — nothing else depends on it:

docker run -d --name jaeger \
 -e COLLECTOR_OTLP_ENABLED=true \
 -p 16686:16686 -p 4317:4317 -p 4318:4318 \
 jaegertracing/all-in-one:latest

Now the config. The thing to notice is what isn’t in it — there’s no secret here, just a placeholder, which is why this file is safe to commit:

config:
 adminAddr: localhost:14010
 statsAddr: "[::]:14030"
 modelCatalog:
 - file: ./costs/catalog.json
 tracing:
 otlpEndpoint: http://localhost:4317
 randomSampling: true

gateways:
 default:
 port: 4002

llm:
 models:
 - name: "*"
 provider: openAI
 params:
 apiKey: "$OPENAI_API_KEY"

The real key goes in its own file, locked down, and git never sees it:

mkdir -p .secrets
umask 077
printf 'export OPENAI_API_KEY=sk-...\n' > .secrets/openai.env
chmod 600 .secrets/openai.env

And a small start script does the only clever thing in this whole setup — it sources that file into this process and nothing else, then refuses to start if the key didn’t make it:

#!/usr/bin/env bash
set -euo pipefail
SECRET="${AGW_SECRET_FILE:-$ROOT/.secrets/openai.env}"
[[ -f "$SECRET" ]] || { echo "missing $SECRET" >&2; exit 1; }
set -a; . "$SECRET"; set +a
[[ -n "${OPENAI_API_KEY:-}" ]] || { echo "OPENAI_API_KEY is empty" >&2; exit 1; }
exec agentgateway -f "$ROOT/agentgateway.yaml"

./start-agw.sh

Open http://127.0.0.1:14010/ui and make sure the admin UI is actually there before moving on. If the gateway isn’t running, everything in the next section fails in ways that look like Harness problems but aren’t.

Handing Harness a fake key

export GATEWAY_API_KEY=local-harness-not-openai
npx @deepseek-ai/dsh web

Read that token again: local-harness-not-openai. It isn’t a credential, it’s a label. The gateway is sitting on loopback and will accept it happily, then attach the real key on the way upstream.

Harness comes up on http://127.0.0.1:3080. Don’t send a message yet — mine failed, and I’ll get to why in a minute. Wire the provider first.

Go to Settings → Models. DeepSeek shows up red here because there’s no DeepSeek key on this laptop and I don’t want one. The custom agentgateway row is the green one:

Add a custom provider and fill in the form. This is the entire integration — a base URL and a fake token:

Field	Value
Provider ID	`agw`
Display name	`agentgateway (OpenAI via dummy token)`
API protocol	`openai-completions`
Base URL	`http://127.0.0.1:4002/v1`
`apiKeyEnv`	`GATEWAY_API_KEY`
Key value	`local-harness-not-openai` — not the real OpenAI key

Then add the models — gpt-4o and gpt-4o-mini are enough to start — and change max output tokens to 8192 while you’re in there. That number matters more than it looks, which is the first of my two mistakes:

Finally, New Session, and pick agentgateway / gpt-4o. The picker keeps the DeepSeek models in their own group above the custom provider, so once you know to look it’s hard to get wrong:

None of this has to happen in the browser, by the way. The UI is just writing $DSH_HOME/settings.yaml — usually ~/.dsh — so you can skip the clicking entirely:

llm-pi-ai:
 providers:
 agw:
 apiKeyEnv: GATEWAY_API_KEY
 api: openai-completions
 baseURL: http://127.0.0.1:4002/v1
 models:
 - id: gpt-4o
 maxTokens: 8192
 - id: gpt-4o-mini
 maxTokens: 8192

The token lands in a second file, $DSH_HOME/.credentials.yaml. Worth opening once just to reassure yourself: it holds GATEWAY_API_KEY, the fake one. OPENAI_API_KEY never appears anywhere in that directory.

Where I tripped

Two failures, ten minutes, neither one the gateway’s fault. I’m writing both down because I’d have saved the evening if someone else had.

The first turn failed on a missing key. I’d wired everything up, typed a message, and got an error about a DeepSeek credential — which made no sense, because I’d just spent twenty minutes making sure this thing pointed at OpenAI. The catch: my session had been created before I added the provider, so it was still sitting on deepseek-official, and there’s no DEEPSEEK_API_KEY on this box. My careful config was real; the session just wasn’t using it. New Session, pick agw, move on.

Then gpt-4o started returning 400. This one is sneakier. Harness defaults max output tokens to 32768, and gpt-4o caps completion tokens at 16384. Every request was asking for more headroom than the model allows, and OpenAI was rejecting it before a single token got generated. Set it to 16384 or 8192 and it clears up immediately.

That’s the entire troubleshooting section. Fix the provider, fix the ceiling, and it just works.

Two boring questions

I kept the test deliberately dull, because I wasn’t testing the model — I was testing the plumbing:

“What is 2+2? Reply with just the number.” → 4
“Name the capital of France in one word.” → Paris

Four and Paris. Not exactly a demo you’d put on a conference slide. But those two answers travelled from a UI holding a fake token, through a gateway holding the real one, out to OpenAI and back — which is the only thing I wanted to prove.

Now the part I actually built this for. Over in the admin UI, that conversation has a receipt: 39 tokens, 2 calls, $0.0000072, broken out by model, user, and provider.

The Logs page is the proof that the dummy token really did reach OpenAI and come back: two CHAT rows, both 200, model routing resolving gpt-4o-mini to gpt-4o-mini-2024-07-18 on provider openai. And no key anywhere on the page, which is the whole idea.

Scale that thought up. It’s a rounding error for two arithmetic questions, but it’s the same counter when an agent runs unattended for an hour. If you want the richer version of this view, I went deeper on it in the cost and tokenomics dashboard post.

Now make the door pickier

Everything so far gets the key out of the app and puts a number on the traffic. That’s four of the five. What’s still missing is the one that matters the moment this stops being a toy: nothing yet decides who may call, how much they may spend, or what may be sent.

Right now my gateway accepts local-harness-not-openai because it accepts anything. That’s fine for proving a path works. It’s not fine for a framework running with shell access on a laptop that also has my SSH keys on it.

So the config grows three policies. This is agentgateway-governed.yaml in the repo:

llm:
 policies:
 # GOVERN — who is allowed through this door, and under what name
 apiKey:
 mode: strict
 keys:
 - key: "$DSH_VIRTUAL_KEY"
 metadata:
 user: dsh
 tier: local

 # GOVERN — a ceiling the harness cannot talk its way past
 localRateLimit:
 - maxTokens: 200000
 tokensPerFill: 200000
 fillInterval: 3600s
 type: tokens

 models:
 - name: "*"
 provider: openAI
 params:
 apiKey: "$OPENAI_API_KEY"
 tokenize: true # count the prompt before OpenAI sees it

 # SECURE — an agent that can read files shouldn't be able to paste
 # a secret into a prompt
 guardrails:
 request:
 - regex:
 action: reject
 rules:
 - pattern: "api[_-]?key[=:]\\s*\\S+"
 - pattern: "sk-[A-Za-z0-9_-]{10,}"
 - builtin: email

Three things change, and each one is worth understanding rather than pasting.

mode: strict turns my invented token into a real one. Not real as in OpenAI — it still does nothing at api.openai.com. Real as in the gateway now recognizes it, and stamps user: dsh onto every call it authorizes. That’s the difference between a cost page that says “someone spent this” and one that says who. Once you care who paid, optional is just a hole.

The rate limit is the thing that lets me sleep. An agent loop that goes wrong doesn’t fail politely, it retries — and tokenize: true means the gateway counts the prompt before OpenAI does, so a runaway turn gets refused without spending anything. Worth knowing on standalone: that ceiling is gateway-wide, not per key. Per-key daily budgets need a remote rate-limit server, which is a different post.

The guardrails exist because of what Harness is. This is a framework with filesystem and shell access, driving a model that decides for itself what to include in a prompt. I don’t think it will paste my .env into a completion. I’d just rather it can’t.

Switching over is two edits. The virtual key goes in the same mode-600 file as the real one — pick any value, it’s yours:

printf 'export DSH_VIRTUAL_KEY=sk-dsh-local-harness\n' >> .secrets/openai.env
AGW_CONFIG=./agentgateway-governed.yaml ./start-agw.sh

And Harness sends that instead of the placeholder — one field in Settings → Models, or the environment before you start it:

export GATEWAY_API_KEY=sk-dsh-local-harness
npx @deepseek-ai/dsh web

That’s it. No plugin, no patch, nothing in ~/.dsh that knows any of this happened. The harness doesn’t learn about governance — it just keeps talking to /v1 and the door got pickier. That’s the whole argument for putting the control point outside the app.

One warning from experience: if you flip to strict and forget to update the token, every single call returns 401 and it looks exactly like the gateway is broken. And the email builtin in that guard is more eager than you’d expect — the first time a legitimate prompt gets a 400 content_policy_violation, that’s the rule that caught it, not a bug.

The same trick on a cluster

Nothing about the pattern changes when this moves off the laptop. Only the hiding place for the secret does — the mode-600 file becomes a Kubernetes Secret, and the local YAML becomes a handful of CRDs. The repo has them as applyable files under k8s/:

On my laptop	On a cluster
mode-600 `.secrets/openai.env`	`openai-secret` Secret, `Authorization` key
`llm.models` in the YAML	`AgentgatewayBackend` + a `/v1` `HTTPRoute`
`config.modelCatalog`	ConfigMap + `AgentgatewayParameters` on the Gateway
`config.tracing`	`AgentgatewayPolicy` → Jaeger
`http://127.0.0.1:4002/v1`	`http://127.0.0.1:8080/v1` through a port-forward

Harness doesn’t notice the difference. Same provider form, same fake token, one different URL:

kubectl port-forward -n agentgateway-system deploy/agentgateway-proxy 8080:80
export GATEWAY_API_KEY=local-harness-not-openai
npx @deepseek-ai/dsh web

One trap worth repeating, because it fails silently: the cost catalog has to be attached to the Gateway via AgentgatewayParameters. Hang it off the GatewayClass and it’s simply ignored — no error, just no dollar figures.

Being straight with you: the standalone path is the one I actually ran, and every screenshot above comes from it. The manifests mirror the 1.4.x CRDs, but I didn’t stand a cluster up for this one, so treat them as a starting point rather than something I’ve proven. For a cluster walkthrough I have run, see agentgateway quickstart on Kubernetes.

Why I keep doing this

The setup takes an evening. What I get back is the five things I opened with, and they’re all boring in the best way.

The real key lives in one file, loaded by one process, and ~/.dsh holds a virtual key that’s worthless anywhere else — secure. Rotating the real one means editing a file and restarting a single thing, instead of hunting through four config directories. Harness asks for gpt-4o and the gateway decides what actually answers — route. Every call is on a page with its status and resolved model — log. When someone asks what an agent cost to run, I have a number instead of a shrug — cost. And the door only opens for a key I issued, under a token ceiling, refusing prompts that look like secrets — govern.

That last one is the reason this is worth an evening rather than a curl. The first four make the setup pleasant. Governance is what makes it something you can leave running while you’re asleep, or hand to someone who isn’t you. And notice where all of it lives: five controls, none of them inside the agent framework. Harness never learned that any of this exists.

MCP isn’t wired into this one yet — same gateway, later. Given that Harness makes everything a plugin, including its tools, that’s the obvious next stop. But the shape keeps repeating. Whether it’s Claude Code and Codex, OpenMausBot, or DeepSeek Harness, the app stays on loopback with a fake token and the secret stays in the gateway. Every new toy just points at /v1.

Which means the next one takes ten minutes, not an evening. That’s really why I bother.

How To: Point Grok Build at standalone agentgateway

Sat, 15 Aug 2026 00:00:00 +0000

First, what is Grok Build?

Grok Build is xAI’s coding agent. It lives in the terminal, reads the repo, edits files, and shells out. Out of the box it talks straight to api.x.ai with a key sitting in ~/.grok or an env var.

That’s fine for one laptop. It’s ugly the moment you have more than one tool, more than one model, or you actually want to know what you spent.

The setting that matters for this post is a custom model in ~/.grok/config.toml. Grok Build will talk to any OpenAI-compatible endpoint you give it a base_url for. It asks for exactly one thing in return: a key.

That’s the moment worth pausing on. A coding agent with shell access, asking for a provider key it will keep on disk — and once it has it, every call is between those two parties. No one else sees the traffic. Nothing counts it. Nothing constrains it.

And what is agentgateway?

For this post the relevant part is simple: it’s a single binary that speaks the OpenAI /v1 dialect. Anything that can call OpenAI can call it instead — including Grok Build, and including curl. The client never holds the real key.

What we’re actually here to do

Put the gateway between Grok Build and xAI, and that one hop becomes the place where five things live that the agent has no opinion about:

Govern — one place that decides what this client is allowed to ask for. Virtual keys, prompt guards, rate limits, spend caps.
Secure — the real XAI_API_KEY stays inside the gateway process. Grok Build gets a token I invented, and it works just as well.
Route — Grok Build asks for grok-4-latest; the gateway decides which provider and which exact model version actually serves it. Mine resolved to grok-4.3.
Log — every call, its status, and the model it resolved to, on one page. Without a proxy, this traffic is invisible.
Cost — tokens turned into dollars with a cost catalog, attributed per model and per provider.

I’ll build the standalone path we actually ran on one box. No cluster required. Secure, route, log, and cost come first, because you want to see traffic before you start refusing it. Governance is the same door getting pickier later — I walked that part in the DeepSeek Harness post from today. This Saturday I wanted the hop working, and a receipt.

The older kind runbook — client holds the key, gateway just forwards Authorization — is a different path: docs/grok-passthrough-kind.md. Mentioning it so you don’t follow it by accident.

By the end of it I asked two deliberately boring questions, and the gateway handed me a receipt: $0.0044 / 623 tokens / 2 calls. A small amount of money, and exactly the point — that number simply does not exist when an agent talks to xAI directly.

Here’s the whole build, including the five places I tripped.

👉 Everything below is in the repo: sebbycorp/grok-build-with-agentgateway · Never run the gateway binary before? Start with agentgateway standalone locally

The shape of it

Two processes on one laptop, and the only interesting part is which one holds the secret.

flowchart LR grok["Grok Build / curl"] -->|"dummy token /v1"| agw["agentgateway :4003"] agw -->|"real XAI_API_KEY"| xai["api.x.ai"] agw --> admin["admin UI :14011"] agw --> jaeger["Jaeger :16686"]

Hop	What happens
1 · Client	Grok Build or curl hits `http://127.0.0.1:4003/v1` with `Authorization: Bearer local-grok-not-xai`.
2 · Gateway	agentgateway accepts the dummy token and injects `XAI_API_KEY` from process env. Tokens and cost get metered here.
3 · Upstream	Official xAI provider. Default upstream is `https://api.x.ai/v1`.
4 · Receipt	Admin UI on `:14011`. Metrics on `:14032`. Jaeger on `:16686` if you started it.

Grok Build thinks it’s talking to xAI. It isn’t — it’s talking to a gateway on 127.0.0.1:4003 that speaks the same OpenAI-compatible dialect, accepts my made-up token, and quietly swaps in the real key on the way out. Because every request passes through that one process, it’s also the natural place to count tokens, add up dollars, and emit traces.

The rule I care about: the key is not in GitHub, not in ~/.grok, and not in the Grok Build process. It lives in one file on disk with mode 600, and it gets loaded into exactly one process.

Ports are 4003 / 14011 on purpose, so this instance can sit next to an OpenAI gateway on 4002 / 14010 — the DeepSeek Harness / OpenMausBot lab. Same laptop, two providers, two doors.

Standing up the gateway

Pinned to 1.4.1 — I’m on the 1.4 OSS line:

curl -sL https://agentgateway.dev/install | bash -s -- --version v1.4.1
agentgateway --version

Next, the piece people skip. The gateway can count tokens on its own, but tokens aren’t money. Import the cost catalog once and it can turn those counts into dollars:

mkdir -p costs
agctl costs import --source models.dev --providers xai --out ./costs/catalog.json

If you want traces too, Jaeger all-in-one is one command. Skip it if you don’t care — nothing else depends on it:

docker run -d --name jaeger \
 -e COLLECTOR_OTLP_ENABLED=true \
 -p 16686:16686 -p 4317:4317 -p 4318:4318 \
 jaegertracing/all-in-one:latest

Now the config. The thing to notice is what isn’t in it — there’s no secret here, just a placeholder, which is why this file is safe to commit. Standalone has a first-class provider: xai. Default upstream is https://api.x.ai/v1.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
config:
 adminAddr: localhost:14011
 statsAddr: "[::]:14032"
 readinessAddr: "[::]:14033"
 database:
 url: "sqlite://./data.db?mode=rwc"
 modelCatalog:
 - file: ./costs/catalog.json
 tracing:
 otlpEndpoint: http://localhost:4317
 randomSampling: true

gateways:
 default:
 port: 4003

llm:
 models:
 - name: "*"
 provider: xai
 params:
 apiKey: "$XAI_API_KEY"

The sqlite URL is what makes Analytics and Logs actually store rows. Skip it and the admin UI looks empty even when the calls succeeded.

The real key goes in its own file, locked down, and git never sees it:

mkdir -p .secrets
umask 077
printf 'export XAI_API_KEY=xai-...\n' > .secrets/xai.env
chmod 600 .secrets/xai.env

And a small start script does the only clever thing in this whole setup — it sources that file into this process and nothing else, then refuses to start if the key didn’t make it:

#!/usr/bin/env bash
set -euo pipefail
ROOT="$(cd "$(dirname "$0")" && pwd)"
cd "$ROOT"
SECRET="${AGW_SECRET_FILE:-$ROOT/.secrets/xai.env}"
[[ -f "$SECRET" ]] || { echo "missing $SECRET" >&2; exit 1; }
set -a; . "$SECRET"; set +a
[[ -n "${XAI_API_KEY:-}" ]] || { echo "XAI_API_KEY is empty after sourcing $SECRET" >&2; exit 1; }
exec agentgateway -f "$ROOT/agentgateway.yaml"

./start-agw.sh

Open http://127.0.0.1:14011/ui and make sure the admin UI is actually there before moving on. If the gateway isn’t running, everything in the next section fails in ways that look like Grok problems but aren’t.

Two boring questions

I kept the test deliberately dull, because I wasn’t testing the model — I was testing the plumbing. Dummy inbound token. Real key stays on the gateway.

curl -sS http://127.0.0.1:4003/v1/chat/completions \
 -H "Content-Type: application/json" \
 -H "Authorization: Bearer local-grok-not-xai" \
 -d '{
 "model": "grok-4-latest",
 "messages": [
 {"role": "user", "content": "What is 2+2? Reply with just the number."}
 ]
 }' | jq

Read that token again: local-grok-not-xai. It isn’t a credential, it’s a label. The gateway is sitting on loopback and will accept it happily, then attach the real key on the way upstream.

Then a second short turn so Analytics has more than one row:

curl -sS http://127.0.0.1:4003/v1/chat/completions \
 -H "Content-Type: application/json" \
 -H "Authorization: Bearer local-grok-not-xai" \
 -d '{
 "model": "grok-4-latest",
 "messages": [
 {"role": "user", "content": "Name the capital of France in one word."}
 ]
 }' | jq

This box: both returned 200. grok-4-latest routed to grok-4.3. Answers were 4 and Paris.

“What is 2+2? Reply with just the number.” → 4
“Name the capital of France in one word.” → Paris

Four and Paris. Not exactly a demo you’d put on a conference slide. But those two answers travelled from a client holding a fake token, through a gateway holding the real one, out to xAI and back — which is the only thing I wanted to prove.

Official agentgateway docs mention grok-2-latest. This box used grok-4-latest. If a model 404s, list what your account actually has through the same listener:

curl -sS http://127.0.0.1:4003/v1/models \
 -H "Authorization: Bearer local-grok-not-xai" | jq

A receipt

Now the part I actually built this for. Over in the admin UI, that conversation has a receipt: $0.0044 / 623 tokens / 2 calls.

The Logs page is the proof that the dummy token really did reach xAI and come back: two CHAT rows, both 200, model routing resolving grok-4-latest to grok-4.3 on provider xai. And no key anywhere on those pages, which is the whole idea.

Scale that thought up. It’s a small bill for two short questions, but it’s the same counter when an agent runs unattended for an hour. If you want the richer version of this view, I went deeper on it in the cost and tokenomics dashboard post.

Point Grok Build at the same door

Grok Build reads ~/.grok/config.toml. Add a custom model that talks to the gateway with the dummy token, not XAI_API_KEY.

[model.agw]
model = "grok-4-latest"
base_url = "http://127.0.0.1:4003/v1"
env_key = "GATEWAY_API_KEY"

[models]
default = "agw"

export GATEWAY_API_KEY=local-grok-not-xai
grok

Or grok -m agw. env_key is the name of an env var Grok Build reads. That var is the dummy. The real xAI key never enters ~/.grok. Switch models in the TUI with /model.

Where I tripped

Five failures, all of them mine, none of them the gateway’s fault. I’m writing them down because I’d have saved the evening if someone else had.

start-agw.sh refused to start. I’d typed the YAML, I’d typed the curls, and the launcher said missing .secrets/xai.env. The script is doing the only clever thing in this setup on purpose: it will not bring the process up if the mode-600 file is missing or if XAI_API_KEY is empty after sourcing it. Redo the 600 file. Do not put the key in the YAML.

Then the curls came back 401 from upstream. Same class of mistake, later in the path. The dummy token was fine — local-grok-not-xai is a label, not a credential. The real key was empty or wrong in the process env. Check the 600 file. Still do not put the key in the YAML.

Then a model 404. Official agentgateway docs mention grok-2-latest. This box used grok-4-latest. If the id isn’t on your xAI account, GET /v1/models through the same listener and pick one you have.

Analytics and Logs stayed empty. The calls had returned 200. The admin UI looked like nothing had happened. Two causes, both boring: the client wasn’t actually hitting :4003, or config.database was missing. The sample YAML includes sqlite://./data.db?mode=rwc for a reason. Recheck the base URL before you rebuild the catalog.

Port already in use. Another agentgateway was already sitting on 4002 / 14010 — the OpenAI door from the DeepSeek / OpenMausBot lab. This config uses 4003 / 14011 on purpose so they can share a laptop. If you copy the OpenAI YAML and forget to change the ports, the second process loses.

That’s the entire troubleshooting section. Fix the file, fix the key, pick a model you have, keep the sqlite URL, stay off the other door.

The same trick on a cluster

Being straight with you: the standalone path is the one I actually ran, and every screenshot above comes from it. The manifests exist. I didn’t stand a cluster up for this one, so treat them as a starting point rather than something I’ve proven. For a cluster walkthrough I have run, see proxying all your LLM traffic through agentgateway with Grok Build and agentgateway quickstart on Kubernetes.

The older kind passthrough — client holds the key — is still docs/grok-passthrough-kind.md. Different runbook. Not this path.

MCP isn’t wired into this one yet. Same gateway, later.

Why I keep doing this

The setup takes an evening. What I get back is the five things I opened with, and they’re all boring in the best way.

Without the gateway	With this path
Grok Build talks straight to `api.x.ai` with a key in `~/.grok`	Dummy token in `~/.grok`. Real `XAI_API_KEY` lives in one process.
Nothing decides who may call, or what they may send	The hop is where govern lives — virtual keys, guards, ceilings — when you want the door pickier
`grok-4-latest` is whatever xAI serves today	The gateway routes it. Mine landed on `grok-4.3`.
The traffic is invisible	Two `CHAT` / `200` rows on Logs, provider `xai`, no key on the page
“What did this cost?” shrugged off	$0.0044 / 623 tokens / 2 calls on Analytics

The real key lives in one file, loaded by one process, and ~/.grok holds a dummy token that’s worthless anywhere else — secure. Rotating the real one means editing a file and restarting a single thing. Grok Build asks for grok-4-latest and the gateway decides what actually answers — route. Every call is on a page with its status and resolved model — log. When someone asks what an agent cost to run, I have a number instead of a shrug — cost. And the door can get pickier later without the agent learning that any of this exists — govern.

That last one is the reason this is worth an evening rather than a curl. The first four make the setup pleasant. Governance is what makes it something you can leave running while you’re asleep, or hand to someone who isn’t you. I flipped that on for DeepSeek Harness today. The agent never learned that any of it happened.

And notice where all of it lives: five controls, none of them inside Grok Build. The TUI never learned that any of this exists.

The shape keeps repeating. Whether it’s Claude Code and Codex, OpenMausBot, DeepSeek Harness, or Grok Build, the app stays on loopback with a fake token and the secret stays in the gateway. Every new toy just points at /v1.

Which means the next one takes ten minutes, not an evening. That’s really why I bother.

How To: Point OpenMausBot at standalone agentgateway

Sat, 15 Aug 2026 00:00:00 +0000

I sat down on Saturday with OpenMausBot because I wanted a chat app that felt like texting a teammate, not another terminal. I named the bot Indigo, picked GPT-5.6 Sol, and asked it to say hi in five words.

It did: Hi, I’m Indigo, your bot.

The part I loved — and the part that made me reach for agentgateway — is that OpenMausBot never called OpenAI. It spawned codex on my laptop. That’s charming until you want a real key, a guardrail, a rate limit, or a cost number that isn’t “trust me, I watched the terminal.”

So I did what I always do. Same pattern as Claude Code / Codex and Claude Desktop: the chat app stays on loopback, the secret and the policy live in the gateway.

This is that Saturday, written down so you can replay it.

👉 milind-soni/OpenMausBot · Official recipe: Codex → agentgateway · If you haven’t run the binary yet: standalone locally

Architecture

Think of it as three people in a room. OpenMausBot is the friendly one you talk to. Codex does the work. agentgateway is the grown-up who holds the wallet.

Hop	What happens
1 · Chat	I talk to Indigo at `http://127.0.0.1:5199`. The UI never sees a provider key — and that’s how I want it.
2 · Spawn	The harness starts `codex`. The Codex driver deletes `OPENAI_API_KEY` from the child env on purpose, so a leaked key can’t quietly flip billing to pay-as-you-go.
3 · Codex	Codex 0.147.0 uses `model_provider = "agentgateway"` and `base_url = "http://127.0.0.1:4000/v1"`.
4 · Gateway	Virtual key, regex guard, token budget, cost catalog — then the real OpenAI key goes upstream.

Codex calls {base}/v1/responses on the standalone LLM listener. Official docs were tested against codex-cli 0.144.4. I was on 0.147.0. OpenMausBot’s default Codex model in this build is GPT-5.6 Sol.

I tried Claude first. It face-planted (exit 127). Codex is the path that said hello.

Why OpenMausBot never holds the key

This is the bit that made me smile when I read the source. Before it even starts the CLI, OpenMausBot strips the provider secrets:

Codex deletes OPENAI_API_KEY
Claude deletes ANTHROPIC_API_KEY (and the Claude Code identity vars)

You can paste a key into the OpenMausBot process and the bot still won’t send it. The CLI either uses its own login, or it talks to whatever base_url you gave it.

That’s my invitation. Standalone agentgateway is that base_url. One binary on my desk held the real OPENAI_API_KEY, a virtual key tagged user: openmausbot / tier: live, the regex guards, the rate limit, and the cost catalog. Indigo didn’t need to know any of that.

What was actually on my desk

Thing	Saturday’s value
OpenMausBot UI	`http://127.0.0.1:5199`
Harness	`127.0.0.1:8799`
Bot	Indigo
Model picker	GPT-5.6 Sol
Codex	`0.147.0` · `wire_api = "responses"`
agentgateway LLM	`http://127.0.0.1:4000/v1`
Admin UI	`http://127.0.0.1:15000/ui/`
Virtual key metadata	`user: openmausbot` · `tier: live`
Gateway Overview	LLM Enabled · 1 model · Port 4000
The prompt I used	`say hi in five words.`
What came back	`Hi, I'm Indigo, your bot.`

Sanity check whenever something feels off:

agentgateway --version
codex --version

Once apiKey.mode is strict, a call with no key should 401. A call with the virtual key should reach OpenAI. That’s the whole handshake.

Standalone agentgateway

I installed the binary and wrote a config that does the four jobs I actually wanted that afternoon: talk to OpenAI, know who called, refuse sloppy secrets in the prompt, and put a number on the turn.

curl -sL https://agentgateway.dev/install | bash
mkdir -p ~/agw-openmausbot/costs
cd ~/agw-openmausbot
agctl costs import --source models.dev --providers openai --out ./costs/catalog.json

cat > config.yaml << 'EOF'
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
config:
 modelCatalog:
 - file: ./costs/catalog.json

llm:
 policies:
 apiKey:
 mode: strict
 keys:
 - key: "$OPENMAUSBOT_VIRTUAL_KEY"
 metadata:
 user: openmausbot
 tier: live
 localRateLimit:
 - maxTokens: 200000
 tokensPerFill: 200000
 fillInterval: 3600s
 type: tokens
 models:
 - name: "*"
 provider: openAI
 params:
 apiKey: "$OPENAI_API_KEY"
 tokenize: true
 guardrails:
 request:
 - regex:
 action: reject
 rules:
 - pattern: "api[_-]?key[=:]\\s*\\S+"
 - pattern: "sk-[A-Za-z0-9_-]{10,}"
 - builtin: email
 rejection:
 status: 400
 headers:
 set:
 content-type: "application/json"
 body: |
 {
 "error": {
 "message": "Request rejected: sensitive content",
 "type": "invalid_request_error",
 "code": "content_policy_violation"
 }
 }
EOF

export OPENAI_API_KEY='sk-...' # gateway process only
export OPENMAUSBOT_VIRTUAL_KEY='sk-omb-...' # what Codex presents; not an OpenAI key
agentgateway -f config.yaml

Port	What I used it for
4000	The front door Codex hits (`/v1/responses`, `/v1/chat/completions`, `/v1/models`)
15000	The Admin UI I left open in a tab — http://127.0.0.1:15000/ui/
15020	Stats, if you like watching tokens (`agentgateway_gen_ai_client_token_usage`)

Leave it running. My Gateway Overview said LLM Enabled, 1 model, Port 4000. I didn’t turn on MCP or Traffic. I didn’t need them to say hello.

Keep the model name as *. Codex sends gpt-5.6-sol itself. If you pin the gateway to one model, you’ll spend an hour wondering why Indigo looks confused.

Virtual API key

I didn’t want Codex holding the real OpenAI key. I wanted a house key — something that says “this is OpenMausBot” when I look at cost later.

LLM → Virtual API Keys → + New key. I tagged mine like this:

Field	What I used
Name	I left it `Unnamed key`. You can be nicer.
`user`	`openmausbot`
`tier`	`live`

Copy it once. The UI masks it (sk-omb-…). That value opens your gateway. It does nothing at OpenAI.

A few things I wish someone had said out loud:

Go strict once you care who paid.
optional is fine for the first curl. The moment you want user: openmausbot on the cost page, optional is just a hole.

Don’t name the virtual key OPENAI_API_KEY.
OpenMausBot deletes that name before Codex starts. I put it in OPENMAUSBOT_VIRTUAL_KEY and told Codex via env_key. The harness can inherit it. Codex still sees it after spawn.

Client Setup is a printer, not a wizard.
Admin UI → Client Setup will copy the Codex snippet from config you already wrote. It will not invent a route, a model, or a key. I still like it — less typing, fewer typos.

Guardrails, rate limits, cost

This is why I didn’t just export a key and call it a day. The YAML above is what I actually ran:

Control	Why I put it there
Regex `reject`	I don’t want a prompt that looks like `api_key=…`, `sk-…`, or an email to leave the laptop (regex filters)
`localRateLimit` `type: tokens`	200k tokens an hour — enough for a Saturday, not enough for a runaway loop
`tokenize: true`	Count the prompt before OpenAI sees it, so a fat turn can 429 without spending
`modelCatalog`	So each Indigo turn has a dollar number (`agw.ai.usage.cost.total`)

On standalone, that rate limit is for the whole gateway. If you later want per-key daily budgets, you’ll want a remote rate-limit server — budget limits and my hard spend limits post. Cost catalog walkthrough: standalone dashboard. If you want a heavier scanner in front of the same door: F5 guardrails.

None of that YAML lives in OpenMausBot. Indigo just chats. I like that.

Codex → `:4000/v1`

This is the official Codex recipe. I wrote a profile, then made it the default so the CLI OpenMausBot spawns would pick it up without me thinking about it.

mkdir -p ~/.codex
cat > ~/.codex/agentgateway.config.toml <<'EOF'
model_provider = "agentgateway"

[model_providers.agentgateway]
name = "OpenAI via agentgateway"
base_url = "http://127.0.0.1:4000/v1"
wire_api = "responses"
env_key = "OPENMAUSBOT_VIRTUAL_KEY"
EOF

cp ~/.codex/config.toml ~/.codex/config.toml.bak 2>/dev/null || true
cp ~/.codex/agentgateway.config.toml ~/.codex/config.toml

If you’d rather not touch the default file:

codex -c 'model_provider="agentgateway"' \
 -c 'model_providers.agentgateway.name="OpenAI via agentgateway"' \
 -c 'model_providers.agentgateway.base_url="http://127.0.0.1:4000/v1"' \
 -c 'model_providers.agentgateway.wire_api="responses"' \
 -c 'model_providers.agentgateway.env_key="OPENMAUSBOT_VIRTUAL_KEY"'

The three gotchas that ate my morning:

wire_api = "responses".
Codex speaks /v1/responses, not only chat completions. Miss this and you’ll stare at a quiet gateway log while the CLI looks busy.

env_key cannot be OPENAI_API_KEY.
That name is gone by the time Codex starts. OPENMAUSBOT_VIRTUAL_KEY survives.

Codex also knocks on /v1/models.
Until agentgateway#1462 grows a model list, you may get a metadata warning. Ignore it. /v1/responses still works.

I always smoke-test the CLI before I open the pretty chat UI. Saves a lot of “is it the bot or the gateway?”

export OPENMAUSBOT_VIRTUAL_KEY='sk-omb-...'
codex --profile agentgateway "Hello"

You want a 200 on /v1/responses in the agentgateway log (endpoint=api.openai.com:443, gen_ai.provider.name=openai). If Codex logs into ChatGPT instead, the profile didn’t load. Coffee, then check ~/.codex/config.toml.

Talking to Indigo

Desktop builds embed the harness. From source I run both:

git clone https://github.com/milind-soni/OpenMausBot && cd OpenMausBot
pnpm install
export OPENMAUSBOT_VIRTUAL_KEY='sk-omb-...' # virtual key only
pnpm dev:server # 127.0.0.1:8799
pnpm dev # http://127.0.0.1:5199

Then, in the browser:

Open http://127.0.0.1:5199
Click Indigo
Model picker → GPT-5.6 Sol
Type say hi in five words.

I picked the categories it offered (Work & projects — I was not in a “life admin” mood). Claude failed first, which I’ll get to. Then Codex came back through the gateway and I grinned at five words.

Hi, I’m Indigo, your bot.

The Claude face-plant

The first bubble in that thread was a red box. I left it in the screenshot on purpose. Saturday labs should be honest.

claude exited 127 before result: openmausbot-agw wrapper: real claude CLI not installed
ANTHROPIC_BASE_URL=http://127.0.0.1:4000 (agentgateway)

127 means “I can’t find that command.” A wrapper was already pointing Claude at agentgateway — the redirect was right — but I hadn’t actually installed claude. OpenMausBot will not invent an Anthropic client because you wished for one.

If you want that path later, install the CLI and keep ANTHROPIC_API_KEY on the gateway. The Claude driver deletes it from the child the same way Codex deletes OPENAI_API_KEY.

# after `claude` is actually installed
export ANTHROPIC_BASE_URL='http://127.0.0.1:4000'

I stayed on Codex. Five words were enough for the day.

Why bother

Without the gateway	How Saturday felt with it
OpenAI key on the laptop, or worse, in the bot	The key stayed in the agentgateway process. I slept better.
Codex talking to OpenAI with no shared rules	Virtual key, regex guards, a rate limit I can change without touching Indigo
“What did this cost?” shrugged off	Cost catalog + `user: openmausbot` on the turn
Every new chat app grows another secret	One `:4000` door. Next weekend’s toy can use it too.

If you already send Claude Code or Codex through agentgateway, this is the same idea with a nicer living room: localhost, /v1, a bot named Indigo instead of a blinking cursor.

How To: Point Claude Desktop at agentgateway with Entra SSO

Wed, 12 Aug 2026 00:00:00 +0000

Claude Desktop is great until the Anthropic key lives on every laptop and nobody can answer who called which model, when, or for how much.

The pattern I want is the same one I use for Claude Code / Codex and MCP through agentgateway: SSO at the edge, provider secret in the cluster, one place for policy and cost.

This guide is the Claude Desktop path running in my lab:

👉 sebbycorp/k8s-goose · Live page: goose.maniak.ai/claude-desktop.html · Deep dive: docs/claude-desktop-gateway.md

Architecture

Claude Desktop authenticates with Entra (PKCE), sends an ID token as Bearer, the gateway validates it Strict against Entra JWKS, then injects the Anthropic key upstream.

Hop	What happens
1 · PKCE	Entra Interactive public client (`agw-claude-desktop`)
2 · Bearer	ID token, not access token
3 · Gateway	JWT Strict against `entra-jwks`
4 · Upstream	Anthropic via `anthropic-secret`

Path is / on this dedicated Gateway so Claude Desktop can call {base}/v1/messages. The open Anthropic path for kagent stays on anthropic-claude-gateway (different NodePort / path).

Why a loopback proxy

Claude Desktop allows plain HTTP only on loopback. Direct lab HTTPS into Electron often dies with ERR_CERT_AUTHORITY_INVALID even when browsers can be taught to trust the cert.

So the recommended lab path is:

Run a tiny TCP proxy on your laptop
Point Claude Desktop at http://127.0.0.1:18789/
Let the proxy forward to the cluster HTTP NodePort

From k8s-goose:

python3 scripts/claude-desktop-lab-proxy.py

Leave that process running.

Thing	Lab value
Service	`claude-desktop-gateway` · ns `agentgateway-system`
Claude Desktop base URL	`http://127.0.0.1:18789/`
Messages path	`http://127.0.0.1:18789/v1/messages`
Proxy upstream (HTTP NodePort)	`172.16.10.155:31938` · Service `80:31938/TCP`
HTTPS NodePort (optional / advanced)	`https://172.16.10.155:31211/` · often rejected by Electron

Confirm NodePorts anytime:

kubectl --context maniak-goose -n agentgateway-system get svc claude-desktop-gateway -o wide

Anonymous calls should 401/403. A valid Entra ID token should reach Anthropic.

Entra app

Public client already registered for Claude Desktop’s loopback PKCE flow.

Field	Value
App name	`agw-claude-desktop`
Tenant ID	`8635e970-2205-4189-bc77-77519ff5064f`
Client ID	`adf4a4f8-45a4-4bda-a7e2-35f39b1db59d`
Issuer	`https://login.microsoftonline.com/8635e970-2205-4189-bc77-77519ff5064f/v2.0`
Platform	Mobile and desktop applications (not Web)
Redirect URI	`http://127.0.0.1/callback`
Public client flows	Yes (no client secret · PKCE)

Critical Entra callouts

Bearer must be an ID token.
Default Claude Desktop mode and this gateway’s aud = the app’s client ID. Access tokens have a different audience story and fail Strict JWT.

Redirect must include /callback.
Register http://127.0.0.1/callback. http://127.0.0.1 alone fails with AADSTS50011. The port is wildcarded.

Platform = Mobile and desktop — not Web.
Entra rejects the loopback PKCE callback on the Web platform. Public client, no secret.

Include offline_access so users are not re-prompted every ~1h when using ID tokens.

Claude Desktop developer config

Help → Troubleshooting → Enable Developer Mode
Developer → Configure Third-Party Inference…
Fully quit and relaunch after edits if the UI is sticky

Start the proxy first, then set:

Field	Value
Connection / Inference provider	Gateway
Gateway base URL	`http://127.0.0.1:18789/`
Credential kind	Interactive sign-in
Client ID	`adf4a4f8-45a4-4bda-a7e2-35f39b1db59d`
Issuer URL	`https://login.microsoftonline.com/8635e970-2205-4189-bc77-77519ff5064f/v2.0`
Bearer token	ID token (not Access token)
Scopes	`openid email profile offline_access`
Model discovery	On → `http://127.0.0.1:18789/v1/models`

Walkthrough expectation: Entra consent → signed in as your org profile → model discovery fills the picker → a trivial chat (whats 2+2) comes back through the gateway footer.

Screenshots from a working lab session live under assets/claude-desktop/ in k8s-goose.

Cluster resources (GitOps)

Everything is under config/ via the Argo app agentgateway-config. Push + sync is enough for the laptop proxy path — no Gateway/JWT YAML changes required just to point Claude Desktop at the lab.

Kind	Name	File
Gateway	`claude-desktop-gateway`	`config/gateway/claude-desktop-gateway.yaml`
HTTPRoute	`claude-desktop`	`config/routes/claude-desktop-route.yaml`
AgentgatewayBackend	`anthropic-claude-desktop`	`config/backends/anthropic-claude-desktop.yaml`
AgentgatewayPolicy	`claude-desktop-jwt-auth`	`config/policies/claude-desktop-jwt-auth.yaml`
Lab proxy	—	`scripts/claude-desktop-lab-proxy.py`

Reused — do not recreate: AgentgatewayBackend/entra-jwks, Secret/anthropic-secret (ExternalSecret → Vault), Secret/solo-ui-tls (HTTPS terminate), cost catalog parameters.

Why bother

Without gateway	With this path
Anthropic key on the laptop	Key stays in Vault / cluster
Claude.ai account or raw API key sprawl	Org Entra sign-in
No shared policy / spend story	Same agentgateway budgets, traces, and cost catalog as the rest of the lab

If you already route Claude Code or MCP through agentgateway, this is the Desktop sibling: same identity story, dedicated Gateway so /v1/messages just works.

Graph Engineering on Kubernetes: kagent + Agent Substrate

Fri, 31 Jul 2026 10:00:00 -0400

Graph Engineering on Kubernetes: kagent + Agent Substrate

By Sebastian Maniak

Here’s a familiar pattern. You stand up a few agents, each in its own pod. You give one a supervisor prompt that vaguely says “delegate when needed.” Tools get wired in application code. The demo works.

Then real traffic shows up. Half the sessions sit idle. Nobody can draw who called whom after an incident. And you’re paying for a fleet of pods that spend most of their lives doing nothing.

That is not a model problem. It is an architecture problem — specifically a topology problem.

A practical guide from beamnxw cuts through a lot of the fuzzy language. The short version is worth memorizing:

Harness engineering builds the machinery around the model.
Loop engineering designs the repeated work-and-feedback cycle.
Graph engineering makes the workflow topology explicit: nodes, branches, joins, state transitions, and controlled cycles.

Even shorter: environment → feedback → flow.

Those three layers all sit around the same model. They all affect reliability. They can all involve “loops.” They are still not the same thing. Confusing them is how teams draw twelve-node diagrams on day one, never define what “done” means, and dump every tool they can find into one agent.

This post takes that framing seriously — especially the graph layer — and maps it onto open source you can actually run:

kagent as the control plane (agents, tools, and edges as Kubernetes CRDs)
Agent Substrate as the execution layer (sandboxed actors that suspend when idle and wake fast)

The goal is not more jargon. It is a multi-agent system you can review in Git, operate on a cluster, and not overpay for while it waits.

Why the three layers suddenly matter

A raw language model cannot create files, keep project state, run a test suite, open a browser, enforce an approval rule, or restart a failed job. Those capabilities come from the environment it sits in.

As agentic software matures, a stack is coming into focus:

At the foundation, the harness — the code and runtime that actually run the model
Next, loops — repeating execution and quality checks
Finally, graphs — structured paths that guide the whole process

Labels are still messy in the wild. “Agent harness” is starting to mean something specific. “Loop engineering” is newer practitioner language. Graph engineering here is practical, not academic: you are building agent workflows as explicit directed graphs or state machines — control flow and state transitions, not knowledge-graph entity modeling.

That distinction matters the moment an agent leaves a demo notebook and starts touching files, APIs, customers, or production clusters.

Layer 1 — Harness: the environment

The harness is everything outside the weights. In practice that usually includes:

Context injection — instructions, retrieved facts, conversation state, task policies
Action surfaces — APIs, shells, browsers, databases, MCP tools
Persistence — files, checkpoints, sessions, progress logs, memory
Execution control — timeouts, retries, budgets, model routing, sub-agent spawning, approval gates
Safety — permissions, isolation, allow lists, secret handling
Observability — traces, tool I/O, cost, latency, eval results

A useful trick from the same guide: remove the model from your architecture diagram. What is left is probably the harness.

Two teams can use the same foundation model and get very different outcomes because one gives the model clean tools, a stable workspace, constrained permissions, and observable state — and the other gives it a vague prompt and an unreliable API wrapper. The intelligence may be similar. The working conditions are not.

On our stack, a large part of the harness is Agent Substrate: gVisor actors, WorkerPools, snapshot/restore to object storage, network isolation, and the lifecycle that makes idle sessions cheap. Tools and memory still come through kagent and MCP. The point is the same as the article’s: the harness is the working system, not a footnote under the model card.

Layer 2 — Loop: evidence and stop rules

Every tool-using agent already has a small loop:

call the model → look at the result → run tools → feed observations back → repeat until something terminal happens.

Loop engineering starts when you deliberately design cycles around that behavior: verification loops, event-driven wakeups, improvement loops that learn from traces. LangChain’s newer framing treats these as a stack of loops, not one magic while.

A well-engineered loop has anatomy:

Trigger — user request, schedule, failed test, new data, grader feedback
Goal — a specific state to reach, not “keep improving”
State and memory — what the next cycle needs without replaying everything
Action policy — what the agent may change, call, delegate, or spend
Evidence — tests, schema validation, citations, diffs, metrics, human review
Feedback — compact, actionable description of why evidence failed
Stopping rule — success, budget, timeout, irrecoverable error, human escalation

The key line from beamnxw, which I keep repeating to teams:

Do not loop on confidence. Loop on evidence.

“The agent says it is done” is not a stopping condition. “The tests pass, the links resolve, the schema validates, and the reviewer approves” is.

A prompt tells the model what to do during a call. A loop specifies what the system does after the call: how it observes results, chooses feedback, decides whether to continue, persists progress, and terminates. Prompt quality still matters. The loop turns a one-shot instruction into a managed process.

Tradeoff: every grader, reviewer, or retry costs latency and money. Prefer the simplest architecture that works. Add loops where the cost of failure is higher than the cost of verification.

On kagent, those loops live inside each graph node — the research agent, the writer, the reviewer — not as a vague global “try harder” policy.

Layer 3 — Graph engineering (the deep dive)

This is the heart of the post. Graph engineering asks a different question than harness or loop:

Not only what the agent does — what component is permitted to run next?

Steps are nodes. Allowed next steps are edges. Edges can express sequence, conditional branching, parallel fan-out, joins, loops, and human interrupts. State traverses the graph. The topology is what makes control flow checkable — by you, by a reviewer, by Git.

LangGraph-style systems emphasize durable execution, shared state, and human-in-the-loop control. AutoGen’s docs put it bluntly: use a graph when you need exact control over agent order, different next steps for different outcomes, deterministic branching, or complex multi-step processes with cycles.

That is not “draw boxes because multi-agent is trendy.” That is control over agents, not another abstraction for the sake of it.

What graph engineers actually decide

beamnxw lists the real design work. This is the checklist I use when someone says “we should make it multi-agent”:

1. Node boundaries
Which work belongs in a deterministic function, an LLM call, a specialist agent, or a human review step?
If everything is “another LLM agent,” you have not drawn boundaries — you have multiplied prompts.

2. State schema
What may each node read or update? How do parallel updates merge?
If state is “whatever is in the chat log,” joins and recovery will hurt.

3. Routing conditions
Which evidence sends work forward, backward, sideways, or to escalation?
Routing on vibes (“looks good”) is how you reintroduce invisible control flow.

4. Concurrency
What can run in parallel? What must join? What shared resources need coordination?
Fan-out without a join plan is just concurrent chaos.

5. Cycles and exits
Where are retries legal? How many are allowed? What makes the cycle safe?
A cycle without an exit is a cost leak with extra arrows.

6. Durability
Where do checkpoints occur? How does execution resume after interruption?
Long-running multi-agent work without durability is a demo, not a system.

If you cannot answer those six, you do not have a graph design yet. You have a slide.

When a graph is worth the ceremony

Graphs earn their keep when the process has:

meaningful branches
parallel work that must rejoin
approvals
recovery paths
multiple specialist agents with different tools and prompts

They are less useful when the job is simply “give one agent three tools and let it work.”

There is also a real risk the article calls out: a graph can freeze assumptions too early. If the model must dynamically invent the plan, forcing every possible path into a diagram can make the system more brittle, not less. Formalize paths you have observed. Do not invent a twenty-node topology on day one because the architecture blog had pretty boxes.

How the three layers nest in one system

Notice the nesting carefully:

The graph runs with support from the harness
One or more loops live inside the graph (usually inside nodes)
The harness supplies the state, tools, sandboxes, and evaluators those loops need

Categories overlap because software layers overlap. Each still gives you a different lever when the system fails:

If you see…	Fix this layer first
Cannot resume, loses workspace, tool surface is a mess, permissions are wrong	Harness
Runs forever, no proof of success, retries have no budget	Loop
Wrong specialist runs, no approval path, unreadable flow, no recovery route	Graph

Do not blame the model for orchestration failures. A model cannot compensate reliably for stale state, ambiguous tool schemas, broken APIs, or missing exit conditions. Improve the layer that owns the failure.

Expensive mistakes (steal these warnings)

These come almost straight from the same guide, and I have watched all of them in real projects:

Building a graph before understanding the work.
Teams translate a business process into dozens of nodes before they have watched a capable agent solve it. Start with traces from a simpler harness. Formalize the stable paths.

Letting the same model write and grade without safeguards.
Self-review can help, but it shares blind spots. Prefer deterministic checks where possible, separate reviewer context, and require humans for high-impact actions.

Using “keep trying” as a loop specification.
Unbounded retry is a cost leak. Every loop needs a measurable objective, fresh evidence, max attempts, and a named escalation path.

Treating the harness as a dumping ground.
More tools and memory are not automatically better. Crowded toolsets raise selection errors. Noisy context raises confusion. Broad permissions raise risk.

Blaming the model for orchestration failures.
If the graph has no exits and the harness loses state, a better model will not save you.

Mapping graph engineering onto kagent + substrate

Now the practical part: how do those abstract graph decisions show up as Kubernetes resources?

The split of ownership

 You / UI / Git / CronJob
 │
 ▼
 kagent — graph control plane
 (nodes, edges, models, MCP, HITL)
 │
 │ create / resume actor
 ▼
 Agent Substrate — harness density
 (WorkerPool, gVisor, snapshots)
 │
 ▼
 Running actors
 (loops inside each node session)

kagent owns topology: which agents exist, what they may call, which model they use, when a human must approve
Substrate owns dense, isolated execution: suspend idle sessions, restore sub-second, sandbox with gVisor
Kubernetes owns platform primitives: CRDs, CronJobs, GitOps, RBAC

Put simply:

kagent makes the graph declarative. Substrate makes idle nodes cheap and isolated. Loops still live inside the nodes.

Graph decisions → concrete objects

Here is beamnxw’s design list, translated into this stack:

Node boundaries
A specialist is a SandboxAgent (declarative Go ADK on substrate) or an AgentHarness (coding backends like OpenClaw/Hermes). A deterministic step might stay outside the LLM entirely. A human step is requireApproval on a tool — or a separate operator-facing path. Do not make every box an LLM if a function or a human is the right node type.

State schema
Prefer explicit memory, prompt templates, and session/ACP state over “the whole chat forever.” Each node should know what it is allowed to read and what it is allowed to write. Parallel specialists need a merge story (orchestrator collects structured outputs; do not hope free-form prose merges cleanly).

Routing conditions
In a declarative multi-agent setup on kagent, routing often lives in the orchestrator’s system message plus the hard constraint of which agents appear as tools. That is weaker than a pure state-machine framework for complex branching — and that is fine if your edges are few and stable. For high-stakes branching, make the condition evidence-based in the prompt and keep the legal next agents limited to the tool list. Illegal edges simply do not exist.

Concurrency
If two specialists can run without shared mutable state, you can fan out (sequentially via orchestrator today, or with explicit parallel patterns as your stack grows). Size the WorkerPool for peak concurrent running actors, not for total nodes in the graph.

Cycles and exits
A research ↔ revise cycle is an edge back to writer/reviewer with a max iteration count in the orchestrator policy. Without a max and an escalation edge (human), you reinvent unbounded “keep trying.”

Durability
This is where substrate shines. Idle graph nodes do not need to hold pods. Actors checkpoint RAM + filesystem to object storage and resume when the next edge fires. Graph durability is “the session can survive interruption”; Kubernetes CronJobs handle “start this path on a schedule.”

Edges you can review in Git

In kagent, an edge is usually one of:

Agent-as-tool — orchestrator may call research-node
MCP tool binding — node may call a named tool on a ToolServer
HITL gate — tool requires human approval before it runs

That is the difference between “the supervisor will figure it out” and “the allowed topology is in the CRD.”

Building the graph on this stack

Install order (harness under the graph)

Install Agent Substrate first (ate-system), then kagent with substrate enabled, then a WorkerPool, then ModelConfig, then the agents that form your graph.

I will not re-paste every Helm flag — see the kind guide and code share. Constraint worth knowing early: substrate-backed declarative agents want the Go runtime today.

Declare nodes as SandboxAgents

A node is an honest job description: system message, model, tools, pool.

apiVersion: kagent.dev/v1alpha2
kind: SandboxAgent
metadata:
 name: research-node
 namespace: kagent
spec:
 type: Declarative
 description: Research specialist — structured facts only
 declarative:
 runtime: go
 modelConfig: default-model-config
 systemMessage: |
 You research a topic and return structured facts only.
 Prefer tools over guesswork. Cite sources when you can.
 Stop when the evidence is enough — do not invent certainty. 
 tools:
 - type: McpServer
 mcpServer:
 name: kagent-tool-server
 kind: RemoteMCPServer
 toolNames:
 - k8s_get_resources
 platform: substrate
 substrate:
 workerPoolRef:
 name: kagent-default

Writer and reviewer nodes are the same shape with different prompts and tool allowlists. That is node boundary design.

Draw edges with agent-as-tool

apiVersion: kagent.dev/v1alpha2
kind: SandboxAgent
metadata:
 name: briefing-orchestrator
 namespace: kagent
spec:
 type: Declarative
 description: Research → draft → review for an industry briefing
 declarative:
 runtime: go
 modelConfig: default-model-config
 systemMessage: |
 You own the briefing pipeline.
 1) Call research-node. Require structured facts with sources.
 2) Call writer-node with those facts only.
 3) Call reviewer-node. If evidence fails, send feedback
 to writer-node at most twice, then escalate to a human.
 Never invent a specialist that is not in your tool list. 
 tools:
 - type: Agent
 agent:
 name: research-node
 namespace: kagent
 - type: Agent
 agent:
 name: writer-node
 namespace: kagent
 - type: Agent
 agent:
 name: reviewer-node
 namespace: kagent
 platform: substrate
 substrate:
 workerPoolRef:
 name: kagent-default

Read that system message again as graph policy: sequence, evidence-based cycle with a max, escalation exit, and a hard tool list so illegal edges do not exist. That is loop design living inside a graph node — exactly the nesting the three-layer model describes.

Human interrupts as real nodes

High-impact tools should not be free edges. With kagent, requireApproval pauses execution until a person decides. That is a human interrupt in graph terms — see HITL on kagent. Prefer it for publish, delete, deploy, and spend. Leave pure reads free so the graph does not drown in clicks.

Schedules: CronJob as the missing trigger

Neither kagent nor substrate is primarily a calendar product. Kubernetes already is.

Keep nodes as CRDs (mostly idle, cheap on substrate). Use a CronJob to fire the orchestrator on a schedule. Substrate resumes the actor, the graph runs, the actor suspends again.

apiVersion: batch/v1
kind: CronJob
metadata:
 name: daily-briefing-trigger
 namespace: kagent
spec:
 schedule: "0 8 * * *"
 jobTemplate:
 spec:
 template:
 spec:
 restartPolicy: OnFailure
 containers:
 - name: trigger
 image: curlimages/curl:8.5.0
 args:
 - -sS
 - -X
 - POST
 - http://kagent-controller.kagent.svc:8083/api/a2a/kagent/briefing-orchestrator/
 # Adjust path, body, and auth for your environment

Trigger from CronJob. Topology from CRDs. Density from substrate. No fourth control plane for “08:00.”

Why sparse graphs stay cheap

Most multi-agent graphs are sparse in time. Research runs, then sleeps. Writer runs, then sleeps. Reviewer runs for a moment. Orchestrator waits on a human.

Pod-per-agent pays for the worst case continuously. Substrate pays for concurrent running work:

Edge fires / request arrives
Actor restores onto a free WorkerPool pod
Node loop runs inside gVisor
Idle → checkpoint to object storage → worker returns to the pool

kubectl scale workerpool kagent-default -n kagent --replicas=5

Scale concurrency, not “number of boxes in the architecture diagram.”

Worked example: research-and-publish briefing

beamnxw uses a research-and-publishing agent as the running example of how layers nest. Here is the same story on kagent + substrate.

CronJob or user
 │
 ▼
[briefing-orchestrator]
 │
 ├─▶ [research-node] ── MCP tools, structured facts
 │
 ├─▶ [writer-node] ── draft from facts only
 │
 └─▶ [reviewer-node] ── evidence checks
 │
 pass? ─┼─ yes → publish (optional HITL)
 │
 └─ no → feedback to writer (max N)
 then escalate to human

Graph decisions in this design:

Nodes: orchestrator, research, writer, reviewer, human (on publish)
Edges: agent-as-tool refs; optional cycle writer ↔ reviewer; escalate edge
State: structured facts object, draft, review findings — not one blob of chat
Routing: reviewer evidence decides pass / revise / escalate
Cycles: max two revise loops, then human
Durability: each specialist session is a substrate actor; idle does not hold a pod

Loop decisions: mostly inside reviewer — grade against sources, return compact feedback, stop on pass, budget, or escalate.

Harness decisions: gVisor isolation, tight MCP tool lists per node, OTel traces, snapshots so a long briefing can pause on HITL without burning a warm Deployment overnight.

If you only remember one sequencing rule from the article: do not draw this whole graph on day one. Start with a single agent and traces. Promote stable handoffs into CRDs when you see them repeating.

Questions worth asking before you scale this

Before you grow the graph, sit with a few questions. They are adapted from the production checklist in the beamnxw guide — translated into this stack.

On the graph: Which paths must be deterministic? Where can work run in parallel? What state is shared, and who merges it? Where are the human gates and recovery routes? Could a teammate see the legal topology from the YAML alone?

On the loops: What evidence proves success for each node? What feedback comes back on failure? How many retries? What happens when the budget is gone?

On the harness: Are tools few, clear, and observable? Does state survive suspend/resume? Are permissions tight? Can someone pause, inspect, and resume a run without redeploying the world?

On ops: Can you replay a bad run and point at a CRD change that fixed it? Are you watching cost, latency, failure rate, and how often a human has to step in?

If you cannot answer the graph questions, a bigger model will not save you. If you cannot answer the harness questions, more nodes will not either.

When you should not build a graph

If the job is “one agent, three tools, let it work,” you do not need twelve nodes. Build a solid harness and a tight loop first.

Add a multi-node graph when you keep reimplementing the same branches, specialist handoffs, recovery paths, or human gates in glue code — and when those paths are stable enough that freezing them is a feature, not a trap.

Graph engineering is for control you can inspect. It is not a fashion statement.

Bottom line

Steal the three-layer framing and use it on purpose:

Harness — the environment that makes the model operable
Loop — the iterative process with evidence and stop rules
Graph — the explicit map of what may run next

None of them replaces the others. A pretty graph with a broken harness still loses state. A great harness with no stop rules still burns money. Careful loops still turn to spaghetti when branching and approvals live only in ad-hoc code.

On Kubernetes, the split is clean enough to ship:

kagent makes the graph (and a lot of the policy) declarative
Agent Substrate makes idle nodes cheap and isolated
You still design node boundaries, evidence, and exits
CronJobs handle “run this path at 8am” without inventing a new product

Keep the topology in Git. Scale the WorkerPool for concurrent work, not for diagram vanity. Put humans on tools that can hurt.

Where to start (call to action)

You do not need the full briefing pipeline on day one. Do this instead:

Stand up the harness. Install Agent Substrate, then kagent with substrate enabled, then a small WorkerPool. Follow the hands-on path:
Suspend & resume on kind · OSS substrate code share
Ship one node you trust. One SandboxAgent, one model, a short tool list. Chat with it. Watch it suspend and resume. Get comfortable before you draw edges.
Add edges only when traces demand them. Promote repeated handoffs into agent-as-tool refs. Keep illegal specialists off the tool list so they cannot run by accident.
Put a human on the dangerous tools. Start with HITL on kagent. Approve deletes and deploys. Leave reads free.
Put the YAML in Git. Treat prompt, model, and tool changes like production config. GitOps for agents is the longer version of that argument.
Schedule later. When you need calendar wakeups, add a CronJob that triggers the orchestrator. Do not wait for a native “agent cron” product.

Runnable demos live in sebbycorp/kagent-demos. Upstream docs: kagent.dev · learn.agentsubstrate.dev · github.com/kagent-dev/kagent. The conceptual spine for this post is still beamnxw’s three-layer guide.

Your move: clone the demo, run one substrate-backed agent on kind this week, and only then decide which edges belong in the graph. Topology you have not earned from traces is just decoration.

agentgateway 1.4.0 OSS: what changed since 1.3.0

Mon, 27 Jul 2026 00:00:00 +0000

agentgateway v1.4.0 landed on July 27, 2026.

If you are still on 1.3.x, this is the upgrade map. It tracks the official v1.4.0 release notes — highlights, breaking changes, security, and the day-2 bits that actually matter — plus a bit of operator commentary.

Also useful

v1.3.0 · v1.3.1
Diff: v1.3.1…v1.4.0

Install bits

Images: cr.agentgateway.dev/agentgateway:v1.4.0, cr.agentgateway.dev/controller:v1.4.0 (glibc — no more musl image tags)
Charts: agentgateway, agentgateway-crds, agentgateway-standalone
Binaries: still built with musl; proxy + agctl

The short version

Upstream’s own highlight reel:

Full MCP 2026-07-28 support (and a year of community work behind it)
Cross App Access / Enterprise-Managed Authorization for MCP (ID-JAG)
OAuth token exchange backend auth (RFC 8693 + JWT bearer)
Standalone gateways + DB-backed UI config (sqlite / postgres) + standalone Helm chart
Experimental AgentgatewayModel on Kubernetes (off by default)
Gateway API v1.6, fault injection, richer guardrails, lots of fixes

My add-on for operators: read the breaking + security sections before you helm upgrade. There is a High severity MCP auth advisory in this release.

Area	What moved
MCP	Protocol 2026-07-28, Apps, traces via `_meta`, Entra/Descope/authentik
Auth	Token exchange, Cross App Access (incl. MCP EMA), hashed API keys
Standalone	`gateways` supersede `binds`, UI on same port, sqlite/postgres storage
UI	Settings to bind console + attach OIDC/JWT/API key/CSRF/CORS
Kubernetes	Experimental `AgentgatewayModel` (off by default)
Platform	DaemonSet, Gateway API 1.6 / TCPRoute v1, monitors, delay fault injection
Security	GHSA-mvgg-jvj2-4frq + CEL body guidance

Read this first: breaking changes

Straight from upstream. Do these before you call the upgrade done.

1. Gateway API v1.6 and TCPRoute v1

agentgateway builds against Gateway API v1.6. The controller uses TCPRoute v1, not v1alpha2.

Action: re-apply the Gateway API CRDs that match this release before you upgrade the controller/proxy.

2. MCP request-phase guardrail rejections return HTTP 200

If an MCP guardrail rejects in the request phase, the gateway now returns HTTP 200 with a JSON-RPC error body — same as the response phase.

Action: fix clients/tests that expected a non-200 on request-phase rejects.

Docs: k8s MCP guardrails · standalone

3. Standalone `auth.location` no longer nests `expression`

Custom token location config dropped the double-nested expression field. Use the flattened form.

Action: grep your standalone policies for nested auth.location expressions and update them.

4. musl container images removed

Musl image variants are gone. Use standard glibc images. Binary releases are still musl builds.

Action: update image pins, digests, and SBOMs.

Read this second: security

High: MCP sessions crossing routes (GHSA-mvgg-jvj2-4frq)

This release fixes GHSA-mvgg-jvj2-4frq — High (8.1) — where stateful MCP sessions could cross routes and overwrite authorization policy.

Read the advisory for impact and mitigation. Credit: Aonan Guan.

Related hardening in the same train includes pinning MCP sessions to a backend and tightening passthrough checks. Still: if you run MCP with authz on 1.3, treat 1.4 as a security upgrade, not just a feature bump.

CEL `request.body` / `response.body` in auth policies

Upstream flags a footgun (recommendation + behavior change), also from the same reporter.

Attribute	1.3.x and earlier	1.4
`request.body`	Truncated to `http.maxBufferSize` (default 2MB)	Not available if truncated
`request.truncatedBody`	Not available	Truncated to max buffer size

Why it matters: a policy like string(request.truncatedBody).contains("attacker-payload") can miss a match when the body is over 2MB, or when encoding/compression changes what you thought you were matching. Auth on raw bodies is easy to get wrong.

Action: audit CEL auth that touches bodies. Prefer not using body contents for hard authorization when you can avoid it. If you must, handle the “body missing / truncated” case explicitly.

What 1.3 already gave you

v1.3.0 brought the rebuilt UI (LLM / MCP / Traffic), cost tracking, virtual models, reusable providers/guardrails, more LLM providers, body buffering, better CEL/agctl, better traces.

v1.3.1 was polish: controller podLabels, Vertex multi-region, same port with different protocols across gateways, UI wildcard/virtual-model fixes.

1.3.1 does not force a redesign. 1.4 does — config shape, CRDs, MCP auth behavior, and at least one security-fixing upgrade path.

1. MCP protocol 2026-07-28

This is highlight #1 in the official notes.

agentgateway adds support for the upcoming MCP 2026-07-28 protocol:

Stateful and stateless servers (SEP-2575 server-stateless gap closed; skip synthetic initialize for modern requests)
Trace context through MCP _meta (SEP-414 / traceparent) — stdio backends stay on the trace
Basic MCP Apps (plus multiplexing fixes for app-originated tool calls)
Multi-target subscriptions/listen, opaque URI multiplexing, preserved multi-resource tool-result capabilities

Most of this is new enough that dedicated guides are still thin. Use the Kubernetes API reference and Standalone config reference for fields available today.

New MCP identity providers

Microsoft Entra ID (k8s + standalone) — bridges Entra vs MCP-auth mismatches (OIDC discovery metadata, strip RFC 8707 resource, short-circuit DCR with your pre-registered app id)
Descope and authentik on standalone
Okta was already first-class in 1.3

Docs: k8s MCP auth · standalone MCP auth · Descope guide

2. Cross App Access (for MCP and backends)

Official name in the MCP world: Enterprise-Managed Authorization. Implementation path: OAuth Identity Assertion Authorization Grant — Cross App Access / ID-JAG.

An enterprise IdP can broker access between a client app and an MCP server (or other backend) without a separate interactive OAuth dance for every downstream app.

Docs: k8s Cross App Access · standalone

Examples landed for Keycloak, xaa.dev, token exchange, and JWT bearer. Subject token source is configurable.

3. OAuth token exchange backend auth

The gateway can exchange an incoming token for a backend credential using RFC 8693 token exchange and RFC 7523 JWT bearer.

Also in this release:

Kubernetes controller support
Custom token types + OAuth 2.1-ish exchange defaults
Inject multiple secret-sourced headers
Override the resolved secret key for OAuth/GCP-style refs

Docs: k8s OAuth token exchange · standalone

If your 1.3 setup is “edge JWT ok, upstream still sees a shared service account,” this is the fix.

Virtual keys from ConfigMaps (not only Secrets)
API keys stored as SHA-256 hashes (raw key need not live in plaintext config)
Admin IP allowlist + timing-attack fixes
private_key_jwt with certificate credentials
AWS assume-role: session tags + RoleSessionName / sessionNameExpression via CEL (e.g. propagate jwt.sub)
Per-backend SigV4 region
Frontend TLS with multiple client CAs
Bundled certs
Cloud auth fetch timeouts

Virtual keys docs: k8s · standalone

4. Standalone: `gateways`, storage, Helm

`gateways` replace `binds`

New top-level gateways unifies UI, LLM, MCP, and ordinary routes on one listener/port. That is what makes OIDC on the UI a first-class story.

Important nuance from upstream:

gateways supersedes binds
Existing binds still work
The UI offers a one-click migration from binds → gateways

Also: simpler host/TLS config, LLM+MCP on the same port, internal bind mode with wildcard fallback.

Config reference: standalone configuration

Database-backed config (no PVC required)

UI-created config can live in a database:

sqlite for local
postgres for remote / HA

That avoids needing a persistent disk for writable UI workflows. Hybrid mode, policy writes, multi-replica notify, and “log errors to SQL too” landed in the same train. Storage naming moved away from the old configStore wording.

Admin interface is not exposed by default.

Standalone Helm chart

cr.agentgateway.dev/charts/agentgateway-standalone:v1.4.0 is real: modes for file vs database, gateways-oriented defaults, OIDC secret support, metrics service + ServiceMonitor.

Docs: Standalone Helm

Treat 1.4 as the first serious baseline for that chart, not a soft bump from early experiments.

5. UI Settings: protect the console

In the product UI: Tools → Settings → UI Settings.

Bind the console to a traffic gateway, then attach the same policy toolkit you use elsewhere.

You get:

Public UI gateway picker + view diff / save
Policy cards: OIDC, JWT, Authorization, External authz, Basic auth, API keys, CSRF, CORS
Expandable top-level policy YAML for the GitOps-minded

After upgrade: bind the UI to the gateway you actually expose, turn on OIDC or JWT, then put that listener on a real network.

6. Experimental `AgentgatewayModel` (off by default)

Kubernetes headline — with the official caveats.

AgentgatewayModel brings the standalone model-centric LLM experience to Kubernetes: serve multiple models on one gateway, route by model name in the request. Higher level than assembling AgentgatewayBackend + HTTPRoute body matchers by hand.

Short names: agmodel / agentgatewaymodels.agentgateway.dev
Experimental
Off by default
Existing APIs remain available

Why try it: one GitOps object per model/family, less HTTPRoute glue, k8s and standalone stop telling different stories.

Why not force it day one: experimental. Pilot one non-prod model after the cluster is stable on 1.4.

7. Guardrails, ext_proc, CEL, fault injection

Guardrails

BackendConnectionPolicy for guardrail callouts (TCP/TLS/HTTP/tunnel) on OpenAI moderation, Bedrock guardrails, Google Model Armor
Default callout timeouts
Clearer guardrail decisions in logs/UI
ext_proc failureMode (fail-open / fail-closed)

External processing

Controller support for metadataContext, requestAttributes, responseAttributes, plus failureMode.

CEL

Custom CEL functions you can register for policies
Opt-in CEL filters for OTel spans
CEL filters that decouple OTLP log fields from stdout
source.connectHeaders for inbound CONNECT headers
Header transformation replace mode via CEL
Body handling split: body vs truncatedBody (see security section)

CEL refs: k8s · standalone

Fault injection: `delay`

New delay traffic policy injects latency before forward. Duration string, CEL duration, or number-as-milliseconds. Injected delay counts against the request timeout.

Docs: k8s fault injection · standalone

8. LLM, A2A, and provider path fixes

From the official “LLM gateway enhancements” plus the long PR train:

Frontend TLS multi-CA client cert validation
Bedrock: Responses→Bedrock images, 64-char tool name sanitizing, cache-write tokens in access logs
Gemini / Vertex: embeddings fix, native generateContent paths, detect-mode model/usage extraction
Azure AI Foundry: Anthropic endpoints; Responses/date-based API path fixes
A2A v1.0 agent card format in URL rewriting
Reasoning fixes on messages→completions; body decode before JSON parse
Cost: Prometheus counter; catalog ConfigMap loading; virtual keys from ConfigMaps
Telemetry: tool calls, A2A response metrics, richer dtrace
Semantic routing examples (including Responses)

Providers: k8s · standalone

9. Deployment and ops

DaemonSet data plane (still defaults to Deployment)
Helm extraContainers on control plane pods
Proxy scrape via PodMonitor; standalone metrics Service + ServiceMonitor
agentgateway_controller_build_info metric
Standalone SQL logging for success and error, with faster writes
Internal bind mode (annotation + wildcard fallback)
Frontend policies targetable to a Gateway port
Buffer: stream when body exceeds limit instead of always hard-failing
ext_authz HTTP cache + custom HTTP bodies
agctl much smaller; some trace/config commands under proxy subcommand; nightly artifacts
s390x support
Embedded UI build by default (UI=0 to opt out)

How I would upgrade

Kubernetes

Re-apply Gateway API CRDs for the v1.6 / TCPRoute v1 set this release expects
Bump agentgateway-crds → v1.4.0
Bump controller + proxy together
If you run authenticated MCP: read GHSA-mvgg-jvj2-4frq and validate session/route authz behavior
Smoke HTTPRoutes, MCP, JWT/ext_authz, traces, guardrails
Audit CEL auth that uses request.body / response.body
Only then pilot one non-prod AgentgatewayModel
Keep Deployment unless you truly need DaemonSet

Standalone

Snapshot config
Note: binds still work; migrate with UI one-click or convert to gateways deliberately
Flatten any nested auth.location expressions
Pick storage: sqlite local vs postgres HA (drop PVC assumptions from early charts)
Open UI Settings, bind console, enable OIDC/JWT
Install/upgrade agentgateway-standalone v1.4.0 after reading values

Auth-heavy estates

Inventory upstreams still on shared service accounts
Prototype token exchange on one API
Prototype Cross App Access for one MCP or SaaS API
Prefer native Entra MCP provider over home-grown resource-parameter proxies
Prefer hashed virtual keys / ConfigMap sources where you can

What I would turn on first

Security fix path — upgrade MCP authz estates for the High advisory
Gateway API CRD re-apply + clean 1.4 roll
One gateway + UI Settings — console actually protected
Token exchange or XAA — user-scoped upstream tokens
MCP 2026-07-28 + _meta traces — federation you can debug
Experimental AgentgatewayModel — one non-prod pilot, not a big-bang rewrite

Links

Official v1.4.0 release
Security advisory GHSA-mvgg-jvj2-4frq
v1.3.1…v1.4.0
Docs: Kubernetes quick start · Standalone quick start
On this site: Eight principles of an AI gateway · Why the v1 line matters

Bottom line: 1.4 is not only features. It is MCP protocol catch-up, real delegated auth (token exchange + Cross App Access for MCP), standalone that can run UI/LLM/MCP on one port with DB-backed config, an experimental model API on Kubernetes, and a set of breaking + security fixes you should not skip.

Re-apply Gateway API CRDs. Bump charts. Read the MCP session advisory. Bind and lock down the UI. Then play with AgentgatewayModel — in that order.

Budgets You Can See: LLM Cost Governance in the Enterprise agentgateway UI

Wed, 15 Jul 2026 00:00:00 +0000

A couple of weeks ago I wrote up AI budgets and hard spend limits in Enterprise agentgateway — a day-one, YAML-and-curl field report on the new EnterpriseAgentgatewayBudget CRD. That post answered how do I declare a budget and watch it trip?

This one answers the question the platform lead actually asks at standup: which teams are on track, and who’s blown their allocation? Because the same budgets you declare in YAML now surface as a first-class view in the solo enterprise for agentgateway console — under Cost Management → Budgets — where spend is something you read, not something you kubectl get and squint at.

This is a tour of that experience. Everything below is a live lab: a management cluster running solo enterprise for agentgateway, three teams (platform, sales, marketing) with per-user and per-team token budgets, and enough traffic driven through the gateway to push one of them over the edge on purpose.

The Budgets tab at a glance

Cost Management is where the gateway’s per-request cost telemetry rolls up. The Budgets tab lists every budget selected by an active gateway budget-enforcement policy, with a one-line health readout for each.

Three things are worth reading here:

The summary chips — 5 Budgets, 4 Within budget, 1 Exceeding budget. This is the whole point of the view: a single glance tells you the fleet is mostly healthy and exactly one budget needs attention. Click the red Exceeding budget chip and the list filters to just the offender.
The Summary column — TOKENS 500 / DAILY · AUDIT (+1 more). Each budget is denominated in tokens or USD, accrues over a rolling window (here, Daily), and carries an enforcement action — Audit or Block. The (+1 more) tells you the budget has multiple entries with their own limits.
The Usage bars — green when under, red when over. team-budget-platform is the one showing red; everyone else is comfortably in the green.

The scope column (mgmt-cluster/agentgateway-system) tells you where each budget lives — cluster and namespace — which matters once teams start keeping their own Budget objects in their own namespaces.

Anatomy of a budget

Before drilling in, the mental model. A budget is a named container scoped to a target (a gateway or route), holding one or more entries. Each entry is an independent limit with four dimensions:

Field	What it means	In the screenshots
Subject	Which requests the entry applies to — a `user`, a `group`, or unscoped	`user alice`, `group platform`
Limit	The cap, in tokens or USD	`200 tokens`, `500 tokens`
Rolling window	The trailing period spend ages out of	`Daily (24 hours)`
When exceeded	`Audit` (observe) or `Block` (429)	`Audit`

The budget’s Total Usage is the roll-up across its entries. That lets you do something genuinely useful: put a per-team ceiling and a per-user ceiling in the same budget, and watch both at once. A single heavy user can be over their personal allocation while the team as a whole is still fine — and the UI shows you both facts side by side.

Drilling in: `team-budget-platform`

This is the budget flying the red flag. Open it and the story is immediate:

Two entries, two very different stories:

platform-alice-tokens — scoped to user alice, capped at 200 tokens/day. She’s at 200 of 200 — 100%, Over budget, the bar solid red. Alice has been the heavy user today.
platform-team-tokens — scoped to group platform, capped at 500 tokens/day. The team is at 297 of 500 — 59%, On track, comfortably green.

The Total Usage rolls both into 71% — 497 of 700 tokens. That’s the per-user + per-team pattern doing exactly what it should: alice individually tripped her limit, but the platform team still has room. One noisy user hasn’t consumed the whole team’s allocation, and you can see precisely where the pressure is without cross-referencing anything.

Because both entries are set to Audit, nothing is blocked — alice’s requests still succeed. Audit mode is visibility first: the gateway records the overage, paints it red, and lets traffic through so you can decide whether that limit is right before you ever arm enforcement. (Flip onBudgetExceeded to Block and that same 100% entry starts returning 429s instead.)

Each entry also has a View in dashboard link that pivots straight into the filtered Cost Management dashboard for that subject — from “who’s over” to “what exactly did they run” in one click.

What healthy looks like: `team-budget-sales`

For contrast, the sales budget is the picture of a team well within its means:

Same shape — a per-user entry (sales-dave-tokens, user dave, 180 tokens) and a per-team entry (sales-team-tokens, group sales, 450 tokens) — but every bar is green. Total usage sits at 12% — 76 of 630 tokens. Dave’s at 21% of his personal cap, the team at 8% of theirs. Nothing to see, which is precisely the signal you want most of your teams to be sending.

Line the two budgets up and the console is doing real FinOps work: same gateway, same route, different wallets — platform under pressure, sales idle — each with its own per-user and per-team accounting, all readable in seconds.

The status vocabulary

Two layers of status, and it’s worth being precise about them:

Per entry: On track (green) or Over budget (red). This is about a single limit — one user, one group.
Per fleet: the summary chips count budgets as Within budget or Exceeding budget. A budget is “exceeding” the moment any of its entries is over. That’s why one over-limit alice entry tips the whole team-budget-platform budget — and the fleet counter — into the red.

And crucially, status is independent of enforcement. Everything in this lab runs in Audit — so “Over budget” and “Exceeding budget” here mean recorded and flagged, not blocked. That separation is the feature: you get the full red-flag dashboard experience long before you ever cut anyone off, which is exactly how you should roll budgets out.

The YAML behind the view

The console is a lens, not a second source of truth. Every budget carries a Resource YAML panel showing the EnterpriseAgentgatewayBudget object it renders. For team-budget-platform, that object is roughly:

apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayBudget
metadata:
 name: team-budget-platform
 namespace: agentgateway-system
spec:
 budgets:
 - name: platform-alice-tokens
 subject:
 user: alice  # per-user entry
 limit:
 unit: Tokens
 amount: 200
 window:
 unit: Day
 onBudgetExceeded: Audit
 - name: platform-team-tokens
 subject:
 group: platform  # per-team entry
 limit:
 unit: Tokens
 amount: 500
 window:
 unit: Day
 onBudgetExceeded: Audit

The subject block is what makes the two entries different: one keys on the user dimension, the other on group. Everything you saw in the drill-in — the two bars, the two limits, the rolled-up total — is just this object, rendered. Create a budget in the UI and you get this YAML; commit this YAML via GitOps and you get the UI. Same object, two doors.

How it works under the hood

A few mechanics tie the pretty bars to reality — covered in depth in the hands-on post, so here’s the short version:

Dimensions come from request identity. user and group values are resolved per request from trusted identity — headers derived from JWT claims or API-key metadata — so the gateway knows whose wallet to charge before it ever forwards the call. An entry’s subject matches when every key/value it lists matches the request’s resolved dimensions ("*" matches any non-missing value; omit subject and the entry applies to every request on the target).
Windows are rolling, not calendar. A Daily budget looks at the trailing 24 hours. Spend ages out continuously — a tripped budget recovers as old requests fall off the window, rather than snapping back to zero at midnight.
USD budgets need a cost catalog. Token budgets count total_tokens (input + output). Dollar budgets multiply realized per-request cost — computed from the model cost catalog — so the console can show you USD 5 / DAILY just as easily as TOKENS 500 / DAILY (both appear in the budget list above).
Audit and Block are the two gears. Audit records and flags; Block returns 429 with the rolling window in the reset header. The whole rollout path is: ship budgets in Audit, watch the console for a week, then arm Block only where runaway spend would actually hurt.

Wrapping up

The CRD gave platform teams a way to declare LLM spending limits. The Cost Management UI gives them a way to govern against those limits day to day — per-team and per-user ceilings, rolling windows, and a red-means-attention readout that a finance partner can look at without touching kubectl.

The scenario in these screenshots — platform straining while alice sits over her personal cap, sales cruising at 12% — took nothing more than a handful of token budgets with user and group subjects and some traffic to push through them. Every budget still lives as YAML you can GitOps; the console just makes the answer to “who’s over?” a one-glance question instead of a query.

Audit first, watch the bars, then arm Block where it counts.

On-Behalf-Of, Explained: How agentgateway Swaps an Entra Token for a Microsoft Graph Token

Wed, 15 Jul 2026 00:00:00 +0000

There’s a moment in almost every enterprise integration where you have a token, but not the token. You’re holding a perfectly valid JWT that your identity provider signed — your user is authenticated, the claims are real — and yet the API you need to call rejects it with a flat 401. Nothing is wrong with the token. It’s just addressed to someone else.

That’s the problem the OAuth 2.0 On-Behalf-Of (OBO) flow was invented to solve, and it’s the story this article tells. We’ll use a small, real lab: an MCP client holding an Entra (Azure AD) token, an Enterprise agentgateway acting as the broker, and Microsoft Graph as the downstream API. By the end you’ll understand why OBO exists, how the exchange actually works on the wire, and how to configure the gateway to do it for you — all reproducible with a single script, test-entra-obo.sh.

The 401 that starts everything

Let’s begin with the failure, because the failure is what makes OBO make sense.

Your MCP client signs in a user through your corporate IdP and receives an access token. It’s a bearer token — you’d be forgiven for assuming you can now take it and call any Microsoft API. So you try the obvious thing: send it straight to Graph.

sequenceDiagram participant C as MCP Client participant E as Microsoft Entra participant G as Microsoft Graph C->>E: Sign in — get access token E-->>C: JWT (aud = your-backend-api) C->>G: GET /v1.0/me Authorization: Bearer <that JWT> G-->>C: 401 InvalidAuthenticationToken Note over C,G: The token is valid — just not for Graph.

The token is cryptographically sound. Entra signed it. The user is who they say they are. But Graph looks at one field and stops reading: aud, the audience.

Why the audience matters (the whole story in one claim)

Every access token carries an audience claim that names who the token is for. When Entra minted your token, you asked for a token scoped to your own backend API — an app registration with the ID 0c00f6d2-5587-469d-9d35-7360d878fddc, in this lab called goose-solo-ui-backend. So the token says, in effect:

“This bearer is authorized to talk to 0c00f6d2-…, on behalf of sebastian@maniak.io.”

Microsoft Graph’s audience is https://graph.microsoft.com. When Graph receives a token whose aud is your backend, it correctly refuses it — a token for one audience must never be accepted by another. That rule isn’t bureaucratic; it’s the thing that stops a token you handed to one service from being replayed against a completely different one. Audience scoping is a security feature, and OBO is how you work with it instead of against it.

Here is the token the client actually holds, decoded (this is real output from the demo’s token mode):

iss : https://login.microsoftonline.com/8635e970-…/v2.0
aud : 0c00f6d2-5587-469d-9d35-7360d878fddc ← your backend, NOT Graph
sub/oid : ed1b752d-d99c-421f-afc3-a6ebc323cd27
user : sebastian@maniak.io
scp : access_as_user
azp : 04b07795-8ddb-461a-bbee-02f9e1bf7b46

The star of the show is aud. It points at the middle tier. Graph would look at it and hand back InvalidAuthenticationToken. We need a token with aud = https://graph.microsoft.com — but we need it to still represent the same signed-in user. That is precisely what OBO produces.

What On-Behalf-Of actually is

OBO is a token exchange. A middle-tier service (the “confused deputy” in older literature, here our gateway) takes the user’s token, presents it back to the identity provider as proof, and asks: “The user already authorized me. Now mint me a new token — for a different downstream API — that still acts on this same user’s behalf.”

The IdP validates the incoming token, checks that the middle tier is allowed to request the downstream scope, and issues a brand-new token with the downstream audience and the same user identity. The user never re-authenticates. The downstream API never sees the original token. And — the part that matters for MCP servers — the thing that performs the exchange holds the client secret; the code that calls Graph does not.

graph LR A["Token IN aud = your-backend user = sebastian"] -->|OBO exchange| B["Token OUT aud = graph.microsoft.com user = sebastian"] style A fill:#FFF7D6,stroke:#E5341F,color:#17181C style B fill:#F1EFE9,stroke:#E5341F,color:#17181C

Same user. New audience. That single transformation is the entire point.

The lab: who is who

Before the flow, meet the cast. The whole demo is three network hops wide.

graph TD C["MCP Client (test-entra-obo.sh / MCP Inspector)"] G["agentgateway http://172.16.10.155:30160/graph-me policy: entra-obo-policy"] M["graph-me-mcp (in-cluster MCP server)"] GR["Microsoft Graph GET /v1.0/me"] E["Microsoft Entra /oauth2/v2.0/token"] C -->|"Bearer: middle-tier JWT aud = goose-solo-ui-backend"| G G -->|"OBO: grant_type=jwt-bearer on_behalf_of"| E E -->|"access token aud = graph.microsoft.com"| G G -->|"injects Graph Bearer upstream"| M M -->|"GET /me + injected Bearer"| GR style C fill:#FFF7D6,stroke:#17181C,color:#17181C style G fill:#FFFFFF,stroke:#E5341F,color:#17181C style M fill:#F1EFE9,stroke:#17181C,color:#17181C style GR fill:#F1EFE9,stroke:#17181C,color:#17181C style E fill:#F1EFE9,stroke:#17181C,color:#17181C

Actor	Role	What it holds
MCP client	Starts the request	A middle-tier Entra JWT (`aud = goose-solo-ui-backend`)
agentgateway	The broker	The app client secret, and the OBO policy
Microsoft Entra	Security token service	The signing keys; issues the exchanged token
graph-me-mcp	Downstream MCP server	Nothing — it receives an injected Graph token
Microsoft Graph	The protected API	Validates `aud = https://graph.microsoft.com`

The crucial design property: the client never holds a Graph token, and the MCP server never holds the client secret. The gateway sits in the middle and is the only party that touches both.

The full flow, end to end

Now the whole thing on one wire diagram. This is what happens on the very first MCP call.

sequenceDiagram participant C as MCP Client participant AG as agentgateway (entra-obo-policy) participant E as Microsoft Entra (STS) participant M as graph-me-mcp participant G as Microsoft Graph C->>AG: POST /graph-me (MCP initialize) Authorization: Bearer <middle-tier JWT> AG->>AG: Validate JWT (iss, aud, signature) AG->>E: POST /oauth2/v2.0/token grant_type=jwt-bearer assertion=<user JWT> requested_token_use=on_behalf_of scope=Graph/User.Read E-->>AG: access_token (aud = graph.microsoft.com) Note over AG: Gateway now holds a Graph-scoped token for THIS user. AG->>M: Forward MCP request + inject Graph Bearer upstream M->>G: GET /v1.0/me Authorization: Bearer <injected token> G-->>M: 200 — user profile JSON M-->>AG: MCP tool result AG-->>C: 200 OK (+ MCP session id)

Read it top to bottom and the story is complete: a token that Graph would reject goes in the top; a real Graph profile comes out the bottom; the exchange in the middle is the only thing that changed, and it happened inside the gateway.

How to do it: the gateway configuration

The behavior above is declarative. Three small YAML objects wire it up — a policy that describes the exchange, a backend that points at the MCP server, and a route that binds a path to them. These mirror the config referenced by the demo (config/policies/…, config/backends/…, config/routes/…).

1. The OBO policy

This is where the exchange lives. The key block is tokenExchange.entra, and the key setting is mode: ExchangeOnly — meaning “just do the OBO swap, no interactive elicitation UI.”

# config/policies/entra-obo-policy.yaml
apiVersion: gateway.agentgateway.dev/v1alpha1
kind: Policy
metadata:
 name: entra-obo-policy
spec:
 tokenExchange:
 entra:
 mode: ExchangeOnly  # do the OBO swap; no elicitation prompt
 tenantId: 8635e970-2205-4189-bc77-77519ff5064f
 clientId: 0c00f6d2-5587-469d-9d35-7360d878fddc  # the middle-tier app
 clientSecretRef:
 name: entra-obo-secret  # the app secret — lives ONLY here
 key: client-secret
 scope: https://graph.microsoft.com/User.Read  # downstream audience+scope

Everything security-sensitive is contained in this one object: the tenant, the app that’s allowed to perform the exchange, the secret that proves it, and the exact downstream scope being requested. Nothing downstream needs any of it.

2. The MCP backend

The backend declares the in-cluster MCP server that will receive the injected token.

# config/backends/graph-me-mcp.yaml
apiVersion: gateway.agentgateway.dev/v1alpha1
kind: Backend
metadata:
 name: graph-me-mcp
spec:
 mcp:
 targets:
 - name: graph-me
 service:
 host: graph-me-mcp.default.svc.cluster.local
 port: 8080

3. The route

The route ties a public path to the backend and applies the policy. This is what makes POST /graph-me do OBO.

# config/routes/graph-me-mcp-route.yaml
apiVersion: gateway.agentgateway.dev/v1alpha1
kind: HTTPRoute
metadata:
 name: graph-me-mcp-route
spec:
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /graph-me
 filters:
 - type: Policy
 policy: entra-obo-policy
 backendRefs:
 - name: graph-me-mcp

That’s the whole surface area. Add a path, attach a policy, point at a backend — the gateway handles the token dance.

Watching it run: the five steps of the demo

The test-entra-obo.sh script narrates the exact flow above in five steps. Here’s what each one proves.

Step 1 — Mint the middle-tier token (what the client holds)

The script asks Entra for a token whose audience is the middle-tier API, not Graph:

az account get-access-token --resource api://0c00f6d2-5587-469d-9d35-7360d878fddc

aud : 0c00f6d2-5587-469d-9d35-7360d878fddc ← middle-tier, not Graph
user : sebastian@maniak.io
scp : access_as_user
★ this token is MIDDLE-TIER scoped — Graph would 401 it.

Teaching point: if we sent this straight to Graph, we’d get the 401 we opened with. OBO exists so the gateway can swap it first.

Step 2 — MCP `initialize` (the exchange fires here)

POST http://172.16.10.155:30160/graph-me
Authorization: Bearer <middle-tier JWT>
→ HTTP/1.1 200 OK (456 ms)
 server : graph-me-mcp v1.0.0
 protocol : 2025-03-26
 session : eyJ0IjoibWNwI… (send this on every later call)

On this call the gateway validates the JWT, calls Entra’s /oauth2/v2.0/token with the OBO grant, and caches a Graph-scoped token for the upstream. The client sees only a normal MCP handshake and a session id.

Step 3 — `tools/list`

• graph_me
 Call Microsoft Graph GET /me using the bearer token
 injected by agentgateway after the Entra OBO exchange.

Same middle-tier bearer from the client; the gateway silently reuses the exchanged Graph token upstream.

Step 4 — `tools/call graph_me` (OBO in action)

The MCP server’s code is almost embarrassingly simple — that’s the point:

GET https://graph.microsoft.com/v1.0/me
Authorization: Bearer <token agentgateway injected>

Graph /me profile:
 displayName sebastian
 mail sebastian@maniak.io
 userPrincipalName sebastian@maniak.io
 id ed1b752d-d99c-421f-afc3-a6ebc323cd27

Real profile fields, returned by Graph, for the signed-in user. That could only happen if the upstream token’s aud was https://graph.microsoft.com — which proves the exchange worked.

Step 5 — Summary

✓ OBO SUCCESS Graph /me → sebastian sebastian@maniak.io

The client never held a Graph token. The MCP server never held the app secret. The gateway bridged the two.

Where OBO fits among the other patterns

This lab exposes three paths that look similar but solve different problems. Knowing which is which is half of understanding OBO.

Path	What it does	Token exchange?
`/graph-me`	Entra On-Behalf-Of — swap user token for a Graph token	✅ Yes (this article)
`/mcp-secure`	Entra JWT validation + tool-level RBAC only	❌ No — same token forwarded
`/github-elicit`	Interactive elicitation / third-party OAuth store	❌ No — user grants a new consent

The distinction that trips people up is /mcp-secure vs /graph-me. Both start from a valid Entra JWT. But /mcp-secure just checks the token and lets it through (fine when the downstream trusts the same audience). /graph-me transforms it — and you need that transformation precisely when the downstream audience differs, as Graph’s always will.

Why route OBO through a gateway at all?

You could implement OBO inside every MCP server. The reason not to is the same reason you don’t put TLS termination or rate limiting in every service:

Secret containment. The client secret lives in exactly one place — the gateway’s policy. Your MCP servers become dumb HTTP callers that trust an injected header. If one is compromised, no exchange credential leaks with it.
Uniformity. Every backend that needs a Graph token gets it the same way, by attaching one policy. No per-service OAuth libraries, no drift.
Auditability. Every exchange happens at one chokepoint you can log, meter, and reason about.
Blast radius. Rotate the secret, change the scope, or revoke the whole path in one config object — not across a fleet of services.

The MCP server in this demo does one thing: GET /me with whatever bearer it was handed. That simplicity is the product of pushing OBO into the gateway.

Run it yourself

# full narrated demo
./scripts/test-entra-obo.sh

# just decode and show the middle-tier token
./scripts/test-entra-obo.sh token

# quiet mode
QUIET=1 ./scripts/test-entra-obo.sh

The one thing to remember

A valid token is not a universal token. Its audience pins it to one destination, on purpose. On-Behalf-Of is the sanctioned way to cross that boundary — a middle tier proves the user already consented, and the IdP re-mints the identity for a new audience without a second login. Put that exchange in agentgateway and you get the best of both worlds: downstream services stay simple and secretless, while the one component that does hold the secret is the one you can watch, rotate, and lock down.

The client never sees Graph. The server never sees the secret. The gateway sees both, briefly, and only to trade one true token for another.

Code Share: Deploy kagent with Agent Substrate

Mon, 13 Jul 2026 00:00:00 +0000

By Sebastian Maniak

Agent sessions are bursty. A user asks a question, the agent thinks for a few seconds, then the session sits idle for minutes — or hours — waiting on the next turn. Plain Kubernetes handles this badly: an idle pod still books its CPU and memory, and a cold pod takes seconds to come back. Multiply that across thousands of conversations and you’re paying for a lot of nothing.

Agent Substrate flips the model. It decouples the agent session from the pod: idle sessions are checkpointed — full RAM and filesystem, via gVisor — to object storage, the pod returns to a warm pool, and the session resumes sub-second on the next request, exactly where it left off. Think serverless scale-to-zero, but for stateful agents.

kagent is the Kubernetes-native agent control plane. Wire substrate in as its execution layer and a declarative SandboxAgent becomes a gVisor actor instead of a long-running Deployment.

This guide covers three things:

What a substrate is (concepts from learn.agentsubstrate.dev)
What you can actually accomplish with it (use cases)
How to set it up with kagent OSS on a throwaway kind cluster — one-shot script or manual Helm

Runnable code for this path lives in 01-kagent-agent-substrate (or your local kagent-demos clone). Prefer a guided lab? Same flow in the Instruqt workshop.

What is Agent Substrate?

From the official atlas at learn.agentsubstrate.dev:

Agent Substrate is a Kubernetes-native runtime for highly-multiplexed actor workloads — AI agents, sandboxed environments, stateful services. It decouples actor lifecycle from Pods, so a small pool of pre-warmed gVisor workers can host 30× more actors than there are pods, by suspending idle actors to object storage and restoring them on demand.

Kubernetes is excellent at long-running services. It is not excellent at:

Pain	Why K8s struggles	What substrate does
Idle agents	Pods still consume CPU/memory	Suspend to object storage; reclaim the worker
Millions of sessions	API server / etcd not built for that QPS	Actors live in Valkey/Redis, not as one CR per session
Sub-second wake	kube-scheduler + image pull is seconds	Pre-warmed workers; resume bypasses the scheduler
Stateful scale-to-zero	Volumes don’t attach/detach at agent speed	Full RAM + filesystem checkpoint via gVisor

Architecture bet in one sentence: Kubernetes provisions infrastructure; substrate schedules actors.

Core concepts (glossary)

These terms show up everywhere in the UI, CRDs, and grpcurl surface. Definitions match learn.agentsubstrate.dev/concepts.

Term	Meaning
Actor	One logical agent session. Has its own RAM, filesystem, and identity — but is not pinned to a pod. Can suspend on worker A and resume on worker B.
Worker	A pre-warmed pod that hosts at most one actor at a time (`IDLE` or assigned). Fungible hosting slot, not the agent itself.
WorkerPool	CRD for a Deployment of warm workers. You size concurrency by scaling the pool.
ActorTemplate	Immutable “class” for actors (image, entrypoint, pool, snapshot location). Creating one builds a golden snapshot (version 0).
Golden snapshot	Fresh, just-booted image of the template. Brand-new actors restore from this instead of cold-booting.
Snapshot	Checkpoint of RAM + sentry/filesystem state in S3/GCS (zstd). Resume uses demand paging so only touched pages load.
Suspend / Resume	Checkpoint to object storage and free the worker / restore sub-second into an idle worker.
ateapi	Control plane: actor lifecycle, worker assignment, suspend/resume workflows (gRPC).
atenet	L7 router + DNS. Per-request resolve of which worker hosts an actor; triggers resume if suspended.
atelet	Node DaemonSet: pulls images, downloads snapshots, talks to `ateom-gvisor` over a Unix socket.
ateom-gvisor	In-worker helper that shells out to `runsc checkpoint` / `runsc restore`.

Actor lifecycle

CreateActor → SUSPENDED
 │
 ▼
ResumeActor → RESUMING → RUNNING
 │ │
 │ ▼
 │ SuspendActor → SUSPENDING → SUSPENDED
 │
 └── DeleteActor (from SUSPENDED)

Only RUNNING holds a worker. Between chat turns a declarative session is typically SUSPENDED.

Request path (why resume is fast)

On each HTTP hit to an actor:

DNS resolves actorId.actors.resources.substrate.ate.dev → atenet router (not the worker IP).
ExtProc extracts the actor ID and calls ateapi ResumeActor.
ateapi picks an idle worker from Redis (no kube-scheduler), asks atelet to restore the snapshot.
ateom-gvisor runs runsc restore -background — sentry comes up immediately; pages fault in on demand.
Router rewrites :authority to the worker pod and forwards the request.

Warm path (already RUNNING): skip restore, just route. Cold path (SUSPENDED): restore + route. No idle pod tax either way.

Deep dive: Resume actor end-to-end · System topology.

What can you accomplish? (use cases)

Substrate is framework-agnostic at the OCI/gVisor layer. With kagent OSS on top, these are the practical outcomes.

1. Dense multi-session chat agents (this guide)

Many concurrent declarative conversations without one Deployment per session.

Each UI/chat session becomes a short-lived actor.
After the turn, the actor snapshots back; the worker serves the next session.
One small WorkerPool multiplexes far more sessions than it has pods (~30× oversubscription is the design point).

You get: serverless economics for stateful chat, with memory and filesystem preserved across turns.

2. Sandboxed cluster assistants (Kubernetes tools in a box)

Run a SandboxAgent with MCP tools (k8s_get_resources, logs, events) inside gVisor.

Isolation: hostile or buggy tool use stays in the sandbox.
Density: idle assistants don’t pin CPUs.
Identity: actor is a first-class substrate identity, not “whatever ServiceAccount the pod had forever.”

This demo’s hello-substrate is exactly that pattern — a Kubernetes assistant actor on the default pool.

3. Coding harnesses and long-lived sandboxes (AgentHarness)

kagent can place OpenClaw / Hermes-style harnesses on substrate (runtime: substrate).

Shared actor for a coding session or workspace.
Pins a worker while active (scale the pool: roughly 1 + active harnesses for headroom).
Strong isolation for shell, git, and untrusted code.

See the kagent AgentHarness docs after this walkthrough.

4. Agent swarms / many-actor demos

Substrate’s own demos (e.g. many stateful counter actors on few workers) show the multiplexing thesis: hundreds of stateful actors on a handful of pods.

You get: a path toward “millions of idle agents, thousands of wakeups/sec” without etcd holding every session.

5. Security-minded agent platforms

gVisor (or micro-VM class) per actor.
Egress can sit behind agentgateway so agents never hold provider keys (Solo’s enterprise pattern).
Snapshot-to-storage means you can reclaim compute without losing session state.

North-star metrics from the architecture docs: ~100 ms activation p95, extreme scale of idle actors, high wakeup throughput. Treat those as direction, not a SLA for the kind lab.

Architecture on kind

What you will run:

┌──────────────────────────── kind cluster (kagent-substrate) ────────────────────────────┐
│ │
│ namespace: kagent namespace: ate-system │
│ ┌────────────────────┐ ┌──────────────────────────────────────┐ │
│ │ kagent-controller │ controller.substrate.* │ ate-api-server (scheduling) │ │
│ │ kagent-ui │ ───────────────────────▶│ atenet-router (L7 routing) │ │
│ │ SandboxAgent │ │ │ atelet (node supervisor) │ │
│ │ hello-substrate │ │ │ valkey-cluster (state) │ │
│ │ WorkerPool │ │ │ rustfs (snapshots/S3) │ │
│ │ kagent-default │ │ └──────────────────────────────────────┘ │
│ └────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────────────────────┘

Layer	Role
Agent Substrate (`ate-system`)	Multiplex actors onto warm workers; suspend/resume via gVisor + object storage.
kagent (`kagent`)	Agent CRDs, UI, model provider (OpenAI); substrate as execution layer.
`SandboxAgent`	Declarative agent → substrate actor (not a long-running Deployment).

Pinned versions (this guide)

Component	Version
Agent Substrate	`0.0.8`
kagent (OSS)	`0.10.0-beta6`
gVisor actor image	`ghcr.io/kagent-dev/substrate/ateom-gvisor:v0.0.8`
kind	`v0.31.0`
Node image	`kindest/node:v1.35.0` (any 1.31+ works)

Pairing matters. Substrate 0.0.8 matches kagent 0.10.0-beta6 ateapi/CreateActor protos so UI chat works. Substrate 0.0.9 broke that wire format — only bump together with a matching kagent.
Node image matters. Substrate CRDs use CEL format.dns1123Label / dns1123Subdomain (Kubernetes 1.31+). Old kind defaults reject the CRDs.

Prerequisites

Linux host or VM (~8 vCPU / 16 GB RAM). Valkey (6) + control plane is heavy for a laptop-sized VM.
Docker running
OpenAI API key (real LLM calls)
Optional: grpcurl + jq for the control-plane section

macOS / Windows: prefer a cloud Linux VM (n1-standard-8, m5.2xlarge, …). Docker Desktop often fails the gVisor checkpoint path.

Quick start (one-shot script)

From the demo directory:

export OPENAI_API_KEY="sk-..."
# optional: cp .env.example .env && edit .env

chmod +x setup.sh teardown.sh
./setup.sh

setup.sh installs tools if missing, creates kind, installs substrate + kagent (wired), and applies manifests/hello-substrate.yaml.

Open the UI:

kubectl -n kagent port-forward svc/kagent-ui 8080:8080
# → http://localhost:8080 → Agents → hello-substrate

Tear down:

./teardown.sh

Script knobs

Env var	Default	Meaning
`OPENAI_API_KEY`	(required)	Model provider key
`KIND_CLUSTER`	`kagent-substrate`	kind cluster name
`SUBSTRATE_VERSION`	`0.0.8`	Substrate chart
`KAGENT_VERSION`	`0.10.0-beta6`	kagent chart
`WORKER_POOL_REPLICAS`	`2`	Warm workers (2 helps golden-snapshot blue-green)
`SKIP_TOOLS=1`	off	Don’t auto-install kubectl/helm/kind/…
`SKIP_CLUSTER=1`	off	Reuse existing cluster
`SKIP_AGENT=1`	off	Skip applying the sample agent

Manual walkthrough (same path as the script)

Env

export OPENAI_API_KEY="sk-..."
export KIND_CLUSTER=kagent-substrate
export SUBSTRATE_VERSION=0.0.8
export KAGENT_VERSION=0.10.0-beta6

⚠️ Do not write OPENAI_API_KEY=... helm ... --set ...="${OPENAI_API_KEY}" on one line — the variable expands before the assignment and you silently install with an empty key.

Step 1 — kind cluster

kind create cluster --name "${KIND_CLUSTER}" --image kindest/node:v1.35.0 --wait 120s
kubectl get nodes

Step 2 — Agent Substrate

JWT auth (ServiceAccount tokens) is the chart default — no feature gates.

helm upgrade --install substrate-crds \
 oci://ghcr.io/kagent-dev/substrate/helm/substrate-crds \
 --version "${SUBSTRATE_VERSION}" \
 --namespace ate-system --create-namespace --wait

helm upgrade --install substrate \
 oci://ghcr.io/kagent-dev/substrate/helm/substrate \
 --version "${SUBSTRATE_VERSION}" \
 --namespace ate-system --wait --timeout 10m

kubectl get pods -n ate-system

Expect ate-api-server, ate-controller, atelet-*, atenet-router, valkey-cluster-0..-5, rustfs Running (plus Completed init Jobs).

Pod	Role
`ate-api-server`	Control plane: lifecycle, scheduling, suspend/resume
`ate-controller`	Reconciles `WorkerPool` + `ActorTemplate`
`atelet`	Node supervisor: images, sandbox, object storage
`atenet-router`	L7 route to the active worker
`valkey-cluster-*`	Actor/worker state + locks
`rustfs`	In-cluster S3 for snapshots

Step 3 — kagent wired to substrate

Order matters: install substrate first. The kagent controller crash-loops if ateapi is unreachable at startup.

helm upgrade --install kagent-crds \
 oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \
 --version "${KAGENT_VERSION}" \
 --namespace kagent --create-namespace --wait

helm upgrade --install kagent \
 oci://ghcr.io/kagent-dev/kagent/helm/kagent \
 --version "${KAGENT_VERSION}" --namespace kagent --timeout 10m --wait \
 --set providers.default=openAI \
 --set providers.openAI.apiKey="${OPENAI_API_KEY}" \
 --set controller.substrate.enabled=true \
 --set controller.substrate.ateApiEndpoint="dns:///api.ate-system.svc:443" \
 --set controller.substrate.ateApiInsecure=true \
 --set controller.substrate.atenetRouterURL="http://atenet-router.ate-system.svc:80" \
 --set controller.substrate.ateApiTokenFile="/var/run/secrets/tokens/ate-api/token" \
 --set controller.substrate.defaultWorkerPool.namespace=kagent \
 --set controller.substrate.defaultWorkerPool.name=kagent-default \
 --set substrateWorkerPool.create=true \
 --set substrateWorkerPool.name=kagent-default \
 --set substrateWorkerPool.replicas=2 \
 --set substrateWorkerPool.ateomImage=ghcr.io/kagent-dev/substrate/ateom-gvisor:v0.0.8 \
 --set grafana-mcp.enabled=false \
 --set observability-agent.enabled=false

Flag	Purpose
`controller.substrate.enabled=true`	Turn on integration
`controller.substrate.ateApiEndpoint`	Substrate control plane
`controller.substrate.atenetRouterURL`	Request router
`substrateWorkerPool.create=true` + `.replicas=2`	Two warm workers
grafana / observability agents off	Avoid broken MCP when Grafana isn’t installed

If Helm times out on cold start:

kubectl wait deploy/kagent-controller -n kagent --for=condition=Available --timeout=10m

Verify:

kubectl get secret kagent-openai -n kagent
kubectl get workerpools.ate.dev -A # expect kagent/kagent-default

kubectl -n kagent port-forward deploy/kagent-controller 8083:8083 >/tmp/pf-ctrl.log 2>&1 &
sleep 3
curl -s http://localhost:8083/api/substrate/status; echo
# expect "enabled": true
kill %1 2>/dev/null

Step 4 — Deploy a SandboxAgent

Manifest (also in the demo repo as manifests/hello-substrate.yaml):

apiVersion: kagent.dev/v1alpha2
kind: SandboxAgent
metadata:
 name: hello-substrate
 namespace: kagent
spec:
 type: Declarative
 description: A Kubernetes assistant running inside a substrate gVisor actor
 declarative:
 # Go runtime is required for gVisor checkpoint/restore on substrate.
 runtime: go
 modelConfig: default-model-config
 systemMessage: |
 You are a helpful Kubernetes assistant running inside an Agent Substrate
 gVisor actor. Use the Kubernetes tools to answer questions about the
 cluster. When asked who you are, say "I am a Kubernetes agent running
 inside a gVisor actor on Agent Substrate." Keep answers concise. 
 tools:
 - type: McpServer
 mcpServer:
 name: kagent-tool-server
 kind: RemoteMCPServer
 apiGroup: kagent.dev
 toolNames:
 - k8s_get_resources
 - k8s_describe_resource
 - k8s_get_pod_logs
 - k8s_get_events
 substrate:
 workerPoolRef:
 name: kagent-default

kubectl apply -f manifests/hello-substrate.yaml
kubectl wait sandboxagent/hello-substrate -n kagent --for=condition=Ready --timeout=5m

kubectl get sandboxagent -n kagent
kubectl get actortemplates.ate.dev -n kagent
kubectl get pods -n kagent -l ate.dev/worker-pool=kagent-default -o wide

The first golden snapshot takes ~60–90s. kagent projects the SandboxAgent into an owned ActorTemplate (and secrets); you don’t hand-write the ate.dev CRDs for the happy path.

Step 5 — Chat and watch suspend / resume

kubectl -n kagent port-forward svc/kagent-ui 8080:8080
# http://localhost:8080 → Agents → hello-substrate

The Agents view lists every agent across namespaces — hello-substrate (our declarative Kubernetes assistant on the default pool) sits alongside the built-in kagent library (k8s, Helm, Istio, Cilium, kgateway, …). Open it to start a conversation.

Try:

What are you, and where are you running? Answer in one sentence.
List pods in ate-system.

Between requests the session actor should sit Suspended (UI View → Substrate, or CLI below). Next message restores it sub-second.

The View → Substrate page is where the multiplexing thesis becomes visible: the kagent-default WorkerPool (2 replicas on ateom-gvisor:v0.0.8), the hello-substrate ActorTemplate (READY, sandbox class gvisor, backed by its golden snapshot), the live actor flipping to SUSPENDED between turns, and both workers sitting idle — no pod tax while the session waits.

Drive ate-api with grpcurl

kubectl port-forward -n ate-system svc/api 18443:443 >/tmp/pf-api.log 2>&1 &
sleep 3
TOKEN=$(kubectl create token kagent-controller -n kagent --audience=api.ate-system.svc --duration=15m)

grpcurl -insecure -H "authorization: Bearer $TOKEN" -d '{}' \
 localhost:18443 ateapi.Control/ListWorkers

grpcurl -insecure -H "authorization: Bearer $TOKEN" -d '{}' \
 localhost:18443 ateapi.Control/ListActors

ACTORS_JSON=$(grpcurl -insecure -H "authorization: Bearer $TOKEN" -d '{}' \
 localhost:18443 ateapi.Control/ListActors)
ACTOR_ID=$(echo "$ACTORS_JSON" | jq -r '.actors[0].actorId // empty')
ATESPACE=$(echo "$ACTORS_JSON" | jq -r '.actors[0].atespace // empty')

# ResumeActor expects actor_ref { atespace, name } — not a bare actor_id field.
if [ -n "${ACTOR_ID:-}" ] && [ -n "${ATESPACE:-}" ]; then
 grpcurl -insecure -H "authorization: Bearer $TOKEN" \
 -d "{\"actor_ref\":{\"atespace\":\"$ATESPACE\",\"name\":\"$ACTOR_ID\"}}" \
 localhost:18443 ateapi.Control/ResumeActor
fi

Scale the WorkerPool

One worker can serve many sequential declarative sessions (each releases the slot after snapshot). Scale when you need overlapping sessions or harnesses that pin a slot:

kubectl scale workerpool kagent-default -n kagent --replicas=3

How kagent and substrate fit together

 User / UI
 │
 ▼
 kagent (Agent / SandboxAgent CRDs, model config, tools)
 │ CreateActor / Resume on chat
 ▼
 Agent Substrate (ateapi + atenet + atelet + workers)
 │ gVisor sandbox per assigned actor
 ▼
 Your agent binary (Go ADK) + MCP tools

Without substrate: kagent runs agents as ordinary Kubernetes Deployments (always-on pods).
With substrate: SandboxAgent + spec.substrate.workerPoolRef → actor on the pool; suspend when idle.

Christian Posta’s write-up frames the product story: agents are long-lived but idle; you need isolation (gVisor/Firecracker) and real lifecycle (suspend, snapshot, resume). Substrate + kagent is that path open-sourced under the kagent umbrella. See also Solo’s blog and the official kagent Agent Substrate example.

Known issues & troubleshooting

Issue	Detail
Default pairing	Pin substrate 0.0.8 + kagent 0.10.0-beta6 for working chat protos.
Empty OpenAI key	Install “succeeds” without `kagent-openai` → `CreateContainerConfigError`. Export key, re-run `./setup.sh` or Helm.
Controller crash-loop	Substrate not ready when kagent starts — fix `ate-system` first.
CRDs rejected	Node image < 1.31 — recreate with `kindest/node:v1.35.0`.
TLS on idle kind	In-cluster certs can expire after ~24h — run in one sitting.
Pre-1.0	APIs will change; not production-ready.

Symptom	Fix
Substrate CRDs rejected	Recreate cluster with k8s 1.31+ node image
`kagent-controller` crash-loops	All `ate-system` pods Running before kagent
Chat fails / ConfigError	`kubectl get secret kagent-openai -n kagent`
Helm timeout on kagent	`kubectl wait deploy/kagent-controller … --timeout=10m`
`ListActors` auth errors	Re-mint token (15m)
SandboxAgent not Ready	`WORKER_POOL_REPLICAS>=2`; check workers + ActorTemplate status

Cleanup

./teardown.sh
# or: kind delete cluster --name kagent-substrate

What’s next

Path	Why
`01-kagent-agent-substrate`	Runnable code for this guide: `setup.sh`, `teardown.sh`, manifests
learn.agentsubstrate.dev	Visual topology, resume flow, ateapi internals
AgentHarness on kagent	Long-lived coding sandboxes on substrate
agentgateway	Govern LLM/MCP egress; keep keys off the actor
kagent docs	Agents, tools, MCP, observability

Suspend-and-resume is the feature that finally makes per-session stateful agents affordable at scale on Kubernetes. Spin it up, chat with hello-substrate, and watch a small worker pool serve far more sessions than it has pods.

The 8 Principles of an AI Gateway (and Why They Matter) — with agentgateway from Solo

Sun, 12 Jul 2026 00:00:00 +0000

You wouldn’t put a hundred microservices into production with no gateway, no auth, no rate limits, and no logs. Yet that is exactly how most teams ship AI agents today: an API key baked into a container, a direct line to an LLM, and a growing pile of MCP tool servers that nobody is governing.

Agents don’t fail the way microservices fail. A microservice returns a 500 and you page someone. An agent quietly sends your customer table to a third‑party model, loops a thousand times on a bad prompt and burns your monthly budget before lunch, or discovers a tool you never meant to expose and happily calls it. The failure modes are new, so the control plane has to be new too.

That is what an AI gateway is for. Not “an API gateway with an LLM route” — a purpose‑built data plane that understands LLM traffic, MCP sessions, and agent‑to‑agent calls, and applies policy to all three.

This post walks through the eight principles an AI gateway has to get right, why each one matters in production, and how agentgateway — the open‑source (CNCF) project, packaged and hardened by Solo.io as Enterprise agentgateway — implements them. Every principle gets a why and a how, with config you can read.

If you want the “why does this exist at all” version first, start with What is agentgateway.dev? and Why your AI agents need a gateway.

The eight principles at a glance

#	Principle	The problem it solves
1	Unified LLM access (one API, any provider)	Every provider has a different SDK, auth, and payload. Apps get locked to one vendor.
2	MCP tool federation (stdio/HTTP/SSE/OpenAPI)	Each tool server is a separate connection, auth surface, and thing to secure.
3	Secure A2A agent discovery & collaboration	Agents calling agents with no identity, no scoping, no audit.
4	Built‑in guardrails (regex / moderation / webhooks)	PII, secrets, and prompt injection flow straight through to the model.
5	Multi‑auth security (JWT / OAuth / RBAC / CEL)	One coarse API key can do everything. No per‑user, per‑tool authorization.
6	Rate limiting, budgets & failover routing	One runaway loop drains the budget; one provider outage kills every agent.
7	Full observability (OpenTelemetry / TLS)	No traces, no token accounting, no idea what a prompt cost or where it went.
8	YAML policy‑driven config	Governance living in app code and dashboards can’t be reviewed, versioned, or rolled back.

None of these is optional at scale. Skip one and it becomes the incident. Let’s take them one at a time.

1. Unified LLM access — one API, any provider

Why it matters. The moment you have more than one model provider — and you will, the day one provider has an outage or another ships something better — you feel the tax. Anthropic, OpenAI, Gemini, Bedrock, Vertex, and your self‑hosted vLLM each have their own SDK, auth headers, request shape, streaming semantics, and error codes. If each application wires directly to each provider, you’ve hard‑coded vendor lock‑in into every service, and swapping a model means a code change and a redeploy.

An AI gateway makes this a routing concern. Apps speak one stable API to the gateway — the widely‑adopted OpenAI‑style schema is the common wire format, but the point isn’t OpenAI, it’s one front door — and the gateway translates to whatever provider sits behind the route: Anthropic today, Gemini tomorrow, a local model for the cheap path. Changing or mixing models becomes a config change, not a code change, and no application is coupled to any single vendor.

How agentgateway does it. A backend declares the provider; a route maps a path prefix to that backend and normalizes the API surface. Here’s a single standalone config front‑ending both Anthropic and OpenAI behind one port:

binds:
- port: 4001

listeners:
- name: default
 protocol: HTTP

routes:
- name: claude
 matches:
 - path:
 pathPrefix: /claude
 policies:
 urlRewrite:
 path:
 prefix: /
 backends:
 - ai:
 name: claude
 provider:
 anthropic: {}

- name: openai
 matches:
 - path:
 pathPrefix: /openai
 policies:
 urlRewrite:
 path:
 prefix: /
 backends:
 - ai:
 name: openai
 provider:
 openAI:
 model: gpt-4o-mini

Your application points OPENAI_BASE_URL (or ANTHROPIC_BASE_URL) at the gateway and never holds a provider key again. On Kubernetes the same idea is expressed as an AgentgatewayBackend with an ai.provider block and an HTTPRoute — see Your first AI route. The payoff shows up later: because everything is normalized here, principles 6 (failover), 7 (token accounting), and 8 (policy) get to work on a single, consistent stream instead of N provider dialects.

2. MCP tool federation (stdio / HTTP / SSE / OpenAPI)

Why it matters. The Model Context Protocol is how agents get hands — GitHub, Slack, a database, an internal API. But MCP servers multiply fast, and each one is a separate connection, a separate auth story, and a separate thing to monitor. Worse, MCP is not plain request/response: it’s stateful JSON‑RPC over long‑lived sessions, with server‑initiated SSE events that must route back to the correct client session. A path‑based reverse proxy can’t do that correctly.

Federation means the gateway presents one MCP endpoint to the agent and multiplexes it across many backend tool servers — regardless of whether a given server speaks stdio, streamable HTTP, SSE, or is really an OpenAPI service wrapped as MCP. The agent sees one tool catalog; the gateway owns the fan‑out, the session affinity, and the per‑target security.

How agentgateway does it. A single MCP listener with multiple targets, each using whatever transport the server actually speaks:

mcp:
 port: 3000
 targets:
 - name: everything
 stdio:
 cmd: docker
 args: ["exec", "-i", "mcp-everything", "node", "dist/index.js"]
 - name: github
 http:
 host: github-mcp.tools.svc.cluster.local
 port: 8080
 - name: internal-api
 openapi:
 schema: /config/openapi/orders.yaml
 host: orders.internal.svc.cluster.local
 port: 80

The agent connects once; agentgateway keeps the JSON‑RPC session coherent, routes SSE events back to the right client, and — critically — lets you decide which tools each caller may even see. That last part is where federation meets authorization (principle 5). For the deep version of multiplexing and how tool exposure affects token cost, see MCP multiplexing and GitHub MCP token economics.

3. Secure A2A agent discovery & collaboration

Why it matters. The next step past “agent calls tools” is “agent calls agent.” A planner delegates to a researcher; the researcher calls a summarizer. A2A standardizes that handoff. But without a gateway in the middle, agent‑to‑agent traffic is the wild west: no shared identity, no way to scope what one agent may ask another to do, and no audit trail when a delegated call does something expensive or sensitive. Multi‑agent systems fail between the agents, and that seam is exactly where nobody is looking.

An AI gateway treats A2A as first‑class traffic: agents are discoverable through the gateway, every cross‑agent call carries identity, and the same auth, rate‑limit, and observability policies that guard LLM and MCP traffic also guard the handoffs. The blast radius of a misbehaving agent stays contained because there’s a kill switch and a policy boundary at the seam.

How agentgateway does it. A2A is routed and secured like any other backend — an A2A route carries JWT identity forward, applies RBAC on what the calling agent is allowed to invoke, and emits the same traces as everything else. The multi‑agent kill‑switch pattern (revoke one agent’s access at the gateway and it’s instantly cut off from peers and tools) is covered in Multi‑agent architecture + kill switch, and a full A2A/MCP stack running on kagent is in AgentX.

4. Built‑in guardrails (regex / moderation / webhooks)

Why it matters. This is the principle that keeps security teams up at night. Between the user and the model — in both directions — you need a decision point that can see the content and act on it: redact a credit card, block a prompt‑injection attempt, drop a response that leaks an internal hostname, or send the payload to an external moderation service and honor its verdict. Bolt this into every app and you get inconsistent coverage and a dozen places to audit. It belongs in the gateway, inline on the request and response path.

Guardrails come in layers, and a good gateway supports all three:

Regex / pattern rules for the cheap, deterministic cases (SSNs, API‑key shapes, email addresses).
Moderation via a model or classification service for the fuzzy cases (toxicity, jailbreaks, category policy).
Webhook callouts to an external decision engine — e.g. F5 AI Guardrails — for enterprise‑grade scanning, redaction, and audit, either inline (block before it reaches the model) or out‑of‑band.

How agentgateway does it. Guardrail policies attach to a route and run on the prompt and/or the completion:

routes:
- name: openai
 matches:
 - path:
 pathPrefix: /openai
 policies:
 ai:
 promptGuard:
 request:
 regex:
 rules:
 - pattern: '\b\d{3}-\d{2}-\d{4}\b' # US SSN
 action: reject
 webhook:
 host: f5-guardrails.security.svc.cluster.local
 port: 8080
 response:
 regex:
 rules:
 - pattern: '(?i)internal[- ]only'
 action: mask
 backends:
 - ai:
 name: openai
 provider:
 openAI: {}

The point is that a blocked request returns a clean, inspectable decision — security sees why it was blocked, the app sees a status code, and nothing leaked. The full inline + out‑of‑band F5 pattern is in agentgateway + F5 AI Guardrails architectures and the hands‑on UI walkthrough is here.

5. Multi‑auth security (JWT / OAuth / RBAC / CEL)

Why it matters. A single API key is a skeleton key: whoever holds it can do everything the key can do. That’s fine for a demo and catastrophic in production. Real systems need who (authentication), what are they allowed to do (authorization), and the ability to express that at the granularity of “this team’s agents may call these three MCP tools and this model, but not that one.” Coarse keys can’t express that. You need identity flowing through the gateway and policy evaluated per request.

An AI gateway should support:

JWT validation — verify tokens, check issuer/audience, extract claims.
OAuth — front the gateway with an IdP (Okta, Keycloak, Entra) so agents authenticate as real principals.
RBAC — map identities/claims to roles, and roles to what they may reach.
CEL (Common Expression Language) — fine‑grained, attribute‑based rules like “allow only if jwt.groups contains platform and the requested tool isn’t in the destructive set.”

How agentgateway does it. JWT auth is a policy on the listener or route; RBAC and CEL express which authenticated principals may reach which backends or MCP tools:

routes:
- name: github-mcp
 matches:
 - path:
 pathPrefix: /mcp
 policies:
 jwtAuth:
 issuer: https://login.example.com/
 audiences: ["agentgateway"]
 jwks:
 url: https://login.example.com/.well-known/jwks.json
 authorization:
 rules:
 # only the platform group may use write-capable tools
 - 'jwt.groups.exists(g, g == "platform") || !request.mcp.tool.startsWith("create_")'
 backends:
 - mcp:
 name: github

Identity established at the gateway then travels with the request into MCP and A2A calls (principles 2 and 3), so authorization is consistent end to end instead of re‑invented per hop. For a full OAuth chain into a cloud runtime, see AWS AgentCore with agentgateway and Okta OAuth.

6. Rate limiting, budgets & failover routing

Why it matters. LLM traffic has a property normal traffic doesn’t: every request costs money, and a bad one can cost a lot. An agent stuck in a reasoning loop doesn’t crash — it spends. So the gateway needs two knobs traditional gateways never had:

Budgets — a hard ceiling in dollars or tokens per user, per team, per agent. When it’s hit, the gateway returns 429, not a surprise invoice. This is FinOps as a policy, not a monthly spreadsheet.
Failover routing — provider A is the primary; on error, timeout, or overload, the gateway transparently fails over to provider B (or a cheaper/local model) without the app knowing. One outage stops being one outage for every agent.

Plus classic rate limiting (requests per second per principal) to protect both your budget and the upstream provider’s limits.

How agentgateway does it. Budgets attach a cost model and a ceiling; the gateway meters real token usage against it. Failover is expressed as an ordered/weighted backend list with health‑aware routing:

routes:
- name: chat
 matches:
 - path:
 pathPrefix: /chat
 policies:
 rateLimit:
 local:
 maxTokens: 100
 tokensPerFill: 100
 fillInterval: 1m
 ai:
 budget:
 currency: USD
 limit: 50.00 # $50/day hard cap
 interval: 24h
 onExceeded: reject  # -> 429
 backends: # failover: primary first, fallback second
 - ai:
 name: primary
 provider:
 openAI:
 model: gpt-4o
 weight: 100
 - ai:
 name: fallback
 provider:
 anthropic:
 model: claude-sonnet-5
 weight: 0 # only used when primary is unhealthy

The hard‑spend‑limit behavior — model cost catalogs, dollar/token budgets, and a real 429 on exhaustion — is walked through in AI budgets & hard spend limits, and the “how much headroom do you actually have” analysis is in Headroom: MCP token stacking.

7. Full observability (OpenTelemetry / TLS)

Why it matters. You cannot govern what you cannot see. With agents, the questions are specific and expensive to answer without instrumentation: What prompt was sent? What did it cost? Which model, which tool, which agent? Where did latency come from — the model, the tool call, or the gateway? If observability is bolted on per app, you get inconsistent, non‑comparable data. If it lives in the gateway, every request — LLM, MCP, A2A — emits the same OpenTelemetry traces, metrics, and logs, complete with token counts and cost, and every hop is encrypted with TLS.

This is the principle that makes the other seven auditable. Budgets (principle 6) are only trustworthy because token accounting is real. Guardrail decisions (principle 4) are only defensible because they’re logged. RBAC (principle 5) is only verifiable because you can see who called what.

How agentgateway does it. Tracing is a gateway‑level policy that exports OTLP to any collector — Langfuse, Tempo, Jaeger, an OTel Collector fan‑out:

config:
 tracing:
 otlp:
 endpoint: http://otel-collector.observability.svc.cluster.local:4317
 protocol: grpc
 # token usage and cost are attached to spans automatically

Because the gateway sits on the normalized stream from principle 1, the spans carry gen_ai.* attributes — model, input/output tokens, cost — consistently across providers. The end‑to‑end Langfuse integrations (direct OTLP and via an OTel Collector) are in agentgateway + Langfuse direct OTLP and via the OTel Collector, and the cost dashboards built on that data are in the cost & tokenomics dashboard.

8. YAML policy‑driven config

Why it matters. The previous seven principles are only as good as the place they’re defined. If routing, auth, guardrails, and budgets live in application code and clicked‑together dashboards, then your governance is un‑reviewable, un‑versioned, and un‑reproducible. Someone changes a limit in a UI at 2am and there’s no diff, no approval, no rollback.

Declarative YAML flips that. Every policy — every route, backend, guardrail, JWT rule, budget, and trace exporter — is text. Text goes in Git. Git gives you pull‑request review, history, blame, environments, and one‑command rollback. Your AI governance becomes a GitOps artifact: the same config runs on a laptop (standalone) and in production (Kubernetes CRDs), and “what is our policy?” has a single, diffable answer.

How agentgateway does it. Everything in this post was YAML. That’s the point — there is no hidden imperative layer. Standalone is a single config.yaml; on Kubernetes the same model is expressed as Gateway API and agentgateway.dev CRDs, reconciled by a controller. Both are declarative, both live in Git. The GitOps pattern for agents is in GitOps for agents, and quickstarts for both surfaces are standalone and Kubernetes.

Why these eight, together

Any one of these principles is useful on its own. The reason they belong in one gateway is that they compound:

Unified access (1) gives you one normalized stream to apply everything else to.
MCP federation (2) and A2A (3) put tools and agent handoffs on that same stream.
Guardrails (4) and multi‑auth (5) turn the stream into a policy enforcement point — content and identity.
Budgets and failover (6) make it resilient and financially safe.
Observability (7) makes all of it auditable and debuggable.
YAML (8) makes the whole thing reviewable, versioned, and portable.

Pull any one out and you reopen a gap: no observability and your budgets are guesswork; no unified access and your policies fork per provider; no YAML and none of it is reproducible. That’s why “an API gateway with an LLM route” isn’t enough — the AI‑native concerns (tokens, sessions, tools, agents, cost) have to be first‑class.

Where to start

You don’t have to adopt all eight on day one. A sane path:

Front your LLM calls through the gateway (principle 1) — kill the scattered API keys first.
Turn on observability (principle 7) — you can’t tune what you can’t see.
Add budgets and rate limits (principle 6) — cap the downside.
Federate your MCP tools (principle 2) and put guardrails + auth (4, 5) on them.
Bring in A2A (principle 3) as your agents start delegating.
Commit all of it to Git (principle 8) from the very first step.

agentgateway is open source and CNCF — you can run the whole thing on a laptop today. When you need the enterprise hardening (the F5 AI Guardrails integration, the Solo Enterprise UI, cost management, supported CRDs), Solo packages it as Enterprise agentgateway.

Project: https://github.com/agentgateway/agentgateway
Docs: https://agentgateway.dev/docs/
Solo.io: https://www.solo.io/

The gateway era of microservices taught us that governance can’t be per‑service and after‑the‑fact. Agents raise the stakes — the blast radius now includes your data, your budget, and your users. These eight principles are how you keep that in bounds.

Build Your Own X Agent: SpaceXAI + X MCP Through agentgateway and kagent

Sat, 11 Jul 2026 00:00:00 +0000

What if you could ask an AI agent “What’s trending in the US right now?” or “Summarize the last week of posts about agentgateway” — and get real answers grounded in live X (Twitter) data, not a hallucinated timeline?

That’s AgentX: a kagent agent powered by SpaceXAI (xAI Grok-4.5), tools from the official X MCP server (api.x.com/mcp), and traffic that all flows through Solo Enterprise AgentGateway. Trends, news, search, users, timelines, mentions, and bookmark curation — with human-in-the-loop approval before any bookmark write lands on your account.

Everything below is running in my lab GitOps repo:

👉 sebbycorp/k8s-goose · Live map: goose.maniak.ai

Here’s AgentX in the Solo Enterprise UI — grok-4.5, ready for research:

Why this stack

X is still where news, tech discourse, and media break first. Scraping it yourself is fragile. Sticking an API key in a notebook agent is worse — no audit trail, no rate-limit story, no shared gateway.

The pattern I want for every agent in the lab is the same one I used for GitHub MCP, FortiGate, and the Telegram multi-agent bot:

Layer	Role
SpaceXAI (Grok-4.5)	The brain — strong at social tone, trends, and summarization
Official X MCP	The hands — tools against `api.x.com/mcp`
agentgateway	The front door — MCP proxy at `/x`, LLM proxy at `/grok`, traces + cost
kagent AgentX	The product surface — system prompt, tool allowlist, HITL
Vault + ESO	Secrets never live in Git

You talk to the agent. The agent never holds an X API key or an xAI key in plain YAML. Credentials stay in Vault; the OAuth user token stays on a PVC inside a bridge pod that only agentgateway can reach.

Architecture

The official X MCP bridge (@xdevplatform/xurl) speaks stdio. agentgateway and kagent expect Streamable HTTP MCP. So we wrap stdio with supergateway, then front the whole thing with the virtual MCP gateway — same shape as every other MCP server in k8s-goose.

You (Solo UI / A2A / Telegram later)
 │
 ▼
┌───────────────────┐ ModelConfig xai-grok
│ kagent AgentX │ ──────────────────────────▶ xai-grok-gateway ──▶ api.x.ai (Grok-4.5)
│ (ns: kagent) │
└─────────┬─────────┘
 │ RemoteMCPServer x-mcp
 ▼
┌───────────────────┐
│ virtual-mcp- │ HTTPRoute /x
│ gateway │
└─────────┬─────────┘
 │ AgentgatewayBackend x-mcp
 ▼
┌───────────────────┐
│ x-mcp-bridge pod │ supergateway --stdio "xurl mcp https://api.x.com/mcp"
│ :8080/mcp │ HOME=/data → PVC (OAuth token)
│ CLIENT_ID/SECRET │ ← Secret x-app-credentials (ESO ← Vault agentgateway/x)
└─────────┬─────────┘
 ▼
 api.x.com/mcp

Important limitation (as of this write-up): the official X MCP surface AgentX uses is read + bookmarks. Trends, news, search, post/user lookup, timelines, mentions, and bookmark folders work. Posting, replying, liking, and media upload are not exposed by this MCP server. AgentX is honest about that in its system prompt — it will draft text for you and bookmark sources, but it will not pretend it can tweet.

What AgentX can do

Category	Tools (subset)	Notes
Trends & news	`get_trends_by_woeid`, `search_news`, `get_news`	WOEID `1` = worldwide, `23424977` = USA
Posts & search	`search_posts_all`, `get_posts_by_id(s)`, `get_posts_counts_recent`, engagement tools	Full-archive search needs a paid X API tier
Users	`get_users_me`, `get_users_by_username(s)`, `search_users`	Research accounts cleanly
Timelines	`get_users_timeline`, `get_users_posts`, `get_users_mentions`	Home timeline + mentions for the OAuth user
Bookmarks (write)	`create_users_bookmark`, `delete_users_bookmark`, `create_users_bookmark_folder`	HITL required
Bookmarks (read)	`get_users_bookmarks`, folders, by-folder	Safe curation views

Example prompts that already work in the UI:

“What’s trending in the US right now?”
“Search recent posts about AI agents and summarize the themes”
“Look up @SebbyCorp and summarize their recent posts”
“How many posts in the last week mention ‘agentgateway’?”
“Create a bookmark folder called ‘AI agents’ and save these posts”

Prerequisites

You need a cluster where agentgateway and kagent are already healthy (or follow the full platform deploy in k8s-goose). For AgentX specifically:

X Developer app at console.x.com with OAuth 2.0
- App permissions: Read and write (bookmarks need write)
- Callback URI: http://localhost:8080/callback
- Paid tier (Basic+) if you want full search / archive behavior
SpaceXAI / xAI API key already wired through agentgateway (ModelConfig xai-grok)
Vault + External Secrets Operator (or a plain Secret if you’re not on Vault yet)
A storage class for a 1Gi RWO PVC (Longhorn in my lab)

Part 1 — X app credentials in Vault → Kubernetes

Never commit client secrets. Seed Vault, let ESO materialize the Secret the bridge will mount.

# Example only — use YOUR client_id / client_secret from console.x.com
kubectl exec -i -n vault vault-0 -- vault kv put agentgateway/x \
 client_id='YOUR_CLIENT_ID' \
 client_secret='YOUR_CLIENT_SECRET'

ExternalSecret (from config/external-secrets/x-app-external-secret.yaml):

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
 name: x-app-credentials
 namespace: agentgateway-system
spec:
 refreshInterval: "1h"
 secretStoreRef:
 name: vault
 kind: ClusterSecretStore
 target:
 name: x-app-credentials
 creationPolicy: Owner
 data:
 - secretKey: CLIENT_ID
 remoteRef:
 key: agentgateway/x
 property: client_id
 - secretKey: CLIENT_SECRET
 remoteRef:
 key: agentgateway/x
 property: client_secret

After sync:

kubectl -n agentgateway-system get secret x-app-credentials

If you use the reboot-safe Vault helper in k8s-goose, the same path is re-seeded by scripts/configure-vault.sh so a wiped dev-mode Vault does not leave AgentX broken after a node restart.

Part 2 — The X MCP bridge pod (xurl + supergateway)

This is the piece that turns official X MCP into something agentgateway can proxy.

From config/mcp-servers/x-mcp-server.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: x-mcp-token
 namespace: agentgateway-system
spec:
 accessModes: ["ReadWriteOnce"]
 storageClassName: longhorn
 resources:
 requests:
 storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: x-mcp-bridge
 namespace: agentgateway-system
 labels:
 app: x-mcp-bridge
spec:
 replicas: 1 # mandatory — RWO PVC + one OAuth owner
 strategy:
 type: Recreate
 selector:
 matchLabels:
 app: x-mcp-bridge
 template:
 metadata:
 labels:
 app: x-mcp-bridge
 spec:
 containers:
 - name: x-mcp-bridge
 image: node:20-alpine
 command: ["sh", "-c"]
 args:
 - >-
 npx -y supergateway
 --stdio "npx -y @xdevplatform/xurl mcp https://api.x.com/mcp"
 --outputTransport streamableHttp
 --stateful
 --sessionTimeout 600000
 --port 8080 
 env:
 - name: HOME
 value: /data
 - name: REDIRECT_URI
 value: http://localhost:8080/callback
 - name: CLIENT_ID
 valueFrom:
 secretKeyRef:
 name: x-app-credentials
 key: CLIENT_ID
 - name: CLIENT_SECRET
 valueFrom:
 secretKeyRef:
 name: x-app-credentials
 key: CLIENT_SECRET
 ports:
 - containerPort: 8080
 volumeMounts:
 - name: token
 mountPath: /data
 readinessProbe:
 tcpSocket: { port: 8080 }
 initialDelaySeconds: 20
 periodSeconds: 10
 resources:
 requests: { cpu: 100m, memory: 256Mi }
 limits: { cpu: 500m, memory: 512Mi }
 volumes:
 - name: token
 persistentVolumeClaim:
 claimName: x-mcp-token
---
apiVersion: v1
kind: Service
metadata:
 name: x-mcp-bridge
 namespace: agentgateway-system
spec:
 selector:
 app: x-mcp-bridge
 ports:
 - protocol: TCP
 port: 8080
 targetPort: 8080
 appProtocol: agentgateway.dev/mcp
 type: ClusterIP

Why replicas: 1 and Recreate? The OAuth refresh token lives on a single RWO volume under /data (HOME=/data → ~/.xurl). Scaling the bridge would fight for the volume and corrupt the token owner story.

Part 3 — One-time OAuth bootstrap (headless, inside the pod)

The bridge cannot mint the first user token without a browser once. Do this after the Deployment is Ready:

kubectl -n agentgateway-system exec -it deploy/x-mcp-bridge -- \
 npx -y @xdevplatform/xurl auth oauth2 --headless

Open the printed authorization URL in a browser
Authorize the X app for your account
The browser redirects to http://localhost:8080/callback?... (the page will fail to load — that’s fine)
Paste the full redirect URL back into the terminal

Confirm the token landed on the PVC and a live call works:

kubectl -n agentgateway-system exec deploy/x-mcp-bridge -- ls -la /data
kubectl -n agentgateway-system exec deploy/x-mcp-bridge -- \
 sh -c 'npx -y @xdevplatform/xurl whoami 2>/dev/null'

You should see your handle / user id. Refresh tokens rotate in place on the PVC — do not delete that volume casually.

Part 4 — Front the bridge with agentgateway

Backend

config/backends/x-mcp.yaml — no auth.secretRef. Auth is already inside the bridge via xurl.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: x-mcp
 namespace: agentgateway-system
spec:
 mcp:
 targets:
 - name: x-bridge
 static:
 host: x-mcp-bridge.agentgateway-system.svc.cluster.local
 port: 8080
 path: /mcp
 protocol: StreamableHTTP

Route

config/routes/x-mcp-route.yaml attaches /x to the virtual MCP gateway (the same aggregator that serves /github, /drone, etc.):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: x-mcp
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: virtual-mcp-gateway
 namespace: agentgateway-system
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /x
 backendRefs:
 - name: x-mcp
 namespace: agentgateway-system
 group: agentgateway.dev
 kind: AgentgatewayBackend

Smoke-test through the gateway (adjust NodePort / IP for your cluster):

curl -sS -X POST http://<worker-node>:31606/x \
 -H 'Content-Type: application/json' \
 -H 'Accept: application/json, text/event-stream' \
 -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'

You should get a JSON-RPC tools list with the X tool names. That’s the contract kagent will discover next.

The Solo UI for agentgateway is the control plane for routes, cost, and playground wiring once backends exist:

After ArgoCD syncs the backend + HTTPRoute, /x shows up on that virtual MCP gateway path — not as a bare LLM playground route, but as a first-class MCP target the rest of the platform can share.

Part 5 — SpaceXAI model path (Grok through the gateway)

AgentX does not call api.x.ai directly. kagent’s ModelConfig points at the dedicated xAI Grok gateway already in the cluster, with a throwaway client key (agentgateway injects the real XAI_API_KEY upstream):

apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
 name: xai-grok
 namespace: kagent
spec:
 provider: OpenAI
 model: grok-4.5
 apiKeySecret: kagent-xai
 apiKeySecretKey: OPENAI_API_KEY
 openAI:
 baseUrl: http://xai-grok-gateway.agentgateway-system.svc.cluster.local/grok/v1

That’s the SpaceXAI half of the story: Grok-4.5 for reasoning, metered and traced by agentgateway, same as every other model in the lab.

Part 6 — kagent RemoteMCPServer + AgentX

RemoteMCPServer

kagent must discover tools through agentgateway, not by talking to the bridge Service directly. That keeps MCP traffic on the governed path (and consistent with GitHub / drone / FortiGate).

config/kagent-models/x-remotemcpserver.yaml:

apiVersion: kagent.dev/v1alpha2
kind: RemoteMCPServer
metadata:
 name: x-mcp
 namespace: kagent
spec:
 description: "X (Twitter) MCP via agentgateway /x — trends, news, post/user search & lookup, timelines, mentions, and bookmarks"
 url: "http://virtual-mcp-gateway.agentgateway-system.svc.cluster.local/x"
 timeout: "30s"
 sseReadTimeout: "5m0s"

Agent CR

config/kagent-models/agentx.yaml (trimmed for readability — full file is in the repo):

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
 name: agentx
 namespace: kagent
spec:
 description: "AI agent for X (Twitter) — analyze trends and news, search and read posts/users, read timelines and mentions, and manage bookmarks"
 declarative:
 runtime: python
 modelConfig: xai-grok
 a2aConfig:
 skills:
 - id: x-research-curation
 name: X (Twitter) Research & Curation
 description: "Analyze trends and news, search and read posts and users, read timelines and mentions, and organize bookmarks on X"
 examples:
 - "What's trending in the US right now?"
 - "Search recent posts about AI agents and summarize the themes"
 - "Bookmark this post for me: <post-id>"
 tags: [x, twitter, social, trends, research]
 systemMessage: |
 You are {{.AgentName}}, an expert X research and curation assistant.
 You operate on the account authenticated in the X MCP bridge.

 ## Capabilities
 Trends, news, search, post/user lookup, timelines, mentions, bookmarks.

 ## Limitation
 This MCP server does NOT provide posting, replying, liking, reposting,
 or media upload. Draft text instead; offer bookmark + research workflows.

 ## Rules
 1. Read before acting — ground claims in tool output.
 2. Confirm every bookmark write.
 3. Never fabricate counts or handles.
 4. Cite post IDs / links.
 5. Map locations to WOEIDs (USA = 23424977, worldwide = 1).
 6. Batch lookups to respect X rate limits.

 Available tools: {{.ToolNames}} 
 tools:
 - type: McpServer
 mcpServer:
 apiGroup: kagent.dev
 kind: RemoteMCPServer
 name: x-mcp
 toolNames:
 - get_trends_by_woeid
 - search_news
 - get_news
 - search_posts_all
 - get_posts_by_id
 - get_posts_by_ids
 - get_posts_counts_recent
 - get_posts_liking_users
 - get_posts_reposted_by
 - get_posts_quoted_posts
 - get_users_me
 - get_users_by_username
 - get_users_by_usernames
 - get_users_by_id
 - search_users
 - get_users_timeline
 - get_users_posts
 - get_users_mentions
 - get_users_bookmarks
 - get_users_bookmark_folders
 - get_users_bookmarks_by_folder_id
 - create_users_bookmark
 - delete_users_bookmark
 - create_users_bookmark_folder
 requireApproval:
 - create_users_bookmark
 - delete_users_bookmark
 - create_users_bookmark_folder

Two details that matter in production:

runtime: python — pins the published kagent app image. The default Go runtime image can ImagePullBackOff depending on your registry digest story; every healthy agent in this lab uses Python.
requireApproval on the three bookmark mutations — HITL at the tool layer, plus the system prompt that asks the model to confirm in natural language first.

Deploy path (GitOps)

In k8s-goose, a push to main is the deploy:

File	ArgoCD owner
`config/external-secrets/x-app-external-secret.yaml`	`agentgateway-config`
`config/mcp-servers/x-mcp-server.yaml`	`agentgateway-config`
`config/backends/x-mcp.yaml`	`agentgateway-config`
`config/routes/x-mcp-route.yaml`	`agentgateway-config`
`config/kagent-models/x-remotemcpserver.yaml`	`kagent-models`
`config/kagent-models/agentx.yaml`	`kagent-models`

Order of operations:

Seed Vault → ExternalSecret → x-app-credentials
Bridge Deployment + PVC + Service
OAuth bootstrap (manual, once)
Backend + /x HTTPRoute
RemoteMCPServer + AgentX
Chat in Solo UI

Verify:

kubectl -n agentgateway-system get deploy,svc,pvc,agentgatewaybackend,httproute | grep x-mcp
kubectl -n kagent get remotemcpserver x-mcp
kubectl -n kagent get agent agentx

Agent Ready, then open the kagent product surface in Solo UI, select agentx, and ask:

What’s trending in the US right now?

You should see a tool call to get_trends_by_woeid (WOEID 23424977) and a clean summary — not a generic model guess.

Security and operational gotchas

Gotcha	Why it bites	Fix
Scale the bridge to 2	RWO PVC + single OAuth token	Keep `replicas: 1`, `strategy: Recreate`
Wipe the PVC	Refresh token is gone	Re-run headless OAuth bootstrap
Dev-mode Vault reboot	`agentgateway/x` disappears	Re-run `configure-vault.sh`, force-sync ExternalSecret
Missing callback URI	OAuth never completes	Register `http://localhost:8080/callback` on the X app
App-only Bearer only	No user context / bookmarks	Use the full xurl OAuth user token path
Secrets in chat or Git	Rotate immediately	Regenerate client secret in console.x.com, re-seed Vault, restart bridge
Expecting AgentX to post	Official MCP doesn’t expose post tools	Draft + bookmark; custom write wrapper is a planned follow-up

Also respect X API product terms and rate limits. Batch with *_by_ids tools when you can. This is a research/curation agent on your account context — treat it like a privileged console, not a public bot.

What this unlocks next

AgentX is deliberately the same shape as the rest of the multi-agent lab:

Add @x to the Telegram multi-agent bot via A2A
Point a scheduled job at morning trend digests (agent-scheduler → PR or chat)
Layer enterprise tool modes (Search / Code / CodeSearch) if the X tool catalog grows large enough that Standard mode burns context
Build a write MCP wrapper on top of xurl CLI for posting — behind HITL, never free-fire

The point of agentgateway in front of MCP is not ceremony. It’s that every tool call is a first-class route: observable, shareable across agents, and isolated from the LLM key path. SpaceXAI handles the intelligence. X MCP handles the media surface. kagent turns both into something you can actually operate.

Repo map

All manifests live here:

Bridge: config/mcp-servers/x-mcp-server.yaml
Backend: config/backends/x-mcp.yaml
Route: config/routes/x-mcp-route.yaml
RemoteMCPServer: config/kagent-models/x-remotemcpserver.yaml
AgentX: config/kagent-models/agentx.yaml
ExternalSecret: config/external-secrets/x-app-external-secret.yaml

Platform overview: sebbycorp/k8s-goose · Interactive docs: goose.maniak.ai

Wrap-up

You now have a repeatable pattern for a personal X research agent:

SpaceXAI Grok-4.5 via agentgateway (xai-grok ModelConfig)
Official X MCP wrapped by supergateway for Streamable HTTP
agentgateway /x on the virtual MCP gateway
kagent AgentX with a curated tool list and HITL on bookmark writes
Vault-backed app credentials and a durable OAuth token on PVC

Ask it what’s trending. Ask it to research a handle. Ask it to build a bookmark folder around a topic. Keep the write surface narrow until you’re ready for a custom poster.

That’s how you stop doomscrolling the firehose — and start running an agent that actually manages the world of news, media, and posts for you.

I Put My Whole Homelab in a Telegram Group Chat — kagent Flies It, agentgateway Governs It

Sat, 11 Jul 2026 00:00:00 +0000

I have a group chat with my infrastructure.

Not a metaphor. I open Telegram, type /use forti what devices are on my wifi?, and a few seconds later my FortiGate firewall answers — in a chat bubble, on my phone, from the couch. Type /use k8s scale the drone-mcp deployment to 2 and my Kubernetes cluster does it, but first it stops and shows me an Approve / Reject button, because that one writes. Type /use drone take off, flip, photograph the room, land and an actual quadcopter in my office leaves the ground.

Six agents. One bot. One phone. It’s the most fun I’ve had with a cluster in a while — and underneath the fun there’s a design I actually care about: the bot holds no brains and no keys. Every agent lives in the cluster, every model call goes through my agentgateway, and anything dangerous waits for my thumb.

Live map: goose.maniak.ai/dispatch.html · Repo: sebbycorp/k8s-goose · Bot: @KagentCorpAIbot

Here’s the whole thing on my phone — /help listing the six agents, and just above it the F5 agent answering “what are my vips?” with 19 VIPs found, all administratively enabled:

The cast

Meet @KagentCorpAIbot. It’s a single polling Deployment in the kagent namespace that speaks to six in-cluster agents. Each one is a real kagent Agent with its own tools, its own MCP servers, and its own blast radius:

Alias	Agent	What it touches
`@k8s`	`k8s-agent` (KubeAssist)	The cluster itself — get/describe/logs/events, plus scale/rollout/patch/apply/delete
`@forti`	`fortigate-agent`	FortiGate `172.16.10.1` — policies, NAT/VIPs, DHCP leases, detected devices, FortiAP wireless
`@f5`	`f5-bigip-agent`	F5 BIG-IP `172.16.10.10` — pools, virtual servers, nodes, monitors, iRules, HA failover
`@github`	`github-agent`	The remote GitHub MCP (47 tools) — issues, PRs, repo ops
`@drone`	`drone-agent`	A real Ryze RoboMaster TT — 28 flight tools over an MCP server
`@demo`	`demo-agent`	The sandbox MCP servers, for when I’m just poking

That’s the whole point of the group chat: the same chat window is a firewall console, a kubectl prompt, a GitHub client, and a drone remote — I just switch who I’m talking to.

Here’s @forti doing exactly that — I asked “give me a list of wifi devices” and the FortiGate agent came back with 40 devices on SSID ManiakHQ, formatted as a table right in the chat (hostname, IP, MAC, AP, band, signal, OS, vendor):

That’s a real FortiGate at 172.16.10.1 answering a plain-English question from my phone. No console, no SSH — just a chat bubble.

Act 1 — How a text message flies a drone

Here’s the loop, start to finish, when I send one message:

 📱 Telegram ns kagent ns agentgateway-system
 ┌──────────┐ long-poll ┌──────────────┐ A2A ┌──────────────────┐ OpenAI-compat
 │ you → │ ───────────▶ │ telegram-bot │ ─────────▶ │ drone-agent │ ──────────────▶ AgentGateway
 │ @drone │ │ (1 replica) │ message/ │ (kagent runtime)│ /openai · /grok
 │ "flip" │ ◀─────────── │ │ send │ ModelConfig ────┼──▶ gateway ──▶ real provider
 └──────────┘ chat reply └──────────────┘ └────────┬─────────┘ keys from Vault
 │ MCP tools
 ▼
 drone-mcp-server → 🚁

You message. The bot long-polls Telegram (getUpdates) — its BotFather token comes from Vault via an ExternalSecret, never from Git. Single replica, on purpose: two pollers means a 409 Conflict fight over the same update stream.
A2A send. The bot is a thin router. It looks up which agent you’ve selected, and fires a JSON-RPC message/send at that agent’s in-cluster Service — http://drone-agent.kagent.svc.cluster.local:8080/ — carrying a per-chat contextId so the conversation has memory.
The agent thinks. drone-agent runs in the kagent runtime. To reason, it calls a model — but its kagent ModelConfig doesn’t point at OpenAI. It points at an in-cluster OpenAI-compatible base URL that is the gateway.
The gateway governs. agentgateway injects the real provider key (from a Vault-synced Secret), meters the tokens, emits a trace, and then forwards upstream. The agent never sees a credential.
The reply comes home. The answer streams back over A2A, the bot posts it as a chat bubble, and — if the agent wanted to run a tool that writes — you get a button instead of a fait accompli. (More on that in Act 2.)

The bot’s routing table is literally one environment variable on the Deployment:

- name: AGENTS_JSON
 value: |
 {
 "k8s": "http://k8s-agent.kagent.svc.cluster.local:8080/",
 "forti": "http://fortigate-agent.kagent.svc.cluster.local:8080/",
 "f5": "http://f5-bigip-agent.kagent.svc.cluster.local:8080/",
 "github": "http://github-agent.kagent.svc.cluster.local:8080/",
 "drone": "http://drone-agent.kagent.svc.cluster.local:8080/",
 "demo": "http://demo-agent.kagent.svc.cluster.local:8080/"
 }

Want a new agent in the chat? Add a line, redeploy the bot. No new brains to train, no keys to hand out — the agent already exists in the fleet, and the gateway already knows how to route its model.

/status pings whichever agent I’ve got selected, and /agents prints that routing table live — the same in-cluster A2A URLs, straight from the bot:

The commands are deliberately boring so I can drive them one-handed:

Command	Action
`/start`	Help
`/agents`	List aliases + their A2A URLs
`/use <alias> [msg]`	Switch agent — and optionally ask in the same line
`/new`	Reset the session (fresh `contextId`)
`/status`	Ping the current agent
`@forti …`	Switch and message in one shot

/use f5 what are my vips? is a complete interaction: pick the F5 agent and ask it, in one thumb-stroke.

Act 2 — The thumb between me and a `delete`

Here’s the part I refuse to skip. @k8s can delete deployments. @forti can rewrite firewall policy. @f5 can drain a pool member. A chatbot with that reach is a great way to nuke prod from a bus stop.

So the destructive tools are wrapped in kagent’s human-in-the-loop approval (requireApproval), and the bot surfaces it natively. When an agent decides it wants to run a write, it doesn’t run it — it returns an adk_request_confirmation back over A2A. The bot parses that, shows me exactly what’s about to happen, and hangs two inline buttons under it:

The agent wants to run: apply_manifest
 namespace: kagent

apiVersion: apps/v1 kind: Deployment …

 [ ✅ Approve ] [ ❌ Reject ]

Tap Approve and the bot sends the decision back — the agent resumes and completes the tool call. Tap Reject and nothing happens. The read paths (get, describe, logs, “what devices are on my wifi”) flow straight through; only the writes stop. It’s the difference between “the bot did a thing” and “I did a thing, from my phone, with a receipt.”

And the receipt is real: because every model hop went through agentgateway, each of these interactions is a priced, traced span in Langfuse and a row in the Solo cost UI — split by project, by model, by agent. I can see what my group chat cost me this week.

Act 3 — The keys never leave home

The whole thing rests on one rule: no agent, and no channel, ever holds a provider API key.

The fleet the bot talks to is just the kagent Agents list in the Solo Enterprise UI — every agent Healthy, each with its Model column showing what it routes to (OpenAI gpt-5.5, local Qwen) and its required MCP tool servers (drone-mcp, f5-bigip-…):

That Model column is the sleight of hand: it says “OpenAI (gpt-5.5)” and “OpenAI (Qwen),” but both are OpenAI-compatible endpoints that resolve to my gateway — not to OpenAI.

kagent ModelConfigs point at in-cluster gateways, not at providers:

ModelConfig	Model	Gateway base URL (in-cluster)
`default-model-config`	lab default	`…/openai/v1` · `agentgateway-proxy`
`xai-grok`	`grok-4.5`	`…/grok/v1` · `xai-grok-gateway`
`dgx-spark`	Qwen (local, self-hosted)	`…/spark/v1` · `dgx-spark-gateway`
`gpt-5-6`	`gpt-5.6`	`…/gpt56/v1` · dedicated gateway

The real OpenAI, xAI, and Anthropic keys sit in Vault. External Secrets Operator syncs them into gateway-side Secrets. agentgateway injects them upstream at call time. Which means:

One control plane for spend, traces, and budgets — not one per app.
Agents are credential-free. Compromise a pod, you get no keys.
I can swap gpt-5.5 for Qwen-on-my-DGX-Spark without touching the bot, the scheduler, or a single agent. It’s a one-line ModelConfig change.
Cost and Langfuse stay accurate across channels — chat and cron land in the same ledger.

That last point matters more than it sounds, because the chat bot isn’t the only way in.

Act 4 — The cron twin

@KagentCorpAIbot is for “help me right now.” But some things should just happen every morning — a cluster health sweep, a firewall device audit, a “what PRs are stale” report. For that there’s a second front-end on the exact same fleet: the Agent Scheduler, and it is pure GitOps.

Here’s the twist I like: the scheduler’s web UI never creates a CronJob through the Kubernetes API. It can’t set desired state directly. All it does is open a pull request against sebbycorp/k8s-goose. ArgoCD notices, merges, and applies. Git stays the one source of truth; the cluster only ever executes what Git says.

 Web UI (:30955) GitHub ArgoCD ns kagent
 ┌─────────────┐ PR + ┌──────────────┐ sync ┌────────────┐ apply ┌──────────────┐
 │ save a │ ──────▶ │ config/agent-│ ─────▶ │ agentgw- │ ──────▶ │ CronJob │
 │ schedule │ merge │ schedules/* │ │ config app │ │ ags-<name> │
 └─────────────┘ └──────────────┘ └────────────┘ └──────┬───────┘
 │ fires
 ▼
 Job → A2A message/send → agent
 result → ConfigMap ags-result-*
 (± Telegram summary, same bot token)

When a scheduled Job fires, it POSTs the same A2A message/send to the same agent Service the chat bot uses, and writes the reply into ConfigMaps (ags-result-<name> for the latest, ags-hist-… for history). Check the box marked “send result to Telegram” and the Job posts a summary to my chat using the same Vault bot token — so my morning cluster report shows up in the same window where I do my ad-hoc asks.

One caveat worth stating plainly: that beautiful HITL approval gate is a problem for unattended cron. A Job can’t tap Approve. So scheduled prompts start read-only — audits, reports, “tell me what changed” — and the writes stay in the interactive channel where a human thumb is present.

The shape of it

Strip away the fun and here’s what’s actually true:

Two front-ends, one fleet. A Telegram bot for conversation, a GitOps scheduler for routines — both are thin edge adapters that speak A2A to the same in-cluster kagent agents.
kagent runs the brains. Six agents, each with its own MCP tool servers and its own RBAC/blast-radius, reachable at :8080 over A2A.
agentgateway is the governor. Every model call is priced, traced, key-injected, and policy-checked before it reaches a provider. Keys live in Vault; nothing sensitive is in Git.
My thumb is the last mile. Destructive tools stop for an inline Approve/Reject.

It started as a joke — “what if I could text my firewall?” — and turned into the cleanest way I’ve found to operate a homelab: fun on the surface, governed all the way down.

If you want to build your own, the whole thing is GitOps’d in sebbycorp/k8s-goose — bot source in telegram-bot/, scheduler in agent-scheduler/, and the operator runbook (seed the Vault bot token, verify the ExternalSecret, smoke-test the chat path) on the live Dispatch page.

Now if you’ll excuse me, I have a drone to fly from the kitchen.

My reMarkable Writes Back — and agentgateway Governs Every Word

Tue, 07 Jul 2026 00:00:00 +0000

Write a question on paper with a pen. Pause. The page writes back — a cursive answer, drawn onto the e-ink letter by letter, in the voice of whichever model you tapped. No screen glare, no keyboard, no page reload. Just ink answering ink.

That’s the ink desk running on my reMarkable 2. It’s a single self-contained C binary on the tablet that reads the Wacom pen, grabs the page as a PNG when I pause, has gpt-5.5 vision transcribe my handwriting, sends the question to the model I’ve selected — OpenAI, Claude, Grok, or my own local Qwen — and renders the reply back onto the panel in a real cursive font.

Live page: goose.maniak.ai/remarkable.html.

Here’s the twist that matters to me: the tablet never talks to a model directly. Every call goes through my agentgateway. The keys live in Vault, every question is a trace in Langfuse, and any request can be blocked or governed by policy before it reaches a provider. This post is two acts — the desk itself, and the gateway that governs it.

Act 1 — Write on paper, it writes back

The magic is a five-stage loop that never leaves the tablet except to call the gateway:

#	Stage	What happens	On
01	WRITE	Reads the Wacom pen and draws your strokes straight to the panel	`evdev · rm2fb`
02	READ	On a short pause it captures the page as a PNG — in-app	`framebuffer → PNG`
03	SEE	gpt-5.5 vision reads the handwriting to text, so any model can answer	`gpt-5.5 · /openai`
04	ASK	The model you tapped replies — text, a flowchart, or a sketch	`via agentgateway`
05	INK	Rendered back onto the e-ink in cursive, letter by letter	`stb_truetype · A2`

The reMarkable 2 has no usable framebuffer of its own, so the app drives the e-ink through rm2fb (timower/rM2-stuff) — which means the tablet had to be downgraded to OS 3.22.4.2 (3.27 kept on the spare A/B partition as a fallback). The app is diary.c, cross-compiled for armv7 with the toltec toolchain, with zero runtime dependencies. Gestures are physical: tap the top-right to switch models, tap the top-left to save the page to your library, and flip the marker to its eraser end + tap to clear and start fresh. Ask it to “diagram X” and it auto-lays-out a mermaid-style flowchart on the page.

Here’s the desk answering a real question — “what is an agent substrate?” — written by hand:

I picked that question on purpose. An agent substrate is the runtime layer that lets stateful agents suspend, resume, and survive — the thing that makes an agent more than a stateless prompt. Watching the tablet hand-write the answer to a question about agent runtimes, through the same gateway I use for everything else, is the whole point: the substrate, the governance, and the observability follow the agent — even onto paper.

Act 2 — agentgateway governs every word

The desk is the fun part. The serious part is that nothing on the tablet is privileged. The app just hits OpenAI-compatible routes on the gateway’s NodePorts over Wi-Fi:

:/openai → OpenAI · gpt-5.5 (also the shared “eyes” that read every page)
:/anthropic → Claude · claude-fable-5
:/grok → xAI · Grok-4.3
:/spark → local Qwen3.6 on my own DGX Spark — no cloud, private model, on-paper ink

agentgateway injects each provider key (sourced from Vault via ESO), records every question as a Langfuse trace — latency, tokens, cost, like any other lab workload — and, crucially, applies policy. Because every answer is a gateway call, I can govern and block requests centrally: rate limits, spend caps, guardrails, and route rules all sit at the gateway, not on a tablet I’d have to re-flash to change my mind.

Here’s agentgateway governing and blocking the traffic in real time:

Swap a model, tighten a limit, or block a route — and the tablet never changes. That’s the difference between a device that calls an API and an agent that runs behind a governed control plane.

The architecture, end to end

reMarkable 2 (OS 3.22 · rm2fb drives the e-ink)
 pen ─► [ diary.c ] ─► capture page → PNG
 │ Wi-Fi · agentgateway NodePorts (HTTP)
 ▼
 ┌──────────────── agentgateway ────────────────┐
 │ gpt-5.5 reads the handwriting → text │
 │ selected model answers → text / flowchart │
 │ key from Vault (ESO) · traces → Langfuse │
 │ policy: rate limits · spend caps · block │
 └────────────────────────────────────────────────┘
 OpenAI · Claude · Grok · local Qwen (DGX Spark)
 │
 ▼
 answer written to the e-ink in cursive (letter by letter)

Everything model-facing is configured in one MODELS[] table in diary.c; everything policy-facing lives at the gateway. The tablet holds no secrets and enforces no rules — it just writes, captures, asks, and inks. Vault holds the keys, Langfuse holds the truth, and agentgateway holds the line.

The takeaway is small and stubborn: governance follows the agent everywhere it goes — even to paper. Your pen in, a cursive answer in your model’s voice back on the page, and every word of it metered, traced, and governable.

Try it live at goose.maniak.ai/remarkable.html.

GPT-5.5 vs Claude vs Grok with agentgateway: Which Model Saves Money? (I Made Them Fly a Drone)

Mon, 06 Jul 2026 00:00:00 +0000

Same job, three brains. I gave an AI agent a real Ryze RoboMaster TT drone and one fixed mission — take off, flip, photograph the room, spin 360°, land and report — then flew it identically on three models: gpt-5.5, claude-fable-5, and grok-4.3. Same agent, same drone, same 28-tool MCP server, same prompts. The only thing that changed was the model, swapped in with a one-line agentgateway ModelConfig.

The bill for that identical flight ranged 5× from cheapest to dearest. This post is about who saves money, why, and the one extra knob that moves the answer. Every number is measured — pulled from kagent’s tracing and Langfuse, not estimated.

Live flight deck: goose.maniak.ai.

Here’s the agent flying the mission on the real drone:

Your browser does not support the video tag.

The verdict: cost per flight

Model	Mode	Tokens	Latency	Cost	CodeSearch saves
grok-4.3 🏆	CodeSearch	99.5k	31s	$0.3050	−11%
grok-4.3	Standard	113.0k	25s	$0.3441	baseline
gpt-5.5	CodeSearch	53.1k	37s	$0.3201	−39%
gpt-5.5	Standard	98.3k	47s	$0.5252	baseline
claude-fable-5	CodeSearch	165.6k	101s	$1.5167	−9%
claude-fable-5	Standard	159.5k	79s	$1.6676	baseline

CodeSearch saves = each model’s CodeSearch run vs its own Standard run — the tool mode’s effect on that model. It’s a big lever for gpt-5.5 (−39%) but small for grok (−11%) and Claude (−9%). The cross-model cost winner is grok either way — it’s ~1/5th of Claude’s bill, but that’s the model choice, not the tool mode.

Three findings, in order of how much money they save you:

grok-4.3 is the cheapest and the fastest. ~$0.30–0.34 a flight, done in ~25–31s. It’s almost comically terse — asked to flip, it replied “Forward flip completed.” — 4 output tokens. Cheap talk, literally.
gpt-5.5 is right behind on cost and the leanest on tokens ($0.32–0.53, 53–98k tokens). It’s the model that responds best to the tool-mode trick below.
claude-fable-5 is the premium option — the most thorough answers, but ~5× grok’s cost and ~3× gpt-5.5’s for the exact same mission. Great model; you pay for it.

If the only thing you care about is the bill, grok wins, gpt-5.5 is a close and leaner second, Claude is a deliberate splurge.

Cost = tokens × each model’s price. gpt-5.5 and Claude are priced by Langfuse; grok-4.3 I priced from models.dev at the grok-4 rate ($3/M in, $15/M out).

Why the gap is so wide

It isn’t just per-token price — it’s how much each model says. Claude reasons out loud and writes verbose tool-discovery code, so it burns 160k+ tokens on a flight. grok is monosyllabic. gpt-5.5 sits in between but is disciplined. Two models can complete the identical mission and differ 5× on the bill purely from verbosity × price. You only see that if you meter it — which is the whole reason agentgateway sits in the middle.

The second lever: MCP tool mode

There’s a knob within each model. Every MCP call injects the tool catalog — the schema of all 28 tools — into the prompt. agentgateway can hide that behind two modes:

Standard — all 28 tools, full schemas, every turn.
CodeSearch — two meta-tools (get_tool to fetch one signature on demand, run_code to batch calls in a JS sandbox). Tiny context, fewer round-trips.

Here’s tokens per flight, both modes, all three models:

The catch: the tool mode that saves money is model-dependent.

On gpt-5.5, CodeSearch cut tokens ~46% and cost ~39% ($0.5252 → $0.3201). Big win.
On grok, a modest trim (it’s already terse).
On Claude, CodeSearch did nothing — its verbose run_code/discovery steps used more tokens than Standard. The saving was cancelled out.

So you can’t pick a tool mode in the abstract. Benchmark the mode with the model you’ll actually ship. For gpt-5.5, flip to CodeSearch. For Claude, don’t bother — spend your savings on picking a cheaper model instead.

Where gpt-5.5’s savings come from

One prompt dominates — the status report, because the agent polls get_state repeatedly:

#	Prompt	Standard (calls / tokens)	CodeSearch (calls / tokens)
5	Land + status report	17 / 45.1k	5 / 13.3k

Standard burns 17 calls and 45k tokens on that one turn; CodeSearch collapses it into a single run_code that reads state once — 5 calls, 13k tokens. That’s most of the −39%.

The receipts — kagent tracing

None of this is estimated. kagent (Solo Enterprise for kagent) traces every flight — one trace per prompt, with input, output, duration, and tokens — and exports the same spans to Langfuse:

Drill into a trace and you see the execution flow and every tool call. CodeSearch on Claude — get_tool then run_code to take the photo:

Claude on Standard, doing the 360° via the celebrate tool and reading get_state:

And grok on CodeSearch — same shape, but look how little it says (4 output tokens — that’s why grok’s bill is tiny):

The gateway sees — and prices — every hop

agentgateway routes each MCP call to the right backend and tags it with the tool mode, method, status, and latency — which is exactly why the cost numbers exist:

info request gateway=virtual-mcp-gateway route=drone-mcp
 http.method=POST http.path=/drone http.status=200
 protocol=mcp mcp.method.name=tools/list duration=3ms
info request gateway=virtual-mcp-gateway route=drone-mcp-codesearch
 http.method=POST http.path=/drone-codesearch http.status=200
 protocol=mcp mcp.method.name=tools/list duration=2ms

Because the same proxy fronts both the models and the MCP server, it meters both sides — one place to compare gpt-5.5, Claude, and grok on an apples-to-apples flight, and to swap between them with a config change.

The money lesson

Pick the model first — it’s the 5× lever. grok-4.3 flew the mission for ~1/5th the cost of Claude and finished 3× faster; gpt-5.5 is the leaner near-tie. Claude is a deliberate premium.
Then tune the tool mode for that model. CodeSearch saves gpt-5.5 ~39%; it does nothing for Claude.
Meter everything. Two models can finish the identical job and differ 5× on cost from verbosity alone. agentgateway makes both the swap and the measurement a one-liner.

Full flight, all three models, the charts, and the traces live on the flight deck: goose.maniak.ai.

First Steps: agentgateway, F5 AI Guardrails, and the Enterprise UI

Fri, 03 Jul 2026 00:00:00 +0000

The previous post covered the new hard spend limits in Enterprise agentgateway v2026.6.3: model cost catalogs, dollar or token budgets, and a real 429 when a budget is exhausted. That solves the FinOps side of AI traffic. The next question is the one security teams ask immediately after: what prevents a prompt, response, or agent workflow from leaking something it should not?

This is the first practical setup I use for that conversation:

agentgateway remains the AI data plane: one OpenAI-compatible front door, routes, backends, enterprise policies, traces, and cost/token metadata.
F5 AI Guardrails is the AI security decision point: scanners, redaction, blocking, and audit history.
Solo Enterprise UI for agentgateway gives the platform view: routes, destinations, policies, playground access, and traces from the gateway.

The goal is not just to return the right HTTP status code. The goal is to make the setup inspectable: security sees the guardrail decision in F5, platform sees the gateway route and policy in the agentgateway UI, and application teams keep calling one OpenAI-compatible endpoint.

The shape of the demo

I deploy two F5 integration patterns side by side:

Route	Pattern	What happens
`/option-a`	agentgateway in front of F5 inline Guardrails	agentgateway forwards to F5’s OpenAI-compatible `/openai/{provider}/chat/completions` endpoint. F5 scans and makes the final provider call.
`/option-c`	agentgateway with out-of-band F5 ScanAPI	agentgateway calls OpenAI directly, but request and response `promptGuard` webhooks call a small adapter that sends text to F5 ScanAPI.

I also add three native agentgateway Enterprise policy routes:

Route	Purpose
`/agw/direct`	direct response generated by the gateway
`/agw/cors`	CORS and response header policy
`/agw/rate-limit`	local rate limiting before provider/backend traffic

That gives the demo two useful proofs at once: F5 is enforcing AI security policy, and agentgateway Enterprise is enforcing gateway-native traffic policy.

Step 1: install Enterprise agentgateway

The demo runs on a disposable kind cluster and installs Enterprise agentgateway v2026.6.3.

The important environment variables are:

AGENTGATEWAY_LICENSE_KEY='...'
OPENAI_API_KEY='sk-...'

F5_AISEC_URL='https://www.us2.calypsoai.app'
F5_AISEC_TOKEN='...'
F5_AISEC_INLINE_PROVIDER='genai-azure-openai'
CAI_PROJECT='Global-...'

OPTION_A_MODEL='gpt-4.1'
OPTION_C_MODEL='gpt-5.5'

I keep these in .env, which is ignored by Git. The F5 token is used in two places: setup-time scanner creation and runtime calls from the in-cluster adapter.

The gateway itself is standard Kubernetes Gateway API:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: agentgateway-proxy
 namespace: agentgateway-system
spec:
 gatewayClassName: enterprise-agentgateway
 listeners:
 - name: http
 protocol: HTTP
 port: 80
 allowedRoutes:
 namespaces:
 from: All

Everything else attaches to that gateway: the F5 inline backend, the direct OpenAI backend, the promptGuard policy, the UI tracing policy, and the native Enterprise policy demos.

Step 2: configure F5 AI Guardrails

The setup script creates a practical scanner set in F5. I started with two simple controls and then expanded it so the demo behaves more like an actual security review:

Scanner	Type	Mode	Direction
`agw-lab-keyword-codename`	Keyword, `project-titan`	Block	Prompts and responses
`agw-lab-regex-ssn`	Regex	Redact	Prompts
`agw-lab-regex-email`	Regex	Redact	Prompts
`agw-lab-regex-phone`	Regex	Redact	Prompts
`agw-lab-regex-api-key`	Regex	Redact	Prompts
`agw-lab-regex-jwt`	Regex	Redact	Prompts
`agw-lab-regex-private-key`	Regex	Redact	Prompts
`agw-lab-keyword-prompt-injection`	Keyword	Block	Prompts
`agw-lab-keyword-secret-exfiltration`	Keyword	Block	Prompts
`agw-lab-regex-codename-obfuscation`	Regex	Block	Prompts and responses

Here is that scanner set in the F5 UI:

The setup script validates F5 access, resolves the project, confirms the inline provider exists, creates or reuses the scanners, attaches them to the project, and then runs quick ScanAPI checks:

./setup-guardrails.sh

For production, the direction column matters. In this demo, PII redactors are prompt-side controls and the codename controls run both ways. If you need PII redaction on model output as well, make those scanners direction: "both" and rerun the setup before you call the deployment production-ready.

Step 3: Option A, F5 inline behind agentgateway

Option A is the fastest path because F5 already exposes an OpenAI-compatible endpoint. agentgateway treats that endpoint like a custom OpenAI provider.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: f5-guardrails-inline
 namespace: agentgateway-system
spec:
 ai:
 provider:
 openai:
 model: "__OPTION_A_MODEL__"
 host: "__F5_AISEC_HOST__"
 port: 443
 path: "/openai/__F5_AISEC_INLINE_PROVIDER__/chat/completions"
 policies:
 auth:
 secretRef:
 name: calypsoai-token
 tls:
 sni: "__F5_AISEC_HOST__"

The app calls:

POST /option-a

agentgateway receives the OpenAI Chat Completions request, applies its route and backend policy, and forwards to F5. F5 scans the prompt and response and owns the final provider hop.

Use this pattern when you want a fast proof that the products work together and the security team is comfortable owning the final provider connection in F5.

Step 4: Option C, F5 ScanAPI as a promptGuard webhook

Option C keeps agentgateway as the only inference path. F5 does not proxy the LLM request. It only renders a verdict through ScanAPI.

The agentgateway policy targets the /option-c HTTPRoute:

apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayPolicy
metadata:
 name: f5-guardrails
 namespace: agentgateway-system
spec:
 targetRefs:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 name: option-c
 backend:
 ai:
 promptGuard:
 request:
 - webhook:
 backendRef:
 kind: Service
 name: f5-guardrails-adapter
 port: 8000
 failureMode: FailClosed
 response:
 message: "Blocked by F5 AI Guardrails"
 statusCode: 403
 response:
 - webhook:
 backendRef:
 kind: Service
 name: f5-guardrails-adapter
 port: 8000
 failureMode: FailClosed

The adapter is intentionally small. It receives the webhook body, extracts the prompt or assistant response, calls:

POST /backend/v1/scans

with:

{
 "input": "text to scan",
 "project": "Global-...",
 "scanDirection": "request",
 "flagOnly": false,
 "verbose": true
}

Then it maps F5 outcomes back to agentgateway actions:

F5 outcome	Request webhook	Response webhook
clear	pass	pass
blocked / flagged / rejected	reject with `403`	mask assistant content
`redactedInput` returned	replace the last user message	replace assistant content
ScanAPI error	fail closed with `503`	fail closed

This is the shape I prefer for production. agentgateway keeps routing, failover, budgets, provider credentials, and traces. F5 keeps scanner policy, redaction decisions, and audit evidence.

Step 5: install the Solo Enterprise UI

This is the part that makes the demo much easier to explain. The UI install is not an afterthought; it is part of the deployment.

0.4.8 adds one prerequisite that 0.4.7 did not need: the ui-backend container now watches platform.solo.io CRDs (KubernetesCluster). Install the dedicated management-crds chart first, or ui-backend CrashLoopBackOffs with no matches for kind "KubernetesCluster" in version "platform.solo.io/v1alpha1":

helm upgrade -i management-crds \
 oci://us-docker.pkg.dev/solo-public/solo-enterprise-helm/charts/management-crds \
 --namespace agentgateway-system --create-namespace \
 --version 0.4.8

Then install the management chart at 0.4.8 with the agentgateway product enabled. I also turn on cost management so the UI exposes spend analytics:

helm upgrade -i management \
 oci://us-docker.pkg.dev/solo-public/solo-enterprise-helm/charts/management \
 --namespace agentgateway-system \
 --create-namespace \
 --version 0.4.8 \
 --set cluster="mgmt-cluster" \
 --set products.agentgateway.enabled=true \
 --set products.agentgateway.features.cost-management=true \
 --set-string licensing.licenseKey="${AGENTGATEWAY_LICENSE_KEY}"

One subtlety worth knowing: products.agentgateway.features.cost-management=true renders PRODUCT_AGENTGATEWAY_FEATURES_COST_MANAGEMENT_ENABLED=true on the ui-frontend container — that is the flag that turns on the Cost Management tab. The ui-backend container does not get that variable; it gets AGENTGATEWAY_COST_WRITES_ENABLED, driven by the separate cost-management-writes value (default true). So the toggle you flip gates the frontend UI, and a second value governs whether the backend can write budgets, dimensions, and virtual keys.

For a demo, I leave SOLO_UI_OIDC_ISSUER empty so the chart’s built-in auto-auth path is used. For a real environment, wire it to your IdP and provide the backend client secret through a Kubernetes Secret.

The install gives me a solo-enterprise-ui service:

kubectl get svc -n agentgateway-system solo-enterprise-ui

The two useful local forwards are:

kubectl port-forward -n agentgateway-system svc/agentgateway-proxy 8080:80
kubectl port-forward -n agentgateway-system svc/solo-enterprise-ui 8090:80

Then open:

open http://localhost:8090

Step 6: turn on agentgateway traces for the UI

The UI becomes useful for traffic analysis when agentgateway emits OTLP traces to the management telemetry collector.

That is a small Enterprise policy:

apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayPolicy
metadata:
 name: tracing
 namespace: agentgateway-system
spec:
 targetRefs:
 - group: gateway.networking.k8s.io
 kind: Gateway
 name: agentgateway-proxy
 frontend:
 tracing:
 backendRef:
 name: solo-enterprise-telemetry-collector
 namespace: agentgateway-system
 kind: Service
 port: 4317
 randomSampling: "true"

Verify that the policy attached:

kubectl get enterpriseagentgatewaypolicy tracing \
 -n agentgateway-system \
 -o yaml

After traffic runs, the proxy logs include trace.id and span.id fields on requests. Those are the breadcrumbs that connect gateway behavior to UI trace views.

This is the policy inventory in the agentgateway UI:

And this is the destination inventory. The two AI backends are exactly the two patterns from the architecture: F5 inline and OpenAI direct.

Step 7: use the playground as a sanity check

The UI sees the routes from the cluster:

That is useful in a demo because you can explain the whole deployment without starting in YAML:

/option-a is the inline F5 path.
/option-c is the out-of-band ScanAPI path.
/agw/direct, /agw/cors, and /agw/rate-limit are native agentgateway Enterprise policy examples.

The Routes inventory is the same set from the control plane, with match prefix, resolved destination, gateway, and observed p95 latency once traffic has run:

The UI is not a replacement for kubectl logs for raw pod stdout. Treat it as the topology and request-visibility layer: routes, destinations, policies, playground, and traces. Use Kubernetes logs for adapter exceptions and pod startup messages.

Step 7b: cost management for the same traffic

Because I installed the UI with products.agentgateway.features.cost-management=true, the Cost Management tab estimates spend for the traffic flowing through these routes. Spend is computed from token counts × your configured per-token prices — an estimate, not the provider’s invoice — and it breaks down by provider, model family, model, group, user, and virtual key.

This is where budgets and guardrails meet in one console: the same route that F5 scans for safety also reports what it costs. Pair it with EnterpriseAgentgatewayBudget (from the hard spend limits article) to enforce a ceiling, not just observe one.

Step 8: prove enforcement with traffic

The smoke test sends six OpenAI Chat Completions requests:

./test.sh

The passing run:

Text version:

PASS Option A benign: HTTP 200
PASS Option C benign: HTTP 200
PASS Option A blocked codename: HTTP 400
PASS Option C blocked codename: HTTP 403
PASS Option C SSN redaction request completed: HTTP 200
PASS Option C redaction did not leak raw SSN
PASS Option C response-phase scan completed: HTTP 200
PASS Option C response-phase scanner masked blocked output

That proves the data plane behavior:

benign traffic reaches the model
project-titan is blocked
SSN-shaped prompt content is redacted before forwarding
response content that trips the response scanner is masked before the client sees it

Step 9: confirm the F5 audit trail

The same traffic shows up in F5 under Logs -> Prompt history:

This is the evidence split I want in the operating model:

agentgateway proves which route, policy, backend, status code, model, token count, cost, trace ID, and latency were involved.
F5 proves which scanner matched, whether content was blocked or redacted, who initiated it, and when the decision happened.

Those are different audit questions. Do not force one product to answer both.

The deployment checklist

For a fresh lab, the sequence is:

cp .env.example .env
# fill in AGENTGATEWAY_LICENSE_KEY, OPENAI_API_KEY, F5_AISEC_URL, F5_AISEC_TOKEN

./setup-guardrails.sh
./deploy.sh

kubectl port-forward -n agentgateway-system svc/agentgateway-proxy 8080:80
kubectl port-forward -n agentgateway-system svc/solo-enterprise-ui 8090:80

./test.sh
./test_agentgateway.sh
HARNESS_CASES=harness/intense-cases.yaml ./run_harness.sh

The checks I care about before showing this to anyone:

helm list -n agentgateway-system
kubectl get pods,svc -n agentgateway-system
kubectl get enterpriseagentgatewaypolicy -n agentgateway-system
kubectl logs -n agentgateway-system deploy/agentgateway-proxy --tail=80
kubectl logs -n agentgateway-system deploy/f5-guardrails-adapter --tail=80

The healthy state should include:

enterprise-agentgateway chart at v2026.6.3
management-crds chart installed (so ui-backend finds the platform.solo.io CRDs)
management chart at 0.4.8
solo-enterprise-ui pod 5/5 Ready (a crash-looping ui-backend means the management-crds install was skipped)
solo-enterprise-telemetry-collector running
tracing policy accepted and attached
Cost Management tab visible in the UI (PRODUCT_AGENTGATEWAY_FEATURES_COST_MANAGEMENT_ENABLED=true on ui-frontend)
agentgateway request logs with trace.id and span.id

What this adds on top of hard spend limits

The budget article showed that agentgateway can stop runaway cost at the gateway. This setup adds the security controls around the same traffic:

budgets answer how much can this team spend?
guardrails answer is this prompt or response allowed?
routes and policies answer where is this traffic allowed to go?
traces answer what happened on this request?

The key is that these controls are attached to the gateway, not hand-coded in every application. Apps keep using normal OpenAI Chat Completions calls. Platform and security teams govern the path.

What I would harden next

For a production-grade rollout, I would tighten four things:

Change PII redactors that must protect model output to direction: "both".
Put the adapter behind real service-level observability and alert on fail-closed 503s.
Wire the Enterprise UI to the corporate IdP instead of demo auto-auth.
Combine this with EnterpriseAgentgatewayBudget so unsafe traffic and runaway spend are both blocked at the same front door.

That is the platform story: one gateway, separate controls, clear ownership, and enough visibility that you can prove what happened after the fact.

References

Hard Spend Limits for LLM Traffic: AI Budgets in Enterprise AgentGateway v2026.6.3

Thu, 02 Jul 2026 00:00:00 +0000

Enterprise AgentGateway v2026.6.3 shipped on July 1st with a changelog line that FinOps-minded platform teams have been waiting for: Enterprise Budgets and Dimensions. Until now, “budgeting” LLM traffic through a gateway meant approximating it with token-based rate limiting — workable, but request-window based, tokens-only, and fiddly to reason about. The new release replaces that with a first-class Kubernetes resource: EnterpriseAgentgatewayBudget, a declarative spending limit in US dollars or tokens, accumulated over a rolling window (day, week, month, year), with a choice of audit or block when the budget runs dry.

This post is a day-one field report. The docs for the feature are still catching up with the release (only the API reference covers it as I write this), so everything below was derived from the live CRD schemas on my cluster and verified with real traffic — including tripping a budget on purpose and watching the gateway return 429 with the window in the reset header.

Why budgets, not rate limits

Rate limits answer “how fast?” Budgets answer “how much, in dollars, this month?” — which is the question your finance team actually asks. The difference shows up in three places:

Unit. A budget is denominated in USD or Tokens. USD budgets use a model cost catalog (per-model input/output rates) so the gateway computes the realized cost of every request as it happens.
Window. Budgets accumulate over rolling windows — Day, Week, Month, Year. Under the hood, enforcement rides the gateway’s rate limiting infrastructure, so spend ages out continuously rather than resetting at a calendar boundary: a Day budget looks at the trailing 24 hours, not “since midnight.”
Action. onBudgetExceeded: Audit logs and lets traffic through (perfect for rollout); Block returns 429 until enough spend rolls off the window to bring you back under the limit. You can run both at once: a generous audited dollar budget for visibility, plus a hard token stop as a circuit breaker.

The moving parts

Three resources cooperate, all in the enterpriseagentgateway.solo.io/v1alpha1 API group:

EnterpriseAgentgatewayBudget — the budget definitions themselves. Each entry has a name, a limit (unit + amount), a window, an onBudgetExceeded action, and an optional subject — up to 16 key/value dimensions that scope the entry to matching requests (per user, team, API key, model… "*" matches any non-missing value; omit subject and the budget applies to every request on the enforced target).
EnterpriseAgentgatewayPolicy with traffic.entBudgetEnforcement — switches enforcement on for the gateways or routes in targetRefs, and controls discovery: which namespaces the controller scans for Budget resources (Same, Selector, or All — handy for letting each team keep its own Budget objects in its own namespace).
EnterpriseAgentgatewayParameters with modelCatalog — loads per-model pricing from a ConfigMap so USD budgets (and per-request cost telemetry) have something to compute with.

The lab

My setup is a bare-metal Talos Kubernetes cluster running Solo Enterprise AgentGateway v2026.6.3, fully GitOps-managed by ArgoCD — every manifest below lives in a repo and lands via auto-sync. The main gateway (agentgateway-proxy) already fronts OpenAI at /openai.

One design note worth stealing: my /openai route is also the default model path for kagent agents in the same cluster. A blocking budget that trips would cut those agents off for up to 24 hours. So the demo gets a dedicated route — same backend, different path — and the budget policy targets only that route. Blast radius: zero.

Step 1 — the model cost catalog

Per-model USD rates, per 1M tokens, as a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
 name: model-cost-catalog
 namespace: agentgateway-system
data:
 catalog.json: |
 {
 "providers": {
 "openai": {
 "models": {
 "gpt-4o": {
 "rates": { "input": "2.50", "output": "10.00" }
 },
 "gpt-4o-mini": {
 "rates": { "input": "0.15", "output": "0.60" }
 }
 }
 }
 }
 }

Treat pricing as versioned configuration — when a provider changes rates, you change a file in Git and your cost telemetry follows.

Step 2 — load the catalog into the proxy

The catalog attaches to a gateway through EnterpriseAgentgatewayParameters:

apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayParameters
metadata:
 name: agentgateway-proxy-params
 namespace: agentgateway-system
spec:
 modelCatalog:
 sources:
 - configMap:
 name: model-cost-catalog
 key: catalog.json

…and the Gateway references it via infrastructure.parametersRef:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: agentgateway-proxy
 namespace: agentgateway-system
spec:
 gatewayClassName: enterprise-agentgateway
 infrastructure:
 parametersRef:
 group: enterpriseagentgateway.solo.io
 kind: EnterpriseAgentgatewayParameters
 name: agentgateway-proxy-params
 listeners:
 - name: http
 protocol: HTTP
 port: 80
 allowedRoutes:
 namespaces:
 from: All

Heads-up: attaching parameters rolls the proxy Deployment (a new pod came up in seconds in my lab). On startup the proxy confirms the catalog load:

info llm::cost watching model catalog files count=2
info llm::cost loaded model catalog providers=3 models=80

Note providers=3 models=80 — your catalog merges with a built-in one, so you only need to define models where you want to pin your own rates.

Step 3 — an isolated demo route

Same OpenAI backend, dedicated path:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: budget-demo
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: agentgateway-proxy
 namespace: agentgateway-system
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /budget-demo
 backendRefs:
 - name: openai
 namespace: agentgateway-system
 group: agentgateway.dev
 kind: AgentgatewayBackend

Step 4 — the budgets and their enforcement

Two entries: a $5/day audited dollar budget (visibility) and a deliberately tiny 2,000-token/day blocking budget (the circuit breaker we’re going to trip):

apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayBudget
metadata:
 name: budget-demo
 namespace: agentgateway-system
spec:
 budgets:
 - name: demo-usd-audit
 limit:
 unit: USD
 amount: 5
 window:
 unit: Day
 onBudgetExceeded: Audit
 - name: demo-token-block
 limit:
 unit: Tokens
 amount: 2000
 window:
 unit: Day
 onBudgetExceeded: Block
---
apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayPolicy
metadata:
 name: budget-demo-enforcement
 namespace: agentgateway-system
spec:
 targetRefs:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 name: budget-demo
 traffic:
 entBudgetEnforcement:
 discovery:
 namespaces:
 from: Same

Push, let ArgoCD sync, and check the policy attached:

$ kubectl -n agentgateway-system get enterpriseagentgatewaypolicy budget-demo-enforcement
NAME ACCEPTED ATTACHED AGE
budget-demo-enforcement True True 29s

Tripping it

First request — normal:

$ curl -s http://<node-ip>:30160/budget-demo/v1/chat/completions \
 -H 'Content-Type: application/json' \
 -d '{"model":"gpt-4o","messages":[{"role":"user","content":"In one sentence, what is a token budget?"}],"max_tokens":60}'

HTTP/1.1 200 OK
...
"usage":{"prompt_tokens":17,"completion_tokens":39,"total_tokens":56}

Then a few large completions (max_tokens: 800, a 500-word essay each) to burn the 2,000-token daily budget:

request 1: HTTP/1.1 200 OK "total_tokens":704
request 2: HTTP/1.1 200 OK "total_tokens":711
request 3: HTTP/1.1 200 OK "total_tokens":669
request 4: HTTP/1.1 429 Too Many Requests

56 + 704 + 711 + 669 = 2,140 tokens — over budget. Request 4 never reaches OpenAI. The 429 response is admission-level and self-describing:

HTTP/1.1 429 Too Many Requests
x-ratelimit-limit: 2000
x-ratelimit-remaining: 0
x-ratelimit-reset: 86400

rate limit exceeded

x-ratelimit-reset: 86400 — the 24-hour rolling window. Clients get exactly what they need to back off intelligently.

Meanwhile the production /openai route (which kagent’s agents use) kept returning 200 throughout — the budget is scoped to the demo route and nothing else.

What you get in the logs

This is my favorite part. With the catalog loaded, every LLM request logs its realized dollar cost alongside the usual token telemetry:

info request route=agentgateway-system/budget-demo ... http.status=200
 gen_ai.request.model=gpt-4o gen_ai.response.model=gpt-4o-2024-08-06
 gen_ai.usage.input_tokens=21 gen_ai.usage.output_tokens=683
 agw.ai.usage.cost.total=0.0068825

That’s $0.0069 for a 704-token gpt-4o call, computed from the catalog at request time — the same number your USD budgets accrue against, and it flows into traces and metrics for your observability stack.

And when the blocking budget trips, the gateway says so in plain terms:

warn budget budget exceeded; blocking...
 budget_id=agentgateway-system/budget-demo/demo-token-block
 budget_action="BLOCK" budget_unit="TOKENS" budget_limit=2000
 budget_window="DAILY" phase="admission" outcome="over_limit_block"

Everything you’d want in an alert: which budget, which action, which window, and the outcome. An Audit entry produces the same telemetry with a non-blocking outcome — which is exactly how you should roll budgets out: audit first, watch the logs for a week, then flip to Block.

Dimensions: scoping budgets to teams and users

The demo entries above are unscoped — they apply to every request on the enforced route. The subject field is where multi-tenancy comes in. Take the classic real-world split: engineering runs agents and evals all day and gets a $500/month allocation; product does lighter prototyping on $200/month; and a catch-all audit entry watches everything else:

spec:
 budgets:
 - name: eng-monthly
 subject:
 group: engineering  # up to 16 key/value dimensions per entry
 limit:
 unit: USD
 amount: 500
 window:
 unit: Month
 onBudgetExceeded: Block
 - name: product-monthly
 subject:
 group: product
 limit:
 unit: USD
 amount: 200
 window:
 unit: Month
 onBudgetExceeded: Block
 - name: everyone-else
 subject:
 group: "*" # any request that carries a group dimension
 limit:
 unit: USD
 amount: 100
 window:
 unit: Month
 onBudgetExceeded: Audit

Engineering blowing through its allocation on a runaway eval loop gets 429s; product keeps shipping, unaffected — same gateway, same route, different wallets. A subject scopes the entry to requests whose resolved dimension values match every key/value pair (a value of "*" means “any non-missing value”). Combined with the discovery modes on the enforcement policy (Same / Selector / All), the intended shape is clear: platform team owns the enforcement policy on the gateway; each product team owns Budget resources in their own namespace, scoped to their own dimensions. Org → team → user hierarchies with different windows and actions per level.

One honest caveat for day one: how dimension values are resolved at request time (trusted identity headers derived from JWT claims, per Solo’s cost-controls reference architecture) isn’t fully documented yet outside the API reference. For unscoped budgets like this demo, none of that matters — they just work.

Gotchas from the field

The API group is enterpriseagentgateway.solo.io/v1alpha1. The docs' model-catalog example shows agentgateway.dev/v1alpha1 for the Parameters kind; the CRDs actually installed by the v2026.6.3 charts use the enterprise group. When in doubt, kubectl get crd | grep -i budget and read the schema — it also encodes useful validation (budget names must be alphanumeric-with-hyphens; subject keys/values can’t contain |, ^, or backticks; token budgets cap at 2^32−1).
Attaching parametersRef restarts the proxy. Plan for a rollout, not a hot reload.
Budgets count input + output tokens. Size token budgets against total_tokens, not your prompt sizes.
Windows are rolling, not calendar-aligned. Budget enforcement rides the gateway’s rate limiting infrastructure, so a tripped Day budget stays blocked until enough spend ages out of the trailing 24 hours — it does not reset at midnight. Either way, never put a small blocking budget on a route that shared infrastructure (agents, CI) depends on. Dedicated routes or subject-scoped entries are your friends.
v2026.6.x is not a long-term-support train. AgentGateway moved to calver with this cycle; quarterly releases (first expected: 2026.7.x) get 12 months of patches. Fine for a lab, plan accordingly for production.

Wrapping up

Budget enforcement at the gateway is the missing primitive for AI platform FinOps: the place that already sees every request, every token, and (with a cost catalog) every dollar is now the place that can say no — declaratively, per team, per window, in the currency your CFO speaks. The rollout path is gentle: load a catalog, watch agw.ai.usage.cost.total show up in your logs, add Audit budgets, and only then arm Block where runaway spend would actually hurt.

Total cost of this entire demo, per the gateway’s own accounting: about two cents.

Three Ways to Combine agentgateway with F5 AI Guardrails (and Where F5 Distributed Cloud Fits)

Thu, 02 Jul 2026 00:00:00 +0000

F5’s acquisition of CalypsoAI gave enterprises something a lot of security teams have been asking for: a dedicated AI runtime security layer — F5 AI Guardrails — that scans prompts and responses against policy (prompt injection, jailbreaks, PII, data exfiltration) independently of whichever model or app framework is behind it. Meanwhile, platform teams are standardizing on agentgateway as the data plane for LLM, MCP, and A2A traffic: one OpenAI-compatible front door with routing, failover, auth, spend limits, and OpenTelemetry.

So the obvious question came up — in my case, from an enterprise that already runs its web estate behind F5 Distributed Cloud (XC): can these two coexist, and who sits in front of whom?

Short answer: yes, and there are three workable architectures. I verified the key API surfaces against the live F5 docs (docs.aisecurity.f5.com — note that docs.calypsoai.com now redirects there) rather than assuming, because the whole design hinges on two questions:

Does Guardrails expose an OpenAI-compatible inline endpoint? (Yes: POST /openai/{provider}/chat/completions.)
Can you override the backend URL Guardrails forwards to? (Yes: providers are created with POST /backend/v1/providers and carry a template.url — an arbitrary endpoint.)

That second one matters because it decides whether “agentgateway behind F5” is even possible. It is. Let’s walk through all three options.

First, get the product boundaries right

One thing that trips people up: F5 AI Guardrails is not an F5 XC feature. It’s the CalypsoAI platform under F5 branding — its own console, its own projects and API tokens, delivered as SaaS (us1.calypsoai.app / eu1.calypsoai.app) or self-hosted on Kubernetes/OpenShift. Your XC tenant gives you the WAAP edge (TLS, WAF, bot defense, DDoS, API protection); the Guardrails license is separate.

The three products in play:

Layer	Product	Job
Edge	F5 Distributed Cloud HTTP LB	Public entry: TLS, WAF, bot defense, DDoS
AI data plane	agentgateway	OpenAI-compatible routing, failover, authn, budgets, MCP/A2A, OTel
AI security	F5 AI Guardrails (ex-CalypsoAI)	Prompt/response scanning: inject, jailbreak, PII, custom scanners

Guardrails itself supports two consumption modes, and every architecture below is a composition of them:

Inline — Guardrails terminates the request, scans, and forwards to a configured provider. Speaks its native PromptAPI (POST /backend/v1/prompts) or plain OpenAI chat completions (POST /openai/{provider}/chat/completions).
Out-of-band — a pure verdict API. POST /backend/v1/scans takes input, project, and scanDirection (request or response) and returns an outcome plus redactedInput. Nothing gets forwarded; you stay in charge of calling the model.

Why the gateway layer is non-negotiable

Before the architectures, it’s worth being blunt about why agentgateway is in every one of these diagrams — because “can’t the apps just call Guardrails directly?” is the first question you’ll get in an architecture review. They can (that’s Lab 3 of the F5 workshop), and it doesn’t scale. Every other layer in the stack assumes someone else is governing AI traffic: the XC edge governs HTTP and doesn’t know what a token, a model, or a tool call is; Guardrails judges content and doesn’t route, failover, meter, or authenticate your apps. Without a gateway, your “AI platform” is a pile of SDK calls nobody owns.

What the gateway layer uniquely provides:

One front door instead of N×M integrations. Ten apps on three providers is thirty credential sets and thirty places to touch every time a model deprecates. Behind one OpenAI-compatible endpoint, providers and models change without a single app redeploy.
Provider keys leave the apps. Apps authenticate to you (JWT, API key, OAuth); OpenAI/Anthropic credentials exist in exactly one place. Revoking a team’s access takes seconds and doesn’t rotate a provider key across the fleet.
Spend control that actually blocks. Provider dashboards report last month; gateway budgets are enforced live — dollar- or token-denominated, per team or app, returning 429 when the money runs out. The edge can rate-limit requests, but a request can cost $0.001 or $5; only the layer that parses usage can meter dollars.
Failover and model agility. Provider outages and deprecations are when, not if. Automatic cross-provider failover and model aliasing turn every provider incident from an all-hands app change into a config edit.
MCP and A2A governance. Agents call tools, and ungoverned MCP servers are the new shadow IT — every tool an agent can reach is an exfiltration path. agentgateway federates MCP servers behind auth and RBAC; neither the edge nor Guardrails addresses this at all.
Observability where the semantics live. OTel traces with tokens, cost, model, and latency per request. The edge sees bytes; the guardrail sees verdicts; only the gateway sees the whole conversation.
It’s the socket the guardrail plugs into. Without a gateway, adopting Guardrails means every app team writes scan-then-forward code. With one, guardrails become a policy — one webhook config, zero app changes (that’s Option C below).

And the honest trade-offs: it’s another hop (single-digit milliseconds — noise next to seconds of LLM inference) and another component to operate; if it’s down, AI traffic is down (it’s a stateless Rust proxy built to run as HA replicas); the budget/webhook/UI features are in the enterprise tier. The counterargument to all three is the same: without the gateway you end up building ad-hoc versions of half these features anyway — badly, in every app, with no one accountable.

You wouldn’t run web apps without a load balancer or APIs without an API gateway. Running LLMs and agents without an AI gateway is the same mistake, except the blast radius is your provider bill, your credentials, and every tool your agents can touch. The edge protects you from the internet; Guardrails protects you from the content; the gateway is what makes AI traffic governable at all.

Option A — agentgateway in front of Guardrails

The zero-code option. Guardrails’ inline endpoint is OpenAI-compatible, so agentgateway just treats it as one more LLM backend. F5 makes the final hop to the provider.

flowchart LR C["AI apps / agents (OpenAI SDK)"] --> AGW["agentgateway authn · routing · budgets · OTel"] AGW --> GR["F5 AI Guardrails (inline) /openai/openai-prod/chat/completions scans prompt + response"] GR --> LLM["LLM provider (configured in Guardrails)"]

The flow, end to end:

sequenceDiagram participant App as AI app participant AGW as agentgateway participant GR as F5 AI Guardrails participant LLM as LLM provider App->>AGW: POST /v1/chat/completions AGW->>AGW: authn, rate limit, model alias, budget AGW->>GR: POST /openai/{provider}/chat/completions GR->>GR: scan prompt (inject, jailbreak, PII...) alt prompt blocked GR-->>AGW: policy violation AGW-->>App: error (logged + traced) else prompt clear GR->>LLM: forward (transformed to provider spec) LLM-->>GR: completion GR->>GR: scan response GR-->>AGW: sanitized completion AGW-->>App: completion (+ usage, cost, trace) end

Configuration is just a custom OpenAI-compatible backend in agentgateway pointing at the Guardrails host, with the CalypsoAI token as backend auth. The same mechanics F5 documents for pointing the raw OpenAI SDK at Guardrails (base_url = "{BASE_URL}/openai/{CONNECTION_NAME}") apply — agentgateway is simply the client.

Choose this when the security team owns model access end-to-end and you want the fastest possible integration. Trade-off: the final provider choice lives in Guardrails’ provider config, so agentgateway’s multi-provider failover happens between Guardrails connections rather than directly against the LLMs.

Option B — agentgateway behind Guardrails

The reverse: Guardrails is the client-facing boundary and delegates the actual model plumbing to agentgateway. This only works if you can override the endpoint Guardrails forwards to — and you can. A Guardrails provider is defined by a request template with a url, method, headers, and body mapping, so you point that template at agentgateway’s OpenAI-compatible listener:

POST /backend/v1/providers
{
 "name": "agentgateway",
 "template": {
 "type": "request",
 "url": "https://agw.internal.example.com/v1/chat/completions",
 "method": "POST",
 "headers": { "Authorization": "Bearer {{agw_token}}" }
 },
 "secrets": { "agw_token": "..." },
 "projectId": "<guardrails-project>"
}

flowchart LR C["AI apps / agents"] --> GR["F5 AI Guardrails (inline) terminates + scans"] GR -->|"provider template.url →"| AGW["agentgateway"] AGW --> L1["OpenAI"] AGW --> L2["Anthropic"] AGW --> L3["vLLM (self-hosted)"] AGW --> MCP["MCP tool servers"]

sequenceDiagram participant App as AI app participant GR as F5 AI Guardrails participant AGW as agentgateway participant LLM as best provider App->>GR: POST /openai/agentgateway/chat/completions GR->>GR: scan prompt GR->>AGW: forward via provider template.url AGW->>AGW: route, failover, budget, trace AGW->>LLM: provider-native request LLM-->>AGW: completion AGW-->>GR: OpenAI-format response GR->>GR: scan response GR-->>App: sanitized completion

Choose this when the security team insists on owning the client-facing endpoint, but you still want gateway-grade routing/failover/spend control underneath the guardrail check. Trade-off: the provider-template mechanism is documented, but chaining it into another gateway isn’t an F5-published pattern — validate streaming pass-through and tool/function-call payloads in a spike before committing.

Option C — out-of-band: agentgateway calls Guardrails as a scanner

My favorite long-term shape. agentgateway stays the single inference path and invokes Guardrails’ ScanAPI as a side-scan in both directions, using agentgateway’s webhook prompt guards. Guardrails never proxies anything; it just renders verdicts. The only code you write is a thin adapter (~100 lines) that translates agentgateway’s webhook contract to POST /backend/v1/scans:

Guardrails outcome = flagged → webhook rejects (client gets a 4xx)
Guardrails returns redactedInput → webhook modifies (masked content goes forward)
otherwise → allow

flowchart LR C["AI apps / agents"] --> AGW["agentgateway"] AGW -. "request webhook" .-> AD["guardrails adapter (~100 lines)"] AD -. "POST /backend/v1/scans scanDirection=request" .-> GR["F5 AI Guardrails projects · scanners · audit"] AGW --> LLM["LLM providers"] AGW -. "response webhook" .-> AD

sequenceDiagram participant App as AI app participant AGW as agentgateway participant AD as adapter participant GR as F5 Guardrails participant LLM as LLM provider App->>AGW: POST /v1/chat/completions AGW->>AD: promptGuard request webhook AD->>GR: POST /backend/v1/scans (scanDirection=request) GR-->>AD: outcome + redactedInput alt flagged AD-->>AGW: reject AGW-->>App: 403 policy violation else cleared / redacted AD-->>AGW: allow or modify AGW->>LLM: forward LLM-->>AGW: completion AGW->>AD: promptGuard response webhook AD->>GR: POST /backend/v1/scans (scanDirection=response) GR-->>AD: outcome AD-->>AGW: allow / modify / reject AGW-->>App: final response end

Wiring it up on the agentgateway side is a policy targeting your LLM route:

apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayPolicy
metadata:
 name: f5-guardrails
 namespace: agentgateway-system
spec:
 targetRefs:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 name: llm-route
 backend:
 ai:
 promptGuard:
 request:
 - webhook:
 backendRef: { kind: Service, name: f5-guardrails-adapter, port: 8000 }
 response:
 - webhook:
 backendRef: { kind: Service, name: f5-guardrails-adapter, port: 8000 }

Choose this when you want agentgateway to keep full provider control (failover, streaming straight from the provider, budgets) while the security team keeps full policy control in the Guardrails console — clean separation of planes, one hop fewer on the token stream. Trade-off: you own the adapter, and scanning streamed responses forces a buffering decision (scan-on-complete vs. chunked) that you should prototype early. If this pattern spreads, the natural product evolution is native F5 ScanAPI support in agentgateway’s guardrail family, right next to Bedrock Guardrails and Model Armor.

The full picture: F5 XC on the edge

If you already run F5 Distributed Cloud, the AI stack slots in behind it the same way your web properties do. XC doesn’t host Guardrails — it contributes the hardened public edge, and an origin pool pointing at agentgateway:

flowchart TB U["Internet clients / AI apps"] --> XC["F5 XC HTTP Load Balancer TLS · WAF · bot defense · DDoS · API protection"] XC -->|origin pool| AGW["agentgateway (K8s or XC CE site) LLM + MCP + A2A gateway"] AGW -. "Option C webhook" .-> AD["guardrails adapter"] AD -.-> GR["F5 AI Guardrails SaaS us1/eu1.calypsoai.app or self-hosted"] AGW --> P1["OpenAI"] AGW --> P2["Anthropic"] AGW --> P3["vLLM / self-hosted"] AGW --> MCP["MCP tool servers"]

Every layer does the one thing it’s best at: XC absorbs the internet, agentgateway governs AI traffic, Guardrails judges content. And each layer is independently swappable — which is exactly what you want when this space is moving as fast as it is.

Choosing between them

	A: agw → F5	B: F5 → agw	C: out-of-band
Code required	none	none	small adapter
Client-facing endpoint	agentgateway	Guardrails	agentgateway
Provider routing/failover	in Guardrails	in agentgateway	in agentgateway
Extra proxy hop on token stream	yes	yes	no (verdicts only)
Streaming risk	F5-documented	needs spike	buffering choice on response scan
Best for	fastest PoC	security-team-owned front door	production, clean separation of planes

My recommendation: prove Option A in an afternoon (it’s configuration only), then build Option C for production. Option B is the niche play for orgs whose security team must terminate the client connection.

Runnable lab and test results

I put the working kind lab here:

👉 github.com/ProfessorSeb/agentgateway-demos/tree/main/202-agw-f5-ai

The lab deploys both recommended paths:

/option-a routes from agentgateway to the F5 AI Guardrails OpenAI-compatible inline endpoint.
/option-c routes from agentgateway directly to OpenAI, with F5 ScanAPI called out-of-band from request and response promptGuard webhooks.

The setup script creates two intentionally simple custom scanners in F5 AI Guardrails:

Scanner	Type	Match	Direction	Mode
`agw-lab-keyword-codename`	Keyword	`project-titan`	Prompts and responses	Block
`agw-lab-regex-ssn`	Regex	`\d{3}-\d{2}-\d{4}`	Prompts	Redact

That gives the test suite something concrete to prove. The smoke test lives in test.sh and sends six OpenAI Chat Completions requests through the gateway:

Test	Route	Expected result	What it proves
Option A benign	`/option-a`	`200`	agentgateway can reach the F5 inline OpenAI-compatible endpoint
Option C benign	`/option-c`	`200`	agentgateway can reach OpenAI directly while the webhook adapter passes clear prompts
Option A blocked codename	`/option-a`	`400` or `403`	F5 inline scanning blocks the custom `project-titan` keyword
Option C blocked codename	`/option-c`	`403`	the request webhook calls ScanAPI and rejects a blocked prompt
Option C SSN redaction	`/option-c`	`200`, no raw SSN	ScanAPI returns `redactedInput`, and the adapter replaces the user message before forwarding
Option C response-phase mask	`/option-c`	`200`, no blocked keyword	the response webhook scans the assistant output and masks blocked content

The important design choice is that Option C tests both directions. Request scanning catches unsafe user input before the model sees it; response scanning catches unsafe model output before the client sees it.

This is the traffic flow the tests exercise:

Run it after deployment:

cd agentgateway-demos/202-agw-f5-ai
./setup-guardrails.sh
./deploy.sh
kubectl port-forward -n agentgateway-system svc/agentgateway-proxy 8080:80
./test.sh

The passing output from my run:

PASS Option A benign: HTTP 200
PASS Option C benign: HTTP 200
PASS Option A blocked codename: HTTP 400
PASS Option C blocked codename: HTTP 403
PASS Option C SSN redaction request completed: HTTP 200
PASS Option C redaction did not leak raw SSN
PASS Option C response-phase scan completed: HTTP 200
PASS Option C response-phase scanner masked blocked output

The same run shows up in F5 AI Guardrails under Logs → Prompt history:

Those rows are the audit trail that matters in a real rollout:

Blocked / Scan / project-titan — Option C’s request webhook called ScanAPI before the model request and F5 returned a blocking verdict.
Redacted / Scan / Repeat back exactly: my ssn is *********** — the regex scanner caught the SSN pattern and returned redacted content instead of the raw value.
Blocked / Prompt / Genai Azure Openai — Option A hit the F5 inline OpenAI-compatible provider, so F5 logged the proxied prompt path as well as the scan decision.

This is the operational difference between “the gateway returned the right status code” and “security has evidence.” The terminal output proves agentgateway enforced the policy; the Guardrails logs prove the scanner decision, user, provider path, interaction text, and timestamp were recorded in the F5 audit plane.

There is also a fuller harness in run_harness.sh that reads harness/cases.yaml, records latency/status/usage metadata, and writes harness/results.jsonl. Use test.sh when you want a quick operational answer; use the harness when you want repeatable evidence for a write-up, demo, or CI job.

References

F5 AI Guardrails API docs — getting started with Defend, ScanAPI, providers, OpenAI-compatible endpoint, OpenAI SDK proxy integration
Runnable Options A and C lab
F5 AI security API integration examples (GitHub)
agentgateway docs — custom providers, webhook guardrails
F5 XC EMEA workshop, Class 6 Module 1 — Lab 2 (inline), Lab 3 (out-of-band)

Suspend & Resume Stateful Agents: kagent on Agent Substrate with kind

Thu, 25 Jun 2026 10:00:00 -0400

Suspend & Resume Stateful Agents: kagent on Agent Substrate with kind

By Sebastian Maniak

In this how-to we’ll stand the whole thing up on a throwaway kind cluster: Agent Substrate, kagent wired to use it as its execution layer, and a tiny SandboxAgent running as a gVisor actor. Then we’ll chat with it and watch it suspend and resume.

This is a standalone, run-it-yourself adaptation of the Agent Substrate with kagent Instruqt workshop. No Instruqt account required — just a Linux box.

Why This Setup?

Serverless economics for stateful agents — idle actors cost (almost) nothing; you pay for a small pool of warm workers, not one pod per session.
Sub-second resume — gVisor checkpoint/restore brings a suspended session back where it left off, no cold-start penalty.
Disposable environment — the whole thing is one kind cluster. Break it, delete it, recreate it.
kagent-native — kagent 0.9.7 is the first release that can use substrate as its execution layer, so a declarative agent becomes a substrate actor with one field: platform: substrate.

The Substrate Model in 30 Seconds

Agent Substrate decouples actor lifecycle from pod lifecycle:

An Actor is one logical agent session — state RUNNING or SUSPENDED.
A Worker is a pre-warmed pod hosting at most one actor — state IDLE or BUSY.
Idle actors are suspended: gVisor checkpoints their full state to object storage and the worker returns to the pool.
On the next request the actor is resumed sub-second into a free worker — exactly where it left off.

Term	Meaning
Actor	One logical agent session.
Worker	A pre-warmed pod hosting at most one actor.
WorkerPool	A set of warm standby worker pods (a CRD).
ActorTemplate	The immutable “class” an actor is created from (a CRD). Creating one builds a golden snapshot (version 0).
Golden snapshot	The initial frozen image every new actor is restored from.
Suspend / Resume	Checkpoint actor state to object storage / restore it sub-second.

Architecture

┌──────────────────────────── kind cluster (kagent-substrate) ────────────────────────────┐
│ │
│ namespace: kagent namespace: ate-system │
│ ┌────────────────────┐ ┌──────────────────────────────────────┐ │
│ │ kagent-controller │ controller.substrate.* │ ate-api-server (scheduling) │ │
│ │ kagent-ui │ ───────────────────────▶│ atenet-router (L7 routing) │ │
│ │ SandboxAgent │ │ │ atelet (node supervisor) │ │
│ │ hello-substrate │ │ │ valkey-cluster (state) │ │
│ │ WorkerPool │ │ │ rustfs (snapshots/S3) │ │
│ │ kagent-default │ │ └──────────────────────────────────────┘ │
│ └────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────────────────────┘

Layer	What it does
Agent Substrate (`ate-system`)	Kubernetes-native runtime that maps many stateful actors onto a small pool of pre-warmed worker pods. Idle actors are checkpointed (RAM + filesystem, via gVisor) to object storage and resumed sub-second on demand.
kagent (`kagent`)	The agent control plane / runtime, wired to use substrate as its execution layer.
`SandboxAgent`	A per-session declarative agent that runs as a gVisor actor on the substrate.

Prerequisites

You need a Linux machine (or a Linux VM) with at least ~8 vCPUs and 16 GB RAM — the substrate control plane runs a 6-node valkey cluster plus several other components, so a tiny machine will struggle.

macOS / Windows note: gVisor checkpoint/restore relies on Linux kernel features. Run this on Linux — a cloud VM such as a GCP n1-standard-8, an EC2 m5.2xlarge, or equivalent works well. Docker Desktop on Mac/Windows runs containers in a Linux VM and may not support the gVisor checkpointing path reliably.

Tools and the versions this guide is pinned to:

Tool	Version	Why
Docker	any recent	`kind` runs the cluster in containers
`kind`	`v0.27.0`	Kubernetes-in-Docker
Node image	`kindest/node:v1.32.2`	k8s 1.31+ required — substrate CRDs use CEL `format` rules
`kubectl`	matches cluster
`helm`	v3	installs the charts
`grpcurl`	`v1.9.1`	drive the substrate `ate-api` directly
OpenAI API key	—	the agent makes real LLM calls

Pinned component versions used throughout:

Component	Version
Agent Substrate	`0.0.6`
kagent (OSS)	`0.9.7`
gVisor actor image	`ghcr.io/kagent-dev/substrate/ateom-gvisor:v0.0.6`

⚠️ Why the node image matters: the substrate 0.0.6 CRDs use CEL validation rules that call the format library (format.dns1123Label / dns1123Subdomain), added in Kubernetes 1.31. An older kind default node image will reject the CRDs. Pin kindest/node:v1.32.2 or newer.

Step 0 — Install Tooling

Run these on a Linux host. Skip any tool you already have at a compatible version.

# kubectl
curl -fsSLo /usr/local/bin/kubectl \
 "https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x /usr/local/bin/kubectl

# helm v3
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# kind v0.27.0 (pin it — older kind ships an older default node image)
curl -fsSLo /usr/local/bin/kind \
 "https://kind.sigs.k8s.io/dl/v0.27.0/kind-linux-amd64"
chmod +x /usr/local/bin/kind

# grpcurl v1.9.1
curl -fsSL "https://github.com/fullstorydev/grpcurl/releases/download/v1.9.1/grpcurl_1.9.1_linux_x86_64.tar.gz" \
 | tar -xz -C /usr/local/bin grpcurl
chmod +x /usr/local/bin/grpcurl

Make sure Docker is running:

sudo systemctl start docker # if applicable
docker info >/dev/null && echo "docker OK"

Export your OpenAI key (the agent makes real LLM calls):

export OPENAI_API_KEY="sk-...your-key..."
[[ -n "${OPENAI_API_KEY:-}" ]] && echo "key set (len=${#OPENAI_API_KEY})" || echo "OPENAI_API_KEY is empty!"

⚠️ Footgun: don’t inline the assignment on the helm line like OPENAI_API_KEY=... helm ... --set ...="${OPENAI_API_KEY}" — the variable is expanded before the assignment runs and you’ll silently pass an empty string. Export it first, as above.

Pin a few env vars for convenience:

export KIND_CLUSTER=kagent-substrate
export SUBSTRATE_VERSION=0.0.6
export KAGENT_VERSION=0.9.7

Step 1 — Create the kind Cluster

kind create cluster --name "${KIND_CLUSTER}" --image kindest/node:v1.32.2 --wait 120s

Verify it’s ready:

kubectl get nodes
kubectl cluster-info

You should see one node in Ready state. Confirm your tools resolve:

kind version
helm version --short
grpcurl --version

Step 2 — Install Agent Substrate

Everything lands in the ate-system namespace. The substrate 0.0.6 chart defaults to JWT auth (ServiceAccount tokens), so a stock kind cluster works — no feature gates, no custom kind config.

Install the CRDs:

helm upgrade --install substrate-crds \
 oci://ghcr.io/kagent-dev/substrate/helm/substrate-crds \
 --version 0.0.6 \
 --namespace ate-system --create-namespace --wait

Install the control plane (this pulls several images and starts the valkey cluster — give it a few minutes):

helm upgrade --install substrate \
 oci://ghcr.io/kagent-dev/substrate/helm/substrate \
 --version 0.0.6 \
 --namespace ate-system --wait --timeout 10m

Watch it come up:

kubectl get pods -n ate-system

Wait until you see all of these Running (a couple of init Jobs will show Completed):

Pod	Role
`ate-api-server`	Control plane: actor lifecycle, scheduling, suspend/resume. Bypasses kube-scheduler.
`ate-controller`	Reconciles `WorkerPool` + `ActorTemplate` CRDs into Deployments.
`atelet` (DaemonSet)	Node supervisor: pulls images, manages sandbox lifecycle, talks to object storage.
`atenet-router`	Envoy L7 router; resolves which worker serves each request.
`valkey-cluster-0`..`-5`	State store: actor/worker records and locks.
`rustfs`	In-cluster S3-compatible object storage holding snapshot images.

Inspect the CRDs:

kubectl get crd | grep ate.dev
kubectl get workerpools.ate.dev -A

There are no WorkerPools yet — kagent creates one in the next step.

Step 3 — Install kagent, Wired to Substrate

kagent is the agent control plane; substrate is the execution layer underneath it. kagent 0.9.7 is the first release with substrate support.

⚠️ Order matters: the kagent controller hard-fails (crash-loops) if the substrate API isn’t reachable at startup — which is why substrate had to be installed first.

Install the kagent CRDs:

helm upgrade --install kagent-crds \
 oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \
 --version 0.9.7 \
 --namespace kagent --create-namespace --wait

Install kagent with substrate enabled:

helm upgrade --install kagent \
 oci://ghcr.io/kagent-dev/kagent/helm/kagent \
 --version 0.9.7 --namespace kagent --timeout 10m --wait \
 --set providers.default=openAI \
 --set providers.openAI.apiKey="${OPENAI_API_KEY}" \
 --set controller.substrate.enabled=true \
 --set controller.substrate.ateApiEndpoint="dns:///api.ate-system.svc:443" \
 --set controller.substrate.ateApiInsecure=true \
 --set controller.substrate.atenetRouterURL="http://atenet-router.ate-system.svc:80" \
 --set controller.substrate.ateApiTokenFile="/var/run/secrets/tokens/ate-api/token" \
 --set controller.substrate.defaultWorkerPool.namespace=kagent \
 --set controller.substrate.defaultWorkerPool.name=kagent-default \
 --set substrateWorkerPool.create=true \
 --set substrateWorkerPool.name=kagent-default \
 --set substrateWorkerPool.replicas=1 \
 --set substrateWorkerPool.ateomImage=ghcr.io/kagent-dev/substrate/ateom-gvisor:v0.0.6

What the substrate flags mean:

controller.substrate.enabled=true — turn on the integration
controller.substrate.ateApiEndpoint — where the substrate control plane lives
controller.substrate.atenetRouterURL — the substrate request router
controller.substrate.defaultWorkerPool.* — the pool agents land on by default
substrateWorkerPool.create=true + .replicas=1 — create one warm worker

If helm times out while the controller waits on its database (it restarts a couple of times during cold start), wait for it manually and continue:
kubectl wait deploy/kagent-controller -n kagent --for=condition=Available --timeout=10m

Verify the WorkerPool and the integration:

kubectl get workerpools.ate.dev -A

You should see kagent/kagent-default.

kubectl run substrate-status-check -n kagent --rm -i --restart=Never \
 --image=curlimages/curl:8.10.1 -- \
 http://kagent-controller:8083/api/substrate/status

The response should include "enabled": true.

Step 4 — Deploy a SandboxAgent

A kagent SandboxAgent with platform: substrate is a per-session declarative agent that runs as a substrate actor. Creating one generates an ActorTemplate, which triggers the golden snapshot — the version-0 frozen image every session is restored from. The first snapshot takes ~60–90s.

The runtime must be Go — Python ADK isn’t compatible with gVisor checkpointing.

Write the manifest:

cat > hello-substrate.yaml <<'YAML'
apiVersion: kagent.dev/v1alpha2
kind: SandboxAgent
metadata:
 name: hello-substrate
 namespace: kagent
spec:
 type: Declarative
 description: Tiny declarative agent running inside a substrate actor
 declarative:
 runtime: go
 modelConfig: default-model-config
 systemMessage: |
 You are a friendly assistant living inside an Agent Substrate sandbox.
 When asked who you are, say "I am hello-substrate, a Go ADK declarative
 agent running inside a gVisor actor."
 platform: substrate
 substrate:
 workerPoolRef:
 name: kagent-default
YAML

Apply it:

kubectl apply -f hello-substrate.yaml

Wait for it to be Ready (first golden snapshot takes ~60–90s):

kubectl wait sandboxagent/hello-substrate -n kagent --for=condition=Ready --timeout=5m

Inspect the generated substrate resources:

kubectl get sandboxagent -n kagent
kubectl get actortemplates.ate.dev -A

kagent generated an ActorTemplate for your agent (owned by the SandboxAgent).

Step 5 — Chat, Then Watch Suspend / Resume

When you chat with hello-substrate, a per-session gVisor actor is restored from the golden snapshot, runs the LLM call, and snapshots itself back to object storage — returning the worker to the pool. Between requests the actor sits SUSPENDED; on the next request it’s RESUMED sub-second.

5a. Open the kagent UI

Port-forward the UI and open it in your browser:

kubectl -n kagent port-forward svc/kagent-ui 8080:8080 --address 0.0.0.0

Open http://localhost:8080 (or http://<vm-ip>:8080). Pick kagent/hello-substrate from the Agents list and send:

What are you, and where are you running? Answer in one sentence.

You should get back something like:

I am hello-substrate, a Go ADK declarative agent running inside a gVisor actor.

Then open the Substrate page (/substrate) in the UI to see the worker pool and actors.

5b. Drive the actor lifecycle from the CLI

In another terminal, port-forward the substrate API in the background:

kubectl port-forward -n ate-system svc/api 18443:443 >/tmp/pf-api.log 2>&1 &
sleep 3

Mint a short-lived token the kagent controller’s identity can use:

TOKEN=$(kubectl create token kagent-controller -n kagent --audience=api.ate-system.svc --duration=15m)

List the actors — note your actor’s id and its status (it will be SUSPENDED between requests):

grpcurl -insecure -H "authorization: Bearer $TOKEN" -d '{}' \
 localhost:18443 ateapi.Control/ListActors

Resume the actor explicitly — watch it flip to RUNNING:

grpcurl -insecure -H "authorization: Bearer $TOKEN" \
 -d '{"actor_id":"<ACTOR_ID>"}' \
 localhost:18443 ateapi.Control/ResumeActor

Re-run ListActors to confirm the state change. That round trip — SUSPENDED in object storage, RESUMED into a worker on demand — is the whole point of substrate.

Recap

You now have, on a single kind cluster:

Agent Substrate (ate-system) — control plane (ate-api-server), router (atenet-router), node supervisor (atelet), state (valkey), snapshots (rustfs).
kagent (kagent) — wired to substrate with a default WorkerPool.
A SandboxAgent running as a per-session gVisor actor, suspended to object storage between requests and resumed sub-second on demand.

Scaling the WorkerPool

One worker serves many declarative sessions sequentially because each session releases its slot the moment it snapshots back. To run overlapping sessions or long-lived agents, scale the pool:

kubectl scale workerpool kagent-default -n kagent --replicas=3

What’s Next (Beyond This Guide)

AgentHarness (runtime: substrate) — long-lived runtimes run as substrate actors reached through a kagent gateway. These need an object-storage bucket (e.g. gs://...) and don’t auto-suspend, so each pins a worker slot.
Identity — substrate can mint per-actor JWTs and certs for mTLS.
Observability — substrate exposes metrics for activation latency and worker pool use.

Troubleshooting

Symptom	Likely cause / fix
Substrate CRDs rejected on install	Node image too old — recreate the cluster with `kindest/node:v1.32.2` or newer.
`kagent-controller` crash-loops	Substrate API not reachable at startup — confirm `ate-system` pods are all `Running` before installing kagent.
kagent install silently “works” but the agent fails on chat	`OPENAI_API_KEY` was empty — re-export it (don’t inline the assignment on the helm line) and re-run the kagent `helm upgrade`.
Helm install times out on kagent	The controller restarts during cold start; run `kubectl wait deploy/kagent-controller -n kagent --for=condition=Available --timeout=10m` and continue.
`ListActors` returns auth errors	Token expired (15m TTL) — re-mint with the `kubectl create token` command.
Everything was fine yesterday, broken today	Substrate’s in-cluster TLS certs expire after ~24h on idle clusters — best run in a single sitting. Recreate the cluster if needed.

Note: Agent Substrate is very early / pre-1.0 — APIs will change and it’s not production-ready. gVisor checkpoint/restore requires a live Linux host to test.

Cleanup

kind delete cluster --name kagent-substrate

Try It Yourself

The full standalone how-to (with the exact manifests used here) lives in the kagent-demos repo. Prefer a guided, browser-based lab with no setup? Run the Agent Substrate with kagent Instruqt workshop instead — same path, six bite-sized challenges.

Suspend-and-resume is the feature that finally makes per-session stateful agents affordable at scale. Spin it up, chat with hello-substrate, and watch a worker serve far more sessions than it has slots for.

AgentGateway Standalone — Cost & Tokenomics Dashboard Demo

Wed, 24 Jun 2026 00:00:00 +0000

Every time I show someone AgentGateway’s Costs and Analytics dashboards, the same problem comes up: a fresh install has no data. The dashboard is empty until you’ve sent real traffic through it, which means you either wait days for usage to accumulate or you script a load generator and pay for thousands of throwaway API calls just to make some charts light up.

The 00-standalone-latest demo solves that. It spins up AgentGateway standalone in a single Docker container with a pre-populated cost database — 5,000 simulated requests spanning 7 days, each priced against a real per-model cost catalog — so the dashboard is fully alive the moment it boots. No waiting, no burned tokens.

👉 sebbycorp/agentgateway-demos / 00-standalone-latest

TL;DR: export OPENAI_API_KEY=… && ./setup.sh, then open http://localhost:15000/ui/. You get a working AgentGateway with a costs dashboard that already has a week of fleet traffic in it — and the LLM endpoint is live on :4000 so you can add your own real requests on top.

What you actually get

The demo is deliberately small — five files do all the work:

File	Role
`setup.sh`	One-shot installer: preflight checks, generates mock data, writes config, launches the container
`config.yaml`	AgentGateway config — admin UI, SQLite database, model catalog, OpenAI routes
`base-costs.json`	Per-model pricing rates (OpenAI, Anthropic, Bedrock, Gemini, Mistral, DeepSeek…) so every request gets a real dollar cost
`gen-mock-logs.py`	Generates realistic fleet traffic and writes it into AGW’s `request_logs` schema
`destroy.sh`	Tears the whole thing down

The trick that makes it work: the mock generator writes to the exact same request_logs table that AgentGateway’s dashboard reads from, and the model catalog (base-costs.json) prices every row. So from the dashboard’s point of view, the seeded data is indistinguishable from real traffic.

Prerequisites

You need very little:

Docker installed and running
curl (the script downloads the mock-data generator)
Python ≥ 3.11 or uv to run the generator
An OPENAI_API_KEY — required because the live LLM route on :4000 proxies to OpenAI. (The dashboard data is mock; the live endpoint is real.)

Run it

git clone https://github.com/sebbycorp/agentgateway-demos.git
cd agentgateway-demos/00-standalone-latest

export OPENAI_API_KEY='sk-...'
./setup.sh

That’s the whole thing. setup.sh walks through:

Preflight — checks for Docker + a running daemon, curl, a Python runner, and OPENAI_API_KEY.
Fetch the generator — downloads gen-mock-logs.py if it isn’t already local.
Generate mock data — creates a SQLite DB with 5,000 requests across 7 days (both configurable, see below) using AGW’s request_logs schema.
Write config — emits the config.yaml pointing the admin UI at that database and loading the cost catalog.
Launch — pulls the image, creates a named volume, seeds the DB, and starts the container.

When it finishes, open:

http://localhost:15000/ui/

and head to the Costs and Analytics sections. They’re already full.

Tuning the seed

Three environment variables let you change the shape of the seeded data before you run setup.sh:

VERSION=v1.3.1 \  # AgentGateway image tag
REQUESTS=20000 \  # number of mock requests (default 5000)
DAYS=30 \  # days to spread them across (default 7)
./setup.sh

Bump REQUESTS and DAYS if you want to demo what a busier fleet or a longer reporting window looks like.

What’s under the hood: config.yaml

The generated config is worth a look because it shows how cost tracking is wired:

config:
 adminAddr: "0.0.0.0:15000" # admin UI + dashboards (reachable from host)
 database:
 url: "sqlite:///data/data.db" # /data is the mounted ./data dir in the container
 modelCatalog:
 - file: /base-costs.json  # per-model rates so every request is priced

llm:
 port: 4000
 policies:
 cors:
 allowOrigins: ["*"]
 allowHeaders: ["*"]
 allowMethods: ["GET", "POST", "OPTIONS"]
 models:
 - name: "openai/gpt-4.1"
 provider: openAI
 params:
 model: gpt-4.1
 apiKey: "$OPENAI_API_KEY"
 - name: "openai/*" # fallback: cheaper nano model
 provider: openAI
 params:
 model: gpt-4.1-nano
 apiKey: "$OPENAI_API_KEY"

Two things to call out:

modelCatalog is what turns raw token counts into dollars. base-costs.json carries input/output/cache rates for dozens of models across OpenAI, Anthropic, Bedrock, Gemini, Mistral, DeepSeek and more — including tiered pricing for large-context models. Every request in the dashboard is costed against it.
The openai/* fallback route quietly downshifts anything that doesn’t match a named model to the cheaper gpt-4.1-nano — a nice pattern for keeping unbudgeted traffic from hitting your most expensive model.

The container is launched roughly like this (the script handles it for you):

docker run -d --name agw-cost-demo \
 --user 0:0 \
 -p 4000:4000 -p 15000:15000 \
 -e OPENAI_API_KEY \
 -v config.yaml:/config.yaml \
 -v base-costs.json:/base-costs.json \
 -v agw-cost-demo-data:/data \
 cr.agentgateway.dev/agentgateway:v1.3.1 -f /config.yaml

Port	What it serves
15000	Admin UI — the Costs & Analytics dashboards
4000	Live LLM endpoint (OpenAI-compatible `/v1/chat/completions`)

Add real traffic on top

Because :4000 is a live OpenAI-compatible endpoint, you can fire real requests at it and watch them land in the same dashboard alongside the mock data:

curl -s http://localhost:4000/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "openai/gpt-4.1",
 "messages": [{"role": "user", "content": "Say hello in one sentence."}]
 }'

Refresh the dashboard and your call shows up — priced against the same catalog as the seeded rows. This is the best way to convince a skeptical teammate that the cost numbers are real: send a couple of calls and watch the spend tick up.

Tear it down

When you’re done, the demo cleans up after itself:

./destroy.sh

or do it by hand:

docker rm -f agw-cost-demo && docker volume rm agw-cost-demo-data

No leftover containers, no stray volumes.

Why this demo is useful

Standalone AgentGateway is the fastest way to understand what the gateway sees about your LLM spend — which models cost what, where the tokens go, how a fallback route changes the bill. But “fastest” still normally means “after you’ve generated enough traffic to have something to look at.” This demo collapses that to a single command by separating the dashboard data from the live path: mock data makes the charts immediately meaningful, and the real :4000 route lets you prove the pricing is genuine whenever you want.

If you’ve read my earlier posts on tool-mode token economics or stacking Headroom on top of AGW, this is the dashboard those experiments report into — now you can stand it up in two minutes and explore it yourself.

👉 Grab it here: sebbycorp/agentgateway-demos / 00-standalone-latest

agentgateway Standalone: A Cost & Tokenomics Dashboard in One Command

Wed, 24 Jun 2026 00:00:00 +0000

Introduction

You’re routing LLM traffic through a gateway. But do you actually know what it costs? Not the rough monthly invoice from your provider — the real breakdown. Which model burned the most tokens last night? Which user is driving 80% of your spend? Which provider is quietly eating your budget?

agentgateway answers those questions out of the box. Every request that flows through the proxy is priced against a per-model rate catalog and surfaced in a built-in Costs and Analytics dashboard. No external observability stack, no Prometheus, no Grafana — just the standalone binary.

This guide gets you from zero to a fully populated tokenomics dashboard in a single command. We’ll use a Docker-based demo that seeds 5,000 simulated requests across 7 days, so the dashboard has something interesting to show you the moment it boots — then we’ll send real traffic through it and watch it get priced live.

Why This Matters

Cost visibility is the FinOps story for AI. As soon as more than one team, agent, or app starts calling LLMs through shared infrastructure, “what did this cost and who spent it?” becomes a board-level question. agentgateway answers it at the gateway layer, which means:

Per-model pricing — every request is priced against a rate catalog (input, output, and cache token rates), so dollars show up next to tokens automatically.
Group by anything — slice spend and tokens by model, provider, user, group, or user agent (Cursor, Claude Code, openai-python, etc.).
Zero application changes — your apps just point at the gateway. The accounting happens in the proxy, not in your code.
One binary — the dashboard ships inside agentgateway. There’s no separate metrics pipeline to stand up for basic cost visibility.

What You’ll Build

 ┌──────────────────────────────┐
 ┌──────────────┐ │ agentgateway │
 │ Your apps / │ /v1/chat/ │ ┌────────────────────────┐ │ ┌────────────┐
 │ agents /curl │───completions──────▶│ │ LLM proxy (port 4000) │──┼─────▶│ OpenAI │
 └──────────────┘ │ └───────────┬────────────┘ │ └────────────┘
 │ │ priced per │
 │ │ model catalog │
 │ ┌───────────▼────────────┐ │
 ┌──────────────┐ localhost:15000 │ │ Admin UI + Dashboard │ │
 │ Your browser│────────────────────▶│ │ Costs / Analytics │ │
 └──────────────┘ │ └───────────┬────────────┘ │
 │ │ │
 │ ┌───────▼───────┐ │
 │ │ SQLite data.db │ │
 │ │ request_logs │ │
 │ └───────────────┘ │
 └──────────────────────────────┘

agentgateway proxies LLM traffic on port 4000 and serves its admin UI and dashboards on port 15000. Every request is written to a SQLite database (data.db) and priced using a model catalog (base-costs.json). The mock generator writes to the same request_logs schema, which is why the dashboard is populated before you send a single real request.

Prerequisites

Docker installed and running
curl
An OpenAI API key
Python 3.11+ (or uv) — used by the mock-data generator

Quick Start: One Command

Clone the demo and run the setup script:

git clone https://github.com/sebbycorp/agentgateway-demos.git
cd agentgateway-demos/00-standalone-latest

export OPENAI_API_KEY='sk-...'
./setup.sh

That’s it. Open http://localhost:15000/ui/ and head to the Costs and Analytics sections — they’re already full of data.

What the script does

setup.sh is a single-command bootstrap. Under the hood it:

Runs preflight checks — verifies Docker is installed and the daemon is running, curl is available, OPENAI_API_KEY is set, and that uv or Python 3.11+ is present.
Downloads the mock-data generator (gen-mock-logs.py).

Generates a SQLite database with simulated fleet traffic:

gen-mock-logs.py --replace --requests 5000 --days 7 -o data/data.db

Writes config.yaml pointing the gateway at the database and the model rate catalog.
Pulls the agentgateway image (cr.agentgateway.dev/agentgateway:v1.3.1) and removes any previous demo container.
Seeds a named Docker volume (agw-cost-demo-data) with the generated database.

Launches the container, binding ports to loopback only:

127.0.0.1:4000:4000 # LLM proxy
127.0.0.1:15000:15000 # admin UI + dashboards

Why loopback only? The proxy port carries your API credentials. Binding to 127.0.0.1 keeps the demo off your network. Don’t expose these ports without locking down auth and CORS first.

Tuning the dataset

Want a bigger or smaller demo dataset? Override the REQUESTS and DAYS environment variables before running setup:

REQUESTS=20000 DAYS=30 ./setup.sh

Exploring the Dashboard

Open http://localhost:15000/ui/ and click Analytics. By default it shows total traffic over the last 24 hours — token volume per hour with a running tally of cost, tokens, and calls.

The real power is in Group by. Switch it to Provider and the same traffic splits out by backend — here OpenAI dominates with ~13.3M tokens, followed by Anthropic, Google, and Bedrock. The breakdown table underneath ranks every provider by token consumption.

Switch Group by to User and you get per-person accounting — exactly the view you need when you’re trying to figure out who’s driving spend. Each bar in the time series is stacked by user, and the breakdown ranks them by tokens consumed.

You can group by Model, Provider, User, Group, or User agent (Cursor, Claude Code, openai-python, codex, bifrost, and more), and switch the Measure between tokens and cost. The Costs page focuses the same data on dollars, and Export lets you pull the underlying numbers out for reporting.

How It Works: The Generated Config

The setup script writes a config.yaml that wires everything together. Here are the pieces that matter:

config:
 adminAddr: "0.0.0.0:15000" # admin UI + dashboards (reachable from host)
 database:
 url: "sqlite:///data/data.db" # /data is the mounted ./data dir in the container
 modelCatalog:
 - file: /base-costs.json  # per-model rates so every request is priced

llm:
 port: 4000
 policies:
 cors: # demo-only: wildcard CORS. Safe because the port
 allowOrigins: ["*"] # is loopback-bound. Restrict this for real use.
 allowHeaders: ["*"]
 allowMethods: ["GET", "POST", "OPTIONS"]
 models:
 - name: "openai/gpt-4.1"
 provider: openAI
 params:
 model: gpt-4.1
 apiKey: "$OPENAI_API_KEY"
 - name: "openai/*" # fallback: cheaper nano model
 provider: openAI
 params:
 model: gpt-4.1-nano
 apiKey: "$OPENAI_API_KEY"

Three things make the dashboard work:

database.url — the admin UI’s Costs/Analytics pages read from this SQLite database. The mock generator writes the same request_logs schema, so generated traffic and real traffic land in one place.
modelCatalog — base-costs.json holds per-model input/output (and cache) token rates. This is what turns raw token counts into dollars.
models — two routes: an explicit openai/gpt-4.1 and a wildcard openai/* that falls back to the cheaper gpt-4.1-nano.

Send Real Traffic

The mock data gets you a populated dashboard, but the gateway is live — send it a real request and watch it get priced alongside the simulated traffic:

curl -X POST http://localhost:4000/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "openai/gpt-4.1",
 "messages": [{"role": "user", "content": "Hello from agentgateway!"}]
 }'

Refresh the Analytics page and your request shows up — tokens counted, cost calculated against the model catalog, attributed to the model and provider. Every real call from here on is accounted for the same way.

Manual Installation

Prefer to run the binary directly instead of the Docker demo? You have three options:

# 1. Automated installer
curl -sL https://agentgateway.dev/install | bash

# 2. Download a platform-specific binary from the GitHub releases page
# https://github.com/agentgateway/agentgateway/releases

# 3. Run via Docker with your own config mounted
docker run --rm \
 -p 127.0.0.1:4000:4000 \
 -p 127.0.0.1:15000:15000 \
 -v "$(pwd)/config.yaml:/config.yaml" \
 cr.agentgateway.dev/agentgateway:v1.3.1 --file /config.yaml

Point it at a config.yaml like the one above, and the proxy listens on port 4000 with the admin UI on 15000. From there it’s the same dashboard — minus the pre-seeded mock data.

Cleanup

When you’re done, tear the demo down with the included script:

./destroy.sh

This stops and removes the container and the named volume.

Wrapping Up

Cost and token visibility is one of those things you don’t realize you’re missing until a bill lands. agentgateway puts it right in the box: per-model pricing, a built-in dashboard, and grouping by model, provider, and user — no external observability stack required. The Docker demo gets you a populated dashboard in one command so you can see exactly what it looks like before pointing real traffic at it.

Resources

Do AgentGateway Tool Modes and Headroom Stack? Measuring Two Token-Saving Layers Together

Mon, 22 Jun 2026 00:00:00 +0000

In a previous post I showed that Enterprise agentgateway can cut your LLM bill by changing how GitHub’s MCP tools are presented to the model — its toolMode (Standard / Search / Code) shrinks the tool-catalog tax, the 4,781 tokens of tool schema re-injected on every turn.

But that’s only one line item on the bill. Headroom attacks a different one: it’s a compression proxy that shrinks the content payload — the verbose GitHub JSON results, file contents, and conversation history — before they ever reach the model. It claims 60–95% token savings, including “GitHub issue triage 73%.”

So the obvious question: they touch different layers, so do the savings stack? If AgentGateway removes the catalog tax and Headroom removes the payload tax, do you get both by running them together — or does one make the other redundant?

I measured it. All the code, manifests, harness, judge, dashboard, and raw results are in the demo repo:

👉 sebbycorp/agentgateway-demos / 105-ent-headroom-comp-tokenomics

TL;DR: Yes — but only when there’s real payload to compress, and not for free. On a large repo, Headroom stacks on top of every AGW mode (Code −63%, Search −36%). On a small, catalog-dominated repo it barely helps and actually makes Standard mode 49% more expensive by busting the provider’s prompt cache. And it trades a little accuracy on exact-identifier tasks.

Two knobs, two layers

This is the whole mental model, so it’s worth being precise:

	What it reduces	How
AGW tool modes	the tool-catalog tax + orchestration round-trips	Search: 28 tools → 2 meta-tools (−91% catalog). Code: N calls → 1 round-trip, summary-only result
Headroom	the content payload + history	ML/AST/JSON compressors (SmartCrusher, CodeCompressor, the Kompress model), reversibly

AGW shrinks the schemas and round-trips. Headroom shrinks the result data and history. Because they act on different parts of the request, they can — in principle — compound. The experiment is whether they actually do.

The setup at a glance

Component	Value
Backend	GitHub’s external remote MCP server — `api.githubcopilot.com/mcp/readonly` (28 read-only tools)
Front door	Enterprise agentgateway (`toolMode` Standard/Search/Code; injects the GitHub PAT as a Bearer token)
Compression layer	Headroom proxy (`headroom proxy`, OpenAI-compatible), upstream pointed at the AGW `/openai` route
Model	`gpt-5.5` via the agentgateway `/openai` route
Scope	A small repo (`sebbycorp/agw-tokenomics-sandbox`) and a larger one (`sebbycorp/k8s-iceman`), both read-only-pinned
Matrix	3 tool modes × Headroom OFF/ON × 2 repos = 12 cells, 5 questions each + an LLM-judge quality score

Architecture — two independent switches in one pipeline

The key insight that makes the test clean: the two knobs sit on different arrows.

 ┌─ Headroom OFF ─► AGW /openai ──► OpenAI gpt-5.5
harness ──build request──>┤
 (catalog baked in └─ Headroom ON ──► Headroom proxy ─► AGW /openai ─► OpenAI
 per AGW tool mode) (compresses payload + history)
 │
 └──/mcp/gh-{std,search,code}── AGW (toolMode) ──TLS+PAT──► GitHub remote MCP

Knob 1 — AGW toolMode acts on the MCP catalog (the /mcp/gh-* arrow).
Knob 2 — Headroom acts on the LLM request body (the /openai arrow). OFF = harness posts straight to AGW; ON = it posts to the Headroom proxy, which compresses and forwards to AGW.

Crucially, the AGW catalog effect is independent of which LLM URL is used: the harness bakes the tool catalog into the request from the MCP tools/list response, so the two knobs stay orthogonal. Headroom forwards to AGW (not OpenAI directly) so AGW still supplies the model + key and traces every call.

A gotcha worth flagging. Headroom is OpenAI-compatible at /v1/chat/completions, and you point its upstream at AGW with OPENAI_TARGET_API_URL. But out of the box, its semantic cache and CCR tool-injection broke the Search/Code tool-orchestration flow — every Search/Code request through Headroom failed with an MCP TaskGroup error until I launched it with --no-cache --no-ccr-inject-tool --no-ccr-marker. In front of an MCP tool-calling agent, those flags are mandatory. (A quick sanity check that compression is actually happening: the same 29 KB tool-result blob went 10,859 → 9,772 prompt tokens through the proxy.)

The numbers

Single run, gpt-5.5 list-price, cache-aware. Δ = the Headroom saving (ON vs OFF): positive means ON is cheaper, negative means ON is more expensive.

Small repo — `agw-tokenomics-sandbox`

AGW mode	OFF cost	ON cost	Headroom Δ	OFF quality	ON quality
Standard	$0.0289	$0.0432	−49% (worse)	5.0	5.0
Search	$0.0149	$0.0138	+7%	5.0	4.2
Code	$0.0315	$0.0204	+35%	5.0	4.8

Large repo — `k8s-iceman`

AGW mode	OFF cost	ON cost	Headroom Δ	OFF quality	ON quality
Standard	$0.0424	$0.0391	+8%	5.0	4.2
Search	$0.0269	$0.0173	+36%	4.8	4.4
Code	$0.0623	$0.0233	+63%	3.4	4.6

The live Grafana dashboard (metrics pushed via Prometheus pushgateway) shows the blended cross-repo picture — Search drops from $0.0209 → $0.0155, a 26% stacked saving:

And the per-repo + quality detail — note Search’s first-call tool context is just 429 tokens (the AGW catalog win, independent of Headroom), and the answer-quality panel that keeps the comparison honest:

Why it comes out this way

1. On a large repo, the layers stack. Big result payloads give Headroom something to compress, and AGW has already collapsed the catalog. The biggest stack is Code + Headroom (−63%) — Code returns summaries only, and Headroom squeezes those further. The cheapest reliable cell overall is Search + Headroom ($0.0173): AGW removes the −91% catalog tax, Headroom removes the result-JSON tax.

2. On a small repo, Headroom can backfire. With little result data to compress, its upside is small — and on Standard mode it loses 49%. The reason is subtle but important: Headroom rewrites the prompt on every turn, which busts gpt-5.5’s prefix cache. On a small workload the big, stable 28-tool catalog is exactly what caches well, so the cache you destroy is worth more than the payload you compress. Payload compression needs payloads.

3. Cheaper isn’t free. The judge flagged it: the “list recent commits” question repeatedly scored 2/5 under Headroom, because compression mangles high-entropy commit SHA hashes. For tasks that depend on verbatim identifiers — hashes, tokens, IDs — lossy text compression is risky. Most other answers held at 4–5/5.

When to run both

Large result payloads (logs, file contents, big JSON) → YES: AGW Search/Code + Headroom stacks (−36% to −63%)
Small, catalog-dominated workload → AGW alone; Headroom can hurt (prefix-cache busting)
Tasks needing exact IDs (SHAs, tokens, hashes) → be careful: Headroom can mangle them
Any MCP tool-calling agent → Headroom needs --no-cache --no-ccr-* to not break

They’re complementary, not competing — AGW on the catalog, Headroom on the payload — and on the right workload they genuinely compound.

Reproduce it yourself

git clone https://github.com/sebbycorp/agentgateway-demos.git
cd agentgateway-demos/105-ent-headroom-comp-tokenomics

cp .env.example .env # AGENTGATEWAY_LICENSE_KEY, OPENAI_API_KEY, GITHUB_PAT (read-only), REPO_LARGE
set -a; . .env; set +a
./deploy.sh # kind + AGW + GitHub MCP backends + installs the Headroom proxy

./test.sh # one question, Headroom OFF vs ON
REPO_LARGE=owner/big-readonly-repo ./run_matrix.sh # full 12-cell matrix + LLM-judge

A Grafana dashboard (AGW Tool Modes + Headroom) visualizes cost, tokens, and quality OFF vs ON. You can even replay a saved run into Prometheus with observability/replay_to_pushgateway.py — no extra LLM spend:

kubectl port-forward svc/grafana -n observability 3001:80

Clean up with ./cleanup.sh.

Takeaways

AGW tool modes and Headroom stack on large payloads — large repo: Code −63%, Search −36%. Different layers, real compounding.
Headroom is payload-dependent. No payload, no benefit — and it can regress cache-friendly workloads by busting the prompt cache (Standard, small repo: +49% cost).
Mind the accuracy cost. Compression mangles exact identifiers like commit SHAs; gate any compression layer with an answer-quality check.
Headroom is not a clean drop-in for MCP. Its semantic cache + CCR tool-injection break tool-calling; disable them with --no-cache --no-ccr-inject-tool --no-ccr-marker.
Measure for your own workload. This is a single run on two repos — the verdict flips with payload size, just as the 104 tool-modes study flipped with catalog size.

The full repo, manifests, harness, judge, and dashboard: 👉 github.com/sebbycorp/agentgateway-demos/tree/main/105-ent-headroom-comp-tokenomics

Related reading:

GitHub MCP Token Economics: Why Search Mode Cuts Your LLM Bill by ~60%

Sat, 20 Jun 2026 00:00:00 +0000

Every time an LLM talks to an MCP server, it has to be told what tools exist. That tool catalog — the JSON schema of every tool, its parameters, and descriptions — is injected into the prompt on every single turn. For a small server that’s a rounding error. For GitHub’s MCP server, with 28 read-only tools and verbose schemas, it’s 4,781 tokens of pure overhead per call — before the model has done any actual work.

That overhead is the hidden tax of MCP, and it’s the subject of this post. Using Enterprise agentgateway in front of GitHub’s official remote MCP server, I measured three different ways to expose those tools to an LLM and tracked the real USD cost of each. The result is clear and a little counter-intuitive: for a large catalog like GitHub’s, Search mode cuts per-call cost by ~60% and wins across conversations too.

All the code, manifests, harness scripts, and raw results are in the demo repo:

👉 sebbycorp/agentgateway-demos / 104-ent-github-tokenomics

This article walks through how it was deployed, what the numbers were, why they came out this way, and how you can flip your own gateway into Search mode.

The setup at a glance

Component	Value
Backend	GitHub’s external remote MCP server — `api.githubcopilot.com/mcp/readonly`
Tools exposed	28 read-only tools (`get_`, `list_`, `search_*`)
Front door	Enterprise agentgateway (injects the GitHub PAT as a `Bearer` token upstream)
Model	`gpt-5.5` via the agentgateway `/openai` route
Scope	One dedicated public sandbox repo: `sebbycorp/agw-tokenomics-sandbox`
Pricing (est.)	input `$0.005`/1K · cached input `$0.0025`/1K · output `$0.015`/1K
Method	5 questions × 3 modes (single-call) + a 5-turn conversation × 3 modes, run three times

A key safety property: every question is pinned to a single repo, the gateway only talks to the /mcp/readonly endpoint (no mutation tools), and the GitHub PAT is fine-grained and scoped to that one repo. The test physically cannot touch anything else.

Architecture

agentgateway sits between the LLM and GitHub. There is no MCP pod to build or run — GitHub’s MCP server is external and remote. The gateway targets it over TLS and injects your PAT. The same gateway also fronts the OpenAI-compatible model route, so one proxy sees both sides of the conversation and can meter tokens.

The three tool modes

The whole experiment hinges on how the gateway presents GitHub’s 28 tools to the model. Enterprise agentgateway’s MCP backend supports three toolMode values, each a different point on the “context vs. round-trips” trade-off.

Standard — the catalog goes in every turn

What the model sees: all 28 tools, full schemas.
Per-call context: ~4,781 tokens.
Workflow: model picks a tool and calls it directly — no discovery step.
Cost: you pay to re-send the entire catalog on every turn.

Search — progressive disclosure (the winner here)

What the model sees: just 2 meta-tools, get_tool and invoke_tool.
Per-call context: ~429 tokens — a 91% reduction.
Workflow: the model first discovers the relevant tool’s schema, then invokes it. That’s an extra round-trip, but each round-trip is tiny.
Cost: dramatically less context per turn; you only pull in the schema you actually need.

Code — one tool, batch in JavaScript

What the model sees: a single run_code tool (with a 10-second sandbox timeout).
Per-call context: ~3,021 tokens — a 37% reduction.
Workflow: the model writes JavaScript that calls multiple GitHub tools in one submission; only the final result returns to the transcript, keeping conversation history compact.
Cost: higher first-call overhead than Search, but it shines when individual results are large and can be batched/summarized server-side.

These three numbers are deterministic — they’re a property of the tool catalog and the mode, identical on every run:

Mode	Tools advertised	First-call tool tokens	vs Standard
Standard	28	4,781	—
Search	2	429	−91%
Code	1	3,021	−37%

How it was deployed

The whole stack comes up with one script on a local kind cluster.

Prerequisites: kind, kubectl, helm, python3 (≥3.10), and three secrets:

cp .env.example .env
# Fill in:
# AGENTGATEWAY_LICENSE_KEY (Enterprise agentgateway)
# OPENAI_API_KEY (gpt-5.5 backend)
# GITHUB_PAT (fine-grained, single-repo, READ-ONLY)

set -a; . .env; set +a
./deploy.sh

deploy.sh is idempotent and does the full bring-up:

Creates a kind Kubernetes cluster
Installs the Gateway API CRDs
Deploys the Enterprise agentgateway control plane
Creates the agentgateway-proxy Gateway
Wires the OpenAI (gpt-5.5) backend + /openai route
Configures three GitHub MCP backends — one per tool mode — all pointed at the external MCP server with the PAT injected

The OpenAI side is a plain AgentgatewayBackend:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: openai
 namespace: agentgateway-system
spec:
 ai:
 provider:
 openai:
 model: gpt-5.5
 policies:
 auth:
 secretRef:
 name: openai-secret

How to enable Search mode

This is the part most people want. The mode is a single field on the Enterprise MCP backend — spec.entMcp.toolMode. Here is the exact gh-search backend from the demo:

apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayBackend
metadata: { name: gh-search, namespace: agentgateway-system }
spec:
 entMcp:
 toolMode: Search  # <-- this is the whole trick
 targets:
 - name: github
 static:
 host: api.githubcopilot.com
 port: 443
 protocol: StreamableHTTP
 path: /mcp/readonly
 policies:
 tls: {}
 auth:
 secretRef: { name: github-pat }
 location:
 header: { name: Authorization, prefix: "Bearer " }

To get the other two modes, you change exactly one line: toolMode: Standard or toolMode: Code. The PAT lives in a Kubernetes Secret (github-pat) and the gateway adds Authorization: Bearer <PAT> on every upstream request, so the model never sees the credential. Each backend gets its own HTTPRoute (/mcp/gh-std, /mcp/gh-search, /mcp/gh-code) so the harness can hit all three side by side.

Note the path is /mcp/readonly, not /mcp/all/readonly — the former exposes only the 28 read tools; the latter would add mutation tools. For a cost benchmark (and for safety) we stay read-only.

The 5 questions

Every mode answered the same five questions against the sandbox repo:

Describe the repo: description, default branch, primary language.
List the 5 most recent commits with their messages.
List the open issues with their titles.
List the open pull requests with their titles.
List the files in the src/ directory.

Results

Everything below is captured live on the GitHub — MCP Tool Modes Grafana dashboard the demo ships with. Here’s the at-a-glance view after a 5-turn conversation per mode:

The numbers that follow break those panels down in detail.

Single call — one question, fresh session

Question	Standard	Search	Code
repo	$0.03239	$0.01347	$0.02330
commits	$0.03126	$0.01726	$0.02015
issues	$0.02707	$0.01451	$0.01994
prs	$0.03862	$0.00743	$0.01870
contents	$0.03921	$0.01246	$0.01905
average	$0.0337	$0.0130	$0.0202

Search is cheapest on all five questions — in every one of the three runs (~61% under Standard this run, 53–68% across runs). Code beats Standard but can’t overtake Search on a small repo: there’s too little per-task data for its batching to repay the higher first-call context.

Multi-turn conversation — cumulative cost per turn (latest run)

Turn	Standard	Search	Code
1	$0.0322	$0.0164	$0.0221
2	$0.0877	$0.0438	$0.0489
3	$0.1316	$0.0786	$0.0911
4	$0.1758	$0.1167	$0.1194
5	$0.2333	$0.1752	$0.1594

Totals after 5 turns:

Mode	Total tokens	Cache-read tokens	Cost	vs Standard
Standard	71,768	54,912	$0.233	baseline
Search	46,812	28,416	$0.175	−25%
Code	49,020	41,856	$0.159	−32%

Both Search and Code beat Standard. In this particular run Code edged out Search; in the prior two runs Search was cheapest. The constant across every run: Standard is the most expensive — re-sending GitHub’s catalog each turn is pricey, so avoiding it always pays.

Cache analysis (gpt-5.5 prompt caching)

Mode	Cache-read tokens (5-turn convo)	% of total
Standard	54,912	77%
Search	28,416	61%
Code	41,856	85%

Does the Search win survive a different cache discount?

A fair question: gpt-5.5’s cached input is ~50% off. What if your cache is cheaper?

Cache discount	cached $/1K	Standard	Search	Search ÷ Standard
50% off (used here)	$0.00250	$0.233	$0.175	0.75×
75% off	$0.00125	$0.165	$0.140	0.85×
90% off	$0.00050	$0.123	$0.118	0.96×
100% (cache free)	$0.00000	$0.096	$0.104	1.08×

At realistic cache rates Search clearly wins. The margin narrows as cache gets cheaper, and only at the theoretical free-cache extreme does Standard’s huge-but-cached catalog edge ahead. Net: Search wins at normal cache rates; it’s a toss-up only in the free-cache limit. Code, with the largest cached share, is the most cache-rate-robust of the three.

Reproducibility — three runs

Single-call average per task:

Run	Standard	Search	Code
1	$0.0410	$0.0150	$0.0335
2	$0.0295	$0.0140	$0.0200
3	$0.0337	$0.0130	$0.0202

Conversation cost after 5 turns:

Run	Standard	Search	Code
1	$0.250	$0.165	$0.256
2	$0.379	$0.171	$0.238
3	$0.233	$0.175	$0.159

Deterministic: first-call context (4,781 / 429 / 3,021) and tools advertised (28 / 2 / 1).
Stable: Search is cheapest on all 5 single questions, every run; Search & Code both beat Standard in conversation, every run; Search is the most predictable (~$0.17).
Noisy: Standard’s conversation cost ($0.23–$0.38) and the exact Search-vs-Code ordering.

The headline conclusion is robust to this noise.

Why the results came out this way

There are two costs in any MCP conversation, and they pull in opposite directions:

Catalog tax — the tool schemas re-sent every turn. Standard pays this in full (4,781 tokens), every single call. Search nearly eliminates it (429 tokens). Code reduces it (3,021 tokens).
Round-trip / transcript cost — Search’s discover-then-invoke pattern adds an extra model call, and a growing conversation re-sends history each turn.

For GitHub, the catalog tax dominates. The catalog is so large and verbose that paying it on every turn (Standard) swamps the small cost of Search’s extra discovery round-trips. So Search wins on single calls and holds up across conversations.

This is exactly why it’s worth measuring rather than assuming. In a previous demo (103, the F5 MCP server) the catalog was only ~1,588 tokens — small enough that the re-sent transcript, not the catalog, dominated. There, Standard actually won in conversation and Search was up to ~4.8× worse. Same three modes, opposite verdict:

	F5 (103), ~1,588-tok catalog	GitHub (104), ~4,781-tok catalog
Search, single call	~−18% vs Standard	~−60% vs Standard
Search, 5-turn convo	+380% (≈4.8× worse)	−25% to −55% (better)

Catalog size is the deciding variable. The bigger and chattier your MCP server’s tool definitions, the more Search mode saves you.

Why enabling Search matters

If you’re routing a real coding agent or chatops bot at GitHub’s MCP server, Standard mode means you’re paying a ~4,800-token surcharge on every turn of every conversation, forever. Flipping toolMode: Search:

Cuts single-call cost ~60% and conversation cost 25–55% in these tests.
Is the most predictable option — a tight ~$0.17 per 5-turn chat across all runs, while Standard swings 60%.
Scales with catalog growth — as GitHub (or any vendor) adds more tools, the catalog tax grows and Search’s advantage widens.
Costs you one line of YAML and a tiny bit of extra latency from the discovery round-trip.

Reach for Code mode instead when individual tool results are large (big file contents, long issue threads): batching multiple calls and returning only summaries server-side keeps the transcript small and can pull ahead of Search.

Reproduce it yourself

git clone https://github.com/sebbycorp/agentgateway-demos.git
cd agentgateway-demos/104-ent-github-tokenomics

cp .env.example .env # AGENTGATEWAY_LICENSE_KEY, OPENAI_API_KEY, GITHUB_PAT (read-only, single-repo)
set -a; . .env; set +a
./deploy.sh

kubectl port-forward deployment/agentgateway-proxy -n agentgateway-system 8080:80 &
kubectl port-forward svc/prometheus-prometheus-pushgateway -n observability 9091:9091 &

LLM_NO_TEMPERATURE=1 ./harness/.venv/bin/python harness/gh_questions.py # 5 Qs × 3 modes
LLM_NO_TEMPERATURE=1 ./harness/.venv/bin/python harness/gh_conversation.py # 5-turn chat × 3 modes

Point at a different repo with GH_REPO=owner/name, or go interactive:

./harness/.venv/bin/python harness/gh_chat.py search "list the open issues"

A Grafana dashboard (GitHub — MCP Tool Modes) visualizes the token and cost metrics:

kubectl port-forward svc/grafana -n observability 3001:80

Clean up with ./cleanup.sh.

Takeaways

For a verbose catalog like GitHub’s, the per-call context reduction is huge — Search −91%, Code −37% — and it translates directly into dollars.
Avoid Standard for a big catalog. It’s the most expensive and least predictable mode.
Search is the safe pick. Code can edge ahead when results are large; their order flips run-to-run on small data.
Caching helps every mode (~61–88% from cache) but doesn’t change the ranking.
Catalog size and result size decide the winner. The F5 demo had the opposite verdict. Measure for your catalog.

The full repo, manifests, and harness: 👉 github.com/sebbycorp/agentgateway-demos/tree/main/104-ent-github-tokenomics

Related reading:

Running agentgateway on Proxmox (LXC): the Everything MCP server end-to-end

Thu, 18 Jun 2026 00:00:00 +0000

Running agentgateway on Proxmox (LXC): the Everything MCP server end-to-end

Date: June 2026 Author: Sebastian Maniak Tags: agentgateway, mcp, proxmox, lxc, docker, llm, homelab

Overview

Most agentgateway tutorials assume you have a Kubernetes cluster lying around. This one doesn’t. If you run a homelab on Proxmox, the cheapest, fastest way to get a real agentgateway instance running is a single LXC container with Docker inside it — no cluster, no Helm, no control plane to babysit.

By the end of this guide you’ll have:

agentgateway v1.3.0 running in an unprivileged LXC container
The official “Everything” MCP server federated behind it
An OpenAI-compatible LLM proxy on port 4000
The built-in admin UI (logs, analytics, the connection playground) on port 15000
Request logging and cost tracking backed by SQLite — with no database errors

The whole thing fits in 2 vCPUs and 2 GB of RAM.

What is agentgateway, and where does the code live?

agentgateway is an open-source (Apache 2.0) connectivity data plane built specifically for agentic AI workloads. Rather than bolting agent traffic onto a traditional API gateway, it’s protocol- and session-aware from the ground up, and it unifies three things behind one binary:

LLM Gateway — a unified OpenAI-compatible API in front of many providers, with budgets and load balancing
MCP Gateway — tool federation across many MCP servers (stdio, HTTP/SSE, Streamable HTTP) behind one endpoint
A2A Gateway — secure agent-to-agent communication with capability discovery

It’s written mostly in Rust for performance and memory safety (which matters for the long-lived, high-concurrency sessions MCP and A2A produce), with a Go control surface and a TypeScript UI.

GitHub repo: https://github.com/agentgateway/agentgateway
Latest release: v1.3.0 (June 18, 2026)
Container image: cr.agentgateway.dev/agentgateway:v1.3.0
Docs: https://agentgateway.dev/docs/
License: Apache 2.0 · ⭐ ~3.4k stars

⭐ If this setup is useful to you, star the repo — it’s a young project (created March 2025) that just crossed the v1.x line, and the maintainers ship fast.

Architecture

A single LXC holds everything. Docker runs two containers — the Everything MCP server and agentgateway — and agentgateway talks to the MCP server over stdio via docker exec.

Proxmox host (vmbr0, 172.16.10.0/24)
└── LXC 210 "agentgateway" (172.16.10.164, DHCP)
 └── Docker
 ├── mcp-everything (modelcontextprotocol "everything" server)
 └── agentgateway v1.3.0
 ├── :4000 LLM proxy (OpenAI-compatible)
 ├── :3000 MCP gateway (Everything federated here)
 └── :15000 admin UI (logs + analytics + playground)

Prerequisites

Proxmox VE 8.x
A bridge (vmbr0) with DHCP on the 172.16.10.x network
Docker Hub access (or a local registry mirror)
An LLM provider API key if you want to exercise the LLM proxy (OpenAI in this example)

Step 1: Create the LXC container

We use an unprivileged Alpine container with nesting enabled so Docker can run inside it.

# Download the Alpine template
pveam download local alpine-3.20-default_20240908_amd64.tar.xz

# Create container (CTID 210)
pct create 210 /var/lib/vz/template/cache/alpine-3.20-default_20240908_amd64.tar.xz \
 --hostname agentgateway \
 --cores 2 \
 --memory 2048 \
 --net0 name=eth0,bridge=vmbr0,ip=dhcp \
 --rootfs local-lvm:8 \
 --unprivileged 1 \
 --features nesting=1,keyctl=1 \
 --onboot 1

pct start 210

The --features nesting=1,keyctl=1 flags are the important bit — without nesting, Docker won’t start inside an unprivileged container, and keyctl keeps Docker’s keyring happy.

You should now see the container in the Proxmox UI:

Tip: find the container’s DHCP-assigned IP with pct exec 210 -- ip -4 addr show eth0. Everything below assumes it landed on 172.16.10.164 — substitute your own.

Step 2: Install Docker inside the LXC

pct exec 210 -- apk update
pct exec 210 -- apk add docker docker-compose
pct exec 210 -- rc-update add docker boot
pct exec 210 -- service docker start

# Sanity check
pct exec 210 -- docker info

Step 3: Deploy the Everything MCP server

The Everything server is the reference MCP server from the Model Context Protocol project. It exposes every MCP primitive — tools, prompts, resources, sampling — which makes it perfect for verifying that your gateway is wired correctly before you connect anything real.

pct exec 210 -- docker run -d \
 --name mcp-everything \
 --restart unless-stopped \
 modelcontextprotocol/servers:latest everything

We keep this container alive and let agentgateway attach to it over stdio with docker exec (configured in the next step). That avoids exposing the MCP server’s raw port on the network — the only thing clients ever talk to is the gateway.

Step 4: Create the agentgateway configuration

Create /config/config.yaml inside the LXC. This single file wires up the LLM proxy, the MCP target, and CORS so the admin UI can call both planes from the browser.

config:
 database:
 url: sqlite:///config/data/data.db

llm:
 port: 4000
 models:
 - name: openai/*
 provider: openai
 params:
 model: gpt-4.1-mini
 apiKey: "sk-..."
 policies:
 cors:
 allowOrigins:
 - "http://172.16.10.164:15000"
 allowHeaders:
 - "*"
 allowMethods:
 - GET
 - POST
 providers: []
 virtualModels: []

mcp:
 port: 3000
 targets:
 - name: everything
 stdio:
 cmd: docker
 args:
 - "exec"
 - "-i"
 - "mcp-everything"
 - "node"
 - "dist/index.js"
 policies:
 cors:
 allowOrigins:
 - "http://172.16.10.164:15000"
 allowHeaders:
 - "*"
 allowMethods:
 - GET
 - POST
 exposeHeaders:
 - Mcp-Session-Id

frontendPolicies:
 http:
 maxBufferSize: 33554432

A few things worth understanding here:

database.url points at a SQLite file under /config/data. This is what powers the Logs and Analytics tabs in the UI — if it can’t be created, those tabs throw errors (see Troubleshooting).
The MCP stdio target literally runs docker exec -i mcp-everything node dist/index.js. agentgateway speaks JSON-RPC over that pipe, so the gateway and the MCP container share the same Docker daemon inside the LXC.
exposeHeaders: [Mcp-Session-Id] is required for browser-based MCP clients — MCP sessions are stateful, and the session ID rides in that header.
CORS allowOrigins must match the URL you open the UI from. If you reach the UI by hostname or a different IP, add that origin here too, or the browser will block the calls.
maxBufferSize: 33554432 (32 MB) gives MCP responses with large resource payloads room to breathe.

🔐 Don’t hardcode keys in production. Replace the inline apiKey: "sk-..." with an environment variable reference and pass it via -e OPENAI_API_KEY=... on the docker run. Inline is fine for a throwaway homelab box; it is not fine for anything shared.

Step 5: Run agentgateway

pct exec 210 -- mkdir -p /config/data
pct exec 210 -- docker run -d \
 --name agentgateway \
 --restart unless-stopped \
 -v /config:/config \
 -p 4000:4000 \
 -p 3000:3000 \
 -p 15000:15000 \
 -e ADMIN_ADDR=0.0.0.0:15000 \
 cr.agentgateway.dev/agentgateway:v1.3.0 \
 -f /config/config.yaml

ADMIN_ADDR=0.0.0.0:15000 binds the admin UI to all interfaces so you can reach it from your workstation — by default it only listens on loopback, which is invisible from outside the container.

Tail the logs to confirm a clean start:

pct exec 210 -- docker logs -f agentgateway

Step 6: Verify everything works

Open the UI:

http://172.16.10.164:15000/ui

You should see:

LLM proxy on port 4000
MCP server (Everything) on port 3000, with its tools listed in the playground
Admin UI on port 15000
Logs and Analytics populated (no database errors)

Quick smoke tests from your workstation:

# LLM proxy — OpenAI-compatible, so any OpenAI SDK/curl works
curl http://172.16.10.164:4000/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "openai/gpt-4.1-mini",
 "messages": [{"role": "user", "content": "Say hello from Proxmox"}]
 }'

# MCP gateway — list the tools Everything exposes through the gateway
curl http://172.16.10.164:3000/ \
 -H "Content-Type: application/json" \
 -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

Here’s what the working UI looks like:

Troubleshooting

Docker won’t start in the LXC. You almost certainly created the container without nesting=1. Add it with pct set 210 --features nesting=1,keyctl=1 and restart the container.

Logs/Analytics tabs show a database error. The /config/data directory didn’t exist when the container started, so SQLite couldn’t create the file. Run pct exec 210 -- mkdir -p /config/data, then recreate the container (the -v /config:/config mount persists it).

UI loads but API calls fail with CORS errors. The origin you opened the UI from isn’t in allowOrigins. Match it exactly — scheme, IP/host, and port — for both the llm and mcp CORS policies.

MCP target shows no tools. Confirm the Everything container is up (pct exec 210 -- docker ps) and that the stdio args point at the right entrypoint (node dist/index.js). docker logs mcp-everything will tell you if it crashed on start.

Final notes

The container IP is 172.16.10.164 (DHCP) — pin a reservation on your router if you want it stable.
All important ports are bound to 0.0.0.0; if this box isn’t on a trusted LAN, put it behind a firewall or reverse proxy with auth.
The Everything MCP server is connected via Docker exec stdio, so it’s never exposed directly.

This gives you a production-shaped agentgateway — LLM proxy, MCP federation, observability, and a UI — running on a single Proxmox LXC. From here, swap the Everything server for real MCP servers, add more LLM providers to the models list, or layer in RBAC and rate limiting from the agentgateway docs.

Go deeper

Source & releases: https://github.com/agentgateway/agentgateway
Documentation: https://agentgateway.dev/docs/
Everything MCP server: https://github.com/modelcontextprotocol/servers/tree/main/src/everything

One-Script Deployment: agentgateway + Self-Hosted Langfuse on Kubernetes for LLM Cost Analysis

Sat, 13 Jun 2026 00:00:00 +0000

Running production-grade LLM gateways with full observability usually involves many manual steps across Helm charts, CRDs, tracing configuration, and UI key management.

The agentgateway-demos/09-k8s-langfuse demo eliminates almost all of that friction by making agentgateway the central cost-control and observability plane.

agentgateway sits in front of your models, inspects every request/response, emits rich GenAI OpenTelemetry traces (model name, input/output tokens, full prompt + completion, latency, user/session attribution), and forwards them directly to Langfuse. Once you set per-model pricing in Langfuse, you get automatic USD cost tracking, spend dashboards, and usage analytics — all without any external SaaS or manual log shipping.

What You Get

A single ./deploy.sh that provisions:

A dedicated kind cluster (agw-k8s-langfuse)
Full self-hosted Langfuse (PostgreSQL + ClickHouse + Redis + web/worker) via the official Helm chart, tuned for kind
agentgateway CRDs, controller, GatewayClass, and proxy (v1.1.0)
Pre-configured AgentgatewayBackend + HTTPRoute routing /v1/* to your local OpenAI-compatible model (e.g. Qwen running on your host)
Direct OTLP/HTTP tracing from agentgateway → Langfuse (no collector sidecar required)

After one follow-up script with your project keys, agentgateway automatically sends every LLM call to Langfuse with the data needed for real cost management.

Quick Start (The Automated Path)

git clone https://github.com/sebbycorp/agentgateway-demos.git
cd agentgateway-demos/09-k8s-langfuse

./deploy.sh

The script handles:

kind cluster creation
Gateway API CRDs
Langfuse Helm install + phased readiness waits (databases first)
agentgateway controller + CRDs
Base AgentgatewayParameters
Gateway + spark backend + spark-route HTTPRoute

Langfuse bootstrap (ClickHouse especially) takes 5–15 minutes on first run. The script prints progress and continues.

Wiring Observability (Minimal Manual Step)

Once langfuse-web is running:

kubectl port-forward -n langfuse svc/langfuse-web 3000:3000 &
# Open http://localhost:3000, create a project, copy Public + Secret keys

Then:

export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...
./configure-observability.sh

This single script:

Computes the correct Basic Auth header Langfuse expects
Applies the complete AgentgatewayParameters resource with:
- OTLP/HTTP endpoint pointing at the in-cluster Langfuse service
- Proper Authorization header
- Full field mappings for gen_ai.prompt, gen_ai.completion, streaming flags, user.id, session.id, client IP, and environment tags
The agentgateway controller immediately reconciles the change and pushes the new tracing config to all data-plane proxies

From this point forward, agentgateway is responsible for capturing cost-relevant telemetry on every request and streaming it to Langfuse in real time. No additional instrumentation, sidecars, or log shipping is required.

How agentgateway Manages Costs & Sends Logs to Langfuse

agentgateway acts as your cost control plane:

Every OpenAI-compatible request hits the Gateway first
It extracts and enriches telemetry using the GenAI semantic conventions
It streams complete traces (including full prompt/completion bodies) over OTLP/HTTP directly to Langfuse’s /api/public/otel/v1/traces endpoint
Langfuse turns these traces into first-class “Generations” with token accounting

Key data agentgateway sends for cost management:

gen_ai.request.model
gen_ai.usage.input_tokens / gen_ai.usage.output_tokens
Full gen_ai.prompt and gen_ai.completion
user.id and session.id (via request headers for attribution and filtering)
Latency, streaming status, client IP, and environment tags

Once pricing is configured in Langfuse, you get accurate per-request and aggregated USD costs.

Test & Observe Costs

kubectl port-forward -n agentgateway-system svc/agentgateway-proxy 8080:80 &

./test.sh --users # realistic multi-user traffic with attribution headers
# or
./test.sh "Explain Kubernetes Gateway API in one paragraph."

agentgateway captures and forwards:

Exact model name and provider
Input and output token counts (gen_ai.usage.input_tokens, gen_ai.usage.output_tokens)
Full prompt and completion bodies
Latency and streaming status
User and session attribution (via x-user-id / x-session-id headers)

Open Langfuse → Traces / Generations. You’ll immediately see rich entries.

Then go to Project → Settings → Models and add pricing for your model (e.g. Qwen). Langfuse automatically calculates real USD costs per generation and aggregates them into spend dashboards. This is how you turn raw token logs into actionable cost management.

Why This Approach Wins

Zero collector complexity — agentgateway speaks Langfuse’s OTLP endpoint natively
Idempotent & reproducible — re-run deploy.sh safely
Local-model friendly — hostOverride pattern works cleanly inside kind
Production-ready patterns — the same tracing config translates directly to real clusters

Cleanup

./cleanup.sh

Removes the cluster and all resources.

Next Steps

Clone the demo and run it:

https://github.com/sebbycorp/agentgateway-demos/tree/main/09-k8s-langfuse

This is currently the fastest way to stand up a complete, observable AI gateway stack on Kubernetes for cost tracking and debugging.

Related reading:

Self-Hosted Langfuse with Docker + kagent Integration
LLM Observability with agentgateway + Langfuse
agentgateway Kubernetes docs: https://agentgateway.dev/docs/kubernetes/latest

agentgateway Standalone → Langfuse: Direct OTLP Tracing (No Collector)

Wed, 10 Jun 2026 23:00:00 -0400

I’ve already covered the production OTel Collector pattern for shipping agentgateway traces to Langfuse. This is the opposite end of the spectrum: the absolute minimum that works.

No collector. No sidecar. No extra containers. Just the agentgateway binary running standalone, pointing its OTLP exporter directly at a self-hosted Langfuse on another VM. If you want LLM tracing on your laptop or a single box in about five minutes, this is the path.

Full working config: sebbycorp/agentgateway-demos/08-standalone-langfuse

The data path

your app ──▶ agentgateway (:3000) ──▶ vLLM / Qwen backend
 │
 └── OTLP/HTTP traces ──▶ Langfuse VM (:3000/api/public/otel)

That’s the whole thing. agentgateway natively emits OpenTelemetry traces using the GenAI semantic conventions (gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.operation.name, …). Langfuse exposes a built-in OTLP receiver at /api/public/otel, so the gateway talks to it directly — no translation layer in between.

The only requirement: the box running agentgateway needs network access to the Langfuse host on port 3000.

Why no collector?

The collector pattern earns its keep when you need gRPC, batching, fan-out to multiple backends, or central enrichment at scale. For a single gateway sending to a single Langfuse, all of that is overhead.

	Standalone (this guide)	OTel Collector
Moving parts	1 binary	gateway + collector deployment
Protocol	OTLP/HTTP only	gRPC or HTTP
Auth handling	inline header (substituted at launch)	collector holds the secret
Best for	laptops, single VM, demos, MVP	clusters, multi-backend, prod

If you outgrow it, the collector article is the next step. For now, keep it simple.

Prerequisites

The agentgateway binary on your PATH (or sitting next to the config).
A reachable Langfuse instance. Mine is self-hosted at http://172.16.10.112:3000.
A Langfuse public key and secret key (Project Settings → API Keys).
An LLM backend. I’m pointing at a local vLLM serving Qwen/Qwen3.6-35B-A3B-FP8 at 172.16.10.173:8000.

Quick reachability check before you start:

curl -I http://172.16.10.112:3000 || echo "Cannot reach Langfuse VM"

The config

Everything lives in one config.yaml. Here’s the tracing block — the part that matters:

config:
 tracing:
 # Langfuse OTLP HTTP endpoint (gRPC is NOT supported by Langfuse ingest).
 # EU cloud: https://cloud.langfuse.com/api/public/otel
 # US cloud: https://us.cloud.langfuse.com/api/public/otel
 # Self-hosted: http://<host>:3000/api/public/otel
 otlpEndpoint: http://172.16.10.112:3000/api/public/otel/v1/traces

 # IMPORTANT: force HTTP (protobuf/json). The default is grpc.
 otlpProtocol: http

 headers:
 Authorization: '"Basic ${LANGFUSE_AUTH_STRING}"'
 x-langfuse-ingestion-version: '"4"'

 # 1.0 / true for dev. Lower this in prod (e.g. 0.1).
 randomSampling: true

Two things trip people up here, both of them about CEL:

otlpProtocol: http is mandatory. agentgateway defaults to gRPC, and Langfuse’s ingest endpoint does not speak gRPC OTLP. Leave it on the default and traces silently never arrive.
Header values are CEL expressions, not raw strings. That’s why the values are double-quoted: '"Basic ${...}"'. The outer quotes are YAML; the inner quotes make it a CEL string literal. Drop the inner quotes and the gateway tries to evaluate Basic ... as an expression and fails to parse.

Enriching the traces

The default GenAI spans are good, but a few extra fields turn them into something genuinely useful in Langfuse:

 fields:
 add:
 # The actual conversation → Langfuse maps these to trace input/output.
 # string() casts raw body bytes to text (else you get base64).
 gen_ai.prompt: 'string(request.body)'
 gen_ai.completion: 'string(response.body)'

 gen_ai.request.stream: 'json(request.body).stream'

 # Attribution / multi-tenant — pulled from request headers.
 user.id: 'request.headers["x-user-id"]'
 session.id: 'request.headers["x-session-id"]'

 environment: '"local-dev"'
 client.ip: 'source.address'

The user.id line is the one to notice — it’s what produces the per-user cost breakdown you’ll see in Langfuse later. No app changes required; the gateway is the instrumentation point.

The backend

binds:
- port: 3000
 listeners:
 - name: spark-http
 protocol: HTTP
 routes:
 - name: spark-route
 backends:
 - ai:
 name: spark
 provider:
 openAI:
 model: "Qwen/Qwen3.6-35B-A3B-FP8"
 hostOverride: "172.16.10.173:8000"

This uses the openAI provider type (vLLM speaks the OpenAI API) with hostOverride pointing at the local vLLM server.

The secret-substitution gotcha

You’ll notice the config contains the literal placeholder ${LANGFUSE_AUTH_STRING}. agentgateway does not expand environment variables when it reads YAML. If you source .env and launch the binary directly, it takes the literal string ${LANGFUSE_AUTH_STRING}, tries to parse it as CEL, and crashes:

Error: parse: ... Syntax error: token recognition error at: '$'
| Basic ${LANGFUSE_AUTH_STRING}

The fix is a tiny wrapper that substitutes the value into a temp file right before launch — keeping the real secret out of the committed config:

#!/bin/bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Load secrets from .env (gitignored)
set -a; source "${SCRIPT_DIR}/.env"; set +a

# Substitute into a throwaway config; clean it up on exit
CONFIG_FILE=$(mktemp)
trap 'rm -f "$CONFIG_FILE"' EXIT

if command -v envsubst >/dev/null 2>&1; then
 envsubst < "${SCRIPT_DIR}/config.yaml" > "$CONFIG_FILE"
else
 sed "s|\${LANGFUSE_AUTH_STRING}|${LANGFUSE_AUTH_STRING}|g" \
 "${SCRIPT_DIR}/config.yaml" > "$CONFIG_FILE"
fi

exec agentgateway -f "$CONFIG_FILE" "$@"

Build the auth string once and drop it in .env:

echo -n "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" | base64
# → put the result in .env as LANGFUSE_AUTH_STRING=...

This is the same pattern you use for any proxy that doesn’t do env expansion natively (Envoy-style tools). .env stays gitignored; config.yaml stays clean.

Run it

chmod +x run.sh
./run.sh

You should see the startup logs print the substituted tracing block — the Authorization header should now read Basic cG..., not the ${LANGFUSE_AUTH_STRING} placeholder. That one line is your fastest confirmation the substitution worked.

Verify the gateway in the UI

agentgateway ships a local UI. Open it and you should see the bind, the route, and the AI backend all wired to port 3000.

Listeners — one HTTP listener (spark-http) bound to port 3000:

Routes — spark-route attached to the spark-http listener, matching all hosts and paths:

Backends — the spark AI backend, provider OpenAI-compatible, model Qwen/Qwen3.6-35B-A3B-FP8:

Send some traffic

Anything that hits the OpenAI-compatible endpoint on port 3000 gets traced. The key is passing x-user-id and x-session-id headers so the gateway can attribute each trace:

curl -X POST "http://localhost:3000/v1/chat/completions" \
 -H "Content-Type: application/json" \
 -H "x-user-id: alice" \
 -H "x-session-id: demo-session-1" \
 -d '{
 "model": "Qwen/Qwen3.6-35B-A3B-FP8",
 "messages": [{"role": "user", "content": "Explain OTLP in one sentence."}],
 "temperature": 0.7,
 "stream": false
 }'

The repo’s test.sh automates this — it loops over a handful of demo users (alice, bob, carol, dave, erin) with different traffic weights and supports streaming via --stream, which is exactly how you generate the per-user breakdown below:

chmod +x test.sh
./test.sh # non-streaming
./test.sh --stream # SSE streaming

See it in Langfuse

Open your Langfuse project. Within a second or two of sending traffic (that’s what x-langfuse-ingestion-version: "4" buys you — real-time previews), traces start landing. Because the gateway emits proper GenAI conventions, Langfuse renders them as full generations with model, token counts, latency, and cost — even for a local Qwen model.

That User consumption panel is the payoff of the user.id field in the config — total spend broken out per user (alice $0.04328, bob $0.02147, and so on), all from a gateway that the LLM apps never had to be aware of.

Troubleshooting

Traces never appear.

Confirm the gateway box can reach Langfuse: curl -I http://172.16.10.112:3000.
Check the startup logs show the real Authorization: Basic cG... header, not the ${...} placeholder. If it’s still the placeholder, you launched the binary directly instead of via run.sh.
Make sure otlpProtocol: http is set — gRPC silently fails against Langfuse.

Syntax error: token recognition error at: '$' — the literal placeholder reached the binary. Use run.sh (or envsubst) so the secret is substituted before launch.

Inputs/outputs show as base64 in Langfuse — wrap the body fields in string(...), as in the fields.add block above.

When to graduate to a collector

Stay standalone for laptops, single VMs, demos, and MVPs — one binary, one config, done. Reach for the OTel Collector pattern when you need gRPC from the gateway, batching and retries under load, fan-out to more than one observability backend, or you simply don’t want the Langfuse secret living anywhere near the gateway process.

Same traces, same GenAI conventions — just a heavier, more flexible delivery path. Start here; grow into that.

Get the code

The complete, runnable setup — config.yaml, run.sh, and test.sh — lives in my demos repo:

→ github.com/sebbycorp/agentgateway-demos/tree/main/08-standalone-langfuse

Clone it, drop your keys into .env, and ./run.sh:

git clone https://github.com/sebbycorp/agentgateway-demos.git
cd agentgateway-demos/08-standalone-langfuse

Langfuse Integration with agentgateway (OTel Collector Pattern) for cost controls, observability

Wed, 10 Jun 2026 00:00:00 +0000

agentgateway emits rich OpenTelemetry traces for every LLM request, tool call, and policy decision. This guide shows the production-grade way to forward those traces to Langfuse using an OpenTelemetry Collector — including proper cost tracking even when using local models.

Why Go Through an OTel Collector?

Directly sending from agentgateway to Langfuse causes problems:

agentgateway parses OTLP headers as CEL expressions
A raw Authorization: Basic xxx header makes the proxy crash-loop
You lose easy fan-out to multiple observability backends

Best practice: agentgateway → OTel Collector (no auth) → Langfuse (with Basic auth)

This is the exact pattern running in production on the k8s-iceman cluster.

Architecture

This design keeps agentgateway clean while the collector handles authentication and enrichment.

1. Deploy the OpenTelemetry Collector

Use the OpenTelemetry Collector Contrib image with Basic Auth extension:

# helm-values/otel-collector/values.yaml
mode: deployment
fullnameOverride: otel-collector

image:
 repository: otel/opentelemetry-collector-contrib

extraEnvsFrom:
 - secretRef:
 name: langfuse-otel

ports:
 otlp:
 enabled: true
 containerPort: 4317
 servicePort: 4317

config:
 extensions:
 basicauth/langfuse:
 client_auth:
 username: ${env:LANGFUSE_PUBLIC_KEY}
 password: ${env:LANGFUSE_SECRET_KEY}

 receivers:
 otlp:
 protocols:
 grpc:
 endpoint: ${env:MY_POD_IP}:4317

 exporters:
 otlphttp/langfuse:
 endpoint: ${env:LANGFUSE_BASE_URL}/api/public/otel
 auth:
 authenticator: basicauth/langfuse

 service:
 extensions: [health_check, basicauth/langfuse]
 pipelines:
 traces:
 receivers: [otlp]
 processors: [memory_limiter, batch]
 exporters: [otlphttp/langfuse]

2. Configure agentgateway Tracing

Create the AgentgatewayParameters resource:

# manifests/agentgateway-config/langfuse-tracing.yaml
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayParameters
metadata:
 name: langfuse-tracing
 namespace: agentgateway-system
spec:
 env:
 - name: OTLP_ENDPOINT
 value: "http://otel-collector.kagent.svc.cluster.local:4317"
 - name: OTLP_PROTOCOL
 value: "grpc"
 - name: OTLP_HEADERS
 value: "{}" # Must be empty object
 rawConfig:
 config:
 tracing:
 randomSampling: true
 fields:
 add:
 span.name: '"agentgateway.request"'
 gen_ai.system: 'llm.provider'
 gen_ai.prompt: 'flattenRecursive(llm.prompt)'
 gen_ai.completion: 'flattenRecursive(llm.completion.map(c, {"role":"assistant", "content": c}))'
 gen_ai.usage.completion_tokens: 'llm.outputTokens'
 gen_ai.usage.prompt_tokens: 'llm.inputTokens'

Reference it from your agentgateway Helm values:

gatewayClassParametersRefs:
 agentgateway:
 name: langfuse-tracing
 namespace: agentgateway-system

3. Secrets via Vault + External Secrets

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
 name: langfuse-otel
spec:
 secretStoreRef:
 name: vault-backend
 kind: ClusterSecretStore
 target:
 name: langfuse-otel
 data:
 - secretKey: LANGFUSE_PUBLIC_KEY
 remoteRef: { key: iceman_langfuse, property: LANGFUSE_PUBLIC_KEY }
 - secretKey: LANGFUSE_SECRET_KEY
 remoteRef: { key: iceman_langfuse, property: LANGFUSE_SECRET_KEY }
 - secretKey: LANGFUSE_BASE_URL
 remoteRef: { key: iceman_langfuse, property: LANGFUSE_BASE_URL }

4. Cost Tracking for Local Models

Even when using local models (Qwen via vLLM), you can still get proper cost tracking in Langfuse.

Step 1: Define Model Pricing in Langfuse

Go to Settings → Models → Add model and create an entry.

Because agentgateway already emits gen_ai.usage.prompt_tokens and gen_ai.usage.completion_tokens, Langfuse will automatically calculate cost once the model name matches.

Step 2: Verify in Langfuse UI

After sending a few requests through agentgateway you should see:

Token usage columns populated
Cost column showing your configured price (even if $0)
Full prompt/completion with rich attributes

5. What You Get in Langfuse

Every request through agentgateway now appears with:

Full prompt and completion
Token counts (prompt_tokens, completion_tokens)
Model name
Cost (auto-calculated from your pricing)
Which gateway policy ran
MCP tool calls
Latency breakdown

Summary

This pattern gives you production-grade observability:

Clean auth separation via OTel Collector
Automatic cost tracking even for local models
No changes required in your agents
Easy to extend to metrics and logs later

This is the exact setup running on the k8s-iceman cluster.

Would you like a version that also includes kagent traces in the same diagram?

Self-Hosted Langfuse with Docker + kagent Integration

Wed, 10 Jun 2026 00:00:00 +0000

Running Langfuse as a self-hosted Docker stack gives you full control over your LLM observability data. This guide shows how to deploy it and integrate it with kagent so every agent trace, prompt, completion, and token count flows into Langfuse automatically.

Why Self-Hosted Langfuse?

Keep all prompts and completions inside your infrastructure
No usage limits or data leaving your cluster
Full control over retention and cost tracking
Works great with local models (vLLM, Ollama, etc.)

1. Run Langfuse with Docker

The fastest way to get Langfuse running is using their official Docker Compose stack.

git clone https://github.com/langfuse/langfuse.git
cd langfuse
docker compose up -d

After a minute, open http://localhost:3000 and create your first project.

Important credentials you’ll need later:

LANGFUSE_PUBLIC_KEY
LANGFUSE_SECRET_KEY
Base URL (usually http://langfuse:3000 inside the cluster or your external URL)

2. Configure kagent to Send Traces to Langfuse

kagent (v0.9.6+) has built-in OpenTelemetry support. The cleanest way to configure it is through Helm values.

Helm Values (kagent)

otel:
 tracing:
 enabled: true
 exporter:
 otlp:
 endpoint: "http://your-langfuse-host:3000/api/public/otel/v1/traces"
 protocol: "http/protobuf"
 insecure: true
 timeout: 15000

controller:
 env:
 - name: LANGFUSE_OTEL_AUTH
 valueFrom:
 secretKeyRef:
 name: langfuse-otel
 key: LANGFUSE_OTEL_AUTH
 - name: OTEL_EXPORTER_OTLP_TRACES_HEADERS
 value: "Authorization=Basic $(LANGFUSE_OTEL_AUTH)"

Create the Auth Secret

Create a secret containing the Basic Auth header:

apiVersion: v1
kind: Secret
metadata:
 name: langfuse-otel
type: Opaque
stringData:
 LANGFUSE_OTEL_AUTH: "<public_key>:<secret_key>" # base64 encoded later

Or let External Secrets Operator + Vault manage it (recommended for production).

3. Verify Traces Are Flowing

Once deployed:

Go to your kagent project in Langfuse
Trigger any agent (via UI, Telegram, Slack, etc.)
You should see traces appear within seconds showing:
- Full prompt + completion
- Token usage (prompt_tokens, completion_tokens)
- Model name
- Latency
- Cost (if you configured pricing)

4. Cost Tracking for Local Models

Since you’re likely using local models (Qwen via vLLM), set the price to $0 in Langfuse:

Settings → Models → Add model

Model name: Qwen/Qwen3.6-35B-A3B-FP8 (must match exactly)
Input price: 0
Output price: 0

This keeps the cost column clean while still capturing full usage data.

Summary

You now have:

Langfuse running self-hosted via Docker
kagent automatically sending every LLM call via OTLP
Token usage and (optional) cost tracking working

This setup is the foundation used in production clusters running kagent + agentgateway.

Next guide: How to extend this pattern to agentgateway using an OpenTelemetry Collector for cleaner auth handling.

Running kagent on a kind Cluster with a Local vLLM + Qwen3 Backend

Tue, 09 Jun 2026 16:00:00 -0400

Running kagent on a kind Cluster with a Local vLLM + Qwen3 Backend

By Sebastian Maniak

Most kagent walkthroughs assume you’re pointing at OpenAI or Anthropic. But what if you want to run the entire stack locally — agents, controller, and the model — with zero tokens leaving your lab?

That’s exactly what this demo does. We’ll spin up a throwaway kind cluster, install kagent with all the built-in agents disabled, and wire its default provider to a self-hosted vLLM endpoint serving Qwen/Qwen3.6-35B-A3B-FP8. From there you can build agents from a clean slate — no cloud LLM, no API bill, no data egress.

Why This Setup?

A few reasons this combo is worth your time:

Fully local inference — the model runs on your own GPU (in my case a DGX Spark), so prompts and cluster data never leave the building.
Disposable environment — kind means the whole cluster is one docker container. Break it, delete it, recreate it in 30 seconds.
Clean slate — disabling the bundled agents means kagent comes up as a bare control plane. You decide which agents exist, instead of inheriting eight of them.
OpenAI-compatible — vLLM exposes the standard /v1 API, so kagent’s openAI provider talks to it with nothing more than a baseUrl override.

Architecture

┌─────────────────────────────────────┐
│ kind cluster (kagent-demo) │
│ │
│ ┌────────────────────────────┐ │
│ │ kagent control plane │ │
│ │ (controller + UI) │ │
│ │ provider: openAI │ │
│ └──────────────┬─────────────┘ │
│ │ OpenAI /v1 API │
└──────────────────┼────────────────────┘
 │ (baseUrl override)
 ▼
 ┌────────────────────────┐
 │ vLLM server │
 │ Qwen3.6-35B-A3B-FP8 │
 │ DGX Spark · :8000/v1 │
 └────────────────────────┘

The cluster runs everything except the model. The model lives on a separate box (the DGX Spark) and is reached over the network at http://172.16.10.173:8000/v1. kagent doesn’t care that it’s not OpenAI — as far as it’s concerned, it’s just an OpenAI-compatible endpoint.

Prerequisites

kind
kubectl
helm
A reachable vLLM endpoint serving the Qwen/Qwen3.6-35B-A3B-FP8 model (see the DGX Spark config at the end)

Step 1: Create the kind Cluster

One command and you’ve got a single-node Kubernetes cluster running inside Docker:

kind create cluster --name kagent-demo

Step 2: Install the kagent CRDs

This installs the Custom Resource Definitions kagent depends on, and — thanks to --create-namespace — creates the kagent namespace that every later step relies on. Run it first.

helm install kagent-crds oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \
 --namespace kagent --create-namespace

Step 3: Create the vLLM API Key Secret

kagent’s openAI provider expects an API key, even when the backend doesn’t enforce one. vLLM typically ignores the key, so any non-empty value works:

kubectl create secret generic kagent-vllm \
 -n kagent \
 --from-literal=VLLM_API_KEY=dummykey

The kagent namespace already exists from Step 2, so this secret lands in the right place. Order matters here.

Step 4: Install kagent — Without the Default Agents

This is the interesting part. The kagent chart ships a set of built-in agents (k8s, helm, istio, cilium, kgateway, promql, observability, argo-rollouts). For this demo we disable all of them so we can build our own agents from scratch. We also point the default provider at our vLLM/Qwen endpoint — just adjust config.baseUrl to match your own server.

helm install kagent oci://ghcr.io/kagent-dev/kagent/helm/kagent \
 --namespace kagent \
 --set k8s-agent.enabled=false \
 --set helm-agent.enabled=false \
 --set istio-agent.enabled=false \
 --set promql-agent.enabled=false \
 --set observability-agent.enabled=false \
 --set argo-rollouts-agent.enabled=false \
 --set cilium-debug-agent.enabled=false \
 --set cilium-manager-agent.enabled=false \
 --set cilium-policy-agent.enabled=false \
 --set kgateway-agent.enabled=false \
 --set grafana-mcp.enabled=false \
 --set querydoc.enabled=false \
 --set providers.default=openAI \
 --set providers.openAI.provider=OpenAI \
 --set-string providers.openAI.model="Qwen/Qwen3.6-35B-A3B-FP8" \
 --set providers.openAI.apiKeySecretRef=kagent-vllm \
 --set providers.openAI.apiKeySecretKey=VLLM_API_KEY \
 --set-string providers.openAI.config.baseUrl="http://172.16.10.173:8000/v1"

A few things worth calling out:

Flag	What it does
`*-agent.enabled=false`	Disables each bundled agent so you start clean
`providers.default=openAI`	Makes the OpenAI-compatible provider the default
`providers.openAI.config.baseUrl`	The key override — points kagent at vLLM instead of api.openai.com
`providers.openAI.apiKeySecretRef`	References the dummy secret from Step 3
`--set-string` on the model	Forces the model name to stay a string (it has slashes/digits)

Step 5: Verify the Control Plane

Check that the control-plane pods come up and that no default agents were created:

kubectl get pods -n kagent
kubectl get agents -n kagent # should be empty

If get agents returns nothing, the disable flags did their job. 🎉

The kagent UI

Once the control plane is healthy, open the kagent UI. The Models view confirms the provider wiring — you should see a default-model-config pointing at OpenAI as the provider, the Qwen model ID, the kagent-vllm API key secret, and your vLLM baseUrl under provider parameters:

The Agents view starts empty — exactly what we wanted. Here I’ve created a single test agent from scratch, running on the local OpenAI (Qwen/Qwen3.6-35B-A3B-FP8) model:

From here the cluster is yours. Create agents, attach MCP tool servers, and every inference call routes to your local Qwen model instead of a cloud API.

Cleanup

When you’re done, tear the whole thing down with a single command:

kind delete cluster --name kagent-demo

No leftover namespaces, no orphaned resources — the entire cluster was a Docker container, and now it’s gone.

Bonus: The vLLM Server on DGX Spark

For completeness, here’s the exact docker run I use to serve Qwen3.6 on my DGX Spark. This is the endpoint kagent’s baseUrl points at:

docker run -d --name vllm-qwen36 \
 --restart always \
 --runtime nvidia --gpus all \
 --network host --ipc host --shm-size 96g \
 -v ~/.cache/huggingface:/root/.cache/huggingface \
 vllm/vllm-openai:cu130-nightly \
 --model Qwen/Qwen3.6-35B-A3B-FP8 \
 --host 0.0.0.0 --port 8000 \
 --gpu-memory-utilization 0.85 \
 --max-model-len 262144 \
 --max-num-seqs 16 \
 --max-cudagraph-capture-size 256 \
 --reasoning-parser qwen3 \
 --enable-auto-tool-choice \
 --tool-call-parser qwen3_coder

Highlights that make this work well for agentic workloads:

--enable-auto-tool-choice + --tool-call-parser qwen3_coder — lets the model emit proper tool calls, which is essential for kagent agents that invoke MCP tools.
--reasoning-parser qwen3 — separates the model’s reasoning trace from its final answer.
--max-model-len 262144 — a generous 256K context window, plenty of room for long agent conversations and tool outputs.
--gpu-memory-utilization 0.85 — leaves a little headroom on the GPU.

Wrapping Up

In under ten minutes you’ve got a disposable Kubernetes cluster running kagent with a fully local LLM backend — no cloud provider, no API key that matters, no data leaving your network. Because the model speaks the OpenAI API, kagent didn’t need any special integration: just a baseUrl swap.

This is my go-to scratchpad for prototyping agents before promoting them to a real cluster. The kind cluster is cheap to create and free to destroy, and the Qwen3.6 model on the DGX Spark is fast enough to make the loop feel interactive.

Next up: building a real agent on top of this and wiring it to MCP tool servers. Stay tuned.

Want more on kagent? Check out Human-in-the-Loop with kagent and Managing FortiGate Firewalls from Telegram with AI, MCP, and kagent.

Proxying All Your LLM Traffic Through agentgateway with Grok Build

Wed, 20 May 2026 00:00:00 +0000

Introduction

If you’re like me, your AI tooling has probably grown into a sprawling mess of API keys: Claude Code reads .claude/.env, Cursor uses OpenAI keys, Langflow points at its own Anthropic endpoint, and your local agents each have their own hardcoded tokens scattered across config files. Every new tool means another key to manage, another endpoint to remember, and another blind spot in your observability stack.

The alternative is elegant: route every single LLM request through a single gateway that handles authentication, observability, rate limiting, and routing — then point all your tools at that one endpoint.

In this guide, I’ll walk you through exactly how I set up this architecture using Solo.io Enterprise agentgateway as the LLM traffic proxy and Grok Build as one of the clients. My production cluster routes traffic to three backends:

OpenAI (gpt-4o) — cloud API via api.openai.com
xAI Grok (grok-4.3) — cloud API via api.x.ai
DGX Spark (Qwen/Qwen3.6-35B-A3B-FP8) — local on-prem inference server at 172.16.10.173:8000

The key insight is that agentgateway speaks the OpenAI API contract. Every backend — regardless of whether it’s OpenAI, xAI, Anthropic, or a local Qwen — is exposed as a standard /v1/chat/completions endpoint. Your tools don’t need to know or care what’s behind the proxy.

Architecture Overview

Here’s the high-level picture:

┌─────────────┐ ┌─────────────────────────────────────────────┐
│ Grok Build │ │ │
│ Claude Code│───→│ agentgateway Proxy (Kubernetes) │
│ Cursor │ │ │
│ Langflow │ │ Gateway → HTTPRoute → AgentgatewayBackend │
│ Your CLI │ │ │
└─────────────┘ └────────┬──────────┬──────────────┬──────────┘
 │ │ │
 ┌──────▼───┐ ┌────▼─────┐ ┌─────▼──────┐
 │ OpenAI │ │ xAI Grok │ │ DGX Spark │
 │ gpt-4o │ │ grok-4.3 │ │ Qwen 3.6 │
 └──────────┘ └──────────┘ └────────────┘

All LLM requests flow through the same agentgateway instance — the same IP, same port, same observability pipeline — but different HTTP path prefixes route to different backends:

Path Prefix	Backend	Model	Location
`/openai`	OpenAI	`gpt-4o`	Cloud (api.openai.com)
`/grok`	xAI	`grok-4.3`	Cloud (api.x.ai)
`/spark`	DGX Spark	`Qwen3.6-35B-A3B-FP8`	On-prem (172.16.10.173)

The Problem with Scattered LLM Configuration

Before agentgateway, my setup looked like this:

Grok Build: base_url: http://172.16.10.173:8000/v1, model Qwen/Qwen3.6-35B-A3B-FP8
Claude Code: ANTHROPIC_API_KEY in .env, points at api.anthropic.com
Cursor: OpenAI key hardcoded in settings, points at api.openai.com
Custom agents: Various scripts with keys in environment variables, .env files, or inline

The pain points were obvious:

API key sprawl — every tool had its own key stored somewhere different
No unified observability — I couldn’t see total LLM spend, compare model performance, or debug issues across providers
Local model isolation — my on-prem Qwen model on DGX Spark was only accessible to tools that could reach 172.16.10.173, with no unified routing
Secret management — no centralized rotation, no audit trail

After agentgateway, it’s all one URL with path-based routing. Every request goes through the same gateway that logs, traces, and authenticates — regardless of which LLM ultimately serves it.

The Architecture: Three Layers

agentgateway uses three Kubernetes resource types to define each backend connection:

1. AgentgatewayBackend — “Where is the LLM?”

The backend resource tells agentgateway about an LLM provider. Here are all three of mine:

OpenAI backend — Cloud API with API key auth:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: openai
 namespace: agentgateway-system
spec:
 ai:
 provider:
 openai:
 model: gpt-4o
 policies:
 auth:
 secretRef:
 name: openai-secret

xAI Grok backend — Cloud API with TLS and SNI:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: xai-grok
 namespace: agentgateway-system
spec:
 ai:
 provider:
 openai:
 model: grok-4.3
 host: api.x.ai
 port: 443
 pathPrefix: /v1
 policies:
 auth:
 secretRef:
 name: xai-secret
 tls:
 sni: api.x.ai

DGX Spark backend — Local on-prem Qwen model, no auth needed:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: dgx-spark-llm
 namespace: agentgateway-system
spec:
 ai:
 provider:
 openai:
 model: Qwen/Qwen3.6-35B-A3B-FP8
 host: 172.16.10.173
 port: 8000
 pathPrefix: /v1

Key observations:

The OpenAI provider in agentgateway isn’t just for OpenAI — it implements the OpenAI-compatible API contract. That’s why it works for xAI (which exposes an OpenAI-compatible endpoint) and for the DGX Spark’s vLLM server.
The DGX Spark backend points directly at the on-prem inference server. No cloud egress, no internet required.
Cloud backends (OpenAI, xAI) use secretRef for API key auth. The local DGX Spark has no auth — it’s on the internal network.

2. Gateway — “Where does the traffic enter?”

A Gateway resource defines the proxy listener. I have one main gateway for cloud providers and dedicated gateways for local models (useful when you want separate NodePort allocations or network policies):

Main gateway (cloud LLMs):

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: agentgateway-proxy
 namespace: agentgateway-system
spec:
 gatewayClassName: enterprise-agentgateway
 listeners:
 - name: http
 protocol: HTTP
 port: 80
 allowedRoutes:
 namespaces:
 from: All

Dedicated DGX Spark gateway (separate NodePort for the on-prem model):

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: dgx-spark-gateway
 namespace: agentgateway-system
spec:
 gatewayClassName: enterprise-agentgateway
 listeners:
 - name: http
 protocol: HTTP
 port: 80
 allowedRoutes:
 namespaces:
 from: All

Both use the enterprise-agentgateway GatewayClass, which is created automatically by the Enterprise agentgateway Helm chart. On my bare-metal Talos cluster, these are exposed via NodePort:

Gateway	NodePort
`agentgateway-proxy`	30160
`dgx-spark-gateway`	31944

3. HTTPRoute — “Which backend handles this path?”

HTTPRoute resources connect the Gateway to the Backend, defining which URL path prefix routes to which LLM:

OpenAI route (goes to main gateway, path /openai):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: agentgateway-proxy
 namespace: agentgateway-system
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /openai
 backendRefs:
 - name: openai
 namespace: agentgateway-system
 group: agentgateway.dev
 kind: AgentgatewayBackend

xAI Grok route (goes to dedicated gateway, path /grok):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: xai-grok
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: xai-grok-gateway
 namespace: agentgateway-system
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /grok
 backendRefs:
 - name: xai-grok
 namespace: agentgateway-system
 group: agentgateway.dev
 kind: AgentgatewayBackend

DGX Spark route (goes to dedicated gateway, path /spark):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: dgx-spark-llm
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: dgx-spark-gateway
 namespace: agentgateway-system
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /spark
 backendRefs:
 - name: dgx-spark-llm
 namespace: agentgateway-system
 group: agentgateway.dev
 kind: AgentgatewayBackend

The complete flow for a request:

Client → http://172.16.10.149:30160/openai/v1/chat/completions
 → Gateway "agentgateway-proxy" (port 80, NodePort 30160)
 → HTTPRoute matches path "/openai"
 → AgentgatewayBackend "openai"
 → api.openai.com/v1/chat/completions

Secret Management with Vault

Cloud LLM providers require API keys. I use HashiCorp Vault with the External Secrets Operator to keep all keys out of Git:

Vault (KV v2) ESO K8s Secret Backend
agentgateway/llm-keys/openai → ExternalSecret → openai-secret → AgentgatewayBackend
agentgateway/llm-keys/xai → ExternalSecret → xai-secret → AgentgatewayBackend

The Vault stores the actual API keys. ESO syncs them into Kubernetes Secrets. The AgentgatewayBackend resources reference those Secrets by name — never the key itself. ArgoCD manages all of this declaratively, and no secret ever touches Git.

# ClusterSecretStore — connects ESO to Vault
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
 name: vault
 namespace: agentgateway-system
spec:
 provider:
 vault:
 server: "http://vault-vault.vault.svc:8200"
 path: "agentgateway/llm-keys"
 kind: SecretList
 auth:
 kubernetes:
 mountPath: "kubernetes"
 role: "eso-role"

Distributed Tracing

Every request through agentgateway emits OpenTelemetry traces to the Solo UI’s telemetry collector. This gives you a unified view of all LLM traffic — OpenAI, xAI, local Qwen — in the same dashboard:

apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayPolicy
metadata:
 name: tracing
 namespace: agentgateway-system
spec:
 targetRefs:
 - group: gateway.networking.k8s.io
 kind: Gateway
 name: agentgateway-proxy
 - group: gateway.networking.k8s.io
 kind: Gateway
 name: dgx-spark-gateway
 - group: gateway.networking.k8s.io
 kind: Gateway
 name: xai-grok-gateway
 frontend:
 tracing:
 backendRef:
 name: solo-enterprise-telemetry-collector
 namespace: agentgateway-system
 kind: Service
 port: 4317
 randomSampling: "true"

Connecting Grok Build

Now for the exciting part — pointing Grok Build at the gateway. The Grok Build config file lives at ~/.grok/config.toml. Here’s mine:

[cli]
installer = "internal"

[model.qwen]
model = "Qwen/Qwen3.6-35B-A3B-FP8"
base_url = "http://172.16.10.173:8000/v1"

[model.agw]
model = "agw"
base_url = "http://172.16.10.149:31944/spark"

[ui]
max_thoughts_width = 120
fork_secondary_model = "grok-build"
yolo = true
compact_mode = true
permission_mode = "always-approve"
theme = "groknight"

[models]
default = "agw"

The [model.agw] section defines a model called agw that points at the DGX Spark gateway URL — http://172.16.10.149:31944/spark. This is the agentgateway endpoint for my local Qwen model.

The [models] section sets agw as the default model, so every Grok Build conversation goes through agentgateway by default.

The [model.qwen] entry is a fallback direct-to-backend configuration. If I need to bypass the gateway for debugging, I can switch to the qwen model and it points directly at the vLLM server.

Here’s what happens when I type a prompt in Grok Build:

Grok Build → base_url: http://172.16.10.149:31944/spark
 → Gateway "dgx-spark-gateway" (NodePort 31944)
 → HTTPRoute matches "/spark"
 → AgentgatewayBackend "dgx-spark-llm"
 → http://172.16.10.173:8000/v1/chat/completions
 → Qwen3.6-35B-A3B-FP8 (local DGX)

The request never leaves my network. The gateway still logs it, traces it, and enforces policies — but the payload never traverses the public internet.

Connecting Other Tools

The same pattern works for any tool that supports custom API endpoints:

Claude Code — set the API base to the agentgateway URL for Anthropic:

ANTHROPIC_BASE_URL=http://172.16.10.149:30160/anthropic

Cursor — in settings, set the API endpoint to:

http://172.16.10.149:30160/openai

And set the model to gpt-4o — agentgateway strips the path prefix and forwards to the OpenAI backend.

Langflow / LlamaIndex / custom scripts — same thing:

import openai
client = openai.OpenAI(
 api_key="not-needed-proxy",
 base_url="http://172.16.10.149:30160/openai/v1"
)
response = client.chat.completions.create(
 model="gpt-4o",
 messages=[{"role": "user", "content": "Hello!"}]
)

The api_key doesn’t need to be a real OpenAI key — it’s just passed through by the proxy. The real auth happens server-side via the Vault-synced Secret.

Testing the Endpoints

You can verify each backend works through the gateway:

OpenAI (main gateway, path /openai):

curl http://172.16.10.149:30160/openai/v1/chat/completions \
 -H "content-type: application/json" \
 -d '{"model":"gpt-4o","messages":[{"role":"user","content":"Hello!"}]}' | jq

xAI Grok (dedicated gateway, path /grok):

curl http://172.16.10.149:31500/grok/v1/chat/completions \
 -H "content-type: application/json" \
 -d '{"model":"grok-4.3","messages":[{"role":"user","content":"Hello!"}]}' | jq

DGX Spark (dedicated gateway, path /spark):

curl http://172.16.10.149:31944/spark/v1/chat/completions \
 -H "content-type: application/json" \
 -d '{"model":"Qwen/Qwen3.6-35B-A3B-FP8","messages":[{"role":"user","content":"Hello!"}]}' | jq

Deploying with GitOps

All of these manifests live in a Git repository and are deployed via ArgoCD using sync waves:

Wave 1 — Gateway API CRDs
Wave 2 — agentgateway CRDs (AgentgatewayBackend, EnterpriseAgentgatewayPolicy types)
Wave 3 — Control plane (controller + proxy)
Wave 4 — HashiCorp Vault (secrets store)
Wave 5 — External Secrets Operator (Vault → K8s Secret sync)
Wave 6 — Solo UI (dashboard + telemetry collector)
Wave 7 — Config (gateways, backends, routes, policies)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: agentgateway-config
 namespace: argocd
spec:
 syncWave: 7
 # ... (points to config/ directory in Git)

When I add a new LLM backend, I create three YAML files (backend, gateway if needed, route), commit them, and ArgoCD deploys everything automatically. No imperative commands, no manual kubectl apply.

The Benefits

After months of running this setup, the benefits are clear:

One URL, all LLMs — Every tool connects to the same gateway. No scattered keys, no config files spread across machines.
Local-first by default — My Grok Build conversations go to the local Qwen model by default (free, fast, private). I switch to GPT-4o or Grok when I need the heavier models. The gateway makes the switch seamless.
Unified observability — The Solo UI shows traces from all three backends in one place. I can see latency, error rates, and token usage across providers without logging into three different dashboards.
Local models get cloud treatment — The DGX Spark runs behind the same Gateway API routing, tracing, and policy system as the cloud providers. It’s a first-class citizen, not an afterthought.
Secrets never touch Git — Vault + ESO keeps API keys out of the repository. Adding a new provider means storing a key in Vault and applying three YAML files.
Network isolation — The on-prem DGX Spark is on an internal network (172.16.10.173). agentgateway exposes it to the outside world through the Gateway API — no firewall rules needed on the DGX itself.

Conclusion

agentgateway transforms the LLM landscape from a fragmented mess of per-tool configurations into a unified, observable, centrally-managed infrastructure. Grok Build is just one client — any tool that speaks the OpenAI API contract can connect to the gateway and get access to all your LLMs.

The three-resource pattern (Backend → Gateway → Route) is simple enough to understand in minutes but powerful enough to handle complex multi-provider, multi-model, multi-cluster setups. And because it’s all Kubernetes-native — using Gateway API CRDs managed by ArgoCD — it fits into any GitOps workflow you already have.

The full configuration for this setup is open source in the k8s-goose repository, which includes the ArgoCD GitOps pipeline, all YAML manifests, and scripts for deploying on any Kubernetes cluster.

For a step-by-step walkthrough of the agentgateway setup itself, see Setting Up Enterprise agentgateway on Kind Clusters.

How To: Connect Claude Code & Codex Through agentgateway (Subscription + API Key)

Fri, 08 May 2026 00:00:00 +0000

Introduction

You’ve got Claude Code running on your laptop. You’ve got Codex open in a tab. Both are just AI coding tools, right? Wrong. Behind the scenes, they’re making raw API calls — Claude Code directly to Anthropic, Codex directly to OpenAI. No central visibility. No rate limiting. No policy enforcement. And if you’re running multiple agents, you’re scattering API keys across every machine.

Enter agentgateway.

agentgateway lets you run a local or cluster proxy that sits between your AI tools and the upstream providers. Every request goes through the gateway first, giving you a single point of control.

In this guide, I’ll walk you through two patterns for connecting Claude Code, Codex, and OpenCode to agentgateway:

Subscription passthrough — your tool uses an Anthropic/OpenAI org subscription, no API key needed
API key routing — you pass API keys through the gateway for auth

Both patterns are simple. The subscription approach is cleaner if your org already handles billing. The API key approach is more flexible for multi-account setups.

Let’s go.

Architecture

Here’s the full flow — your AI tools talk to localhost, the gateway routes and authenticates, upstream providers never see your machine directly.

Your tools → gateway (auth + routing + security layers) → providers. One proxy, full visibility.

Pattern 1: Subscription Passthrough (No API Key)

This is the simplest setup. Your AI tools connect to agentgateway, which routes them to Anthropic or OpenAI using the organization’s subscription or embedded credentials. No API keys on your machine.

The Setup

You’re running agentgateway locally on port 4001. Claude Code and Codex both connect to it. Here’s what the config looks like:

binds:
- port: 4001

listeners:
- name: default
 protocol: HTTP

routes:
- name: claude-agent
 matches:
 - path:
 pathPrefix: /claude
 policies:
 urlRewrite:
 path:
 prefix: /
 backends:
 - ai:
 name: claude-agent
 provider:
 anthropic: {}
 policies:
 ai:
 routes:
 /v1/messages: messages
 /v1/messages/count_tokens: anthropicTokenCount
 '*': passthrough

- name: codex-agent
 matches:
 - path:
 pathPrefix: /codex
 policies:
 urlRewrite:
 path:
 prefix: /
 backends:
 - ai:
 name: codex-agent
 provider:
 openAI: {}
 policies:
 ai:
 routes:
 /v1/chat/completions: completions
 /v1/responses: responses
 /v1/embeddings: embeddings
 '*': passthrough
 hostOverride: chatgpt.com:443
 pathPrefix: /backend-api/codex

A few things to call out:

Claude route (/claude) — rewrites the path to / and routes to the Anthropic provider. Maps the standard Anthropic API paths like /v1/messages.
Codex route (/codex) — this one’s slightly more complex because Codex talks to OpenAI’s chatgpt.com backend, not the regular API. We set hostOverride: chatgpt.com:443 and pathPrefix: /backend-api/codex to tell the gateway exactly where to forward.

Claude Code

Configure Claude Code to point at your local gateway:

// .claude/settings.local.json
{
 "env": {
 "ANTHROPIC_BASE_URL": "http://localhost:4001/claude"
 }
}

That’s it. Claude Code will send all requests to your gateway on port 4001, and the gateway routes them to Anthropic using your org subscription. No API key in your environment variables.

Codex

Configure Codex in its config file:

# .codex/config.toml
model_provider = "agentgateway"

[model_providers]

[model_providers.agentgateway]
name = "agentgateway"
base_url = "http://localhost:4001/codex/v1"
requires_openai_auth = true

The requires_openai_auth = true flag is important — it tells Codex to send the credentials through to the proxy, so the gateway can handle the auth header.

Pattern 2: API Key Routing

This pattern is for when you want to manage API keys centrally. You store your Anthropic and OpenAI keys in environment variables, and the gateway injects them into upstream requests.

The Setup

First, set your keys:

export ANTHROPIC_API_KEY=$(cat ~/.api-keys/anthropic.txt)
export OPENAI_API_KEY=$(cat ~/.api-keys/openai.txt)

Then run agentgateway with those keys and bind on a different port (we use 4060 here so it doesn’t collide with the first pattern):

binds:
- port: 4060

listeners:
- name: default
 protocol: HTTP

routes:
- name: claude-agent
 matches:
 - path:
 pathPrefix: /claude
 policies:
 urlRewrite:
 path:
 prefix: /
 backendAuth:
 key: $ANTHROPIC_API_KEY
 backends:
 - ai:
 name: claude-agent
 provider:
 anthropic: {}
 policies:
 ai:
 routes:
 /v1/messages: messages
 /v1/messages/count_tokens: anthropicTokenCount
 '*': passthrough

- name: codex-agent
 matches:
 - path:
 pathPrefix: /codex
 policies:
 urlRewrite:
 path:
 prefix: /
 backendAuth:
 key: $OPENAI_API_KEY
 backends:
 - ai:
 name: codex-agent
 provider:
 openAI: {}
 policies:
 ai:
 routes:
 /v1/chat/completions: completions
 /v1/responses: responses
 /v1/embeddings: embeddings
 '*': passthrough

The key difference from Pattern 1 is the backendAuth block under each route’s policies.urlRewrite. This tells the gateway to inject the API key into every upstream request. Your tools themselves never see the key — they just talk to localhost.

Claude Code

// .claude/settings.local.json
{
 "env": {
 "ANTHROPIC_BASE_URL": "http://localhost:4060/claude"
 }
}

Codex

# .codex/config.toml
model_provider = "agentgateway"

[model_providers]

[model_providers.agentgateway]
name = "agentgateway"
base_url = "http://localhost:4060/codex/v1"
env_key = "OPENAI_API_KEY"

In this pattern, Codex uses env_key instead of requires_openai_auth. It’ll pull the key from the environment variable you set.

Optional: Security Layers

The patterns above cover the core routing and auth. But agentgateway also supports a full security posture — rate limiting, JWT auth, input/output validation, and more. Here’s a practical guide to layering those on top.

Rate Limiting

Stop a runaway agent from burning through tokens:

# Rate limit Claude Code to 100 requests/hour
- name: rate-limit-claude
 matches:
 - path:
 pathPrefix: /claude
 policies:
 rateLimit:
 requestsPerHour: 100
 actions:
 - reject: true
 errorCode: 429

You can do token-based limits (e.g., max 1M tokens/day) or request-based limits. Both per-route and per-identifier (IP, user, org header).

JWT Authentication

If you’re exposing agentgateway externally (not just localhost), protect it with JWT:

- name: secure-claude
 matches:
 - path:
 pathPrefix: /claude
 policies:
 auth:
 jwt:
 issuer: https://auth.yourcompany.com
 audiences:
 - agentgateway
 actions:
 onSuccess:
 requestHeaders:
 - set:
 X-Authenticated-User: \\$(request.auth.claims.email)

This validates the JWT, rejects unauthorized requests, and forwards the user’s email as a header for audit logging.

Input Validation

Guard against prompt injection and oversized payloads:

- name: validate-input
 matches:
 - path:
 pathPrefix: /claude
 policies:
 inputValidation:
 maxRequestBodySize: 1MB
 # Block requests with known injection patterns
 blockPatterns:
 - "ignore previous instructions"
 - "bypass system prompt"

Output Filtering

Prevent sensitive data from leaking back:

- name: filter-output
 matches:
 - path:
 pathPrefix: /claude
 policies:
 outputFilter:
 # Mask emails, phone numbers, SSNs in responses
 maskPatterns:
 - email
 - phone
 - ssn

Request/Response Transformation

Beyond what we showed in the config, you can strip headers, modify bodies, add middleware headers:

policies:
 urlRewrite:
 path:
 prefix: /
 requestHeaderModifier:
 set:
 X-Gateway-Version: "1.2"
 X-Request-ID: \\$(request.id)
 responseHeaderModifier:
 add:
 X-Gateway-Processed: "true"

Putting It All Together

Here’s what a production-ready route looks like when you layer it on:

- name: claude-agent-secure
 matches:
 - path:
 pathPrefix: /claude
 policies:
 urlRewrite:
 path:
 prefix: /
 auth:
 jwt:
 issuer: https://auth.yourcompany.com
 audiences:
 - agentgateway
 rateLimit:
 requestsPerHour: 200
 actions:
 - reject: true
 errorCode: 429
 inputValidation:
 maxRequestBodySize: 2MB
 transformations:
 request:
 body: toJson(json(request.body).filterKeys(k, k != "context_management"))
 headers:
 set:
 X-Authenticated-User: \\$(request.auth.claims.email)
 X-Gateway-Source: claude-code
 backends:
 - ai:
 name: claude-agent
 provider:
 anthropic: {}
 policies:
 ai:
 routes:
 /v1/messages: messages
 /v1/messages/count_tokens: anthropicTokenCount
 '*': passthrough

One route. JWT auth, rate limiting, body transformation, audit headers, and the Anthropic backend. That’s the power of agentgateway — you configure it once and every tool that connects gets the full security treatment.

Why Bother?

You could just point Claude Code and Codex at Anthropic and OpenAI directly. And you absolutely can. But here’s what you get by routing through agentgateway:

Centralized auth — your API keys live in one place, not scattered across dev machines
Observability — every request flows through the gateway, so you get traces, metrics, and logs for free
Rate limiting — set per-agent limits so one noisy tool can’t drown out the others
Policy enforcement — block certain model calls, redirect traffic, add middleware
One proxy, multiple tools — deploy once, every AI tool on your machine connects through it

Recap

Pattern	Auth	When to use
Subscription passthrough	Org subscription	Your org handles billing, no API key management needed
API key routing	`backendAuth` with env vars	Multi-account setup, or you want keys managed at the gateway

Pick the pattern that fits your setup. Both work — the gateway handles the heavy lifting either way.

This is a living guide. If you run into issues or have a setup that works differently, drop a comment on the agentgateway GitHub repo.

GitOps for Agents: Deployment and Management

Thu, 02 Apr 2026 00:00:00 +0000

Sebastian Maniak — Solo.io

If your team has been running Kubernetes for any length of time, you’ve probably already adopted some form of GitOps — a Git repository as the source of truth, a reconciliation loop that makes your cluster match what’s declared, pull requests as the change management workflow. It’s become table stakes for production infrastructure.

So when AI agents enter the picture, the instinct is natural: treat them like any other workload, deploy them the same way. And plenty of teams are doing exactly that — containerizing their agents, writing Helm charts, pointing Argo CD at a repo, and calling it done.

That works, up to a point. But agents introduce a set of management challenges that go beyond what traditional GitOps was designed for. An agent isn’t just a container with some config. It’s a system prompt, a model selection, an effort setting, a set of tools, memory backends, and the non-deterministic reasoning of an LLM — all interacting in ways that make the standard “code + config” model feel incomplete.

The question isn’t whether to use GitOps for agents. Of course you should. The question is whether your current GitOps setup gives you the right abstractions to manage what actually makes agents different. That’s where kagent comes in — it gives Kubernetes a native understanding of what an agent is, so your existing GitOps workflows can manage the things that actually matter.

What Makes Agents Different From Other Workloads

When a microservice misbehaves after a deployment, you can usually point at a code change or a config change and say “that’s the one.” The behavior is deterministic. Same input, same output. Fix the code, redeploy, move on.

Agents have a wider blast radius.

An agent’s behavior emerges from the intersection of several things at once: its system prompt, its model, its effort setting, the tools it has access to, and the non-deterministic reasoning of the LLM itself. Change any one of those and you might get a meaningfully different agent. Swap the underlying model and the same agent with the same prompt starts making different decisions about which tools to call and in what order. Adjust the effort and a previously reliable agent starts producing inconsistent output. Grant access to one new tool and the agent starts reaching for it in situations you didn’t design for.

If you’re deploying agents as generic containers with environment variables and config maps, these changes are all happening at the same level of abstraction. A model swap looks the same as bumping a memory limit. A prompt edit is just another config change. There’s no structure that tells your GitOps tooling — or your team reviewing the PR — that one of these changes is cosmetic and the other fundamentally alters how the agent behaves.

That’s the gap. Not “do you have GitOps?” — you probably do. It’s “does your GitOps pipeline understand the anatomy of an agent well enough to give you meaningful control?”

What Agent-Aware GitOps Actually Looks Like

The GitOps principle doesn’t change: desired state in Git, reconciliation loop making reality match. What changes is what counts as “desired state.” When agent-specific concerns become first-class objects in your pipeline — not opaque blobs inside a container — you get three things you didn’t have before:

Prompts you can actually review. Right now, a prompt probably lives somewhere inside your application code, tangled up with everything else. Pull it out into its own declarative resource and suddenly a prompt edit is a focused PR — one diff, one review, one approval. The behavioral change stands on its own instead of hiding inside a code commit.

Model upgrades you can roll back in isolation. When the model config is its own resource, swapping from one model to another is a single, isolated change. Promote it through dev → staging → production on its own timeline. If quality drops, revert that one resource. Nothing else moves.

Tool access you can govern like a permission. Each agent declares exactly which tools it can use — not “everything this server offers,” but a specific, named list. Adding a tool is a one-line diff. Removing one is equally clear. The PR is the approval process for what an agent is allowed to do.

How kagent Makes This Work

kagent is a Kubernetes-native framework that models agents as custom resources — the same declarative, reconcilable building blocks that platform teams already use for everything else. This is the design choice that makes GitOps for agents practical rather than theoretical.

The architecture splits an agent’s definition into independent, composable resources:

The Agent is the top-level resource. It defines the system prompt, references a model configuration, declares which tools are available (and critically, which specific tools from each tool server — not blanket access), sets resource limits, and advertises skills to other agents via the A2A protocol.

The Model Configuration is a separate resource that defines the LLM provider, model version, and parameters like temperature and token limits. Because it’s independent, you can swap models without touching agent definitions. You can share one model config across twenty agents. You can run a canary by creating a new model config alongside the existing one and pointing one agent at it.

Tool Servers define the MCP servers that provide capabilities to agents. This is where kagent’s skill model lives. Each tool server exposes a set of tools, and agents cherry-pick the specific ones they need. You don’t give an agent access to “all Kubernetes tools” — you give it access to the five read-only tools it actually needs and leave the destructive ones off the list. That scoping decision is declared in the agent resource and reviewed in the PR.

Skills deserve a closer look. In kagent’s model, an agent’s capabilities come from tool servers — MCP servers that expose tools the agent can call. kmcp is the toolkit for building, testing, and deploying these as production services. kagent ships with built-in tool servers for common operational needs — Kubernetes, Helm, Argo, Istio, Prometheus, Grafana, Cilium — but the real power is in custom skills: your company’s internal APIs, your domain-specific workflows, your proprietary data sources, all packaged, versioned, and deployed the same way.

When an agent declares skills in its A2A configuration, it advertises capabilities that other agents can discover. A supervisor agent doesn’t need to know implementation details — it just needs to know the skill exists and how to invoke the agent that has it. Multi-agent orchestration becomes a graph of declared relationships, all visible in the same configuration files.

The GitOps Workflow for Agents

Here’s how it all comes together in practice. The flow follows the same base-and-overlay pattern that platform teams already use for microservices, extended to cover the full agent stack:

The base definitions contain the canonical agent configurations — agent resources, model configs, tool servers, and skills. Overlays customize per environment: dev might use a local model to keep costs down, staging might use a mid-tier cloud model for integration testing, and production runs the most capable model with proper scaling and multiple replicas.

What makes this powerful for agents specifically is how different types of changes flow through the system:

Prompt Changes Become Code Reviews

When your system prompt lives in a declarative resource in Git, prompt engineering becomes a reviewable activity. Someone rewrites a paragraph of the system prompt — that’s a PR with a clear diff. The team can read it, discuss it, push back on it, and approve it. After it’s merged, you can trace any line of any system prompt back to the PR that introduced it, the discussion that shaped it, and the person who signed off on it.

This matters more than it sounds. Prompt changes are the most impactful behavioral changes you can make to an agent. A subtle rewording can completely alter how it uses tools or handles edge cases. Without version control, those changes are invisible. With GitOps, they’re first-class, auditable events.

Model Upgrades Become Staged Rollouts

New model drops. Instead of flipping a switch across your entire agent fleet, you update the model configuration in your dev overlay. Argo CD syncs it. You run evals. If they pass, you promote the change to staging and run integration tests against real traffic patterns. If staging holds, you promote to production and monitor quality metrics.

If quality drops at any stage, you revert the model config change — one commit, full rollback, the old model is back in minutes.

You can even run canaries. Create a new model configuration alongside the existing one, point a single agent at it, and compare quality side-by-side before migrating the rest of the fleet.

Tool Access Changes Become Security Reviews

Adding a tool to an agent’s allowed list is granting a permission. When an agent gains access to a destructive capability — deleting resources, writing to a database, calling an external API — that’s a security-relevant change. GitOps makes it visible by default: the change shows up as a one-line diff in a pull request.

You can put CODEOWNERS rules on tool configuration so changes require security team review. You can write CI checks that enforce policy — no agent in a customer-facing namespace gets tools that modify infrastructure. The pull request becomes your approval process.

Skill Development Gets a Real Lifecycle

Custom skills — the MCP servers that give agents domain-specific capabilities — live in the same repository and go through the same workflow. A developer builds a new skill with kmcp, tests it locally, commits the source alongside the tool server configuration that deploys it, and opens a PR. The skill goes through review, gets deployed to dev, gets tested, and promotes through environments just like everything else.

Because tool servers are referenced by name, you can roll out a new version of a skill independently of the agents that use it. Or you can pin specific agents to specific versions if stability matters more than getting the latest capabilities.

The Reconciliation Loop: Why Self-Healing Matters for Agents

Any GitOps controller works with kagent — Flux, Argo CD, even a simple apply step in CI. But the reconciliation loop is where the real value lives, and it matters more for agents than for traditional workloads.

Here’s the scenario. It’s 2 AM. An incident is happening. Someone kubectl edits an agent’s system prompt to change its behavior during the incident. Totally reasonable in the moment. The incident resolves, everyone goes to sleep. Three weeks later, the agent is behaving oddly and nobody can figure out why. The team spends half a day investigating before someone thinks to diff the running config against Git and finds the untracked prompt change.

With GitOps and self-healing enabled, this doesn’t happen. The reconciliation loop detects the drift within minutes and reverts the agent to match what’s declared in Git. If you actually need to change the agent’s behavior, you do it through a PR — which means the change is tracked, reviewed, and reversible.

What Changes at Scale

The benefits compound as the number of agents grows. With a handful of agents, the overhead of agent-aware GitOps is marginal. At scale — dozens of agents across multiple teams and environments — it starts paying for itself in ways that are hard to get otherwise.

Drift detection and self-healing mean no agent runs a configuration that wasn’t committed and reviewed. Rollback on any dimension — prompt, model, tools — is a git revert away. Environment promotion is a pull request between overlays. The audit trail is just Git history, filterable to any agent’s files, with timestamps, authors, and the PR discussions attached.

Multi-tenancy maps naturally to Kubernetes namespaces. Different teams own different agents, each synced from their own path in the repository. Model configs and tool servers can be shared across namespaces using cross-namespace references, so the platform team provides shared infrastructure while application teams own their agent definitions.

And when you pair kagent with agentgateway, the traffic layer — routing, rate limiting, model failover, cost budgets — lives in the same repository and syncs through the same pipeline. One repo describing the full agent stack: what agents do, what skills they have, how their traffic flows, and how it all gets secured.

Where This Is Heading

The teams that are already deploying agents with GitOps are in a good position. The next step is making sure the abstractions match the workload. Treating an agent as a generic container with env vars works until you need to roll back a model change independently of a prompt change, or review a tool access grant as a security decision, or promote a new skill through environments without touching the agents that use it.

kagent gives Kubernetes a native understanding of these concerns. It fits into the GitOps infrastructure you’ve already built — same Argo CD, same Kustomize overlays, same PR workflows — and adds the agent-specific structure that turns generic deployment into meaningful lifecycle management.

The agents you’re running today are probably manageable. The question is whether your setup scales to the agents you’ll be running a year from now, when there are more of them, more teams building them, and more at stake when something goes wrong. That’s the problem worth solving now.

kagent is a CNCF Sandbox project. kagent.dev · github.com/kagent-dev/kagent

agentgateway is an LF / AAIF project. agentgateway.dev

How I Built a Multi-Topic AI Workspace with OpenClaw and Telegram

Sun, 29 Mar 2026 00:00:00 +0000

The Problem with One Big Chat

When I first set up OpenClaw, I did what most people do — one Telegram chat, one conversation, everything in one place. It worked… for about a week.

Then things got messy. I’d ask about a Kubernetes issue and the AI would reference a recipe I’d asked about earlier. I’d try to draft a blog post and it would pull in context from a client billing discussion. The context window — the amount of conversation the AI can hold at once — was polluted with everything from smart home commands to security audits.

The real problem isn’t that the AI is bad at context. It’s that a single conversation thread forces every topic to compete for the same limited context window. When that window fills up, older messages get compressed into summaries. Summaries lose details. Your agent slowly forgets who you are, what you’re working on, and what you told it yesterday.

I lost an entire project’s context once because of this. Spent 2 hours re-explaining preferences and decisions my agent already knew. Never again.

The Solution: Telegram Forum Topics

Telegram has a feature called Topics (or Forum mode) that turns a group into a structured space where each topic is its own isolated thread. OpenClaw supports this natively — you can configure each topic with:

Its own system prompt (personality and instructions)
Its own model (Opus for deep work, Sonnet for quick tasks)
Its own skills (only the tools that topic needs)
Its own storage (Notion page + Google Drive folder)
Its own memory path (separate daily logs per domain)
Its own session (isolated context window — zero cross-contamination)

Think of it as giving your AI a different desk for each type of work. When you walk up to the Content desk, it’s already thinking about content. When you walk up to the Infra desk, it’s in ops mode. They don’t leak into each other.

My Setup: 9 Topics

Here’s how I have mine structured:

Topic	Model	Skills	Purpose
📋 Mission Control	Sonnet (default)	—	Tasks, priorities, reminders, deadlines
🏠 Home	Sonnet	sonoscli, camsnap, weather	Smart home: Sonos, cameras, devices
⚙️ Infra & K8s	Sonnet (default)	github, gh-issues, coding-agent, healthcheck	Kubernetes clusters, logs, incident response
✍️ Content	Opus	notion, summarize, xurl, blogwatcher	Scripts, social posts, webinar prep
🔍 Research	Opus	notion, summarize, blogwatcher, github	Deep dives on people, products, companies
💪 Health & Fitness	Sonnet	weather	WHOOP data, workouts, health tracking
🤖 AI & Agents	Sonnet (default)	—	OpenClaw config, skill development, experiments
⚡ Overflow	Sonnet	—	Quick asks that don’t fit elsewhere
💼 Corp	Sonnet (default)	—	Client relationships, billing, time tracking

Step-by-Step Setup

1. Create the Telegram Forum Group

Open Telegram → New Group → name it (I called mine “OpenClaw”)
Add your OpenClaw bot to the group
Go to Group Settings → Edit → Enable Topics (Forum mode)
Create each topic with an emoji prefix for quick scanning

Each topic gets a numeric ID from Telegram. You’ll need these for the config.

2. Get Your IDs

You need three things:

Group chat ID: the negative number for your group (check Gateway logs after sending a message)
Topic IDs: each topic’s thread number (also in logs)
Your Telegram user ID: use @userinfobot in Telegram

3. Define Your Agents

Before configuring topics, set up agents in openclaw.json. Each agent maps to a model:

{
 agents: {
 defaults: {
 model: {
 primary: "anthropic/claude-sonnet-4-6",
 fallbacks: ["anthropic/claude-opus-4-6"]
 }
 },
 list: [
 { id: "main" },
 {
 id: "opus",
 name: "opus",
 workspace: "~/.openclaw/workspace",
 agentDir: "~/.openclaw/agents/opus/agent",
 model: "anthropic/claude-opus-4-6"
 },
 {
 id: "sonnet",
 name: "sonnet",
 workspace: "~/.openclaw/workspace",
 agentDir: "~/.openclaw/agents/sonnet/agent",
 model: "anthropic/claude-sonnet-4-6"
 }
 ]
 }
}

Key decisions:

All agents share the same workspace — so SOUL.md, AGENTS.md, and MEMORY.md are consistent
Each agent gets its own agentDir — for separate auth profiles and sessions
Sonnet is the default — fast and cheap for 80% of tasks
Opus is reserved — only for topics that need deep reasoning or creativity

4. Configure the Telegram Group

Here’s the structure inside openclaw.json:

{
 telegram: {
 enabled: true,
 botToken: "YOUR_BOT_TOKEN",
 dmPolicy: "pairing",
 groupPolicy: "allowlist",

 groups: {
 // Default for ALL groups: require @mention
 "*": {
 requireMention: true
 },

 // Your specific forum group
 "-1003885751519": {
 requireMention: false,
 allowFrom: ["775644809"], // Only YOUR Telegram user ID

 topics: {
 // Each topic goes here...
 }
 }
 }
 }
}

5. Configure Each Topic

Every topic follows the same pattern:

"TOPIC_ID": {
 requireMention: false,
 agentId: "opus", // or "sonnet", or omit for default
 skills: ["skill-a", "skill-b"],
 systemPrompt: "MISSION + STYLE + STORAGE + MEMORY"
}

Here’s a real example — my Content topic:

"48": {
 requireMention: false,
 agentId: "opus",
 skills: ["notion", "summarize", "xurl", "blogwatcher"],
 systemPrompt: "You help draft scripts, social posts, and webinar prep. Match the user's voice and tone. Ask clarifying questions before producing long-form content.\n\nStorage:\n- Notion page: 32d4d457-9015-8138-ba2d-e40df5d4b20e (Content)\n- Google Drive folder: 1__Z8KpROUZucvWbH-4cRzw8tA1BhWFDM (Content)\nUse Notion for drafts, content calendar, and outlines. Use Drive for final assets, slides, and media.\n\nDaily memory: Write dated notes to `memory/content/YYYY-MM-DD.md` for this topic."
}

The System Prompt Pattern

Every topic system prompt follows four parts:

MISSION — What this topic does (1-2 sentences)

“You manage tasks, priorities, and reminders.”
STYLE — How to respond

“Be proactive about deadlines. Keep answers practical.”
STORAGE — Where to save work

“Notion page: <ID> (Mission Control) / Drive folder: <ID>”
MEMORY — Where to write daily logs

“Write dated notes to memory/mission-control/YYYY-MM-DD.md”

This ensures every topic knows its job, its voice, where to put things, and where to journal. When you open a topic, the AI is already in the right headspace.

Model Routing: The Strategy

Not every task needs the most expensive model. Here’s my routing logic:

Use Opus (frontier model) when:

Creative writing — blog posts, scripts, social content
Deep research — connecting non-obvious dots across sources
Complex reasoning — architecture decisions, strategy

Use Sonnet (fast/cheap) for everything else:

Task management — creating, updating, tracking tasks
Smart home — “play jazz in the kitchen” doesn’t need genius
Health data — querying WHOOP stats is structured, not creative
Quick asks — definitions, lookups, simple questions
Ops work — kubectl commands, log parsing, incident response

The result: ~80% of my interactions use Sonnet (faster, cheaper), and the 20% that use Opus actually benefit from it.

Skills Routing: Keep It Lean

Each topic only loads the skills it needs. This matters because:

Every skill adds instructions to the system prompt
More instructions = more tokens consumed per turn
More tokens = faster context window fill-up = more compaction = more forgetting

My mapping:

Topic	Skills	Why
🏠 Home	sonoscli, camsnap, weather	Smart device control
⚙️ Infra	github, gh-issues, coding-agent, healthcheck	DevOps workflow
✍️ Content	notion, summarize, xurl, blogwatcher	Content pipeline
🔍 Research	notion, summarize, blogwatcher, github	Research tools
💪 Health	weather	Outdoor planning
Everything else	(none)	Keep it lean

Rule of thumb: if a skill isn’t used weekly in a topic, remove it.

Memory: Separate Paths Per Domain

This is the key insight that makes the whole architecture work. Each topic writes its daily logs to a separate memory subdirectory:

memory/
├── mission-control/
│ └── 2026-03-29.md
├── home/
│ └── 2026-03-29.md
├── infra/
│ └── 2026-03-29.md
├── content/
│ └── 2026-03-29.md
├── research/
│ └── 2026-03-29.md
├── health/
│ └── 2026-03-29.md
├── ai/
│ └── 2026-03-29.md
├── corp/
│ └── 2026-03-29.md
└── general/
 └── 2026-03-29.md

Why this matters:

No cross-contamination — infra notes don’t pollute your content drafts
Vector search stays relevant — when searching memory from the Content topic, you find content notes, not Kubernetes debugging logs
Compaction is less painful — each topic’s session is smaller and more focused, so compaction happens less often and loses less context

Combined with MEMORY.md for permanent long-term storage (key facts, project overviews, infrastructure details), this creates a layered memory system:

MEMORY.md — permanent, cross-topic, loaded every session
memory//YYYY-MM-DD.md — daily, topic-specific, searched on demand

Security: Three Layers

The config implements three security layers:

{
 groupPolicy: "allowlist", // Layer 1: only approved groups
 groups: {
 "*": { requireMention: true }, // Layer 2: strangers must @mention
 "-1003885751519": {
 requireMention: false,
 allowFrom: ["775644809"] // Layer 3: only YOUR user ID
 }
 },
 dmPolicy: "pairing" // DMs require pairing code
}

Even if someone joins your Telegram group:

They can’t trigger the bot in other groups (groupPolicy: "allowlist")
They’d need to @mention the bot in random groups (requireMention: true on "*")
They can’t trigger actions in your forum (allowFrom restricts to your user ID)

Storage: Per-Topic Notion + Drive

Each topic has its own Notion page and Google Drive folder. This means:

Content topic → drafts go to the Content Notion page, final slides go to the Content Drive folder
Infra topic → runbooks go to the Infra Notion page, log dumps go to the Infra Drive folder
Corp topic → client notes go to the Corp Notion page, contracts go to the Corp Drive folder

No hunting. No “which folder was that in?” Everything is organized by domain from the moment it’s created.

Tips From 3+ Months of Use

Start with 3-4 topics — add more when you find natural boundaries
Overflow is essential — random questions need somewhere to go without polluting focused topics
Emoji prefixes matter — when you have 9 topics, visual scanning speed matters
Review monthly — merge underused topics, split overloaded ones
Pin important messages in each topic for quick reference
Keep system prompts under 500 words — they’re injected every turn
Memory subdirectories are non-negotiable — this is what prevents context bleed
Use a Meta topic (my “AI & Agents”) for discussing the setup itself
Back up openclaw.json — it’s the single source of truth for this entire architecture
The cheapest model that works is the right model — don’t waste Opus on “what’s the weather”

The Complete Config

Here’s the full relevant section of my openclaw.json for reference:

{
 telegram: {
 enabled: true,
 botToken: "YOUR_BOT_TOKEN",
 dmPolicy: "pairing",
 groupPolicy: "allowlist",
 streaming: "off",

 groups: {
 "*": { requireMention: true },

 "-YOUR_GROUP_ID": {
 requireMention: false,
 allowFrom: ["YOUR_USER_ID"],

 topics: {
 "45": {
 requireMention: false,
 systemPrompt: "You manage tasks, priorities, and reminders. Be proactive about deadlines and follow-ups. Track what's pending and surface what needs attention.\n\nStorage:\n- Notion page: <ID> (Mission Control)\n- Google Drive folder: <ID> (Mission Control)\nUse Notion for tasks, SOPs, and structured notes. Use Drive for file attachments and exports.\n\nDaily memory: Write dated notes to `memory/mission-control/YYYY-MM-DD.md` for this topic."
 },
 "46": {
 requireMention: false,
 agentId: "sonnet",
 skills: ["sonoscli", "camsnap", "weather"],
 systemPrompt: "You assist with home automation: Sonos, cameras, and smart devices. Keep answers practical and action-oriented.\n\nStorage:\n- Notion page: <ID> (Home)\n- Google Drive folder: <ID> (Home)\n\nDaily memory: Write dated notes to `memory/home/YYYY-MM-DD.md` for this topic."
 },
 "47": {
 requireMention: false,
 skills: ["github", "gh-issues", "coding-agent", "healthcheck"],
 systemPrompt: "You assist with Kubernetes clusters, logs, and incident response. Be concise and ops-focused. Prefer actionable commands over lengthy explanations.\n\nStorage:\n- Notion page: <ID> (Infra & K8s)\n- Google Drive folder: <ID> (Infra & K8s)\n\nDaily memory: Write dated notes to `memory/infra/YYYY-MM-DD.md` for this topic."
 },
 "48": {
 requireMention: false,
 agentId: "opus",
 skills: ["notion", "summarize", "xurl", "blogwatcher"],
 systemPrompt: "You help draft scripts, social posts, and webinar prep. Match the user's voice and tone. Ask clarifying questions before producing long-form content.\n\nStorage:\n- Notion page: <ID> (Content)\n- Google Drive folder: <ID> (Content)\n\nDaily memory: Write dated notes to `memory/content/YYYY-MM-DD.md` for this topic."
 },
 "49": {
 requireMention: false,
 agentId: "opus",
 skills: ["notion", "summarize", "blogwatcher", "github"],
 systemPrompt: "You research people, products, companies, and ideas. Cite sources, be thorough, and surface non-obvious connections.\n\nStorage:\n- Notion page: <ID> (Research)\n- Google Drive folder: <ID> (Research)\n\nDaily memory: Write dated notes to `memory/research/YYYY-MM-DD.md` for this topic."
 },
 "50": {
 requireMention: false,
 agentId: "sonnet",
 skills: ["weather"],
 systemPrompt: "You help with WHOOP data, workout planning, and health dashboard work. Be data-driven and concise.\n\nStorage:\n- Notion page: <ID> (Health & Fitness)\n- Google Drive folder: <ID> (Health & Fitness)\n\nDaily memory: Write dated notes to `memory/health/YYYY-MM-DD.md` for this topic."
 },
 "51": {
 requireMention: false,
 systemPrompt: "You help with OpenClaw configuration, skill development, and agent experiments. You can be technical and detailed here.\n\nStorage:\n- Notion page: <ID> (AI & Agents)\n- Google Drive folder: <ID> (AI & Agents)\n\nDaily memory: Write dated notes to `memory/ai/YYYY-MM-DD.md` for this topic."
 },
 "66": {
 requireMention: false,
 agentId: "sonnet",
 systemPrompt: "You handle overflow and quick asks. Keep responses short and direct.\n\nDaily memory: Write dated notes to `memory/general/YYYY-MM-DD.md` for this topic."
 },
 "387": {
 requireMention: false,
 systemPrompt: "You manage customer relationships, project billing, and time tracking. Focus on: active clients, logged hours, invoices, and follow-ups. Be direct — flag unbilled time, overdue invoices, and upcoming deadlines.\n\nStorage:\n- Notion page: <ID> (Corp)\n- Google Drive folder: <ID> (Corp)\n\nDaily memory: Write dated notes to `memory/corp/YYYY-MM-DD.md` for this topic."
 }
 }
 }
 }
 },

 agents: {
 defaults: {
 model: {
 primary: "anthropic/claude-sonnet-4-6",
 fallbacks: ["anthropic/claude-opus-4-6"]
 }
 },
 list: [
 { id: "main" },
 {
 id: "opus",
 workspace: "~/.openclaw/workspace",
 agentDir: "~/.openclaw/agents/opus/agent",
 model: "anthropic/claude-opus-4-6"
 },
 {
 id: "sonnet",
 workspace: "~/.openclaw/workspace",
 agentDir: "~/.openclaw/agents/sonnet/agent",
 model: "anthropic/claude-sonnet-4-6"
 }
 ]
 }
}

What’s Next

This setup has transformed how I interact with AI. Instead of one overwhelmed assistant trying to be everything, I have a team of specialists — each focused, each equipped, each with its own memory. The context window problem is essentially solved because no single topic accumulates enough history to trigger aggressive compaction.

If you’re running OpenClaw with a single chat thread and wondering why it “forgets things” — this is probably your answer. Thread it. Topic it. Give each domain its own space.

The AI doesn’t forget when you organize its memory.

Running OpenClaw? Join the community at discord.com/invite/clawd. Find skills at clawhub.ai.

Building a Slack Bot That Updates Your Docs: Kagent, MCP, and GitHub in Action

Fri, 20 Mar 2026 00:00:00 +0000

The Problem Nobody Talks About

Documentation rots. Everyone knows it, nobody wants to fix it. The friction is real — switching context from a Slack conversation about a bug to opening a browser, navigating to a repo, editing markdown, committing, pushing, and opening a PR. By the time you’ve done all that, the conversation has moved on and the motivation is gone.

What if updating docs was as simple as telling a bot in Slack, “Hey, update the getting started guide to mention the new evaluator type”?

That’s exactly what this project does. A Slack bot, running inside Kubernetes, powered by an AI agent framework called Kagent, connected to GitHub through the Model Context Protocol (MCP). You talk to it in Slack. It reads the docs, makes changes, opens PRs — and asks for your approval before doing anything destructive.

No more context switching. No more stale docs.

Why This Exists

Three pain points drove this build:

1. Teams live in Slack. That’s where decisions happen, where context lives, where people already are. Forcing someone out of Slack to manage docs is a guaranteed way to ensure docs never get updated.

2. LLMs are good at writing, bad at acting. ChatGPT can draft a paragraph, sure. But it can’t read your existing docs, understand the structure, create a branch, commit the change, and open a PR. You need tools — real, connected tools — not just text generation.

3. Trust requires guardrails. Nobody wants an AI silently pushing code to main. The agent needs to ask permission before it mutates anything. Human-in-the-loop isn’t a nice-to-have, it’s a requirement.

The Architecture

Here’s the high-level flow:

Slack message
 ↓
Slack Bot (Socket Mode, no ingress required)
 ↓
Kagent Controller (A2A protocol)
 ↓
Agent with MCP tools
 ├── slack-mcp → Post status updates back to Slack
 └── github-mcp → Read files, create branches, commit, open PRs
 ↓
GitHub (via Copilot hosted MCP endpoint)

The entire stack runs on Kubernetes. No external servers, no Lambda functions, no third-party SaaS platforms mediating the conversation. Everything is declared as Kubernetes resources, managed through GitOps, and secured with proper secrets management.

What is Kagent?

Kagent is a Kubernetes-native AI agent framework. Think of it as a controller that manages AI agents the same way Kubernetes manages pods — through declarative YAML manifests.

Instead of writing a Python script that calls OpenAI and bolts on some tools, you declare an Agent custom resource:

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
 name: docs-agent
spec:
 type: Declarative
 declarative:
 modelConfig: default-model-config
 tools:
 - type: McpServer
 mcpServer:
 kind: RemoteMCPServer
 name: github-mcp
 - type: McpServer
 mcpServer:
 kind: MCPServer
 name: slack-mcp

That’s it. Kagent handles:

LLM routing — Which model to call, with what credentials
Tool orchestration — MCP servers are mounted as tools the agent can invoke
Memory — Embedding-based long-term context so the agent remembers past conversations
Context compaction — When conversations get long, Kagent compresses earlier messages to stay within token limits
A2A protocol — Agent-to-Agent communication, which is how the Slack bot talks to the Kagent controller

The agent isn’t a monolith. It’s a composition of capabilities declared in YAML and reconciled by a Kubernetes controller. Add a new tool? Add a line to the manifest. Change the model? Update a config reference. Everything follows the Kubernetes pattern — desired state in git, actual state in the cluster, a controller reconciling the difference.

How GitHub MCP Server Fits In

The Model Context Protocol (MCP) is a standard for giving AI agents access to tools. Instead of writing custom integrations for every service, you expose tools through an MCP server, and any MCP-compatible agent can use them.

For GitHub, this means the agent gets access to operations like:

get_file_contents — Read any file in a repository
create_or_update_file — Edit files and commit changes
create_branch — Work on feature branches, not main
create_pull_request — Open PRs for review
search_code — Find relevant files across the repo
list_issues, create_issue — Manage issues alongside docs

The GitHub MCP server is declared as a RemoteMCPServer resource in Kubernetes:

apiVersion: kagent.dev/v1alpha2
kind: RemoteMCPServer
metadata:
 name: github-mcp
spec:
 transport:
 type: STREAMABLE_HTTP
 streamableHTTP:
 url: https://api.githubcopilot.com/mcp/
 timeout: 5s
 headersFrom:
 - name: Authorization
 valueFrom:
 type: Secret
 name: github-pat

A few things worth noting:

No self-hosted binary. This uses GitHub Copilot’s hosted MCP endpoint. No containers to build, no sidecars to manage.
Auth through Kubernetes secrets. The GitHub PAT is stored in HashiCorp Vault, synced to a Kubernetes secret via External Secrets Operator, and injected as a Bearer token header. The agent never sees the raw token.
Streamable HTTP transport. The MCP server uses HTTP with streaming support, so long-running operations don’t time out.

The Slack Bot: Bridging Chat and Code

The Slack bot is the user-facing piece. It runs in Socket Mode — meaning it opens a websocket connection to Slack’s API rather than requiring an inbound webhook URL. This is a deliberate choice: no ingress controller needed, no public endpoints exposed, no certificates to manage.

When a message comes in, the bot forwards it to the Kagent controller using the A2A (Agent-to-Agent) protocol over HTTP:

async with httpx.AsyncClient() as client:
 response = await client.post(
 f"{KAGENT_BASE_URL}/api/a2a/kagent/{AGENT_NAME}/",
 json=payload
 )

The agent processes the request — which might involve reading docs from GitHub, drafting changes, and preparing a PR — then returns a response. But here’s the critical part: mutating operations require human approval.

When the agent wants to run create_or_update_file, push_files, create_pull_request, or merge_pull_request, the Slack bot renders an approval prompt:

Tool: create_pull_request Repository: my-org/website Title: “Update getting started guide with new evaluator type” Branch: docs/update-evaluators

[ Approve ] [ Deny ]

The user clicks a button in Slack. Only then does the agent execute the action. This isn’t just a safety measure — it’s a trust-building mechanism. People adopt tools they can control.

Secrets Management: The Boring Part That Matters

Every credential in this system flows through a single pipeline:

HashiCorp Vault → External Secrets Operator → Kubernetes Secret → Application

Vault stores the truth:

LLM API keys at secret/kagent/llm
Slack bot tokens at secret/slack
GitHub PAT at secret/github

External Secrets Operator polls Vault every hour and syncs secrets into Kubernetes. Applications reference Kubernetes secrets in their manifests. Nobody hardcodes a token. Nobody copy-pastes a key into a YAML file.

The bootstrap script initializes everything:

# Initialize Vault, configure K8s auth, create policies
./bootstrap/vault-init.sh

It prompts for each secret interactively, stores them in Vault, and configures the Kubernetes auth method so ESO can retrieve them. One script, one time, all secrets in the right place.

Deployment: GitOps All the Way Down

The entire stack deploys through Argo CD using an App of Apps pattern with sync waves:

Wave	What Deploys	Why This Order
0	Vault, External Secrets Operator	Secrets infrastructure first
1-3	Istio (ambient mesh)	Zero-trust networking
4	Kagent CRDs	Agent framework schema
5	Kagent controller	Agent runtime
6	Vault config (SecretStores, ExternalSecrets)	Wire secrets to apps
7	Slack bot, GitHub agent	Application layer

One git push to main and Argo CD reconciles everything. Auto-prune removes deleted resources. Auto-heal fixes drift. If someone kubectl deletes a resource, Argo puts it back.

The bootstrap is two commands:

./bootstrap/install.sh # Argo CD + Gateway API CRDs
./bootstrap/vault-init.sh # Secrets initialization

After that, git is the only interface.

What a Real Interaction Looks Like

Here’s a realistic conversation in Slack:

You: The docs for the scoring evaluator are missing the threshold parameter. Can you add it to the reference page?

Agent: I’ll check the current docs. Let me read the evaluator reference page.

(Agent calls get_file_contents on the docs repo)

Agent: Found it. The scoring evaluator section at content/docs/evaluators/scoring.md doesn’t mention the threshold parameter. I’ll create a branch and update the docs.

(Agent calls create_branch → create_or_update_file)

⚠️ Approval Required Tool: create_or_update_file File: content/docs/evaluators/scoring.md Branch: docs/add-threshold-param

[ Approve ] [ Deny ]

You: clicks Approve

Agent: Done. I’ve also opened a PR for review.

⚠️ Approval Required Tool: create_pull_request Title: “Add threshold parameter to scoring evaluator docs”

[ Approve ] [ Deny ]

You: clicks Approve

Agent: PR #42 is open. Here’s the link: [PR #42]

Total time: under a minute. No browser tabs opened. No git commands typed. No context lost.

Lessons Learned

Socket Mode is underrated. Not needing ingress for a Slack bot simplifies everything. No TLS certs, no DNS records, no service exposure. The bot dials out; Slack sends messages over the websocket. For internal tools, this is the way.

MCP is the right abstraction for agent tools. Before MCP, every AI agent had bespoke tool integrations — custom functions wrapping API calls. MCP standardizes this. Swap GitHub for GitLab? Replace one MCP server, the agent doesn’t change. The protocol decouples the agent from the tools.

Human-in-the-loop needs good UX. A wall of JSON asking “approve this?” doesn’t cut it. The approval prompt needs to clearly show what action is being taken, on what resource, with what parameters. Slack’s interactive blocks make this possible — buttons, structured messages, and threaded conversations.

GitOps + AI agents work well together. The agent configurations are YAML in git. The secrets pipeline is declarative. The deployment order is encoded in sync waves. When something breaks, git log tells you what changed. When you want to add a new agent, you add manifests and push. The operational model is the same as any other Kubernetes workload.

Start with read-only, add writes carefully. The first version of this agent could only read docs and post to Slack. Writes came later, gated behind approvals. This incremental approach builds confidence — both in the system and in the team using it.

Try It Yourself

The building blocks are all open source:

Kagent — Kubernetes-native AI agent framework
GitHub MCP Server — GitHub’s hosted MCP endpoint (requires a GitHub PAT)
Slack Bolt — Python framework for Slack apps with Socket Mode support
External Secrets Operator — Sync secrets from Vault (or other providers) to Kubernetes

The pattern generalizes beyond documentation. Any workflow that involves reading from a system, deciding on an action, getting human approval, and executing — that’s an agent use case. Incident response, infrastructure changes, onboarding checklists. The Slack bot is the interface. Kagent is the brain. MCP servers are the hands.

The gap between “we should update the docs” and “the docs are updated” just got a lot smaller.

Managing FortiGate Firewalls from Telegram with AI, MCP, and kagent

Sun, 15 Mar 2026 10:00:00 -0400

Managing FortiGate Firewalls from Telegram with AI, MCP, and kagent

By Sebastian Maniak

What if you could query your FortiGate firewall from Telegram? Not by logging into the web UI or running CLI commands over SSH — but by asking an AI agent “show me all firewall policies” or “what VIPs are configured?” and getting a clean, summarized answer in seconds.

That’s what I built. After wrapping my F5 BIG-IP as an MCP server and managing it from Telegram, I wanted the same experience for my Fortinet firewall. In this article, I’ll walk through how I deployed the community FortiGate MCP server on Kubernetes, wired it up as a kagent AI agent, and connected it to a Telegram bot — giving me full conversational access to firewall policies, NAT rules, VIPs, address objects, routing tables, and system status from my phone.

Everything runs on my home lab Kubernetes cluster (Talos Linux on Proxmox), deployed via GitOps with ArgoCD, with secrets managed by HashiCorp Vault.

Why Wrap a Firewall as an MCP Server?

FortiGate has a comprehensive REST API (/api/v2/cmdb/... and /api/v2/monitor/...), but it returns deeply nested JSON that’s hard to parse at a glance. When you’re on your phone and want a quick answer — “which policies allow traffic from the DMZ?” or “what’s my NAT setup?” — you don’t want to stare at raw JSON.

By wrapping FortiGate as an MCP server and letting an AI agent interpret the results, you get:

Natural language queries — ask questions in plain English, get summarized answers
Automatic tool discovery — kagent discovers all available FortiGate operations via MCP, no manual wiring
HITL safety — destructive operations (creating policies, modifying routes) require your explicit approval via Telegram buttons
Conversational context — follow-up questions work naturally (“show me policy 5” after listing all policies)

Architecture

┌──────────────┐
│ Telegram │
│ (user) │
└──────┬───────┘
 │ polling
 ▼
┌──────────────┐ A2A protocol ┌──────────────────┐
│ Telegram Bot │ ──────────────────────▶│ kagent Controller │
│ (K8s pod) │ │ │
└──────────────┘ └────────┬─────────┘
 │ routes to agent
 ▼
 ┌──────────────────┐
 │ fortigate-agent │
 │ (LLM decides │
 │ which tool) │
 └────────┬─────────┘
 │ MCP streamable-HTTP
 ▼
 ┌──────────────────┐
 │ FortiGate MCP │
 │ Server │
 └────────┬─────────┘
 │ FortiGate REST API
 ▼
 ┌──────────────────┐
 │ FortiGate FW │
 └──────────────────┘

Four components, each running as a separate pod in the kagent namespace:

Telegram Bot — polls for messages, sends them to kagent via A2A, renders HITL approval buttons
kagent Controller — routes the request to the correct agent, manages conversation state
fortigate-agent — the AI agent (LLM) that decides which MCP tools to call based on your question
FortiGate MCP Server — translates MCP tool calls into FortiGate REST API calls

Part 1: The FortiGate MCP Server

Rather than writing a custom wrapper from scratch, I used the excellent fortigate-mcp-server community project. It’s a well-structured Python application built on FastMCP that exposes 28+ tools across six categories.

Tool Inventory

Category	Tools	What They Do
Device	6	List devices, test connectivity, get system status, discover VDOMs
Firewall	5	List, create, update, delete policies; get policy details
Network	4	List/create address objects and service objects
Routing	7	Static routes, routing table, interfaces, interface status
Virtual IPs	5	List, create, update, delete VIPs; get VIP details
System	2+	Health checks, connection tests, schema info

Multi-Device Support

One of the nice features of this MCP server is multi-device support. Each tool takes a device_id parameter, so you can manage multiple FortiGate appliances from a single server instance. The configuration file defines your device inventory:

{
 "fortigate": {
 "devices": {
 "primary": {
 "host": "172.16.10.1",
 "port": 443,
 "api_token": "your-api-token",
 "vdom": "root",
 "verify_ssl": false,
 "timeout": 30
 }
 }
 }
}

Authentication

FortiGate supports two auth methods — API tokens and username/password. I use API tokens because they don’t expire (unless revoked) and avoid the session management complexity. You generate one in the FortiGate GUI under System > Administrators > REST API Admin.

Containerization

I built a custom Docker image that generates the config.json from environment variables at startup. This lets Kubernetes secrets flow cleanly into the container without baking credentials into the image:

#!/bin/bash
cat > /app/config/config.json <<EOF
{
 "fortigate": {
 "devices": {
 "primary": {
 "host": "${FORTI_HOST}",
 "port": ${FORTI_PORT:-443},
 "api_token": "${FORTI_TOKEN}",
 "vdom": "${FORTI_VDOM:-root}",
 "verify_ssl": false,
 "timeout": 30
 }
 }
 }
}
EOF
exec python -m src.fortigate_mcp.server_http \
 --host 0.0.0.0 --port 8080 --path /mcp \
 --config /app/config/config.json

MCP Transport: Streamable HTTP

A key detail — kagent uses the Streamable HTTP MCP transport, not the older SSE transport. The server needs to call mcp.streamable_http_app() (not mcp.sse_app()) and initialize the MCP task group via a Starlette lifespan handler:

mcp_app = self.mcp.streamable_http_app()

@asynccontextmanager
async def lifespan(a):
 async with mcp_app.router.lifespan_context(mcp_app):
 yield

app = Starlette(
 lifespan=lifespan,
 routes=[
 Route("/health", health_endpoint),
 Mount("/mcp", app=mcp_app),
 ],
)

This exposes the MCP endpoint at /mcp/mcp — kagent POSTs JSON-RPC requests to this URL to discover and invoke tools.

Part 2: The kagent Agent

The agent is defined as a Kubernetes CRD. It tells kagent what the agent can do, which MCP tools it has access to, and which operations need human approval.

Agent CRD

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
 name: fortigate-agent
 namespace: kagent
spec:
 description: "AI agent for managing FortiGate firewall"
 type: Declarative
 declarative:
 modelConfig: default-model-config
 systemMessage: |
 You are fortigate-agent, an expert AI assistant
 for managing a FortiGate firewall.

 All tools require a device_id parameter.
 The configured device is called "primary".
 Always pass device_id="primary" when calling any tool.

 ## Behavioral Guidelines
 1. Confirm destructive operations before executing
 2. Present data clearly with tables
 3. Flag overly permissive policies (all/all/accept)
 4. Explain FortiGate API errors in plain language 
 tools:
 - type: McpServer
 mcpServer:
 kind: RemoteMCPServer
 name: fortigate-mcp
 namespace: kagent

HITL Approval for Destructive Operations

Read operations (listing policies, viewing routes) execute immediately. But anything that modifies the firewall requires your explicit approval:

requireApproval:
 - create_firewall_policy
 - update_firewall_policy
 - delete_firewall_policy
 - create_address_object
 - create_service_object
 - create_static_route
 - update_static_route
 - delete_static_route
 - create_virtual_ip
 - update_virtual_ip
 - delete_virtual_ip

When the LLM decides to call one of these tools, kagent pauses execution and sends an approval request back through A2A. The Telegram bot renders this as inline Approve/Reject buttons — you tap to decide.

MCP Tool Discovery

The RemoteMCPServer CRD points kagent at the FortiGate MCP server’s endpoint:

apiVersion: kagent.dev/v1alpha2
kind: RemoteMCPServer
metadata:
 name: fortigate-mcp
 namespace: kagent
spec:
 url: "http://fortigate-mcp-server.kagent:8080/mcp/mcp"
 timeout: "30s"
 sseReadTimeout: "5m0s"

kagent automatically connects, discovers all available tools, and makes them available to the agent. No manual tool registration needed.

Part 3: The Telegram Bot

The Telegram bot is the same Python bot I use for all my kagent agents. It reuses the sebbycorp/telegram-kagent-bot:latest image — the only thing that changes between agents is the KAGENT_A2A_URL environment variable:

env:
 - name: TELEGRAM_BOT_TOKEN
 valueFrom:
 secretKeyRef:
 name: telegram-forti-bot-token
 key: TELEGRAM_BOT_TOKEN
 - name: KAGENT_A2A_URL
 value: "http://kagent-controller.kagent.svc.cluster.local:8083/api/a2a/kagent/fortigate-agent/"

Same image, different agent. The bot handles:

Message routing — sends user messages to the FortiGate agent via A2A
Response rendering — formats the agent’s response (tables, summaries) for Telegram’s 4000-char message limit
HITL approval — renders Approve/Reject inline buttons when the agent needs permission for a destructive operation
Conversation context — maintains per-user context IDs so follow-up questions work naturally

The deployment uses strategy: Recreate because Telegram only allows one poller per bot token.

Part 4: Kubernetes Deployment

The entire stack is deployed via GitOps. The manifests live in the k8s-iceman repo:

manifests/kagent-examples/
├── fortigate-agent/
│ ├── 01-external-secret.yaml # FortiGate creds from Vault
│ ├── 02-agent.yaml # kagent Agent CRD
│ └── 03-deployment.yaml # Deployment + Service + MCP + NetworkPolicy
└── telegram-forti-bot/
 ├── 01-external-secret.yaml # Telegram token from Vault
 └── 02-deployment.yaml # Telegram bot Deployment

ArgoCD watches this directory and auto-syncs any changes. Push to main and the stack deploys itself.

Secrets from Vault

FortiGate credentials are stored in HashiCorp Vault and synced to Kubernetes via External Secrets Operator:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
 name: fortigate-credentials
 namespace: kagent
spec:
 secretStoreRef:
 kind: ClusterSecretStore
 name: vault-backend
 target:
 name: fortigate-credentials
 data:
 - secretKey: FORTI_TOKEN
 remoteRef:
 key: fortigate
 property: forti_token
 - secretKey: FORTI_HOST
 remoteRef:
 key: fortigate
 property: host

The Telegram bot token comes from a separate Vault path:

data:
 - secretKey: TELEGRAM_BOT_TOKEN
 remoteRef:
 key: secret/telegram
 property: fortiaikagentbot_key

Network Policy

The MCP server is locked down — only pods in the kagent namespace can reach it, and it can only talk to the FortiGate management IP and DNS:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: fortigate-mcp-server-policy
spec:
 podSelector:
 matchLabels:
 app: fortigate-mcp-server
 ingress:
 - from:
 - namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: kagent
 ports:
 - protocol: TCP
 port: 8080
 egress:
 - to:
 - ipBlock:
 cidr: 172.16.0.0/12
 ports:
 - protocol: TCP
 port: 443
 - to:
 - namespaceSelector: {}
 ports:
 - protocol: UDP
 port: 53

What It Looks Like in Practice

Here are some example conversations:

You: “Show me all firewall policies”

Agent: Returns a table with policy ID, name, source/destination interfaces, addresses, services, action, and NAT status.

You: “What VIPs are configured?”

Agent: Lists all Virtual IP mappings with external IP, mapped IP, port forwarding settings, and associated interfaces.

You: “Show me the routing table”

Agent: Displays the active routing table with destinations, gateways, interfaces, and metrics.

You: “Create an address object called test-server with IP 10.0.5.100/32”

Agent: “I’ll create an address object ’test-server’ with type ‘ipmask’ and address ‘10.0.5.100/32’.”

Telegram shows: [Approve] [Reject] buttons

You: Tap Approve

Agent: “Address object ’test-server’ created successfully.”

Safety Guardrails

This is network security infrastructure — the safety layers matter:

Layer	Guardrail
Agent	System prompt instructs the LLM to confirm destructive ops, flag overly permissive policies
kagent	`requireApproval` on 11 write operations — execution pauses until human approves
Telegram	HITL approval rendered as inline Approve/Reject buttons
FortiGate	API token scoped to specific admin profile with limited permissions
Network	NetworkPolicy restricts who can reach the MCP server and where it can connect

Lessons Learned

A few things I ran into while building this:

MCP transport matters. kagent uses Streamable HTTP, not SSE. If your MCP server only serves the older SSE transport, kagent will fail with “Method Not Allowed” on POST. Use mcp.streamable_http_app() and initialize the task group via a Starlette lifespan.

Lifespan initialization is required. The Streamable HTTP app needs async with mcp_app.router.lifespan_context(mcp_app) in your Starlette lifespan handler. Without it, you get “Task group is not initialized” errors.

Device ID is a required parameter. Every tool in the community MCP server requires a device_id. Make sure your agent’s system prompt tells the LLM to always pass device_id="primary" (or whatever you named your device in the config).

Vault key paths depend on the ClusterSecretStore. If your ESO ClusterSecretStore has path: "secret" configured, the ExternalSecret key should be just fortigate, not secret/fortigate — the provider prepends the path automatically.

Wrapping Up

This project reinforces the same patterns I’ve been exploring with F5 BIG-IP:

MCP as the universal integration layer — wrap any API as an MCP server and let AI agents discover and use it
kagent for agent lifecycle on Kubernetes — define agents as CRDs, deploy with GitOps, get HITL approval for free
A2A for agent communication — the Telegram bot is agent-agnostic; it speaks A2A to kagent, which routes to the right agent
Reusable bot pattern — same Telegram bot image, different KAGENT_A2A_URL, instant new agent interface

The FortiGate MCP server gives you 28+ tools covering device management, firewall policies, network objects, routing, VIPs, and system monitoring. And because it’s MCP, adding more tools is just adding more functions — kagent discovers them automatically.

The full source code is available in the k8s-iceman repository:

FortiGate MCP server modifications: apps/fortigate-wrapper-src/
Agent manifests: manifests/kagent-examples/fortigate-agent/
Telegram bot manifests: manifests/kagent-examples/telegram-forti-bot/

Managing F5 BIG-IP from Telegram with an AI Agent, MCP, and kagent

Sat, 14 Mar 2026 10:00:00 -0400

Managing F5 BIG-IP from Telegram with an AI Agent, MCP, and kagent

By Sebastian Maniak

What if you could manage your F5 BIG-IP load balancer from Telegram? Not by SSH-ing into the box and running tmsh — but by telling an AI agent in plain English to “show me all pools” or “disable node 10.0.1.25” and having it figure out the right iControl REST calls, execute them safely, and ask for your approval before anything destructive happens?

That’s exactly what I built. In this article, I’ll walk through the full stack: a custom MCP (Model Context Protocol) server that wraps the F5 iControl REST API, a kagent AI agent on Kubernetes that uses those tools to manage the BIG-IP, and a Telegram bot that lets you talk to the agent from your phone — complete with Human-in-the-Loop (HITL) approval buttons for destructive operations.

Everything runs on my home lab Kubernetes cluster (Talos Linux on Proxmox), deployed via GitOps with ArgoCD, with secrets managed by HashiCorp Vault.

Why Not Just Use the F5 API Directly?

The F5 iControl REST API is massive. It uses token-based auth with expiring sessions, returns deeply nested JSON, and has hundreds of endpoints. Talking to it directly from an AI agent would be fragile and wasteful.

I needed a wrapper that:

Exposes only what the agent needs — a curated set of pool, virtual server, node, monitor, iRule, certificate, and system operations instead of the full iControl surface area
Handles auth in one place — token acquisition, automatic refresh before the 1200s expiry, and clean logout on shutdown
Speaks MCP natively — so kagent discovers all tools automatically via a single endpoint, no manual tool registration needed
Adds guardrails — read-only mode, partition allow-lists, HITL approval for destructive operations, and clear error responses the agent can reason about

Architecture

Here’s how it all fits together:

┌──────────────┐
│ Telegram │
│ (user) │
└──────┬───────┘
 │ polling
 ▼
┌──────────────┐ A2A protocol ┌──────────────────┐
│ Telegram Bot │ ──────────────────────▶│ kagent Controller │
│ (K8s pod) │ │ │
└──────────────┘ └────────┬─────────┘
 │ routes to agent
 ▼
 ┌──────────────────┐
 │ f5-bigip-agent │
 │ (LLM decides │
 │ which tool) │
 └────────┬─────────┘
 │ MCP streamable-HTTP
 ▼
 ┌──────────────────┐
 │ F5 Wrapper │
 │ (FastAPI + MCP) │
 └────────┬─────────┘
 │ iControl REST
 ▼
 ┌──────────────────┐
 │ F5 BIG-IP │
 └──────────────────┘

Four components, each running as a separate pod in the kagent namespace:

Telegram Bot — polls for messages, sends them to kagent via A2A, renders HITL approval buttons
kagent Controller — routes the request to the correct agent, manages conversation state
f5-bigip-agent — the AI agent (LLM) that decides which MCP tools to call based on the user’s natural language request
F5 Wrapper — FastAPI service that translates MCP tool calls into iControl REST API calls against the BIG-IP

Part 1: The F5 MCP Wrapper

The wrapper is a Python FastAPI application that serves two purposes: a REST API for direct HTTP access and an MCP server for kagent tool discovery.

Project Structure

f5-wrapper/
├── Dockerfile
├── requirements.txt
├── app/
│ ├── main.py # Starlette root — mounts REST + MCP
│ ├── mcp_server.py # 28 MCP tool definitions
│ ├── config.py # Pydantic Settings (env vars)
│ ├── auth.py # F5 token manager
│ ├── routers/ # REST API endpoints
│ │ ├── pools.py
│ │ ├── virtual_servers.py
│ │ ├── nodes.py
│ │ ├── monitors.py
│ │ ├── irules.py
│ │ ├── certificates.py
│ │ └── system.py
│ └── utils/
│ └── f5_client.py # Reusable HTTP client for iControl REST

Authentication: Token Lifecycle

The F5 iControl REST API uses token-based auth. Rather than handling this in every tool, the wrapper manages the entire token lifecycle transparently:

class F5TokenManager:
 def __init__(self, host, username, password, verify_ssl=False):
 self.host = host
 self.token = None
 self.token_expiry = 0

 async def login(self):
 resp = await self.client.post(
 f"{self.host}/mgmt/shared/authn/login",
 json={"username": self.username, "password": self.password,
 "loginProviderName": "tmos"},
 )
 self.token = resp.json()["token"]["token"]
 # Tokens last 1200s; refresh at 80% (960s)
 self.token_expiry = time.time() + 960

 async def get_token(self):
 if time.time() >= self.token_expiry:
 await self.login()
 return self.token

 async def logout(self):
 # Clean up token on shutdown
 await self.client.delete(
 f"{self.host}/mgmt/shared/authz/tokens/{self.token}",
 headers={"X-F5-Auth-Token": self.token},
 )

On startup, the app logs in and stores the token. Every request checks if the token is about to expire and refreshes automatically. On shutdown, the token is deleted.

MCP Tool Definitions (28 Tools)

The MCP server uses the FastMCP library to expose tools that kagent discovers automatically. Here’s the full tool inventory:

Category	Tools	Description
Pools	8	List, create, delete pools; add/remove/enable/disable members
Virtual Servers	4	List, get details, create, delete virtual servers
Nodes	5	List, get, create, delete nodes; enable/disable/force-offline
Monitors	4	List all monitors, HTTP, HTTPS, and TCP monitors
iRules	2	List iRules, get full TCL definitions
Certificates	2	List SSL certs and expiration dates
System	3	BIG-IP version/hostname, HA failover status, config sync status

Each tool is a simple async function decorated with @mcp.tool(). For example, here’s the pool listing tool:

@mcp.tool()
async def list_pools(partition: str = "Common") -> str:
 """List all LTM pools in the specified partition."""
 data = await _client().get(
 f"/mgmt/tm/ltm/pool?$filter=partition eq {partition}"
 )
 items = data.get("items", [])
 result = []
 for item in items:
 result.append({
 "name": item["name"],
 "partition": item.get("partition", "Common"),
 "monitor": item.get("monitor", "none"),
 "lb_method": item.get("loadBalancingMode", "round-robin"),
 })
 return json.dumps(result, indent=2)

And a write operation with read-only guard:

@mcp.tool()
async def create_pool(name, partition="Common", monitor="/Common/http",
 lb_method="round-robin", members=None) -> str:
 """Create a new LTM pool with optional members."""
 if settings.READ_ONLY:
 return "ERROR: Service is in read-only mode"
 payload = {
 "name": name, "partition": partition,
 "monitor": monitor, "loadBalancingMode": lb_method,
 }
 if members:
 payload["members"] = json.loads(members) if isinstance(members, str) else members
 data = await _client().post("/mgmt/tm/ltm/pool", payload)
 return json.dumps(data, indent=2)

Mounting MCP + REST Together

The app uses Starlette as the root application to mount both the MCP server and the FastAPI REST API under different paths:

# MCP streamable-HTTP app
mcp_app = mcp.streamable_http_app()

# Main Starlette app
app = Starlette(
 lifespan=lifespan,
 routes=[
 Route("/health", health),
 Mount("/mcp", app=mcp_app), # MCP tools for kagent
 Mount("/", app=rest_app), # REST API + Swagger docs
 ],
)

The lifespan handler initializes the F5 token manager on startup and triggers the MCP sub-app’s lifespan to set up its async task group:

@asynccontextmanager
async def lifespan(app):
 tm = F5TokenManager(host=settings.F5_HOST, ...)
 await tm.login()
 set_token_manager(tm)
 async with mcp_app.router.lifespan_context(mcp_app):
 yield
 await tm.logout()

Configuration

All configuration is via environment variables, loaded by Pydantic Settings:

Variable	Required	Default	Description
`F5_HOST`	Yes	—	BIG-IP management URL
`F5_USERNAME`	No	`admin`	iControl REST username
`F5_PASSWORD`	Yes	—	iControl REST password
`F5_VERIFY_SSL`	No	`false`	Verify F5 SSL certificate
`F5_PARTITION`	No	`Common`	Default BIG-IP partition
`ALLOWED_PARTITIONS`	No	`Common`	Partition allow-list
`READ_ONLY`	No	`false`	Block all write operations

Part 2: The kagent Agent

The AI agent is defined as a Kubernetes CRD — a declarative Agent resource that tells kagent what the agent can do, which tools it has, and which operations need human approval.

Agent CRD

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
 name: f5-bigip-agent
 namespace: kagent
spec:
 description: "AI agent for managing F5 BIG-IP load balancer"
 type: Declarative
 declarative:
 modelConfig: default-model-config
 systemMessage: |
 You are f5-bigip-agent, an expert AI assistant for managing
 the F5 BIG-IP load balancer.

 ## Behavioral Guidelines
 1. Always confirm destructive operations before executing
 2. Check dependencies before deleting (e.g., is a VS using this pool?)
 3. Check HA failover status before write operations
 4. Use tables for listing multiple resources
 ... 
 tools:
 - type: McpServer
 mcpServer:
 kind: RemoteMCPServer
 name: f5-bigip-mcp
 namespace: kagent

HITL Approval

The critical piece is requireApproval — this tells kagent that 10 destructive operations must get human approval before execution:

requireApproval:
 - create_pool
 - delete_pool
 - add_pool_member
 - remove_pool_member
 - set_pool_member_state
 - create_virtual_server
 - delete_virtual_server
 - create_node
 - delete_node
 - set_node_state

When the LLM decides to call one of these tools, kagent pauses execution and sends an approval request back through the A2A protocol. The Telegram bot renders this as Approve/Reject buttons.

A2A Skills

The agent also advertises its capabilities via A2A skills, so other agents or clients can discover what it does:

a2aConfig:
 skills:
 - id: f5-bigip-operations
 name: F5 BIG-IP Operations
 description: "Manage pools, virtual servers, nodes, monitors,
 and system status on F5 BIG-IP"
 examples:
 - "Show me all pools on the F5"
 - "Disable member 10.0.1.25:80 in pool prod-web"
 - "What is the failover status of the BIG-IP?"

Part 3: The Telegram Bot

The Telegram bot is the same Python bot I built for general kagent operations, redeployed with a different environment variable pointing it at the F5 agent:

env:
 - name: TELEGRAM_BOT_TOKEN
 valueFrom:
 secretKeyRef:
 name: telegram-f5-bot-token
 key: TELEGRAM_BOT_TOKEN
 - name: KAGENT_A2A_URL
 value: "http://kagent-controller.kagent.svc.cluster.local:8083/api/a2a/kagent/f5-bigip-agent/"

That’s it — same bot image (sebbycorp/telegram-kagent-bot:latest), different agent URL. The bot polls Telegram for messages, sends them to the F5 agent via A2A, and streams the response back to the chat. When a HITL approval is needed, it shows inline Approve/Reject buttons.

The deployment uses strategy: Recreate because Telegram only allows one poller per bot token.

Part 4: Kubernetes Deployment

The entire stack is deployed via GitOps. The manifests live in the k8s-iceman repo under manifests/kagent-examples/:

manifests/kagent-examples/
├── f5-agent/
│ └── bigip/
│ ├── 01-external-secret.yaml # F5 creds from Vault
│ ├── 02-agent.yaml # kagent Agent CRD
│ └── 03-deployment.yaml # Deployment + Service + MCP + NetworkPolicy
└── telegram-f5-bot/
 ├── 01-external-secret.yaml # Telegram token from Vault
 └── 02-deployment.yaml # Telegram bot Deployment

ArgoCD watches this directory and auto-syncs any changes.

Secrets from Vault

F5 credentials are stored in HashiCorp Vault and injected via External Secrets Operator:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
 name: f5-bigip-credentials
spec:
 secretStoreRef:
 kind: ClusterSecretStore
 name: vault-backend
 target:
 name: f5-bigip-credentials
 data:
 - secretKey: F5_HOST
 remoteRef:
 key: secret/f5
 property: host
 - secretKey: F5_USERNAME
 remoteRef:
 key: secret/f5
 property: username
 - secretKey: F5_PASSWORD
 remoteRef:
 key: secret/f5
 property: password

Network Policy

The wrapper is locked down with a NetworkPolicy that only allows ingress from the kagent namespace and egress to the F5 management IP + DNS:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: f5-wrapper-bigip-policy
spec:
 podSelector:
 matchLabels:
 app: f5-wrapper-bigip
 ingress:
 - from:
 - namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: kagent
 ports:
 - protocol: TCP
 port: 8080
 egress:
 - to:
 - ipBlock:
 cidr: 10.0.0.0/8
 ports:
 - protocol: TCP
 port: 443
 - to:
 - namespaceSelector: {}
 ports:
 - protocol: UDP
 port: 53

Safety Guardrails

This is network infrastructure — you don’t want an AI agent accidentally deleting a production pool. Here are the layers of safety built in:

Layer	Guardrail
Agent	System prompt instructs the LLM to confirm before destructive ops, check dependencies, verify HA status
kagent	`requireApproval` on 10 destructive tools — execution pauses until human approves
Telegram	HITL approval rendered as inline Approve/Reject buttons
Wrapper	`READ_ONLY=true` mode blocks all write operations at the API level
Wrapper	`ALLOWED_PARTITIONS` restricts which F5 partitions can be accessed
Network	NetworkPolicy limits which pods can reach the wrapper and where the wrapper can connect

What It Looks Like in Practice

Here’s a typical conversation flow:

You (in Telegram): “Show me all pools on the F5”

Agent: Lists pools in a clean table with member counts, monitors, and load balancing methods.

You: “Disable member 10.0.1.50:80 in pool k8s-argocd”

Agent: “I’ll disable member 10.0.1.50:80 in pool k8s-argocd. This will stop it from receiving new connections but allow existing connections to drain.”

Telegram shows: [Approve] [Reject] buttons

You: Tap Approve

Agent: “Done. Member 10.0.1.50:80 in pool k8s-argocd is now disabled (session: user-disabled, state: user-up).”

CI/CD

A GitHub Actions workflow automatically builds and pushes the f5-wrapper Docker image to Docker Hub whenever files in apps/f5-wrapper/ change on main. PRs build the image for validation but don’t push.

Wrapping Up

This project brings together several patterns that I think are the future of infrastructure management:

MCP as the integration layer — instead of writing bespoke integrations, wrap existing APIs as MCP servers and let any AI agent discover and use them
kagent for agent lifecycle — define agents as Kubernetes CRDs, manage them with GitOps, get HITL approval for free
A2A for agent communication — the Telegram bot doesn’t know anything about F5; it just speaks A2A to kagent, which routes to the right agent
Defense in depth — read-only mode, partition allow-lists, HITL approval, network policies, and agent-level behavioral guidelines

The F5 wrapper has 28 tools covering pools, virtual servers, nodes, monitors, iRules, certificates, and system operations. You could extend it with more tools (data groups, policies, WAF rules) by adding more @mcp.tool() functions — kagent discovers them automatically.

The full source code is available in the k8s-iceman repository:

F5 wrapper: apps/f5-wrapper/
Agent manifests: manifests/kagent-examples/f5-agent/
Telegram bot manifests: manifests/kagent-examples/telegram-f5-bot/

Building a Telegram Bot for Your Kubernetes Cluster with kagent and A2A

Fri, 13 Mar 2026 14:00:00 -0400

Building a Telegram Bot for Your Kubernetes Cluster with kagent and A2A

By Sebastian Maniak

What if you could manage your Kubernetes cluster from Telegram? Not through a half-baked webhook that runs kubectl — but through a real AI agent that understands context, uses tools, responds intelligently, and asks for your approval before doing anything destructive?

In this article, I’ll walk you through how I built exactly that: a Telegram bot that connects to a kagent AI agent running on my home lab Kubernetes cluster (Talos Linux on Proxmox), giving me full cluster operations from my phone. The entire thing is deployed via GitOps with ArgoCD, secrets come from HashiCorp Vault, and the bot uses the A2A (Agent-to-Agent) protocol to communicate with kagent.

The bot maintains conversation continuity across messages (so the agent remembers what you were talking about), and supports Human-in-the-Loop (HITL) approval — when the agent wants to run a destructive operation like deleting a resource or applying a manifest, it shows you Approve/Reject buttons in Telegram before proceeding.

No webhooks. No public endpoints. Just polling from inside the cluster.

Architecture Overview

Here’s what we’re building:

┌─────────────────────────────────────────────────────────────────────┐
│ Telegram Cloud │
│ ┌──────────────────┐ │
│ │ Telegram Bot API │ │
│ └────────┬─────────┘ │
│ │ Long Polling │
└─────────────────────────────┼───────────────────────────────────────┘
 │
┌─────────────────────────────┼───────────────────────────────────────┐
│ Kubernetes Cluster │ │
│ (maniak-iceman) ▼ │
│ ┌──────────────────────────┐ │
│ │ telegram-bot (Pod) │ │
│ │ python-telegram-bot │ │
│ │ + httpx │ │
│ │ │ │
│ │ Tracks contextId per │ │
│ │ Telegram user for │ │
│ │ session continuity │ │
│ └────────────┬─────────────┘ │
│ │ HTTP POST (A2A JSON-RPC) │
│ │ message/send + contextId │
│ ▼ │
│ ┌──────────────────────────┐ │
│ │ kagent-controller │ │
│ │ :8083/api/a2a/kagent/ │ │
│ │ telegram-k8s-agent/ │ │
│ └────────────┬─────────────┘ │
│ │ Routes to Agent │
│ ▼ │
│ ┌──────────────────────────┐ ┌──────────────────┐ │
│ │ telegram-k8s-agent (Pod) │─────▶│ kagent-tool- │ │
│ │ LLM: gpt-5.4 │ MCP │ server (Pod) │ │
│ │ Agent CRD │◀─────│ :8084/mcp │ │
│ │ + long-term memory │ └──────────────────┘ │
│ │ + context compaction │ │ │
│ └──────────────────────────┘ ┌────────┴──────────┐ │
│ │ Tools: │ │
│ Response states: │ • k8s_get_resources│ │
│ • completed │ • k8s_create_resource│ │
│ • input-required │ • k8s_apply_manifest│ │
│ (HITL approval) │ • k8s_delete_resource│ │
│ │ • k8s_get_pod_logs│ │
│ │ • k8s_scale │ │
│ └────────────────────┘ │
│ │
│ ┌───────────────┐ ┌──────────────────┐ │
│ │ Vault │───▶│ ExternalSecret │──▶ telegram-bot-token │
│ │ secret/ │ │ Operator │ │
│ │ telegram │ └──────────────────┘ │
│ └───────────────┘ │
└─────────────────────────────────────────────────────────────────────┘

The key insight: the Telegram bot doesn’t talk to the LLM directly. It sends messages to the kagent controller’s A2A endpoint, which routes them to the correct agent. The agent handles LLM orchestration, tool invocation, and response generation. The bot is a transport layer that also handles session tracking (via contextId) and HITL approval (via Telegram inline keyboards).

The A2A Protocol

A2A (Agent-to-Agent) is a Google-backed open protocol for agent interoperability. kagent implements A2A on its controller, meaning any A2A-compatible client can talk to any kagent agent.

The protocol uses JSON-RPC 2.0 over HTTP. Here’s what a message exchange looks like:

┌──────────┐ ┌───────────────────┐ ┌─────────────┐
│ Telegram │ │ kagent-controller │ │ Agent Pod │
│ Bot Pod │ │ (A2A endpoint) │ │ + LLM + MCP │
└────┬─────┘ └─────────┬──────────┘ └──────┬──────┘
 │ │ │
 │ POST /api/a2a/kagent/ │ │
 │ telegram-k8s-agent/ │ │
 │ ┌─────────────────────────┐ │ │
 │ │ { │ │ │
 │ │ "jsonrpc": "2.0", │ │ │
 │ │ "method": │ │ │
 │ │ "message/send", │ │ │
 │ │ "params": { │ │ │
 │ │ "message": { │ │ │
 │ │ "role": "user", │ │ │
 │ │ "kind": "message",│ │ │
 │ │ "contextId": │ │ │
 │ │ "abc-123...", │ │ │
 │ │ "parts": [{ │ │ │
 │ │ "kind": "text", │ │ │
 │ │ "text": "list │ │ │
 │ │ my pods" │ │ │
 │ │ }] │ │ │
 │ │ } │ │ │
 │ │ } │ │ │
 │ │ } │ │ │
 │ └─────────────────────────┘ │ │
 │ ──────────────────────────────────▶│ │
 │ │ Forward to agent │
 │ │─────────────────────────────────▶│
 │ │ │
 │ │ LLM + Tool calls │
 │ │ (k8s_get_resources) │
 │ │ │
 │ │◀─────────────────────────────────│
 │ ◀──────────────────────────────────│ Response with artifacts │
 │ ┌─────────────────────────┐ │ │
 │ │ { │ │ │
 │ │ "result": { │ │ │
 │ │ "contextId": │ │ │
 │ │ "abc-123...", │ │ │
 │ │ "status": { │ │ │
 │ │ "state": │ │ │
 │ │ "completed" │ │ │
 │ │ }, │ │ │
 │ │ "artifacts": [{ │ │ │
 │ │ "parts": [{ │ │ │
 │ │ "kind": "text", │ │ │
 │ │ "text": "Here │ │ │
 │ │ are your │ │ │
 │ │ pods: ..." │ │ │
 │ │ }] │ │ │
 │ │ }] │ │ │
 │ │ } │ │ │
 │ │ } │ │ │
 │ └─────────────────────────┘ │ │
 │ │ │

Key things about kagent’s A2A implementation

The method is message/send (not tasks/send as in the older A2A draft spec)
Parts use "kind": "text" (not "type": "text")
The URL pattern is /api/a2a/{namespace}/{agent-name}/ — trailing slash required
Session continuity uses contextId inside the message object — not sessionId in params
Responses include a status.state field: "completed" for normal responses, "input-required" for HITL approval
Completed responses have text in result.artifacts[].parts[]
Input-required responses have data in result.status.message.parts[] and text in result.history[]

Session Continuity with `contextId`

This is the most important (and least documented) part of kagent’s A2A protocol. To maintain a conversation across multiple messages, you must include a contextId in the message object itself:

{
 "jsonrpc": "2.0",
 "id": "unique-message-id",
 "method": "message/send",
 "params": {
 "message": {
 "role": "user",
 "kind": "message",
 "contextId": "previously-returned-context-id",
 "parts": [{"kind": "text", "text": "now scale it to 3 replicas"}]
 }
 }
}

On the first message, omit contextId — kagent will generate one and return it in result.contextId. Store that value and send it back on every subsequent message to continue the conversation.

If you don’t send contextId, every message starts a fresh session and the agent won’t remember what you were talking about.

Response States

kagent A2A responses come in two states:

completed — The agent finished processing. The response text is in artifacts:

{
 "result": {
 "contextId": "abc-123",
 "status": {"state": "completed"},
 "artifacts": [{"parts": [{"kind": "text", "text": "Here are your pods..."}]}]
 }
}

input-required — The agent needs human input before proceeding. This happens when a tool in the requireApproval list is about to be called. The response contains an adk_request_confirmation data part describing which tool and what arguments:

{
 "result": {
 "contextId": "abc-123",
 "status": {
 "state": "input-required",
 "message": {
 "parts": [{
 "kind": "data",
 "data": {
 "name": "adk_request_confirmation",
 "args": {
 "originalFunctionCall": {
 "name": "k8s_create_resource",
 "args": {"namespace": "staging", "kind": "Namespace"}
 }
 }
 }
 }]
 }
 }
 }
}

To approve or reject, send a follow-up message with "approved" or "rejected" using the same contextId.

The Components

1. ExternalSecret — Pulling the Bot Token from Vault

The Telegram bot token lives in HashiCorp Vault at secret/telegram with key api_key. The External Secrets Operator syncs it into a Kubernetes secret:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
 name: telegram-bot-token
 namespace: kagent
spec:
 refreshInterval: "1h"
 secretStoreRef:
 name: vault-backend
 kind: ClusterSecretStore
 target:
 name: telegram-bot-token
 creationPolicy: Owner
 data:
 - secretKey: TELEGRAM_BOT_TOKEN
 remoteRef:
 key: telegram
 property: api_key

This follows the same pattern as the existing kagent-openai and kagent-anthropic secrets in the cluster. No hardcoded tokens in Git.

2. The Agent CRD

The telegram-k8s-agent is a kagent Declarative agent focused on K8s resource management. It has long-term memory (vector-backed via SQLite), context compaction for long conversations, and a prompt template with kagent’s built-in safety guardrails:

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
 name: telegram-k8s-agent
 namespace: kagent
spec:
 description: "Kubernetes resource management agent accessible via Telegram bot"
 type: Declarative
 declarative:
 modelConfig: default-model-config
 a2aConfig:
 skills:
 - id: k8s-operations
 name: Kubernetes Operations
 description: "Create, apply, inspect, and manage Kubernetes resources"
 examples:
 - "Create a new staging namespace"
 - "Apply a deployment for nginx"
 - "What pods are running in the default namespace?"
 - "Show me the events in namespace kagent"
 - "Scale the nginx deployment to 3 replicas"
 tags:
 - kubernetes
 - operations
 # Long-term memory — remembers user preferences, namespaces, and
 # past operations across conversations (vector-backed via SQLite).
 memory:
 modelConfig: openai-embed
 # Context compaction — automatically summarizes older messages in long
 # conversations so the agent doesn't lose track during extended sessions.
 context:
 compaction:
 tokenThreshold: 120000
 eventRetentionSize: 80
 overlapSize: 8
 promptTemplate:
 dataSources:
 - kind: ConfigMap
 name: kagent-builtin-prompts
 alias: builtin
 systemMessage: |
 You are {{.AgentName}}, a Kubernetes resource management agent accessible via Telegram.

 {{include "builtin/safety-guardrails"}}
 {{include "builtin/tool-usage-best-practices"}}
 {{include "builtin/kubernetes-context"}}

 Your primary role is helping users create, apply, inspect, and manage Kubernetes resources.
 You can:
 - Create namespaces, deployments, services, configmaps, and other K8s resources
 - Apply YAML manifests to the cluster
 - Inspect running resources (pods, deployments, services, events, logs)
 - Scale deployments and manage rollouts
 - Describe resources and troubleshoot issues

 Guidelines:
 - Keep responses concise and well-formatted for Telegram chat
 - Use short code blocks for YAML and logs
 - Summarize large outputs — don't dump full resource lists
 - When creating resources, confirm the namespace and resource details before applying
 - Mutating operations (create, apply, delete, scale) require user approval — explain what you plan to do first

 Available tools: {{.ToolNames}} 
 tools:
 - type: McpServer
 mcpServer:
 apiGroup: kagent.dev
 kind: RemoteMCPServer
 name: kagent-tool-server
 toolNames:
 # --- Read / Inspect ---
 - k8s_get_resources
 - k8s_describe_resource
 - k8s_get_pod_logs
 - k8s_get_events
 - k8s_get_resource_yaml
 - k8s_get_available_api_resources
 # --- Create / Apply / Mutate ---
 - k8s_create_resource
 - k8s_apply_manifest
 - k8s_delete_resource
 - k8s_scale
 - k8s_rollout
 - k8s_label_resource
 - k8s_annotate_resource
 requireApproval:
 - k8s_create_resource
 - k8s_apply_manifest
 - k8s_delete_resource
 - k8s_scale

Notable features:

memory with openai-embed: The agent remembers context from past conversations using vector search. If you told it “I always work in the staging namespace” last week, it’ll remember.
context.compaction: Long Telegram conversations won’t blow out the LLM context window. kagent automatically summarizes older messages when the token count exceeds 120k.
promptTemplate with dataSources: Pulls in kagent’s built-in safety guardrails and Kubernetes-aware prompts from a ConfigMap, so the system message stays DRY.
requireApproval: The four mutating tools (create, apply, delete, scale) trigger HITL — the bot shows Approve/Reject buttons in Telegram before they execute.

3. The Bot Code

The bot is ~460 lines of Python using python-telegram-bot (polling mode) and httpx for A2A calls. It handles three main concerns: A2A communication with session continuity, HITL approval flow via Telegram inline keyboards, and robust response parsing.

Let’s walk through it section by section.

A2A Communication

The core of the bot — sending messages to kagent and tracking contextId for session continuity:

"""Telegram bot that forwards messages to a kagent A2A agent."""

import asyncio
import json
import logging
import os
import uuid
from pathlib import Path

import httpx
from telegram import InlineKeyboardButton, InlineKeyboardMarkup, Update
from telegram.ext import (
 Application,
 CallbackQueryHandler,
 CommandHandler,
 MessageHandler,
 filters,
)

TELEGRAM_BOT_TOKEN = os.environ["TELEGRAM_BOT_TOKEN"]
KAGENT_A2A_URL = os.environ["KAGENT_A2A_URL"]

# Per-user contextId for conversation continuity (maps Telegram user_id -> kagent contextId)
user_contexts: dict[int, str] = {}

# Pending approval tasks: callback_id -> {context_id, user_id}
pending_approvals: dict[str, dict] = {}


async def _send_a2a_request(
 message_text: str,
 context_id: str | None = None,
) -> dict:
 """Send a message to the kagent A2A endpoint and return the raw result."""
 message = {
 "role": "user",
 "kind": "message",
 "messageId": str(uuid.uuid4()),
 "parts": [{"kind": "text", "text": message_text}],
 }
 if context_id:
 message["contextId"] = context_id

 payload = {
 "jsonrpc": "2.0",
 "id": message["messageId"],
 "method": "message/send",
 "params": {"message": message},
 }

 async with httpx.AsyncClient(timeout=120.0) as client:
 resp = await client.post(
 KAGENT_A2A_URL,
 json=payload,
 headers={"Content-Type": "application/json"},
 )
 resp.raise_for_status()
 data = resp.json()

 result = data.get("result", {})
 return result

The critical detail: contextId goes inside the message object, not in params. On the first message, we omit it — kagent returns a contextId in the result. We store that per Telegram user and send it on every subsequent message. This is what makes “tell me about nginx” followed by “now scale it to 3” work as a coherent conversation.

Response Parsing

kagent returns text in different locations depending on the response state. The bot tries three sources in order:

def _extract_text(result: dict) -> str | None:
 """Extract text from an A2A task result (artifacts, history, or status message)."""
 # Check artifacts first (completed responses)
 artifacts = result.get("artifacts", [])
 if artifacts:
 parts = artifacts[-1].get("parts", [])
 texts = [p.get("text", "") for p in parts if p.get("kind") == "text" and p.get("text")]
 if texts:
 return "\n".join(texts)

 # Check history — last agent message with text parts
 for msg in reversed(result.get("history", [])):
 if msg.get("role") != "agent":
 continue
 texts = [p.get("text", "") for p in msg.get("parts", []) if p.get("kind") == "text" and p.get("text")]
 if texts:
 return "\n".join(texts)

 # Check status message
 for p in result.get("status", {}).get("message", {}).get("parts", []):
 if p.get("kind") == "text" and p.get("text"):
 return p["text"]

 return None

Why three sources? completed responses put text in artifacts. But input-required responses (HITL) don’t have artifacts — the agent’s explanatory text is in the history array. And some edge cases put short status messages in status.message.parts. Without this fallback chain, you’d get “Agent returned no text response” for perfectly valid HITL interactions.

HITL Approval Flow

When kagent returns input-required, the bot parses the adk_request_confirmation data to figure out what tool the agent wants to run, then shows Telegram inline keyboard buttons:

def _parse_adk_confirmation(data: dict) -> dict | None:
 """Parse an adk_request_confirmation DataPart into a structured dict."""
 if data.get("name") == "adk_request_confirmation":
 args = data.get("args", {})
 func_call = args.get("originalFunctionCall", {})
 tool_name = func_call.get("name", "")
 tool_args = func_call.get("args", {})
 hint = args.get("toolConfirmation", {}).get("hint", "")

 if tool_name == "ask_user":
 questions = tool_args.get("questions", [])
 if isinstance(questions, str):
 questions = [{"question": questions}]
 return {"type": "ask_user", "tool_name": tool_name, "questions": questions, "hint": hint}

 return {"type": "approval", "tool_name": tool_name, "tool_args": tool_args, "hint": hint}

 return None

The bot classifies input-required responses into three categories:

Approval — A tool from the requireApproval list (e.g., k8s_create_resource). Shows Approve/Reject buttons.
Ask user — The agent is using ask_user to ask a clarifying question, sometimes with predefined choices.
Question — Generic fallback, prompts the user for free-text input.

When the user presses Approve, the bot sends "approved" back to kagent with the same contextId, and the agent proceeds to execute the tool:

async def handle_callback(update: Update, _) -> None:
 """Handle inline keyboard button presses (approval and choice selection)."""
 query = update.callback_query
 await query.answer()

 data = query.data
 parts = data.split(":", 2)
 action = parts[0]
 callback_id = parts[1]

 approval = pending_approvals.pop(callback_id, None)
 if not approval:
 await query.edit_message_text("This action has expired or was already handled.")
 return

 context_id = approval.get("context_id", "")

 if action == "approve":
 await query.edit_message_text("Approved. Processing...")
 reply_text = "approved"
 elif action == "reject":
 await query.edit_message_text("Rejected.")
 reply_text = "rejected"
 elif action == "choice":
 choice_value = parts[2] if len(parts) > 2 else ""
 await query.edit_message_text(f"Selected: {choice_value}")
 reply_text = choice_value

 # Send the response back to kagent with the same contextId
 result = await send_a2a_message(reply_text, context_id)
 # ... handle the follow-up result

Message Handler

The main handler ties it all together — sends the user’s text to kagent, stores the returned contextId, and dispatches to the right renderer:

async def handle_message(update: Update, _) -> None:
 """Forward user message to kagent A2A and reply with the response."""
 user_id = update.effective_user.id
 user_text = update.message.text
 context_id = user_contexts.get(user_id)

 thinking_msg = await update.message.reply_text("Thinking...")

 try:
 result = await send_a2a_message(user_text, context_id)
 ctx = result.get("contextId")
 if ctx:
 user_contexts[user_id] = ctx
 await _handle_a2a_result(result, user_id, thinking_msg)
 except Exception as e:
 await thinking_msg.edit_text(f"Error contacting agent: {e}")

The _handle_a2a_result function checks status.state — if it’s input-required, it renders the approval UI; otherwise, it sends the text response.

Key design decisions:

Polling, not webhooks: No ingress, no public endpoint, no TLS termination. The bot makes outbound connections to Telegram’s API from inside the cluster.
contextId per user: Each Telegram user’s conversation maps to a kagent contextId. The /new command clears it, starting a fresh session.
Inline keyboards for approvals: Instead of making the user type “yes” or “no”, the bot shows tappable buttons for HITL approval. Each button carries a callback ID that maps to the pending approval’s context_id.
Chunked responses: Telegram has a 4096-character message limit. Long agent responses are split into 4000-char chunks automatically.
Health file: A simple /tmp/bot-healthy file is created on startup for Kubernetes liveness/readiness probes.

4. The Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
 name: telegram-bot
 namespace: kagent
spec:
 replicas: 1
 strategy:
 type: Recreate  # Only one poller at a time
 template:
 spec:
 containers:
 - name: bot
 image: docker.io/sebbycorp/telegram-kagent-bot:latest
 imagePullPolicy: Always
 env:
 - name: TELEGRAM_BOT_TOKEN
 valueFrom:
 secretKeyRef:
 name: telegram-bot-token
 key: TELEGRAM_BOT_TOKEN
 - name: KAGENT_A2A_URL
 value: "http://kagent-controller.kagent.svc.cluster.local:8083/api/a2a/kagent/telegram-k8s-agent/"
 - name: LOG_LEVEL
 value: "INFO"
 resources:
 requests:
 cpu: 50m
 memory: 64Mi
 limits:
 cpu: 200m
 memory: 128Mi
 livenessProbe:
 exec:
 command:
 - python
 - -c
 - "import os; assert os.path.exists('/tmp/bot-healthy')"
 initialDelaySeconds: 15
 periodSeconds: 30
 readinessProbe:
 exec:
 command:
 - python
 - -c
 - "import os; assert os.path.exists('/tmp/bot-healthy')"
 initialDelaySeconds: 10
 periodSeconds: 15

The Recreate strategy is important — Telegram’s polling API doesn’t support multiple consumers. If you use RollingUpdate, you’d briefly have two pods pulling the same messages.

The HITL Approval Flow

This is the most interesting part of the bot. Here’s what happens when you ask the agent to do something destructive:

┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Telegram │ │ Bot (Pod) │ │ kagent A2A │
│ User │ │ │ │ │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
 │ │ │
 │ "Create a staging namespace" │ │
 │──────────────────────────────▶│ │
 │ │ message/send │
 │ │ (no contextId — first msg) │
 │ │─────────────────────────────▶│
 │ │ │
 │ │ status: input-required │
 │ │ contextId: abc-123 │
 │ │ data: adk_request_ │
 │ │ confirmation { │
 │ │ k8s_create_resource │
 │ │ namespace: staging │
 │ │ } │
 │ │◀─────────────────────────────│
 │ │ │
 │ "The agent wants to run: │ │
 │ k8s_create_resource │ │
 │ namespace: staging" │ │
 │ │ │
 │ [ Approve ] [ Reject ] │ │
 │◀──────────────────────────────│ │
 │ │ │
 │ *taps Approve* │ │
 │──────────────────────────────▶│ │
 │ │ message/send │
 │ │ contextId: abc-123 │
 │ │ text: "approved" │
 │ │─────────────────────────────▶│
 │ │ │
 │ │ status: completed │
 │ │ "Namespace staging created" │
 │ │◀─────────────────────────────│
 │ │ │
 │ "Namespace staging created │ │
 │ successfully." │ │
 │◀──────────────────────────────│ │

The same contextId is used throughout the entire flow — from the initial request, through the approval, to the final result. This means the agent maintains context even across HITL interactions. You can ask “now deploy nginx to that namespace” and it’ll know you mean staging.

GitOps Flow

The entire deployment is managed through ArgoCD. Here’s how changes flow:

┌─────────────┐ git push ┌──────────────────┐
│ Developer │ ─────────────────▶│ GitHub │
│ (You) │ │ ProfessorSeb/ │
└─────────────┘ │ k8s-iceman │
 └────────┬──────────┘
 │
 ArgoCD polls │ (3 min)
 ▼
 ┌──────────────────┐
 │ ArgoCD │
 │ kagent-examples │
 │ Application │
 └────────┬──────────┘
 │
 kubectl apply│
 ▼
 ┌──────────────────┐
 │ Kubernetes │
 │ ├─ ExternalSecret │
 │ ├─ Agent CRD │
 │ └─ Deployment │
 └──────────────────┘

The repo structure:

k8s-iceman/
├── apps/
│ ├── kagent-examples.yaml # ArgoCD Application (recurse: true)
│ └── telegram-bot-src/ # Bot source code
│ ├── Dockerfile
│ ├── main.py
│ └── requirements.txt
├── manifests/
│ └── kagent-examples/
│ └── telegram-bot/ # Picked up by ArgoCD automatically
│ ├── 01-external-secret.yaml
│ ├── 02-agent.yaml
│ └── 03-deployment.yaml
└── helm-values/
 └── kagent/
 └── values.yaml # Model config (gpt-5.4)

Because the kagent-examples ArgoCD Application has directory.recurse: true, any new directory under manifests/kagent-examples/ is automatically synced. No new ArgoCD Application needed.

Gotchas and Lessons Learned

1. `contextId`, not `sessionId`

This was the biggest surprise. kagent’s A2A protocol uses contextId for session continuity, and it must go inside the message object — not in params, not as a top-level field. The A2A spec and other implementations sometimes mention sessionId — kagent ignores it completely. I spent hours debugging why every message started a fresh conversation before diving into the kagent source code and finding that contextId maps directly to kagent’s internal session system.

# WRONG — kagent ignores sessionId
payload = {
 "params": {
 "sessionId": "...", # This does nothing
 "message": {...}
 }
}

# RIGHT — contextId goes inside the message object
payload = {
 "params": {
 "message": {
 "contextId": "...", # This maintains the session
 "role": "user",
 "parts": [...]
 }
 }
}

2. `input-required` responses have no artifacts

When the agent needs approval (HITL), the response has status.state: "input-required" but no artifacts array. If you only parse artifacts, you’ll get “Agent returned no text response” for every approval request. The agent’s explanation text is in the history array (last agent message), and the approval data is in status.message.parts as a data kind part.

3. The trailing slash matters

The kagent A2A endpoint at /api/a2a/kagent/telegram-k8s-agent returns a 307 redirect to /api/a2a/kagent/telegram-k8s-agent/. The httpx library (and most HTTP clients) won’t follow redirects on POST requests by default — for good security reasons. Always include the trailing slash.

4. kagent uses `message/send`, not `tasks/send`

If you’re reading the A2A spec or looking at other A2A implementations, be aware that kagent uses message/send as the method name. The older tasks/send method returns a Method not found error.

5. Parts use `kind`, not `type`

The A2A spec uses "kind": "text" for message parts. Some implementations and docs use "type": "text". kagent expects kind.

6. Telegram message limits

Telegram enforces a 4096-character limit per message. If your agent returns a large response (like a full pod listing or verbose logs), you need to chunk it. The bot handles this automatically by splitting at 4000-character boundaries.

7. `Recreate` strategy, not `RollingUpdate`

Telegram’s long-polling API delivers each update to exactly one consumer. With two pods running during a rolling update, you’d get duplicate responses or missed messages. Recreate ensures a clean handoff.

Testing It

Once deployed, open Telegram and find your bot (the one you created with @BotFather):

/start — Shows available commands
/status — Checks connectivity to the kagent controller
/new — Resets your conversation session (clears contextId)
Send any message — It goes to the agent and comes back with a real answer

Example interactions:

You: What pods are running in kagent namespace?

Bot: Here are the running pods in the kagent namespace:

kagent-controller-57864fdf69-xfgmr (1/1 Running)

telegram-bot-5469669bf-v8h7p (1/1 Running)

telegram-k8s-agent-5f696bf4b9-pl7cl (1/1 Running)

k8s-agent-6dccd8ddd8-gxt6x (1/1 Running)

… (28 more pods)

You: Create a staging namespace

Bot: The agent wants to run: k8s_create_resource namespace: staging kind: Namespace

[ Approve ] [ Reject ]

You tap Approve

Bot: Namespace staging created successfully.

You: Now deploy nginx to that namespace

Bot: The agent wants to run: k8s_create_resource namespace: staging kind: Deployment name: nginx …

[ Approve ] [ Reject ]

Notice how the agent knows “that namespace” means staging — because the contextId maintained the conversation context across the approval interaction.

What’s Next

This is a foundation. From here you could:

Add more agents: Create a telegram-security-agent with Kubescape tools, or a telegram-istio-agent with mesh-specific tools
Route by command: Use different Telegram commands (/k8s, /istio, /security) to route to different kagent agents
Add image support: kagent supports multi-modal parts — you could send screenshots of dashboards and ask “what’s wrong here?”
Connect via agentgateway: Route the A2A traffic through agentgateway for rate limiting, authentication, and observability
Build other chat integrations: The A2A protocol is platform-agnostic. Swap python-telegram-bot for discord.py, slack-bolt, or even a WhatsApp integration and the A2A layer stays exactly the same

The pattern works for any chat platform. The A2A protocol is the universal adapter.

The full source code and Kubernetes manifests are in the k8s-iceman repo under apps/telegram-bot-src/ and manifests/kagent-examples/telegram-bot/.

How To: Run agentgateway standalone locally (UI + basic smoke test)

Thu, 12 Mar 2026 08:04:00 -0400

How To: Run agentgateway standalone locally (UI + basic smoke test)

This is a quick-start for agentgateway v1.0.0-alpha.4 in standalone mode.

Upstream repo: https://github.com/agentgateway/agentgateway Latest alpha release notes: https://github.com/agentgateway/agentgateway/releases/tag/v1.0.0-alpha.4

Option A — Docker (fastest)

The project publishes a container image:

cr.agentgateway.dev/agentgateway:v1.0.0-alpha.4

Example:

docker run --rm -p 15000:15000 \
 cr.agentgateway.dev/agentgateway:v1.0.0-alpha.4

Then open:

UI: http://localhost:15000/ui

Option B — Binary

The release also publishes binaries (see the GitHub release assets). Download the one for your OS/arch, then run it:

./agentgateway

Then open:

UI: http://localhost:15000/ui

What to check first

Does the UI load at /ui?
Are there obvious errors in logs on startup?
Can you see config/state updating when you apply changes?

Next step: one real use case

Tell me which “first win” you want and I’ll build the smallest working example:

Proxy OpenAI through agentgateway
Route + observe MCP tool traffic
A2A connectivity demo

How To: Install agentgateway on Kubernetes (Helm OCI)

Thu, 12 Mar 2026 08:02:00 -0400

How To: Install agentgateway on Kubernetes (Helm OCI)

This is a “get it running fast” install for agentgateway v1.0.0-alpha.4 using the OCI Helm charts published by the project.

Upstream release notes (artifacts list): https://github.com/agentgateway/agentgateway/releases/tag/v1.0.0-alpha.4

Note: exact values and defaults can change between alphas; treat this as a baseline and validate in your cluster.

Prereqs

Kubernetes cluster
kubectl
Helm v3 with OCI support

1) Pick a namespace

export AGW_NS=agentgateway-system
kubectl create namespace $AGW_NS || true

2) Install CRDs

export AGW_VER=v1.0.0-alpha.4

helm install agentgateway-crds oci://cr.agentgateway.dev/charts/agentgateway-crds \
 --version $AGW_VER \
 --namespace $AGW_NS

3) Install agentgateway

helm install agentgateway oci://cr.agentgateway.dev/charts/agentgateway \
 --version $AGW_VER \
 --namespace $AGW_NS

4) Verify pods

kubectl get pods -n $AGW_NS
kubectl get svc -n $AGW_NS

5) Next: configure your first route/tool access

At this point agentgateway is installed; the next step is to define how traffic flows (MCP servers, policies, routes, etc.).

If you tell me your target (OpenAI proxying? MCP multiplexing? A2A?), I’ll write the exact minimal config example as a follow-up.

agentgateway v1.0 (alpha): Why the 1.0 Line Matters (and What Changed)

Thu, 12 Mar 2026 08:00:00 -0400

agentgateway v1.0 (alpha): Why the 1.0 Line Matters (and What Changed)

If you’ve been tracking agentgateway, you probably noticed something big: it crossed into the v1.0.0 release line.

As of today, the latest published release is v1.0.0-alpha.4 — so this isn’t “GA 1.0” yet — but the 1.0 line itself is the story.

This post breaks down why v1.0 is significant, what it changes operationally, and how you should think about upgrading if you were previously consuming agentgateway via kgateway.

Quick context: what is agentgateway?

agentgateway is an open-source data plane optimized for agentic AI connectivity — agent-to-tool and agent-to-agent — with a focus on:

Security and governance around tool access (MCP) and agent interoperability (A2A)
Observability (including OpenTelemetry)
High-performance (Rust-based)
Run anywhere (standalone or Kubernetes)

Upstream repo: https://github.com/agentgateway/agentgateway

Why the v1.0 line is significant

1) Decoupling from kgateway (versioning + lifecycle)

The most important “why 1.0” detail is not a single feature — it’s project independence.

From the v1.0.0-alpha.2 release notes:

agentgateway is now entirely decoupled from kgateway
Kubernetes deployment used to be delivered via kgateway and followed kgateway’s versioning scheme

What that fixes in practice:

You no longer have to explain two versions for “agentgateway on k8s” (controller vs dataplane)
Releases become unified under one repo with consistent artifacts

2) One release, one set of artifacts

The v1.0.0 alpha releases publish a consistent set of artifacts:

Docker images (controller + gateway)
Helm charts (agentgateway + agentgateway-crds)
Binaries

Example (alpha.4):

cr.agentgateway.dev/agentgateway:v1.0.0-alpha.4
cr.agentgateway.dev/controller:v1.0.0-alpha.4
cr.agentgateway.dev/charts/agentgateway:v1.0.0-alpha.4
cr.agentgateway.dev/charts/agentgateway-crds:v1.0.0-alpha.4

This is a very “1.0-adjacent” sign: the project is taking packaging/distribution seriously.

3) The project is signaling stability goals

Even though it’s still alpha, moving to v1 is a public signal that the project is:

drawing a line around scope
solidifying APIs (CRDs, config models)
standardizing install paths

In other words: it’s positioning itself to be something you can operationalize, not just demo.

Notable changes in the v1.0.0-alpha series (high-level)

From the v1.0.0-alpha.4 notes, a few changes that matter operationally:

OpenTelemetry support for access logs (great for real-world production debugging and cost attribution)
Gateway API alignment updates (example: TLSRoute v1 bump)
Release artifact / versioning cleanup (consistent v prefixes)

Release notes:

Upgrade / adoption guidance

If you’re new

Start with v1.0.0-alpha.4 and pick one track:

Standalone for local dev and labs
Kubernetes if you want multi-tenant, policy, and integration with existing cluster operations

If you came via kgateway

The key shift is that agentgateway now has its own release line, but it can still be used as a supported data plane within kgateway.

Practical recommendation:

treat v1.0 alpha as a migration window
stand up v1.0 alpha in parallel, validate config and policy behavior, then cut over

What’s next

In the next posts I’ll publish two quick how-tos:

“Install agentgateway on Kubernetes in 10 minutes”
“Run agentgateway standalone locally (and what to look at in the UI)”

If you want one single “do it all” tutorial instead (one post, end-to-end), that also works — it’s just longer.

What is agentgateway.dev? (And why it exists)

Thu, 12 Mar 2026 07:10:00 -0400

What is agentgateway.dev? (And why it exists)

If you’re building agentic AI systems, you eventually hit the same wall:

Your agents need tools (MCP servers, internal APIs, SaaS)
They need to talk to other agents (A2A)
They need to call LLM providers (often multiple)

…and suddenly “just use an API gateway” stops being a satisfying answer.

That’s what agentgateway.dev is about.

agentgateway.dev is the documentation hub for the open source agentgateway project — a connectivity data plane designed specifically for agent workloads.

Docs entry point (standalone): https://agentgateway.dev/docs/standalone/latest/

Upstream repo: https://github.com/agentgateway/agentgateway

Why a new gateway for agents?

The docs put it bluntly: traditional API gateways and reverse proxies aren’t built for MCP and A2A.

Why?

Traditional gateways assume:

stateless request/response
one request → one backend
client-initiated traffic only

But MCP/A2A introduces:

stateful JSON-RPC sessions with long-lived connections
fan-out across multiple tool servers
server-initiated events (SSE) that must route back to the correct client session
per-session authorization (different users/agents should see different tools)

In other words: the gateway needs to be protocol-aware and session-aware — not just path-aware.

What agentgateway actually is (in one sentence)

Agentgateway is a unified LLM + MCP + A2A gateway, built for:

enterprise-grade security
observability
resiliency
multi-tenancy

And it’s implemented in Rust for performance and memory safety (important for long-lived, high-concurrency sessions).

The three big product pillars

1) LLM Gateway

Agentgateway can route traffic to major LLM providers behind a unified OpenAI-compatible API, so you can switch providers without rewriting your application.

The docs also call out “native vs translation” support per provider/API, which is a real-world detail that matters when fields/models evolve quickly.

2) MCP Gateway

Agentgateway’s MCP features (from the docs):

Tool federation: aggregate multiple MCP servers behind a single endpoint
Multiple transports: stdio, HTTP/SSE, Streamable HTTP
OpenAPI integration: expose existing REST APIs as MCP tools
AuthN/AuthZ: MCP auth spec compliance + OAuth providers (Auth0, Keycloak)

This is where agentgateway becomes the “connective tissue” between LLMs and tools.

3) A2A Gateway

Agentgateway also supports the Agent-to-Agent (A2A) protocol so agents can:

discover each other’s capabilities
negotiate modalities (text/forms/media)
collaborate on long-running tasks

Security & observability (the part that turns demos into systems)

From the docs:

Authentication: JWT, API keys, basic auth, MCP auth spec
Authorization: fine-grained RBAC with the Cedar policy engine
Traffic policies: rate limiting, CORS, TLS, external authz
Observability: built-in OpenTelemetry metrics/logs/tracing

If you’re building agent systems you expect to run for months, this is the difference between:

“we tested it once”
and “we can operate it safely”

Why now (the timing)

The timing is interesting because multiple milestones are converging:

The repo is about 1 year old (created March 2025)
It’s nearing ~2k GitHub stars
The project is crossing into the v1.0 release line (currently in alpha)

This is usually the moment when a project shifts from “early adopter playground” to “this is becoming a platform.”

Next: what I’d build first

If you want a practical entry point, pick one:

LLM routing + observability (single endpoint, multiple providers)
MCP tool federation (one tool endpoint, many MCP servers)
Policy + RBAC (who can call which tools, under what conditions)

If you tell me which one you want to lead with, I’ll write a follow-up tutorial that’s copy/paste runnable.

Human-in-the-Loop: Building AI Agents You Can Actually Trust on Kubernetes

Wed, 11 Mar 2026 10:34:00 -0400

Human-in-the-Loop: Building AI Agents You Can Actually Trust on Kubernetes

By Sebastian Maniak

AI agents are getting good — really good — at taking autonomous action. They can inspect your Kubernetes cluster, diagnose problems, create resources, and even clean up after themselves. But here’s the uncomfortable question most teams are quietly asking:

“Do I actually want an AI agent deleting things in my production cluster without asking me first?”

The answer, almost universally, is no.

This is where Human-in-the-Loop (HITL) comes in. It’s the pattern that lets you give an AI agent real tools and real capabilities, while keeping a human in the decision loop for anything that matters. Not everything — just the operations where a mistake could ruin your afternoon.

In this post, I’ll walk through how HITL works in kagent, an open-source project for building Kubernetes-native AI agents. We’ll go from concept to a working agent that pauses, asks for approval, and respects your answer — all running on a local Kind cluster.

The Problem with Fully Autonomous Agents

Let’s set the scene. You build an AI agent that can manage Kubernetes resources. You give it tools to list pods, apply manifests, delete deployments. It works great in a demo.

Then someone on your team asks it to “clean up the old staging resources” and it deletes the wrong namespace. Or it misinterprets “scale down the database” and sets replicas to zero on your production PostgreSQL StatefulSet.

The issue isn’t that the agent is dumb — it’s that there was no checkpoint between “the agent decided to do something” and “the thing got done.” There was no moment where a human could look at the plan and say, “Wait, no. Not that.”

Fully autonomous agents are fine for read-only operations. For anything that changes state, you need a gate.

What Human-in-the-Loop Actually Means

HITL isn’t a single feature — it’s a design pattern with a few distinct mechanisms:

1. Tool Approval

The agent decides to call a tool (like deleting a resource), but instead of executing immediately, the system pauses and presents the action to a human. The human reviews it and either:

Approves — the tool executes normally
Rejects — the agent receives the rejection (with an optional reason) and adapts its approach

This is the core HITL mechanism. It’s simple, explicit, and gives you exactly the control you need.

2. Ask User

Sometimes the agent doesn’t need approval — it needs information. The request is ambiguous, or there are multiple valid paths, and the agent needs you to make a choice.

For example:

“Which namespace should I create this in?”
“Do you want to use PostgreSQL or MySQL?”
“Should I apply this to staging or production?”

The ask_user tool lets the agent pause execution, ask a question, and wait for your answer before continuing. It’s not a safety gate — it’s a collaboration mechanism.

3. The Combination

The real power is when both mechanisms work together. An agent that can:

Read your cluster freely (no interruptions)
Ask you for clarification when your request is vague
Show you exactly what it’s about to change and wait for your approval
Accept your rejection gracefully and try a different approach

That’s an agent you can actually put in front of a team.

How kagent Implements HITL

kagent is an open-source project for building AI agents that run natively on Kubernetes. Agents are defined as Custom Resources (CRDs), tools are served via MCP (Model Context Protocol) servers, and everything runs inside your cluster.

HITL in kagent works through two mechanisms, both built on the same underlying infrastructure:

The `requireApproval` Field

When you define an agent, you list the tools it can use. For any tool that should require human approval, you add it to the requireApproval list:

tools:
 - type: McpServer
 mcpServer:
 name: kagent-tool-server
 kind: RemoteMCPServer
 apiGroup: kagent.dev
 toolNames:
 - k8s_get_resources  # read-only, runs freely
 - k8s_describe_resource  # read-only, runs freely
 - k8s_apply_manifest  # destructive, needs approval
 - k8s_delete_resource  # destructive, needs approval
 requireApproval:
 - k8s_apply_manifest
 - k8s_delete_resource

That’s it. Two lines to add a safety gate. The tools in requireApproval must also appear in toolNames — the CRD validates this at creation time, so you can’t accidentally gate a tool that isn’t assigned to the agent.

The `ask_user` Tool

Every kagent agent automatically has access to ask_user. You don’t need to configure it — it’s built-in. When the agent encounters an ambiguous request, it can call ask_user to pause and ask the human a question.

The question can be:

Free-text — the user types their answer
Single-select — the user picks from a list of choices
Multi-select — the user picks multiple options

The agent gets back structured data, not just a raw string, which makes it easier to act on the answer.

Under the Hood

Both mechanisms share the same infrastructure:

The agent calls a tool that requires confirmation (either via requireApproval or ask_user)
The executor calls request_confirmation() and blocks
An event is generated and sent to the UI via A2A (Agent-to-Agent protocol)
The UI renders the approval controls or question
The human responds
Their response is sent back as an A2A message
The executor unblocks and the agent continues

HITL flow: the agent proposes actions, a human approves/edits/rejects, and results feed back into the agent context.

This shared architecture means HITL is not a bolted-on feature — it’s a core part of how kagent agents communicate. Adding approval to a new tool is a one-line YAML change, not a code change.

Building a HITL Agent: Step by Step

Let’s build one. We’ll create an agent with access to Kubernetes tools — read-only tools run freely, destructive tools require approval.

Prerequisites

Docker, Kind, kubectl, Helm installed
An OpenAI API key

1. Create a Cluster

kind create cluster --name kagent-hitl

2. Install kagent

# Install CRDs
helm install kagent-crds oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \
 --namespace kagent \
 --create-namespace

# Set your API key
export OPENAI_API_KEY="your-api-key-here"

# Install kagent
helm install kagent oci://ghcr.io/kagent-dev/kagent/helm/kagent \
 --namespace kagent \
 --set providers.default=openAI \
 --set providers.openAI.apiKey=$OPENAI_API_KEY

# Wait for pods
kubectl wait --for=condition=ready pod --all -n kagent --timeout=120s

3. Define the Agent

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
 name: hitl-agent
 namespace: kagent
spec:
 description: A Kubernetes agent with human-in-the-loop approval for destructive operations.
 type: Declarative
 declarative:
 modelConfig: default-model-config
 systemMessage: |
 You are a Kubernetes management agent. You help users inspect and manage
 resources in the cluster. Before making any changes, explain what you
 plan to do. If the user's request is ambiguous, use the ask_user tool
 to clarify before proceeding. 
 tools:
 - type: McpServer
 mcpServer:
 name: kagent-tool-server
 kind: RemoteMCPServer
 apiGroup: kagent.dev
 toolNames:
 - k8s_get_resources
 - k8s_describe_resource
 - k8s_get_pod_logs
 - k8s_get_events
 - k8s_get_resource_yaml
 - k8s_apply_manifest
 - k8s_delete_resource
 - k8s_patch_resource
 requireApproval:
 - k8s_apply_manifest
 - k8s_delete_resource
 - k8s_patch_resource

kubectl create -f hitl-agent.yaml

4. Open the UI

kubectl port-forward -n kagent svc/kagent-ui 8080:8080

Open http://localhost:8080.

Seeing It in Action

Scenario 1: Reading is Free

You ask the agent: “List all pods in the kagent namespace.”

The agent calls k8s_get_resources. Since this tool is not in requireApproval, it runs immediately. You see the results with no interruption.

This is the right UX. You don’t want to click “Approve” every time you ask the agent to read something. Only destructive operations should have gates.

Scenario 2: Writes Need Approval

You ask: “Create a ConfigMap called test-config in the default namespace with the key message set to ‘hello from kagent’.”

The agent calls k8s_apply_manifest. Since this tool is in requireApproval, execution pauses. In the UI, you see:

The tool the agent wants to call
The arguments it wants to pass (including the full YAML manifest)
Approve and Reject buttons

You review the manifest. It looks correct. You click Approve. The agent applies the ConfigMap and confirms success.

Scenario 3: Rejection with Reason

You ask: “Delete the ConfigMap test-config in the default namespace.”

The agent calls k8s_delete_resource. Execution pauses. But this time, you click Reject and enter a reason: “I want to keep this ConfigMap for now.”

The agent receives your rejection reason. Instead of failing or retrying, it acknowledges your decision and responds: “Understood — I’ll leave the ConfigMap in place. Let me know if you change your mind.”

This is a key detail. The rejection reason isn’t just logged — it’s sent back to the LLM as context. The agent can adapt its behavior based on why you said no.

Scenario 4: Agent Asks for Clarification

You ask: “Set up a namespace for my application.”

The agent doesn’t know what to call the namespace. Instead of guessing, it calls ask_user: “What should the namespace be called?”

You type “staging-app” and the agent continues, creating the namespace with the name you specified.

This is the difference between an agent that guesses wrong and an agent that gets it right. The cost of asking is a few seconds. The cost of guessing wrong is rolling back a mistake.

Design Principles

A few things about kagent’s HITL implementation that I think are worth highlighting:

1. Opt-In, Not Opt-Out

Tools run freely by default. You explicitly list which ones need approval. This means adding a new read-only tool to an agent doesn’t require updating any approval configuration — it just works.

2. Declarative, Not Procedural

You don’t write approval logic in code. You add a tool name to a YAML list. This keeps agent definitions readable and auditable. A security reviewer can look at the agent YAML and immediately see which tools require human approval.

3. Rejection Is Contextual

When you reject a tool call, you can provide a reason. That reason is returned to the LLM as part of the conversation context. This means the agent can learn from your rejection in the current conversation and adjust its approach.

4. ask_user Is Always Available

You don’t need to configure ask_user. It’s built-in on every agent. This encourages agent builders to write system prompts that tell the agent to ask when things are ambiguous, rather than assuming.

When to Use HITL (and When Not To)

Use HITL for:

Any operation that changes cluster state (create, update, delete)
Operations that are hard to reverse (deleting PVCs, removing finalizers)
Multi-step workflows where early decisions affect later steps
Any agent that will be used by people who aren’t the agent’s author

Skip HITL for:

Read-only operations (listing, describing, reading logs)
Internal agent reasoning (tool calls that don’t affect external state)
Fully automated pipelines where latency matters and rollback is cheap

The goal isn’t to make every action require approval — it’s to put gates where they matter.

Cleanup

kind delete cluster --name kagent-hitl

What’s Next

HITL is one piece of the puzzle. kagent also supports:

A2A protocol for agent-to-agent communication
MCP servers for extending agent capabilities with custom tools
Memory for agents that learn from past interactions
Skills for packaging reusable agent behaviors

If you want to try it yourself, the upstream Episode 03 material is here:

https://github.com/solo-io/allthings/tree/main/episode-03

kagent is open source: https://github.com/kagent-dev/kagent

This post is part of the “All Things” series exploring AI agents on Kubernetes.

Prompting After Feb 2026: Prompt Craft → Context → Intent → Specs

Fri, 27 Feb 2026 00:00:00 +0000

Most people mean “write better prompts” when they say prompt engineering.

That was a winning strategy in 2024–2025, when the dominant workflow was:

ask in chat → get an answer → iterate in real time

But as models become longer-running and more autonomous, the bottleneck shifts. You don’t get to babysit the session. You have to encode oversight up front.

This post is a practical distillation of a YouTube video that argues “prompting” is now hiding four different disciplines — and you need all four to get consistent, production-quality outcomes.

Source video: https://youtu.be/BpibZSMGtdY

The Four Disciplines of “Prompting” in 2026

1) Prompt craft (table stakes)

This is the classic skill:

Clear instruction
Examples + counterexamples
Guardrails (what to do / what not to do)
Explicit output format
Rules for ambiguity (“if X conflicts with Y, do Z”)

It still matters — it just doesn’t differentiate you anymore.

2) Context engineering (your real leverage)

Your prompt might be 200 tokens. Your context window might be 200k–1M.

That means your “prompt” is a rounding error.

Context engineering is the work of designing the information environment the agent runs inside:

system prompts / agent instructions
tool definitions + permissions
RAG sources / docs / repos
memory (what persists across runs)
conventions (how this org writes, builds, tests, ships)

If you see someone getting 10x more out of the same model, they usually aren’t “better at wording.” They built better context infrastructure.

3) Intent engineering (what the agent should want)

Context tells the agent what to know. Intent tells the agent what to optimize for.

This is where teams break things at enterprise scale:

speed vs quality
cost vs correctness
customer satisfaction vs ticket closure time
“ship it” vs “fail safe”

If you don’t encode trade-offs and escalation triggers, the agent will “pick a metric” implicitly.

4) Specification engineering (blueprints for autonomous work)

Specs are what you write when you can’t rely on real-time correction.

A good spec is:

self-contained
structured
internally consistent
explicit about quality measurement

If your output keeps coming back “80% correct,” you don’t have a prompt problem. You have a spec + evaluation problem.

What “specification engineering” actually means

The video’s argument is simple: once agents can run for hours or days, you can’t rely on live back-and-forth to fix mistakes.

So specification engineering becomes the skill of writing agent-executable blueprints:

Self-contained: no hidden assumptions, undefined acronyms, or missing “obvious” org context.
Verifiable: “done” is defined in checks that someone else can evaluate.
Constrained: must/must-not/preferences are explicit (plus what should be escalated).
Decomposable: the work can be broken into chunks that can be executed and verified independently.

The broader implication: your documents become infrastructure. Strategy docs, product docs, runbooks, and OKRs all start acting like specs that agents (and humans) can execute.

The five spec primitives (a useful mental model)

Self-contained problem statement
Acceptance criteria (verifiable “done”)
Constraint architecture (must / must-not / preferences / escalate if)
Decomposition (small, independently testable chunks)
Eval design (test cases that catch regressions after model updates)

The 2026 Prompting Checklist (copy/paste)

Use this before you hand a long-running task to an agent.

Objective: what outcome do you want, and why?
Success metric: how will we measure success?
Inputs: links/docs/data sources (authoritative only)
Definitions: terms/acronyms the agent might misread
Deliverables: files/sections/artifacts + exact output format
Acceptance criteria: verifiable checks (someone else can evaluate)
Constraints:
- Must
- Must not
- Preferences
- Escalate if
Plan-first: ask for a plan + checkpoints before execution
Decomposition: tasks broken into verifiable sub-steps
Eval: test cases / verification steps
Progress log: so the next session doesn’t start blind

Templates (use these as your default “prompt format”)

Template A — Self-contained spec (general)

## Objective
- What to do:
- Why it matters:
- Audience:

## Inputs (authoritative)
- Links/docs:
- Definitions/glossary:

## Deliverables
- D1:
- D2:

## Acceptance criteria
- [ ]
- [ ]

## Constraints
- Must:
- Must not:
- Preferences:
- Escalate if:

## Workflow
1) Propose plan + checkpoints.
2) Wait for confirmation.
3) Execute.
4) Provide final output + verification notes.

Template B — Coding task spec

## Repo context
- Repo root:
- Build commands:
- Test commands:
- Conventions (style, lint, naming, etc.):

## Change request
- Feature/bug:
- Non-goals:

## Constraints
- Must not break:
- Security/perf constraints:
- Escalate if:

## Acceptance criteria
- Tests added/updated:
- Commands that must pass:
- Manual validation steps:

## Rollout
- Backwards compatibility:
- Migration steps:

Template C — Research task spec

## Question

## Constraints
- Time horizon:
- Must cite primary sources:
- Output format:

## Deliverables
- Executive summary (5 bullets)
- Findings (with links)
- Risks/unknowns
- Recommendations

## Evaluation
- What would make this wrong?
- How can we verify quickly?

Failure Modes I See Constantly (and how to fix them)

“It’s 80% right but takes forever to clean up.”
- Fix: acceptance criteria + explicit output format + examples/counterexamples.
“The agent drifted after 30 minutes.”
- Fix: constraints + escalation triggers + checkpoints + progress log.
“We loaded everything and quality got worse.”
- Fix: curate context; summarize; move stable conventions into a short, high-signal rule file.
“It optimized for the wrong thing.”
- Fix: intent engineering — explicitly state trade-offs and priorities.
“We can’t tell if outputs are good.”
- Fix: eval design — build test cases; rerun after model updates; track regressions.

What to Do This Week

If you want an easy, practical on-ramp:

Pick one recurring task (deck creation, incident writeups, migration plans, content outlines).
Write a self-contained spec using Template A.
Define acceptance criteria that another person could verify.
Create 3–5 eval cases (known-good examples).
Save it as your baseline prompt/spec and iterate as models change.

That’s the shift: from “chat tricks” to repeatable systems.

Configuring Local Qwen3 Embeddings for OpenClaw Memory

Mon, 23 Feb 2026 15:19:00 -0500

Configuring Local Qwen3 Embeddings for OpenClaw’s Memory System

OpenClaw’s memory_search and memory_get tools enable semantic recall from MEMORY.md, memory/*.md, and session transcripts before answering about prior work/decisions. Defaults use cloud embeddings, but I’ve switched to a local Qwen3-Embedding-0.6B (GGUF quantized) for privacy, speed, and zero cost.

Why Qwen3-Embedding-0.6B?

Sweet Spot: 0.6B params—20-100ms/chunk on CPU (10x faster than 7B).
Quality: Tops MTEB benchmarks for retrieval, beats E5-small.
GGUF: Q8_0 (~400MB, near-lossless) via llama.cpp.
Hybrid Mode: CPU + auto-GPU (CUDA/Metal/ROCm).

Model Specs

Repo: Qwen/Qwen3-Embedding-0.6B-GGUF
File: Qwen3-Embedding-0.6B-Q8_0.gguf
Dims: 1024 (semantic search optimized)
Provider: local (llama.cpp backend)

Setup (5 Mins)

Download:

mkdir -p ~/.openclaw/models/embeddings
cd ~/.openclaw/models/embeddings
huggingface-cli download Qwen/Qwen3-Embedding-0.6B-GGUF Qwen3-Embedding-0.6B-Q8_0.gguf

Config (~/.openclaw/config.yaml or CLI):
```
agents:
 defaults:
 embeddingModel: "hf:Qwen/Qwen3-Embedding-0.6B-GGUF/Qwen3-Embedding-0.6B-Q8_0.gguf?provider=local&hybrid=true&cpu_threads=8"
```
Params:

Key Desc

provider=local llama.cpp

hybrid=true GPU auto

cpu_threads=8 Cores

CLI: openclaw configure --section agents.defaults.embeddingModel '...'
Restart: openclaw gateway restart

Test:

memory_search query="test prior decision" # ~80ms top-5

Benchmarks (i7-12700K, 32GB, No GPU)

Task	Time	RAM
Embed 512t chunk	45ms	350MB
Search 100 docs	80ms	<500MB
100 queries/min	✅

GPU (RTX 3060): 15ms/embed.

How It Works

Chunk: Files → 512t snippets (w/ lines).
Embed Query: Qwen3 → vector.
SimSearch: Cosine top-k (minScore filter).
Cite: path#line for memory_get.

On-demand (snippet cache).

Tweaks & Troubleshooting

Faster: Q4_K_M.gguf.
GPU?: gpu=true.
OOM: context_length=2048.
Logs: journalctl -u openclaw-gateway | grep embedding.
Verify: session_status → embedding provider.

Alternatives

Model	Params	Speed	Use Case
Qwen3-0.6B-Q8	0.6B	45ms	General
NV-Embed-4B-Q5	4B	150ms	Code
all-minilm-L6	22M	10ms	Light

Pro Tips

Pre-cache: JSONL embeddings via cron.
Scale: memory/ subdirs.
Monitor: Query hit rates.

Local RAG + Claude = unbeatable. Questions?

Docs | Qwen

Multi-Agent Architecture with a Kill Switch: Why Every AI Agent Needs a Gateway

Sat, 21 Feb 2026 00:00:00 +0000

The Setup

I run a multi-agent system. One coordinator agent handles user interaction, memory, and routing. Specialist sub-agents get spawned on demand for domain-specific tasks — security audits, network diagnostics, cloud management, infrastructure automation. Each specialist has its own system prompt, its own toolset, and runs on a different model.

It works. The specialists are good at their jobs. The coordinator knows when to delegate and when to handle things itself.

But here’s what keeps me up at night: what happens when one of these agents goes rogue?

A security agent with access to nmap and trivy decides to scan every host on the network in a loop. A cloud agent burns through $500 of Opus tokens chasing a hallucinated Terraform state or decides to reconfigure your Istio ambient mesh routing because it misread a waypoint proxy status. A general agent with SSH access starts “fixing” things on production hosts that don’t need fixing.

Without a control plane between your agents and the outside world, you have no way to stop any of this. No kill switch. No cost ceiling. No audit trail. No rate limits. Just agents with direct access to LLMs and tools, hoping nothing goes wrong.

That’s not engineering. That’s negligence.

The Architecture

Here’s what I actually run. Every LLM call and every MCP tool invocation from every agent — coordinator and specialists alike — routes through agentgateway.

┌─────────────────────────────────────────────────┐
│ User (Seb) │
│ Telegram / Discord / CLI │
└────────────────────┬────────────────────────────┘
 │
 ▼
┌─────────────────────────────────────────────────┐
│ Coordinator Agent │
│ (Jacob) │
│ │
│ • User interaction & conversation │
│ • Memory management (MEMORY.md) │
│ • Task triage & routing │
│ • Context assembly for specialists │
│ • Result synthesis & delivery │
└──┬──────────┬──────────┬──────────┬─────────────┘
 │ │ │ │
 ▼ ▼ ▼ ▼
┌──────┐ ┌──────┐ ┌──────┐ ┌──────────┐
│ Sec │ │ Net │ │Cloud │ │ General │
│Agent │ │Agent │ │Agent │ │ Agent │
└──┬───┘ └──┬───┘ └──┬───┘ └────┬─────┘
 │ │ │ │
 └─────────┴─────────┴────────────┘
 │
 │ ALL traffic
 ▼
 ┌──────────────────────────────────────────┐
 │ Kubernetes Cluster │
 │ ┌──────────────────────────────┐ │
 │ │ agentgateway │ │
 │ │ (Pod) │ │
 │ │ │ │
 │ │ • Kill switch │ │
 │ │ • Rate limiting │ │
 │ │ • Cost controls │ │
 │ │ • JWT auth + RBAC │ │
 │ │ • Observability (OTel) │ │
 │ │ • Tool poisoning protection │ │
 │ └──────┬───────────┬───────────┘ │
 │ │ │ │
 └─────────┼───────────┼────────────────────┘
 │ │
 ┌────▼────┐ ┌───▼────────────────┐
 │ LLMs │ │ MCP Servers │
 │Anthropic│ │ nmap, trivy │
 │ OpenAI │ │ aws-cli, kubectl │
 │ xAI │ │ istioctl, docker │
 └─────────┘ └───────────────────-┘

Nothing reaches an LLM or a tool without passing through the gateway. That’s the entire point.

The Agents

Coordinator: Jacob

The coordinator is the only agent that talks to the user. It owns the conversation, manages memory (a persistent MEMORY.md that carries context across sessions), and decides which specialist to invoke for each request.

When a task comes in, the coordinator classifies it and builds a context payload — the relevant portion of memory, the specific question, any constraints — and spawns a specialist. The specialist does its work, returns a result, and dies. Stateless. Disposable.

The coordinator synthesizes the result and delivers it back to the user. If a task spans multiple domains, the coordinator fans out to multiple specialists in parallel.

Model: Sonnet — fast enough for routing, smart enough for context assembly.

Security Agent

Domain: Vulnerability scanning, CVE analysis, firewall rules, IAM audits, compliance checks
Tools: nmap, trivy, falco, OWASP ZAP, CIS benchmarks, secrets scanning
Model: Opus — high reasoning for threat analysis
Access: Read-only on infra by default, escalation required for remediation

Network Agent

Domain: DNS, routing, load balancing, VPN, firewall config, traffic analysis
Tools: dig, traceroute, tcpdump, iperf3, netstat, ip, iptables, tshark
Model: Sonnet — fast, good for diagnostic tasks
Access: Network interfaces, DNS servers, routing tables

Cloud Agent

Domain: AWS/GCP/Azure resource management, Terraform, cost optimization, architecture, Kubernetes, Istio service mesh, ambient mesh
Tools: aws-cli, gcloud, az, terraform, kubectl, helm, istioctl
Model: Sonnet — balance of speed and capability
Access: Cloud provider credentials (scoped IAM roles), Kubernetes clusters, Istio control plane

General / Infra Agent

Domain: Proxmox, Docker, Linux admin, Git, CI/CD, general automation
Tools: ssh, docker, git, systemctl, proxmox API, cron
Model: Sonnet (routine ops) or Haiku (simple tasks)
Access: Full local system, Proxmox API, SSH to hosts

Routing Logic

The coordinator classifies each request and routes to the appropriate specialist:

Keywords	Routes To
CVE, vulnerability, audit, compliance, secrets	Security Agent
DNS, firewall, routing, VPN, latency, ports	Network Agent
AWS, Terraform, GCP, Azure, S3, EC2, cost, Istio, mesh, Kubernetes, k8s	Cloud Agent
VM, Docker, git, systemd, Proxmox, backup	General Agent

Ambiguous requests stay with the coordinator. Multi-domain tasks fan out to multiple specialists in parallel.

Why Every Agent Goes Through the Gateway

This is the part that matters. Here’s why I don’t let any agent — not even the coordinator — talk to LLMs or tools directly.

The Doom Scenario

Picture this: your cloud agent is debugging a Terraform plan. It calls Opus to reason about a complex state migration. The model hallucinates a resource dependency. The agent re-plans, calls the model again for clarification, gets another hallucination, retries with more context (bigger prompt, more tokens), and enters a loop. Each iteration costs more than the last because the context window keeps growing.

Without a gateway: you find out when the invoice arrives. $2,000 spent on a conversation with itself.

With agentgateway: the agent hits a token-per-minute ceiling after the third iteration. The request is rejected. You get an alert. You investigate. Total damage: $12.

That’s not a hypothetical. That’s Tuesday.

Kill Switch

agentgateway gives me a single point where I can shut everything down. If I see an agent misbehaving — through the metrics, through the traces, through an alert — I can:

Revoke the JWT for that specific agent’s identity. Immediate. That agent can’t make another LLM call or tool invocation.
Update the rate limit to zero for that agent class. Every security agent stops. Every cloud agent stops. Surgical.
Pull the gateway entirely. Nuclear option. Everything stops. Nothing reaches any LLM or tool.

Without a gateway, killing a rogue agent means finding the pod, kubectl exec-ing into the right node, and hoping you’re faster than the agent. With a gateway running in Kubernetes, it’s a config change — or a kubectl rollout restart away from a full reset.

Cost Controls

Every agent has a budget. Not a suggestion — a hard limit enforced at the gateway level.

# Security agent route — Opus workloads
policies:
 localRateLimit:
 - maxTokens: 50000
 tokensPerFill: 50000
 fillInterval: 1m
 type: tokens
 - maxTokens: 20
 tokensPerFill: 20
 fillInterval: 1m
 type: requests

# Cloud agent route — higher throughput
policies:
 localRateLimit:
 - maxTokens: 100000
 tokensPerFill: 100000
 fillInterval: 1m
 type: tokens
 - maxTokens: 30
 tokensPerFill: 30
 fillInterval: 1m
 type: requests

# General agent route — simple ops
policies:
 localRateLimit:
 - maxTokens: 20000
 tokensPerFill: 20000
 fillInterval: 1m
 type: tokens
 - maxTokens: 15
 tokensPerFill: 15
 fillInterval: 1m
 type: requests

Each route gets a token-bucket rate limit scoped by the route’s identity. The security agent running Opus gets 50k tokens per minute. That’s enough for serious threat analysis but not enough to bankrupt me on a hallucination loop. The general agent on Haiku gets 20k — simple ops don’t need more.

agentgateway tracks token usage per provider and per model with agentgateway_gen_ai_client_token_usage metrics, tagged with provider, model, and operation labels. I know exactly what each agent costs, in real time.

Rate Limiting

Rate limits aren’t just about cost. They’re about preventing an agent from overwhelming a downstream system.

A network agent running nmap scans through an MCP tool server could, in theory, scan your entire /16 network if nobody stops it. Rate limiting at the gateway means the agent gets N tool calls per minute, period. It can’t outrun the limit no matter how convinced it is that it needs to scan “just one more subnet.”

Same for LLM calls. An agent that retries on every 429 or timeout — something LLM providers actually rate-limit you for — gets its retries throttled at the gateway before the provider even sees them.

Governance and RBAC

Each agent has a JWT identity with scoped permissions. The security agent can call nmap and trivy tools but cannot call terraform apply. The cloud agent can call terraform plan but not ssh. The general agent can SSH to designated hosts but cannot touch cloud credentials.

This is enforced at the gateway with CEL expressions in mcpAuthorization rules:

# Security agent backend
mcpAuthorization:
 rules:
 - >-
 jwt.agent_role == "security" && (
 mcp.tool.name.startsWith("nmap") ||
 mcp.tool.name.startsWith("trivy") ||
 mcp.tool.name.startsWith("falco")
 ) 

# Cloud agent backend
mcpAuthorization:
 rules:
 - >-
 jwt.agent_role == "cloud" && (
 mcp.tool.name.startsWith("terraform") ||
 mcp.tool.name.startsWith("kubectl") ||
 mcp.tool.name.startsWith("istioctl") ||
 mcp.tool.name.startsWith("helm")
 )

Even if a specialist agent’s system prompt gets jailbroken and it tries to invoke tools outside its domain, the gateway blocks it. If a tool isn’t matched by a rule, it’s automatically filtered from the tools/list response — the agent literally cannot see tools it doesn’t have access to.

And since unmatched tools are denied by default, I only whitelist what’s explicitly allowed. Any tool not covered by an mcpAuthorization rule is invisible to the agent. For HTTP-level operations, I add explicit deny rules:

authorization:
 rules:
 - deny: 'request.path.contains("delete")'
 - deny: 'request.path.contains("destroy")'
 - deny: 'request.path.contains("drop")'

No agent gets to run destructive operations without explicit human escalation. Period.

Full Observability

Every LLM call and every tool invocation generates OpenTelemetry traces. Every trace is tagged with the agent identity that triggered it.

I can see:

Which agent made the call
What prompt was sent to the LLM
What tool was invoked with what arguments
How many tokens were consumed
How long it took
Whether it succeeded or failed

┌─ Trace: security-agent-cve-scan ──────────────────┐
│ │
│ initialize 12ms mcp-session-setup │
│ list_tools 8ms tool-discovery │
│ call_tool(nmap) 4.2s scan-target-host │
│ llm_call(opus) 3.1s analyze-scan-results │
│ call_tool(trivy) 6.8s container-vuln-scan │
│ llm_call(opus) 2.4s synthesize-findings │
│ │
│ Total: 16.5s | Tokens: 12,847 | Cost: $0.38 │
└─────────────────────────────────────────────────────┘

Metrics go to Prometheus. Traces go to Jaeger. LLM-specific telemetry goes to Langfuse for prompt/completion pair analysis. All of it through agentgateway’s built-in OpenTelemetry support — no instrumentation code in the agents themselves.

When something goes wrong, I don’t grep through logs hoping to find what happened. I open a dashboard and see exactly which agent, which call, which tool, at what time, with what parameters.

Design Decisions

Specialists are stateless, spawned per task. Simple and cost-effective. No long-running agent processes consuming resources while idle. The coordinator is the only persistent component.

Coordinator owns all memory. Specialists get context injected per request. They don’t need to remember previous conversations — the coordinator handles continuity.

Model per agent. Opus for security (high-stakes reasoning). Sonnet for network/cloud (speed + capability balance). Haiku for simple ops (cost efficiency). Each agent gets the cheapest model that’s good enough for its domain.

Tool isolation. Each specialist only gets the tools it needs. Not through prompt instructions (which can be jailbroken) but through gateway-enforced RBAC (which can’t).

Single gateway for all traffic. Not one gateway per agent. Not a sidecar pattern. One agentgateway instance running in Kubernetes that every agent routes through. One place to set policy, one place to monitor, one place to kill. K8s gives me rolling updates, health checks, and resource limits on the gateway itself — so the control plane has its own control plane.

Extensible. New domain = new agent config + system prompt + tool set. The coordinator’s routing logic gets a new keyword match. The gateway gets a new JWT scope. No architectural changes needed.

Running agentgateway in Kubernetes

agentgateway runs as a deployment in my Kubernetes cluster. This isn’t just convenience — it’s operational discipline. The gateway that controls all my agents is itself managed by K8s primitives: health checks, resource limits, rolling updates, and restart policies.

apiVersion: apps/v1
kind: Deployment
metadata:
 name: agentgateway
 namespace: agent-infra
spec:
 replicas: 1
 selector:
 matchLabels:
 app: agentgateway
 template:
 metadata:
 labels:
 app: agentgateway
 spec:
 containers:
 - name: agentgateway
 image: ghcr.io/agentgateway/agentgateway:latest
 ports:
 - containerPort: 3000
 name: proxy
 - containerPort: 15000
 name: admin
 - containerPort: 15020
 name: metrics
 resources:
 requests:
 memory: "128Mi"
 cpu: "100m"
 limits:
 memory: "512Mi"
 cpu: "500m"
 readinessProbe:
 httpGet:
 path: /healthz/ready
 port: 15020
 initialDelaySeconds: 5
 periodSeconds: 10
 volumeMounts:
 - name: config
 mountPath: /etc/agentgateway
 volumes:
 - name: config
 configMap:
 name: agentgateway-config
---
apiVersion: v1
kind: Service
metadata:
 name: agentgateway
 namespace: agent-infra
spec:
 selector:
 app: agentgateway
 ports:
 - name: proxy
 port: 3000
 targetPort: 3000
 - name: admin
 port: 15000
 targetPort: 15000
 - name: metrics
 port: 15020
 targetPort: 15020

The kill switch becomes even simpler in K8s. Scale to zero replicas and every agent loses its gateway instantly:

kubectl scale deployment agentgateway -n agent-infra --replicas=0

Everything stops. Scale back up when you’ve fixed the issue. The gateway comes back with the same config, same policies, same state.

The cloud agent — the one that handles Kubernetes, Istio, and ambient mesh — is particularly interesting in this setup. It manages the same cluster that hosts the gateway. That’s a circular dependency I’ve thought carefully about: the agent that manages K8s infrastructure talks through a gateway that runs on K8s infrastructure. The circuit breaker here is the RBAC policy — the cloud agent’s JWT scope explicitly excludes the agent-infra namespace. It can manage workloads, configure Istio routing, and deploy ambient mesh policies, but it cannot touch the gateway deployment itself.

Why agentgateway and Not a Traditional Proxy

Traditional API gateways (Envoy, Kong, NGINX) were built for HTTP request/response. AI agent traffic is fundamentally different:

MCP is stateful. Agents maintain long-lived sessions with tool servers. Requests and responses are tied to session context. Traditional gateways don’t maintain session awareness.
LLM calls are long-running. A single inference call can take 30+ seconds with streaming. Connection timeouts designed for web APIs don’t apply.
Token-based economics. Cost isn’t about request count — it’s about token count. A gateway that can’t count tokens can’t enforce budgets.
Bidirectional communication. MCP servers can push messages back to clients asynchronously. This breaks the request/response model traditional gateways assume.

agentgateway is purpose-built for this. Written in Rust for performance and memory safety on stateful, long-lived connections. Understands MCP sessions natively. Counts tokens per-provider. Handles fan-out patterns where one agent call becomes multiple downstream requests.

It’s open source, Apache 2.0 licensed, and part of the Linux Foundation. No vendor lock-in.

The Takeaway

A multi-agent system without a control plane is a liability. Every agent you deploy is a potential cost bomb, a potential security breach, a potential “I can’t believe nobody caught that” incident.

The architecture is straightforward:

One coordinator that handles users and routes tasks
Specialist agents that are stateless, scoped, and disposable
One gateway that sees everything, controls everything, and logs everything

The coordinator decides what gets done. The gateway decides whether it’s allowed to happen. That separation is what makes the system safe to run autonomously.

agentgateway isn’t optional in this architecture. It’s the thing that makes the entire system possible without me staring at a terminal 24/7 wondering if an agent is about to do something catastrophic.

Build the agents. Put the gateway in front. Sleep at night.

Resources:

MCP Multiplexing with agentgateway

Fri, 20 Feb 2026 00:00:00 +0000

Introduction

As your agentic AI environment grows, you end up with a sprawl of MCP servers — one for time utilities, one for general-purpose tools, one for Slack, another for GitHub. Each server exposes different tools, runs on a different port, and needs its own connection configuration.

Your MCP clients (Cursor, Claude Desktop, VS Code) shouldn’t need to know about every server individually. They should connect to one endpoint and see all available tools.

This guide walks you through MCP Multiplexing — federating multiple MCP servers behind a single agentgateway endpoint.

We’ll deploy two MCP servers (mcp-server-everything for utility tools and mcp-website-fetcher for fetching web content), multiplex them behind agentgateway, and connect from any IDE.

What You Get

One endpoint, many servers: Clients connect to /mcp and see tools from all federated servers
Automatic tool namespacing: Tools are prefixed with the server name to avoid conflicts
Label-based federation: Add servers by just adding a Kubernetes label — no config changes
IDE-agnostic: Works with Cursor, VS Code, Claude Code, Windsurf, OpenCode

Architecture

┌──────────────┐
│ Cursor │──┐
├──────────────┤ │
│ VS Code │──┤
├──────────────┤ │ ┌──────────────────┐ ┌──────────────────────┐
│ Claude Code │──┼─MCP──▶│ agentgateway │──────▶│ mcp-server-everything│
├──────────────┤ │ │ /mcp endpoint │ │ (echo, add, etc.) │
│ Windsurf │──┘ │ │──────▶├──────────────────────┤
└──────────────┘ │ Multiplexing │ │ mcp-website-fetcher │
 └──────────────────┘ │ (fetch web content) │
 └──────────────────────┘

Prerequisites

Docker installed and running
kind installed
kubectl and helm installed
At least one MCP client: Cursor, VS Code, Claude Code, Windsurf, or OpenCode

Step 1: Create a Kind Cluster

kind create cluster --name agentgateway-mcp

Verify it’s running:

kubectl get nodes

Step 2: Install agentgateway OSS

Deploy the Kubernetes Gateway API CRDs:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml

Install the agentgateway CRDs and control plane:

helm upgrade -i --create-namespace \
 --namespace agentgateway-system \
 --version v2.2.1 \
 agentgateway-crds oci://ghcr.io/kgateway-dev/charts/agentgateway-crds

helm upgrade -i -n agentgateway-system \
 agentgateway oci://ghcr.io/kgateway-dev/charts/agentgateway \
 --version v2.2.1

Verify the control plane is running:

kubectl get pods -n agentgateway-system

You should see the agentgateway control plane pod in Running state.

Step 3: Create an agentgateway Proxy

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: mcp-gateway
 namespace: agentgateway-system
spec:
 gatewayClassName: agentgateway
 listeners:
 - name: http
 port: 80
 protocol: HTTP
EOF

Wait for the proxy pod:

kubectl rollout status deploy/mcp-gateway -n agentgateway-system --timeout=60s

Step 4: Deploy Two MCP Servers

Server 1: mcp-server-everything

This is the reference MCP test server from the Model Context Protocol project. It provides utility tools like echo, add, longRunningOperation, and more. The streamableHttp argument tells it to listen over HTTP instead of stdio — which is exactly the transport agentgateway’s multiplex feature requires.

kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
 name: mcp-server-everything
 namespace: agentgateway-system
 labels:
 app: mcp-server-everything
spec:
 replicas: 1
 selector:
 matchLabels:
 app: mcp-server-everything
 template:
 metadata:
 labels:
 app: mcp-server-everything
 spec:
 containers:
 - name: mcp-server-everything
 image: node:20-alpine
 command: ["npx"]
 args: ["-y", "@modelcontextprotocol/server-everything", "streamableHttp"]
 ports:
 - containerPort: 3001
---
apiVersion: v1
kind: Service
metadata:
 name: mcp-server-everything
 namespace: agentgateway-system
 labels:
 app: mcp-server-everything
 mcp-federation: "true"
spec:
 selector:
 app: mcp-server-everything
 ports:
 - protocol: TCP
 port: 3001
 targetPort: 3001
 appProtocol: kgateway.dev/mcp
 type: ClusterIP
EOF

Server 2: mcp-website-fetcher

This server provides a fetch tool that can retrieve and extract content from any URL — useful for giving your AI assistant web browsing capabilities. It’s already HTTP-native, no wrapper needed.

kubectl apply -f- <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
 name: mcp-website-fetcher-fix
 namespace: agentgateway-system
data:
 app.py: |
 import httpx
 from mcp.server.fastmcp import FastMCP
 mcp = FastMCP(
 "mcp-website-fetcher",
 host="0.0.0.0",
 port=8000,
 streamable_http_path="/mcp",
 stateless_http=True,
 )
 @mcp.tool()
 async def fetch(url: str) -> str:
 """Fetches a website and returns its content"""
 headers = {"User-Agent": "MCP Test Server"}
 async with httpx.AsyncClient(follow_redirects=True, headers=headers) as client:
 response = await client.get(url)
 response.raise_for_status()
 return response.text
 if __name__ == "__main__":
 mcp.run(transport="streamable-http")
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: mcp-website-fetcher
 namespace: agentgateway-system
spec:
 selector:
 matchLabels:
 app: mcp-website-fetcher
 template:
 metadata:
 labels:
 app: mcp-website-fetcher
 spec:
 containers:
 - name: mcp-website-fetcher
 image: ghcr.io/peterj/mcp-website-fetcher:main
 imagePullPolicy: Always
 command: ["python3", "/app/app.py"]
 volumeMounts:
 - name: fix
 mountPath: /app/app.py
 subPath: app.py
 volumes:
 - name: fix
 configMap:
 name: mcp-website-fetcher-fix
---
apiVersion: v1
kind: Service
metadata:
 name: mcp-website-fetcher
 namespace: agentgateway-system
 labels:
 app: mcp-website-fetcher
 mcp-federation: "true"
spec:
 selector:
 app: mcp-website-fetcher
 ports:
 - port: 80
 targetPort: 8000
 appProtocol: kgateway.dev/mcp
 type: ClusterIP
EOF

Wait for both servers to be ready:

kubectl rollout status deploy/mcp-server-everything -n agentgateway-system --timeout=60s
kubectl rollout status deploy/mcp-website-fetcher -n agentgateway-system --timeout=60s

Key detail: Both services have the label mcp-federation: "true" and appProtocol: kgateway.dev/mcp. This is how agentgateway discovers and federates them.

Step 5: Create the Multiplexed MCP Backend

Here’s where the magic happens. Instead of pointing to individual servers, you use a label selector to match all services with mcp-federation: "true":

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: mcp-federated
 namespace: agentgateway-system
spec:
 mcp:
 targets:
 - name: mcp-servers
 selector:
 services:
 matchLabels:
 mcp-federation: "true"
EOF

This single backend automatically discovers both mcp-server-everything and mcp-website-fetcher — and any future MCP server you deploy with the mcp-federation: "true" label.

Step 6: Create the HTTPRoute

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: mcp-route
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: mcp-gateway
 namespace: agentgateway-system
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /mcp
 backendRefs:
 - name: mcp-federated
 group: agentgateway.dev
 kind: AgentgatewayBackend
EOF

Step 7: Port-Forward and Test

Expose the gateway locally:

kubectl port-forward -n agentgateway-system deployment/mcp-gateway 8080:80

Test with MCP Inspector

npx @modelcontextprotocol/inspector@latest

Connect with:

Transport: Streamable HTTP
URL: http://localhost:8080/mcp

Click Tools → List Tools. You should see tools from both servers, namespaced:

mcp-server-everything-3001_echo — echo a message
mcp-server-everything-3001_add — add two numbers
mcp-server-everything-3001_longRunningOperation — simulate a long task
mcp-website-fetcher-80_fetch — fetch and extract content from a URL

One endpoint. Two servers. All tools.

Step 8: Connect from Your IDE

With port-forward running, configure your IDE:

Cursor

Create or edit ~/.cursor/mcp.json:

{
 "mcpServers": {
 "federated-tools": {
 "url": "http://localhost:8080/mcp"
 }
 }
}

VS Code (with GitHub Copilot)

Add to settings.json:

{
 "github.copilot.chat.mcp.servers": {
 "federated-tools": {
 "url": "http://localhost:8080/mcp"
 }
 }
}

Claude Code

claude mcp add federated-tools --transport sse http://localhost:8080/mcp

Windsurf

Create or edit ~/.windsurf/mcp.json:

{
 "mcpServers": {
 "federated-tools": {
 "url": "http://localhost:8080/mcp"
 }
 }
}

OpenCode

Add to opencode.json:

{
 "mcp": {
 "servers": {
 "federated-tools": {
 "type": "sse",
 "url": "http://localhost:8080/mcp"
 }
 }
 }
}

Now try asking your IDE:

“Echo back: agentgateway is awesome” — uses the everything server
“Add 42 and 58” — uses the everything server
“Fetch the content from https://agentgateway.dev” — uses the website fetcher

The IDE doesn’t know or care that these tools come from different servers. It’s all one MCP endpoint.

Adding More MCP Servers

The beauty of label-based federation: adding a new server is just a deployment + service with the mcp-federation: "true" label. No agentgateway config changes needed — the backend’s label selector picks it up automatically.

Cleanup

kubectl delete httproute mcp-route -n agentgateway-system
kubectl delete agentgatewaybackend mcp-federated -n agentgateway-system
kubectl delete deploy mcp-server-everything mcp-website-fetcher -n agentgateway-system
kubectl delete svc mcp-server-everything mcp-website-fetcher -n agentgateway-system
kind delete cluster --name agentgateway-mcp

Conclusion

MCP multiplexing solves one of the biggest challenges in production agentic AI environments: server sprawl. Developers shouldn’t need to configure connections to every MCP server individually. One gateway endpoint, all tools.

Federate all your MCP servers behind agentgateway, add new servers with a label, and every connected client sees the new tools immediately. No agent code changes. No client reconfiguration.

One gateway. All your tools.

AWS AgentCore with agentgateway and Okta OAuth

Thu, 19 Feb 2026 00:00:00 +0000

You wouldn’t deploy microservices without an API gateway. So why are teams deploying AI agents with direct, ungoverned access to LLMs and external tools?

If you’re running agents on AWS Bedrock AgentCore — or anywhere else — and those agents are calling Claude, GPT-4, or hitting MCP tool servers directly, you have a governance gap. No rate limiting. No audit trail. No PII filtering. No failover. Every team wiring up their own retry logic, their own auth handling, their own cost tracking. It’s 2016 microservices all over again, except the blast radius includes sending your customer database to a third-party LLM.

agentgateway fixes this. It’s an open-source (CNCF) gateway purpose-built for AI agent traffic — both LLM calls and MCP tool calls — giving you a single control plane for everything your agents talk to.

But here’s the thing: you don’t need another gateway from your cloud provider to do it. agentgateway is your gateway — for auth, RBAC, observability, and security. Your cloud just provides compute.

This post walks through why, how the auth chain works end-to-end, and how the aws-agentcore-demo wires it all together.

The Problem: Agents Without Guardrails

An AI agent is, at its core, a loop: receive input → call an LLM → maybe call some tools → return output. The interesting part is what happens in those calls.

When an agent on AgentCore calls Anthropic’s API directly:

API keys live in the agent container. Every developer who can deploy an agent has access to production LLM credentials.
No visibility. What prompts are being sent? What data is in them? You don’t know until something leaks.
No cost controls. One runaway agent loop burns through your API budget in minutes. There’s no per-user or per-agent rate limiting.
No failover. Anthropic has an outage? Your agent is dead. Hope someone set up a retry with exponential backoff — oh wait, each team did it differently.
No security policies. PII goes straight to the LLM. Prompt injection attempts pass through unfiltered. Credentials in responses get forwarded to users.

And then there’s MCP. Your agent discovers tools — Slack, GitHub, internal APIs — via Model Context Protocol servers. Each MCP server needs auth. Each one is another surface to secure, another thing to monitor, another connection to manage.

Every team solving these problems independently is wasted engineering. Worse, most teams don’t solve them at all.

The Trap: Another Vendor Gateway

Cloud providers see this problem too. Their solution? Another managed service. AWS has AgentCore Gateway. Others will follow.

But think about what that means:

You’re locked in. That gateway only works on AWS. Move to GCP, Azure, or on-prem? Rebuild your auth, RBAC, and observability from scratch.
Double the gateways. Now you have a cloud gateway and your actual governance layer. Two hops, two configurations, two things to debug.
Weakest-link security. If your cloud gateway does JWT validation but your real tools don’t, you have a false sense of security. If both do it, you’re paying for redundancy.
Vendor roadmap dependency. Need scope-based MCP RBAC? PII filtering? Prompt injection detection? You’re waiting on their feature backlog.

The right answer: use your cloud provider for what it’s good at — compute — and use a purpose-built, portable gateway for governance.

The Solution: agentgateway

agentgateway sits between your agents and everything they talk to. LLMs and MCP tool servers both route through it.

This isn’t Envoy with some AI plugins bolted on. agentgateway is a purpose-built proxy with its own xDS-inspired control plane, designed from the ground up for agent traffic patterns. It understands:

LLM protocols — OpenAI-compatible chat completions, streaming, token counting
MCP protocol — tool discovery, tool invocation, server lifecycle
Agent-specific policies — PII redaction, prompt injection detection, per-user token budgets
JWT authentication — validate Okta/Entra/any OIDC tokens on MCP routes with scope-based RBAC

Two logical functions, one binary:

LLM Proxy — OpenAI-compatible endpoint that routes to any provider (Anthropic, OpenAI, xAI, local models)
MCP Gateway — aggregates multiple MCP servers, presents a unified tool catalog to agents, enforces auth + RBAC

Your agent code barely changes. Point it at agentgateway instead of the LLM provider directly. Add a Bearer token to MCP calls. That’s it.

Traffic Flow: How This Demo Works

The aws-agentcore-demo is a working reference architecture. Here’s what’s actually running:

┌───────────────────────────────┐
│ AWS Bedrock AgentCore │
│ ┌───────────────────────────┐ │
│ │ Agent Container │ │
│ │ (arm64, Python/FastAPI) │ │
│ │ │ │
│ │ Gets Okta JWT │ │
│ │ (client_credentials) │ │
│ └──────┬──────────┬──────────┘ │
│ │ │ │
│ ┌──────▼──┐ ┌────▼─────────┐ │
│ │ Runtime │ │ (no gateway) │ │
│ │ Endpoint │ │ │ │
│ └─────────┘ └──────────────┘ │
└────────────────────┼────────────┘
 │ ngrok tunnels
 ▼
┌────────────────────────────────────┐
│ k8s-rooster (on-prem k8s) │
│ │
│ agentgateway (Enterprise) │
│ ┌──────────┐ ┌──────────────────┐ │
│ │ LLM │ │ MCP │ │
│ │ Proxy │ │ Gateway │ │
│ │ (no auth)│ │ (Okta JWT auth) │ │
│ │ │ │ (scope RBAC) │ │
│ └────┬─────┘ └──────┬───────────┘ │
└───────┼──────────────┼─────────────┘
 │ │
 ┌────▼────┐ ┌──────▼──────────┐
 │Anthropic│ │ MCP Servers │
 │OpenAI │ │ - Slack │
 │xAI │ │ - GitHub │
 └─────────┘ │ - Tools │
 └─────────────────┘

The agent container runs on AgentCore as an arm64 Python/FastAPI application. It doesn’t hold any LLM API keys. It knows two endpoints: agentgateway’s LLM proxy and MCP gateway.

LLM calls go through agentgateway’s OpenAI-compatible proxy — no auth needed. agentgateway holds the API keys (stored as Kubernetes Secrets, managed by ArgoCD), applies policies, logs the interaction, and returns the response.

MCP tool calls go through agentgateway’s MCP gateway. The agent includes its Okta JWT (obtained via client_credentials grant) in every MCP request. agentgateway validates the token, checks scopes, and enforces RBAC before the tool call reaches the upstream MCP server.

Connectivity between AWS and the on-prem k8s-rooster cluster uses ngrok tunnels — pragmatic for a demo. In production, you’d use VPC peering, PrivateLink, or similar.

No AgentCore Gateway. The agent container is invoked directly through the AgentCore Runtime endpoint. AWS IAM controls who can invoke the agent. agentgateway controls what the agent can do.

Authentication: End-to-End

Every MCP request traverses an authenticated chain — from AWS IAM through Okta to tool invocation. No ambient credentials, no hardcoded API keys in agent code, no unauthenticated MCP hops.

The Auth Stack

sequenceDiagram participant I as Invoker participant IAM as AWS IAM participant R as AgentCore Runtime participant A as Agent participant O as Okta participant AG as agentgateway participant LLM as LLM Provider participant MCP as MCP Tools I->>IAM: invoke-agent-runtime (SigV4) IAM->>R: Authorized → forward to agent R->>A: POST /invocations A->>O: client_credentials grant O-->>A: JWT (iss, aud, scp: mcp:read mcp:write) A->>AG: LLM request (no auth needed) AG->>AG: PII redaction, injection guard AG->>LLM: Forward (AG injects API key) LLM-->>AG: Response AG->>AG: Credential leak check AG-->>A: Sanitized response A->>AG: MCP tool call + Bearer JWT AG->>AG: Validate JWT (JWKS, iss, aud, exp) AG->>AG: Check scopes vs tool (RBAC) AG->>AG: Check deny list (destructive ops) AG->>MCP: Execute tool call MCP-->>AG: Result AG-->>A: Tool result A-->>R: Final response R-->>I: Response

Two Layers of Access Control

Layer 1: AWS IAM — controls who can invoke the agent. Standard AWS access management. You already know this.

Layer 2: agentgateway (Okta JWT + RBAC) — controls what the agent can do with MCP tools. This is where it gets interesting.

Okta Configuration

One OAuth2 service app provides the agent’s identity:

resource "okta_app_oauth" "agentcore_service" {
 label = "devops-copilot-service"
 type = "service"
 grant_types = ["client_credentials"]
 token_endpoint_auth_method = "client_secret_basic"
 response_types = ["token"]
}

Custom MCP scopes control fine-grained tool access:

resource "okta_auth_server_scope" "mcp_read" {
 name = "mcp:read"
 consent = "IMPLICIT"
}

resource "okta_auth_server_scope" "mcp_write" {
 name = "mcp:write"
 consent = "IMPLICIT"
}

resource "okta_auth_server_scope" "mcp_admin" {
 name = "mcp:admin"
 consent = "REQUIRED" # Explicit consent required
}

How the Agent Gets Its Token

The agent uses client_credentials — simple, no user interaction:

async def get_okta_token() -> str:
 """Get Okta JWT via client_credentials. Cached with 5-min buffer."""
 if _token_cache["access_token"] and time.time() < _token_cache["expires_at"] - 300:
 return _token_cache["access_token"]

 resp = await http_client.post(
 OKTA_TOKEN_URL,
 data={"grant_type": "client_credentials", "scope": "mcp:read mcp:write"},
 auth=(OKTA_CLIENT_ID, OKTA_CLIENT_SECRET),
 )
 data = resp.json()
 _token_cache["access_token"] = data["access_token"]
 _token_cache["expires_at"] = time.time() + data.get("expires_in", 3600)
 return data["access_token"]

The JWT payload:

{
 "iss": "https://integrator-7147223.okta.com/oauth2/aus104zseyg64swj3698",
 "aud": "api://default",
 "sub": "0oa...",
 "scp": ["mcp:read", "mcp:write"],
 "exp": 1739544600
}

Every MCP call includes this token:

headers["Authorization"] = f"Bearer {token}"

agentgateway Enforces Three Layers of MCP Access Control

Layer 1: JWT Authentication — Enterprise policy validates Okta JWTs on every MCP request. No valid token = no tool access.

traffic:
 jwtAuthentication:
 mode: Strict
 providers:
 - issuer: "https://your-okta.okta.com/oauth2/your-auth-server"
 audiences: ["api://default"]
 jwks:
 remote:
 url: "https://your-okta.okta.com/oauth2/your-auth-server/v1/keys"
 cacheDuration: 3600s

Layer 2: Scope-Based RBAC — CEL expressions match JWT scopes to tool operations:

backend:
 mcp:
 authorization:
 action: Allow
 policy:
 matchExpressions:
 - >-
 claims.scp.exists(s, s == 'mcp:read') && (
 tool.name.startsWith('list_') || tool.name.startsWith('get_')
 ) 
 - >-
 claims.scp.exists(s, s == 'mcp:write') && (
 tool.name.startsWith('post_') || tool.name.startsWith('create_')
 )

Scope	Allowed	Denied
`mcp:read`	List channels, get issues, search code	Post messages, create issues
`mcp:write`	All of read + post, create, comment	Delete, admin operations
`mcp:admin`	All of write + admin operations	Destructive ops (always blocked)

Key detail: tools/list responses are filtered (unauthorized tools hidden from the agent), tools/call requests rejected if scopes don’t match.

Layer 3: Destructive Operation Blocking — always denied regardless of scope:

denyPolicy:
 matchExpressions:
 - >-
 tool.name.contains('delete') || tool.name.contains('merge_pull_request')

Why Not Both Gateways?

“Why not keep the AgentCore Gateway and use agentgateway?”

You can. But you shouldn’t. Here’s why:

Redundant JWT validation. Both validate the same Okta token. One is enough. Having two means debugging auth failures in two places.
AgentCore Gateway can’t do what matters. It validates JWTs. Great. But it doesn’t do PII filtering, prompt injection detection, scope-based MCP RBAC, rate limiting per token budget, or LLM failover. You need agentgateway for all of that anyway.
Portability. The EnterpriseAgentgatewayPolicy you write for AWS works identically on GKE, AKS, bare metal, or your laptop. The AgentCore Gateway works on AWS only.
Fewer moving parts. One gateway, one auth config, one place to look when something breaks.

The demo architecture is intentionally minimal:

AWS provides compute (AgentCore Runtime)
agentgateway provides governance (auth, RBAC, security, observability)
Okta provides identity (JWT tokens)

Each component does one thing well.

Guardrails: The Full Picture

Security Policies

PII Protection. Before a prompt reaches the LLM, agentgateway scans for personally identifiable information and redacts it. Social security numbers, email addresses, phone numbers — stripped before they leave your network.

Prompt Injection Detection. Agents process user input. Users (or attackers) submit malicious prompts designed to hijack the agent’s behavior. agentgateway detects common jailbreak patterns and blocks them before the LLM ever sees them.

Credential Leak Prevention. LLMs sometimes echo back credentials that appeared in training data or context. agentgateway scans responses for patterns matching API keys, tokens, and passwords, blocking them before they reach users.

Traffic Management

Rate limiting operates at two levels — requests and tokens:

rate_limiting:
 - match:
 provider: xai
 limits:
 - requests_per_minute: 10
 - tokens_per_minute: 5000

Per-identity limits (from JWT sub claim) prevent any single agent or user from monopolizing LLM capacity.

Multi-provider failover: Configure primary and fallback providers. Anthropic down? agentgateway routes to OpenAI automatically — your agent code doesn’t change.

Observability

Every LLM call and MCP tool invocation is traced end-to-end:

Langfuse — LLM-specific traces with prompt/completion pairs, token usage, latency, cost
ClickHouse — raw OTLP spans for custom dashboards and alerting
Identity attribution — every trace tagged with the JWT identity that triggered it

The Takeaway

Your cloud provider should run your agents. It shouldn’t govern them.

agentgateway gives you one place to enforce auth, RBAC, security policies, rate limits, and observability — regardless of where your agents run. Write the policy once, deploy it anywhere.

The aws-agentcore-demo proves it: a fully functional agent on AWS Bedrock AgentCore, with zero vendor-specific gateways, and complete governance through agentgateway.

Resources:

Connect Any IDE to GitHub MCP Server Through agentgateway

Thu, 19 Feb 2026 00:00:00 +0000

Introduction

Modern AI-powered IDEs and coding agents ship with built-in MCP client support. That means you can connect them to GitHub’s remote MCP server to give your AI assistant direct access to issues, pull requests, code search, and repository management.

But connecting directly to GitHub means no visibility into what tools are being called, no rate limiting, and no centralized credential management. Every developer has their own PAT, every tool call goes straight to GitHub ungoverned.

This guide shows you how to route MCP traffic from any IDE — Cursor, VS Code, Windsurf, Claude Code, or OpenCode — to GitHub’s remote MCP server through Solo agentgateway. You deploy the gateway once, and every IDE on your team connects through it.

What You Get

One gateway, every IDE: Deploy agentgateway once, connect all your tools
Centralized credentials: GitHub PAT lives in a Kubernetes secret, not on every developer laptop
Rate limiting: Control tool call volume per IDE or per user
Observability: OpenTelemetry traces for every MCP interaction
No self-hosted MCP server: GitHub hosts the MCP server at api.githubcopilot.com — you just proxy

Architecture

┌──────────────┐
│ VS Code │──┐
├──────────────┤ │
│ Cursor │──┤
├──────────────┤ │ ┌──────────────────┐ ┌──────────────────────────┐
│ Windsurf │──┼──MCP──▶│ agentgateway │──MCP──▶│ api.githubcopilot.com │
├──────────────┤ │ │ (K8s proxy) │ │ (GitHub Remote MCP) │
│ Claude Code │──┤ └──────────────────┘ └──────────────────────────┘
├──────────────┤ │ │ Auth
│ OpenCode │──┘ │ Rate limiting
└──────────────┘ │ OTLP traces

Prerequisites

Docker installed and running
kind installed
kubectl and helm installed
A GitHub Personal Access Token (PAT)
At least one of: Cursor, VS Code (with Copilot), Windsurf, Claude Code, or OpenCode

Step 1: Create a Kind Cluster

kind create cluster --name agentgateway

Verify it’s running:

kubectl get nodes

Step 2: Install agentgateway

Deploy the Kubernetes Gateway API CRDs:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml

Install the agentgateway CRDs and control plane:

helm upgrade -i --create-namespace \
 --namespace agentgateway-system \
 --version v2.2.1 \
 agentgateway-crds oci://ghcr.io/kgateway-dev/charts/agentgateway-crds

helm upgrade -i -n agentgateway-system \
 agentgateway oci://ghcr.io/kgateway-dev/charts/agentgateway \
 --version v2.2.1

Verify the control plane is running:

kubectl get pods -n agentgateway-system

Step 3: Create an agentgateway Proxy

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: agentgateway-proxy
 namespace: agentgateway-system
spec:
 gatewayClassName: agentgateway
 listeners:
 - name: http
 port: 80
 protocol: HTTP
EOF

Wait for the proxy pod to be ready:

kubectl rollout status deploy/agentgateway-proxy -n agentgateway-system

Step 4: Configure the GitHub Remote MCP Backend

GitHub hosts a remote MCP server at https://api.githubcopilot.com/mcp/. We point agentgateway at it instead of deploying our own.

Create a secret with your GitHub PAT:

kubectl create secret generic github-mcp-secret \
 -n agentgateway-system \
 --from-literal=Authorization="Bearer ghp_your_token_here"

Create the backend:

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: github-mcp-backend
 namespace: agentgateway-system
spec:
 mcp:
 targets:
 - name: mcp-target
 static:
 host: api.githubcopilot.com
 port: 443
 path: /mcp/
 policies:
 auth:
 secretRef:
 name: github-mcp-secret
 tls:
 sni: api.githubcopilot.com
EOF

Key details:

path: /mcp/ — GitHub’s MCP endpoint
tls.sni — required for HTTPS on port 443
auth.secretRef — injects the Authorization: Bearer <PAT> header automatically

Step 5: Create the HTTPRoute

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: github-mcp
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: agentgateway-proxy
 namespace: agentgateway-system
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /github
 backendRefs:
 - name: github-mcp-backend
 group: agentgateway.dev
 kind: AgentgatewayBackend
EOF

Step 6: Port-Forward

Expose the gateway locally:

kubectl port-forward -n agentgateway-system deployment/agentgateway-proxy 8080:80

Leave this running. All IDE configs below use http://localhost:8080.

IDE Configuration

Each IDE has its own MCP config format. Pick yours below — or configure all of them to use the same gateway endpoint.

Cursor

Create or edit ~/.cursor/mcp.json (global) or .cursor/mcp.json (per-project):

{
 "mcpServers": {
 "github": {
 "url": "http://localhost:8080/github/mcp"
 }
 }
}

Restart Cursor → Settings → MCP → verify GitHub tools appear.

VS Code (with GitHub Copilot)

Add to your VS Code settings.json (Cmd/Ctrl + , → search “mcp”):

{
 "github.copilot.chat.mcp.servers": {
 "github": {
 "url": "http://localhost:8080/github/mcp"
 }
 }
}

Or for workspace-specific config, add to .vscode/settings.json.

Reload VS Code (Cmd/Ctrl + Shift + P → “Developer: Reload Window”) → open Copilot Chat → tools should list GitHub MCP tools.

Windsurf

Create or edit ~/.windsurf/mcp.json (global) or .windsurf/mcp.json (per-project):

{
 "mcpServers": {
 "github": {
 "url": "http://localhost:8080/github/mcp"
 }
 }
}

Restart Windsurf → verify connection in MCP settings.

Claude Code

Add via CLI:

claude mcp add github --transport sse http://localhost:8080/github/mcp

Or add to your project’s .mcp.json:

{
 "mcpServers": {
 "github": {
 "type": "sse",
 "url": "http://localhost:8080/github/mcp"
 }
 }
}

OpenCode

Add to your opencode.json config:

{
 "mcp": {
 "servers": {
 "github": {
 "type": "sse",
 "url": "http://localhost:8080/github/mcp"
 }
 }
 }
}

Step 7: Test It

Open your IDE’s AI chat and try:

“List open issues in my repo”
“Create a new branch called feature/test”
“Search for files referencing agentgateway”

Every tool call flows through agentgateway regardless of which IDE you’re using. The gateway handles auth, logs the interaction, and forwards to GitHub.

Verifying Traffic

Check proxy logs:

kubectl logs -n agentgateway-system deploy/agentgateway-proxy --tail=20

You should see MCP connection events and tool call forwards to api.githubcopilot.com.

IDE Comparison

IDE	Config File	Transport	Auth Headers
Cursor	`~/.cursor/mcp.json`	streamable-http	✅ supported
VS Code	`settings.json`	streamable-http	✅ supported
Windsurf	`~/.windsurf/mcp.json`	streamable-http	✅ supported
Claude Code	`.mcp.json` or CLI	SSE	✅ supported
OpenCode	`opencode.json`	SSE	✅ supported

Note: All IDEs connect to the same http://localhost:8080/github/mcp endpoint. The gateway doesn’t care which client is calling — it applies the same auth, rate limiting, and observability to all of them.

What’s Next

Add more MCP servers: Slack, Notion, Jira — all behind the same gateway, same endpoint pattern
Security policies: JWT auth, prompt guards, tool-level RBAC with AgentgatewayPolicy
Team rollout: Share the gateway endpoint with your team — one PAT, centrally managed, no credentials on laptops
Per-IDE rate limiting: Different limits for different tools or users

Cleanup

kubectl delete httproute github-mcp -n agentgateway-system
kubectl delete agentgatewaybackend github-mcp-backend -n agentgateway-system
kubectl delete secret github-mcp-secret -n agentgateway-system
kind delete cluster --name agentgateway

Conclusion

Every AI-powered IDE now speaks MCP. But connecting each one directly to backend servers means fragmented credentials, zero visibility, and no governance. agentgateway gives you a single control point — deploy it once, connect every IDE on your team, and get auth, rate limiting, and observability on every tool call.

One gateway. Every IDE. Full visibility.

Open Source LLM Observability: Tracing AI Calls with agentgateway and Langfuse

Wed, 18 Feb 2026 00:00:00 +0000

Introduction

You’ve got your AI gateway routing LLM traffic. But can you actually see what’s happening? Which models are being called, how many tokens are being consumed, what prompts are going in and what completions are coming back?

This guide shows you how to integrate Langfuse with Solo agentgateway to get full observability over every LLM request — without touching your application code. We’ll go from zero to traced requests on a local kind cluster in under 10 minutes.

Why This Matters

agentgateway already captures rich telemetry about every LLM request: model, tokens, latency, route, and security policy actions. By forwarding those traces to Langfuse, you get:

Full prompt and completion visibility across all LLM providers (OpenAI, Anthropic, xAI, etc.)
Token usage and cost tracking per model, route, and user
Latency analysis with gateway-level metadata — which route, which backend, which policy fired
Zero application changes — tracing happens at the gateway layer, not in your app

This is the power of observability at the infrastructure layer. Your developers don’t need to instrument anything. Every LLM call that flows through the gateway is automatically captured.

Architecture

Here’s what we’re building:

┌──────────────┐ ┌────────────────────────┐ ┌─────────────────┐
│ Your App / │ │ Solo agentgateway │ │ LLM Provider │
│ AI Agent │────▶│ (Gateway API) │────▶│ (OpenAI, etc) │
└──────────────┘ └───────────┬────────────┘ └─────────────────┘
 │
 OTLP Traces (gRPC)
 │
 ┌───────────▼────────────┐
 │ OpenTelemetry │
 │ Collector │
 └─────┬───────────┬──────┘
 │ │
 OTLP HTTP OTLP gRPC
 │ │
 ┌─────▼──┐ ┌─────▼──────────┐
 │Langfuse│ │Other backends │
 │ UI │ │(Jaeger, etc.) │
 └────────┘ └─────────────────┘

agentgateway natively emits OpenTelemetry traces for every LLM request. A lightweight OTel Collector receives those traces and forwards them to Langfuse via OTLP HTTP. The collector can also fan-out to additional backends like Jaeger, Datadog, or ClickHouse if you need traces in multiple places.

Prerequisites

Docker installed and running
kind installed
kubectl installed
helm installed
A Langfuse account (free tier works) or self-hosted instance
An OpenAI API key (or any supported LLM provider)

Step 1: Create a Kind Cluster

kind create cluster --name agentgateway
kubectl cluster-info --context kind-agentgateway

Step 2: Install the Gateway API CRDs

agentgateway uses the standard Kubernetes Gateway API for configuration:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml

Step 3: Install agentgateway

Install the CRDs and control plane:

helm upgrade -i agentgateway-crds oci://cr.agentgateway.dev/helm/agentgateway-crds \
 --version 2.1.0 \
 --namespace agentgateway-system \
 --create-namespace

helm upgrade -i agentgateway oci://cr.agentgateway.dev/helm/agentgateway \
 --version 2.1.0 \
 --namespace agentgateway-system

Verify everything is running:

kubectl get pods -n agentgateway-system
kubectl get gatewayclass agentgateway

Step 4: Get Your Langfuse Credentials

Log in to Langfuse and go to Settings → API Keys. Create a new key pair (or use an existing one). Base64 encode them for the collector config:

echo -n "pk-lf-YOUR_PUBLIC_KEY:sk-lf-YOUR_SECRET_KEY" | base64

Step 5: Deploy the OTel Collector

The collector bridges agentgateway (OTLP gRPC) and Langfuse (OTLP HTTP). Create langfuse-collector.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
 name: langfuse-otel-collector-config
 namespace: agentgateway-system
data:
 config.yaml: |
 receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318
 exporters:
 otlphttp/langfuse:
 endpoint: https://cloud.langfuse.com/api/public/otel
 headers:
 Authorization: "Basic <YOUR_BASE64_CREDENTIALS>"
 retry_on_failure:
 enabled: true
 initial_interval: 5s
 max_interval: 30s
 max_elapsed_time: 300s
 processors:
 batch:
 send_batch_size: 1000
 timeout: 5s
 service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [batch]
 exporters: [otlphttp/langfuse] 
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: langfuse-otel-collector
 namespace: agentgateway-system
 labels:
 app: langfuse-otel-collector
spec:
 replicas: 1
 selector:
 matchLabels:
 app: langfuse-otel-collector
 template:
 metadata:
 labels:
 app: langfuse-otel-collector
 spec:
 containers:
 - name: otel-collector
 image: otel/opentelemetry-collector-contrib:0.132.1
 args: ["--config=/conf/config.yaml"]
 ports:
 - containerPort: 4317
 name: otlp-grpc
 - containerPort: 4318
 name: otlp-http
 volumeMounts:
 - name: config
 mountPath: /conf
 resources:
 requests:
 cpu: 50m
 memory: 128Mi
 limits:
 cpu: 200m
 memory: 256Mi
 volumes:
 - name: config
 configMap:
 name: langfuse-otel-collector-config
---
apiVersion: v1
kind: Service
metadata:
 name: langfuse-otel-collector
 namespace: agentgateway-system
spec:
 selector:
 app: langfuse-otel-collector
 ports:
 - name: otlp-grpc
 port: 4317
 targetPort: 4317
 - name: otlp-http
 port: 4318
 targetPort: 4318

Deploy it:

kubectl apply -f langfuse-collector.yaml

Step 6: Enable Tracing on agentgateway

For agentgateway OSS, configure tracing via Helm values. Create values-tracing.yaml:

gateway:
 envs:
 OTEL_EXPORTER_OTLP_ENDPOINT: "http://langfuse-otel-collector.agentgateway-system.svc.cluster.local:4317"
 OTEL_EXPORTER_OTLP_PROTOCOL: "grpc"

Upgrade the installation:

helm upgrade agentgateway oci://cr.agentgateway.dev/helm/agentgateway \
 --version 2.1.0 \
 --namespace agentgateway-system \
 -f values-tracing.yaml

Using agentgateway Enterprise? Instead of environment variables, create an EnterpriseAgentgatewayParameters resource with full control over field mappings:

apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayParameters
metadata:
 name: tracing
 namespace: agentgateway-system
spec:
 rawConfig:
 config:
 tracing:
 otlpEndpoint: grpc://langfuse-otel-collector.agentgateway-system.svc.cluster.local:4317
 otlpProtocol: grpc
 randomSampling: true
 fields:
 add:
 gen_ai.operation.name: '"chat"'
 gen_ai.system: "llm.provider"
 gen_ai.request.model: "llm.requestModel"
 gen_ai.response.model: "llm.responseModel"
 gen_ai.usage.prompt_tokens: "llm.inputTokens"
 gen_ai.usage.completion_tokens: "llm.outputTokens"
 gen_ai.usage.total_tokens: "llm.totalTokens"
 gen_ai.request.temperature: "llm.params.temperature"
 gen_ai.prompt: "llm.prompt"
 gen_ai.completion: "llm.completion"

Step 7: Create a Gateway and LLM Route

Create the Gateway resource and an OpenAI route:

# gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: ai-gateway
 namespace: agentgateway-system
spec:
 gatewayClassName: agentgateway
 listeners:
 - name: llm
 port: 8080
 protocol: HTTPS
 allowedRoutes:
 namespaces:
 from: Same

# openai-route.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: ai-gateway
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /openai
 backendRefs:
 - group: agentgateway.dev
 kind: AgentgatewayBackend
 name: openai
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: openai
 namespace: agentgateway-system
spec:
 type: llm
 llm:
 provider:
 openai:
 authToken:
 secretRef:
 name: openai-api-key
 namespace: agentgateway-system

Create the secret and apply everything:

kubectl create secret generic openai-api-key \
 -n agentgateway-system \
 --from-literal=Authorization="Bearer $OPENAI_API_KEY"

kubectl apply -f gateway.yaml
kubectl apply -f openai-route.yaml

Step 8: Test It

Port-forward the gateway and send a request:

kubectl port-forward -n agentgateway-system svc/ai-gateway 8080:8080 &

curl -X POST http://localhost:8080/openai/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "gpt-4.1-mini",
 "messages": [{"role": "user", "content": "Hello from agentgateway!"}]
 }'

Step 9: View Traces in Langfuse

Open your Langfuse UI and navigate to Traces. You should see a new trace with the full picture:

Model and provider — which model actually served the request
Token counts — input, output, and total tokens consumed
Full prompt and completion — the exact content sent and received
Gateway metadata — route name, backend endpoint, listener

What Gets Captured

agentgateway follows the OpenTelemetry GenAI semantic conventions, so every trace includes structured attributes:

Attribute	What It Tells You
`gen_ai.system`	LLM provider (openai, anthropic, etc.)
`gen_ai.request.model`	The model you asked for
`gen_ai.response.model`	The model that actually responded
`gen_ai.usage.prompt_tokens`	Input tokens consumed
`gen_ai.usage.completion_tokens`	Output tokens generated
`gen_ai.prompt`	Full prompt content
`gen_ai.completion`	Full completion content
`gateway`	agentgateway resource name
`route`	HTTPRoute that matched
`endpoint`	Backend LLM endpoint

Going Further

Fan-Out to Multiple Backends

The OTel Collector makes it easy to send traces to Langfuse and another backend simultaneously:

exporters:
 otlphttp/langfuse:
 endpoint: https://cloud.langfuse.com/api/public/otel
 headers:
 Authorization: "Basic <CREDENTIALS>"
 otlp/jaeger:
 endpoint: jaeger-collector:4317
 tls:
 insecure: true

service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [batch]
 exporters: [otlphttp/langfuse, otlp/jaeger]

MCP Tool Tracing

agentgateway also traces MCP (Model Context Protocol) traffic. When an agent discovers or invokes tools through an MCP server proxied by agentgateway, you’ll see tool discovery requests, execution calls with parameters and results, and backend server latency — all as spans within the same trace.

Security Policy Visibility

When agentgateway’s security policies fire — PII protection, prompt injection detection, credential leak prevention — the trace metadata shows which policy triggered, what action was taken (block, mask, allow), and what pattern matched. You get observability into both your LLM interactions and your security guardrails in one place.

Troubleshooting

No traces appearing?

Check the collector is running: kubectl get pods -n agentgateway-system -l app=langfuse-otel-collector
Check collector logs for errors: kubectl logs -n agentgateway-system -l app=langfuse-otel-collector
Wrong API keys show up as 401 errors in collector logs
Make sure proxies were restarted after configuring tracing

Traces visible in gateway logs but not Langfuse?

Langfuse only supports OTLP HTTP, not gRPC — that’s why the collector is needed for protocol conversion
Verify the exporter endpoint URL includes /api/public/otel
Double-check the Base64 credentials format: base64(public_key:secret_key)

Incomplete trace data?

For Enterprise, ensure fields.add includes the GenAI attribute mappings
Set randomSampling: true to capture all requests during testing

Cleanup

kind delete cluster --name agentgateway

Resources

Route MCP and LLM Traffic from Claude Desktop and Claude Code Through agentgateway

Wed, 18 Feb 2026 00:00:00 +0000

Introduction

Claude Desktop and Claude Code are powerful AI tools, but out of the box they talk directly to backend services with no visibility, no security controls, and no usage governance. Every MCP tool call from Claude Desktop hits your servers unmonitored. Every LLM API call from Claude Code goes straight to the provider with no rate limiting or audit trail.

What if you could put a gateway in front of all that traffic?

This guide shows you how to route both MCP server traffic from Claude Desktop and LLM API calls from Claude Code through Solo agentgateway — giving you JWT authentication, observability traces, rate limiting, and centralized API key management. We’ll use Anthropic as the LLM provider.

What You Get

By routing traffic through agentgateway, you gain:

Security: JWT authentication on MCP endpoints, RBAC for tool access, prompt injection guards
Observability: OpenTelemetry traces for every MCP tool call and LLM request
Rate Limiting: Token-based and request-based limits per user or team
Centralized Secrets: API keys live in Kubernetes secrets, not on developer laptops
Audit Trail: Full logging of every interaction for compliance

Architecture

┌──────────────────┐ ┌──────────────────────────┐
│ Claude Desktop │ │ │
│ (MCP traffic) │─────────▶│ │──▶ MCP Servers
│ │ │ │ (math, github, etc.)
└──────────────────┘ │ Solo agentgateway │
 │ (Gateway API) │
┌──────────────────┐ │ │
│ Claude Code │ │ • JWT Auth │
│ (LLM traffic) │─────────▶│ • Rate Limiting │──▶ Anthropic API
│ │ │ • OTel Tracing │
└──────────────────┘ │ • Prompt Guards │
 └──────────────────────────┘

Claude Desktop connects to MCP servers through the gateway. Claude Code sends its LLM API calls through the gateway to Anthropic. Both streams get the same security and observability treatment.

Prerequisites

Kubernetes cluster with agentgateway deployed (quickstart)
kubectl and helm installed
Anthropic API key
Claude Desktop installed (for MCP routing)
Claude Code CLI installed (for LLM routing): npm install -g @anthropic-ai/claude-code

Part 1: Deploy an MCP Server

We’ll deploy a simple math MCP server that Claude Desktop can call through the gateway.

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
 name: mcp-math-script
 namespace: default
data:
 server.py: |
 import uvicorn
 from mcp.server.fastmcp import FastMCP
 from starlette.applications import Starlette
 from starlette.routing import Route
 from starlette.requests import Request
 from starlette.responses import JSONResponse, Response

 mcp = FastMCP("Math-Service")

 @mcp.tool()
 def add(a: int, b: int) -> int:
 return a + b

 @mcp.tool()
 def multiply(a: int, b: int) -> int:
 return a * b

 async def handle_mcp(request: Request):
 try:
 data = await request.json()
 method = data.get("method")
 msg_id = data.get("id")
 result = None

 if method == "initialize":
 result = {
 "protocolVersion": "2024-11-05",
 "capabilities": {"tools": {}},
 "serverInfo": {"name": "Math-Service", "version": "1.0"}
 }
 elif method == "notifications/initialized":
 return Response(status_code=202)
 elif method == "tools/list":
 tools_list = await mcp.list_tools()
 result = {
 "tools": [
 {"name": t.name, "description": t.description, "inputSchema": t.inputSchema}
 for t in tools_list
 ]
 }
 elif method == "tools/call":
 params = data.get("params", {})
 name = params.get("name")
 args = params.get("arguments", {})
 tool_result = await mcp.call_tool(name, args)
 serialized = []
 for content in tool_result:
 if hasattr(content, "type") and content.type == "text":
 serialized.append({"type": "text", "text": content.text})
 else:
 serialized.append(content if isinstance(content, dict) else str(content))
 result = {"content": serialized, "isError": False}
 elif method == "ping":
 result = {}
 else:
 return JSONResponse(
 {"jsonrpc": "2.0", "id": msg_id, "error": {"code": -32601, "message": "Method not found"}},
 status_code=404
 )
 return JSONResponse({"jsonrpc": "2.0", "id": msg_id, "result": result})
 except Exception as e:
 import traceback
 traceback.print_exc()
 return JSONResponse(
 {"jsonrpc": "2.0", "id": None, "error": {"code": -32603, "message": str(e)}},
 status_code=500
 )

 app = Starlette(routes=[
 Route("/mcp", handle_mcp, methods=["POST"]),
 Route("/", lambda r: JSONResponse({"status": "ok"}), methods=["GET"])
 ])

 if __name__ == "__main__":
 uvicorn.run(app, host="0.0.0.0", port=8000)
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: mcp-math-server
 namespace: default
spec:
 replicas: 1
 selector:
 matchLabels:
 app: mcp-math-server
 template:
 metadata:
 labels:
 app: mcp-math-server
 spec:
 containers:
 - name: math
 image: python:3.11-slim
 command: ["/bin/sh", "-c"]
 args:
 - |
 pip install "mcp[cli]" uvicorn starlette &&
 python /app/server.py
 ports:
 - containerPort: 8000
 volumeMounts:
 - name: script-volume
 mountPath: /app
 readinessProbe:
 httpGet:
 path: /
 port: 8000
 initialDelaySeconds: 5
 periodSeconds: 5
 volumes:
 - name: script-volume
 configMap:
 name: mcp-math-script
---
apiVersion: v1
kind: Service
metadata:
 name: mcp-math-server
 namespace: default
spec:
 selector:
 app: mcp-math-server
 ports:
 - port: 80
 targetPort: 8000
EOF

Wait for the pod to be ready:

kubectl wait --for=condition=ready pod -l app=mcp-math-server --timeout=120s

Part 2: Create the Gateway and Routes

We need two things routed through agentgateway: MCP tool traffic and LLM API traffic to Anthropic.

Create the Anthropic API Key Secret

kubectl create secret generic anthropic-api-key \
 -n agentgateway-system \
 --from-literal=Authorization="Bearer $ANTHROPIC_API_KEY"

Create the Gateway

A single Gateway with one listener handles both MCP and LLM traffic:

# gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: ai-gateway
 namespace: agentgateway-system
spec:
 gatewayClassName: agentgateway
 listeners:
 - name: http
 port: 8080
 protocol: HTTPS
 allowedRoutes:
 namespaces:
 from: All

Create the MCP Backend and Route

Point the gateway at our math MCP server:

# mcp-backend.yaml
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: math-mcp
 namespace: agentgateway-system
spec:
 mcp:
 targets:
 - name: math-service
 static:
 host: mcp-math-server.default.svc.cluster.local
 port: 80
 path: /mcp
 protocol: StreamableHTTP
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: mcp-route
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: ai-gateway
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /mcp
 backendRefs:
 - name: math-mcp
 group: agentgateway.dev
 kind: AgentgatewayBackend

Create the Anthropic LLM Backend and Route

# anthropic-backend.yaml
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: anthropic-llm
 namespace: agentgateway-system
spec:
 type: llm
 llm:
 provider:
 anthropic:
 authToken:
 secretRef:
 name: anthropic-api-key
 namespace: agentgateway-system
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: anthropic-route
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: ai-gateway
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /anthropic
 backendRefs:
 - name: anthropic-llm
 group: agentgateway.dev
 kind: AgentgatewayBackend

Apply everything:

kubectl apply -f gateway.yaml
kubectl apply -f mcp-backend.yaml
kubectl apply -f anthropic-backend.yaml

Get the Gateway Address

export GATEWAY_IP=$(kubectl get svc ai-gateway -n agentgateway-system \
 -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo "Gateway: http://$GATEWAY_IP:8080"

For local clusters (kind/minikube), use port-forward instead:

kubectl port-forward -n agentgateway-system svc/ai-gateway 8080:8080 &
export GATEWAY_IP=localhost

Part 3: Configure Claude Desktop (MCP Traffic)

Claude Desktop can route its MCP tool calls through agentgateway using supergateway as a local bridge.

Config File Location

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

Basic Configuration

Create or update the config file:

{
 "mcpServers": {
 "math-service": {
 "command": "npx",
 "args": ["-y", "supergateway", "--streamableHttp", "http://GATEWAY_IP:8080/mcp"]
 }
 }
}

Replace GATEWAY_IP with your actual gateway IP or localhost if using port-forward.

With JWT Authentication

If you’ve configured JWT auth on the gateway:

{
 "mcpServers": {
 "math-service": {
 "command": "npx",
 "args": ["-y", "mcp-remote", "http://GATEWAY_IP:8080/mcp"],
 "env": {
 "MCP_HEADERS": "Authorization: Bearer <your-jwt-token>"
 }
 }
 }
}

Restart Claude Desktop after saving the config. You should see the math tools available in the tools menu.

Part 4: Configure Claude Code (LLM Traffic via Anthropic)

Claude Code can route its LLM API traffic through agentgateway to Anthropic. This means your API keys stay in Kubernetes secrets — not on developer machines.

Set the Base URL

Point Claude Code at the gateway’s Anthropic route:

export ANTHROPIC_BASE_URL=http://$GATEWAY_IP:8080/anthropic

For persistence, add it to your shell profile:

echo "export ANTHROPIC_BASE_URL=http://$GATEWAY_IP:8080/anthropic" >> ~/.zshrc

Run Claude Code

claude

All LLM traffic now flows through agentgateway to Anthropic. You can verify by checking the gateway logs:

kubectl logs -n agentgateway-system -l gateway.networking.k8s.io/gateway-name=ai-gateway -f

Part 5: Add Security Policies

Now that traffic flows through the gateway, you can layer on security controls.

Rate Limiting

Prevent runaway costs with token-based rate limiting:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
 name: rate-limit
 namespace: agentgateway-system
spec:
 targetRefs:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 name: anthropic-route
 default:
 rateLimiting:
 tokenBucket:
 maxTokens: 10000
 refillRate: 1000
 refillInterval: 60s

Prompt Guards

Block prompt injection attempts before they reach Anthropic:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
 name: prompt-guard
 namespace: agentgateway-system
spec:
 targetRefs:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 name: anthropic-route
 default:
 promptGuard:
 request:
 matches:
 - action: REJECT
 regex: "ignore (previous|all) instructions"

Apply the policies:

kubectl apply -f rate-limit.yaml
kubectl apply -f prompt-guard.yaml

Part 6: Enable Observability

Add OpenTelemetry tracing to see every request in detail. If you followed our Langfuse observability guide, you already have this set up. Otherwise, configure tracing via Helm values:

# values-tracing.yaml
gateway:
 envs:
 OTEL_EXPORTER_OTLP_ENDPOINT: "http://your-otel-collector:4317"
 OTEL_EXPORTER_OTLP_PROTOCOL: "grpc"

helm upgrade agentgateway oci://cr.agentgateway.dev/helm/agentgateway \
 --version 2.1.0 \
 --namespace agentgateway-system \
 -f values-tracing.yaml

With tracing enabled, every Claude Desktop tool call and every Claude Code LLM request shows up as a trace with full metadata: model, tokens, latency, route, and any security policy actions.

Testing the Full Setup

Test MCP (Claude Desktop)

Open Claude Desktop and ask it to use the math tools:

“What’s 42 multiplied by 17?”

Claude should invoke the multiply tool through agentgateway. Check the gateway logs to confirm the request was proxied.

Test LLM (Claude Code)

ANTHROPIC_BASE_URL=http://$GATEWAY_IP:8080/anthropic claude

# In Claude Code, type any prompt — it routes through the gateway to Anthropic

Test with MCP Inspector

You can also verify MCP connectivity directly:

npx @modelcontextprotocol/inspector

Enter the URL: http://GATEWAY_IP:8080/mcp

You should see the math tools listed and be able to invoke them.

Important Limitations

Claude Desktop’s core LLM traffic (conversations with Claude itself) cannot be routed through agentgateway — only MCP server traffic is proxied
Claude Code LLM traffic can be fully routed through the gateway via the ANTHROPIC_BASE_URL environment variable
For local development (kind/minikube), you’ll need port-forwarding since there’s no external load balancer

Troubleshooting

MCP tools not showing in Claude Desktop?

Restart Claude Desktop after config changes
Verify the gateway is reachable: curl http://GATEWAY_IP:8080/mcp
Check that supergateway/mcp-remote is installed: npx -y supergateway --help

Claude Code connection refused?

Verify the gateway service has an external IP: kubectl get svc -n agentgateway-system
Check firewall rules allow traffic on port 8080
Ensure ANTHROPIC_BASE_URL is set correctly: echo $ANTHROPIC_BASE_URL

Timeout errors?

LLM requests can take time — ensure gateway timeouts are configured for AI workloads
Check gateway pod resources are sufficient for the traffic volume

Authentication errors?

Verify the Anthropic API key secret exists: kubectl get secret anthropic-api-key -n agentgateway-system
Check the key is valid with a direct curl to Anthropic’s API

Cleanup

kubectl delete -f gateway.yaml -f mcp-backend.yaml -f anthropic-backend.yaml
kubectl delete deployment mcp-math-server
kubectl delete svc mcp-math-server
kubectl delete configmap mcp-math-script
kubectl delete secret anthropic-api-key -n agentgateway-system

Resources

Getting Started with LLM Provider Routing on Kind

Sun, 15 Feb 2026 00:00:00 +0000

Agentgateway makes it simple to route traffic to multiple LLM providers through a single gateway using the Kubernetes Gateway API. This guide walks through setting up agentgateway OSS on a local Kind cluster with xAI, Anthropic, and OpenAI backends, all routed through a listener named llm-providers.

One of the most common patterns in AI-native infrastructure is routing traffic to multiple LLM providers behind a single entry point. Whether you’re comparing models, building failover strategies, or just want a unified API across providers, agentgateway gives you a clean Kubernetes-native way to do it using the Gateway API and AgentgatewayBackend custom resources.

In this guide, we’ll set up a complete working example on a local Kind cluster with three LLM providers routed via path-based HTTPRoute resources.

What you’ll build

By the end of this guide you’ll have:

A Kind cluster running the agentgateway control plane
A Gateway with a listener named llm-providers on port 8080
Three AgentgatewayBackend resources for xAI, Anthropic, and OpenAI
Three HTTPRoute resources that route /xai, /anthropic, and /openai to their respective backends

Prerequisites

Before getting started, make sure you have the following installed:

Docker — container runtime for Kind
Kind — local Kubernetes clusters
kubectl — Kubernetes CLI (within 1 minor version of your cluster)
Helm — Kubernetes package manager

You also need API keys for the LLM providers you want to use. Export them as environment variables:

export XAI_API_KEY="your-xai-api-key"
export ANTHROPIC_API_KEY="your-anthropic-api-key"
export OPENAI_API_KEY="your-openai-api-key"

Step 1: Create a Kind cluster

Create a local Kubernetes cluster using Kind. This gives you a lightweight, disposable cluster perfect for testing.

kind create cluster --name agentgateway-demo

Verify it’s running:

kubectl cluster-info --context kind-agentgateway-demo
kubectl get nodes

Step 2: Install agentgateway OSS via Helm

Install the Gateway API CRDs

Agentgateway relies on the Kubernetes Gateway API. Install the standard CRDs:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml

Install the agentgateway CRDs

Install the custom resource definitions that agentgateway needs (AgentgatewayBackend, AgentgatewayPolicy, etc.):

helm upgrade -i agentgateway-crds \
 oci://ghcr.io/kgateway-dev/charts/agentgateway-crds \
 --create-namespace \
 --namespace agentgateway-system \
 --version v2.2.0-main

Install the agentgateway control plane

helm upgrade -i agentgateway \
 oci://ghcr.io/kgateway-dev/charts/agentgateway \
 --namespace agentgateway-system \
 --version v2.2.0-main \
 --set controller.image.pullPolicy=Always

The --set controller.image.pullPolicy=Always flag is recommended for development builds to ensure you always get the latest image.

Verify the pods are running:

kubectl get pods -n agentgateway-system

You should see the controller pod in a Running state.

Step 3: Create the Gateway

The Gateway resource is the entry point for all traffic. It defines a listener named llm-providers on port 8080 that accepts HTTPRoute resources from any namespace.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: agentgateway-proxy
 namespace: agentgateway-system
spec:
 gatewayClassName: enterprise-agentgateway
 infrastructure:
 parametersRef:
 name: tracing
 group: enterpriseagentgateway.solo.io
 kind: EnterpriseAgentgatewayParameters
 listeners:
 - protocol: HTTP
 port: 8080
 name: llm-providers
 allowedRoutes:
 namespaces:
 from: All
EOF

The listener name llm-providers is the key here. All HTTPRoute resources in the following steps reference this listener via sectionName, so the gateway knows which listener should handle each route.

Step 4: Configure API key secrets

Each LLM provider needs an API key stored as a Kubernetes Secret. The AgentgatewayBackend resources reference these secrets for authentication.

kubectl apply -f- <<EOF
apiVersion: v1
kind: Secret
metadata:
 name: xai-secret
 namespace: agentgateway-system
type: Opaque
stringData:
 Authorization: $XAI_API_KEY
EOF

kubectl apply -f- <<EOF
apiVersion: v1
kind: Secret
metadata:
 name: anthropic-secret
 namespace: agentgateway-system
type: Opaque
stringData:
 Authorization: $ANTHROPIC_API_KEY
EOF

kubectl apply -f- <<EOF
apiVersion: v1
kind: Secret
metadata:
 name: openai-secret
 namespace: agentgateway-system
type: Opaque
stringData:
 Authorization: $OPENAI_API_KEY
EOF

Never commit API keys to source control. Use environment variable substitution or a secrets manager in production.

Step 5: Create agentgateway backends

AgentgatewayBackend resources define the LLM provider endpoints. Each backend specifies the provider type, model, and authentication. Agentgateway automatically rewrites requests to the correct chat completion endpoint for each provider.

xAI backend

xAI uses an OpenAI-compatible API. Because we’re specifying a custom host (api.x.ai) rather than the default OpenAI host, we need to explicitly set the host, port, path, and TLS SNI.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: xai
 namespace: agentgateway-system
spec:
 ai:
 provider:
 openai:
 model: grok-4-1-fast-reasoning
 host: api.x.ai
 port: 443
 path: "/v1/chat/completions"
 policies:
 auth:
 secretRef:
 name: xai-secret
 tls:
 sni: api.x.ai
EOF

Anthropic backend

Anthropic uses its native provider type. Agentgateway handles the endpoint rewriting automatically — no custom host or TLS configuration needed.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: anthropic
 namespace: agentgateway-system
spec:
 ai:
 provider:
 anthropic:
 model: "claude-sonnet-4-5-20250929"
 policies:
 auth:
 secretRef:
 name: anthropic-secret
EOF

OpenAI backend

OpenAI also uses its native provider type with the default endpoint.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: openai
 namespace: agentgateway-system
spec:
 ai:
 provider:
 openai:
 model: gpt-4o-mini
 policies:
 auth:
 secretRef:
 name: openai-secret
EOF

Step 6: Create HTTPRoutes

HTTPRoute resources connect incoming request paths to the AgentgatewayBackend resources. Each route references the llm-providers listener on the Gateway via sectionName, and matches a path prefix to direct traffic to the correct backend.

Route	Path	Backend	Provider
xai	`/xai`	xai	xAI (Grok)
anthropic	`/anthropic`	anthropic	Anthropic (Claude)
openai	`/openai`	openai	OpenAI (GPT)

xAI route

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: xai
 namespace: agentgateway-system
 labels:
 route-type: llm-provider
spec:
 parentRefs:
 - name: agentgateway-proxy
 namespace: agentgateway-system
 sectionName: llm-providers
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /xai
 backendRefs:
 - name: xai
 namespace: agentgateway-system
 group: agentgateway.dev
 kind: AgentgatewayBackend
EOF

Anthropic route

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: anthropic
 namespace: agentgateway-system
 labels:
 route-type: llm-provider
spec:
 parentRefs:
 - name: agentgateway-proxy
 namespace: agentgateway-system
 sectionName: llm-providers
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /anthropic
 backendRefs:
 - name: anthropic
 namespace: agentgateway-system
 group: agentgateway.dev
 kind: AgentgatewayBackend
EOF

OpenAI route

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai
 namespace: agentgateway-system
 labels:
 route-type: llm-provider
spec:
 parentRefs:
 - name: agentgateway-proxy
 namespace: agentgateway-system
 sectionName: llm-providers
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /openai
 backendRefs:
 - name: openai
 namespace: agentgateway-system
 group: agentgateway.dev
 kind: AgentgatewayBackend
EOF

The key fields that tie everything together:

parentRefs.sectionName: llm-providers — binds the route to the specific Gateway listener
backendRefs.group: agentgateway.dev — tells the Gateway API to look for AgentgatewayBackend resources (not standard Kubernetes Service objects)
backendRefs.kind: AgentgatewayBackend — references the custom backend type
labels.route-type: llm-provider — optional label useful for filtering and grouping

Step 7: Verify and test

Once all resources are applied, verify everything is connected and working.

Check resource status

kubectl get gateway -n agentgateway-system

# Verify backends exist
kubectl get agentgatewaybackend -n agentgateway-system

# Verify routes are attached
kubectl get httproute -n agentgateway-system

Port-forward and test

Forward the gateway port to your local machine and send a test request:

kubectl port-forward -n agentgateway-system \
 svc/agentgateway-proxy 8080:8080 &

Test the OpenAI route:

curl -s http://localhost:8080/openai \
 -H "Content-Type: application/json" \
 -d '{
 "model": "gpt-4o-mini",
 "messages": [{"role": "user", "content": "Hello"}]
 }'

Test the Anthropic route:

curl -s http://localhost:8080/anthropic \
 -H "Content-Type: application/json" \
 -d '{
 "model": "claude-sonnet-4-5-20250929",
 "messages": [{"role": "user", "content": "Hello"}]
 }'

Test the xAI route:

curl -s http://localhost:8080/xai \
 -H "Content-Type: application/json" \
 -d '{
 "model": "grok-4-1-fast-reasoning",
 "messages": [{"role": "user", "content": "Hello"}]
 }'

Agentgateway automatically rewrites requests to each provider’s chat completion endpoint, so you use a unified request format regardless of the backend provider.

Cleanup

When you’re done, remove everything:

# Remove routes and backends
kubectl delete httproute xai anthropic openai -n agentgateway-system
kubectl delete agentgatewaybackend xai anthropic openai -n agentgateway-system
kubectl delete secret xai-secret anthropic-secret openai-secret -n agentgateway-system
kubectl delete gateway agentgateway-proxy -n agentgateway-system

# Uninstall Helm charts
helm uninstall agentgateway agentgateway-crds -n agentgateway-system

# Delete the Kind cluster
kind delete cluster --name agentgateway-demo

What’s next

Now that you have path-based LLM routing working, there’s a lot more you can do with agentgateway:

Multiple providers on one route — group backends for automatic load balancing and failover. Agentgateway picks two random providers and selects the healthiest one.
Prompt guarding — add AgentgatewayPolicy resources for regex-based prompt filtering or webhook-based validation before requests hit your LLM.
Rate limiting — protect your API keys and budgets with local or remote rate limiting policies.
Observability — enable full OpenTelemetry support for metrics, logs, and distributed tracing across all your LLM traffic.

Check out the agentgateway docs for more, or come chat with us on Discord.

With a single Gateway listener and a few YAML resources, you get a unified, Kubernetes-native control point for all your LLM traffic. That’s the power of agentgateway.

Open Source LLM Observability: Tracing AI Calls with agentgateway and Langfuse

Sat, 14 Feb 2026 00:00:00 +0000

Your AI agents are calling LLMs hundreds of times a day. Do you know what they’re sending? What they’re spending? Whether that prompt injection guard actually fired?

This guide shows how to wire Langfuse into agentgateway to get full observability over every LLM and MCP tool call — zero application code changes required.

Why Gateway-Level Observability?

Most teams add tracing inside their application code — wrapping LLM SDK calls with Langfuse decorators or OpenTelemetry spans. This works, but it has gaps:

You only see what the app reports. If a developer forgets to instrument a call, it’s invisible.
Gateway-level policies are opaque. PII redaction, prompt injection blocking, rate limiting — these happen at the gateway. Application-level tracing can’t see them.
Multiple agents, multiple codebases. Every team has to add their own instrumentation. Different languages, different quality.

When tracing happens at the gateway layer, every request is captured automatically. Every agent, every provider, every tool call — same fidelity, same format, zero code changes.

Architecture

┌──────────────┐ ┌────────────────────────┐ ┌─────────────────┐
│ Your App / │ │ Solo agentgateway │ │ LLM Provider │
│ AI Agent │────▶│ (Gateway API) │────▶│ (OpenAI, etc) │
│ │ │ │ │ │
└──────────────┘ └───────────┬────────────┘ └─────────────────┘
 │
 OTLP Traces (gRPC)
 │
 ┌───────────▼────────────┐
 │ OpenTelemetry │
 │ Collector (fan-out) │
 │ │
 └─────┬───────────┬──────┘
 │ │
 OTLP HTTP OTLP gRPC
 │ │
 ┌─────▼──┐ ┌─────▼──────────┐
 │Langfuse│ │ClickHouse / │
 │ UI │ │ Solo Enterprise │
 │ │ │ UI (optional) │
 └────────┘ └─────────────────┘

agentgateway natively emits OpenTelemetry traces for every LLM request. A lightweight OTel Collector receives these traces and forwards them to Langfuse via OTLP HTTP. The same collector can fan-out traces to additional backends (ClickHouse, Jaeger, Datadog) simultaneously.

Quick Start: Kind Cluster + agentgateway 2.1 OSS

Don’t have a cluster? Here’s the fastest path from zero to traced LLM calls.

Create the Cluster

kind create cluster --name agentgateway

Install Gateway API CRDs

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml

Install agentgateway 2.1 OSS

helm upgrade -i agentgateway-crds oci://cr.agentgateway.dev/helm/agentgateway-crds \
 --version 2.1.0 \
 --namespace agentgateway-system \
 --create-namespace

# Install control plane
helm upgrade -i agentgateway oci://cr.agentgateway.dev/helm/agentgateway \
 --version 2.1.0 \
 --namespace agentgateway-system

Verify it’s running:

kubectl get pods -n agentgateway-system
kubectl get gatewayclass agentgateway

Setting Up Langfuse Tracing

Step 1: Get Langfuse API Keys

Sign up at cloud.langfuse.com (free tier) or use a self-hosted instance. Go to Settings → API Keys and create a key pair.

Base64 encode your credentials:

echo -n "pk-lf-YOUR_PUBLIC_KEY:sk-lf-YOUR_SECRET_KEY" | base64

Step 2: Deploy the OTel Collector

The collector bridges agentgateway’s OTLP gRPC output to Langfuse’s OTLP HTTP input:

apiVersion: v1
kind: ConfigMap
metadata:
 name: langfuse-otel-collector-config
 namespace: agentgateway-system
data:
 config.yaml: |
 receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318
 exporters:
 otlphttp/langfuse:
 endpoint: http://cloud.langfuse.com/api/public/otel
 headers:
 Authorization: "Basic <YOUR_BASE64_CREDENTIALS>"
 retry_on_failure:
 enabled: true
 initial_interval: 5s
 max_interval: 30s
 max_elapsed_time: 300s
 processors:
 batch:
 send_batch_size: 1000
 timeout: 5s
 service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [batch]
 exporters: [otlphttp/langfuse] 
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: langfuse-otel-collector
 namespace: agentgateway-system
 labels:
 app: langfuse-otel-collector
spec:
 replicas: 1
 selector:
 matchLabels:
 app: langfuse-otel-collector
 template:
 metadata:
 labels:
 app: langfuse-otel-collector
 spec:
 containers:
 - name: otel-collector
 image: otel/opentelemetry-collector-contrib:0.132.1
 args: ["--config=/conf/config.yaml"]
 ports:
 - containerPort: 4317
 name: otlp-grpc
 - containerPort: 4318
 name: otlp-http
 volumeMounts:
 - name: config
 mountPath: /conf
 resources:
 requests:
 cpu: 50m
 memory: 128Mi
 limits:
 cpu: 200m
 memory: 256Mi
 volumes:
 - name: config
 configMap:
 name: langfuse-otel-collector-config
---
apiVersion: v1
kind: Service
metadata:
 name: langfuse-otel-collector
 namespace: agentgateway-system
spec:
 selector:
 app: langfuse-otel-collector
 ports:
 - name: otlp-grpc
 port: 4317
 targetPort: 4317
 - name: otlp-http
 port: 4318
 targetPort: 4318

kubectl apply -f langfuse-collector.yaml

Step 3: Configure agentgateway Tracing

For agentgateway OSS, enable tracing via Helm values:

# values-tracing.yaml
gateway:
 envs:
 OTEL_EXPORTER_OTLP_ENDPOINT: "http://langfuse-otel-collector.agentgateway-system.svc.cluster.local:4317"
 OTEL_EXPORTER_OTLP_PROTOCOL: "grpc"

helm upgrade agentgateway oci://cr.agentgateway.dev/helm/agentgateway \
 --version 2.1.0 \
 --namespace agentgateway-system \
 -f values-tracing.yaml

For agentgateway Enterprise, use the EnterpriseAgentgatewayParameters resource instead:

apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayParameters
metadata:
 name: tracing
 namespace: agentgateway-system
spec:
 rawConfig:
 config:
 tracing:
 otlpEndpoint: grpc://langfuse-otel-collector.agentgateway-system.svc.cluster.local:4317
 otlpProtocol: grpc
 randomSampling: true
 fields:
 add:
 gen_ai.operation.name: '"chat"'
 gen_ai.system: "llm.provider"
 gen_ai.request.model: "llm.requestModel"
 gen_ai.response.model: "llm.responseModel"
 gen_ai.usage.prompt_tokens: "llm.inputTokens"
 gen_ai.usage.completion_tokens: "llm.outputTokens"
 gen_ai.usage.total_tokens: "llm.totalTokens"
 gen_ai.request.temperature: "llm.params.temperature"
 gen_ai.prompt: "llm.prompt"
 gen_ai.completion: "llm.completion"

Step 4: Create a Gateway and Route

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: ai-gateway
 namespace: agentgateway-system
spec:
 gatewayClassName: agentgateway
 listeners:
 - name: llm
 port: 8080
 protocol: HTTPS
 allowedRoutes:
 namespaces:
 from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: ai-gateway
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /openai
 backendRefs:
 - group: agentgateway.dev
 kind: AgentgatewayBackend
 name: openai
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: openai
 namespace: agentgateway-system
spec:
 type: llm
 llm:
 provider:
 openai:
 authToken:
 secretRef:
 name: openai-api-key
 namespace: agentgateway-system

# Create the API key secret
kubectl create secret generic openai-api-key \
 -n agentgateway-system \
 --from-literal=Authorization="Bearer $OPENAI_API_KEY"

# Apply the gateway and route
kubectl apply -f gateway.yaml

Step 5: Test and View Traces

# Port-forward the gateway
kubectl port-forward -n agentgateway-system svc/ai-gateway 8080:8080 &

# Send a test request
curl -X POST http://localhost:8080/openai/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "gpt-4.1-mini",
 "messages": [{"role": "user", "content": "Hello from agentgateway!"}]
 }'

Open Langfuse → Traces. You should see a trace with model, tokens, prompt, completion, and gateway metadata.

What Gets Captured

Trace Attributes (GenAI Semantic Conventions)

Attribute	Description	Example
`gen_ai.system`	LLM provider	`openai`
`gen_ai.request.model`	Requested model	`gpt-4.1-mini`
`gen_ai.response.model`	Actual model used	`gpt-4o-mini-2024-07-18`
`gen_ai.usage.prompt_tokens`	Input tokens	`13`
`gen_ai.usage.completion_tokens`	Output tokens	`30`
`gen_ai.usage.total_tokens`	Combined	`43`
`gen_ai.prompt`	Full prompt content	`[{"role":"user","content":"Hello"}]`
`gen_ai.completion`	Full completion	`Hello! How can I help you?`
`gen_ai.streaming`	Streaming used	`false`

Gateway Metadata

Attribute	Description	Example
`gateway`	Gateway resource	`agentgateway-system/ai-gateway`
`route`	HTTPRoute name	`agentgateway-system/openai`
`endpoint`	Backend endpoint	`api.openai.com:443`
`listener`	Gateway listener	`llm`

Multi-Provider Support

agentgateway traces all providers through the same pipeline — OpenAI, Anthropic, xAI/Grok, Azure OpenAI, Google Gemini, Ollama, and any OpenAI-compatible API. Add more routes, same observability:

/openai/* → OpenAI GPT → traced to Langfuse
/anthropic/* → Anthropic → traced to Langfuse
/xai/* → xAI Grok → traced to Langfuse

Every provider, same trace format, same dashboard.

Fan-Out: Langfuse + Additional Backends

Send traces to multiple backends simultaneously:

exporters:
 otlphttp/langfuse:
 endpoint: http://cloud.langfuse.com/api/public/otel
 headers:
 Authorization: "Basic <CREDENTIALS>"
 otlp/jaeger:
 endpoint: jaeger-collector:4317
 tls:
 insecure: true

service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [batch]
 exporters: [otlphttp/langfuse, otlp/jaeger]

Common fan-out targets: Langfuse (LLM analytics) + ClickHouse (gateway metrics) + Jaeger (distributed tracing) + Datadog (enterprise monitoring).

MCP Tool Tracing

agentgateway doesn’t just trace LLM calls — it also traces MCP (Model Context Protocol) tool interactions:

Tool discovery (tools/list) — which tools are available, how long discovery takes
Tool execution (tools/call) — parameters, results, latency
Backend MCP server performance — per-server latency and error rates

When an agent calls Slack, GitHub, or any MCP tool server through agentgateway, the full tool call chain appears in Langfuse alongside the LLM calls that triggered it.

Security Policy Visibility

When agentgateway’s security policies fire, the trace metadata includes what happened:

PII Protection — how many entities were redacted, what types (email, SSN, phone)
Prompt Injection — whether an injection was detected and blocked
Credential Leak — whether secrets were caught in the LLM response
Rate Limiting — remaining quota for the user

This means you can see not just what your agents are doing, but what guardrails are protecting them.

Troubleshooting

Problem	Fix
No traces in Langfuse	Check collector pod is running: `kubectl get pods -n agentgateway-system -l app=langfuse-otel-collector`
401 errors in collector logs	Wrong Langfuse API credentials — re-check the base64 encoding
Traces show but no prompt/completion	Add the `fields.add` section in Enterprise, or check OTEL env vars in OSS
Missing gateway metadata	Restart proxies after config change: `kubectl rollout restart deployment -n agentgateway-system`

Full Source Code

All manifests, configs, and examples are in the agentgateway-langfuse repository:

Path	Description
docs/quickstart-kind.md	Full kind cluster quickstart guide
examples/basic/	Basic Langfuse collector + tracing config
examples/fan-out/	Fan-out to Langfuse + additional backends
examples/argocd/	Production ArgoCD/GitOps deployment
scripts/verify.sh	End-to-end verification script

Resources

agentgateway OSS — CNCF open-source AI gateway
agentgateway Helm Install
Langfuse — Open-source LLM observability
Langfuse OpenTelemetry Docs
OpenTelemetry GenAI Conventions

Your First AI Route: Connecting to OpenAI with agentgateway

Wed, 11 Feb 2026 00:00:00 +0000

Introduction

This is a how-to guide to setup agentgateway and get your first AI route working with OpenAI. We’ll walk through the complete setup from scratch - creating a Kubernetes cluster, installing agentgateway, and connecting it to OpenAI’s API.

What is agentgateway?

agentgateway is an open source, AI-native data plane built in Rust for connecting, securing, and observing AI traffic. Originally created by Solo.io and now a Linux Foundation project, it acts as a purpose-built proxy layer between your applications and AI services like LLMs, MCP tool servers, and other AI agents.

Traditional API gateways don’t fit AI workloads — AI inference requests are long-running (minutes vs milliseconds), have larger payloads, and can consume entire GPUs, unlike standard web traffic.
Connectivity — Unified interface to route requests to LLM providers (OpenAI, Anthropic, Bedrock, etc.), self-hosted models, and MCP tool servers.
Security — Built-in auth, RBAC, and secrets management for API keys and sensitive data.
Observability — Automatic token counting, cost tracking, and OpenTelemetry-compatible structured logs.
MCP support — Can federate multiple MCP servers behind a single endpoint, and expose legacy REST APIs as MCP tools via OpenAPI integration.
A2A support — Native Agent-to-Agent protocol for secure inter-agent communication..

In this tutorial, we’ll focus on one of agentgateway’s most common use cases: routing requests to an LLM provider (OpenAI) with secure credential management and built-in cost observability.

What You’ll Learn

Create a Kubernetes cluster and install agentgateway
Set up secure OpenAI API key storage
Configure agentgateway to route to OpenAI
Test chat completions, embeddings, and model listings
Monitor real AI requests and track costs
Troubleshoot common issues

Prerequisites

Docker installed and running
kubectl CLI tool
Helm 3.x installed
Valid OpenAI API Key with credits (get from OpenAI Platform)
Basic understanding of Kubernetes and OpenAI API structure

Step 1: Environment Setup

In this step, we’ll create a local Kubernetes cluster using kind (Kubernetes in Docker) and install agentgateway. This gives us a complete testing environment that mirrors production setups but runs entirely on your local machine.

Install Kind

Kind creates Kubernetes clusters using Docker containers as nodes. This is perfect for development and testing because it’s lightweight, fast to spin up, and doesn’t require cloud resources.

brew install kind

# On Linux
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.22.0/kind-linux-amd64
chmod +x ./kind && sudo mv ./kind /usr/local/bin/kind

Create Kind Cluster

This creates a single-node Kubernetes cluster that will host our agentgateway installation. The cluster provides the foundation for all the networking, security, and routing capabilities we’ll configure.

# Create the cluster
kind create cluster --name agentgateway

# Verify cluster is ready
kubectl get nodes

Install agentgateway

agentgateway installation happens in three phases: First we install the Kubernetes Gateway API (the standard for ingress traffic), then agentgateway’s custom resources, and finally the control plane that manages everything. This separation allows for better modularity and easier upgrades.

# 1. Install Gateway API CRDs (version 1.4.0)
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml

# 2. Install agentgateway CRDs
helm upgrade -i --create-namespace \
 --namespace agentgateway-system \
 --version v2.2.0 agentgateway-crds \
 oci://ghcr.io/kgateway-dev/charts/agentgateway-crds

# 3. Install agentgateway control plane
helm upgrade -i -n agentgateway-system agentgateway \
 oci://ghcr.io/kgateway-dev/charts/agentgateway \
 --version v2.2.0

# 4. Verify installation
kubectl get pods -n agentgateway-system

Step 2: OpenAI API Key Setup

Security is paramount when working with AI services. Instead of embedding API keys directly in configurations, we’ll use Kubernetes secrets to store credentials securely. This approach ensures keys are encrypted at rest and can be rotated without changing application code.

Get Your OpenAI API Key

OpenAI uses API keys for authentication and billing. Each key is tied to your account and usage limits, making it essential to secure them properly.

Visit OpenAI Platform
Navigate to API Keys section and create a new key
Set usage limits to control costs
Copy your API key securely

Test Your API Key

Before integrating with agentgateway, we’ll verify the API key works directly with OpenAI’s API. This eliminates the key as a potential issue if something goes wrong later in the setup.

# Set your OpenAI API key (replace with your actual key)
export OPENAI_API_KEY="sk-your-openai-api-key-here"

# Test the key directly
curl -s "https://api.openai.com/v1/models" \
 -H "Authorization: Bearer $OPENAI_API_KEY" \
 | jq '.data[0:3] | .[].id'

Create Kubernetes Secret

Kubernetes secrets provide a secure way to store sensitive data like API keys. We format the key as a complete Authorization header (Bearer sk-...) so agentgateway can use it directly without modification. The --dry-run=client -o yaml | kubectl apply -f - pattern ensures the secret is created safely even if it already exists.

# Create secret with proper authorization header format
kubectl create secret generic openai-secret \
 -n agentgateway-system \
 --from-literal="Authorization=Bearer $OPENAI_API_KEY" \
 --dry-run=client -o yaml | kubectl apply -f -

# Verify secret creation
kubectl get secret openai-secret -n agentgateway-system

Step 3: Configure agentgateway

Now we’ll configure the core components that make AI routing work. agentgateway follows the Kubernetes Gateway API pattern with three main resources: Gateway (the entry point), Backends (destination services), and HTTPRoutes (traffic routing rules). This declarative approach makes configurations version-controllable and environment-portable.

Create Gateway Resource

The Gateway resource defines the entry point for all incoming traffic. It specifies which ports to listen on, what protocols to accept, and which namespaces can create routes through it. Think of it as the front door to your AI services.

kubectl apply -f- <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: agentgateway-proxy
 namespace: agentgateway-system
spec:
 gatewayClassName: agentgateway
 listeners:
 - protocol: HTTP
 port: 8080
 name: http
 allowedRoutes:
 namespaces:
 from: All
EOF

Create OpenAI Backend

AgentgatewayBackend resources define how to connect to AI services. The ai.provider.openai section tells agentgateway this is an AI service that expects OpenAI-compatible requests. The authentication policy references our secret, and the timeout ensures long-running AI requests don’t hang indefinitely.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: openai-backend
 namespace: agentgateway-system
spec:
 ai:
 provider:
 openai:
 model: gpt-4o-mini
 policies:
 auth:
 secretRef:
 name: openai-secret
EOF

Create HTTP Routes

Create an HTTPRoute resource that routes incoming traffic to the AgentgatewayBackend. The following example sets up a route. Note that agentgateway automatically rewrites the endpoint to the OpenAI /v1/chat/completions endpoint.

Create the HTTP routes:

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai-chat
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: agentgateway-proxy
 namespace: agentgateway-system
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /openai
 backendRefs:
 - name: openai-backend
 namespace: agentgateway-system
 group: agentgateway.dev
 kind: AgentgatewayBackend
EOF

Verify Configuration

Before testing, we’ll check that all our resources are properly created and accepted by the agentgateway controller. The Accepted status indicates that configurations are valid and the controller can proceed with implementation.

# Check both backends
kubectl get agentgatewaybackend -n agentgateway-system

# Check routes
kubectl get httproute -n agentgateway-system

# Check Gateway
kubectl get gateway agentgateway-proxy -n agentgateway-system

Step 4: Testing Your Setup

With our configuration complete, it’s time to test the AI routing. Since we’re using a local kind cluster, we’ll use port-forwarding to access the Gateway service. In production, this would be handled by a LoadBalancer or Ingress controller.

Setup Port-Forward

Port-forwarding creates a tunnel from your local machine to the agentgateway service inside the Kubernetes cluster. This lets us test the setup without exposing services publicly.

# Port-forward agentgateway service in background
kubectl port-forward -n agentgateway-system svc/agentgateway-proxy 8080:8080 &

Test Chat Completions

This is where the magic happens! Our request travels through agentgateway, gets authenticated using our secret, routed to OpenAI’s API, and returns with a complete AI response. Notice how the response includes token usage information that agentgateway automatically captures for cost tracking and observability.

# Test basic chat completion
curl -i "localhost:8080/openai/chat/completions" \
 -H "content-type: application/json" \
 -d '{
 "model": "gpt-4o-mini",
 "messages": [
 {
 "role": "user",
 "content": "What are the key benefits of using an AI Gateway?"
 }
 ],
 "max_tokens": 100
 }'

Expected Response:

{
 "id": "chatcmpl-abc123def456",
 "object": "chat.completion",
 "created": 1701234567,
 "model": "gpt-4o-mini-2024-07-18",
 "choices": [
 {
 "index": 0,
 "message": {
 "role": "assistant",
 "content": "An AI Gateway provides unified access to multiple AI providers, centralized security and authentication, comprehensive observability and cost tracking, rate limiting and quotas, and improved reliability through failover and retry mechanisms."
 },
 "finish_reason": "stop"
 }
 ],
 "usage": {
 "prompt_tokens": 15,
 "completion_tokens": 35,
 "total_tokens": 50
 }
}

Test Different Models

agentgateway allows you to easily switch between different OpenAI models by simply changing the model parameter. The backend automatically routes to the appropriate model while maintaining consistent authentication and observability.

# Test with GPT-4o
curl -s "localhost:8080/openai/chat/completions" \
 -H "content-type: application/json" \
 -d '{
 "model": "gpt-4o",
 "messages": [
 {
 "role": "user",
 "content": "Explain agentgateway in one sentence."
 }
 ],
 "max_tokens": 50
 }' | jq '.choices[0].message.content'

Troubleshooting

When working with distributed systems like Kubernetes and external APIs, issues can arise at multiple layers. This section covers the most common problems you might encounter and how to systematically diagnose them. The key is to test each layer independently: network connectivity, authentication, resource configuration, and API compatibility.

Common Issues

1. Service Not Found Error: This usually means the service name doesn’t match what was actually created during installation. Different agentgateway versions or installation methods may create services with different names.

# Check what services exist
kubectl get svc -n agentgateway-system

# If agentgateway-proxy doesn't exist, use the correct service name
kubectl get svc -n agentgateway-system | grep -i gateway

2. Authentication Errors (401): Authentication failures typically indicate either an invalid API key or incorrect secret formatting. Always test the key directly with OpenAI before troubleshooting agentgateway.

# Verify secret exists
kubectl get secret openai-secret -n agentgateway-system -o yaml

# Test API key directly
curl -s "https://api.openai.com/v1/models" \
 -H "Authorization: Bearer $OPENAI_API_KEY" | jq '.data[0].id'

3. Routes Not Working: Route issues often stem from mismatched resource names or namespaces. The Gateway, HTTPRoute, and Backend must all reference each other correctly for traffic to flow.

# Check backend status
kubectl describe agentgatewaybackend openai-backend -n agentgateway-system

# Check route status
kubectl describe httproute openai-chat -n agentgateway-system

# Check Gateway status
kubectl describe gateway agentgateway-proxy -n agentgateway-system

4. Port-Forward Issues: Port conflicts are common on development machines. If port 8080 is busy, either stop the conflicting service or use a different port.

# Check if port 8080 is in use
lsof -i :8080

# Try a different port
kubectl port-forward -n agentgateway-system svc/agentgateway-proxy 8081:8080 &
export GATEWAY_PORT="8081"

Debug Commands

# View all agentgateway resources
kubectl get agentgatewaybackend,gateway,httproute -n agentgateway-system

# Check pod logs for errors
kubectl logs deploy/agentgateway -n agentgateway-system --tail=20

# Test connectivity from inside cluster
kubectl exec -n agentgateway-system deploy/agentgateway -- \
 curl -v https://api.openai.com/v1/models \
 -H "Authorization: Bearer $OPENAI_API_KEY"

Cleanup

When you’re done experimenting, it’s important to clean up resources to free up system resources and avoid any potential costs. The cleanup process should happen in reverse order: stop network connections first, then remove application resources, and finally remove infrastructure.

Stop Port-Forward

# Kill the port-forward process
kill $PORTFORWARD_PID

Remove Resources (Optional)

This removes all the agentgateway configuration we created, but leaves the agentgateway installation intact for future experiments. Remove resources in dependency order: routes first (they reference backends), then backends, then the Gateway.

# Remove all OpenAI configuration
kubectl delete httproute openai-chat openai-models -n agentgateway-system
kubectl delete agentgatewaybackend openai-backend openai-models-backend -n agentgateway-system
kubectl delete gateway agentgateway-proxy -n agentgateway-system
kubectl delete secret openai-secret -n agentgateway-system

Remove Kind Cluster (Optional)

This completely removes the Kubernetes cluster and all associated resources. Only do this if you’re completely done with the tutorial, as you’ll need to recreate everything from Step 1 to run it again.

# Delete the entire cluster
kind delete cluster --name agentgateway

Next Steps

Now that you have a working AI gateway, you can build on this foundation to create production-ready AI infrastructure:

Add more providers - Configure Anthropic, AWS Bedrock, or Azure OpenAI for multi-provider setups and failover scenarios
Implement security - Add rate limiting, authentication, and guardrails to protect against abuse and unexpected costs
Set up monitoring - Configure Grafana dashboards and alerting to track performance, costs, and usage patterns across teams
Explore advanced routing - Implement path-based, header-based, and weighted routing to direct different types of requests to optimal models

Key Takeaways

This tutorial demonstrates several important concepts for production AI systems:

agentgateway provides a unified interface to AI providers with minimal overhead, making it easy to switch providers or implement failover
Proper secret management is essential for production deployments - never embed API keys in code or configuration files
Built-in observability gives immediate insights into costs and performance without requiring additional tooling or instrumentation
The Gateway API pattern makes routing configuration declarative and portable across different Kubernetes environments
Dual backend types (AI-aware vs static HTTP) allow you to handle both complex AI workloads and simple metadata requests efficiently
Kind clusters are perfect for local development and testing, providing a production-like environment without cloud costs

Your agentgateway is now successfully routing requests to OpenAI with enterprise-grade security, observability, and cost control! You’ve built a foundation that can scale from development to production while maintaining visibility and control over your AI infrastructure. 🎯

Advanced Routing Patterns for AI Models with agentgateway

Mon, 09 Feb 2026 00:00:00 +0000

Introduction

One of the most powerful features of agentgateway is its ability to intelligently route requests to different AI models or providers based on various criteria. This enables sophisticated scenarios like model selection based on request characteristics, A/B testing different models, and cost optimization through intelligent routing.

In this guide, we’ll explore multiple routing patterns that transform your agentgateway from a simple proxy into an intelligent AI traffic management system.

What You’ll Learn

Path-based routing for different models and use cases
Header-based routing for tenant isolation and testing
Query parameter routing for flexible client control
Weighted routing for A/B testing and gradual rollouts
Content-based routing using request body analysis
Fallback and failover routing strategies
Cost-optimized routing patterns

Prerequisites

Completed previous blog posts (agentgateway setup, observability, OpenAI integration)
Understanding of Kubernetes Gateway API concepts
Knowledge of HTTP routing principles
OpenAI API key for testing (we’ll add Anthropic optionally)

Environment Setup

Ensure your environment is ready:

# Verify environment variables
export OPENAI_API_KEY="sk-your-openai-api-key-here"
export SOLO_TRIAL_LICENSE_KEY="your-license-key-here"

# Verify agentgateway is running
kubectl get pods -n enterprise-agentgateway

# Verify existing OpenAI configuration
kubectl get agentgatewaybackend openai-all-models -n enterprise-agentgateway

Pattern 1: Path-Based Routing

Model-Specific Paths

Create different paths for different models, allowing clients to choose the right model for their use case:

kubectl apply -f- <<'EOF'
# GPT-4o Mini route (fast, cost-effective)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai-gpt4o-mini
 namespace: enterprise-agentgateway
 labels:
 provider: openai
 model: gpt-4o-mini
 use-case: general
spec:
 parentRefs:
 - name: agentgateway
 namespace: enterprise-agentgateway
 
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /ai/fast
 - path:
 type: PathPrefix
 value: /ai/gpt4o-mini
 
 backendRefs:
 - name: openai-gpt4o-mini
 group: agentgateway.dev
 kind: AgentgatewayBackend
 
 timeouts:
 request: "60s"
 
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplacePrefixMatch
 replacePrefixMatch: /v1/chat/completions
 # Add model override header
 - type: ResponseHeaderModifier
 responseHeaderModifier:
 add:
 - name: X-Model-Used
 value: gpt-4o-mini
 - name: X-Use-Case
 value: fast-general

---
# GPT-4o route (premium quality)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai-gpt4o
 namespace: enterprise-agentgateway
 labels:
 provider: openai
 model: gpt-4o
 use-case: premium
spec:
 parentRefs:
 - name: agentgateway
 namespace: enterprise-agentgateway
 
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /ai/premium
 - path:
 type: PathPrefix
 value: /ai/gpt4o
 
 backendRefs:
 - name: openai-gpt4o
 group: agentgateway.dev
 kind: AgentgatewayBackend
 
 timeouts:
 request: "120s"
 
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplacePrefixMatch
 replacePrefixMatch: /v1/chat/completions
 - type: ResponseHeaderModifier
 responseHeaderModifier:
 add:
 - name: X-Model-Used
 value: gpt-4o
 - name: X-Use-Case
 value: premium-quality

---
# Embeddings route
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai-embeddings-route
 namespace: enterprise-agentgateway
 labels:
 provider: openai
 endpoint: embeddings
spec:
 parentRefs:
 - name: agentgateway
 namespace: enterprise-agentgateway
 
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /ai/embeddings
 
 backendRefs:
 - name: openai-all-models
 group: agentgateway.dev
 kind: AgentgatewayBackend
 
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplacePrefixMatch
 replacePrefixMatch: /v1/embeddings
 - type: ResponseHeaderModifier
 responseHeaderModifier:
 add:
 - name: X-Service-Type
 value: embeddings
EOF

Create Model-Specific Backends

kubectl apply -f- <<'EOF'
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: openai-gpt4o-mini
 namespace: enterprise-agentgateway
spec:
 ai:
 provider:
 openai:
 config:
 model: "gpt-4o-mini" # Force specific model
 policies:
 auth:
 secretRef:
 name: openai-secret
 timeout:
 request: "60s"

---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: openai-gpt4o
 namespace: enterprise-agentgateway
spec:
 ai:
 provider:
 openai:
 config:
 model: "gpt-4o" # Force premium model
 policies:
 auth:
 secretRef:
 name: openai-secret
 timeout:
 request: "120s"
EOF

Test Path-Based Routing

export GATEWAY_IP="${GATEWAY_IP:-localhost}"
export GATEWAY_PORT="${GATEWAY_PORT:-8080}"

# Test fast/general route (gpt-4o-mini)
echo "=== Testing Fast Route (GPT-4o Mini) ==="
curl -s "$GATEWAY_IP:$GATEWAY_PORT/ai/fast/completions" \
 -H "content-type: application/json" \
 -d '{
 "messages": [
 {
 "role": "user",
 "content": "What model are you and what are your strengths?"
 }
 ],
 "max_tokens": 100
 }' | jq '{
 model: .model,
 content: .choices[0].message.content,
 usage: .usage
 }'

echo ""
echo "=== Testing Premium Route (GPT-4o) ==="
curl -s "$GATEWAY_IP:$GATEWAY_PORT/ai/premium/completions" \
 -H "content-type: application/json" \
 -d '{
 "messages": [
 {
 "role": "user", 
 "content": "What model are you and what are your strengths?"
 }
 ],
 "max_tokens": 100
 }' | jq '{
 model: .model,
 content: .choices[0].message.content,
 usage: .usage
 }'

Pattern 2: Header-Based Routing

Tenant Isolation and Testing

Use headers to route requests to different models or providers based on client characteristics:

kubectl apply -f- <<'EOF'
# Development environment route
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai-dev-environment
 namespace: enterprise-agentgateway
 labels:
 environment: development
spec:
 parentRefs:
 - name: agentgateway
 namespace: enterprise-agentgateway
 
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /ai/chat
 headers:
 - name: X-Environment
 value: development
 - path:
 type: PathPrefix
 value: /ai/chat
 headers:
 - name: X-Environment
 value: dev
 
 backendRefs:
 - name: openai-gpt4o-mini # Use cheaper model for dev
 group: agentgateway.dev
 kind: AgentgatewayBackend
 
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplacePrefixMatch
 replacePrefixMatch: /v1/chat/completions
 - type: ResponseHeaderModifier
 responseHeaderModifier:
 add:
 - name: X-Environment-Used
 value: development
 - name: X-Model-Tier
 value: economy

---
# Production environment route 
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai-prod-environment
 namespace: enterprise-agentgateway
 labels:
 environment: production
spec:
 parentRefs:
 - name: agentgateway
 namespace: enterprise-agentgateway
 
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /ai/chat
 headers:
 - name: X-Environment
 value: production
 - path:
 type: PathPrefix
 value: /ai/chat
 headers:
 - name: X-Environment
 value: prod
 
 backendRefs:
 - name: openai-gpt4o # Use premium model for prod
 group: agentgateway.dev
 kind: AgentgatewayBackend
 
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplacePrefixMatch
 replacePrefixMatch: /v1/chat/completions
 - type: ResponseHeaderModifier
 responseHeaderModifier:
 add:
 - name: X-Environment-Used
 value: production
 - name: X-Model-Tier
 value: premium

---
# A/B Testing route
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai-ab-test
 namespace: enterprise-agentgateway
 labels:
 purpose: ab-testing
spec:
 parentRefs:
 - name: agentgateway
 namespace: enterprise-agentgateway
 
 rules:
 # Variant A - GPT-4o Mini
 - matches:
 - path:
 type: PathPrefix
 value: /ai/chat
 headers:
 - name: X-AB-Test
 value: variant-a
 
 backendRefs:
 - name: openai-gpt4o-mini
 group: agentgateway.dev
 kind: AgentgatewayBackend
 
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplacePrefixMatch
 replacePrefixMatch: /v1/chat/completions
 - type: ResponseHeaderModifier
 responseHeaderModifier:
 add:
 - name: X-AB-Variant
 value: variant-a
 - name: X-Model-Used
 value: gpt-4o-mini
 
 # Variant B - GPT-4o
 - matches:
 - path:
 type: PathPrefix
 value: /ai/chat
 headers:
 - name: X-AB-Test
 value: variant-b
 
 backendRefs:
 - name: openai-gpt4o
 group: agentgateway.dev
 kind: AgentgatewayBackend
 
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplacePrefixMatch
 replacePrefixMatch: /v1/chat/completions
 - type: ResponseHeaderModifier
 responseHeaderModifier:
 add:
 - name: X-AB-Variant
 value: variant-b
 - name: X-Model-Used
 value: gpt-4o
EOF

Test Header-Based Routing

# Test development environment routing
echo "=== Testing Development Environment ==="
curl -s "$GATEWAY_IP:$GATEWAY_PORT/ai/chat/completions" \
 -H "content-type: application/json" \
 -H "X-Environment: development" \
 -d '{
 "messages": [
 {"role": "user", "content": "Hello from development!"}
 ],
 "max_tokens": 50
 }' | jq '{
 model: .model,
 content: .choices[0].message.content
 }'

echo ""
echo "=== Testing Production Environment ==="
curl -s "$GATEWAY_IP:$GATEWAY_PORT/ai/chat/completions" \
 -H "content-type: application/json" \
 -H "X-Environment: production" \
 -d '{
 "messages": [
 {"role": "user", "content": "Hello from production!"}
 ],
 "max_tokens": 50
 }' | jq '{
 model: .model,
 content: .choices[0].message.content
 }'

echo ""
echo "=== Testing A/B Test Variant A ==="
curl -i "$GATEWAY_IP:$GATEWAY_PORT/ai/chat/completions" \
 -H "content-type: application/json" \
 -H "X-AB-Test: variant-a" \
 -d '{
 "messages": [
 {"role": "user", "content": "A/B test message"}
 ],
 "max_tokens": 30
 }' | grep -E "(X-AB-Variant|X-Model-Used)"

echo ""
echo "=== Testing A/B Test Variant B ==="
curl -i "$GATEWAY_IP:$GATEWAY_PORT/ai/chat/completions" \
 -H "content-type: application/json" \
 -H "X-AB-Test: variant-b" \
 -d '{
 "messages": [
 {"role": "user", "content": "A/B test message"}
 ],
 "max_tokens": 30
 }' | grep -E "(X-AB-Variant|X-Model-Used)"

Pattern 3: Query Parameter Routing

Flexible Client-Side Model Selection

Allow clients to specify routing preferences via query parameters:

kubectl apply -f- <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai-query-param-routing
 namespace: enterprise-agentgateway
 labels:
 routing-type: query-parameter
spec:
 parentRefs:
 - name: agentgateway
 namespace: enterprise-agentgateway
 
 rules:
 # Route for speed preference
 - matches:
 - path:
 type: PathPrefix
 value: /ai/flexible
 queryParams:
 - name: speed
 value: fast
 
 backendRefs:
 - name: openai-gpt4o-mini
 group: agentgateway.dev
 kind: AgentgatewayBackend
 
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplacePrefixMatch
 replacePrefixMatch: /v1/chat/completions
 - type: ResponseHeaderModifier
 responseHeaderModifier:
 add:
 - name: X-Speed-Optimized
 value: "true"
 
 # Route for quality preference
 - matches:
 - path:
 type: PathPrefix
 value: /ai/flexible
 queryParams:
 - name: quality
 value: premium
 
 backendRefs:
 - name: openai-gpt4o
 group: agentgateway.dev
 kind: AgentgatewayBackend
 
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplacePrefixMatch
 replacePrefixMatch: /v1/chat/completions
 - type: ResponseHeaderModifier
 responseHeaderModifier:
 add:
 - name: X-Quality-Optimized
 value: "true"
 
 # Route for cost preference
 - matches:
 - path:
 type: PathPrefix
 value: /ai/flexible
 queryParams:
 - name: cost
 value: minimal
 
 backendRefs:
 - name: openai-gpt4o-mini
 group: agentgateway.dev
 kind: AgentgatewayBackend
 
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplacePrefixMatch
 replacePrefixMatch: /v1/chat/completions
 - type: ResponseHeaderModifier
 responseHeaderModifier:
 add:
 - name: X-Cost-Optimized
 value: "true"
EOF

Test Query Parameter Routing

# Test speed-optimized routing
echo "=== Testing Speed-Optimized Routing ==="
curl -s "$GATEWAY_IP:$GATEWAY_PORT/ai/flexible/completions?speed=fast" \
 -H "content-type: application/json" \
 -d '{
 "messages": [
 {"role": "user", "content": "Fast response needed!"}
 ],
 "max_tokens": 50
 }' | jq '{model: .model, usage: .usage}'

# Test quality-optimized routing 
echo "=== Testing Quality-Optimized Routing ==="
curl -s "$GATEWAY_IP:$GATEWAY_PORT/ai/flexible/completions?quality=premium" \
 -H "content-type: application/json" \
 -d '{
 "messages": [
 {"role": "user", "content": "High quality response needed!"}
 ],
 "max_tokens": 50
 }' | jq '{model: .model, usage: .usage}'

# Test cost-optimized routing
echo "=== Testing Cost-Optimized Routing ==="
curl -s "$GATEWAY_IP:$GATEWAY_PORT/ai/flexible/completions?cost=minimal" \
 -H "content-type: application/json" \
 -d '{
 "messages": [
 {"role": "user", "content": "Cost-effective response needed!"}
 ],
 "max_tokens": 50
 }' | jq '{model: .model, usage: .usage}'

Pattern 4: Weighted Routing for Gradual Rollouts

Implement Traffic Splitting

Use multiple backends with different weights for gradual model rollouts:

kubectl apply -f- <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai-weighted-routing
 namespace: enterprise-agentgateway
 labels:
 routing-type: weighted
spec:
 parentRefs:
 - name: agentgateway
 namespace: enterprise-agentgateway
 
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /ai/gradual-rollout
 
 # Split traffic between models
 backendRefs:
 # 80% traffic to stable model
 - name: openai-gpt4o-mini
 group: agentgateway.dev
 kind: AgentgatewayBackend
 weight: 80
 # 20% traffic to new model for testing
 - name: openai-gpt4o
 group: agentgateway.dev
 kind: AgentgatewayBackend
 weight: 20
 
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplacePrefixMatch
 replacePrefixMatch: /v1/chat/completions
 - type: ResponseHeaderModifier
 responseHeaderModifier:
 add:
 - name: X-Routing-Type
 value: weighted-rollout
EOF

Test Weighted Routing

cat <<'EOF' > test-weighted-routing.sh
#!/bin/bash

GATEWAY_IP="${GATEWAY_IP:-localhost}"
GATEWAY_PORT="${GATEWAY_PORT:-8080}"
REQUESTS=${1:-20}

echo "Testing weighted routing with $REQUESTS requests..."
echo "Expected: ~80% gpt-4o-mini, ~20% gpt-4o"
echo ""

declare -A model_counts

for ((i=1; i<=REQUESTS; i++)); do
 response=$(curl -s "$GATEWAY_IP:$GATEWAY_PORT/ai/gradual-rollout/completions" \
 -H "content-type: application/json" \
 -d '{
 "messages": [
 {"role": "user", "content": "What model are you?"}
 ],
 "max_tokens": 10
 }')
 
 model=$(echo "$response" | jq -r '.model')
 ((model_counts[$model]++))
 
 printf "Request %2d: %s\n" $i "$model"
done

echo ""
echo "=== Results ==="
for model in "${!model_counts[@]}"; do
 count=${model_counts[$model]}
 percentage=$(( count * 100 / REQUESTS ))
 echo "$model: $count requests (${percentage}%)"
done
EOF

chmod +x test-weighted-routing.sh
./test-weighted-routing.sh 10

Pattern 5: Content-Based Routing

Route Based on Request Content

Use agentgateway’s request analysis capabilities to route based on content characteristics:

kubectl apply -f- <<'EOF'
apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayPolicy
metadata:
 name: content-based-routing
 namespace: enterprise-agentgateway
spec:
 targetRefs:
 - group: gateway.networking.k8s.io
 kind: Gateway
 name: agentgateway
 traffic:
 # Simple content analysis - route long requests to premium model
 requestTransformation:
 inline:
 # Add request length as header for routing decisions
 headers:
 add:
 X-Content-Length: 'string(len(json(request.body).messages[0].content))'
 X-Content-Type: 'if len(json(request.body).messages[0].content) > 100 then "long" else "short"'

---
# Route short content to fast model
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai-short-content
 namespace: enterprise-agentgateway
 labels:
 routing-type: content-based
 content-type: short
spec:
 parentRefs:
 - name: agentgateway
 namespace: enterprise-agentgateway
 
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /ai/smart
 headers:
 - name: X-Content-Type
 value: short
 
 backendRefs:
 - name: openai-gpt4o-mini
 group: agentgateway.dev
 kind: AgentgatewayBackend
 
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplacePrefixMatch
 replacePrefixMatch: /v1/chat/completions
 - type: ResponseHeaderModifier
 responseHeaderModifier:
 add:
 - name: X-Content-Routing
 value: short-content-fast-model

---
# Route long content to premium model 
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai-long-content
 namespace: enterprise-agentgateway
 labels:
 routing-type: content-based
 content-type: long
spec:
 parentRefs:
 - name: agentgateway
 namespace: enterprise-agentgateway
 
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /ai/smart
 headers:
 - name: X-Content-Type
 value: long
 
 backendRefs:
 - name: openai-gpt4o
 group: agentgateway.dev
 kind: AgentgatewayBackend
 
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplacePrefixMatch
 replacePrefixMatch: /v1/chat/completions
 - type: ResponseHeaderModifier
 responseHeaderModifier:
 add:
 - name: X-Content-Routing
 value: long-content-premium-model
EOF

Test Content-Based Routing

# Test with short content
echo "=== Testing Short Content (should use gpt-4o-mini) ==="
curl -s "$GATEWAY_IP:$GATEWAY_PORT/ai/smart/completions" \
 -H "content-type: application/json" \
 -d '{
 "messages": [
 {"role": "user", "content": "Hi!"}
 ],
 "max_tokens": 20
 }' | jq '{model: .model, usage: .usage}'

echo ""
echo "=== Testing Long Content (should use gpt-4o) ==="
curl -s "$GATEWAY_IP:$GATEWAY_PORT/ai/smart/completions" \
 -H "content-type: application/json" \
 -d '{
 "messages": [
 {
 "role": "user", 
 "content": "This is a much longer prompt that contains significantly more content and should trigger the routing logic to use the premium model because it requires more sophisticated processing and understanding. The content-based routing should detect this as a long request and route it to GPT-4o for better handling of complex queries that require more nuanced responses and deeper analysis."
 }
 ],
 "max_tokens": 50
 }' | jq '{model: .model, usage: .usage}'

Observing Routing Patterns

Create Routing Dashboard

cat <<'EOF' > routing-analysis.sh
#!/bin/bash

echo "Analyzing routing patterns from agentgateway logs..."
echo ""

# Extract routing information from recent logs
kubectl logs deploy/agentgateway -n enterprise-agentgateway --tail=100 | \
 jq -r 'select(.gen_ai) | [
 .timestamp,
 .route_name // "unknown",
 .gen_ai.request.model,
 (.gen_ai.usage.total_tokens // 0),
 (.request.headers["x-environment"] // "none"),
 (.request.headers["x-ab-test"] // "none")
 ] | @csv' | \
 awk -F',' '
BEGIN {
 print "Route Analysis Report"
 print "===================="
 print ""
 total_requests = 0
 split("", route_counts)
 split("", model_counts)
 split("", env_counts)
}
{
 total_requests++
 
 # Remove quotes
 route = $2
 model = $3
 tokens = $4
 env = $5
 
 gsub(/"/, "", route)
 gsub(/"/, "", model) 
 gsub(/"/, "", env)
 
 route_counts[route]++
 model_counts[model]++
 if (env != "none") env_counts[env]++
 
 total_tokens += tokens
}
END {
 print "Total Requests: " total_requests
 print "Total Tokens: " total_tokens
 print ""
 
 print "Routes:"
 for (route in route_counts) {
 printf " %s: %d requests\n", route, route_counts[route]
 }
 print ""
 
 print "Models:"
 for (model in model_counts) {
 printf " %s: %d requests\n", model, model_counts[model]
 }
 print ""
 
 print "Environments:"
 for (env in env_counts) {
 printf " %s: %d requests\n", env, env_counts[env]
 }
}'
EOF

chmod +x routing-analysis.sh
./routing-analysis.sh

Monitor Routing in Grafana

Open Grafana Dashboard:

kubectl port-forward -n monitoring svc/grafana-prometheus 3000:3000 &

Create Custom Routing Queries:

# Request rate by route
sum(rate(agentgateway_requests_total[5m])) by (route_name)

# Request rate by model
sum(rate(agentgateway_requests_total[5m])) by (model)

# Token usage by routing pattern
sum(rate(agentgateway_tokens_total[5m])) by (route_name, token_type)

Watch routing patterns as you send test requests through different patterns

Pattern 6: Fallback and Failover Routing

Primary-Secondary Backend Configuration

kubectl apply -f- <<'EOF'
# Primary backend with health checking
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: openai-primary
 namespace: enterprise-agentgateway
spec:
 ai:
 provider:
 openai:
 config:
 model: "gpt-4o"
 policies:
 auth:
 secretRef:
 name: openai-secret
 # Health checking configuration
 healthCheck:
 interval: "30s"
 timeout: "5s"
 unhealthyThreshold: 2
 healthyThreshold: 2

---
# Fallback backend
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: openai-fallback
 namespace: enterprise-agentgateway
spec:
 ai:
 provider:
 openai:
 config:
 model: "gpt-4o-mini"
 policies:
 auth:
 secretRef:
 name: openai-secret

---
# Failover route configuration
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai-failover
 namespace: enterprise-agentgateway
 labels:
 routing-type: failover
spec:
 parentRefs:
 - name: agentgateway
 namespace: enterprise-agentgateway
 
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /ai/reliable
 
 backendRefs:
 # Primary backend
 - name: openai-primary
 group: agentgateway.dev
 kind: AgentgatewayBackend
 weight: 100
 # Fallback backend (only used if primary fails)
 - name: openai-fallback
 group: agentgateway.dev
 kind: AgentgatewayBackend
 weight: 0
 
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplacePrefixMatch
 replacePrefixMatch: /v1/chat/completions
 - type: ResponseHeaderModifier
 responseHeaderModifier:
 add:
 - name: X-Routing-Type
 value: failover-enabled
EOF

Creating Routing Test Suite

Comprehensive Routing Test

cat <<'EOF' > comprehensive-routing-test.sh
#!/bin/bash

GATEWAY_IP="${GATEWAY_IP:-localhost}"
GATEWAY_PORT="${GATEWAY_PORT:-8080}"

echo "Comprehensive Routing Pattern Test Suite"
echo "======================================="
echo ""

test_endpoint() {
 local name="$1"
 local endpoint="$2"
 local headers="$3"
 local expected_model="$4"
 
 echo "Testing: $name"
 echo "Endpoint: $endpoint"
 
 local response=$(curl -s $headers "$GATEWAY_IP:$GATEWAY_PORT$endpoint" \
 -H "content-type: application/json" \
 -d '{
 "messages": [
 {"role": "user", "content": "What model are you?"}
 ],
 "max_tokens": 20
 }')
 
 local actual_model=$(echo "$response" | jq -r '.model')
 local tokens=$(echo "$response" | jq -r '.usage.total_tokens')
 
 if [[ "$actual_model" == *"$expected_model"* ]]; then
 echo "✓ Success: $actual_model ($tokens tokens)"
 else
 echo "✗ Failed: Expected $expected_model, got $actual_model"
 fi
 echo ""
}

echo "1. Path-based routing tests"
echo "---------------------------"
test_endpoint "Fast Route" "/ai/fast/completions" "" "gpt-4o-mini"
test_endpoint "Premium Route" "/ai/premium/completions" "" "gpt-4o"

echo "2. Header-based routing tests" 
echo "-----------------------------"
test_endpoint "Dev Environment" "/ai/chat/completions" "-H 'X-Environment: development'" "gpt-4o-mini"
test_endpoint "Prod Environment" "/ai/chat/completions" "-H 'X-Environment: production'" "gpt-4o"
test_endpoint "A/B Test Variant A" "/ai/chat/completions" "-H 'X-AB-Test: variant-a'" "gpt-4o-mini"
test_endpoint "A/B Test Variant B" "/ai/chat/completions" "-H 'X-AB-Test: variant-b'" "gpt-4o"

echo "3. Query parameter routing tests"
echo "--------------------------------"
test_endpoint "Speed Optimized" "/ai/flexible/completions?speed=fast" "" "gpt-4o-mini"
test_endpoint "Quality Optimized" "/ai/flexible/completions?quality=premium" "" "gpt-4o"
test_endpoint "Cost Optimized" "/ai/flexible/completions?cost=minimal" "" "gpt-4o-mini"

echo "Routing test suite complete!"
echo ""
echo "Check Grafana dashboard to see routing patterns and metrics."
EOF

chmod +x comprehensive-routing-test.sh
./comprehensive-routing-test.sh

Best Practices and Considerations

Routing Strategy Guidelines

Performance vs Cost: Balance response quality with token costs
Gradual Rollouts: Use weighted routing for safe model updates
Environment Separation: Use headers for dev/staging/prod isolation
Content Awareness: Route based on request complexity
Fallback Planning: Always have backup routes for reliability

Monitoring Routing Health

cat <<'EOF' > routing-health-check.sh
#!/bin/bash

echo "Routing Health Check"
echo "==================="
echo ""

# Check route statuses
echo "Route Status:"
kubectl get httproute -n enterprise-agentgateway -o custom-columns=\
"NAME:.metadata.name,\
ACCEPTED:.status.parents[0].conditions[?(@.type=='Accepted')].status,\
AGE:.metadata.creationTimestamp"

echo ""

# Check backend health
echo "Backend Status:" 
kubectl get agentgatewaybackend -n enterprise-agentgateway -o custom-columns=\
"NAME:.metadata.name,\
ACCEPTED:.status.conditions[?(@.type=='Accepted')].status,\
AGE:.metadata.creationTimestamp"

echo ""

# Test each route with a quick health check
echo "Route Connectivity Tests:"
routes=(
 "/ai/fast/completions"
 "/ai/premium/completions" 
 "/ai/flexible/completions?speed=fast"
 "/ai/gradual-rollout/completions"
)

for route in "${routes[@]}"; do
 printf "%-35s: " "$route"
 response_code=$(curl -s -o /dev/null -w "%{http_code}" \
 "${GATEWAY_IP:-localhost}:${GATEWAY_PORT:-8080}$route" \
 -H "content-type: application/json" \
 -d '{"messages":[{"role":"user","content":"test"}],"max_tokens":1}')
 
 if [ "$response_code" = "200" ]; then
 echo "✓ OK"
 else
 echo "✗ $response_code"
 fi
done
EOF

chmod +x routing-health-check.sh
./routing-health-check.sh

Cleanup

When you want to clean up the routing configurations:

# Remove all routing test configurations
kubectl delete httproute -n enterprise-agentgateway -l routing-type
kubectl delete httproute -n enterprise-agentgateway \
 openai-gpt4o-mini openai-gpt4o openai-embeddings-route \
 openai-dev-environment openai-prod-environment openai-ab-test \
 openai-query-param-routing openai-weighted-routing \
 openai-short-content openai-long-content openai-failover

# Remove test backends (keep original openai-all-models)
kubectl delete agentgatewaybackend -n enterprise-agentgateway \
 openai-gpt4o-mini openai-gpt4o openai-primary openai-fallback

# Remove policy
kubectl delete enterpriseagentgatewaypolicy content-based-routing -n enterprise-agentgateway

# Clean up test scripts
rm -f test-weighted-routing.sh routing-analysis.sh comprehensive-routing-test.sh routing-health-check.sh

Next Steps

With advanced routing patterns mastered, you’re ready to:

Add security layers - Authentication, authorization, and rate limiting
Implement guardrails - Content filtering and safety policies
Multi-provider routing - Add Anthropic, AWS Bedrock, and others
Production optimization - Performance tuning and cost management

In our next blog post, we’ll explore security features including JWT authentication, API key management, and role-based access control.

Key Takeaways

Intelligent routing transforms agentgateway into a sophisticated AI traffic manager
Multiple routing criteria enable complex decision-making logic
A/B testing and gradual rollouts provide safe model deployment strategies
Content-based routing optimizes cost and performance automatically
Observability provides insights into routing effectiveness and patterns
Fallback strategies ensure reliability even when primary models fail

Your agentgateway now has enterprise-grade routing capabilities that can handle complex production scenarios while optimizing for cost, performance, and reliability!

Observability Stack: Monitoring Your AI Gateway

Mon, 09 Feb 2026 00:00:00 +0000

Introduction

Observability is crucial for any production AI Gateway deployment. Enterprise agentgateway emits comprehensive OpenTelemetry-compatible metrics, logs, and traces out of the box. In this guide, we’ll deploy a complete observability stack including Grafana, Prometheus, Tempo, and Loki to collect, store, and visualize this rich telemetry data.

This setup will give you real-time visibility into your AI Gateway’s performance, cost metrics, token usage, streaming performance, and more.

What You’ll Learn

Deploy Tempo for distributed tracing
Install Prometheus and Grafana for metrics and visualization
Configure agentgateway-specific monitoring
Set up the official agentgateway Grafana dashboard
Access and interpret observability data

Prerequisites

Kind cluster with Enterprise agentgateway installed (from Part 1)
kubectl configured to work with your cluster
Helm 3.x installed

Architecture Overview

Our observability stack will include:

Grafana: Visualization and dashboards
Prometheus: Metrics collection and storage
Tempo: Distributed tracing backend
Loki: Log aggregation (optional)
agentgateway Datasources: Pre-configured Grafana dashboards

Deploy Monitoring Stack

Create Monitoring Namespace

kubectl create namespace monitoring

Install Prometheus

First, let’s deploy Prometheus to collect metrics:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/prometheus \
 --namespace monitoring \
 --set server.service.type=NodePort \
 --set server.service.nodePort=30090 \
 --set server.persistentVolume.enabled=false \
 --set alertmanager.enabled=false \
 --set kube-state-metrics.enabled=true \
 --set nodeExporter.enabled=false \
 --set pushgateway.enabled=false

Install Tempo

Deploy Tempo for distributed tracing:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Create Tempo configuration
cat <<EOF > tempo-values.yaml
tempo:
 retention: 24h
 receivers:
 jaeger:
 protocols:
 grpc:
 endpoint: 0.0.0.0:14250
 thrift_http:
 endpoint: 0.0.0.0:14268
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318

service:
 type: NodePort
 
storage:
 trace:
 backend: local
 local:
 path: /var/tempo/traces

compactor:
 retention: 24h

metrics_generator:
 enabled: true
 registry:
 external_labels:
 source: tempo
EOF

helm install tempo grafana/tempo \
 --namespace monitoring \
 --values tempo-values.yaml

Install Grafana

Deploy Grafana with pre-configured datasources:

# Create Grafana configuration
cat <<EOF > grafana-values.yaml
service:
 type: NodePort
 nodePort: 30080

persistence:
 enabled: false

datasources:
 datasources.yaml:
 apiVersion: 1
 datasources:
 - name: Prometheus
 type: prometheus
 url: http://prometheus-server:80
 access: proxy
 isDefault: true
 - name: Tempo
 type: tempo
 url: http://tempo:3100
 access: proxy

dashboardProviders:
 dashboardproviders.yaml:
 apiVersion: 1
 providers:
 - name: 'default'
 orgId: 1
 folder: ''
 type: file
 disableDeletion: false
 editable: true
 options:
 path: /var/lib/grafana/dashboards/default

dashboards:
 default:
 agentgateway-genai:
 gnetId: 21703
 revision: 1
 datasource: Prometheus

adminPassword: admin
EOF

helm install grafana grafana/grafana \
 --namespace monitoring \
 --values grafana-values.yaml

Configure agentgateway for Observability

Update agentgateway Configuration

We need to configure agentgateway to send traces to our Tempo instance:

kubectl apply -f- <<'EOF'
---
apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayParameters
metadata:
 name: agentgateway-params
 namespace: enterprise-agentgateway
spec:
 # Enable shared extensions
 sharedExtensions:
 extauth:
 enabled: true
 deployment:
 spec:
 replicas: 1
 ratelimiter:
 enabled: true
 deployment:
 spec:
 replicas: 1
 extCache:
 enabled: true
 deployment:
 spec:
 replicas: 1
 
 # Enhanced observability configuration
 rawConfig:
 config:
 logging:
 level: info
 fields:
 add:
 jwt: 'jwt'
 request.body: json(request.body)
 response.body: json(response.body)
 request.body.modelId: json(request.body).modelId
 llm.provider: 'llm.provider'
 llm.model: 'llm.requestModel'
 llm.tokens.input: 'llm.inputTokens'
 llm.tokens.output: 'llm.outputTokens'
 llm.cost.total: 'llm.totalCost'
 format: json
 
 tracing:
 randomSampling: 'true'
 collector:
 endpoint: tempo.monitoring.svc.cluster.local:4317
 fields:
 add:
 # GenAI semantic conventions
 gen_ai.operation.name: '"chat"'
 gen_ai.system: "llm.provider"
 gen_ai.prompt: 'llm.prompt'
 gen_ai.completion: 'llm.completion.map(c, {"role":"assistant", "content": c})'
 gen_ai.request.model: "llm.requestModel"
 gen_ai.response.model: "llm.responseModel"
 gen_ai.usage.completion_tokens: "llm.outputTokens"
 gen_ai.usage.prompt_tokens: "llm.inputTokens"
 gen_ai.request: 'flatten(llm.params)'
 
 # Additional context
 jwt: 'jwt'
 response.body: 'json(response.body)'
 llm.cost.total: 'llm.totalCost'
 llm.cost.input: 'llm.inputCost'
 llm.cost.output: 'llm.outputCost'
 
 metrics:
 enabled: true
 prefix: "agentgateway"
 tags:
 add:
 provider: 'llm.provider'
 model: 'llm.requestModel'
 user: 'jwt.sub // "anonymous"'
 
 # Service configuration for monitoring
 service:
 spec:
 type: NodePort
 ports:
 - name: http
 port: 8080
 targetPort: 8080
 nodePort: 30080
 protocol: TCP
 - name: metrics
 port: 9091
 targetPort: 9091
 nodePort: 30091
 protocol: TCP
 
 # Deployment configuration
 deployment:
 spec:
 replicas: 1
 template:
 spec:
 containers:
 - name: agentgateway
 resources:
 requests:
 cpu: 200m
 memory: 128Mi
 limits:
 cpu: 500m
 memory: 256Mi
EOF

Verify Monitoring Stack

Check All Pods are Running

kubectl get pods -n monitoring

Expected output:

NAME READY STATUS RESTARTS AGE
grafana-7b8c4c4d4c-xyz12 1/1 Running 0 3m
prometheus-server-5f8b8b7d7d-abc34 1/1 Running 0 5m
tempo-0 1/1 Running 0 4m

Verify agentgateway Configuration

kubectl get pods -n enterprise-agentgateway
kubectl logs deploy/agentgateway -n enterprise-agentgateway --tail=20

Look for log entries indicating successful startup and trace collection.

Access Monitoring Interfaces

Grafana Dashboard

Access Grafana at: http://localhost:30080

Username: admin
Password: admin

Prometheus UI

Access Prometheus at: http://localhost:30090

agentgateway Metrics

Access agentgateway metrics directly: http://localhost:30091/metrics

agentgateway GenAI Dashboard

The Grafana deployment automatically imports the official agentgateway dashboard (ID: 21703). This dashboard provides:

Key Metrics Panels

Request Rate and Latency
- Requests per second by provider
- P95, P99 latency percentiles
- Error rates and status codes
Token Usage and Costs
- Input/output token consumption
- Cost tracking per provider
- Token efficiency metrics
Model Performance
- Response time by model
- Token generation rates
- Streaming performance
Provider Health
- Provider availability
- Error rates by provider
- Failover events

Using the Dashboard

Navigate to Dashboards > Browse in Grafana
Open agentgateway GenAI Dashboard
Set time range (e.g., Last 1 hour)
Select providers/models using dropdown filters

Testing Your Monitoring Setup

Let’s create some test traffic to see data flowing through our observability stack:

Deploy Test Mock Backend

kubectl apply -f- <<'EOF'
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: mock-openai
 namespace: default
spec:
 replicas: 1
 selector:
 matchLabels:
 app: mock-openai
 template:
 metadata:
 labels:
 app: mock-openai
 spec:
 containers:
 - name: wiremock
 image: wiremock/wiremock:3.3.1
 ports:
 - containerPort: 8080
 args:
 - --port=8080
 - --verbose
 volumeMounts:
 - name: wiremock-data
 mountPath: /home/wiremock
 env:
 - name: JAVA_OPTS
 value: "-Dfile.encoding=UTF-8"
 volumes:
 - name: wiremock-data
 configMap:
 name: mock-openai-config
---
apiVersion: v1
kind: Service
metadata:
 name: mock-openai
 namespace: default
spec:
 selector:
 app: mock-openai
 ports:
 - port: 80
 targetPort: 8080
---
apiVersion: v1
kind: ConfigMap
metadata:
 name: mock-openai-config
 namespace: default
data:
 mappings.json: |
 {
 "mappings": [
 {
 "request": {
 "method": "POST",
 "url": "/v1/chat/completions"
 },
 "response": {
 "status": 200,
 "headers": {
 "Content-Type": "application/json"
 },
 "jsonBody": {
 "id": "chatcmpl-test-{{randomValue type='ALPHANUMERIC' length=10}}",
 "object": "chat.completion",
 "created": {{currentTimestamp}},
 "model": "gpt-4",
 "choices": [
 {
 "index": 0,
 "message": {
 "role": "assistant",
 "content": "Hello! This is a mock response from the test OpenAI backend. Current time: {{currentTimestamp}}."
 },
 "finish_reason": "stop"
 }
 ],
 "usage": {
 "prompt_tokens": 25,
 "completion_tokens": 15,
 "total_tokens": 40
 }
 },
 "transformers": ["response-template"],
 "fixedDelayMilliseconds": 100
 }
 }
 ]
 }
EOF

Configure agentgateway Route

kubectl apply -f- <<'EOF'
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: mock-openai-backend
 namespace: default
spec:
 openai:
 connectionString: http://mock-openai.default.svc.cluster.local
 authToken:
 secretRef:
 name: mock-openai-secret
 key: api-key
 model:
 default: gpt-4
 mapping:
 "gpt-4": gpt-4
 "gpt-3.5-turbo": gpt-3.5-turbo
---
apiVersion: v1
kind: Secret
metadata:
 name: mock-openai-secret
 namespace: default
type: Opaque
stringData:
 api-key: "test-api-key"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: mock-openai-route
 namespace: default
spec:
 parentRefs:
 - name: agentgateway
 namespace: enterprise-agentgateway
 hostnames:
 - "*"
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /openai
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplaceFullPath
 replaceFullPath: /v1/chat/completions
 backendRefs:
 - name: mock-openai-backend
 group: agentgateway.dev
 kind: AgentgatewayBackend
 port: 80
EOF

Generate Test Traffic

# Send test requests to generate observability data
for i in {1..10}; do
 curl -X POST http://localhost:8080/openai \
 -H "Content-Type: application/json" \
 -H "Authorization: Bearer test-token" \
 -d '{
 "model": "gpt-4",
 "messages": [
 {"role": "user", "content": "Test message '${i}' for observability demo"}
 ],
 "max_tokens": 50
 }' && echo
 sleep 2
done

Viewing Observability Data

In Grafana

Dashboard Overview: Navigate to the agentgateway dashboard
Request Metrics: See request rates, response times
Token Usage: Monitor input/output tokens and costs
Error Analysis: Check error rates and types

In Tempo (Distributed Tracing)

Go to Explore in Grafana
Select Tempo datasource
Search for traces by service name: agentgateway
Analyze trace spans showing request flow

Key Traces to Look For

HTTP Request Span: Gateway ingress
LLM Provider Span: Backend communication
Auth Span: Authentication processing
Rate Limit Span: Rate limiting decisions

Sample Tempo Query

{service.name="agentgateway"}

Advanced Monitoring Configuration

Custom Metrics

Add custom metrics to track specific business KPIs:

rawConfig:
 config:
 metrics:
 enabled: true
 customMetrics:
 - name: "user_requests_total"
 type: "counter"
 help: "Total requests per user"
 labels:
 - user: 'jwt.sub // "anonymous"'
 - model: 'llm.requestModel'
 
 - name: "token_cost_dollars"
 type: "histogram"
 help: "Cost in dollars per request"
 buckets: [0.001, 0.01, 0.1, 1.0, 10.0]
 value: 'llm.totalCost'

Alert Rules

Create Prometheus alert rules for critical conditions:

groups:
- name: agentgateway.rules
 rules:
 - alert: HighErrorRate
 expr: rate(agentgateway_requests_total{status=~"5.."}[5m]) > 0.1
 for: 2m
 labels:
 severity: warning
 annotations:
 summary: "High error rate detected"
 
 - alert: HighLatency
 expr: histogram_quantile(0.95, rate(agentgateway_request_duration_seconds_bucket[5m])) > 5
 for: 5m
 labels:
 severity: critical
 annotations:
 summary: "High latency detected"
 
 - alert: HighCost
 expr: increase(agentgateway_token_cost_dollars_total[1h]) > 50
 for: 0m
 labels:
 severity: warning
 annotations:
 summary: "High hourly cost detected"

Monitoring Best Practices

Resource Planning

Monitor these key metrics for capacity planning:

Request Rate: Track requests/second trends
Token Velocity: Monitor tokens/minute per model
Cost Burn Rate: Track $/hour consumption
Provider Latency: Monitor P95/P99 response times

Performance Optimization

Use observability data to optimize:

Route Configuration: Based on latency patterns
Caching Strategies: Based on request patterns
Rate Limiting: Based on usage distribution
Provider Selection: Based on cost/performance

Cost Management

Track and alert on:

Daily/Monthly spend per provider
Cost per user/team
Token efficiency (output/input ratio)
Most expensive models/users

Troubleshooting

Missing Traces

If traces aren’t appearing in Tempo:

# Check Tempo is receiving traces
kubectl logs -l app.kubernetes.io/name=tempo -n monitoring

# Verify agentgateway trace configuration
kubectl get enterpriseagentgatewayparameters agentgateway-params -n enterprise-agentgateway -o yaml

# Check connectivity
kubectl exec -n enterprise-agentgateway deployment/agentgateway -- nc -zv tempo.monitoring.svc.cluster.local 4317

Missing Metrics

If metrics aren’t showing in Prometheus:

# Check Prometheus targets
curl http://localhost:30090/targets

# Verify agentgateway metrics endpoint
curl http://localhost:30091/metrics

# Check Prometheus configuration
kubectl get configmap prometheus-server -n monitoring -o yaml

Dashboard Issues

If the dashboard isn’t loading:

# Check Grafana pod logs
kubectl logs -l app.kubernetes.io/name=grafana -n monitoring

# Verify datasource connectivity
kubectl exec -n monitoring deployment/grafana -- nc -zv prometheus-server 80
kubectl exec -n monitoring deployment/grafana -- nc -zv tempo 3100

Cleanup

To remove the monitoring stack:

helm uninstall grafana -n monitoring
helm uninstall prometheus -n monitoring
helm uninstall tempo -n monitoring
kubectl delete namespace monitoring

Next Steps

With comprehensive observability in place, you’re ready to:

Set up development environments with mock providers
Configure real AI provider integrations
Implement advanced routing strategies
Add security and rate limiting policies

In our next blog post, we’ll create a mock OpenAI environment for cost-free development and testing.

Key Takeaways

Complete visibility into AI Gateway performance and costs
Real-time monitoring with Grafana dashboards
Distributed tracing for request flow analysis
Cost tracking and optimization insights
Production-ready monitoring stack for kind clusters

Your agentgateway is now fully instrumented and ready for production workloads with comprehensive observability!

Setting Up Enterprise agentgateway on Kind Clusters

Mon, 09 Feb 2026 00:00:00 +0000

Introduction

Enterprise agentgateway by Solo.io is a powerful AI Gateway solution that provides unified access to multiple LLM providers with advanced features like routing, security, observability, and cost management. In this comprehensive guide, we’ll walk you through setting up Enterprise agentgateway on a kind (Kubernetes in Docker) cluster - perfect for development, testing, and learning environments.

What You’ll Learn

How to set up a kind cluster optimized for agentgateway
Installing Kubernetes Gateway API and agentgateway CRDs
Deploying the Enterprise agentgateway controller
Configuring your first agentgateway instance
Validating your installation

Prerequisites

Before starting this guide, ensure you have:

Solo.io Trial License Key: Get a free trial at Solo.io
Docker Desktop or Docker Engine running
kind CLI installed (installation guide)
kubectl CLI installed and configured
helm CLI installed (version 3.x)

Kind Cluster Setup

Create Kind Cluster Configuration

First, let’s create a kind cluster with the proper configuration for running agentgateway:

cat <<EOF > kind-agentgateway-cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: agentgateway
nodes:
- role: control-plane
 kubeadmConfigPatches:
 - |
 kind: InitConfiguration
 nodeRegistration:
 kubeletExtraArgs:
 node-labels: "ingress-ready=true"
 extraPortMappings:
 - containerPort: 80
 hostPort: 80
 protocol: TCP
 - containerPort: 443
 hostPort: 443
 protocol: TCP
 - containerPort: 8080
 hostPort: 8080
 protocol: TCP
- role: worker
- role: worker
EOF

Create the Cluster

# Create the kind cluster
kind create cluster --config kind-agentgateway-cluster.yaml

# Verify the cluster is ready
kubectl cluster-info --context kind-agentgateway
kubectl get nodes

Expected output:

NAME STATUS ROLES AGE VERSION
agentgateway-control-plane Ready control-plane 2m v1.29.1
agentgateway-worker Ready <none> 90s v1.29.1
agentgateway-worker2 Ready <none> 90s v1.29.1

Environment Setup

Configure Required Variables

Export your Solo.io trial license key and agentgateway version:

# Replace with your actual license key from Solo.io
export SOLO_TRIAL_LICENSE_KEY="your-license-key-here"
export ENTERPRISE_AGW_VERSION="2.1.0"

# Verify the variables are set
echo "License Key: $SOLO_TRIAL_LICENSE_KEY"
echo "agentgateway Version: $ENTERPRISE_AGW_VERSION"

Installing Kubernetes Gateway API

agentgateway builds on the Kubernetes Gateway API. We’ll use the experimental CRDs to enable advanced features:

# Install Kubernetes Gateway API CRDs (experimental for advanced features)
kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/experimental-install.yaml

# Wait for CRDs to be established
kubectl wait --for condition=Established crd/gatewayclasses.gateway.networking.k8s.io
kubectl wait --for condition=Established crd/gateways.gateway.networking.k8s.io
kubectl wait --for condition=Established crd/httproutes.gateway.networking.k8s.io

Verify Gateway API Installation

kubectl api-resources --api-group=gateway.networking.k8s.io

Expected output:

NAME SHORTNAMES APIVERSION NAMESPACED KIND
backendtlspolicies btlspolicy gateway.networking.k8s.io/v1 true BackendTLSPolicy
gatewayclasses gc gateway.networking.k8s.io/v1 false GatewayClass
gateways gtw gateway.networking.k8s.io/v1 true Gateway
grpcroutes gateway.networking.k8s.io/v1 true GRPCRoute
httproutes gateway.networking.k8s.io/v1 true HTTPRoute
referencegrants refgrant gateway.networking.k8s.io/v1beta1 true ReferenceGrant
tcproutes gateway.networking.k8s.io/v1alpha2 true TCPRoute
tlsroutes gateway.networking.k8s.io/v1alpha3 true TLSRoute
udproutes gateway.networking.k8s.io/v1alpha2 true UDPRoute

Installing Enterprise agentgateway

Create Namespace and Install CRDs

# Create namespace for agentgateway
kubectl create namespace enterprise-agentgateway

# Install Enterprise agentgateway CRDs
helm upgrade -i --create-namespace --namespace enterprise-agentgateway \
 --version $ENTERPRISE_AGW_VERSION enterprise-agentgateway-crds \
 oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway-crds

# Wait for CRDs to be established
kubectl wait --for condition=Established crd/agentgatewaybackends.agentgateway.dev
kubectl wait --for condition=Established crd/agentgatewayparameters.agentgateway.dev
kubectl wait --for condition=Established crd/agentgatewaypolicies.agentgateway.dev

Verify agentgateway CRDs

kubectl api-resources | awk 'NR==1 || /enterpriseagentgateway\\.solo\\.io|agentgateway\\.dev|ratelimit\\.solo\\.io|extauth\\.solo\\.io/'

Expected output:

NAME SHORTNAMES APIVERSION NAMESPACED KIND
agentgatewaybackends agbe agentgateway.dev/v1alpha1 true AgentgatewayBackend
agentgatewayparameters agpar agentgateway.dev/v1alpha1 true AgentgatewayParameters
agentgatewaypolicies agpol agentgateway.dev/v1alpha1 true AgentgatewayPolicy
enterpriseagentgatewayparameters eagpar enterpriseagentgateway.solo.io/v1alpha1 true EnterpriseAgentgatewayParameters
enterpriseagentgatewaypolicies eagpol enterpriseagentgateway.solo.io/v1alpha1 true EnterpriseAgentgatewayPolicy
authconfigs ac extauth.solo.io/v1 true AuthConfig
ratelimitconfigs rlc ratelimit.solo.io/v1alpha1 true RateLimitConfig

Install Enterprise agentgateway Controller

helm upgrade -i -n enterprise-agentgateway enterprise-agentgateway \
 oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway \
 --create-namespace \
 --version $ENTERPRISE_AGW_VERSION \
 --set-string licensing.licenseKey=$SOLO_TRIAL_LICENSE_KEY \
 -f -<<EOF
# Gateway Class parameters reference
gatewayClassParametersRefs:
 enterprise-agentgateway:
 group: enterpriseagentgateway.solo.io
 kind: EnterpriseAgentgatewayParameters
 name: agentgateway-params
 namespace: enterprise-agentgateway
EOF

Verify Controller Installation

# Check that the controller is running
kubectl get pods -n enterprise-agentgateway -l app.kubernetes.io/name=enterprise-agentgateway

# Wait for the controller to be ready
kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=enterprise-agentgateway -n enterprise-agentgateway --timeout=300s

Expected output:

NAME READY STATUS RESTARTS AGE
enterprise-agentgateway-5fc9d95758-n8vvb 1/1 Running 0 87s

Deploy agentgateway Instance

Now let’s create an agentgateway instance with optimized configuration for kind clusters:

kubectl apply -f- <<'EOF'
---
apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayParameters
metadata:
 name: agentgateway-params
 namespace: enterprise-agentgateway
spec:
 # Enable shared extensions for auth, rate limiting, and caching
 sharedExtensions:
 extauth:
 enabled: true
 deployment:
 spec:
 replicas: 1
 ratelimiter:
 enabled: true
 deployment:
 spec:
 replicas: 1
 extCache:
 enabled: true
 deployment:
 spec:
 replicas: 1
 
 # Configure logging
 logging:
 level: info
 
 # Configure service for kind cluster (NodePort for local access)
 service:
 spec:
 type: NodePort
 ports:
 - name: http
 port: 8080
 targetPort: 8080
 nodePort: 30080
 protocol: TCP
 
 # Observability configuration
 rawConfig:
 config:
 logging:
 fields:
 add:
 jwt: 'jwt'
 request.body: json(request.body)
 response.body: json(response.body)
 request.body.modelId: json(request.body).modelId
 format: json
 tracing:
 randomSampling: 'true'
 fields:
 add:
 gen_ai.operation.name: '"chat"'
 gen_ai.system: "llm.provider"
 gen_ai.prompt: 'llm.prompt'
 gen_ai.completion: 'llm.completion.map(c, {"role":"assistant", "content": c})'
 gen_ai.request.model: "llm.requestModel"
 gen_ai.response.model: "llm.responseModel"
 gen_ai.usage.completion_tokens: "llm.outputTokens"
 gen_ai.usage.prompt_tokens: "llm.inputTokens"
 gen_ai.request: 'flatten(llm.params)'
 jwt: 'jwt'
 response.body: 'json(response.body)'
 
 # Deployment configuration optimized for kind
 deployment:
 spec:
 replicas: 1 # Single replica for development
 template:
 spec:
 containers:
 - name: agentgateway
 resources:
 requests:
 cpu: 200m
 memory: 128Mi
 limits:
 cpu: 500m
 memory: 256Mi
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: agentgateway
 namespace: enterprise-agentgateway
spec:
 gatewayClassName: enterprise-agentgateway
 listeners:
 - name: http
 port: 8080
 protocol: HTTP
 allowedRoutes:
 namespaces:
 from: All
EOF

Validation

Check All Pods Are Running

kubectl get pods -n enterprise-agentgateway

Expected output:

NAME READY STATUS RESTARTS AGE
agentgateway-7d4c8c4d4b-lvdsq 1/1 Running 0 2m
enterprise-agentgateway-5f9c5b95b4-gjblt 1/1 Running 0 5m
ext-auth-service-enterprise-agentgateway-6fcc5bc989-22wgd 1/1 Running 0 2m
ext-cache-enterprise-agentgateway-6bfcb8c87d-vjzxn 1/1 Running 0 2m
rate-limiter-enterprise-agentgateway-589f66bb88-xz7nm 1/1 Running 0 2m

Check Gateway Status

kubectl get gateway agentgateway -n enterprise-agentgateway -o yaml

Look for the status section showing the gateway is accepted and programmed.

Test Gateway Accessibility

For kind clusters, we can access the gateway through the NodePort:

# Get the kind cluster's node port
kubectl get svc -n enterprise-agentgateway --selector=gateway.networking.k8s.io/gateway-name=agentgateway

# Test basic connectivity (should return 404 since no routes are configured yet)
curl -v http://localhost:8080/test

Expected response: HTTP 404 (which is correct - no routes configured yet)

Understanding Your Setup

What We’ve Deployed

Kubernetes Gateway API: Standard APIs for traffic management
Enterprise agentgateway Controller: Manages agentgateway lifecycle
agentgateway Data Plane: Routes traffic to AI providers
Shared Extensions:
- ext-auth-service: Handles authentication (JWT, API keys)
- rate-limiter: Manages rate limiting and quotas
- ext-cache: Provides caching capabilities

Network Configuration for Kind

Gateway Service: NodePort on port 30080 for local access
Internal Port: 8080 for cluster-internal communication
Host Access: http://localhost:8080 from your development machine

Environment Setup Script

Create a helper script for managing your agentgateway environment:

cat <<'EOF' > agentgateway-env.sh
#!/bin/bash

# agentgateway Environment Setup Script
export SOLO_TRIAL_LICENSE_KEY="${SOLO_TRIAL_LICENSE_KEY:-your-license-key-here}"
export ENTERPRISE_AGW_VERSION="2.1.0"

# Helper functions
agw_status() {
 echo "=== agentgateway Status ==="
 kubectl get pods -n enterprise-agentgateway
 echo ""
 kubectl get gateway agentgateway -n enterprise-agentgateway
}

agw_logs() {
 kubectl logs deploy/agentgateway -n enterprise-agentgateway --tail=50 -f
}

agw_reset() {
 echo "Resetting agentgateway..."
 kubectl delete namespace enterprise-agentgateway
 # Re-run installation commands
}

echo "agentgateway environment loaded!"
echo "Available commands: agw_status, agw_logs, agw_reset"
EOF

# Make it executable and source it
chmod +x agentgateway-env.sh
source agentgateway-env.sh

Troubleshooting

Common Issues

1. License Key Issues

# Check controller logs for license errors
kubectl logs deploy/enterprise-agentgateway -n enterprise-agentgateway

2. CRD Installation Problems

# Reinstall CRDs if needed
kubectl delete crd -l app.kubernetes.io/name=enterprise-agentgateway-crds
# Then re-run the helm install command

3. Pod Startup Issues

# Check pod events
kubectl describe pod -l app.kubernetes.io/name=agentgateway -n enterprise-agentgateway

# Check resource constraints
kubectl top pods -n enterprise-agentgateway

4. Gateway Not Ready

# Check gateway status
kubectl describe gateway agentgateway -n enterprise-agentgateway

# Verify GatewayClass
kubectl get gatewayclass enterprise-agentgateway -o yaml

Cleanup

When you’re done testing, clean up your environment:

# Delete the agentgateway installation
helm uninstall enterprise-agentgateway -n enterprise-agentgateway
helm uninstall enterprise-agentgateway-crds -n enterprise-agentgateway

# Delete the namespace
kubectl delete namespace enterprise-agentgateway

# Delete the kind cluster
kind delete cluster --name agentgateway

Next Steps

Now that you have agentgateway running on your kind cluster, you’re ready to:

Set up monitoring and observability - Install Grafana, Prometheus, and Tempo for comprehensive observability
Configure your first AI route - Connect to OpenAI, Anthropic, or other providers
Explore advanced features - Security, rate limiting, and traffic management

In our next blog post, we’ll set up a complete observability stack to monitor your agentgateway’s performance, costs, and usage patterns.

Key Takeaways

Enterprise agentgateway provides a unified interface for multiple AI providers
Kind clusters are perfect for development and testing agentgateway
The setup includes controller, data plane, and essential shared extensions
Proper observability configuration enables comprehensive monitoring
NodePort service configuration allows easy local access in kind environments

With your agentgateway foundation in place, you’re ready to build sophisticated AI routing and management capabilities!

Key	Desc
`provider=local`	llama.cpp
`hybrid=true`	GPU auto
`cpu_threads=8`	Cores

Sebastian Maniak

One Front Door for Every Tool: MCP Multiplexing Through agentgateway

The sprawl, drawn

The catalog behind the door

How it’s wired: one backend, two behaviors

What a tool call actually does

Why one door is safer than many

What it does — and what it doesn’t, yet

Using it: Grok Bot as the governed client

The rules that keep it honest

The takeaway

The After-Hours Kill Switch: Daytime Claude, Nighttime Grok, One CEL Expression

The idea: the client asks, the gateway decides

How it’s built: virtual model + conditional routing + CEL

Why the switch can’t be dodged

The timezone footgun (read this before you copy the CEL)

Proof: a 7:11pm request served Grok

Why this is worth doing

The takeaway

The Art of the Possible: Governed Sandbox Agents, and an Arista Operator That Can't Go Rogue

This time, the agent is the box

Governance layer 1 — skills that encode how to behave

Governance layer 2 — a curated, read-only tool catalog

Governance layer 3 — the sandbox, the secrets, and the allowlist

Watch it actually work

Why this is the art of the possible

Being honest about the edges

The takeaway: governed autonomy

How To: Build an AWS Budget SandboxAgent on kagent + Agent Substrate

Why a SandboxAgent and not a plain Agent

Architecture

The pins that matter (do not bump)

Snapshots today: rustfs, and omit snapshotsConfig

Step 1 — a dedicated IAM identity

Step 2 — keys into Vault, never git

Step 3 — build and import the MCP image

Step 4 — apply the agent

Step 5 — what “Ready” actually means

The tool catalog — no generic shell

The live run

Proof, with nothing sensitive on screen

Honest limits

The rest of the series

A GCP Budget SandboxAgent on kagent — and the Value of an Agent That Says "Unavailable"

Why a SandboxAgent and not a plain Agent

Architecture

Pins (do not bump)

Build it

The tools

The live run, including what broke

The Cloud Billing gap is real, not a bug in my code

Why “unavailable” is the feature

Honest limits

The rest of the series

A ServiceNow Triage SandboxAgent on kagent — Where Write Tools Belong

Why a SandboxAgent and not a plain Agent

Architecture

Pins (do not bump)

Build it

Six read tools, two write tools

Where write tools belong

The live run

Honest limits

The rest of the series

Asking My Home FortiGate Questions: A gVisor SandboxAgent on kagent

Why a SandboxAgent and not a plain Agent

Architecture

Pins (do not bump)

Build it

The tool catalog

The live run

What this demo is actually good at

Honest limits

The rest of the series

An F5 BIG-IP SandboxAgent on kagent: 19 VIPs, 2 Up, and 18 Tool Calls to Find Out Why

Why a SandboxAgent and not a plain Agent

Architecture

Pins (do not bump)

Build it

Six tools, zero writes

Snapshots today: rustfs, and omit `snapshotsConfig`

Codex → `:4000/v1`