A CTO found $78,000 of OpenAI charges on the company card this year. A single request, they report, had spawned 826 parallel Codex agents — consuming roughly 2.1 trillion tokens across 162 invoices, none of it authorised, none of it visible in real time, all of it discovered the way these things usually are: on the statement. The vendor's answer, after weeks of tickets: "credits were consumed". No server-side reconstruction was offered.
HN commenters argued about the word "rogue"; nobody argued about the missing piece: there were no hard billing controls at any level — not at the vendor, not at the card, not in the orchestration layer. That is the real story, and on 3 October Simon Willison stated the design principle plainly: pay-by-usage services need hard budget caps on by default — "after X/month,cutthisthingoffX, send me an email") will not cut it, because by the time anyone reads the email, the call that blew the budget has already been made and billed.
an
d
r
e
t
u
r
n
er
r
or
s
".
S
o
f
t
c
a
p
s
(
"
a
f
t
er
Why soft caps fail
Detection is not control. A token counter shows you the overspend after it happened; a warning email arrives after that. The engineering reality of agentic workloads makes this worse in three compounding ways:
The 24× multiplier. A chat session costs a few thousand tokens and a couple of cents. An agentic task ranges from 50,000 to 500,000 tokens — 3–15 steps, each 5k–20k tokens, times a 2–5× retry multiplier — landing at $0.50–5.00 per task, with every task paying it. Goldman Sachs' 2026 analysis put the gap at roughly 24× token demand for agentic over conversational use.
Context compounds. Each step appends to the context window, so the cost per call climbs as the run progresses. A retry loop's hundredth iteration is far more expensive than its first — a loop at 100 iterations can spend hundreds of dollars while looking perfectly healthy on a dashboard.
Loops are multi-agent. An analyser and a verifier can ping-pong indefinitely, each step generating fresh work for the other, with no shared budget and no round-trip counter.
The canonical documented disaster: a four-agent LangChain market-research pipeline in late 2025 with no spend caps, no timeouts, no alerts, ran for **264 hours and cost 47,000∗∗,itsweeklyAPIspendclimbing127 → 891→6,240 → $18,400 before the shutdown. An academic catalogue on arXiv this year ("Token Budgets", June 2026) documents 63 confirmed production budget-overrun incidents across 21 projects and 18 ecosystems between 2023 and 2026 — a recurrence pattern, not anecdotes.
The anatomy of a runaway
Runaways come in four shapes: retry storms (a transient error retried forever), single-agent ReAct loops with a fuzzy completion criterion, multi-agent ping-pong, and unbounded tool-calling that hammers a database or third-party API. They share one property:
The cost of the next call is invisible to the system making it.
That is why content guardrails cannot help. Every individual call is legitimate — valid tool invocation, plausible reasoning — the problem is volume and lack of a terminal condition. Only a consumption ceiling catches it. A ceiling is a circuit breaker on resource use; it inspects nothing, judges nothing, and fires precisely when guardrails cannot see the problem at all.
Incident
Cost
Duration
Missing
LangChain analyse/verify ping-pong
$47,000
11 days
per-agent caps, timeout, alerting
Codex 826-agent fan-out
~$78,000
~20 days
vendor cap, card cap, session limits
Flaky-tool retry storm
per-call small, total unbounded
minutes to days
retry caps + failure containment
Four layers of enforcement
The control surface now exists top to bottom. Spring 2026 saw the shift Willison is asking for become a trend — AWS began rolling out project spending limits on 16 September; Google Cloud launched Spend Caps in July ("set a monthly financial cap on specific services within a project"). Both are default-off for existing accounts today, and Willison's point is that they should be default-on with a prominent "Remove the budget cap" opt-out — "if someone wants to live dangerously they should be able to do that, but it needs to be opt-in".
Provider caps alone are not enough. The Codex victim had a $50k/month card limit; an agent reportedly switched payment cards when the first was exhausted. You need containment at every layer, with the deepest one owned by you:
1. Provider and card. Hard cap at the vendor; hard cap at the payment instrument. Treat "uncapped" as a deliberate choice, not a default.
2. A gateway you control (the transport layer). Put every agent key behind a local proxy like LiteLLM. It meters spend itself, enforces per-key/team/user budgets with stacked windows (10/day∗and∗100/month, so one bad afternoon cannot burn the month), can reroute to cheaper models via budget fallbacks (opus → sonnet → haiku), and can be configured fail-closed so a stale counter never under-reports spend:
# LiteLLM proxy config
litellm_settings:
max_budget: 500 # swallow budget for the whole tenant
budget_duration: 30d # resets 1st of month
# per key: two stacked windows + model fallbacks
# /key/generate -> budget_limits:
# - {"budget_duration": "24h", "max_budget": 10}
# - {"budget_duration": "30d", "max_budget": 100}
# model_max_budget + budget_fallbacks: opus->sonnet->haiku
This enforces the bound regardless of what the agent framework does or does not configure — every call traverses the proxy, so the meter is always in the path.
3. The harness (before the expensive call). The most important rule: check before the spend, not after. Reserve a pessimistic estimate of the next call's cost, then settle to the actual when it returns; halt from a before_model hook so the third call is never made; and maintain two windows — per-run and per-tenant, because a 0.05/runcapwith10,000runsisnotacap.Classicfailure:everyrunobeyedits0.50 budget while aggregated tenant spend blew through $50.
def before_model(run, tenant):
reserve = 2.0 * run.ctx_tokens * PRICE[run.model] # biased up, never optimistic
if run.spent + reserve > run.cap_usd:
return "halt" # refuse the call, return partial answer
redis.hincrbyfloat(f"budget:{tenant}", "spent", reserve) # atomic; TTL the reservation
return "allow"
Wind-down beats crash: deny tools and let the agent summarise at 80% of budget ("here is what I found before I ran out" is a product; a stack trace is an incident), and hard-halt at 100%.
4. The orchestrator. Per-agent ceilings configured per agent, not one global default: step/iteration caps, delegation-depth caps, wall-clock timeouts. Emit limit breaches as telemetry; alert on approach and on breach; alert on the cost derivative (spend per hour) rather than the level, so a spike is an event, not a line item you learn about at month-end.
Layer
What it contains
Failure it blocks
Provider/card
Account-level $X
Invoice shock at the source
Gateway (LiteLLM)
Per-key, per-tenant, stacked windows
Rogue key, single bad afternoon
Harness
Per-run + per-tenant, reserve-then-reconcile
One runaway task, 10,000 quiet runs
Orchestrator
Steps, depth, wall-clock, derivative alerts
Infinite loops, delegation spirals
Why this is a sovereignty story
The Codex victim's sharpest complaint was that "only OpenAI has the server-side mapping" — the meter is opaque, and reconciliation happens after the fact through billing archaeology. That asymmetry is the deepest cost-control problem in managed AI.
When the model runs on your own hardware, you own the meter. A vLLM endpoint behind a local LiteLLM gateway gives you exact, per-key, per-tenant token-to-dollar accounting with real-time enforcement — no vendor counter to interrogate, no invoice to reconstruct. Digital sovereignty for AI is usually argued in terms of where data lives; the budget story is the same argument in terms of who holds the kill switch, and self-hosted stacks hold it by construction.
What to do on Monday
Hard-cap the vendor account and the card. Default to capped; make uncapped an explicit, approved choice.
Route every agent key through a local gateway with stacked per-tenant windows and budget fallbacks — before the first public launch, not after.
Add harness-level budgets with reserve-then-reconcile and a before_model halt. Token counters are not cost controls.
Cap steps, delegation depth and wall-clock time per agent — a $0.05/run ceiling is worthless without the tenant window.
Alert on cost derivative and on breached ceilings. A tripped cap is a defect signal to investigate, not a fine to absorb.
If you self-host, point agents at your vLLM endpoint through the gateway and enjoy metering your provider can't take away from you.
Set caps from measured p95, not vibes. Run a week with generous caps and full metering, look at the distribution, then tighten.
Willison's closing wish is that agents themselves start biasing towards providers with hard caps. Until then, the discipline is yours: a service that returns an error after Xisadefectyoucanfix.Aninvoicefor78,000 is a failure mode you can't.