Load-Testing Production Without Blowing It Up: an 8 GB CEILING lesson

Load tests feel safe because they are "just requests." No CI pipeline I know of warns you that the request you're blasting at production is itself the most dangerous code in the deployment — until five of them line up and the kernel decides which process to kill.

This is the story of a CI load test that took down a production graph store, and the design lessons that turned "speed budget" thinking into "blast radius" thinking.

The incident, compressed

A lead-scoring service exposed GET /api/graph/score. Each request scans and scores the entire company graph in memory — a legitimately expensive operation, normally a second or two on a healthy box. A CI load-test case fired five concurrent such requests.

At 06:50:53 the host OOM-killed the Dgraph container:

ExitCode: 255, OOMKilled: true

The container had no memory ceiling (mem_limit unset), so the kernel didn't recycle dgraph — it let dgraph balloon, then killed whichever process it picked. It picked dgraph. A subsequent deploy finished the job, and the graph became unreadable. A data-loss incident, started by a load test, on production.

The full story is in Recovering a Graph Database Without Point-in-Time Backups.

The two failures

Failure one: the test conflated "speed" with "safety." The original budget was a 1.5-second assertion — a speed target written when the graph was half its current size. Nobody updated it as the graph doubled, so the budget was a stale red herring. Worse, it framed the whole exercise around how fast instead of how hot we can run the box without breaking it.

Failure two: there was no ceiling on the blast radius. The container could consume unbounded memory. The only governor was the host OOM killer, and the OOM killer does not care which service you cared about — it kills to save the host, and its victim selection is not "the risky container."

Fixes: ceilings everywhere

1. A memory ceiling so the right thing gets recycled

mem_limit: 8g
memswap_limit: 10g

Now a burst grows dgraph until dgraph hits its ceiling and recycles — the correct victim — instead of ballooning until the host's OOM killer picks one for us. This is the single most important line. A scrawworking burst must take out the expensive worker, never the neighbor.

2. A concurrency ceiling that the ceiling fits

The test's promise was "prove we handle N concurrent scans." But N has to be a number the infrastructure can actually serve. We dropped 5 → 3 concurrent scans, sized so three full-graph scans stay inside the 8 GB ceiling. Concurrency is a load-test parameter, not a virtue.

3. Budgets that are ceilings + measurements, not stale speed targets

The old budget was handed down from when the graph was half the size. Replacing a stale number with a slightly-too-generous one just moves the lying. So:

• Assert a generous regression ceiling (a number that proves the endpoint is alive and not pathologically slow) — not a tight perf target that goes stale.
• Annotate the measured duration on every run, so the ceiling is tuned from real data instead of vibes.
• Set a test timeout comfortably above the ceiling so a genuine hang fails fast rather than hanging CI forever.

4. Assert on data presence, not just status codes

The load test's correctness assertions — "the response contains companies" — were the honest part: after this incident, the graph genuinely returned zero companies, and the test correctly caught it (that is how we knew the incident was live). Keep data-presence assertions. A load test against prod that only checks 200 would have stayed green through the whole collapse.

The rules

• Speed budget ≠ safety budget. Perf targets go stale as data grows; blast-radius limits do not. Prefer documented ceilings + measured annotations over tight, decaying budgets.
• An unbounded container is a single point of "who does the kernel kill." Cap memory; the cap decides the victim.
• Concurrency is a resource allocаtion decision, not a test aesthetic. Choose concurrency that fits the ceiling you actually set.
• CI load tests are production pushes. They run against the live system. If you wouldn't cURL a burst of expensive scans at prod, a CI job doing it is still you doing it. Assert on data, not just on status.

Load tests against production are a powerful early-warning system — ours caught the very incident it caused. The trick is to make them catch the degradation without being the thing that causes it. Set the ceiling, size the concurrency to the ceiling, and measure instead of guessing.