Your Post-Deploy Health Check Is a Cold-Start Lie

The error was unambiguous: the deploy's health check failed.

Error: Health check — exit code 28

Exit code 28 is curl's CURLE_OPERATION_TIMEDOUT. A fresh container, a freshly built image, and the ready probe fired a single request the moment the container came up — and the container, busy warming caches, loading a graph namespace from disk, and resolving first-page routes, did not answer inside the window. The pipeline declared the deploy broken. Seconds later the same endpoint answered in 150 milliseconds.

This is the cold-start lie, and it is quietly one of the most common reasons graph services fail their own deployments.

Why graph deployments are especially prone to it

Cold-start latency is worst exactly where a graph store's data model is strongest: rich, interconnected state that must be materialized before a query can be answered.

• Namespace / index load from disk. Right after boot, a graph database may rebuild or page in indexes, adjacency structures, and caches before the first query can be served at speed.
• Application warm-up. The application layer initializes connection pools, compiles query plans, and warms caches. The first request often pays the cost of all of that.
• TLS / routing. Behind a reverse proxy, the very first request may also race certificate provisioning and route registration.

So the naive health check — "boot container, immediately curl once, fail if not 200" — measures warm-up, not availability. The service was up and callable; it was just not yet fast.

How to tell a warm-up from a real outage

The distinguishing question is not did it answer in time but did it converge to healthy quickly:

1. Introduce backoff before you first conclude failure. Replace the single request with a bounded retry loop. Twelve attempts with a few seconds of sleep cover a cold start without hiding a persistent outage:

   for i in $(seq 1 12); do
     if curl -s -o /dev/null -w "%{http_code}" "$URL" | grep -q "200"; then
       echo "health check passed (attempt $i)"
       exit 0
     fi
     echo "waiting… attempt $i/12"
     sleep 5
   done
   echo "health check failed"
   exit 1
   

2. Use a warm-up endpoint that is cheap but representative. A headers-only or status probe that forces the same route construction as a real page gives you availability without a full payload.

3. Time-box against convergence, not instant success. If the service is healthy within the retry window, that is success. Failing only when it never converges is the correct semantic — and it is also what your incident paging should watch.

The second-order benefit: it makes flaky deploys actionable

A health check that fails on warm-up sees a single failure mode covered with noise: every deploy either passes instantly or "fails" then recovers. That trains everyone to ignore the check. Once backoff is in, the check converges cleanly, so a genuine red — exit 28 persisting across all twelve attempts — finally means something, and paging on it is credible.

In our graph deployment, the fix was precisely this: the container was warming a graph namespace with several minutes of index materialization. The first instant probe tripped curl's timeout; a backoff loop stopped treating cold-start latency as downtime. The deploy pipeline went from a post-deploy bespoke failure to a boring green, and the next time a health check was slow we could reason about it instead of assuming the site was down.

Rules for health checks that don't lie

• A single immediate probe is not a health check — it's a warm-up speed test. Availability is a convergence property, so measure convergence.
• Bound the retry, don't loop forever. A bounded backoff (e.g., 12×5s in the snippet above) absorbs cold starts and still fails loudly on real outages.
• Never let a warm-up-timeout survivor of one deploy define your baseline. Log the convergence time and alert on trends, not on the first-slow-ever case.

The exit-28 that "broke" our deploy was never an outage. It was a measurement that confused not-yet-fast with down. Once the health check started asking the right question — did the service converge — it stopped lying, and the pipeline stopped breaking deployments that were, in every meaningful way, already healthy.