Recovering a Graph Database Without Point-in-Time Backups

Graph databases are wonderful at representing connected data and — as this incident showed — also at making you acutely aware of the difference between "the data is on disk" and "the data is queryable." When the graph at the heart of a lead-scoring product lost the entire company graph one morning, we discovered that our point-in-time backups were not the insurance we thought they were. This is how we recovered a 5,000-node knowledge graph without a working point-in-time restore — and the layered strategy that made it recoverable at all.

Three layers of "but the data is saved"

Recovery fantasy goes deepest on graph stores, because graph data is not just rows — it is relationships, and relationships are what a partial or mis-restored backup tends to destroy. As we worked through the incident we kept hitting a pattern: each layer of backup was real in one sense and worthless in another.

Layer 1 — the live store. The graph was unreadable. Not deleted — the postings directory held 2.7 million keys. But type(Company) and has(Company.description) queries returned zero rows. The restore tool reported success on top of this, which was the trap: it wrote two million keys and the running cluster served none of them.

The forensic detail that mattered: individual uid() lookups resolved (bare nodes came back), while predicate and type-index lookups returned empty. That asymmetry — "you can fetch a node by id but not by its type" — is the signature of postings that were materialized in a format the actively-serving store cannot index/serve. On this image, this is what an Enterprise-format backup restore produces on a community build.

Layer 2 — Postgres. The app's other store. It had every application table you'd expect — contacts, campaigns, agents, a scoring ledger — and not one row of the graph. A graph rebuilt from Postgres was impossible by construction. It was scaffolding, not a source.

Layer 3 — the migration source. This was the actual recovery. The graph had been migrated from Neo4j months earlier, and that Neo4j container was still running (labeled "stale leftover"). It held the original 5,267 companies. Its restore path was not a backup at all — it was the original migration, replayed.

Recovery via the migration source

Reaching Neo4j was the hardest part, because it was bridge-only with no published port. The topology was two hosts with no direct route between the graph server and the archive host, so I chained SSH tunnels through a workstation as a relay:

# hop 1: laptop → neo4j host, forward the container's bridge IP
ssh -N -L 17687:172.21.0.2:7687 weiss@178.254.2.90 &
# hop 2: laptop → dgraph host, reverse-bind the same port
ssh -N -R 17687:localhost:17687 weiss@195.90.216.159 &

The migration script re-ran in three steps — schema, seed, copy:

1. Re-apply the schema and seed the owning company + products (the application's own reimport routine, minus the live ingestion pipeline).
2. Read every company/product/application/signal out of Neo4j.
3. Write them through the application's repository layer so predicates get the exact prefixes the GraphQL layer reads — the same lesson as graph-data-modeling-patterns: a graph's storage format and its query interface must stay aligned.

35 minutes and 5,267 companies / 20,958 signals later, the scoring endpoint was healthy: 5,240 companies, tiers computed, responses in ~2 seconds.

The layered rule for graph recovery

The single most important practice professional teams get wrong:

Do not tear down the system you migrated from until you have restored from it.

The "dead" system is your cheapest, most-proven restore. Beyond that, build recovery in layers so no single format owns your fate:

1. Portable text exports, not just binary blobs. Binary/Enterprise formats that only a subset of your images can read back are not backups you can use. A plain RDF/N-Quads + schema export loads into any store via a plain mutate endpoint. It is restorable by construction.
2. Verify content, not presence. A backup cron that only checks "file exists and is N MB" would have passed every night before this incident. Check minimum node/type counts; ideally test-restore into a scratch instance on a schedule.
3. Keep the migration source warm. A migrate-away system holding lineage is an insurance policy. Give it a cheap keep-alive lease and a documented replay procedure, not a deletion ticket.
4. A running store and a store you can serve are different things. "The data is on disk" from dgraph debug or any *_status tool is forensics, not availability. Only a query through the interface users actually use proves recovery.

Graph stores reward a layered recovery posture more than relational ones do, precisely because their value is in relationships, and relationships are what an unrestorable backup destroys first. Lean on the migration source, keep a portable export, and verify by restoring — not by reading a size column.