Deploying GraphRAG in Production: Architecture, Cost, and Operations
graphragproductionneo4jknowledge-graphsretrieval-augmented-generationarchitecturedeployment
Loading…
GraphRAG demos are compelling. Show a system that answers "How does this API change affect the three downstream services that depend on it?" — a multi-hop question that vector search cannot answer — and stakeholders are convinced. The hard part comes after the demo: turning a Jupyter notebook into a system that runs reliably at 100 queries per second, costs a predictable amount to operate, and stays fresh as documents change.
By mid-2026, enough GraphRAG systems have been deployed in production to separate myth from pattern. This article synthesises real deployment lessons into a practical guide covering architecture decisions, cost modelling, incremental indexing, query reliability, and the operational practices that keep GraphRAG systems healthy.
The first decision in any production deployment is when to use GraphRAG at all. Every deployment that has scaled successfully applies GraphRAG as an additive backend, not a wholesale replacement for vector RAG.
| Query Type | Recommended Method | Why |
|---|---|---|
| Simple fact retrieval | Vector RAG | Cheaper, faster, no graph overhead |
| Single-document Q&A | Vector RAG | Dense retrieval is sufficient |
| Multi-hop (2–3 hops) | GraphRAG + vector hybrid | Graph traversal narrows candidates, vectors fill context |
| Cross-document synthesis | GraphRAG (global mode) | Community summaries aggregate across documents |
| Relational queries ("which X depends on Y?") | GraphRAG (template Cypher) | Graph traversal is the only reliable approach |
| High-frequency factual lookups | Cache-Augmented Generation | No retrieval needed for known patterns |
The well-documented rule from production deployments: roughly 7% of queries genuinely require graph traversal. For the other 93%, vector RAG with a good reranker is faster and cheaper. A routing classifier — a small LLM or a traditional classifier trained on query logs — directs each query to the appropriate pipeline.
After a year of production GraphRAG adoption, three architectural patterns recur across successful deployments.
The most common pattern. A routing layer decides per-query whether to activate graph traversal or stay in vector space. The graph and the vector index operate on the same corpus but serve different query profiles.
┌─────────────────┐
│ Query Router │
│ (Classifier) │
└──┬──────────┬───┘
│ │
┌────▼──┐ ┌────▼────┐
│Vector │ │ Graph │
│Retrie.│ │Traversal│
│(Qdrant│ │ (Neo4j) │
│/pgvec)│ │ │
└───┬───┘ └────┬────┘
│ │
└────┬─────┘
┌────▼─────┐
│ Reranker │
│ (Cross- │
│ encoder) │
└────┬─────┘
┌────▼─────┐
│ LLM │
│ Synthesis│
└──────────┘
The reranker is critical: it merges graph traversal results (entity IDs, relationship paths) with vector search results (document chunks) into a unified context window, deduplicating and scoring by relevance. Production systems using a cross-encoder reranker in this fusion topology report 34% higher answer accuracy than vector-only retrieval while maintaining p95 latency under one second for 90% of queries.
Extraction cost — the LLM calls needed to turn documents into entities, relationships, and community summaries — dominates GraphRAG operating budgets. A layered extraction pattern uses a small, cheap model (a 1–3B parameter model) for bulk entity extraction, and reserves a frontier model only for community summarisation and high-value relationship inference.
# Layered extraction: small model for bulk, frontier model for synthesis
from neo4j_graphrag.experimental.pipeline.kg_builder import SimpleKGPipeline
from neo4j_graphrag.llm import OpenAILLM
# Small model for entity/relation extraction
extraction_llm = OpenAILLM(model_name="gpt-4o-mini", max_tokens=2000)
# Frontier model for community summarisation
summary_llm = OpenAILLM(model_name="gpt-4o", max_tokens=8000)
pipeline = SimpleKGPipeline(
driver=driver,
llm=extraction_llm,
embedder=embedder,
entities=["Customer", "Order", "Product", "Supplier"],
relations=["PLACED", "CONTAINS", "SOURCED_FROM"],
enforce_schema="STRICT"
)
The saving is roughly an order of magnitude on extraction cost. Community summarisation remains the most expensive single stage — but it runs on the aggregated graph, not on every document, so its cost scales with graph complexity rather than corpus size.
In enterprise environments where different departments own different data domains, a single monolithic graph creates coordination bottlenecks. The federated pattern lets each domain maintain its own graph with its own schema, plus an upper graph of common entities that bridges them.
This pattern survives reorganisations without full re-ingestion: when one department restructures its ontology, only its graph needs rebuilding. The bridging graph handles cross-domain queries by mapping entities to a shared vocabulary.
One of the most debated decisions in GraphRAP production deployments is the storage backend. The conventional choice is Neo4j, but PostgreSQL — particularly with pgvector and the AGE extension — has gained significant ground.
| Criterion | Neo4j | PostgreSQL + pgvector | PostgreSQL + AGE |
|---|---|---|---|
| Native graph traversal | ✓ | ✗ (app-level joins) | ✓ (Cypher via AGE) |
| Vector index | ✓ (HNSW in GDS) | ✓ (HNSW/IVF via pgvector) | ✗ (use pgvector alongside) |
| Multi-hop query latency | ~45ms p50 | ~200ms p50 (app joins) | ~80ms p50 |
| Ops complexity | Additional DB to manage | Familiar to most teams | Familiar + graph extension |
| Scaling cost (1K QPS, monthly) | ~$4,800 (6 replicas) | ~$1,800 | ~$2,400 |
| Community detection | ✓ (GDS: Leiden, Louvain) | ✗ | ✗ (limited) |
For mid-size graphs under a million entities, PostgreSQL with well-tuned indexes is operationally sensible and economically attractive. You avoid adding another database to your inventory, and many teams already run PostgreSQL in production. The trade-off is that deep multi-hop traversals (4+ hops) become expensive because they require application-level joins rather than constant-time pointer chasing.
For graphs exceeding a million entities, or when you need pathfinding algorithms (shortest path, PageRank, community detection), Neo4j's native graph engine and the Graph Data Science library justify the additional operational cost. As covered in Graph Algorithms in Production with Neo4j GDS, GDS provides over 50 in-memory algorithms that cannot be replicated in PostgreSQL without significant engineering effort.
The single biggest surprise for teams deploying GraphRAG in production is the 10–30x ingestion cost premium over vectorising the same corpus. LLM calls for entity extraction, relationship inference, and community summarisation dominate the budget. In representative production runs, these two stages account for approximately 90% of total build cost and runtime.
Build a cost model before you deploy. Run one full indexing pass on a representative corpus sample, then extrapolate:
| Stage | Cost Driver | Optimisation Levers |
|---|---|---|
| Document chunking | Negligible | — |
| Entity extraction | LLM tokens per document | Use small model; trim prompts; batch with async I/O |
| Relationship inference | LLM tokens per chunk pair | Dependency parsing as first pass; LLM only for low-confidence edges |
| Embedding generation | API cost per chunk | Negligible compared to LLM stages |
| Community detection | Free (Leiden in GDS) | — |
| Community summarisation | LLM tokens per community | Most expensive single stage; limit to top-N communities |
| Query-time LLM call | Tokens per query | Cache frequent queries; use smaller synthesis model |
The practical optimisation hierarchy:
The most common production failure in GraphRAG is unreliable Cypher generation. Letting an LLM write Cypher queries directly produces correct results roughly 77% of the time — impressive for a demo, unacceptable for production.
The proven alternative is a template-based approach:
-- Template for: "Which [target] are connected to [source] via [path]?"
MATCH (source:{{source_label}} {name: $source_name})
{{path_template}}
RETURN DISTINCT target.name AS result
LIMIT $limit
This approach drops query errors from 23% to under 4%. The trade-off is flexibility — new query patterns require new templates. Production systems maintain roughly 30 templates and add two to three new ones per month based on query log analysis.
Additional reliability measures:
Full re-indexing of a production knowledge graph is expensive. Incremental indexing — processing only document deltas — is essential for operational viability.
The key insight from production deployments is that freshness in a knowledge graph is not just about keeping data current. In systems where users ask "What changed?" or "Why does this behave differently now?", the graph must be designed so that evolution is visible and traceable, not just overwritten.
# Incremental update: process only new and modified documents
def incremental_index(pipeline, last_run_timestamp: datetime):
new_docs = fetch_documents(updated_since=last_run_timestamp)
for doc_batch in batched(new_docs, size=50):
asyncio.run(pipeline.run_async(doc_batch))
# Recompute only affected communities
recompute_communities(affected_entity_ids=detect_changed_entities())
An explicit operational policy should specify:
Community structure is particularly sensitive to incremental updates. Most production systems use a hybrid approach: schedule community re-detection on affected subgraphs weekly, while preserving stable community IDs elsewhere to maintain query consistency.
GraphRAG requires observability dimensions that vector-only RAG does not. If you are monitoring GraphRAG with the same metrics you use for vector RAG, you are blind to the failure modes that matter.
| Metric | Why It Matters | Alert Threshold |
|---|---|---|
| Graph traversal p50/p95 | Deep traversals can explode without bounds | Alarm if p95 > 2s |
| Community selection accuracy | Wrong community → irrelevant context | Sample 100 queries weekly |
| Edge confidence distribution | Low-confidence edges degrade quality | Flag edges < 0.5 confidence |
| Stale-node ratio | Outdated entities mislead queries | Rebuild if > 10% stale |
| Cache hit rate (community summaries) | Repeated LLM calls waste budget | Optimise if < 60% |
| Query routing accuracy | Wrong pipeline → wrong answer | Monitor via A/B comparison |
Semantic caching is more important in GraphRAG than in basic RAG because many queries repeatedly touch the same high-value entities, communities, and summaries. Cache not just final answers but also intermediate artifacts: normalised entities, extracted triples, community summaries, and graph traversal results.
Cache invalidation policies must account for graph drift. Production systems typically implement time-based expiration combined with change detection on community membership: when a community grows or shrinks beyond a threshold, its cached summary is marked stale and regenerated on the next access.
A production GraphRAG deployment that embodies these patterns looks like this:
| Layer | Component | Purpose |
|---|---|---|
| Ingestion | Async extraction pipeline (small LLM) | Entity and relationship extraction from source documents |
| Storage | Neo4j (entity graph) + pgvector (dense vectors) | Graph traversal and semantic search |
| Indexing | GDS Leiden community detection | Hierarchical community partitioning |
| Query classification | Lightweight classifier (~30 templates) | Route query to graph, vector, or cache |
| Retrieval | Template-based Cypher + vector HNSW | Reliable graph traversal and semantic search |
| Fusion | Cross-encoder reranker | Merge and rank graph + vector results |
| Generation | LLM synthesis (small model for most queries) | Context-grounded answer generation |
| Cache | Redis (semantic + summary cache) | Reduce repeated LLM calls |
| Observability | OpenTelemetry + custom GraphRAG metrics | Traversal depth, routing accuracy, cost per query |
| CI/CD | Versioned Cypher templates + eval suite | Regression-free schema and query updates |
Scale this architecture incrementally. Start with a single Neo4j instance and a vector index, prove value on a bounded domain, then expand. The teams that succeed with GraphRAG in production are the ones that treat it as an operational platform from day one — with explicit cost budgets, update cadences, monitoring, and rollback plans — rather than as a research project that somehow escaped the notebook.
For a deeper look at the GraphRAG variants available today, see GraphRAG Variants Compared in 2026. For a cost-efficient alternative, read Lazy GraphRAG on $30.