Introduction to GraphRAG: Combining Knowledge Graphs with RAG

The Problem with Vanilla RAG

Retrieval-augmented generation (RAG) solved a real problem: LLMs hallucinate when asked about facts not in their training data. By retrieving relevant documents and injecting them into the prompt, RAG grounds the model in external knowledge.

But vanilla RAG has blind spots:

• Chunk isolation: Documents are split into chunks and embedded independently. Related facts across chunks are lost. A passage about "the CEO's resignation" in one chunk and "the company's stock drop" in another are never connected.
• Entity ambiguity: "Apple" — the fruit or the company? Vector similarity cannot distinguish between homographs without explicit entity resolution.
• Multi-hop reasoning: "Which employees of companies founded in 2015 worked on GraphRAG papers?" requires joining facts across multiple documents. Vanilla RAG retrieves the most similar chunks independently, then leaves the LLM to piece together relationships the retriever never saw.
• No structure: Retrieved chunks are flat text. Relationships, hierarchies, and provenance are invisible to the LLM. The model must infer connections from raw text, which is exactly the task that leads to hallucination.

Consider a concrete scenario. A medical researcher asks: "Which drugs approved by the FDA in 2024 target the same biological pathway as Drug X?" A vanilla RAG system retrieves chunks about Drug X and about FDA approvals, but it cannot connect the dots because the relationship "targets_same_pathway" exists only as an implicit inference across separate documents. The LLM must guess — and guessing is where hallucination begins.

What Is GraphRAG?

GraphRAG replaces flat vector retrieval with graph-based retrieval. Instead of relying solely on embedding similarity to find relevant text, it explicitly models entities and their relationships in a knowledge graph, then traverses that graph during retrieval.

The core idea is simple:

1. Build a knowledge graph from your documents (entities become nodes, relationships become edges)
2. Index both the graph structure and the document text (vector indexes for semantic search, graph indexes for structural traversal)
3. On query: Traverse the graph to find relevant sub-graphs AND retrieve related documents
4. Feed both structured sub-graph and document context to the LLM

Query: "What security issues exist in GraphRAG implementations?"

Vector search: [3 chunks about security, partially relevant]
Graph search:  [sub-graph: GraphRAG → implementations → security_audit → CVE-2026-XX]
               [linked documents: "Security audit of GraphRAG v2.1", "CVE report 2026"]

LLM context: structured data + related documents
Output: grounded, multi-hop answer

The critical distinction is that the graph traversal is deterministic — the relationships between entities are explicit edges, not probabilistic similarity scores. When the graph says (:Vulnerability {id: "CVE-2026-XX"})-[:AFFECTS]->(:Technology {name: "GraphRAG"}), that fact was either extracted from a source document or it does not exist. There is no "maybe."

The GraphRAG Pipeline

1. Entity Extraction

The first stage processes source documents to identify entities and the relationships between them. In production, this is typically done by an LLM guided by a structured extraction schema:

import json
from openai import OpenAI

EXTRACTION_SCHEMA = {
    "type": "object",
    "properties": {
        "entities": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "type": {"type": "string", "enum": ["Technology", "Vulnerability", "Organisation", "Person", "Concept"]},
                    "description": {"type": "string"}
                },
                "required": ["name", "type"]
            }
        },
        "relationships": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "source": {"type": "string"},
                    "target": {"type": "string"},
                    "type": {"type": "string"}
                },
                "required": ["source", "target", "type"]
            }
        }
    }
}

def extract_graph(text: str) -> dict:
    client = OpenAI()
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "system",
            "content": "Extract entities and relationships from the text. "
                       "Be precise — only extract facts explicitly stated."
        }, {
            "role": "user",
            "content": text
        }],
        response_format={"type": "json_schema", "json_schema": EXTRACTION_SCHEMA}
    )
    return json.loads(response.choices[0].message.content)

The schema-first approach is important. Without a schema, the LLM produces inconsistent entity types — sometimes "Company", sometimes "Organisation", sometimes "Organization". A constrained schema normalises the output and makes the graph queryable without guesswork.

2. Graph Construction

Once entities and relationships are extracted, they are loaded into a graph database. For Neo4j, this means creating nodes with labels, edges with types, and database-level constraints to enforce data integrity:

// Create constraints to prevent duplicates
CREATE CONSTRAINT IF NOT EXISTS FOR (t:Technology) REQUIRE t.name IS UNIQUE;
CREATE CONSTRAINT IF NOT EXISTS FOR (v:Vulnerability) REQUIRE v.id IS UNIQUE;

// Create indexes for fast lookups
CREATE INDEX IF NOT EXISTS FOR (e:Entity) ON (e.type);
CREATE FULLTEXT INDEX entities_text IF NOT EXISTS
FOR (n:Technology|Vulnerability|Organisation)
ON EACH [n.name, n.description];

Batch insertion is handled in application code. In Python:

def load_extractions(tx, entities, relationships):
    for entity in entities:
        tx.run(
            f"MERGE (e:{entity['type']} {{name: $name}}) "
            "SET e.description = $description",
            name=entity["name"], description=entity.get("description", "")
        )
    for rel in relationships:
        tx.run(
            "MATCH (s {name: $source}), (t {name: $target}) "
            f"MERGE (s)-[r:{rel['type']}]->(t) "
            "SET r.source_document = $doc_id",
            source=rel["source"], target=rel["target"], doc_id=doc_id
        )

with driver.session() as session:
    for doc_id, text in enumerate(documents):
        extracted = extract_graph(text)
        session.execute_write(load_extractions, extracted["entities"], extracted["relationships"])

The MERGE pattern is critical — it creates nodes and relationships only if they do not already exist, making the ingestion idempotent.

3. Hybrid Indexing

A production GraphRAG system needs three kinds of index:

Index Type	Purpose	Neo4j Implementation
Vector index	Semantic similarity on document chunks and entity descriptions	CREATE VECTOR INDEX ... FOR ()-[r:TEXT_EMBEDDING]-() ON r.embedding
Full-text index	Exact keyword and phrase matching	CREATE FULLTEXT INDEX ...
Graph index	Relationship traversal for structural queries	Native property graph (edges are first-class)

The vector index on entity descriptions enables a crucial hybrid search pattern: find entities semantically related to the query, then traverse their relationships to collect structured context. This combines the flexibility of embedding similarity with the precision of graph traversal.

CREATE VECTOR INDEX entities_vector IF NOT EXISTS
FOR (e:Entity) ON (e.description_embedding)
OPTIONS {indexConfig: {
  `vector.dimensions`: 1536,
  `vector.similarity_function`: 'cosine'
}};

4. Hybrid Retrieval

When a query arrives, the system performs both vector search and graph traversal, then merges the results. This is the heart of GraphRAG.

A hybrid retrieval query in Cypher:

// Step 1: Find entities semantically related to the query
CALL db.index.vector.queryNodes('entities_vector', 5, $query_embedding)
YIELD node AS entity, score
WITH entity, score

// Step 2: Traverse from matched entities to collect context
OPTIONAL MATCH (entity)-[r]->(related)
WHERE score > 0.7

// Step 3: Return structured results
RETURN entity.name AS entity_name,
       labels(entity)[0] AS entity_type,
       type(r) AS relationship_type,
       related.name AS related_entity,
       score AS similarity
ORDER BY score DESC

The same operation in Python using neo4j-graphrag:

from neo4j_graphrag.retrievers import VectorCypherRetriever

retriever = VectorCypherRetriever(
    driver=driver,
    index_name="entities_vector",
    retrieval_query="""
        OPTIONAL MATCH (node)-[r]->(related)
        RETURN node.name AS entity,
               labels(node)[0] AS type,
               collect({rel: type(r), target: related.name}) AS relationships
    """,
)

results = retriever.search(query_text="What security issues affect GraphRAG?", top_k=5)

For even better results, weighted reciprocal rank fusion (WRRF) merges the ranked lists from vector, full-text, and graph signals — boosting results that rank well across multiple retrieval methods.

5. Context Assembly

The final stage combines retrieved sub-graphs and document chunks into a structured prompt:

def assemble_context(subgraph, chunks):
    context = "# Relevant Documents\n\n"
    for chunk in chunks:
        context += f"{chunk.text}\n\n"

    context += "# Knowledge Graph Context\n\n"
    for row in subgraph:
        context += f"- {row['entity']} ({row['type']})"
        for rel in row['relationships']:
            context += f" --[{rel['rel']}]--> {rel['target']}"
        context += "\n"

    return context

The full prompt instructs the LLM to answer only from the provided context:

You are a helpful assistant. Answer the question using ONLY the information
provided in the documents and knowledge graph below.
If the information is insufficient, say so.

{context}

Question: {question}
Answer:

This separation of concerns — facts come from deterministic graph operations, the LLM handles only language generation — is what makes GraphRAG auditable and legally defensible.

Local vs Global Search

Microsoft's GraphRAG paper introduced a distinction that has become standard in the field:

• Local search focuses on a specific entity or small sub-graph. The query "What is the latest CVE affecting Neo4j?" traverses from the Neo4j entity node, follows AFFECTED_BY relationships to vulnerability nodes, and returns their properties. Local search is fast — typically one or two hops — and produces highly precise answers.

• Global search summarises themes across the entire corpus. A query like "What security trends emerged in graph databases during 2026?" requires community detection. The Leiden algorithm partitions the graph into communities of related entities. Each community is summarised by an LLM, and the summaries are merged to answer the query.

Property	Local Search	Global Search
Query scope	Single entity or small sub-graph	Entire corpus themes
Traversal depth	1-3 hops	Uses pre-computed community summaries
Latency	10-100 ms	1-10 seconds (amortised by pre-computation)
Best for	Factual Q&A, entity lookups	Trend analysis, corpus-level summarisation
Example	"What does Company X do?"	"What are the major trends in AI regulation?"

In practice, most production GraphRAG deployments default to local search and fall back to global search only when the query is clearly thematic. The community summaries for global search are pre-computed during indexing.

Implementation Options

Approach	Indexing Cost	Query Cost	Best For	Trade-off
Microsoft GraphRAG	High (LLM for entity extraction + community summaries)	Low (pre-computed)	Large corpora, global queries	Expensive indexing, powerful retrieval
Custom (Neo4j + LangChain)	Medium (schema-based extraction)	Low	Domain-specific apps, precise control	More engineering work upfront
LazyGraphRAG	Near zero (noun-phrase extraction only)	Medium (LLM at query time)	Streaming data, cost-sensitive teams	Higher per-query latency
LightRAG	Medium (efficient extraction)	Low	Balanced cost-quality	Less mature ecosystem

Microsoft's GraphRAG

Microsoft Research's GraphRAG paper introduced an automated pipeline that:

• Extracts entity communities using Leiden clustering
• Generates community summaries at multiple hierarchical levels
• Answers queries at both global and local scope

Best for: Large document corpora, question answering over broad topics

Custom GraphRAG with Neo4j + LangChain

For production systems, a custom implementation gives more control:

from langchain_community.graphs import Neo4jGraph
from langchain.chains import GraphCypherQAChain

graph = Neo4jGraph(url="bolt://localhost:7687", username="neo4j", password="...")
chain = GraphCypherQAChain.from_llm(llm=llm, graph=graph)
result = chain.invoke("What security issues affect GraphRAG?")

Best for: Domain-specific applications, precise control over graph schema

LazyGraphRAG

LazyGraphRAG defers LLM extraction to query time, eliminating indexing costs entirely. For a million-document corpus, indexing drops from approximately $30,000 to roughly$30.

Best for: Cost-sensitive teams, streaming data, exploratory analysis

Production Considerations

Entity resolution. Two documents about the same company might use different names ("Google", "Alphabet", "GOOGL"). A production pipeline needs a normalisation step: lower-case, strip legal suffixes, or use an external entity linking service like Wikidata.

Incremental updates. Documents change. If you re-extract entities from an updated document, old entities may become orphaned. Use MERGE with document-level soft deletion: mark old entities as stale: true rather than deleting them, so queries can filter with WHERE e.stale <> true.

Chunking strategy for hybrid retrieval. Document chunks used for vector search should overlap by 10-15%% to avoid splitting entity mentions across chunk boundaries. Entity extraction should run on the full document, not per-chunk, to capture cross-chunk relationships.

Measuring coverage. Track what percentage of query entities resolve to graph nodes. If coverage drops below 80%%, the extraction pipeline is missing too many entities and retrieval quality will suffer:

MATCH (q:QueryLog)
WHERE q.timestamp > datetime() - duration({days: 7})
RETURN q.resolved_entities / q.total_entities AS coverage_rate

When to Use GraphRAG

GraphRAG shines when:

• Your data has rich entity relationships
• Multi-hop reasoning is required
• Entity disambiguation matters
• You need provenance (which document supports this fact?)
• Legal or compliance requirements demand auditable AI outputs

It is overkill when:

• You are answering simple factoid questions
• Your documents have no relational structure
• Latency is the primary concern (graph traversal adds overhead)
• Your total corpus fits in a single LLM context window

The emerging consensus from production deployments is hybrid: route simple factual lookups to vector search and complex reasoning queries to graph traversal. The retrieval cost savings alone justify the dual architecture.

Further Reading

• GraphRAG Reality Check: When It Fails, Why, and How to Fix It — Evidence-based analysis of GraphRAG's failure modes with concrete mitigations
• GraphRAG for $30: Lazy Extraction That Actually Works — How LazyGraphRAG collapses indexing costs by deferring extraction to query time
• Knowledge Graphs Are the Antidote to AI Hallucination Liability — Why GraphRAG is the only legally defensible architecture after the Munich ruling
• GraphRAG at Mythos Scale: What Claude Fable 5 Means for Knowledge Graphs — How Mythos-class reasoning lifts the last bottleneck on GraphRAG adoption
• Knowledge Graphs as the Context Layer for AI Agents — Why every major AI platform is adopting graph-based context layers
• What Are Knowledge Graphs? — The foundational concepts behind GraphRAG's core data structure