Chapter 12 · concept 4 of 4 · show all

GraphRAG, structured data and multimodal RAG

Not all knowledge is plain text. Relationships across documents, exact data in tables and databases, and images each need their own structure. When to use a graph, when to query data directly, and when chunks are enough.

A real question

Some answers live in links between documents, in database tables or in images. Do chunks still work?

Sometimes yes, sometimes no. Small tables kept whole work as chunks. Exact current data is better queried directly. Questions about relationships across many documents may need a graph. Pick the structure the question needs.

The problem

Why this matters

The belief

Every serious RAG system needs a knowledge graph.

What really happens

All 8 of our table questions found their answer in the top 3 with plain 512 token chunks. Graphs help with questions about relationships and themes across many documents, and cost an extraction step over the whole corpus.

Why not the simpler way

Text chunks are a good default, and this course showed how far they go. Some questions break them: "summarise all incidents this year", "which suppliers are linked to this contract", "what is the balance right now", "what does this diagram show". Each needs a different kind of retrieval.

The idea

The idea in plain words

GraphRAG (Edge et al., "From Local to Global: A Graph RAG Approach to Query-Focused Summarization", arXiv 2404.16130) uses a model to extract entities and relations from the corpus, builds a graph, groups it into communities and summarises them, so questions about the whole corpus can be answered. Structured RAG retrieves from databases and APIs. Multimodal RAG retrieves images, scans and diagrams alongside text.

A detective's board with photos and string: useful for connecting people and events, unnecessary for looking up one phone number.

How it works

Step by step

1
Entities and dates as metadata
Incident ids, product codes and dates on each chunk let you filter and order, a light form of structure (RAG-18).
2
Query exact data
Prices, balances and stock levels live in databases. Let the model call a query tool (text-to-SQL or a fixed API) instead of embedding them.
3
Images and scans
Caption images or extract their text at ingestion, or use a model that embeds images; keep the link to the original.

In detail

Graph options. Light: entity and date metadata on chunks, filtered and grouped at query time. Full: model-extracted entities and relations, community summaries, graph traversal. Full graphs cost a model pass over the corpus and need rebuilding as it changes.

Structured data. A query tool with a fixed schema, called by the model (RAG-53). Prefer parameterised queries over free text-to-SQL; validate and restrict to read-only.

Multimodal. At ingestion: OCR for scans (RAG-08), captions or descriptions for images, or a joint image-text embedding model. Store the image link so the answer can show it.

The code

Try it in Python

import sqlite3
from datetime import date

PRICE_TOOL = {
    "name": "plan_price",
    "description": "Monthly price of a plan code on a date. Use for exact prices.",
    "input_schema": {"type": "object",
                     "properties": {"plan": {"type": "string"}, "on": {"type": "string", "format": "date"}},
                     "required": ["plan"]},
}

def plan_price(db: sqlite3.Connection, plan: str, on: str | None = None):
    on = on or date.today().isoformat()
    row = db.execute(
        "SELECT price_eur, source_doc FROM prices WHERE plan = ? AND valid_from <= ? "
        "AND (valid_to IS NULL OR valid_to > ?) ORDER BY valid_from DESC LIMIT 1",
        (plan, on, on)).fetchone()                       # parameterised, read-only
    return {"plan": plan, "on": on, "price_eur": row[0], "source": row[1]} if row else None

See it

What we measured

Measured · question types on our corpus

Polder corpus, 34 recursive 512 token chunks, 47 golden questions, vector search (bge-small).

Table questions in the top 38 of 8

Multi-document questions in the golden set1

Price similarity, EUR 20.00 against 21.000.942

Images in the corpus0

GraphRAG on this corpusnot measured

Measured 2026-11-04 with corpus/scripts/measure-parts-4-7.mts. Our corpus is text with tables and has no images, so multimodal retrieval is described, not measured.

Try it on a real example

Three kinds of question, and the structure each needs

Question

“What was the cost to income ratio in 2024?”

The bank's key figures table, kept whole in one 512 token chunk.

Plain chunks work: table questions were 8 of 8 in the top 3. Keep tables whole (RAG-09, RAG-14).

Choosing

Use it when, avoid it when

Use it when

Plain chunks with whole tables as the default
Query tools for exact, live data
A graph for relationship and corpus-wide questions the golden set shows failing

Avoid it when

Embedding live numbers that change daily
Building a full knowledge graph before measuring the need

What it costs · Metadata: cheap. Query tools: a schema and a tool. Full GraphRAG: a model pass over the corpus and regular rebuilds.

In production

Add question types to the golden set before adding structure: relationship questions, corpus-wide summaries, live data, images. Measure plain chunks on them first. If you build a graph, budget its extraction cost and refresh cycle, and test that answers cite documents, not only graph nodes.

Watch out

Common mistakes

Tables need special RAG. Kept whole, ours worked as plain text.
Graphs replace chunks. Most graph systems still retrieve text chunks to answer.
Multimodal means sending every image to the model. Index captions and text; fetch images when needed.

Check yourself

Remember this

Remember these

Small tables kept whole work as chunks: 8 of 8 here.
Query exact, current data instead of embedding it.
Graphs help with relationship and corpus-wide questions.
Images need captions, extracted text or an image embedding model.

Say it in an interview

01I match the retrieval structure to the question: chunks, a database query or a graph.
02On my corpus all 8 table questions worked with plain 512 token chunks kept whole.
03Exact live values like prices come from a query tool, not from embeddings.
04I would add a graph for relationship and corpus-wide questions, after measuring that chunks fail.
05For images I index captions or extracted text and keep a link to the original.

Quick quiz

Three questions

Our 8 table questions with 512 token chunks found how many answers in the top 3?

"What does PT-MOB-25GB cost today?" is best answered from where, if the data exists in a database?

Q3 · from RAG-09

Why can a table split across chunks give wrong numbers?

Saved on this device. No account needed.