Chapter 11 · concept 6 of 6 · show all

Reliability, degraded modes and launch readiness

Every part of a RAG system can fail: the embedder, the reranker, the model, the index. Plan a fallback for each and know its quality in advance. On our corpus, keyword search alone still found all 47 answers in the top 3.

A real question

The reranker service goes down at 9 am. What do users see?

Ideally, slightly worse answers, not an error. For each part, decide in advance what the system does without it, measure that mode on the golden set, and test the switch before launch.

The problem

Why this matters

The belief

If a part fails, the safest thing is to show an error.

What really happens

Each part has a fallback with known quality. Without the embedder, BM25 alone found 47 of 47 answers in the top 3 on our corpus. Without the reranker, the fused order still had all 47; only rank 1 fell from 40 to 33.

Why not the simpler way

A RAG system depends on several services: an embedding model, a search index, maybe a reranker, a language model API. Each has its own outages and slow periods. Without planning, any one of them takes the whole feature down. With planning, most failures become a small drop in quality.

The idea

The idea in plain words

Degraded modes are planned, lower-quality ways of working when a part fails. Timeouts and circuit breakers decide when to switch. Launch readiness is the checklist that proves the system, its fallbacks and its operations are ready for real users.

A lift breaks; the building still has stairs, with signs. Nobody is told the building is closed.

How it works

Step by step

1
A fallback per part
Embedder down: keyword search. Reranker down: fused order. Model down: show sources. Index down: last good build.
2
Timeouts everywhere
Every external call gets a timeout and a retry budget. A slow reranker is skipped, not waited for.
3
A launch checklist
Golden set passes, fallbacks tested, budgets met, rollback rehearsed, owners and alerts in place.

In detail

Fallback table. Part, failure signal, fallback, measured quality, user message. Keep it next to the runbook.

Circuit breakers. After N failures in a window, stop calling the part for a while and use the fallback; probe to recover.

Model fallback. A second model or provider, tested on the golden set; or sources only, clearly labelled.

Index fallback. Keep the previous build (RAG-47) and switch the alias back.

Launch checklist. Golden set and slice bars pass; injection and role tests pass (RAG-45, RAG-46); degraded modes measured and tested by switching parts off; latency and cost budgets met (RAG-49); dashboards and alerts live (RAG-48); rollback rehearsed; owners named.

The code

Try it in Python

import concurrent.futures as cf

POOL = cf.ThreadPoolExecutor(4)

def with_timeout(fn, seconds, fallback, *args):
    fut = POOL.submit(fn, *args)
    try:
        return fut.result(timeout=seconds), None
    except Exception as e:
        return fallback(*args), type(e).__name__

def retrieve(question, trace):
    vec, err = with_timeout(vector_search, 0.5, lambda q: [], question)
    trace["vector_error"] = err
    candidates = rrf([bm25_search(question), vec]) if vec else bm25_search(question)
    ranked, err = with_timeout(rerank, 2.0, lambda q, c: c, question, candidates[:10])
    trace["rerank_error"] = err                     # degraded: fused order
    return ranked[:3]

See it

What we measured

Measured · answers in the top 3 and at rank 1 per degraded mode

Polder corpus, 34 recursive 512 token chunks, 47 golden questions. Reranker: bge-reranker-base over the hybrid top 10.

Hybrid + rerank (normal)40 first47 in top 3

Hybrid, reranker off33 first47 in top 3

BM25 only, embedder off35 first47 in top 3

Vector only, keyword index off31 first44 in top 3

Measured 2026-11-04 with corpus/scripts/measure-parts-4-7.mts. Model fallbacks were not measured (not measured yet).

Try it on a real example

Degraded modes on 47 golden questions, measured

Mode

1.hybrid + rerank · 47 in top 3 · 40 firstNormal operation.✓

The baseline.

Choosing

Use it when, avoid it when

Use it when

A measured fallback for every part
Timeouts and circuit breakers on external calls
A written launch checklist

Avoid it when

Error pages when a fallback exists
Fallbacks that were never tested

What it costs · A fallback table, a few timeouts and a staging test that switches parts off.

In production

Test fallbacks on purpose: switch the reranker off in staging and confirm the system answers, the trace shows the degraded mode, and the dashboard alerts. Show users when an answer was produced in a degraded mode only if it changes what they should trust, for example "sources only, no summary".

Watch out

Common mistakes

High availability means no degraded mode. Parts fail; plan how.
Fallback quality can be assumed. Measure it; ours ranged from 44 to 47 in the top 3.
Launch readiness is a code review. It includes data, tests, budgets and people.

Check yourself

Remember this

Remember these

Plan a fallback for every part and measure it on the golden set.
Our BM25-only mode kept all 47 answers in the top 3.
When the model is down, show sources instead of an error.
Launch with a checklist: gates, fallbacks, budgets, rollback, owners.

Say it in an interview

01Each part has a fallback with measured quality on the golden set.
02On my corpus BM25 alone kept 47 of 47 answers in the top 3; vectors alone kept 44.
03Without the reranker, rank 1 fell from 40 to 33 but every answer stayed in the top 3.
04When the model is down, users get the sources, clearly labelled.
05Launch needs passing gates, tested fallbacks, met budgets, a rehearsed rollback and owners.

Quick quiz

Three questions

The embedding service is down. What is a measured fallback on our corpus?

The answer model times out. What can the system still do?

Q3 · from RAG-48

What does a trace record?

Saved on this device. No account needed.