31  The Golden Layer

The RAG pipeline from the last few chapters handles any question, but “any question” is not how a doc-bot gets used. A handful of questions (how do I enable batch consumption? what’s the connection-pool setting?) arrive hundreds of times a day, and for each one the pipeline embeds, searches, calls the model, and produces a slightly different wording. For your highest-traffic, highest-stakes questions you want the answer a human already reviewed, served unchanged — and you want it before a model gets a chance to rephrase it.

The golden layer is a small set of curated, human-reviewed answers that a query checks before RAG. A close, current match ends the run with the curated answer: no document retrieval, no chat-model generation, no rewording. (The golden check still embeds the question to find its match and verifies the source versions.) Anything less than close-and-current falls through to the ordinary pipeline.

One thing to know about the design before any of its mechanics: a store of vetted answers is a cache, and caches lie. They lie confidently, instantly, and with no model in the loop to hedge. So this chapter builds the cache and, in the same node, the guard that refuses it.

31.1 Two stores, one cascade

The doc-bot now keeps two knowledge bases side by side, and they are not the same kind of thing:

Store What it holds Character
docs the source documents the source of truth
golden curated Q&A, distilled from feedback and human-reviewed fast, trusted, but derived, so it can go stale

Putting one in front of the other is a single conditional edge, the primitive from Chapter 5. check_golden runs first; a hit ends the run, a miss routes onward.

%%{init: {'securityLevel': 'loose', 'flowchart': {'diagramPadding': 24}, 'themeCSS': 'foreignObject{overflow:visible}span.bot{display:flex;justify-content:flex-start;align-items:flex-start;width:100%;height:0;overflow:visible;line-height:0}span.bot img{display:block!important;width:18px!important;height:18px!important;padding:2px;background:#fff;border-radius:50%;box-shadow:0 0 0 1px #999}span.bot.r img{margin:0;transform:translate(-24px,-24px)}span.bot.d{justify-content:center}span.bot.d img{margin-top:-50%;transform:translateY(-24px)}'}}%%
flowchart LR
    START(["START"]) --> check_golden["check_golden"]
    check_golden -->|"hit: close + fresh"| END(["END"])
    check_golden -->|"miss / stale"| retrieve_docs["retrieve_docs"]
    retrieve_docs --> generate["<span class='bot r'><img src='data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIyNCIgaGVpZ2h0PSIyNCIgdmlld0JveD0iMCAwIDI0IDI0IiBmaWxsPSJub25lIiBzdHJva2U9IiMzMzMiIHN0cm9rZS13aWR0aD0iMiIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIj48cGF0aCBkPSJNMTIgOFY0SDgiLz48cmVjdCB3aWR0aD0iMTYiIGhlaWdodD0iMTIiIHg9IjQiIHk9IjgiIHJ4PSIyIi8+PHBhdGggZD0iTTIgMTRoMiIvPjxwYXRoIGQ9Ik0yMCAxNGgyIi8+PHBhdGggZD0iTTE1IDEzdjIiLz48cGF0aCBkPSJNOSAxM3YyIi8+PC9zdmc+'/></span>generate"]
    generate --> END(["END"])

On the top branch the run reaches END without touching retrieve_docs or generate: no document retrieval, no generation call, no output-token bill — though the golden check itself embeds the question to find its match. The bottom branch is the RAG pipeline you already have, waiting for everything the golden layer declines.

31.2 Building the golden store

The golden store is an ordinary vector collection, the same local embedding model and Chroma setup the RAG chapters used (Chapter 27), with one twist: we embed the question, not the answer. A user arrives with a new phrasing (“How do I set up Kafka batch consumption?”) and we want it to match a curated entry written for a differently-worded but equivalent question (“How do I configure Kafka batch consumption?”). Embedding the questions is what makes that paraphrase match possible; the vetted answer rides along in metadata, next to the provenance we will need to judge whether it has gone stale.

That makes two stores from here on, and keeping them apart is the whole design — one holds documents, the other holds blessed question/answer pairs. They are separate Chroma collections, not two names for one. The document store is this chapter’s own, seeded with a topic and a version on every document — the two fields the freshness guard below reads, which Chapter 27’s collection does not carry:

code/ch29_golden.py
doc_store = Chroma(collection_name="docs",     # seeded below, not Chapter 25, RAG Foundations's store
                   embedding_function=emb, persist_directory=".../ch29_docs",
                   collection_metadata={"hnsw:space": "cosine"})

golden_store = Chroma(collection_name="golden",   # curated questions, answers in metadata
                      embedding_function=emb, persist_directory=".../ch29_golden",
                      collection_metadata={"hnsw:space": "cosine"})

emb is Chapter 27’s NomicEmbeddings, unchanged — task prefixes and check_embedding_ctx_length=False included. Leave that flag off and OpenAIEmbeddings tokenizes first and sends token IDs, which a local server answers with 'input' field must be a string or an array of strings. Same embedding model, same distance metric, different contents and different jobs: a question is asked of golden_store first and only reaches doc_store on a miss.

code/ch29_golden.py
golden_store.add_texts(
    texts=[  # embed the QUESTIONS, not the answers
        "How do I configure Kafka batch consumption?",
        "How do I configure a Redis connection pool?",
    ],
    metadatas=[
        {"answer": "[Golden] Set spring.kafka.listener.type=batch, tune throughput "
                   "with max-poll-records, and take a List parameter in the batch listener.",
1         "topic": "kafka", "doc_version": "v2"},
        {"answer": "[Golden (old)] Configure the connection pool via spring.redis.pool.max-active.",
         "topic": "redis", "doc_version": "v1"},    # v1 != current v2 -> STALE
    ],
)

def current_doc_version(store: Chroma, topic: str) -> str | None:
    """What version a DOCUMENT STORE holds for a topic, read from its metadata."""
    got = store.get(where={"topic": topic}, include=["metadatas"])
2    versions = {m.get("version") for m in got["metadatas"]}
    return next(iter(versions), None) if len(versions) == 1 else None


def is_stale(golden: dict, store: Chroma) -> bool:
    """Freshness guard: the doc version this golden entry (its metadata) was derived
    from no longer matches what `store` currently holds for its topic.

    Fails closed on an entry missing its stamps — see the note below.
    """
    topic = golden.get("topic")
    golden_version = golden.get("doc_version")
    if not topic or not golden_version:
        return True                      # untagged entry: nothing to check against
    current = current_doc_version(store, topic)
    if current is None:
        return True                      # source gone, or no single current version
    return golden_version != current
1
doc_version is the golden entry’s own record of which version of the source document its answer was reviewed against — written by whoever approved the answer, never touched by a sync. The kafka entry says v2, which is what the store holds today; the redis entry says v1, and the store has moved on.
2
That version is the stamp Chapter 30 writes when a document changes — not a snapshot number bumped across the whole store on every sync, which is why an untouched document there kept its old tag. That is exactly the property this guard needs: the tag moves only when the content moved, so “the golden entry’s doc_version still equals the topic’s current one” means “the source has not changed since this answer was reviewed.”

Reading it back from the store is the point: one source of truth. The obvious alternative — a small CURRENT_DOC_VERSION dict maintained beside the store — is shorter and wrong in a way that never announces itself. A hand-kept table drifts from what the index actually holds, and then is_stale is wrong in both directions: refusing fresh entries because nobody bumped the dict, and, worse, serving answers derived from documents that moved. A freshness guard that depends on somebody remembering is not a guard.

NoteWhat the two early return True lines are for

On a well-formed entry the guard is its last line — golden.get("doc_version") != current_doc_version(store, topic) — and that line alone. Whether every entry is well-formed is not the guard’s job: the review that adds a golden answer stamps it with its topic and doc_version, and a test that every entry carries both is one line beside the store.

The two branches above only decide what happens if that process slips. An entry with no stamp, or a topic whose documents have since been deleted, would compare None against None, and None != None is False — the bare comparison would call it fresh. The branches call it stale instead, so the slip costs one retrieval rather than a curated answer served at confidence with nothing behind it. Two lines for a case the process should already prevent: cheap enough to keep, not something to build the guard around. (A topic legitimately spanning several documents needs a rule for what “the current version” means across them — current_doc_version returns None when they disagree, which lands in the same branch. And note the assumption the guard rests on: one topic, one source. Once Chapter 28 has cut a document into pieces, or an answer draws on two, “every remaining row still says v2” no longer means “everything this answer was reviewed against is still here” — a deleted chunk leaves the survivors’ tags untouched. A golden entry that depends on several sources needs to record which ones, ids and content hashes, and check for absence as well as for a changed tag.)

Two golden entries are enough for one run to show both outcomes. The kafka answer was derived from doc version v2, which is still current. The redis answer was derived from v1, and the redis docs have since moved to v2, so it must never be served even though its question will match well.

31.3 The check: two guards

The third question the runnable file asks is “How do I configure connection pooling for Redis?”, and the curated store has an answer written for almost that question. Here is what the lookup node prints:

   [check_golden] closest golden question similarity=0.860 stale=True
   [check_golden] similar enough, but stale -> guard refuses, falling back to docs

A 0.860 match, comfortably over the 0.80 bar, and the node turns it down. The answer it declined to serve is [Golden (old)] Configure the connection pool via spring.redis.pool.max-active; the answer the pipeline generated instead, on the same run, names spring.data.redis.lettuce.pool.max-active. The curated entry was written against one version of the Redis docs and the docs moved on. The question did not change, and the wrong answer would have matched it better than the right one.

That refusal is the second guard doing its job.

check_golden embeds the incoming question, finds the single closest curated question, and decides between two outcomes: return the curated answer, or route to RAG. A hit requires both guards to pass, similar enough and fresh.

code/ch29_golden.py
GOLDEN_THRESHOLD = 0.80   # set the bar high; a false hit skips the model
                          # (how to pick it for real: Section 30.7, A/B test: which pipeline ships?)


class State(TypedDict):
    question: str
    docs: str          # what retrieval found on the miss path — not the answer
    answer: str
    source: str        # "golden" or "generated" — which path produced the answer
    golden_route: str  # "hit" / "miss" — what the conditional edge branches on


def check_golden_node(state: State) -> dict:
    # One audience, so no tenant filter here -- on purpose, and said out loud. A
    # store that serves several tenants carries the caller's scope on this lookup
    # too (Section 25.4, Security: data leak across tenants): a golden answer that leaks is still a leak.
    hits = golden_store.similarity_search_with_score(state["question"], k=1)
    if hits:
        doc, dist = hits[0]
        sim = 1 - dist                       # cosine distance -> similarity
        stale = is_stale(doc.metadata, doc_store)
        print(f"   [check_golden] closest golden question similarity={sim:.3f} stale={stale}")
        # Guard 1: close enough.  Guard 2: not stale.
        if sim >= GOLDEN_THRESHOLD and not stale:
1            return {"answer": doc.metadata["answer"], "source": "golden", "golden_route": "hit"}
2        if sim >= GOLDEN_THRESHOLD and stale:
            print("   [check_golden] similar enough, but stale -> guard refuses, falling back to docs")
3    return {"golden_route": "miss"}
1
A clean hit writes three fields: the curated answer, source="golden" (so a caller can see the answer skipped the model), and golden_route="hit".
2
This if exists for the operator: a close-but-stale match is logged before it falls through, because a silent refusal is a mystery when you are staring at a golden entry that “should” have matched.
3
Anything else, too far away, close but stale, or an empty store, writes only golden_route="miss" and leaves the answer for the pipeline to produce.

The routing function returns the field, and the wiring is the conditional edge from Chapter 5 pointed at the two outcomes:

code/ch29_golden.py
def route_golden(state: State) -> str:
    return state["golden_route"]

The three exits, and the trap the second guard exists for:

%%{init: {'securityLevel': 'loose', 'flowchart': {'diagramPadding': 24}, 'themeCSS': 'foreignObject{overflow:visible}span.bot{display:flex;justify-content:flex-start;align-items:flex-start;width:100%;height:0;overflow:visible;line-height:0}span.bot img{display:block!important;width:18px!important;height:18px!important;padding:2px;background:#fff;border-radius:50%;box-shadow:0 0 0 1px #999}span.bot.r img{margin:0;transform:translate(-24px,-24px)}span.bot.d{justify-content:center}span.bot.d img{margin-top:-50%;transform:translateY(-24px)}'}}%%
flowchart LR
    q(["question"]) --> cg("check_golden")
    cg -->|"hit, docs unchanged"| serve["serve the golden answer<br/>no model call, deterministic"]
    cg -->|"hit, but docs updated<br/>since the answer was written"| stale["log the refusal,<br/>fall through"]
    cg -->|miss| rag
    stale --> rag("<span class='bot r'><img src='data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIyNCIgaGVpZ2h0PSIyNCIgdmlld0JveD0iMCAwIDI0IDI0IiBmaWxsPSJub25lIiBzdHJva2U9IiMzMzMiIHN0cm9rZS13aWR0aD0iMiIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIj48cGF0aCBkPSJNMTIgOFY0SDgiLz48cmVjdCB3aWR0aD0iMTYiIGhlaWdodD0iMTIiIHg9IjQiIHk9IjgiIHJ4PSIyIi8+PHBhdGggZD0iTTIgMTRoMiIvPjxwYXRoIGQ9Ik0yMCAxNGgyIi8+PHBhdGggZD0iTTE1IDEzdjIiLz48cGF0aCBkPSJNOSAxM3YyIi8+PC9zdmc+'/></span>full RAG pipeline")
    rag --> ans(["answer"])
    serve --> ans
    serve -. "the trap: skip the staleness check<br/>and this exit fires on outdated docs" .-> trap(["✗ stale answer served"])
    classDef bad fill:#f8d7da,stroke:#b02a37,color:#000;
    classDef good fill:#d6e6f7,stroke:#3b6ea5,color:#000;
    class trap bad;
    class serve good;

WarningGotcha: caches lie

The guard only refuses; it does not repair. The moment a source doc is re-indexed under a new version, every golden entry derived from it stops qualifying until someone re-reviews and re-stamps it, so a document upgrade needs a step in its process that revisits the golden layer. That re-review is the recurring cost of caching answers over ground truth that keeps changing; budget for it.

NoteDeep Dive: why the similarity bar is set high

GOLDEN_THRESHOLD = 0.80 is conservative because the two errors cost different amounts. A miss is cheap: you fall through to the pipeline you already trust and generate an answer from the docs. A false hit hands back a curated answer to a question it was never meant for, at full confidence, with no model to notice the mismatch and no citation for the reader to check. So the threshold leans toward “when in doubt, generate.” Set it too low and the golden layer starts answering the wrong questions (the Silver challenge makes this bite); push it toward 1.0 and you throw away the paraphrase matching that made embedding the question worthwhile — and you do not get “exact match” in exchange. 1.0 would reject even a verbatim repeat here, because the stored question was embedded as search_document: and the incoming one as search_query: (Chapter 27): same words, two different embedding inputs, cosine 0.884. If you want exact match, compare normalized text and let the vector lookup be the fallback. The right similarity value is empirical, which is what the next chapter’s evaluation harness is for.

TipTry it: promote the stale answer

Simulate a re-review, which means doing what a reviewer would actually do: fix the answer first, then re-stamp it. Edit the redis entry’s answer so it names the key the current docs name — spring.data.redis.lettuce.pool.max-active, not the retired spring.redis.pool.max-active — and only then set its doc_version to "v2" so it matches what the document store holds. Re-run. The third case flips: same similarity=0.860, but now stale=False, so the guard lets it through and the answer comes back source=golden with no model call. The guard has now gone both ways on the same match, refusing it stale and admitting it current.

Bump the tag without fixing the answer and you have built the failure the whole chapter is about: a stale answer wearing a current stamp, which no guard downstream can catch. The version is a claim that somebody checked; it is only as true as the review behind it.

31.4 Falling through to RAG

Everything below the top branch is the ordinary RAG path — retrieve, then generate — a baseline of the previous chapters’ pipeline, not their full rewrite-and-rerank composition. retrieve_docs pulls from the source store (doc_store, not the golden one), and generate writes the answer from those docs and marks its provenance source="generated":

code/ch29_golden.py
def retrieve_docs_node(state: State) -> dict:
    docs = doc_store.similarity_search(state["question"], k=2)
    print(f"   [retrieve_docs] recalled {len(docs)} document(s) from the source store")
    return {"docs": "\n".join(d.page_content for d in docs)}


NOT_COVERED = "Not covered in the docs."


def generate_node(state: State) -> dict:
    # No similarity threshold on this path: retrieve_docs hands back the nearest
    # two documents whatever the question, so the refusal is the model's — and a
    # fixed sentence, so a double miss has a checkable end.
    prompt = f"""
        Answer the question concisely, using only the sources below. If the
        sources do not cover the question, reply exactly "{NOT_COVERED}".

        Question: {state['question']}

        Sources:
        {state['docs']}

        Answer:"""
    answer = llm.invoke(prompt).content.strip()
    return {"answer": answer, "source": "generated"}

The wiring is a conditional edge whose "hit" target is END itself:

code/ch29_golden.py
builder = StateGraph(State)
builder.add_node("check_golden", check_golden_node)
builder.add_node("retrieve_docs", retrieve_docs_node)
builder.add_node("generate", generate_node)

builder.add_edge(START, "check_golden")
builder.add_conditional_edges(
    "check_golden", route_golden, {"hit": END, "miss": "retrieve_docs"}
)
builder.add_edge("retrieve_docs", "generate")
builder.add_edge("generate", END)
graph = builder.compile()

When check_golden returns golden_route="hit", the answer it wrote is the answer the caller gets, and neither retrieve_docs nor generate runs.

NoteAside: the runnable file wraps these nodes in a retry

Every node here calls the same shared local server (check_golden and retrieve_docs embed the query, generate runs the chat model), and a shared server can swap a model out mid-run, which surfaces as a transient error while it reloads. So code/ch29_golden.py attaches a RetryPolicy to all three nodes to ride those blips out. That is infrastructure for a shared machine, not golden-layer logic, so it is elided from the wiring above; retries are the subject of Section 21.2. (The chat model and embedding connection are the local setup from Chapter 12.)

31.5 Running it

The runnable file, code/ch29_golden.py, seeds the two entries above and runs four questions: one that hits a fresh golden answer, one that misses entirely, the stale Redis case from the top of the chapter, and one that misses both stores:

code/ch29_golden.py
if __name__ == "__main__":
    cases = [
        ("How do I set up Kafka batch consumption?",
         "frequent question -> hits a fresh golden answer, returned directly (no LLM call)"),
        ("How do I optimize slow database queries?",
         "rare question -> not in the golden layer, falls back to document generation"),
        ("How do I configure connection pooling for Redis?",
         "matches a golden answer, but it is stale -> guard refuses, falls back to docs"),
        ("How do I rotate the TLS certificates on the gateway?",
         "double miss -> no golden entry and no covering document; generate must refuse"),
    ]
    for question, expect in cases:
        print(f"\n{'=' * 56}\nQuestion: {question}\nExpect:   {expect}\n{'=' * 56}")
        result = graph.invoke({"question": question})
        print(f"   >>> source = {result['source']}")
        print(f"   >>> answer: {excerpt(result['answer'])}")   # first line, trimmed

(excerpt: first line, trimmed — Chapter 12.)

code/ch29_golden.py
========================================================
Question: How do I set up Kafka batch consumption?
Expect:   frequent question -> hits a fresh golden answer, returned directly (no LLM call)
========================================================
   [check_golden] closest golden question similarity=0.861 stale=False
   >>> source = golden
   >>> answer: [Golden] Set spring.kafka.listener.type=batch, tune throughput with max-poll-records, and take a List parameter in the batch listener.

========================================================
Question: How do I optimize slow database queries?
Expect:   rare question -> not in the golden layer, falls back to document generation
========================================================
   [check_golden] closest golden question similarity=0.476 stale=True
   [retrieve_docs] recalled 2 document(s) from the source store
   >>> source = generated
   >>> answer: Add a B+ tree index on frequently queried columns.

========================================================
Question: How do I configure connection pooling for Redis?
Expect:   matches a golden answer, but it is stale -> guard refuses, falls back to docs
========================================================
   [check_golden] closest golden question similarity=0.860 stale=True
   [check_golden] similar enough, but stale -> guard refuses, falling back to docs
   [retrieve_docs] recalled 2 document(s) from the source store
   >>> source = generated
   >>> answer: Configure it via `spring.data.redis.lettuce.pool.max-active`.

========================================================
Question: How do I rotate the TLS certificates on the gateway?
Expect:   double miss -> no golden entry and no covering document; generate must refuse
========================================================
   [check_golden] closest golden question similarity=0.485 stale=True
   [retrieve_docs] recalled 2 document(s) from the source store
   >>> source = generated
   >>> answer: Not covered in the docs.

The four check_golden lines are the part that does not vary between runs: the same embedding model gives the same similarities (0.861, 0.476, 0.860, 0.485), and the freshness flags are metadata, so the routing is fixed.

  • Case one clears both guards: source=golden, and retrieve_docs never prints because it never ran.
  • Case two is too far from any curated question (0.476 < 0.80), so it misses and generates.
  • Case three is the refusal from the opening, and the run shows what the refusal bought: a stale golden answer naming spring.redis.pool.max-active stayed in the store, and the caller got a generated answer naming spring.data.redis.lettuce.pool.max-active, which is what the current docs say.
  • Case four is the double miss, and the reason this simplified fall-through needs a contract of its own. Nothing in the golden store is close (0.485), and nothing in the docs covers it either — but retrieve_docs has no threshold, so the two nearest documents go to generate anyway. The run ends in the fixed refusal sentence, source=generated, and that sentence is the checkable end: a smoke test asserts it verbatim, not “the answer was non-empty”. The previous chapters’ pipeline refused earlier, at retrieval (Chapter 27’s SIM_THRESHOLD) or at the empty shortlist (Chapter 29); this one refuses last, in the model. Either is a design; having neither is the bug.

What does vary is the prose of the two generated answers (one representative run — Chapter 12). The golden answer comes back character-for-character as it was stored.

31.6 The operational rule

A golden hit skips document retrieval, generation, and the citation — though it still embeds the question to look the entry up — so the checks those stages were carrying are gone on that path: nothing re-reads the source, and nothing re-writes the answer for the question actually asked. Two checks stand in for them, and both have to be in the node, not in a process around it: a high similarity bar, because a false hit is a wrong answer with nothing downstream to catch it; and a freshness stamp compared against the current doc version, because a correct entry becomes a wrong one the day its source is re-indexed. Embedding the question rather than the answer is what makes a paraphrase reach the entry at all. Wiring-wise the whole layer is one conditional edge, {"hit": END, "miss": "retrieve_docs"}; the store is the only new piece.

Whether 0.80 is the right bar, whether a fresh golden answer actually beats what generation would have produced, and whether adding an entry regressed an answer that used to be fine are all hunches until they are measured. The next chapter builds the instrument that measures them — an evaluation harness (Chapter 32): a fixed question set scored against expected answers, run as a repeatable gate — on a miniature RAG without a golden layer, so that the harness itself stays small enough to read. Pointing it at this chapter’s graph is the reader’s move: add a case whose question sits in the golden store, and the same gate turns the threshold, the freshness policy, and the choice to short-circuit at all into numbers you can defend.

New pieces in this chapter

No new pieces in this chapter — it composes ones you already know.

Code in this chapter

File What it runs
code/ch29_golden.py A reviewed-answer layer in front of retrieval, with the two guards that keep a cached answer from outliving its documents.

Challenge

The golden layer lives or dies on two dials: how close counts as a hit, and how you know an answer is still true.

🥉 Bronze — add a golden entry and watch it short-circuit

The doc store already knows about MySQL indexing, but the golden store does not, so a MySQL question falls through to generation. Add a third golden entry — question "How do I add a database index in MySQL?", a curated answer of your choosing, topic "mysql", and doc_version "v2" (fresh, since current_doc_version(doc_store, "mysql") reads v2 back from the document store). Then ask a paraphrase the store has never seen, e.g. "How do I create an index on a MySQL table?". Predict first: will it clear the 0.80 bar? Run it and read the check_golden line. You should see a high similarity, stale=False, and source=golden with retrieve_docs never printing: a brand-new phrasing served by a curated answer, with no model call, because you embedded the question.

office hours Worked solution with tests: code/answers/ch29_add_a_golden_entry.py.

🥈 Silver — make a false hit bite

Lower GOLDEN_THRESHOLD to something reckless — say 0.40 — and ask "How do I handle Kafka consumer failures?". Its nearest curated question is the fresh batch-consumption entry (~0.67), so the freshness guard stays out of it and the threshold alone decides. Then restore the threshold to 0.80 and ask it again.

Read the two runs against each other. The 0.40 run should report source=golden and hand back the batch-consumption answer — set spring.kafka.listener.type=batch..., which says nothing about handling failures — with no model call anywhere in the trace; the restored run should fall back through to generation. Then write down the lesson: on a path that bypasses the model, and once the entry is fresh and was reviewed, the similarity bar is what stands between “fast and right” and “fast and wrong.”

Hints: the Deep Dive claimed the high threshold is the whole safety margin of a path that skips the model — this challenge is that claim put under load. Watch for whether anything in the loop is left to notice that the reply answered a different question.

office hours Worked solution with tests: code/answers/ch29_make_a_false_hit_bite.py.