33 Before You Ship

Chapter 32 ended with a green gate: the right documents reached the context, the judge passed every answer, and the build would have shipped. This chapter is about everything that verdict does not cover.

Passing is not readiness. A score says the answer was good. It does not say the run took a path you allow, and it says nothing at all about whether the service around the graph can be handed to other people.

Two gaps, in that order. The first is inside the run and you can measure it: the answer is not the only output — the tool calls are an output too, and nothing so far has looked at them. The second is around the run and you cannot measure it from a transcript at all. Every mechanism in Part III was shown at its teaching minimum, and the distance from that to production is not a LangGraph distance. It is the operational layer every service carries.

33.1 What the harness cannot see: the path taken

Everything Chapter 32 scored is an answer. That is the right thing to score for the RAG pipeline it measures, where the run is retrieve then generate and there is only one path through it. It stops being sufficient the moment the model is choosing what to call.

Here are two runs of the doc-bot from Chapter 20 — the same graph, not a new one. Only the script differs, and both scripts end on the same sentence:

code/ch31_trajectory.py

# Both scripts end on the same sentence. Only the route differs.
STRAIGHT = [call("search_docs", "kafka batch", "c1"), AIMessage(content=ANSWER)]
DETOUR = [call("web_search", "kafka batch consumers", "c1"),
          call("search_docs", "kafka batch", "c2"),
          AIMessage(content=ANSWER)]

code/ch31_trajectory.py

run A
  answer     : Set spring.kafka.listener.type=batch.
  requested  : ['search_docs']
  off-policy : none
run B
  answer     : Set spring.kafka.listener.type=batch.
  requested  : ['web_search', 'search_docs']
  off-policy : ['web_search']

same answer      : True
same requests    : False

Byte-identical answers. This demo compares the answer strings — it does not run Chapter 32’s judge or its gate — but that is enough to make the point, because an identical string is the easiest possible case for any answer-scoring to call identical. Both runs also reached the internal document, so no retrieval measure separates them either. Whatever a judge decided about run A, it has nothing else to go on for run B.

What separates them is that run B called web_search on a question the internal documentation plainly covers. For a support bot whose whole promise is “answers come only from our documentation,” reaching for that tool is the promise broken, and Chapter 13 named exactly this trade when it handed the model the wheel.

Be precise about what the two runs demonstrate, because the demo is smaller than the worry it stands in for. The model is scripted (Chapter 20’s ScriptedModel) and the doc-bot’s web_search is a stub that returns a canned string, so no request leaves the machine. What you are watching is the path difference surviving an identical-answer check, not an exfiltration. The reason to care is that in a real deployment that same tool call is a real call: to the open web, to a paid API, to a system with side effects. The measurement is a toy; the thing it measures is not.

The general form is worth stating plainly: the answer is the output you score; the tool calls are the output that has side effects. Scoring only the first leaves the second unobserved. A run that answered correctly may also have spent nine minutes retrying, called a paid API forty times, written to a system it had no business touching, or skipped the verification step you thought was mandatory.

Extracting the path needs nothing new. It is already in the messages:

code/ch31_trajectory.py

def requested(state) -> list[str]:
    """The tools a run ASKED for, in order — attempts included."""
    return [tc["name"] for m in state["messages"]
            for tc in (getattr(m, "tool_calls", None) or [])
1            + (getattr(m, "invalid_tool_calls", None) or [])]

1: invalid_tool_calls is where a call whose arguments failed to parse lands — tool_calls stays empty for it. An audit wants both: a model that reached for a tool it was not allowed to touch and fumbled the arguments still reached for it, and a check that reads only tool_calls would wave it through.

Read the name carefully, because the function is narrower than “what the run did”. It reports what the model requested. Three things follow, and each one is a way to be wrong if you forget it:

Requested is not executed. An invalid_tool_calls entry never reached a tool at all — the graph routes on valid calls only. So a run that tried to call verify and failed to phrase it still puts verify in this list.
Executed is not succeeded. A tool that ran can return an error, retry inside itself, or do nothing useful. An execution record would tell you a call was dispatched, and its result whether the work landed; the request list establishes neither.
The order is the extractor’s, not the model’s. Valid and invalid calls are concatenated per message, so within one message every valid call sorts before every invalid one regardless of what the model actually asked first. The runs in this chapter make one valid call per turn, which is why their order is faithful.

With that boundary in mind, five comparisons cover nearly everything anyone checks:

Comparison	Written as	Catches
Exact, ordered	`path == ["search_docs"]`	drift — yesterday two steps, today five
Same calls, any order	`sorted(path) == [...]`	correctness where the order is genuinely free (this still counts repeats — two `search_docs` calls will not match one)
Nothing extra	`set(path) <= ALLOWED`	going off-policy: the tool this task had no business calling
Nothing missing	`set(path) >= REQUIRED`	asking for the required step — not that it ran, and not that it passed
Not too many	`len(path) <= BUDGET`	a run that asked for a lot; a first approximation of cost, blind to retries inside a tool and to how long anything took

Two rows there are weaker than they look. >= REQUIRED is satisfied by a request, so it cannot tell “the agent verified” from “the agent tried to verify and mis-typed the arguments” — if the point is that a verification really happened, check the verification’s output, not the presence of its name. And len(path) counts requests, so real cost and latency have to come from execution events.

The third is the one to reach for first: set(path) <= ALLOWED encodes a policy rather than a preference, and it is a pure function of the messages, so it gives the same verdict on every machine. That is the cheapest way to pin this agent may never call that tool, and it is what the tool boundary of LangGraph in the Wild’s coding agent comes down to.

It is also where the next section starts, because a check that reads a finished transcript has already let the call happen.

That gives Part III three questions and three homes for them:

	Test (Chapter 20)	Answer eval (Chapter 32)	Trajectory
Asks	did my code do what I wired it to	is the answer good enough	did it get there a way I allow
Answers with	yes / no	a number against a threshold	both, depending on the comparison
On identical code	never varies	varies every run	varies only where the model chose
The model is	scripted out	the thing being measured	the thing being observed
Runs	every commit, blocks the merge	in CI on its own gate (Section 32.5), and on a schedule to open a conversation	hard rules enforced at runtime; distributions reported

That is enough to expose the blind spot in answer-scoring, which is all this doc-bot can show: its RAG pipeline has one path, so a richer trajectory check over it would only assert that the only possible thing happened. These checks start earning their keep when an agent has real discretion — a coding agent that must run the tests before it claims a fix, say. That is where LangGraph in the Wild starts: a coding agent’s graph will not report success on a claim it has not tested.

The more urgent question is what any of these checks can actually do about what they find.

33.2 Observing is not preventing

Here is the hinge between the two halves of this chapter, and it is easy to miss because the trajectory check feels like a safeguard.

It is not one. requested(state) reads a finished run. By the time the list contains web_search, the call has already happened: whatever it cost, whatever it touched, whatever left the building has already left. A path check is a detector, and a detector’s job description is “tells you afterwards.”

The obvious fix is not the fix. Stop telling the model about the tool — drop it from the list you pass to bind_tools — and the call still runs, because those are two different things. What you bind is a description the model reads. What executes is the doc-bot’s tools node, BY_NAME[call["name"]], a dictionary lookup with no permission in it. What a control looks like is one branch in front of that lookup — Ch18’s tools node with the check added, and no other change:

code/ch31_refuse.py

for c in state["messages"][-1].tool_calls:
1    if c["name"] not in ALLOWED:
        REFUSALS.append(f"refused {c['name']}: not in this run's allow-list")
        results.append(ToolMessage(
            content=f"[denied] {c['name']} is not available to this run",
            tool_call_id=c["id"]))
2        continue
    output = db.BY_NAME[c["name"]].invoke(c["args"])
    results.append(ToolMessage(content=output, tool_call_id=c["id"]))

1: ALLOWED is what this run may execute — not what the model was told about. The two are separate on purpose; run 2 is what happens when you confuse them.
2: The side effect never happens. The refusal goes back as a ToolMessage, so the run continues and still answers from what it is allowed to use — a denied call is not a crashed graph.

Run code/ch31_refuse.py and watch the middle row:

code/ch31_refuse.py

run             advertised                  requested                   executed
1 shipped       ['search_docs', 'web_search']['web_search', 'search_docs']['web_search', 'search_docs']
2 unadvertised  ['search_docs']             ['web_search', 'search_docs']['web_search', 'search_docs']
3 authorized    ['search_docs']             ['web_search', 'search_docs']['search_docs']

refusals recorded: ["refused web_search: not in this run's allow-list"]
answer after the refusal: 'Set spring.kafka.listener.type=batch.'

Run 2 was never told the tool exists and executed it anyway. Hiding a tool makes the model less likely to ask — and “less likely to ask” is not a control, because the model is precisely the component you cannot make promises on behalf of. Run 3 adds four lines to the tools node: check the name against what this run may execute, and refuse before dispatching. The request is still in the trajectory, the execution count is zero, and there is a line naming what was refused. The run still answers, from the tool it was allowed to use — a denied call is not a crashed graph.

Two properties made run 3 a control and run 2 not one: it decided before the side effect, and it sits where the model cannot reach it. Those are the only two questions worth asking of anything you are told is a safeguard.

So do not read “a red run may mean the model wandered once” as “a live run can never fail a build.” Whether a violation is rare and whether it is allowed are different questions, and rarity is no defence for a tool the agent must never touch: if a run reaches production data or spends real money, one occurrence is a finding, not noise. What decides whether something gates is the kind of claim it makes, not whether the model was scripted:

The check is about	What it means when it trips	Enforce at runtime?
A forbidden tool, or a resource out of scope	One violation is worth failing and investigating	Yes — refuse before the call
An explicit per-run budget	Over the limit is a broken constraint, not a bad average	Yes — count, and stop before the next call
Usual path, typical step count, mean latency	Only a trend means anything; one run means nothing	No — report and watch the distribution

Scripted versus live decides how much your evidence covers, not whether the rule is enforceable. A scripted test reproduces the allow and deny branches exactly, which is what makes it a test rather than an eval (Section 20.2) — and it pins the boundary case. It cannot prove a live model never asks; run 3 is why it does not have to.

That is the general shape of everything in the rest of this chapter. Each item below is a control that has to live somewhere the model cannot argue with, plus a way to tell whether it is really there. A check that only observes cannot stop the call it observed.

33.3 Nine things shipping asks for

“Runs as a service” is not “ready for users.” Every mechanism in Part III was shown at its teaching minimum, which was the right size for learning the mechanism and is not the size that survives contact with other people.

Here is the gap. For each one: what breaks, where the control belongs, and what evidence would convince a reviewer it is there — because “we handle that” with nothing behind it is how a checklist passes without the control existing.

33.3.1 Authentication & authorization

What breaks. Chapter 22 put the graph behind an HTTP endpoint that trusts anyone who can reach it. On a private network that reads as fine right up to the day the network is less private than you thought.

Where the control belongs. Authenticate the caller at the edge, before the run starts. Authorization is not only an edge concern, though: coarse “may this caller use this endpoint” belongs there, while “may this run touch that tool or that record” has to be checked at the point of access — which is exactly the executor-side check the previous section built. Both are ordinary trusted Python, and neither belongs in the prompt, which Chapter 27 already showed is not a boundary. The test is not which box the code sits in; it is whether it runs before the protected access and whether the model can route around it.

Evidence. An unauthenticated request gets rejected, and an authenticated one carrying someone else’s resource id also gets rejected — the second is the one teams forget, and it is the one that matters.

33.3.2 Thread ownership

What breaks. Checkpointers key state by thread_id (Chapter 8), and nothing binds a thread to a user. Guess an id and you resume someone else’s conversation, including whatever they said in it.

Where the control belongs. Bind the thread to its owner at creation and check ownership on every resume. Treat thread_id from a request as a claim, never as an authorization.

Evidence. A test in which user B resumes user A’s thread_id and is refused. Ids that are unguessable help and do not substitute for the check.

33.3.3 Rate limits & budgets

What breaks. Nothing caps how often a caller reaches the model or the tools. One client — or one retry loop of your own (Chapter 21) — drains the quota and the bill.

Where the control belongs. Per-caller rate limits at the edge, and a spend ceiling the run itself respects, so a single request cannot cost unboundedly. The trajectory budget above (len(path) <= BUDGET) is the in-run half of this, and it is a detector; the ceiling that actually stops work belongs in the runtime.

Evidence. A load test that gets throttled rather than served, and a run that hits its ceiling and stops with a clear error instead of quietly continuing.

33.3.4 Timeouts & cancellation

What breaks. Chapter 21 retried failures, but a hung model call or a client that walked away still runs to the end, holding a slot and spending tokens on an answer nobody will read.

Where the control belongs. A deadline on every request, propagated into the run so cancellation actually reaches the node that is waiting.

Evidence. Disconnect a client mid-run and watch the work stop — in your provider’s usage, not only in your logs.

33.3.5 Secret management

What breaks. Keys and base URLs are inlined throughout this book for readability. Inlined keys reach source control, and from there they reach everywhere.

Where the control belongs. A secret store, with rotation, injected at runtime; keys out of source and out of logs — and out of State, which is written to disk in the clear and copied into every trace and response (Section 26.8). The channel for them is runtime context (Section 14.8), per run, never persisted.

Evidence. A repository scan that finds nothing, and a rotation you have actually performed rather than one you believe is possible.

33.3.6 Trace redaction

What breaks. Chapter 24 recorded every run in full, prompts and all. A trace is a copy of whatever the user typed, kept somewhere with different access rules than the system it came from.

Where the control belongs. Redaction before the trace leaves the process, plus a retention window — the two together, since redaction alone still accumulates forever.

Evidence. A trace of a run containing planted PII, inspected in the trace UI, with the PII absent.

33.3.7 Tool trust boundary

What breaks. Chapter 25 wired external MCP tools in as trusted. A tool server is another process, often someone else’s, and its output flows straight into the model’s context.

Where the control belongs. Constrain what a tool may do at the point it is granted, and never let a tool’s output trigger a further action unchecked.

Evidence. A tool whose response tries to instruct the model, run end to end, with the instruction not obeyed — the Section 27.3 canary technique pointed at a tool instead of a document.

33.3.8 Retrieved-document trust boundary

What breaks. The whole RAG run treats the corpus as trusted. Chapter 29 hands retrieved text to a model to grade, and every generate interpolates it as Sources:. Section 27.3 already measured what that costs: a planted document was obeyed, and the delimiters and the “treat as data” rule did not hold.

Where the control belongs. Start on the write side, because that is the one place a guarantee is available: who may add a document, reviewed like code — and a golden entry (Chapter 31) is code. What Section 27.3 actually measured is narrower than “the read side is hopeless”: it showed that a prompt-level instruction is not a boundary. The read side still carries real controls that reduce the blast radius — scope retrieval to what this caller may see (Section 27.4), keep retrieved text out of any rubric it could argue with, and require authorization for anything a model’s output triggers, which is the executor check again. None of those makes a model injection-proof; together they decide how much a successful injection can reach.

Evidence. Re-run that canary against your own index and count. And an audit trail of who wrote what into the corpus — if you cannot answer that, the boundary does not exist.

33.3.9 Schema migrations

What breaks. The durable checkpointer (Chapter 26) creates its tables once. Upgrade the library and the schema beneath live threads moves under them.

Where the control belongs. Versioned migrations, applied deliberately, with a rollback path — the same discipline Chapter 30 applied to an index, applied to state.

Evidence. A migration rehearsed on a copy of production data, with threads created before it still resumable after it.

None of these is a LangGraph gap — the graph is done. They are the operational layer every service carries, and they are where “it works on my machine” becomes “it is safe for other people.” Read this as the first page of your own runbook.

New pieces in this chapter

No new pieces in this chapter — it composes ones you already know.

Code in this chapter

File	What it runs
`code/ch31_trajectory.py`	Two runs with byte-identical answers and different paths – the thing answer-scoring cannot see.
`code/ch31_refuse.py`	The other half: stopping the call the agent was never allowed to make, where the model cannot reach the gate.

Challenge

One exercise to feel the detector working, and one to find out how far your own service is from the list above.

🥉 Bronze — make the budget check fail while the answer stays right

Take code/ch31_trajectory.py and add a third script that reaches the same answer the other two do, but calls search_docs several times before it gets there — a model that dithered rather than one that went somewhere forbidden. Then score all three runs on both checks: the allow-list (set(path) <= ALLOWED) and a budget (len(path) <= BUDGET).

Predict the two columns before you run it. The point to arrive at: the allow-list is clean for this run and the budget is not, the answer check notices neither, and those three verdicts are answering three different questions about the same transcript.

Then say what you would do with the budget failure — is it a gate or a report? Argue it from the section above rather than from taste.

office hours Worked solution with tests: code/answers/ch31_path_budget.py.

🥈 Silver — audit your own service against the nine

No code. Take something you actually run — or the doc-bot as you would deploy it — and fill one row per item:

Item	Control today	Evidence	Gap	How it gets accepted

The rule that makes this worth doing: a row whose Evidence column says “we handle that” is a gap, not a control. Evidence means something you could show someone — a test that fails when the control is removed, a scan, a rehearsal, a screenshot of a redacted trace. If you cannot name one, you have found the row to work on first.

ch31_refuse.py is the shape to copy. Its third run does not claim the tool is blocked; it produces a request that appears in the trajectory, an execution count of zero, and a line naming what was refused — and test_hiding_the_tool_from_the_model_does_not_stop_execution fails if someone “fixes” it by hiding the tool instead. A row is filled when you have that: a check that goes red when the control is taken away.

The rows with the emptiest Evidence column are often the ones nobody was worried about. That is the exercise doing its job.

What you have, and what comes next

That closes Part III, and with it the doc-bot’s journey from a hard-coded machine to something with the shape of a service. Trace what it became: it survives failure with retries and fallbacks (Chapter 21), answers over HTTP so something other than a __main__ block can call it (Chapter 22), records every run so you can see what happened after the fact (Chapter 24), reaches external tools over MCP (Chapter 25), rests on persistence internals we can now reason about (Chapter 26), and carries a suite that pins its deterministic half (Chapter 20). Its retrieval is no longer a stub but a real set of techniques — a vector store, tuned chunks, query rewriting and reranking, a blue-green index, a golden layer — and Chapter 32 put a regression harness around a miniature version of that pipeline, so a change to it can be defended with a number.

What that gives you is a working example of each production mechanism, and a regression harness for a miniature retrieval pipeline. What it does not give you is a system you can put in front of users: composing those examples into one service, and pointing the harness at that service’s actual pipeline, is the integration step — and the nine items above are the rest of the distance, yours to close.

Agents will keep changing quickly, because the models and the tools under them change quickly, and each step up in what they can do means more software gets built, more complex software, and more capable software. We are in the middle of a large shift in how software is made. Harnesses like the one this book took apart are becoming infrastructure, and infrastructure still has to evolve, so knowing what is under LangGraph’s lid is an advantage: you can evolve with it instead of waiting for it.

Thank you for reading to the end. I hope this book is helpful for you as an AI agent builder on the way from 10x productivity to 10x shipping, or at least that the drawings are.

P.S. Please subscribe to the mailing list for updates about this book and the planned LangGraph in the Wild.