Reliability and Production: Evaluation, Monitoring, and Cost · Milestone Companion: No Eval, No Iteration

Date: October 19, 2026 · Path: Agent Builder · Stage s5

What this milestone asks

Take the MCP agent from Stage 4 and make it measurable. Three deliverables:

1. A golden set of twenty cases. Not twenty happy-path questions — twenty cases spanning straightforward tool use, multi-tool composition, questions your system should refuse, deliberately ambiguous inputs, and at least two cases designed to make the tools fail (unknown entity, malformed argument). Each case carries its expected behavior — expected tool calls, expected properties of the answer, or an expected refusal — written before you run anything.
2. Tracing on every run. Wire Langfuse (or LangSmith) so every evaluation run records full traces: model inputs, tool calls and results, outputs, latency, and token counts.
3. A first evaluation report. Pass rate per case category, a failure taxonomy (retrieval miss, wrong tool, wrong argument, hallucinated citation, timeout), plus cost and latency per case. One page, honest numbers.

Why this milestone

Until now, every improvement you made was a superstition: you changed a prompt, tried three questions, and felt better. A golden set converts that loop into engineering — run the suite, get a number, change one thing, run again, diff the numbers. This single habit is worth more than any framework, and it is the dividing line between a demo and a system that can be iterated without fear.

The report forces two uncomfortable truths of production agents into the open. First, LLM-as-judge has biases: a judge model grades with its own blind spots, so critical cases get human review, and you track where judge and human disagree. Second, cost is a feature of the design: the report's cost column is where the cost triad — model tiering (small models for routing and grading where quality allows), caching, and context trimming — stops being a slogan and becomes a line-item decision. Behind all of it stand the four defense lines of a production agent: input filtering, tool whitelisting, output validation, and human fallback. Your eval set is how you find out whether each line actually holds.

How to use the resources

1. Concepts first: the evaluation chapters of Chip Huyen's AI Engineering. Among public resources this is one of the most thorough treatments of evaluation design. Read it before writing your twenty cases — it will change what you count as a case.
2. Methodology: Stanford CS329S (Machine Learning Systems Design). Its treatment of evaluation, monitoring, and iterative ML systems is the general discipline of which agent evals are a special case. Sample the relevant lectures rather than the whole course.
3. Tooling concepts: the LangSmith documentation. Even if you run Langfuse, LangSmith's docs are the cleanest explanation of the dataset / evaluator / trace triad. Learn the vocabulary there, then apply it in whichever tool you chose.
4. Framework design: the OpenAI Evals repository. You will probably not use it directly against an MCP agent, but reading how a general eval framework structures tasks and graders will improve your own harness.
5. Then build, in this order: five cases run manually against traces; twenty cases in a scripted runner; scoring (rule-based checks first, judge model second); the report.

Common failure modes

• Twenty happy paths. An eval set with no refusal cases and no tool-failure cases measures whether your demo still demos. Adversarial and error cases are where the information is.
• Expectations written after the run. Writing expected outputs after seeing what the agent did is grading to the curve. Expectations are frozen before execution.
• Unexamined judges. A judge model from the same family as the agent, with no rubric and no human spot-check, will grade your agent's accent as fluency. Rubric first; hand-check at least the critical cases; record agreement.
• Changing two things at once. New prompt and new model in one run — the diff is now uninterpretable. One variable per run.
• A report without a taxonomy. "85% pass" hides everything that matters. The failure categories are the actionable output; the pass rate is decoration.
• Ignoring the meter. No per-case cost and latency means you cannot even see the routing or caching decision you should be making.

Acceptance checklist

• Twenty frozen cases covering success, refusal, tool-error, multi-tool, and ambiguous/adversarial inputs, each with expectations written in advance.
• Every run traced in Langfuse or LangSmith: inputs, tool calls, outputs, latency, tokens — inspectable per case.
• Rule-based checks used wherever possible; LLM-as-judge only with a written rubric.
• At least five critical cases hand-checked, with judge-versus-human agreement noted in the report.
• The report includes per-category pass rates, a failure taxonomy, and per-case cost and latency.
• One deliberate change (prompt or model), one re-run, one diff — demonstrating the loop actually loops.

Going further

Move the suite into CI so every change is gated, and add a regression alarm on cost per case. Experiment with the cost triad and let the suite arbitrate: does a small model on routing hold the pass rate? Extend tracing into production with sampling, and let real traffic nominate case number twenty-one. You have now closed the loop that defines this entire path: build, measure, iterate — with the numbers to prove it.