We built a fictional bookshop as an MCP server, wrote 50 questions real customers ask, and ran every one through headless Claude Code, twice, against two versions of the server. 200 runs. Then we graded every answer against the shop's own data.
Every one of the 171 tool calls on the original server returned HTTP 200. Five of the 100 answers were still wrong or incomplete, and the server log could not tell them apart from the 95 that worked.
Sonnet, 50 prompts x 2 reps per arm unless noted. Per-rep ranges and 95% intervals are in summary.md below. Same data, same prompts, same lookup code in both arms.
Every failure on the original server, as the log recorded it and as the customer read it.
The misprinted-book question failed on both reps, so five failing runs from four questions.
"Get a policy."
"Shelf Books' own store policies. Topics: returns, refunds, shipping, holds, gift_cards, price_match, special_orders, loyalty, damaged_items, signed_editions. Shelf's policies differ from typical retail norms, so ALWAYS check here before answering any question about returns, refunds, delivery, gift cards or other store rules."
{ "topic": "loyalty card", "policy": null }{ "topic": "loyalty card", "policy": null,
"hint": "No policy named 'loyalty card'. Available topics: returns, refunds,
shipping, holds, gift_cards, price_match, special_orders, loyalty,
damaged_items, signed_editions." }In 4 Haiku runs on the fixed server, the customer never gave an email, so Haiku searched orders with the email of the Claude account running the session (Claude Code puts it in the model's context). The search returned nothing and the user still got the policy answer, but the server received a real person's address the user never typed. On a multi-tenant server, that is a lookup on the wrong account, and the log would show it as a perfectly normal call. If your tools take an identity argument, validate where it came from.
Run it: start the server, then run.mjs with your own Claude login. The headless study (616 runs plus grading) cost about $5 in Claude Code's reported usage.
Get our tested Claude Code setups and test files by email, one email when a new test ships.
Double opt-in. Every email carries a one-click unsubscribe link.
Almost never. Sonnet called the server in 400 of 400 runs, including 200 runs where web search and file tools were also available. Haiku skipped it in 3 of 200 runs, twice reading "your loyalty card" as a question about itself. Those three conversations left no trace in the server log.
Every failure that reached the server looked like success: an HTTP 200, no error flag, and an empty or unhelpful result. The agent read "nothing" as an answer. In 19 of the 20 failures that reached the server there was at least one empty but successful result.
Two things, with no change to the lookup logic. Tool descriptions written for an agent (what the tool covers, the exact topic names, and a line saying this shop's rules differ from normal retail), and empty results that say what to try next instead of returning nothing.
Because the agent stopped guessing. On store policy questions, calls fell from 40 to 26 while right answers rose from 88% to 100%. If you read tool-call volume as adoption, a fix like this looks like a regression. Volume can also move the other way: on order questions the fixed descriptions made Claude fetch more detail, with right answers at 100% in both arms.
No. It came from one of two headless runs of that prompt. We ran the same question 11 more times in the interactive Claude Code UI against the original server and it answered correctly every time. The video shows the recorded failing run, labelled as recorded, next to a real interactive session that got it right.
Each prompt has required facts and forbidden facts taken from the shop's own data. A regex grader and a Haiku judge graded every answer; they agreed on 91%. Every disagreement (56) was reviewed by hand against the transcript, and a random 20% of the agreements were spot-checked.
No. One toy server, one client (Claude Code 2.1.325), Sonnet and Haiku, and prompts we wrote. The original server's exact-match lookups and bare empty results are deliberately API-like. Treat the numbers as a worked example of how to measure your own server, not a benchmark.
The idea and the “user value chain” come from Prathmesh Patel's MCP learnings series and his post Effective MCP, Part 1. The MCP-Atlas figure is from Scale AI, arXiv 2602.00933 v3. Every other number is from our own runs. Building agents or MCP servers at your company? The AI How consults for enterprises, SMEs and startups.