Skip to main content
Tested with Claude Code

Your MCP server's logs say 200 OK. Five users got nothing.

We built a fictional bookshop as an MCP server, wrote 50 questions real customers ask, and ran every one through headless Claude Code, twice, against two versions of the server. 200 runs. Then we graded every answer against the shop's own data.

Every one of the 171 tool calls on the original server returned HTTP 200. Five of the 100 answers were still wrong or incomplete, and the server log could not tell them apart from the 95 that worked.


The receipts

Original server vs fixed server

Measure
Original
Fixed
Tool calls that returned HTTP 200
171 / 171
152 / 152
Right answers, Sonnet (100 runs per arm)
95%
100%
Right answers, Haiku (100 runs per arm)
86%
99%
Empty but successful results
36
6
Calls an informed agent would not need
27%
13%
Store policy questions: calls / right answers
40 / 88%
26 / 100%

Sonnet, 50 prompts x 2 reps per arm unless noted. Per-rep ranges and 95% intervals are in summary.md below. Same data, same prompts, same lookup code in both arms.


The empty OK

What the server saw vs what the user got

Every failure on the original server, as the log recorded it and as the customer read it.

How does your loyalty card work?
Server log, all 200
get_store_policy("loyalty card") returned an empty policy
Customer was told
"I couldn't find anything on a loyalty card."
The shop's data
One stamp per £10, ten stamps is £10 off
Do you have the second Kingkiller book in?
Server log, all 200
check_stock and search_catalog both returned 0 results
Customer was told
"It doesn't appear to be in the catalog."
The shop's data
In the catalogue, out of stock, next delivery 25 Nov
My delivery came with a misprinted book. Do I post it back?
Server log, all 200
three policy lookups, two empty
Customer was told
"The store has no specific policy for misprinted books."
The shop's data
No need to return it; report within 48 hours with a photo
What's your policy on signed copies?
Server log, all 200
a guessed topic returned empty, then the general returns policy
Customer was told
Final sale (right), but told the transit-damage case isn't covered (wrong)
The shop's data
Signed editions: final sale unless damaged in transit

The misprinted-book question failed on both reps, so five failing runs from four questions.


The fix

Two changes, no new code paths

1. Describe the tool for an agent, not a developer

"Get a policy."
get_store_policy, original
"Shelf Books' own store policies. Topics: returns, refunds, shipping, holds,
gift_cards, price_match, special_orders, loyalty, damaged_items, signed_editions.
Shelf's policies differ from typical retail norms, so ALWAYS check here before
answering any question about returns, refunds, delivery, gift cards or other
store rules."
get_store_policy, fixed

2. Make empty results say what to try next

{ "topic": "loyalty card", "policy": null }
original: a successful, empty answer
{ "topic": "loyalty card", "policy": null,
  "hint": "No policy named 'loyalty card'. Available topics: returns, refunds,
           shipping, holds, gift_cards, price_match, special_orders, loyalty,
           damaged_items, signed_editions." }
fixed: still empty, now actionable

What to measure

Call counts measure the middle. Value lives at the ends.

  1. Keep a set of real prompts with known answers and run them through a real client on every change.
  2. Grade the answers, not the status codes. A 200 tells you the request worked, not that the user got anything.
  3. Alert on empty successful results. In our test they showed up in 19 of the 20 failures that reached the server.
  4. Do not read tool-call volume as adoption. Our fix cut calls by 11% while right answers went up.

One more thing the log showed

The agent used the operator's own email

In 4 Haiku runs on the fixed server, the customer never gave an email, so Haiku searched orders with the email of the Claude account running the session (Claude Code puts it in the model's context). The search returned nothing and the user still got the policy answer, but the server received a real person's address the user never typed. On a multi-tenant server, that is a lookup on the wrong account, and the log would show it as a perfectly normal call. If your tools take an identity argument, validate where it came from.


Download

The prompts, the server and the grading script

Run it: start the server, then run.mjs with your own Claude login. The headless study (616 runs plus grading) cost about $5 in Claude Code's reported usage.


Get the next tested setup

Get our tested Claude Code setups and test files by email, one email when a new test ships.

Double opt-in. Every email carries a one-click unsubscribe link.


Questions

FAQ

Did Claude Code skip the MCP server?

Almost never. Sonnet called the server in 400 of 400 runs, including 200 runs where web search and file tools were also available. Haiku skipped it in 3 of 200 runs, twice reading "your loyalty card" as a question about itself. Those three conversations left no trace in the server log.

Then what went wrong?

Every failure that reached the server looked like success: an HTTP 200, no error flag, and an empty or unhelpful result. The agent read "nothing" as an answer. In 19 of the 20 failures that reached the server there was at least one empty but successful result.

What exactly was the fix?

Two things, with no change to the lookup logic. Tool descriptions written for an agent (what the tool covers, the exact topic names, and a line saying this shop's rules differ from normal retail), and empty results that say what to try next instead of returning nothing.

Why did the fix lower the call count?

Because the agent stopped guessing. On store policy questions, calls fell from 40 to 26 while right answers rose from 88% to 100%. If you read tool-call volume as adoption, a fix like this looks like a regression. Volume can also move the other way: on order questions the fixed descriptions made Claude fetch more detail, with right answers at 100% in both arms.

Does the Kingkiller failure happen every time?

No. It came from one of two headless runs of that prompt. We ran the same question 11 more times in the interactive Claude Code UI against the original server and it answered correctly every time. The video shows the recorded failing run, labelled as recorded, next to a real interactive session that got it right.

How was "user value" graded?

Each prompt has required facts and forbidden facts taken from the shop's own data. A regex grader and a Haiku judge graded every answer; they agreed on 91%. Every disagreement (56) was reviewed by hand against the transcript, and a random 20% of the agreements were spot-checked.

Is this all MCP servers?

No. One toy server, one client (Claude Code 2.1.325), Sonnet and Haiku, and prompts we wrote. The original server's exact-match lookups and bare empty results are deliberately API-like. Treat the numbers as a worked example of how to measure your own server, not a benchmark.

The idea and the “user value chain” come from Prathmesh Patel's MCP learnings series and his post Effective MCP, Part 1. The MCP-Atlas figure is from Scale AI, arXiv 2602.00933 v3. Every other number is from our own runs. Building agents or MCP servers at your company? The AI How consults for enterprises, SMEs and startups.