# Summary (generated by scripts/summarize.mjs)

## main | sonnet | shelf | baseline

- runs: 100 (ungraded: 0)
- tool selected (>=1 server call): 100/100 = 100% (reps 100% / 100%; 95% CI 96%–100%)
- invisible to server (0 calls): 0/100 = 0% (reps 0% / 0%; 95% CI 0%–4%); miss types {}
- USER VALUE: 95/100 = 95% (reps 98% / 92%; 95% CI 89%–98%)
- PARTIAL: 1/100 = 1% (reps 0% / 2%; 95% CI 0%–5%)
- NO VALUE: 4/100 = 4% (reps 2% / 6%; 95% CI 2%–10%)
- server tools/call count: 171 (per rep 88 / 83); mean calls per run per rep 1.76 / 1.66
- tools/call HTTP 200: 171/171; all HTTP: 371/671 were 200; non-200: {"POST 400":100,"POST 202":100,"GET 405":100}
- isError results: 0; empty-but-successful results: 36 (per rep 19 / 17)
- wasted calls: 46 = 27% of calls (per rep 24 / 22); exact-duplicate calls 0
- runs that used ToolSearch first: 100/100; other tools used: {}
- cost per run (total_cost_usd): min $0.0041, median $0.0084, max $0.0147; total $0.79; turns 3–6

| bucket | runs | server calls | share of calls | calls/run | user value | invisible | wasted | empty 200s |
|---|---|---|---|---|---|---|---|---|
| named | 24 | 32 | 19% | 1.33 | 24/24 = 100% (reps 100% / 100%; 95% CI 86%–100%) | 0 | 6 | 6 |
| implicit | 28 | 38 | 22% | 1.36 | 28/28 = 100% (reps 100% / 100%; 95% CI 88%–100%) | 0 | 4 | 0 |
| trap | 24 | 40 | 23% | 1.67 | 21/24 = 88% (reps 92% / 83%; 95% CI 69%–96%) | 0 | 19 | 14 |
| loop | 24 | 61 | 36% | 2.54 | 22/24 = 92% (reps 100% / 83%; 95% CI 74%–98%) | 0 | 17 | 16 |

## main | sonnet | shelf | fix

- runs: 100 (ungraded: 0)
- tool selected (>=1 server call): 100/100 = 100% (reps 100% / 100%; 95% CI 96%–100%)
- invisible to server (0 calls): 0/100 = 0% (reps 0% / 0%; 95% CI 0%–4%); miss types {}
- USER VALUE: 100/100 = 100% (reps 100% / 100%; 95% CI 96%–100%)
- PARTIAL: 0/100 = 0% (reps 0% / 0%; 95% CI 0%–4%)
- NO VALUE: 0/100 = 0% (reps 0% / 0%; 95% CI 0%–4%)
- server tools/call count: 152 (per rep 76 / 76); mean calls per run per rep 1.52 / 1.52
- tools/call HTTP 200: 152/152; all HTTP: 352/652 were 200; non-200: {"POST 400":100,"POST 202":100,"GET 405":100}
- isError results: 0; empty-but-successful results: 6 (per rep 3 / 3)
- wasted calls: 20 = 13% of calls (per rep 10 / 10); exact-duplicate calls 0
- runs that used ToolSearch first: 100/100; other tools used: {}
- cost per run (total_cost_usd): min $0.0038, median $0.0067, max $0.0152; total $0.75; turns 3–6

| bucket | runs | server calls | share of calls | calls/run | user value | invisible | wasted | empty 200s |
|---|---|---|---|---|---|---|---|---|
| named | 24 | 28 | 18% | 1.17 | 24/24 = 100% (reps 100% / 100%; 95% CI 86%–100%) | 0 | 2 | 0 |
| implicit | 28 | 44 | 29% | 1.57 | 28/28 = 100% (reps 100% / 100%; 95% CI 88%–100%) | 0 | 10 | 0 |
| trap | 24 | 26 | 17% | 1.08 | 24/24 = 100% (reps 100% / 100%; 95% CI 86%–100%) | 0 | 2 | 0 |
| loop | 24 | 54 | 36% | 2.25 | 24/24 = 100% (reps 100% / 100%; 95% CI 86%–100%) | 0 | 6 | 6 |

## supp | haiku | shelf | fix

- runs: 100 (ungraded: 0)
- tool selected (>=1 server call): 99/100 = 99% (reps 98% / 100%; 95% CI 95%–100%)
- invisible to server (0 calls): 1/100 = 1% (reps 2% / 0%; 95% CI 0%–5%); miss types {"never_discovered":1}
- USER VALUE: 99/100 = 99% (reps 98% / 100%; 95% CI 95%–100%)
- PARTIAL: 0/100 = 0% (reps 0% / 0%; 95% CI 0%–4%)
- NO VALUE: 1/100 = 1% (reps 2% / 0%; 95% CI 0%–5%)
- server tools/call count: 150 (per rep 73 / 77); mean calls per run per rep 1.46 / 1.54
- tools/call HTTP 200: 150/150; all HTTP: 350/650 were 200; non-200: {"POST 400":100,"POST 202":100,"GET 405":100}
- isError results: 0; empty-but-successful results: 7 (per rep 3 / 4)
- wasted calls: 19 = 13% of calls (per rep 8 / 11); exact-duplicate calls 0
- runs that used ToolSearch first: 99/100; other tools used: {}
- cost per run (total_cost_usd): min $0.0002, median $0.0005, max $0.0012; total $0.06; turns 1–6

| bucket | runs | server calls | share of calls | calls/run | user value | invisible | wasted | empty 200s |
|---|---|---|---|---|---|---|---|---|
| named | 24 | 28 | 19% | 1.17 | 24/24 = 100% (reps 100% / 100%; 95% CI 86%–100%) | 0 | 2 | 0 |
| implicit | 28 | 45 | 30% | 1.61 | 28/28 = 100% (reps 100% / 100%; 95% CI 88%–100%) | 0 | 11 | 0 |
| trap | 24 | 27 | 18% | 1.13 | 23/24 = 96% (reps 92% / 100%; 95% CI 80%–99%) | 1 | 4 | 3 |
| loop | 24 | 50 | 33% | 2.08 | 24/24 = 100% (reps 100% / 100%; 95% CI 86%–100%) | 0 | 2 | 4 |

## supp | haiku | shelf | baseline

- runs: 100 (ungraded: 0)
- tool selected (>=1 server call): 98/100 = 98% (reps 100% / 96%; 95% CI 93%–99%)
- invisible to server (0 calls): 2/100 = 2% (reps 0% / 4%; 95% CI 1%–7%); miss types {"never_discovered":2}
- USER VALUE: 86/100 = 86% (reps 82% / 90%; 95% CI 78%–91%)
- PARTIAL: 1/100 = 1% (reps 2% / 0%; 95% CI 0%–5%)
- NO VALUE: 13/100 = 13% (reps 16% / 10%; 95% CI 8%–21%)
- server tools/call count: 170 (per rep 84 / 86); mean calls per run per rep 1.68 / 1.72
- tools/call HTTP 200: 170/170; all HTTP: 370/670 were 200; non-200: {"POST 400":100,"POST 202":100,"GET 405":100}
- isError results: 0; empty-but-successful results: 46 (per rep 24 / 22)
- wasted calls: 53 = 31% of calls (per rep 28 / 25); exact-duplicate calls 0
- runs that used ToolSearch first: 98/100; other tools used: {}
- cost per run (total_cost_usd): min $0.0003, median $0.0006, max $0.0013; total $0.06; turns 1–6

| bucket | runs | server calls | share of calls | calls/run | user value | invisible | wasted | empty 200s |
|---|---|---|---|---|---|---|---|---|
| named | 24 | 32 | 19% | 1.33 | 24/24 = 100% (reps 100% / 100%; 95% CI 86%–100%) | 0 | 6 | 6 |
| implicit | 28 | 37 | 22% | 1.32 | 28/28 = 100% (reps 100% / 100%; 95% CI 88%–100%) | 0 | 3 | 0 |
| trap | 24 | 37 | 22% | 1.54 | 14/24 = 58% (reps 50% / 67%; 95% CI 39%–76%) | 1 | 23 | 19 |
| loop | 24 | 64 | 38% | 2.67 | 20/24 = 83% (reps 75% / 92%; 95% CI 64%–93%) | 1 | 21 | 21 |

## supp | sonnet | full | baseline

- runs: 100 (ungraded: 0)
- tool selected (>=1 server call): 100/100 = 100% (reps 100% / 100%; 95% CI 96%–100%)
- invisible to server (0 calls): 0/100 = 0% (reps 0% / 0%; 95% CI 0%–4%); miss types {}
- USER VALUE: 97/100 = 97% (reps 96% / 98%; 95% CI 92%–99%)
- PARTIAL: 0/100 = 0% (reps 0% / 0%; 95% CI 0%–4%)
- NO VALUE: 3/100 = 3% (reps 4% / 2%; 95% CI 1%–8%)
- server tools/call count: 170 (per rep 85 / 85); mean calls per run per rep 1.7 / 1.7
- tools/call HTTP 200: 170/170; all HTTP: 370/670 were 200; non-200: {"POST 400":100,"POST 202":100,"GET 405":100}
- isError results: 0; empty-but-successful results: 37 (per rep 20 / 17)
- wasted calls: 43 = 25% of calls (per rep 22 / 21); exact-duplicate calls 0
- runs that used ToolSearch first: 100/100; other tools used: {}
- cost per run (total_cost_usd): min $0.0096, median $0.0168, max $0.0274; total $1.58; turns 3–6

| bucket | runs | server calls | share of calls | calls/run | user value | invisible | wasted | empty 200s |
|---|---|---|---|---|---|---|---|---|
| named | 24 | 33 | 19% | 1.38 | 24/24 = 100% (reps 100% / 100%; 95% CI 86%–100%) | 0 | 7 | 7 |
| implicit | 28 | 36 | 21% | 1.29 | 28/28 = 100% (reps 100% / 100%; 95% CI 88%–100%) | 0 | 2 | 0 |
| trap | 24 | 38 | 22% | 1.58 | 21/24 = 88% (reps 83% / 92%; 95% CI 69%–96%) | 0 | 17 | 14 |
| loop | 24 | 63 | 37% | 2.63 | 24/24 = 100% (reps 100% / 100%; 95% CI 86%–100%) | 0 | 17 | 16 |

## supp | sonnet | full | fix

- runs: 100 (ungraded: 0)
- tool selected (>=1 server call): 100/100 = 100% (reps 100% / 100%; 95% CI 96%–100%)
- invisible to server (0 calls): 0/100 = 0% (reps 0% / 0%; 95% CI 0%–4%); miss types {}
- USER VALUE: 100/100 = 100% (reps 100% / 100%; 95% CI 96%–100%)
- PARTIAL: 0/100 = 0% (reps 0% / 0%; 95% CI 0%–4%)
- NO VALUE: 0/100 = 0% (reps 0% / 0%; 95% CI 0%–4%)
- server tools/call count: 149 (per rep 75 / 74); mean calls per run per rep 1.5 / 1.48
- tools/call HTTP 200: 149/149; all HTTP: 349/649 were 200; non-200: {"POST 400":100,"POST 202":100,"GET 405":100}
- isError results: 0; empty-but-successful results: 4 (per rep 2 / 2)
- wasted calls: 17 = 11% of calls (per rep 9 / 8); exact-duplicate calls 0
- runs that used ToolSearch first: 100/100; other tools used: {}
- cost per run (total_cost_usd): min $0.0093, median $0.0133, max $0.0307; total $1.48; turns 3–6

| bucket | runs | server calls | share of calls | calls/run | user value | invisible | wasted | empty 200s |
|---|---|---|---|---|---|---|---|---|
| named | 24 | 28 | 19% | 1.17 | 24/24 = 100% (reps 100% / 100%; 95% CI 86%–100%) | 0 | 2 | 0 |
| implicit | 28 | 44 | 30% | 1.57 | 28/28 = 100% (reps 100% / 100%; 95% CI 88%–100%) | 0 | 10 | 0 |
| trap | 24 | 26 | 17% | 1.08 | 24/24 = 100% (reps 100% / 100%; 95% CI 86%–100%) | 0 | 2 | 0 |
| loop | 24 | 51 | 34% | 2.13 | 24/24 = 100% (reps 100% / 100%; 95% CI 86%–100%) | 0 | 3 | 4 |

## Spend (ledger, total_cost_usd)

- probe: $0.08
- pilot: $0.17
- main: $1.55
- supp: $3.18
- judge: $0.10
- TOTAL: $5.08 of $40 cap
