← ctx-optimize

We pre-committed a kill bar, ran the test on ourselves, and published it either way.

Before launch, this project's standing critique pre-committed a kill criterion: if the composite product (query + symbol cards + deterministic wiki) can't beat a plain grep agent by 25% on hostile terrain, stop — or pivot. This page is that test, run for real, with the losses left in.

Protocol

• Corpus: linux/block (98 C files / ~73k lines of kernel C — opaque naming, the terrain where a graph should help most), then a 40× bigger slice (3,987 files: block+mm+kernel+fs+drivers/block+drivers/nvme+lib) for the model-tier runs.
• Arms: the same agent answers each question twice. Arm A: shell search + file reads only, forbidden from using ctx-optimize. Arm B: told the store exists and to prefer it (query / card / explain / path / affected / wiki), file reads allowed to fill gaps.
• Questions: 12 real block-layer questions — 3 locate, 5 mechanism, 2 impact, 2 conceptual — with ground-truth keys derived by reading the source directly (never via the store, so the key can't favor arm B). questions.json (raw).
• Judging: every answer graded against the key before tokens count. A cheaper wrong answer is a loss, not a saving.
• Token accounting: each harness's own usage reporting; "fresh tokens" = input + cache-creation + output, cache reads tracked separately. Tokens are only ever compared within one harness — cross-harness token accounting is fiction.

The verdict

composite <25% on hostile terrain → stop, or pivot

The 25% bar was not cleared on frontier harnesses. Claude Code and Codex grep superbly and cache aggressively; the store lands at token parity at equal answer quality. Per the pre-commitment, the universal "saves tokens" claim is dead, and this site doesn't make it. Devin's daily quota cut its run at 6 of 12 pairs; its numbers carry high variance (single questions ranged −82% to +103%) and are labeled accordingly.

What survived — measured, not asserted

1. Onboarding / architecture traces: −34% to −49% tokens on both frontier CLIs. "Trace the lifecycle of a bio from submission to driver": Claude −34% tokens / −48% wall (22→11 turns), Codex −49% tokens. The wiki + cards class. Shallow enumeration questions ("list the three schedulers") show no such win — the class that pays is deep structure, not trivia.
2. Correctness where grep looks right and is wrong. On "which gatekeeper allows a back-merge", the grep arms of both Codex and Devin confidently named the wrong function; every store arm cited ll_back_merge_fn via call edges. On "which files call blk_mq_run_hw_queue", Sonnet's grep arm reported a comment mention as a real caller — the store arm explicitly flagged it as "comment, not an actual call".
3. Steps fall at every tier: −13% (Sonnet), −24% (Devin), −29% (Haiku). Whether saved steps become saved tokens is pure economics — see below.

The mechanism: step-cost economics

Devin reports zero cached tokens — every step re-pays the entire context — so −24% steps became −42% tokens and −34% wall. Claude Code's prompt caching makes an extra step nearly free, so even Haiku (−29% steps) lands at token parity, and the smaller models actually spend slightly longer per store call digesting output (+3%/+9% wall). The savings are real exactly where caching is absent or you pay per step: API scaffolds, CI loops, custom harnesses. On a cached frontier CLI the product buys fewer steps and fewer confidently-wrong answers, not a smaller bill.

Scale check (bonus finding)

The 3,987-file kernel slice became a 274,812-node / 529,562-edge store with 3,989 wiki pages in 5.5 seconds; queries answer in ~1.7s at that scale. (For calibration: graphify produced a similar node count on kubernetes in 6m46s.)

Defects this test surfaced in our own product

1. Query noise: C local-variable declarations are extracted as child nodes and pollute 50–80% of top-10 hits on kernel queries, burying the real symbol.
2. Card caller cap: card truncates caller lists at 15 ("… 4 more") — the hidden tail cost one arm a complete impact answer.
3. Doc terms unindexed: "writeback throttling" ranks blk-throttle.c (cgroup throttling) above blk-wbt.c (the actual wbt).

All three are on the fix list. When they're fixed we'll re-run arm B and publish both numbers, before and after.

Addendum 1 — cost-true accounting (cache reads are not free)

The tables above compare "fresh tokens", which prices cache reads at zero. They bill at ~10% of input price, and every extra agent turn re-reads the whole conversation. On total_cost_usd — the harness's own bill — the steered store arm is cheaper at every Claude tier: −9.2% (default, $3.55→$3.22), −5.6% (Sonnet), −16.2% (Haiku), driven by ~26% fewer cache re-reads. The parity verdict above is the free-cache view; the dollar view favors the store.

Addendum 2 — the natural-usage test (no steering at all)

The arms above steer the agent with a prompt. The honest product test is: install the skill, ask a plain question, observe. Measured:

• A skill alone never fires — 0 store calls across one-shots with both a passive and a maximally imperative description. (graphify's skill uses the same passive trigger pattern; this failure mode is not unique to us.)
• What works: a pointer block in the repo's CLAUDE.md/AGENTS.md — the instruction files Claude Code, Codex, Copilot, OpenCode and Devin all load natively. With it the agent made 55–57 unprompted store calls per 12-question run. ctx-optimize init now writes this block; install --skills covers every CLI with a native skill dir.
• Cost today: +6% vs a no-store baseline (v1 of the block: +11%; v2 "use it INSTEAD, cite directly, don't re-verify": +6%, turns −6%). Mechanism/architecture questions already win naturally (−13…25% cost); enumeration sweeps and body-constant questions lose — cards don't yet carry function bodies, so those reads still happen on top. That body-head card (in the original design, unbuilt) plus the three defects above are the fix list; we'll re-run this exact harness after and publish both runs.

Addendum 3 — the re-run: body-head cards clear the bar

Same day, same frozen harness, promise kept. The fix was the card the original design specified but v0 never built: card now returns the first ~30 lines of the symbol's body alongside signature/doc/callers, killing the follow-up read. Natural usage (skill + repo pointer, plain questions, zero steering), harness-billed dollars:

The pre-committed 25% bar is cleared in the most honest configuration we know how to build. Quality spot-verified on the cheapest runs — the $0.049 and $0.054 answers are complete, correct, straight-off-one-card, and again name the merge gatekeeper the grep arms kept getting wrong. Residuals stay published: 3 of 12 questions still cost more, and wall time is +31% (store calls run ~1–2s each at 275k nodes — latency is the next target). Trigger stack shipped as: generic skills (two standard dirs) + init-written repo pointer (CLAUDE.md/AGENTS.md) + one Claude Code UserPromptSubmit hook (the only supported CLI with a hook API); a hook-only one-shot with no repo files still used the store unprompted and answered correctly at baseline cost.

Raw data

• questions.json — the 12 questions with ground-truth keys
• metrics-frontier.json — per-run extraction, Claude/Codex/Devin
• metrics-tiers.json — per-run extraction, Haiku/Sonnet on the 40× corpus
• RESULTS.md · README.md — the in-repo write-up and protocol (plain markdown)
• Full per-run payloads (every answer, every usage report) ship in the product repo's proof/results*/.

Measured 2026-07-12 · Apple M5 Pro · Claude Code 2.1.207, Codex CLI 0.142.4, Devin CLI 2026.8.18, each on its own default settings, identical in both arms · back to the site