Don't trust our numbers — run them. This harness clones a small repo, builds both knowledge stores offline (no LLM), and lets the same model answer a set of questions three ways over OpenRouter. Tokens and dollars are OpenRouter's own accounting (usage.include=true), not our estimate.
| arm | tools the model gets | store built with |
|---|---|---|
| a · shell | run_shell (grep/rg/find/sed/cat) | — |
| b · ctx-optimize | query/card/affected/path/explain + shell for gaps | ctx-optimize add . |
| c · graphify | query/explain/affected/path + shell for gaps | graphify update . --no-cluster |
gorilla/mux, openai/gpt-4o-minigorilla/mux is a small, well-named repo — plain grep's best case, i.e. the hardest terrain for a graph to win on. Deltas are vs arm a (shell).
| metric | a shell | b ctx-optimize | c graphify | b vs a | c vs a |
|---|---|---|---|---|---|
| tokens | 15,078 | 9,659 | 18,352 | −36% | +22% |
| cost | $0.0024 | $0.0016 | $0.0025 | −31% | +7% |
| tool calls | 11 | 4 | 8 | −64% | −27% |
| wall time | 16.5s | 11.8s | 18.7s | −29% | +13% |
query/card call. graphify cuts tool calls too — but its query returns a raw BFS dump of 100+ nodes, so the model burns more tokens wading through it, landing graphify above plain grep on tokens and cost. Head-to-head, ctx-optimize uses about half graphify's tokens, cost, and steps. Both arms answered correctly with file:line citations — a cheaper wrong answer would be a loss, not a saving.This table wasn't typed by hand — a GitHub Actions job produced it, headless, on a clean runner. Open the run and read every log line yourself:
▶ Watch this run on GitHub Actions →
The workflow lives in this public repo, so the run is fully viewable and forkable — no private access needed. Below are the exact raw records it wrote, straight from the job's artifact: each file is the model's full transcript for one arm × one question — the tools it called, OpenRouter's token/cost accounting, and the final answer.
| field | value |
|---|---|
| workflow | benchmark.yml · public, forkable |
| run | #29193645340 · commit 9529e8e |
| date | 2026-07-12 · conclusion success · headless |
| model | openai/gpt-4o-mini via OpenRouter |
The 12 raw records (arm × question) + the generated summary — open any one:
The harness runs on the published CLI. Both stores build offline; only the model calls cost anything (cents on a cheap model).
npm i -g @muthuishere/ctx-optimize # the store CLI pipx install graphifyy # the competitor (arm c; optional) export OPENROUTER_API_KEY=sk-or-... # read from env only, never logged # grab the harness (the five files below) into proof/agent/, then: bash proof/agent/run-bench.sh # defaults: gorilla/mux, openai/gpt-4o-mini # options: --model <slug> --repo <url> --name <short> --questions <file>
It prints a per-question table, the totals above, and writes one raw JSON record per run (inspect the answer field and judge quality for yourself). arm c is skipped automatically if graphify isn't installed.
Every file, plain and readable — nothing hidden:
We run benchmark.yml on every push; it produces this exact table in the job summary and uploads the raw records as an artifact.