Skip to content
toolnexus

Performance benchmarks

Every number on this page is [MEASURED] on the machine below, with a reproducible harness committed under benchmarks/. We publish only figures we actually measured, and we report where a competitor edges toolnexus rather than hide it. A partial-but-real table beats a complete-but-fake one.

Measured July 2026 across the ports and their real competitors. Frameworks move fast — re-run before quoting.

TL;DR

Section titled “TL;DR”

Every framework points at the same local mock LLM and the same stdio MCP server, running one fixed tool-calling scenario (2 LLM round-trips + 2 tool calls). We measure only the framework’s own per-request overhead — a real model call (100 ms–seconds) would swamp it. Across 18 framework configurations in 6 languages:

  • toolnexus has the lowest or joint-lowest per-request overhead in Python, JavaScript, Go, and Elixir, and sits within noise of the fastest in C# and Java — where a lighter native-tool competitor edges it by tenths of a millisecond at the sub-few-ms floor, while toolnexus is doing real MCP.
  • toolnexus keeps the smallest or near-smallest install footprint in every language — often dramatically (vs CrewAI 828 MB, Google ADK 207 MB, LangChain.js 106 MB).
  • Every competitor that ships an MCP client did real MCP here; toolnexus does real MCP in all six languages.

Results — pick your language

Section titled “Results — pick your language”

All cells [MEASURED], N = 30 runs, 5 warmup, single machine (see Hardware). ok = the loop produced the correct final answer. Lower is better. Tables are split by tool source — an Over MCP row discovers tools over the real shared stdio MCP server (so it pays the subprocess spawn + tools/list), a Native tools row uses in-process tools with the same names/behavior. Comparing across the two is apples-to-oranges, so they’re separate. Click any column header to sort (ascending / descending).

Every Python competitor ships a first-class MCP client, so all rows are over the real MCP server.

Over MCP

Section titled “Over MCP”
Framework Init (ms) p50 (ms) p95 (ms) mean (ms) RSS (MB) Tools ok
toolnexus 17 1.00 1.23 1.04 59.6 3 ✅
toolnexus (+ skill + native) 19 1.07 1.37 1.12 59.4 5 ✅
OpenAI Agents SDK 19 4.14 4.55 4.13 109 3 ✅
Pydantic AI 37 5.45 6.34 6.43 123 3 ✅
LangGraph (persistent session) 144 7.00 9.72 8.32 127 3 ✅
Google ADK 23 8.80 10.53 8.87 292 3 ✅
CrewAI 41 10.46 15.36 11.06 211 3 ✅
LangGraph (reconnect-per-call)¹ 154 31.3 33.5 32.9 127 3 ✅

Install footprint: toolnexus 33 MB (2 deps) · OpenAI Agents 64 MB · LangGraph 83 MB · Pydantic AI 96 MB · Google ADK 207 MB · CrewAI 828 MB (heaviest; also needed a manual litellm add). ¹ LangGraph’s documented default re-spawns the MCP subprocess on every tool call; the persistent-session row is the apples-to-apples one.

Methodology

Section titled “Methodology”

Why a mock LLM

Section titled “Why a mock LLM”

Point every framework at the same local mock LLM and the same stdio MCP server, and measure only the framework’s own cost. The mock answers one fixed question with exactly one tool-calling turn (two parallel tool calls: get_weather, add), then a final message — so every run is 2 LLM round-trips + 2 tool executions through that framework’s own agent loop. Zero network, zero cost, deterministic. It picks tool names from each framework’s advertised list, so prefixed names work.

Metrics

Section titled “Metrics”
Metric How
Cold init Wall time to build the toolkit/agent including MCP connect + tool discovery. Honest per-language timers (perf_counter, time.Now, System.nanoTime, Stopwatch, monotonic_time).
p50/p95/mean Wall time per scenario run, over N = 30 measured runs after 5 warmup.
Peak RSS Max resident set of the runner process (/usr/bin/time -l).
Footprint Isolated per-framework install (du -sh of each venv/module/dll set).

Hardware / OS

Section titled “Hardware / OS”
Machine Apple Mac17,8 (arm64), 18 cores, 48 GB RAM
OS macOS 26.4
Runtimes CPython 3.11 · Node 22 · Go 1.23 · OpenJDK 21 · .NET 10 · Elixir 1.20/OTP 29
Network none — mock LLM + MCP server are both local, offline, deterministic

MCP vs native

Section titled “MCP vs native”

Where a framework ships a first-class MCP client (toolnexus everywhere; LangGraph, ADK, OpenAI Agents, Pydantic AI, CrewAI, Vercel AI SDK, LangChain.js, Mastra, LangChain4j, Eino), rows use the real shared stdio MCP server. Where MCP isn’t first-class (Semantic Kernel, Microsoft.Extensions.AI, langchaingo, Elixir LangChain), the row uses in-process native tools with identical names/behavior and is tagged (native). Native skips the stdio spawn + discovery, so a (native) row has an inherent small advantage over an MCP row — kept explicit so the comparison stays honest.

Reproduce it

Section titled “Reproduce it”

Full instructions + every runner: benchmarks/README.md. Each competitor installs in its own isolated venv/module (toolnexus from the local repo); start mock_llm.py, then run each runner (or run_all.py) — N = 30, 5 warmup, one JSON line each.

Verdict

Section titled “Verdict”

On pure framework overhead for a fixed MCP tool-calling scenario, toolnexus is the lowest or joint-lowest in Python, JavaScript, Go, and Elixir, and within noise of the fastest in C# and Java — where a lighter native-tool or MCP competitor edges it by fractions of a millisecond at a floor where a real model call is 100–1000× larger. What holds in every language: toolnexus does real MCP, carries the smallest or near-smallest install footprint, and never falls into the heavy tier (Spring AI, LangGraph, ADK, CrewAI). That matches its design goal — a small, right-sized library, not a runtime — and the honest asterisks (LangChain4j edges Java; SK/MEAI native edge C#) are exactly the kind of thing we’d rather publish than hide.