Tool calling

Tool calling is what turns a 0.6B model in a tab from a toy into something useful. The model doesn’t need to know the answer — it needs to know which of your functions to call. That works at sizes that fit in a browser.

Register a tool

chat.tool(
  'get_weather',                          // name
  'Get current weather for a city',       // description — the model reads this
  { city: 'string' },                     // shorthand JSON schema
  async ({ city }) => (await fetch(`/api/weather?c=${city}`)).json(),
);

Parameters take a shorthand: { city: 'string' } expands to { city: { type: 'string' } }. Pass a full property object when you need more:

chat.tool('search', 'Search the catalog',
  { q: 'string', limit: { type: 'number', description: 'max results' } },
  handler,
  { required: ['q'] },   // default: every property is required
);

tool() returns this, so registrations chain. The emitted schema is the standard OpenAI function envelope — { type: 'function', function: { name, description, parameters } } — readable at chat.toolSchemas.

The loop

chat.chat(prompt) runs the whole round trip:

1. Render your tools into the model’s chat template.
2. Generate. Strip <think>…</think> if it’s a reasoning model.
3. Parse any tool calls out of the raw output.
4. Run your handlers.
5. Feed the results back as tool messages and generate again.
6. Repeat up to chat.maxRounds (default 4), then return the grounded answer.

Multi-round and multi-tool: a model can call two tools in one turn, or call one, see the result, and call another. Watch it happen:

chat.on('token', t => render(t));
chat.on('toolCall', (call, result) => console.log(call.name, call.arguments, result));

The default system prompt shows the model the exact bytes of a tool call rather than describing them — small models copy a worked example far better than they follow an instruction. Two prompts are used: chat.systemPrompt while deciding whether to call, and chat.answerPrompt once results are in context. Both are overridable.

Making the call happen

A small model will sometimes answer a question it should have used a tool for. The library does not accept that. If a turn produces no tool call while tools are registered, it generates once more with the call syntax already open —

<tool_call>
{"name": "

— leaving no continuation that isn’t a call. The completion is parsed leniently (unterminated JSON, trailing prose and a name in the wrong case are all recovered), and the result is kept only if it names a tool you registered.

await chat.chat(question);                              // auto: force only if needed
await chat.chat(question, { toolChoice: 'required' });  // skip the free turn
await chat.chat(question, { toolChoice: 'none' });      // never force

A hallucinated name is never remapped

If the forced turn names a tool that does not exist, it is discarded — even when you registered exactly one tool. Calling the wrong tool is worse than calling none, and “it asked for launch_missiles, we have delete_account, close enough” is how that goes wrong. You get the model’s original answer, plus a tool_calls_force_failed metric and a forced event carrying the raw text.

Just give me a model that calls tools

load() picks the first quantization the host serves. That is a fact about the host and says nothing about whether the model can call a tool — and which quantization works is model-specific and inverted between models, so there is nothing to guess with.

loadForTools() asks the only question you care about. It loads a candidate, verifies it against a throwaway tool returning an unguessable token, and keeps the first one that both calls the tool and answers from the result:

const chat = await NexusChat.loadForTools({ hub: 'onnx-community/Qwen3-0.6B-ONNX' });
chat.tool('get_weather', 'Current weather', { city: 'string' }, getWeather);

await chat.chat('Weather in Chennai?');   // the model is already proven to call

A rejected candidate is disposed before the next is tried, so only one model is ever resident. If nothing works it throws, naming every dtype and why each failed — it will not hand you a chat object that silently cannot do the job:

no dtype of HuggingFaceTB/SmolLM2-360M-Instruct could call a tool on cpu. Tried:
  q4: ... does not call tools. Try another quantization, or a larger model.
  q8: ... does not call tools. Try another quantization, or a larger model.
Models below ~0.5B generally cannot pick a tool name from a list; try onnx-community/Qwen3-0.6B-ONNX.

Narrate the retry so a second download doesn’t look like a hang, and tighten the bar if you want a model that volunteers calls without priming:

await NexusChat.loadForTools(source, {
  onAttempt: (a) => console.log(a.dtype, a.ok, a.detail),
  allowForcing: false,        // reject models that only call when the syntax is primed
  dtype: 'q4',                // still verified — just not retried past
});

The tradeoff is honest: a rejected candidate was still downloaded. Weights are cached, so it is paid once per dtype per browser. If you already know the answer for your model — run npm run test:models once — pass dtype and skip the search.

When the model can’t produce the format

Some models understand the question and cannot emit the syntax. Priming the call fixes many of them; for the rest, the library stops asking for JSON altogether and escalates automatically — you don’t configure anything:

1. Inline — one generation, the whole call as JSON. What a tool-trained model does natively.
2. Primed — regenerate with <tool_call>\n{"name": " already open, so prose is not a possible continuation.
3. Stepwise — ask a closed question to pick the tool, then one question per argument, and assemble the call here.

Stepwise is structurally safer, not just statistically. The tool name is chosen from your registered list, never written by the model, so a hallucinated name cannot be dispatched. And each argument is asked for as a bare value with the value primed (city = "), so there is no JSON for a small model to malform.

await chat.chat(q);                              // auto: escalate only if needed
await chat.chat(q, { strategy: 'inline' });      // opt out of the extra round trips
await chat.chat(q, { strategy: 'stepwise' });    // skip inline entirely

Watch it happen — stepwise is many small generations, so a wrong argument is invisible without this:

chat.on('step', (which, raw, resolved) => console.log(which, resolved, raw));
// select   get_weather  "The tool needed is get_weather."
// arg:city get_weather  'Chennai"'

What it does and does not buy

Measured on real weights, same three questions:

Model	dtype	inline	with escalation
Qwen3-0.6B	q4	3/3 called	3/3 called, 0 rescues, no slower
Qwen2.5-0.5B	q8	1/3 called	2/3 called, 2 rescues

Two honest limits. Escalation costs nothing on a model that never needs it — Qwen3 spent zero extra generations. And it fixes format failures only: a model that picks the wrong tool, or ignores a result sitting in its context, is a capability limit that no decomposition repairs. Qwen2.5-0.5B at q8 improved at calling and did not improve at answering.

Stepwise is usually not slower

It costs 1+N generations instead of 1 — but each is a few tokens instead of up to 256, so measured wall-clock came out the same or faster. Concurrency does not help either way: three generations on one ONNX session measured a 0.99 ratio against sequential, because a session is a single compute stream.

When the model loops instead of answering

Decoding is greedy (do_sample: false) — deterministic, which is what you want when the output has to parse as JSON. But greedy decoding has no escape from a repetition loop: once a phrase becomes the highest-probability continuation it stays the highest-probability continuation, forever, and small models fall into that regularly. No system prompt fixes it, because it isn’t a comprehension failure.

So every generation carries a repetition penalty of 1.1 by default — the call turn, the forced turn and the answer turn alike:

await chat.chat(q, { repetitionPenalty: 1.3 });   // stubborn looper
await chat.chat(q, { repetitionPenalty: 1 });     // off

1.1 is deliberately mild. The penalty divides the logit of tokens already emitted, and JSON legitimately repeats ", , and : — past roughly 1.2 those structural tokens start losing their positions and tool-call JSON malforms. Raise it for prose-heavy turns, not for calls.

Which models actually call tools

Forcing raises the floor; it does not create ability the weights don’t have. Measured with npm run test:models — three questions whose answers are unguessable, so a correct reply proves a real call happened:

Model	q4	q8	fp16
Qwen2.5-0.5B-Instruct	3/3 usable	0/3	2/3 flaky
Qwen3-0.6B-ONNX	3/3 usable	3/3 usable	0/3
SmolLM2-360M-Instruct	0/3	0/3	0/3

Three things worth taking from that table.

The best quantization is model-specific. Qwen2.5 is reliable at q4 and useless at q8. There is no ordering that wins everywhere, so auto picks q4 first — the best single default, not a guarantee. Rather than trusting either the default or this table, use loadForTools(), which measures it on the machine the model is actually running on.

360M is below the floor. SmolLM2-360M never produced a usable call at any quantization; primed with open call syntax it invented tool names and degenerated. Around 0.5B is where tool calling starts working at all.

A failing combination fails loudly, not subtly. You get tool_calls_force_failed in metrics and a forced event with the raw text, so you can see the model naming a tool that doesn’t exist rather than wondering why answers are vague.

Ask the model itself, at runtime

The table above cannot cover a model your user just supplied — an archive they uploaded, a folder you serve, something converted this morning. So don’t guess: run a real tool call and see.

const chat = await NexusChat.load({ archive: fileTheUserPicked });

const check = await chat.selfCheck();
if (!check.ok) {
  showWarning(check.detail);
  // "…does not call tools. Try another quantization, or a larger model —
  //  below ~0.5B this usually cannot be fixed."
}

It registers a throwaway tool whose result is a token nothing could guess, asks for it, and checks whether that token comes back in the answer. Costs about one question — trivial next to loading the weights.

interface ToolCallCheck {
  ok: boolean;              // called a tool AND answered from its result
  called: boolean;          // did it call at all
  grounded: boolean;        // did the real value reach the answer
  needed_forcing: boolean;  // only called because the syntax was primed
  model, device, dtype: string;
  answer: string;
  detail: string;           // one sentence, safe to show a user
}

The three outcomes are genuinely different. called && !grounded is the dangerous one: the tool ran, and the model reported something else. Answers look right and are wrong. needed_forcing still passes, but the model is closer to the edge than one that volunteers a call.

Your registered tools and conversation history are saved and restored, so this is safe to run immediately after load(). The token / toolCall / raw hooks do fire during the check — the demo suppresses its diagnostics panel while it runs.

Check a model from the command line

Point it at any model and it tells you which quantization to use — or that the model can’t do the job:

npm run test:models -- --models your-org/your-model --dtypes q4,q8,fp16

VERDICT
------------------------------------------------------------------------------
onnx-community/Qwen2.5-0.5B-Instruct
  USE dtype 'q4' — 3/3 tool calls correct
  await NexusChat.load({ hub: 'onnx-community/Qwen2.5-0.5B-Instruct' }, { dtype: 'q4' });

A flaky model reports its best score and says it isn’t safe to ship; one that never calls says so outright. The run exits non-zero when nothing is usable, so it works as a CI gate before you pin a model.

Every call format, one parser

Small open models each invented their own way to say “call this function.” The parser handles all of them, in order, and stops at the first that yields calls:

Family	What it emits
Qwen, Hermes	<tool_call>{"name":…,"arguments":…}</tool_call>
Mistral	[TOOL_CALLS] [{"name":…,"arguments":…}]
Llama, others	bare JSON, or JSON in a ```json fence
OpenAI-shaped	nested { function: { name, arguments } }

Argument objects that arrive as a string of JSON are decoded too — a very common failure mode with quantized models. You can use the parser on its own:

import { parseToolCalls, stripThinking } from 'browser-llm-nexus/toolcalls';

Reasoning models

Models with a thinking mode are handled: generation passes enable_thinking: false where the template supports it, and any <think>…</think> block that still appears is stripped before parsing. You get the answer, not the monologue.

Tools in one file

Keep every tool in a single plain .js file and load it by URL:

await chat.loadTools('./tools.js');

// tools.js — no exports, no build step. `tool` is injected.
tool('get_weather', 'Get current weather for a city', { city: 'string' },
  async ({ city }) => (await fetch(`/api/weather?c=${city}`)).json());

tool('get_time', 'Get the current local time in a city', { city: 'string' },
  async ({ city }) => ({ city, time: new Date().toLocaleTimeString() }));

The file stays a file — editable, diffable, servable, swappable at runtime without a rebuild. loadTools also reports failures against the file: a 404, a network error, or a server that returns its HTML error page with status 200 all name the path, instead of surfacing as tool is not defined from inside an eval, which points at the wrong thing entirely.

Pass your own loader when you need auth headers, a bundler’s ?raw import, or a test double:

await chat.loadTools('/api/tools.js', { fetch: (u) => fetch(u, { headers }) });

Tools written at runtime

evalTools is the same thing one level down — it takes JavaScript you already have as a string and registers whatever it defines. loadTools is a fetch in front of it. Useful for letting an app’s own users extend the agent:

const names = await chat.evalTools(`
  tool('get_watch_count', 'How many watches the user owns', {},
    async () => (await db.watches.count()));
`);

It replaces the current tool set and returns the registered names.

Caution

evalTools runs the string with new Function. That is code execution with the privileges of your page. Only feed it code the current user authored for themselves — never a string that arrived from another user or a server you don’t control.

Related

• Together with toolnexus — the same schema, server-side, in six languages
• Knowledge & offline bundles — grounding in documents instead of functions