Which models actually work

Every row here was produced by npm run test:models against real weights, asking 3 questions whose answers are unguessable tokens — so a correct reply proves a tool call really happened rather than the model guessing well.

Measured 2026-08-02 on cpu, library 0.7.0.

Model	Params	q4	q8	fp16
onnx-community/Qwen3-0.6B-ONNX	0.6B	✅ usable · 3/3	✅ usable · 3/3	❌ broken · 0/3
onnx-community/Qwen2.5-0.5B-Instruct	0.5B	✅ usable · 3/3	⚠️ poor · 2/3	⚠️ flaky · 3/3
HuggingFaceTB/SmolLM2-360M-Instruct	0.36B	❌ 0/3 · 0/3	❌ 0/3 · 0/3	❌ 0/3 · 0/3

Why the marked ones fail

• onnx-community/Qwen3-0.6B-ONNX @ fp16 — Transformers.js throws ‘Tensor.data must be a typed array (4) for float16 tensors’ on this model — a runtime defect, not a model one.
• onnx-community/Qwen2.5-0.5B-Instruct @ q8 — Inline it emits {“name”: “rain”} — a tool that does not exist. Stepwise escalation gets it calling 2/3, but it still answers from its own guess instead of the tool result.
• onnx-community/Qwen2.5-0.5B-Instruct @ fp16 — Read a tool result of 1096637 back as 109,663,700.

Retrieval — which embedding models find the right chunk

npm run test:rag measures the other half: given a question, does the right document come back? The corpus is built to be hard — every question’s key phrase appears in two documents and only a qualifier (standard/express, production/staging, severity one/two) picks the right one, so lexical overlap alone cannot answer it. Answers are unguessable tokens, and the harness is checked against itself: swapping in random vectors drops recall@1 from 10/10 to 1/10, so a good score is not an artefact of an easy corpus.

• Xenova/all-MiniLM-L6-v2 @ q8 — recall@1 100%, margin 0.106 — Largest separation from the near-miss distractor.
• Xenova/bge-small-en-v1.5 @ q8 — recall@1 100%, margin 0.084
• Xenova/gte-small @ q8 — recall@1 100%, margin 0.038 — Correct every time, but only 0.038 ahead of the wrong look-alike — the thinnest margin measured.

Margin is the number to read, not recall. All three retrieve correctly on every question; what separates them is how far ahead the right document scores over the near-miss. Xenova/all-MiniLM-L6-v2 leads by 0.106, Xenova/gte-small by only 0.038 — correct today, and much closer to a coin flip on a corpus slightly harder than this one.

End to end

With Xenova/all-MiniLM-L6-v2 q8 retrieving and onnx-community/Qwen3-0.6B-ONNX q4 answering: 10/10 grounded answers — the reply contained the unguessable token from the correct document, so retrieval and generation both did their job. The whole knowledge base exported to a 31 KB zip and, after reimport, returned identical retrieval.

Note the contrast with tool calling above: retrieval is far more reliable at these model sizes. Summarising a retrieved paragraph is a much easier task than choosing a function and emitting valid JSON, which is why an offline knowledge system is a better fit for a small in-browser model than an agent is.

Reading this table

Only models we measured appear here. Anything else is untested rather than bad — It may work perfectly — run chat.selfCheck() and find out on your own machine.

Tool calling starts working around 0.5B. Below that, a model primed with open call syntax invents tool names rather than choosing one.

The important thing this table shows is that the best quantization is model-specific and does not transfer: Qwen2.5-0.5B is reliable at q4 and poor at q8, while Qwen3-0.6B is fine at both and broken at fp16. There is no ordering that wins everywhere, which is exactly why loadForTools() measures instead of assuming.

Check your own model

This table cannot cover a model you converted yourself or loaded from your own host. Ask it directly — one throwaway tool, one unguessable token, a few seconds:

const chat = await NexusChat.loadForTools(source);  // picks a working dtype, or throws
const check = await chat.selfCheck();
// { ok, called, grounded, needed_forcing, dtype, detail }

To reproduce this whole table locally:

npm run test:models