Skip to content
browser-llm-nexus

GPU or CPU — same API

Most browser-LLM projects are a WebGPU showcase: no adapter, no product. That rules out the managed corporate laptop with GPU acceleration disabled by policy — which is precisely the machine where “this data must not leave the page” is a hard requirement rather than a nice idea.

GPU is an accelerator here, never a requirement. The same page works on a workstation and on a locked-down laptop, with the same code.

How the backend is chosen

Section titled “How the backend is chosen”

detectDevice asks the browser for a GPU adapter and falls back to WASM if there isn’t one — or if requesting the adapter throws:

const chat = await NexusChat.load(source);                     // auto: webgpu → wasm
const chat = await NexusChat.load(source, { device: 'wasm' }); // force CPU
const chat = await NexusChat.load(source, { device: 'webgpu' }); // force GPU

After loading, the resolved backend is readable:

console.log(chat.device);   // 'webgpu' | 'wasm'

NexusEmbedder.load takes the same option and does the same detection, so an embedder and a chat model in one page can be forced onto different backends if you want to.

dtype follows the backend

Section titled “dtype follows the backend”

Quantization that is fast on a GPU is not the quantization that is fast on a CPU, so the preferred order differs:

Backend Order tried
webgpu q4 → fp16 → q8 → fp32
wasm / CPU q4 → q8 → fp16 → fp32

q4 comes first on both backends, which is not the conventional choice — a GPU has the bandwidth for fp16, so preferring it looks obviously right. On small models it isn’t. Measured on Qwen2.5-0.5B-Instruct across three tool-calling questions (npm run test:models reproduces this):

dtype Called the tool Answered correctly from the result
q4 3/3 3/3
fp16 3/3 2/3 — read 1096637 back as 109,663,700
q8 0/3 0/3 — narrates “I would need to use a specific tool” and never calls

A faster answer that silently corrupts a tool result is worse than a slower correct one, so correctness picks the default. But note this is the best single default, not a universal truth — Qwen3-0.6B is the reverse, reliable at q8 and unusable at fp16. See which models actually call tools for the measured matrix, and state the dtype yourself once you have verified your own model:

await NexusChat.load(source, { dtype: 'fp16' });

The library probes which variants your source actually serves — a HEAD request per candidate against <base>/<model>/onnx/<file> — and takes the first that exists. It never assumes a variant is there.

dtype File probed
q4 model_q4.onnx
q8 model_quantized.onnx
fp16 model_fp16.onnx
fp32 model.onnx

Skip probing when you already know what you’re serving — one fewer round trip per candidate:

await NexusChat.load(source, { dtype: 'q4' });

Probing needs a base URL to probe against. If you hit cannot probe dtypes without a base URL, pass an explicit dtype.

The CPU number nobody tells you about

Section titled “The CPU number nobody tells you about”

If CPU inference feels far slower than it should, check this before blaming the model:

console.log(self.crossOriginIsolated);   // false = single-threaded WASM

onnxruntime-web uses WASM threads, which need SharedArrayBuffer, which browsers only expose on a cross-origin isolated page. That requires two response headers:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

Without them the runtime silently falls back to one thread — on any machine, however many cores it has. Nothing errors. It is just several times slower, and it looks like the library being slow.

Measured on one Apple-silicon machine (18 cores), Qwen2.5-0.5B at q4, same weights throughout, generating the same prompt:

  • ~51 tok/s — native ONNX Runtime on CPU (Node, threaded)
  • ~14–17 tok/s — browser WebGPU
  • ~4.7 tok/s — browser WASM with threads (cross-origin isolated)
  • ~1.5 tok/s — browser WASM without threads (a plain static host)

Two things in that list are worth pausing on. Cross-origin isolation is worth about 3× on the CPU path and costs two response headers. And for a model this small, native CPU beats browser WebGPU by roughly 3× — the GPU is not the bottleneck at 0.5B; per-call overhead is. Do not assume WebGPU is the fast path until you have measured your model.

Enabling isolation has a real cost: every cross-origin resource on the page then needs Cross-Origin-Resource-Policy or CORP-enabled CORS, which can break third-party embeds. That trade is why it isn’t the default anywhere.

WebGPU is unaffected — it doesn’t use WASM threads — which is one reason the GPU path can look dramatically better on a hosted page than the CPU path, even where the hardware gap is much smaller.

What this buys you

Section titled “What this buys you”
  • One build, one code path, every machine — no “unsupported browser” branch.
  • A CPU fallback that is slow but correct, instead of a blank page.
  • Freedom to ship a small quantized model as the floor and let GPU machines pick fp16.
Section titled “Related”