Run a model in-process (ONNX, no server, no socket)

Why you need this

Not every model lives behind a URL. Yours might be an ONNX Runtime session, a llama.cpp binding, a fine-tune loaded straight into your process, or a router you wrote yourself. toolnexus never assumes a network: the same seam that lets you supply a proxy lets you supply the model itself. Everything above it — the tool-calling loop, MCP tools, skills, sub-agents, retries, memory — works unchanged, because none of it knows whether a socket was involved.

A local OpenAI-compatible server — llama-server, Ollama, vLLM, LM Studio — is just a baseUrl, so it needs no recipe. This page is about the case that does: the model runs inside your process and there is no server at all.

One constructor. No wire to configure.

const client = createInProcessClient({
  model: "my-local",
  generate: (request) => ({ content: "…" }),
})

That is the whole integration. No baseUrl — there is no URL. No apiKey — there is no auth. No style — there is no wire to be neutral about. And no Response, no RoundTripper, no HttpMessageHandler: the HTTP envelope is the library’s job, not yours.

It is an ordinary client

createInProcessClient is a second constructor, not a second seam — it is built on the same injectable transport that fetch / httpClient uses. So the tool-calling loop, MCP servers, agent skills, sub-agents, hooks, metrics, conversation memory and the completion gate all behave exactly as they do against a hosted model.

The contract

Your generate receives the assembled request and returns exactly one assistant message.

You receive:

field
messages	the conversation so far, including the system prompt and tool results
tools	the tool schemas offered for this call
model	the model name you passed
body	every other key the client assembled, if you want it

You return either an answer, or a request for tools — never both:

{ content: "The answer is 5." }                                  // finish the run
{ toolCalls: [{ name: "add", arguments: { a: 2, b: 3 } }] }      // call a tool

Optionally add usage. Omit it and the run reports zero tokens rather than failing.

Three things the library does so you do not have to: it derives finish_reason, it builds the choices envelope, and it encodes tool arguments — pass a structured value and it is serialized for you, or pass an already-encoded string and it goes through untouched.

Failures are final, not retried

A network client retries transient failures. An in-process one does not, by default: there is no wire, so there is nothing transient — whatever your generate throws, it will throw again. Retrying only buys backoff before you see your own bug. Concretely, the streaming refusal below took 3.7 seconds to surface before this default, and now takes 1 millisecond.

A model that genuinely is flaky (a GPU that OOMs, say) can opt back in: set retries, or in Go — where Retries: 0 already means “use the default of 2” — supply OnError.

Streaming is refused, not faked

generate returns a whole answer, so a stream would be one chunk pretending to be many — and a buffered body is indistinguishable from a real stream by content or by delta count. The streaming path therefore raises with a message naming the limitation. If you need real streaming, supply a transport instead.

The exception is Clojure, which has no streaming entry point at all, so there is nothing there to refuse.

In every language

Each of these runs a full tool-calling turn — the model asks for add, the tool executes, the model answers — with zero sockets opened and no wire configuration.

JavaScript

import { createInProcessClient, createToolkit } from "toolnexus"

let turn = 0
const client = createInProcessClient({
  model: "my-local",
  generate: (req) => {
    if (++turn === 1) return { toolCalls: [{ name: "add", arguments: { a: 2, b: 3 } }] }
    return { content: `The answer is ${req.messages.at(-1).content}.` }
  },
})

const r = await client.run("What is 2 + 3?", { toolkit: await createToolkit({ builtins: false, extraTools: [add] }) })

Python

from toolnexus import create_in_process_client, create_toolkit

turn = 0

def generate(req):
    global turn
    turn += 1
    if turn == 1:
        return {"tool_calls": [{"name": "add", "arguments": {"a": 2, "b": 3}}]}
    return {"content": f"The answer is {req['messages'][-1]['content']}."}

client = create_in_process_client(model="my-local", generate=generate)
r = await client.run("What is 2 + 3?", toolkit=await create_toolkit(builtins=False, extra_tools=[add]))

Go

turn := 0
client := tn.CreateInProcessClient(tn.InProcessOptions{
    Model: "my-local",
    Generate: func(req tn.InProcessRequest) (tn.InProcessResponse, error) {
        turn++
        if turn == 1 {
            return tn.InProcessResponse{ToolCalls: []tn.InProcessToolCall{
                {Name: "add", Arguments: map[string]any{"a": 2, "b": 3}},
            }}, nil
        }
        return tn.InProcessResponse{Content: "The answer is 5."}, nil
    },
})

r, err := client.Run(ctx, "What is 2 + 3?", tk)

Java

AtomicInteger turn = new AtomicInteger();

LlmClient client = InProcess.createClient(new InProcess.Options()
        .model("my-local")
        .generate(req -> turn.incrementAndGet() == 1
                ? InProcess.Response.toolCalls(
                        new InProcess.ToolCall("add", Map.of("a", 2, "b", 3)))
                : InProcess.Response.content("The answer is 5.")));

LlmClient.RunResult r = client.run("What is 2 + 3?", tk);

No HttpClient subclass: the eleven abstract methods this port used to demand are implemented once, inside the library.

C#

var turn = 0;

var client = InProcess.CreateClient(new InProcess.Options
{
    Model = "my-local",
    Generate = req => ++turn == 1
        ? InProcess.Response.FromToolCalls(new InProcess.ToolCall
        {
            Name = "add",
            Arguments = new Dictionary<string, object?> { ["a"] = 2, ["b"] = 3 },
        })
        : InProcess.Response.FromContent("The answer is 5."),
});

var r = await client.RunAsync("What is 2 + 3?", tk);

Elixir

{:ok, turn} = Agent.start_link(fn -> 0 end)

client =
  Toolnexus.Client.create_in_process(
    model: "my-local",
    generate: fn req ->
      if Agent.get_and_update(turn, &{&1, &1 + 1}) == 0 do
        %{tool_calls: [%{name: "add", arguments: %{"a" => 2, "b" => 3}}]}
      else
        %{content: "The answer is #{List.last(req.messages)["content"]}."}
      end
    end
  )

r = Toolnexus.Client.run(client, "What is 2 + 3?", tk)

Clojure

(require '[toolnexus.client :as client])

(def turn (atom 0))

(def c
  (client/create-in-process-client
   {:model "my-local"
    :generate (fn [req]
                (if (= 1 (swap! turn inc))
                  {:tool-calls [{:name "add" :arguments {:a 2 :b 3}}]}
                  {:content (str "The answer is "
                                 (:content (last (:messages req))) ".")}))}))

(def r (client/run c "What is 2 + 3?" {:toolkit tk}))

All seven were executed, not written from memory

Each tab was run against its port before publication. Every port also ships a test suite for this constructor covering the same seven behaviours: no wire config needed · generate sees messages/tools/model · tool calls loop back · arguments structured or pre-encoded · usage optional and derived · streaming refused · generate required.

A real model, end to end

The constructor is model-agnostic. Two runnable examples in the repo wire it to an actual ONNX model — Qwen/Qwen2.5-1.5B-Instruct, int8, 1.7 GB — including the ChatML template and <tool_call> parsing a real model needs:

Python	python/examples/local_onnx_model.py
Go	golang/examples/onnx/

tool call : get_weather({'city': 'Chennai'}) -> {"city": "Chennai", "tempC": 31, "sky": "clear"}
answer    : The current weather in Chennai is 31°C with clear skies.
turns     : 2 | sockets opened: 0

One warning worth inheriting: pick a model actually trained for tool calling. A 135M instruct model will hallucinate an answer rather than call, and no amount of transport code fixes that.

Still want the raw transport?

It has not gone anywhere. fetch / httpClient / transport remains the answer when you need a proxy, mTLS, credential injection or record-replay — see Bring your own HTTP client. The in-process constructor is built on it, not beside it.

Appendix: the raw transport’s body shape, per port

You do not need this to run an in-process model — createInProcessClient handles it. It matters only if you drop to the transport directly (proxy, mTLS, record-replay), because each port’s is idiomatic and the body shape differs:

Port	Seam	Request body you receive	Response body you return
JavaScript	fetch	init.body — a JSON string	a Response (JSON string)
Python	http_transport (post/open)	a dict	a dict
Go	HTTPClient (http.RoundTripper)	req.Body — bytes	*http.Response (bytes)
Java	httpClient (subclass HttpClient)	HttpRequest — bytes	HttpResponse<String>
C#	HttpHandler (HttpMessageHandler)	HttpRequestMessage — bytes	HttpResponseMessage
Elixir	transport (1-arg fn)	req.body — an un-marshalled map	a map
Clojure	:http-client ((url headers body))	a JSON string	{:status :headers :body}, body a JSON string

Python and Elixir hand you the body already parsed, so an in-process model there pays no serialization at all. The other five hand you bytes, because that is what their HTTP client hands them.

Streaming, honestly

The recipes above implement the non-streaming path, which is what a tool-calling loop uses. If you also want runStream, your seam must return an incremental body — a buffered one produces the same final text and the same delta count as a real stream, so a test asserting on content cannot tell them apart. Python’s seam splits this explicitly (post vs open); the others distinguish it by stream: true in the request body.

What you do not have to change

This is the part worth being explicit about: you swap one option, and nothing else in your setup moves.

The seam is scoped to the LLM path on purpose. Your MCP servers, agent skills, native and HTTP tools, built-ins and outbound A2A agents keep using their own clients and keep working exactly as they did against a hosted model — you do not re-register them, re-wire them, or run them in a special mode:

// The ONLY line that changes when you move from GPT-4o to a model in your process.
const client = createClient({
  baseUrl: "http://local.invalid", style: "openai", model: "local", apiKey: "unused",
  fetch: localFetch(new LocalModel()),   // ← this line, and nothing else
})

// Untouched. Still real MCP servers over stdio and HTTP, still real skills.
const toolkit = await createToolkit({
  mcpConfig: "./mcp.json",
  skillsDir: "./skills",
})

That separation is deliberate, and it is what makes a local model useful rather than merely isolated. A model running in your process still needs to read files, query GitHub, call your REST API and hand work to another agent — so the MCP servers and HTTP tools should open their real connections. A seam that silenced them along with the model would have turned a working agent into a sandbox.

Run against the shared examples/mcp.json fixture, that is exactly what you get — a socket-free model calling a real stdio MCP server it knows nothing about:

mcp status : {"example-remote":"disabled","everything":"connected"}
tools      : 24 | has everything_echo: true | has skill: true
text       : MCP replied: Echo: hello from a local model
toolCalls  : [ [ 'everything_echo', 'Echo: hello from a local model' ] ]

The MCP server was spawned, connected and called normally. The model never opened a connection. Neither half knows about the other, which is the whole design.

So it is not an 'offline mode' switch

One consequence to be precise about: because tool sources keep their own clients, an in-process model makes model calls socket-free, not the whole process. If what you actually want is a fully hermetic test, that is a different job — reach for the toolkit’s own controls (builtins: false, disableTools, an empty mcp.json) and choose which sources exist, rather than expecting the model seam to mute them.

Related: Bring your own HTTP client for the proxy / mTLS / credentials version of the same seam.