Observability — metrics & Prometheus

You can’t run an agent in production you can’t see. toolnexus instruments the loop once and gives you two outputs from it — both opt-in, both zero-cost when unused.

The onMetric event feed

Pass a sink and it receives a readable, semantic record at each significant point: one llm event per model call, one tool event per tool call, and one terminal run event per run/ask. Forward it to statsd, structured logs, or OpenTelemetry — the library holds no opinion.

JavaScript

const client = createClient({
  baseUrl, style: "openai", model,
  onMetric: (ev) => {
    if (ev.event === "run") console.log(`run: ${ev.turns} turns, ${ev.totalTokens} tokens, ${ev.ms}ms`)
  },
})

Python

def on_metric(ev):
    if ev["event"] == "run":
        print(f"run: {ev['turns']} turns, {ev['total_tokens']} tokens, {ev['ms']}ms")

client = create_client(base_url=base, style="openai", model=model, on_metric=on_metric)

Go

c := toolnexus.CreateClient(toolnexus.ClientOptions{
    BaseURL: baseURL, Style: toolnexus.StyleOpenAI, Model: model,
    OnMetric: func(ev toolnexus.MetricEvent) {
        if ev.Event == "run" {
            log.Printf("run: %d turns, %d tokens, %dms", ev.Turns, ev.TotalTokens, ev.Ms)
        }
    },
})

Java

LlmClient client = LlmClient.create(new LlmClient.Options()
        .baseUrl(base).style("openai").model(model)
        .onMetric(ev -> {
            if ("run".equals(ev.event())) System.out.println("run: " + ev.turns() + " turns");
        }));

C#

var client = LlmClient.Create(new LlmClient.Options
{
    BaseUrl = baseUrl, Style = "openai", Model = model,
    OnMetric = ev =>
    {
        if (ev.Event == "run") Console.WriteLine($"run: {ev.Turns} turns");
    },
});

Elixir

on_metric = fn ev ->
  if ev.event == "run" do
    IO.puts("run: #{ev.turns} turns, #{ev.total_tokens} tokens, #{ev.ms}ms")
  end
end

client = Toolnexus.Client.create(base_url: base, style: "openai", model: model,
                                 on_metric: on_metric)

Clojure

(require '[toolnexus.client :as client])

(def llm
  (client/create-client
    {:base-url base-url :style "openai" :model model
     :on-metric (fn [m]
                  (when (= "run" (:event m))
                    (println (str "run: " (:turns m) " turns, "
                                  (:total_tokens m) " tokens, " (:ms m) "ms"))))}))

The three event shapes: { event: "llm", model, status, ms, promptTokens, completionTokens }, { event: "tool", tool, source, isError, ms, pending? }, and { event: "run", model, turns, toolCalls, totalTokens, ms, error? } — names idiomatic per port (snake_case in Python and Elixir; Clojure emits plain maps keyed :event / :prompt_tokens / :total_tokens).

On an agent run

The examples above build a client directly. When you run sub-agents the runtime builds that client for you, so onMetric is set on the runtime — covering every agent — or on an individual agent, which replaces the runtime-wide sink for that agent alone. Setting it per agent is how you attribute events to the agent that produced them.

The events are identical on both paths: same names, same fields. The runtime forwards the sink verbatim and adds, renames, drops, buffers, reorders and aggregates nothing.

// every agent in this runtime reports to one sink…
const rt = new agents.AgentRuntime({ registry, onMetric: ev => track(ev) })

// …or give one agent its own, and it wins for that agent only
const researcher = agents.agent("researcher", {
  does: "digs through long documents",
  onMetric: ev => track({ ...ev, agent: "researcher" }),
})

hooks resolves the same way and independently — an agent may override one and inherit the other. See Sub-agents → hooks & metrics.

Built-in Prometheus — client.metrics()

The same events feed a tiny in-memory registry that renders the Prometheus text exposition format — no third-party dependency. Mount it at GET /metrics:

// in your HTTP handler:
res.setHeader("Content-Type", "text/plain")
res.end(client.metrics())        // client.Metrics() in Go / Java / C#

It exposes toolnexus_llm_requests_total, toolnexus_llm_tokens_total, toolnexus_tool_calls_total, plus the toolnexus_llm_request_duration_seconds and toolnexus_tool_duration_seconds histograms.

Note

The rendered Prometheus text is byte-identical across all six published ports — same metric names, labels, and bucket layout — so a dashboard built against one language works against any of them. OTLP push is a planned future companion.

The Clojure port ships the :on-metric event feed but no metrics() renderer: there is no in-memory registry and no Prometheus text. Aggregate the events into whatever registry your service already exposes — that seam is where the vocabulary belongs. It is a declared gap, not a rename.

A suspension is never counted as a tool error: its tool event carries isError: false and a pending: true marker, so error-rate metrics and circuit-breakers don’t trip on a human wait (see Suspension).