LlmClient.MetricEvent

Java · package io.github.muthuishere:toolnexus · SPEC §8 · LlmClient.java

public sealed interface MetricEvent permits MetricEvent.Llm, MetricEvent.Tool, MetricEvent.Run {
    String event(); // "llm" | "tool" | "run"

    record Llm(String model, String status, long ms, long promptTokens, long completionTokens) implements MetricEvent {}
    record Tool(String tool, String source, boolean isError, long ms, boolean pending) implements MetricEvent {}
    record Run(String model, int turns, int toolCalls, long totalTokens, long ms, String error) implements MetricEvent {}
}

// LlmClient.Options
public Options onMetric(Consumer<MetricEvent> v)
// LlmClient
public String metrics() // Prometheus text exposition

Semantic, readable observability events — NOT raw counters — pushed to Options.onMetric as the loop runs: one Llm event per model round trip, one Tool event per tool call, one Run event per completed run/ask. The same events also feed a built-in, zero-dependency Prometheus registry, rendered by client.metrics().

When to use it

Forward onMetric to your own logs/statsd/OTel pipeline when you want domain-shaped events (which model, how many tokens, was it an error, was it a §10 suspension) rather than pre-aggregated counters. Call client.metrics() directly when you just want a Prometheus /metrics endpoint and don’t need per-event granularity — no external dependency required.

Why this and not the alternative

metrics() when you only need the scrape endpoint

If the only requirement is “expose /metrics for Prometheus to scrape,” skip onMetric entirely — client.metrics() already accumulates from the same internal events and renders valid text exposition, empty-but-valid before any activity. Reach for onMetric when you need to forward individual events elsewhere, or act on one as it happens (e.g. alert on a Run event with a non-null error).

Examples

1. The smallest useful call — one run, two events

import com.sun.net.httpserver.HttpServer;
import io.github.muthuishere.toolnexus.*;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;

public class Example {
    public static void main(String[] args) throws Exception {
        HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
        server.createContext("/", ex -> {
            byte[] b = ("{\"choices\":[{\"message\":{\"content\":\"ok\"}}],"
                    + "\"usage\":{\"prompt_tokens\":4,\"completion_tokens\":2,\"total_tokens\":6}}")
                    .getBytes(StandardCharsets.UTF_8);
            ex.getResponseHeaders().add("Content-Type", "application/json");
            ex.sendResponseHeaders(200, b.length);
            try (OutputStream os = ex.getResponseBody()) { os.write(b); }
        });
        server.start();
        int port = server.getAddress().getPort();

        List<LlmClient.MetricEvent> events = new ArrayList<>();
        try (Toolkit tk = Toolkit.create(new Toolkit.Options())) {
            LlmClient client = LlmClient.create(new LlmClient.Options()
                    .baseUrl("http://127.0.0.1:" + port)
                    .style("openai")
                    .model("test-model")
                    .apiKey("test-key")
                    .onMetric(events::add));

            client.run("hello", tk);

            long llmEvents = events.stream().filter(e -> e.event().equals("llm")).count();
            long runEvents = events.stream().filter(e -> e.event().equals("run")).count();
            if (llmEvents != 1) throw new AssertionError("expected 1 llm event, got " + llmEvents);
            if (runEvents != 1) throw new AssertionError("expected 1 run event, got " + runEvents);

            LlmClient.MetricEvent.Llm llm = (LlmClient.MetricEvent.Llm) events.stream()
                    .filter(e -> e.event().equals("llm")).findFirst().orElseThrow();
            if (!"ok".equals(llm.status())) throw new AssertionError(llm.status());
            if (!"test-model".equals(llm.model())) throw new AssertionError(llm.model());
            if (llm.promptTokens() != 4 || llm.completionTokens() != 2) throw new AssertionError(llm);

            System.out.println("ok: " + events.size() + " metric event(s)");
        } finally {
            server.stop(0);
        }
    }
}

2. The realistic case — a Tool event, plus the Prometheus text

import com.sun.net.httpserver.HttpServer;
import io.github.muthuishere.toolnexus.*;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;

public class Example {
    public static void main(String[] args) throws Exception {
        AtomicInteger hits = new AtomicInteger(0);
        HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
        server.createContext("/", ex -> {
            int n = hits.incrementAndGet();
            String body = n == 1
                    ? "{\"choices\":[{\"message\":{\"content\":null,\"tool_calls\":[{\"id\":\"c1\",\"type\":\"function\","
                      + "\"function\":{\"name\":\"ping\",\"arguments\":\"{}\"}}]},\"finish_reason\":\"tool_calls\"}]}"
                    : "{\"choices\":[{\"message\":{\"content\":\"pong received\"}}]}";
            byte[] b = body.getBytes(StandardCharsets.UTF_8);
            ex.getResponseHeaders().add("Content-Type", "application/json");
            ex.sendResponseHeaders(200, b.length);
            try (OutputStream os = ex.getResponseBody()) { os.write(b); }
        });
        server.start();
        int port = server.getAddress().getPort();

        Tool ping = NativeTool.of("ping", "Health check",
                Map.of("type", "object", "properties", Map.of()),
                (Map<String, Object> a) -> "pong");

        List<LlmClient.MetricEvent> events = new ArrayList<>();
        try (Toolkit tk = Toolkit.create(new Toolkit.Options().extraTools(ping))) {
            LlmClient client = LlmClient.create(new LlmClient.Options()
                    .baseUrl("http://127.0.0.1:" + port)
                    .style("openai")
                    .model("test-model")
                    .apiKey("test-key")
                    .onMetric(events::add));

            client.run("ping the service", tk);

            LlmClient.MetricEvent.Tool tool = (LlmClient.MetricEvent.Tool) events.stream()
                    .filter(e -> e.event().equals("tool")).findFirst().orElseThrow();
            if (!"ping".equals(tool.tool())) throw new AssertionError(tool.tool());
            if (!"native".equals(tool.source())) throw new AssertionError(tool.source());
            if (tool.isError() || tool.pending()) throw new AssertionError(tool);

            // The same events also feed the built-in, zero-dependency Prometheus registry.
            String prom = client.metrics();
            if (!prom.contains("toolnexus_tool_calls_total{")) throw new AssertionError(prom);
            if (!prom.contains("tool=\"ping\"")) throw new AssertionError(prom);

            System.out.println("ok: " + tool.tool() + " -> tool metric + prometheus text");
        } finally {
            server.stop(0);
        }
    }
}

3. The full surface — cumulative counts across runs, and a Run error

import com.sun.net.httpserver.HttpServer;
import io.github.muthuishere.toolnexus.*;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;

public class Example {
    public static void main(String[] args) throws Exception {
        AtomicInteger hits = new AtomicInteger(0);
        HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
        server.createContext("/", ex -> {
            int n = hits.incrementAndGet();
            if (n == 3) { // the third call always fails; retries(0) below means no retry either
                byte[] b = "server exploded".getBytes(StandardCharsets.UTF_8);
                ex.sendResponseHeaders(500, b.length);
                try (OutputStream os = ex.getResponseBody()) { os.write(b); }
                return;
            }
            byte[] b = "{\"choices\":[{\"message\":{\"content\":\"ok\"}}]}".getBytes(StandardCharsets.UTF_8);
            ex.getResponseHeaders().add("Content-Type", "application/json");
            ex.sendResponseHeaders(200, b.length);
            try (OutputStream os = ex.getResponseBody()) { os.write(b); }
        });
        server.start();
        int port = server.getAddress().getPort();

        List<LlmClient.MetricEvent> events = new ArrayList<>();
        try (Toolkit tk = Toolkit.create(new Toolkit.Options())) {
            LlmClient client = LlmClient.create(new LlmClient.Options()
                    .baseUrl("http://127.0.0.1:" + port)
                    .style("openai")
                    .model("test-model")
                    .apiKey("test-key")
                    .retries(0)
                    .onMetric(events::add));

            client.run("first", tk);
            client.run("second", tk);
            try {
                client.run("third", tk); // hits the 500, no retries -> throws
                throw new AssertionError("expected the third call to throw");
            } catch (RuntimeException expected) {
                // fine — the interesting part is what got recorded
            }

            long runErrors = events.stream()
                    .filter(e -> e.event().equals("run") && ((LlmClient.MetricEvent.Run) e).error() != null)
                    .count();
            if (runErrors != 1) throw new AssertionError("expected exactly 1 run error, got " + runErrors);

            String prom = client.metrics();
            if (!prom.contains("toolnexus_llm_requests_total{model=\"test-model\",status=\"ok\"} 2")) {
                throw new AssertionError(prom);
            }
            if (!prom.contains("# TYPE toolnexus_run_errors_total counter")) throw new AssertionError(prom);

            System.out.println("ok: " + runErrors + " run error(s) recorded, cumulative metrics rendered");
        } finally {
            server.stop(0);
        }
    }
}

Fields

Member	Type	What it is
MetricEvent.event()	String	"llm" | "tool" | "run" — the discriminator.
Llm.model/status/ms/promptTokens/completionTokens	—	One LLM round trip; status is "ok" or "error".
Tool.tool/source/isError/ms/pending	—	One tool call; a §10 suspension sets pending=true and is NEVER counted as isError.
Run.model/turns/toolCalls/totalTokens/ms/error	—	One completed run/ask; error is null on success.
Options.onMetric(v)	—	Receives every event as the loop runs. null ⇒ no sink (zero cost).
client.metrics()	String	Prometheus text exposition, cumulative, byte-identical across ports. Valid (HELP/TYPE only) even before any activity.

See also

• LlmClient.create — The unified client: system prompt, skills injection, parallel and chained tool calls, retries, memory.
• LlmClient.run — Send a prompt, let the loop call tools until the model stops, get a RunResult.
• LlmClient.stream — The streaming loop: text deltas, tool-call events, and suspension events as they happen.
• LlmClient.Hooks — Intercept before/after model calls and tool calls: audit, redact, veto, or rewrite.