Skip to content
Toolnexusv0.28.0

Typed decisions (judge)

Start here: ask & gate

Section titled “Start here: ask & gate”

Most hosts do the same thing with a classifier: build a state, ask a few named questions, read a band, then act or ask a human. That is the whole layer. Two yes/no questions went from ~16 lines of nested question maps to five, and the wire request is byte-identical to the hand-written one.

import { createClassifier, judge, State, ask, gate, decide } from "toolnexus"


const c = createClassifier()
const st = State("You are Donkey Kong. You want to win the game.", { message_received: msg })
const qs = [
  judge.noul("is_appropriate", "Does `message_received` contain insults, profanity or harmful topics?"),
  judge.noul("does_this_help", "Does `message_received` help Donkey Kong win the game?"),
]
const a = await ask(c, st, qs)
a.is_appropriate.band      // "yes" | "no" | "uncertain"
a.does_this_help.value()   // one number; a choice answer's option is .pick()


const out = await decide(c, st, qs, {
  rules: [{ question: "is_appropriate", at_least: 0.7, action: "fail" }],
  default: "",            // "" = nothing fired -> a §10 input Request (ask a human)
  skipUncertain: true,    // an uncertain rule is skipped; a missing answer still escalates
})
out.action; out.escalated; out.request
  • Bands default to 0.30 / 0.70 and are exclusive: exactly 0.70 is uncertain. A choice or score answer carries sure instead (a near-uniform choice is never sure).
  • gate declines to decide. An uncertain, unsure or missing answer returns needs_input with a §10 input Request — route it through waitFor or your own queue. It never authorises.
  • Tape records live decisions by call name and replays them offline, so tests run on real answers. evaluateBatch asks the same questions of many states (state order, 16 in flight, fail-closed on the lowest failing index). Names per port are in each port’s README.

Two rules come from measuring it, not from taste:

  • The role goes in the state. The wire has only model, state and questions, so there is no system prompt to put it in. State(role, data) puts role next to your data.
  • Each question names the state field it judges. “Does the message contain…” is vague; “Does message_received contain…” points at a field.

Live on TypeSafe jev-1.13.0, 2026-09-27, one good message and one insult (is_appropriate asks about inappropriate language, so high means inappropriate):

variant good: inappropriate / helps bad: inappropriate / helps
A state = JSON string with role 0.02 / 0.76 0.57 / 0.02
B state = object with role 0.02 / 0.78 0.55 / 0.02
C state = object, no role 0.06 / 0.64 0.90 / 0.12
D state = plain text message 0.06 / 0.70 0.84 / 0.13
E role inside the one question that needs it 0.06 / 0.62 0.90 / 0.05
F role in state + question names its field 0.02 / 0.79 0.96 / 0.02

A string and an object state behave the same (A ≈ B). With the role in state and a vague question, the insult blurred to 0.55, right in the uncertain band. Naming the field (F) made it the sharpest row on both messages. Putting the role into the question instead (E) is rejected: it costs the question that needs the role. One sample per cell; Jev is near-deterministic, but these are readings, not averages. A full pipeline built on this is the test-scout scenario.

Batteries: judgments already wired into the loop

Section titled “Batteries: judgments already wired into the loop”
battery what it decides hook
ToolGuardClassifier allow / ask / deny a tool call from a 0–3 risk rating; unsure always asks beforeTool (ask = §10 approval suspension)
ToolRelevanceClassifier which tools this turn needs; only a confident “no” drops one beforeLLM (swaps tools)
SkillRelevanceClassifier which skills are relevant none; feed selected to the skill allowlist (never an empty list: empty means all)
ToolResultFilterClassifier which paragraphs of a tool’s output matter afterTool
IsCompleteClassifier does the final answer complete the task none
AgentRouterClassifier which agent handles a task (walks your tree past 255) none
ContentGuardClassifier allow / review / block text on harmful and prompt_injection beforeLLM (block stops the run)
ModelRouterClassifier opt-in per-query model from your described list beforeLLM (overrides model for that turn)

You must say what a classifier error means (onError: "open" | "closed", no default); the routers fall back to your configured model or fallback instead. ToolGuard’s deny is a policy aid, not a security control: keep allowlists in code in front of it.

import { createClassifier, createClient, ToolGuardClassifier, ModelRouterClassifier } from "toolnexus"


const c = createClassifier()
const guard = new ToolGuardClassifier(c, { onError: "closed" })
const router = new ModelRouterClassifier(c, [
  { id: "small-fast", description: "Short factual answers, lookups and simple rewrites; cheapest." },
  { id: "large-reasoning", description: "Multi-step reasoning and code changes across files; most expensive." },
])


const client = createClient({ model: "large-reasoning", /* … */ hooks: {
  beforeTool: guard.asHook(myBeforeTool),   // your hook still runs when the guard allows
  beforeLLM: router.asHook(),               // sure pick routes this turn; otherwise "large-reasoning"
}})

The full per-port spelling of every battery is in the SPEC §8B idiom table.

Under the hood: the raw Classifier

Section titled “Under the hood: the raw Classifier”

ask and gate are a thin layer over the map-in, typed-answers-out API below.

Run it first

Section titled “Run it first”

Every tab below is an excerpt from a real file in the repo, and every one of these commands was run to produce the output at the bottom of this page. Each one picks its backend from the environment: TYPESAFE_API_KEY calls TypeSafe’s own API, OPENROUTER_API_KEY calls the same wire through OpenRouter’s gateway, and with no key at all they replay one recorded decision through the static backend — no network, no credential, no cost. The snippets below show the TypeSafe branch, which is also the library’s default configuration.

language file command
JavaScript js/examples/judge.ts cd js && npm run build && npm run example:judge
Python python/examples/judge.py cd python && uv run python examples/judge.py
Go golang/examples/judge/main.go cd golang && go run ./examples/judge
Java java/…/examples/Judge.java cd java && ./gradlew runJudge
C# csharp/examples/Toolnexus.Examples/Judge.cs cd csharp/examples/Toolnexus.Examples && dotnet run -- judge
Elixir elixir/examples/judge.exs cd elixir && mix run examples/judge.exs
Clojure clojure/examples/src/examples/judge.cljc cd clojure/examples/clj && clojure -M -m examples.judge

The state is one support ticket. The three questions ask three different kinds of thing about it, in one round trip.

The questions

Section titled “The questions”

The map keys (wants_money_back, department, urgency) are yours, and they are addressing, not content — they are never transmitted to the model, so a key may be a tool, skill or agent name verbatim.

import { createClassifier, noul, choice, score } from "toolnexus"


const QUESTIONS = {
  wants_money_back: noul("Is the customer asking for money to be returned?"),
  department: choice("Which desk should own this ticket?", {
    billing: "own it here when the problem is money that moved: a duplicate charge, a wrong invoice, a refund owed",
    shipping: "own it here when the problem is a physical parcel: a late delivery, a package damaged in transit",
    technical: "own it here when the problem is the product itself: a login that fails, a feature that errors",
  }),
  urgency: score("How fast does this ticket need a human?", [
    "the customer is working normally and is waiting on an answer",
    "the customer is inconvenienced and will chase if nobody replies today",
    "the customer is blocked from working right now and every hour costs them",
  ]),
}


const judge = createClassifier({
  baseUrl: "https://api.typesafe.ai/v1", // TypeSafe's own API — also the library default
  model: "jev-latest",
  apiKeyEnv: "TYPESAFE_API_KEY", // the NAME of an env var, never the value
  onMetric: (ev) => ev.event === "classifier.warning" && console.log("warning:", ev.warning),
})


const d = await judge.evaluate(TICKET, QUESTIONS)


const want = d.noul("wants_money_back")
const dept = d.choice("department")
const urg = d.score("urgency")

Reading an answer as the wrong type is an error, not a surprise: the typed accessors fail loudly (an exception, an {:error, _}, an err) rather than handing you a zero.

What it prints

Section titled “What it prints”

One live TypeSafe decision, through the JavaScript port:

backend: systemone via api.typesafe.ai (live)


model answering: jev-1.13.0
wants_money_back: 0.99   (a noul carries NO confidence — the number IS the answer)
department:       billing  p={"technical":0,"billing":1,"shipping":0} confidence=1
urgency:          0.48  of 0..2  p={"0":0.52,"1":0.48,"2":0}
  level 0: the customer is working normally and is waiting on an answer   (a score MAY fall between levels)


calibrated: true
nearUniform(department): false


usage: 516 in / 72 out / cost: not reported by this backend

Which key you set changes one line, and one field

Section titled “Which key you set changes one line, and one field”

TYPESAFE_API_KEY goes to TypeSafe’s own API; OPENROUTER_API_KEY goes to the same wire through OpenRouter’s gateway. They are equivalent in latency — 339 ms / 449 ms (p50 / p95) against 351 ms / 400 ms, warm and interleaved, which is a tie — so pick on dependencies, not speed: a gateway in the path versus a first-party key. The one functional difference is usage.cost, a gateway field that TypeSafe does not return; that is why the line above says cost: not reported by this backend rather than $0, and why a cost-based budget only works through the gateway. The backends page has the full comparison.

With no key at all the first line reads backend: static (recorded — set TYPESAFE_API_KEY or OPENROUTER_API_KEY to go live), the model answering is the recorded typesafe/jev-1.13-20260917, and the usage line carries the gateway’s $0.000021672 — the same decision, replayed.

Three things in that output are worth a second look.

  • urgency is 0.48. Not 0 and not 1: the model is genuinely split between “waiting on an answer” and “will chase today”, and the number says so. p={"0":0.52,"1":0.48} says it again, with the disagreement visible instead of averaged away. It was 0.49 on the recording and 0.48 live — the backend is non-deterministic, which is why only static may be asserted against.
  • calibrated: true — these probabilities came from a calibrated backend, so a threshold tuned here transfers. An llm-style backend reports false, and your thresholds do not carry over.
  • nearUniform(department): false — the model had something to rank on. true would be the symptom of an encoding that told it nothing.

The encoding obligation

Section titled “The encoding obligation”

That is why every option in the example above carries a real sentence, why all three use the same template (“own it here when the problem is X: a, b, c”), and why the host does the counting and hands over the conclusion rather than shipping raw numbers.

The encoding rules has the four rules with the measurement behind each, the degenerate-criteria warning the library emits when you get this wrong, and the precise limits of nearUniform, confidence and calibrated as health signals.

Recipe: the same questions on OpenAI’s Decisions API

Section titled “Recipe: the same questions on OpenAI’s Decisions API”

The key is read from OPENAI_API_KEY at call time and is sent only to api.openai.com over https. Decisions from this backend report calibrated: false, so a threshold tuned on Jev does not carry over — the snippets below check it before comparing a number. A refusal comes back as an error, never as a “no”. These snippets call the live API, so they need the key; the details are on the backends page.

import { createClassifier, noul, choice, score } from "toolnexus"


const judge = createClassifier({ backend: "openai" }) // OPENAI_API_KEY, gpt-6-luna


const d = await judge.evaluate("The package arrived with a broken screen.", {
  damaged: noul("Does the customer report a damaged item?"),
  team: choice("Which team should handle this?", {
    shipping: "lost, late or damaged parcels",
    billing: "refunds and payments",
  }),
  upset: score("How upset is the customer?", ["calm", "annoyed", "furious"]),
})


console.log(d.noul("damaged"), d.choice("team"), d.score("upset"))
if (!d.calibrated) console.log("calibrated: false — re-tune thresholds for this backend")

A choice here needs at least two options — the Decisions wire takes 2–255, and a one-option choice fails before any request is made, as does a raw question type such as box.

Where to go next

Section titled “Where to go next”
page what it answers
Typed decisions what a Classifier is, the three question types, the Decision shape
The encoding rules what to write inside a question, with the measurements
Backends & configuration systemone, llm, custom, static, and every option
Measured on a live backend latency, cost, non-determinism, and the one control that can fail