Classifier.Create

C# · package Toolnexus · SPEC §8B · Classifier.cs

public sealed class Classifier
{
    public Classifier(ClassifierOptions? options = null);
    public static Classifier Create(ClassifierOptions? options = null);

    public Task<Decision> EvaluateAsync(
        object? state,
        IReadOnlyDictionary<string, Question> questions,
        CancellationToken cancellationToken = default);

    public ClassifierStyle Style { get; }
    public string BaseUrl { get; }
    public string Model { get; }
    public string ApiKeyEnv { get; }
    public TimeSpan Timeout { get; }

    public const string DefaultBaseUrl   = "https://api.typesafe.ai/v1";
    public const string DefaultModel     = "jev-latest";
    public const string DefaultApiKeyEnv = "TYPESAFE_API_KEY";
    public static readonly TimeSpan DefaultTimeout = TimeSpan.FromSeconds(10);

    public const int    MaxChoiceOptions     = 255;
    public const int    MinScoreLevels       = 2;
    public const int    MaxScoreLevels       = 10;
    public const double NearUniformTolerance = 0.05;

    public const string MetricEvaluate = "classifier.evaluate";
    public const string MetricWarning  = "classifier.warning";

    public static bool NearUniform(IReadOnlyDictionary<string, double> probabilities);
    public static byte[] CanonicalRequest(string model, IReadOnlyDictionary<string, Question> questions);
    public static string CanonicalRequestString(string model, IReadOnlyDictionary<string, Question> questions);
}

A sibling of the client: pre-declared typed questions in, calibrated answers out — no messages, no tool calling, no loop. ITool is the contract for an action; Classifier is the contract for a judgment. It never enters the client loop and is never selected as a model for RunAsync.

state is whatever the host already has — a string, a dictionary, a list. questions is a map from caller-chosen keys to question definitions; the keys are addressing, not content, and are never transmitted, so a key may be a tool, skill or agent name verbatim. Questions are independent: one answer is never context for another.

A classifier interprets; it never authorises

The output is a reading of intent inside a boundary, never the boundary. Numeric limits, permission checks and allowlists stay in code. Schema validity is not correctness — a decision can be confidently wrong, and “cannot hallucinate” means only that the returned value is inside the declared schema. Nothing here is a security control. See Typed decisions.

When to use it

Reach for a Classifier when a step in your agent produces a judgment that steers something else rather than an action that changes the world: is this shell command risky, which desk owns this ticket, does this turn need the billing skill, how urgent is this. Those questions do not need a model that can call tools and write prose — they need one number you can threshold, and the System One wire returns it in a single POST with no free text.

Use the client instead whenever the step must act.

Why this and not the alternative

Reach for a plain `if` when the rule is a rule

A threshold on a price, a permission check, an allowlist — those are code, and a classifier is a strictly worse if: slower, non-deterministic, and it cannot be audited. A classifier earns its place only where the input is meaning that no regex can reach. And when you do use one, pin Model (e.g. jev-1.13.0) once a threshold is tuned: Decision.Model echoes what actually answered, which may be more specific than what you asked for, and a threshold tuned against one backend does not transfer to another.

The four backends

Set by ClassifierOptions.Style; the full comparison lives on Backends.

ClassifierStyle	What answers	Needs
SystemOne (default)	one POST {BaseUrl}/systemone with the canonical body	a key in ApiKeyEnv
Llm	the three question types rendered as one JSON-schema structured-output call on a §8 client	Client
Custom	your own function — a fine-tuned encoder, a rules engine, a cache	Evaluate
Static	a recorded corpus, keyed by the canonical request and the state	Decisions

Static is what CI runs: no network, no credential. It is not a convenience — the live backend is non-deterministic, so it is the only backend a test may assert a number against. Every example on this page uses it.

Examples

1. The smallest useful call — one question, one recorded decision, no network

using Toolnexus;

var ticket = "My card was charged twice for the annual plan and the second charge has not been refunded.";

var questions = new Dictionary<string, Question>
{
    ["wants_money_back"] = new NoulQuestion { Instructions = "Is the customer asking for money to be returned?" },
};

// `static` replays a recorded body, matched on the canonical request AND the state.
var judge = Classifier.Create(new ClassifierOptions
{
    Style = ClassifierStyle.Static,
    Model = "typesafe/jev-1.13",
    Decisions = new[]
    {
        new RecordedDecision
        {
            State = ticket,
            Questions = questions,
            Response = """
            {"model":"typesafe/jev-1.13-20260917",
             "answers":{"wants_money_back":{"type":"noul","noul":0.99}},
             "usage":{"input_tokens":112,"output_tokens":9}}
            """,
        },
    },
});

var decision = await judge.EvaluateAsync(ticket, questions);

// A noul carries NO confidence — the number IS the answer.
var noul = decision.Noul("wants_money_back").Noul;
if (noul < 0.9) throw new Exception($"noul: {noul}");

// The authority stays in code. The classifier only supplied the reading.
var route = noul > 0.8 ? "billing" : "general";

Console.WriteLine($"ok: model={decision.Model} noul={noul} calibrated={decision.Calibrated} route={route}");

2. The realistic case — a Custom backend, so a test never touches a wire at all

using Toolnexus;

var questions = new Dictionary<string, Question>
{
    ["department"] = new ChoiceQuestion
    {
        Instructions = "Which desk should own this ticket?",
        Criteria = new Dictionary<string, string>
        {
            // Criteria[id] is the ONLY thing that differentiates one option from another to the
            // model. Passing the id itself is schema-valid — and ranks at chance. See /judge/encoding/.
            ["billing"] = "own it here when the problem is money that moved: a duplicate charge, a refund owed",
            ["technical"] = "own it here when the problem is the product itself: a login that fails, a feature that errors",
        },
    },
};

var calls = 0;

var judge = Classifier.Create(new ClassifierOptions
{
    Style = ClassifierStyle.Custom,
    // Every wire option is ignored under `custom`. The host owns the whole evaluation.
    Evaluate = (state, qs, ct) =>
    {
        calls++;
        var billing = state is string s && s.Contains("charged", StringComparison.OrdinalIgnoreCase);
        return Task.FromResult(new Decision
        {
            Model = "rules-engine-v1",
            // A backend that cannot guarantee independent questions reports Calibrated = false,
            // which carries BOTH caveats: the numbers and any threshold tuned on them.
            Calibrated = false,
            Answers = new Dictionary<string, DecisionAnswer>
            {
                ["department"] = new ChoiceAnswer
                {
                    Choice = billing ? "billing" : "technical",
                    Probabilities = billing
                        ? new Dictionary<string, double> { ["billing"] = 0.9, ["technical"] = 0.1 }
                        : new Dictionary<string, double> { ["billing"] = 0.1, ["technical"] = 0.9 },
                    Confidence = 0.9,
                    NearUniform = false,
                },
            },
        });
    },
});

var decision = await judge.EvaluateAsync("I was charged twice", questions);

if (calls != 1) throw new Exception($"calls: {calls}");
if (decision.Choice("department").Choice != "billing") throw new Exception("routed wrong");
if (decision.Calibrated) throw new Exception("a rules engine is not calibrated");

Console.WriteLine($"ok: {decision.Choice("department").Choice} via {decision.Model} (calibrated={decision.Calibrated})");

3. Full surface — every option spelled out, and the degenerate-criteria warning observed

using Toolnexus;

// Degenerate criteria: every value equals its own key. The library DETECTS and REPORTS this,
// once per question key, and sends the request BYTE-UNCHANGED. It never repairs it — repairing
// would invent option descriptions you did not write.
var questions = new Dictionary<string, Question>
{
    ["lane"] = new ChoiceQuestion
    {
        Instructions = "Which lane?",
        Criteria = new Dictionary<string, string> { ["left"] = "left", ["right"] = "right" },
    },
};

var warnings = new List<string>();
var errors = 0;

var judge = Classifier.Create(new ClassifierOptions
{
    Style = ClassifierStyle.Static,              // default: SystemOne
    BaseUrl = "https://api.typesafe.ai/v1",      // the default, spelled out so it is visible
    Model = "typesafe/jev-1.13",                 // default "jev-latest"; PIN it once a threshold is tuned
    ApiKeyEnv = "TYPESAFE_API_KEY",              // the NAME of an env var, never the value
    Timeout = TimeSpan.FromSeconds(10),          // bounds ONE request; a classifier has no loop to bound
    Retries = 2,
    RetryableStatuses = new[] { 418 },           // ADDS to the defaults; it can never remove one
    OnError = info => info.Retryable ? LlmClient.Tier.Retry : LlmClient.Tier.Fail,
    RequestParams = new Dictionary<string, object?> { ["x_tenant"] = "acme" },
    BodyTransform = body => body,                // base body -> RequestParams merge -> BodyTransform -> marshal
    OnMetric = ev =>
    {
        // The warning travels in `Warning`, never in `Error`: a consumer filtering the §8 sink on
        // "has an error" must NOT count one.
        if (ev.Event == Classifier.MetricWarning) warnings.Add(ev.Question ?? "");
        if (!string.IsNullOrEmpty(ev.Error)) errors++;
    },
    Decisions = new[]
    {
        new RecordedDecision
        {
            State = "merging",
            Questions = questions,
            Response = """
            {"model":"typesafe/jev-1.13-20260917",
             "answers":{"lane":{"type":"choice","choice":"left","probabilities":{"left":0.51,"right":0.49},"confidence":0.5}},
             "usage":{"input_tokens":40,"output_tokens":6}}
            """,
        },
    },
});

await judge.EvaluateAsync("merging", questions);
await judge.EvaluateAsync("merging", questions);   // detection is ONCE per question key per classifier

if (warnings.Count != 1 || warnings[0] != "lane") throw new Exception(string.Join(",", warnings));
if (errors != 0) throw new Exception("a warning is not a failure");

// And the symptom the warning predicts: a flat distribution.
var choice = judge.Style == ClassifierStyle.Static ? (await judge.EvaluateAsync("merging", questions)).Choice("lane") : null!;
Console.WriteLine($"ok: warned once on \"{warnings[0]}\", errors={errors}, nearUniform={choice.NearUniform}");

ClassifierOptions

Mirrors LlmClient.Options field-for-field wherever a field makes sense, so a host that has configured one has configured the other.

Option	Type	Default	What it does
Style	ClassifierStyle	SystemOne	Which backend answers: SystemOne, Llm, Custom, Static.
BaseUrl	string?	https://api.typesafe.ai/v1	The System One endpoint’s base. OpenRouter serves the same wire at https://openrouter.ai/api/v1; self-hosted and open-weights implementations speak it too.
Model	string?	"jev-latest"	The model asked for. Pin it once thresholds are tuned; Decision.Model echoes what actually answered.
ApiKeyEnv	string?	"TYPESAFE_API_KEY"	The name of an env var, read at call time and never logged. Deliberately not a value, unlike §8’s ApiKey.
Headers	IDictionary<string, string>?	—	Extra headers; values expand ${ENV_VAR} from the environment at call time and are never logged.
Timeout	TimeSpan?	10 s	Bounds one request, not a run — a classifier has no loop to bound.
HttpClient	HttpClient?	—	(§8 Gap 2) Overrides the transport. Scope is the classifier path only.
Retries	int?	2	Attempts after the first, on a retryable status or a network fault.
RetryBaseMs	int?	500	Base of the retry backoff, in ms: the delay is base * 2^attempt, no jitter, and a Retry-After header still wins. null or <= 0 falls back to the default. The classifier’s half of the §8 Options.RetryBaseMs it mirrors.
RetryableStatuses	IReadOnlyCollection<int>?	—	Adds to the default set; it can never remove one, so a host cannot drop 429 and lose Retry-After handling with it.
OnError	Func<LlmClient.ErrorInfo, LlmClient.Tier>?	—	Reuses the §8 error classifier verbatim, and decides each attempt. There is no Suspend tier here.
RequestParams	IReadOnlyDictionary<string, object?>?	—	Merged into the body the classifier builds; a RequestParams key wins on collision.
BodyTransform	Func<IDictionary<string, object?>, IDictionary<string, object?>?>?	—	Last stop before marshalling: base body → RequestParams merge → BodyTransform → wire.
OnMetric	Action<MetricEvent>?	—	Emits classifier.evaluate (latency, tokens, model, status; Error set only on a failed evaluate) and classifier.warning into the same §8 sink.
Client	LlmClient?	—	Style = Llm only — the §8 client to emulate over.
Evaluate	Func<object?, IReadOnlyDictionary<string, Question>, CancellationToken, Task<Decision>>?	—	Style = Custom only — your own function. Every wire option is ignored.

Port-local extras (not in the cross-language options manifest, C# only):

Option	Type	What it does
HttpHandler	HttpMessageHandler?	A handler to build the classifier’s HttpClient from. Ignored when HttpClient is set.

Decisions (IReadOnlyList<RecordedDecision>?) — the recorded corpus for Style = Static — is not port-local: it is named in §8B and gated at core tier by the cross-language options manifest, since it is the backend CI runs on. Each port spells it idiomatically.

Every option also has a fluent With… setter (new ClassifierOptions().WithStyle(…).WithModel(…)).

Retries

The default retryable set is {408, 429, 500, 502, 503, 504, 529} plus network faults — note the 408, which is the classifier’s one addition over the §8 client’s set, because a classifier is a single bounded request where a request timeout is worth one more attempt. Retry-After is honoured with the same delay-seconds rule as §8. There is no second retry policy in the library.

Secrets

The credential resolves at call time from the named environment variable, and ${ENV_VAR} header references expand at call time. No credential value and no expanded header value appears in any log, metric, error message or returned value: an authentication failure names the status and the endpoint and nothing else.

Constructing no Classifier leaves behaviour byte-identical to a build without §8B, and constructing one alters no request the client loop makes.

See also

• NoulQuestion — The three question types, the criteria each one needs, and the limits enforced client-side before the request.
• Decision — One answer per question under the caller’s own keys, read through typed accessors that fail loudly rather than hand back a zero.
• Typed decisions — Why a judgment is a different contract from an action.
• Encoding a choice — The measured cost of undescribed options.
• Backends — What each of the four actually costs.
• Cookbook: a judge in the loop — The end-to-end recipe.