ClientOptions.tool_tiers

Rust · crate toolnexus · SPEC §8

pub struct ToolTierOptions {
    pub core: Vec<String>,                 // always sent with full schemas
    pub deferred: Vec<String>,             // named only; every other tool is "listed"
    pub budget_tokens: usize,              // cap on what is sent up front; 0 ⇒ context_window / 20
    pub context_window: usize,
    pub usage_count: Option<UsageCountFn>, // Arc<dyn Fn(&str) -> usize>, read when the plan is built
    pub count_tokens: Option<CountTokensFn>, // Arc<dyn Fn(&Value) -> usize>; default ceil(bytes/4)
}

// ClientOptions.tool_tiers: Option<ToolTierOptions> — None ⇒ off, every schema sent as before.

impl Client {
    pub fn tool_tier_plan(&self, tk: &Toolkit) -> Option<HashMap<String, ToolTier>>; // Core | Listed | Deferred
    pub fn reset_tool_tiers(&self, tk: &Toolkit);
}

With tool_tiers set, the tools array is the core tools sorted by name, then tool_search and call_tool. Every other tool is listed (- name — first line inside tool_search’s description) or deferred (name only there).

• tool_search {query, limit?} — select:a,b for exact names, else keyword scoring (default 5 results). It returns the full definitions as JSON in the tool result, so the tools array — and the cached prefix — never changes mid-session. It is engine-internal (no before_tool).
• A loaded tool is called by name, or as call_tool {name, arguments}. The loop unwraps call_tool before hooks, so before_tool, hints and permission gates see the real tool; stream tool_call / tool_result events carry the real name with via: "call_tool".
• The plan is built on a toolkit’s first request and then frozen. reset_tool_tiers(&tk) rebuilds it — call it at session start.
• Budget: while core schemas plus the two meta tools exceed it, the least-used core tool moves to listed (ties: larger schema, then name), then the least-used listed tool to deferred.

When to use it

A host with many tools — dozens of MCP tools, a full builtin pack, plugins — where sending every schema on every request costs tokens and dilutes the model’s choice, or where tools are added mid-session and would otherwise break the prompt cache.

Why this and not the alternative

A handful of tools? Use disable_tools or nothing at all

Tiers add a search round trip whenever the model needs a tool outside the core set. With a small, fixed tool set, sending every schema is simpler. To drop tools you never want offered, filter them with ToolkitOptions.disable_tools instead.

Examples

1. The smallest useful call — only core schemas go out

use std::sync::{Arc, Mutex};

use serde_json::Value;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
use toolnexus::client::{ClientOptions, Style, ToolTierOptions, create_client};
use toolnexus::{BuiltinsConfig, Tool, ToolkitOptions, create_toolkit, define_tool};

/// A loopback HTTP stub: `reply(n, raw_request)` answers the n-th request (1-based)
/// with `(status, extra header lines, body)`.
async fn stub<F>(reply: F) -> String
where
    F: Fn(usize, &str) -> (u16, String, String) + Send + Sync + 'static,
{
    let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
    let url = format!("http://{}", listener.local_addr().unwrap());
    let reply = Arc::new(reply);
    tokio::spawn(async move {
        let mut n = 0;
        while let Ok((mut sock, _)) = listener.accept().await {
            n += 1;
            let (reply, i) = (reply.clone(), n);
            tokio::spawn(async move {
                let (mut buf, mut chunk) = (Vec::new(), [0u8; 4096]);
                loop {
                    let k = sock.read(&mut chunk).await.unwrap_or(0);
                    if k == 0 {
                        break;
                    }
                    buf.extend_from_slice(&chunk[..k]);
                    let s = String::from_utf8_lossy(&buf).to_string();
                    if let Some(h) = s.find("\r\n\r\n") {
                        let len = s[..h]
                            .lines()
                            .find_map(|l| l.to_ascii_lowercase().strip_prefix("content-length:").map(|v| v.trim().parse().unwrap_or(0)))
                            .unwrap_or(0);
                        if buf.len() >= h + 4 + len {
                            break;
                        }
                    }
                }
                let (status, extra, body) = reply(i, &String::from_utf8_lossy(&buf));
                let head = format!(
                    "HTTP/1.1 {status} STUB\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n{extra}\r\n",
                    body.len()
                );
                let _ = sock.write_all(format!("{head}{body}").as_bytes()).await;
            });
        }
    });
    url
}

fn tool(name: &'static str, desc: &str) -> Tool {
    define_tool(name, desc, None, move |_: Value, _| async move { Ok::<_, String>(format!("{name} ran")) })
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let tk = create_toolkit(ToolkitOptions {
        builtins: Some(BuiltinsConfig::off()),
        extra_tools: vec![tool("read_file", "Read a file"), tool("weather", "Current weather for a city"), tool("stocks", "Stock quote")],
        ..Default::default()
    })
    .await?;

    let body = Arc::new(Mutex::new(Value::Null));
    let seen = body.clone();
    let url = stub(move |_, raw| {
        *seen.lock().unwrap() = serde_json::from_str(raw.split("\r\n\r\n").nth(1).unwrap_or("")).unwrap_or_default();
        (200, String::new(), r#"{"choices":[{"message":{"role":"assistant","content":"hi"}}]}"#.into())
    })
    .await;

    let client = create_client(ClientOptions {
        base_url: url, style: Style::OpenAI, model: "m".into(), api_key: "YOUR_KEY_HERE".into(),
        tool_tiers: Some(ToolTierOptions { core: vec!["read_file".into()], deferred: vec!["stocks".into()], ..Default::default() }),
        ..Default::default()
    });
    client.run("hello", &tk).await?;

    let body = body.lock().unwrap().clone();
    let tools = body["tools"].as_array().unwrap();
    let sent: Vec<&str> = tools.iter().map(|t| t["function"]["name"].as_str().unwrap()).collect();
    assert_eq!(sent, ["read_file", "tool_search", "call_tool"]);
    let search = tools[1]["function"]["description"].as_str().unwrap();
    assert!(search.contains("weather") && search.contains("stocks"), "{search}");
    let plan = client.tool_tier_plan(&tk).unwrap();
    println!("ok: {sent:?} weather = {:?} stocks = {:?}", plan["weather"], plan["stocks"]);
    Ok(())
}

2. The realistic case — search, then call_tool, under a permission hook

The hook sees weather, never call_tool.

use std::sync::{Arc, Mutex};

use serde_json::{Value, json};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
use toolnexus::client::{BeforeToolEvent, ClientOptions, Hooks, Style, ToolTierOptions, create_client};
use toolnexus::{BuiltinsConfig, ToolkitOptions, create_toolkit, define_tool};

/// A loopback HTTP stub: `reply(n, raw_request)` answers the n-th request (1-based)
/// with `(status, extra header lines, body)`.
async fn stub<F>(reply: F) -> String
where
    F: Fn(usize, &str) -> (u16, String, String) + Send + Sync + 'static,
{
    let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
    let url = format!("http://{}", listener.local_addr().unwrap());
    let reply = Arc::new(reply);
    tokio::spawn(async move {
        let mut n = 0;
        while let Ok((mut sock, _)) = listener.accept().await {
            n += 1;
            let (reply, i) = (reply.clone(), n);
            tokio::spawn(async move {
                let (mut buf, mut chunk) = (Vec::new(), [0u8; 4096]);
                loop {
                    let k = sock.read(&mut chunk).await.unwrap_or(0);
                    if k == 0 {
                        break;
                    }
                    buf.extend_from_slice(&chunk[..k]);
                    let s = String::from_utf8_lossy(&buf).to_string();
                    if let Some(h) = s.find("\r\n\r\n") {
                        let len = s[..h]
                            .lines()
                            .find_map(|l| l.to_ascii_lowercase().strip_prefix("content-length:").map(|v| v.trim().parse().unwrap_or(0)))
                            .unwrap_or(0);
                        if buf.len() >= h + 4 + len {
                            break;
                        }
                    }
                }
                let (status, extra, body) = reply(i, &String::from_utf8_lossy(&buf));
                let head = format!(
                    "HTTP/1.1 {status} STUB\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n{extra}\r\n",
                    body.len()
                );
                let _ = sock.write_all(format!("{head}{body}").as_bytes()).await;
            });
        }
    });
    url
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let weather = define_tool(
        "weather",
        "Current weather for a city",
        Some(json!({"type": "object", "properties": {"city": {"type": "string"}}})),
        |a: Value, _| async move { Ok::<_, String>(format!("sunny in {}", a["city"].as_str().unwrap_or("?"))) },
    );
    let tk = create_toolkit(ToolkitOptions {
        builtins: Some(BuiltinsConfig::off()),
        extra_tools: vec![weather],
        ..Default::default()
    })
    .await?;

    let replies = [
        r#"{"choices":[{"message":{"role":"assistant","content":null,"tool_calls":[{"id":"c1","type":"function","function":{"name":"tool_search","arguments":"{\"query\":\"select:weather\"}"}}]}}]}"#,
        r#"{"choices":[{"message":{"role":"assistant","content":null,"tool_calls":[{"id":"c2","type":"function","function":{"name":"call_tool","arguments":"{\"name\":\"weather\",\"arguments\":{\"city\":\"Paris\"}}"}}]}}]}"#,
        r#"{"choices":[{"message":{"role":"assistant","content":"It is sunny in Paris."}}]}"#,
    ];
    let url = stub(move |n, _| (200, String::new(), replies[n - 1].to_owned())).await;

    let gated = Arc::new(Mutex::new(Vec::<String>::new()));
    let gate = gated.clone();
    let client = create_client(ClientOptions {
        base_url: url, style: Style::OpenAI, model: "m".into(), api_key: "YOUR_KEY_HERE".into(),
        tool_tiers: Some(ToolTierOptions::default()),
        hooks: Hooks {
            before_tool: Some(Arc::new(move |ev: BeforeToolEvent| {
                gate.lock().unwrap().push(ev.name);
                Box::pin(async { Ok(None) })
            })),
            ..Default::default()
        },
        ..Default::default()
    });
    let res = client.run("weather in Paris?", &tk).await?;
    assert_eq!(*gated.lock().unwrap(), ["weather"], "before_tool sees only the real tool");
    assert_eq!(res.tool_calls.last().unwrap().output, "sunny in Paris");
    println!("ok: {}", res.text);
    Ok(())
}

3. The full surface — a token budget, usage stats, and reset_tool_tiers

The plan is frozen once built; changing the usage stat only takes effect after reset_tool_tiers. No request is made: tool_tier_plan builds the plan on its own.

use std::collections::HashMap;
use std::sync::{Arc, Mutex};

use serde_json::Value;
use toolnexus::client::{ClientOptions, Style, ToolTier, ToolTierOptions, create_client};
use toolnexus::{BuiltinsConfig, ToolkitOptions, create_toolkit, define_tool};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let tools = ["alpha", "beta", "gamma"]
        .map(|n| define_tool(n, format!("Tool {n} {}", "x".repeat(200)), None, move |_: Value, _| async move { Ok::<_, String>(n.to_owned()) }))
        .to_vec();
    let tk = create_toolkit(ToolkitOptions { builtins: Some(BuiltinsConfig::off()), extra_tools: tools, ..Default::default() }).await?;

    let usage = Arc::new(Mutex::new(HashMap::from([("alpha", 10), ("beta", 1), ("gamma", 5)])));
    let stats = usage.clone();
    let client = create_client(ClientOptions {
        base_url: "http://127.0.0.1:1".into(), style: Style::OpenAI, model: "m".into(), api_key: "YOUR_KEY_HERE".into(),
        tool_tiers: Some(ToolTierOptions {
            core: vec!["alpha".into(), "beta".into(), "gamma".into()],
            budget_tokens: 300,                         // the two meta tools plus exactly one core schema
            count_tokens: Some(Arc::new(|_: &Value| 100)), // every definition "costs" 100
            usage_count: Some(Arc::new(move |name: &str| *stats.lock().unwrap().get(name).unwrap_or(&0))),
            ..Default::default()
        }),
        ..Default::default()
    });

    let first = client.tool_tier_plan(&tk).unwrap();
    assert!(first["alpha"] == ToolTier::Core && first["beta"] != ToolTier::Core, "{first:?}");

    usage.lock().unwrap().insert("beta", 100); // new stats are ignored while the plan is frozen
    assert_eq!(client.tool_tier_plan(&tk).unwrap()["beta"], first["beta"]);
    client.reset_tool_tiers(&tk); // e.g. at a new session start
    let after = client.tool_tier_plan(&tk).unwrap();
    assert_eq!(after["beta"], ToolTier::Core, "{after:?}");
    println!("ok: beta {:?} -> {:?}", first["beta"], after["beta"]);
    Ok(())
}

See also

• prompt_cache — tiers keep the tools array stable, so the cache keeps hitting.
• Hooks — before_tool sees the real tool behind call_tool.
• Client.stream — tool_call events carry via: "call_tool".
• create_client — The unified client: system prompt, skills injection, parallel and chained tool calls, retries, memory.