Skip to content
toolnexus

The verify-before-commit gate

An agent writes a database migration. It looks right. It reads well. It is wrong in a way that only shows up when it runs — a missing NOT NULL default that locks the table on deploy. The same agent that produced it is the worst possible reviewer of it: it already believes it is correct, because it just argued itself into writing it.

The fix is the oldest one in engineering, moved inside the agent: nothing leaves until an independent check passes. A worker produces an artifact; a separate adversarial verifier sub-agent tries to break it; only a pass lets the result out. On a fail, the verifier’s critique is fed back and the worker retries — a bounded number of times. When the retries run out, the agent returns its best effort loudly flagged as unverified, never a silent pass.

When to reach for this

Section titled “When to reach for this”
Reach for the gate when… Skip it when…
A step’s wrongness is expensive and checkable — a diff that must compile, a refund that must be policy-legal, a research claim that must be sourced The work is trivial or cheap to reverse — the check costs more than just fixing a mistake later
There is a cheaper independent check than redoing the work — running the tests, re-deriving the number, following the citation The only way to verify is to do the whole task again — you have not saved anything
The artifact leaves the agent — it commits, it ships, it answers a customer The output is a draft a human reviews anyway; you already have a verifier (you)
The failure is plausible-but-wrong — it passes a glance and fails on contact Failures are obvious and loud on their own (a crash, an empty result)

Why toolnexus makes the verifier independent for free

Section titled “Why toolnexus makes the verifier independent for free”

Here is the property this whole page rests on, and the reason it belongs to toolnexus rather than to a prompt-engineering blog post.

When the worker delegates to the verifier through task (or when you call verifier.run(...) yourself), the verifier runs on a fresh transcript and returns its final text only. The worker’s reasoning — the chain of thought that talked it into believing the artifact is correct — is never sent to the verifier. The verifier sees the artifact, not the argument for it.

That is a structural guarantee, not a prompt instruction. You cannot forget to add it, the model cannot leak past it, and it does not depend on the verifier being told to ignore anything. An independent reviewer that never saw the persuasive case for the work cannot be talked into agreeing by it.

This is not the self-improving loop — it is its mirror image

Section titled “This is not the self-improving loop — it is its mirror image”

There is a sibling page about a scoped reviewer sub-agent, and it is easy to think this page repeats it. It does not. They use the same primitive — an isolated child reading a finished result — placed at opposite points in time.

This page — the gate Self-improving agent
When it runs Before the result leaves — synchronously After the result has already left — asynchronously
Blocking? Blocking. The answer is withheld until it passes Fire-and-forget. The user already has the answer
What it changes This result — retries or flags it The next run — writes a lesson to MEMORY.md
Failure mode it fights Shipping a wrong artifact now Repeating a known mistake tomorrow
Cost model On the critical path — latency you pay every time Off the critical path — a background tax

Same isolated-child mechanic, opposite placement: verify-before-commit here, learn-after-commit there. A serious agent often has both — the gate stops today’s bad diff, and the reviewer makes sure it does not write that diff again next week.

Why the isolated verifier beats a same-context self-check

Section titled “Why the isolated verifier beats a same-context self-check”

The tempting shortcut is to append “now double-check your work” to the same conversation. It fails on the same axes the self-improving page lays out in full — in short:

Self-check (same context) Isolated verifier sub-agent
Bias Grades its own reasoning while committed to it — rationalizes Sees the artifact, never the reasoning — nothing to defend
Cost Re-sends the whole transcript at the worker’s model price Reads a short artifact, on a cheap model, under a hard budget
Contamination The self-review stays in the transcript and steers later turns The child’s tokens never flow back up — one result crosses the boundary
Blast radius Inherits every tool the worker had, including the destructive ones uses scopes it to inspection tools — it cannot write or commit

The point specific to a gate: the whole value is the verifier’s independence, and same-context self-check has none. The isolation above is not an optimization here — it is the feature.

The verifier’s soul — try to refute, default to FAIL

Section titled “The verifier’s soul — try to refute, default to FAIL”

A verifier is defined by its prose. The one instruction that matters more than any other: when uncertain, FAIL. A pass must be a positive claim — “I ran the checks that would expose this and could not break it” — not the absence of an objection.

verifier/SOUL.md — the adversarial contract
You are an adversarial verifier. Your job is to REFUTE the artifact, not to approve it.


You are given a TASK and an ARTIFACT produced by another agent. You did NOT see how it
was produced, and you must not assume it is correct.


Work to find a reason it FAILS:
- Check every clause of the task. Partial satisfaction is a FAIL.
- Run the checks that would expose it: compile it, run the tests, re-derive the number,
  follow the cited source. A claim you did not check is a claim you cannot pass.
- Hunt the plausible-but-wrong: the off-by-one, the unhandled case, the citation that
  does not actually say what it is quoted as saying.


Reply on the FIRST line with exactly `PASS` or `FAIL`. If you FAIL, the rest of the reply
is the critique: specific, actionable, ordered by severity — what is wrong and what would
make it pass.


When you are uncertain, FAIL. A verifier that waves through work it could not confirm is
worse than none: it manufactures false confidence. Passing means you TRIED to break it and
could not.

The second half of the verifier is its tool scope, and this is a security boundary, not a convenience. A verifier’s job is to inspect: read files, run tests, follow a link. It must never be able to change the artifact it is judging or commit anything. uses gives it a view with the mutating tools dropped — it is not asked to behave, it is structurally incapable of writing.

import { createToolkit, agents } from "toolnexus"


// Inspection-only: it can read and run the tests, but never write, edit, patch, or commit.
const inspect = await createToolkit({
  builtins: { tools: { write: false, edit: false, apply_patch: false } },
})


const verifier = agents.agent("verifier", {
  does: "Adversarially verifies an artifact against a task. Replies PASS or FAIL + critique. Read/test-only.",
  soulFile: "./verifier/SOUL.md",
  uses: { tools: inspect.tools() },        // read, grep, glob, bash(tests) — no write/edit/patch
  model: "openai/gpt-4o-mini",             // cheap: verifying is not a frontier-model task
  budget: { maxTurns: 10, maxTokens: 40_000 },
})

The gate loop — run, verify, retry, or flag

Section titled “The gate loop — run, verify, retry, or flag”

This is the whole pattern, and it is deliberately shown in full: a for loop in your code, not a library call. Run the worker, hand the artifact only to the verifier, parse its first line. On PASS, return. On FAIL, feed the critique back into the worker’s next attempt. Cap the retries; on exhaustion, return the best artifact with the unresolved critique attached and a loud unverified marker.

const MAX_RETRIES = 2                       // total attempts = MAX_RETRIES + 1


interface Gated { text: string; verified: boolean; critique?: string }


async function verifiedRun(task: string): Promise<Gated> {
  let critique: string | undefined
  let artifact = ""


  for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
    // Feed the previous critique back in — this is the only way the worker improves.
    const prompt = critique
      ? `${task}\n\nA reviewer REJECTED your last attempt. Fix EXACTLY this and return the corrected artifact:\n${critique}`
      : task
    artifact = (await worker.run(prompt, { llm })).text


    // The verifier sees the ARTIFACT — never the worker's reasoning (that is the isolation).
    const verdict = await verifier.run(
      `Verify this artifact against the task.\n\nTASK:\n${task}\n\nARTIFACT:\n${artifact}`,
      { llm },
    )
    if (/^\s*PASS\b/i.test(verdict.text)) return { text: artifact, verified: true }
    critique = verdict.text                  // carry the critique into the next attempt
  }


  // Retries exhausted. Return the best effort — LOUDLY marked, never a silent pass.
  return {
    text: `⚠️ UNVERIFIED — failed ${MAX_RETRIES + 1} checks.\n\n${artifact}\n\nUnresolved critique:\n${critique}`,
    verified: false,
    critique,
  }
}

Escalation — perspective-diverse verification

Section titled “Escalation — perspective-diverse verification”

For a high-stakes artifact, one verifier is a single point of failure: it has one lens, and what it does not think to check, it does not catch. Run N independent verifiers, each with a distinct lens — does it satisfy the spec?, is it safe?, does it actually reproduce? — and accept only on a majority. Because parallel work is just multiple task calls (or, in userland, concurrent runs), the fan-out is free — the concurrency primitive is the one from the research orchestrator.

// Three lenses, three fresh transcripts, run concurrently. Each is the verifier agent above
// with a different SOUL.md (correctness / security / reproduces).
const lenses = [correctnessVerifier, securityVerifier, reproVerifier]


const verdicts = await Promise.all(
  lenses.map((v) =>
    v.run(`Verify this artifact against the task.\n\nTASK:\n${task}\n\nARTIFACT:\n${artifact}`, { llm })),
)
const passes = verdicts.filter((r) => /^\s*PASS\b/i.test(r.text)).length
const accepted = passes > lenses.length / 2           // strict majority
const critique = verdicts.filter((r) => !/^\s*PASS\b/i.test(r.text)).map((r) => r.text).join("\n---\n")

This is an escalation, not the default: it costs N extra passes. Reach for it when a single wrong artifact is genuinely costly (a production migration, a financial action), and keep the single verifier everywhere else.

Budget the verifier — it should be a rounding error on the worker

Section titled “Budget the verifier — it should be a rounding error on the worker”

The gate adds at least one LLM pass to every run, so the verifier must be cheap and bounded or the gate becomes the dominant cost. Three levers, same as any scoped sub-agent:

Lever Why
A cheaper model Checking an artifact against a task is not a frontier-model job. Run the worker on the strong model and the verifier on a small one.
A small maxTurns / maxTokens A verifier that needs sixty turns is not verifying, it is re-doing the work. Bound it low; a budget stop returns status: "incomplete" — loud, so you see a runaway.
Hand it the artifact, not the world The verifier reads the artifact and runs the checks. It does not need the worker’s transcript (it never gets it) or the whole repo in context.

A verifier whose cost is more than a small fraction of the worker’s is misconfigured. And because every agent’s tokens roll up into the parent’s totalTokens, you can measure the gate’s true overhead directly: run a task with the gate and without, and compare the root totalTokens.

The full assembly

Section titled “The full assembly”

A worker that edits a repository, an adversarial verifier that must compile it and pass the tests before the change is accepted, and the bounded gate loop around both. The verifiedRun helper is the one from above.

import { createToolkit, agents } from "toolnexus"


const llm = { baseUrl: "https://openrouter.ai/api/v1", style: "openai" as const }


// --- tool views: worker can change the tree; verifier can only inspect + test ---
const full = await createToolkit()                                   // read/write/edit/bash/...
const inspect = await createToolkit({
  builtins: { tools: { write: false, edit: false, apply_patch: false } },
})


// --- the worker: strong model, full hands ---
const worker = agents.agent("worker", {
  does: "Implements a change in this repository and returns the diff.",
  soulFile: "./AGENTS.md",
  uses: { tools: full.tools() },
  model: "anthropic/claude-sonnet-4",
  budget: { maxTurns: 40, maxTokens: 300_000 },
})


// --- the verifier: cheap model, inspection-only, adversarial soul ---
const verifier = agents.agent("verifier", {
  does: "Adversarially verifies a diff. Compiles + runs the tests. PASS or FAIL + critique.",
  soulFile: "./verifier/SOUL.md",
  uses: { tools: inspect.tools() },
  model: "openai/gpt-4o-mini",
  budget: { maxTurns: 10, maxTokens: 40_000 },
})


// --- the gate (verifiedRun as defined above), then act on the flag ---
const result = await verifiedRun("Fix the child-process leak on MCP stdio reconnect. Add a test.")


if (result.verified) {
  await git.commit(result.text)                 // it passed the gate — safe to ship
} else {
  await slack.post("#eng", result.text)         // loudly unverified — a human decides
}

What you bring — and what this is not

Section titled “What you bring — and what this is not”

The library gives you the hard parts: isolated sub-agents (the verifier structurally cannot see the worker’s reasoning), tool scoping (it structurally cannot mutate the artifact), hierarchical budgets (it cannot outspend its ceiling), and usage roll-up (you can measure the gate’s true cost) — identical in six languages. The gate itself is yours, and so are its limits:

Not shipped What you do instead
A verify() primitive or a pass/fail type There is none by design. The verifier returns text; you define the PASS/FAIL contract and parse it.
A scoring model or threshold The library never decides “good enough”. Your loop defines what a pass is and how many retries you allow.
Detection of a lazy verifier Nothing notices a verifier that has drifted to rubber-stamping. Spot-check its critiques; seed it with known-bad artifacts and confirm it FAILs them.
Free verification The gate is at least one extra LLM pass per run — real latency and tokens on the critical path. Budget the verifier cheap; use perspective-diverse only for high stakes.
A guarantee the artifact is correct A gate raises the floor; it does not certify. A verifier is only as good as its soul and the checks it can actually run.
A sandbox Tool scoping drops the mutating tools, but bash is a real shell. If “cannot commit” must be structural, drop bash or gate it — process isolation is the host’s job.

See also

Section titled “See also”
Back to all scenarios