Evaluate a corpus

The failure this prevents: you swap the embedding model, the answer rate goes up, everyone celebrates, and what actually happened is that the system started answering questions it should have refused. Answer rate on its own is a metric that rewards hallucination.

Read this before you write a golden set

evaluate() cannot score abstention, and scores it backwards. A row with a blank expected counts as “expected-supported” iff the row was answered (evaluate.py:76-77) — so a correct refusal earns nothing, and because expected_support_rate divides by total rather than by answered, every refusal drags it down.

Must-refuse rows therefore do not belong in the CSV. Split the job: evaluate() scores the answerable rows, and you assert the refusals yourself. Everything below is built that way.

Python

1. Write a golden CSV of rows the corpus can answer. The column the library reads is expected — not expected_support, which is silently ignored. The question column may be question or query.

   question,expected
   "Can the employee disclose confidential information?","shall not disclose"
   "What notice does termination require?","thirty days"

   A row passes when the content tokens of expected are a subset of the answer’s — a lightweight check that the right evidence made it into the text.

2. Run it.

   report = rag.evaluate("golden.csv")
   
   print(report.groundedness_rate)      # grounded / answered
   print(report.citation_rate)          # cited / answered
   print(report.expected_support_rate)  # expected_supported / TOTAL

3. Keep the must-refuse questions in a separate list and assert the decision directly. This is the abstention regression gate.

   from citenexus.answer.result import Decision
   
   MUST_REFUSE = [
       "What is the capital of France?",           # not in the corpus at all
       "What is the maximum security deposit?",    # adjacent topic, no clause
   ]
   
   for question in MUST_REFUSE:
       result = rag.ask(question)
       assert result.evidence.decision is Decision.refused, question

4. Gate CI on the direction of travel, not on an absolute:

   assert report.groundedness_rate >= baseline.groundedness_rate
   assert report.citation_rate >= baseline.citation_rate
   assert refused_count == len(MUST_REFUSE)

Go

evaluate() does not exist in this port — there is no CSV front door, no groundedness_rate and no audit log. But re-read the warning at the top of this page: the half of the job that actually protects you is the must-refuse assertions, and those are a plain loop over Ask. That much is real here, and it runs:

package main

import (
  "fmt"
  "os"

  "github.com/muthuishere/citenexus/golang/answer"
  "github.com/muthuishere/citenexus/golang/result"
)

func main() {
  corpus := []answer.Doc{
    {DocumentID: "help/returns", Text: "Unopened items may be returned within 30 days of delivery."},
  }
  mustRefuse := []string{
    "What is the capital of France?",         // not in the corpus at all
    "What is the maximum security deposit?",  // adjacent topic, no clause
  }

  for _, q := range mustRefuse {
    res := answer.Ask(corpus, q, answer.DefaultTopK)
    if res.Evidence.Decision != result.DecisionRefused {
      fmt.Println("REGRESSION: answered a must-refuse row:", q)
      os.Exit(1)
    }
    fmt.Println("refused, as required:", q)
  }
}

refused, as required: What is the capital of France?
refused, as required: What is the maximum security deposit?

The rates are what you lose: to compute groundedness or citation rate you would have to aggregate Results yourself, and the scoring rules (and the expected subset check) live only in Python.

JavaScript

evaluate() does not exist in this port — no CSV front door, no rates, no audit log. The must-refuse gate — the half this page argues matters most — is a plain loop over ask, and it works here:

import assert from "node:assert/strict"
import { ask, Decision } from "@muthuishere/citenexus"

const corpus = [
  { document_id: "help/returns", text: "Unopened items may be returned within 30 days of delivery." },
]
const MUST_REFUSE = [
  "What is the capital of France?",        // not in the corpus at all
  "What is the maximum security deposit?", // adjacent topic, no clause
]

for (const q of MUST_REFUSE) {
  const res = ask(corpus, q)
  assert.equal(res.evidence.decision, Decision.refused, q)
  console.log("refused, as required:", q)
}

refused, as required: What is the capital of France?
refused, as required: What is the maximum security deposit?

The rates are what you lose: aggregating Results into groundedness / citation / expected-support numbers is Python-only.

Why the must-refuse list is the one that matters

A model that answers everything scores well on any metric that only looks at answers — which is exactly what groundedness_rate and citation_rate do: both divide by answered, so a system that refuses everything it is unsure of and answers three questions perfectly reads 100% / 100%. Those two rates prove that what it did say was grounded and cited. They say nothing about what it should not have said.

The must-refuse assertions are the other half. A change that makes the system more willing to answer will flip one of those rows before it moves any rate, and a flipped assertion names the question. That is the early warning; the rates are the confirmation.

Per-claim scoring changes what a partial answer means

Verification is per atomic claim, so an answer can come back trimmed — its supported claims only, with unsupported_claims_removed counting the rest. Trimmed answers are scored on the claims they actually returned, not on what the model originally produced.

`evaluate()` passes nothing through to `ask()`

It calls ask(question) bare — no mode, k, answer_language or search_languages (client.py:883-885, evaluate.py:72). On a multilingual corpus its counts will not match a harness that fans out over search_languages, and that gap is the missing fan-out rather than a regression. If you need those options, drive ask() in a loop and aggregate yourself.

Reading a regression

When a row flips, the signals say why before you open anything:

Python

sig = response.evidence

sig.decision                    # answered | refused | partial
sig.supporting_sources          # 0 -> retrieval failed, not generation
sig.distinct_documents          # 1 -> single-source, no corroboration
sig.retrieval_score_spread      # near 0 -> everything ranked alike; ranking is not discriminating
sig.unsupported_claims_removed  # >0 -> the generator over-reached and got trimmed
sig.conflicts_detected          # >0 -> the corpus disagrees with itself
sig.unsupported_scripts         # non-empty -> a capability gap, not an evidence gap
sig.authority_floor_applied     # True -> a refusal may mean "no standing", not "no evidence"

Go

EvidenceSignals is conformance-pinned, so every field above exists here with the same meaning:

sig := res.Evidence

sig.Decision                 // answered | refused | partial
sig.SupportingSources        // 0 -> retrieval failed, not generation
sig.DistinctDocuments        // 1 -> single-source, no corroboration
sig.RetrievalScoreSpread     // near 0 -> ranking is not discriminating
sig.UnsupportedClaimsRemoved // >0 -> the generator over-reached and got trimmed

ConflictsDetected is a real signal here too — ADR-0007 detection is native in this port, so a non-zero count means the corpus contradicted itself in the window that was about to be cited.

Two fields are always zero/empty in this port and diagnose nothing: UnsupportedScripts (never populated on a Result — call tokenize.UnsupportedScripts instead) and AuthorityFloorApplied (authority selection runs in the Python ask() facade). They are carried for wire parity, not as signals you can read.

JavaScript

EvidenceSignals is conformance-pinned, so the fields are the same:

const sig = res.evidence

sig.decision                    // answered | refused | partial
sig.supporting_sources          // 0 -> retrieval failed, not generation
sig.distinct_documents          // 1 -> single-source, no corroboration
sig.retrieval_score_spread      // near 0 -> ranking is not discriminating
sig.unsupported_claims_removed  // >0 -> the generator over-reached and got trimmed

conflicts_detected is a real signal here too — ADR-0007 detection is native in this port.

Two are always zero/empty here and diagnose nothing: unsupported_scripts (never populated on a Result — call unsupportedScripts() instead) and authority_floor_applied (authority selection is in the Python facade). They exist for wire parity only.

supporting_sources = 0 and unsupported_claims_removed > 0 are different bugs in different halves of the pipeline. The first is retrieval, the second is generation — and treating them the same is how teams spend a week tuning a prompt to fix an indexing problem.

A refusal is not always an evidence gap. In strict mode it can also mean the authority floor withheld everything it found (authority_floor_applied), or that two grounded sources contradict each other (conflicts_detected). Those are three different failures with three different fixes.

Retrieval on its own

retrieve() is the engine under ask(), exposed so you can measure ranking without a generator in the loop:

Python

for c in rag.retrieve("termination notice period", k=10):
    print(f"{c.score:.4f}  {c.document_id}  {c.citable_text[:60]}")

Go

There is no retrieve() in this port — ranking is internal to Ask and not exposed. The closest thing is topK: answer.Ask(corpus, q, 1) versus answer.Ask(corpus, q, answer.DefaultTopK) tells you whether the right passage was ranked first or merely somewhere in the top five. The ranking primitives themselves — bm25.Rank and rrf.Fuse — are exported and can be driven directly if you need the scores.

JavaScript

There is no retrieve() in this port — ranking is internal to ask. Narrowing topK (askWith(corpus, q, { topK: 1 })) tells you whether the right passage ranked first. The bm25 and rrfFuse primitives are exported if you want the scores themselves.

If the right passage is not in the top-k here, no amount of prompt work will save the answer.

A worked example on a real corpus

The law worked example runs this end to end against California landlord–tenant statutes with live models — including the reason its own expected_support_rate reads 45% while groundedness and citation are both 100%.