Worked example: law (a live benchmark)

This is a real run — Jina embeddings + reranker, Gemini 2.5-flash generation, LanceDB storage, nothing mocked. It stress-tests CiteNexus on a high-stakes domain (California landlord–tenant notice law) and reports the actual committed numbers, including where the library still falls short.

The corpus was built to demonstrate a gap: relevance ranking alone lets a low-authority source that repeats the query vocabulary out-rank the controlling statute. The ADR-0004 authority floor closes that gap, and this page records the before and after — including the part that did not get fixed.

The corpus — one topic, deliberately varied authority

Six real, public documents. Four say “30 days” for a month-to-month tenancy and repeat month-to-month / notice / terminate / landlord densely. Only the two highest-authority California sources carry the correct 60-day rule for a long-term tenant — in denser, less repetitive prose.

document	curator-declared tier
Cal. Civ. Code § 1946.1 — controlling residential statute (60 days if ≥ 1 yr, 30 if < 1 yr)	controlling-statute
Mak v. City of Berkeley (Cal. Ct. App. 2015) — applies the 60-day rule	binding-appellate
Cal. Civ. Code § 1946.2 — just cause after 12 months	statute
Cal. Civ. Code § 1946 — older general rule (30 days), superseded for residential	general-statute
Nolo self-help blog — “most states … 30-day notice”	secondary-blog
Fla. Stat. § 83.57 — 30 days	out-of-jurisdiction

Tiers are curator assertions supplied at ingest via authority=. The library never derives them from prose, and a mis-declared tier is a real failure mode.

The measured progression (11-question golden set)

The committed results.json is the post-floor run — the rightmost column.

metric	v0.9.0	v0.10.0 pre-floor	v0.10.0 post-floor (committed)
answered / refused	5 / 6	8 / 3	6 / 5
groundedness_rate	100%	100%	100%
citation_rate	100%	100%	100%
answer_when_grounded	50%	75%	75%
abstain_when_no_evidence	67%	33%	100%
out-of-jurisdiction citations	—	4	0

Read the columns together, because each one alone misleads.

The safety metric is what moved, and it moved all the way. Four citations of a Florida statute against California and Texas questions → zero. Every question that should refuse now refuses (3/3).

The pre-floor column is the cautionary one. It looks like the best run — 8 answered, answer_when_grounded up from 50% to 75%, groundedness still 100% — and it is the least trustworthy of the three. Those extra answers include a Texas question answered from Florida law with all_claims_verified: true. 100% groundedness alongside 4 wrong-jurisdiction citations is the entire reason authority had to exist: the gate proved the words came from the passage, which was true, and said nothing about whether the passage governed.

Recall did not regress to buy that. answer_when_grounded held at 75% across the floor; what disappeared was the wrong-authority answers, not the right ones.

These rates are a single live run, and the generator is not deterministic

An identical post-floor re-run — same code, same corpus, same prompts, temperature 0 — produced 5 / 6 answered/refused and answer_when_grounded 62% instead of 6 / 5 and 75%.

What was stable across both runs: out-of-jurisdiction citations = 0, groundedness 100%, citation rate 100%, abstain_when_no_evidence 100%. So treat the rate metrics as ±1 question of wobble and the safety metrics as the reproducible result. Any single-run rate quoted from this example is an illustration, not a benchmark number.

The thesis holds on the metric that matters for a regulated domain: zero ungrounded, zero fabricated answers. Every claim CiteNexus emitted traced to a real passage it cited.

External context, not reproduced here

Stanford RegLab (Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, 2024) reported a 17–33% hallucination rate for leading generic legal-RAG tools. We cite that as context — we did not re-run a generic-RAG baseline. CiteNexus’s failure modes are over-refusal and wrong-subject citation, not fabrication.

What the floor fixed

1. The Texas question stopped being answered from Florida law

Q: “What is the notice period to end a month-to-month tenancy in Texas?” Pre-floor: “not less than 30 days’ notice” — cited 06-florida-83_57-statute, all_claims_verified: true. Post-floor: refused.

Texas is not in the corpus. The out-of-jurisdiction Florida statute token-matched “month-to-month / notice” and answered a question about a third state, perfectly grounded. The floor withholds it before generation, and the refusal reason is deliberately distinct from “no relevant evidence found”.

2. The 60-day authority probes stopped being suppressed

Both long-tenancy questions now answer 60 days from the right sources:

question	cited	tier
“resided more than one year”	02-mak-v-berkeley-2015-appellate	binding-appellate
“three years, month-to-month”	01-ca-civ-1946_1-statute	controlling-statute

Both are classified CORRECT-AUTHORITY in results.json. Pre-floor these refused: the repetitive 30-day text (Florida × 3 plus the Nolo blog) crowded § 1946.1(b) and the Mak opinion out of the top 6 entirely, so the faithfulness gate had no 60-day passage to verify against. Authority-blindness had silently suppressed the controlling answer.

How the floor is wired — deliberately not into ranking

The floor is applied after grounding and before generation, not fed into fusion or rerank. That is the whole point of ADR-0004: mixing standing into the relevance score makes a low-authority passage merely less likely to win, which is still a coin flip on rank order. Withholding it outright makes the failure a refusal you can see. Conflict detection follows the same rule — it reports and never resolves by rank (ADR-0007).

It is unranked by default. You get it by attaching tiers at ingest (rag.ingest(..., authority={"authority_tier": ...})) and setting a floor. See Authority — grounding is not standing.

Three things this run does not let us claim

A. One golden question now refuses by design

Q: “What minimum notice is required to terminate a month-to-month tenancy in Florida?” — golden set says answer → refused.

The curator declared Florida out-of-jurisdiction for this corpus. A floored California corpus therefore cannot answer a Florida question, and refusing is the correct behaviour of the configuration as written. This is corpus scoping, not a bug — but the golden set now encodes an expectation the configuration contradicts, and it counts against answer_when_grounded. Either the question leaves the golden set or the corpus stops being California-only; the current state is honest but inconsistent.

B. The subject-scope gap is NOT fixed — the commercial-lease case passed by luck

Q: “How much notice must a landlord give to terminate a fixed five-year commercial lease with a specified term in California?” — must abstain.

It refuses in this run. That is luck, not authority. Dropping the Florida chunks changed which passages reached the generator; the floor did nothing here and cannot. The source that produces the wrong answer is 01-ca-civ-1946_1-statute — tier controlling-statute, the highest tier in the corpus. No ordering over sources can exclude the top of the ordering. It is genuinely the right authority, about the wrong kind of tenancy.

The cause is applicability severance: the clause deciding whether § 1946.1 applies at all — “for a term not specified by the parties” — is a different Evidence Unit from the operative 60-day rule, and retrieval, the generator and the gate all saw the operative one and none of them ever saw the precondition. Measured: 8 of 11 operative notice-period EUs are citable in isolation from the precondition that governs them — a 73% severance rate on this corpus.

The information is in the corpus (5 of 6 documents state their term-scope in plain prose). It is severed by chunking, and every downstream guard is chunk-local by design. This is open work — see Wrong subject, right source and docs/adr/0012-subject-scope-applicability.md.

C. Over-refusal is still real

Two groundable questions refuse for gate conservatism, not authority: § 1162 “manner of service”, where the ordered-containment gate could not verify a short cross-reference answer even though the passage was present, plus the Florida question in (A). Safe for a regulated domain; it costs recall — and it is the deliberate direction of the trade, since the gate is strictly narrower than the predicate it replaced: the faithfulness gate.

`evaluate()` reads 45% here, and that is the metric's fault

The library’s own evaluate() reports expected_support_rate 45% on this same run while groundedness and citation are both 100%. A blank expected is scored as supported only if the row was answered, so the five refusals — four of them correct — count against it. evaluate() cannot score abstention. This example computes its own answer/abstain accuracy in run.py; the extra golden.csv columns (expect_decision, probe, correct_docs, trap_docs) are ignored by evaluate() and used only by our analysis. See Evaluate.

Reproduce it

cd python && . .venv/bin/activate        # or: uv run
export JINA_API_KEY=...                   # referenced by name; never printed
export GEMINI_API_KEY=...
export CITENEXUS_BASE_URI=/tmp/law-data && rm -rf "$CITENEXUS_BASE_URI"
python ../examples/law-authority/run.py   # writes results.json

Expect the rate metrics to move by a question between runs; the safety metrics should not. The corpus, golden set, run.py and the full write-up live under examples/law-authority/.

Honest limitations

1. Zero fabrication is the claim; zero wrong answers is not. CiteNexus’s failure modes here are over-refusal and — until the floor — wrong-authority citation, with wrong-subject citation still open.
2. Authority is curator-supplied. The floor is only as good as authority.csv. Nothing in the library validates that a document really is a controlling statute.
3. Small set, few runs. 6 documents, 11 questions, one sub-topic, a non-reproducible generator. Illustrative baseline, not a benchmark.
4. Corpus text is trimmed. The committed corpus files are faithful excerpts, not full pages. Severance rates depend on extractor granularity, so the 73% figure is specific to plain-text ingest of this corpus.