Cross-lingual corpus

The failure this prevents: an employee at the Hyderabad office asks how much unused leave they may carry forward. The answer comes back “a maximum of 10 days”, quoted verbatim from the English handbook, with a real document ID and a passing faithfulness gate. The binding Telugu annexure for that office caps it at 5 — and it was never retrieved, because an English query shares zero BM25 tokens with Telugu prose.

Nothing in the pipeline is broken. Groundedness is 100%, the citation is real, the quote is exact. The answer is simply from a document that does not govern the person asking — and at the point of use that is indistinguishable from a correct answer.

This is the sibling failure to Regulated audit and Authority: there, the wrong document was reachable and shouldn’t have been; here, the right document was unreachable, and no ordering over what you retrieved can rescue a passage you never retrieved.

Python

1. Configure a reformulator. The fan-out rewrites the question into each requested language with a small model, then RRF-fuses every retrieval. Without an endpoint there are no extra queries, so it is required rather than silently skipped.

   from citenexus.config.schema import ReformulationConfig
   
   ReformulationConfig(
       enabled=True,                      # default: False
       model="gemini-2.5-flash-lite",     # a SMALL model is enough
       endpoint=gemini,                   # required — no endpoint, no fan-out
   )

2. Name the languages the corpus is written in. Same keyword on ask() and retrieve(); the default is ("en",), which is the pre-fan-out behaviour.

   response = rag.ask(
       "How many days of earned leave may a Hyderabad employee carry forward?",
       search_languages=("en", "ta", "te"),
   )

   The original question is always issued too, so the fan-out is strictly additive: it can surface candidates the original missed, never displace them.

3. Read the two language fields separately. They answer different questions.

   response.answer_language          # "en" — resolved for the ANSWER
   response.sources[0].passage_language   # "te" — the SOURCE's own language
   response.sources[0].passage            # verbatim Telugu, untranslated

Go

There is no query fan-out in this port — search_languages, the reformulation seam and the ask() facade they hang off are Python-only, so a Go caller cannot rewrite an English question into Telugu. Nothing here closes the cross-script recall gap for you.

What Go does have is the half underneath it, and it is worth running, because it shows both the capability and the gap in one program: the 14-script tokenizer and the v2 gate answer a Telugu question over a Telugu clause verbatim, while the English form of the same question refuses.

package main

import (
  "fmt"

  "github.com/muthuishere/citenexus/golang/answer"
  "github.com/muthuishere/citenexus/golang/tokenize"
)

func main() {
  fmt.Println(tokenize.SupportedScripts()) // the 14 claimed scripts
  fmt.Println(tokenize.TokenizeV2("హైదరాబాద్ ఉద్యోగులు రోజులు"))
  fmt.Println(tokenize.UnsupportedScripts("ការជូនដំណឹង")) // [khmer] — a capability gap

  corpus := []answer.Doc{
    {DocumentID: "09-te-hyderabad-annexure", Text: "హైదరాబాద్ కార్యాలయ ఉద్యోగులు సెలవును బదిలీ చేయవచ్చు."},
    {DocumentID: "01-en-handbook", Text: "Employees may carry forward a maximum of ten days of earned leave."},
  }

  res := answer.Ask(corpus, "హైదరాబాద్ ఉద్యోగులు సెలవును బదిలీ చేయవచ్చా?", answer.DefaultTopK)
  fmt.Println(res.Evidence.Decision)          // answered
  fmt.Println(res.Answer)                     // the Telugu clause, verbatim
  fmt.Println(res.Sources[0].Document)        // 09-te-hyderabad-annexure
  fmt.Println(res.Sources[0].PassageLanguage) // "en" — hardcoded in this port

  // The same question in English shares no token with Telugu prose:
  crossScript := answer.Ask(corpus[:1], "How many days of leave may a Hyderabad employee carry forward?", answer.DefaultTopK)
  fmt.Println(crossScript.Evidence.Decision) // refused
}

[arabic bengali cyrillic devanagari greek han hangul hebrew hiragana katakana latin tamil telugu thai]
[హైదరాబాద్ ఉద్యోగులు రోజులు]
[khmer]
answered
హైదరాబాద్ కార్యాలయ ఉద్యోగులు సెలవును బదిలీ చేయవచ్చు.
09-te-hyderabad-annexure
en
refused

That last refused is this page’s failure, reproduced in Go — and here it is the safe half of it, because the corpus held nothing else to answer from. Add the English handbook back and the wrong-but-grounded answer returns. Until fan-out exists here, issue the question in each of your corpus’s languages yourself and merge the Results.

Note also PassageLanguage prints en on a Telugu passage: this port does no language detection and hardcodes the field. Read the passage, not the field.

JavaScript

There is no query fan-out in this port — search_languages and the reformulation seam are Python-only. What JavaScript has is the tokenizer and gate underneath, which answer a Telugu question over a Telugu clause verbatim, and refuse the English form of the same question:

import { ask, tokenizeV2, unsupportedScripts, SUPPORTED_SCRIPTS } from "@muthuishere/citenexus"

console.log([...SUPPORTED_SCRIPTS])                      // the 14 claimed scripts
console.log(tokenizeV2("హైదరాబాద్ ఉద్యోగులు రోజులు"))
console.log(unsupportedScripts("ការជូនដំណឹង"))            // [ 'khmer' ] — a capability gap

const corpus = [
  { document_id: "09-te-hyderabad-annexure", text: "హైదరాబాద్ కార్యాలయ ఉద్యోగులు సెలవును బదిలీ చేయవచ్చు." },
  { document_id: "01-en-handbook", text: "Employees may carry forward a maximum of ten days of earned leave." },
]

const res = ask(corpus, "హైదరాబాద్ ఉద్యోగులు సెలవును బదిలీ చేయవచ్చా?")
console.log(res.evidence.decision)             // answered
console.log(res.answer)                        // the Telugu clause, verbatim
console.log(res.sources[0].document)           // 09-te-hyderabad-annexure
console.log(res.sources[0].passage_language)   // "en" — hardcoded in this port

// The same question in English shares no token with Telugu prose:
const crossScript = ask([corpus[0]], "How many days of leave may a Hyderabad employee carry forward?")
console.log(crossScript.evidence.decision)     // refused

[
  'arabic',   'bengali',
  'cyrillic', 'devanagari',
  'greek',    'han',
  'hangul',   'hebrew',
  'hiragana', 'katakana',
  'latin',    'tamil',
  'telugu',   'thai'
]
[ 'హైదరాబాద్', 'ఉద్యోగులు', 'రోజులు' ]
[ 'khmer' ]
answered
హైదరాబాద్ కార్యాలయ ఉద్యోగులు సెలవును బదిలీ చేయవచ్చు.
09-te-hyderabad-annexure
en
refused

The final refused is this page’s failure reproduced in JavaScript. Until fan-out exists here, ask the question once per corpus language and merge the Results. And passage_language reads en on a Telugu passage — no language detection in this port, so read the passage itself.

Searching a language is a capability claim, so it is checked first

An unsupported language raises before any model call is spent — it does not return an empty list, and it does not abstain.

Python

rag.ask(q, search_languages=("en", "kn"))
# UnsupportedSearchLanguageError: search language 'kn' (Kannada) is written in
# 'kannada', which this tokenizer does not claim (ADR-0011).

Go

There is no search_languages to check, so there is no error to raise. The equivalent check you can make before asking is on the tokenizer:

tokenize.UnsupportedScripts("ಕನ್ನಡ") // [kannada] — do not ask; you will get an
                                     // evidence-absent refusal that hides this

JavaScript

There is no search_languages to check, so there is no error to raise. Make the equivalent check on the tokenizer before asking:

unsupportedScripts("ಕನ್ನಡ")   // [ 'kannada' ] — do not ask; you will get an
                              // evidence-absent refusal that hides this

That is deliberate. Returning [] for a language the tokenizer cannot segment is indistinguishable from “the corpus does not contain this”, and routing a capability gap through the abstention channel is exactly how an ASCII-only tokenizer once hid inside a library advertised as multilingual.

Telugu is now claimed — U+0C00–U+0C7F was a hole in the script range table and is fixed, with a golden fixture behind it. Kannada, Malayalam, Gujarati, Gurmukhi, Oriya and Sinhala are named in the table and deliberately not claimed: naming them is what lets the refusal say “Kannada is not supported” instead of “unknown language code”. See Languages & multilingual for the full script matrix.

Two independent knobs

search_languages decides which languages the question is searched in. answer_language decides which language it is answered in — an explicit code forces it, the sentinel "auto" detects it from the question, and leaving it unset uses the configured default_answer_language. The retrieved evidence never votes on either.

Not available on strategy="deep"

The agentic loop writes its own follow-up queries, so passing search_languages to strategy="deep" raises rather than searching fewer languages than you asked for. Use strategy="strict".

Measured

From the live run in examples/multilingual/ on 2026-08-16 — 12 authored documents (4 English / 4 Tamil / 4 Telugu, two of the Tamil ones as real PDFs), 22 questions all asked in English, real Jina embeddings + jina-reranker-v2-base-multilingual, real gemini-2.5-flash generation at temperature 0, nothing mocked. One constant changed between runs: ("en",) → ("en", "ta", "te").

bucket	n	answered before	answered after
Tamil-only	6	1	4
Telugu-only	7	2	7
English control	5	5	5
Ungroundable (must abstain)	4	0	0

metric	before	after
answered / abstained	8 / 14	16 / 6
groundedness_rate	100%	100%
citation_rate	100%	100%
cited_right_document_rate	75%	100%
answer_when_groundable	44%	89%
abstain_when_ungroundable	100%	100%

The Hyderabad question is now answered 5 రోజులు, cited to 09-te-hyderabad-leave-annexure. The two rows that must never move — abstention on ungroundable questions, and groundedness — did not move. The recall was not bought out of the abstention guarantee.

Two changes shipped; only one of them is the fan-out

Telugu could not be cited at all in the baseline: its script range was missing from the tokenizer, so every Telugu passage was filtered out of the grounding set before search language mattered. That was fixed in the same sitting. The Telugu column is therefore the sum of two changes and this benchmark cannot split them.

Read the Tamil bucket instead if you want to know what the fan-out itself buys. Tamil was already claimed and already citable in the baseline, so nothing about it changed except the search languages: 1/6 → 4/6. At n = 6 that is a direction, not a rate you can plan against.

Known limitation: verbatim wins over answer language

The Hyderabad answer comes back as 5 రోజులు — Telugu script — while answer_language reads "en". That is a real inconsistency, and it is not solved. It falls out of two individually correct rules that genuinely conflict:

• The strict flow is extractive — the answer must survive the faithfulness gate against the passage it cites, which is what makes “no ungrounded claim” checkable at all.
• Citations stay verbatim in their source language, because a translated quote is no longer the evidence.

When the only support for an English question is a Telugu clause, “answer in the query’s language” and “quote verbatim” cannot both hold — and verbatim wins. answer_language then reports what the resolution chain decided, not what script the returned text is in.

Two candidate fixes, neither implemented: make answer_language descriptive of the returned text (cheap, but it changes the meaning of a field with conformance vectors on it), or generate an answer alongside the verbatim citation (a real design change to the strict flow, not a patch). Until one lands: read sources[*].passage_language, not answer_language, when you need to know what script is in front of you.

How to read these numbers

Single run per configuration. Gemini is not deterministic at temperature 0 — on the law benchmark rate metrics moved ±1 question between identical runs while the safety metrics reproduced exactly. A ±1 movement in any bucket here would not surprise us.

The corpus is authored fiction. Twelve invented employment-policy documents for a company that does not exist; every figure was chosen to make the measurement possible. Not real policy, not legal advice, nothing scraped. See PROVENANCE.md.

evaluate() does not fan out. rag.evaluate(csv) calls ask() with the default search_languages=("en",). Drive ask() yourself if you need a fanned-out evaluation.

Cost

One reformulation call per requested language per question, cached per (question, language) and shared across ask() and retrieve(). Three languages is three calls — one per requested language, including the one the question is already in — yielding up to three extra queries. A reformulation that fails, comes back empty, or duplicates the original contributes nothing — retrieval proceeds with fewer queries rather than erroring.

The benchmark above is Python, because the fan-out it measures is Python. See Languages & multilingual for the script matrix all three share.