Languages & multilingual

CiteNexus supports 14 scripts, and it is the same 14 in all three ports — the Unicode tokenizer (v2) reads its script table from one canonical source, so Python, Go and JavaScript claim exactly the same set. Ask in French over an English corpus and the answer comes back in French — but the citation stays verbatim in English, because a translated quote is no longer the evidence. Any translation is additive and clearly marked.

Text in a script CiteNexus does not claim abstains with an explicit unsupported_scripts signal — a capability gap, never an evidence judgement. See Script support below.

Script support

Script	Languages	Python · Go · JS
Latin	English, Dutch, German, French, Spanish, Portuguese, …	supported
Cyrillic	Russian, Ukrainian, …	supported
Greek	Greek	supported
Han	Chinese	supported (bigram)
Hiragana / Katakana	Japanese	supported (bigram)
Hangul	Korean	supported
Arabic	Arabic	supported
Hebrew	Hebrew	supported
Devanagari	Hindi, Marathi, …	supported
Bengali	Bengali	supported
Tamil	Tamil	supported
Telugu	Telugu	supported
Thai	Thai	supported (bigram)
Kannada, Malayalam, Gujarati, Gurmukhi, Oriya, Sinhala	—	not claimed
Khmer, Lao, Myanmar, Georgian, Armenian	—	not claimed

One column, because there is only one answer: the claimed set is generated from the same canonical script table in every port — tokenize_v2 / tokenize.TokenizeV2 / tokenizeV2, with SUPPORTED_SCRIPTS / tokenize.SupportedScripts() / SUPPORTED_SCRIPTS reading it back. That tokenizer is what BM25 and the v2 faithfulness predicate consume in Python, Go and JavaScript alike. The frozen v1 tokenizer ([a-z0-9]+) still ships in all three, unchanged, so its conformance vectors keep passing — but it is not what these algorithms call.

Every claimed script is backed by a golden conformance fixture: a script is claimed only when its evidence exists, and the fixture generator fails if the two disagree in either direction.

Khmer, Lao and Myanmar are deliberately not claimed even though the bigram path mechanically works for them. Answering through a segmentation no fixture has checked is worse than refusing — it is indistinguishable, at the point of use, from a verified answer.

Space-less scripts (Han, Kana, Thai) are indexed by character bigrams, which do not cross script boundaries: Japanese 従業員は yields 従業, 業員, は.

Reading the script table from code

The three functions that answer “can you read this?” exist in every port, over the same table, and return the same answer for the same input:

Python

from citenexus.tokenize import (
    tokenize_v2, scripts_in, unsupported_scripts, SUPPORTED_SCRIPTS,
)

tokenize_v2("従業員は機密情報を開示してはならない")
# ['従業', '業員', 'は', '機密', '密情', '情報', 'を', '開示', 'して', 'ては', 'はな', 'なら', 'らな', 'ない']

scripts_in("従業員は")            # ('han', 'hiragana')
unsupported_scripts("ಕನ್ನಡ")      # ('kannada',)  → a capability gap, by name
len(SUPPORTED_SCRIPTS)            # 14

Go

import "github.com/muthuishere/citenexus/golang/tokenize"

tokenize.TokenizeV2("従業員は機密情報を開示してはならない")
// [従業 業員 は 機密 密情 情報 を 開示 して ては はな なら らな ない]

tokenize.ScriptsIn("従業員は")          // [han hiragana]
tokenize.UnsupportedScripts("ಕನ್ನಡ")    // [kannada]
len(tokenize.SupportedScripts())        // 14

JavaScript

import {
  tokenizeV2, scriptsIn, unsupportedScripts, SUPPORTED_SCRIPTS,
} from "@muthuishere/citenexus";

tokenizeV2("従業員は機密情報を開示してはならない");
// ['従業','業員','は','機密','密情','情報','を','開示','して','ては','はな','なら','らな','ない']

scriptsIn("従業員は");             // ['han', 'hiragana']
unsupportedScripts("ಕನ್ನಡ");       // ['kannada']
SUPPORTED_SCRIPTS.size;            // 14

The token lists above are not illustrative — they are the output, and the same output, from all three. TOKENIZER_VERSION / tokenize.TokenizerVersion is 2 in each, and it is stamped on an index so a tokenizer/index mismatch is detectable rather than silent.

Answering in a non-Latin script

The ports’ ask flow runs the v2 gate over the v2 tokenizer — the relevance-overlap-v2 and faithfulness-v2 pair, the same pair Python uses (HasRelevanceOverlapV2 + IsSupportedV2 in Go, hasRelevanceOverlapV2 + isSupportedV2 in JavaScript) — so a Go or JavaScript answer grounded in a Japanese passage is emitted, not refused:

Python

# The facade — retrieval, authority, conflicts and per-claim verification behind it.
response = rag.ask("従業員は機密情報を開示できますか？", answer_language="auto")
response.sources[0].passage   # verbatim Japanese

Go

corpus := []answer.Doc{
  {DocumentID: "handbook-ja", Text: "従業員は機密情報を開示してはならない。"},
}
res := answer.Ask(corpus, "従業員は機密情報を開示してはならない", answer.DefaultTopK)

res.Evidence.Decision      // "answered"
res.Sources[0].Passage     // 従業員は機密情報を開示してはならない。

JavaScript

const corpus = [
  { document_id: "handbook-ja", text: "従業員は機密情報を開示してはならない。" },
];
const res = ask(corpus, "従業員は機密情報を開示してはならない", 5);

res.evidence.decision;      // "answered"
res.sources[0].passage;     // 従業員は機密情報を開示してはならない。

The hermetic port flow always reports answer_language: "en" and passage_language: "en" — it does no detection at all, and those fields are placeholders there, not a claim about the text. Language detection is the Python facade’s job.

What is still Python-only here

The 14-script tokenizer, BM25, the v2 faithfulness predicate and the ports’ ask path are at parity — a Go or JavaScript answer over a non-Latin passage is emitted, not refused.

Atomic-claim decomposition is now at parity too. split_claims / SplitClaims / splitClaims (deterministic core) is called by all three answer flows: each splits an answer into claims, verifies every claim against its own passage, and drops the unsupported ones. A trimmed answer comes back answered with all_claims_verified: false and unsupported_claims_removed above zero — in every port.

search_languages fan-out, query reformulation, citation translation and the detector seam are Python-only — the ports have no ask() facade to hang them on. Background: ADR-0011.

Translation: three knobs, all off by default

CiteNexus never translates anything unless you ask it to, and the three places it can are separate settings with different risk profiles.

1. Query reformulation — helps cross-lingual retrieval

A French question over an English corpus aligns imperfectly in embedding space and shares zero BM25 tokens, which is a leading cause of unnecessary abstention. Enabling reformulation has a small model rewrite the query in English, then retrieves with both the original and the rewrite and RRF-fuses the two ranked lists.

from citenexus.config.schema import ReformulationConfig, HttpEndpoint

ReformulationConfig(
    enabled=True,                          # default: False
    model="gemini-2.5-flash-lite",
    endpoint=HttpEndpoint(base_url="..."),  # required — no endpoint, no reformulation
)

The original query is always kept. Translation damages exactly the tokens lexical retrieval depends on — names, IDs, clause and article numbers — so the rewrite is additive: it can surface candidates the original missed, but it can never displace them. It is enhancement-only in the strict sense too: a failure, an empty reply, or a rewrite identical to the original degrades to single-query retrieval rather than erroring.

This cannot weaken the guarantee. Reformulation affects which passages are retrieved, never whether an answer is grounded — the faithfulness gate still verifies the answer against the passage it cites.

2. Citation translation — additive, never in place

from citenexus.config.schema import MultilingualConfig

MultilingualConfig(translate_citations=True)   # default: False

Even enabled, SourceRef.passage stays verbatim in the source language. The translation lands on a separate, clearly marked translation field. A translated quote is not the evidence, so it never overwrites the evidence.

The field is on SourceRef in every port, so a reviewer reads the same two values wherever the answer was produced:

Python

src = response.sources[0]
src.passage        # verbatim — this is what you show and what a reviewer diffs
src.translation    # optional, marked, additive — a reading aid, not a citation

Go

src := res.Sources[0]
src.Passage        // verbatim — this is what you show and what a reviewer diffs
src.Translation    // *string, nil when absent — a reading aid, not a citation

JavaScript

const src = res.sources[0];
src.passage;       // verbatim — this is what you show and what a reviewer diffs
src.translation;   // string | null — a reading aid, not a citation

Only Python populates translation: filling it needs the reformulation endpoint and the config layer, and the ports have neither. They carry the field so the wire shape matches, and leave it empty.

3. Search-language fan-out — reformulation, per language, on demand

search_languages on ask() and retrieve() names the languages a question is searched in. Each one gets its own reformulation, and every ranked list is fused through the same RRF merge; the original question is always issued too.

response = rag.ask(question, search_languages=("en", "ta", "te"))   # default: ("en",)

A language whose script is not claimed raises UnsupportedSearchLanguageError before any model call is spent — by name, so the error says “Kannada is not supported” rather than “unknown language code”. Codes are never guessed: an unknown code raises too. Requesting two or more languages with no reformulation endpoint configured also raises, because a fan-out that issues no extra queries looks exactly like a search that found nothing.

Measured on a 12-document Tamil/Telugu corpus, this took answered-when-groundable from 44% to 89% with abstention on ungroundable questions unchanged at 100% — Cross-lingual corpus has the full before/after and its caveats.

Two things to know before you plan against those rates

evaluate() does not fan out. rag.evaluate(csv) calls ask(question) bare — no mode, k, answer_language or search_languages — so a golden CSV is always scored at the default ("en",) and will read lower than a fanned-out ask() on the same corpus. Drive ask() yourself when you need a fanned-out evaluation.

Every rate on this site is a single live run. Gemini is not deterministic at temperature 0: on the law benchmark an identical re-run moved the rate metrics by a question (RESULTS.md). Only the safety metrics — groundedness, citation rate, abstain-when-no-evidence, out-of-jurisdiction citations — reproduced exactly. Read the rates as illustrations, the safety metrics as results.

The answer-language rule

search_languages decides where CiteNexus looks; answer_language decides what language it answers in, and the caller states it. The retrieved evidence never votes.

1. an explicit answer_language="de" — the caller’s word outranks everything;
2. the sentinel answer_language="auto" — detect it from the question, via the injected detector;
3. the language of the ongoing conversation, if any;
4. the client’s default_answer_language ("en" by default).

Leaving answer_language unset is rung 4 — a fixed, predictable default, never a quiet inference.

rag.ask("Quelle est la durée de conservation ?", answer_language="auto")   # → French
rag.ask("What is the retention period?", answer_language="de")             # forced German
rag.ask("What is the retention period?")                                   # default_answer_language

Verbatim beats the answer language

The strict flow is extractive: the answer has to survive the faithfulness gate against the passage it cites. When the only support for an English question is a Telugu clause, “answer in the query’s language” and “quote verbatim” genuinely conflict — and verbatim wins. answer_language then describes what the chain resolved, not what script the returned text is in. Read sources[*].passage_language when you need the latter. This is a known open limitation, written up in Cross-lingual corpus.

On the Result, answer_language is the language the chain resolved and the generator was asked for, and each SourceRef keeps passage verbatim with its own passage_language; an optional translation field is additive and never replaces the passage.

Honest scope of the invariant

Citations are never rewritten, and the answer is never re-generated to force a language. The strict flow will offer the generator the next-ranked passage (bounded, a handful of attempts) when an answer fails the faithfulness gate — so one badly-phrased top passage does not cost you the answer — but every returned answer still has to be supported by the passage it cites, and when nothing passes the flow refuses. Abstention over an ungrounded answer is the intended behaviour.

Detecting language

Two detectors implement the same plugin; inject one via detector= (or let from_config pick FastTextDetector):

• FastTextDetector — the real fastText lid.176 model, lazily downloaded and cached on first use. FastTextDetector(*, threshold=0.50, model_dir=None). Requires the optional fasttext package.
• HeuristicDetector — a network-free Unicode-script detector, the offline / test default. HeuristicDetector(*, threshold=0.50, default_language="en").

from citenexus.lang.detect import HeuristicDetector

rag = CiteNexus("./citenexus-data", embedder=..., generator=..., detector=HeuristicDetector())

Both return a LanguageResult (language, confidence, is_reliable); a detection below the 0.50 threshold is treated as unreliable and falls through the chain above.