Multilingual desk

The failure this prevents: a French-speaking analyst asks a question, the system answers in French, and the “quote” it shows is a French translation of an English clause. That translation is not the evidence. Nobody can diff it against the source, and the moment it is wrong there is nothing to catch it.

Python

1. Ingest a corpus in whatever languages it is written in. Language is detected per document; nothing needs declaring.

   rag.ingest("handboek-nl.pdf")        # Dutch
   rag.ingest("policy-en.pdf")          # English
   rag.ingest("richtlinie-de.pdf")      # German

2. Ask in the reader’s language.

   response = rag.ask("Quelle est la durée de conservation des données ?")

3. Read the two languages separately. They are different fields on purpose.

   print(response.answer_language)      # "fr" — follows the QUESTION
   print(response.answer)               # the answer, in French
   
   src = response.sources[0]
   print(src.passage_language)          # "nl" — the SOURCE's language
   print(src.passage)                   # verbatim Dutch, untranslated
   print(src.translation)               # optional, additive, clearly marked

Go

Go runs the same 14-script tokenizer and the same v2 faithfulness gate, so a non-Latin corpus is answered verbatim in its own script — which is the part of this page that actually protects you:

package main

import (
  "fmt"

  "github.com/muthuishere/citenexus/golang/answer"
  "github.com/muthuishere/citenexus/golang/tokenize"
)

func main() {
  corpus := []answer.Doc{
    {DocumentID: "ja-policy", Text: "個人データの保持期間は五年間です。"},
    {DocumentID: "en-policy", Text: "Invoices are retained for seven years."},
  }

  res := answer.Ask(corpus, "個人データの保持期間は？", answer.DefaultTopK)
  fmt.Println(res.Evidence.Decision)   // answered
  fmt.Println(res.Answer)              // the Japanese sentence, verbatim
  fmt.Println(res.Sources[0].Document) // ja-policy

  // A script the tokenizer does not claim is nameable BEFORE you ask:
  fmt.Println(tokenize.UnsupportedScripts("ការជូនដំណឹង")) // [khmer]

  khmer := answer.Ask(corpus, "ការជូនដំណឹង?", answer.DefaultTopK)
  fmt.Println(khmer.Evidence.Decision)           // refused
  fmt.Println(khmer.Evidence.UnsupportedScripts) // [] — NOT populated by this port
}

answered
個人データの保持期間は五年間です。
ja-policy
[khmer]
refused
[]

Two honest differences. There is no language detection: Sources[0].PassageLanguage and Result.AnswerLanguage are both hardcoded "en" in this port, so read the passage, not the field. And a Khmer question refuses through the ordinary evidence-absent path — Evidence.UnsupportedScripts stays empty, so call tokenize.UnsupportedScripts yourself to tell a capability gap from an evidence gap.

JavaScript

JavaScript runs the same 14-script tokenizer and v2 gate as Go and Python, so the non-Latin passage is answered verbatim:

import { ask, unsupportedScripts } from "@muthuishere/citenexus"

const corpus = [
  { document_id: "ja-policy", text: "個人データの保持期間は五年間です。" },
  { document_id: "en-policy", text: "Invoices are retained for seven years." },
]

const res = ask(corpus, "個人データの保持期間は？")
console.log(res.evidence.decision)   // answered
console.log(res.answer)              // the Japanese sentence, verbatim
console.log(res.sources[0].document) // ja-policy

// A script the tokenizer does not claim is nameable BEFORE you ask:
console.log(unsupportedScripts("ការជូនដំណឹង"))   // [ 'khmer' ]

const khmer = ask(corpus, "ការជូនដំណឹង?")
console.log(khmer.evidence.decision)             // refused
console.log(khmer.evidence.unsupported_scripts)  // [] — NOT populated by this port

answered
個人データの保持期間は五年間です。
ja-policy
[ 'khmer' ]
refused
[]

Two honest differences. No language detection — sources[0].passage_language and answer_language are both hardcoded "en" here. And a Khmer question takes the ordinary evidence-absent refusal: evidence.unsupported_scripts stays [], so call unsupportedScripts() yourself to separate “I cannot read this script” from “the evidence isn’t there”.

The rule that makes this defensible

The answer follows the question. The citation never moves. A translated quote is no longer the evidence, so passage always stays verbatim in the source language and any translation is additive on translation — never a replacement.

The answer language is stated by the caller, never inferred from the corpus: an explicit answer_language="de" wins outright, the sentinel answer_language="auto" detects it from the question, then the ongoing conversation, then the client’s default_answer_language. The retrieved evidence does not vote.

Python

rag.ask("What is the retention period?", answer_language="de")     # forced German
rag.ask("Quelle est la durée de conservation ?", answer_language="auto")  # detected

Go

No answer-language resolution in this port. answerLanguage is the constant "en" in golang/answer, there is no config layer to hold a default_answer_language, and Ask takes no such argument. What you get is the half that cannot lie: the verbatim passage in its own script.

JavaScript

No answer-language resolution in this port. ask emits answer_language: "en" unconditionally and takes no language argument — there is no config layer behind it. The verbatim passage is still the source’s own script.

When verbatim and the answer language collide

The strict flow is extractive, so if the only support for a question is a passage in another script, the verbatim quote wins and the returned text is in the source’s script even though answer_language says otherwise. Read sources[*].passage_language for what is actually in front of you. Known open limitation — see Cross-lingual corpus.

Script support is explicit, and narrower than “multilingual”

Fourteen scripts are supported, the same fourteen in Python, Go and JavaScript — Latin, Cyrillic, Greek, Han, Hiragana, Katakana, Hangul, Arabic, Hebrew, Devanagari, Bengali, Tamil, Telugu and Thai. Space-less scripts (Han, Kana, Thai) are indexed by character bigrams.

Each claimed script is backed by a golden conformance fixture. A script is claimed only when its evidence exists — the fixture generator fails if the claim set and the fixture set disagree in either direction.

Python

response.evidence.unsupported_scripts   # ('khmer',) — a CAPABILITY gap

Go

The tokenizer names the gap; the Result does not carry it. Ask the tokenizer directly — the run in the first tab shows both halves:

tokenize.UnsupportedScripts("ការជូនដំណឹង") // [khmer]
res.Evidence.UnsupportedScripts            // [] — always, in this port

JavaScript

The tokenizer names the gap; the Result does not carry it. Ask the tokenizer directly:

unsupportedScripts("ការជូនដំណឹង")   // [ 'khmer' ]
res.evidence.unsupported_scripts    // [] — always, in this port

That signal is deliberately distinct from “no supporting evidence”. An abstention carrying unsupported_scripts means “I cannot read this script”, which is a different fact from “the evidence isn’t there” — and conflating the two is exactly how an ASCII-only tokenizer hid in a library advertised as multilingual for months.

Khmer, Lao and Myanmar are deliberately NOT claimed

The bigram path mechanically works for them. That is precisely why they are not claimed: answering through a segmentation no fixture has ever checked produces an answer indistinguishable, at the point of use, from a verified one. Refusing is the honest output until a golden fixture exists.

Known sharp edge

Arabic and Hebrew diacritics are not stripped — marks are word-forming, so a diacritized spelling and an undiacritized one are different tokens. This is conservative (it can only cause a false abstention, never a false answer) and is correct for the abugidas, but it means a corpus mixing both spellings will under-retrieve. Normalise at ingest if that describes your data.

When the question and the corpus share no tokens at all

This page assumes retrieval reaches the evidence. Across a script boundary it often does not: an English query contributes zero BM25 tokens against Tamil or Telugu prose, and a reachable-but-superseded English document answers instead — verbatim, correctly cited, and wrong for the person asking. That is a separate build, with a measured before/after: Cross-lingual corpus.