Ingest anything

Ingest a file, raw text, or a URL. Python’s ingest() writes into a persistent store and is idempotent by content hash. Go and JavaScript have no store on the plain go get / npm install surface, so they do the same job in memory: turn a source into chunked corpus documents and pass that corpus to Ask / ask.

Python

from citenexus.extract.types import SourceType

rag.ingest("report.pdf")                          # a file → extractor picked by extension
rag.ingest(text="Clause 4: 90 days notice.", document_id="c4")   # raw text (plain)
rag.ingest("https://example.com/policy")          # a URL → fetched, HTML-extracted
rag.ingest(image_bytes, source_type=SourceType.image, document_id="fig-1")  # an image

Go

// No store on the plain `go get` surface: a source becomes corpus documents.
import (
  "fmt"
  "os"

  "github.com/muthuishere/citenexus/golang/answer"
  "github.com/muthuishere/citenexus/golang/chunker"
)

func ingestFile(path, documentID string) []answer.Doc {
  raw, err := os.ReadFile(path)
  if err != nil {
    panic(err)
  }
  var docs []answer.Doc
  for i, chunk := range chunker.ChunkText(string(raw), chunker.DefaultMaxTokens, chunker.DefaultOverlap) {
    docs = append(docs, answer.Doc{DocumentID: fmt.Sprintf("%s::%d", documentID, i), Text: chunk})
  }
  return docs
}

corpus := ingestFile("corpus/nda.txt", "nda")
corpus = append(corpus, answer.Doc{DocumentID: "clause-4", Text: "Clause 4: 90 days notice."})

res := answer.Ask(corpus, "Can the employee disclose confidential information?", answer.DefaultTopK)
fmt.Println(len(corpus), "documents |", res.Evidence.Decision, "|", res.Sources[0].Document)
//  2 documents | answered | nda::0

JavaScript

// No store on the plain `npm install` surface: a source becomes corpus documents.
import { readFileSync } from "node:fs";
import { ask, chunkText } from "@muthuishere/citenexus";

function ingestFile(path, documentId) {
  const raw = readFileSync(path, "utf8");
  return chunkText(raw).map((text, i) => ({ document_id: `${documentId}::${i}`, text }));
}

const corpus = [
  ...ingestFile("corpus/nda.txt", "nda"),
  { document_id: "clause-4", text: "Clause 4: 90 days notice." },
];

const res = ask(corpus, "Can the employee disclose confidential information?");
console.log(corpus.length, "documents |", res.evidence.decision, "|", res.sources[0].document);
//  2 documents | answered | nda::0

What comes back

Python

r = rag.ingest("corpus/nda.txt")
print(r.status, r.n_units)     # "ingested" 1   (or "unchanged" 0 on a re-ingest)
print(r.document_id, r.eu_ids) # "nda" ('nda::0::0',)

Re-ingesting the same document_id with identical bytes is a no-op:

rag.ingest(text="Clause 4: 90 days notice.", document_id="c4").status   # "ingested"
rag.ingest(text="Clause 4: 90 days notice.", document_id="c4").status   # "unchanged"

Go

// The corpus slice *is* the result: one Doc per chunk, with the ids you assigned.
docs := ingestFile("corpus/nda.txt", "nda")
for _, d := range docs {
  fmt.Println(d.DocumentID, len(d.Text))
}
//  nda::0 148

There is no status/unchanged to read because nothing is persisted — see bulk & batch ingest for the content-hash dedup you add yourself with euid.Checksum.

JavaScript

// The corpus array *is* the result: one document per chunk, with the ids you assigned.
const docs = ingestFile("corpus/nda.txt", "nda");
for (const d of docs) console.log(d.document_id, d.text.length);
//  nda::0 148

There is no status/unchanged to read because nothing is persisted — see bulk & batch ingest for the content-hash dedup you add yourself with sha256Hex.

One source at a time

Python’s ingest() takes a single source. It does not walk a directory and there is no no-argument “ingest the whole bucket/folder” call — pass each file, URL, or text yourself. For whole-site intake, use crawl() below. The Go and JavaScript helpers above are per-file for the same reason; loop them yourself (bulk & batch ingest).

Ingest has an inverse: rag.delete(document_id) (alias revoke) surgically removes one document and everything derived from it — idempotently, and guarding raw blobs shared by identical bytes.

Crawl a site

Python

crawl() does a same-domain breadth-first walk, ingesting each page as HTML:

results = rag.crawl("https://docs.example.com", max_pages=50, max_depth=3)
print(len(results), "pages ingested")

Go

No crawler in the Go port — fetch the pages with net/http, then feed each body through the same chunker.ChunkText → []answer.Doc path as a file. The port ships no HTML extractor on the plain go get surface either (that lives in the Rust core, behind the citenexus_ffi build tag), so you strip markup yourself.

JavaScript

No crawler in the JavaScript port — fetch() the pages, then feed each body through the same chunkText → corpus path as a file. The port ships no HTML extractor on the plain npm install surface either (that lives in the Rust core, behind the @muthuishere/citenexus/ingest subpath), so you strip markup yourself.

Supported formats

Extraction dispatches by explicit source_type first, then file extension, then falls back to plain text:

Extension	Extractor
.pdf	PDF (text + figures, with page and — at extraction only — bbox)
.docx .pptx .xlsx	Office (OOXML); tables become GFM Markdown
.html .htm	HTML
.md .markdown	Markdown
.csv	CSV
.txt	Plain text
anything else	Plain-text fallback — never a hard failure

Images and OCR — the honest version

Standalone image files are not routed by extension (.png/.jpg fall through to plain text). To ingest an image, pass source_type=SourceType.image explicitly. Figures inside PDFs/DOCX/PPTX are handled by the injected vision model during ingest. CiteNexus ships no OCR engine — text inside a scanned image is only recovered if your vision model returns it.

Which indexes get built from the extracted units is governed by the signals you declared.

Where extraction actually runs

Go and JavaScript extract through the shared Rust core (citenexus_extract) — reachable behind the citenexus_ffi build tag (Go) and the @muthuishere/citenexus/ingest subpath (JavaScript), both of which need the Rust cdylib built. The Python reference does not use it — it uses pdfplumber for PDF and openpyxl for XLSX, and holds itself to the Rust twin with parity tests instead. Today XLSX byte-identity is test-pinned; PDF has neither a parity test nor a conformance case, so do not assume a PDF extracts identically across the three languages.