Enrichment: LLMs & VLMs

In the Enrich stage (05), indx asks a model to read your content and add useful metadata: a detected document type, topics, tags, and a summary. The stage uses two swappable slots — an LLM handles text, and an optional VLM handles images.

This guide covers how to choose and configure those models, how to disable enrichment, and how to keep content private.

For where this fits in the run, see the Enrich stage reference and the pipeline overview.

What enrichment produces

Enrich reads the chunks and documents already in the SpaceContext and writes structured metadata back onto them:

Field	Lands on	Example
type	Document.type, Source.type	"policy", "guide"
topics	Document.topics, Chunk.metadata.topics	["retention", "compliance"]
tags	Document.tags	["gdpr", "data"]
summary	Document.summary, Chunk.metadata.summary	"Defines the 90-day retention rule…"

The LLM handles all four fields. The VLM contributes descriptions of images, diagrams, and scanned pages that feed into the same fields.

The default enricher is a deterministic, offline scaffold (LLM-free): it derives type, topics, tags, and summary from cheap, reproducible signals, so the zero-config path never makes a model call. A swappable LLM-backed enricher processes each item and skips on failure: if one document’s LLM call times out, that document is skipped by appending a StageErrorRecord(kind="skip") to the run context and the run continues. Use --strict to promote any such skip to a fatal StageError instead.

Cloud default, local available

The default LLM is openai:gpt-5-mini, the cloud-backed path. For air-gapped enrichment, switch to ollama:qwen2.5 or disable the LLM with none. See the local & air-gapped guide.

The LLM slot

indx ships a thin per-provider adapter behind a single LLM protocol. The core package depends on no LLM SDK — each backend’s client installs only via an extra. The protocol is small:

@runtime_checkable
class LLM(Protocol):
    """Text generation for enrichment (type, topics, tags, summaries).
    Default: openai:gpt-5-mini."""
    def complete(self, prompt: str, *, system: str | None = None,
                 max_tokens: int = 512, temperature: float = 0.0) -> str: ...

Choosing a backend

Name string	Backend	Extra	Best for
openai:<model> (default openai:gpt-5-mini)	OpenAI	indx[openai] + OPENAI_API_KEY	Cloud default for text enrichment.
ollama:<model> (ollama:qwen2.5 in the local profile)	Ollama	— (local runtime)	Local, no key, air-gapped enrichment.
vllm:<model>	vLLM	local serving	High-throughput / GPU-server deployments.
anthropic:<model>	Anthropic	indx[anthropic]	Cloud alternative with long context.
azure:<model>	Azure OpenAI	indx[openai]	OpenAI models through Azure governance.
litellm:<model>	LiteLLM	indx[litellm]	Opt-in unified backend routing 100+ providers through one adapter.
none	—	—	Disable enrichment entirely.

The name string carries an optional :model suffix. The base name selects the adapter; the suffix selects the model. So ollama:qwen2.5, openai:gpt-4o-mini, and anthropic:claude-3-5-haiku are all valid.

Why adapters, not one client

A mandatory unified client would bloat the core and tie indx’s release cadence to a third party. Adapters keep the core SDK-free.

The litellm backend stays available as an opt-in escape hatch for teams already standardized on it. See the rationale in the design principles.

Selecting the LLM

On the CLI, use --llm:

# Default: cloud openai:gpt-5-mini (needs OPENAI_API_KEY)
indx ./docs --out ./ai-ready

# Local path — no key, air-gapped
indx ./docs --out ./ai-ready --llm ollama:qwen2.5

# Switch to a different cloud provider/model (key via env var, never the config file)
export INDX_LLM__API_KEY="sk-…"
indx ./docs --out ./ai-ready --llm openai:gpt-4o-mini

# Turn enrichment OFF
indx ./docs --out ./ai-ready --llm none

In indx.toml, the slot lives under [enrich]:

[enrich]
llm = "openai:gpt-5-mini" # name[:model] or "none". Default: "openai:gpt-5-mini".

In the SDK, pass llm= to the pipeline (or drop the stage outright):

from indx import DirectoryPipeline

# Named backend
space = DirectoryPipeline(llm="anthropic:claude-3-5-haiku").run("./docs", "./out")

# Disable enrichment, two equivalent ways:
DirectoryPipeline(llm="none").run("./docs", "./out")
DirectoryPipeline().drop("enrich").run("./docs", "./out")

--llm none and drop("enrich") both produce a fully valid knowledge space. The graph, chunks, neighbors, relations, and embeddings are all still built. Only the LLM-derived type, topics, tags, and summary are left empty.

The VLM slot

The VLM slot adds vision-language descriptions of figures, diagrams, scans, and screenshots — the images the parser captured into ParsedDoc.images. It defaults to none, so image-description latency and cost stay opt-in.

@runtime_checkable
class VLM(Protocol):
    """Vision-language enrichment for images/layout. Default: none (disabled)."""
    def describe(self, image: bytes, *, prompt: str | None = None) -> str: ...

Name string	Backend	Enable for
none (default)	disabled	Text-only corpora; fastest, cheapest.
qwen-vl	local Qwen-VL	Local image understanding, no egress.
gpt4o (model defaults to gpt-4o; pin with gpt4o:<model>)	cloud (OpenAI)	High-quality figure/diagram descriptions.
<local adapter>	other local VLM	Any installed vision model.

Enable it with --vlm or the [enrich].vlm key:

indx ./docs --out ./ai-ready --vlm qwen-vl

[enrich]
vlm = "qwen-vl"   # name or "none". Default: "none".

DirectoryPipeline(llm="ollama:qwen2.5", vlm="qwen-vl").run("./scans", "./out")

When a VLM pays off

Turn the VLM on when meaning lives in the pixels: architecture diagrams, charts, screenshots, or scanned/OCR-resistant PDFs. For clean text documents it adds cost without adding signal.

Controlling which metadata is produced

The [enrich].metadata key selects which of the four enrichments to compute. It defaults to all four — trim it to save time and tokens.

[enrich]
llm      = "openai:gpt-5-mini"
metadata = ["type", "topics", "summary"]   # skip "tags"

The allowed values are exactly the subset ["type", "topics", "tags", "summary"]. A shorter list narrows the work. Setting llm = "none" skips the stage altogether.

Determinism and concurrency

Enrichment is built to be reproducible and well-behaved against rate limits:

• temperature=0.0 by default. The LLM.complete signature defaults temperature to 0.0, so reruns are as stable as the provider allows. See reproducibility for the full determinism story.
• Provenance is recorded. Some providers aren’t bit-reproducible. To keep runs auditable, the resolved config snapshot is written into index.json.metadata and the archive manifest.json. That snapshot includes the model name.
• No model calls in the default scaffold. The shipped enricher is a single-pass, deterministic, LLM-free stage: it derives topics, tags, summary, and type from cheap local heuristics, so it issues no network requests and has no concurrency or rate-limit knobs. An LLM/VLM-backed enricher is a swappable stage; parse-time parallelism (the --jobs worker pool, default os.cpu_count()) applies to the parse stage, not enrich. See performance.

Privacy: cloud LLMs egress

The default stack is cloud-backed, so the default Enrich run sends chunk text to OpenAI and requires OPENAI_API_KEY.

Several backends egress during Enrich:

• The cloud LLMs openai, anthropic, and azure transmit chunk text to the provider.
• A cloud VLM like gpt-4o transmits images as well.

To keep everything on-device, switch to the local profile: a local parser, the ollama:qwen2.5 LLM, a local embedder, and a local/JSONL store, all running fully air-gapped.

Redact before you enrich

If you must use a cloud LLM or VLM over sensitive content, insert a redaction stage before Enrich. Secrets are then stripped before any egress-capable component sees them. Redaction is a first-class extension point.

from indx import DirectoryPipeline, SpaceContext

class PiiRedactStage:
    name = "pii-redact"
    def run(self, ctx: SpaceContext) -> SpaceContext:
        for chunk in ctx.chunks:
            chunk.text = redact(chunk.text)
        return ctx   # MUST return the same context

pipeline = DirectoryPipeline(llm="openai:gpt-4o-mini")
pipeline.insert(3, PiiRedactStage())   # lands after Chunk (index 2) and before Relate (which shifts from index 3 to 4) — and therefore before Enrich
space = pipeline.run("./docs", "./out")

Secrets themselves stay out of the config file. API keys come from environment variables such as INDX_LLM__API_KEY, and indx never logs or serializes them.

See custom stages for the full redaction recipe.

Next steps

• Enrich stage reference — exactly what stage 05 reads and writes.
• Component protocols — the full LLM and VLM contracts.
• Extras — which pip install indx[...] each backend needs.
• Reproducibility — determinism with cloud and local models.
• Bring your own stack — swap any slot without a rewrite.