05 · Enrich

Enrich is the fifth pipeline stage. It adds AI-ready metadata — detected type, topics, tags, and summary — to every document using a deterministic, fully-offline scaffold: topics from term frequency, a summary from lead text, and a type derived from file extension, folder lineage, and content cues. It runs with no LLM call and no network egress, which keeps the zero-config path reproducible and air-gapped. A real LLM-backed enricher is a swappable Stage, not wired in by default.

Two things set this stage apart:

• It is the only stage with LLM and VLM component slots bound for provenance, even though the default scaffold does not call them.
• It is fully optional — you can drop it entirely for a graph-only run with no LLM calls.

The [enrich] config still records an llm (default openai:gpt-5-mini) and vlm (default none) for provenance and forward compatibility, but the default scaffold enricher does not invoke them — its output is identical whether the configured LLM is cloud, local, or none.

What Enrich produces

Enrich reads the Documents and Chunks assembled by earlier stages from the shared SpaceContext, then writes its results back onto the same context — following the stage contract run(ctx: SpaceContext) -> SpaceContext.

The enrichments land on each Document.

Field	Where it lands	Description
type	Document.type (and Source.type)	Detected/refined document type, e.g. policy, guide, table.
topics	Document.topics	Salient subjects covered by the document.
tags	Document.tags	Short, keyword-style labels for filtering.
summary	Document.summary	A concise natural-language summary.

Choose only what you need

Every enrichment adds latency and, for cloud providers, cost. Keep the default metadata set when you want rich browsing and filtering. Trim it when you only need a summary or when the corpus is large.

A resulting Document looks like this in index.json:

{
  "id": "doc_0007",
  "path": "policies/data/retention.pdf",
  "type": "policy",
  "topics": ["retention", "compliance"],
  "tags": ["gdpr", "data"],
  "summary": "Defines the 90-day retention rule…"
}

See index.json and the data models for the full shape.

The components it uses

Enrich is the only stage that records the LLM and VLM component slots. They are swappable adapters behind typed protocols, reserved for a future LLM-backed enricher; the default scaffold does not call them. Swap them by name in indx.toml, or pass an instance via the SDK — see bring your own stack for the SDK path.

The LLM protocol

A future LLM-backed enricher would use the text model for type, topics, tags, and summaries; the default scaffold derives these deterministically.

@runtime_checkable
class LLM(Protocol):
    """Text generation for enrichment (type, topics, tags, summaries).
    Default: openai:gpt-5-mini."""
    def complete(self, prompt: str, *, system: str | None = None,
                 max_tokens: int = 512, temperature: float = 0.0) -> str: ...

The default openai:gpt-5-mini uses the OpenAI adapter. The name string carries an optional :model suffix:

• openai:gpt-5-mini selects the openai adapter with the gpt-5-mini model.
• ollama:qwen2.5 selects the local Ollama adapter.
• none resolves the null adapter, which skips text enrichment.

The VLM protocol

The optional vision-language model would describe images and layout captured during Parse and carried on ParsedDoc.images. It is reserved for a future LLM-backed enricher and is not invoked by the default scaffold; the VLM is off by default.

@runtime_checkable
class VLM(Protocol):
    """Vision-language enrichment for images/layout. Default: none (disabled)."""
    def describe(self, image: bytes, *, prompt: str | None = None) -> str: ...

When to enable a VLM

Turn the VLM on for image-heavy corpora such as slide decks, scanned figures, and diagrams, where a caption adds retrievable signal. Image description adds latency, plus cost for cloud VLMs — which is why it stays opt-in. See enrichment with LLM & VLM.

Choosing which enrichments run

The [enrich] section of indx.toml controls the models. The metadata key controls which enrichments are produced.

[enrich]
llm      = "openai:gpt-5-mini" # LLM name[:model] or "none"
vlm      = "none"             # VLM name or "none"
metadata = ["type", "topics", "tags", "summary"]

Key	Type	Default	Allowed values
llm	string	openai:gpt-5-mini	<name>[:model], none
vlm	string	none	<name>, none
metadata	list of strings	["type","topics","tags","summary"]	any subset of those four

Trim metadata to skip work you don’t need. For example, metadata = ["summary"] produces summaries only, saving LLM calls. The same values can be overridden on the CLI with --llm and --vlm. See the CLI reference.

Concurrency

The default scaffold enricher runs in-process with no model calls, so there is no concurrency knob; it is O(1) per document.

Determinism

Enrichment is built to be reproducible:

• Fully deterministic by construction. The scaffold derives metadata locally: term-frequency topics sorted by (-count, word), a lead-text summary, and a type from extension, folder lineage, and content cues. No model call, no randomness.
• Recorded provenance. The resolved llm/vlm names are recorded in index.json.metadata and the archive manifest for auditability even though the default scaffold does not call them.

Re-running with identical inputs, config, and component versions yields a byte-identical index.json (modulo the created_at timestamp). See reproducibility for the full determinism contract.

Privacy: a forward-looking note

The default scaffold enricher performs no network egress. If you replace it with a custom LLM-backed enricher pointed at a cloud provider, that swapped-in stage becomes your egress boundary — insert a redaction stage before it.

Redact before a cloud-backed enricher

If you swap in a cloud-backed enricher, insert a redaction stage before it, so sensitive content is stripped before any egress-capable component sees it. Stages are ordinary objects that mutate the shared context, so a custom stage slots cleanly into the pipeline:

class PiiRedactStage:
    name = "pii-redact"
    def run(self, ctx: SpaceContext) -> SpaceContext:
        for chunk in ctx.chunks:
            chunk.text = redact(chunk.text)
        return ctx            # MUST return the same context

# Lands after Chunk and before Relate and Enrich.
pipeline.insert(3, PiiRedactStage())

See writing a custom stage for the full recipe.

Running with no LLM

Enrich is fully optional. Drop the stage to omit the AI-derived metadata entirely; the pipeline still produces a complete knowledge space — document graph, chunks, relations, and embeddings.

from indx import DirectoryPipeline

pipeline = (
    DirectoryPipeline(embedder="openai:text-embedding-3-small", store="qdrant")
    .drop("enrich")
)
space = pipeline.run("./docs", "./ai-ready")

A future LLM-backed enricher would record a per-item failure — such as an LLM call that times out for a single document — as a skip error on ctx.errors and continue, with --strict promoting those skips to fatal. The current deterministic scaffold runs offline and has no such per-item failures.

Error handling

Per-item, non-fatal failures use the shared error channel ctx.errors — a list of StageErrorRecord entries on the SpaceContext, where a record with kind="skip" means one item was skipped without aborting the run. The Parse stage records such skip entries today (a per-file parse failure does not abort the build); a future LLM-backed enricher would do the same for a model call that fails on a single document. The current deterministic, LLM-free Enrich scaffold derives metadata locally and does not call out, so it has no per-item failures to record.

Misconfiguration, by contrast, is fatal and aborts before any stage runs — for example, an unknown LLM name. See errors and exit codes.

Next stage

With metadata attached, the context flows into the final stage, which vectorizes every chunk, writes it to the store, and seals the archive.

→ 06 · Embed + Pack