The Pipeline & Stages

DirectoryPipeline is the engine that turns a directory into a KnowledgeSpace. It is an ordered list of stages you can inspect, extend, and trim. The CLI, SDK, and archive format all expose the same pipeline model.

One pipeline, one context

A pipeline is an ordered set of stages. Each stage receives and returns one shared, mutable object: the SpaceContext. Every stage obeys the same contract:

def run(self, ctx: SpaceContext) -> SpaceContext: ...

A stage returns the same SpaceContext instance it received, mutated in place. Stages never communicate through globals or side channels, only through the context.

That single rule gives every stage a clear view of the work done so far, and makes insertion, replacement, and removal predictable.

The mental model

DirectoryPipeline owns orchestration. The heavy lifting lives elsewhere: parsing, generation, embedding, and storage run in swappable components behind typed protocols. Stages decide when work happens; components decide how that work is performed.

The six stages

A fresh DirectoryPipeline registers six built-in stages in canonical order. Each stage drives a component slot, or is fully built-in:

#	Stage (name)	One-line responsibility	Component slot
01	walk	Traverse the folder/ZIP, build the directory graph, detect file types	— (built-in)
02	parse	Run each file through the chosen parser → text, tables, layout, images	Parser
03	chunk	Split content with structure intact; keep lineage and neighbor links	— (built-in)
04	relate	Resolve references, siblings, parents, duplicates → typed Relations	— (built-in)
05	enrich	LLM/VLM add detected type, topics, tags, and summaries	LLM, VLM
06	embed-pack	Vectorize chunks, write to the store, seal the .indx archive	Embedder, Store, OutputWriter

One slot, five names

The output-writer slot wears a different label in each surface, but it is all the same thing. The names are: CLI --format, config [output], SDK output=, the OutputWriter protocol, and the writer field on SpaceContext. The entry-point group is indx.outputs.

For a deep dive into the inputs, outputs, and tuning of each stage, see the pipeline reference and the per-stage pages: Walk, Parse, Chunk, Relate, Enrich, and Embed+Pack.

How the context accumulates

The SpaceContext carries the inputs (root, out, config), the resolved components, and a set of collections that fill up phase by phase. Each stage appends to the collection relevant to its work. Later stages read what earlier stages produced:

01 Walk        → ctx.space.documents_  (one Document per file, with lineage + detected type)
02 Parse       → ctx.parsed            (doc_id → ParsedDoc)
03 Chunk       → ctx.space.chunks      (retrievable units, with prev_id/next_id links)
04 Relate      → ctx.space.relations   (typed edges between docs/chunks)
05 Enrich      → enriches ctx.space.documents_ (adds doc_type, topics, tags, summary)
06 Embed+Pack  → sets Chunk.embedding on ctx.space.chunks + sealed .indx manifest

The document list (ctx.space.documents_) is built up across stages 01–05, and chunk vectors land last. The context already holds a live KnowledgeSpace (ctx.space) that the stages fill in place:

documents_ → parsed → chunks → relations → chunk embeddings  ⇒  ctx.space (KnowledgeSpace)

That arrow diagram shows how collections nest inside ctx.space (a KnowledgeSpace). It is not a second execution order. Stages still run 01–06, and ctx.space.documents_ is refined in place across stages 01–05 rather than created in one step.

Per-item failures collected along the way are recorded on ctx.errors and surfaced on the result under space.metadata["errors"].

Stages are addressable by name

Every stage has a stable name: walk, parse, chunk, relate, enrich, embed-pack. Because stages are an ordered list keyed by name, you can reshape the pipeline with a small API on DirectoryPipeline:

Method	What it does
stages()	Return the current ordered stage list.
insert(index, stage)	Insert a custom stage at a 0-based position.
append(stage)	Add a stage to the end.
replace(name, stage)	Swap out the stage with the given name.
drop(name)	Remove the named stage entirely.

These return the pipeline for chaining. Components, not stages, are swapped separately with use(parser=..., llm=..., store=...). See Bring your own stack.

from indx import DirectoryPipeline
# PiiRedactStage and MyChunker are user-defined — see /guides/custom-stage/.

pipeline = (
    DirectoryPipeline(embedder="bge-m3", store="chroma")
    .drop("enrich")                  # skip all LLM work
    .insert(3, PiiRedactStage())     # after Chunk, before Relate and Enrich
)
space = pipeline.run("./notes", "./out")

Common reshapes

• drop("enrich") — skip LLM/VLM work entirely. A fully supported, common operation. Useful when no model is available, or when you only need the structural graph of folders, neighbors, and relations without topics, tags, or summaries.
• drop("embed-pack") — produce a graph-only space with no vectors. Helps when a downstream store self-embeds, or when you want to inspect structure before committing to an embedder. The CLI exposes the same intent as --no-embed.
• insert(i, stage) — add a custom pass. For example, redaction before Enrich so sensitive content is stripped before any egress-capable component sees it, or a deduplication step before Relate.
• replace("chunk", MyChunker()) — substitute a built-in stage with your own implementation.

Writing your own

A custom stage needs only a name attribute and a run(ctx) -> ctx method that returns the same context. See the Stage protocol for the exact contract, and Write a custom stage for a worked example.

Per-item skip vs. fatal errors

Not every failure should stop a build. indx distinguishes two kinds:

• Per-item (skip) — a single file fails to parse, or one document’s LLM call times out. The item is skipped, a skip-kind error is appended to ctx.errors, and the pipeline continues. This is the default behaviour for Parse and Enrich.
• Fatal — misconfiguration, an unreachable store, an unresolvable component name, or a stage raising an unhandled exception. The pipeline aborts and re-raises a wrapping error.

The --strict CLI flag, and strict=True in the SDK, promotes every skip into a fatal error, so any single failure aborts the run.

Either way, errors are visible. Nothing is silently swallowed, and ctx.errors ends up on space.metadata["errors"] for inspection.

Strict mode

Under --strict, a single unparseable file will fail the whole build with exit code 1. Use it in CI to catch regressions; leave it off for best-effort runs over messy, real-world directories.

Why this design

Because the pipeline is an ordered list and every stage shares one typed contract, the same code can run on a laptop, CI worker, or air-gapped server. Swap cloud components for local ones without rewriting orchestration.

Stages stay replaceable, components stay selectable by name or object, and the output remains deterministic and portable.

Next steps

• Core objects — the data models (KnowledgeSpace, Document, Chunk, Relation) the context accumulates.
• Bring your own stack — swap the component behind any slot.
• Pipeline overview — per-stage inputs, outputs, and tuning.
• Stage protocol and Write a custom stage — build your own.