03 · Chunk

Stage 03 turns each ParsedDoc from Parse into ordered Chunks. Chunks are the retrievable units used by relation building, enrichment, embedding, and query.

Chunk is a built-in stage with no component slot, but it is still replaceable as a Stage. Its job is to split content while preserving provenance, position, and neighbor links.

What Chunk does

Chunk reads parsed documents from ctx.parsed (keyed by doc_id) on the shared SpaceContext, and writes Chunk objects to ctx.chunks. Like every stage it obeys run(ctx: SpaceContext) -> SpaceContext and returns the same mutated context. See the pipeline overview and protocols reference.

For each document it:

1. Splits content with structure intact. A ParsedDoc carries more than flat text. It has blocks (headings, paragraphs, list items), tables, and images. Chunk uses these structural artifacts to split along natural boundaries instead of slicing a raw character stream. A heading stays with its section, and a table is not cut mid-row.
2. Stamps provenance. Every chunk copies the document’s Source (path, folder, type). A chunk always knows which file and folder it came from.
3. Records position. Each chunk gets a doc_id for its parent Document and a 0-based index for its position within that document.
4. Links neighbors. Each chunk’s neighbors list holds the adjacent chunk ids (previous, next) within the same document. This adjacency forms the implicit continues sequence that Relate uses and that surfaces at query time.

Built-in, but not a black box

Chunk has no Parser-style protocol slot, but it is still a regular Stage. You can insert a custom stage before or after it, such as a PII-redaction pass. You can also replace the whole stage with your own chunking logic via pipeline.replace("chunk", MyChunkStage()). See custom stages.

The Chunk model

Chunk is a Pydantic v2 model. The fields populated at this stage:

Field	Type	Set by	Meaning
id	str	Chunk	Stable 16-char hex digest derived from the parent doc id and in-document position, e.g. a1b2c3d4e5f6a7b8.
text	str	Chunk	The retrievable text payload.
source	Source	Chunk	Originating document provenance (path, folder, type).
doc_id	str	Chunk	Id of the parent Document (also a stable 16-char hex digest).
index	int	Chunk	0-based position within the parent document.
neighbors	list[str]	Chunk	Adjacent chunk ids (prev, next).
metadata	dict[str, Any]	Enrich	Topics, summary, tags (added later).
relations	list[Relation]	Relate	Outgoing typed edges (added later).
embedding	list[float] or None	Embed+Pack	Vector (added later).

So Chunk owns id, text, source, doc_id, index, and neighbors; the remaining fields are filled by later stages. Each chunk records its parent via doc_id, so the chunks of a given document are recoverable by filtering on that id. Full field definitions live in the data models reference.

Stable, deterministic chunk ids

Chunk ids are stable, deterministic 16-character hex digests derived from the parent doc id and the in-document position (e.g. a1b2c3d4e5f6a7b8). They are assigned in a fixed, deterministic traversal order:

folder lineage → then path → then in-document index

The order never depends on parallelism or filesystem iteration quirks, so re-running indx over unchanged input yields identical chunk ids.

The chunk id is distinct from the on-disk filename: the per-chunk files written under chunks/ are named chunk_0000.json, chunk_0001.json, … by enumeration order (see src/indx/output/indx_writer.py), not by the chunk’s id.

This is a core part of indx’s reproducibility contract: the same inputs, config, and component versions produce a byte-identical index.json, modulo the created_at timestamp.

Why this matters

Stable ids let a --resume run skip already-chunked documents. Golden-file tests can assert byte-equality. External systems can reference a chunk’s id across rebuilds without churn. See Reproducibility for the full determinism story.

Earlier stages can run in parallel; parsing is embarrassingly parallel. That work is re-sorted into this deterministic order before ids are assigned, so concurrency never affects the resulting ids or their ordering.

Example chunk

After Chunk runs, a chunk for a retention policy looks like this. The metadata and relations fields are populated by later stages; the continues adjacency is captured first as neighbors.

{
  "id": "a1b2c3d4e5f6a7b8",
  "text": "Enterprise data is retained for 90 days…",
  "source": {
    "path": "policies/data/retention.pdf",
    "folder": "policies/data",
    "type": "policy"
  },
  "doc_id": "0f1e2d3c4b5a6978",
  "index": 1,
  "neighbors": ["9a8b7c6d5e4f3021", "b2c3d4e5f6a70819"],
  "metadata": {},
  "relations": []
}

Here this chunk is the second chunk (index: 1) of its parent document. Its neighbors point at the chunk before and after — the raw material for the implicit continues sequence.

At query time, space.search(...) returns each hit as a SearchHit. Its resolved neighbors give an agent a ready-made context window around the match.

Chunking concerns

How content is split is the part that most affects retrieval quality. The Chunk stage has to balance several concerns:

• Chunk size. Chunks must be small enough to be precise retrieval targets, yet large enough to stand alone as meaningful context.
• Overlap. A sliding overlap between adjacent chunks can preserve context across boundaries, at the cost of some redundancy.
• Structure-awareness. Respecting headings, paragraphs, list items, and table boundaries from the ParsedDoc keeps related content together. Splitting on raw length alone can break it apart.

Do not couple to exact boundaries

Chunk boundaries are an implementation detail. Size, overlap, and how aggressively structure overrides length may change as the default strategy improves. Depend on stable ids, provenance, neighbors, and relations, not on a specific character boundary. If your retrieval needs differ, replace the Chunk stage with a custom implementation. See custom stages and the performance guide.

Where chunks go next

02 Parse ──▶ 03 Chunk ──▶ 04 Relate ──▶ 05 Enrich ──▶ 06 Embed+Pack
            ctx.chunks    relations     metadata       vectors + .indx

• 04 Relate reads chunk adjacency to materialize continues edges and resolves references, sibling, parent, and duplicate-of relations.
• 05 Enrich attaches topics, tags, and summary to each chunk’s metadata.
• 06 Embed+Pack vectorizes chunk.text, writes vectors to the store, and seals the .indx archive, with one JSON file per chunk under chunks/.

See also

• Data models reference — the full Chunk, Source, and Relation definitions.
• Reproducibility — deterministic id assignment and byte-stable output.
• 04 Relate — how neighbor links become typed continues relations.