Concepts Overview

This page gives you the mental model behind indx. There are four core objects, one shared context, and a pipeline of replaceable stages. Learn those pieces first; the CLI, SDK, and archive format then become different views of the same system.

The four ideas are KnowledgeSpace, Document, Chunk, and Relation. The CLI, the SDK, and the output format are all different views onto these same objects.

The whole idea in one sentence

indx turns a directory into a knowledge space. It walks a folder, parses each file with the parser you choose, models how files, folders, and chunks relate, and writes the result in a layout built for retrieval.

That positioning shapes every concept below. indx composes parsers rather than replacing them:

• A parser answers “what does this one file say?”
• indx answers the question agents actually need: “how does this body of knowledge fit together, and what context should I carry with this result?”

The four core objects

These four Pydantic models are the entire data model you need to be productive. Field-by-field tables live in the data models reference. This page stays conceptual.

KnowledgeSpace is the top-level result of processing a directory. It holds the document graph, the chunks, the embeddings, and the build metadata, and serializes to a single portable .indx archive.

It is also a first-class object: inspect its stats, filter its documents(type=...), and run search(query, k=5) against it. It is what DirectoryPipeline.run(...) returns, and what KnowledgeSpace.load(path) reconstructs.

Document is one source file, enriched. Beyond the original path, a Document carries its folder lineage (root-to-leaf ancestry), a detected type, and the LLM-derived topics, tags, and summary.

It also records the references it points to and the references that point back to it. Those file-to-file edges are what most pipelines throw away at parse time.

Chunk is a retrievable unit that never forgets where it came from. Every Chunk keeps its source provenance, its parent document id, its 0-based position within that document, the ids of its immediate neighbor chunks, and any typed relations.

Because that context travels with the text, retrieval returns something an agent can reason over rather than an orphaned fragment.

Relation is a typed, directed edge in the knowledge graph. A Relation can connect a document to a document, a chunk to a chunk, or a chunk to a document. The five RelationType values are the structure indx works to preserve:

| Type | Meaning | |------|---------| | sibling | Same folder / same logical group | | parent | Folder lineage / containment | | references | Outgoing citation, link, or mention | | continues | Next unit in a split sequence | | duplicate-of | Near or exact duplicate content |

Two supporting ideas

Two more concepts round out the model:

• SpaceContext is the carrier — a single mutable object that flows through the pipeline and accumulates work stage by stage (see below).
• Slot is a swappable, typed component interface with a default. The six slots — parser, LLM, VLM, embedder, store, and output — are how indx stays vendor-free. See Bring your own stack.

A chunk that remembers everything

Here is the canonical chunk as it appears in index.json. It is the clearest single illustration of why indx exists. Almost every concept above shows up in this one object.

{
  "id": "chunk_0481",
  "doc_id": "doc_0007",
  "position": 12,
  "text": "Enterprise data is retained for 90 days…",
  "prev_id": "chunk_0480",
  "next_id": "chunk_0482",
  "source": {
    "path": "policies/data/retention.pdf",
    "folder": "policies/data",
    "type": "policy"
  },
  "metadata": {
    "topics": ["retention", "compliance"],
    "summary": "90-day retention rule…",
    "tags": ["data-retention", "gdpr"]
  },
  "relations": [
    { "src": "chunk_0481", "dst": "legal/gdpr.md", "type": "references", "score": 1.0 }
  ]
}

Walking through what each field shows:

• id is a stable, zero-padded identifier (chunk_0481). Ids are assigned in a deterministic traversal order, so rerunning over unchanged input yields identical ids. That is what makes a knowledge space diffable and reproducible.
• doc_id and position are the parent document id (doc_0007) and the chunk’s 0-based position within that document (12). Together they pin every chunk back to its source document and its place in the original reading order.
• text is the retrievable payload. This is the part a file-level parser would give you on its own. Everything around it is what indx adds.
• source is provenance. The chunk knows its original path, its containing folder, and the detected document type. An agent can filter by location (“only chunks under policies/”) or by type (“only policy documents”) without guessing.
• metadata is the semantic layer added during Enrich: topics, summary, and tags. This is what lets retrieval reason instead of pattern-matching raw text.
• prev_id and next_id are the ids of the adjacent chunks (chunk_0480, chunk_0482). Context travels with the chunk, so an agent can expand the window around a hit instead of retrieving it in isolation. (The Python Chunk object also exposes convenience read-properties index and neighbors, but those are not keys in index.json.)
• relations are typed edges leaving this chunk. Here a references edge points to legal/gdpr.md. This is the structure most pipelines discard. It is the reason an agent can follow knowledge rather than just match it.

Embeddings live elsewhere

Vectors are not inlined in index.json. They live in the embeddings/ directory, and inside the .indx archive, as a float32 matrix. This keeps the graph human-legible and the vectors memory-mappable. See the archive reference.

One shared context, a pipeline of stages

Under the hood, a DirectoryPipeline is an ordered set of six stages with a single shared object flowing through them. That object is the SpaceContext. Every stage obeys the contract run(ctx: SpaceContext) -> SpaceContext and returns the same object it received, mutated in place.

Each stage appends its own work to the context:

• Walk appends the directory graph.
• Parse appends ParsedDocs.
• Chunk appends Chunks.
• Relate appends Relations.
• Enrich fills in metadata.
• Embed+Pack adds vectors and seals the archive.

The six ordered, replaceable stages are:

01 Walk → 02 Parse → 03 Chunk → 04 Relate → 05 Enrich → 06 Embed+Pack

The pipeline is a list you control. Insert a custom stage, such as a redaction pass before enrichment, or drop a stage entirely, such as skipping Enrich when no LLM is available. The full walkthrough lives in Pipeline and stages and the pipeline overview.

Where to go next

Core objects, in depth KnowledgeSpace, Document, Chunk, Relation, and the supporting types — how they connect into one graph.

The pipeline and stages How the six stages share one SpaceContext, and why every stage is replaceable.

Bring your own stack The Slot model: swap parser, LLM, VLM, embedder, store, and output by name or by object.

If you would rather see the model in action first, two pages help:

• The quickstart builds and queries a real knowledge space in under a minute.
• Your first knowledge space walks through the output directory file by file.