06 · Embed + Pack

Embed+Pack is the final stage. It vectorizes every chunk through the Embedder slot, writes those vectors into the Store, materializes the accumulated SpaceContext into a KnowledgeSpace, and asks the OutputWriter to write the portable .indx archive plus the expanded on-disk layout.

At a glance

This is the only stage that uses three swappable slots at once — Embedder, Store, and OutputWriter. It is also the stage behind --no-embed, which produces a graph-only space with no vectors.

What this stage does

By the time the context reaches stage 06, earlier stages have already produced the document graph, chunks, relations, and any enrichment metadata. Embed+Pack then performs two steps:

1. Embed — collect chunk.text for every chunk, pass them to Embedder.embed(texts), and set each Chunk.embedding on the live ctx.space.
2. Upsert — write the embedded chunks into the active store via Store.upsert(chunks), then record the embedder name and dim on the space manifest (manifest.embedding_model / manifest.embedding_dim).

Serialization happens after the stage, not inside it: when an out path is passed to DirectoryPipeline.run(...), the chosen OutputWriter.write(space, out) writes index.json, the per-chunk files under chunks/, the embeddings/ vector matrix (built directly from each Chunk.embedding), and the sealed .indx archive. The space itself is the live ctx.space field — there is no separate to_space() materialization step, and the store is not asked to persist into the output.

ctx.space.chunks ──► Embedder.embed(batch) ──► Store.upsert(chunks) ──► manifest.embedding_model/dim
                                                                              (stage 06 ends here)

# afterward, only when run(src, out) is given an out path:
OutputWriter.write(ctx.space, out) ──► handbook.indx + expanded layout (index.json, chunks/, embeddings/)

The three component protocols

Embed+Pack is defined entirely by these protocols. Each is a typed Protocol with a named default. For the complete set, see the protocols reference.

Embedder — text to vectors

@runtime_checkable
class Embedder(Protocol):
    """Turns text into vectors. Default: openai:text-embedding-3-small."""
    dim: int
    def embed(self, texts: list[str]) -> list[list[float]]: ...

The default is openai:text-embedding-3-small, a cloud embedder with 1536-dimensional vectors. The dim attribute is pinned into the archive manifest so a consumer can detect a dimension mismatch before querying.

For other embedding paths:

• Local semantic — bge-m3 from the local profile.
• Zero-dependency offline — --offline, which selects the lexical hash embedder.

To compare BGE-M3, E5, OpenAI, and Cohere, see choosing an embedder.

Store — the vector database adapter

@runtime_checkable
class Store(Protocol):
    """Vector database adapter. Default: qdrant.
    Also: pgvector, chroma, lancedb, jsonl."""
    name: str
    def upsert(self, chunks: list[Chunk]) -> None: ...
    def search(self, vector: list[float], k: int = 5) -> list[SearchHit]: ...
    def delete(self, chunk_ids: list[str]) -> None: ...

The default is qdrant. It runs embedded (in-process and on-disk) or against a server — the client code is the same either way. Alternatives are pgvector, chroma, lancedb, and the jsonl store.

The three methods divide the work:

• upsert ingests embedded chunks during the run (it takes the Chunk objects, whose .embedding is already populated).
• search powers space.search(...) and indx query, returning a list of SearchHit objects in descending score order.
• delete removes chunks from the store by id.

See choosing a store.

OutputWriter — serializing the space to disk

@runtime_checkable
class OutputWriter(Protocol):
    """Serializes a KnowledgeSpace to disk. Default: indx (.indx archive).
    Also: jsonl, langchain, llamaindex."""
    name: str
    def write(self, space: KnowledgeSpace, dest: Path, *, name: str = "handbook") -> None: ...

The default writer’s identity attribute is name (e.g. indx, jsonl); the indx writer seals the .indx archive. Alternatives emit jsonl shards or adapters for langchain / llamaindex. The sealed archive’s base filename defaults to handbook via the name= parameter (<name>.indx). See output formats.

Concurrency: batching beats parallelism

Parse (stage 02) fans out across files in a worker pool. Embedding instead relies on batching — the single biggest performance lever for stage 06, whether the embedder is local or an API.

Local embedders are far more efficient on batches, thanks to GPU/CPU vectorization, and store upserts are batched to amortize round-trips.

Param	Default	Notes
Embed batch size	64	Chunk texts are grouped into batches of 64 and submitted to Embedder.embed(list[str]).
Embed max concurrency	--jobs	Network-bound embedders use a bounded concurrency limit; local ones lean on batch size.
Store upsert	batched	Vectors are written in batches to amortize round-trips.

Store upserts are batched in lockstep so vectors land as soon as each batch is embedded. Batch size is overridable via adapter sub-tables / kwargs.

Tuning throughput

For the local profile, raising the batch size usually beats raising --jobs. For a network-bound embedder or store, the bounded concurrency limit is tied to --jobs and protects you from provider rate limits. More in the performance guide.

The on-disk output

Running indx ./docs --out ./ai-ready writes both the sealed archive and an expanded layout beside it. Downstream tools can read either form:

ai-ready/
├── handbook.indx        # the portable archive (a ZIP container)
├── index.json           # the knowledge graph
├── chunks/              # agent-readable chunks + per-chunk context
└── embeddings/          # vectors + manifest

Inside handbook.indx (a deflate ZIP), the layout is:

handbook.indx
├── manifest.json            # archive metadata + sha256 checksums
├── index.json               # the knowledge graph
├── chunks/
│   ├── chunk_0000.json
│   └── …
└── embeddings/
    ├── manifest.json         # model, dim, count, backend
    └── vectors.f32           # contiguous little-endian float32 matrix (count × dim)

Embeddings are not inlined in index.json. They live in embeddings/ as a contiguous little-endian float32 matrix, memory-mapped on load.

The embeddings manifest pins the embedder name and dim, so a loaded archive can validate query-time compatibility. The full format is documented in the .indx archive reference.

Archives are self-contained

The OutputWriter materializes the embeddings/ layout from the chunk embeddings regardless of which backend ran the upserts. A space built against Qdrant or pgvector still seals a self-contained archive whose vectors travel with it — no live database is required to open it later. Store adapters implement only upsert / search / delete; there is no persist() method.

Skipping the stage: --no-embed

Pass --no-embed (CLI) or drop("embed-pack") (SDK) to skip vectorization entirely. The result is a graph-only space: documents, chunks, relations, and enrichment, but no vectors.

indx ./docs --out ./graph-only --no-embed

pipeline = DirectoryPipeline().drop("embed-pack")
space = pipeline.run("./docs", "./graph-only")

A graph-only space cannot answer space.search(...) or indx query. It is still useful when you only need the directory graph, relations, and metadata, or when you plan to embed later with a different model.

Dimension and model identity are pinned

The embedder dim and identity are recorded in both index.json.metadata and the archive manifest. If the embedder changes, re-embedding is required. A query against vectors from a different model is a compatibility error, not a silent mismatch. See reproducibility.

A minimal run

# default cloud stack: requires OPENAI_API_KEY
from indx import DirectoryPipeline

pipeline = DirectoryPipeline(
    embedder="openai:text-embedding-3-small",   # dim 1536
    store="qdrant",
    output=".indx",
)
space = pipeline.run("./docs", "./ai-ready")

print(space.stats.embeddings, space.stats.embed_dim)   # e.g. 1042 1536

for hit in space.search("how long is data retained?", k=3):
    print(hit.score, hit.source.path)

Embed+Pack produces exactly one vector per chunk, so for a fully embedded space space.stats.embeddings == space.stats.chunks. A --no-embed run is the only case where they diverge: vectors drop to zero while chunks remain.

The progress summary for this stage looks like:

  06 embed   1042 vectors → qdrant, sealed handbook.indx
done: 1042 chunks, 128 docs, embed_dim=1536  (12.4s)

Where to go next

• The .indx archive — full container layout, manifest, sealing and loading.
• Choosing an embedder — bge-m3 vs. E5, OpenAI, Cohere.
• Choosing a store — Qdrant, pgvector, Chroma, LanceDB, JSONL.
• Output formats — .indx, JSONL, LangChain, LlamaIndex writers.
• Pipeline overview — how all six stages fit together.