The .indx Archive Format

A .indx file is the single portable artifact for a knowledge space — a self-describing container carrying everything a downstream tool needs: the knowledge graph, agent-readable chunks, and the vector matrix.

Under the hood it is an ordinary ZIP (deflate) with a defined internal layout and a manifest, so you can open it with unzip as readily as with KnowledgeSpace.load.

At a glance

Property	Value
Container	ZIP, deflate compression
Access	Random access to individual members (no full decompress)
Default base name	handbook → handbook.indx (set with --name)
Produced by	The default indx writer (IndxWriter, stage 06), or space.save(path)
Opened by	KnowledgeSpace.load(path), indx inspect, indx query
Integrity	Per-member SHA-256 checksums in a dedicated checksums.json member
Versioning	schema_version (on-disk layout); indx_version (producing build)

Internal layout

A sealed archive contains the manifest, the serialized graph, and a per-member integrity record. The graph is split into newline-delimited members for documents, chunks, and relations.

handbook.indx                 (ZIP container)
├── manifest.json             # archive metadata (Manifest model)
├── documents.jsonl           # one Document per line
├── chunks.jsonl              # one Chunk per line (embeddings included)
├── relations.jsonl           # one Relation per line
└── checksums.json            # per-member SHA-256 digests

A few details make the format predictable:

• documents.jsonl, chunks.jsonl, and relations.jsonl are the serialized knowledge graph, one JSON object per line. They hold the Document, Chunk, and Relation models respectively. JSON keys are sorted and member order is fixed, so the same space produces a byte-identical archive across runs and machines.
• chunks.jsonl carries each chunk’s full record, including its embedding vector. The expanded on-disk layout (see below) instead splits chunks into per-file form under chunks/ and stores vectors separately under embeddings/. Inside the sealed archive they stay together in chunks.jsonl.
• checksums.json records { "algo": "sha256", "members": { <member>: <hex digest>, … } }. It is written last and is the one member it does not cover, since it carries no self-digest. A reader can therefore detect corruption or tampering of every other member on load.

Two artifacts, one writer

The sealed .indx container described here is what space.save(path) and single-file mode write. Running a build to an output directory also writes the expanded layout (index.json, chunks/, embeddings/) alongside the archive. See The expanded on-disk layout below.

manifest.json

The top-level manifest.json holds the archive’s metadata, serialized from the Manifest model. Read it first to learn what an archive contains and whether you can open it. Integrity digests live separately in checksums.json.

{
  "schema_version": "1",
  "indx_version": "0.1.0",
  "source_root": "/abs/path/docs",
  "components": {
    "parser": "plaintext",
    "llm": "none",
    "embedder": "openai:text-embedding-3-small",
    "store": "qdrant"
  },
  "embedding_model": "openai:text-embedding-3-small",
  "embedding_dim": 1536
}

Field	Type	Meaning
schema_version	string	On-disk layout version. Bumped only on a breaking layout change; readers check it.
indx_version	string	The producing indx build (e.g. 0.1.0). Recorded for diagnostics/auditing.
source_root	string	Path of the walked directory/ZIP.
components	object	The resolved slot → name map (parser, llm, vlm, embedder, store, …).
embedding_model	string | null	The embedder name pinned for the run.
embedding_dim	int | null	Vector dimensionality, so a reader can validate compatibility before querying.

Self-describing vectors

Because embedding_model and embedding_dim are pinned in the manifest, a consumer can detect “these vectors came from a different model” before querying. It can reject a query embedded with a mismatched embedder rather than return garbage results.

checksums.json

Integrity is a dedicated member, not a manifest field, so the record can name manifest.json itself without a self-referential digest:

{
  "algo": "sha256",
  "members": {
    "manifest.json": "…",
    "documents.jsonl": "…",
    "chunks.jsonl": "…",
    "relations.jsonl": "…"
  }
}

Sealing and loading

Sealing

Sealing happens in stage 06 (Embed+Pack) via the indx writer (IndxWriter), or explicitly through space.save(path). Sealing the container:

1. Writes manifest.json — the serialized Manifest model (with schema_version set).
2. Writes documents.jsonl, chunks.jsonl, and relations.jsonl — the graph, one JSON object per line, with sorted keys and fixed member order.
3. Computes a SHA-256 digest of each of those members and writes them into checksums.json (the last member, with no self-digest).
4. Deflates everything into the .indx ZIP container with zeroed timestamps, so the same space yields a byte-identical archive.

Loading

KnowledgeSpace.load(archive) reverses the process, with verification gates:

1. Verify schema_version compatibility — see Versioning below. An incompatible layout fails fast.
2. Validate checksums — each member is checked against the SHA-256 digest recorded in checksums.json; a mismatch is an archive error.
3. Reconstruct the in-memory models — the Manifest plus the Document, Chunk, and Relation Pydantic v2 models are rebuilt from the JSONL members (write and read share one schema, so the data validates on load).

Checksum and archive failures

A corrupt or checksum-mismatched archive surfaces as an archive error (CLI exit code 4). A nonexistent archive path is a usage error (2), validated before the command runs. See errors and exit codes.

Versioning

.indx archives are versioned through two fields, and only one of them gates compatibility.

schema_version identifies the on-disk layout and controls whether an archive can be loaded. It is bumped deliberately on a breaking change to the member set or format; a reader checks it on load and rejects an unrecognized layout. Within a layout version, fields are only added, never removed or retyped, and consumers ignore unknown values rather than fail.

indx_version records the producing build (e.g. 0.1.0) for diagnostics and auditing. It never affects whether an archive loads.

For an end-to-end view of what makes a rebuild byte-identical, see reproducibility.

Auditability

The resolved config snapshot — the components map plus embedding_model and embedding_dim — is recorded in the archive manifest.json (and mirrored into index.json.metadata in the expanded layout), so you can always reconstruct exactly how an archive was produced even when an upstream model is not bit-reproducible.

The expanded on-disk layout (unsealed)

Running a build writes the expanded layout alongside the sealed archive, so downstream tools can read either the portable container or the loose files directly — whichever is more convenient.

indx ./docs --out ./ai-ready

ai-ready/
├── handbook.indx        # the portable archive (sealed)
├── index.json           # the knowledge graph (version, root, metadata, stats, documents, chunks, relations)
├── chunks/              # one chunk_NNNN.json per chunk (embedding excluded)
└── embeddings/
    ├── vectors.f32      # contiguous little-endian float32 matrix (count × dim)
    └── manifest.json    # { model, dim, count, backend }

The loose files are a convenience projection of the same space; the sealed handbook.indx is the canonical artifact. A few details:

• index.json is the full serialized graph plus a stats block; embeddings are never inlined here — they live under embeddings/. Its schema is documented in the index.json reference.
• chunks/chunk_NNNN.json holds one file per chunk, named by its zero-padded index (chunk_0000, chunk_0001, …), with the embedding excluded.
• embeddings/vectors.f32 is a raw, contiguous matrix of little-endian float32 values laid out row-major as count × dim — no delimiters or headers; its shape comes from the embeddings manifest.
• embeddings/manifest.json records { model, dim, count, backend } — the embedder name, vector dimensionality, vector count, and the store backend — so a reader can validate and reshape vectors.f32 without guessing.

The .indx file is what you ship or version-control as a single unit; the expanded form is handy for grepping, diffing, or wiring into a tool that wants plain files.

Why ZIP (and not tar.gz or SQLite)

The container choice is deliberate, and serves the goal of an open artifact with no lock-in:

• Random access to individual members. ZIP lets a consumer read just manifest.json, or just one chunk, without decompressing the whole file — something a streamed tar.gz cannot do.
• Stdlib and cross-platform. ZIP is handled by Python’s stdlib zipfile, so reading and writing archives adds no dependency to the light core.
• Inspectable with ubiquitous tooling. Anyone can unzip handbook.indx and read the JSON by hand — important for an artifact meant to be a public contract.

SQLite was considered (single-file and queryable) but is opaque to non-SQLite tooling and would couple the artifact to a query engine. So SQLite remains a store option rather than the archive format.

The trade-off accepted for ZIP is slightly less compression efficiency for many tiny files than a solid tar.gz stream — a fair price for random access and tooling ubiquity.

Fully portable, fully offline

A .indx built with the zero-dependency jsonl store is completely self-contained: vectors travel inside the archive and it opens with no external service. See local and air-gapped usage.

See also

• The index.json schema — the full structure of the knowledge graph inside the archive.
• Inspect and query an archive — indx inspect and indx query over a .indx file.
• Output formats — .indx vs jsonl, langchain, and llamaindex writers.
• Reproducibility — what makes a rebuilt archive byte-identical.