Performance & Scaling

This guide shows how to keep large runs fast and memory-stable. The key idea: tune the right lever at the right stage — parallelism for parsing, batching for embedding, and content-addressed caching for cheap re-runs.

indx is built to turn whole document estates into a knowledge space, not just a handful of files. It’s engineered against these headline targets:

• Time-to-first-space: under 60 seconds on a small directory (~10 docs) on a typical laptop with defaults.
• Scale: directories of 10k+ files processed crash-free in over 99% of runs. The estate is streamed rather than loaded whole into memory.
• Memory: a 2 GB folder must not require 2 GB of RAM. Files are processed as a stream.

The mental model: one lever per stage

The six stages (Walk → Parse → Chunk → Relate → Enrich → Embed+Pack) each have a different workload profile. There’s no single concurrency knob — indx applies a strategy suited to each one:

Stage	Workload	Strategy	Default
01 Walk	I/O, CPU	Single-pass; may parallelise per-folder traversal	—
02 Parse	Blocking native / CPU-bound	Worker pool across files (thread pool; process pool for GIL-bound parsers)	--jobs (CPU count)
03 Chunk	CPU-bound	Single-pass	—
04 Relate	CPU-bound	Single-pass	—
05 Enrich	Single-pass scaffold (offline, LLM-free by default)	Single-pass; a swappable LLM enricher is future work	—
06 Embed+Pack	Model + store I/O	Batched embed + batched Store.upsert	adapter batch_size (OpenAI 256, e5/bge-m3 12; store upsert 256)

Two principles run through this table:

• Parse is embarrassingly parallel across files. A worker pool of --jobs runs Parser.parse concurrently and merges results into the context keyed by doc_id. Parsers that hold native or GIL-bound resources, such as Docling, may run in a process pool instead of a thread pool.
• Batching beats parallelism for embeddings. Local embedders like bge-m3 are far more efficient on batches via CPU/GPU vectorisation, and vector-store upserts are batched to amortise round-trips. This is the single biggest performance lever, and it applies whether the embedder is local or a cloud API.

Today, the only stage-level parallelism is in Parse: a ThreadPoolExecutor of --jobs workers runs Parser.parse across files and merges results keyed by doc_id. Embedders amortise cost through internal request batching (e.g. the OpenAI embedder batches per request), not pipeline-level concurrency. Asyncio-based bounded concurrency and per-provider rate limiting for cloud calls are planned but not yet implemented.

Batching & concurrency parameters

These are the defaults; each is overridable via adapter sub-tables in indx.toml or component kwargs in the SDK.

Stage	Parameter	Default
Embed	batch size	adapter batch_size (OpenAI 256; e5/bge-m3 12)
Parse	workers	--jobs

--jobs (alias -j) defaults to the CPU count and controls the Parse worker pool:

# Use 8 parse workers
indx ./docs --out ./ai-ready --jobs 8

Tuning order

Start with the defaults. If a run is slow, profile which stage dominates the summary timings, then turn the one lever that matters for that stage. Use --jobs for a parse-heavy corpus, and batch size for embedding throughput. Avoid raising everything at once.

Caching & resume

indx keeps a content-addressed cache under <out>/.indx-cache/. Each entry is keyed on:

(stage, sha256(input), component-id)

Pass --resume to reuse any cache entry whose key is unchanged, skipping recomputation for unmodified files and unchanged components:

indx ./docs --out ./ai-ready --resume

Because the key includes the component identity (its resolved name, including any :model suffix), invalidation is precise — scoped to what actually changed:

You change…	Invalidates
The parser (--parser)	Parse and everything downstream
The embedder (--embedder)	Only Embed
A single source file	That file’s entries (and their downstream)
Nothing	Nothing — the whole run is cache hits

This is why re-running over a large estate after a small edit is cheap: only the touched files and the stages affected by your change are recomputed. The cache also makes stages idempotent — re-running a stage on its own output never duplicates work or corrupts state.

Use --verbose to raise log verbosity to DEBUG during a resume run:

indx ./docs --out ./ai-ready --resume --verbose

Note

The cache is keyed on the resolved config, including model identifiers. Examples are the cloud default openai:gpt-5-mini / openai:text-embedding-3-small, or the local-profile ollama:qwen2.5 / bge-m3. Pinning model identity is also what makes runs reproducible. See Reproducibility.

Streaming & memory rules

indx streams the estate rather than materialising it. Walk and the downstream stages process files as an iterator, holding the working set in memory rather than the whole directory. Files are read incrementally where the parser allows, and large intermediate buffers are released promptly.

This is what keeps a 2 GB folder from needing 2 GB of RAM:

• Walk yields files lazily; nothing assumes the full file list fits in memory.
• Parse runs in bounded worker pools, so only --jobs documents are in flight at once.
• Embed and upsert flow in adapter-sized batches (the embedder sub-batches; the store upserts in batches of 256), so vectors are written and dropped rather than accumulated.
• Vectors in a sealed .indx archive are memory-mapped on demand from vectors.f32 on load, not read whole.

Stream + batch pattern (SDK)

When building a custom stage or your own ingestion loop, follow the stream-then-batch shape. Don’t load everything and process one item at a time.

from itertools import islice

def batched(iterable, size):
    it = iter(iterable)
    while batch := list(islice(it, size)):
        yield batch

# Stream the walk, embed and upsert in batches, resume on cache hits.
for batch in batched(ctx.chunks, size=256):
    if cache.has(batch):           # resume: skip completed work
        continue
    vectors = embedder.embed([c.text for c in batch])
    for chunk, vector in zip(batch, vectors):
        chunk.embedding = vector
    store.upsert(batch)

The anti-pattern is the opposite. Do not read every file up front and embed one chunk at a time:

# Anti-pattern: materialises the whole estate, one round-trip per chunk
texts = [p.read_text() for p in all_files]
vectors = [embedder.embed([t]) for t in texts]

Making the local profile fast enough

The shipped zero-config defaults are cloud-backed, so model-heavy work runs on managed APIs out of the box. The opt-in local profile (docling + ollama:qwen2.5 + bge-m3) supports the air-gapped path with zero network calls.

On commodity hardware, local models can be the slow part of a large run. When you run the local profile, these mitigations apply, in order of impact:

1. Parallelise parsing. Raise --jobs to match your cores; parsing is usually the first bottleneck on document-heavy corpora.

2. Lean on batching. Keep embedding batched (local e5/bge-m3 default batch_size 12); increase the batch size if you have GPU/CPU headroom.

3. Skip what you do not need. When you only need structure, use --no-embed to produce a graph-only space (Walk → Relate, no vectors). When you do not want LLM work, drop the Enrich stage:

   # Graph only — no embedding, fastest path to structure
   indx ./docs --out ./ai-ready --no-embed

   from indx import DirectoryPipeline
   
   # Drop enrichment to skip all LLM calls
   space = DirectoryPipeline().drop("enrich").run("./docs", "./out")

4. Lean back on hosted models for the heaviest stages when policy allows. These are the same cloud backends used by the zero-config defaults. A hosted LLM for Enrich or a hosted embedder can dramatically cut wall-clock time on large estates. See Enrichment with LLM/VLM and Choosing an embedder.

5. Resume aggressively. Combine --resume with the cache so iterative runs only pay for what changed.

Determinism vs. speed

Concurrency never changes output ordering. Parallel results are re-sorted into deterministic order before chunk/document ids are assigned. Do not trade reproducibility for speed without an explicit, documented flag. Keeping index.json byte-stable is a core guarantee. See Reproducibility.

Quick reference

Goal	Lever
Parse faster	--jobs / -j (parse workers)
Embed faster	embed batch_size (per-adapter: OpenAI 256, e5/bge-m3 12)
Cheap re-runs	--resume (+ --verbose for DEBUG logs)
Skip vectors	--no-embed (graph-only space)
Skip LLM work	drop the enrich stage
Stay memory-stable	stream + batch; let the defaults do their job

For the full flag list see the CLI reference (--jobs, --resume, --no-embed). For keeping runs auditable and byte-stable see Reproducibility.