02 · Parse

Parse is the second stage of the pipeline. It runs each discovered file through the configured Parser slot and produces one normalized ParsedDoc per document. Its input is the directory graph from Walk.

The big idea

indx composes existing parsers behind one typed interface — it does not reimplement PDF, Office, or layout extraction.

What this stage does

For each file in the directory graph, the Parse stage:

1. Resolves the file path as ctx.root / doc.path for the Document record seeded by stage 01.
2. Calls Parser.parse(path) on the configured parser (default: docling).
3. Stores the returned ParsedDoc in the shared context under ctx.parsed, keyed by doc_id.

Like every stage, Parse obeys the Stage contract: run(ctx: SpaceContext) -> SpaceContext. It mutates and returns the same SpaceContext it received. The Chunk stage then reads ctx.parsed to split content with structure intact.

01 Walk            02 Parse           03 Chunk
 dir graph  ──────▶ Parser.parse  ──────▶ split + lineage
 + Documents        → ParsedDoc           (uses blocks/tables)

Compose, don’t reimplement

Every parser — Docling, Unstructured, LlamaParse, MarkItDown — is a swappable adapter behind the Parser protocol. Each adapter’s only job is to map its native output into the parser-agnostic ParsedDoc shape. The rest of the pipeline never knows which parser is active.

The Parser protocol

A parser is any object that structurally satisfies the Parser protocol. It needs no base class and no import of indx internals.

from pathlib import Path
from typing import Protocol, runtime_checkable
from indx import ParsedDoc

@runtime_checkable
class Parser(Protocol):
    """Converts a single file into a ParsedDoc. Default: docling."""
    name: str
    def parse(self, path: Path) -> ParsedDoc: ...

Parse passes a pathlib.Path (resolved as ctx.root / doc.path); the parser reads that file and returns a ParsedDoc. Each parser also exposes a name attribute. See the protocols reference for the full contract and the custom components guide for a bring-your-own-parser walkthrough.

The ParsedDoc it returns

ParsedDoc is the raw, pre-chunking output of a parser for one source file. It is a Pydantic v2 model with these fields:

Field	Type	Description
source_path	str	Path of the source file this was parsed from. Required.
parser	str	Name of the parser that produced this doc (default "plaintext").
parser_version	str	Version string of that parser (default "0").
blocks	list[Block]	Structural units of parsed content (default empty). Each Block has kind: str (default "text" — e.g. text, heading, table, caption, code), text: str (default ""), and order: int (default 0).
text (computed)	str	Read-only property: the block text joined in order. Not a settable field.

Only source_path is required. parser/parser_version describe the producing parser, and blocks defaults to an empty list, populated to whatever depth the active parser supports. text is derived from blocks, so a parser produces structure by appending Blocks rather than setting text directly.

So a minimal parser can return just normalized text. A richer parser like Docling fills in layout, reading order, tables, and figures — giving downstream stages more structure to preserve.

from indx import ParsedDoc
from indx.core.parsed import Block

doc = ParsedDoc(
    source_path="policies/data/retention.pdf",
    parser="docling",
    blocks=[
        Block(kind="heading", text="Retention", order=0),
        Block(text="Enterprise data is retained for 90 days…", order=1),
    ],
)
# doc.text -> "Retention\n\nEnterprise data is retained for 90 days…"

Available parsers

Each non-fallback parser ships as an optional extra so the core install stays light. Select one three ways: with --parser, the [parser] engine key in indx.toml, or DirectoryPipeline(parser=...).

Name	Class	Install	Best for
docling (default)	DoclingParser	pip install "indx[docling]"	High-fidelity layout, reading order, tables, and figures from PDFs/Office docs. Fully local.
unstructured	UnstructuredParser	pip install "indx[unstructured]"	Heterogeneous corpora with many odd file formats.
llamaparse	LlamaParseParser	pip install "indx[llamaparse]"	Hard, messy PDFs via a hosted service (cloud).
markitdown	MarkItDownParser	pip install "indx[markitdown]"	Fast Markdown conversion. The lightest local install.
plaintext	PlainTextParser	ships in core	Zero-dependency fallback so a run works offline.

Install the extra for your parser

Docling is the richest local option but also the heaviest, since it can pull models and Torch — so it is not in the core install.

If you run a build without the chosen parser’s extra installed, indx raises a single MissingDependencyError naming the exact command, e.g. pip install "indx[docling]". The plaintext parser always works with no extras at all.

See choosing a parser for a deeper comparison and decision guide.

Concurrency

Parse is embarrassingly parallel across files. A worker pool of size --jobs (default = CPU count) runs Parser.parse concurrently, and results are merged back into ctx.parsed keyed by doc_id.

• Blocking and native parsers (Docling and Unstructured wrap native code) run in a bounded thread pool that keeps the run responsive.
• GIL-bound, CPU-heavy parsers may run in a process pool to sidestep the GIL. This costs the pickling of ParsedDocs across process boundaries.

Concurrency never affects output. Parallel results are re-sorted into deterministic order (folder lineage, then path) before any chunk or document ids are assigned, so reruns over unchanged input yield identical ids. See performance and reproducibility for tuning and guarantees.

# Parse (and embed) with 8 workers
indx ./docs --out ./ai-ready --jobs 8

Error model

Parse uses per-item skip as its default failure mode. When a single file fails to parse, indx does not abort the whole run:

• The file is skipped.
• A StageError(kind="skip") is appended to ctx.errors (recording the stage, the offending item path, and a message).
• The pipeline continues with the remaining files.

Skipped items are recorded as StageErrorRecord(kind="skip") entries on the pipeline context (ctx.errors), and the build summary reports the count:

  02 parse   127 ok, 1 skipped

Passing --strict (or strict=True in the SDK) promotes every skip to a fatal error. The first bad file aborts the run with a PipelineError, and the CLI exits with code 1. Use strict mode in CI, or when a corrupt input must never silently disappear.

Some conditions are always fatal regardless of strict mode, such as an unresolvable parser name or a missing extra. See errors and exit codes for the full table.

Where it fits

Reads from context	Writes to context
ctx.space.documents_ (files seeded by Walk)	ctx.parsed — doc_id → ParsedDoc
(parser is bound to the stage instance at pipeline construction, not read from context)	ctx.errors — per-item skips

Next, the Chunk stage consumes each ParsedDoc and splits it into retrievable Chunks while preserving the structure that Parse captured.