01 · Walk

Walk is the first pipeline stage. It turns a directory or .zip into a navigable inventory: files, folders, lineage, basic metadata, and detected file types. Every later stage depends on this inventory being complete and deterministic.

The big idea

Most knowledge doesn’t live in one file — it lives in the arrangement of files. Walk preserves that arrangement before any parser sees the content.

Responsibility

Walk is a built-in stage with no swappable component slot of its own. It is pure orchestration that ships in the dependency-light core, and it runs with zero extras installed.

By contrast, Parse delegates to the Parser protocol, and Embed+Pack delegates to the Embedder, Store, and OutputWriter protocols.

Its job is to:

• Traverse the input directory or ZIP, discovering every candidate file.
• Capture folder lineage (the ordered list of folder names from root down to each file) and basic file metadata (size_bytes).
• Seed one Document per file with a stable id, relative path, lineage, and size_bytes.
• Apply include/exclude filters so ignored paths never enter the pipeline.

What Walk writes to SpaceContext

Every stage receives and returns the same shared, mutable SpaceContext, obeying the contract run(ctx: SpaceContext) -> SpaceContext — each returns the same object it received, mutated in place. Walk is the first writer into that context.

Field	Walk’s contribution
ctx.space.documents_	Walk’s only output: the seeded list of Document records (id, path, lineage, size_bytes), one per discovered file. Enrichment fields (doc_type, topics, tags, summary) are filled in later stages. The folder structure is carried by each Document’s lineage.

Each Document’s lineage carries the folder structure forward. Relate uses that lineage to derive sibling and parent edges. The Document objects Walk seeds are progressively enriched across stages 01–05.

How lineage feeds relations later

The folder ancestry Walk records is raw material for the Relate stage, not just metadata for display:

• Documents in the same folder become sibling relations.
• A document’s containing folder lineage becomes parent relations.

Both edge kinds are RelationType members emitted in stage 04. Capturing lineage faithfully in Walk is what makes “only contracts under /2024/acme” filtering and sibling-aware retrieval possible downstream. See the data models reference for the Document.lineage and Relation shapes.

Walk’s handoff to Parse

Walk does not parse anything itself. For each discovered file, it seeds a Document record (id, relative path, folder lineage, size_bytes). The Parse stage then resolves each file as ctx.root / doc.path (a pathlib.Path) and hands it to the parser. The Parser protocol is defined as:

@runtime_checkable
class Parser(Protocol):
    """Converts a single file into a ParsedDoc. Default: docling."""
    name: str
    def parse(self, path: Path) -> ParsedDoc: ...

Each Document Walk seeds carries everything Walk resolved:

• the relative path of the file (joined to ctx.root by Parse),
• the folder lineage captured during traversal,
• the size_bytes of the file.

This keeps responsibilities clean: Walk decides what exists and where it sits; the parser decides what it says.

Because the parser simply receives a pathlib.Path, a custom parser or custom component gets exactly the same handoff a built-in parser does.

Filtering — include / exclude

Walk honours include/exclude filters so that ignored paths — build artifacts, lockfiles, vendored directories — never reach Parse.

Filtering at the Walk boundary keeps the working set small. indx also streams files as an iterator rather than materializing a 10k-file estate into memory, so a 2 GB folder does not require 2 GB of RAM.

Plan before you run

Because Walk is cheap and runs before any parser, model, or store is loaded, it is the best place to catch a bad scope. The build summary reports Walk’s findings up front:

indx ./docs → ./ai-ready
  01 walk    128 files, 14 folders
  02 parse   128 ok, 0 skipped
  ...

Walk runs first regardless of which components you select, so a mis-scoped directory or over-broad filter surfaces before expensive work starts.

Security: treat scanned directories as untrusted

A scanned directory is untrusted input, and a ZIP especially so. Walk is the boundary where that input first enters indx, so it is where the guards live.

Untrusted input handling

• Guard against zip-slip / path traversal. When reading a ZIP, entry names are validated so a crafted archive cannot write or resolve outside the intended root, such as ../../etc/....
• Cap recursion depth. Traversal depth is bounded so pathological or cyclic structures cannot exhaust resources.
• Never execute file contents. Walk reads bytes and metadata only. It never runs anything it finds.

These are non-negotiable contributor rules, not optional hardening. Any code that walks a directory or expands a ZIP must apply them.

Determinism

Walk traverses in a deterministic order: folder lineage, then path. This ordering is what makes chunk and document ids stable later — ids are assigned by traversal order, so re-running over unchanged input yields identical ids and a byte-stable index.json.

Parallel per-folder traversal, where used, never affects the final ordering. Results are re-sorted into the canonical order before any id is assigned. See reproducibility for the full guarantee.

Where it sits in the pipeline

#	Stage	Component
01	Walk	— (built-in)
02	Parse	Parser
03	Chunk	— (built-in)
04	Relate	— (built-in)
05	Enrich	LLM, VLM
06	Embed+Pack	Embedder, Store, OutputWriter

Walk feeds its dir_graph and seeded Documents straight into 02 Parse, which resolves each file’s path and runs it through the configured parser.

For the full stage contract and the SpaceContext shape, see the pipeline overview and the data models reference.