Bring Your Own Parser, Embedder, and Store

The user story

As an integrator, I want to plug my own parser and store into indx without subclassing or rewriting the pipeline, so I can handle in-house file formats without forking anything.

Priya’s team keeps their internal knowledge in a custom line-oriented format — think structured plain text with a proprietary header block — that none of the built-in parsers recognize. She also has a Chroma instance already running in staging. She doesn’t want to maintain a fork, and she refuses to subclass something just to swap one stage.

The blocker isn’t capability; it’s friction. What she needs is a seam.

What you’ll build

A pipeline that:

• Parses her custom file format with a class she wrote in twenty lines.
• Embeds with bge-m3 (built-in, no extra wiring).
• Stores into her existing chroma instance.
• Skips LLM enrichment entirely — her documents are already well-structured.

By the end you’ll have a KnowledgeSpace built from scratch, with every component either hers or named from the built-in registry.

The principle

Every slot in DirectoryPipeline — parser, llm, vlm, embedder, store, output — is backed by a typed Protocol. Protocols use structural typing: if your class implements the right methods with the right signatures, it satisfies the protocol. You don’t subclass anything. You don’t register anything. You just pass an instance.

Duck typing all the way down

Structural typing means the protocol is a contract, not a base class. Write a class with the required method, and indx treats it as a first-class component. isinstance checks are never involved.

This makes adapters tiny — an adapter only needs to import the slot it implements, so there are no transitive dependencies on the rest of the pipeline.

The build

1. Install indx.

   Priya’s team uses Chroma, so she pulls in the store extra. The core is always included.

   pip install "indx[chroma]"

   No LLM key needed for this story — she’s dropping the enrich stage entirely.

2. Write the custom Parser.

   The Parser protocol requires one method: parse(self, file) -> ParsedDoc. Import ParsedDoc and Source from indx, and import Parser from indx.parsers if you want to type-annotate the class — it is never required to inherit from it.

   from indx import DirectoryPipeline, ParsedDoc, Source
   from indx.parsers import Parser
   
   class MyMarkdownParser:
       """Custom Parser: satisfies the Parser protocol structurally."""
       def parse(self, file) -> ParsedDoc:
           text = file.read_text()
           return ParsedDoc(
               source=Source(path=file.path, folder=file.folder, type="markdown"),
               text=text,
               blocks=[{"kind": "paragraph", "text": p} for p in text.split("\n\n")],
           )

   That’s the whole adapter. ParsedDoc carries a Source (path, folder, type), the raw full text, and a blocks list — each block is a dict with at least "kind" and "text". Paragraph splitting on blank lines is a valid and common starting point; replace text.split("\n\n") with your own segmentation logic.

3. Wire it into the pipeline — two equivalent ways.

   Pass the instance at construction time:

   pipeline = DirectoryPipeline(
       parser=MyMarkdownParser(),
       embedder="bge-m3",
       store="chroma",
   )

   Or start with a pipeline and swap the slot with .use():

   pipeline = (
       DirectoryPipeline(embedder="bge-m3", store="chroma")
       .use(parser=MyMarkdownParser())
   )

   Both forms are identical at runtime. .use() returns self, so it chains naturally.

4. Drop the stages you don’t need.

   Priya’s notes are already structured — she doesn’t need LLM enrichment. .drop("enrich") removes that stage before the pipeline runs. Named stages are: walk, parse, chunk, relate, enrich, embed-pack.

   pipeline = (
       DirectoryPipeline(embedder="bge-m3", store="chroma")
       .use(parser=MyMarkdownParser())
       .drop("enrich")                   # skip LLM enrichment entirely
   )

   Dropping relate or chunk affects downstream stages

   chunk feeds relate, and both feed embed-pack. Drop them only if you understand what each stage produces — dropping relate removes the structural edge graph from the resulting space.

5. Run the pipeline.

   space = pipeline.run("./notes", "./out")

   indx ./notes → ./out
     01 walk     38 files, 4 folders
     02 parse    38 ok, 0 skipped   (MyMarkdownParser)
     03 chunk    291 chunks
     04 relate   88 relations
     06 embed    291 vectors → chroma, sealed notes.indx
   done: 291 chunks, 38 docs, embed_dim=1024  (3.2s)

   Stage 05 enrich is absent — .drop("enrich") took effect. The (MyMarkdownParser) annotation in the parse line confirms her class ran, not the built-in.

Mixing custom and built-in freely

Priya used a custom parser with built-in embedder and store. She could just as easily flip any other combination:

• Custom embedder, built-in parser and store — write a class with the Embedder protocol’s embed(self, texts) -> list[list[float]] method.
• Custom store, everything else built-in — implement the Store protocol; the slot accepts any instance that satisfies it.
• All custom, no built-ins — pass instances for every slot; the pipeline becomes a typed orchestrator over your own stack.

The other protocols follow the exact same pattern as Parser:

| Slot | Protocol | Import path | |---|---|---| | parser | Parser | from indx.parsers import Parser | | llm | LLM | from indx.llm import LLM | | vlm | VLM | from indx.llm import VLM | | embedder | Embedder | from indx.embed import Embedder | | store | Store | from indx.store import Store | | output | OutputWriter | from indx.output import OutputWriter |

Each protocol lives in its own sub-package so an adapter imports only what it needs — no transitive dependency on the rest of the pipeline.

Resolution precedence

When a slot is set in multiple places, code and .use() always win, then CLI flags, then INDX_* environment variables, then indx.toml, then documented defaults. Passing an instance at construction time is the highest-precedence form. An unknown name string raises a fatal error before any stage runs, so typos surface immediately.

The full example

from indx import DirectoryPipeline, ParsedDoc, Source
from indx.parsers import Parser

class MyMarkdownParser:
    """Custom Parser: satisfies the Parser protocol structurally."""
    def parse(self, file) -> ParsedDoc:
        text = file.read_text()
        return ParsedDoc(
            source=Source(path=file.path, folder=file.folder, type="markdown"),
            text=text,
            blocks=[{"kind": "paragraph", "text": p} for p in text.split("\n\n")],
        )

pipeline = (
    DirectoryPipeline(embedder="bge-m3", store="chroma")
    .use(parser=MyMarkdownParser())   # BYO parser instance
    .drop("enrich")                   # skip LLM enrichment entirely
)
space = pipeline.run("./notes", "./out")

Twenty lines. No base class. No registration. No fork.

What you learned

• Every DirectoryPipeline slot is a typed Protocol — satisfy it structurally and pass an instance; inheritance is never required.
• DirectoryPipeline(parser=..., embedder=..., store=...) and .use(parser=...) are equivalent; code-time assignment is the highest-precedence form.
• .drop("enrich") (or any named stage) removes that stage cleanly; the pipeline adapts around the gap.
• Custom and built-in components mix freely — name strings resolve to the built-in registry, instances bypass it entirely.

Where to go next

Custom components guide The full reference for writing, testing, and distributing custom parsers, embedders, stores, and output writers.

Bring your own stack The design principle behind structural typing in indx — why protocols, and what the seams guarantee.

Component protocols Exact method signatures for every slot: Parser, LLM, VLM, Embedder, Store, and OutputWriter.

Python SDK reference DirectoryPipeline constructor, .use(), .drop(), .run(), KnowledgeSpace — the full API surface.

Story: The vendor-neutral migration Devin builds once and re-exports to LangChain, LlamaIndex, and pgvector without re-deriving anything — a natural next step after you own your components.