Use indx as a Vendor-Neutral Migration Layer

The user story

As a platform lead locked into a vendor’s ingestion, I want to build my knowledge once and re-export it to LangChain, LlamaIndex, or any vector DB — without re-deriving anything next year.

Devin runs platform engineering at a mid-size fintech. Eighteen months ago the team committed to a managed vector store and a proprietary ingestion SDK. Last quarter that vendor raised prices. This quarter the ML team wants to trial a different embedding provider. Next quarter someone will ask for LlamaIndex instead of LangChain.

Every time the stack shifts, someone re-parses the document archive, re-chunks it, re-derives the relationship graph, and re-embeds everything from scratch. It takes a week and never produces quite the same result twice.

The real problem isn’t the vendor. It is that the pipeline is the knowledge — each choice of tool is baked into the artifact. Change the tool, lose the artifact.

What you’ll build

A single indx build that:

• Parses, chunks, relates, and enriches Devin’s document archive once, into a portable handbook.indx archive.
• Re-emits to LangChain, LlamaIndex, JSONL, or any combination — without touching the source documents again.
• Re-stores into pgvector, Chroma, LanceDB, or Qdrant by changing one flag or one line of config.
• Stays model-neutral via the litellm adapter, so the LLM and embedder are as swappable as the store.

By the end, Devin can hand the archive to a new team, re-target a different store, and re-export a different format — all without re-deriving anything.

Not a database, not a runtime

indx writes to stores (Qdrant, pgvector, Chroma, LanceDB, JSONL) but does not store or serve vectors at runtime. Serving, retrieval orchestration, and agent hosting all stay downstream. indx’s job ends when the export is written.

The core idea: every component is a swappable slot

indx pipelines are composed of six typed slots: parser, LLM, VLM, embedder, store, and output. Each slot has a default, but every slot can be replaced independently — on the CLI, in indx.toml, via environment variable, or in Python.

That design makes indx a neutral intermediate layer. The knowledge space — its document graph, chunk boundaries, relationship edges, and enriched metadata — is derived once and stored in a portable .indx archive. The output format and vector store are a separate, downstream concern. Change the store or the framework adapter and you are not touching the knowledge; you are only changing where it lands.

source documents
        │
        ▼
  ┌──────────────────────────────────────────────────────┐
  │  indx pipeline                                       │
  │  parser → chunk → relate → enrich (LLM) → embed     │
  └───────────────────────────┬──────────────────────────┘
                              │ KnowledgeSpace (portable)
              ┌───────────────┼────────────────┐
              ▼               ▼                ▼
         --store          --format         SDK save()
         pgvector         langchain        any target
         chroma           llamaindex
         lancedb          jsonl
         qdrant

The build

1. Install indx.

   The cloud extra pulls in the default OpenAI LLM and embedder. Add litellm if you want to route to Bedrock, Azure OpenAI, or another provider (covered in step 5).

   pip install "indx[cloud]"
   export OPENAI_API_KEY="sk-..."

2. Build the knowledge space once.

   Point indx at the document archive. It walks the tree, parses every file, chunks and relates documents, enriches each one with a type and summary, and embeds everything. The result is a sealed, portable handbook.indx archive.

   indx ./docs --out ./ai-ready

   indx ./docs → ./ai-ready
     01 walk    318 files, 24 folders
     02 parse   318 ok, 0 skipped
     03 chunk   2 741 chunks
     04 relate  891 relations
     05 enrich  318 documents (openai:gpt-5-mini)
     06 embed   2 741 vectors → qdrant, sealed handbook.indx
   done: 2741 chunks, 318 docs, embed_dim=1536  (31.4s)

   That handbook.indx is the stable artifact. Everything below re-uses it.

3. Re-emit to a framework-native format with --format.

   The --format flag controls the output writer. Three writers ship out of the box:

   | --format | What it writes | |---|---| | langchain | Framework-native structures — Document objects with metadata, ready to drop into a LangChain retriever or vector store loader | | llamaindex | Framework-native structures — TextNode objects with relationships, ready for a LlamaIndex index or query engine | | jsonl | Newline-delimited JSON; zero dependencies, portable everywhere |

   Re-export to LangChain without touching the source documents:

   indx ./docs --out ./ai-ready --format langchain

   Re-export to LlamaIndex instead:

   indx ./docs --out ./ai-ready --format llamaindex

   The portable fallback — useful for streaming into any downstream pipeline:

   indx ./docs --out ./ai-ready --format jsonl

   Re-run from the built space

   If the archive already exists and the source documents haven’t changed, you can re-run with a different --format or --store and indx will skip re-deriving structure. Only the output step runs again.

4. Re-store into a different vector DB with --store.

   The --store flag is independent of --format. You can combine them freely.

   # Move from the default Qdrant to pgvector
   indx ./docs --out ./ai-ready --store pgvector
   
   # pgvector + LangChain output in one pass
   indx ./docs --out ./ai-ready --store pgvector --format langchain
   
   # Chroma for local dev, no network required
   indx ./docs --out ./ai-ready --store chroma
   
   # JSONL store — flat files, works anywhere
   indx ./docs --out ./ai-ready --store jsonl

   Supported store backends: qdrant (default), pgvector, chroma, lancedb, jsonl.

   Changing --store does not change the knowledge — the document graph, chunk text, and metadata are identical across all backends. Only the vector index changes.

5. Stay model-neutral with the litellm adapter.

   The store and format slots decouple the output side. The litellm extra decouples the input side — LLM and embedder — so the build itself has no hard dependency on any single provider.

   pip install "indx[litellm]"

   Name strings follow the pattern litellm:<provider>/<model-id>. The adapter routes to 100+ providers through one interface.

   Bedrock + Azure Embedder

   indx ./docs --out ./ai-ready \
     --llm      litellm:bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0 \
     --embedder litellm:azure/my-embedding-deployment \
     --store    pgvector \
     --format   langchain

   Ollama (fully local)

   indx ./docs --out ./ai-ready \
     --llm      litellm:ollama/qwen2.5 \
     --embedder litellm:ollama/nomic-embed-text \
     --store    lancedb \
     --format   jsonl

   indx.toml

   # indx.toml — commit this; override per-environment with INDX_* env vars or CLI flags
   [llm]
   backend = "litellm:bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0"
   
   [embedder]
   backend = "litellm:azure/my-embedding-deployment"
   
   [store]
   backend = "pgvector"
   
   [output]
   format = "langchain"

   With litellm in the LLM and embedder slots and a different backend in the store and format slots, the pipeline has no hard dependency on any vendor at any layer.

6. Do it in Python — load once, re-export many times.

   The SDK gives you the same control with more composability. Load the built space and re-save it to a different target without rebuilding from source.

   from indx import KnowledgeSpace
   
   # Load the archive built in step 2
   space = KnowledgeSpace.load("ai-ready/handbook.indx")
   print(space.stats)
   # KnowledgeSpaceStats(docs=318, chunks=2741, relations=891, embed_dim=1536)
   
   # Re-save to a different archive (e.g. for a different team's store)
   space.save("ai-ready/handbook-pgvector.indx")

   Or re-run the pipeline with a swapped store and output via .use():

   from indx import DirectoryPipeline
   
   pipeline = DirectoryPipeline()
   
   # Swap any slot without rebuilding the pipeline object
   pipeline.use(store="pgvector", output="langchain")
   
   space = pipeline.run("./docs", out="./ai-ready", name="handbook")

   .use(**components) returns self, so you can chain it. Every slot — parser, llm, vlm, embedder, store, output — is swappable the same way.

Why this protects Devin next year

The knowledge space — parsed documents, chunk boundaries, relationship edges, enriched metadata, embeddings — is captured in one portable artifact the first time. When the vendor changes:

• New framework? Re-run with --format llamaindex. Structure unchanged.
• New vector store? Re-run with --store chroma. Chunks unchanged.
• New model provider? Swap --llm and --embedder via litellm. One flag, any of 100+ providers.
• New team? Hand them handbook.indx. They call KnowledgeSpace.load() and re-save to their stack.

None of those changes touch the source documents or re-derive the graph. That’s the only way to guarantee the knowledge is the same across environments.

Embedding dimensions are store-bound

If you change the embedder — e.g. from openai:text-embedding-3-small (dim 1536) to a different model — the vector index must be rebuilt, because dimensions don’t match. The document graph and chunk text are still reusable; only the embed step re-runs. Plan your embedder choice as carefully as your store choice.

What you learned

• Every indx component is a typed, swappable slot. The knowledge space is derived independently of where it lands.
• --format (langchain, llamaindex, jsonl) controls the output writer. --store (pgvector, chroma, lancedb, qdrant, jsonl) controls the vector backend. They compose freely.
• KnowledgeSpace.load() / space.save() and DirectoryPipeline(...).use(...) give you the same control in Python.
• The litellm adapter decouples the input side — LLM and embedder — so every layer of the pipeline is vendor-neutral.

Where to go next

Output formats & integrations The full reference for every --format writer and how framework-native structures are shaped.

Bring your own stack How the slot system works under the hood — structural typing, precedence, and the adapter contract.

Plug into an AI agent Wire the built space into an agent framework — every connector, MCP, and LiteLLM backends.

Bring your own components Priya plugs a custom parser and store into the pipeline with structural typing — no subclassing.