Redact PII Before It Reaches the Cloud

The user story

As a compliance engineer, I want secrets and PII stripped from chunks before any cloud LLM or embedder sees them — as a first-class pipeline step, not a hack.

Aisha runs compliance at a fintech. Her team wants cloud-quality enrichment and embeddings — the summaries, type detection, and semantic search that come from sending text to a capable model. The problem is the source documents: HR records, contracts, and support tickets that contain email addresses, national-ID numbers, API keys, and other data that must never egress to an external provider.

The default pipeline is cloud-backed. Stages enrich (index 4) and embed-pack (index 5) both send chunk text to the configured LLM and embedder. If Aisha can transform every chunk before those stages run, she gets cloud enrichment on clean text — not a bypass, not a fork.

The pipeline’s stage model makes this exact extension point available.

What you’ll build

A single PiiRedactStage class that:

• Rewrites .text on every chunk in place, stripping emails, SSNs, and other patterns.
• Inserts at index 3 — after chunking, before relation-building, enrichment, and embedding.
• Works with structural typing: no subclassing, no registration, no special base class.

By the end you’ll have a one-liner pipeline where only redacted text ever leaves your machine.

The build

1. Install the cloud extra.

   pip install "indx[cloud]"
   export OPENAI_API_KEY="sk-..."

   Running offline?

   Replace "indx[cloud]" with "indx[local]" and pass llm="ollama:qwen2.5" to DirectoryPipeline. The stage code and insertion call are identical — the egress concern simply goes away.

2. Understand where in the pipeline redaction must land.

   indx runs six built-in stages in this order:

   | Index | Name | What it does | |-------|------|--------------| | 0 | walk | Discovers files in the source tree | | 1 | parse | Converts each file into a document | | 2 | chunk | Splits documents into chunks, populating ctx.chunks | | 3 | relate | Derives sibling, parent, and cross-reference edges | | 4 | enrich | Sends chunk text to the LLM for type, summary, metadata | | 5 | embed-pack | Embeds chunk text and seals the archive |

   Stages enrich and embed-pack are the egress-capable stages. Redaction must happen after chunk (index 2, which creates ctx.chunks) and before relate (index 3, the first downstream consumer). Inserting at index 3 pushes the existing relate → 4, enrich → 5, and embed-pack → 6.

   Don’t insert at index 2 or earlier

   Before chunk runs, ctx.chunks is empty — there is nothing to iterate. The target index is 3.

3. Write the PiiRedactStage.

   A stage is any object with a name: str attribute and a run(self, ctx: SpaceContext) -> SpaceContext method that returns the same context, mutated. That’s the full protocol — structural typing, no subclassing required.

   import re
   from indx import SpaceContext
   
   # Patterns to redact — extend to match your data classification policy.
   _PATTERNS = [
       (re.compile(r'[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}'), '[EMAIL]'),
       (re.compile(r'\b\d{3}-\d{2}-\d{4}\b'), '[SSN]'),          # US SSN
       (re.compile(r'\b\d{16}\b'), '[CARD]'),                     # 16-digit card
       (re.compile(r'(?i)(api[_-]?key|secret|token)\s*[:=]\s*\S+'), '[SECRET]'),
   ]
   
   def redact(text: str) -> str:
       for pattern, replacement in _PATTERNS:
           text = pattern.sub(replacement, text)
       return text
   
   class PiiRedactStage:
       name = "pii-redact"
   
       def run(self, ctx: SpaceContext) -> SpaceContext:
           for chunk in ctx.chunks:
               chunk.text = redact(chunk.text)
           return ctx          # MUST return the same context object

   The redact() helper uses plain re — no extra dependencies. Add or replace patterns to match your data classification policy. Aisha’s version also covers national ID formats for the jurisdictions her company operates in.

4. Insert the stage and run the pipeline.

   from indx import DirectoryPipeline, SpaceContext
   
   pipeline = DirectoryPipeline(llm="openai:gpt-5-mini")
   pipeline.insert(3, PiiRedactStage())     # after chunk (2), before relate/enrich/embed
   space = pipeline.run("./docs", "./ai-ready")

   Or as a one-liner:

   space = (
       DirectoryPipeline(llm="openai:gpt-5-mini")
       .insert(3, PiiRedactStage())
       .run("./docs", "./ai-ready")
   )

   Stage management methods return self, so calls chain. The resulting space is built entirely from redacted text.

5. Verify the stage is in position before you build.

   Call .stages() to inspect the live pipeline before committing a run:

   pipeline = DirectoryPipeline(llm="openai:gpt-5-mini")
   pipeline.insert(3, PiiRedactStage())
   
   for i, stage in enumerate(pipeline.stages()):
       print(i, stage.name)

   0  walk
   1  parse
   2  chunk
   3  pii-redact      ← your stage
   4  relate
   5  enrich
   6  embed-pack

   pii-redact sits between chunk and relate. Every downstream stage — including both egress-capable ones — operates on already-redacted text.

6. Run with --strict so a redaction failure is fatal.

   By default, a stage error on one item is logged and skipped. In a compliance context you want a failure to be loud and terminal. Pass --strict on the CLI, or add strict=True when constructing the pipeline if the SDK supports it directly:

   indx ./docs --out ./ai-ready --strict

   With --strict, any unhandled exception inside PiiRedactStage.run causes the pipeline to exit with code 1 rather than continuing with a partially-redacted chunk.

Why secrets stay out of the config

indx never reads API keys from indx.toml or any other config file. Provider credentials come exclusively from environment variables (OPENAI_API_KEY, INDX_LLM__API_KEY, and so on), and indx never logs or serializes them. That means your config file is safe to commit; the secrets are in your secrets manager where they belong.

What you learned

• The six built-in stages run in fixed order: walk(0) → parse(1) → chunk(2) → relate(3) → enrich(4) → embed-pack(5). .insert(3, …) lands a custom stage between chunk and relate, before any egress.
• A stage is any object with name: str and run(self, ctx) -> ctx. No base class, no registration, no framework ceremony.
• ctx.chunks is the live list of chunks; mutate .text in place, then return ctx.
• Stage management methods (.insert(), .append(), .replace(), .drop()) all return self, so they chain.
• --strict turns a per-item skip into a fatal exit — essential when a missed redaction is a compliance violation.

Where to go next

Writing a custom stage The full stage protocol — lifecycle hooks, error handling, and testing patterns.

Enrichment: LLMs & VLMs What enrich does with chunk text, and how to configure or replace the LLM slot.

Python SDK reference DirectoryPipeline, SpaceContext, and every stage management method in full.

Bring your own components Priya plugs a custom parser and store into the pipeline with structural typing — the same pattern that makes PiiRedactStage work.