Read Scanned PDFs with a Vision Model

The user story

As an archivist, I want a shelf of scanned, image-only reports turned into searchable text — including their charts and figures, so researchers can query the full meaning of a document, not just the words that survived OCR.

Marco manages the archive for a regional environmental agency. Over two decades, the agency digitised field reports by scanning them: stacks of pages that are images, not text. Each report contains the usual prose, but also bar charts of pollutant readings, maps with annotated regions, and hand-drawn diagrams that are the actual finding. A plain text extraction pipeline either returns nothing or returns the sparse OCR fragments the scanner could recover — neither is useful for retrieval.

The problem is not the model or the embedder. It’s that text-only pipelines treat a scanned page as a blank. The meaning lives in the pixels, and nothing in the pipeline looks at them.

What you’ll build

An archive where every scanned report — text, charts, figures, and maps — is:

• Parsed into both text fragments and captured images by the docling parser.
• Described in natural language by a vision-language model (VLM) that reads those images.
• Enriched with the same type, topics, tags, and summary fields as any other document.
• Queryable so that a question about a chart gets the chart’s described content, not silence.

By the end you’ll have a reports.indx archive where indx query surfaces answers derived from figures — not just from text.

How the VLM slot works

indx’s pipeline has two model slots in the Enrich stage (05 enrich):

• LLM slot — always active; handles text → type, topics, tags, summary.
• VLM slot — off by default (none); activates with --vlm; describes images the parser captured into ParsedDoc.images.

The docling parser (the default) pulls images and figures out of each file into ParsedDoc.images during the Parse stage. When the VLM slot is active, those images are passed to the vision model and the resulting descriptions feed the same enrichment fields the LLM populates from text. The two contributions are merged — one document, one set of enriched metadata.

When to turn the VLM on

Enable it when meaning is in the pixels: architecture diagrams, bar charts, scanned or OCR-resistant pages, screenshots, annotated maps. For clean text-only PDFs it adds cost and latency with no additional signal — leave it at none.

The VLM slot has two values:

| Value | Where it runs | Network egress | |---|---|---| | qwen-vl | Local (Qwen-VL via Ollama) | None | | gpt4o | Cloud (OpenAI) | Images sent to provider | | gpt4o:<model> | Cloud, pinned model | Images sent to provider |

The build

1. Install indx.

   Choose the stack that matches your privacy and infrastructure requirements. Marco’s agency has sensitive records, so the fully-local option matters — but the cloud path is shown first because it’s the fastest starting point.

   Cloud (OpenAI VLM)

   pip install "indx[cloud]"
   export OPENAI_API_KEY="sk-..."

   Fully local (no egress)

   pip install "indx[local]"
   # Qwen-VL must be available through Ollama:
   # ollama pull qwen-vl

2. Run the build.

   Point indx at the reports folder. Add --vlm to activate the vision slot.

   Cloud build

   indx ./reports --out ./ai-ready --name reports --vlm gpt4o

   Fully local build

   indx ./reports --out ./ai-ready --name reports --llm ollama:qwen2.5 --vlm qwen-vl

   Here is what a representative run looks like (cloud path, gpt4o VLM):

   indx ./reports → ./ai-ready  (--name reports)
     01 walk    318 files, 24 folders
     02 parse   318 ok, 0 skipped
     03 chunk   2740 chunks
     04 relate  894 relations
     05 enrich  318 documents (openai:gpt-5-mini + gpt-4o vision)
     06 embed   2740 vectors → qdrant, sealed reports.indx
   done: 2740 chunks, 318 docs, embed_dim=1536  (48.2s)

   Notice the 05 enrich line: both the LLM model and the vision model are at work. Every document that had captured images in ParsedDoc.images received VLM descriptions alongside the LLM’s text-derived enrichment.

3. Sanity-check retrieval.

   Before connecting an agent or exporting to another system, query the archive directly. Pick a question whose answer would only be in a chart.

   indx query ./ai-ready/reports.indx "what did revenue do in Q3?" -k 3

   #1  score 0.81  reports/2023/q3-annual-summary.pdf  (reports/2023 · financial-report)
       "The Q3 bar chart shows revenue rising from €2.1 M in July to €2.8 M in September,
        a 33 % increase over the quarter. The trend line projects continued growth into Q4."
       neighbors: chunk_1042, chunk_1044

   That answer is derived from the chart image, not from surrounding prose — the scanned page had no machine-readable text at that position. Without the VLM the chunk would have been empty or absent from the results entirely.

4. Optional — use the SDK instead of the CLI.

   from indx import DirectoryPipeline
   
   # Cloud
   DirectoryPipeline(vlm="gpt4o").run("./reports", "./ai-ready")
   
   # Fully local
   DirectoryPipeline(llm="ollama:qwen2.5", vlm="qwen-vl").run("./reports", "./ai-ready")

Why scanned PDFs need a separate model slot

A standard RAG pipeline operates on extracted text. For a clean PDF that text is accurate and complete. For a scanned PDF it is absent, sparse, or garbled by OCR. The semantic content — what a chart shows, what a diagram labels, what a table of figures reports — is encoded in pixels.

indx separates the two concerns deliberately:

• The parser (docling by default) is responsible for pulling images out of files and collecting them into ParsedDoc.images. It does not describe them.
• The VLM slot is responsible for description. It is opt-in because most archives do not need it, and activating it for a clean-text corpus adds cost without benefit.

Because the VLM’s descriptions feed the same enrichment fields as the LLM, downstream retrieval, filtering, and agent tools work identically whether a chunk’s metadata came from text or from a described figure.

Cloud VLMs transmit image data

When you set --vlm gpt4o, the captured images from ParsedDoc.images are sent to OpenAI’s API. For sensitive or regulated archives, use --vlm qwen-vl to keep all processing local. The qwen-vl option requires Ollama and the qwen-vl model to be available locally; no image data leaves the machine.

When to use the VLM — and when not to

| Your content | Recommendation | |---|---| | Scanned / image-only PDFs | Enable VLM — text extraction returns nothing useful | | PDFs with embedded charts and diagrams | Enable VLM — figures carry signal the text does not | | Screenshots, annotated maps, architecture diagrams | Enable VLM | | Clean text-only PDFs, Markdown, plain text | Leave VLM at none — it adds cost with no signal | | Mixed corpus (some scanned, some clean) | Enable VLM — it only acts on documents that have captured images |

What you learned

• The docling parser captures images and figures into ParsedDoc.images during the Parse stage; the VLM slot then describes them during Enrich.
• The VLM slot is none by default. Set --vlm qwen-vl for fully local processing or --vlm gpt4o (optionally gpt4o:<model> to pin a specific model) for cloud processing.
• LLM and VLM contribute to the same enrichment fields — type, topics, tags, summary — so the rest of the pipeline and all downstream tools are unaware of the distinction.
• A question whose answer lives in a chart now returns that answer; without the VLM the chunk would be empty.

Where to go next

Enrichment: LLMs & VLMs Full reference for the LLM and VLM slots — models, cost trade-offs, and how descriptions merge into enrichment fields.

Choosing a parser When docling is the right choice and when you might bring your own.

Enrich stage How the 05 enrich stage orchestrates the LLM and VLM slots and writes enrichment fields.

The citable research archive Dr. Chen turns mixed PDFs and notes into a graph with citation and duplicate edges — a close cousin of Marco's archive.