Turn a Research Archive into a Citable Knowledge Graph

The user story

As a researcher, I want my messy archive of papers, notebooks, and notes turned into a navigable, citable knowledge graph I can hand to a collaborator as a single file.

Dr. Chen has years of accumulated research: PDFs of published papers, lab notebooks, and loose notes — several hundred files across a tangle of subfolders. Papers cite each other, versions of the same paper coexist under different filenames, and shared datasets connect documents that live in completely different parts of the tree. Every tool she’s tried indexes the files in isolation. There’s no edge that says “this preprint is a duplicate of that published paper,” no edge that says “this notebook’s results are what this methods section describes.”

The bigger frustration is portability. Whenever she hands an archive to a collaborator, they have to repeat the whole process. She wants something she can send once and her collaborator can immediately query.

What you’ll build

A self-contained knowledge graph of a research archive that:

• Surfaces references edges connecting papers that cite each other within the archive.
• Surfaces duplicate-of edges that collapse multiple versions of the same paper into one node.
• Is inspectable at the terminal — see the relation histogram before writing a single line of code.
• Ships as a single .indx file a collaborator can load and search on their own machine, no re-processing.

The build

1. Install indx with the cloud extras.

   PDFs need the docling parser, which is the default. No extra flags required.

   pip install "indx[cloud]"
   export OPENAI_API_KEY="sk-..."

   Working fully offline?

   Swap to pip install "indx[local]" and build with --llm ollama:qwen2.5 --embedder bge-m3. The archive format, relation edges, and KnowledgeSpace API are identical — only the build flags change.

2. Build the knowledge space from the archive folder.

   Point indx at the archive root. It walks the tree, parses each file (PDF pages via docling by default), chunks the text, derives the relation graph, enriches each document with a type and summary, and embeds everything — then seals it into one portable archive.

   indx ./archive --out ./ai-ready

   indx ./archive → ./ai-ready
     01 walk    318 files, 41 folders
     02 parse   315 ok, 3 skipped
     03 chunk   8 204 chunks
     04 relate  2 371 relations
     05 enrich  315 documents (openai:gpt-5-mini)
     06 embed   8 204 vectors → qdrant, sealed handbook.indx
   done: 8 204 chunks, 315 docs, embed_dim=1536  (4m 12s)

   The 04 relate 2 371 relations line is where the graph comes from. The Relate stage compares documents pairwise, resolves in-archive citation links, and detects near-duplicate content — producing references and duplicate-of edges alongside the structural sibling and parent edges that come from the folder tree.

3. Inspect the archive to confirm the graph looks right.

   Before opening a notebook, look at what the pipeline actually built.

   indx inspect ./ai-ready/handbook.indx

   documents : 315
   chunks    : 8 204
   relations : 2 371
   embeddings: 8 204  (dim 1536)
   bytes_source: 1.2 GB
   
   type histogram
     paper        187
     notebook      61
     note          42
     dataset       25
   
   relations sample
     references    attention-is-all-you-need.pdf → scaling-laws-nlp.pdf
     references    scaling-laws-nlp.pdf → chinchilla.pdf
     duplicate-of  attention-v1-preprint.pdf → attention-is-all-you-need.pdf
     duplicate-of  scaling-laws-draft.pdf → scaling-laws-nlp.pdf
     sibling       papers/transformers/ → papers/scaling/
     parent        papers/ → papers/transformers/

   The references rows confirm that in-archive citations were resolved into real edges. The duplicate-of rows confirm that preprint/published-version pairs were detected and linked — so downstream search won’t return both as independent hits.

4. Query the graph by topic.

   Semantic search works immediately. The -k flag controls how many hits come back (default 5).

   indx query ./ai-ready/handbook.indx "transformer attention scaling" -k 5

   #1  score 0.91  papers/transformers/attention-is-all-you-need.pdf  (papers/transformers · paper)
       "We propose a new simple network architecture, the Transformer, based solely on attention mechanisms…"
       neighbors: chunk_1042, chunk_1044
   
   #2  score 0.88  papers/scaling/scaling-laws-nlp.pdf  (papers/scaling · paper)
       "We study empirical scaling laws for language model performance on the cross-entropy loss…"
       neighbors: chunk_2301, chunk_2303
   
   #3  score 0.84  notebooks/transformer-replication/experiment-log.ipynb  (notebooks/transformer-replication · notebook)
       "Replicated attention mechanism from Vaswani et al. — confirmed scaling behaviour matches…"
       neighbors: chunk_5812, chunk_5813

   Each hit carries its source path, detected type, and neighbor chunk ids. The notebook that replicates the paper surfaces alongside the paper itself — because they share vocabulary, not because anyone hand-linked them.

   Filter by document type

   Add --type paper to restrict results to a single type. Useful when you want only primary sources, not notes or replications.

   indx query ./ai-ready/handbook.indx "transformer attention scaling" -k 5 --type paper

5. Hand the archive to a collaborator.

   The .indx file is a self-contained ZIP container: manifest, index, per-chunk content, and memory-mappable embeddings — everything. Your collaborator doesn’t need the original files, doesn’t need to re-run the pipeline, and doesn’t need the same environment.

   # On your collaborator's machine — pip install "indx[cloud]" is all they need
   from indx import KnowledgeSpace
   
   space = KnowledgeSpace.load("handbook.indx")
   # Validates the manifest checksum and version, then memory-maps the embedding matrix.
   
   hits = space.search("attention mechanism scaling laws", k=5)
   for hit in hits:
       print(hit.score, hit.source.path, hit.source.type)
       print(hit.chunk[:200])
       print()

   0.91  papers/transformers/attention-is-all-you-need.pdf  paper
   We propose a new simple network architecture, the Transformer…
   
   0.88  papers/scaling/scaling-laws-nlp.pdf  paper
   We study empirical scaling laws for language model performance…

   KnowledgeSpace.load() validates checksums and memory-maps the embedding matrix, so the first search is fast even for a large archive. The .indx archive reference describes the full container layout.

6. Save a modified view back to disk.

   space.save() is the reverse of load() — useful for trimming, re-annotating, or re-archiving a subset of the space.

   space.save("handbook-trimmed.indx")

   The saved archive is a valid .indx container: your collaborator can load() it the same way.

Why a graph beats a flat index for research

A flat index answers “what chunks are semantically close to this query?” A relation graph answers additional questions that matter for research:

• references edges let you ask “what does this paper actually cite within my corpus?” — tracing an intellectual lineage without manually parsing every bibliography.
• duplicate-of edges mean a search for a topic returns the canonical paper, not both the preprint and the published version as independent hits that dilute each other’s score.
• parent and sibling edges from the folder structure mean the graph knows that papers/transformers/ and papers/scaling/ are sibling sub-collections — context a flat embedder throws away.

The relation types the Relate stage can produce are sibling, parent, references, continues, and duplicate-of. See Relate stage for the full detection logic.

PDFs that are scanned images

The default docling parser handles text-layer PDFs. If your archive contains scanned PDFs with no embedded text, docling will extract little or nothing. See Read scanned PDFs with a vision model for the VLM-slot approach that handles those.

What you learned

• A mixed archive of PDFs, notebooks, and notes becomes a typed, relation-aware knowledge graph with one build command — no pipeline code to write.
• The Relate stage produces references and duplicate-of edges automatically, turning a pile of files into something you can navigate rather than just search.
• indx inspect gives you a relation sample and type histogram before you write any code — a quick sanity-check that costs nothing.
• The .indx container is the portable artifact: one file, validated checksums, memory-mapped vectors. A collaborator with KnowledgeSpace.load() is up and running immediately. See Data models for the full type surface.

Where to go next

Read scanned PDFs with a vision model Enable the VLM slot to extract text from scanned pages — same pipeline, one extra flag.

Relate stage How indx detects citation links, duplicates, and structural edges between documents.

.indx archive reference The full container layout — manifest, index, chunk files, and the embedding matrix.

Data models SearchHit, KnowledgeSpace.stats, RelationType, and every type the SDK surfaces.