spiderpdf

Local citation graph viewer plus reference manager. Runs as a FastAPI server you open in a browser.

What it does

• 3D citation graph for any seed paper, scored by bibliographic coupling + co-citation, rendered with 3d-force-graph.
• Reference manager: each saved paper lives in its own folder with paper.pdf + metadata.json + notes.md + graph.json.
• Workspaces: pick a directory, get an isolated library + SQLite cache + BibTeX file inside it. Multiple workspaces, switchable from the UI.
• Auto-generated BibTeX at <workspace>/references.bib, regenerated on every library change.
• Background graph pre-building: when you save a paper, its graph builds in the background so the next view is instant.
• PDF ingest: drop files into the workspace dir, or upload via the UI. DOI / arXiv id is extracted and resolved via Semantic Scholar (OpenAlex fallback).

Install & run

Requires Python 3.10+ on Linux or macOS (Windows: use WSL or Git Bash).

git clone https://github.com/<you>/spiderpdf
cd spiderpdf
./start.sh

start.sh creates a .venv/, installs dependencies once, and launches the server. Subsequent runs reuse the venv. By default it picks port 8000 (or the next free one) and opens your browser to it.

Optional flags pass through:

./start.sh --port 9000        # explicit port
./start.sh --no-browser       # don't auto-open the browser
./start.sh --host 0.0.0.0     # listen on all interfaces (LAN access)

Optional but recommended — get a free Semantic Scholar API key and put it in .env next to start.sh:

echo "SEMANTIC_SCHOLAR_API_KEY=s2k-..." > .env
chmod 600 .env

Without a key, S2 lookups are throttled to the shared anonymous tier (~1 req/sec).

Manual install (no script)

If you'd rather not use start.sh:

python3 -m venv .venv
.venv/bin/pip install -e .
.venv/bin/spiderpdf serve

After install, python -m spiderpdf serve works as a synonym.

Try it without any PDFs

Once the server is running, paste any DOI into the Seed input at the top of the left sidebar and hit Enter. A good first example:

10.1145/3764119

That's the Fray (OOPSLA '25) concurrency-testing paper. The graph builds around it in ~1–2 min (S2 rate-limited on first hit), then renders ~40 related papers. Re-seed onto any of them by right-clicking, or save them to your library from the floating menu.

Usage

1. Workspace (top-left dropdown). Create one via + New workspace… — pick a name and a parent directory; the workspace dir is <parent>/<sanitized-name>/.
2. Add papers:
   • Drop PDFs into the workspace dir → indexed on next server start (DOI/arXiv only).
   • Or click + Add PDF for a file picker. Each upload becomes <workspace>/<pdf-stem>/paper.pdf with metadata + notes; title-search fallback is allowed.
3. Open the graph: click any library paper. First build takes ~1–2 min (S2 rate-limited); cached after.
4. Navigate:
   • Hover a node → tooltip.
   • Left-click a node → floating menu (Save, View details, Re-seed, Zoom) + right-panel detail.
   • Right-click a node → re-seed the graph around it.
5. Save a candidate to your library: Save in the floating menu. The PDF is downloaded if Semantic Scholar reports an open-access URL or the paper has an arXiv id.
6. Export BibTeX: 📋 .bib button in the Library header, or <workspace>/references.bib directly.

Configuration

Variable	Default	Purpose
SEMANTIC_SCHOLAR_API_KEY	unset	Faster, dedicated S2 rate slot
LIBRARY_DIR	./papers	Initial workspace path on first launch (ignored once a workspace registry exists)
XDG_CONFIG_HOME	~/.config	Where the workspace registry (<XDG>/spiderpdf/workspaces.json) lives

CLI

After install, the spiderpdf command lives in .venv/bin/spiderpdf. Run it directly, or activate the venv first (source .venv/bin/activate) and drop the prefix.

spiderpdf serve [--host 127.0.0.1] [--port 8000] [--no-browser]
spiderpdf index /path/to/pdfs                       # walk a dir, resolve PDFs, add to library
spiderpdf resolve "10.1145/3292500.3330701"         # debug: resolve a DOI / arXiv id

python -m spiderpdf works as a synonym for the spiderpdf script.

API

Endpoint	Purpose
GET /	Web UI
GET /library	List library items + background-build status
POST /library/upload	Upload one or more PDFs (multipart)
POST /library/add?paper_id=	Save a paper from a graph node into the workspace
GET /resolve?q=	Resolve DOI / arXiv id / S2 paperId
GET /graph?seed=&top_n=&min_edge=	Build the 3D graph for a seed paper
GET /bibtex	Download references.bib for the active workspace
GET /workspaces	List workspaces + active
POST /workspaces?name=&path=&subfolder=true	Create + activate a workspace
POST /workspaces/activate?name=	Switch active workspace
DELETE /workspaces/{name}	Remove from registry (files left on disk)
GET /fs/browse?path=	List subdirectories (for the New Workspace modal)

File layout

<workspace>/
├── .cp/papers.db                  # SQLite: papers, edges, expansions, library, resolutions
├── references.bib                 # auto-regenerated
├── any-file.pdf                   # flat PDFs auto-indexed on server start
└── <paper-folder>/                # created by Save / Add PDF
    ├── paper.pdf
    ├── metadata.json              # full S2 metadata
    ├── notes.md                   # title + abstract + your notes
    └── graph.json                 # 3D graph snapshot (nodes + links + scores)

spiderpdf/
├── pyproject.toml
├── README.md
└── spiderpdf/
    ├── __init__.py
    ├── __main__.py                # python -m spiderpdf
    ├── app.py                     # FastAPI + CLI + background worker
    ├── pipeline/
    │   ├── cache.py               # SQLite cache layer
    │   ├── sources.py             # S2 + OpenAlex client, rate-limited
    │   ├── ingest.py              # PDF → DOI / arXiv / title
    │   ├── graph.py               # bib-coupling + co-citation, top-N
    │   ├── bibtex.py              # paper dict → BibTeX entry
    │   └── workspaces.py          # workspace registry + manager
    └── static/index.html          # 3D viewer + UI

How resolution works

PDF → pypdf metadata + regex on pages 1-2 → DOI / arXiv id / fallback title
  → SemanticScholar /paper/{DOI:... | ARXIV:... | hash}
  → on miss: OpenAlex /works/...                     (only at resolution time)
  → on miss: SemanticScholar /paper/search?query=    (manual uploads only)

Auto-index skips title-search to avoid matching non-academic PDFs (class notes, scans, reports) to unrelated papers.

Limitations

• Semantic Scholar's free tier is 1 req/sec, cumulative — first graph for a cold paper takes minutes. Subsequent views are instant (SQLite cache).
• PDFs without a DOI or arXiv id in their first two pages are skipped by auto-index; use + Add PDF to title-search them.
• Heavy survey papers with hundreds of refs may show fewer edges than expected — the batch refs endpoint caps nested arrays in the response.
• Workspace paths are stored absolute. Moving a workspace dir requires editing ~/.config/spiderpdf/workspaces.json.

License

MIT.