Efficient Index Rebuilding for Visual Document Retriever Upgrades
Architecture · Reproduction · Experiments
ReprForge rebuilds visual document indexes by reusing intermediate computation that remains valid across retriever upgrades. It identifies the visual state needed by later computation, validates its dependencies and page inputs, and lets the target retriever complete the remaining computation with its own parameters.
Initial build page → visual computation → retained state → source representation
Upgrade validate retained state → target continuation → target representation
The implementation includes PyTorch execution tracing, state capture and replay, a ColQwen2.5 integration, and experiment runners. See the architecture and implementation notes for the replay contract and supported execution paths.
On 20,946 pages, four released retriever upgrades achieve 4.35–4.48× rebuild speedups and reduce total initial-build-and-upgrade time by 61%, including initial state retention. These measurements use a common processing configuration and a fixed input contract; all rebuilt representations are bitwise identical to full target encoding. Timing covers representation generation and publication, excluding physical ANN index construction.
A separate 200-page experiment measures generic input validation, which reruns the target processor: 1.878× speedup, or 1.818× including the one-time dependency check. Protocols and implementation paths are documented in the reproduction guide and paper map.
From a checkout, install the library and run the CPU example:
git clone https://github.com/visionary-5/reprforge.git
cd reprforge
python -m pip install -e '.[dev]'
python examples/basic_replay.py
python -m pytest -qThe core library requires Python 3.10+ and NumPy. The example uses synthetic tensors to demonstrate capture, validation, replay and publication. Install PyTorch to include the tracing and replay tests:
python -m pip install torch==2.5.1
python -m pytest tests/test_traced_replay.py -qFor real models, follow the GPU reproduction guide. The command-line guide covers source indexing, target replay and comparison with independently encoded target representations.
| Evaluation | Entry |
|---|---|
| Released-pair validity and interface discovery | Audit and tracing |
| Exact target reconstruction | Reconstruction |
| Full lifecycle rebuild cost | Lifecycle |
| Manual interfaces, retrieval quality and validation cost | Interface and validation |
| Controlled state and dependency changes | Ablation |
The experiment index also lists auxiliary workflows. The paper map identifies coverage, required inputs and historical workflows that are not fully distributed. Each experiment compares routes under its own input contract and timing scope.
reprforge/: tracing, replay, model integration and index utilities.scripts/,examples/: command-line tools and CPU examples.experiments/,configs/: evaluation runners and input specifications.tests/: implementation checks.docs/: architecture, protocols and reproduction instructions.
Keep datasets, full model checkpoints, retained states and generated results outside the checkout. Four small projection inputs needed for input preparation are bundled with their source and license information.
Software citation metadata is available in CITATION.cff. Paper citation details will be added when the manuscript is publicly available. Code is licensed under Apache-2.0; third-party models and datasets retain their own terms.