PiPNN 4/6: integrate disk build pipeline

[SeliMeli]

Connects PiPNN adjacency output to the production disk-index build pipeline.

Code map

1. build_algorithm.rs adds the tagged PiPNN configuration and converts it into the core crate's validated parameters.
2. builder/build.rs dispatches on the explicit algorithm choice. Memory-budget estimation may choose Vamana only for the existing automatic path; an explicit PiPNN request is never rewritten.
3. builder/build/pipnn.rs validates the dataset/configuration boundary, constructs the core build context, and receives adjacency for real point IDs.
4. The adapter writes that adjacency through the existing graph header/layout and serializer. PQ generation, disk layout, start metadata, and other storage policy remain in the common outer pipeline.
5. The new storage entry point accepts canonical adjacency rows; it does not expose PiPNN internals or a second graph format.

Review path

• Follow one explicit PiPNN request from deserialization through BuildAlgorithm, builder dispatch, core invocation, and graph serialization.
• Check that graph degree/metric configuration is shared with the existing pipeline rather than reconstructed in the adapter.
• Verify real-point counts, IDs, and headers at the core/storage boundary, including configuration/dataset mismatch errors.
• Compare one-shot and sharded selection tests: only the automatic estimator path may fall back, while explicit PiPNN remains PiPNN.

Stack 4/6: #1290 → #1294

[Copilot]

Pull request overview

This PR integrates PiPNN graph construction into the DiskANN disk-index build pipeline and exposes it through the benchmark CLI inputs, including automated fixture-based tests to validate the new JSON configuration and execution paths.

Changes:

• Add a BuildAlgorithm selector (Vamana vs PiPNN) with JSON-facing PiPNN parameters, plus memory-estimate-based fallback to Vamana for disk builds.
• Implement PiPNN graph building adapters for disk builds and in-memory benchmark builds, including vector-store “flat prefix” access for dense layouts.
• Add a shared cached CLI fixture runner and new PiPNN benchmark fixtures (disk + in-memory) to validate end-to-end CLI wiring.

Reviewed changes

Copilot reviewed 33 out of 37 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
diskann-providers/src/utils/rayon_util.rs	Exposes underlying Rayon pool reference for APIs that need &rayon::ThreadPool.
diskann-providers/src/storage/mod.rs	Re-exports new adjacency-graph save helper.
diskann-providers/src/storage/bin.rs	Adds save_adjacency_graph for canonical adjacency graph serialization.
diskann-providers/src/model/graph/provider/async_/fast_memory_vector_provider.rs	Exposes unsafe dense-prefix slice access to support PiPNN build path.
diskann-providers/src/model/graph/provider/async_/common.rs	Implements AlignedMemoryVectorStore::flat_prefix and adds a contiguous-prefix test.
diskann-disk/src/lib.rs	Re-exports PiPNN parameters (feature-gated) and exposes build-algorithm selection.
diskann-disk/src/build/mod.rs	Re-exports BuildAlgorithm and (feature-gated) PiPNN parameters.
diskann-disk/src/build/configuration/mod.rs	Introduces build-algorithm configuration module and re-exports its types.
diskann-disk/src/build/configuration/disk_index_build_parameter.rs	Adds build_algorithm, PiPNN constructor, PiPNN config conversion, and memory-estimate-based fallback.
diskann-disk/src/build/configuration/build_algorithm.rs	New: serde-driven BuildAlgorithm enum and PiPNN JSON parameter struct.
diskann-disk/src/build/builder/build/pipnn.rs	New: PiPNN graph build adapter producing a canonical on-disk adjacency graph.
diskann-disk/src/build/builder/build/pipnn/tests.rs	New: disk-build integration tests for PiPNN selection, fallback, and header correctness.
diskann-disk/src/build/builder/build.rs	Routes graph construction to PiPNN (when selected) while keeping existing Vamana pipeline as default.
diskann-disk/Cargo.toml	Adds optional diskann-pipnn dependency + feature, plus test deps for new tests.
diskann-benchmark/src/main.rs	Refactors registry creation and adds PiPNN fixture test hook (feature-gated).
diskann-benchmark/src/inputs/graph_index.rs	Adds PiPNN build-algorithm selection for graph-index build inputs (feature-gated).
diskann-benchmark/src/inputs/disk.rs	Adds alpha, makes quantization_type optional, and adds disk build-algorithm selection + validation.
diskann-benchmark/src/index/build.rs	Adds PiPNN in-memory build path leveraging dense flat-prefix when possible.
diskann-benchmark/src/index/benchmarks.rs	Wires PiPNN build path selection into benchmark build flow.
diskann-benchmark/src/disk_index/build.rs	Wires disk BuildAlgorithm into disk-index build parameter construction.
diskann-benchmark/Cargo.toml	Adds optional PiPNN dependency/feature and enables fixture runner for dev-tests.
diskann-benchmark/tests/pipnn/inmemory/stdout.txt	New: expected output for in-memory PiPNN CLI fixture.
diskann-benchmark/tests/pipnn/inmemory/stdin.txt	New: commands for in-memory PiPNN CLI fixture.
diskann-benchmark/tests/pipnn/inmemory/input.json	New: PiPNN in-memory input JSON fixture.
diskann-benchmark/tests/pipnn/disk/stdout.txt	New: expected output for disk PiPNN CLI fixture.
diskann-benchmark/tests/pipnn/disk/stdin.txt	New: commands for disk PiPNN CLI fixture.
diskann-benchmark/tests/pipnn/disk/input.json	New: PiPNN disk input JSON fixture.
diskann-benchmark-runner/src/lib.rs	Exposes new fixture runner module for tests/downstream crates (feature-gated).
diskann-benchmark-runner/src/fixture.rs	New: shared cached CLI fixture runner implementation.
diskann-benchmark-runner/src/fixture/tests.rs	New: unit tests for fixture runner behavior.
diskann-benchmark-runner/src/app.rs	Updates integration-test documentation and switches tests to the shared fixture runner.
diskann-benchmark-runner/Cargo.toml	Adds test-fixtures feature + optional tempfile dep for downstream fixture usage.
Cargo.lock	Adds/records diskann-pipnn and serde_json dependency changes.
.cargo/mutants.toml	Excludes new fixture overwrite behavior from mutation testing.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

diskann-disk/src/build/configuration/disk_index_build_parameter.rs:17

• estimate_pipnn_peak_memory uses size_of::<...>() unqualified, but this module doesn’t import std::mem::size_of. This will fail to compile under the repo’s Rust toolchain.

use std::num::NonZeroUsize;

use diskann::ANNError;
#[cfg(feature = "pipnn")]
use diskann::ANNResult;
use thiserror::Error;

#[cfg(feature = "pipnn")]
use super::PiPNNParameters;
use super::{BuildAlgorithm, QuantizationType};

diskann-disk/src/build/configuration/disk_index_build_parameter.rs:113

• DiskIndexBuildParameters is publicly re-exported (via diskann-disk’s public API) and this change removes Copy from its derives. That is a breaking API change for downstream users that relied on implicit copies; if this is intentional, it should be called out explicitly (and potentially gated or versioned accordingly).

/// Parameters specific for disk index construction.
#[derive(Clone, PartialEq, Debug)]
pub struct DiskIndexBuildParameters {

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

diskann-disk/src/build/configuration/disk_index_build_parameter.rs:204

• disk_index_build_parameter.rs enables #![warn(missing_docs)], and the new pub(crate) fn pipnn_config has no doc comment. If CI promotes warnings to errors, this will fail builds; even if not, it adds new lint noise.

    #[cfg(feature = "pipnn")]
    pub(crate) fn pipnn_config(&self) -> Option<diskann_pipnn::PiPNNConfig> {
        match &self.build_algorithm {
            BuildAlgorithm::PiPNN(config) => Some(config.into()),
            BuildAlgorithm::Vamana => None,
        }
    }

diskann-disk/src/build/builder/build/pipnn.rs:49

• pipnn::build_graph reads the full dataset into memory (read_bin) and then calls find_medoid_with_sampling, which performs another full pass over the dataset via VectorDataIterator (plus a sampled centroid pass). On large datasets this doubles I/O and parsing work even though the vectors are already in memory.

    let data =
        read_bin::<Data::VectorDataType>(&mut builder.storage_provider.open_reader(&data_path)?)?;
    let context = PiPNNBuildContext::new(
        config,
        &builder.index_configuration.config,

diskann-providers/src/storage/bin.rs:388

• save_adjacency_graph is re-exported as a public API, but its signature exposes diskann::graph::AdjacencyList<u32>, forcing downstream callers to take a dependency on diskann graph types just to use diskann-providers serialization. If the goal is a narrow cross-crate adjacency entry point, consider changing this API to accept adjacency rows as plain slices (or any type that derefs/borrows to &[u32]) to avoid leaking diskann internals into diskann-providers’ public surface.

pub fn save_adjacency_graph<P>(
    adjacency: &[AdjacencyList<u32>],
    max_degree: u32,
    provider: &P,
    start_point: u32,

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 23.21429% with 43 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (pipnn-stack/03-core@857e200). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
diskann-providers/src/storage/bin.rs	0.00%	33 Missing ⚠️
...nn-disk/src/build/configuration/build_algorithm.rs	66.66%	4 Missing ⚠️
.../build/configuration/disk_index_build_parameter.rs	50.00%	3 Missing ⚠️
diskann-providers/src/utils/rayon_util.rs	0.00%	3 Missing ⚠️

❌ Your patch status has failed because the patch coverage (23.21%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@                  Coverage Diff                   @@
##             pipnn-stack/03-core    #1291   +/-   ##
======================================================
  Coverage                       ?   90.72%           
======================================================
  Files                          ?      521           
  Lines                          ?   101084           
  Branches                       ?        0           
======================================================
  Hits                           ?    91709           
  Misses                         ?     9375           
  Partials                       ?        0           

Flag	Coverage Δ
miri	90.72% <23.21%> (?)
unittests	90.40% <23.21%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-disk/src/build/builder/build.rs	92.43% <100.00%> (ø)
.../build/configuration/disk_index_build_parameter.rs	94.69% <50.00%> (ø)
diskann-providers/src/utils/rayon_util.rs	96.32% <0.00%> (ø)
...nn-disk/src/build/configuration/build_algorithm.rs	66.66% <66.66%> (ø)
diskann-providers/src/storage/bin.rs	72.95% <0.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Suppressed comments (2)

diskann-providers/src/storage/bin.rs:388

• save_adjacency_graph is a new public entry point that can currently write an invalid graph file: it does not validate that start_point is within adjacency.len(), and it trusts the caller-provided max_degree even if some rows are longer. load_graph sizes its buffer from the header max-degree and will panic if any stored row length exceeds that header value, so accepting max_degree < observed degree can produce a graph that crashes on load.

pub fn save_adjacency_graph<P>(
    adjacency: &[AdjacencyList<u32>],
    max_degree: u32,
    provider: &P,
    start_point: u32,

diskann-disk/src/build/builder/build/pipnn.rs:88

• build_graph reads the entire dataset into memory via read_bin, but then immediately re-reads/scans the dataset again via find_medoid_with_sampling to pick the start node. For large datasets/providers this doubles I/O and can become a noticeable build-time cost. Consider adding/using a medoid helper that operates over the already-loaded matrix view (or a row iterator over it) so PiPNN can reuse the in-memory data instead of reopening/scanning the file again.

    let mut rng = diskann_providers::utils::create_rnd_from_optional_seed(
        builder.index_configuration.random_seed,
    );
    let (_, start_id) = find_medoid_with_sampling::<Data::VectorDataType, _>(
        &data_path,

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Suppressed comments (2)

diskann-providers/src/storage/bin.rs:388

• save_adjacency_graph can write a graph file that load_graph cannot read if any row has more than max_degree neighbors: load_graph allocates a buffer of length max_degree and slices it to num_neighbors, which will panic/out-of-bounds when num_neighbors > max_degree. The wrapper should ensure the header max degree is at least the observed maximum (or fail early).

pub fn save_adjacency_graph<P>(
    adjacency: &[AdjacencyList<u32>],
    max_degree: u32,
    provider: &P,
    start_point: u32,

diskann-providers/src/utils/rayon_util.rs:81

• as_rayon is documented as a borrow, but it consumes self. While RayonThreadPoolRef is Copy, taking &self better matches the API intent and avoids forcing a move in call sites that hold the wrapper in a non-Copy context later.

    /// Borrow the underlying pool for APIs that retain a caller-owned pool.
    pub fn as_rayon(self) -> &'a rayon::ThreadPool {
        self.0
    }

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Suppressed comments (2)

diskann-providers/src/storage/bin.rs:398

• save_adjacency_graph is a new public entry point but it doesn’t validate that start_point is within adjacency.len() or that any row’s degree is <= max_degree. If callers pass an out-of-range start point or an overfull row, this will write a graph whose header/contents are inconsistent and can later fail (or mis-size allocations) when loading.

    save_graph(
        &AdjacencyGraph {
            adjacency,
            max_degree,
        },

diskann-disk/src/build/configuration/disk_index_build_parameter.rs:113

• DiskIndexBuildParameters previously implemented Copy; adding build_algorithm: BuildAlgorithm (which can contain heap data like Vec) drops Copy. This is an API-breaking change for downstream users that may have relied on implicit copies (e.g., passing params by value multiple times). If this is intended, it likely needs an explicit release note / versioning consideration; otherwise consider keeping the algorithm config outside this public params struct (or exposing a separate PiPNN params type) to preserve the prior Copy API surface.

/// Parameters specific for disk index construction.
#[derive(Clone, PartialEq, Debug)]
pub struct DiskIndexBuildParameters {
    /// Memory budget for disk-index pipeline stages that support bounded work.
    /// Explicit one-shot PiPNN selection is never silently replaced.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Suppressed comments (2)

diskann-providers/src/storage/bin.rs:398

• save_adjacency_graph writes a header max_degree that is later trusted by load_graph to size its read buffer; if any adjacency row is longer than max_degree, load_graph will panic when slicing the buffer. Since this is now a public entry point, it should defensively validate that the observed max row length does not exceed the configured max_degree (and return an error if it does) before delegating to save_graph.

    save_graph(
        &AdjacencyGraph {
            adjacency,
            max_degree,
        },

diskann-disk/src/build/configuration/disk_index_build_parameter.rs:113

• DiskIndexBuildParameters used to be Copy, but the new build_algorithm: BuildAlgorithm field (and the PiPNN variant’s Vec) forces dropping Copy for all builds. Since pipnn is an optional feature and the default build still only has BuildAlgorithm::Vamana, consider preserving Copy for the common non-pipnn build via cfg_attr(not(feature = "pipnn"), derive(Copy)) (and similarly making BuildAlgorithm Copy when pipnn is disabled). This reduces downstream breakage for consumers not enabling PiPNN.

/// Parameters specific for disk index construction.
#[derive(Clone, PartialEq, Debug)]
pub struct DiskIndexBuildParameters {
    /// Memory budget for disk-index pipeline stages that support bounded work.
    /// Explicit one-shot PiPNN selection is never silently replaced.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

diskann-providers/src/storage/bin.rs:398

• save_adjacency_graph() forwards a caller-supplied max_degree into the file header without validating it against the actual adjacency row lengths. If any row is longer than max_degree, load_graph() will panic because it allocates a buffer of size max_degree and then slices it to num_neighbors. Consider validating and returning an ANNError when observed_max_degree > max_degree.

    save_graph(
        &AdjacencyGraph {
            adjacency,
            max_degree,
        },

diskann-disk/src/build/builder/build/pipnn/tests.rs:154

• This test currently hard-codes the serialized graph header max_degree to the pruned degree (32). If the adapter writes the canonical max_degree (Config::max_degree_u32) like the Vamana pipeline, this assertion will fail even though the header is correct.

    assert_eq!(u32::from_le_bytes(header[8..12].try_into().unwrap()), 32);