PiPNN 3/6: add core graph construction

[SeliMeli]

Adds the provider-independent PiPNN graph-construction pipeline.

Code map

1. lib.rs defines configuration, validates graph-policy compatibility, and orchestrates partition → leaf candidates → finalization inside the caller-owned Rayon pool.
2. partitioning.rs runs deterministic replicas. Each oversized work item samples leaders, gathers point/leader rows, uses GEMM plus partition_kernel for assignments, and recurses only on oversized clusters.
3. Large assignment scatters use at most one partial per pool worker, then merge each leader in parallel. Assignment row tiles are rounded down to power-of-two sizes under the 512 KiB target to avoid repeated GEMM tail shapes.
4. global_merge_small combines sub-c_min leaves without exceeding c_max; final validation rejects empty/oversized leaves.
5. leaf_build.rs gives each Rayon job reusable buffers. Active prefixes are passed explicitly because numeric buffers retain their high-water length. Each leaf gathers rows, computes lower A · Aᵀ, runs the dual-endpoint top-k kernel, and merges symmetric candidates.
6. finalization.rs leaves degree-bounded rows unchanged and sends only overfull rows through the shared RobustPrune kernel.

Review path

• Verify seed derivation and output ordering across replicas, recursion levels, worker-count scatter, and small-leaf merge.
• In leaf_build, check the sorted-ID duplicate fast path and its HashSet fallback, plus every active-prefix slice after grow-only buffer reuse.
• Confirm scratch ownership: partition chunks lease buffers from a stage-owned pool and return them after computation; the mutex is held only for pop/push, never across GEMM. Leaf jobs still own their MapInit state, and the outer leaf value is consumed by the stage. There is no TLS or cleanup broadcast.
• The crate boundary intentionally contains no provider, start-point, PQ, serialization, or search lifecycle.

Stack 3/6: #1288 → #1291

[Copilot]

Pull request overview

This PR introduces the “core” PiPNN build pipeline in the diskann-pipnn crate, wiring together deterministic partitioning, leaf-local candidate construction, and final pruning into a public build_graph API with a validated build context.

Changes:

• Adds PiPNNConfig validation and a PiPNNBuildContext that binds PiPNN policy to DiskANN graph pruning policy and a caller-owned Rayon thread pool.
• Implements the three main stages: partitioning (partitioning.rs), leaf candidate construction (leaf_build.rs), and final pruning via shared Vamana robust prune (finalization.rs).
• Adds comprehensive unit/integration tests and a Criterion benchmark for core scenarios; updates dependencies, lockfile, and mutation-test exclusions.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
diskann-pipnn/src/lib.rs	Adds public PiPNN API (PiPNNConfig, PiPNNBuildContext, build_graph) and stage orchestration.
diskann-pipnn/src/partitioning.rs	Implements deterministic overlapping partition construction and leader assignment/scatter.
diskann-pipnn/src/partitioning/tests.rs	Adds unit tests covering partition determinism, invariants, error cases, and helpers.
diskann-pipnn/src/leaf_build.rs	Builds leaf-local symmetric k-NN candidates and accumulates global candidates safely in parallel.
diskann-pipnn/src/leaf_build/tests.rs	Adds unit tests for candidate correctness, invariants, type support, and error handling.
diskann-pipnn/src/finalization.rs	Orders/prunes candidate rows using shared robust_prune and validates candidate IDs/shape.
diskann-pipnn/src/finalization/tests.rs	Adds unit tests for pruning behavior and candidate validation failures.
diskann-pipnn/src/tests.rs	Tests effective_metric behavior for integer cosine-normalized handling.
diskann-pipnn/tests/config.rs	Integration tests for config validation and graph-policy compatibility checks.
diskann-pipnn/tests/build_graph.rs	Integration tests for end-to-end graph building, invariants, determinism, and type/metric support.
diskann-pipnn/benches/core.rs	Adds a Criterion benchmark for stage-focused core build scenarios.
diskann-pipnn/Cargo.toml	Updates crate dependencies/dev-dependencies and registers the new core benchmark target.
Cargo.lock	Records dependency graph changes for the updated diskann-pipnn crate dependencies.
.cargo/mutants.toml	Adds mutation-test exclusions for key PiPNN public boundary checks and partitioning invariants.

Comments suppressed due to low confidence (1)

diskann-pipnn/src/partitioning.rs:604

• size_of::<f32>() is used without being in scope (no use std::mem::size_of; and not qualified), so this function won’t compile as written.

fn assignment_stripe_rows(leaders: usize) -> usize {
    (ASSIGNMENT_CACHE_TARGET_BYTES / (leaders.max(1) * size_of::<f32>()))
        .clamp(MIN_ASSIGNMENT_STRIPE_ROWS, MAX_ASSIGNMENT_STRIPE_ROWS)
}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

diskann-pipnn/src/partitioning.rs:637

• size_of is used without being in scope (std::mem::size_of), which will not compile. Qualify the call or import it.

fn assignment_stripe_rows(leaders: usize) -> usize {
    (ASSIGNMENT_CACHE_TARGET_BYTES / (leaders.max(1) * size_of::<f32>()))
        .clamp(MIN_ASSIGNMENT_STRIPE_ROWS, MAX_ASSIGNMENT_STRIPE_ROWS)
}

diskann-pipnn/src/partitioning.rs:19

• Norm is imported but never used in this module, which will trip unused_imports warnings (and can become CI failures under -D warnings). Remove it from the import list.

use diskann::{utils::VectorRepr, ANNError, ANNResult};
use diskann_linalg::Transpose;
use diskann_utils::views::MatrixView;
use diskann_vector::{distance::Metric, norm::FastL2NormSquared, Norm};
use rand::{prelude::IndexedRandom, SeedableRng};
use rayon::prelude::*;

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

diskann-pipnn/src/partitioning.rs:390

• gather_rows uses TypeId::of::<T>(), which implicitly requires T: 'static. Making that bound explicit here avoids surprising/indirect trait-bound errors later and matches the public build_graph boundary (which already requires 'static).

fn gather_rows<T>(data: MatrixView<'_, T>, indices: &[u32], output: &mut [f32]) -> ANNResult<()>
where
    T: VectorRepr,

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

diskann-pipnn/src/partitioning.rs:641

• size_of::<f32>() is used without being imported or qualified, which will fail to compile. Qualify it with std::mem::size_of (or add an explicit import).

    let rows = ASSIGNMENT_CACHE_TARGET_BYTES / (leaders.max(1) * size_of::<f32>());

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (4)

diskann-pipnn/src/leaf_build.rs:222

• build_leaf is executed from a Rayon parallel context (via build_leaf_candidates), so it should also explicitly require T: Send + Sync to reflect the actual thread-safety requirement.

where
    T: VectorRepr + 'static,
{

diskann-pipnn/src/leaf_build/tests.rs:154

• assert_source_type forwards T into the parallel leaf build path, so it should also include Send + Sync bounds to match the production requirements.

fn assert_source_type<T>(data: &[T])
where
    T: diskann::utils::VectorRepr + 'static,
{

diskann-pipnn/src/leaf_build.rs:193

• build_leaf_candidates uses Rayon parallel iteration over data, so T must be Send + Sync. Making this explicit in the signature avoids confusing trait-bound errors at call sites and documents the thread-safety requirement.

This issue also appears on line 220 of the same file.

where
    T: VectorRepr + 'static,
{

diskann-pipnn/src/leaf_build/tests.rs:35

• This test helper calls build_leaf_candidates, which (via Rayon) requires T: Send + Sync. Add the bounds here so the test continues to compile once the production signature is tightened.

This issue also appears on line 151 of the same file.

where
    T: diskann::utils::VectorRepr + 'static,
{

[Copilot]

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (1)

diskann-pipnn/src/partitioning.rs:492

• For Metric::Cosine, row_scales are computed via FastL2NormSquared, which goes through the SIMD-optimized InnerProduct implementation. This can reassociate the reduction and change low bits across architectures/builds, undermining the determinism guarantee you’re already preserving for leader_scales via a scalar sum.

Since these row norms participate in the normalized cosine score, they can also affect near-tie leader ordering. Consider computing the squared norm with the same scalar reduction order used for leaders.

    let row_scales = if metric == Metric::Cosine {
        grow_fallible(row_scale_buffer, rows, 0.0)?;
        let row_scales = &mut row_scale_buffer[..rows];
        for (scale, row) in row_scales
            .iter_mut()
            .zip(point_values.chunks_exact(dimensions))
        {
            *scale = FastL2NormSquared.evaluate(row);
        }
        &*row_scales

[Copilot]

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

diskann-pipnn/src/leaf_build.rs:170

• This doc comment says “duplicate removal are deferred”, but candidates.extend_from_slice(additions) already enforces uniqueness on insertion (and de-dupes within additions). Only sorting is deferred until the end, so the comment is misleading.

/// A point may appear in several overlapping leaves, so workers lock only the
/// destination list long enough to append one leaf's additions. Sorting and
/// duplicate removal are deferred until all leaves finish; doing either under
/// the lock would lengthen the contended section for no semantic benefit.