feat(concurrent): Implement ConcurrentDynamicVamana orchestrator and associated tests

[rfsaliev]

This pull request introduces significant improvements to the concurrent dynamic Vamana index, focusing on allocator flexibility, type safety, and enhanced assembly and construction options. The changes make it easier to use custom allocators for graph storage, provide robust mechanisms for assembling indices from existing data, and improve the underlying type traits and loading mechanisms to ensure correctness and extensibility.

Key changes include:

Allocator and Storage Flexibility

• SimpleBlockedGraph is now templated on its allocator type, allowing users to specify custom allocators instead of being limited to HugepageAllocator. All loading and construction methods in SimpleBlockedGraph now accept an allocator parameter, making it easier to control memory management.
• The graph's reverse-edge storage now uses a conditional allocator: it attempts to use the graph's base allocator if possible, otherwise falls back to std::allocator. This is determined by a new reverse_edges_rebindable_v trait. [1] [2] [3]

Type Traits and Compile-Time Safety

• Added the is_segmented_blocked_v trait to detect at compile-time whether an allocator or dataset uses segmented blocked storage, ensuring that only compatible types are used with the concurrent index.

Index Construction and Assembly

• Introduced a new auto_dynamic_build function that builds a concurrent dynamic Vamana index using grow-stable (segmented blocked) storage for both data and graph, enforcing allocator compatibility at compile time.
• Added a new constructor to MutableVamanaIndex that assembles an index from arbitrary data, graph, status, and ID translator, with robust precondition checks to ensure consistency and correctness.

API Improvements and Documentation

• Exposed new methods in MutableVamanaIndex for accessing the ID translator and obtaining a snapshot of slot statuses, improving observability and debugging.
• Updated the documentation (README.md) to describe the new type-erased wrapper and allocator selection mechanisms for concurrent dynamic Vamana.

These changes collectively make the concurrent dynamic Vamana index more flexible, safer, and easier to use in a variety of memory management scenarios.

[Copilot]

Copilot review overview

🟡 Changes recommended

Assembly can accept relocation-prone graph storage, and allocator detection and constructor overload resolution have compile-time defects.

Review effort: Balanced
Findings: 3 · 1

Open (4)

• Exclude logger-convertible arguments from allocator overload · New
• Make allocator rebind fallback SFINAE-safe · New
• Enforce segmented graph storage in concurrent assembly · New
• Add coverage for state-assembly validation and slot reuse · New

What changed in this PR

Adds the type-erased concurrent dynamic Vamana orchestrator with allocator-aware graph storage, assembly support, and integration tests.

Changes:

• Adds concurrent build, mutation, iteration, persistence, and concurrency APIs/tests.
• Generalizes blocked graph allocators and reverse-edge allocation.
• Adds segmented-storage traits and state-based index assembly.

File	Description
tests/​svs/​orchestrators/​concurrent_dynamic_vamana.cpp	Adds orchestrator integration tests.
tests/​CMakeLists.txt	Registers the new tests.
include/​svs/​orchestrators/​vamana_iterator.h	Selects index-specific iterator types.
include/​svs/​orchestrators/​concurrent_dynamic_vamana.h	Adds the concurrent type-erased wrapper.
include/​svs/​concurrent/​README.md	Documents allocator and wrapper APIs.
include/​svs/​concurrent/​graph.h	Adds allocator-flexible blocked graphs.
include/​svs/​concurrent/​dynamic_index.h	Adds allocator-aware build and state assembly.
include/​svs/​concurrent/​blocked_data.h	Adds segmented allocator detection.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

Allocator selection, translator validation, and iterator factory handling contain unresolved correctness and performance issues.

Review effort: Balanced
Findings: 1 · 3

Open (4)

• Enforce segmented graph storage in concurrent assembly
• Validate translator coverage for every valid slot · New
• Const-correct allocator equality to preserve huge-page allocation · New
• Initialize iterator implementation from the index factory · New

Resolved since last review (3)

• Make allocator rebind fallback SFINAE-safe
• Exclude logger-convertible arguments from allocator overload
• Add coverage for state-assembly validation and slot reuse

[ahuber21]

I am worried about drift between sequential and concurrent code and I'm wondering why we can't pick the same approach as in #380. I have a suggestion below. Let's discuss why it's not the right choice:

Suggestion: pass the graph allocator the way #380 does for the sequential index. In #380 the sequential auto_dynamic_build builds the graph and hands it to a ready-graph build constructor. Here the same capability comes in a second way: an allocator overload of the build constructor, a std::monostate sentinel, and make_graph. Mirroring the sequential shape would:

• leave the existing build constructor exactly as it is on `main;
• drop the requires clause, so the nullptr logger problem Copilot raised goes away;
• allocate the graph once on the allocator path instead of twice.

I tried it locally to check that it holds. auto_dynamic_build becomes:

auto verified_parameters = parameters;
verify_and_set_default_index_parameters(verified_parameters, distance);
graph_type graph{data.size(), verified_parameters.graph_max_degree, graph_allocator};
auto entry_point = data.size() == 0 ? 0 : extensions::compute_entry_point(data, threadpool);
return MutableVamanaIndex<graph_type, data_type, Distance>(
    verified_parameters, std::move(graph), std::move(data), lib::narrow<Idx>(entry_point),
    std::move(distance), external_ids, std::move(threadpool), std::move(logger));

Points I'm unsure about:

• Sequential has no reverse edges. Not sure if that already breaks the option to mirror #380.
• In sequential, direct callers of MutableVamanaIndex cannot pass an allocator to the build-from-scratch constructor. They have to allocate the graph and compute the entry point themselves, then call the ready-graph constructor. Is this a limitation we could fix for both versions, assuming we decide to mirror the patterns?

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 3 · 3

Open (6)

• reverse_edges_rebindable_v intends to evaluate to false for allocators that cannot rebind to… · New
• ConcurrentDynamicVamana::build(...) takes std::span<const size_t> but this header does not… · New
• This test file uses std::span (line ~200) and std::min (line ~300) but does not include… · New
• The assemble(...) overload takes const Distance&, but later forwards into make(...) and uses… · New
• Initialize iterator implementation from the index factory
• Const-correct allocator equality to preserve huge-page allocation

Resolved since last review (2)

• Enforce segmented graph storage in concurrent assembly
• Validate translator coverage for every valid slot

[Copilot]

Copilot review overview

🟡 Changes recommended

Construction retains costly reverse-edge bookkeeping, and important concurrency, allocator, and state-restoration paths need stronger coverage.

Review effort: Balanced
Findings: 4

Open (4)

• Reverse-edge bookkeeping remains enabled during construction · New
• Tests do not cover loading graphs with empty or deleted slots · New
• Tests do not cover custom graph allocator propagation · New
• Concurrency test lacks a barrier to ensure overlapping operations · New

Resolved since last review (6)

• This test file uses std::span (line ~200) and std::min (line ~300) but does not include…
• ConcurrentDynamicVamana::build(...) takes std::span<const size_t> but this header does not…
• reverse_edges_rebindable_v intends to evaluate to false for allocators that cannot rebind to…
• The assemble(...) overload takes const Distance&, but later forwards into make(...) and uses…
• Initialize iterator implementation from the index factory
• Const-correct allocator equality to preserve huge-page allocation

[rfsaliev]

I am worried about drift between sequential and concurrent code and I'm wondering why we can't pick the same approach as in #380.
@ahuber21, I've updated concurrent .ctors to match non-concurrent MutableVamanaIndex.

[Copilot]

🔵 Needs a closer look

Empty-index construction can retain an invalid entry point, and key allocator and hole-aware paths lack coverage.

0 open findings

4 resolved since last review

• Concurrency test lacks a barrier to ensure overlapping operations
• Tests do not cover custom graph allocator propagation
• Tests do not cover loading graphs with empty or deleted slots
• Reverse-edge bookkeeping remains enabled during construction

🧠 Review effort: Balanced

[ahuber21]

Two more small comments.

[Copilot]

🟡 Changes recommended

The assembly constructor breaks an existing public signature, and key concurrent and hole-aware behavior lacks deterministic coverage.

3 open findings

• Preserve existing post-reload constructor overload · New
• Correct namespace for SegmentedBlockedData · New
• Correct documented namespace for allocator · New

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[mergify]

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.