ruvector-delta-index test_insert_and_search hangs indefinitely (DeltaHnsw insert/search)

[ruvnet]

Summary

ruvector-delta-index::tests::test_insert_and_search (crates/ruvector-delta-index/src/lib.rs:731) hangs indefinitely rather than failing. It is not slow — it never terminates.

Evidence

Observed on CI run 31675583441 (PR #822, ci/split-core-platform). The core-platform shard sat in a nextest SLOW loop on this single test for 3h 52m until the job was cancelled at the 240-minute cap. No other test in the shard reported progress after it started.

Reproduction / nondeterminism

The test builds its input from random_vector(128), which uses an unseeded rand::thread_rng():

fn random_vector(dim: usize) -> Vec<f32> {
    use rand::Rng;
    let mut rng = rand::thread_rng();
    (0..dim).map(|_| rng.gen()).collect()
}

#[test]
fn test_insert_and_search() {
    let mut index = DeltaHnsw::new(128, DeltaHnswConfig::default());
    for i in 0..100 {
        let vec = random_vector(128);
        index.insert(&format!("vec_{}", i), vec).unwrap();
    }
    assert_eq!(index.len(), 100);
    let query = random_vector(128);
    let results = index.search(&query, 10).unwrap();
    assert_eq!(results.len(), 10);
}

Because the vectors are unseeded, the hang is nondeterministic — it reproduces only for graph topologies that trigger the bug. This is consistent with the test having passed on earlier runs before the core-platform shard split surfaced it.

The hang is almost certainly an unterminated loop in DeltaHnsw::insert or DeltaHnsw::search — a candidate-set/visited-set traversal that fails to make progress (e.g. a neighbor list containing a cycle, or a while over a candidate heap that re-pushes an already-visited node).

Impact on CI

Until this is fixed, a hung test burns the full 240-minute job budget. Two mitigations landed in PR #822:

1. .config/nextest.toml now sets slow-timeout = { period = "120s", terminate-after = 5 }, so any test is killed and reported as a failure after 10 minutes instead of running forever.
2. ruvector-delta-index is temporarily held out of CI tests entirely — removed from the core-platform shard's package list and kept in the core-and-rest catch-all's --exclude list — so the crate does not block the pipeline.

Fix checklist

• Root-cause the non-terminating loop in DeltaHnsw::insert / DeltaHnsw::search
• Add a progress/iteration bound so a malformed graph fails loudly instead of hanging
• Seed the test RNG (or use a fixed fixture) so the case is deterministic and regression-testable
• Re-add -p ruvector-delta-index to the core-platform shard and drop its --exclude from core-and-rest in .github/workflows/ci.yml

[ruvnet]

Confirming this is still live, and that it is very likely the reason every PR shows a red Tests (core-and-rest).

Still failing today

Tests (core-and-rest) fails at exactly the 240-minute cap on every PR I sampled — not intermittently:

PR	result
#926 (opened today)	fail, 4h00m20s
#834	fail, 4h00m16s
#804	fail, 4h00m17s

A job that dies at precisely timeout-minutes on every branch is a hang, not slowness — which matches this issue's "it never terminates" exactly.

The shard-truncation bug is not the cause

Worth separating, because the two get conflated: the #-inside-a-folded-scalar bug that silently ate all the --exclude flags was fixed on 2026-08-13 in 09cccfebe. I re-checked main — no packages: scalar contains a #, and core-and-rest carries its full 162 excludes. That fix is in and holding. The 4h failures above are all after it, so something else is hanging, and this issue describes it.

#787 appears to be the fix

#787 identifies DeltaHnsw::connect_node taking a write guard on a node and then a read() on that same node — parking_lot's RwLock is not reentrant, so the thread parks at 0% CPU forever. That is precisely the symptom reported here.

It also catches a second one the obvious fix would have missed: prune_neighbors calls distance, which reads every node in the list being pruned — including, for any list containing its own owner, the node whose guard is still held. So the fix has to move the prune outside the guard, not just the vector read.

The circular part

#787 is MERGEABLE, and its only failing check is Tests (core-and-rest) — the check it fixes. It cannot go green until it lands, and it will not land while people wait for it to go green. That is worth breaking deliberately: the change is one file, the reasoning is specific, and the alternative is that the repo keeps accumulating PRs against a permanently red required check (100 open right now, oldest from 2026-05-30).

Suggest merging #787 on the strength of its targeted evidence and then confirming core-and-rest on main, rather than treating the red check as a gate on its own fix.

[ruvnet]

Dream-cycle evidence correction (2026-08-25)

The DeltaHnsw deadlock and PR #787 remain credible as a separate defect/fix pair, but current evidence rejects the claim that #787 is the cause-level fix for every present core-and-rest 240-minute cancellation.

On current main, ruvector-delta-index is explicitly excluded from both the core-platform and core-and-rest test package sets. In PR #926's Workspace run, 16 of 17 jobs succeeded, while core-and-rest job 97356253593 reached the timeout while Cargo/Rustc were still compiling ruvector-temporal-tensor-wasm. No DeltaHnsw test executed in that job.

Frozen hypothesis: PR #787 alone will stop the current core-and-rest 240-minute cancellations.

Acceptance required both:

1. the focused delta-index suite to finish under two minutes; and
2. an attributable core-and-rest run to complete under 240 minutes.

The first is supported by #787's focused evidence; the second is not, and the active compiler trace points elsewhere.

Verdict: REJECT

Recommended separation:

• Keep this issue and fix(delta-index): connect_node self-deadlocks pruning a neighbour list #787 focused on the independently reproduced DeltaHnsw self-deadlock.
• Track the present core-and-rest timeout as a compile-time/sharding investigation: separate compile from test duration, identify the active compiler bottleneck, then produce one complete run below the 240-minute cap before attributing recovery to fix(delta-index): connect_node self-deadlocks pruning a neighbour list #787.

This correction does not dispute #787's targeted lock-order fix; it narrows the unsupported portfolio-wide causal claim.

[ruvnet]

Retracting my comment above. The causal claim in it is wrong, and the recommendation that followed from it should not be acted on.

I claimed the DeltaHnsw hang was "very likely the reason every PR shows a red Tests (core-and-rest)" and suggested merging #787 to unblock CI. Both halves fail on evidence that was available when I wrote it:

ruvector-delta-index is excluded from that shard. .github/workflows/ci.yml on main carries --exclude ruvector-delta-index in the core-and-rest package set, plus two comments saying so explicitly — one of which cites this issue by number as the reason it was held out. A crate that never runs in a shard cannot be hanging it.

The job I cited never reached tests. Job 97231570606 was still emitting Compiling ruvector-temporal-tensor-wasm, ruvector-profiler, ruvector-dither and others when it hit the 240-minute cap. It timed out compiling, not hanging.

The reasoning error is worth naming, because the shape of it is reusable: I saw a hang-forever issue and a job dying at exactly timeout-minutes, matched the symptom, and treated "this issue is referenced near the shard config" as evidence of causation — when the same file, a few lines away, said the crate was excluded because of this issue. The reference was a record of the problem already being handled, and I read it as the problem still being live.

What survives: #787 is still a real fix for a real self-deadlock (connect_node taking a write guard then a read guard on the same node, parking_lot being non-reentrant), and it is still the prerequisite for putting ruvector-delta-index back into CI at all. That case stands on its own merits. What does not survive is any expectation that merging it turns core-and-rest green — the 4-hour cancellations are a compile-time/shard-size problem, tracked separately in #928.

Apologies to anyone who read the earlier comment as a green light to merge #787 for CI reasons.