Repository navigation
fix(bench): stable CELLD_NODE + lab same-host supersede (pre-v4) - #1456
Open
danielsmithdevelopment wants to merge 11 commits into
Open
danielsmithdevelopment wants to merge 11 commits into
danielsmithdevelopment wants to merge 11 commits into
Conversation
Stop minting a fresh node id on every restart (v3's 712-registration spiral on stock 0.6.3 without lease-sweep). Board defaults to fleet_A/ fleet_B with post-fence TTL wait; FLEET_PHASED_FRESH_NODE_IDS=1 restores legacy. Add deploy-aware soak that asserts unique nodes/* ≤2 under rebuild pressure before the next N=1000. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
240 rebuild cycles on stock 0.6.3 with stable CELLD_NODE: unique nodes/* stayed at 2 (primary gate pass). Record 14 self-fences + 6 ready flakes honestly; score ok on node ceiling, not zero readyFails. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Primary gate PASS (unique nodes=2 over 240/~3.75h); note ready flakes and self-fence recoveries without claiming upstream GC or authorizing v4. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Node-ceiling-only passed the multi-hour soak but ready≈2.5% and self-fence≈5.8% each already beat the board's 2% ceiling. Require all three: nodes≤2, readyFail rate≤0.5%, selfFence rate≤0.5%. Re-score evidence as failed_v4_rate_gates; document stable+incarnation+sweep+ alive path; do not start v4 until measured_v4_ready. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Require ≥1000 cycles for measured_v4_ready (600 rule-of-three floor; Wilson zero-event needs ~765). Report Wilson 95% UBs with each rate. Define incarnation supersede only when pid+startTime+bootId prove the old same-host process is gone; cross-host wait/refuse. Order before v4 frozen in SOAK_GATES.md. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
At 1000 cycles, Wilson UB≤0.5% means no readyFails and no selfFences — one stray event (~0.56% UB) fails. Keep that bar deliberately for v4 headroom after v2/v3; do not relax to ~1% after the fact. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Align STATUS and run-node-soak honesty with SOAK_GATES: at n=1000, Wilson UB≤0.5% means 0 readyFails and 0 selfFences. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Stop unfixed overnight soak (cannot authorize v4); land baseline evidence at 60/1000 with readyFails=2 selfFences=2. Add vendor/celld-lab from stock 0.6.3 with /alive + timed readiness, maintenance phase budget, and lease host_process stamps for upcoming same-host supersede. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Wire incarnation supersede into predecessor recovery, dead-leader sweep, and initial lease CAS: fence/replace only when expiry passed or same-host pid+start+boot proves the prior process is gone; live/cross-host wait TTL. 12-cycle lab smoke: 0 readyFails, 0 selfFences with supersede log lines. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
cursor
Bot
changed the title
Sticky lastExit.selfFence re-waited TTL every restart and risked re-counting. Abort evidence documents correlated dual watchdog fences after ~17m host gap (not supersede) on the first lab authorizing attempt. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Second attempt hit the same correlated watchdog fence after ~15m VM pause while the agent was idle on a 30m progress timer. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pre-v4 availability path on stable
CELLD_NODE, with a lab celld that carries the agreed product order (not stock 0.6.3).Product order
cycles ≥ 1000, 0 readyFails, 0 selfFences)This PR
CELLD_NODEsoak harness + strict SOAK_GATESnode-soak-baseline-unfixed-0.6.3.json) — non-authorizingvendor/celld-lab: tip 0.6.3 + alive/sweep +host_process+ same-host supersede claim pathnode_lease_watchdog_fencefollowing a ~17.7 min host suspend gap (not supersede) — seenode-soak-lab-authorizing-suspend-aborted.jsonCELLD_BIN→node-soak-lab-authorizing.jsonLab build
cargo build -p celld --release --manifest-path vendor/celld-lab/Cargo.toml CELLD_BIN=$PWD/vendor/celld-lab/target/release/celld npm run benchmark:fleet-node-soak