Repository navigation
feat(bench): digest-equal statePreserved + trialAccounting before N=1000 - #1449
Open
danielsmithdevelopment wants to merge 31 commits into
Open
danielsmithdevelopment wants to merge 31 commits into
danielsmithdevelopment wants to merge 31 commits into
Conversation
Tighten claim-path fleet measurement: exact step/result digest equality vs per-trial no-crash twins (deterministic mock owner ids), explicit trialAccounting denominators, and honest read/model-call limits in docs. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Commit claim-path pass gates in durability STATUS before the lab run lands; document that fleet digest equality uses a deterministic mock while real APIs depend on journal replay; HOLD density climb until swap/live-heap characterize the ~8GB@102.5k resident slope. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Survivor SELF-FENCE previously hung the N=1000 lab in unbounded ready waits. Cap rebuild/restart attempts, catch twin/rebuild failures, and mark mid-run evidence status=partial with ok=false until all arms finish. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Classify outer misses as availability (deadline / pause fetch) not state divergence; state miss is twin_capture_failed not digest_mismatch. Document next-run options without loosening frozen criteria. Make twin capture retry-safe and record per-trial reason for the next board. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
…re re-run Option 2 chosen: split wrong outcomes (correctness, zero) from no-outcome-in-time (availability, ≤2% ceiling + resume p99/max). Criteria dated before the next board. Harness FLEET_PHASED_SCORE_MODE=correctness_availability_v2; v1 zero-deadline fail stays published. Celld availability fixes remain parallel, not blocking. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Fresh CELLD_NODE + port settle on restart; twin prefer ensureSurvivor over rebuild storms; v2 gates report recoveryMs p99/max; document lease-sweep / alive-vs-ready / pause-aware tracks as parallel product work that does not block the correctness/availability board. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Freeze the resume deadline clock while ensureSurvivor/rebuild runs (FLEET_PHASED_PAUSE_AWARE_DEADLINE default on). Wall resumeMs still includes recovery so multi-minute stalls stay visible; evidence also records resumeActiveMs. Completes the in-repo half of the parallel celld triad from option-2 (lease-sweep/alive-vs-ready remain upstream). Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Reverts the ensureSurvivor/rebuild deadline freeze — a multi-minute orchestrator stall is still a miss from the user's point of view. resumeActiveMs/recoveryMs remain diagnostic only (no pass/fail ceiling). Clarify pause-aware clocks as host pause/suspend (still open on celld). Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Lock FLEET_PHASED_RESUME_DEADLINE_MS=60000 and availability denominator = 750 write-kill + 250 await before the re-run. Record frozenCriteria in v2 evidence; note ~1/12 binomial chance of honest >2% at ~1.5% true rate. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Echo locked deadline, 2% ceiling, and 1000-trial denominator into the evidence JSON so post-hoc loosening is visible against the run file. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Final measured evidence under frozen zero-deadline criteria: 0 dups / 0 divergence / 0 write reruns; 7 deadline misses; outer 369/373; statePreserved 368/369 (twin_capture_failed); await okRate 0.28. STATUS fail report finalized. v2 correctness/availability board is next. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
…await v1 ran all 250 await trials (70 ok); 178 were silent park_failed continues with no failureClass/log/rebuild — not an incomplete arm and not correctness. Classify+log+rebuildFleet on park fail; keep await failureByReason in evidence. STATUS fail report + awaitArmForensics updated; v2 restarted on fixed harness (same frozen criteria, fresh R2 prefix). Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Grep for 900/15min across celld+harness did not find a lease/R2 15-minute timeout. The cluster (~875–909s) matches programAttempts=180 × fetchJson timeout 5s when /health accepts but does not respond. waitUrl/waitNode now use wallMs + 1.5s polls; resume-path ensureSurvivor respects deadlineAt. Criteria unchanged; running v2 process unaffected until next start. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Document 15 availability fails at ~500 write-kill (5×~900s deadline, 4 pause cascade, 5 no_expected, 1 twin). p99 spike is the discrete 900s class (p95 still ~14s), not rising typical recovery. Treat board as likely avail fail; finish for correctness. Twin capture now clears expectOk so trialFailures includes it. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Harness wait-cap only speeds failure classification — celld hang (/health accepts, never ready) is the real incident (lease-sweep / alive-vs-ready). Report 15 fails as ~5 stall clusters. during_call 404×5: writeCount=0, no divergence, 4/5 in cascades — not proven lost committed state; correctness stays 0. Next harness retries mid-flight 404 until wall deadline and stores programId for later status checks. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
Read-only check on the live v2 prefix shows trials 134/286/390/442 never landed meta+intent in DO storage; 494 has create meta only without a durable write intent. Do not publish correctness=0 for this board. Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
…on-zero Board finished under frozen criteria: availability 34/1000 (3.4%) above 2% ceiling; harness correctness 0 predates mid-flight storage check; R2 settle keeps correctness non-zero (five during_call lost-intent). Evidence JSON committed; v1 fail remains published. Mechanism claim stays paused pending output-gate/epoch-fence fork (PR #1453). Co-authored-by: Daniel Smith <danielsmithdevelopment@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
N≈1000 claim-path v2 board under frozen correctness/availability criteria.
v2 result — honest fail (do not loosen criteria)
Evidence:
phased-n1000-correctness-availability.json· measured 2026-10-30 21:40 PDT · prefixtier-c-phased-20261030212001.during_calltrials lost mid-flight journal (R2 settle) — do not publish corr=0v1 zero-deadline fail remains published at
phased-n1000-claim-path.json.Also in this PR
correctness_availability_v2)Next