Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
179e6d7
feat(bench): digest-equal statePreserved + trialAccounting before N=1000
cursoragent Oct 29, 2026
3904c27
style: apply Prettier [prettier-autofix]
github-actions[bot] Oct 29, 2026
e8c9164
docs: fix N=1000 pass criteria before results; scope digest claim
cursoragent Oct 29, 2026
04acb5b
style: apply Prettier [prettier-autofix]
github-actions[bot] Oct 29, 2026
c4f6441
fix(bench): bound fleet rebuild after self-fence; mark partial evidence
cursoragent Oct 30, 2026
6f2ca0a
docs+bench: N=1000 fail report; twin retry and reason classification
cursoragent Oct 30, 2026
247719f
style: apply Prettier [prettier-autofix]
github-actions[bot] Oct 30, 2026
a3ec312
docs+bench: lock claim-path v2 correctness/availability criteria befo…
cursoragent Oct 30, 2026
47bcdae
style: apply Prettier [prettier-autofix]
github-actions[bot] Oct 30, 2026
c7e8630
bench: parallel availability hardenings for claim-path v2
cursoragent Oct 30, 2026
ca239e7
bench: pause-aware resume deadline during orchestrator recovery
cursoragent Oct 30, 2026
c833b47
style: apply Prettier [prettier-autofix]
github-actions[bot] Oct 30, 2026
25d0134
fix(bench): score availability on wall-clock resume, not frozen deadline
cursoragent Oct 30, 2026
3954a31
style: apply Prettier [prettier-autofix]
github-actions[bot] Oct 30, 2026
7f73325
docs+bench: pin v2 wall deadline 60s and 1000-trial avail denominator
cursoragent Oct 30, 2026
992c826
bench: record frozenCriteria in v2 fleet evidence
cursoragent Oct 30, 2026
7d00527
style: apply Prettier [prettier-autofix]
github-actions[bot] Oct 30, 2026
7578d68
docs(bench): publish failed N=1000 v1 claim-path board (keep as fail)
cursoragent Oct 30, 2026
eef2df7
style: apply Prettier [prettier-autofix]
github-actions[bot] Oct 30, 2026
5ed442d
docs+bench: explain v1 await 70/250; fix park-fail harness before v2 …
cursoragent Oct 30, 2026
38a29e2
style: apply Prettier [prettier-autofix]
github-actions[bot] Oct 30, 2026
43dda78
bench: cap waitNode wall time — 180×5s was the ~900s resume tail
cursoragent Oct 30, 2026
7ead780
style: apply Prettier [prettier-autofix]
github-actions[bot] Oct 30, 2026
2795e5b
docs+bench: v2 interim failure breakdown; twin_capture forces !ok
cursoragent Oct 31, 2026
e7b9922
style: apply Prettier [prettier-autofix]
github-actions[bot] Oct 31, 2026
462620e
docs+bench: cluster avail fails; settle during_call 404; label two fixes
cursoragent Oct 31, 2026
0721972
style: apply Prettier [prettier-autofix]
github-actions[bot] Oct 31, 2026
9c867a6
docs: R2 settle during_call 404s as durability (correctness non-zero)
cursoragent Oct 31, 2026
b8549e6
style: apply Prettier [prettier-autofix]
github-actions[bot] Oct 31, 2026
54e5fa3
docs+bench: finalize N=1000 v2 — avail 3.4% fail; settle keeps corr n…
cursoragent Oct 31, 2026
0872eb0
style: apply Prettier [prettier-autofix]
github-actions[bot] Oct 31, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 135 additions & 1 deletion docs/benchmarks/durability-proof/STATUS.md

Large diffs are not rendered by default.

24 changes: 24 additions & 0 deletions docs/benchmarks/program-cell-fleet-kill/AVAILABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,17 @@ So the TTL pulls two ways:

**celld 0.4.0 knobs:** only `CELLD_TTL_MS` is exposed. There is no `CELLD_RENEW_*` env. Renew cadence and lease-store placement are production asks / future celld surfaces — mark **planned** until run at the same sample size as stock. Do not pretend a 30s TTL is the preferred production tune.

## Parallel product work (does not block claim-path v2)

Claim-path N≈1000 v2 scores **correctness** (zero wrong outcomes) separately from **availability** (≤2% failure-rate ceiling on **all 1000** trials = 750 write-kill + 250 await; resume/recovery tails **reported without a ceiling**). “In time” = wall clock **`FLEET_PHASED_RESUME_DEADLINE_MS=60000`**, including orchestrator recovery. These celld items run **in parallel** so a later re-measure can show deadline-miss rate falling from X% to Y%:

| Track | Meaning | Where |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| **Lease renew / regional store** | Cut false `SELF-FENCE` under R2 RTT (preferred levers above) | Upstream celld — no `CELLD_RENEW_*` yet |
| **Lease-sweep** | Retire dead node-session leases so a restarted peer is not blocked by a fenced identity | Upstream (`dead_node_gc`); harness mitigates with fresh `CELLD_NODE` per start |
| **Alive vs ready** | `/health` can answer while `ready_gate` is still closed — callers must wait `/program/health` (or ready_gate), not confuse alive with serving | Harness already probes `/program/health?cell=health`; product docs / SDK should match |
| **Pause-aware clocks** | Timers that handle the **host** being paused/suspended (wall time jumps while the machine was not running) — behind pause-fetch failures; **not** freezing the resume deadline during harness recovery | Still open on the celld / product list. Harness reports diagnostic `resumeActiveMs` + `recoveryMs` but scores availability on wall `resumeMs` |

## Stall forensics (stock 10s / R2)

| Knob / observation | Value |
Expand All @@ -31,6 +42,19 @@ So the TTL pulls two ways:
| Observed renew RTT (`elapsed_ms`) | typically **270–560ms** under peer-down load |
| Fence | Watchdog if lease not renewed within TTL |

### ~900s resume tails — harness wait vs product hang

Stalls land at **~875–921s**. Grep found no celld/R2 **15-minute** lease TTL. The arithmetic is harness-side; the _reason_ the harness waited is product-side.

| Layer | Finding | Fix does |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| **Harness (explains the 15 minutes)** | `waitNode` used `programAttempts=180` × `fetchJson` **5s** when `/health` accepts TCP but never responds → **180×5s=900s**. Also bypassed the 60s resume deadline. | **Wall-cap + `deadlineAt`** → faster **classified failure**, better p99 optics. Does **not** make the node healthy. |
| **celld (explains why `/health` hung)** | Same pattern as the N=1000 dead-leader-sweep spiral: process accepts connections, does not become ready. | **Bounded lease-sweep** + **alive vs ready** (still on the parallel product list). This is what removes the _incident_, not just the wait. |

**v2 mid-run (~500 write-kill):** five ~900s stalls seed cascades; **15 failures ≈ 5 stall incidents + twin + 1 lone mid-flight 404**. **p95≈14s / p99≈870s** = discrete stall class (1% of trials), not rising typical recovery. Running v2 process still has the pre-wall-cap harness.

**`during_call` resume 404 (correctness check):** intent+meta are journaled before the hanging upstream call; all five no_expected trials had **writeCount=0** (no committed outside write) and no journal divergence. Four sit in stall cascades. Harness stops on first 404, so visibility lag vs permanent meta loss is not separable from this evidence — **not reclassified as correctness**; residual noted for harness retry-on-404 + retain `programId`.

## Three columns

| Column | Property |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
{
"kind": "during-call-404-r2-settle",
"measuredAt": "2026-10-31T02:11:14.931Z",
"prefix": "tier-c-phased-20261030212001",
"cell": "ProgramCellDO:8b3f8c259c1b2bf1aed4a4270920817997c5d8f760e4dc8e59f3f428e2c89b4f",
"method": [
"read-only copy of live DO SQLite _cf_KV",
"scan of 1879 local /tmp/celld-*/…/db.sqlite generation snapshots",
"GET /program/status on live survivor for present programId",
"no PutObject / kv put / mutate against the live prefix"
],
"trials": [
134,
286,
390,
442,
494
],
"results": {
"134": {
"present": false,
"intent": false,
"verdict": "durability_missing",
"notes": "never in any in-window generation; neighbors 130 and 138 present"
},
"286": {
"present": false,
"intent": false,
"verdict": "durability_missing",
"notes": "current-run window empty; prior-run ts 1793374057485 ignored"
},
"390": {
"present": false,
"intent": false,
"verdict": "durability_missing",
"notes": "current-run window empty; prior-run ts ignored"
},
"442": {
"present": false,
"intent": false,
"verdict": "durability_missing",
"notes": "current-run window empty; prior-run ts ignored"
},
"494": {
"present": true,
"programId": "phased_during_call_494_1793409345554",
"intent": false,
"pauseKind": null,
"status": "running",
"entries": 0,
"intents": 0,
"createdAt": "2026-10-31T01:15:45.557Z",
"updatedAt": "2026-10-31T01:15:45.557Z",
"verdict": "durability_intent_missing",
"notes": "create meta only; intent/pauseKind never durable across all local gens for this ts"
}
},
"duringCallExpectedVsPresent": {
"expected": 150,
"presentMeta": 146,
"missingIndices": [
134,
286,
390,
442
],
"presentCreateOnly": [
494
]
},
"classificationImpact": {
"harnessScoredAs": "availability/no_expected_outcome",
"afterSettle": "correctness/durability — do not publish correctness=0",
"harnessFollowUp": "retry mid-flight 404 until wall deadline; retain programId; later /status"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# during_call 404 ×5 — read-only R2/DO settle (v2 prefix)

**Prefix:** `s3://clawql-streams-state/tier-c-phased-20261030212001`
**Cell:** `ProgramCellDO:8b3f8c259c1b2bf1aed4a4270920817997c5d8f760e4dc8e59f3f428e2c89b4f`
**Method:** read-only copies of live `_cf_KV` SQLite + scan of 1879 local generation DBs; live `GET /program/status` for present IDs. No writes to the prefix.

## Expected vs present (current run)

Among completed `during_call` trials (`i % 4 === 2`), live KV holds meta for **146/150**. The **only** gaps are the five 404 failures' indices — except **494**, which is present as create-only meta.

| Trial | Current-run keys in live KV / in-window local gens | Intent journaled? | Live `/program/status` |
| ----- | -------------------------------------------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------- |
| 134 | **MISSING** (never in any local gen in neighbor window; e130–e144 never had 134) | no | N/A (no programId retained; neighbors 130+138 present) |
| 286 | **MISSING** in current-run window (prior-run ts `…057485` only) | no | N/A |
| 390 | **MISSING** in current-run window (prior-run ts only) | no | N/A |
| 442 | **MISSING** in current-run window (prior-run ts only) | no | N/A |
| 494 | **PRESENT** `phased_during_call_494_1793409345554:meta` only | **never** (0 intent keys across all gens for this ts) | `ok:true`, `status:running`, `pauseKind` absent, `intents:[], entries:[]` |

Neighbor successful `during_call` programs (e.g. 130, 490) retain `pauseKind=during_call` and post-resume entries — the shape expected after journal-before-hang.

## Verdict

- **134, 286, 390, 442:** journaled intent **and** meta for this run are **absent** from R2-backed DO storage. Not a visibility-timeout 404 — durable state was never present (or was lost before any generation snapshot). → **durability / correctness** (not availability-only).
- **494:** create meta survived; the `during_call` intent + `pauseKind` meta update **never** became durable. Resume-time 404 can still be availability/visibility, but the mid-flight journal contract was not met in storage. → **durability / correctness** for intent; do **not** publish correctness=0 on the strength of “404 alone.”

Prior-run keys for indices 286/390/442/494 (different `Date.now()` suffixes, full meta+intent+entries) exist in older local gens and are **out of** the current-run timestamp windows — ignored for this settle.

## Harness follow-up (unchanged)

Retry mid-flight 404 until wall deadline; retain `programId`; later `/status` check. That remains the right permanent harness fix either way.
Loading