Skip to content

fix(cms): discard empties the draft index first; time out slow R2 draft writes - #8419

Merged
tlgimenes merged 5 commits into
mainfrom
fix/draft-write-order
Nov 10, 2026
Merged

tlgimenes merged 5 commits into
mainfrom
fix/draft-write-order

Conversation

@tlgimenes

@tlgimenes tlgimenes commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes to hosted CMS drafts, on top of current main:

  1. Discard empties the draft index before it deletes any file (apps/api/src/hosted/cms-drafts.ts). A discard used to delete every listed draft file and then the index. A reader (the SDK, through the CDN) that read the index between those steps found listed names whose files were already gone, and the CDN can keep those 404s. Now a discard writes { names: [] } first, then deletes the files that write found listed, then the index, then the row. Because the names come from the emptying write, a name that a concurrent save listed between our read and that write is deleted too. If a file delete fails, the names not yet deleted are put back into the index (best effort) before the error is rethrown. The row is kept, the leftover files stay listed (a later save guards against them instead of hitting a hidden file with ifMatch: null), and a retried discard deletes them.

  2. Conditional R2 draft writes time out, and slow conditional writes are logged (apps/api/src/hosted/delivery-store.ts). The conditional S3 client runs with maxAttempts: 1 and had no request timeout, so a stalled R2 PUT kept the save open until R2 answered, and nothing logged it. Each conditional PUT of a draft object (a key under drafts/<slug>/: the draft index and its files) now stops after 15 s (CONDITIONAL_PUT_TIMEOUT_MS) and throws the existing DeliveryUnavailableError. A save already turns that error into PartialApplyError (-32008 with landed), which the editor retries, so no new error shape is introduced. A PUT that times out, gets a 429/5xx, or takes 2 s or more is logged with console.warn("hosted delivery: slow or failed conditional write", { key, outcome, durationMs, bytes, timeoutMs }). That is the repo's usual structured console.* call, and it is exported as OTel logs. The draft slug in key is replaced with <slug> because the slug is the draft's read grant.

  3. A save that meets its own late write counts it as landed. A PUT given up on can still land, and the editor retries the rest right away with the same guards. On the server (cms-storage.ts, apply), a 412 on a file that already holds the version being written counts as landed, not as a lost race. In the editor (content-protocol-api.ts, applyProtocolPatch), when the retry is refused with a Conflict and every refused block already holds the exact body this save sends (same cmsBlockVersion), those blocks are adopted as landed and the rest is sent once more, guarded on those versions. Any other conflict is shown as before.

Why 15 s, and only for drafts: R2 answers a PUT the size of a draft well under a second (p99 is a few hundred ms). The largest body one PUT can carry is one block of a request capped at 8 MiB (MAX_REQUEST_BYTES), which takes about 13 s on a poor 5 Mbit/s uplink. So 15 s does not cut off a healthy draft write, and the ~25 s stall from the incident now fails fast. Publish writes (block objects, a revision's index and meta, latest.json) can be up to 16 MiB (MAX_BLOCK_BYTES, MAX_INDEX_BYTES), which can take longer than 15 s on a poor link. A publish that gave up on a latest.json PUT that then landed would leave the site live but not purged. So those PUTs keep no timeout, as on main. They are still logged when slow or failed, with timeoutMs: null.

Decision: keeps index-first

An earlier version of this PR reversed the save order (draft files first, index last). That went against the index-first design the user decided on 2026-11-05. That reversal is dropped: saves still write the draft index first, then the files, exactly as on main. This PR does not change the save path.

Why the reversal was dropped (worst case against worst case):

  • Files-first can leave a file written but not listed when the index write fails. No reader sees that file. A later save of the same name then hits a 412 from it, so the editor gets a Conflict it can't explain, or, if the conflict is resolved by overwriting, the edit can be silently lost.
  • Index-first can leave a name listed whose file isn't written yet. Readers treat a listed name with no file as "not in the draft" and fall back to production for that block. The save answers -32008 with what landed, the failure is visible, and the editor's retry writes the missing file.
  • The symptom that prompted the change (osklenbr-faststore-studio draft SWYdvLFwqQ5t8D60VODzJQ: index at 07:30:18Z, file at 07:30:44Z) was a stalled R2 PUT between two back-to-back writes. Cloudflare caching the file's 404 for about 180 s made it worse. The read side is fixed in fix(blocks): read draft files past shared caches (no CDN-cached 404s) blocks#804 (a fresh ?r=<nonce> on every draft GET). This PR makes the stall fail fast and shows it in the logs.

How to spike it

  • cd apps/api && bun test src/hosted src/api/routes/cms
  • Discard order: cms-drafts.test.ts, "discard empties the index, then deletes every listed file, then the index, then the row" (the log starts with put …/index.json), "discard never leaves a listed name whose file is gone" (every file delete sees an empty index), and "discard deletes a name a save listed while the index was being emptied".
  • Timeout: delivery-store-s3.test.ts, "conditional PUT timeout". A fake S3 holds a PUT for 2 s against a 300 ms test timeout. The PUT is sent once and rejected with DeliveryUnavailableError ("no answer within 300 ms") in under 1.5 s, and it is logged with outcome: "timeout", durationMs ≥ 300, bytes and a key with the slug hidden. A slow PUT that lands is logged with outcome: "ok", a quick one is not logged, a 503 is logged as unavailable, and a 412 is not logged.
  • Publish PUTs are not cut off: delivery-store-s3.test.ts, "a publish PUT (not a draft key) is never cut off, only logged when slow". A 500 ms PUT to latest.json against the 300 ms test timeout lands, is sent once, and is logged with timeoutMs: null.
  • Discard with a failed delete: cms-drafts.test.ts, "a failed file delete puts the names it didn't delete back into the index…". With b's delete answering 503, a is gone, the index lists ["b", "c"], the row is kept, and a second discard removes everything.
  • Editor path: cms-storage.test.ts, "a timed-out file PUT is -32008 with what landed…". The save answers -32008 with landed: { a }, the index already lists b (index-first), and the retry of b lands.
  • Late write, server side: cms-storage.test.ts, "a retry whose PUT meets the first try's own late write counts it as landed…". The given-up PUT commits after the retry read the draft. When it holds the same body the retry answers versions: { b }. When it holds another body it is still -32008 with landed: {}.
  • Late write, editor side: hosted-content.test.ts, "a retry refused by the first try's own late write". The retry's Conflict reports home at the version of the body being sent, so home is adopted and sent once more with ifMatch: { home: <that version> }. A Conflict at any other version is thrown with no further send.
  • Save order on main is unchanged: apply in cms-storage.ts calls updateDraftIndex(… added) before any writeDraftFile, and the existing test "writes targets before referrers, then the deletes" still expects put …/index.json first.

Evidence

  • bun test src/hosted src/api/routes/cms (apps/api): 219 pass, 0 fail (15 files)
  • bun test src/components/sections-editor (apps/web): 1711 pass, 0 fail (127 files)
  • tsc --noEmit -p apps/api and -p apps/web: clean · oxlint apps/api/src/hosted apps/web/src/components/sections-editor: 0 warnings, 0 errors · biome check on the changed files: clean

Known limitations

  • A PUT that times out may still land in R2. Like any Unavailable answer, it is treated as neither a lost race nor a success (contract §2.6). The retry handles a late write that holds the same body (see 3). If the late write lands only after the retry is done, and the user's next save of that block has a different body, that save gets a Conflict that shows the user's own earlier body as the draft's copy. The conflict dialog resolves it, and nothing is lost silently.
  • Putting names back after a failed discard delete is best effort. If that index write also fails, the leftover files stay unlisted until the draft is written again.
  • An autosave that runs while a publish or discard is in flight can still race with it. That race exists on main and is out of scope here.
  • The timeout covers conditional PUTs of draft keys only. Publish writes keep no timeout, plain writes keep the SDK's default retries, and reads and deletes are unchanged.

🤖 Generated with Claude Code

A discard deleted every listed draft file, then the index. A reader
(the SDK through the CDN) that read the index in between met listed
names whose files were gone, and a CDN may hold those 404s.

Now the index is emptied ({ names: [] }) first, then the files it
listed are deleted, then the index, then the row. The names come from
the emptying write itself, so a name a save listed between the read
and that write is deleted too. A failed delete still keeps the row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tlgimenes
tlgimenes force-pushed the fix/draft-write-order branch from 03dec99 to 479eb71 Compare November 10, 2026 19:02
@tlgimenes tlgimenes changed the title fix(cms): write draft files before the draft index, index last fix(cms): discard empties the draft index first; time out slow R2 draft writes Nov 10, 2026
tlgimenes and others added 4 commits November 10, 2026 16:10
Conditional PUTs go through an S3 client with the SDK retries off and
no request timeout, so a stalled R2 PUT held the save open until R2
answered (one sat ~25 s before landing), and nothing logged it.

Each conditional PUT now gives up after 15 s (CONDITIONAL_PUT_TIMEOUT_MS)
and throws DeliveryUnavailableError, the Unavailable answer a save
already turns into PartialApplyError (-32008 with what landed) and the
editor retries. 15 s is far above R2 latency for a draft-sized body and
still fits the largest block an 8 MiB request can carry on a slow link.

A conditional PUT that times out, answers 429/5xx, or takes 2 s or
more is logged through console.warn (exported as OTel logs) with its
outcome, duration, size and key, the draft slug hidden since it is the
draft read grant. Save order is unchanged: draft index first, then files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Discard empties the draft index before it deletes the files. When one
delete fails, the names not yet deleted are put back into the index
(best effort) before the error is rethrown. The files left behind stay
listed, so a later save guards against them and a retried discard
deletes them. Without this they were hidden files that a save with
ifMatch null hit as a conflict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 15 s timeout is sized for draft bodies (one block of a save capped
at 8 MiB). Publish writes go up to 16 MiB, and a publish that gave up on
a latest.json PUT that then landed would leave the site live but not
purged. So only keys under drafts/<slug>/ get the timeout. Every other
conditional PUT keeps no timeout, as on main, and is only logged when
slow or failed (timeoutMs null in the log).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A draft PUT given up on after the timeout can still land. The editor
retries the rest right away with the same guards, so the retry could
meet the first try's own write and be refused.

- Server: in blocks.apply, a 412 on a file that already holds the
  version being written is counted as landed, not a lost race.
- Editor: when the retry is refused with a Conflict, and every refused
  block already holds the exact body this save sends (same version),
  those blocks are adopted as landed and the rest is sent once more,
  guarded on those versions. Any other conflict is shown as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tlgimenes
tlgimenes force-pushed the fix/draft-write-order branch from 479eb71 to d4d9495 Compare November 10, 2026 19:10
@tlgimenes
tlgimenes marked this pull request as ready for review November 10, 2026 19:58
@tlgimenes
tlgimenes merged commit 7d19ef4 into main Nov 10, 2026
29 of 30 checks passed
@tlgimenes
tlgimenes deleted the fix/draft-write-order branch November 10, 2026 19:59
decocms Bot pushed a commit that referenced this pull request Nov 10, 2026
PR: #8419 fix(cms): discard empties the draft index first; time out slow R2 draft writes
Bump type: patch

- decocms (apps/api/package.json): 4.568.1 -> 4.568.2

Deploy-Scope: both
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude PR authored by a coding agent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant