Repository navigation
fix(cms): discard empties the draft index first; time out slow R2 draft writes - #8419
Merged
Merged
Conversation
A discard deleted every listed draft file, then the index. A reader
(the SDK through the CDN) that read the index in between met listed
names whose files were gone, and a CDN may hold those 404s.
Now the index is emptied ({ names: [] }) first, then the files it
listed are deleted, then the index, then the row. The names come from
the emptying write itself, so a name a save listed between the read
and that write is deleted too. A failed delete still keeps the row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tlgimenes
force-pushed
the
fix/draft-write-order
branch
from
November 10, 2026 19:02
03dec99 to
479eb71
Compare
Conditional PUTs go through an S3 client with the SDK retries off and no request timeout, so a stalled R2 PUT held the save open until R2 answered (one sat ~25 s before landing), and nothing logged it. Each conditional PUT now gives up after 15 s (CONDITIONAL_PUT_TIMEOUT_MS) and throws DeliveryUnavailableError, the Unavailable answer a save already turns into PartialApplyError (-32008 with what landed) and the editor retries. 15 s is far above R2 latency for a draft-sized body and still fits the largest block an 8 MiB request can carry on a slow link. A conditional PUT that times out, answers 429/5xx, or takes 2 s or more is logged through console.warn (exported as OTel logs) with its outcome, duration, size and key, the draft slug hidden since it is the draft read grant. Save order is unchanged: draft index first, then files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Discard empties the draft index before it deletes the files. When one delete fails, the names not yet deleted are put back into the index (best effort) before the error is rethrown. The files left behind stay listed, so a later save guards against them and a retried discard deletes them. Without this they were hidden files that a save with ifMatch null hit as a conflict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 15 s timeout is sized for draft bodies (one block of a save capped at 8 MiB). Publish writes go up to 16 MiB, and a publish that gave up on a latest.json PUT that then landed would leave the site live but not purged. So only keys under drafts/<slug>/ get the timeout. Every other conditional PUT keeps no timeout, as on main, and is only logged when slow or failed (timeoutMs null in the log). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A draft PUT given up on after the timeout can still land. The editor retries the rest right away with the same guards, so the retry could meet the first try's own write and be refused. - Server: in blocks.apply, a 412 on a file that already holds the version being written is counted as landed, not a lost race. - Editor: when the retry is refused with a Conflict, and every refused block already holds the exact body this save sends (same version), those blocks are adopted as landed and the rest is sent once more, guarded on those versions. Any other conflict is shown as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tlgimenes
force-pushed
the
fix/draft-write-order
branch
from
November 10, 2026 19:10
479eb71 to
d4d9495
Compare
tlgimenes
marked this pull request as ready for review
November 10, 2026 19:58
decocms Bot
pushed a commit
that referenced
this pull request
Nov 10, 2026
PR: #8419 fix(cms): discard empties the draft index first; time out slow R2 draft writes Bump type: patch - decocms (apps/api/package.json): 4.568.1 -> 4.568.2 Deploy-Scope: both
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes to hosted CMS drafts, on top of current
main:Discard empties the draft index before it deletes any file (
apps/api/src/hosted/cms-drafts.ts). A discard used to delete every listed draft file and then the index. A reader (the SDK, through the CDN) that read the index between those steps found listed names whose files were already gone, and the CDN can keep those 404s. Now a discard writes{ names: [] }first, then deletes the files that write found listed, then the index, then the row. Because the names come from the emptying write, a name that a concurrent save listed between our read and that write is deleted too. If a file delete fails, the names not yet deleted are put back into the index (best effort) before the error is rethrown. The row is kept, the leftover files stay listed (a later save guards against them instead of hitting a hidden file withifMatch: null), and a retried discard deletes them.Conditional R2 draft writes time out, and slow conditional writes are logged (
apps/api/src/hosted/delivery-store.ts). The conditional S3 client runs withmaxAttempts: 1and had no request timeout, so a stalled R2 PUT kept the save open until R2 answered, and nothing logged it. Each conditional PUT of a draft object (a key underdrafts/<slug>/: the draft index and its files) now stops after 15 s (CONDITIONAL_PUT_TIMEOUT_MS) and throws the existingDeliveryUnavailableError. A save already turns that error intoPartialApplyError(-32008withlanded), which the editor retries, so no new error shape is introduced. A PUT that times out, gets a 429/5xx, or takes 2 s or more is logged withconsole.warn("hosted delivery: slow or failed conditional write", { key, outcome, durationMs, bytes, timeoutMs }). That is the repo's usual structuredconsole.*call, and it is exported as OTel logs. The draft slug inkeyis replaced with<slug>because the slug is the draft's read grant.A save that meets its own late write counts it as landed. A PUT given up on can still land, and the editor retries the rest right away with the same guards. On the server (
cms-storage.ts,apply), a 412 on a file that already holds the version being written counts as landed, not as a lost race. In the editor (content-protocol-api.ts,applyProtocolPatch), when the retry is refused with a Conflict and every refused block already holds the exact body this save sends (samecmsBlockVersion), those blocks are adopted as landed and the rest is sent once more, guarded on those versions. Any other conflict is shown as before.Why 15 s, and only for drafts: R2 answers a PUT the size of a draft well under a second (p99 is a few hundred ms). The largest body one PUT can carry is one block of a request capped at 8 MiB (
MAX_REQUEST_BYTES), which takes about 13 s on a poor 5 Mbit/s uplink. So 15 s does not cut off a healthy draft write, and the ~25 s stall from the incident now fails fast. Publish writes (block objects, a revision's index and meta,latest.json) can be up to 16 MiB (MAX_BLOCK_BYTES,MAX_INDEX_BYTES), which can take longer than 15 s on a poor link. A publish that gave up on alatest.jsonPUT that then landed would leave the site live but not purged. So those PUTs keep no timeout, as onmain. They are still logged when slow or failed, withtimeoutMs: null.Decision: keeps index-first
An earlier version of this PR reversed the save order (draft files first, index last). That went against the index-first design the user decided on 2026-11-05. That reversal is dropped: saves still write the draft index first, then the files, exactly as on
main. This PR does not change the save path.Why the reversal was dropped (worst case against worst case):
-32008with what landed, the failure is visible, and the editor's retry writes the missing file.SWYdvLFwqQ5t8D60VODzJQ: index at 07:30:18Z, file at 07:30:44Z) was a stalled R2 PUT between two back-to-back writes. Cloudflare caching the file's 404 for about 180 s made it worse. The read side is fixed in fix(blocks): read draft files past shared caches (no CDN-cached 404s) blocks#804 (a fresh?r=<nonce>on every draft GET). This PR makes the stall fail fast and shows it in the logs.How to spike it
cd apps/api && bun test src/hosted src/api/routes/cmscms-drafts.test.ts, "discard empties the index, then deletes every listed file, then the index, then the row" (the log starts withput …/index.json), "discard never leaves a listed name whose file is gone" (every file delete sees an empty index), and "discard deletes a name a save listed while the index was being emptied".delivery-store-s3.test.ts, "conditional PUT timeout". A fake S3 holds a PUT for 2 s against a 300 ms test timeout. The PUT is sent once and rejected withDeliveryUnavailableError("no answer within 300 ms") in under 1.5 s, and it is logged withoutcome: "timeout",durationMs ≥ 300,bytesand a key with the slug hidden. A slow PUT that lands is logged withoutcome: "ok", a quick one is not logged, a 503 is logged asunavailable, and a 412 is not logged.delivery-store-s3.test.ts, "a publish PUT (not a draft key) is never cut off, only logged when slow". A 500 ms PUT tolatest.jsonagainst the 300 ms test timeout lands, is sent once, and is logged withtimeoutMs: null.cms-drafts.test.ts, "a failed file delete puts the names it didn't delete back into the index…". Withb's delete answering 503,ais gone, the index lists["b", "c"], the row is kept, and a second discard removes everything.cms-storage.test.ts, "a timed-out file PUT is -32008 with what landed…". The save answers-32008withlanded: { a }, the index already listsb(index-first), and the retry ofblands.cms-storage.test.ts, "a retry whose PUT meets the first try's own late write counts it as landed…". The given-up PUT commits after the retry read the draft. When it holds the same body the retry answersversions: { b }. When it holds another body it is still-32008withlanded: {}.hosted-content.test.ts, "a retry refused by the first try's own late write". The retry's Conflict reportshomeat the version of the body being sent, sohomeis adopted and sent once more withifMatch: { home: <that version> }. A Conflict at any other version is thrown with no further send.mainis unchanged:applyincms-storage.tscallsupdateDraftIndex(… added)before anywriteDraftFile, and the existing test "writes targets before referrers, then the deletes" still expectsput …/index.jsonfirst.Evidence
bun test src/hosted src/api/routes/cms(apps/api): 219 pass, 0 fail (15 files)bun test src/components/sections-editor(apps/web): 1711 pass, 0 fail (127 files)tsc --noEmit -p apps/apiand-p apps/web: clean ·oxlint apps/api/src/hosted apps/web/src/components/sections-editor: 0 warnings, 0 errors ·biome checkon the changed files: cleanKnown limitations
mainand is out of scope here.🤖 Generated with Claude Code