epic 2.1: a type change stops leaving the old body on disk (T2.1.9) - #52
Open
kuyazee wants to merge 2 commits into
Open
epic 2.1: a type change stops leaving the old body on disk (T2.1.9)#52kuyazee wants to merge 2 commits into
kuyazee wants to merge 2 commits into
Conversation
saveArtifact only ever wrote, so a PUT converting html to a redirect left index.html and source.html on disk and the reverse left source.url. Nothing served them, because every serve path keys off meta.type, but they stayed at rest and on the git backend they stayed in commit history after the artifact was deleted. Adds delete(key) to the storage interface on all five backends: local removes the file with force so a missing key is not an error, git delegates to local, s3 reuses the delete helper that already treats 404 as done, and the two SQL stores take an equality match rather than the range predicate the prefix operations need. lib/artifact-files.js owns which objects each type owns and what a conversion drops, so a unit test can walk all twenty ordered pairs without a server. server.js runs the deletes after meta.json lands and before flush: a crash in between leaves the old record whole rather than a listed artifact with no body, and git carries the deletions in the same commit as the write. A delete that fails is logged, not thrown, so a cleanup problem cannot turn a replace that already landed into a 500 the caller would retry. Red first: five conversions on one slug left source.html, source.jsx, source.md, source.tsx and source.url side by side; each hop now leaves exactly what the new type owns.
…o it The four review lenses on 3fa2a31. Two gaps were real and both are closed here. The cleanup had no test proving anything called it. Replacing the loop body with `for (const key of [])` left all 96 unit tests and the whole smoke suite green on every backend, because a stale object is not reachable over HTTP and no test boots server.js. The loop moved into `dropStaleObjects` in lib/artifact-files.js, where a fake storage can record the keys it asks for. The same neuter now fails 3 tests. A backend with no `delete` raised "storage.delete is not a function" straight into the catch that exists so a failed cleanup cannot 500 a write that already landed, so it would have warned once per conversion forever and failed nothing. `createStorage` now checks every method the app calls and refuses the boot, which is how the backend already treats a store it cannot reach. test/storage-contract.test.js drives local and sqlite for real and the shared SQL wrapper for postgres; s3 and git are covered by the boot check under the CI matrix. Smaller findings from the same pass: - pipeStream answered 500 when a read opened after the object was deleted. local stats then opens, so a conversion landing in between produced a hard 500 on a plain GET. It answers 404 now, which the comment above serveObject already promised and which the rename path has always given during its own window. - ownedKeys and the /a/:slug/source route both looked SOURCE_EXT up bare, so a hand-edited meta.type of "constructor" built a key out of a function body. Both use Object.hasOwn now. - TYPES is read off SOURCE_EXT, so a sixth type cannot be publishable and invisible to the cleanup. dashboard-check.mjs imports the same table instead of grepping server.js for the array. - The update_artifact description said title resets when omitted, which T2.1.7 made false, and neither it nor docs/mcp.md said a type change now deletes files. - The warn on a failed delete says what it means for the artifact and that nothing retries, matching the other operator lines in this repo. - Dropped a smoke assertion that could not fire, and the duplicate ok-line label. The commit message on 3fa2a31 says the old bytes "stayed in commit history"; that reads as if this fixes it. It does not. A deletion on the git backend is another commit, so the bytes stay in history and `git log -p` on the remote still hands them back. lib/artifact-files.js says so now.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Items
One item, not the usual batch. T2.1.9 is the storage-interface item the 2026-08-13 run sized and
deferred, and it filled the branch on its own: 556 lines against
main, past the 400-line cap, sotask/epic-2-1stays where it is and this is a separate PR.Evidence page: https://artifacts.zonily.cloud/a/epic-2-1-b-type-cleanup
T2.1.9 - Type changes leave the old body on disk
What the item was
saveArtifactwrote the new type's objects and never removed the old type's. APUTconvertinghtml to a redirect left
index.htmlandsource.htmlon disk, and the reverse leftsource.url.Nothing served them, because
/a/:slugand/a/:slug/sourceboth key offmeta.type, so it wasat-rest bloat that grew with every conversion.
Red first, on a real server: one artifact walked html to redirect to md to jsx to tsx ended up
holding
source.html,source.jsx,source.md,source.tsxandsource.urlside by side, oneof them live. Each hop now leaves exactly what the new type owns. Both listings are on the
evidence page.
The sizing note was wrong about the hard part
The item said the interface needed a single-key delete plus a subtree delete, because converting a
zip site to html has to drop a
site/prefix rather than a key. That premise does not hold.storeArtifactanswers 400 onexisting.type === 'zip'andstoreZipArtifactanswers 409 on aslug that already has a meta, so no API path converts a zip to anything or anything to a zip. A
site/tree only ever goes away whole, through thedeleteSlugthat already exists. Adding aprefix delete now would be an interface nobody calls.
What changed
delete(key)on all five backends:fs.rmwithforce, so a key that is already gone is not an error.stageAllcallsgit.removewhen a file leaves the working copy,and the deletes run before
flush, so they ride the same commit as the write.need, so there is no collation question and a row that is not there is a no-op.
lib/artifact-files.jsowns which objects each type owns (ownedKeys), what a conversion drops(
staleKeys), and the loop that issues the deletes (dropStaleObjects).server.jscalls it aftermeta.jsonlands and beforeflush. That ordering is deliberate: a crash in between leaves thepre-fix state, a new record with old orphans, rather than a listed artifact whose body is gone. A
delete that fails is logged, not thrown, because the write has already landed and throwing would
turn a successful replace into a 500 the caller would retry.
createStoragenow checks that a backend answers every method the app calls and refuses the boototherwise, the way it already refuses a store it cannot reach.
Tests
npm testsmoke.shtest/artifact-files.test.jswalks all twenty ordered pairs of types and asserts the invariant:what a conversion drops plus what the new type owns covers everything the old type owned, and
nothing the new type still serves is on the drop list.
test/storage-local.test.jsandtest/storage-sql.test.jscover the delete itself on local andsqlite, including a key that is already gone and one shaped like a SQL
LIKEpattern.test/storage-contract.test.jsproves local, sqlite and the shared SQL wrapper answer the wholeinterface. s3 and git need a bucket and a remote, so neither is built on a laptop; the boot check
covers them under the CI backend matrix.
up front what it does not prove: a stale object is not reachable over HTTP, so smoke can only
show the cleanup never takes a file the new type still needs.
Review findings
Four lenses on 3fa2a31, since the diff was over 200 lines. Two findings were serious enough to
change the shape of the fix.
Fixed in 0a90379:
for (const key of [])left all96 unit tests and the whole smoke suite green on every backend. No test boots
server.js, and astale object is invisible over HTTP, so the entire fix could have been deleted with CI green. The
loop moved into
dropStaleObjectswhere a fake storage records the keys it asks for. The sameneuter now fails 3 tests.
deletefailed nothing. It raisedstorage.delete is not a functionstraight into the catch that exists so a failed cleanup cannot 500 a landed write, so it would
have warned once per conversion forever.
createStoragechecks the interface at boot now.landing in between produced a hard 500 on a plain GET. Measured at 13 misses in 311 reads, one of
them a 500. It answers 404 now, which the comment above
serveObjectalready promised and whichthe rename path has always given during its own window. Re-measured after the fix: 216 404s,
text/plain, no 500s.SOURCE_EXT[type]on ahand-edited
meta.typeofconstructorproducedsource.function Object() { [native code] }.ownedKeysand the/a/:slug/sourceroute both useObject.hasOwnnow.TYPESandSOURCE_EXTwere the same list in two files. A sixth type added to one wouldhave been publishable and invisible to the cleanup.
TYPESreads offSOURCE_EXT, anddashboard-check.mjsimports the same table instead of greppingserver.jsfor the array.update_artifactdescription was wrong. It said title resets when omitted, which T2.1.7made false, and neither it nor
docs/mcp.mdsaid a type change now deletes files. Both fixed.the file, and that nothing retries.
ok:label. Both dropped.Filed, not fixed:
PUTthat omitstypesilently converts the artifact to html. Pre-existing anddocumented, but this branch raised the cost: the old type's bytes are deleted now rather than left
on disk. Making
typesticky is the title precedent from T2.1.7, and it is a behaviour change, soit is a call for you rather than the runner.
duplicatecopies themforward because
copySlugprunes nothing. The prune half needs no decision; the sweep forexisting installs does.
putanddeleteskip the realpath containment that readsapply, so a symlink planted under
DATA_DIRcan be written through or deleted through. Pre-existingin
putsince the backend shipped and needs write access toDATA_DIRout of band. It turns onwhether an operator with that access is inside the threat model.
One claim corrected. The message on 3fa2a31 says the stale bytes "stayed in commit history",
which reads as if this fixes that. It does not. A deletion on the git backend is another commit, so
the old bytes stay in history and
git log -pon the remote still hands them back, private andpassword artifacts included.
lib/artifact-files.jssays so now.Verification
Local only, port 3011,
DATA_DIR=/tmp/epic21b-data. Nothing was written under the worktree or underany real data directory, and no existing artifact was touched. No dashboard UI changed, so there is
no browser capture.