Skip to content

epic 2.1: a type change stops leaving the old body on disk (T2.1.9) - #52

Open
kuyazee wants to merge 2 commits into
mainfrom
task/epic-2-1-b
Open

epic 2.1: a type change stops leaving the old body on disk (T2.1.9)#52
kuyazee wants to merge 2 commits into
mainfrom
task/epic-2-1-b

Conversation

@kuyazee

@kuyazee kuyazee commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Items

  • T2.1.9 - a type change deletes the bytes the old type owned (commits 3fa2a31, 0a90379)

One item, not the usual batch. T2.1.9 is the storage-interface item the 2026-08-13 run sized and
deferred, and it filled the branch on its own: 556 lines against main, past the 400-line cap, so
task/epic-2-1 stays where it is and this is a separate PR.

Evidence page: https://artifacts.zonily.cloud/a/epic-2-1-b-type-cleanup

T2.1.9 - Type changes leave the old body on disk

What the item was

saveArtifact wrote the new type's objects and never removed the old type's. A PUT converting
html to a redirect left index.html and source.html on disk, and the reverse left source.url.
Nothing served them, because /a/:slug and /a/:slug/source both key off meta.type, so it was
at-rest bloat that grew with every conversion.

Red first, on a real server: one artifact walked html to redirect to md to jsx to tsx ended up
holding source.html, source.jsx, source.md, source.tsx and source.url side by side, one
of them live. Each hop now leaves exactly what the new type owns. Both listings are on the
evidence page.

The sizing note was wrong about the hard part

The item said the interface needed a single-key delete plus a subtree delete, because converting a
zip site to html has to drop a site/ prefix rather than a key. That premise does not hold.
storeArtifact answers 400 on existing.type === 'zip' and storeZipArtifact answers 409 on a
slug that already has a meta, so no API path converts a zip to anything or anything to a zip. A
site/ tree only ever goes away whole, through the deleteSlug that already exists. Adding a
prefix delete now would be an interface nobody calls.

What changed

delete(key) on all five backends:

  • local: fs.rm with force, so a key that is already gone is not an error.
  • git: delegates to local. stageAll calls git.remove when a file leaves the working copy,
    and the deletes run before flush, so they ride the same commit as the write.
  • s3: reuses the delete helper that already treats a 404 as done.
  • sqlite and postgres: an equality match, not the range predicate the prefix operations
    need, so there is no collation question and a row that is not there is a no-op.

lib/artifact-files.js owns which objects each type owns (ownedKeys), what a conversion drops
(staleKeys), and the loop that issues the deletes (dropStaleObjects). server.js calls it after
meta.json lands and before flush. That ordering is deliberate: a crash in between leaves the
pre-fix state, a new record with old orphans, rather than a listed artifact whose body is gone. A
delete that fails is logged, not thrown, because the write has already landed and throwing would
turn a successful replace into a 500 the caller would retry.

createStorage now checks that a backend answers every method the app calls and refuses the boot
otherwise, the way it already refuses a store it cannot reach.

Tests

Before After
npm test 89 108, 0 fail
smoke.sh 192 ok-lines 194 ok-lines, all pass
  • test/artifact-files.test.js walks all twenty ordered pairs of types and asserts the invariant:
    what a conversion drops plus what the new type owns covers everything the old type owned, and
    nothing the new type still serves is on the drop list.
  • test/storage-local.test.js and test/storage-sql.test.js cover the delete itself on local and
    sqlite, including a key that is already gone and one shaped like a SQL LIKE pattern.
  • test/storage-contract.test.js proves local, sqlite and the shared SQL wrapper answer the whole
    interface. s3 and git need a bucket and a remote, so neither is built on a laptop; the boot check
    covers them under the CI backend matrix.
  • The smoke block walks one artifact through every type and reads it back after each hop. It says
    up front what it does not prove: a stale object is not reachable over HTTP, so smoke can only
    show the cleanup never takes a file the new type still needs.

Review findings

Four lenses on 3fa2a31, since the diff was over 200 lines. Two findings were serious enough to
change the shape of the fix.

Fixed in 0a90379:

  1. Nothing proved the cleanup ran. Replacing the loop body with for (const key of []) left all
    96 unit tests and the whole smoke suite green on every backend. No test boots server.js, and a
    stale object is invisible over HTTP, so the entire fix could have been deleted with CI green. The
    loop moved into dropStaleObjects where a fake storage records the keys it asks for. The same
    neuter now fails 3 tests.
  2. A backend with no delete failed nothing. It raised storage.delete is not a function
    straight into the catch that exists so a failed cleanup cannot 500 a landed write, so it would
    have warned once per conversion forever. createStorage checks the interface at boot now.
  3. A reader mid-conversion could get a 500. local stats a file then opens it, so a conversion
    landing in between produced a hard 500 on a plain GET. Measured at 13 misses in 311 reads, one of
    them a 500. It answers 404 now, which the comment above serveObject already promised and which
    the rename path has always given during its own window. Re-measured after the fix: 216 404s,
    text/plain, no 500s.
  4. A prototype-chain lookup built a key out of a function body. SOURCE_EXT[type] on a
    hand-edited meta.type of constructor produced source.function Object() { [native code] }.
    ownedKeys and the /a/:slug/source route both use Object.hasOwn now.
  5. TYPES and SOURCE_EXT were the same list in two files. A sixth type added to one would
    have been publishable and invisible to the cleanup. TYPES reads off SOURCE_EXT, and
    dashboard-check.mjs imports the same table instead of grepping server.js for the array.
  6. The update_artifact description was wrong. It said title resets when omitted, which T2.1.7
    made false, and neither it nor docs/mcp.md said a type change now deletes files. Both fixed.
  7. The failure log said nothing useful. It now says the artifact is fine, that nothing serves
    the file, and that nothing retries.
  8. A smoke assertion that could not fire, and a duplicate ok: label. Both dropped.

Filed, not fixed:

  • T2.1.19: a PUT that omits type silently converts the artifact to html. Pre-existing and
    documented, but this branch raised the cost: the old type's bytes are deleted now rather than left
    on disk. Making type sticky is the title precedent from T2.1.7, and it is a behaviour change, so
    it is a call for you rather than the runner.
  • T2.1.20: artifacts converted before this keep their orphans, and duplicate copies them
    forward because copySlug prunes nothing. The prune half needs no decision; the sweep for
    existing installs does.
  • T2.1.21: on the local backend, put and delete skip the realpath containment that reads
    apply, so a symlink planted under DATA_DIR can be written through or deleted through. Pre-existing
    in put since the backend shipped and needs write access to DATA_DIR out of band. It turns on
    whether an operator with that access is inside the threat model.

One claim corrected. The message on 3fa2a31 says the stale bytes "stayed in commit history",
which reads as if this fixes that. It does not. A deletion on the git backend is another commit, so
the old bytes stay in history and git log -p on the remote still hands them back, private and
password artifacts included. lib/artifact-files.js says so now.

Verification

Local only, port 3011, DATA_DIR=/tmp/epic21b-data. Nothing was written under the worktree or under
any real data directory, and no existing artifact was touched. No dashboard UI changed, so there is
no browser capture.

saveArtifact only ever wrote, so a PUT converting html to a redirect left index.html
and source.html on disk and the reverse left source.url. Nothing served them, because
every serve path keys off meta.type, but they stayed at rest and on the git backend they
stayed in commit history after the artifact was deleted.

Adds delete(key) to the storage interface on all five backends: local removes the file
with force so a missing key is not an error, git delegates to local, s3 reuses the delete
helper that already treats 404 as done, and the two SQL stores take an equality match
rather than the range predicate the prefix operations need.

lib/artifact-files.js owns which objects each type owns and what a conversion drops, so a
unit test can walk all twenty ordered pairs without a server. server.js runs the deletes
after meta.json lands and before flush: a crash in between leaves the old record whole
rather than a listed artifact with no body, and git carries the deletions in the same
commit as the write. A delete that fails is logged, not thrown, so a cleanup problem
cannot turn a replace that already landed into a 500 the caller would retry.

Red first: five conversions on one slug left source.html, source.jsx, source.md,
source.tsx and source.url side by side; each hop now leaves exactly what the new type
owns.
…o it

The four review lenses on 3fa2a31. Two gaps were real and both are closed here.

The cleanup had no test proving anything called it. Replacing the loop body with
`for (const key of [])` left all 96 unit tests and the whole smoke suite green on
every backend, because a stale object is not reachable over HTTP and no test boots
server.js. The loop moved into `dropStaleObjects` in lib/artifact-files.js, where a
fake storage can record the keys it asks for. The same neuter now fails 3 tests.

A backend with no `delete` raised "storage.delete is not a function" straight into
the catch that exists so a failed cleanup cannot 500 a write that already landed, so
it would have warned once per conversion forever and failed nothing. `createStorage`
now checks every method the app calls and refuses the boot, which is how the backend
already treats a store it cannot reach. test/storage-contract.test.js drives local
and sqlite for real and the shared SQL wrapper for postgres; s3 and git are covered
by the boot check under the CI matrix.

Smaller findings from the same pass:

- pipeStream answered 500 when a read opened after the object was deleted. local
  stats then opens, so a conversion landing in between produced a hard 500 on a
  plain GET. It answers 404 now, which the comment above serveObject already
  promised and which the rename path has always given during its own window.
- ownedKeys and the /a/:slug/source route both looked SOURCE_EXT up bare, so a
  hand-edited meta.type of "constructor" built a key out of a function body. Both
  use Object.hasOwn now.
- TYPES is read off SOURCE_EXT, so a sixth type cannot be publishable and invisible
  to the cleanup. dashboard-check.mjs imports the same table instead of grepping
  server.js for the array.
- The update_artifact description said title resets when omitted, which T2.1.7 made
  false, and neither it nor docs/mcp.md said a type change now deletes files.
- The warn on a failed delete says what it means for the artifact and that nothing
  retries, matching the other operator lines in this repo.
- Dropped a smoke assertion that could not fire, and the duplicate ok-line label.

The commit message on 3fa2a31 says the old bytes "stayed in commit history"; that
reads as if this fixes it. It does not. A deletion on the git backend is another
commit, so the bytes stay in history and `git log -p` on the remote still hands them
back. lib/artifact-files.js says so now.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant