Repository navigation
seed-dataset: land the dataset seeder on main with canonical contentHash - #48
Open
JamesCarnley wants to merge 12 commits into
Open
JamesCarnley wants to merge 12 commits into
JamesCarnley wants to merge 12 commits into
Conversation
Preserve the exact June 23-26 working-tree snapshot without presenting it as production or merge-ready work. This is unreviewed v1 archival evidence. It predates the active v2 redesign and retains the known raw-keccak contentHash behavior. No file contents were changed during recovery, and hooks were skipped so they could not rewrite the snapshot. Verification: staged blobs matched working-tree blobs; diff check passed. Permanence-tier: Ephemeral Co-authored-by: Codex GPT-5 <noreply@openai.com>
Preserve the exact June 23 working-tree snapshot, including generated local addresses and debug-only presentation and transport experiments. This is unreviewed v1 archival evidence. It is not an adopted client change, is not approved for merge, and predates the active v2 redesign. No file contents were changed during recovery, and hooks were skipped so they could not rewrite the snapshot. Verification: staged blobs matched working-tree blobs; diff check passed. Permanence-tier: Ephemeral Co-authored-by: Codex GPT-5 <noreply@openai.com>
…place The dataset seeder — the tool that seeded the 67 demo files on Sepolia (/games, /whitepapers, /standards, /cypherpunk) — wrote `keccak256(bytes)` as contentHash: a bare `0x…` digest that does not say which algorithm made it. specs/10 §8 required the canonical multibase multihash (`f1220` + sha2-256) "before any durable data is seeded". The SDK reports the bare form as `malformed-claim`, so its fail-closed readText/readBytes/readJson refuse every one of those files. Two changes, because the first alone would do real damage: - Write the canonical form (`canonicalContentHash`). - HEAL, don't re-seed. Idempotence compared hash STRINGS, and a legacy `0x`+keccak claim never equals the local `f1220…`, so a re-run would have treated all 67 files as changed content: re-pin to IPFS, mint new DATA, mirrors, key anchors, properties and placements for bytes that did not change. New decision `heal` (reason `legacy-content-hash`): when the active claim equals keccak256 of the SAME local bytes, bind a canonical contentHash PROPERTY on the EXISTING DATA and re-PIN it (binding PIN is cardinality-1, so it supersedes; ADR-0052). 2 attestations per file, no IPFS pin, DATA identity preserved. A legacy claim of different bytes is still a real content change and is re-seeded as before. Verified: - Against LIVE Sepolia, read-only: all 67 placed files carry a bare `0x` claim, and this branch's real decideSeedFileAction plans heal for all 67, re-seed for none (~134 attestations total). The plan was computed from live claims + local dataset bytes without the curator key. - End to end on a pinned Sepolia fork: the ORIGINAL seeder created the legacy state; the NEW seeder planned heal x2, executed healed=2 with ipfsPins=0, txs=4, attestations=4; afterwards both claims equal f1220+sha256(local bytes) (checked independently), DATA UIDs and size unchanged; a re-run plans skip x2. - 630 contract tests (a fixture that encoded the old keccak "match" now uses the canonical claim), hardhat lint at --max-warnings=0, check-types. Noted, not changed: this seeder mints DATA and its key anchors in separate txs, the same front-running window fixed in the SDK (efs-project/sdk#1); low risk for a curator-run tool. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
A heal re-binds one PROPERTY on an existing file; a write re-seeds the whole file. The plan summary lumped them into one 'writes' count, which hid the one real hazard of the heal run: with the WRONG signer every file looks unplaced under that account, so the plan shows writes instead of heals and executing would re-seed all 67 under it. The summary now prints the signer and heals/writes separately — the check before --execute is 'signer=<curator> heals=67 writes=0'. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex archived two unrelated things on `codex/content-deploy-checkout` on 2026-08-21: the dataset seeder (be5078d) and some unreviewed v1 debug-client scratch (495e8de: FileBrowser, ExplorerClient, transports, containers, fetchFileContent, deployedContracts). Only the seeder belongs in this PR. The scratch was never reviewed, touches files that #47/#50/#51 have since changed on main, and edits the committed contract addresses, so it is reverted here rather than carried into main. Reverting instead of rewriting the branch keeps the original commits in history for anyone tracing where the seeder came from. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… mis-sign Every chain read in the seeder is lens-scoped to the loaded key's address. That made `--plan` useless without the curator key: run with any other key, it read an empty lens and reported every live file as a fresh write. Worse, `--execute` with the wrong key would publish a second copy of the whole dataset under a lens the app never reads. `--attester <address>` names the lens the dataset is published under. `--plan` reads as that address, so anyone can preview a run without the key. `--execute` refuses before any chain read, IPFS pin or write unless the loaded key is that address. Omitting it keeps the old behaviour. Verified: 16/16 seeder tests (3 new, each failing before the change); hardhat lint and check-types. Live, read-only against Sepolia as the curator 0x11Cb…9912 with no key loaded: 40+10+2+15 files across the four manifests plan heals=67 writes=0 skipped=0. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
JamesCarnley
changed the base branch from
codex/content-deploy-checkout
to
main
October 27, 2026 20:26
An independent review of the seeder before its first public run found ways a normal-looking command could publish the wrong thing, permanently: - Source. With no --manifest, and for relative paths, it guessed sibling folders. From packages/hardhat that resolved to an untracked, stale copy of the datasets (verified: pong.html hashed to the old bytes). Now --plan/--execute need --manifest or DATASET_ROOT, paths resolve against INIT_CWD, every run logs the absolute source and its git commit, and --execute refuses a folder that isn't a clean git checkout (--allow-uncommitted to override). - Signer. hardhat.config.ts falls back to the public Hardhat key when none is loaded, on every network. --execute on a non-local chain now requires --attester and refuses that key. The config comment claiming it "fails closed" is corrected. The yarn wrapper no longer asks for the key password for read-only runs. - Tags. A tag the chain rejects used to revert mid-file, after the file's DATA and PROPERTYs were already permanent, and every rerun minted another orphan. All manifests and tags are now validated before the first read or write. - Upload bytes. The file was hashed from one read and uploaded from a second; now one buffer feeds both, so CID, contentHash and size can't disagree. - Removal. --prune revokes the attester's placement PIN for files under a dataset anchor that the manifest no longer lists, in one multiRevoke. Without it those files are reported as warnings. Refused with --only. WHITEOUT isn't needed: the curator is the only lens. Verified: 25/25 seeder tests (9 new, each failing first); hardhat lint and check-types. Live, read-only, via `yarn hardhat:seed:dataset --plan --prune --attester <curator> --network sepolia` with DATASET_ROOT=content/datasets: no password prompt, source commit logged, heals=59 writes=31 removals=0. Reviewed-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Two owner decisions for the Devcon content refresh (2026-10-27): - Licences matter for media, not documents. Files served as games, images, audio, video, 3D models or wasm now need a `license` (on the entry or in `defaults`; NOASSERTION doesn't count) before the seeder will plan or publish them. Whitepapers, standards and essays are exempt and may be hosted without one. - Removed and replaced files should be unpinned from the VPS IPFS node afterwards. Each run now prints the `ipfs://` CIDs it stops serving: the old copy of every file it republishes with new bytes, and every file --prune removes. --plan prints the same list in advance. Heals keep the same bytes, so they retire nothing. Verified: 28/28 seeder tests (3 new; the two behaviour tests failed first); hardhat lint and check-types. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 27, 2026
A fork run of the full Devcon publish (independent review) found that tags were only ever written on new files. Skip and heal returned before looking at tags, so a changed tag list in the manifest never reached chain, and stale tags lived on: 9 kept PDFs would still have carried `interoperability` and lacked `proof-of-stake`. Republished files left their tags on the old DATA, so /tags/payment would have listed 10 files (6 in the manifest) and /tags/flagged still 2. Now, for the curator's lens: - skip and heal diff the file's active TAGs (definitions under /tags) against the manifest: missing tags are attested, extra ones revoked, one batch call each; - a write over an existing placement revokes the old DATA's tags; - --prune revokes a removed file's tags with its placement PIN; - --plan reports `plan tags <file> +a -b` and `plan untag <file> (old copy)`, and the summary counts tagChanges, so drift is visible before anything is sent. Verified: 33/33 seeder tests (5 new; each failed before the change). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
From the fork run of the full publish: - A crash lost the unpin list, which was only printed at the end. Each --execute now appends an NDJSON record as it goes, to the content repo's deployments/<chainId>/seed-<time>.ndjson (or --record-dir): run header (source and seeder commits, signer, attester, datasets), every file action with its DATA UID, mirror and hash, every tx hash, tag changes, removals, CIDs to unpin, and a closing summary. It sits outside the dataset folders, so a re-run after a crash still passes the clean-checkout check. - A stalled IPFS upload hung the run. Uploads now time out after 10 minutes and fail like any other pin error, before anything for that file is written; re-running resumes. - docs/DEPLOYMENT.md described the old flow (../datasets, no --attester or --prune). It now gives the plan / execute / re-plan runbook from the content repo, the safety rails, the run record, and the warning that a devnet reset deletes the Kubo volume. Verified: 34/34 seeder tests (1 new for the record); hardhat lint, check-types and docs:check. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
yarn hardhat:seed:dataset) onmainfor the first time. It's the tool that published the 67 sample files on Sepolia (/whitepapers40,/games15,/standards10,/cypherpunk2).f1220sha2-256contentHash(specs/10, ADR-0064) instead of a barekeccak256digest, and adds ahealstep that corrects the existing files' claims in place.--attester <address>:--planreads as the curator without its key, and--executerefuses to run under any other key.8c36dfa):--manifestorDATASET_ROOTis required, never a guessed sibling folder, which once resolved to a stale untracked copy. Each run logs the source and its git commit, and--executerefuses uncommitted content.--attesteris required on live chains, and the public Hardhat dev key is refused. Note thathardhat.config.tsfalls back to that key when no key is loaded.--prunerevokes the curator's placement of files the manifest no longer lists, in onemultiRevoke.4715e34):Why
The seeder lived only on
codex/content-deploy-checkout, an archive branch for unreviewed v1 work, so it never had CI or review. It wrote bare0x…keccak digests, which don't say which algorithm produced them. The SDK reports those asmalformed-claim, so its strictreadText/readBytes/readJsonrefuse all 67 files.Two parts to the hash fix. First, write the canonical form (
canonicalContentHash). Second, heal instead of re-seed. The seeder decides "already done" by comparing hash strings, and a legacy0xclaim never equalsf1220…. Switching the format alone would make a re-run treat all 67 files as changed, re-pinning them to IPFS and minting new DATA, mirrors and placements for unchanged bytes. So when the live claim equals keccak256 of the same local bytes, the seeder binds a canonicalcontentHashto the existing DATA and re-pins it. The new PIN supersedes the old one (ADR-0052). That's 2 attestations per file, no IPFS, and file identity is preserved. A legacy claim of different bytes is still a real change.Why
--attester. Every read the seeder does is scoped to the loaded key's address. Without the curator key,--planread an empty lens and reported every live file as a fresh write. And--executewith the wrong key would publish a second copy of the whole dataset under a lens the app never reads.How this branch was rebuilt (history kept on purpose)
This PR used to target
codex/content-deploy-checkout. Rather than rewrite the branch, it was brought tomainadditively, so every original commit stays traceable:be5078d495e8de73ee047,7990f91b6d9cde495e8de. The scratch was never reviewed, touches files #47/#50/#51 changed since, and editsdeployedContracts.ts58d6f4cmaine531e08--attester8c36dfa--prune4715e34Net diff against
mainis the 11 seeder files only (scripts, task, tests, root/hardhatpackage.jsonscript entries,runHardhatDeployWithPK.tsgaining an opt-inEFS_USE_DEPLOYER_KEY_ON_LOCALHOSTand a non-zero exit when no key is set, plusdocs/DEPLOYMENT.mdand the June plan doc).Permanence tier
Ephemeral: deploy tooling. What it writes is durable (PROPERTY values are non-revocable), which is why the canonical form and the
--attesterguard matter. No contract, schema or spec changes.Specs / ADRs touched or checked
specs/10-file-metadata-encoding.md§2 and §8: implemented, unchangedf1220): confirmedTest plan
SeedDataset.test.ts,SeedDatasetLib.test.ts); each new behaviour test failed before its change--max-warnings=0,hardhat:check-types,docs:checknpx hardhat seed:dataset --plan --attester 0x11CbE1b619bb9fe79e2F4C22c9A62412b3E79912 --network sepolia(withDATASET_ROOTpointing at the datasets folder) →heals=67 writes=0 skipped=0healed=2 ipfsPins=0 txs=4 attestations=4; a re-run plannedskip×2--executeon Sepolia (needs the curator key)To publish
Content now comes from the
efs-project/contentrepo (see efs-project/content#1 for the Devcon refresh). From a contracts checkout ofmain, after this merges:Read the plan, then run the same command with
--execute --pinin place of--plan, using the curator key andIPFS_API_URL. Against efs-project/content#1 the plan isheals=29 writes=61 removals=0, plus a list of 38 old CIDs to unpin. Don't usedatasets/deploy(the old Codex checkout) or the untrackeddatasets/folder again.Not in this PR
cidPROPERTY. specs/10 §4.2 says it SHOULD be written; it isn't yet.MAX_ANCHOR_DEPTH = 32in the seeder's visibility walk is stricter than the contract's 256 (ADR-0068). Harmless for these paths.Agents involved
main,--attester, safety rails,--prune, licence gate, unpin list, live plans: Claude Opus 5.5 (2026-10-27)🤖 Generated with Claude Code