Skip to content

seed-dataset: land the dataset seeder on main with canonical contentHash - #48

Open
JamesCarnley wants to merge 12 commits into
mainfrom
fix/dataset-seeder-canonical-hash
Open

JamesCarnley wants to merge 12 commits into
mainfrom
fix/dataset-seeder-canonical-hash

Conversation

@JamesCarnley

@JamesCarnley JamesCarnley commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Lands the dataset seeder (yarn hardhat:seed:dataset) on main for the first time. It's the tool that published the 67 sample files on Sepolia (/whitepapers 40, /games 15, /standards 10, /cypherpunk 2).
  • Makes it write the canonical f1220 sha2-256 contentHash (specs/10, ADR-0064) instead of a bare keccak256 digest, and adds a heal step that corrects the existing files' claims in place.
  • Adds --attester <address>: --plan reads as the curator without its key, and --execute refuses to run under any other key.
  • Safety rails from an independent review (8c36dfa):
    • Explicit content source: --manifest or DATASET_ROOT is required, never a guessed sibling folder, which once resolved to a stale untracked copy. Each run logs the source and its git commit, and --execute refuses uncommitted content.
    • Signer checks: --attester is required on live chains, and the public Hardhat dev key is refused. Note that hardhat.config.ts falls back to that key when no key is loaded.
    • Tags checked first: tags are validated before the first write.
    • Single upload buffer: the upload uses the same buffer that was hashed.
  • Removal: --prune revokes the curator's placement of files the manifest no longer lists, in one multiRevoke.
  • Owner decisions (4715e34):
    • Licences: media (games, images, audio, video) must carry a licence; documents need not.
    • Unpin list: every run prints the IPFS CIDs it stops serving, for unpinning.

Why

The seeder lived only on codex/content-deploy-checkout, an archive branch for unreviewed v1 work, so it never had CI or review. It wrote bare 0x… keccak digests, which don't say which algorithm produced them. The SDK reports those as malformed-claim, so its strict readText/readBytes/readJson refuse all 67 files.

Two parts to the hash fix. First, write the canonical form (canonicalContentHash). Second, heal instead of re-seed. The seeder decides "already done" by comparing hash strings, and a legacy 0x claim never equals f1220…. Switching the format alone would make a re-run treat all 67 files as changed, re-pinning them to IPFS and minting new DATA, mirrors and placements for unchanged bytes. So when the live claim equals keccak256 of the same local bytes, the seeder binds a canonical contentHash to the existing DATA and re-pins it. The new PIN supersedes the old one (ADR-0052). That's 2 attestations per file, no IPFS, and file identity is preserved. A legacy claim of different bytes is still a real change.

Why --attester. Every read the seeder does is scoped to the loaded key's address. Without the curator key, --plan read an empty lens and reported every live file as a fresh write. And --execute with the wrong key would publish a second copy of the whole dataset under a lens the app never reads.

How this branch was rebuilt (history kept on purpose)

This PR used to target codex/content-deploy-checkout. Rather than rewrite the branch, it was brought to main additively, so every original commit stays traceable:

Commit What
be5078d Codex GPT-5 archives the seeder (2026-08-21)
495e8de Codex GPT-5 archives unrelated debug-client scratch, on the same branch
73ee047, 7990f91 Canonical hash and in-place heal (Claude Opus 5, 2026-09-21)
b6d9cde Reverts 495e8de. The scratch was never reviewed, touches files #47/#50/#51 changed since, and edits deployedContracts.ts
58d6f4c Merge main
e531e08 --attester
8c36dfa Safety rails from the independent review, plus --prune
4715e34 Media licence gate and the per-run unpin list

Net diff against main is the 11 seeder files only (scripts, task, tests, root/hardhat package.json script entries, runHardhatDeployWithPK.ts gaining an opt-in EFS_USE_DEPLOYER_KEY_ON_LOCALHOST and a non-zero exit when no key is set, plus docs/DEPLOYMENT.md and the June plan doc).

Permanence tier

Ephemeral: deploy tooling. What it writes is durable (PROPERTY values are non-revocable), which is why the canonical form and the --attester guard matter. No contract, schema or spec changes.

Specs / ADRs touched or checked

  • specs/10-file-metadata-encoding.md §2 and §8: implemented, unchanged
  • ADR-0064 (canonical f1220): confirmed
  • ADR-0052 (PROPERTY non-revocable; supersede at the PIN): the heal relies on it
  • ADR-0049 (DATA is pure identity; hash and size are PROPERTYs): confirmed
  • ADR-0055 (WHITEOUT): not used yet; see "Not in this PR"

Test plan

  • 28/28 seeder tests (SeedDataset.test.ts, SeedDatasetLib.test.ts); each new behaviour test failed before its change
  • Hardhat lint at --max-warnings=0, hardhat:check-types, docs:check
  • Live Sepolia, read-only, as the curator with no key loaded:
    npx hardhat seed:dataset --plan --attester 0x11CbE1b619bb9fe79e2F4C22c9A62412b3E79912 --network sepolia (with DATASET_ROOT pointing at the datasets folder) → heals=67 writes=0 skipped=0
  • Earlier (2026-09-21, Claude Opus 5): end to end on a pinned Sepolia fork, the original seeder created the legacy state and the new one executed healed=2 ipfsPins=0 txs=4 attestations=4; a re-run planned skip ×2
  • Not run: a real --execute on Sepolia (needs the curator key)

To publish

Content now comes from the efs-project/content repo (see efs-project/content#1 for the Devcon refresh). From a contracts checkout of main, after this merges:

DATASET_ROOT=../content/datasets yarn hardhat:seed:dataset --plan --prune --attester 0x11CbE1b619bb9fe79e2F4C22c9A62412b3E79912 --network sepolia

Read the plan, then run the same command with --execute --pin in place of --plan, using the curator key and IPFS_API_URL. Against efs-project/content#1 the plan is heals=29 writes=61 removals=0, plus a list of 38 old CIDs to unpin. Don't use datasets/deploy (the old Codex checkout) or the untracked datasets/ folder again.

Not in this PR

  • Unpinning. The seeder lists CIDs to unpin but doesn't unpin; that's done on the VPS node.
  • cid PROPERTY. specs/10 §4.2 says it SHOULD be written; it isn't yet.
  • Heal batching. Heals are 2 transactions per file and could be batched.
  • DATA and its key anchors are minted in separate transactions: the same front-running window fixed in the SDK (Scaffold the EFS SDK monorepo sdk#1). Low risk for a curator-run tool.
  • MAX_ANCHOR_DEPTH = 32 in the seeder's visibility walk is stricter than the contract's 256 (ADR-0068). Harmless for these paths.

Agents involved

  • Seeder: Codex GPT-5 (archived 2026-08-21)
  • Canonical hash + heal: Claude Opus 5 (2026-09-21)
  • Rebuild onto main, --attester, safety rails, --prune, licence gate, unpin list, live plans: Claude Opus 5.5 (2026-10-27)
  • Independent adversarial review: Claude Opus 5.5 subagent (2026-10-27)
  • Orchestrator: James

🤖 Generated with Claude Code

claude and others added 3 commits August 21, 2026 23:46
Preserve the exact June 23-26 working-tree snapshot without
presenting it as production or merge-ready work.

This is unreviewed v1 archival evidence. It predates the active v2
redesign and retains the known raw-keccak contentHash behavior. No file
contents were changed during recovery, and hooks were skipped so they
could not rewrite the snapshot.

Verification: staged blobs matched working-tree blobs; diff check
passed.

Permanence-tier: Ephemeral
Co-authored-by: Codex GPT-5 <noreply@openai.com>
Preserve the exact June 23 working-tree snapshot, including generated
local addresses and debug-only presentation and transport experiments.

This is unreviewed v1 archival evidence. It is not an adopted client
change, is not approved for merge, and predates the active v2 redesign.
No file contents were changed during recovery, and hooks were skipped so
they could not rewrite the snapshot.

Verification: staged blobs matched working-tree blobs; diff check
passed.

Permanence-tier: Ephemeral
Co-authored-by: Codex GPT-5 <noreply@openai.com>
…place

The dataset seeder — the tool that seeded the 67 demo files on Sepolia
(/games, /whitepapers, /standards, /cypherpunk) — wrote `keccak256(bytes)`
as contentHash: a bare `0x…` digest that does not say which algorithm made
it. specs/10 §8 required the canonical multibase multihash (`f1220` +
sha2-256) "before any durable data is seeded". The SDK reports the bare
form as `malformed-claim`, so its fail-closed readText/readBytes/readJson
refuse every one of those files.

Two changes, because the first alone would do real damage:

- Write the canonical form (`canonicalContentHash`).
- HEAL, don't re-seed. Idempotence compared hash STRINGS, and a legacy
  `0x`+keccak claim never equals the local `f1220…`, so a re-run would have
  treated all 67 files as changed content: re-pin to IPFS, mint new DATA,
  mirrors, key anchors, properties and placements for bytes that did not
  change. New decision `heal` (reason `legacy-content-hash`): when the
  active claim equals keccak256 of the SAME local bytes, bind a canonical
  contentHash PROPERTY on the EXISTING DATA and re-PIN it (binding PIN is
  cardinality-1, so it supersedes; ADR-0052). 2 attestations per file, no
  IPFS pin, DATA identity preserved. A legacy claim of different bytes is
  still a real content change and is re-seeded as before.

Verified:
- Against LIVE Sepolia, read-only: all 67 placed files carry a bare `0x`
  claim, and this branch's real decideSeedFileAction plans heal for all
  67, re-seed for none (~134 attestations total). The plan was computed
  from live claims + local dataset bytes without the curator key.
- End to end on a pinned Sepolia fork: the ORIGINAL seeder created the
  legacy state; the NEW seeder planned heal x2, executed healed=2 with
  ipfsPins=0, txs=4, attestations=4; afterwards both claims equal
  f1220+sha256(local bytes) (checked independently), DATA UIDs and size
  unchanged; a re-run plans skip x2.
- 630 contract tests (a fixture that encoded the old keccak "match" now
  uses the canonical claim), hardhat lint at --max-warnings=0, check-types.

Noted, not changed: this seeder mints DATA and its key anchors in separate
txs, the same front-running window fixed in the SDK (efs-project/sdk#1);
low risk for a curator-run tool.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

A heal re-binds one PROPERTY on an existing file; a write re-seeds the
whole file. The plan summary lumped them into one 'writes' count, which
hid the one real hazard of the heal run: with the WRONG signer every file
looks unplaced under that account, so the plan shows writes instead of
heals and executing would re-seed all 67 under it. The summary now prints
the signer and heals/writes separately — the check before --execute is
'signer=<curator> heals=67 writes=0'.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex archived two unrelated things on `codex/content-deploy-checkout`
on 2026-08-21: the dataset seeder (be5078d) and some unreviewed v1
debug-client scratch (495e8de: FileBrowser, ExplorerClient,
transports, containers, fetchFileContent, deployedContracts). Only the
seeder belongs in this PR. The scratch was never reviewed, touches
files that #47/#50/#51 have since changed on main, and edits the
committed contract addresses, so it is reverted here rather than
carried into main. Reverting instead of rewriting the branch keeps the
original commits in history for anyone tracing where the seeder came
from.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… mis-sign

Every chain read in the seeder is lens-scoped to the loaded key's
address. That made `--plan` useless without the curator key: run with
any other key, it read an empty lens and reported every live file as a
fresh write. Worse, `--execute` with the wrong key would publish a
second copy of the whole dataset under a lens the app never reads.

`--attester <address>` names the lens the dataset is published under.
`--plan` reads as that address, so anyone can preview a run without the
key. `--execute` refuses before any chain read, IPFS pin or write
unless the loaded key is that address. Omitting it keeps the old
behaviour.

Verified: 16/16 seeder tests (3 new, each failing before the change);
hardhat lint and check-types. Live, read-only against Sepolia as the
curator 0x11Cb…9912 with no key loaded: 40+10+2+15 files across the
four manifests plan heals=67 writes=0 skipped=0.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@JamesCarnley
JamesCarnley changed the base branch from codex/content-deploy-checkout to main October 27, 2026 20:26
@JamesCarnley JamesCarnley changed the title fix(seed-dataset): mint canonical contentHash; heal legacy claims in place seed-dataset: land the dataset seeder on main with canonical contentHash Oct 27, 2026
@JamesCarnley JamesCarnley reopened this Oct 27, 2026
An independent review of the seeder before its first public run found
ways a normal-looking command could publish the wrong thing, permanently:

- Source. With no --manifest, and for relative paths, it guessed
  sibling folders. From packages/hardhat that resolved to an untracked,
  stale copy of the datasets (verified: pong.html hashed to the old
  bytes). Now --plan/--execute need --manifest or DATASET_ROOT, paths
  resolve against INIT_CWD, every run logs the absolute source and its
  git commit, and --execute refuses a folder that isn't a clean git
  checkout (--allow-uncommitted to override).
- Signer. hardhat.config.ts falls back to the public Hardhat key when
  none is loaded, on every network. --execute on a non-local chain now
  requires --attester and refuses that key. The config comment claiming
  it "fails closed" is corrected. The yarn wrapper no longer asks for
  the key password for read-only runs.
- Tags. A tag the chain rejects used to revert mid-file, after the
  file's DATA and PROPERTYs were already permanent, and every rerun
  minted another orphan. All manifests and tags are now validated
  before the first read or write.
- Upload bytes. The file was hashed from one read and uploaded from a
  second; now one buffer feeds both, so CID, contentHash and size can't
  disagree.
- Removal. --prune revokes the attester's placement PIN for files under
  a dataset anchor that the manifest no longer lists, in one
  multiRevoke. Without it those files are reported as warnings. Refused
  with --only. WHITEOUT isn't needed: the curator is the only lens.

Verified: 25/25 seeder tests (9 new, each failing first); hardhat lint
and check-types. Live, read-only, via `yarn hardhat:seed:dataset
--plan --prune --attester <curator> --network sepolia` with
DATASET_ROOT=content/datasets: no password prompt, source commit
logged, heals=59 writes=31 removals=0.

Reviewed-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Two owner decisions for the Devcon content refresh (2026-10-27):

- Licences matter for media, not documents. Files served as games,
  images, audio, video, 3D models or wasm now need a `license` (on the
  entry or in `defaults`; NOASSERTION doesn't count) before the seeder
  will plan or publish them. Whitepapers, standards and essays are
  exempt and may be hosted without one.
- Removed and replaced files should be unpinned from the VPS IPFS node
  afterwards. Each run now prints the `ipfs://` CIDs it stops serving:
  the old copy of every file it republishes with new bytes, and every
  file --prune removes. --plan prints the same list in advance. Heals
  keep the same bytes, so they retire nothing.

Verified: 28/28 seeder tests (3 new; the two behaviour tests failed
first); hardhat lint and check-types.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
JamesCarnley and others added 3 commits November 12, 2026 15:16
A fork run of the full Devcon publish (independent review) found that
tags were only ever written on new files. Skip and heal returned before
looking at tags, so a changed tag list in the manifest never reached
chain, and stale tags lived on: 9 kept PDFs would still have carried
`interoperability` and lacked `proof-of-stake`. Republished files left
their tags on the old DATA, so /tags/payment would have listed 10 files
(6 in the manifest) and /tags/flagged still 2.

Now, for the curator's lens:
- skip and heal diff the file's active TAGs (definitions under /tags)
  against the manifest: missing tags are attested, extra ones revoked,
  one batch call each;
- a write over an existing placement revokes the old DATA's tags;
- --prune revokes a removed file's tags with its placement PIN;
- --plan reports `plan tags <file> +a -b` and `plan untag <file> (old
  copy)`, and the summary counts tagChanges, so drift is visible before
  anything is sent.

Verified: 33/33 seeder tests (5 new; each failed before the change).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
From the fork run of the full publish:

- A crash lost the unpin list, which was only printed at the end. Each
  --execute now appends an NDJSON record as it goes, to the content
  repo's deployments/<chainId>/seed-<time>.ndjson (or --record-dir):
  run header (source and seeder commits, signer, attester, datasets),
  every file action with its DATA UID, mirror and hash, every tx hash,
  tag changes, removals, CIDs to unpin, and a closing summary. It sits
  outside the dataset folders, so a re-run after a crash still passes
  the clean-checkout check.
- A stalled IPFS upload hung the run. Uploads now time out after 10
  minutes and fail like any other pin error, before anything for that
  file is written; re-running resumes.
- docs/DEPLOYMENT.md described the old flow (../datasets, no
  --attester or --prune). It now gives the plan / execute / re-plan
  runbook from the content repo, the safety rails, the run record, and
  the warning that a devnet reset deletes the Kubo volume.

Verified: 34/34 seeder tests (1 new for the record); hardhat lint,
check-types and docs:check.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants