Skip to content

🐛 fix(okf): hash-checked, rollback-safe node mv and migrate --apply - #10

Open
LGuichet wants to merge 39 commits into
nextfrom
fix/hash-checked-writes
Open

LGuichet wants to merge 39 commits into
nextfrom
fix/hash-checked-writes

Conversation

@LGuichet

@LGuichet LGuichet commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

fix(okf): hash-checked, rollback-safe node mv and migrate --apply

The branch is up to date with next (PR #5 merged there, so the diff is only this PR). Commits that are this PR:

✨ feat(fsutil): hash-checked, rollback-safe multi-file transaction
🐛 fix(okf): refuse stale plans and roll back partial node mv and migrate --apply
🐛 fix(okf): double-quote a node new value that holds a control character

The branch also carried CI fixes for problems that came from #5 (lint, the Go 1.26.10 bump, the touch duplicate-key fuzz crasher). next has since landed its own versions of those (#12, cecb642, 53524f6). In the merge, next's versions won wherever the two overlapped.

The bug

PR #5 made each file write atomic. Neither command was atomic across several files, and neither noticed files changing after it read them:

  • node mv rewrote the linking nodes one by one, then created the destination directory and renamed the node. If the directory step or the rename failed, the rewrites stayed: links pointed at a node that never moved. A node edited after the bundle was loaded was overwritten without notice.
  • migrate --apply wrote each node as it went. A failure on a late node left the bundle half-migrated. The plan was applied to whatever the files held at apply time, even after they were edited after phase 1.

The fix

  • fsutil.Tx (new, standard library only). It stages writes and renames, then:

    1. checks every target against the SHA-256 the caller planned against, before writing anything;
    2. applies the steps in order, re-checking each one right before it runs;
    3. on a failure, undoes the applied steps newest first: restores the original bytes, removes files it created, renames back, and removes only the directories it created.

    A rollback that cannot finish is reported (rollback incomplete: …), never silent.

  • Load records each node's SHA-256 (Node.SHA256).

  • ApplyMove checks the moved node and every rewritten node against those hashes, then commits the rewrites and the move as one Tx.

  • migrate:

    • Phase 1 writes a sha256 per plan node.
    • --apply renders every node and both version markers (.okf, root index.md) first, then commits them as one Tx. A node that no longer matches refuses the whole plan before any write.
    • --dry-run runs the same checks, so the preview fails exactly where the apply would.

No validate or lint rule changes. Nothing about what a bundle may contain changes.

Decision for review: plans without sha256

sha256 is omitempty. A plan written before this change still applies, unchecked, with a warning on stderr:

warning: 3 plan node(s) carry no sha256, so edits made since phase 1 cannot be detected for them; re-run phase 1 to add the check.

The alternative was refusing such plans, which would break any saved plan. The reverse is a one-line change if you prefer it.

Limits

  • The undo runs in-process from in-memory copies of the originals. It covers a step that fails, not a crash or a kill part-way through a commit. A crash-safe journal is a larger change.
  • A check and its write are separate syscalls, so an edit landing in the microseconds between them is not caught. The re-check right before each step narrows that window; it does not close it.
  • node promote has its own writer and is not changed here.

Spec

This is write safety, not OKF-defined behavior: no spec section governs atomicity or drift. The parts that touch spec-defined content are unchanged:

The v0.1 fallbacks are untouched.

Gate

1. Conformance

$ go test ./internal/okf/ ./cmd/ -run Conformance -count=1 -v
16 × --- PASS (TestConformance_*, including MigrateV01ToV02ValidatesAndReadsBack,
              V01NegativeControl_UnchangedAndFallbacks_Section13_1, MigrateCommandV01ToV02Validates)
ok  github.com/cwest/okfctl/internal/okf
ok  github.com/cwest/okfctl/cmd

2. Full suite

$ gofmt -l <changed .go files, CRLF-normalised>   -> empty
$ go vet ./...                                     -> clean
$ CGO_ENABLED=0 go build .                         -> ok
$ go test ./... -count=1
ok  cmd/okfctl-api, cmd/okfctl-search, internal/agentplugin, internal/apiserver,
    internal/fsutil, internal/okfconfig, internal/search
FAIL cmd, internal/okf, internal/plugin: 17 tests, the SAME 17 that fail on unchanged next (efc58b3)
     on this machine (diffed by name): shell-script plugin stubs and $EDITOR scripts
     ("%1 is not a valid Win32 application", "not found on PATH") and git CRLF checkouts
     (TestConnect_*, TestClone_FromLocalBareRepo, TestPullFastForward_PicksUpNewCommit).

-race: it ran in CI, because this machine has no cgo toolchain. See CI below.

3. Real corpus: market-fr/knowledge-base, next binary vs branch binary

Read-only. The output files are byte-identical before and after:

bundle validate before → after lint --strict before → after
ifpos-interface-cases OK → OK 0 → 0
middleware OK → OK 0 → 0
nf525-lne OK → OK 2 → 2 (broken-link ×2, exit 1)
plateforme-agree OK → OK 1 → 1 (tag-hygiene, exit 1)

Nothing moved: the change touches no read path.

Write smoke on cp -r scratch copies of nf525-lne. The node is mapping/contract-mapping.md, which 37 files link to:

successful move   next:   exit 0, 49 inbound link(s) rewritten, validate exit 0
                  branch: exit 0, 49 inbound link(s) rewritten, validate exit 0
                  -> the two resulting trees are byte-identical

failing move      (destination parent `blocker` is a regular file)
                  next:   exit 1; tree CHANGED: 36 file(s) differ   <- links point at a node that never moved
                  branch: exit 1; tree unchanged
leftover .tmp- files: 0

v0.1 fixture, migrate through the CLI. Three nodes with legacy timestamp and body # Citations. c.md is edited after phase 1:

drifted apply   next:   exit 0; tree CHANGED: 5 file(s)   <- stale plan applied silently
                branch: exit 1; tree unchanged
                okfctl: migrate plan node c.md: file changed since the plan was made; nothing was written (re-run phase 1 to plan against the current files)
fresh plan      next / branch: apply exit 0, validate exit 0
plan sha256     next: 0   branch: 3

CI

  • -race: go test ./... -race passed in every package in CI (build-test). It cannot run locally: this Windows machine has no cgo toolchain.
  • Earlier failures came from 🐛 fix(write): write-safety audit—atomic, confined, injection-free writes #5. lint, govulncheck and a touch fuzz crasher all came from 🐛 fix(write): write-safety audit—atomic, confined, injection-free writes #5, and next has fixed them since.
  • fuzz, after the merge with next: FuzzNewNodeFrontmatter found that node new --title with a tab, a newline, then 0 wrote a yaml.v3 literal block whose tab-led first line does not parse back.
    • Fix: a238165 writes any value containing a control character double-quoted, which escapes it and round-trips. Every other value keeps the plain style, so normal nodes are written byte-for-byte as before; the existing node new and conformance tests are unchanged.
    • The crasher is committed as the seed testdata/fuzz/FuzzNewNodeFrontmatter/tab-led-multiline-title. It fails before the fix and passes after. A local 45-second fuzz run found nothing new.
  • Merge with next (5aa4404):
    • mutate.go was unchanged on next, so this PR's version stands.
    • In migrate.go, next's control-character quoting of the okf_version target (53524f6) is re-applied inside the renamed stageRootIndexMarker.
    • writepath_fuzz_test.go keeps next's new seeds and this PR's transaction-driven marker commit.
    • Conformance passes after the merge. The full local suite fails only the Windows-environment tests that fail on next too (plugin stubs, $EDITOR scripts, git CRLF).

Controls

Failed first. Base-compatible copies of the positive controls, run against unchanged next (efc58b3):

--- FAIL: TestFailFirst_MigrateLateFailureWritesNothing
    half-migrated: a.md now "---\ntype: Concept\ntitle: A\ngenerated: {by: casey, ...}"
--- FAIL: TestFailFirst_MoveEditedSinceLoadRefused
    a node edited after the load was moved without a check
--- FAIL: TestFailFirst_MoveFailedDirectoryLeavesTreeUnchanged
    links rewritten to a node that never moved: a.md now "...See [x](blocker/foo.md)."

On the branch, the same scenarios pass as:

positive control (the defect is caught) negative control (legitimate case stays silent)
fsutil.Tx RefusesDriftBeforeAnyWrite, RollsBackWritesWhenRenameFails, RollsBackWritesWhenMkdirFails, RollbackRemovesCreatedFile, RefusesExistingRenameDestination CommitsWhenNothingChanged, RollbackKeepsPreexistingDir
node mv RefusesLinkerEditedSinceLoad, RefusesMovedNodeEditedSinceLoad, FailedDirectoryLeavesTreeUnchanged UnchangedBundleStillMoves, InMemoryNodeSkipsHashCheck, every existing ApplyMove/PlanMove test
migrate --apply RefusesNodeEditedSincePlan (apply and CheckMigratePlan), LateFailureWritesNothing, CLI ApplyRefusesNodeEditedSincePlan (apply and --dry-run) RecordsNodeHashes, UnchangedBundleMigrates, LegacyPlanWithoutHashesApplies, CLI ApplyLegacyPlanWarnsAndApplies, ApplyHashedPlanNoWarning, every existing migrate test

FuzzRootIndexMarker now drives the staged marker write through Tx.Commit.

On Linux, FailedDirectoryLeavesTreeUnchanged is refused at the up-front stat (ENOTDIR) before any write. On Windows it reaches the directory step and exercises the rollback. Both end with the same assertion. The rollback path itself is covered on every platform by the fsutil tests, which inject the failure.

Notes for review

  • testdata/fuzz/FuzzTouchModifiedRoundTrip/duplicate-key-block repeats a case next now seeds inline. It is kept as a file seed and does no harm.

cwest and others added 30 commits September 12, 2026 03:04
lint warned that Concept and Concepts are one type spelled twice but had
no opinion about Wine/wine or runbook/runbooks/run-book — even though tags
is the other §4.1 field every consumer routes and filters on. The --tag
scope filter and the /api/v1/search tag parameter match exactly and
case-sensitively, so one tag spelled three ways is three disjoint result
sets, and analyze's cluster detector counted the variants separately and
missed the cluster.

Extract the shared base fold (case, trim, single trailing 's') as canonFold;
canonType is now a thin wrapper over it, so type-hygiene is byte-identical.
canonTag layers separator-insensitivity (-, _, space) on top — for tags
only. Add lintTagHygiene as a sibling of lintTypeHygiene: bundle-level,
advisory, "tag-hygiene" in --json, --strict treats it like any finding;
the message lists each variant with its per-node count, variants sorted.
analyze adopts canonTag for cluster detection so it and lint agree on when
two spellings are one tag, and it now dedups the node set per cluster
(the old per-occurrence append double-counted a tag repeated within one
node's list.

The check never rejects a value (PRD §7.4 anti-taxonomy): it flags only
spellings that fold to one value, never two genuinely distinct ones.

Closes cwest#166
EOF
)
…yze labels

The bare trailing-'s' fold merged distinct short tags (new/news, op/ops,
aws/aw, js/j, k8s/k8) and missed real variants (-es and -ies plurals,
ci/cd, node.js). Fold tags instead through tagSurface (case, trim, and
-, _, space, / separators, plus a full stop between letters per UAX cwest#29)
and tagKeys, which applies KStem's plural rules (Lucene KStemmer.java,
derived from the UMass CIIR KStemmer): letters-only, more than two
letters, a plain 's' dropped only after more than three letters and not
after 'ss' or 'ous', and the Snowball English invariant forms (news,
atlas, ...). KStem's dictionary lookup is stood in for by the bundle's
own tag vocabulary: a spelling folds onto a candidate only when that
candidate is itself a tag, so non-word stems never become keys. The
partition depends only on the tag set, never its order. The type path
(canonType) is unchanged.

analyze labels clusters with the lower-cased variant and orders them by
label, so case-only and separator-only clusters read exactly as before
the fold swap (main labelled Wine/wine as "wine"; the raw variant made
it "Wine"). lint's long help now describes the fold as a heuristic
instead of claiming it never flags distinct tags.

Tests: the negative controls are pairs that folded under the old rule
(each fails against it), positive controls cover the new merges, the
tag-hygiene message and cluster labels are pinned exactly, and the
digraph control that could not fail is replaced.
On Windows with core.autocrlf=true a fresh clone checks index.md out as
CRLF, and `index check` reported every index stale although the content
was identical, because it compared bytes with the LF render.

- index check compares with CRLF folded to LF; any other difference is
  still drift.
- index build keeps the line endings of the index.md it replaces: an
  all-CRLF file stays CRLF, a new or mixed-ending file is written LF.
  The choice depends only on that file, so it is deterministic and
  git sees no whole-file ending flip.
- the subdirectory-description parser strips CR before matching, so a
  CRLF index no longer captures the old shape suffix into the curator
  description (which made check fail and build double the suffix).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d the §8 index suffix

Lift the per-directory aggregation that `index build` renders as the
subdirectory shape suffix (OKF §8 progressive disclosure) out of dirShape
into a structured DirShape: own and subtree concept counts, the sorted
distinct `type` values (§7.4 open vocabulary, kept as written), and the
§4.1 shared tags with a truncation flag.

Shape computes it for one directory and Shapes for every IndexDirs
entry. dirShape now renders FROM the struct, so the index suffix and any
other consumer (the PRD §6.1 "neighborhoods" view of `bundle info`)
cannot drift apart. No behavior change to the rendered suffix.

Tests cover own-vs-subtree counting, own-concept-only types and tags,
tag-cap truncation, byte-equality between Shape and the rendered suffix,
a build→validate round trip, and Shapes covering every index dir.
…ex.md

OKF §12 defines where a bundle declares the version it targets: an
`okf_version` key in the bundle-root index.md frontmatter block. Load
read the version only from the .okf sidecar, which is okfctl's own pin
and not part of the spec, so a spec-conforming bundle carrying the
marker and no sidecar was reported as SpecVersion whatever it declared.
index build already resolved the marker first (bundleRootOkfVersion);
load and build disagreed about the same bundle.

Resolve Bundle.OkfVersion in the order index build writes the marker:
the root-index marker, else the sidecar, else SpecVersion. Only the
root index counts (§12 permits the marker nowhere else). The declared
string is reported as declared: per §12 an unrecognised version is
consumed best-effort, never rejected.

Tests pin both directions: marker-only, marker-over-sidecar, bare
scalar and unrecognised values are read (positive); a nested index.md
marker, a blank marker and a sidecar-only bundle leave the result
unchanged (negative). The conformance suite gains the consumer half of
the §12 round trip: the marker index build preserves is the one Load
reads back.
…ally

Every bundle writer in internal/okf (node new, TouchModifiedFile, node mv
and node promote link rewrites, migrate apply, Scaffold, index build, log
append) used os.WriteFile, which truncates the target in place: a crash or
a full disk mid-write left a half-written node behind. They now go through
fsutil.WriteFileAtomic (temp file in the same directory, sync, rename).

Each rewrite path is pinned by a hard-link test that fails on an in-place
write; a v0.1 node (legacy timestamp, body # Citations) still migrates and
touches as before.
…tomically

okfconfig.Save, search.(*Store).Save and the migrate --plan writer used
os.WriteFile, which truncates in place; a crash or a concurrent reader
could observe a partial file. Route them through fsutil.WriteFileAtomic
(temp file plus rename). Each swap is pinned by a hard-link test that
fails against the in-place writer.
OKF §3 defines a bundle as its directory tree. node new checked only the
lexical path, so a symlinked or junctioned directory inside the bundle
could carry the write outside it; node edit did no check at all and
launched the editor on ../outside.md. Both now resolve the user path
through fsutil.ResolveInRoot before touching disk. A nested a/b/c path
and a title carrying --- in prose still write as before.
ApplyMove joined the destination onto the root with no containment check,
so node mv a ../x.md renamed the node out of the bundle (OKF §3) after
already rewriting its inbound links. Both ends now go through
fsutil.ResolveInRoot before any write; a control character in the
destination is refused too, since the path is echoed into link text.
A nested in-bundle destination still moves.
…one segment

PromotePlan accepted any --name, so ../x, a/b or index wrote the promoted
concept outside its directory, outside the bundle (OKF §3), or over a §3.1
reserved file; a line break in it landed raw in every rewritten link.
--name must now be one ordinary, non-reserved file-name segment.
PromoteApply confines every path a plan names (old index, new concept,
rewrite targets) through fsutil.ResolveInRoot before any write. The
default name, an ordinary name and one with --- in it are still accepted.
The plan file is user-editable between phases. MigrateApply joined each
node path onto the root unchecked, so a plan naming ../victim.md rewrote
a file outside the bundle (OKF §3), and wrote target_version verbatim into
the line-keyed .okf sidecar, so a line break forged a second key (§12).
Every plan node path now goes through fsutil.ResolveInRoot and a target
carrying a control character is refused, both before any write.
AppendLog flattened a message to its first line by cutting at LF only.
CommonMark also ends a line at a bare CR, so a message like
"a<CR>## 1999-01-01" wrote a forged date heading into the §9 log. Cut at
the first CR or LF.
…aths

Four fuzz targets assert the write-safety properties on untrusted input:
no injected frontmatter key from node new, no lost key or body from a
modified refresh, a key-stable root index.md okf_version rewrite, and
exactly one entry and heading per log append. Registered in the CI fuzz
job.
Name ErrInvalidPath and the junction check in the ResolveInRoot contract,
list the fuzz targets and tests as shipped, and correct the linkDir test
comment: EvalSymlinks does not follow a junction.
With core.autocrlf=true (the Windows default) every text file checks out
as CRLF, so gofmt -l listed all 218 Go files, the command-reference drift
test failed on docs/commands/README.md, and index check saw every index as
stale. Pin eol=lf for all text files so a clone is LF regardless of local
config. No stored content changes: git add --renormalize is a no-op.
…d honour the umask

- On Windows an O_EXCL create under a directory junction fails with "file
  exists" (os.CreateTemp included), so every write into an in-bundle
  junctioned directory that ResolveInRoot accepts was refused. Open the temp
  without O_EXCL there, behind an Lstat check on a 64-bit random name.
- Create the temp with perm instead of os.CreateTemp's forced 0600 plus a
  chmod, so a new file goes through the umask as it did with os.WriteFile.
  An existing file still keeps its own mode.
- Retry a Windows rename briefly on access-denied, when another process
  holds the target open.
- Sync the parent directory after the rename so the replace survives a
  crash (best effort; not possible on Windows).
- Document that a hard-linked target is replaced rather than written
  through.
…efore any write

- §3: ApplyMove now confines every node it rewrites, not only the two ends
  of the move, so a node reached through a directory link out of the bundle
  is never written through.
- PromoteApply checks on disk that each destination is free before step 1
  rewrites inbound links. The check used to run after the rewrites, so a
  refused promotion left links pointing at a concept that was never
  written.
- §3.1: IsReservedPath cleans the path with the native separator first, so
  `node edit sub\index.md` on Windows is refused like sub/index.md.
fsutil.Tx stages writes and renames, checks every target against the
SHA-256 the caller planned against before writing anything, applies
the steps in order, and undoes the applied steps newest first when one
fails. Each write stays atomic through WriteFileAtomic; directories a
rename created are removed on rollback, pre-existing ones never are.

The undo runs in-process from in-memory copies, so it covers a failed
step, not a crash part-way through a commit.
…ate --apply

node mv rewrote inbound links one file at a time and then renamed the
node: a failure on the directory step or the rename left every rewrite
in place, pointing at a node that never moved, and a node edited after
the load was overwritten without notice. migrate --apply wrote each
node as it went, so a failure on a late node left the bundle
half-migrated, and a plan was applied to files edited since phase 1.

Load now records each node's SHA-256. ApplyMove checks every file it
touches against it and commits the rewrites and the move as one
fsutil.Tx. The migrate plan records a sha256 per node; --apply and its
--dry-run refuse a node that no longer matches, and the apply renders
every node and both version markers before committing them as one
transaction. A plan without sha256 (written before this change) still
applies, with a warning on stderr.
… cutset, apply De Morgan in the tag fold

golangci-lint (errorlint, staticcheck SA1024 and QF1001) flagged four
findings already on next and on #5, which failed the lint job here.
The cutset is "/" on Unix and "/\\" on Windows, as before.
govulncheck failed build-test on GO-2026-6809 (crypto/x509 distrusts
compromised intermediate certificates), fixed in 1.26.10 and reported
reachable from fsutil.createTemp's rand.Read. Bump the go.mod directive
and every actions/setup-go pin, as cwest#190 did. go mod tidy leaves go.sum
unchanged; govulncheck on 1.26.10 reports no vulnerabilities.
FuzzTouchModifiedRoundTrip found that TouchModifiedFile stamped
'modified' into a block with a duplicate key: decoding into a yaml.Node
accepts it, so the file was rewritten and still did not parse. Check the
block with ParseFrontmatter, the parser validate uses, and refuse it
unchanged. The crasher is committed as a regression seed.
FuzzNewNodeFrontmatter found that a title of tab, newline, '0' was
written as a yaml.v3 literal block whose first line starts with a tab,
which does not parse back. Write any value containing a control
character double-quoted, which escapes it and round-trips; every other
value keeps the plain style. The crasher is committed as a regression
seed.
# Conflicts:
#	CHANGELOG.md
#	cmd/lint.go
#	docs/commands/README.md
#	internal/fsutil/confine.go
#	internal/okf/analyze.go
#	internal/okf/analyze_clusters_test.go
#	internal/okf/frontmatter_edit.go
#	internal/okf/index_eol_test.go
#	internal/okf/lint.go
#	internal/okf/lint_taghygiene_test.go
#	internal/okf/migrate.go
#	internal/okf/mutate.go
#	internal/okf/neighborhoods_test.go
#	internal/okf/reserved_lifecycle.go
#	internal/okf/writepath_fuzz_test.go
The merge with next kept internal/okf/tagfold.go alive because this
branch carried a one-line lint edit to it, but next had moved its code
elsewhere, so every symbol in it was unused and failed golangci-lint.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants