Repository navigation
Conversation
lint warned that Concept and Concepts are one type spelled twice but had no opinion about Wine/wine or runbook/runbooks/run-book — even though tags is the other §4.1 field every consumer routes and filters on. The --tag scope filter and the /api/v1/search tag parameter match exactly and case-sensitively, so one tag spelled three ways is three disjoint result sets, and analyze's cluster detector counted the variants separately and missed the cluster. Extract the shared base fold (case, trim, single trailing 's') as canonFold; canonType is now a thin wrapper over it, so type-hygiene is byte-identical. canonTag layers separator-insensitivity (-, _, space) on top — for tags only. Add lintTagHygiene as a sibling of lintTypeHygiene: bundle-level, advisory, "tag-hygiene" in --json, --strict treats it like any finding; the message lists each variant with its per-node count, variants sorted. analyze adopts canonTag for cluster detection so it and lint agree on when two spellings are one tag, and it now dedups the node set per cluster (the old per-occurrence append double-counted a tag repeated within one node's list. The check never rejects a value (PRD §7.4 anti-taxonomy): it flags only spellings that fold to one value, never two genuinely distinct ones. Closes cwest#166 EOF )
…yze labels The bare trailing-'s' fold merged distinct short tags (new/news, op/ops, aws/aw, js/j, k8s/k8) and missed real variants (-es and -ies plurals, ci/cd, node.js). Fold tags instead through tagSurface (case, trim, and -, _, space, / separators, plus a full stop between letters per UAX cwest#29) and tagKeys, which applies KStem's plural rules (Lucene KStemmer.java, derived from the UMass CIIR KStemmer): letters-only, more than two letters, a plain 's' dropped only after more than three letters and not after 'ss' or 'ous', and the Snowball English invariant forms (news, atlas, ...). KStem's dictionary lookup is stood in for by the bundle's own tag vocabulary: a spelling folds onto a candidate only when that candidate is itself a tag, so non-word stems never become keys. The partition depends only on the tag set, never its order. The type path (canonType) is unchanged. analyze labels clusters with the lower-cased variant and orders them by label, so case-only and separator-only clusters read exactly as before the fold swap (main labelled Wine/wine as "wine"; the raw variant made it "Wine"). lint's long help now describes the fold as a heuristic instead of claiming it never flags distinct tags. Tests: the negative controls are pairs that folded under the old rule (each fails against it), positive controls cover the new merges, the tag-hygiene message and cluster labels are pinned exactly, and the digraph control that could not fail is replaced.
On Windows with core.autocrlf=true a fresh clone checks index.md out as CRLF, and `index check` reported every index stale although the content was identical, because it compared bytes with the LF render. - index check compares with CRLF folded to LF; any other difference is still drift. - index build keeps the line endings of the index.md it replaces: an all-CRLF file stays CRLF, a new or mixed-ending file is written LF. The choice depends only on that file, so it is deterministic and git sees no whole-file ending flip. - the subdirectory-description parser strips CR before matching, so a CRLF index no longer captures the old shape suffix into the curator description (which made check fail and build double the suffix). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d the §8 index suffix Lift the per-directory aggregation that `index build` renders as the subdirectory shape suffix (OKF §8 progressive disclosure) out of dirShape into a structured DirShape: own and subtree concept counts, the sorted distinct `type` values (§7.4 open vocabulary, kept as written), and the §4.1 shared tags with a truncation flag. Shape computes it for one directory and Shapes for every IndexDirs entry. dirShape now renders FROM the struct, so the index suffix and any other consumer (the PRD §6.1 "neighborhoods" view of `bundle info`) cannot drift apart. No behavior change to the rendered suffix. Tests cover own-vs-subtree counting, own-concept-only types and tags, tag-cap truncation, byte-equality between Shape and the rendered suffix, a build→validate round trip, and Shapes covering every index dir.
…ex.md OKF §12 defines where a bundle declares the version it targets: an `okf_version` key in the bundle-root index.md frontmatter block. Load read the version only from the .okf sidecar, which is okfctl's own pin and not part of the spec, so a spec-conforming bundle carrying the marker and no sidecar was reported as SpecVersion whatever it declared. index build already resolved the marker first (bundleRootOkfVersion); load and build disagreed about the same bundle. Resolve Bundle.OkfVersion in the order index build writes the marker: the root-index marker, else the sidecar, else SpecVersion. Only the root index counts (§12 permits the marker nowhere else). The declared string is reported as declared: per §12 an unrecognised version is consumed best-effort, never rejected. Tests pin both directions: marker-only, marker-over-sidecar, bare scalar and unrecognised values are read (positive); a nested index.md marker, a blank marker and a sidecar-only bundle leave the result unchanged (negative). The conformance suite gains the consumer half of the §12 round trip: the marker index build preserves is the one Load reads back.
…ally Every bundle writer in internal/okf (node new, TouchModifiedFile, node mv and node promote link rewrites, migrate apply, Scaffold, index build, log append) used os.WriteFile, which truncates the target in place: a crash or a full disk mid-write left a half-written node behind. They now go through fsutil.WriteFileAtomic (temp file in the same directory, sync, rename). Each rewrite path is pinned by a hard-link test that fails on an in-place write; a v0.1 node (legacy timestamp, body # Citations) still migrates and touches as before.
…tomically okfconfig.Save, search.(*Store).Save and the migrate --plan writer used os.WriteFile, which truncates in place; a crash or a concurrent reader could observe a partial file. Route them through fsutil.WriteFileAtomic (temp file plus rename). Each swap is pinned by a hard-link test that fails against the in-place writer.
OKF §3 defines a bundle as its directory tree. node new checked only the lexical path, so a symlinked or junctioned directory inside the bundle could carry the write outside it; node edit did no check at all and launched the editor on ../outside.md. Both now resolve the user path through fsutil.ResolveInRoot before touching disk. A nested a/b/c path and a title carrying --- in prose still write as before.
ApplyMove joined the destination onto the root with no containment check, so node mv a ../x.md renamed the node out of the bundle (OKF §3) after already rewriting its inbound links. Both ends now go through fsutil.ResolveInRoot before any write; a control character in the destination is refused too, since the path is echoed into link text. A nested in-bundle destination still moves.
…one segment PromotePlan accepted any --name, so ../x, a/b or index wrote the promoted concept outside its directory, outside the bundle (OKF §3), or over a §3.1 reserved file; a line break in it landed raw in every rewritten link. --name must now be one ordinary, non-reserved file-name segment. PromoteApply confines every path a plan names (old index, new concept, rewrite targets) through fsutil.ResolveInRoot before any write. The default name, an ordinary name and one with --- in it are still accepted.
The plan file is user-editable between phases. MigrateApply joined each node path onto the root unchecked, so a plan naming ../victim.md rewrote a file outside the bundle (OKF §3), and wrote target_version verbatim into the line-keyed .okf sidecar, so a line break forged a second key (§12). Every plan node path now goes through fsutil.ResolveInRoot and a target carrying a control character is refused, both before any write.
AppendLog flattened a message to its first line by cutting at LF only. CommonMark also ends a line at a bare CR, so a message like "a<CR>## 1999-01-01" wrote a forged date heading into the §9 log. Cut at the first CR or LF.
…aths Four fuzz targets assert the write-safety properties on untrusted input: no injected frontmatter key from node new, no lost key or body from a modified refresh, a key-stable root index.md okf_version rewrite, and exactly one entry and heading per log append. Registered in the CI fuzz job.
Name ErrInvalidPath and the junction check in the ResolveInRoot contract, list the fuzz targets and tests as shipped, and correct the linkDir test comment: EvalSymlinks does not follow a junction.
With core.autocrlf=true (the Windows default) every text file checks out as CRLF, so gofmt -l listed all 218 Go files, the command-reference drift test failed on docs/commands/README.md, and index check saw every index as stale. Pin eol=lf for all text files so a clone is LF regardless of local config. No stored content changes: git add --renormalize is a no-op.
…d honour the umask - On Windows an O_EXCL create under a directory junction fails with "file exists" (os.CreateTemp included), so every write into an in-bundle junctioned directory that ResolveInRoot accepts was refused. Open the temp without O_EXCL there, behind an Lstat check on a 64-bit random name. - Create the temp with perm instead of os.CreateTemp's forced 0600 plus a chmod, so a new file goes through the umask as it did with os.WriteFile. An existing file still keeps its own mode. - Retry a Windows rename briefly on access-denied, when another process holds the target open. - Sync the parent directory after the rename so the replace survives a crash (best effort; not possible on Windows). - Document that a hard-linked target is replaced rather than written through.
…efore any write - §3: ApplyMove now confines every node it rewrites, not only the two ends of the move, so a node reached through a directory link out of the bundle is never written through. - PromoteApply checks on disk that each destination is free before step 1 rewrites inbound links. The check used to run after the rewrites, so a refused promotion left links pointing at a concept that was never written. - §3.1: IsReservedPath cleans the path with the native separator first, so `node edit sub\index.md` on Windows is refused like sub/index.md.
fsutil.Tx stages writes and renames, checks every target against the SHA-256 the caller planned against before writing anything, applies the steps in order, and undoes the applied steps newest first when one fails. Each write stays atomic through WriteFileAtomic; directories a rename created are removed on rollback, pre-existing ones never are. The undo runs in-process from in-memory copies, so it covers a failed step, not a crash part-way through a commit.
…ate --apply node mv rewrote inbound links one file at a time and then renamed the node: a failure on the directory step or the rename left every rewrite in place, pointing at a node that never moved, and a node edited after the load was overwritten without notice. migrate --apply wrote each node as it went, so a failure on a late node left the bundle half-migrated, and a plan was applied to files edited since phase 1. Load now records each node's SHA-256. ApplyMove checks every file it touches against it and commits the rewrites and the move as one fsutil.Tx. The migrate plan records a sha256 per node; --apply and its --dry-run refuse a node that no longer matches, and the apply renders every node and both version markers before committing them as one transaction. A plan without sha256 (written before this change) still applies, with a warning on stderr.
… cutset, apply De Morgan in the tag fold golangci-lint (errorlint, staticcheck SA1024 and QF1001) flagged four findings already on next and on #5, which failed the lint job here. The cutset is "/" on Unix and "/\\" on Windows, as before.
govulncheck failed build-test on GO-2026-6809 (crypto/x509 distrusts compromised intermediate certificates), fixed in 1.26.10 and reported reachable from fsutil.createTemp's rand.Read. Bump the go.mod directive and every actions/setup-go pin, as cwest#190 did. go mod tidy leaves go.sum unchanged; govulncheck on 1.26.10 reports no vulnerabilities.
FuzzTouchModifiedRoundTrip found that TouchModifiedFile stamped 'modified' into a block with a duplicate key: decoding into a yaml.Node accepts it, so the file was rewritten and still did not parse. Check the block with ParseFrontmatter, the parser validate uses, and refuse it unchanged. The crasher is committed as a regression seed.
FuzzNewNodeFrontmatter found that a title of tab, newline, '0' was written as a yaml.v3 literal block whose first line starts with a tab, which does not parse back. Write any value containing a control character double-quoted, which escapes it and round-trips; every other value keeps the plain style. The crasher is committed as a regression seed.
# Conflicts: # CHANGELOG.md # cmd/lint.go # docs/commands/README.md # internal/fsutil/confine.go # internal/okf/analyze.go # internal/okf/analyze_clusters_test.go # internal/okf/frontmatter_edit.go # internal/okf/index_eol_test.go # internal/okf/lint.go # internal/okf/lint_taghygiene_test.go # internal/okf/migrate.go # internal/okf/mutate.go # internal/okf/neighborhoods_test.go # internal/okf/reserved_lifecycle.go # internal/okf/writepath_fuzz_test.go
The merge with next kept internal/okf/tagfold.go alive because this branch carried a one-line lint edit to it, but next had moved its code elsewhere, so every symbol in it was unused and failed golangci-lint.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(okf): hash-checked, rollback-safe
node mvandmigrate --applyThe branch is up to date with
next(PR #5 merged there, so the diff is only this PR). Commits that are this PR:The branch also carried CI fixes for problems that came from #5 (lint, the Go 1.26.10 bump, the touch duplicate-key fuzz crasher).
nexthas since landed its own versions of those (#12,cecb642,53524f6). In the merge,next's versions won wherever the two overlapped.The bug
PR #5 made each file write atomic. Neither command was atomic across several files, and neither noticed files changing after it read them:
node mvrewrote the linking nodes one by one, then created the destination directory and renamed the node. If the directory step or the rename failed, the rewrites stayed: links pointed at a node that never moved. A node edited after the bundle was loaded was overwritten without notice.migrate --applywrote each node as it went. A failure on a late node left the bundle half-migrated. The plan was applied to whatever the files held at apply time, even after they were edited after phase 1.The fix
fsutil.Tx(new, standard library only). It stages writes and renames, then:A rollback that cannot finish is reported (
rollback incomplete: …), never silent.Loadrecords each node's SHA-256 (Node.SHA256).ApplyMovechecks the moved node and every rewritten node against those hashes, then commits the rewrites and the move as oneTx.migrate:sha256per plan node.--applyrenders every node and both version markers (.okf, rootindex.md) first, then commits them as oneTx. A node that no longer matches refuses the whole plan before any write.--dry-runruns the same checks, so the preview fails exactly where the apply would.No
validateorlintrule changes. Nothing about what a bundle may contain changes.Decision for review: plans without
sha256sha256isomitempty. A plan written before this change still applies, unchecked, with a warning on stderr:The alternative was refusing such plans, which would break any saved plan. The reverse is a one-line change if you prefer it.
Limits
node promotehas its own writer and is not changed here.Spec
This is write safety, not OKF-defined behavior: no spec section governs atomicity or drift. The parts that touch spec-defined content are unchanged:
renderMigratedNode.The v0.1 fallbacks are untouched.
Gate
1. Conformance
2. Full suite
-race: it ran in CI, because this machine has no cgo toolchain. See CI below.3. Real corpus:
market-fr/knowledge-base,nextbinary vs branch binaryRead-only. The output files are byte-identical before and after:
--strictbefore → afterifpos-interface-casesmiddlewarenf525-lneplateforme-agreeNothing moved: the change touches no read path.
Write smoke on
cp -rscratch copies ofnf525-lne. The node ismapping/contract-mapping.md, which 37 files link to:v0.1 fixture, migrate through the CLI. Three nodes with legacy
timestampand body# Citations.c.mdis edited after phase 1:CI
-race:go test ./... -racepassed in every package in CI (build-test). It cannot run locally: this Windows machine has no cgo toolchain.lint,govulncheckand a touch fuzz crasher all came from 🐛 fix(write): write-safety audit—atomic, confined, injection-free writes #5, andnexthas fixed them since.fuzz, after the merge withnext:FuzzNewNodeFrontmatterfound thatnode new --titlewith a tab, a newline, then0wrote a yaml.v3 literal block whose tab-led first line does not parse back.a238165writes any value containing a control character double-quoted, which escapes it and round-trips. Every other value keeps the plain style, so normal nodes are written byte-for-byte as before; the existingnode newand conformance tests are unchanged.testdata/fuzz/FuzzNewNodeFrontmatter/tab-led-multiline-title. It fails before the fix and passes after. A local 45-second fuzz run found nothing new.next(5aa4404):mutate.gowas unchanged onnext, so this PR's version stands.migrate.go,next's control-character quoting of theokf_versiontarget (53524f6) is re-applied inside the renamedstageRootIndexMarker.writepath_fuzz_test.gokeepsnext's new seeds and this PR's transaction-driven marker commit.nexttoo (plugin stubs,$EDITORscripts, git CRLF).Controls
Failed first. Base-compatible copies of the positive controls, run against unchanged
next(efc58b3):On the branch, the same scenarios pass as:
fsutil.TxRefusesDriftBeforeAnyWrite,RollsBackWritesWhenRenameFails,RollsBackWritesWhenMkdirFails,RollbackRemovesCreatedFile,RefusesExistingRenameDestinationCommitsWhenNothingChanged,RollbackKeepsPreexistingDirnode mvRefusesLinkerEditedSinceLoad,RefusesMovedNodeEditedSinceLoad,FailedDirectoryLeavesTreeUnchangedUnchangedBundleStillMoves,InMemoryNodeSkipsHashCheck, every existingApplyMove/PlanMovetestmigrate --applyRefusesNodeEditedSincePlan(apply andCheckMigratePlan),LateFailureWritesNothing, CLIApplyRefusesNodeEditedSincePlan(apply and--dry-run)RecordsNodeHashes,UnchangedBundleMigrates,LegacyPlanWithoutHashesApplies, CLIApplyLegacyPlanWarnsAndApplies,ApplyHashedPlanNoWarning, every existing migrate testFuzzRootIndexMarkernow drives the staged marker write throughTx.Commit.On Linux,
FailedDirectoryLeavesTreeUnchangedis refused at the up-front stat (ENOTDIR) before any write. On Windows it reaches the directory step and exercises the rollback. Both end with the same assertion. The rollback path itself is covered on every platform by thefsutiltests, which inject the failure.Notes for review
testdata/fuzz/FuzzTouchModifiedRoundTrip/duplicate-key-blockrepeats a casenextnow seeds inline. It is kept as a file seed and does no harm.