Skip to content

CI check: rebased next on cwest/okfctl main (do not merge) - #14

Closed
LGuichet wants to merge 29 commits into
mainfrom
next
Closed

LGuichet wants to merge 29 commits into
mainfrom
next

Conversation

@LGuichet

@LGuichet LGuichet commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Summary

CI check for the rebased next (do not merge). next is now cwest/okfctl main (5bb1fac) plus the fork's 29 commits replayed as linear history. CI runs only on pushes to main and on pull requests, so this draft gets -race, golangci-lint and fuzz to run on it.

Dropped in the replay (already upstream)

Conflicts resolved (internal/okf/)

  • reserved_lifecycle.go: AppendLog keeps upstream's CRLF handling (🐛 fix(log): handle CRLF log.md — normalize on read, preserve on write cwest/okfctl#207) and writes through the fork's atomic writer. WriteIndex keeps upstream's CRLF preserve-on-write and adds the fork's skip of an index whose content is current.
  • index_eol_test.go: upstream's version, with its subdirectory links moved to the dir/index.md form, plus the fork's TestWriteIndex_UnchangedIndexIsNotRewritten in upstream's helper style.
  • The fork's lint fix drops its tagfold.go hunk, since that file does not exist upstream.

Evidence

  • Before: fork next at ce38a7e (fork history merged with an older upstream).
    After: next at 53524f6. git diff ce38a7e 53524f6 contains only upstream changes since 1fee902 plus the resolutions above.

Local checks on Windows: gofmt -l . is empty, go vet ./... is clean, and the -run Conformance suite passes. The full suite fails only on tests that fail identically on the old next on this host (plugin and editor exec, CRLF git clones). The two TestConnect_* tests now pass, thanks to cwest#204.

Real corpus: copies of three production bundles, old next build against this branch.

Bundle validate lint --strict index check analyze --json
nf525-lne identical (rc 0) identical (rc 1, 4 lines) rc 0 → 0 identical
plateforme-agree identical (rc 0) identical (rc 1, 3 lines) rc 0 → 0 same 44 clusters, different order
ifpos-interface-cases identical (rc 0) identical (rc 0) rc 0 → 0 identical

The cluster order changes because upstream's cwest#167 sorts clusters on the separator-insensitive key, so e-reporting now follows en16931.

Merge Danger

Door: two-way

Blast Radius: none (draft)

  • Not for merging: main is a mirror of upstream main.

🤖 Generated with Claude Code

LGuichet and others added 29 commits October 31, 2026 22:54
…d the §8 index suffix

Lift the per-directory aggregation that `index build` renders as the
subdirectory shape suffix (OKF §8 progressive disclosure) out of dirShape
into a structured DirShape: own and subtree concept counts, the sorted
distinct `type` values (§7.4 open vocabulary, kept as written), and the
§4.1 shared tags with a truncation flag.

Shape computes it for one directory and Shapes for every IndexDirs
entry. dirShape now renders FROM the struct, so the index suffix and any
other consumer (the PRD §6.1 "neighborhoods" view of `bundle info`)
cannot drift apart. No behavior change to the rendered suffix.

Tests cover own-vs-subtree counting, own-concept-only types and tags,
tag-cap truncation, byte-equality between Shape and the rendered suffix,
a build→validate round trip, and Shapes covering every index dir.
…ex.md

OKF §12 defines where a bundle declares the version it targets: an
`okf_version` key in the bundle-root index.md frontmatter block. Load
read the version only from the .okf sidecar, which is okfctl's own pin
and not part of the spec, so a spec-conforming bundle carrying the
marker and no sidecar was reported as SpecVersion whatever it declared.
index build already resolved the marker first (bundleRootOkfVersion);
load and build disagreed about the same bundle.

Resolve Bundle.OkfVersion in the order index build writes the marker:
the root-index marker, else the sidecar, else SpecVersion. Only the
root index counts (§12 permits the marker nowhere else). The declared
string is reported as declared: per §12 an unrecognised version is
consumed best-effort, never rejected.

Tests pin both directions: marker-only, marker-over-sidecar, bare
scalar and unrecognised values are read (positive); a nested index.md
marker, a blank marker and a sidecar-only bundle leave the result
unchanged (negative). The conformance suite gains the consumer half of
the §12 round trip: the marker index build preserves is the one Load
reads back.
…red core package

The tokenizer, stopword set and light stemmer move verbatim from internal/search to a stdlib-only internal/lexical package so core search can rank with the same chain without depending on the plugin. internal/search keeps its names as thin forwarders; --lexical-gate output and tests are unchanged.
okfctl search keeps its substring match set (PRD §6.3) and now orders it by BM25F (k1=1.2, b=0.75, per-token DF, title > tags > type > body), ties broken by path. --sort path reproduces the previous output byte for byte; --json gains a score field. Ranking is consumer-side (OKF §1); reserved files stay out (§3.1) and the open type vocabulary is ranked, never gated (§4.1). okf.RankBM25 and internal/lexical expose the ranking for hybrid fusion.
…ally

Every bundle writer in internal/okf (node new, TouchModifiedFile, node mv
and node promote link rewrites, migrate apply, Scaffold, index build, log
append) used os.WriteFile, which truncates the target in place: a crash or
a full disk mid-write left a half-written node behind. They now go through
fsutil.WriteFileAtomic (temp file in the same directory, sync, rename).

Each rewrite path is pinned by a hard-link test that fails on an in-place
write; a v0.1 node (legacy timestamp, body # Citations) still migrates and
touches as before.
…tomically

okfconfig.Save, search.(*Store).Save and the migrate --plan writer used
os.WriteFile, which truncates in place; a crash or a concurrent reader
could observe a partial file. Route them through fsutil.WriteFileAtomic
(temp file plus rename). Each swap is pinned by a hard-link test that
fails against the in-place writer.
OKF §3 defines a bundle as its directory tree. node new checked only the
lexical path, so a symlinked or junctioned directory inside the bundle
could carry the write outside it; node edit did no check at all and
launched the editor on ../outside.md. Both now resolve the user path
through fsutil.ResolveInRoot before touching disk. A nested a/b/c path
and a title carrying --- in prose still write as before.
ApplyMove joined the destination onto the root with no containment check,
so node mv a ../x.md renamed the node out of the bundle (OKF §3) after
already rewriting its inbound links. Both ends now go through
fsutil.ResolveInRoot before any write; a control character in the
destination is refused too, since the path is echoed into link text.
A nested in-bundle destination still moves.
…one segment

PromotePlan accepted any --name, so ../x, a/b or index wrote the promoted
concept outside its directory, outside the bundle (OKF §3), or over a §3.1
reserved file; a line break in it landed raw in every rewritten link.
--name must now be one ordinary, non-reserved file-name segment.
PromoteApply confines every path a plan names (old index, new concept,
rewrite targets) through fsutil.ResolveInRoot before any write. The
default name, an ordinary name and one with --- in it are still accepted.
The plan file is user-editable between phases. MigrateApply joined each
node path onto the root unchecked, so a plan naming ../victim.md rewrote
a file outside the bundle (OKF §3), and wrote target_version verbatim into
the line-keyed .okf sidecar, so a line break forged a second key (§12).
Every plan node path now goes through fsutil.ResolveInRoot and a target
carrying a control character is refused, both before any write.
AppendLog flattened a message to its first line by cutting at LF only.
CommonMark also ends a line at a bare CR, so a message like
"a<CR>## 1999-01-01" wrote a forged date heading into the §9 log. Cut at
the first CR or LF.
…aths

Four fuzz targets assert the write-safety properties on untrusted input:
no injected frontmatter key from node new, no lost key or body from a
modified refresh, a key-stable root index.md okf_version rewrite, and
exactly one entry and heading per log append. Registered in the CI fuzz
job.
Name ErrInvalidPath and the junction check in the ResolveInRoot contract,
list the fuzz targets and tests as shipped, and correct the linkDir test
comment: EvalSymlinks does not follow a junction.
…d honour the umask

- On Windows an O_EXCL create under a directory junction fails with "file
  exists" (os.CreateTemp included), so every write into an in-bundle
  junctioned directory that ResolveInRoot accepts was refused. Open the temp
  without O_EXCL there, behind an Lstat check on a 64-bit random name.
- Create the temp with perm instead of os.CreateTemp's forced 0600 plus a
  chmod, so a new file goes through the umask as it did with os.WriteFile.
  An existing file still keeps its own mode.
- Retry a Windows rename briefly on access-denied, when another process
  holds the target open.
- Sync the parent directory after the rename so the replace survives a
  crash (best effort; not possible on Windows).
- Document that a hard-linked target is replaced rather than written
  through.
…efore any write

- §3: ApplyMove now confines every node it rewrites, not only the two ends
  of the move, so a node reached through a directory link out of the bundle
  is never written through.
- PromoteApply checks on disk that each destination is free before step 1
  rewrites inbound links. The check used to run after the rewrites, so a
  refused promotion left links pointing at a concept that was never
  written.
- §3.1: IsReservedPath cleans the path with the native separator first, so
  `node edit sub\index.md` on Windows is refused like sub/index.md.
With core.autocrlf=true (the Windows default) every text file checks out
as CRLF, so gofmt -l listed all 218 Go files, the command-reference drift
test failed on docs/commands/README.md, and index check saw every index as
stale. Pin eol=lf for all text files so a clone is LF regardless of local
config. No stored content changes: git add --renormalize is a no-op.
…y platform

- Go may fuse a multiply-add (arm64, ppc64, s390x), so raw float64 scores
  differed in their low bits by platform. --json was no longer CI-diffable,
  and near-equal hits could swap order. Round every score to 6 decimals where
  it is produced (lexical.Index.Scores), so both the output and the
  score-then-path order are the same everywhere.
- Reword the search help and guide. BM25F weights a title hit over tags,
  type and body, but repeated body mentions still add up, so it does not
  guarantee a strict title > body order.
…rectory

`index build` now writes a Subdirectories entry as `* [Wine](wine/index.md)`
instead of `* [Wine](wine/)`. OKF §8 leaves the relative URL of an index
entry open (its `subdir/` is an example, not a rule; §6.1 only asks for a
standard Markdown path), and a file target resolves for readers that can only
open files: an agent's read tool, a plain Markdown viewer. It also matches
the reference producer (reference_agent/bundle/index.py).

The on-disk parser accepts both spellings and keys them on the same
directory, so a curator description written against a legacy `wine/` bullet
carries over on the first rebuild. Existing bundles report stale once under
`index check` until rebuilt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- confine.go: wrap the underlying cause with %w beside ErrEscapesRoot, so
  errors.Is still matches ErrEscapesRoot and now also sees the cause.
- confine.go: trim leading separators with TrimLeftFunc; on Unix the cutset
  `string(filepath.Separator)+"/"` repeated '/' (SA1024).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An explicit empty key (`? `) is a null key, which a map decode drops. The
`modified` refresh round-trips the block through yaml.v3, whose encoder
writes that empty plain scalar back as a quoted empty string, so the touched
node gained a "" key it never had. FuzzTouchModifiedRoundTrip caught it in CI
(`touch added keys: [] -> [ modified]`).

Rewrite such keys as `~` before encoding, at every depth. A key that already
is a quoted empty string is real data and keeps its quotes (negative control).
The failing input joins the fuzz seeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… fix

The index guide and the changelog describe the link form index build writes since the merge of #11, how an older index migrates, and the empty-null-key fix in this PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decoding into a yaml.Node skips map-level checks such as a duplicate key, so the modified refresh rewrote an invalid block (`ype: 0` then `ype:`) instead of refusing it and leaving the file byte-identical. FuzzTouchModifiedRoundTrip caught it in CI on #12. Run the frontmatter parser first and return its error; the input joins the fuzz seeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
index build rewrote every index.md on every run, and an index with mixed line endings (an editor or agent wrote LF lines into a CRLF file) came out LF even when nothing changed. On a core.autocrlf=true checkout git then reports the file modified with an empty diff, since its size no longer matches the CRLF size git recorded. Compare the render with the on-disk file the way index check does (line endings are not content) and skip the write when they match. A changed index still keeps CRLF when it was pure CRLF, and falls back to LF otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…racter

bumpRootIndexMarker left the scalar style to yaml.v3, which writes a target such as a tab then a newline as a literal block whose tab-led line no parser reads back, so migrate --apply could leave an unparseable bundle-root index.md. FuzzRootIndexMarker caught it in CI on #13. Force double-quoted style when the target holds a control character; every other target is written as before. The input joins the fuzz seeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@LGuichet

LGuichet commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

CI check done: all 10 checks pass on the rebased next (53524f6). Closing without merging; main stays a mirror of cwest/okfctl main.

@LGuichet LGuichet closed this Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant