Repository navigation
π fix(index): handle CRLF index.md β normalize on read, preserve on write - #205
Conversation
β¦rite
On Windows with core.autocrlf=true an index.md is checked out CRLF, and
okfctl mishandled it three ways. All three are fixed at the index read/
write seam with a single decided policy: NORMALIZE ON READ, PRESERVE ON
WRITE.
(a) Data damage: subdirBulletRE / toolShapeSuffixRE are $-anchored, so a
trailing \r stopped the tool-owned shape suffix from matching; the
stale suffix was captured into the preserved curator description and
`index build` appended a fresh one, doubling it
(`... (3 concepts Β· reference) (3 concepts Β· reference)`). A bug
regardless of line-ending policy. Fixed by normalizing idx.Body
CRLF->LF before the Β§8 bullet parse in preservedSubdirDescriptions.
(b) False staleness: `index check` compared raw bytes against the LF
render, so every CRLF index reported stale and Windows teams could
not use it as a gate. Fixed by normalizing the on-disk bytes before
the staleness comparison in IndexInSync β any non-line-ending
difference is still drift, so genuine staleness is still caught.
(c) Churn: `index build` rewrote every CRLF index to LF with no content
change. Fixed by detecting the existing file's dominant line ending
and writing the render back in that ending; a new or mixed-ending
file stays LF. An up-to-date CRLF index is now left byte-identical.
Two small helpers (normalizeEOL, usesCRLF) carry the policy. LF/Linux
behavior is unchanged; the choice depends only on the one file, so it is
deterministic on any machine.
Closes #185
cwest
left a comment
There was a problem hiding this comment.
The fix is correct and the controls hold. Normalize-on-read closes both (a) and (b) at the parse/compare seam, and preserve-on-write closes (c) without touching LF behavior. usesCRLF requiring every break to be CRLF is the right call β a mixed or new file stays LF, so a stray CR never flips a file to CRLF on write.
I reverted each seam in isolation and confirmed the matching test goes red: dropping normalize-on-read in preservedSubdirDescriptions reproduces the exact doubled suffix from the issue ((1 concept Β· reference) (1 concept Β· reference)); dropping it in IndexInSync brings back the false staleness; dropping preserve-on-write rewrites CRLF to LF. The three boundary tests (genuine drift still caught, new index LF, LF index unchanged) correctly stay green with the fix reverted, so they're real negative controls, not duplicates of the positive ones.
Ran the issue repro end to end against a before/after binary: CRLF index check goes exit 1 to exit 0, the audit bullet keeps a single suffix, and an up-to-date CRLF index is byte-identical after build. On the 303-node LF corpus, validate and lint --strict output is byte-identical between the two binaries, index build introduces zero CR and is idempotent on rebuild.
One thing to note that's outside this change: LoadLog/log read paths still compare raw bytes, so a CRLF log.md would hit the same class of issue. Out of scope for #185 and not a blocker here β worth a separate card if Windows log handling comes up.
The branch is behind base now that siblings have merged. Rebase onto the current base and force-push with lease; the diff is unchanged so this is a mechanical update, not a re-review.
β¦#207) A CRLF-checked-out log.md (Windows with core.autocrlf=true, or an editor that writes CRLF) hit the same bug class #185 fixed for index.md: AppendLog read the raw bytes and parseLog split on "\n", so every preserved line kept a trailing \r while the newly rendered date group and entry were pure LF. The result was a mangled mixed-ending file on every append (whole-file git churn) and a stray \r captured into a folded legacy bullet's message. Fixed at the single log read/write seam with the same decided policy as #205: NORMALIZE ON READ, PRESERVE ON WRITE, reusing the normalizeEOL / usesCRLF helpers #205 added rather than a second implementation. - Normalize on read: parseLog is fed normalizeEOL(existing), so no \r leaks into the verbatim preamble/entries or defeats the $-anchored legacy-bullet regex. Any non-line-ending difference is still real content, so a genuine legacy fold / newest-first insert still happens. - Preserve on write: an up-to-date CRLF log is written back CRLF (zero churn); a new file, a mixed-ending file, or a lone-CR file stays LF. The choice depends only on the one file, so it is deterministic on any machine and leaves LF/Linux behavior unchanged. Closes #185
On Windows with
core.autocrlf=trueanindex.mdis checked out CRLF, and okfctl mishandled it three ways. All three are fixed at the index read/write seam with one decided policy: normalize on read, preserve on write.What changed
Two small helpers in
internal/okf/reserved_lifecycle.gocarry the policy:normalizeEOL(s)β folds\r\nβ\n(only the pair; a lone\ris left alone).usesCRLF(s)β true only when every line break is CRLF; a new or mixed-ending file is not CRLF.Wired into the three seams:
subdirBulletRE/toolShapeSuffixREare$-anchored, so a trailing\rstopped the tool-owned shape suffix from matching; the stale suffix was captured into the preserved curator description andindex buildappended a fresh one. Fixed by normalizingidx.BodyCRLFβLF before the Β§8 bullet parse inpreservedSubdirDescriptions. This is a bug regardless of line-ending policy.index checkcompared raw bytes to the LF render, so every CRLF index reported stale. Fixed by normalizing the on-disk bytes before the staleness comparison inIndexInSyncβ any non-line-ending difference is still drift, so genuine staleness is still caught.index buildrewrote every CRLF index to LF with no content change. Fixed by detecting the existing file's dominant ending and writing the render back in that ending; a new or mixed-ending file stays LF. An up-to-date CRLF index is now left byte-identical.LF/Linux behavior is unchanged; the choice depends only on the one file, so it is deterministic on any machine.
Issue repro β before/after
Exact steps from the issue (
bundle init, threetype: referencenodes underaudit/, add a curator description, convert everyindex.mdto CRLF, thenindex check+index build):5c7ce66)index check, LF curatedOK: index.md is current, exit 0OK: index.md is current, exit 0index check, CRLFindex.md is out of date, exit 1OK: index.md is current, exit 0index build, CRLF β line endingsindex build, CRLF β curator suffix(3 concepts Β· reference) (3 concepts Β· reference)(3 concepts Β· reference)(once)Before (main) β doubled suffix and LF rewrite:
After (this PR) β fresh, single suffix, no churn:
Controls (both directions, per AGENTS.md)
New
internal/okf/index_eol_test.goβ every test fails without the fix, passes with it:TestUsesCRLF_Tableβ dominant-ending detector (pure LF / pure CRLF / mixed / no-newline / single-line).TestNormalizeEOLβ CRLF folded, LF untouched, lone CR left alone.TestIndexInSync_CRLFIndexIsFreshβ a CRLF index matching the render β in sync.TestIndexInSync_CRLFGenuineDriftStillCaughtβ a CRLF index with a node added after the build still reports out of sync (normalizing endings does not mask real drift).TestWriteIndex_PreservesCRLF/TestWriteIndex_PreservesLFβ up-to-date index byte-identical after rebuild, in its own ending.TestWriteIndex_NewIndexIsLFβ a newly written nested index is LF.TestWriteIndex_CuratorSuffixIdempotentOnCRLFβ curator description round-trips through a CRLF rebuild with the shape suffix exactly once.Conformance gate (all three layers)
Real-corpus control (LF corpus, ~303 nodes)
Ran a
5c7ce66(main-HEAD) binary and this PR's binary on~/src/knowledge-base/bundles/knowledge:validateandlint --strictoutput are byte-identical between the two binaries;index buildon the LF corpus produces output identical to the main-HEAD binary and introduces zero CRLF.v0.1 fallback control
A forced
okf_version: 0.1bundle still builds, checks, and validates clean (exit 0), and a CRLF round-trip of its indexes reports fresh (exit 0) β the legacy path is unaffected.Closes #185