Skip to content

✨ feat(index): link subdirectories to their index.md, not the bare directory - #11

Merged
LGuichet merged 23 commits into
nextfrom
feat/index-subdir-index-links
Oct 30, 2026
Merged

LGuichet merged 23 commits into
nextfrom
feat/index-subdir-index-links

Conversation

@LGuichet

Copy link
Copy Markdown
Owner

Summary

okfctl index build now links each subdirectory entry to the child's own index.md (* [Wine](wine/index.md)) instead of the bare directory (* [Wine](wine/)). That way readers that can only open files (an agent's read tool, a plain Markdown viewer) can follow the link.

Index output (OKF §8)

  • Subdirectory links point at a file: the target is child/index.md. §8 leaves an entry's relative URL open (its subdir/ is an example, not a rule, and §6.1 only asks for a standard Markdown path), so both forms conform. The new one matches the reference producer, reference_agent/bundle/index.py, line 98.
  • Old indexes migrate cleanly: the on-disk parser accepts topics/index.md and the legacy topics/, and treats both as the same subdirectory. A curator description written on an old topics/ entry carries over to the new link on the first rebuild.

Index rendering (internal/okf/)

  • reserved_lifecycle.go: renders child/index.md, and subdirBulletRE accepts both spellings ((?:index\.md)?).
  • Tests: subdir_description_test.go adds TestSubdirDescription_LegacyTrailingSlashLinkMigrates (positive control: a legacy topics/ description survives as topics/index.md) and TestSubdirBulletRE_OnlyDirectoryLinks (negative control: nouns.md, topics/nouns.md, index.md and topics/index.md.bak are not taken for subdirectory entries). The existing fixtures move to the new link form.
  • Tests that pin the new output: index_eol_test.go, index_shape_render_test.go, neighborhoods_test.go, nested_index_test.go, reserved_lifecycle_test.go

CLI tests (cmd/)

  • index_nested_test.go, index_shape_test.go: pin the new link form through okfctl index build.

Evidence

  • Before: index build writes * [Api](api/) - … (8 concepts · …).
    After: it writes * [Api](api/index.md) - … (8 concepts · …).

Spec-conformance suite (go test ./internal/okf/ ./cmd/ -run Conformance):

ok  	github.com/cwest/okfctl/internal/okf	5.144s
ok  	github.com/cwest/okfctl/cmd	1.916s

gofmt -l . is empty and go vet ./... is clean. All other packages pass (okfctl-api, okfctl-search, agentplugin, apiserver, fsutil, lexical, search). In internal/okf and cmd, every index test passes. The 13 tests that fail locally fail identically on next, because of this Windows host: plugins not found on PATH, shell-script editors, true missing, and CRLF in git clones. -race was not run locally (no C compiler on the host), so CI covers it.

Real corpus: two production bundles, each copied and run with the next binary (before) and this branch (after). "Subdir links" = Subdirectories entries in every index.md after index build.

Bundle validate rc broken-link lint --strict output Subdir links dir/ → dir/index.md Descriptions kept
plateforme-agree 0 → 0 0 → 0 3 lines, identical 10/0 → 0/10 10 → 10
nf525-lne 0 → 0 2 → 2 4 lines, identical 7/0 → 0/7 7 → 7

The two broken-link findings on nf525-lne predate this PR; that count not moving is the control.

Merge Danger

Door: two-way

Blast Radius: index files

  • An existing bundle fails okfctl index check once (rc 0 → 1) until index build is re-run; after that, rc 0. The rebuild rewrites every Subdirectories link and keeps curator descriptions.
  • validate and lint are unaffected: lint's broken-link check scans concept nodes only, never index files, and node promote already resolved both dir/ and dir/index.md (promote.go).

Not in this PR: no --link-style flag to keep emitting dir/, and the dated design records in docs/specs/ still describe the child/ form they decided at the time.

🤖 Generated with Claude Code

LGuichet and others added 23 commits October 12, 2026 02:03
…ally

Every bundle writer in internal/okf (node new, TouchModifiedFile, node mv
and node promote link rewrites, migrate apply, Scaffold, index build, log
append) used os.WriteFile, which truncates the target in place: a crash or
a full disk mid-write left a half-written node behind. They now go through
fsutil.WriteFileAtomic (temp file in the same directory, sync, rename).

Each rewrite path is pinned by a hard-link test that fails on an in-place
write; a v0.1 node (legacy timestamp, body # Citations) still migrates and
touches as before.
…tomically

okfconfig.Save, search.(*Store).Save and the migrate --plan writer used
os.WriteFile, which truncates in place; a crash or a concurrent reader
could observe a partial file. Route them through fsutil.WriteFileAtomic
(temp file plus rename). Each swap is pinned by a hard-link test that
fails against the in-place writer.
OKF §3 defines a bundle as its directory tree. node new checked only the
lexical path, so a symlinked or junctioned directory inside the bundle
could carry the write outside it; node edit did no check at all and
launched the editor on ../outside.md. Both now resolve the user path
through fsutil.ResolveInRoot before touching disk. A nested a/b/c path
and a title carrying --- in prose still write as before.
ApplyMove joined the destination onto the root with no containment check,
so node mv a ../x.md renamed the node out of the bundle (OKF §3) after
already rewriting its inbound links. Both ends now go through
fsutil.ResolveInRoot before any write; a control character in the
destination is refused too, since the path is echoed into link text.
A nested in-bundle destination still moves.
…red core package

The tokenizer, stopword set and light stemmer move verbatim from internal/search to a stdlib-only internal/lexical package so core search can rank with the same chain without depending on the plugin. internal/search keeps its names as thin forwarders; --lexical-gate output and tests are unchanged.
okfctl search keeps its substring match set (PRD §6.3) and now orders it by BM25F (k1=1.2, b=0.75, per-token DF, title > tags > type > body), ties broken by path. --sort path reproduces the previous output byte for byte; --json gains a score field. Ranking is consumer-side (OKF §1); reserved files stay out (§3.1) and the open type vocabulary is ranked, never gated (§4.1). okf.RankBM25 and internal/lexical expose the ranking for hybrid fusion.
…one segment

PromotePlan accepted any --name, so ../x, a/b or index wrote the promoted
concept outside its directory, outside the bundle (OKF §3), or over a §3.1
reserved file; a line break in it landed raw in every rewritten link.
--name must now be one ordinary, non-reserved file-name segment.
PromoteApply confines every path a plan names (old index, new concept,
rewrite targets) through fsutil.ResolveInRoot before any write. The
default name, an ordinary name and one with --- in it are still accepted.
The plan file is user-editable between phases. MigrateApply joined each
node path onto the root unchecked, so a plan naming ../victim.md rewrote
a file outside the bundle (OKF §3), and wrote target_version verbatim into
the line-keyed .okf sidecar, so a line break forged a second key (§12).
Every plan node path now goes through fsutil.ResolveInRoot and a target
carrying a control character is refused, both before any write.
AppendLog flattened a message to its first line by cutting at LF only.
CommonMark also ends a line at a bare CR, so a message like
"a<CR>## 1999-01-01" wrote a forged date heading into the §9 log. Cut at
the first CR or LF.
…aths

Four fuzz targets assert the write-safety properties on untrusted input:
no injected frontmatter key from node new, no lost key or body from a
modified refresh, a key-stable root index.md okf_version rewrite, and
exactly one entry and heading per log append. Registered in the CI fuzz
job.
Name ErrInvalidPath and the junction check in the ResolveInRoot contract,
list the fuzz targets and tests as shipped, and correct the linkDir test
comment: EvalSymlinks does not follow a junction.
…d honour the umask

- On Windows an O_EXCL create under a directory junction fails with "file
  exists" (os.CreateTemp included), so every write into an in-bundle
  junctioned directory that ResolveInRoot accepts was refused. Open the temp
  without O_EXCL there, behind an Lstat check on a 64-bit random name.
- Create the temp with perm instead of os.CreateTemp's forced 0600 plus a
  chmod, so a new file goes through the umask as it did with os.WriteFile.
  An existing file still keeps its own mode.
- Retry a Windows rename briefly on access-denied, when another process
  holds the target open.
- Sync the parent directory after the rename so the replace survives a
  crash (best effort; not possible on Windows).
- Document that a hard-linked target is replaced rather than written
  through.
…efore any write

- §3: ApplyMove now confines every node it rewrites, not only the two ends
  of the move, so a node reached through a directory link out of the bundle
  is never written through.
- PromoteApply checks on disk that each destination is free before step 1
  rewrites inbound links. The check used to run after the rewrites, so a
  refused promotion left links pointing at a concept that was never
  written.
- §3.1: IsReservedPath cleans the path with the native separator first, so
  `node edit sub\index.md` on Windows is refused like sub/index.md.
…y platform

- Go may fuse a multiply-add (arm64, ppc64, s390x), so raw float64 scores
  differed in their low bits by platform. --json was no longer CI-diffable,
  and near-equal hits could swap order. Round every score to 6 decimals where
  it is produced (lexical.Index.Scores), so both the output and the
  score-then-path order are the same everywhere.
- Reword the search help and guide. BM25F weights a title hit over tags,
  type and body, but repeated body mentions still add up, so it does not
  guarantee a strict title > body order.
…rectory

`index build` now writes a Subdirectories entry as `* [Wine](wine/index.md)`
instead of `* [Wine](wine/)`. OKF §8 leaves the relative URL of an index
entry open (its `subdir/` is an example, not a rule; §6.1 only asks for a
standard Markdown path), and a file target resolves for readers that can only
open files: an agent's read tool, a plain Markdown viewer. It also matches
the reference producer (reference_agent/bundle/index.py).

The on-disk parser accepts both spellings and keys them on the same
directory, so a curator description written against a legacy `wine/` bullet
carries over on the first rebuild. Existing bundles report stale once under
`index check` until rebuilt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@LGuichet
LGuichet merged commit 5d27721 into next Oct 30, 2026
5 of 8 checks passed
LGuichet added a commit that referenced this pull request Oct 30, 2026
… fix

The index guide and the changelog describe the link form index build writes since the merge of #11, how an older index migrates, and the empty-null-key fix in this PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant