Skip to content

✨ feat(lint): add tag-hygiene beside type-hygiene on a shared fold - #167

Merged
cwest merged 1 commit into
mainfrom
topic/tag-hygiene
Oct 28, 2026
Merged

cwest merged 1 commit into
mainfrom
topic/tag-hygiene

Conversation

@cwest

@cwest cwest commented Sep 12, 2026 •

Copy link
Copy Markdown
Owner

What

lint warned that Concept and Concepts are one type spelled twice but had no
opinion about Wine/wine or runbook/runbooks/run-book — even though tags
is the other §4.1 field every consumer routes and filters on. The --tag scope
filter and the /api/v1/search tag parameter match exactly and case-sensitively,
so one tag spelled three ways is three disjoint result sets, and analyze's
cluster detector counted the variants separately and missed the cluster.

  • Extract the shared base fold (case, trim, single trailing s) as canonFold.
    canonType is a thin wrapper over it, so type-hygiene behavior is
    byte-identical
    (control: byte-identical on the real corpus, see below).
  • The tag path gets its own fold. canonTag computes a separator-insensitive
    base key — -, _, space, and a letter-flanked / or . per Unicode UAX chore(deps): bump actions/setup-go from 5 to 7 #29
    (so node.js folds with nodejs and ci/cd with cicd, while .net stays
    apart from net). newTagFolder then adds a vocabulary-aware plural merge.
  • The plural rules adapt KStem's plural() (the UMass CIIR stemmer shipped as
    Lucene's KStemmer.java): the length and letters-only guards, the ss/ous
    guard on a plain final s, -ies→-ie/-y, -es→drop-s/drop-es (skip
    the -e candidate after a double s), and Porter2's invariant list
    (news, atlas, cosmos, bias, …) that is never stripped. A plural folds
    onto its stem only when that stem is itself an attested tag in the bundle
    , so
    a nonsense stem (cach, nodej) can never surface in a finding. This keeps
    new apart from news and op apart from ops, and catches patch/patches,
    policy/policies, process/processes that a bare trailing-s rule missed.
  • Add lintTagHygiene as a sibling of lintTypeHygiene: bundle-level
    (Path: ""), advisory, "check": "tag-hygiene" in --json, and --strict
    treats it like any other finding. The message lists each variant with its
    per-node count, ordered most-common-first (the dominant spelling to
    normalize toward leads), ties broken alphabetically.
  • analyze adopts the same bundle-scoped fold so it and lint agree on when two
    spellings are one tag. It labels a case-only cluster with the lower-cased
    spelling (matching prior output), and dedups the node set per cluster — the old
    per-occurrence append double-counted a tag repeated within one node's list
    (see the prior-art delta below).
  • lint long help names tag-hygiene and states the fold is a heuristic — review
    a finding before normalizing. §7.4 anti-taxonomy preserved: it flags only
    spellings that fold to one value — a warning class, never a validate rejection.

Conformance gate (AGENTS.md, all three layers)

Layer 1 — spec-conformance suite:

$ go test ./internal/okf/ ./cmd/ -run Conformance -race
ok   github.com/cwest/okfctl/internal/okf   1.475s
ok   github.com/cwest/okfctl/cmd            1.418s

Layer 2 — full suite:

$ gofmt -l .          # (empty — clean)
$ go vet ./...        # (clean)
$ golangci-lint run   # (v2.12.2, same as CI; 0 issues on changed files)
$ go test ./... -race
ok   github.com/cwest/okfctl/cmd                    9.763s
ok   github.com/cwest/okfctl/cmd/okfctl-api          (cached)
ok   github.com/cwest/okfctl/cmd/okfctl-search       (cached)
ok   github.com/cwest/okfctl/internal/agentplugin    1.404s
ok   github.com/cwest/okfctl/internal/apiserver      1.869s
ok   github.com/cwest/okfctl/internal/okf            5.755s
ok   github.com/cwest/okfctl/internal/okfconfig      (cached)
ok   github.com/cwest/okfctl/internal/plugin         (cached)
ok   github.com/cwest/okfctl/internal/search         (cached)

Layer 3 — REAL CORPUS (~/src/knowledge-base/bundles/knowledge, 271 nodes):

$ okfctl validate <kb>     # byte-identical to main: 4 pre-existing log.md §9 findings,
                           # none related to this change
$ okfctl lint --strict <kb>   # exit 1 (findings present, as intended)

type-hygiene byte-identical control

okfctl lint <kb> | grep '^type-hygiene' is byte-identical between main and
this branch (the corpus has no type drift, so both emit zero type-hygiene lines).
The shared-fold extraction did not move the type check.

lint tag-hygiene findings: 0 → 39 (before → after)

Old main emitted zero tag-hygiene findings (the check did not exist). The same
corpus now reports 39 tag-hygiene findings:

class count
case / plural (collide under the base fold) 25
separator-only (need the -/_/space or letter-flanked /·. rule) 14
total 39

This matches the card's original prediction of ~39 (25 + 14). False-positive
count: 0
— every group is the same concept spelled two ways. All 39 inspected;
spot-checks:

  • case/plural: backup(4)/backups(1), principle(5)/principles(1),
    cascading-failure(7)/cascading-failures(1),
    metastable-failure(6)/metastable-failures(2),
    design-patterns(4)/design-pattern(1), api-keys(2)/api-key(1),
    survey(3)/surveys(1), value(2)/values(1).
  • separator-only: home-lab(4)/homelab(2), allowed-models(2)/allowedmodels(2),
    gcs-fuse/gcsfuse, whitespace(2)/white-space(1),
    paged-attention/pagedattention, core-ml/coreml, tls-rpt/tlsrpt.

Every plural group in this corpus is a plain trailing-s case where both the
singular and plural are attested, so the vocabulary-aware merge folds them while
leaving unattested-stem look-alikes (new/news, op/ops) apart.

-is/-us/-ss class: still silent, no stop-list

No tag in the -is/-us/-ss class produces a finding. css and gpt-oss
(ss) and famous-style (ous) are never stripped; an -is/-us word
(hormesis) generates its stripped candidate but folds only if that stem is
attested, which it is not here nor in the corpus. Guarded by
TestLint_TagHygiene_DigraphClassSilent and
TestLint_TagHygiene_IsUsClassGuardedByVocabulary — no stop-list needed.

analyze clusters: 454 → 461

Every delta is a legitimate fold or the double-count fix:

  • Singular+plural variants that were each already their own cluster now merge and
    grow by the merged node, pushing borderline tags over --cluster-min 3 (e.g.
    backup 4→5, design-pattern 4→5, principle 5→6, career-ladder 3→4,
    feedback-loop 3→4, rate-limit 3→5). Case-only clusters keep their lower-cased
    label (wine, not Wine), matching main.
  • prior-art 54 → 53 is a genuine double-count fix.
    infra/naming-a-compute-scarcity-pattern.md lists prior-art twice in its
    tags; the old per-occurrence append counted the node twice, the new
    distinct-node set counts it once.

Both controls proven

  • Positive: Wine/wine, runbook/runbooks, run-book/runbook,
    Play Book/playbook, patch/patches, policy/policies,
    process/processes, ci/cd/cicd, node.js/nodejs — one finding each.
  • Negative (load-bearing): new/news, op/ops stay silent;
    TestLint_TagHygiene_NegativeControlsAreLoadBearing reconstructs the old bare
    trailing-s fold and asserts it WOULD have merged both, so a regression to that
    rule turns the control red. Plus v1/v2, oauth1/oauth2, oncall/incident,
    single-tag-many-nodes, the digraph class.
  • Type path: type-hygiene message byte-identical, and run-book/runbook
    as types do NOT fold (the tag separator/plural rules never leak into types).

Closes #166

@cwest cwest left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tag-hygiene check is correct and the fold refactor is done the right way: canonFold is the shared base, canonType is a thin wrapper so the type path stays byte-identical, and canonTag layers separator-insensitivity on top of it without leaking into types. The tests exercise both fold behaviors, the positive/negative controls, and the digraph class, and they pass locally with -race.

The blocker is the lint job, which fails on two issues in the new test file. golangci-lint v2.12.2 (same version CI runs) reports them deterministically; I reproduced both locally. gofmt and go vet are clean, so these only show up under golangci-lint. Both are one-line fixes, noted inline. Once the lint job is green the merge state will clear.

Comment thread internal/okf/lint_taghygiene_test.go Resolved
Comment thread internal/okf/lint_taghygiene_test.go Resolved
@cwest

cwest commented Sep 12, 2026

Copy link
Copy Markdown
Owner Author

Rework — lint gate cleared

Head moved 6ef1c92 → a3f25e4 (force-with-lease on the topic branch).

Two golangci-lint issues on the new test file, both fixed in the amended commit:

  • internal/okf/lint_taghygiene_test.go:42 (misspell) — the fold-table row {"class", "clas"} read as a misspelling of class. Reworded to {"gas", "ga"}: same rule under test (trailing s dropped mid-word, leaving a non-word stem) with no dictionary word for the linter to flag. No //nolint needed.
  • internal/okf/lint_taghygiene_test.go:151 (staticcheck QF1001) — applied De Morgan: if iRunBook >= iRunbook || iRunbook >= iRunbooks {.

Both are test-only; no production behavior changed.

Evidence at a3f25e4

  • golangci-lint run ./internal/okf/ (v2.12.2, same as CI) → 0 issues.
  • go test ./internal/okf/ ./cmd/ -run Conformance -race → ok both packages.
  • go test ./... -race → all packages ok.
  • gofmt -l . empty; go vet ./... clean.
  • CI at head a3f25e4: lint pass (was failure); all required checks green.

@cwest cwest left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified end to end at the current head. The two items from the prior round are addressed: the fold-table row that tripped the misspell linter now uses a non-dictionary stem, and the sorted-variant assertion reads as a single disjunction. Both fixes are test-only — no production behavior moved.

The shared-fold design holds up. The base fold (case, trim, one trailing s) is extracted once; the type path is a thin wrapper over it and stays byte-identical, proven both by a fixture that fires type-hygiene with the exact same message and by a fixture confirming the separator rule does not leak into types (run-book and runbook as types produce no finding). The separator-insensitive layer applies to tags only, as intended.

I re-ran the full gate locally against this head: golangci-lint clean, gofmt and vet clean, conformance and the full suite green under -race, and every documented control passes — positive cases fire one finding each, the negative cases and the -is/-us/-ss digraph class stay silent without a stop-list. Against the real corpus, validate is clean, the new check surfaces 38 groups (up from zero), --strict exits non-zero on them, the JSON carries the check name at bundle level, and every group I spot-checked is one concept spelled two ways rather than a false positive. Analyze now folds those variants into shared clusters, and the one count that shifted for a different reason is a genuine double-count fix.

No changes needed.

@cwest
cwest marked this pull request as ready for review September 12, 2026 07:17
LGuichet

This comment was marked as duplicate.

Comment thread internal/okf/lint.go Resolved
Comment thread internal/okf/analyze.go Resolved
Comment thread internal/okf/lint_taghygiene_test.go Resolved
@cwest
cwest force-pushed the topic/tag-hygiene branch from a3f25e4 to 6e11f81 Compare October 28, 2026 21:35
@cwest
cwest marked this pull request as draft October 28, 2026 21:35
lint warned that Concept and Concepts are one type spelled twice but had
no opinion about Wine/wine or runbook/runbooks/run-book — even though tags
is the other §4.1 field every consumer routes and filters on. The --tag
scope filter and the /api/v1/search tag parameter match exactly and
case-sensitively, so one tag spelled three ways is three disjoint result
sets, and analyze's cluster detector counted the variants separately and
missed the cluster.

Extract the shared base fold (case, trim, single trailing 's') as canonFold;
canonType is a thin wrapper over it, so type-hygiene stays byte-identical.
The tag path gets its own fold: canonTag computes a separator-insensitive
base key (-, _, space, and a letter-flanked / or . per UAX #29, so node.js
folds with nodejs while .net stays apart from net), and newTagFolder adds a
vocabulary-aware plural merge. The plural rules adapt KStem's plural()
(Lucene KStemmer.java): the -is/-us/-ss and length guards, -ies/-es/-s
handling, and Porter2's invariant list (news, atlas, cosmos, bias). A plural
folds onto its stem only when that stem is itself an attested tag in the
bundle, so a nonsense stem can never surface in a finding. This keeps new
apart from news and op apart from ops, and merges patch/patches,
policy/policies and process/processes, which the bare trailing-'s' rule
missed.

Add lintTagHygiene as a sibling of lintTypeHygiene: bundle-level, advisory,
"tag-hygiene" in --json, --strict treats it like any finding; the message
lists each variant with its per-node count, ordered most-common-first so the
dominant spelling to normalize toward leads. analyze adopts the same
bundle-scoped fold so it and lint agree on when two spellings are one tag,
labels a case-only cluster with the lower-cased spelling (matching prior
output), and dedups the node set per cluster, fixing a double-count where a
tag was repeated within one node's list.

The check never rejects a value (PRD §7.4 anti-taxonomy): it flags only
spellings that fold to one value. The fold is a heuristic, so the help text
tells a curator to review a finding before normalizing.

Closes #166
@cwest
cwest force-pushed the topic/tag-hygiene branch from 6e11f81 to 1cdd4b7 Compare October 28, 2026 21:38
@cwest
cwest marked this pull request as ready for review October 28, 2026 21:51

@cwest cwest left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rework holds up. The bare trailing-s rule is gone, replaced by a vocabulary-aware plural fold on the tag path: a plural folds onto its stem only when that stem is itself a tag in the bundle, so new/news and op/ops stay apart while patch/patches, policy/policies and process/processes now merge. canonFold/canonType and the type path are untouched, and the separator rule (hyphen, underscore, space, and a letter-flanked . or /) stays tag-only. analyze now labels a case-only cluster with its lower-cased spelling again, matching the prior output, and the per-node dedup is a genuine fix for the prior-art double-count.

Ran the branch against the real corpus: tag-hygiene goes 0 to 39 findings (25 case/plural, 14 separator-only), and every group is one concept spelled two ways — no false positives. type-hygiene output is unchanged. The -is/-us/-ss class stays silent without a stop-list, carried by the ss/ous guards and the vocabulary check. The negative controls are load-bearing now: the test reconstructs the old fold inline and proves it would have merged the pairs the new fold keeps apart, so a regression turns it red.

Full suite and conformance pass with -race, golangci-lint is clean on the diff, docs regenerate with no drift. Nothing to change.

No changes needed.

@cwest
cwest merged commit 5bb1fac into main Oct 28, 2026
10 checks passed
@cwest
cwest deleted the topic/tag-hygiene branch October 28, 2026 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

✨ feat(lint): add tag-hygiene beside type-hygiene—lint.go:620 folds type only, and --tag Wine and --tag wine return different nodes

2 participants