Repository navigation
✨ feat(lint): add tag-hygiene beside type-hygiene on a shared fold - #167
Conversation
cwest
left a comment
There was a problem hiding this comment.
The tag-hygiene check is correct and the fold refactor is done the right way: canonFold is the shared base, canonType is a thin wrapper so the type path stays byte-identical, and canonTag layers separator-insensitivity on top of it without leaking into types. The tests exercise both fold behaviors, the positive/negative controls, and the digraph class, and they pass locally with -race.
The blocker is the lint job, which fails on two issues in the new test file. golangci-lint v2.12.2 (same version CI runs) reports them deterministically; I reproduced both locally. gofmt and go vet are clean, so these only show up under golangci-lint. Both are one-line fixes, noted inline. Once the lint job is green the merge state will clear.
6ef1c92 to
a3f25e4
Compare
Rework — lint gate clearedHead moved Two golangci-lint issues on the new test file, both fixed in the amended commit:
Both are test-only; no production behavior changed. Evidence at
|
cwest
left a comment
There was a problem hiding this comment.
Verified end to end at the current head. The two items from the prior round are addressed: the fold-table row that tripped the misspell linter now uses a non-dictionary stem, and the sorted-variant assertion reads as a single disjunction. Both fixes are test-only — no production behavior moved.
The shared-fold design holds up. The base fold (case, trim, one trailing s) is extracted once; the type path is a thin wrapper over it and stays byte-identical, proven both by a fixture that fires type-hygiene with the exact same message and by a fixture confirming the separator rule does not leak into types (run-book and runbook as types produce no finding). The separator-insensitive layer applies to tags only, as intended.
I re-ran the full gate locally against this head: golangci-lint clean, gofmt and vet clean, conformance and the full suite green under -race, and every documented control passes — positive cases fire one finding each, the negative cases and the -is/-us/-ss digraph class stay silent without a stop-list. Against the real corpus, validate is clean, the new check surfaces 38 groups (up from zero), --strict exits non-zero on them, the JSON carries the check name at bundle level, and every group I spot-checked is one concept spelled two ways rather than a false positive. Analyze now folds those variants into shared clusters, and the one count that shifted for a different reason is a genuine double-count fix.
No changes needed.
a3f25e4 to
6e11f81
Compare
lint warned that Concept and Concepts are one type spelled twice but had no opinion about Wine/wine or runbook/runbooks/run-book — even though tags is the other §4.1 field every consumer routes and filters on. The --tag scope filter and the /api/v1/search tag parameter match exactly and case-sensitively, so one tag spelled three ways is three disjoint result sets, and analyze's cluster detector counted the variants separately and missed the cluster. Extract the shared base fold (case, trim, single trailing 's') as canonFold; canonType is a thin wrapper over it, so type-hygiene stays byte-identical. The tag path gets its own fold: canonTag computes a separator-insensitive base key (-, _, space, and a letter-flanked / or . per UAX #29, so node.js folds with nodejs while .net stays apart from net), and newTagFolder adds a vocabulary-aware plural merge. The plural rules adapt KStem's plural() (Lucene KStemmer.java): the -is/-us/-ss and length guards, -ies/-es/-s handling, and Porter2's invariant list (news, atlas, cosmos, bias). A plural folds onto its stem only when that stem is itself an attested tag in the bundle, so a nonsense stem can never surface in a finding. This keeps new apart from news and op apart from ops, and merges patch/patches, policy/policies and process/processes, which the bare trailing-'s' rule missed. Add lintTagHygiene as a sibling of lintTypeHygiene: bundle-level, advisory, "tag-hygiene" in --json, --strict treats it like any finding; the message lists each variant with its per-node count, ordered most-common-first so the dominant spelling to normalize toward leads. analyze adopts the same bundle-scoped fold so it and lint agree on when two spellings are one tag, labels a case-only cluster with the lower-cased spelling (matching prior output), and dedups the node set per cluster, fixing a double-count where a tag was repeated within one node's list. The check never rejects a value (PRD §7.4 anti-taxonomy): it flags only spellings that fold to one value. The fold is a heuristic, so the help text tells a curator to review a finding before normalizing. Closes #166
6e11f81 to
1cdd4b7
Compare
cwest
left a comment
There was a problem hiding this comment.
The rework holds up. The bare trailing-s rule is gone, replaced by a vocabulary-aware plural fold on the tag path: a plural folds onto its stem only when that stem is itself a tag in the bundle, so new/news and op/ops stay apart while patch/patches, policy/policies and process/processes now merge. canonFold/canonType and the type path are untouched, and the separator rule (hyphen, underscore, space, and a letter-flanked . or /) stays tag-only. analyze now labels a case-only cluster with its lower-cased spelling again, matching the prior output, and the per-node dedup is a genuine fix for the prior-art double-count.
Ran the branch against the real corpus: tag-hygiene goes 0 to 39 findings (25 case/plural, 14 separator-only), and every group is one concept spelled two ways — no false positives. type-hygiene output is unchanged. The -is/-us/-ss class stays silent without a stop-list, carried by the ss/ous guards and the vocabulary check. The negative controls are load-bearing now: the test reconstructs the old fold inline and proves it would have merged the pairs the new fold keeps apart, so a regression turns it red.
Full suite and conformance pass with -race, golangci-lint is clean on the diff, docs regenerate with no drift. Nothing to change.
No changes needed.
What
lintwarned thatConceptandConceptsare one type spelled twice but had noopinion about
Wine/wineorrunbook/runbooks/run-book— even thoughtagsis the other §4.1 field every consumer routes and filters on. The
--tagscopefilter and the
/api/v1/searchtagparameter match exactly and case-sensitively,so one tag spelled three ways is three disjoint result sets, and
analyze'scluster detector counted the variants separately and missed the cluster.
s) ascanonFold.canonTypeis a thin wrapper over it, sotype-hygienebehavior isbyte-identical (control: byte-identical on the real corpus, see below).
canonTagcomputes a separator-insensitivebase key —
-,_, space, and a letter-flanked/or.per Unicode UAX chore(deps): bump actions/setup-go from 5 to 7 #29(so
node.jsfolds withnodejsandci/cdwithcicd, while.netstaysapart from
net).newTagFolderthen adds a vocabulary-aware plural merge.plural()(the UMass CIIR stemmer shipped asLucene's
KStemmer.java): the length and letters-only guards, thess/ousguard on a plain final
s,-ies→-ie/-y,-es→drop-s/drop-es(skipthe
-ecandidate after a doubles), and Porter2's invariant list(
news,atlas,cosmos,bias, …) that is never stripped. A plural foldsonto its stem only when that stem is itself an attested tag in the bundle, so
a nonsense stem (
cach,nodej) can never surface in a finding. This keepsnewapart fromnewsandopapart fromops, and catchespatch/patches,policy/policies,process/processesthat a bare trailing-srule missed.lintTagHygieneas a sibling oflintTypeHygiene: bundle-level(
Path: ""), advisory,"check": "tag-hygiene"in--json, and--stricttreats it like any other finding. The message lists each variant with its
per-node count, ordered most-common-first (the dominant spelling to
normalize toward leads), ties broken alphabetically.
analyzeadopts the same bundle-scoped fold so it andlintagree on when twospellings are one tag. It labels a case-only cluster with the lower-cased
spelling (matching prior output), and dedups the node set per cluster — the old
per-occurrence
appenddouble-counted a tag repeated within one node's list(see the
prior-artdelta below).lintlong help namestag-hygieneand states the fold is a heuristic — reviewa finding before normalizing. §7.4 anti-taxonomy preserved: it flags only
spellings that fold to one value — a warning class, never a
validaterejection.Conformance gate (AGENTS.md, all three layers)
Layer 1 — spec-conformance suite:
Layer 2 — full suite:
Layer 3 — REAL CORPUS (
~/src/knowledge-base/bundles/knowledge, 271 nodes):type-hygienebyte-identical controlokfctl lint <kb> | grep '^type-hygiene'is byte-identical betweenmainandthis branch (the corpus has no type drift, so both emit zero
type-hygienelines).The shared-fold extraction did not move the type check.
linttag-hygiene findings: 0 → 39 (before → after)Old
mainemitted zero tag-hygiene findings (the check did not exist). The samecorpus now reports 39 tag-hygiene findings:
-/_/space or letter-flanked/·.rule)This matches the card's original prediction of ~39 (25 + 14). False-positive
count: 0 — every group is the same concept spelled two ways. All 39 inspected;
spot-checks:
backup(4)/backups(1),principle(5)/principles(1),cascading-failure(7)/cascading-failures(1),metastable-failure(6)/metastable-failures(2),design-patterns(4)/design-pattern(1),api-keys(2)/api-key(1),survey(3)/surveys(1),value(2)/values(1).home-lab(4)/homelab(2),allowed-models(2)/allowedmodels(2),gcs-fuse/gcsfuse,whitespace(2)/white-space(1),paged-attention/pagedattention,core-ml/coreml,tls-rpt/tlsrpt.Every plural group in this corpus is a plain trailing-
scase where both thesingular and plural are attested, so the vocabulary-aware merge folds them while
leaving unattested-stem look-alikes (
new/news,op/ops) apart.-is/-us/-ssclass: still silent, no stop-listNo tag in the
-is/-us/-ssclass produces a finding.cssandgpt-oss(
ss) andfamous-style (ous) are never stripped; an-is/-usword(
hormesis) generates its stripped candidate but folds only if that stem isattested, which it is not here nor in the corpus. Guarded by
TestLint_TagHygiene_DigraphClassSilentandTestLint_TagHygiene_IsUsClassGuardedByVocabulary— no stop-list needed.analyzeclusters: 454 → 461Every delta is a legitimate fold or the double-count fix:
grow by the merged node, pushing borderline tags over
--cluster-min 3(e.g.backup4→5,design-pattern4→5,principle5→6,career-ladder3→4,feedback-loop3→4,rate-limit3→5). Case-only clusters keep their lower-casedlabel (
wine, notWine), matchingmain.prior-art54 → 53 is a genuine double-count fix.infra/naming-a-compute-scarcity-pattern.mdlistsprior-arttwice in itstags; the old per-occurrenceappendcounted the node twice, the newdistinct-node set counts it once.
Both controls proven
Wine/wine,runbook/runbooks,run-book/runbook,Play Book/playbook,patch/patches,policy/policies,process/processes,ci/cd/cicd,node.js/nodejs— one finding each.new/news,op/opsstay silent;TestLint_TagHygiene_NegativeControlsAreLoadBearingreconstructs the old baretrailing-
sfold and asserts it WOULD have merged both, so a regression to thatrule turns the control red. Plus
v1/v2,oauth1/oauth2,oncall/incident,single-tag-many-nodes, the digraph class.
type-hygienemessage byte-identical, andrun-book/runbookas types do NOT fold (the tag separator/plural rules never leak into types).
Closes #166