From 2b3a1978e9051bf6e04fb5c137783c3e181fb1d9 Mon Sep 17 00:00:00 2001 From: Mohammad Wahbeh Date: Mon, 31 Aug 2026 01:09:34 +0300 Subject: [PATCH] chore: split docs/knowledge/ into a harvest-only tree and a notes tree (#56) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A `` sha digests the whole source file, not the entry. Every entry harvested from one document therefore carries the same value, and an edit to an entry's text changes no sha. The next harvest cannot see the edit: it regenerates the original text, or it writes a duplicate. Hand-written knowledge under the harvested tree was scheduled for silent deletion. The corpus is now two trees under docs/knowledge/: - harvested/ — what the documents say. Roles spec, design, styleguide. Generated by knowledge-harvest. Never hand-edited. 38 topic files, 1457 entries, INDEX.md, SOURCES.md. - notes/ — what the implementation found. Role review, a manual sha: marker. Hand-written. 3 files, 5 entries. A note overrides the harvested entry it names. docs/knowledge/README.md states that contract once. INDEX.md and SOURCES.md point at it rather than restating it, because a harvest overwrites both. Query surface (scripts/knowledge.mjs): - Reads both trees. Every entry carries an `origin`, and `--origin` selects one. - `--prefix HTTP` selects a whole requirement family. It is the audit-scale filter that `--req` cannot express. An unknown prefix exits 2, validated against appendix C. - Every entry carries a stable key, /<8 hex>, digested from the entry text. A note cites that key. The CLI resolves the citation, prints `[overridden by notes/...]` on the harvested entry, and resolves one on demand with `--key`. The key changes when the rule's text changes, which is when the note needs a fresh check. - An empty filter value is refused. `--topic ''` and the one-character typo `--topic 'a,b,'` matched every entry and printed the whole corpus. - A zero-result query tests each filter alone before it blames one. It used to report "PAGE-11 is canonical but no entry cites it yet" for `--prefix HTTP --req PAGE-11`, which is false: three entries cite PAGE-11, and none of them cites an HTTP ID. - The parser strips a BOM and accepts CRLF. A CRLF topic file parsed to zero entries, with no error. Two checks: - `bun run verify:knowledge-structure` — blocking, and the first step in ci.yml because it is pure Node over Markdown and needs no build. It rejects a review or invented role under harvested/, a Superseded entry there, an entry with no provenance line, a `` that cites outside the three source roots SOURCES.md names, a note that is not review-role, and a .md stranded at the root of docs/knowledge/ — which is where a harvest without `--corpus docs/knowledge/harvested` writes. It refuses to report OK below 1000 harvested entries, so a failing parse cannot pass vacuously. - `bun run knowledge:drift` — a hand-run report, deliberately not in CI. It compares each SOURCES.md sha against the file on disk, and it reports every note citation that no entry carries any more. Three sources drift today, all in docs/sdk-design-nodejs/. The 16 styleguide sources are NOT VERIFIABLE off the harvest machine, which is expected and never a failure. Migration: - Four resolved Conflicts entries are split. Each statement stays in harvested/. Each resolution is a note that names the statement by key. - The PAGE-11 erratum moves to notes/pagination.md, under Superseded. The inline erratum marker it duplicated is removed from the harvested Reference entry, which now prints the override tag instead. - deliberate-deviations.md leaves the corpus, with its SOURCES.md row. A register accumulates rows, so any harvest of one is a stale fraction of it. The copy held 13 entries against a 17-item register, at a three-revision-old sha, with two entries substantively false. notes/deliberate-deviations.md is the pointer. docs/open-items.md is the second register under the same rule. Documentation: - .claude/skills/knowledge-lookup/SKILL.md rewritten. Phase start is two queries, `--origin note` and `--section conflicts`; they are different sets, and a conflict with no override tag is still open. Adds the audit query as a stated exception to the rule against broad topic queries, the API-surface audit group, the audit loop, and the prefix-to-chapter table that completes the roll-up path. Frozen counts removed. - CLAUDE.md records the two trees, both checks, and the citation convention. - 107 corpus citations in packages/, test/, tests/ and .changeset/ are repointed to harvested/. Two cited the deleted file and now cite the ledger. docs/superpowers/ keeps its pre-split paths: those files are dated records of what a phase planned, and they are not retro-edited. docs/open-items.md section O records three gaps: the global knowledge-harvest skill still defaults --corpus to the tree no query reads (O1), a note's key citation is checked by a report and not by a gate (O2), and the docs/superpowers/ paths are deliberate (O3). H13 is raised in priority: a blocking gate now depends on a parser that no CI job tests. No changeset. No published package surface changes. Verified: test:scripts 85 pass, lint 0, build 0, test 2175 pass / 0 fail, structure gate OK, drift 44 OK / 3 DRIFT / 5 note citations resolve. --- .changeset/2026-08-26-execution-context.md | 2 +- .../2026-08-27-configuration-review-pass-3.md | 2 +- .claude/skills/ci-preflight/run-ci.mjs | 8 + .claude/skills/knowledge-lookup/SKILL.md | 250 ++++-- .github/workflows/ci.yml | 10 + CLAUDE.md | 59 +- docs/deviations.md | 8 +- docs/knowledge/README.md | 31 + docs/knowledge/deliberate-deviations.md | 62 -- docs/knowledge/{ => harvested}/INDEX.md | 20 +- docs/knowledge/{ => harvested}/SOURCES.md | 8 +- docs/knowledge/{ => harvested}/api-design.md | 0 docs/knowledge/{ => harvested}/assertions.md | 0 .../{ => harvested}/authentication.md | 0 .../cancellation-and-timeouts.md | 0 .../{ => harvested}/concurrency-and-async.md | 0 .../{ => harvested}/configuration.md | 0 .../cross-cutting-invariants.md | 0 .../{ => harvested}/data-modeling.md | 0 .../{ => harvested}/documentation.md | 0 .../{ => harvested}/error-handling.md | 0 .../{ => harvested}/execution-context.md | 0 .../{ => harvested}/function-design.md | 0 .../{ => harvested}/http-domain-model.md | 0 .../{ => harvested}/io-and-byte-streams.md | 0 .../{ => harvested}/message-bodies.md | 0 .../{ => harvested}/module-organization.md | 0 .../{ => harvested}/naming-conventions.md | 0 .../{ => harvested}/observability.md | 0 .../package-and-dependency-layout.md | 0 docs/knowledge/{ => harvested}/pagination.md | 8 +- docs/knowledge/{ => harvested}/performance.md | 0 docs/knowledge/{ => harvested}/pipeline.md | 0 .../{ => harvested}/redaction-and-security.md | 0 .../{ => harvested}/redirect-handling.md | 0 .../{ => harvested}/resource-management.md | 0 .../{ => harvested}/retry-and-resilience.md | 0 .../{ => harvested}/sdk-positioning.md | 0 .../seams-and-extensibility.md | 0 docs/knowledge/{ => harvested}/serde.md | 0 .../{ => harvested}/sse-streaming.md | 0 .../{ => harvested}/styleguide-overview.md | 0 docs/knowledge/{ => harvested}/testing.md | 0 .../tooling-and-quality-gates.md | 12 +- .../{ => harvested}/transport-adapter.md | 0 docs/knowledge/{ => harvested}/type-system.md | 0 .../{ => harvested}/typescript-idioms.md | 0 .../{ => harvested}/url-and-query-encoding.md | 0 .../variables-and-declarations.md | 0 docs/knowledge/notes/deliberate-deviations.md | 7 + docs/knowledge/notes/pagination.md | 11 + .../notes/tooling-and-quality-gates.md | 13 + docs/open-items.md | 98 ++- ...-deviations-from-the-reference-contract.md | 2 +- package.json | 2 + .../codec-json/src/tristate-schema.test.ts | 2 +- packages/core/src/auth/auth-step.test.ts | 2 +- packages/core/src/auth/auth-step.ts | 6 +- packages/core/src/auth/bearer-cache.ts | 4 +- packages/core/src/auth/credential.ts | 4 +- packages/core/src/auth/digest.ts | 2 +- packages/core/src/auth/errors.ts | 4 +- packages/core/src/auth/md5.ts | 2 +- packages/core/src/auth/preset.ts | 2 +- packages/core/src/auth/requirement.ts | 2 +- packages/core/src/config/build-info.ts | 4 +- .../core/src/config/client-identity-step.ts | 2 +- .../core/src/config/configuration.test.ts | 2 +- packages/core/src/config/configuration.ts | 4 +- packages/core/src/config/duration.test.ts | 2 +- packages/core/src/config/duration.ts | 2 +- packages/core/src/config/identifiers.test.ts | 2 +- packages/core/src/config/identifiers.ts | 2 +- packages/core/src/config/proxy.test.ts | 2 +- packages/core/src/context/store.test.ts | 2 +- packages/core/src/context/store.ts | 4 +- packages/core/src/index.ts | 6 +- packages/core/src/invariant.test.ts | 2 +- packages/core/src/invariant.ts | 4 +- packages/core/src/pipeline/builder.test.ts | 2 +- packages/core/src/pipeline/errors.ts | 2 +- packages/core/src/recovery/cancellation.ts | 2 +- packages/core/src/recovery/outcome.test.ts | 2 +- packages/core/src/recovery/response-chain.ts | 2 +- packages/core/src/recovery/status-mapping.ts | 2 +- packages/core/src/redirect/errors.test.ts | 2 +- packages/core/src/redirect/errors.ts | 2 +- packages/core/src/retry/engine.ts | 2 +- packages/core/src/seams/index.ts | 2 +- packages/core/src/serde/tristate.test.ts | 2 +- packages/core/src/sse/stream.test.ts | 2 +- packages/core/src/sse/stream.ts | 6 +- packages/core/src/suppress.test.ts | 2 +- packages/core/src/suppress.ts | 6 +- packages/shrink-test/src/fixture-app.ts | 2 +- .../shrink-test/src/run-shrink-guard.test.ts | 6 +- .../transport-fetch/src/fetch-transport.ts | 2 +- .../transport-undici/src/undici-transport.ts | 2 +- scripts/knowledge-drift.mjs | 195 +++++ scripts/knowledge.mjs | 727 ++++++++++++++---- scripts/knowledge.test.mjs | 378 ++++++++- scripts/verify-knowledge-structure.mjs | 251 ++++++ scripts/verify-knowledge-structure.test.mjs | 207 +++++ .../xcut/error-taxonomy.conformance.test.ts | 2 +- tests/node-conformance/README.md | 2 +- .../node-conformance/recovery-chain.test.mjs | 4 +- 106 files changed, 2082 insertions(+), 417 deletions(-) create mode 100644 docs/knowledge/README.md delete mode 100644 docs/knowledge/deliberate-deviations.md rename docs/knowledge/{ => harvested}/INDEX.md (73%) rename docs/knowledge/{ => harvested}/SOURCES.md (90%) rename docs/knowledge/{ => harvested}/api-design.md (100%) rename docs/knowledge/{ => harvested}/assertions.md (100%) rename docs/knowledge/{ => harvested}/authentication.md (100%) rename docs/knowledge/{ => harvested}/cancellation-and-timeouts.md (100%) rename docs/knowledge/{ => harvested}/concurrency-and-async.md (100%) rename docs/knowledge/{ => harvested}/configuration.md (100%) rename docs/knowledge/{ => harvested}/cross-cutting-invariants.md (100%) rename docs/knowledge/{ => harvested}/data-modeling.md (100%) rename docs/knowledge/{ => harvested}/documentation.md (100%) rename docs/knowledge/{ => harvested}/error-handling.md (100%) rename docs/knowledge/{ => harvested}/execution-context.md (100%) rename docs/knowledge/{ => harvested}/function-design.md (100%) rename docs/knowledge/{ => harvested}/http-domain-model.md (100%) rename docs/knowledge/{ => harvested}/io-and-byte-streams.md (100%) rename docs/knowledge/{ => harvested}/message-bodies.md (100%) rename docs/knowledge/{ => harvested}/module-organization.md (100%) rename docs/knowledge/{ => harvested}/naming-conventions.md (100%) rename docs/knowledge/{ => harvested}/observability.md (100%) rename docs/knowledge/{ => harvested}/package-and-dependency-layout.md (100%) rename docs/knowledge/{ => harvested}/pagination.md (93%) rename docs/knowledge/{ => harvested}/performance.md (100%) rename docs/knowledge/{ => harvested}/pipeline.md (100%) rename docs/knowledge/{ => harvested}/redaction-and-security.md (100%) rename docs/knowledge/{ => harvested}/redirect-handling.md (100%) rename docs/knowledge/{ => harvested}/resource-management.md (100%) rename docs/knowledge/{ => harvested}/retry-and-resilience.md (100%) rename docs/knowledge/{ => harvested}/sdk-positioning.md (100%) rename docs/knowledge/{ => harvested}/seams-and-extensibility.md (100%) rename docs/knowledge/{ => harvested}/serde.md (100%) rename docs/knowledge/{ => harvested}/sse-streaming.md (100%) rename docs/knowledge/{ => harvested}/styleguide-overview.md (100%) rename docs/knowledge/{ => harvested}/testing.md (100%) rename docs/knowledge/{ => harvested}/tooling-and-quality-gates.md (91%) rename docs/knowledge/{ => harvested}/transport-adapter.md (100%) rename docs/knowledge/{ => harvested}/type-system.md (100%) rename docs/knowledge/{ => harvested}/typescript-idioms.md (100%) rename docs/knowledge/{ => harvested}/url-and-query-encoding.md (100%) rename docs/knowledge/{ => harvested}/variables-and-declarations.md (100%) create mode 100644 docs/knowledge/notes/deliberate-deviations.md create mode 100644 docs/knowledge/notes/pagination.md create mode 100644 docs/knowledge/notes/tooling-and-quality-gates.md create mode 100644 scripts/knowledge-drift.mjs create mode 100644 scripts/verify-knowledge-structure.mjs create mode 100644 scripts/verify-knowledge-structure.test.mjs diff --git a/.changeset/2026-08-26-execution-context.md b/.changeset/2026-08-26-execution-context.md index 369f49d..546a7c4 100644 --- a/.changeset/2026-08-26-execution-context.md +++ b/.changeset/2026-08-26-execution-context.md @@ -37,7 +37,7 @@ Three design calls worth recording: Two known deviations, both already in the deferral register (`docs/open-items.md`): - `contextStore` is a module-level mutable singleton, which - `docs/knowledge/variables-and-declarations.md:22` bans. Accepted because threading a store handle through + `docs/knowledge/harvested/variables-and-declarations.md:22` bans. Accepted because threading a store handle through builder → runtime → every step would be a wide API change for no observable gain; logged in the design's Deviation Ledger for Phase 10. Tests build their own `new ContextStore()` rather than asserting through the singleton, which is shared by every file in a `bun test` run. diff --git a/.changeset/2026-08-27-configuration-review-pass-3.md b/.changeset/2026-08-27-configuration-review-pass-3.md index c89a6e8..55d108f 100644 --- a/.changeset/2026-08-27-configuration-review-pass-3.md +++ b/.changeset/2026-08-27-configuration-review-pass-3.md @@ -8,7 +8,7 @@ which is the whole of the `etc/core.api.md` diff: - `Clock.sleep(ms, signal)` becomes `Clock.sleep(durationMs, signal)`. A bare `ms` is a unit with no concept attached, and the report carried it two lines above `composeSignal(userSignal, timeoutMs)` — the same package stating the same kind of quantity two different ways - (`docs/knowledge/naming-conventions.md:36`). + (`docs/knowledge/harvested/naming-conventions.md:36`). - `Configuration.getDuration(key, fallback)` becomes `getDuration(key, fallbackMs)`. The accessor returns and accepts milliseconds, and said so only in prose while its own private collaborator is named `parseDurationMs`. diff --git a/.claude/skills/ci-preflight/run-ci.mjs b/.claude/skills/ci-preflight/run-ci.mjs index d39be13..e7f7678 100644 --- a/.claude/skills/ci-preflight/run-ci.mjs +++ b/.claude/skills/ci-preflight/run-ci.mjs @@ -38,6 +38,14 @@ const STEPS = [ tier: 'install', fix: 'bun install (then commit the updated bun.lock)', }, + { + // First of the gates, mirroring ci.yml: pure Node over Markdown, no build, + // so a corpus mistake reports in seconds rather than after the pipeline. + id: 'verify:knowledge-structure', + ci: 'Knowledge-corpus structure check', + cmd: 'bun run verify:knowledge-structure', + tier: 'gate', + }, {id: 'typecheck', ci: 'Typecheck', cmd: 'bun run typecheck', tier: 'build'}, { id: 'lint', diff --git a/.claude/skills/knowledge-lookup/SKILL.md b/.claude/skills/knowledge-lookup/SKILL.md index 3e4855b..91d373f 100644 --- a/.claude/skills/knowledge-lookup/SKILL.md +++ b/.claude/skills/knowledge-lookup/SKILL.md @@ -1,78 +1,120 @@ --- name: knowledge-lookup -description: Use when starting a numbered task from a docs/superpowers/plans/ file, implementing or reviewing against a requirement ID (HTTP-7, SEAM-1, RETRY-13, NFR-5), or resolving a styleguide citation such as "styleguide 6.7" or "ch08". +description: Use when starting a phase or a numbered task from a docs/superpowers/plans/ file, implementing or reviewing against a requirement ID (HTTP-7, SEAM-1, RETRY-13, NFR-5), resolving a styleguide citation such as "styleguide 6.7" or "ch08", auditing a subsystem against every rule the corpus holds for it, or recording what an implementation found in docs/knowledge/ — which is a note under notes/, never an edit to harvested/. --- # Knowledge Lookup ## Overview -`docs/knowledge/` is 39 topic files and ~1470 harvested entries, past what belongs +`docs/knowledge/` is 39 topics across two trees and ~1457 harvested entries, past what belongs in context. `bun run knowledge` filters it. A requirement-ID query runs ~120–580 tokens (median ~230) against a topic file of ~1800–5200 (median ~2300): roughly 9× smaller, and much more than that when the ID you want lives in a file you'd never have guessed. -Every entry is one bullet plus a `` line carrying role, source path, line range, and -sha — the citation a test-file header or deferral note needs. +Every entry is one bullet plus a `` line, and a stable key — `/<8 hex>`, printed +after the section name. The `` usually carries role, source path, line range and sha, which +is the citation a test-file header or deferral note needs; a Conflicts entry carries two sources +and no sha, and a note carries a manual `sha:` marker. Copy what is there. -## Start of a phase: run this once +**Two trees, one query surface.** -```bash -bun run knowledge --section conflicts --brief # 6 entries corpus-wide, ~1.1k tokens -``` - -Six entries exist. They are where a design-vs-styleguide contradiction is recorded as -resolved or still open, and a plan's Global Constraints may assert as settled something the -corpus still lists **unresolved**. Nothing else in this workflow will surface them. +| Tree | Holds | Rule | +|---|---|---| +| `docs/knowledge/harvested/` | What the source documents say. Roles `spec`, `design`, `styleguide`. | Generated by `knowledge-harvest`. **Never hand-edit it.** A `` sha digests the whole source file, not the entry, so your edit changes no sha and the next harvest deletes or duplicates it. | +| `docs/knowledge/notes/` | What the implementation found. Role `review`, a manual `sha:` marker. | Hand-written. Small by design — a note earns its place only by overriding a harvested rule. | -## Starting a numbered task: one query, not six +**How to read a result:** a role of `spec`, `design`, or `styleguide` states what the +documents say; a role of `review` states what the implementation found, and it overrides the +first. `verify:knowledge-structure` is the CI gate that keeps the two apart. -Plan tasks list their requirement IDs in the task header. Pass the whole set at once — -`--req` accepts commas and ORs within itself: +## Start of a phase: run these two ```bash -bun run knowledge --req HTTP-13,HTTP-14,HTTP-15,HTTP-16,HTTP-3,HTTP-5 +bun run knowledge --origin note --brief # what the implementation found +bun run knowledge --section conflicts --brief # what two documents still disagree about ``` +They are different sets and you need both. The first is every note: the places where a plan's +Global Constraints may assert as settled something the implementation found to be otherwise. +The second is every recorded design-vs-styleguide contradiction — a resolved one prints +`[overridden by notes/…]` on its location line, and one with no such tag is **still open**. A +plan that assumes an open conflict is settled is the failure both queries exist to catch, and +nothing else in this workflow surfaces either. + Different filters AND together; multiple values inside one filter OR. So `--req A --req B --topic headers` means "(cites A or B) and is in a headers file". ## Check the result is real before trusting it -**A `--req` hit is not proof the corpus knows anything.** 255 of the 645 canonical IDs resolve +**A `--req` hit is not proof the corpus knows anything.** 256 of the 645 canonical IDs resolve *only* to an appendix-B conformance roll-up — one sentence naming three to five IDs and -stating none of them. It exits 0, so nothing else will warn you. - -The CLI tags these `[appendix-B roll-up]` and prints a WARNING when every hit is one. When -you see it, stop querying and go to the source: - -```bash -grep -n '^| HTTP-10 ' docs/product-spec/appendix-c-consolidated-normative-requirement-index.md -``` - -The leading `| ` and trailing space are load-bearing — `grep 'HTTP-1'` matches HTTP-10 -through HTTP-19. +stating none of them. It exits 0, so nothing else will warn you. (`--coverage` reports the +substantive / roll-up-only / uncited split per prefix; run it rather than trusting a +remembered number.) + +The CLI tags these `[appendix-B roll-up]` and prints a WARNING when every hit is one. When you +see it, follow the roll-up path below — all three steps, not just the first. + +### The roll-up path + +1. `bun run knowledge --req ` came back all roll-up. +2. Get the canonical text: + + ```bash + grep -n '^| HTTP-10 ' docs/product-spec/appendix-c-consolidated-normative-requirement-index.md + ``` + + The leading `| ` and the trailing space are load-bearing — `grep 'HTTP-1'` matches HTTP-10 + through HTTP-19. +3. Read the owning chapter. The row from step 2 carries a Subsystem cell; the prefix maps to a + chapter file in `docs/product-spec/`: + + | Prefix | Chapter | + |---|---| + | SEAM | `03-pluggable-seams-and-extension-model.md` | + | HTTP | `04-core-http-domain-model.md` | + | IO | `05-i-o-contracts.md` | + | BODY | `06-request-and-response-body-lifecycle.md` | + | CTX | `07-execution-context-model.md` | + | PIPE, RECOV | `08-execution-pipelines.md` | + | RETRY | `09-retry-and-resilience.md` | + | REDIR | `10-redirect-handling.md` | + | AUTH | `11-authentication.md` | + | PAGE | `12-pagination.md` | + | SSE | `13-server-sent-events-and-streaming.md` | + | SERDE | `14-serialization-serde.md` | + | OBS | `15-instrumentation-and-observability.md` | + | CFG | `16-configuration.md` | + | TRANSPORT | `17-transport-adapter-conformance-contract.md` | + | ASYNC | `18-asynchronous-runtime-adapter-contract.md` | + | XCUT | `19-cross-cutting-invariants-and-policies.md` | + | NFR | `20-non-functional-requirements-and-quality-bar.md` | + + A prefix not listed here is new; derive its chapter from the Subsystem cell. ## Two entry points -**ID-first — you have requirement IDs.** This is the plan-task case. +**ID-first — you have requirement IDs.** This is the plan-task case. Plan tasks list their IDs +in the task header; pass the whole set in one call, never six. -1. `bun run knowledge --req ` — what the corpus concluded. Design-role entries quote - `docs/sdk-design-nodejs/` inline, so this usually covers the TypeScript mapping too; add - `--role design` to isolate them. Open the design doc only to follow a line range. -2. `grep -n '^| ' docs/product-spec/appendix-c-…md` — canonical text, when the query - came back a roll-up or you need the normative wording verbatim. +1. `bun run knowledge --req HTTP-13,HTTP-14,HTTP-15` — what the corpus concluded. Design-role + entries quote `docs/sdk-design-nodejs/` inline, so this usually covers the TypeScript + mapping too; add `--role design` to isolate them. Open the design doc only to follow a line + range. +2. `grep -n '^| ' docs/product-spec/appendix-c-…md` — canonical text, when the query came + back a roll-up or you need the normative wording verbatim. **Topic-first — you have an area, or a styleguide citation.** ```bash -bun run knowledge --list-topics # 39 topics, entry and ID counts +bun run knowledge --list-topics # topics, entry, ID and note counts bun run knowledge --topic pipeline --section rules --brief cursor fork ``` -**15 of the 39 topics carry no requirement ID at all** — every styleguide-derived one, -including `data-modeling`, `error-handling`, `assertions`, `testing`, `api-design`. ID-first -cannot reach them. `--list-topics` shows which; don't work from a memorised list. +**15 of the 38 harvested topics carry no requirement ID at all** — every styleguide-derived +one, including `data-modeling`, `error-handling`, `assertions`, `testing`, `api-design`. +ID-first cannot reach them. `--list-topics` shows which; don't work from a memorised list. For "styleguide 6.7" / "ch08", use `--chapter`: @@ -83,45 +125,107 @@ bun run knowledge --chapter 6 interface class # styleguide 6.7 → the clas Entries record a chapter file and line range, never a section number, so `--chapter 6.7` queries chapter 6 and tells you it dropped the `.7`. Narrow with bare words instead. -## Never read a whole topic file — with two stated exceptions +## Auditing: the one case where a broad query is right + +A lookup wants the smallest answer. An audit wants a **complete group of rules**, because a +rule it never read is a rule it never checked. The CLI applies no cap, so a group query is +complete as soon as the group is. + +Two forms: + +```bash +bun run knowledge --topic api-design,documentation,type-system --section rules --brief +bun run knowledge --prefix HTTP --section rules --brief +``` + +`--prefix` takes a whole ID family and is validated against appendix C, so a typo fails +loudly instead of returning nothing. + +**One topic is not a group.** A search of `Rules` for the word "public" hits twelve topics; an +audit that queries `api-design` alone reports clean over an incomplete set. Use the recorded +groups, and extend this table rather than improvising a group per audit — an audit is only +repeatable if its group is written down. + +| Group | Topics | +|---|---| +| API surface | `api-design`, `http-domain-model`, `documentation`, `type-system`, `module-organization`, `error-handling`, `tooling-and-quality-gates`, `styleguide-overview` | + +To build a group that is not in the table: for an ID-bearing subsystem use `--prefix`, which is +exact. Otherwise grep the corpus for the subject word (`bun run knowledge --section rules +--brief `), take the topic files that came back, and add the row here before running the +audit — an audit whose group is not written down cannot be repeated. The API-surface group's +`--section rules` is ~183 entries and ~12k tokens; budget for it. + +### The audit loop + +1. **Read the notes first** — the phase-start query above. A rule that already carries a note + is already known-broken; don't re-report it. A harvested entry that prints + `[overridden by notes/…]` is the same signal inline. +2. **Read the group.** One query, `--section rules`, from the table above. +3. **Check the system** against each rule. +4. **Write a note for each broken rule**, in `docs/knowledge/notes/.md`, naming the + rule by the key the query printed (`api-design/e0f4662b`). Backtick the key: that is how + the CLI links the two, so the harvested entry then prints `[overridden by notes/…]` and + `bun run knowledge --key api-design/e0f4662b` resolves it. The key changes exactly when the + rule's text changes — including on a re-harvest that rewords it, which is when the note + needs revisiting; `bun run knowledge:drift` reports a citation that has gone stale. Never + edit the harvested entry. + +A note's shape: a topic heading, a section, one bullet, role `review`, a source path, a manual +`sha:manual-` marker. + +```markdown +# pagination — notes + +Hand-written. `../harvested/pagination.md` is what the documents say; this is what the +implementation found, and it wins. + +## Superseded +- **What we found**, superseding `pagination/81881061`. … + review · `docs/superpowers/specs/2026-07-28-phase6c-pagination-design.md` · high · sha:manual-6c-erratum +``` + +Which section: `## Superseded` when following the harvested rule would cause damage, +`## Conflicts` when you are recording which of two documents won, `## Reference` when the note +only points somewhere (the deviation register's pointer is the one example). A note is not for +a preference. + +**Re-harvesting.** Always `--corpus docs/knowledge/harvested`; the skill's default is +`docs/knowledge/`, which no query reads and the structure gate rejects. If the harvest emits a +`supersede` resolution, move it to `notes/` by hand — a `## Superseded` entry under +`harvested/` fails CI. -**Reading a topic file when a filtered query answers the question is the failure this tool -exists to prevent.** Not "I'll grep it myself" — `grep` has no section, role, or exact-token -ID matching. Not "I need surrounding context" — widen the filter first. +## Never read a whole topic file -The two cases where reading is correct, and how: +A filtered query answers the question for a fraction of the tokens, and `grep` is not a +substitute — it has no section, role, or exact-token ID matching. Two exceptions, both narrow: -- **An unnarrowed `--topic` costs more than the file.** `--topic http-domain-model` is - 22,196 bytes; the file is 20,157. A topic query without `--section`, `--chapter`, or bare - words is not a filter. Add one, or read the file — don't run the query. +- **An unnarrowed `--topic` costs more than the file.** Add `--section`, `--chapter`, or bare + words, or read the file — but don't run a bare `--topic` on a subsystem topic. (Two stated + exceptions: an audit group, which is deliberately broad and narrowed by `--section rules`; + and a topic `--list-topics` shows to be tiny, such as a note-only one.) - **Following up a located entry, when you need the exact bytes.** The bullet sits at the - printed line, its `` at line+1, entries are 2 lines with no blank between — so read - an even span starting on the bullet or you will split a rule from its citation, and stop - at the section boundary or you silently cross into Constraints. - - Reach for this last. A neighbouring entry is only related to the one you found about half - the time (54% of adjacent pairs share a source line or one within 3), so "read around it" - is a weak way to find the rest of a rule cluster. Two better moves first: - - **The cluster is defined by ID, not by file position.** Landed on HTTP-5 and want the - rule it belongs to? `--req HTTP-3,HTTP-4,HTTP-5`. Appendix C numbers related - requirements together; the topic file does not order them for you. - - **Pull the section.** `--topic X --section rules` — the median section is 7 entries - (~550 tokens). Only the big subsystem `Rules` sections (pipeline 58, retry 43, auth 41) - are expensive enough to need narrowing with bare words. + printed line; read from it to the next `` (usually line+1, but a multi-paragraph entry + runs longer), and stop at the section boundary. Reach for this last: a neighbouring entry is + related only about half the time. Prefer `--req` on the ID cluster (appendix C numbers + related requirements together) or `--topic X --section rules`, whose median section is small. ## Quick reference | Flag | Effect | |---|---| | `--req HTTP-7,HTTP-8` | Entries citing any of these. Exact-token: never matches `HTTP-70`. | +| `--prefix HTTP` | A whole ID family. The audit filter; validated against appendix C. | +| `--key pagination/81881061` | The one entry with that key. How a note's citation is resolved; an unknown key means the rule was reworded. | +| `--origin harvested\|note` | Which tree. `note` is the phase-start query. | | `--topic pipeline,retry` | Topic files by substring — matches broadly and silently. | -| `--section rules,…` | rules, constraints, conclusions, reference, conflicts. (superseded is empty.) | -| `--role spec\|design\|styleguide\|review` | Filter by provenance role. | +| `--section rules,…` | rules, constraints, conclusions, reference, conflicts, superseded. | +| `--role spec\|design\|styleguide\|review` | Provenance role. `review` only ever appears in `notes/`. | | `--chapter 6` | Styleguide chapter. The only way in from a "styleguide N.M" citation. | | `--grep ` / bare words | Case-insensitive; regex is real, bare words are literal. | | `--brief` | Drop `` lines, ~30% smaller — but you lose the citation. | -| `--json` | Records, each with a `rollup` boolean. | -| `--list-topics` | The 39 topics with entry and distinct-ID counts. | +| `--json` | Records, each with `origin`, `key` and a `rollup` boolean. | +| `--list-topics` | Every topic with entry, distinct-ID and note counts. | | `--list-reqs` | ID → location map. **~6k tokens, bigger than any topic file.** Prefer `--coverage`. | | `--coverage` | Substantive vs roll-up-only vs uncited, per prefix. A report, not a gate. | @@ -135,12 +239,12 @@ The `` line is the citation, but it comes in two tiers: - **spec / design** — repo-relative, quote verbatim: `` docs/product-spec/09-retry-and-resilience.md:28 · sha:9efbe276001e `` - **styleguide** — an absolute path to a sibling repo on the harvest machine - (`/home/…/styleguide/typescript/11-testing.md:110-114`). **Strip the machine prefix** - before committing it: `styleguide/typescript/11-testing.md:110-114`. Pasting it raw - produces a citation that resolves on one laptop. + (`/home/…/styleguide/typescript/11-testing.md:110-114`). **Strip the machine prefix** before + committing it: `styleguide/typescript/11-testing.md:110-114`. Pasting it raw produces a + citation that resolves on one laptop. -Shape is not uniform: Conflicts entries carry two sources and no sha; one `review` entry has -no line range. Copy what is there, don't assume four fields. +Shape is not uniform: Conflicts entries carry two sources and no sha; a note carries a manual +`sha:` marker and sometimes no line range. Copy what is there, don't assume four fields. Drop `--brief` whenever the result will be cited. @@ -148,10 +252,12 @@ Drop `--brief` whenever the result will be cited. | Mistake | Fix | |---|---| -| Trusting a `--req` hit that is all roll-up | Watch for the WARNING; go to appendix C and the owning `product-spec/NN` chapter. | +| Editing an entry under `harvested/` to record what you found | Write a note in `notes/` naming the rule by its key. The harvested sha is per-file; your edit is invisible to the next harvest and CI rejects it. | +| Trusting a `--req` hit that is all roll-up | Watch for the WARNING; follow all three steps of the roll-up path. | | Six sequential `--req` calls for one task | One comma-separated call. | +| Auditing one topic and reporting "clean" | One topic is not a group. Use the group table, or `--prefix`. | | ID-first on `data-modeling` / `error-handling` / `testing` | Those carry zero IDs. Topic- or chapter-first. | | Pasting a styleguide `` path verbatim | Strip the machine prefix first. | -| `--topic X` with nothing else | Not a filter; costs more than the file. | -| Reading `--coverage`'s total as "the corpus knows this" | 385/645 are substantive; 256 more are roll-up only. | -| Treating `--coverage` as a gate | Hand-run report. Nothing in CI runs it. | +| `--topic X` with nothing else | Not a filter; costs more than the file — unless `--list-topics` shows the topic is tiny. | +| Looking for the deviation register in the corpus | It is not harvested — a register goes stale on the next append. `bun run knowledge --topic deliberate-deviations` gives the pointer; read §10 itself. | +| Treating `--coverage` or `knowledge:drift` as a gate | Hand-run reports. The only CI gate here is `verify:knowledge-structure`. | diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c4c1bc7..c9331cc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -19,6 +19,16 @@ jobs: - name: Install (frozen lockfile) run: bun install --frozen-lockfile + # First of the gates: pure Node over Markdown, no build, no dist/, so a + # corpus mistake reports in seconds instead of after the whole pipeline. + # Keeps docs/knowledge/'s two trees apart — no hand-written entry under + # harvested/, whose `` shas digest whole source files and so cannot + # record an edit; the next harvest would delete it silently. The companion + # drift report is deliberately NOT here: 16 of the 47 sources are a + # sibling styleguide repository that no CI checkout has. + - name: Knowledge-corpus structure check + run: bun run verify:knowledge-structure + - name: Typecheck run: bun run typecheck diff --git a/CLAUDE.md b/CLAUDE.md index bdf371c..81102b2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -125,6 +125,7 @@ bun run verify:seam-1 # zero runtime dependencies in EVERY package, bun run verify:sse-37 # no serde dependency and no reconnect path in core SSE bun run verify:runtime-floor # tsconfig target vs package engines.node consistency bun run verify:test-partition # the five files that keep tests/ and tests/node-conformance/ apart +bun run verify:knowledge-structure # docs/knowledge/'s two trees stay separate (see below) bun run verify:reproducible-build # two clean builds of one source tree agree, dist/ and tarball (NFR-12) bun run test:scripts # the gates' OWN tests (node --test scripts/*.test.mjs) bun run audit # bun audit --audit-level=high --prod @@ -134,10 +135,10 @@ bun run audit # bun audit --audit-level=high --prod work is done — `bun run test` passing is not sufficient evidence. `test:scripts` tests the *gates themselves* — the knowledge CLI, `verify-seam-1.mjs`, `verify-sse-37.mjs`, -`verify-test-partition.mjs`. Phase 10 made it a blocking CI step, closing `docs/open-items.md` H13. It was -not one before, and the proof that it should have been is that `knowledge.test.mjs` had been failing on -`main` since `36c3f96` with nobody noticing. A gate whose own logic degrades still exits 0, so nothing else -in the run would. +`verify-knowledge-structure.mjs`, `verify-test-partition.mjs`. Phase 10 made it a blocking CI step, closing +`docs/open-items.md` H13. It was not one before, and the proof that it should have been is that +`knowledge.test.mjs` had been failing on `main` since `36c3f96` with nobody noticing. A gate whose own logic +degrades still exits 0, so nothing else in the run would. ### HARD RULE — the `tests/` partition @@ -179,13 +180,14 @@ passes. The gate checks this too. ## Documentation hierarchy -Four distinct trees, easy to confuse: +Five distinct trees, easy to confuse — `docs/knowledge/` being two of them: | Path | Role | |---|---| | `docs/product-spec/` | **Normative.** Numbered requirements (`HTTP-7`, `SEAM-1`, `RETRY-13`, `NFR-5`, …). The source of truth. | | `docs/sdk-design-nodejs/` | How each spec area maps to idiomatic TypeScript. Non-normative but binding by convention. | -| `docs/knowledge/` | Harvested styleguide + spec knowledge, topic-indexed (`INDEX.md`). Cited as "styleguide 6.7", "ch08". | +| `docs/knowledge/harvested/` | Harvested styleguide + spec knowledge, topic-indexed (`INDEX.md`). Cited as "styleguide 6.7", "ch08". Generated; never hand-edited. | +| `docs/knowledge/notes/` | What the implementation found, hand-written, role `review`. Overrides a harvested entry. | | `docs/superpowers/specs/` + `plans/` | Per-phase design doc, task-by-task implementation plan, and a requirement-coverage checklist. | `docs/product-spec/appendix-c-consolidated-normative-requirement-index.md` is the fastest way to locate a @@ -193,27 +195,58 @@ requirement ID. ### Querying `docs/knowledge/` -`docs/knowledge/` is 39 topic files — never read a topic file whole when a filtered query -answers the question. `bun run knowledge` parses the corpus into entries and filters them; a requirement-ID +`docs/knowledge/` is two trees and 39 topics — never read a topic file whole when a filtered query +answers the question. `bun run knowledge` parses both trees into entries and filters them; a requirement-ID query returns ~170 tokens against a ~5700-token file read. ```bash bun run knowledge --req HTTP-13,HTTP-14,HTTP-15 # a whole task's IDs in one call (exact-token) +bun run knowledge --origin note --brief # start of a phase: everything the implementation found +bun run knowledge --prefix HTTP --section rules # an audit group: a whole ID family, uncapped bun run knowledge --chapter 6 interface class # a "styleguide 6.7" citation -bun run knowledge --section conflicts --brief # open design-vs-styleguide calls; 6 entries corpus-wide ``` Different filters AND together, values within one filter OR; `--help` lists the rest. Each result carries its `` provenance line — the citation for test-file headers and deferral notes, though styleguide paths are -absolute to a sibling repo and need their machine prefix stripped first. **A `--req` hit is not proof of -knowledge:** 255 of 645 IDs are named only by an appendix-B conformance roll-up, tagged `[appendix-B roll-up]` -in output; 386 have a substantive entry and 4 are cited nowhere at all (`--coverage` breaks this down). 15 of -the 39 topics carry no requirement ID at all and are reachable only via `--topic`/`--chapter` +absolute to a sibling repo and need their machine prefix stripped first — and a stable key, `/<8 hex>`, +digested from the entry text, which is how a note names the rule it overrides. **A `--req` hit is not proof of +knowledge:** 256 of 645 IDs are named only by an appendix-B conformance roll-up, tagged `[appendix-B roll-up]` +in output; 385 have a substantive entry and 4 are cited nowhere at all (`--coverage` breaks this down). 15 of +the 38 harvested topics carry no requirement ID at all and are reachable only via `--topic`/`--chapter` (`--list-topics`). Every count in this paragraph moves when the corpus is edited, so `scripts/knowledge.test.mjs` pins all four against the live corpus and its failure message names the two docs to update alongside. No CI step gates corpus *content*; CI does run the CLI's own suite (`test:scripts`), which parses the real corpus. The `.claude/skills/knowledge-lookup` skill carries the full workflow. +**Two trees, and the split is a CI gate.** `docs/knowledge/README.md` is the contract; the short version is +that `harvested/` is `knowledge-harvest`'s output and is never hand-edited, because a `` sha digests the +whole source file rather than the entry — an edit inside an entry changes no sha, and the next harvest +regenerates or duplicates it. Record what the implementation found in `docs/knowledge/notes/.md` +instead: role `review`, a manual `sha:` marker, and a backticked `/<8 hex>` key naming the harvested +rule, which makes that rule print `[overridden by notes/…]` in every query result. + +`bun run verify:knowledge-structure` (blocking, in CI) keeps the trees apart: no `review` or invented role +under `harvested/`, no `Superseded` entry there, every harvested entry carrying a `` that cites one of +the three source roots `SOURCES.md` names, every note carrying `review`, and no `.md` stranded at the root of +`docs/knowledge/` — the CLI reads the two trees only, so a file there is invisible rather than wrong, and that +is exactly where a `--corpus`-less harvest run lands. + +`bun run knowledge:drift` is the hand-run companion, deliberately not in CI. It reports source drift +(`OK` / `DRIFT` / `NOT VERIFIABLE` / `UNREADABLE` per `SOURCES.md` row) and stale note citations (a key no +entry carries any more). Not a gate: the styleguide root is a sibling repository at an absolute path, so 16 of +the 47 sources are absent from any CI checkout, and drift is normal — a design chapter a phase edits to record +an outcome *should* drift, and the fix is a re-harvest. + +**A register is not harvested.** `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md` +and `docs/open-items.md` are ledgers that every phase appends to, so any harvest of one is a stale fraction of +it. Read them directly; `notes/deliberate-deviations.md` is the pointer. When you re-harvest, point the skill +at the harvested tree — `--corpus docs/knowledge/harvested` — and hand-move any `supersede` entry it emits +into `notes/`. + +**Citations into the corpus** are written `docs/knowledge/harvested/.md:`. `docs/superpowers/` +is the exception: its plans and specs are dated records of what was true when they were written, are never +retro-edited, and so still carry pre-split paths. + ## Requirement-ID conventions (enforced by review, not tooling) - Every source file opens with `// SPDX-License-Identifier: MIT` on **line 1** (NFR-13). diff --git a/docs/deviations.md b/docs/deviations.md index ea46c0e..d2205e3 100644 --- a/docs/deviations.md +++ b/docs/deviations.md @@ -20,9 +20,11 @@ linked the two, and the numbering they share had no stated owner): `file:line` evidence, and records which of §10's claims did not survive contact with the code. - **A new deviation is recorded in neither.** It goes in the owning phase spec's own `## Deviation Ledger (for Phase 10)` section; §10 is the consolidated **output** of those, not their intake. -- **Do not confuse either with `docs/knowledge/deliberate-deviations.md`.** Despite the near-identical name it - is a harvested corpus topic file queried by `bun run knowledge`, derived from an older revision of §10 and - currently stale. It is `knowledge-harvest`'s output and is never hand-edited as a ledger. +- **The corpus no longer carries a copy.** `docs/knowledge/deliberate-deviations.md` was a harvested topic + file derived from an older revision of §10 — a third of the register, mis-anchored, two entries false. It + was dropped on 2026-08-31: a register accumulates rows and a harvest of it is one stale revision, so §10 is + read directly. What remains under the corpus is a pointer, `docs/knowledge/notes/deliberate-deviations.md`, + which says exactly that. Audited 2026-08-29 against `25-phase-10-deviation-reconciliation` @ `d8217af`; the audit's own changes landed on that branch as `27fb81f`, which is the tree this file describes. diff --git a/docs/knowledge/README.md b/docs/knowledge/README.md new file mode 100644 index 0000000..caf89ce --- /dev/null +++ b/docs/knowledge/README.md @@ -0,0 +1,31 @@ +# docs/knowledge/ + +Two trees, one query surface. `bun run knowledge` reads both; nothing else should read either by hand. + +| Tree | Holds | Rule | +| --- | --- | --- | +| `harvested/` | What the source documents say. Roles `spec`, `design`, `styleguide`. | Generated by the `knowledge-harvest` skill. **Never hand-edited.** | +| `notes/` | What the implementation found. Role `review`, a manual `sha:` marker. | Hand-written. A note overrides the harvested entry it names. | + +**Why `harvested/` is never hand-edited.** A `` line's sha digests the whole *source file*, not the +entry, so every entry harvested from one document carries the same value and an edit to an entry's text +changes no sha. The next harvest cannot see the edit: it regenerates the original text, or writes a +duplicate. A correction made in place is therefore scheduled for silent deletion. Make it in `notes/`. + +**How a note names its target.** Every entry has a stable key, `/<8 hex>`, digested from the entry's +own text. A note cites that key; the CLI resolves it, tags the harvested entry `[overridden by notes/…]`, +and `bun run knowledge --key ` looks one up. The key changes exactly when the rule's text does — so a +re-harvest that rewords a rule breaks the citation on purpose, and `bun run knowledge:drift` reports it. + +**A register is not harvested.** `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md` +and `docs/open-items.md` are ledgers every phase appends to; any harvest of one is a stale fraction of it. +Read them directly. `notes/deliberate-deviations.md` is the pointer. + +**Gates.** `bun run verify:knowledge-structure` is blocking in CI and keeps the two trees apart. +`bun run knowledge:drift` is a hand-run report over source shas and note citations; it never fails a build. + +Re-harvesting: `--corpus docs/knowledge/harvested`, always. The skill's default is this directory, which no +query reads — the structure gate rejects a topic file stranded here for that reason. A `supersede` +resolution the harvest emits must be moved to `notes/` by hand, or the gate rejects it too. + +The workflow for reading and writing all of this is `.claude/skills/knowledge-lookup/SKILL.md`. diff --git a/docs/knowledge/deliberate-deviations.md b/docs/knowledge/deliberate-deviations.md deleted file mode 100644 index cc051c3..0000000 --- a/docs/knowledge/deliberate-deviations.md +++ /dev/null @@ -1,62 +0,0 @@ -# deliberate-deviations - -> **STALE — do not treat these entries as current (flagged 2026-08-30, Phase 10).** Every entry below was -> harvested on 2026-07-25 from the **pre-implementation** revision of -> `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md` (`05d649a`) — the 12-item -> *prediction*, not the as-built ledger. That source has been rewritten and then corrected repeatedly since: -> Phase 10's reconciliation replaced the 12 predictions with a 16-item as-built ledger (`293f2e5`), Phase 8 -> added item 17 for `TransportFailureError`'s third error-tree level (`a0d734d`), and Phase 10's as-built audit -> corrected items 4, 7, 11 and 14 against source (`27fb81f`). Consequences for a `bun run knowledge` consumer: -> -> - **The `sha:f9ecb6e7d87b` pin on every `` line matches only `05d649a`, the oldest revision in that -> file's history.** The source file's current sha-256 prefix is `301f1d519cd8`. -> - **Every `file:line` anchor below is wrong.** They point into a 62-line file; the source is now 238 lines and -> restructured, so the ranges resolve to unrelated text. -> - **Two entries are substantively false**, not merely mis-anchored — the structural-interfaces mitigation and -> the "three tiers, not four" configuration claim. Both carry an inline `**Stale…**` marker below so the -> correction travels with the entry in CLI output. -> -> **Trigger for removing this banner:** a `knowledge-harvest` run over -> `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md`, re-pinning -> `docs/knowledge/SOURCES.md` and the `` lines to the current sha and refreshing -> `docs/knowledge/INDEX.md`'s row. Phase 10 did **not** re-harvest: `knowledge-harvest` is explicitly -> user-invoked only, which its own design doc records as a scope boundary. Until then, read §10 and -> `docs/deviations.md` directly — see §10's own three-file disambiguation table. - -## Rules - -## Constraints - -## Conclusions -- None of the port's Node-idiomatic mechanism substitutions narrow a MUST-level correctness guarantee from the reference contract; each substitutes an equivalent, differently-shaped mechanism where the JVM-specific one the requirement was worded around does not exist in Node. - design · `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md:3-6` · high · sha:f9ecb6e7d87b -- The synchronous transport seam and the asynchronous transport seam collapse into a single `Promise`-returning `Transport.send()` satisfying both requirements' letter simultaneously, because Node has no blocking-I/O execution model to give the sync/async distinction meaning. - design · `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md:8-12` · high · sha:f9ecb6e7d87b -- The byte-stream provider seam is no longer pluggable because Web Streams are a runtime standard rather than a third-party library, so `@dexpace/core` implements the byte-stream contracts directly with no discovery/installation machinery. - design · `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md:13-16` · high · sha:f9ecb6e7d87b -- Async-runtime adapter fragmentation does not exist in the port because `Promise` is Node's only ecosystem-wide async primitive, so no bridge modules equivalent to the JVM reference's coroutine/reactor/netty/virtual-threads adapters are shipped, and the one optional adapter shipped (`@dexpace/rx`) is sugar over a genuinely different push-based data shape rather than plumbing for the request/response pivot. - design · `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md:17-21` · high · sha:f9ecb6e7d87b -- The two retry stacks collapse into one, with the total-timeout budget made explicitly opt-in, a substitution the spec itself sanctions by requiring that a port that unifies retry entry points MUST make that budget explicitly opt-in. - design · `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md:22-24` · high · sha:f9ecb6e7d87b -- True runtime encapsulation of domain models is not fully achievable because TypeScript's structural typing means a hand-built object literal can still impersonate a public interface type and bypass builder validation, even though ECMAScript `#private` fields close the "official construction path" hole; this acknowledged language-level limitation is mitigated, not eliminated, by exporting only concrete classes rather than bare structural interfaces from each package's public entry point. **Stale (Phase 10 audit, 2026-08-30): the limitation holds; the mitigation clause is false as stated.** §10 item 4 was corrected against the API report — `packages/core/etc/core.api.md` exports 61 interfaces against 58 classes, and `Configuration` is a builder-built type exported as a bare interface and accepted structurally by `setGlobalConfiguration()`. The mitigation is real for the `http/` wire models it was written about, not a package-wide property. See `docs/deviations.md` §4. - design · `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md:25-30` · high · sha:f9ecb6e7d87b -- The generic-erasure defense uses schema-as-witness rather than reflective type capture, because TypeScript erases types more completely than JVM generic erasure and leaves no raw class token to reflect over, and this substitution is argued to be at least as strong a guarantee, not a weaker one. - design · `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md:31-35` · high · sha:f9ecb6e7d87b -- Single-threaded execution collapses the JVM reference's atomic compare-and-set guard for the materialize-once body race into a synchronous check-and-set, correct only if the guard executes before the guarded async function's first `await`, a precondition stated explicitly because it is the one place the simplification could be silently misapplied. - design · `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md:36-40` · high · sha:f9ecb6e7d87b -- Digest MD5 needs a vendored implementation while SHA-256 does not, because the Web Crypto API deliberately excludes MD5, so the port vendors a small, dependency-free MD5 implementation for RFC 7616 interoperability and uses `crypto.subtle` directly for SHA-256/SHA-256-sess. - design · `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md:41-43` · high · sha:f9ecb6e7d87b -- Configuration layering has three tiers rather than four because the system-property tier is lost outright, Node having no ambient key/value store distinct from environment variables to fill that slot, and the port does not fabricate one. **Stale (Phase 10 audit, 2026-08-30): false as stated.** §10 item 7 was corrected against the code — all four of `CFG-1`'s tiers are implemented, and the property layer is a first-class caller-supplyable seam (`ConfigurationBuilder.withPropertySource()` and `getRawProperty()` are both public API), so a host that *does* have an ambient store can bind it. What deviates is only the default production wiring: `defaultConfiguration()` binds a property source that always returns `undefined`. See §10 item 7; this item is not in `docs/deviations.md` precisely because it was correctable and was corrected. - design · `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md:44-46` · high · sha:f9ecb6e7d87b -- Cancellation is `AbortController`/`AbortSignal` end-to-end rather than an interrupt-and-restore-a-flag discipline, composing the same signal type across the transport call, the retry backoff wait, and a derived per-call timeout; since `Promise` has no public `cancel()` unlike `CompletableFuture`, cancellation is cooperative end-to-end, and a `send()` implementation must check `signal.aborted` after resuming from an `await` before treating a resolved value as deliverable. - design · `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md:47-53` · high · sha:f9ecb6e7d87b -- Frozen collections are computed once rather than wrapped on every read, satisfied by `Object.freeze`-ing each collection exactly once at construction and returning the same frozen reference from every subsequent getter call, cheaper than the reference's per-access unmodifiable-wrapper pattern because the port's models never change after construction. - design · `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md:54-57` · high · sha:f9ecb6e7d87b -- The dead-code-survival gate targets a different risk than the JVM reference, since JS bundlers have no reflection blind spot to guard against, so `@dexpace/shrink-test` instead targets the dual-package hazard of two copies of `@dexpace/core` breaking cross-package `instanceof` checks after a bundle-and-tree-shake round trip. - design · `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md:58-62` · high · sha:f9ecb6e7d87b - -## Reference - -## Conflicts - -## Superseded diff --git a/docs/knowledge/INDEX.md b/docs/knowledge/harvested/INDEX.md similarity index 73% rename from docs/knowledge/INDEX.md rename to docs/knowledge/harvested/INDEX.md index 5511baa..3cde49d 100644 --- a/docs/knowledge/INDEX.md +++ b/docs/knowledge/harvested/INDEX.md @@ -1,5 +1,7 @@ # Knowledge Index +Generated. The two-tree contract, and why nothing here is hand-edited, is in `../README.md`. + | topic | file | entries | roles | conflicts | last harvest | | --- | --- | --- | --- | --- | --- | | api-design | `api-design.md` | 32 | styleguide | 0 | 2026-07-25 | @@ -10,7 +12,6 @@ | configuration | `configuration.md` | 51 | design, spec | 0 | 2026-07-25 | | cross-cutting-invariants | `cross-cutting-invariants.md` | 7 | spec | 0 | 2026-07-25 | | data-modeling | `data-modeling.md` | 27 | styleguide | 0 | 2026-07-25 | -| deliberate-deviations | `deliberate-deviations.md` | 13 | design | 0 | 2026-07-25 — **STALE, see below** | | documentation | `documentation.md` | 21 | styleguide | 0 | 2026-07-25 | | error-handling | `error-handling.md` | 43 | spec, styleguide | 0 | 2026-07-25 | | execution-context | `execution-context.md` | 33 | spec | 0 | 2026-07-25 | @@ -22,7 +23,7 @@ | naming-conventions | `naming-conventions.md` | 32 | styleguide | 0 | 2026-07-25 | | observability | `observability.md` | 63 | design, spec | 0 | 2026-07-25 | | package-and-dependency-layout | `package-and-dependency-layout.md` | 31 | design, spec | 0 | 2026-07-25 | -| pagination | `pagination.md` | 61 | design, spec | 0 | 2026-07-25 | +| pagination | `pagination.md` | 61 | design, spec | 1 | 2026-07-25 | | performance | `performance.md` | 35 | styleguide | 0 | 2026-07-25 | | pipeline | `pipeline.md` | 83 | design, spec | 1 | 2026-07-25 | | redaction-and-security | `redaction-and-security.md` | 22 | spec | 0 | 2026-07-25 | @@ -42,15 +43,6 @@ | url-and-query-encoding | `url-and-query-encoding.md` | 20 | design, spec | 0 | 2026-07-25 | | variables-and-declarations | `variables-and-declarations.md` | 14 | styleguide | 0 | 2026-07-25 | -## Stale topics - -One row above is annotated rather than refreshed, because re-harvesting is `knowledge-harvest`'s job and that -skill is user-invoked only. Flagged 2026-08-30 by Phase 10. - -- **`deliberate-deviations`** — harvested from the **12-item pre-implementation prediction** in - `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md` (`05d649a`, 62 lines). That - source is now a 238-line, 17-item as-built ledger, corrected twice since (`a0d734d`, `27fb81f`). All 13 - entries are mis-anchored, the `sha:f9ecb6e7d87b` pin in `SOURCES.md` no longer matches (current: - `301f1d519cd8`), and two entries are substantively false — both carry an inline `**Stale…**` marker so the - correction reaches `bun run knowledge` output. Full detail in the topic file's own head banner. **Unblock:** - a `knowledge-harvest` run over that one source, which re-pins `SOURCES.md`, the `` lines, and this row. +38 topic files, 1457 entries — 1451 in the type sections plus +6 conflict statements. A conflict statement records what two documents each say; its +resolution, where one exists, is a note under `../notes/`. diff --git a/docs/knowledge/SOURCES.md b/docs/knowledge/harvested/SOURCES.md similarity index 90% rename from docs/knowledge/SOURCES.md rename to docs/knowledge/harvested/SOURCES.md index f14c5ab..95b8c43 100644 --- a/docs/knowledge/SOURCES.md +++ b/docs/knowledge/harvested/SOURCES.md @@ -1,5 +1,12 @@ # Harvested Sources +Every source `harvested/` is derived from, with the sha256 of the whole file at harvest time — per **file**, +never per entry, which is why an entry here cannot be hand-corrected. See `../README.md`. +`bun run knowledge:drift` compares each digest below against the file on disk; the 16 styleguide rows are +`NOT VERIFIABLE` off the harvest machine, which is expected and never a failure. +`docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md` was harvested once and +dropped on 2026-08-31: it is a register, not a description. + | source | role | sha256 | last harvest | | --- | --- | --- | --- | | `/home/mohammad/Projects/dexpace/styleguide/typescript/01-formatting-and-tooling.md` | styleguide | `640652667e83` | 2026-07-25 | @@ -49,4 +56,3 @@ | `docs/sdk-design-nodejs/07-pagination-sse-and-serialization.md` | design | `d546f9973c4e` | 2026-07-25 | | `docs/sdk-design-nodejs/08-instrumentation-and-configuration.md` | design | `35281a426195` | 2026-07-25 | | `docs/sdk-design-nodejs/09-toolchain-and-quality-gates.md` | design | `2d2fd9dcfee4` | 2026-07-25 | -| `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md` | design | `f9ecb6e7d87b` — **stale, source is now `301f1d519cd8`** | 2026-07-25 — **needs re-harvest**, see `INDEX.md`'s "Stale topics" | diff --git a/docs/knowledge/api-design.md b/docs/knowledge/harvested/api-design.md similarity index 100% rename from docs/knowledge/api-design.md rename to docs/knowledge/harvested/api-design.md diff --git a/docs/knowledge/assertions.md b/docs/knowledge/harvested/assertions.md similarity index 100% rename from docs/knowledge/assertions.md rename to docs/knowledge/harvested/assertions.md diff --git a/docs/knowledge/authentication.md b/docs/knowledge/harvested/authentication.md similarity index 100% rename from docs/knowledge/authentication.md rename to docs/knowledge/harvested/authentication.md diff --git a/docs/knowledge/cancellation-and-timeouts.md b/docs/knowledge/harvested/cancellation-and-timeouts.md similarity index 100% rename from docs/knowledge/cancellation-and-timeouts.md rename to docs/knowledge/harvested/cancellation-and-timeouts.md diff --git a/docs/knowledge/concurrency-and-async.md b/docs/knowledge/harvested/concurrency-and-async.md similarity index 100% rename from docs/knowledge/concurrency-and-async.md rename to docs/knowledge/harvested/concurrency-and-async.md diff --git a/docs/knowledge/configuration.md b/docs/knowledge/harvested/configuration.md similarity index 100% rename from docs/knowledge/configuration.md rename to docs/knowledge/harvested/configuration.md diff --git a/docs/knowledge/cross-cutting-invariants.md b/docs/knowledge/harvested/cross-cutting-invariants.md similarity index 100% rename from docs/knowledge/cross-cutting-invariants.md rename to docs/knowledge/harvested/cross-cutting-invariants.md diff --git a/docs/knowledge/data-modeling.md b/docs/knowledge/harvested/data-modeling.md similarity index 100% rename from docs/knowledge/data-modeling.md rename to docs/knowledge/harvested/data-modeling.md diff --git a/docs/knowledge/documentation.md b/docs/knowledge/harvested/documentation.md similarity index 100% rename from docs/knowledge/documentation.md rename to docs/knowledge/harvested/documentation.md diff --git a/docs/knowledge/error-handling.md b/docs/knowledge/harvested/error-handling.md similarity index 100% rename from docs/knowledge/error-handling.md rename to docs/knowledge/harvested/error-handling.md diff --git a/docs/knowledge/execution-context.md b/docs/knowledge/harvested/execution-context.md similarity index 100% rename from docs/knowledge/execution-context.md rename to docs/knowledge/harvested/execution-context.md diff --git a/docs/knowledge/function-design.md b/docs/knowledge/harvested/function-design.md similarity index 100% rename from docs/knowledge/function-design.md rename to docs/knowledge/harvested/function-design.md diff --git a/docs/knowledge/http-domain-model.md b/docs/knowledge/harvested/http-domain-model.md similarity index 100% rename from docs/knowledge/http-domain-model.md rename to docs/knowledge/harvested/http-domain-model.md diff --git a/docs/knowledge/io-and-byte-streams.md b/docs/knowledge/harvested/io-and-byte-streams.md similarity index 100% rename from docs/knowledge/io-and-byte-streams.md rename to docs/knowledge/harvested/io-and-byte-streams.md diff --git a/docs/knowledge/message-bodies.md b/docs/knowledge/harvested/message-bodies.md similarity index 100% rename from docs/knowledge/message-bodies.md rename to docs/knowledge/harvested/message-bodies.md diff --git a/docs/knowledge/module-organization.md b/docs/knowledge/harvested/module-organization.md similarity index 100% rename from docs/knowledge/module-organization.md rename to docs/knowledge/harvested/module-organization.md diff --git a/docs/knowledge/naming-conventions.md b/docs/knowledge/harvested/naming-conventions.md similarity index 100% rename from docs/knowledge/naming-conventions.md rename to docs/knowledge/harvested/naming-conventions.md diff --git a/docs/knowledge/observability.md b/docs/knowledge/harvested/observability.md similarity index 100% rename from docs/knowledge/observability.md rename to docs/knowledge/harvested/observability.md diff --git a/docs/knowledge/package-and-dependency-layout.md b/docs/knowledge/harvested/package-and-dependency-layout.md similarity index 100% rename from docs/knowledge/package-and-dependency-layout.md rename to docs/knowledge/harvested/package-and-dependency-layout.md diff --git a/docs/knowledge/pagination.md b/docs/knowledge/harvested/pagination.md similarity index 93% rename from docs/knowledge/pagination.md rename to docs/knowledge/harvested/pagination.md index 4311f3c..05c07a3 100644 --- a/docs/knowledge/pagination.md +++ b/docs/knowledge/harvested/pagination.md @@ -127,13 +127,11 @@ spec · `docs/product-spec/appendix-b-conformance-test-checklist.md:15` · high · sha:0451cc7f3bb4 - The pagination conformance suite verifies that per-call options reach every page exchange (PAGE-36). spec · `docs/product-spec/appendix-b-conformance-test-checklist.md:16` · high · sha:0451cc7f3bb4 -- The port's item-level generator wraps `yield* page.items` in a `finally` block that awaits `page.close()`, so an early `break` in the consumer's loop automatically triggers page closure with no wrapper type or documented convention required. **Erratum (Phase 6c, 2026-07-28): correct about the `.return()`-on-abandon mechanism, wrong about close ordering — see Conflicts below.** +- The port's item-level generator wraps `yield* page.items` in a `finally` block that awaits `page.close()`, so an early `break` in the consumer's loop automatically triggers page closure with no wrapper type or documented convention required. design · `docs/sdk-design-nodejs/07-pagination-sse-and-serialization.md:19-31` · high · sha:d546f9973c4e ## Conflicts -- **Item-view close ordering.** The Rules entry above (`PAGE-11`, `docs/product-spec/12-pagination.md:38`) requires the item-level view to eager-close each page **before** yielding any of that page's items, after copying them. The Reference entry above (`sdk-design-nodejs/07` §7.1) shows the opposite ordering — `yield*` inside a `try`, `close()` in the `finally` — which holds the response open for the entire time a consumer walks that page's items. **`PAGE-11` governs**, per the standing tie-breaker that the normative spec wins over an illustrative snippet; the cost is zero because `PAGE-2` guarantees materialized items survive close. Resolved in Phase 6c, which implements copy-items → close → yield and writes the erratum into `sdk-design-nodejs/07` §7.1. - - The reason this needed recording rather than silently correcting: **the conformance test is weaker than the requirement.** Appendix B's `PAGE-11` check ("take one item from a multi-item first page and stop; assert the first page's response was closed") *passes* under the snippet's ordering, because an early `break` drives `.return()` and therefore the `finally`. Following the design doc would have shipped a MUST violation the checklist could not catch. Phase 6c's `lifecycle.test.ts` adds the assertion appendix B does not make — that the close is observed *before* the first item is yielded. - review · `docs/superpowers/specs/2026-07-28-phase6c-pagination-design.md` · high · sha:manual-6c-erratum +- **spec vs design: item-view close ordering** — `PAGE-11` requires the item-level view to eager-close each page **before** yielding any of that page's items, after copying them. The §7.1 snippet shows the opposite ordering — `yield*` inside a `try`, `close()` in the `finally` — which holds the response open for the entire time a consumer walks that page's items. + spec `docs/product-spec/12-pagination.md:38` · design `docs/sdk-design-nodejs/07-pagination-sse-and-serialization.md:19-31` · resolved 2026-07-28 ## Superseded diff --git a/docs/knowledge/performance.md b/docs/knowledge/harvested/performance.md similarity index 100% rename from docs/knowledge/performance.md rename to docs/knowledge/harvested/performance.md diff --git a/docs/knowledge/pipeline.md b/docs/knowledge/harvested/pipeline.md similarity index 100% rename from docs/knowledge/pipeline.md rename to docs/knowledge/harvested/pipeline.md diff --git a/docs/knowledge/redaction-and-security.md b/docs/knowledge/harvested/redaction-and-security.md similarity index 100% rename from docs/knowledge/redaction-and-security.md rename to docs/knowledge/harvested/redaction-and-security.md diff --git a/docs/knowledge/redirect-handling.md b/docs/knowledge/harvested/redirect-handling.md similarity index 100% rename from docs/knowledge/redirect-handling.md rename to docs/knowledge/harvested/redirect-handling.md diff --git a/docs/knowledge/resource-management.md b/docs/knowledge/harvested/resource-management.md similarity index 100% rename from docs/knowledge/resource-management.md rename to docs/knowledge/harvested/resource-management.md diff --git a/docs/knowledge/retry-and-resilience.md b/docs/knowledge/harvested/retry-and-resilience.md similarity index 100% rename from docs/knowledge/retry-and-resilience.md rename to docs/knowledge/harvested/retry-and-resilience.md diff --git a/docs/knowledge/sdk-positioning.md b/docs/knowledge/harvested/sdk-positioning.md similarity index 100% rename from docs/knowledge/sdk-positioning.md rename to docs/knowledge/harvested/sdk-positioning.md diff --git a/docs/knowledge/seams-and-extensibility.md b/docs/knowledge/harvested/seams-and-extensibility.md similarity index 100% rename from docs/knowledge/seams-and-extensibility.md rename to docs/knowledge/harvested/seams-and-extensibility.md diff --git a/docs/knowledge/serde.md b/docs/knowledge/harvested/serde.md similarity index 100% rename from docs/knowledge/serde.md rename to docs/knowledge/harvested/serde.md diff --git a/docs/knowledge/sse-streaming.md b/docs/knowledge/harvested/sse-streaming.md similarity index 100% rename from docs/knowledge/sse-streaming.md rename to docs/knowledge/harvested/sse-streaming.md diff --git a/docs/knowledge/styleguide-overview.md b/docs/knowledge/harvested/styleguide-overview.md similarity index 100% rename from docs/knowledge/styleguide-overview.md rename to docs/knowledge/harvested/styleguide-overview.md diff --git a/docs/knowledge/testing.md b/docs/knowledge/harvested/testing.md similarity index 100% rename from docs/knowledge/testing.md rename to docs/knowledge/harvested/testing.md diff --git a/docs/knowledge/tooling-and-quality-gates.md b/docs/knowledge/harvested/tooling-and-quality-gates.md similarity index 91% rename from docs/knowledge/tooling-and-quality-gates.md rename to docs/knowledge/harvested/tooling-and-quality-gates.md index f646ee1..2e3e011 100644 --- a/docs/knowledge/tooling-and-quality-gates.md +++ b/docs/knowledge/harvested/tooling-and-quality-gates.md @@ -117,11 +117,11 @@ design · `docs/sdk-design-nodejs/09-toolchain-and-quality-gates.md:37-41` · high · sha:2d2fd9dcfee4 ## Conflicts -- **design vs styleguide: package manager and lockfile** — RESOLVED in favor of the styleguide (2026-07-25, confirmed 2026-07-28 Phase 9 audit). The scaffold implements Bun (`bun.lock`, `.bun-version`, `bun install --frozen-lockfile` as the CI gate) throughout; the design's pnpm/`catalog:` framing describes a toolchain this repository does not use. Decision recorded at `docs/superpowers/plans/2026-07-23-scaffold-milestone-checklist.md:54`; the enforcement properties pnpm's layout gave for free (isolated linker, workspace catalogs) were restored separately — see the Bun workspace catalogs adopted in Phase 6a and the isolated linker set at the 2026-07-25 checkpoint. - design `docs/sdk-design-nodejs/02-package-and-workspace-layout.md:50-51` · styleguide `/home/mohammad/Projects/dexpace/styleguide/typescript/01-formatting-and-tooling.md:112-120` · resolved 2026-07-25, backported 2026-07-28 -- **design vs styleguide: test runner and whether coverage gates the build** — RESOLVED as a split (2026-07-25, confirmed 2026-07-28 Phase 9 audit). Runner: `bun test` with `bun:test` symbol imports (the styleguide's choice) — the design's `c8`/`vitest` framing is dead. Gating: `NFR-5`/`NFR-17` are spec conformance obligations that outrank the styleguide's general "coverage is a trend, never a pass/fail gate" default; `bunfig.toml`'s `coverageThreshold = 0.8` blocks the build, as the scaffold's own plan already implemented. - design `docs/sdk-design-nodejs/09-toolchain-and-quality-gates.md:12` · styleguide `/home/mohammad/Projects/dexpace/styleguide/typescript/11-testing.md:47-48,210-213` · resolved 2026-07-25, backported 2026-07-28 -- **design vs styleguide: gts as the lint and format baseline** — RESOLVED in favor of the styleguide (2026-07-25, confirmed 2026-07-28 Phase 9 audit). The plans extend `gts` in `eslint.config.js` and layer `@typescript-eslint`'s `strict-type-checked`/`stylistic-type-checked` tiers on top as the single permitted overlay, satisfying the design's rule set as well; the design's table never mentioning `gts` describes a toolchain this repository does not use. - design `docs/sdk-design-nodejs/09-toolchain-and-quality-gates.md:8-10` · styleguide `/home/mohammad/Projects/dexpace/styleguide/typescript/01-formatting-and-tooling.md:66-83,122-129` · resolved 2026-07-25, backported 2026-07-28 +- **design vs styleguide: package manager and lockfile** — the design puts every version and tooling coordinate in one place via pnpm's `catalog:` protocol (pnpm ≥9), the direct analog of `gradle/libs.versions.toml` (**NFR-14**), and leans on pnpm's dedupe to keep one `@dexpace/core` instance; the styleguide mandates Bun — a committed `bun.lock`, `bun install --frozen-lockfile` as the CI gate, and a committed `.bun-version` — and is explicit that Bun's default flat `node_modules` buys reproducibility, not pnpm-style isolation. + design `docs/sdk-design-nodejs/02-package-and-workspace-layout.md:50-51` · styleguide `/home/mohammad/Projects/dexpace/styleguide/typescript/01-formatting-and-tooling.md:112-120` · resolved 2026-07-25 +- **design vs styleguide: test runner and whether coverage gates the build** — the design's gate table maps Kover's 80% aggregate line-coverage floor onto `c8`/`@vitest/coverage-v8` with a `coverage.thresholds` aggregate floor wired into the default `test` script; the styleguide makes `bun test` the runner and not a per-project choice, with `bun:test` symbols imported explicitly, and reports coverage as a floor and a trend, never as a pass/fail target. + design `docs/sdk-design-nodejs/09-toolchain-and-quality-gates.md:12` · styleguide `/home/mohammad/Projects/dexpace/styleguide/typescript/11-testing.md:47-48,210-213` · resolved 2026-07-25 +- **design vs styleguide: gts as the lint and format baseline** — the design's gate table names ESLint with `@typescript-eslint`'s `strict-type-checked` and `stylistic-type-checked` configs and never mentions `gts`; the styleguide makes `gts` the whole toolchain, bans a standalone Prettier or ESLint config, and takes gts's Prettier defaults as final. + design `docs/sdk-design-nodejs/09-toolchain-and-quality-gates.md:8-10` · styleguide `/home/mohammad/Projects/dexpace/styleguide/typescript/01-formatting-and-tooling.md:66-83,122-129` · resolved 2026-07-25 ## Superseded diff --git a/docs/knowledge/transport-adapter.md b/docs/knowledge/harvested/transport-adapter.md similarity index 100% rename from docs/knowledge/transport-adapter.md rename to docs/knowledge/harvested/transport-adapter.md diff --git a/docs/knowledge/type-system.md b/docs/knowledge/harvested/type-system.md similarity index 100% rename from docs/knowledge/type-system.md rename to docs/knowledge/harvested/type-system.md diff --git a/docs/knowledge/typescript-idioms.md b/docs/knowledge/harvested/typescript-idioms.md similarity index 100% rename from docs/knowledge/typescript-idioms.md rename to docs/knowledge/harvested/typescript-idioms.md diff --git a/docs/knowledge/url-and-query-encoding.md b/docs/knowledge/harvested/url-and-query-encoding.md similarity index 100% rename from docs/knowledge/url-and-query-encoding.md rename to docs/knowledge/harvested/url-and-query-encoding.md diff --git a/docs/knowledge/variables-and-declarations.md b/docs/knowledge/harvested/variables-and-declarations.md similarity index 100% rename from docs/knowledge/variables-and-declarations.md rename to docs/knowledge/harvested/variables-and-declarations.md diff --git a/docs/knowledge/notes/deliberate-deviations.md b/docs/knowledge/notes/deliberate-deviations.md new file mode 100644 index 0000000..a5dde3d --- /dev/null +++ b/docs/knowledge/notes/deliberate-deviations.md @@ -0,0 +1,7 @@ +# deliberate-deviations — notes + +Hand-written. There is no harvested counterpart to this file, deliberately: see the entry below. + +## Reference +- **The deviation register is not harvested. Read the register itself, at the start of every phase.** `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md` is a numbered ledger that each phase appends to, and a harvest of a ledger is a snapshot of one revision that goes stale on the next append. The snapshot this corpus used to carry proved the point: 13 entries against a 17-item register, roughly a third of it, mis-anchored, two entries substantively false, and pinned to a sha three revisions old. A description of an approach is harvestable because it changes slowly; a register is not. `docs/deviations.md` is the as-built audit of that same ledger, and `docs/open-items.md` is the second register under the same rule — read them, do not expect `bun run knowledge` to know them. + review · `docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md` · high · sha:manual-register-pointer diff --git a/docs/knowledge/notes/pagination.md b/docs/knowledge/notes/pagination.md new file mode 100644 index 0000000..41b9ce3 --- /dev/null +++ b/docs/knowledge/notes/pagination.md @@ -0,0 +1,11 @@ +# pagination — notes + +Hand-written. `docs/knowledge/harvested/pagination.md` is what the documents say; this file is what +the implementation found, and it wins. Each entry names the harvested entry it answers by that +entry's stable key. + +## Superseded +- **Item-view close ordering: `PAGE-11` governs, and the `sdk-design-nodejs/07` §7.1 snippet does not.** Supersedes `pagination/81881061` (the Reference entry describing the snippet) and resolves the conflict statement `pagination/d108714e`. The item-level view copies the page's items, closes the page, and only then yields — never the snippet's `yield*` inside a `try` with `close()` in the `finally`. The standing tie-breaker applies: a normative MUST beats an illustrative snippet, and the cost is zero because `PAGE-2` guarantees materialized items survive close. Phase 6c implements copy-items → close → yield and wrote the erratum into `sdk-design-nodejs/07` §7.1. + + The reason this needed recording rather than silently correcting: **the conformance test is weaker than the requirement.** Appendix B's `PAGE-11` check ("take one item from a multi-item first page and stop; assert the first page's response was closed") *passes* under the snippet's ordering, because an early `break` drives `.return()` and therefore the `finally`. Following the design doc would have shipped a MUST violation the checklist could not catch. Phase 6c's `lifecycle.test.ts` adds the assertion appendix B does not make — that the close is observed *before* the first item is yielded. + review · `docs/superpowers/specs/2026-07-28-phase6c-pagination-design.md` · high · sha:manual-6c-erratum diff --git a/docs/knowledge/notes/tooling-and-quality-gates.md b/docs/knowledge/notes/tooling-and-quality-gates.md new file mode 100644 index 0000000..fe156e0 --- /dev/null +++ b/docs/knowledge/notes/tooling-and-quality-gates.md @@ -0,0 +1,13 @@ +# tooling-and-quality-gates — notes + +Hand-written. `docs/knowledge/harvested/tooling-and-quality-gates.md` states each design-vs-styleguide +contradiction; this file records how the repository resolved it, and it wins. Each entry names the +harvested statement it resolves by that statement's stable key. + +## Conflicts +- **Package manager and lockfile: the styleguide wins.** Resolves `tooling-and-quality-gates/4f1a46a5`. The scaffold implements Bun throughout — `bun.lock`, `.bun-version`, and `bun install --frozen-lockfile` as the CI gate; the design's pnpm/`catalog:` framing describes a toolchain this repository does not use. The two enforcement properties pnpm's layout gave for free were restored separately: Bun workspace catalogs in Phase 6a, and the isolated linker at the 2026-07-25 checkpoint. Decided 2026-07-25, confirmed by the 2026-07-28 Phase 9 audit. + review · `docs/superpowers/plans/2026-07-23-scaffold-milestone-checklist.md:54` · high · sha:manual-2026-07-25-package-manager +- **Test runner and coverage gating: a split decision.** Resolves `tooling-and-quality-gates/99637a28`. Runner: `bun test` with `bun:test` symbol imports, the styleguide's choice — the design's `c8`/`vitest` framing is dead. Gating: the styleguide's general "coverage is a trend, never a pass/fail gate" default loses here, because `NFR-5`/`NFR-17` are spec conformance obligations that outrank a general style default; `bunfig.toml`'s `coverageThreshold = 0.8` blocks the build. Decided 2026-07-25, confirmed by the 2026-07-28 Phase 9 audit. + review · `bunfig.toml` · high · sha:manual-2026-07-25-test-runner +- **gts as the lint and format baseline: the styleguide wins.** Resolves `tooling-and-quality-gates/90367d73`. `eslint.config.js` extends `gts` and layers `@typescript-eslint`'s `strict-type-checked` and `stylistic-type-checked` tiers on top as the single permitted overlay, which satisfies the design's rule set as well; the design's table never mentioning `gts` describes a toolchain this repository does not use. The corollary is load-bearing and easy to undo by accident: no root Prettier config, so `eslint.config.js` sources `gts/.prettierrc.json` itself. Decided 2026-07-25, confirmed by the 2026-07-28 Phase 9 audit. + review · `eslint.config.js` · high · sha:manual-2026-07-25-gts-baseline diff --git a/docs/open-items.md b/docs/open-items.md index f60d4de..3949976 100644 --- a/docs/open-items.md +++ b/docs/open-items.md @@ -125,7 +125,7 @@ Two ways out, both defensible: 1. **Give default keys a distinguishing description** — `Symbol('dispatch-context#' + n)` from a module-scoped counter. The counter would label only the description; `Symbol()` remains the identity, so CTX-4/5/6's uniqueness is untouched and the ledger's rejection of a `traceId:spanId`+counter *string key* still stands. - Costs a second module-level mutable binding (`docs/knowledge/variables-and-declarations.md:22`), on top of + Costs a second module-level mutable binding (`docs/knowledge/harvested/variables-and-declarations.md:22`), on top of the `contextStore` singleton that already takes that deviation. 2. **Record a deliberate partial deviation** in the Phase 4a design's Deviation Ledger, on the grounds that a symbol has no unique rendering and the typed `.key` field identifies the key more precisely than a message @@ -343,7 +343,7 @@ Phase 5's retry step being the first real third-party-shaped consumer. ### F3 — Zero `invariant()` assertions across `recovery/` — **SCHEDULED** (Phase 10) -`docs/knowledge/assertions.md:6-7` sets a 2-per-function module average; this phase ships none across roughly a +`docs/knowledge/harvested/assertions.md:6-7` sets a 2-per-function module average; this phase ships none across roughly a dozen functions. Project-wide inconsistency rather than 4b's — Phases 1/2/3b/4a ship zero, 4c's plan ships fifteen — so adding them to 4b alone would deepen the split. Recorded in the phase design's Deviation Ledger. @@ -376,7 +376,7 @@ step lands without giving it behavior, inline it there and carry the disposition `suppress()` returns the native `SuppressedError` where the runtime has one and `FallbackSuppressedError` where it does not. No test forces the other branch by deleting the global — that cannot survive parallel execution -(`docs/knowledge/testing.md:50`). Coverage comes from the `test:node` matrix instead: `lts/*` exercises the +(`docs/knowledge/harvested/testing.md:50`). Coverage comes from the `test:node` matrix instead: `lts/*` exercises the native branch, the pinned `20.3.0` exercises the fallback. **Trigger:** if the matrix ever collapses to one runtime, or the floor rises past Node 24 (where the fallback becomes dead code to be deleted, not guarded). @@ -706,10 +706,10 @@ reachable through the public barrels, which `index.public.test.ts` and `cross-pa ### H6 — assertion density — **DEFERRED to Phase 10, project-wide** This phase ships one `invariant()` call across roughly fifteen functions, against -`docs/knowledge/assertions.md:6-7`'s 2-per-function module average. Phase 4b raised the identical gap and +`docs/knowledge/harvested/assertions.md:6-7`'s 2-per-function module average. Phase 4b raised the identical gap and resolved it to a ledger row rather than fixing 4b alone, on the grounds that the split is project-wide (Phases 1/2/3b/4a ship zero, 4c ships fifteen) and half-migrating it is what -`docs/knowledge/styleguide-overview.md:32-33` forbids. 6a follows 4b, deliberately. +`docs/knowledge/harvested/styleguide-overview.md:32-33` forbids. 6a follows 4b, deliberately. **Trigger:** Phase 10, which settles the density rule once. @@ -827,7 +827,7 @@ The positional form is what the plan's Task 2 Interfaces block specifies and it — three parameters, inside `max-params`, and an SPI a third-party codec *implements*, where a positional shape is the smaller burden. The object form exists because positionally the handlers would be four parameters, which is a lint error. So each layer's choice is locally right and the pair is globally inconsistent: a codec author -implements one spelling, a caller uses the other. `docs/knowledge/api-design.md:14` ("optional parameters +implements one spelling, a caller uses the other. `docs/knowledge/harvested/api-design.md:14` ("optional parameters collected into a single options object rather than a positional list past two parameters") points at the object form for both. @@ -862,7 +862,7 @@ mid-implementation. Raised in the Phase 6a shape review as F11. ### H12 — `seams/index.ts` is an unimported internal barrel — **OPEN** -`docs/knowledge/module-organization.md:18` bans internal folder-level barrels outright ("never create internal +`docs/knowledge/harvested/module-organization.md:18` bans internal folder-level barrels outright ("never create internal barrels; import the specific file directly"), and `api-design.md:6` makes `index.ts` the package's *sole* barrel. `packages/core/src/seams/index.ts` is one, from Phase 2, and 6a grew it by three re-exports (`Deserializer`, `Schema`, `Serializer`). @@ -882,8 +882,8 @@ maintain it; out of scope for a review pass because it is Phase 2 surface. Closed by the `Gate self-tests (scripts/*.test.mjs)` step in the `ci` job, placed directly after the `Test` step and mirrored in `.claude/skills/ci-preflight/run-ci.mjs` as step id `test:scripts`. The glob now covers -`knowledge.test.mjs`, `verify-seam-1.test.mjs`, `verify-sse-37.test.mjs` and `verify-test-partition.test.mjs` -— 64 cases. +`knowledge.test.mjs`, `verify-seam-1.test.mjs`, `verify-sse-37.test.mjs`, +`verify-knowledge-structure.test.mjs` and `verify-test-partition.test.mjs` — 123 cases. **The trigger had already fired when this was closed.** `knowledge.test.mjs` was failing on `main`: `--list-topics` reports how many of the 39 topic files carry no requirement ID, the test pinned that at 16, @@ -901,7 +901,15 @@ Original finding kept below for provenance. `scripts/*.test.mjs` runs only under `bun run test:scripts`, and `.github/workflows/ci.yml` has no step that invokes it. As of 6b that glob covered `scripts/knowledge.test.mjs`, `scripts/verify-seam-1.test.mjs`, and -`scripts/verify-sse-37.test.mjs`; Phase 10's `tests/` merge added `scripts/verify-test-partition.test.mjs`. +`scripts/verify-sse-37.test.mjs`; the 2026-08-31 knowledge-corpus split added +`scripts/verify-knowledge-structure.test.mjs`, and Phase 10's `tests/` merge added +`scripts/verify-test-partition.test.mjs`. + +**Raised in priority 2026-08-31.** `verify:knowledge-structure` is now a blocking CI step, and it reads the +corpus through `scripts/knowledge.mjs`'s parser. A parser regression that makes `loadCorpus` return fewer +entries turns that gate into a no-op that prints OK — the failure mode a hand edit to `harvested/` is supposed +to hit. The gate carries its own floor assertion against a vacuous parse (`MIN_HARVESTED_ENTRIES`), which is a +backstop, not a substitute for running the parser's tests. The script was named `test:knowledge` until the Phase 6a reader pass renamed it: the glob had outgrown the name the moment `verify-seam-1.test.mjs` landed, and both places that cite it had to explain the mismatch in @@ -982,7 +990,7 @@ other callers exposed. Raised by the Phase 6a adversarial review as G10. Ledgered rather than fixed: adding `{signal}` to four public APIs is a design decision, not a review-pass edit, and some of the sites would breach `max-params`. -`docs/knowledge/concurrency-and-async.md:18` ("every long-running async API must accept an options object with +`docs/knowledge/harvested/concurrency-and-async.md:18` ("every long-running async API must accept an options object with `{ signal }`"), `:20` (accepting must be paired with honoring), and `:44` (a signal must reach the actual I/O primitive) all apply. Four sites accept none: `Serializer.serializeTo`, `Deserializer.deserializeFrom` (`packages/core/src/seams/serde.ts`), `decodeResponse` and `decodeSuccessResponse` @@ -1302,7 +1310,7 @@ a missing export. ### K10 — `CFG-24`'s warning half is not emitted — **SCHEDULED** (Phase 7b) `CFG-24` requires that malformed proxy configuration resolve to null *with a warning* — "MUST NOT throw on -malformed input (invalid config → null + warning)", restated at `docs/knowledge/configuration.md:52`. The +malformed input (invalid config → null + warning)", restated at `docs/knowledge/harvested/configuration.md:52`. The shipped `resolveProxyOptions` gets the null half right on all three rejection paths (`parseProxyUrl`'s `catch`, an unusable port, an absent host) and emits no diagnostic on any of them, because there is no `Logger` in the package until 7b ships the `OBS-*` seam. The consequence today: a typo'd `HTTPS_PROXY` @@ -1321,7 +1329,7 @@ surface-widening the phase declined to do for K1: 1. **It is the sole outbound `config/ → pipeline/` edge.** Three of its four imports leave the folder — `../http/headers.js`, `../http/request.js`, `../pipeline/step.js` — and only `./build-info.js` stays. Every other module under `config/` imports nothing beyond `../invariant.js` and `../generated/version.js`. - The file is grouped by which phase built it, not by feature (`docs/knowledge/module-organization.md:12`). + The file is grouped by which phase built it, not by feature (`docs/knowledge/harvested/module-organization.md:12`). There is no cycle today; the risk is that 5a's `RetryConfig.clock` and 7b's logging step both create the return edge, and nothing in CI would catch the loop (see K12). 2. **Its `RECOV-32` sibling would land elsewhere.** The idempotency-key step — the adjacent requirement, the @@ -1332,15 +1340,15 @@ surface-widening the phase declined to do for K1: Also noted here, since it is the same class of question: the barrel comment at `packages/core/src/index.ts` explains the absence of a `config/index.ts`. This repo carries both patterns — `http/`, `body/`, `io/`, and `seams/` have internal barrels; `pipeline/`, `context/`, and `config/` do not — and so does the knowledge -corpus, where `docs/knowledge/module-organization.md:18` bans internal barrels outright while -`docs/knowledge/api-design.md:8` endorses one per feature folder, with nothing in the corpus's +corpus, where `docs/knowledge/harvested/module-organization.md:18` bans internal barrels outright while +`docs/knowledge/harvested/api-design.md:8` endorses one per feature folder, with nothing in the corpus's `--section conflicts` reconciling them. 7a followed its design doc, which rules a `config/index.ts` out by name. **Trigger:** the same phase as K1 — whichever one promotes the pipeline authoring surface. Settle the file's folder and the barrel question together there, and record the corpus tension at that point. ### K12 — No import-cycle gate exists in CI — **WATCH** (repo tooling, not this phase) -`docs/knowledge/module-organization.md:20` treats any import cycle as a bug rather than a style nit, and +`docs/knowledge/harvested/module-organization.md:20` treats any import cycle as a bug rather than a style nit, and `:22` requires it be gated in CI with `madge --circular src` or `eslint-plugin-import/no-cycle` as a required check. Neither string appears anywhere in `package.json`, `eslint.config.js`, or `.github/workflows/ci.yml` — verified 2026-08-27. Every other rule in that topic is either enforced or @@ -1442,7 +1450,7 @@ cases and `http/`'s own), so a divergence surfaces as a test failure rather than ### K19 — No `fast-check` property test logs its seed — **WATCH** (repo tooling, not this phase) -`docs/knowledge/testing.md:44` requires the seed of a failing seeded `fast-check` property test to reach CI +`docs/knowledge/harvested/testing.md:44` requires the seed of a failing seeded `fast-check` property test to reach CI output, "or the shrunk counterexample that found the bug is lost". No `fc.assert` call anywhere under `packages/core/src` passes a `seed`, `numRuns`, or a `reporter` — verified 2026-08-27 across all 20 `fc.assert` sites, of which Phase 7a contributes 12. A property failure in CI today reports the shrunk @@ -1649,6 +1657,62 @@ Not fixed in Phase 9: it edits another package's manifest and changes the lockfi Owner: Phase 10, alongside its own dependency pass. +## Section O — Knowledge-corpus split (2026-08-31) + +### O1 — The `knowledge-harvest` skill's default `--corpus` still points at the tree no query reads — **WATCH** + +`docs/knowledge/` is now two trees, `harvested/` and `notes/`, and `bun run knowledge` reads only those two. +The producing skill lives outside this repository (`~/.claude/skills/knowledge-harvest/`, user-global, shared +across projects) and its documented default is `/docs/knowledge/`, with its canonical stored-run command +naming `--corpus docs/knowledge`. A run that forgets `--corpus docs/knowledge/harvested` therefore writes a +third copy of the corpus at the root, which no query reads. + +Separately, `merge.py` emits a `## Superseded` heading unconditionally and offers `supersede` as one of its +four conflict resolutions. A harvest that writes one into `harvested/` produces an entry the structure gate +rejects, correctly — the resolution has to be hand-moved to `notes/`. + +Neither can be fixed from inside this repository. The compensating controls are all here and all blocking: +`verify:knowledge-structure` rejects a `.md` stranded at the root of `docs/knowledge/` and rejects a +`Superseded` entry under `harvested/`, and the invocation is stated in `CLAUDE.md`, `docs/knowledge/README.md` +and the `knowledge-lookup` skill. + +**Trigger:** anyone changing the global skill — make `harvested/` its default when the two-tree layout is +present, and stop emitting `Superseded` into a harvest target that has a sibling `notes/`. + +### O2 — A note's key citation is checked by a report, not by a gate — **OPEN** + +A note names the harvested rule it overrides by that rule's stable key (`/<8 hex>`, digested from the +entry's text). `bun run knowledge:drift` reports a citation that no entry carries any more, `--key` resolves +one on demand, and a harvested entry that a note overrides prints `[overridden by notes/…]`. None of that is +blocking: a re-harvest that rewords a rule silently breaks every note citing it, and only a hand-run report +says so. + +The issue that introduced the split specified three structural rules and this is not among them, so it was not +added unilaterally. Two candidate mechanisms were reviewed: + +1. **Fail `verify:knowledge-structure` on an unresolvable key.** Ten lines, no new file, and it makes a + re-harvest that orphans a note a red build rather than a silent rot. It also means a legitimate re-harvest + cannot land until the notes it invalidates are updated in the same commit — which is arguably the point. +2. **Commit a key manifest** (`harvested/KEYS.md`) regenerated only by a harvest, and fail when the live key + set diverges without a matching `SOURCES.md` sha change. This catches strictly more: it detects *any* hand + edit to harvested text, including one that keeps the role, the section and the source and so passes all + four current rules. Cost is a new generated artifact and a coupling to a skill this repo does not own. + +**Trigger:** the next re-harvest, whichever lands first. Until then `bun run knowledge:drift` is the check, and +it is named in the phase-start section of the `knowledge-lookup` skill. + +### O3 — `docs/superpowers/` still carries pre-split corpus paths — **WON'T FIX** + +The move to `harvested/` invalidated every `docs/knowledge/.md:` citation in the repository. The +107 in `packages/`, `test/`, `tests/` and `.changeset/` were repointed in the same commit, per the rule that a +comment that no longer matches is corrected with the change that staled it. + +The ~200 in `docs/superpowers/plans/` and `docs/superpowers/specs/` were deliberately left. Those files are +dated records of what a phase planned and found at the time; they are not retro-edited, and a phase plan +citing the path that existed when it was written is accurate about its own moment. `CLAUDE.md` states the +split so a reader does not read it as an oversight. + + ## Maintaining this file Add an entry the moment a gap is found, not when it is fixed — the failure mode this file prevents is a diff --git a/docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md b/docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md index 537a152..3def142 100644 --- a/docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md +++ b/docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md @@ -14,7 +14,7 @@ Phase 10 (`docs/superpowers/specs/2026-07-28-phase10-deviation-reconciliation-de |---|---|---| | **This section (§10)** | The **normative ledger** of deliberate deviations — the canonical, as-built list. Every item's number is the one the other files cite. | Owns items 1-17 | | `docs/deviations.md` | The **as-built audit** of this ledger, performed against source rather than against the phase specs that produced it. Carries the `file:line` evidence for each item, and the record of which items this ledger got wrong. Restates §10's item numbers; it does not assign its own. | Follows §10's | -| `docs/knowledge/deliberate-deviations.md` | Neither. A **harvested corpus topic file** queried by `bun run knowledge`, derived from an *older* revision of this section. Confusingly named; it is not a ledger and must not be edited as one — it is `knowledge-harvest`'s output. **Currently stale** — see its own head banner. | None | +| `docs/knowledge/notes/deliberate-deviations.md` | Neither. A one-entry **pointer** in the knowledge corpus saying that this register is not harvested and must be read here. The harvested copy that used to sit at `docs/knowledge/deliberate-deviations.md` was dropped on 2026-08-31: it held about a third of this section, at a three-revision-old sha, with two entries substantively false. A register accumulates rows; a harvest of one is a snapshot that goes stale on the next append. | None | **Renumbering this section renumbers `docs/deviations.md`.** Its section headings and its two summary tables are keyed to the numbers above, with no independent identity to fall back on; change one and the other must change in diff --git a/package.json b/package.json index 19aa31d..acc6fc1 100644 --- a/package.json +++ b/package.json @@ -66,6 +66,8 @@ "verify:sse-37": "node scripts/verify-sse-37.mjs", "verify:runtime-floor": "node scripts/verify-runtime-floor.mjs", "verify:test-partition": "node scripts/verify-test-partition.mjs", + "verify:knowledge-structure": "node scripts/verify-knowledge-structure.mjs", + "knowledge:drift": "node scripts/knowledge-drift.mjs", "verify:reproducible-build": "node scripts/verify-reproducible-build.mjs" } } diff --git a/packages/codec-json/src/tristate-schema.test.ts b/packages/codec-json/src/tristate-schema.test.ts index b7347db..c6c248c 100644 --- a/packages/codec-json/src/tristate-schema.test.ts +++ b/packages/codec-json/src/tristate-schema.test.ts @@ -103,7 +103,7 @@ test('a field schema rejection propagates out of tristateObject', () => { test("tristateObject preserves each field's element type through the mapped return (SERDE-16)", () => { // `tristateObject`'s return is a mapped-plus-conditional type built behind an `as never`, so a // runtime test cannot catch an inference regression here — only `expectTypeOf` can - // (docs/knowledge/testing.md:30). + // (docs/knowledge/harvested/testing.md:30). const parsed = tristateObject({age: numberSchema, name: stringSchema}).parse( {}, ); diff --git a/packages/core/src/auth/auth-step.test.ts b/packages/core/src/auth/auth-step.test.ts index f7462ca..89598b0 100644 --- a/packages/core/src/auth/auth-step.test.ts +++ b/packages/core/src/auth/auth-step.test.ts @@ -73,7 +73,7 @@ interface DriveOverrides { // `Transport`, not `FakeTransport`: the only thing this helper does with it is hand it to `Cursor`, // and narrowing to what is actually used is what lets the gated double below be driven through it too -// (`docs/knowledge/api-design.md` -- accept the narrowest interface describing the members used). +// (`docs/knowledge/harvested/api-design.md` -- accept the narrowest interface describing the members used). function runThrough( descriptor: StepDescriptor, transport: Transport, diff --git a/packages/core/src/auth/auth-step.ts b/packages/core/src/auth/auth-step.ts index 8bb15c9..80b9e94 100644 --- a/packages/core/src/auth/auth-step.ts +++ b/packages/core/src/auth/auth-step.ts @@ -177,7 +177,7 @@ export type ChallengeHook = ( * The calling request's cancellation, threaded straight through from `StepContext.signal`. * * A hook is the sanctioned place to run a custom OAuth2 refresh-token grant, which is external - * I/O on the request path -- and `docs/knowledge/concurrency-and-async.md` is explicit that a + * I/O on the request path -- and `docs/knowledge/harvested/concurrency-and-async.md` is explicit that a * signal accepted at the top of a call chain must reach the actual I/O primitive, or it is * decoration. Without this a hung hook pinned the auth step, every retry attempt nested under * it, and the whole request, with no way for the caller to abort. @@ -293,7 +293,7 @@ function withHeader(request: Request, name: string, value: string): Request { * compile, which is the compiler being confidently wrong about mutable external state. Routing every * read through a call re-reads the getter each time. * - * `docs/knowledge/concurrency-and-async.md`: "state checked before an `await` must be re-validated + * `docs/knowledge/harvested/concurrency-and-async.md`: "state checked before an `await` must be re-validated * after every `await` that could have let the world move." */ function isAborted(signal: AbortSignal | undefined): boolean { @@ -312,7 +312,7 @@ function answerHeaderName(isProxy: boolean): string { * parsed challenge, and `NO_AUTH` has nothing to stamp. * * An exhaustive `switch` closing on `assertNever`, not an if-chain: `AuthScheme` is a closed - * discriminant, and `docs/knowledge/data-modeling.md` bars an if-chain over one because it gives no + * discriminant, and `docs/knowledge/harvested/data-modeling.md` bars an if-chain over one because it gives no * exhaustiveness guarantee and falls through silently when a variant is added — and the value that * would fall through here is a credential-stamping decision. */ diff --git a/packages/core/src/auth/bearer-cache.ts b/packages/core/src/auth/bearer-cache.ts index b793975..7709da6 100644 --- a/packages/core/src/auth/bearer-cache.ts +++ b/packages/core/src/auth/bearer-cache.ts @@ -18,7 +18,7 @@ const NO_EVICTION_GENERATION = -1; * One token fetch's inputs. * * Bundled rather than passed positionally: `max-params` is 3, and - * `docs/knowledge/function-design.md` requires an options object at three or more parameters anyway. + * `docs/knowledge/harvested/function-design.md` requires an options object at three or more parameters anyway. * * @internal */ @@ -87,7 +87,7 @@ async function invokeProvider(provider: TokenProvider): Promise { * assignment and the generation bump un-interleaved. * * `new Promise` with a synchronous executor adapting an event-emitter callback is the one shape - * `docs/knowledge/concurrency-and-async.md` sanctions for it, and the listener is removed on every + * `docs/knowledge/harvested/concurrency-and-async.md` sanctions for it, and the listener is removed on every * exit so a long-lived caller signal does not accumulate one per token fetch. * * A `pending` that rejects after losing the race is still settled through `Promise.race`'s own diff --git a/packages/core/src/auth/credential.ts b/packages/core/src/auth/credential.ts index f0581ee..5eec7a7 100644 --- a/packages/core/src/auth/credential.ts +++ b/packages/core/src/auth/credential.ts @@ -184,7 +184,7 @@ export function credentialKey( * this is a class with a private field and deliberately NO `equals` override: `===`, the language * default, already gives exactly those semantics. * - * `#key`, not `private key`, is the deliberate exception to `docs/knowledge/data-modeling.md`'s + * `#key`, not `private key`, is the deliberate exception to `docs/knowledge/harvested/data-modeling.md`'s * `private`-by-default rule, and the same note requires the justification be written down: AUTH-8's * redaction is a RUNTIME-privacy requirement, not a compile-time one. `private` is erased, leaving the * secret reachable through `credential['key']`, `Object.keys`, `JSON.stringify`, and a default @@ -306,7 +306,7 @@ export class NameKeyCredential { * const provider: TokenProvider = () => fetchToken({signal: AbortSignal.timeout(5_000)}); * ``` * - * `docs/knowledge/concurrency-and-async.md`'s "every external I/O call must carry a deadline" is the + * `docs/knowledge/harvested/concurrency-and-async.md`'s "every external I/O call must carry a deadline" is the * rule this discharges; its "pass the caller's signal down to the I/O primitive" rule is the one * deliberately not applied here, because the premise it rests on -- that the call owns the I/O -- is * false for a coalesced fetch. Recorded in the phase checklist's Deviation Ledger. diff --git a/packages/core/src/auth/digest.ts b/packages/core/src/auth/digest.ts index 93ffddd..92fde74 100644 --- a/packages/core/src/auth/digest.ts +++ b/packages/core/src/auth/digest.ts @@ -115,7 +115,7 @@ async function hashHex({base, input, isUtf8}: HashInput): Promise { * * The eviction is an insert-THEN-drain, never a pre-insert check-then-evict: `next()` admits the * nonce first and only then brings the map back under the cap, which is how - * `docs/knowledge/concurrency-and-async.md` (XCUT-14) and AUTH-19 both word it — "drained back under + * `docs/knowledge/harvested/concurrency-and-async.md` (XCUT-14) and AUTH-19 both word it — "drained back under * the cap after admitting a nonce". The key space is the SERVER's, since it picks the nonces, so a * pre-insert evict would leave a burst sitting above the cap rather than converging to it. * diff --git a/packages/core/src/auth/errors.ts b/packages/core/src/auth/errors.ts index 174e36d..57d7145 100644 --- a/packages/core/src/auth/errors.ts +++ b/packages/core/src/auth/errors.ts @@ -13,7 +13,7 @@ import {DexpaceError} from '../http/errors.js'; * * The scheme lists are `readonly` FIELDS, not only interpolated prose. AUTH-6 requires the error to * carry both the required schemes in preference order and the available schemes, and - * `docs/knowledge/error-handling.md` requires identifying inputs to be `readonly` fields "so they + * `docs/knowledge/harvested/error-handling.md` requires identifying inputs to be `readonly` fields "so they * survive serialization and appear in structured logs". Both are `undefined` on the AUTH-35 * construction path, which has no scheme lists to carry. * @@ -79,7 +79,7 @@ export class AuthResolutionError extends DexpaceError { * AUTH-28: a credential would have been attached to a non-HTTPS URL. * * The offending URL is deliberately NOT carried — a URL can hold userinfo and query-string secrets, - * and `docs/knowledge/error-handling.md` bars interpolating secrets into a message that travels into + * and `docs/knowledge/harvested/error-handling.md` bars interpolating secrets into a message that travels into * logs. The step name and scheme identify the fault without that risk. * * @public diff --git a/packages/core/src/auth/md5.ts b/packages/core/src/auth/md5.ts index 445361b..6653893 100644 --- a/packages/core/src/auth/md5.ts +++ b/packages/core/src/auth/md5.ts @@ -20,7 +20,7 @@ const SHIFTS = [ 21, ] as const; -// `/*#__PURE__*/`, because this is a top-level CALL, and `docs/knowledge/performance.md` is explicit +// `/*#__PURE__*/`, because this is a top-level CALL, and `docs/knowledge/harvested/performance.md` is explicit // that modules must do no work at import time — a top-level call is a side effect the bundler must // preserve, and that pins the module in the bundle. `@dexpace/core` declares `"sideEffects": false`; // without the annotation a bundler cannot prove these 64 `Math.sin` calls are pure, so `md5.ts` and diff --git a/packages/core/src/auth/preset.ts b/packages/core/src/auth/preset.ts index bb47b1d..9455dd9 100644 --- a/packages/core/src/auth/preset.ts +++ b/packages/core/src/auth/preset.ts @@ -35,7 +35,7 @@ export interface StandardResilienceOptions { } // Built lazily rather than as a top-level `const NO_AUTH_SETTINGS = ...`: a module-scope factory call -// is import-time work a bundler must preserve (`docs/knowledge/performance.md`), and it would pin +// is import-time work a bundler must preserve (`docs/knowledge/harvested/performance.md`), and it would pin // descriptor.ts/requirement.ts into every bundle that imports the preset. The allocation is per call, // but the preset is constructed once per client, not per request. function noAuthSettings(): AuthStepSettings { diff --git a/packages/core/src/auth/requirement.ts b/packages/core/src/auth/requirement.ts index f69f2c1..89ea94c 100644 --- a/packages/core/src/auth/requirement.ts +++ b/packages/core/src/auth/requirement.ts @@ -36,7 +36,7 @@ export function createAuthRequirement( scopes: readonly string[] = [], params: ReadonlyMap = new Map(), ): AuthRequirement { - // `Object.freeze` is SHALLOW. `docs/knowledge/data-modeling.md` requires a frozen value object to + // `Object.freeze` is SHALLOW. `docs/knowledge/harvested/data-modeling.md` requires a frozen value object to // hold only primitives or already-frozen/read-only values, never a mutable object that stays // writable behind the freeze. `new Map(params)` satisfies AUTH-2's literal clause -- caller-side // mutation cannot reach the stored value -- but leaves the copy itself writable behind the diff --git a/packages/core/src/config/build-info.ts b/packages/core/src/config/build-info.ts index 700643b..4ba9a67 100644 --- a/packages/core/src/config/build-info.ts +++ b/packages/core/src/config/build-info.ts @@ -70,7 +70,7 @@ function toUsableToken(value: unknown): string | null { * * Takes the host explicitly rather than reading `globalThis` inline: the branches this phase cannot * execute on its own runners are then reachable from a test without deleting a global, which no test - * may do (`docs/knowledge/testing.md:50` -- tests must survive parallel execution). + * may do (`docs/knowledge/harvested/testing.md:50` -- tests must survive parallel execution). * * @param host - the ambient global object to interrogate. * @returns a non-blank runtime identity token. @@ -107,7 +107,7 @@ function resolveBuildInfo(): BuildInfo { } /** - * Module-level mutable state, which `docs/knowledge/variables-and-declarations.md:22` bans outright. + * Module-level mutable state, which `docs/knowledge/harvested/variables-and-declarations.md:22` bans outright. * Deliberate: CFG-36's descriptor is resolved once per process, and the alternative -- re-running the * feature detection per request -- is the cost the memo exists to avoid. Safe against the rule's * stated hazard because `resolveBuildInfo` is deterministic within a process, so no test can observe diff --git a/packages/core/src/config/client-identity-step.ts b/packages/core/src/config/client-identity-step.ts index 3f4e14a..45bb528 100644 --- a/packages/core/src/config/client-identity-step.ts +++ b/packages/core/src/config/client-identity-step.ts @@ -51,7 +51,7 @@ function composeFirstValue(existingFirst: string, tokenLine: string): string { /** * The header name and mode {@link composeHeaders} needs, after {@link clientIdentityStep} has applied * its defaults. One object rather than two more parameters, so `composeHeaders` stays inside the - * three-parameter cap (`docs/knowledge/function-design.md:22`). + * three-parameter cap (`docs/knowledge/harvested/function-design.md:22`). */ interface ResolvedComposition { readonly headerName: string; diff --git a/packages/core/src/config/configuration.test.ts b/packages/core/src/config/configuration.test.ts index a65cae8..d1a7901 100644 --- a/packages/core/src/config/configuration.test.ts +++ b/packages/core/src/config/configuration.test.ts @@ -362,7 +362,7 @@ describe('substitutable seams (CFG-11)', () => { test('delegates the production default to the ambient environment', () => { // A fresh key per run, drawn from the package's own generator rather than the clock: two // same-millisecond runs of this file would otherwise pick the same name and race on real - // `process.env` (`docs/knowledge/testing.md:36`, `:50`). + // `process.env` (`docs/knowledge/harvested/testing.md:36`, `:50`). const key = `DEXPACE_TEST_${randomUuid().replaceAll('-', '')}`; const host = globalThis as { process?: {env?: Record}; diff --git a/packages/core/src/config/configuration.ts b/packages/core/src/config/configuration.ts index 9249e68..3db4ef8 100644 --- a/packages/core/src/config/configuration.ts +++ b/packages/core/src/config/configuration.ts @@ -329,7 +329,7 @@ export function defaultConfiguration(): Configuration { } /** - * Module-level mutable state, which `docs/knowledge/variables-and-declarations.md:22` bans outright. + * Module-level mutable state, which `docs/knowledge/harvested/variables-and-declarations.md:22` bans outright. * Deliberate: CFG-13 *specifies* a process-wide, last-write-wins slot, so the shared-by-every-importer * property the rule warns about is the requirement rather than a side effect. The rule's real cost -- * state carried between test cases in one process -- is live and unmitigated: there is no reset hook, @@ -363,7 +363,7 @@ export function setGlobalConfiguration(config: Configuration): void { // A `typeof` check, not just a null check: every other CFG-37 guard in this module tests the shape // it needs (`string` for an override, `function` for a seam or a mutator). Accepting `42` here put // a number in the process-wide slot, where it surfaced as a failure in an unrelated consumer far - // from the fault (`docs/knowledge/error-handling.md:36`). + // from the fault (`docs/knowledge/harvested/error-handling.md:36`). invariant( typeof supplied === 'object' && supplied !== null, 'setGlobalConfiguration: config is required', diff --git a/packages/core/src/config/duration.test.ts b/packages/core/src/config/duration.test.ts index 29d593c..f9c00fc 100644 --- a/packages/core/src/config/duration.test.ts +++ b/packages/core/src/config/duration.test.ts @@ -54,7 +54,7 @@ describe('parseDurationMs (CFG-7)', () => { }); test('reads the three grammars onto one scale', () => { - // The canonical law CFG-7 implies (`docs/knowledge/testing.md:28`): ISO-8601, shorthand, and a + // The canonical law CFG-7 implies (`docs/knowledge/harvested/testing.md:28`): ISO-8601, shorthand, and a // bare number are three spellings of one duration, so for any whole number of seconds all three // must land on the same milliseconds. A totality property cannot see the two scales drift apart. fc.assert( diff --git a/packages/core/src/config/duration.ts b/packages/core/src/config/duration.ts index 5aa65ad..c90b67c 100644 --- a/packages/core/src/config/duration.ts +++ b/packages/core/src/config/duration.ts @@ -65,7 +65,7 @@ function parseShorthandDuration(raw: string): number | null { * unit, a negative, anything else -- yields `null` so the caller falls back to its default. * * Its own module rather than a helper inside `configuration.ts`: the grammar is a concept separate - * from the layered lookup that happens to consume it (`docs/knowledge/module-organization.md:42`). + * from the layered lookup that happens to consume it (`docs/knowledge/harvested/module-organization.md:42`). * `Configuration.getDuration` is its only caller today. * * @param raw - the candidate duration; surrounding whitespace is tolerated. diff --git a/packages/core/src/config/identifiers.test.ts b/packages/core/src/config/identifiers.test.ts index 00f2a34..f546504 100644 --- a/packages/core/src/config/identifiers.test.ts +++ b/packages/core/src/config/identifiers.test.ts @@ -36,7 +36,7 @@ describe('randomUuid (CFG-32)', () => { test('names the missing dependency when the runtime exposes no WebCrypto', () => { // The random source is passed in rather than read off `globalThis`, so this branch is reachable // without deleting or reassigning a global -- which would break parallel execution - // (`docs/knowledge/testing.md:50`). Reading `getRandomValues` off `undefined` would otherwise + // (`docs/knowledge/harvested/testing.md:50`). Reading `getRandomValues` off `undefined` would otherwise // report only `TypeError: Cannot read properties of undefined`. expect(() => randomUuidFrom(undefined)).toThrow(InvariantViolation); expect(() => randomUuidFrom(undefined)).toThrow(/globalThis\.crypto/u); diff --git a/packages/core/src/config/identifiers.ts b/packages/core/src/config/identifiers.ts index b12f97d..1458425 100644 --- a/packages/core/src/config/identifiers.ts +++ b/packages/core/src/config/identifiers.ts @@ -11,7 +11,7 @@ const HEX_BY_BYTE: readonly string[] = Array.from({length: 256}, (_, byte) => * * Parameterized for the same reason `build-info.ts`'s `detectRuntimeIdentity(host)` is: the * missing-WebCrypto branch is then reachable from a test without deleting or reassigning a global, - * which no test may do (`docs/knowledge/testing.md:50` -- every test must survive parallel + * which no test may do (`docs/knowledge/harvested/testing.md:50` -- every test must survive parallel * execution). * * @param webCrypto - the WebCrypto implementation, or `undefined` on a runtime that exposes none. diff --git a/packages/core/src/config/proxy.test.ts b/packages/core/src/config/proxy.test.ts index 728a9c7..70abba4 100644 --- a/packages/core/src/config/proxy.test.ts +++ b/packages/core/src/config/proxy.test.ts @@ -45,7 +45,7 @@ function hasOwnToString(options: ProxyOptions): boolean { * A fresh bypass-options value per test. Never a shared `const` at describe scope: the array is * mutable, and `proxy.ts` keys its compiled-glob cache by that exact array *identity*, so a shared * fixture would be shared into module-level state that outlives the describe - * (`docs/knowledge/testing.md:52`, `:50`). + * (`docs/knowledge/harvested/testing.md:52`, `:50`). */ function bypassOptions( ...nonProxyHosts: string[] diff --git a/packages/core/src/context/store.test.ts b/packages/core/src/context/store.test.ts index 17db477..fdf3ee3 100644 --- a/packages/core/src/context/store.test.ts +++ b/packages/core/src/context/store.test.ts @@ -16,7 +16,7 @@ // Every test builds its own `new ContextStore()`. The exported `contextStore` singleton is module-level // mutable state shared by every test file in a `bun test` run -- 4c's runtime.test.ts installs into that // same object -- so an absolute `size` assertion against it reads a counter a sibling file can move, and a -// blanket clear() wipes a sibling's entries. docs/knowledge/testing.md:50,52. The singleton gets exactly +// blanket clear() wipes a sibling's entries. docs/knowledge/harvested/testing.md:50,52. The singleton gets exactly // one assertion here: that it is a ContextStore. import {describe, expect, test} from 'bun:test'; import {Protocol} from '../http/protocol.js'; diff --git a/packages/core/src/context/store.ts b/packages/core/src/context/store.ts index 54e103f..49ab4a4 100644 --- a/packages/core/src/context/store.ts +++ b/packages/core/src/context/store.ts @@ -31,7 +31,7 @@ export class ContextStore { constructor(maxEntries: number = DEFAULT_MAX_ENTRIES) { // A bad cap is a violated precondition — a programmer error — so it crashes at the fault via the // project's one assertion primitive rather than an ad-hoc `if (!x) throw` - // (docs/knowledge/assertions.md:4, docs/knowledge/error-handling.md:36). + // (docs/knowledge/harvested/assertions.md:4, docs/knowledge/harvested/error-handling.md:36). invariant( Number.isInteger(maxEntries) && maxEntries >= 1, `maxEntries must be a positive integer, got ${String(maxEntries)}`, @@ -123,7 +123,7 @@ export class ContextStore { /** * The one registry 4c's `Runtime.send()` installs into. Module-level mutable state, which - * `docs/knowledge/variables-and-declarations.md:22` bans — accepted here because threading a store handle + * `docs/knowledge/harvested/variables-and-declarations.md:22` bans — accepted here because threading a store handle * through builder → runtime → every step would be a wide API change for no observable gain, and logged in * the design's Deviation Ledger for Phase 10. Tests must build their own `new ContextStore()` rather than * asserting through this one: it is shared by every test file in a `bun test` run. diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 32a67b8..a6f8e7b 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -21,7 +21,7 @@ export * from './http/index.js'; // Deliberately NOT `export * from './seams/index.js';`. That file is a folder-level barrel, which -// docs/knowledge/module-organization.md:18 bans outright and api-design.md:6 makes this file the sole +// docs/knowledge/harvested/module-organization.md:18 bans outright and api-design.md:6 makes this file the sole // one of; nothing imports it and the right end state is deleting it (docs/open-items.md H12). Naming // each public export here keeps the package's surface a decision made in one place rather than a // consequence of what a folder happens to re-export. @@ -242,8 +242,8 @@ export {PaginationError} from './pagination/errors.js'; // design doc rules one out by name. Not because the question is settled: this repo carries both // patterns — `http/`, `body/`, `io/`, and `seams/` each have an internal barrel, while `pipeline/`, // `context/`, and `config/` do not — and so does the knowledge corpus, where -// docs/knowledge/module-organization.md:18 bans internal barrels outright and -// docs/knowledge/api-design.md:8 endorses one per feature folder, with no entry in the corpus's +// docs/knowledge/harvested/module-organization.md:18 bans internal barrels outright and +// docs/knowledge/harvested/api-design.md:8 endorses one per feature folder, with no entry in the corpus's // `--section conflicts` reconciling them. 7a followed its design doc and names each symbol here // against its own file; see docs/open-items.md K11 for the standing note. // Deliberately NOT exported: `config/equality.js`'s deepEqual/deepHash — no requirement gives a diff --git a/packages/core/src/invariant.test.ts b/packages/core/src/invariant.test.ts index 118975c..7b55da9 100644 --- a/packages/core/src/invariant.test.ts +++ b/packages/core/src/invariant.test.ts @@ -1,7 +1,7 @@ // SPDX-License-Identifier: MIT // packages/core/src/invariant.test.ts // Exercises: the project's sole assertion primitive (styleguide 5.6), its error class, and the -// discriminated-union exhaustiveness helper docs/knowledge/data-modeling.md requires every switch to +// discriminated-union exhaustiveness helper docs/knowledge/harvested/data-modeling.md requires every switch to // close with. import {describe, expect, test} from 'bun:test'; import {assertNever, invariant, InvariantViolation} from './invariant.js'; diff --git a/packages/core/src/invariant.ts b/packages/core/src/invariant.ts index d450d00..39a427e 100644 --- a/packages/core/src/invariant.ts +++ b/packages/core/src/invariant.ts @@ -32,7 +32,7 @@ export function invariant(cond: unknown, msg: string): asserts cond { /** * Closes an exhaustive discriminated-union `switch`'s `default` case - * (`docs/knowledge/data-modeling.md`). If a new union variant is ever added without a matching + * (`docs/knowledge/harvested/data-modeling.md`). If a new union variant is ever added without a matching * `case`, the call stops type-checking; if one reaches this at runtime anyway — a value crossing a * seam that the type says cannot exist — it crashes loudly rather than falling through silently. * @@ -47,7 +47,7 @@ export function assertNever(value: never, message?: string): never { /** * `String(value)` is not total: it throws on a null-prototype object (no `toString` to reach) and * on any value whose `toString`/`Symbol.toPrimitive` throws — the same hazard - * `docs/knowledge/error-handling.md:18` makes `toError` guard. An assertion helper that throws from + * `docs/knowledge/harvested/error-handling.md:18` makes `toError` guard. An assertion helper that throws from * its own message construction reports the wrong failure at the worst moment, so the fallback is a * fixed string. */ diff --git a/packages/core/src/pipeline/builder.test.ts b/packages/core/src/pipeline/builder.test.ts index 1cdaa87..ad22b1a 100644 --- a/packages/core/src/pipeline/builder.test.ts +++ b/packages/core/src/pipeline/builder.test.ts @@ -348,7 +348,7 @@ describe('PipelineBuilder edit-order independence (PIPE-22)', () => { // The two ordering laws the design calls for (PIPE-38's split across an append and a prepend test, one act // each). `build()` is an invariant-bearing assembler, which -// docs/knowledge/testing.md:29 puts in property-test territory; the examples above pin concrete regressions, +// docs/knowledge/harvested/testing.md:29 puts in property-test territory; the examples above pin concrete regressions, // these prove the law over generated input. Generated over the non-pillar stages only: a generator that also // emitted pillar stages would spend most of its cases hitting PIPE-5's collision instead of exercising order. const editableStages = STAGE_ORDER.filter( diff --git a/packages/core/src/pipeline/errors.ts b/packages/core/src/pipeline/errors.ts index d49770c..05fc8da 100644 --- a/packages/core/src/pipeline/errors.ts +++ b/packages/core/src/pipeline/errors.ts @@ -22,7 +22,7 @@ export class PillarCollisionError extends DexpaceError { ) { // PIPE-5: the error names BOTH step types and points at the replace path. Symbols are rendered with // String() (`Symbol(retry)`) -- a bare symbol field is invisible in a stack trace or log line - // (docs/knowledge/error-handling.md:40), the same reason 4a's DuplicateContextKeyError renders its key. + // (docs/knowledge/harvested/error-handling.md:40), the same reason 4a's DuplicateContextKeyError renders its key. super( `pillar stage '${stage}' already holds ${String(existingType)}; cannot install ${String(incomingType)} (use replace() to swap it)`, options, diff --git a/packages/core/src/recovery/cancellation.ts b/packages/core/src/recovery/cancellation.ts index edd053b..7ef3216 100644 --- a/packages/core/src/recovery/cancellation.ts +++ b/packages/core/src/recovery/cancellation.ts @@ -15,7 +15,7 @@ import {failure, type Outcome} from './outcome.js'; * It deliberately does **not** crash on a `CancellationError` whose paired signal never aborted. * `Transport` is a pluggable seam, so that mismatch is a misbehaving third-party implementation — * an operational failure, not a violated precondition of this codebase, and crash-loud treatment is - * reserved for the latter (`docs/knowledge/error-handling.md`). It would also break RECOV-2: this + * reserved for the latter (`docs/knowledge/harvested/error-handling.md`). It would also break RECOV-2: this * runs inside `dispatchWithRecovery`'s own `catch`, so throwing here would let a transport failure * skip the response and recovery chains entirely, which is precisely what RECOV-2 forbids. A * transport that aborts its in-flight requests from `close()` — which SEAM-14 permits — produces diff --git a/packages/core/src/recovery/outcome.test.ts b/packages/core/src/recovery/outcome.test.ts index 9cc38d4..3b3e9a3 100644 --- a/packages/core/src/recovery/outcome.test.ts +++ b/packages/core/src/recovery/outcome.test.ts @@ -68,7 +68,7 @@ describe('fold (RECOV-1)', () => { }); describe('fold identity law (RECOV-1)', () => { - // Canonical law for an invariant-bearing function (docs/knowledge/testing.md): folding a success + // Canonical law for an invariant-bearing function (docs/knowledge/harvested/testing.md): folding a success // through the identity success-handler, and a failure through the identity failure-handler, must // each recover the original payload, for arbitrary values. test('fold(success(x), id, _) === x for arbitrary x', () => { diff --git a/packages/core/src/recovery/response-chain.ts b/packages/core/src/recovery/response-chain.ts index f62349f..fbc1d07 100644 --- a/packages/core/src/recovery/response-chain.ts +++ b/packages/core/src/recovery/response-chain.ts @@ -111,7 +111,7 @@ export class ResponseRecoveryChain { * A close failure rides along as `suppressed` on the ORIGINAL throwable — built by hand through * {@link suppress}, original first — never via `using` / `await using`, whose auto-generated * `SuppressedError` puts the *teardown* failure first and would silently invert which error the - * caller ends up seeing (`docs/knowledge/resource-management.md:72`). + * caller ends up seeing (`docs/knowledge/harvested/resource-management.md:72`). * * **This function is total: it never throws, for any argument.** RECOV-8 makes "`apply()` MUST NOT * throw under any input" absolute, and this runs inside both phases' `catch` blocks — the last place diff --git a/packages/core/src/recovery/status-mapping.ts b/packages/core/src/recovery/status-mapping.ts index 40678ff..76ba577 100644 --- a/packages/core/src/recovery/status-mapping.ts +++ b/packages/core/src/recovery/status-mapping.ts @@ -30,7 +30,7 @@ export async function statusMappingStep(response: Response): Promise { } // A named declaration, not `const statusMappingStep: ResponseStep = async response => ...`: arrows -// are reserved for inline callbacks (docs/knowledge/function-design.md:18-21), and a named +// are reserved for inline callbacks (docs/knowledge/harvested/function-design.md:18-21), and a named // declaration survives in stack traces — which a function whose whole job is to throw actually // depends on. `func-style`'s `allowArrowFunctions: true` would not have flagged the arrow form, so // this is on the author, not the gate. diff --git a/packages/core/src/redirect/errors.test.ts b/packages/core/src/redirect/errors.test.ts index 39a0c7f..b9442b6 100644 --- a/packages/core/src/redirect/errors.test.ts +++ b/packages/core/src/redirect/errors.test.ts @@ -19,7 +19,7 @@ describe('NonReplayableBodyError', () => { }); test('carries the target as a readonly field, not only in the message', () => { - // docs/knowledge/error-handling.md: identifying inputs are fields so they survive serialization + // docs/knowledge/harvested/error-handling.md: identifying inputs are fields so they survive serialization // and reach a structured log without anyone parsing the message back apart. const error = new NonReplayableBodyError('https://example.com/next'); expect(error.targetUrl).toBe('https://example.com/next'); diff --git a/packages/core/src/redirect/errors.ts b/packages/core/src/redirect/errors.ts index f7d0e4f..99573e9 100644 --- a/packages/core/src/redirect/errors.ts +++ b/packages/core/src/redirect/errors.ts @@ -15,7 +15,7 @@ export class NonReplayableBodyError extends DexpaceError { * The redirect target that would have received the re-send. * * Carried as a field, not only interpolated into the message, per - * `docs/knowledge/error-handling.md` -- so it survives serialization and reaches a structured log + * `docs/knowledge/harvested/error-handling.md` -- so it survives serialization and reaches a structured log * without anyone parsing the message back apart. Phase 7b's rejection event reads it directly. */ readonly targetUrl: string; diff --git a/packages/core/src/retry/engine.ts b/packages/core/src/retry/engine.ts index 7dae51a..ef9e161 100644 --- a/packages/core/src/retry/engine.ts +++ b/packages/core/src/retry/engine.ts @@ -275,7 +275,7 @@ function withTrail( * port), and rejects promptly for a signal that aborted earlier. Duplicating it here would be the * same second-implementation the Phase 7a retrofit removed for the RFC 1123 parser and the * retryable-status set, and it would put the wait outside the injected seam -- forcing real timers - * into a unit suite `docs/knowledge/testing.md` requires to be deterministic. + * into a unit suite `docs/knowledge/harvested/testing.md` requires to be deterministic. * * A non-positive delay short-circuits before `sleep` is reached: it continues inline with no timer * (RETRY-31), which is reachable after RETRY-17's past-instant hint and after the budget clamp, and diff --git a/packages/core/src/seams/index.ts b/packages/core/src/seams/index.ts index 0c63905..90fe964 100644 --- a/packages/core/src/seams/index.ts +++ b/packages/core/src/seams/index.ts @@ -6,7 +6,7 @@ // Phase 6a closed SEAM-21, so the serde seam it re-exports is public, just promoted through // index.ts rather than through here. // -// Nothing imports this file. docs/knowledge/module-organization.md:18 bans internal folder-level +// Nothing imports this file. docs/knowledge/harvested/module-organization.md:18 bans internal folder-level // barrels outright, so the right end state is deleting it rather than maintaining it; recorded at // docs/open-items.md H12 for the next phase that touches packages/core/src/seams/. Phase 6a's own // new folder, packages/core/src/serde/, correctly has no barrel at all. diff --git a/packages/core/src/serde/tristate.test.ts b/packages/core/src/serde/tristate.test.ts index 4d9d066..ec5119b 100644 --- a/packages/core/src/serde/tristate.test.ts +++ b/packages/core/src/serde/tristate.test.ts @@ -90,7 +90,7 @@ test('values are frozen — a Tristate cannot be mutated after construction', () }); test('isTristate accepts only branded values — truth table', () => { - // A custom type guard needs the full table, not just the happy case (docs/knowledge/testing.md:34). + // A custom type guard needs the full table, not just the happy case (docs/knowledge/harvested/testing.md:34). expect([ isTristate(absent()), isTristate(nullValue()), diff --git a/packages/core/src/sse/stream.test.ts b/packages/core/src/sse/stream.test.ts index a798d69..7f3b866 100644 --- a/packages/core/src/sse/stream.test.ts +++ b/packages/core/src/sse/stream.test.ts @@ -257,7 +257,7 @@ test('aborting the signal closes the stream, ending an idle iterator cleanly (SS }); test('sseStreamFrom releases the byte source as well as the response (SSE-23, SSE-32)', async () => { - // docs/knowledge/sse-streaming.md:84 — the facade's release must reach `response.body.cancel()` exactly once. + // docs/knowledge/harvested/sse-streaming.md:84 — the facade's release must reach `response.body.cancel()` exactly once. // The BufferedSource holds the reader lock on that body, so unless the facade closes the *source*, a real // Response.close() would be cancelling a locked stream. A close-counting double cannot catch this; asserting // the body's own cancel hook fired is what does. diff --git a/packages/core/src/sse/stream.ts b/packages/core/src/sse/stream.ts index 8f10e4a..980f4ad 100644 --- a/packages/core/src/sse/stream.ts +++ b/packages/core/src/sse/stream.ts @@ -45,7 +45,7 @@ export interface SseStreamFromOptions extends SseStreamOptions { /** * Cancellation for this long-running operation - * (`docs/knowledge/concurrency-and-async.md:18`, `docs/knowledge/api-design.md:34`). + * (`docs/knowledge/harvested/concurrency-and-async.md:18`, `docs/knowledge/harvested/api-design.md:34`). * * Aborting closes the stream, which is the only cancellation a pull-based reader needs: an iterator sitting * *between* pulls then ends cleanly (SSE-27) and one blocked *in* a read surfaces an `IoError` (SSE-31). @@ -254,11 +254,11 @@ export function sseStreamFrom( * Make an abort close the stream. * * This lives here rather than in `SseStream`'s constructor because a constructor may only assign its arguments - * to fields — no branching, no listener registration (`docs/knowledge/data-modeling.md:24`). The listener is + * to fields — no branching, no listener registration (`docs/knowledge/harvested/data-modeling.md:24`). The listener is * registered with `{once: true}` and removed upon close, and the close promise is explicitly * discarded with `void` plus a `.catch`, because an unhandled rejection on this path would take the process * down under Node's default `unhandledRejection` policy - * (`docs/knowledge/cancellation-and-timeouts.md:26`). + * (`docs/knowledge/harvested/cancellation-and-timeouts.md:26`). */ function bindAbort( stream: SseStream, diff --git a/packages/core/src/suppress.test.ts b/packages/core/src/suppress.test.ts index e675796..c697c5f 100644 --- a/packages/core/src/suppress.test.ts +++ b/packages/core/src/suppress.test.ts @@ -5,7 +5,7 @@ // priority. // // Neither branch of the guard is forced here by mutating `globalThis` — a test that deletes a -// global does not survive parallel execution, which docs/knowledge/testing.md:50 requires. The +// global does not survive parallel execution, which docs/knowledge/harvested/testing.md:50 requires. The // branch selection is covered where it is real instead: `suppress()` is asserted on its shape, // which holds on either runtime, `FallbackSuppressedError` is constructed directly, and the // `test:node` matrix runs both legs — `lts/*` has the native class, the pinned `20.3.0` floor does diff --git a/packages/core/src/suppress.ts b/packages/core/src/suppress.ts index 99c407f..01940b1 100644 --- a/packages/core/src/suppress.ts +++ b/packages/core/src/suppress.ts @@ -39,7 +39,7 @@ type SuppressedErrorConstructor = new ( * * Never built via `using`/`await using`: native disposal constructs * `new SuppressedError(disposalError, originalError)`, making the *teardown* failure primary - * (`docs/knowledge/resource-management.md:72`) — the inverse of what RECOV-12 requires. + * (`docs/knowledge/harvested/resource-management.md:72`) — the inverse of what RECOV-12 requires. * * @param error - the primary throwable; stays primary. * @param suppressed - the secondary throwable raised while unwinding. @@ -65,14 +65,14 @@ export function suppress( * The stand-in {@link suppress} builds on runtimes without the native class. Mirrors its observable * shape — `name`, `error`, `suppressed` — so a caller never has to branch on which one it received. * - * `name` is pinned to `'SuppressedError'` rather than following `docs/knowledge/error-handling.md`'s + * `name` is pinned to `'SuppressedError'` rather than following `docs/knowledge/harvested/error-handling.md`'s * `this.name = new.target.name`: the point of this class is to be indistinguishable from the native * one, and reporting `FallbackSuppressedError` in a stack trace would make the runtime the reader is * on part of the error's identity. * * Exported so its shape is unit-testable directly. The alternative — deleting * `globalThis.SuppressedError` inside a test to force the fallback branch — would not survive - * parallel execution, which `docs/knowledge/testing.md:50` requires of every test. + * parallel execution, which `docs/knowledge/harvested/testing.md:50` requires of every test. * * @internal */ diff --git a/packages/shrink-test/src/fixture-app.ts b/packages/shrink-test/src/fixture-app.ts index 1f0b21e..5208661 100644 --- a/packages/shrink-test/src/fixture-app.ts +++ b/packages/shrink-test/src/fixture-app.ts @@ -107,7 +107,7 @@ function probeDisposalSymbol(transport: Transport): boolean { * `instanceof`: `TransportFailureError` is thrown by `@dexpace/transport-fetch` and its base class * `IoError` is imported here from `@dexpace/core`, so the check passes only if the bundle contains * exactly ONE copy of core's class identity. Two copies -- the dual-package hazard - * `docs/knowledge/tooling-and-quality-gates.md` names, and the risk this port substitutes for the + * `docs/knowledge/harvested/tooling-and-quality-gates.md` names, and the risk this port substitutes for the * reference's reflective keep-rules (`NFR-8`, deviation-ledger item 10) -- make it silently false * while every type still checks. Second, that a real serde round trip still works once the codec has * been through the same minifier. Third, that the module-scope disposal installs are still there -- diff --git a/packages/shrink-test/src/run-shrink-guard.test.ts b/packages/shrink-test/src/run-shrink-guard.test.ts index fc1a6fc..030e79b 100644 --- a/packages/shrink-test/src/run-shrink-guard.test.ts +++ b/packages/shrink-test/src/run-shrink-guard.test.ts @@ -6,8 +6,10 @@ // declared Node floor are still present and callable after tree-shaking -- see // `fixture-app.ts`'s `probeDisposalSymbol`). // Substitutes for NFR-8's keep-configuration, which this port ships nothing for by design -- see the -// Phase 9 deviation ledger and docs/knowledge/deliberate-deviations.md:55 (that corpus file is flagged -// stale as of 2026-08-30; docs/deviations.md section 10 is the current statement). +// Phase 9 deviation ledger and item 11 of +// docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md, the normative ledger +// (docs/deviations.md is its as-built audit). The knowledge corpus deliberately holds no copy: a +// register goes stale on the next append -- see docs/knowledge/notes/deliberate-deviations.md. import {describe, expect, test} from 'bun:test'; import {runShrinkGuard} from './run-shrink-guard.js'; diff --git a/packages/transport-fetch/src/fetch-transport.ts b/packages/transport-fetch/src/fetch-transport.ts index 7900f88..e76236d 100644 --- a/packages/transport-fetch/src/fetch-transport.ts +++ b/packages/transport-fetch/src/fetch-transport.ts @@ -326,7 +326,7 @@ if (typeof Symbol.asyncDispose === 'symbol') { * `close()` is a sanctioned no-op and `send()` keeps working after it (SEAM-15). There is no proxy * support at all; see {@link FetchTransportOptions}. * - * `close()` is the single teardown path `docs/knowledge/resource-management.md` asks for. A + * `close()` is the single teardown path `docs/knowledge/harvested/resource-management.md` asks for. A * `[Symbol.asyncDispose]` delegating to it is installed at run time **when the runtime has the * symbol**, which this package's declared floor (`engines.node >=20.3`) does not — it arrived in Node * 20.4. The return type therefore does not promise `AsyncDisposable`: claiming it would type-check diff --git a/packages/transport-undici/src/undici-transport.ts b/packages/transport-undici/src/undici-transport.ts index d5cdfd8..a901c97 100644 --- a/packages/transport-undici/src/undici-transport.ts +++ b/packages/transport-undici/src/undici-transport.ts @@ -576,7 +576,7 @@ if (typeof Symbol.asyncDispose === 'symbol') { * Creates a `Transport` backed by `undici` — the full-featured option, with connection-pool control, * proxy support, and real `close()` semantics over the dispatchers it owns. * - * `close()` is the single teardown path `docs/knowledge/resource-management.md` asks for, and the one + * `close()` is the single teardown path `docs/knowledge/harvested/resource-management.md` asks for, and the one * that actually destroys the dispatchers this transport owns. A `[Symbol.asyncDispose]` delegating to * it is installed at run time **when the runtime has the symbol**, which this package's declared floor * (`engines.node >=20.3`) does not — it arrived in Node 20.4. The return type therefore does not diff --git a/scripts/knowledge-drift.mjs b/scripts/knowledge-drift.mjs new file mode 100644 index 0000000..50f12d8 --- /dev/null +++ b/scripts/knowledge-drift.mjs @@ -0,0 +1,195 @@ +// SPDX-License-Identifier: MIT +// scripts/knowledge-drift.mjs +// +// Reports what has gone stale in `docs/knowledge/`, in two dimensions: +// +// sources — every sha256 recorded in `harvested/SOURCES.md` against the file +// on disk, so you can see which harvested entries describe a +// document that has since changed. +// keys — every `/<8 hex>` a note cites, against the corpus, so you +// can see which notes name a rule whose text no longer exists. A +// key digests entry text; a re-harvest that rewords a rule breaks +// the citation, and that is precisely when the note needs revisiting. +// +// Named for its subject rather than a verb, like `knowledge.mjs` and +// `changeset.mjs`: the `verify-*.mjs` prefix in this directory belongs to the +// blocking gates, and this is a report. +// +// A report, not a gate, and deliberately not in CI: no drift state fails it. +// (A manifest that is missing or malformed still exits 2 — that is the report +// being unable to run, not a state it reports.) Two reasons. The styleguide +// root is a sibling repository addressed by an absolute path on the harvest +// machine, so 16 of the sources simply do not exist in a CI checkout — they are +// NOT VERIFIABLE, never a failure. And drift is normal: a design chapter that a +// phase edits to record an outcome SHOULD drift, and the fix is a re-harvest, +// which is a user-invoked skill rather than something CI can do. +// +// The states are OK, DRIFT, NOT VERIFIABLE and UNREADABLE. +import {createHash} from 'node:crypto'; +import {readFileSync} from 'node:fs'; +import {join} from 'node:path'; +import {fileURLToPath} from 'node:url'; + +import { + danglingKeys, + derivePrefixes, + loadCanonicalIds, + loadCorpus, +} from './knowledge.mjs'; + +const repoRoot = fileURLToPath(new URL('..', import.meta.url)); +const sourcesPath = join( + repoRoot, + 'docs', + 'knowledge', + 'harvested', + 'SOURCES.md', +); + +// `| \`path\` | role | \`sha\` | date |` — the sha cell sometimes carries an +// annotation after the digest, so take the first backticked token in it. +const SOURCE_ROW = /^\|\s*`([^`]+)`\s*\|\s*([^|]*?)\s*\|\s*`([0-9a-f]+)`/; +// Any data row of the manifest table, parseable or not. Counting these is what +// turns "47 sources OK" from a count of rows that happened to match into a +// statement about the table: a row with an upper-case or empty digest used to +// vanish, and the summary line reported the smaller number as if it were all. +const ANY_ROW = /^\|\s*`([^`]+)`\s*\|/; +// A digest short enough to collide by accident is not a pin. The manifest +// records 12 hex; comparing at the recorded width alone would let a truncated +// row compare clean forever. +const MIN_SHA_LENGTH = 12; + +function parseSources(text) { + const rows = []; + let listed = 0; + for (const line of text.split('\n')) { + if (!ANY_ROW.test(line)) continue; + listed += 1; + const match = SOURCE_ROW.exec(line); + if (match) rows.push({path: match[1], role: match[2], sha: match[3]}); + } + if (rows.length === 0) { + throw new Error( + `parsed zero source rows out of ${sourcesPath}; its table format changed`, + ); + } + if (rows.length < listed) { + throw new Error( + `${sourcesPath} lists ${listed} sources but only ${rows.length} parse; ` + + 'the rest carry a malformed digest cell and would be silently skipped', + ); + } + for (const row of rows) { + if (row.sha.length >= MIN_SHA_LENGTH) continue; + throw new Error( + `${row.path} is pinned to a ${row.sha.length}-character digest; at least ` + + `${MIN_SHA_LENGTH} are needed for the comparison to mean anything`, + ); + } + return rows; +} + +// The manifest records a truncated digest, so compare at the recorded width. +// +// Only ENOENT is NOT VERIFIABLE. An unreadable or wrong-typed path reported as +// "not present" would hide inside the one state this report teaches the reader +// to ignore — off the harvest machine, 16 sources are legitimately absent. +function stateOf(row) { + let bytes; + try { + bytes = readFileSync( + row.path.startsWith('/') ? row.path : join(repoRoot, row.path), + ); + } catch (error) { + if (error.code === 'ENOENT') return {state: 'NOT VERIFIABLE', actual: null}; + return {state: 'UNREADABLE', actual: null, detail: error.code}; + } + const actual = createHash('sha256') + .update(bytes) + .digest('hex') + .slice(0, row.sha.length); + return {state: actual === row.sha ? 'OK' : 'DRIFT', actual}; +} + +function detailFor(row, result) { + if (result.state === 'DRIFT') { + return `recorded ${row.sha}, actual ${result.actual}`; + } + if (result.state === 'UNREADABLE') return `read failed: ${result.detail}`; + return 'file not present in this checkout'; +} + +// A note names the rule it overrides by key. Report every citation that no +// longer resolves — the rule was reworded, so the note is describing something +// that is not there any more. +function reportKeys() { + const entries = loadCorpus(derivePrefixes(loadCanonicalIds())); + const dangling = danglingKeys(entries); + for (const {note, cited} of dangling) { + process.stdout.write( + `STALE KEY\t${note}\tcites ${cited}, which no entry carries\n`, + ); + } + const cited = entries + .filter(entry => entry.origin === 'note') + .reduce((total, note) => total + note.overrides.length, 0); + process.stdout.write( + `\n${cited} note citation(s) resolve, ${dangling.length} do not.\n`, + ); + if (dangling.length > 0) { + process.stdout.write( + 'A stale key means the harvested rule was reworded or re-harvested. ' + + 'Re-read the rule, then update the note to the key it prints now.\n', + ); + } +} + +function main() { + const rows = parseSources(readFileSync(sourcesPath, 'utf8')); + const counts = {OK: 0, DRIFT: 0, 'NOT VERIFIABLE': 0, UNREADABLE: 0}; + + for (const row of rows) { + const result = stateOf(row); + counts[result.state] += 1; + if (result.state === 'OK') continue; + process.stdout.write( + `${result.state}\t${row.path}\t${detailFor(row, result)}\n`, + ); + } + + process.stdout.write( + `\n${rows.length} harvested sources: ${counts.OK} OK, ` + + `${counts.DRIFT} DRIFT, ${counts['NOT VERIFIABLE']} NOT VERIFIABLE, ` + + `${counts.UNREADABLE} UNREADABLE.\n`, + ); + if (counts.DRIFT > 0) { + process.stdout.write( + 'A drifted source means the harvested entries derived from it describe an ' + + 'older revision. Re-harvest that source, or record what changed as a ' + + 'note under docs/knowledge/notes/. This check never fails the build.\n', + ); + } + if (counts['NOT VERIFIABLE'] > 0) { + process.stdout.write( + 'NOT VERIFIABLE is expected off the harvest machine: the styleguide root ' + + 'is a sibling repository at an absolute path. It is not a failure.\n', + ); + } + reportKeys(); + return 0; +} + +export {parseSources, stateOf, detailFor}; + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + try { + process.exitCode = main(); + } catch (error) { + const messages = []; + for (let current = error; current; current = current.cause) { + messages.push(current.message); + } + process.stderr.write(`${messages.join('\n caused by: ')}\n`); + process.exitCode = 2; + } +} diff --git a/scripts/knowledge.mjs b/scripts/knowledge.mjs index fd24927..b550ac2 100644 --- a/scripts/knowledge.mjs +++ b/scripts/knowledge.mjs @@ -1,15 +1,25 @@ // SPDX-License-Identifier: MIT // scripts/knowledge.mjs // -// Query surface over `docs/knowledge/`. The corpus is 39 topic files and ~1470 -// harvested entries; without a filter, answering "what do we already know about -// RETRY-12" means reading a 20 KB file. This turns that into a query that -// returns the handful of entries that actually cite the requirement. +// Query surface over `docs/knowledge/`. The corpus is ~1.5k entries across two +// trees; without a filter, answering "what do we already know about RETRY-12" +// means reading a 20 KB file. This turns that into a query that returns the +// handful of entries that actually cite the requirement. // -// Not a gate. Nothing in `.github/workflows/` runs this — `--coverage` is a -// report you run by hand when annotating the corpus, not a blocking check. +// Two trees, one query surface: +// harvested/ generated by `knowledge-harvest`, never hand-edited. A `` +// sha there digests the WHOLE source file, so a hand edit inside +// an entry is invisible to the next harvest — it is regenerated +// or duplicated. Hence: no hand edits, ever. +// notes/ hand-written during implementation. Role `review`, a manual +// `sha:` marker. What the implementation found, which outranks +// what the documents say. +// +// `verify-knowledge-structure.mjs` is the gate that keeps the two apart. This +// file is not a gate — `--coverage` is a report you run by hand. // // Zero dependencies, plain Node ESM, same shape as the `verify-*.mjs` scripts. +import {createHash} from 'node:crypto'; import {readFileSync, readdirSync} from 'node:fs'; import {join, basename} from 'node:path'; import {fileURLToPath} from 'node:url'; @@ -17,6 +27,8 @@ import {parseArgs} from 'node:util'; const repoRoot = fileURLToPath(new URL('..', import.meta.url)); const knowledgeDir = join(repoRoot, 'docs', 'knowledge'); +const harvestedDir = join(knowledgeDir, 'harvested'); +const notesDir = join(knowledgeDir, 'notes'); const appendixCPath = join( repoRoot, 'docs', @@ -24,9 +36,10 @@ const appendixCPath = join( 'appendix-c-consolidated-normative-requirement-index.md', ); -// `INDEX.md` is a generated topic table and `SOURCES.md` a provenance manifest; -// neither holds entries, and both would parse as noise. -const NON_TOPIC_FILES = new Set(['INDEX.md', 'SOURCES.md']); +// `INDEX.md` is a generated topic table, `SOURCES.md` a provenance manifest and +// `README.md` the two-tree contract; none holds entries, and all would parse as +// noise. +const NON_TOPIC_FILES = new Set(['INDEX.md', 'SOURCES.md', 'README.md']); // Appendix B is the conformance-test checklist. Its entries roll several // requirement IDs into one "the suite verifies X, Y, Z" sentence, so they make @@ -34,10 +47,20 @@ const NON_TOPIC_FILES = new Set(['INDEX.md', 'SOURCES.md']); // resolve ONLY to a roll-up — a silent wrong answer unless it is called out. const ROLLUP_SOURCE = 'appendix-b-conformance-test-checklist'; -// Provenance roles the corpus uses, most common first. +// Provenance roles the corpus uses, most common first. `review` is the notes +// tree's role and appears nowhere under `harvested/`. const ROLES = ['spec', 'design', 'styleguide', 'review']; -// The six sections every harvested topic file carries, in emission order. +// The two trees, and the value of an entry's `origin` field for each. +const ORIGINS = ['harvested', 'note']; +const TREES = [ + {origin: 'harvested', dir: harvestedDir, required: true}, + {origin: 'note', dir: notesDir, required: false}, +]; + +// The six section names the parser recognises, in emission order. No file +// carries all six: `Superseded` exists only under `notes/`, where an override +// is recorded — `verify-knowledge-structure.mjs` rejects one under `harvested/`. const SECTIONS = [ 'Rules', 'Constraints', @@ -177,16 +200,59 @@ function parseSub(inner) { }; } -function parseFile(path, prefixes) { +// A citable name for one entry, stable across a re-order and a re-harvest. +// The line number is not that name: an entry moves whenever a neighbour is +// added. The `` sha is not either — it digests the whole source file, so +// every entry harvested from one file shares it. The digest of the entry's own +// text changes when, and only when, the rule changes, which is what a note +// naming a broken rule wants: a changed rule needs a fresh check. +function entryKey(topic, text) { + const hash = createHash('sha256').update(text.trimEnd()).digest('hex'); + return `${topic}/${hash.slice(0, 8)}`; +} + +// A topic file authored on Windows, or saved with a BOM, is the one input that +// fails silently rather than loudly: `\r` defeats BULLET_START's `$` and a BOM +// defeats SECTION_HEADING, so the file parses to zero entries or to entries +// with no section, and every downstream gate then reports OK over a hole. The +// repo has no `.gitattributes`, so `core.autocrlf=true` is one contributor +// away. Normalize both at the door. +function readTopicFile(path) { + let text; + try { + text = readFileSync(path, 'utf8'); + } catch (cause) { + throw new Error(`cannot read the topic file ${path}`, {cause}); + } + return text.replace(/^\uFEFF/, '').split(/\r?\n/); +} + +// One bullet carrying two `` lines is malformed, but overwriting the first +// with the second drops its sources — and a source is what the structural gate +// checks, so the malformed half would escape the check. Accumulate instead, and +// let the gate see everything the entry cites. +function mergeSub(entry, parsed, subLine) { + if (entry.subLine === null) { + Object.assign(entry, parsed, {subLine}); + return; + } + entry.roles.push(...parsed.roles); + entry.sources.push(...parsed.sources); + entry.subLine += ` ${subLine}`; +} + +function parseFile(path, prefixes, origin = 'harvested') { const entries = []; - const lines = readFileSync(path, 'utf8').split('\n'); + const lines = readTopicFile(path); const file = basename(path); + const topic = file.replace(/\.md$/, ''); let section = null; let current = null; const flush = () => { if (!current) return; current.reqs = extractIds(current.text, prefixes); + current.key = entryKey(topic, current.text); entries.push(current); current = null; }; @@ -201,7 +267,7 @@ function parseFile(path, prefixes) { const sub = SUB_LINE.exec(line); if (sub && current) { - Object.assign(current, parseSub(sub[1]), {subLine: line.trim()}); + mergeSub(current, parseSub(sub[1]), line.trim()); continue; } @@ -210,6 +276,11 @@ function parseFile(path, prefixes) { flush(); current = { file, + topic, + origin, + key: null, + overriddenBy: [], + overrides: [], line: index + 1, section, text: bullet[1], @@ -239,13 +310,86 @@ function parseFile(path, prefixes) { } function loadCorpus(prefixes) { - return topicFiles().flatMap(name => - parseFile(join(knowledgeDir, name), prefixes), + const entries = topicFiles().flatMap(({path, origin}) => + parseFile(path, prefixes, origin), ); + return linkOverrides(entries); +} + +// A note names the harvested rule it overrides by that rule's key. Resolve those +// references once, at load, and hang the answer on both ends. +// +// Without this the override is prose inside one note and nothing else: a query +// that lands on the harvested entry — by `--section reference`, by a bare word, +// by anything that does not also return the note — hands back a rule the +// implementation has already overruled, with no sign that it did. The corpus's +// highest-stakes entry is exactly that case: the pagination snippet whose note +// records that following it "would have shipped a MUST violation the checklist +// could not catch". +// +// `overriddenBy` is also how a stale reference becomes visible. A key digests +// entry text, so a re-harvest that rewords a rule leaves the citing note +// pointing at nothing; `danglingKeys` is that list, and `--key` reports it. +function linkOverrides(entries) { + const byKey = new Map(entries.map(entry => [entry.key, entry])); + for (const note of entries) { + if (note.origin !== 'note') continue; + for (const [, cited] of note.text.matchAll(CITED_KEY)) { + const target = byKey.get(cited); + if (target === undefined || target === note) continue; + target.overriddenBy.push(entryLocation(note)); + note.overrides.push(cited); + } + } + return entries; +} + +// A key inside an entry's prose, backticked: `pagination/81881061`. +const CITED_KEY = /`([a-z0-9-]+\/[0-9a-f]{8})`/g; + +function danglingKeys(entries) { + const known = new Set(entries.map(entry => entry.key)); + const dangling = []; + for (const note of entries) { + if (note.origin !== 'note') continue; + for (const [, cited] of note.text.matchAll(CITED_KEY)) { + if (!known.has(cited)) dangling.push({note: entryLocation(note), cited}); + } + } + return dangling; } +// Every topic file in both trees, harvested first, as records rather than bare +// names: a caller needs the tree an entry came from, and `pagination.md` exists +// in both. function topicFiles() { - return readdirSync(knowledgeDir) + return TREES.flatMap(({origin, dir, required}) => + readTree(dir, required).map(name => ({ + file: name, + topic: name.replace(/\.md$/, ''), + path: join(dir, name), + origin, + })), + ); +} + +function readTree(dir, required) { + let names; + try { + names = readdirSync(dir); + } catch (cause) { + if (required) { + throw new Error( + `cannot read the harvested corpus at ${dir}; docs/knowledge/ is two ` + + 'trees (harvested/ and notes/) and the harvested one is not optional', + {cause}, + ); + } + // The notes tree holds a file only where a note exists. None yet is a + // legitimate state, not a broken checkout. + return []; + } + return names .filter(name => name.endsWith('.md') && !NON_TOPIC_FILES.has(name)) .sort(); } @@ -282,74 +426,169 @@ function chaptersOf(entry) { // Filtering // --------------------------------------------------------------------------- -function buildFilters(options, positionals, canonicalIds) { - const reqs = (options.req ?? []).flatMap(value => value.split(',')); +// `--req A,B --req C` and `--req A --req B` mean the same thing: values inside +// one filter OR. +// +// An empty value is dropped, and a filter whose values were ALL empty throws. +// `--topic ''` and the one-character typo `--topic 'a,b,'` would otherwise match +// every file — `''` is a substring of everything and a regex that matches +// everything — so a trailing comma in the skill's own comma-separated audit form +// silently turns a 55-entry query into the whole corpus, which is the one thing +// this tool exists to prevent. +function splitValues(values, flag) { + const supplied = (values ?? []).flatMap(value => value.split(',')); + const kept = supplied.filter(value => value.trim() !== ''); + if (supplied.length > 0 && kept.length === 0) { + throw new Error( + `${flag} was given only empty values; an empty value matches every ` + + 'entry, so this would print the whole corpus. Drop the flag, or a ' + + 'stray comma.', + ); + } + return kept; +} + +function warnUncanonicalIds(reqs, canonicalIds) { for (const id of reqs) { - if (!canonicalIds.has(id)) { - process.stderr.write( - `warning: ${id} is not in appendix C — it is not a canonical ` + - 'requirement ID, so no entry can legitimately cite it\n', + if (canonicalIds.has(id)) continue; + process.stderr.write( + `warning: ${id} is not in appendix C — it is not a canonical ` + + 'requirement ID, so no entry can legitimately cite it\n', + ); + } +} + +function parseRoles(options) { + return splitValues(options.role, '--role').map(value => { + if (!ROLES.includes(value)) { + throw new Error( + `unknown role '${value}'; the roles are ${ROLES.join(', ')}`, ); } - } + return value; + }); +} - const topics = (options.topic ?? []).flatMap(value => value.split(',')); +function parseOrigins(options) { + return splitValues(options.origin, '--origin').map(value => { + if (!ORIGINS.includes(value)) { + throw new Error( + `unknown origin '${value}'; the origins are ${ORIGINS.join(', ')} — ` + + 'harvested/ is what the documents say, notes/ what the ' + + 'implementation found', + ); + } + return value; + }); +} - const roles = (options.role ?? []) - .flatMap(value => value.split(',')) - .map(value => { - if (!ROLES.includes(value)) { - throw new Error( - `unknown role '${value}'; the roles are ${ROLES.join(', ')}`, - ); - } - return value; - }); +// An ID family, for the audit query: `--prefix HTTP` is every HTTP-N entry, +// which `--req` cannot express without listing all 92 of them. Validated +// against appendix C for the same reason `--req` warns — `--prefix UTF` would +// otherwise be a silent empty result rather than a typo. +function parseIdPrefixes(options, known) { + return splitValues(options.prefix, '--prefix').map(value => { + const prefix = value.trim().toUpperCase(); + if (!known.has(prefix)) { + throw new Error( + `'${prefix}' is not a requirement-ID prefix in appendix C, so no ` + + 'entry can legitimately cite one. The prefixes are ' + + `${[...known].sort().join(' ')}`, + ); + } + return prefix; + }); +} - // "styleguide 6.7" — the chapter is queryable, the sub-section number is not, - // so take the chapter and say plainly that the rest was dropped. - const chapters = (options.chapter ?? []) - .flatMap(value => value.split(',')) - .map(value => { - const match = /^(\d{1,2})(?:\.(\d+))?$/.exec(value.trim()); - if (!match) { - throw new Error( - `unknown chapter '${value}'; expected a styleguide chapter like 6 or 6.7`, - ); - } - if (match[2] !== undefined) { - process.stderr.write( - 'note: entries record a chapter file and line range, not section ' + - `numbers — querying chapter ${match[1]}, ignoring .${match[2]}. ` + - 'Narrow with bare words.\n', - ); - } - return String(Number(match[1])); - }); +// A key is `/<8 hex>`. Not validated against the corpus here: a key that +// resolves to nothing is exactly what a reader needs told — the note citing it +// is describing a rule whose text has changed — so it flows through to +// renderNoMatches, which says so. +const ENTRY_KEY = /^[a-z0-9-]+\/[0-9a-f]{8}$/; + +function parseKeys(options) { + return splitValues(options.key, '--key').map(value => { + const key = value.trim(); + if (!ENTRY_KEY.test(key)) { + throw new Error( + `'${key}' is not an entry key; a key is /<8 hex>, as printed ` + + 'after the section name on every result', + ); + } + return key; + }); +} - const sections = (options.section ?? []) - .flatMap(value => value.split(',')) - .map(value => { - const resolved = SECTIONS.find( - name => name.toLowerCase() === value.toLowerCase(), +// "styleguide 6.7" — the chapter is queryable, the sub-section number is not, +// so take the chapter and say plainly that the rest was dropped. +function parseChapters(options) { + return splitValues(options.chapter, '--chapter').map(value => { + const match = /^(\d{1,2})(?:\.(\d+))?$/.exec(value.trim()); + if (!match) { + throw new Error( + `unknown chapter '${value}'; expected a styleguide chapter like 6 or 6.7`, ); - if (!resolved) { - throw new Error( - `unknown section '${value}'; the six sections are ${SECTIONS.join(', ')}`, - ); - } - return resolved; - }); + } + if (match[2] !== undefined) { + process.stderr.write( + 'note: entries record a chapter file and line range, not section ' + + `numbers — querying chapter ${match[1]}, ignoring .${match[2]}. ` + + 'Narrow with bare words.\n', + ); + } + return String(Number(match[1])); + }); +} +function parseSections(options) { + return splitValues(options.section, '--section').map(value => { + const resolved = SECTIONS.find( + name => name.toLowerCase() === value.toLowerCase(), + ); + if (!resolved) { + throw new Error( + `unknown section '${value}'; the six sections are ${SECTIONS.join(', ')}`, + ); + } + return resolved; + }); +} + +// An empty pattern matches every entry, exactly as an empty `--topic` does, so +// it gets the same refusal rather than printing the corpus. +function parsePatterns(options, positionals) { + const sources = [...(options.grep ?? []), ...positionals]; + if (sources.length > 0 && sources.every(value => value.trim() === '')) { + throw new Error( + '--grep was given only empty values; an empty pattern matches every ' + + 'entry, so this would print the whole corpus.', + ); + } const patterns = []; for (const source of options.grep ?? []) { - patterns.push(new RegExp(source, 'i')); + if (source.trim() !== '') patterns.push(new RegExp(source, 'i')); } for (const word of positionals) { - patterns.push(new RegExp(escapeRegExp(word), 'i')); + if (word.trim() !== '') patterns.push(new RegExp(escapeRegExp(word), 'i')); } + return patterns; +} + +function buildFilters(options, positionals, canonicalIds) { + const reqs = splitValues(options.req, '--req'); + warnUncanonicalIds(reqs, canonicalIds); - return {reqs, topics, roles, chapters, sections, patterns}; + return { + reqs, + topics: splitValues(options.topic, '--topic'), + keys: parseKeys(options), + origins: parseOrigins(options), + prefixes: parseIdPrefixes(options, derivePrefixes(canonicalIds)), + roles: parseRoles(options), + chapters: parseChapters(options), + sections: parseSections(options), + patterns: parsePatterns(options, positionals), + }; } function escapeRegExp(value) { @@ -359,12 +598,30 @@ function escapeRegExp(value) { // Every supplied filter must hold — they AND, never OR. Within one filter, // multiple values OR (`--req A --req B` is "cites A or B"). function matches(entry, filters) { - const {reqs, topics, roles, chapters, sections, patterns} = filters; + const { + reqs, + topics, + keys, + origins, + prefixes, + roles, + chapters, + sections, + patterns, + } = filters; + if (keys.length > 0 && !keys.includes(entry.key)) return false; if (reqs.length > 0 && !reqs.some(id => entry.reqs.includes(id))) return false; + if ( + prefixes.length > 0 && + !entry.reqs.some(id => prefixes.includes(splitId(id)[0])) + ) { + return false; + } if (topics.length > 0 && !topics.some(t => entry.file.includes(t))) { return false; } + if (origins.length > 0 && !origins.includes(entry.origin)) return false; if (sections.length > 0 && !sections.includes(entry.section)) return false; if (roles.length > 0 && !roles.some(r => entry.roles.includes(r))) { return false; @@ -380,6 +637,9 @@ function matches(entry, filters) { function isEmptyFilter(filters) { return ( filters.reqs.length === 0 && + filters.keys.length === 0 && + filters.prefixes.length === 0 && + filters.origins.length === 0 && filters.topics.length === 0 && filters.roles.length === 0 && filters.chapters.length === 0 && @@ -421,7 +681,7 @@ function renderListReqs(index) { for (const id of [...index.keys()].sort(compareIds)) { const locations = index .get(id) - .map(entry => `${entry.file}:${entry.line}`) + .map(entry => entryLocation(entry)) .join(' '); out.push(`${id}\t${locations}`); } @@ -433,20 +693,33 @@ function renderListReqs(index) { function renderListTopics(entries) { const stats = new Map(); for (const entry of entries) { - const name = entry.file.replace(/\.md$/, ''); - if (!stats.has(name)) stats.set(name, {entries: 0, ids: new Set()}); - stats.get(name).entries += 1; - entry.reqs.forEach(id => stats.get(name).ids.add(id)); + const name = entry.topic; + if (!stats.has(name)) { + stats.set(name, {entries: 0, notes: 0, ids: new Set()}); + } + const row = stats.get(name); + if (entry.origin === 'note') row.notes += 1; + else row.entries += 1; + entry.reqs.forEach(id => row.ids.add(id)); } - const out = ['topic\tentries\tdistinct IDs']; - for (const [name, {entries: count, ids}] of [...stats].sort()) { - out.push(`${name}\t${count}\t${ids.size}`); + const out = ['topic\tentries\tdistinct IDs\tnotes']; + for (const [name, {entries: count, notes, ids}] of [...stats].sort()) { + out.push(`${name}\t${count}\t${ids.size}\t${notes}`); } - const idless = [...stats].filter(([, v]) => v.ids.size === 0).length; + // Count only harvested topics as styleguide-derived: a note-only topic has no + // requirement ID either, and calling it styleguide-derived is simply wrong. + const idless = [...stats].filter( + ([, v]) => v.ids.size === 0 && v.entries > 0, + ).length; + const noted = [...stats].filter(([, v]) => v.notes > 0).length; + const harvested = [...stats].filter(([, v]) => v.entries > 0).length; out.push(''); out.push( - `${stats.size} topic files. ${idless} carry no requirement ID at all — ` + - 'those are styleguide-derived and are only reachable topic-first.', + `${stats.size} topics, ${harvested} of them harvested. ${idless} ` + + 'harvested topics carry no requirement ID at all — those are ' + + 'styleguide-derived and are only reachable topic-first. ' + + `${noted} topics carry a hand-written note (\`--origin note\`), which ` + + 'states what the implementation found and overrides the harvested entry.', ); return out.join('\n'); } @@ -501,17 +774,49 @@ function renderCoverage(canonicalIds, index) { // Output // --------------------------------------------------------------------------- +// `pagination.md` exists in both trees, so a bare basename is ambiguous. A +// harvested entry prints as `pagination.md:134`, a note as +// `notes/pagination.md:5`. +function entryLocation(entry) { + const prefix = entry.origin === 'note' ? 'notes/' : ''; + return `${prefix}${entry.file}:${entry.line}`; +} + +function plural(count, noun) { + return `${count} ${noun}${count === 1 ? '' : 's'}`; +} + +// The tags an entry can carry, in the order that matters to a reader: an +// override first, because it changes whether the entry is still true. +function tagsOf(entry) { + const tags = (entry.overriddenBy ?? []).map(at => ` [overridden by ${at}]`); + if (isRollup(entry)) tags.push(' [appendix-B roll-up]'); + return tags.join(''); +} + function renderEntries(results, brief, filters) { const out = []; for (const entry of results) { - const tag = isRollup(entry) ? ' [appendix-B roll-up]' : ''; - out.push(`${entry.file}:${entry.line} (${entry.section})${tag}`); + const key = entry.key ? ` ${entry.key}` : ''; + out.push( + `${entryLocation(entry)} (${entry.section})${key}${tagsOf(entry)}`, + ); out.push(`- ${entry.text}`); if (!brief && entry.subLine) out.push(` ${entry.subLine}`); out.push(''); } - const files = new Set(results.map(entry => entry.file)); - out.push(`${results.length} entries across ${files.size} topic files`); + const files = new Set( + results.map(entry => entryLocation(entry).split(':')[0]), + ); + const notes = results.filter(entry => entry.origin === 'note').length; + out.push( + `${plural(results.length, 'entry').replace('entrys', 'entries')} across ` + + `${plural(files.size, 'topic file')}` + + (notes > 0 + ? `, ${notes} of them notes — a note states what the implementation ` + + 'found and overrides the harvested entry where it names one' + : ''), + ); // The silent wrong answer this tool can give: a `--req` that "hits" but whose // every hit merely names the ID in a conformance-checklist sentence. Exit 0 @@ -529,104 +834,214 @@ function renderEntries(results, brief, filters) { } // A zero-result query must never look like "the corpus has nothing to say" when -// it is really a typo or the wrong topic name, so spend the tokens on saying -// what nearby things do exist. +// it is really a typo, the wrong topic name, or two filters that cannot both +// hold. Spend the tokens on saying which of those it was. +// +// The dimension a hint blames has to be the dimension that emptied the result. +// A per-filter hint that fires unconditionally states falsehoods: with +// `--prefix HTTP --req PAGE-11` it used to report "PAGE-11 is canonical but no +// entry cites it yet", which is wrong — three entries cite it, none of them an +// HTTP one. So each dimension is first tested alone, and only a dimension that +// matches nothing on its own gets to explain itself. +const DIMENSIONS = [ + 'reqs', + 'keys', + 'prefixes', + 'origins', + 'topics', + 'roles', + 'sections', + 'chapters', + 'patterns', +]; + +function soloFilters(filters, dimension) { + const solo = Object.fromEntries(DIMENSIONS.map(name => [name, []])); + solo[dimension] = filters[dimension]; + return solo; +} + function renderNoMatches(filters, entries, index, canonicalIds) { const out = ['no matching entries.']; + const used = DIMENSIONS.filter(name => filters[name].length > 0); + const barren = used.filter( + name => !entries.some(entry => matches(entry, soloFilters(filters, name))), + ); + + if (barren.length === 0) { + const alone = used + .map( + name => + `${name} ${entries.filter(entry => matches(entry, soloFilters(filters, name))).length}`, + ) + .join(', '); + out.push( + ' every filter matches something on its own; no entry satisfies all ' + + `of them at once (matching alone: ${alone}). Filters AND together — ` + + 'drop one.', + ); + return out.join('\n'); + } + for (const name of barren) { + out.push(...HINTS[name](filters, entries, index, canonicalIds)); + } + return out.join('\n'); +} + +function hintForReqs(filters, entries, index, canonicalIds) { + const out = []; for (const id of filters.reqs) { const [prefix, number] = splitId(id); - if (!canonicalIds.has(id)) { - out.push( - ` ${id} is not a canonical requirement ID (not in appendix C).`, - ); - } else { - out.push(` ${id} is canonical but no entry cites it yet.`); - } + out.push( + canonicalIds.has(id) + ? ` ${id} is canonical but no entry cites it yet.` + : ` ${id} is not a canonical requirement ID (not in appendix C).`, + ); + // Never offer the queried ID back as its own nearest neighbour. const nearest = [...index.keys()] - .filter(other => splitId(other)[0] === prefix) + .filter(other => other !== id && splitId(other)[0] === prefix) .sort((a, b) => { const distance = Math.abs(splitId(a)[1] - number) - Math.abs(splitId(b)[1] - number); return distance === 0 ? compareIds(a, b) : distance; }) .slice(0, 5); - if (nearest.length > 0) { - out.push(` nearest cited ${prefix} IDs: ${nearest.join(' ')}`); - } else { - const prefixes = [ - ...new Set([...index.keys()].map(id2 => splitId(id2)[0])), - ]; - out.push( - ` no ${prefix} ID is cited anywhere. cited prefixes: ${prefixes.sort().join(' ')}`, - ); - } + out.push( + nearest.length > 0 + ? ` nearest cited ${prefix} IDs: ${nearest.join(' ')}` + : ` no ${prefix} ID is cited anywhere. cited prefixes: ${[ + ...new Set([...index.keys()].map(other => splitId(other)[0])), + ] + .sort() + .join(' ')}`, + ); const topics = topicsForPrefix(entries, prefix); if (topics.length > 0) { out.push(` topics carrying ${prefix} knowledge: ${topics.join(' ')}`); } } + return out; +} - for (const topic of filters.topics) { - const known = [...new Set(entries.map(entry => entry.file))]; - if (!known.some(file => file.includes(topic))) { - out.push( - ` no topic file matches '${topic}'. available: ` + - known.map(file => file.replace(/\.md$/, '')).join(' '), - ); - } - } +// A key that resolves to nothing is not a typo, it is news: the entry it named +// has been reworded, so whatever cites it — a note, an audit — is describing a +// rule that no longer exists in that form and needs a fresh check. +function hintForKeys(filters, entries) { + return filters.keys.map(key => { + const [topic] = key.split('/'); + const citedBy = entries + .filter(entry => entry.text.includes(`\`${key}\``)) + .map(entry => entryLocation(entry)); + const known = entries.some(entry => entry.topic === topic); + const where = + citedBy.length > 0 ? ` It is cited by ${citedBy.join(' ')}` : ''; + return known + ? ` no entry carries the key ${key}. A key digests the entry's text, ` + + `so a reworded or re-harvested rule gets a new one.${where}` + + (citedBy.length > 0 ? ', which needs updating.' : '') + : ` no entry carries the key ${key}, and no topic '${topic}' exists.`; + }); +} - for (const section of filters.sections) { - if (!entries.some(entry => entry.section === section)) { - out.push( - ` the ${section} section is empty across all 39 topic files — ` + - 'nothing has been harvested into it.', - ); - } - } +function hintForPrefixes(filters) { + return filters.prefixes.map(prefix => ` no entry cites any ${prefix} ID.`); +} - for (const chapter of filters.chapters) { - const known = [ - ...new Set(entries.flatMap(entry => chaptersOf(entry))), - ].sort((a, b) => Number(a) - Number(b)); - if (!known.includes(chapter)) { - out.push( - ` no entry cites styleguide chapter ${chapter}. harvested ` + - `chapters: ${known.join(' ')}`, - ); - } - } +function hintForOrigins(filters) { + return filters.origins.map(origin => + origin === 'note' + ? ' the notes tree carries only what an implementation found that ' + + 'overrides a harvested rule; it is meant to be small.' + : ' no harvested entry matches. harvested/ is the whole corpus bar the ' + + 'notes, so this is a filter combination, not an empty tree.', + ); +} - if (filters.patterns.length > 0 && filters.reqs.length === 0) { - out.push( - ' text filters are applied to entry text only; try --grep with a ' + - 'looser pattern, or drop --section/--topic.', - ); - } - return out.join('\n'); +function hintForTopics(filters, entries) { + const known = [...new Set(entries.map(entry => entry.topic))].sort(); + return filters.topics.map( + topic => + ` no topic file matches '${topic}'. available: ${known.join(' ')}`, + ); +} + +function hintForRoles(filters) { + return filters.roles.map(role => + role === 'review' + ? " no review-role entry matches. review is the notes tree's role; " + + 'under harvested/ it is a structural violation, so there are none.' + : ` no entry carries the role ${role}.`, + ); } +function hintForSections(filters) { + return filters.sections.map( + section => + ` the ${section} section holds no entry in either tree` + + (section === 'Superseded' + ? ' — it exists only under notes/, where an override is recorded.' + : '.'), + ); +} + +function hintForChapters(filters, entries) { + const known = [...new Set(entries.flatMap(entry => chaptersOf(entry)))].sort( + (a, b) => Number(a) - Number(b), + ); + return filters.chapters.map( + chapter => + ` no entry cites styleguide chapter ${chapter}. harvested chapters: ` + + known.join(' '), + ); +} + +function hintForPatterns() { + return [ + ' text filters are applied to entry text only; try --grep with a ' + + 'looser pattern, or fewer bare words.', + ]; +} + +const HINTS = { + reqs: hintForReqs, + keys: hintForKeys, + prefixes: hintForPrefixes, + origins: hintForOrigins, + topics: hintForTopics, + roles: hintForRoles, + sections: hintForSections, + chapters: hintForChapters, + patterns: hintForPatterns, +}; + function topicsForPrefix(entries, prefix) { const counts = new Map(); for (const entry of entries) { for (const id of entry.reqs) { if (splitId(id)[0] !== prefix) continue; - counts.set(entry.file, (counts.get(entry.file) ?? 0) + 1); + counts.set(entry.topic, (counts.get(entry.topic) ?? 0) + 1); } } return [...counts.entries()] .sort((a, b) => b[1] - a[1]) .slice(0, 3) - .map(([file]) => file.replace(/\.md$/, '')); + .map(([topic]) => topic); } const USAGE = `Usage: bun run knowledge [options] [words...] -Query docs/knowledge/. Different filters AND together; values within one filter OR. +Query docs/knowledge/ — both harvested/ (what the documents say) and notes/ +(what the implementation found; it overrides). Different filters AND together; +values within one filter OR. --req entries citing that requirement ID (repeatable, comma-ok). Comma form is the whole-task query: --req HTTP-13,HTTP-14 + --key the one entry with that key — how a note's citation is + resolved. An unknown key is reported, not an error: it + means the entry was reworded and the citation is stale. + --prefix a whole ID family: --prefix HTTP. The audit-scale filter. + --origin ${ORIGINS.join(' | ')} --topic topic files whose name contains any of these (substring) --section ${SECTIONS.map(s => s.toLowerCase()).join(' | ')} --role ${ROLES.join(' | ')} @@ -635,17 +1050,28 @@ Query docs/knowledge/. Different filters AND together; values within one filter bare words: case-insensitive substrings, all must match --brief drop provenance lines (~30% less output) --json machine-readable records - --list-topics the 39 topics with entry and distinct-ID counts + --list-topics every topic with entry, distinct-ID and note counts --list-reqs requirement-ID -> location map (~6k tokens; prefer --coverage) --coverage substantive vs roll-up-only vs uncited, per prefix --help +Every result carries a stable key, /<8 hex>, digested from the entry +text. Name a rule by that key in a note: it survives a re-order, and it changes +exactly when the rule's text does — including on a re-harvest that rewords it, +which is when the note needs revisiting. Resolve one with --key. A harvested +entry a note overrides prints [overridden by notes/...]. + +An unknown --role, --section, --chapter, --origin or --prefix exits 2: a typo +there is a silent empty result. An unknown --req only warns, because a +not-yet-canonical ID is a legitimate thing to ask about. + Exits 1 when a query matches nothing. Examples: bun run knowledge --req HTTP-13,HTTP-14,HTTP-15 # one task's whole ID set + bun run knowledge --origin note --brief # start of a phase: what we found + bun run knowledge --prefix HTTP --section rules # an audit group, by ID family bun run knowledge --chapter 6 interface class # "styleguide 6.7" - bun run knowledge --section conflicts --brief # open design-vs-styleguide calls bun run knowledge --topic pipeline --section rules --brief cursor fork `; @@ -655,6 +1081,9 @@ function main(argv) { allowPositionals: true, options: { req: {type: 'string', multiple: true}, + key: {type: 'string', multiple: true}, + prefix: {type: 'string', multiple: true}, + origin: {type: 'string', multiple: true}, topic: {type: 'string', multiple: true}, section: {type: 'string', multiple: true}, role: {type: 'string', multiple: true}, @@ -720,9 +1149,12 @@ function main(argv) { } export { + ROLES, + SECTIONS, loadCanonicalIds, derivePrefixes, extractIds, + entryKey, parseSub, parseFile, loadCorpus, @@ -732,8 +1164,11 @@ export { renderCoverage, renderEntries, renderListTopics, + renderNoMatches, isRollup, chaptersOf, + danglingKeys, + entryLocation, topicFiles, compareIds, main, diff --git a/scripts/knowledge.test.mjs b/scripts/knowledge.test.mjs index f289c9b..c8145bb 100644 --- a/scripts/knowledge.test.mjs +++ b/scripts/knowledge.test.mjs @@ -15,7 +15,10 @@ import { buildFilters, citationIndex, compareIds, + danglingKeys, derivePrefixes, + entryKey, + entryLocation, extractIds, loadCanonicalIds, loadCorpus, @@ -25,6 +28,7 @@ import { renderCoverage, renderEntries, renderListTopics, + renderNoMatches, isRollup, chaptersOf, topicFiles, @@ -163,7 +167,9 @@ test('the standard four-field splits role from source correctly', () => { test('the real corpus parses with one per bullet and no orphans', () => { const entries = loadCorpus(prefixes); - assert.equal(entries.length, 1470); + const harvested = entries.filter(entry => entry.origin === 'harvested'); + assert.equal(harvested.length, 1457); + assert.ok(entries.length > harvested.length, 'the notes tree is non-empty'); for (const entry of entries) { assert.ok(entry.subLine, `${entry.file}:${entry.line} lost its line`); assert.ok( @@ -361,25 +367,44 @@ test('an unknown chapter or role fails loudly, like an unknown section', () => { // --- topic listing ---------------------------------------------------------- -test('--list-topics covers every topic file and counts the ID-less ones', () => { +test('--list-topics covers every topic file in both trees', () => { const entries = loadCorpus(prefixes); const report = renderListTopics(entries); - assert.equal(topicFiles().length, 39); - for (const name of topicFiles()) { - assert.ok( - report.includes(name.replace(/\.md$/, '')), - `${name} missing from --list-topics`, - ); + const files = topicFiles(); + assert.equal( + files.filter(file => file.origin === 'harvested').length, + 38, + 'the harvested corpus is 38 topic files, the register having been dropped', + ); + assert.ok( + files.some(file => file.origin === 'note'), + 'the notes tree is discovered too', + ); + for (const {topic} of files) { + assert.ok(report.includes(topic), `${topic} missing from --list-topics`); } + const topics = new Set(files.map(file => file.topic)); + const harvested = new Set( + files.filter(file => file.origin === 'harvested').map(file => file.topic), + ); + assert.match( + report, + new RegExp( + `^${topics.size} topics, ${harvested.size} of them harvested\\.`, + 'm', + ), + 'the topic count is the union of both trees — deliberate-deviations is note-only', + ); // The prose count must agree with the table it summarises. This half tests the renderer, and it - // holds whatever the corpus says. + // holds whatever the corpus says. The count is scoped to harvested topics: a note-only topic is + // not styleguide-derived, so its row (entries 0) is not an ID-less harvested topic. const stated = Number( - /(\d+) carry no requirement ID at all/.exec(report)?.[1], + /(\d+) harvested topics carry no requirement ID at all/.exec(report)?.[1], ); const zeroIdRows = report .split('\n') - .filter(line => /^\S+\t\d+\t0$/.test(line)).length; + .filter(line => /^\S+\t[1-9]\d*\t0\t\d+$/.test(line)).length; assert.equal( stated, zeroIdRows, @@ -398,6 +423,7 @@ test('--list-topics covers every topic file and counts the ID-less ones', () => 15, 'ID-less topic count changed — update CLAUDE.md and knowledge-lookup/SKILL.md with it', ); + assert.match(report, /carry a hand-written note/); }); test('--coverage pins the substantive / roll-up / uncited split the docs quote', () => { @@ -430,7 +456,335 @@ test('--coverage pins the substantive / roll-up / uncited split the docs quote', ); assert.deepEqual( {substantive, rollup, uncited}, - {substantive: 386, rollup: 255, uncited: 4}, + {substantive: 385, rollup: 256, uncited: 4}, 'corpus coverage changed — update CLAUDE.md and knowledge-lookup/SKILL.md with the new numbers', ); }); + +test('a note and its harvested topic share a name but not a tree', () => { + const files = topicFiles(); + const pagination = files.filter(file => file.topic === 'pagination'); + assert.deepEqual( + pagination.map(file => file.origin).sort(), + ['harvested', 'note'], + 'pagination.md exists in both trees', + ); + assert.ok(pagination.every(file => file.path.endsWith('pagination.md'))); +}); + +test('an entry location names the tree, so the two are never confused', () => { + const entries = loadCorpus(prefixes); + const note = entries.find(entry => entry.origin === 'note'); + const harvested = entries.find(entry => entry.origin === 'harvested'); + assert.match(entryLocation(note), /^notes\/[a-z-]+\.md:\d+$/); + assert.match(entryLocation(harvested), /^[a-z-]+\.md:\d+$/); +}); + +// --- the two trees ---------------------------------------------------------- + +test('an entry records which tree it came from', () => { + const body = [ + '## Superseded', + '- A hand-written note.', + ' review · `docs/superpowers/specs/x.md` · high · sha:manual-note', + '', + ].join('\n'); + + const [harvested] = parseFile(fixture(body), prefixes, 'harvested'); + const [note] = parseFile(fixture(body), prefixes, 'note'); + + assert.equal(harvested.origin, 'harvested'); + assert.equal(note.origin, 'note'); +}); + +test('--origin selects one tree and rejects an unknown name', () => { + const harvested = {origin: 'harvested', roles: ['spec'], reqs: []}; + const note = {origin: 'note', roles: ['review'], reqs: []}; + + assert.ok(matches(note, filtersFor({origin: ['note']}))); + assert.ok(!matches(harvested, filtersFor({origin: ['note']}))); + assert.ok(matches(harvested, filtersFor({origin: ['harvested']}))); + assert.throws( + () => filtersFor({origin: ['notes']}), + /unknown origin 'notes'/, + ); +}); + +test('the corpus is both trees, and only notes carry the review role', () => { + const entries = loadCorpus(prefixes); + const notes = entries.filter(entry => entry.origin === 'note'); + + assert.ok(notes.length > 0, 'the notes tree should hold entries'); + assert.ok( + notes.every(entry => entry.roles.includes('review')), + 'every note is a review-role entry', + ); + assert.ok( + entries + .filter(entry => entry.origin === 'harvested') + .every(entry => !entry.roles.includes('review')), + 'no harvested entry carries the review role', + ); +}); + +// --- --prefix --------------------------------------------------------------- + +test('--prefix selects a whole requirement family', () => { + const http = {reqs: ['HTTP-7'], roles: ['spec']}; + const retry = {reqs: ['RETRY-1'], roles: ['spec']}; + const idless = {reqs: [], roles: ['styleguide']}; + + assert.ok(matches(http, filtersFor({prefix: ['HTTP']}))); + assert.ok(!matches(retry, filtersFor({prefix: ['HTTP']}))); + assert.ok(!matches(idless, filtersFor({prefix: ['HTTP']}))); + assert.ok(matches(retry, filtersFor({prefix: ['HTTP,RETRY']}))); +}); + +test('--prefix rejects a name appendix C does not define', () => { + assert.throws( + () => filtersFor({prefix: ['UTF']}), + /'UTF' is not a requirement-ID prefix in appendix C/, + ); +}); + +test('--prefix beats a --req list at reaching a whole family', () => { + const entries = loadCorpus(prefixes); + const hits = entries.filter(entry => + matches(entry, filtersFor({prefix: ['PAGE']})), + ); + assert.ok(hits.length > 0); + assert.ok( + hits.every(entry => entry.reqs.some(id => id.startsWith('PAGE-'))), + 'every hit cites a PAGE id', + ); +}); + +// --- the stable entry key --------------------------------------------------- + +test('an entry key is /<8 hex> derived from the entry text alone', () => { + const [entry] = parseFile( + fixture( + [ + '## Rules', + '- A rule about HTTP-1.', + ' spec · `docs/product-spec/04-core-http-domain-model.md:9` · high · sha:abc123', + '', + ].join('\n'), + ), + prefixes, + 'harvested', + ); + + assert.match(entry.key, /^topic\/[0-9a-f]{8}$/); + assert.equal(entry.key, entryKey('topic', 'A rule about HTTP-1.')); +}); + +test('an entry key survives a re-order but not a re-wording', () => { + const rule = '- A rule about HTTP-1.'; + const sub = + ' spec · `docs/product-spec/04-core-http-domain-model.md:9` · high · sha:abc123'; + + const first = parseFile( + fixture(['## Rules', rule, sub, '- Another rule.', sub, ''].join('\n')), + prefixes, + 'harvested', + ); + const moved = parseFile( + fixture(['## Rules', '- Another rule.', sub, rule, sub, ''].join('\n')), + prefixes, + 'harvested', + ); + + assert.notEqual(first[0].line, moved[1].line, 'the entry did move'); + assert.equal(first[0].key, moved[1].key, 'the key does not follow the line'); + assert.notEqual( + first[0].key, + first[1].key, + 'a different rule, a different key', + ); + assert.notEqual( + entryKey('topic', 'A rule about HTTP-1.'), + entryKey('topic', 'A rule about HTTP-2.'), + ); +}); + +test('the key ignores trailing whitespace and the topic scopes it', () => { + assert.equal(entryKey('t', 'A rule. '), entryKey('t', 'A rule.')); + assert.notEqual(entryKey('a', 'A rule.'), entryKey('b', 'A rule.')); +}); + +// --- parser robustness ------------------------------------------------------ + +test('a CRLF topic file parses, rather than silently yielding nothing', () => { + const body = [ + '## Rules', + '- A rule about HTTP-1.', + ' spec · `docs/product-spec/04-core-http-domain-model.md:9` · high · sha:abc123', + '', + ].join('\r\n'); + const entries = parseFile(fixture(body), prefixes, 'harvested'); + + assert.equal(entries.length, 1, 'CRLF must not empty the file'); + assert.equal(entries[0].section, 'Rules'); + assert.equal(entries[0].sources.length, 1); + assert.equal(entries[0].text, 'A rule about HTTP-1.'); +}); + +test('a BOM does not orphan every entry from its section', () => { + const body = + '' + + [ + '## Rules', + '- A rule.', + ' spec · `docs/product-spec/04-core-http-domain-model.md:9` · high · sha:abc', + '', + ].join('\n'); + const [entry] = parseFile(fixture(body), prefixes, 'harvested'); + assert.equal(entry.section, 'Rules'); +}); + +test('a second adds its sources instead of hiding the first', () => { + const [entry] = parseFile( + fixture( + [ + '## Rules', + '- A rule.', + ' review · `docs/superpowers/specs/x.md` · high · sha:manual', + ' spec · `docs/product-spec/04-core-http-domain-model.md:9` · high · sha:abc', + '', + ].join('\n'), + ), + prefixes, + 'harvested', + ); + assert.deepEqual(entry.roles, ['review', 'spec']); + assert.equal(entry.sources.length, 2); +}); + +test('an unreadable topic file names itself in the error', () => { + assert.throws( + () => parseFile(join(tmpdir(), 'knowledge-test-nonexistent.md'), prefixes), + /cannot read the topic file .*knowledge-test-nonexistent\.md/, + ); +}); + +// --- empty filter values ---------------------------------------------------- + +test('an empty filter value is rejected, not treated as "match everything"', () => { + assert.throws( + () => filtersFor({topic: ['']}), + /--topic was given only empty/, + ); + assert.throws(() => filtersFor({}, ['']), /--grep was given only empty/); + assert.throws(() => filtersFor({req: [',']}), /--req was given only empty/); +}); + +test('a trailing comma is dropped, not turned into a whole-corpus query', () => { + const filters = filtersFor({topic: ['pipeline,']}); + assert.deepEqual(filters.topics, ['pipeline']); +}); + +// --- --key and note overrides ---------------------------------------------- + +test('--key selects the single entry with that key', () => { + const entries = loadCorpus(prefixes); + const target = entries.find(entry => entry.origin === 'harvested'); + const hits = entries.filter(entry => + matches(entry, filtersFor({key: [target.key]})), + ); + assert.deepEqual( + hits.map(entry => entry.key), + [target.key], + ); +}); + +test('--key rejects anything that is not /<8 hex>', () => { + assert.throws(() => filtersFor({key: ['pagination']}), /is not an entry key/); + assert.throws( + () => filtersFor({key: ['pagination/xyz']}), + /is not an entry key/, + ); +}); + +test('a note links to the harvested entry it names, in both directions', () => { + const entries = loadCorpus(prefixes); + const note = entries.find( + entry => entry.origin === 'note' && entry.section === 'Superseded', + ); + assert.ok(note.overrides.length > 0, 'the note cites at least one key'); + + for (const key of note.overrides) { + const target = entries.find(entry => entry.key === key); + assert.ok(target, `${key} resolves`); + assert.ok( + target.overriddenBy.includes(entryLocation(note)), + 'the harvested entry points back at the note', + ); + } +}); + +test('every key a note cites resolves — a dangling one is a stale note', () => { + assert.deepEqual(danglingKeys(loadCorpus(prefixes)), []); +}); + +test('danglingKeys reports a citation whose entry has been reworded', () => { + const note = { + origin: 'note', + file: 'pagination.md', + line: 8, + text: 'Supersedes `pagination/deadbeef`.', + }; + assert.deepEqual(danglingKeys([note]), [ + {note: 'notes/pagination.md:8', cited: 'pagination/deadbeef'}, + ]); +}); + +test('an overridden harvested entry says so in its rendered header', () => { + const entries = loadCorpus(prefixes); + const overridden = entries.find(entry => entry.overriddenBy.length > 0); + const rendered = renderEntries([overridden], true, {reqs: []}); + assert.match(rendered, /\[overridden by notes\//); +}); + +// --- zero-result diagnosis -------------------------------------------------- + +test('an empty intersection is reported as one, not blamed on a filter', () => { + const entries = loadCorpus(prefixes); + const filters = filtersFor({prefix: ['HTTP'], req: ['PAGE-11']}); + const index = citationIndex(entries); + + assert.equal(entries.filter(entry => matches(entry, filters)).length, 0); + const rendered = renderNoMatches(filters, entries, index, canonicalIds); + assert.match(rendered, /every filter matches something on its own/); + assert.ok( + !rendered.includes('no entry cites it yet'), + 'PAGE-11 is cited; blaming it would be a false statement', + ); +}); + +test('a genuinely uncited ID is still named, and never as its own neighbour', () => { + const entries = loadCorpus(prefixes); + const filters = filtersFor({req: ['PAGE-9999']}); + const rendered = renderNoMatches( + filters, + entries, + citationIndex(entries), + canonicalIds, + ); + assert.match(rendered, /PAGE-9999 is not a canonical requirement ID/); + assert.match(rendered, /nearest cited PAGE IDs/); + assert.ok(!/nearest cited PAGE IDs:.*PAGE-9999/.test(rendered)); +}); + +test('a stale key is diagnosed as a reworded rule, not as a typo', () => { + const entries = loadCorpus(prefixes); + const filters = filtersFor({key: ['pagination/deadbeef']}); + const rendered = renderNoMatches( + filters, + entries, + citationIndex(entries), + canonicalIds, + ); + assert.match(rendered, /no entry carries the key pagination\/deadbeef/); + assert.match(rendered, /a key digests the entry's text/i); +}); diff --git a/scripts/verify-knowledge-structure.mjs b/scripts/verify-knowledge-structure.mjs new file mode 100644 index 0000000..0e0a739 --- /dev/null +++ b/scripts/verify-knowledge-structure.mjs @@ -0,0 +1,251 @@ +// SPDX-License-Identifier: MIT +// scripts/verify-knowledge-structure.mjs +// +// `docs/knowledge/` is two trees, and this is the gate that keeps them apart. +// +// harvested/ what the source documents say. Generated by `knowledge-harvest`, +// never hand-edited. +// notes/ what the implementation found. Hand-written, role `review`, a +// manual `sha:` marker, and it outranks the harvested entry it +// names. +// +// Why a gate rather than a convention: a `` sha digests the WHOLE source +// file, not the entry, so every entry harvested from one file carries the same +// value and an edit to an entry's text does not change it. The next harvest +// cannot see the edit — it regenerates the original text or writes a duplicate. +// Hand-written knowledge under `harvested/` is therefore not just untidy, it is +// scheduled for silent deletion. The three rules below are the smallest set +// that catches it, and none of them needs a file outside the repository. +// +// Structural only. It says nothing about whether an entry is TRUE — that is +// what the drift check (`bun run knowledge:drift`) and a re-harvest are for. +import {readFileSync, readdirSync} from 'node:fs'; +import {dirname, join, normalize} from 'node:path'; +import {fileURLToPath} from 'node:url'; + +import { + ROLES, + derivePrefixes, + entryLocation, + loadCanonicalIds, + loadCorpus, +} from './knowledge.mjs'; + +// The roles a harvested entry may carry: everything but `review`, which is the +// notes tree's. Derived from the CLI's list so a new role is a one-line change +// there, not two. +const HARVESTED_ROLES = ROLES.filter(role => role !== 'review'); + +// Below this the corpus is not small, it is broken — a parse that silently +// yields nothing (a CRLF topic file, a moved directory) would otherwise let this +// gate print OK over an empty tree, which is worse than failing. The floor is +// deliberately far below the real count (1457) so ordinary editing never trips +// it. +const MIN_HARVESTED_ENTRIES = 1000; + +const repoRoot = fileURLToPath(new URL('..', import.meta.url)); +const knowledgeDir = join(repoRoot, 'docs', 'knowledge'); +const sourcesPath = join( + repoRoot, + 'docs', + 'knowledge', + 'harvested', + 'SOURCES.md', +); + +const SOURCE_ROW = /^\|\s*`([^`]+)`\s*\|/; +// A `` source is `path/to/file.md:12-18`, `…:12`, or a bare path. +const LINE_SUFFIX = /:[\d,-]+$/; + +// The three source roots are not hardcoded: they are the directories the +// harvest manifest itself names. One of them is an absolute path on the harvest +// machine (the styleguide is a sibling repository), so hardcoding it would make +// this gate machine-specific — and deriving it means a fourth root is a +// SOURCES.md edit, reviewable in the same diff as the entries that use it. +function sourceRoots(text) { + const roots = new Set(); + for (const line of text.split('\n')) { + const match = SOURCE_ROW.exec(line); + if (match) roots.add(dirname(match[1])); + } + if (roots.size === 0) { + throw new Error( + `parsed zero source rows out of ${sourcesPath}; its table format changed ` + + 'and the allowlist of source roots cannot be derived', + ); + } + + // One row at a root's parent would widen the allowlist to everything beneath + // it — a future `docs/open-items.md` row makes `docs` a root, and then every + // `docs/superpowers/...` citation passes. Refuse rather than silently widen. + const sorted = [...roots].sort(); + for (const root of sorted) { + const swallowed = sorted.find( + other => other !== root && other.startsWith(`${root}/`), + ); + if (swallowed !== undefined) { + throw new Error( + `${sourcesPath} derives the source root '${root}', which contains ` + + `'${swallowed}'. A root that contains another admits everything ` + + 'beneath it; list sources at one level, not two.', + ); + } + } + return sorted; +} + +// `normalize` first: a `..` segment makes a string prefix test meaningless, and +// the point of this check is where the path actually lands. +function isUnderRoot(source, roots) { + const path = normalize(source.replace(LINE_SUFFIX, '')); + return roots.some(root => path.startsWith(`${normalize(root)}/`)); +} + +// Returns one line per violation, empty when the trees are clean. +function structuralViolations(entries, roots) { + const violations = []; + for (const entry of entries) { + if (entry.origin !== 'harvested') continue; + const at = `${entryLocation(entry)} (${entry.section})`; + + if (entry.roles.includes('review')) { + violations.push( + `${at}: role \`review\` under harvested/. A review-role entry is ` + + 'hand-written knowledge; move it to docs/knowledge/notes/.', + ); + } + // `review` is the label an honest hand edit wears. An invented role is what + // a careless one wears, and it used to pass every rule here. + for (const role of entry.roles) { + if (role === 'review' || HARVESTED_ROLES.includes(role)) continue; + violations.push( + `${at}: role \`${role}\` under harvested/, which is not one of ` + + `${HARVESTED_ROLES.join(', ')}. A harvested entry carries the role of ` + + 'the document it came from.', + ); + } + // The per-source loop below iterates zero times on an entry with no + // provenance line at all, so the strongest rule here never ran on the one + // shape a hand-written bullet is most likely to have. + if (entry.sources.length === 0) { + violations.push( + `${at}: no source. Every harvested entry carries a \`\` naming ` + + 'the document it came from; a bullet without one was written by hand.', + ); + } + if (entry.section === 'Superseded') { + violations.push( + `${at}: a Superseded entry under harvested/. Superseding is a ` + + 'judgement the implementation made; it belongs in docs/knowledge/notes/.', + ); + } + for (const source of entry.sources) { + if (isUnderRoot(source, roots)) continue; + violations.push( + `${at}: cites \`${source}\`, which is under none of the harvested ` + + `source roots (${roots.join(', ')}). Only a harvest of those roots ` + + 'belongs in harvested/.', + ); + } + } + return violations; +} + +// A fourth rule, beyond the three the trees themselves imply, because the +// accident it catches is silent: `knowledge-harvest` defaults its `--corpus` to +// `/docs/knowledge/`, so a run that forgets `--corpus docs/knowledge/harvested` +// writes a third copy of the corpus at the root. No query reads it — the CLI +// walks harvested/ and notes/ — so the knowledge is not wrong, it is invisible. +function strayTopicFiles(dir = knowledgeDir) { + return readdirSync(dir, {withFileTypes: true}) + .filter( + entry => + entry.isFile() && + entry.name.endsWith('.md') && + entry.name !== 'README.md', + ) + .map( + entry => + `docs/knowledge/${entry.name}: a topic file at the root of ` + + 'docs/knowledge/, which is neither tree — no query reads it. Move it ' + + 'into harvested/ or notes/. (A `knowledge-harvest` run without ' + + '`--corpus docs/knowledge/harvested` writes here.)', + ); +} + +// The mirror of rule 1, and fatal for the same reason rather than advisory: a +// note that does not say `review` reads, in a query result, exactly as though a +// source document had said it — and a warning inside a step that exits 0 is +// invisible in a green log. +function noteViolations(entries) { + return entries + .filter(entry => entry.origin === 'note' && !entry.roles.includes('review')) + .map( + entry => + `${entryLocation(entry)}: a note whose role is ` + + `\`${entry.role ?? 'none'}\`, not \`review\`. A note states what the ` + + 'implementation found; that is what the role says.', + ); +} + +// The scripts here wrap-and-rethrow with `{cause}`; printing only the outer +// message throws away the half that says which file. +function formatCauses(error) { + const messages = []; + for (let current = error; current; current = current.cause) { + messages.push(current.message); + } + return messages.join('\n caused by: '); +} + +function main() { + const roots = sourceRoots(readFileSync(sourcesPath, 'utf8')); + const entries = loadCorpus(derivePrefixes(loadCanonicalIds())); + const violations = [ + ...structuralViolations(entries, roots), + ...noteViolations(entries), + ...strayTopicFiles(), + ]; + + const harvested = entries.filter(entry => entry.origin === 'harvested'); + if (harvested.length < MIN_HARVESTED_ENTRIES) { + throw new Error( + `only ${harvested.length} harvested entries parsed, below the floor of ` + + `${MIN_HARVESTED_ENTRIES}. The corpus did not shrink by hand — a parse ` + + 'is failing (a CRLF or BOM topic file, a moved directory), and every ' + + 'rule below would pass vacuously over the hole.', + ); + } + + if (violations.length > 0) { + for (const violation of violations) { + process.stderr.write(`knowledge-structure violation: ${violation}\n`); + } + process.stderr.write( + `${violations.length} violation(s). docs/knowledge/harvested/ carries ` + + 'only harvested entries; hand-written knowledge lives in ' + + 'docs/knowledge/notes/ with role `review` and a manual sha marker.\n', + ); + return 1; + } + + const notes = entries.length - harvested.length; + process.stdout.write( + `knowledge structure OK: ${harvested.length} harvested entries, each with ` + + `a source under one of ${roots.length} roots and a role among ` + + `${HARVESTED_ROLES.join('/')}, none Superseded; ${notes} notes, all ` + + 'review-role; nothing stranded at the root.\n', + ); + return 0; +} + +export {sourceRoots, structuralViolations, noteViolations, strayTopicFiles}; + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + try { + process.exitCode = main(); + } catch (error) { + process.stderr.write(`${formatCauses(error)}\n`); + process.exitCode = 2; + } +} diff --git a/scripts/verify-knowledge-structure.test.mjs b/scripts/verify-knowledge-structure.test.mjs new file mode 100644 index 0000000..abfbe5d --- /dev/null +++ b/scripts/verify-knowledge-structure.test.mjs @@ -0,0 +1,207 @@ +// SPDX-License-Identifier: MIT +// scripts/verify-knowledge-structure.test.mjs +// +// Run with `bun run test:scripts`. Tests the detector, not the corpus: the +// corpus being clean today is what `bun run verify:knowledge-structure` says in +// CI, and a gate nobody has seen fail is a gate nobody trusts. +import assert from 'node:assert/strict'; +import {mkdirSync, mkdtempSync, writeFileSync} from 'node:fs'; +import {tmpdir} from 'node:os'; +import {join} from 'node:path'; +import {test} from 'node:test'; + +import { + noteViolations, + sourceRoots, + strayTopicFiles, + structuralViolations, +} from './verify-knowledge-structure.mjs'; + +const ROOTS = [ + '/home/u/styleguide/typescript', + 'docs/product-spec', + 'docs/sdk-design-nodejs', +]; + +function entry(overrides) { + return { + file: 'topic.md', + topic: 'topic', + origin: 'harvested', + line: 4, + section: 'Rules', + roles: ['spec'], + role: 'spec', + sources: ['docs/product-spec/04-core-http-domain-model.md:9'], + ...overrides, + }; +} + +test('the source roots are derived from the manifest, not hardcoded', () => { + const roots = sourceRoots( + [ + '# Harvested Sources', + '', + '| source | role | sha256 | last harvest |', + '| --- | --- | --- | --- |', + '| `/home/u/styleguide/typescript/01-formatting.md` | styleguide | `aaa` | 2026-07-25 |', + '| `docs/product-spec/04-core-http-domain-model.md` | spec | `bbb` | 2026-07-25 |', + '| `docs/product-spec/12-pagination.md` | spec | `ccc` | 2026-07-25 |', + ].join('\n'), + ); + assert.deepEqual(roots, [ + '/home/u/styleguide/typescript', + 'docs/product-spec', + ]); +}); + +test('a manifest that parses to nothing fails loudly', () => { + assert.throws(() => sourceRoots('# Harvested Sources\n'), /zero source rows/); +}); + +test('a clean pair of trees reports no violation', () => { + const entries = [ + entry({}), + entry({ + origin: 'note', + section: 'Superseded', + roles: ['review'], + role: 'review', + sources: [ + 'docs/superpowers/specs/2026-07-28-phase6c-pagination-design.md', + ], + }), + ]; + assert.deepEqual(structuralViolations(entries, ROOTS), []); + assert.deepEqual(noteViolations(entries), []); +}); + +test('a review-role entry under harvested/ is a violation', () => { + const [violation, ...rest] = structuralViolations( + [entry({roles: ['review'], role: 'review'})], + ROOTS, + ); + assert.equal(rest.length, 0); + assert.match(violation, /role `review` under harvested\//); + assert.match(violation, /topic\.md:4/); +}); + +test('a Superseded entry under harvested/ is a violation', () => { + const [violation] = structuralViolations( + [entry({section: 'Superseded'})], + ROOTS, + ); + assert.match(violation, /Superseded entry under harvested\//); +}); + +test('a harvested citing outside the three roots is a violation', () => { + const [violation] = structuralViolations( + [entry({sources: ['docs/superpowers/plans/2026-07-28-phase9.md:1097']})], + ROOTS, + ); + assert.match(violation, /cites `docs\/superpowers\/plans/); + assert.match(violation, /under none of the harvested source roots/); +}); + +test('a line range or a bare path both resolve to their root', () => { + const ranged = entry({ + sources: ['/home/u/styleguide/typescript/06-classes.md:168-183'], + }); + const bare = entry({sources: ['docs/sdk-design-nodejs/04-domain.md']}); + assert.deepEqual(structuralViolations([ranged, bare], ROOTS), []); +}); + +test('a prefix match is on a path segment, not on characters', () => { + const sibling = entry({sources: ['docs/product-spec-draft/04-core.md:9']}); + const [violation] = structuralViolations([sibling], ROOTS); + assert.match(violation, /docs\/product-spec-draft/); +}); + +test('the same shapes are allowed under notes/, which is hand-written', () => { + const note = entry({ + origin: 'note', + section: 'Superseded', + roles: ['review'], + role: 'review', + sources: ['docs/superpowers/plans/2026-07-28-phase9.md:1097'], + }); + assert.deepEqual(structuralViolations([note], ROOTS), []); +}); + +test('a topic file stranded at the root is caught', () => { + const dir = mkdtempSync(join(tmpdir(), 'knowledge-stray-')); + mkdirSync(join(dir, 'harvested')); + writeFileSync(join(dir, 'harvested', 'api-design.md'), '## Rules\n'); + assert.deepEqual(strayTopicFiles(dir), [], 'a tree is not a stray file'); + + writeFileSync(join(dir, 'README.md'), '# the two trees\n'); + assert.deepEqual( + strayTopicFiles(dir), + [], + 'README.md is the contract, not a topic', + ); + + writeFileSync(join(dir, 'pagination.md'), '## Rules\n'); + const [violation, ...rest] = strayTopicFiles(dir); + assert.equal(rest.length, 0); + assert.match(violation, /pagination\.md: a topic file at the root/); + assert.match(violation, /--corpus docs\/knowledge\/harvested/); +}); + +test('the live corpus has nothing stranded at its root', () => { + assert.deepEqual(strayTopicFiles(), []); +}); + +test('a harvested entry with no provenance line at all is caught', () => { + const [violation, ...rest] = structuralViolations( + [entry({sources: [], roles: [], role: null, subLine: null})], + ROOTS, + ); + assert.equal(rest.length, 0); + assert.match(violation, /no source/); + assert.match(violation, /written by hand/); +}); + +test('an invented role under harvested/ is caught, not just `review`', () => { + const [violation, ...rest] = structuralViolations( + [entry({roles: ['impl'], role: 'impl'})], + ROOTS, + ); + assert.equal(rest.length, 0); + assert.match(violation, /role `impl` under harvested\//); + assert.match(violation, /spec, design, styleguide/); +}); + +test('a source root that contains another root is refused, not widened', () => { + assert.throws( + () => + sourceRoots( + [ + '| `docs/product-spec/04-core.md` | spec | `aaa` | 2026-07-25 |', + '| `docs/open-items.md` | design | `bbb` | 2026-07-25 |', + ].join('\n'), + ), + /which contains/, + ); +}); + +test('a `..` segment cannot walk out of a source root', () => { + const escaping = entry({ + sources: ['docs/product-spec/../../etc/passwd'], + }); + const [violation] = structuralViolations([escaping], ROOTS); + assert.match(violation, /under none of the harvested source roots/); +}); + +test('a note that is not review-role is a violation, not a warning', () => { + const note = entry({origin: 'note', roles: ['design'], role: 'design'}); + assert.deepEqual( + structuralViolations([note], ROOTS), + [], + 'the harvested rules do not apply to a note', + ); + const [violation, ...rest] = noteViolations([note]); + assert.equal(rest.length, 0); + assert.match(violation, /notes\/topic\.md:4/); + assert.match(violation, /`design`, not `review`/); +}); diff --git a/tests/conformance/xcut/error-taxonomy.conformance.test.ts b/tests/conformance/xcut/error-taxonomy.conformance.test.ts index e8f9a31..a434b0c 100644 --- a/tests/conformance/xcut/error-taxonomy.conformance.test.ts +++ b/tests/conformance/xcut/error-taxonomy.conformance.test.ts @@ -103,7 +103,7 @@ describe('XCUT-6: a custom error type participates without editing the classifie test('retries an error type declared in this test file, unknown to classify.ts', async () => { // The port's retryability capability is subtyping, not a duck-typed `isRetryable` flag: the // cause-walk returns true for anything `instanceof IoError`, so extending it is what opts a new - // failure in with no classifier edit (deviation ledger item 17, deliberate-deviations.md:168). + // failure in with no classifier edit (deviation ledger item 17, docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md). class CustomTransientError extends IoError {} const pipeline = buildComposedPipeline({ transport: new ThrowingTransport(new CustomTransientError('transient')), diff --git a/tests/node-conformance/README.md b/tests/node-conformance/README.md index 68d3671..d46fb0b 100644 --- a/tests/node-conformance/README.md +++ b/tests/node-conformance/README.md @@ -20,7 +20,7 @@ grep proves the code is runtime-*agnostic in its imports*, which is a much weake on Node*. This layer is **thin and additive**, not a second unit suite. `bun test` stays the unit-test runner, unchanged -— `docs/knowledge/testing.md` mandates `bun:test` symbol imports, `setSystemTime`, and `--concurrent`, so +— `docs/knowledge/harvested/testing.md` mandates `bun:test` symbol imports, `setSystemTime`, and `--concurrent`, so migrating the suite to `node:test` would be a styleguide deviation plus a whole-suite rewrite, and it buys nothing for the pure-logic majority (`Headers`/`MediaType`/`QueryParams` parsing cannot behave differently on Node). diff --git a/tests/node-conformance/recovery-chain.test.mjs b/tests/node-conformance/recovery-chain.test.mjs index 849768c..73d5dbf 100644 --- a/tests/node-conformance/recovery-chain.test.mjs +++ b/tests/node-conformance/recovery-chain.test.mjs @@ -6,7 +6,7 @@ // Management proposal and is absent on this package's declared floor (`engines.node ">=20.3"`). A // `new SuppressedError(...)` written straight into `response-chain.ts` would pass `bun test` and then throw // `ReferenceError: SuppressedError is not defined` at a consumer's call time — exactly the `NFR-10` trap -// `docs/knowledge/tooling-and-quality-gates.md:60-61` describes. `suppress()` guards on the global; this +// `docs/knowledge/harvested/tooling-and-quality-gates.md:60-61` describes. `suppress()` guards on the global; this // file is what proves the guarded path actually works on the runtime the SDK ships to, at both ends of the // matrix. // @@ -51,7 +51,7 @@ describe('suppress() on the declared Node floor', () => { it('takes the branch this runtime actually has, and both legs of the matrix are covered', () => { // Not forced by deleting the global — that would not survive parallel execution - // (docs/knowledge/testing.md:50). The matrix is the forcing function: the pinned 20.3.0 leg has + // (docs/knowledge/harvested/testing.md:50). The matrix is the forcing function: the pinned 20.3.0 leg has // no native class and takes the fallback, `lts/*` has one and takes the native branch. Either // way the result must be usable without the caller knowing which. const native = globalThis.SuppressedError;