known-issues: fixed-in registry, version schemes and a ledger re-measure queue by 88plug · Pull Request #256 · 88plug/k3d-gpu

88plug · 2026-10-29T21:30:27Z

The known-issues part of the freshness CI/CD design. Every upstream bug, behaviour change and advisory k3d-gpu depends on now has a registry entry with its fixed-in release. CI fails when an "ours" pin stays on a version that has a known fix. A read-only queue lists the SUPPORTED ledger rows that were measured on versions older than today's pins.

What is added

hack/pinver.py: the version library. It is stdlib only, has a small CLI, and later freshness tooling imports it.
- Schemes: semver, zerover, k8s (v1.37.1+k3s1 / -k3s1), llama (bNNNNN builds and vX.Y.Z releases), calver, pep440, docker-suffix, eat (release-X.Y.Z and release-X.Y.Z-hotfix in one stream), apt (dpkg order) and hf-sha.
- API: parse, compare, comparable, is_prerelease, in_range (*, comma AND, || unions), check_range, update_type, breaking, classify and latest.
- update_type reports where two versions differ. breaking applies each scheme's rule: a zerover 0.24 -> 0.25 is a minor that is breaking, a k8s minor is breaking, and a llama.cpp build bump is a minor that is not breaking.
- latest skips prereleases, tags of another docker-suffix flavour, and anything not newer than the current version.
pins/known-issues.toml and hack/known-issues.
- check is offline and deterministic, with its own stdlib validator. It fails on:
  - a schema error, including a fixed_in that is inside affected;
  - a marker and its entry that disagree, in either direction;
  - a marker in a ledger or historical file;
  - a known-issue-begin with no known-issue-end;
  - a site regex that matches nothing;
  - an "ours" site inside affected while fixed_in is set. The message is bump <file> to >= <fixed_in>;
  - workaround markers that remain after every "ours" site reached fixed_in, while retained is empty.
- check only reports eat-stock sites in range and ledger_refs in range. It never fails on them. --json prints the full evaluation.
- poll reads GitHub through gh and writes nothing.
  - It finds the fix commit from fix.sha, the pull request's merge commit, or the issue timeline's closing commit.
  - It walks the fix repo's stable releases in version order, starting from the lowest site version. If that release already contains the fix, it walks down to the first release that does. A release counts as fixed when compare/<tag>...<sha> reads behind or identical.
  - For an entry with an image template, the image tag of the fixing release must resolve (scripts/images-lock digest).
- Two more fix forms:
  - {repo, path, contains}: fixed in the first release whose file contains the string. This is how kserve-4592 is tracked. The upstream PR merged in January, and every release since contains its commit, but no release's runtimes chart lists kserve-openvino.
  - {ecosystem, package}: reads the GitHub advisory database for advisories.
hack/ledger-freshness: read-only.
- It parses the Hypotheses table with ledger() from tests/ledger_drift_check.py. For each SUPPORTED row, it follows the Probe and Verdict cells' run anchors to the # Reproduce / # provenance blocks and **Reproduce:** paragraphs.
- It extracts the versions of vLLM CPU/XPU, OVMS (a version, or a digest that images.lock maps), llama.cpp, k3s, the node image and HF revisions. It compares them with today's "ours" pins from images.lock and VERSION.
- Queue order:
  1. rows in console/ledger-rules.json with classes;
  2. rows that docs claims cite;
  3. the rest.
- Within each tier, rows inside a known issue's range (priority-stale) come first.
tests/claims_check.sh --cited: prints the ledger rows that claims cite, using the guard's own claim and reference index, so the queue does not re-implement it.
Markers (known-issue: and known-issue-begin/end) at the workaround sites:
- console/model_opt.py (vllm-pr-51493; a single marker, since the workaround is one clause of a note string inside an if, and deleting a begin/end block there would leave the block empty);
- console/audit.py (eat-9);
- scripts/k3d-gpu, docs/first-run.md and docs/quickstart.md (linux-npu-driver-56);
- examples/kserve/openvino-runtime.yaml and install.sh (kserve-4592).
No markers are under bench/ or in EXPERIMENTS.md.
ci.yml: a new known-issues job at the end of the file. It has timeout-minutes: 10, contents: read, and checkout at the v7.0.1 SHA (gh api: latest release). It runs the three new suites and hack/known-issues check.

Seeded entries (each upstream state re-read with gh api on 2026-10-29)

id	upstream state	affected	fixed_in
vllm-45844	PR merged 2026-07-04 as 26eb8720; compare v0.25.0...26eb8720 = behind 106 / ahead 0, v0.24.0 diverged	<0.25.0	0.25.0
vllm-50801 (change)	PR merged 2026-08-03 as c8602c79; v0.28.0 behind, v0.27.0 diverged	<0.28.0	0.28.0
vllm-pr-51493	PR open	*	""
linux-npu-driver-56	issue open	*	""
kserve-4592	PR merged 2026-01-16; latest release v0.21.0; neither the kustomization nor the kserve-runtime-configs chart lists kserve-openvino at v0.21.0 or master	*	""
eat-9	issue open	*	""
litellm-1.83.14-advisories	7 advisories (see note)	<1.100.4 \|\| >=1.101.0,<1.101.3 \|\| >=1.102.0,<1.102.2 \|\| >=1.103.0,<1.103.1 \|\| >=1.104.0rc1,<1.104.0rc2	1.100.4
vllm-0.32-num-spec-tokens	not filed upstream (notes: "File upstream")	>=0.32.0	""

Sites by class:

ours: AMX_TEST_IMAGE and console/moe_predict.py (both v0.32.0), console/model_opt.py's vLLM version, HS_NPU_REL and KSERVE_VERSION.
eat-stock: integrations/eat/vllm-values.yaml (v0.24.0), eat-up.sh's EAT_TAG, gateway-values.yaml and the litellm-gateway requirements and constraints.

ledger_refs: H54a (bench/h54/h54.py) and H53 (bench/eatflow/live.sh) for vllm-45844, and H54q for the num_speculative_tokens bug.

Acceptance

hack/known-issues check on this branch (rebased on main 74ff63e): known-issues: ok. It prints 8 issues, 12 sites, 10 markers, and 12 report lines: the eat-stock vLLM v0.24.0 and LiteLLM 1.83.14 sites, the H54a and H53 ledger refs, and the unfixed ours sites.

Moving the ours vLLM CPU pin to v0.24.0 in a copy of the registry and its files fails (tests/known_issues_test.py, RealRepo):

FAIL scripts/k3d-gpu: vllm-openai-cpu 0.24.0 is inside vllm-45844's affected range <0.25.0, fixed in 0.25.0; bump scripts/k3d-gpu to >= 0.25.0
known-issues: 1 problem(s)

Live, read-only hack/known-issues poll --json, condensed (exit 0, 19 s, re-run after the rebase):

vllm-45844                  released     fixed_in 0.25.0  tag v0.25.0  2026-07-11T20:06:44Z  https://github.com/vllm-project/vllm/releases/tag/v0.25.0
                                         image vllm/vllm-openai-cpu:v0.25.0 -> sha256:3a667d6f3c5ad9d0456e1575bed760976f0820beb6db476259d0eff08c2c2188  mismatch false
vllm-50801                  released     fixed_in 0.28.0  tag v0.28.0  2026-08-26T09:46:30Z  mismatch false
vllm-pr-51493               open         fixed_in ""
linux-npu-driver-56         open         fixed_in ""
kserve-4592                 not-shipped  fixed_in ""
eat-9                       open         fixed_in ""
litellm-1.83.14-advisories  advisory     fixed_in 1.100.4  (recorded 1.100.4, no new advisories)  https://pypi.org/project/litellm/1.100.4/
vllm-0.32-num-spec-tokens   untracked    fixed_in ""

hack/ledger-freshness --md (excerpt):

| 1 | H57dm | ledger-rules (drives optimizer defaults) | llama.cpp b11749 -> b11960 (minor) | H57r |
| 2 | H6    | cited by a docs claim | rancher/k3s v1.34.1+k3s1 -> v1.37.1-k3s1 (minor) | H6r |
| 3 | H30   | cited by a docs claim | llama.cpp b11706 -> b11960 (minor) | H30r |
| 4 | H31   | cited by a docs claim | llama.cpp b11706 -> b11960 (minor) | H31r |
| 5 | H32   | cited by a docs claim | llama.cpp b11706 -> b11960 (minor) | H32r |
| 6 | H1b   | other | priority-stale: vllm-openai-cpu 0.24.0 is inside vllm-45844 (<0.25.0, fixed in 0.25.0); ... | H1br |

| H34   | cryptoandcoffee/k3d-gpu 0.1.55; openvino/model_server 2026.4.1 | no |
| H57dm | cryptoandcoffee/k3d-gpu 0.1.63; ...; llama.cpp b11749; openvino/model_server 2026.4.1 | yes |

H34 is measured on OVMS 2026.4.1, the current pin, so it is not queued. H57dm is queued because its llama.cpp b11749 is behind the b11960 pin.

The protected files are byte-identical after --json and --md (sha256 before and after):

EXPERIMENTS.md: 436895781433b624316029350d0c2eeecb7b92e5e6550620830da3675b2d3964 (main at 74ff63e)
console/ledger-rules.json: a83e85937868c0dfa97a872241fff9f2eb68765df27d2924e92ec67db47842cc

Tests run

Suite	Result
tests/pinver_test.py	53 OK
tests/known_issues_test.py	41 OK
tests/ledger_freshness_test.py	14 OK
tests/claims_check_test.sh	83 passed (80 passed and 0 failed under Ubuntu's mawk; `--cited` output identical)
tests/claims_check.sh	clean
tests/ledger_drift_check.py / _test.py	pass
tests/ci_wiring_test.sh	72 passed
console_test / audit_test / model_opt_test	61 / 49 / 37 passed
detect_test / eat_test / host_setup_test / platforms_doc_test	1185 / 64 / 269 / 195 passed
quickstart_test	174 passed
kserve_test	29 passed (kubeconform step skipped locally)
docs_test (mkdocs --strict, hash-locked docs-build toolchain)	63 passed
scripts/images-lock check and pins; tests/image_pins_test.sh	ok; 43 passed
actionlint	1.7.12 (ACTIONLINT_IMAGE digest): clean
shellcheck	v0.11.0 -S warning on every changed shell file: clean

Second commit

The first CI run failed one job, "images.lock complete, every image pinned by digest": the re-measure test fixture's recorded node image cryptoandcoffee/k3d-gpu:0.1.70 is a name:tag reference, which images-lock pins flags. It is now allowed in images.unpinned as a test input that is never pulled, like the other test fixtures there.

The same commit:

replaces the model_opt.py begin/end pair with one marker (above);
makes the two real-repo tests independent of today's pins. The vLLM CPU pin is moved to v0.24.0 by pattern, and H34's recorded OVMS version is checked by shape. A freshness bump that substitutes version strings in tests/ therefore cannot turn them red.

CI on 56ce2e4 (rebased on main 74ff63e): 78 checks pass, including the image pin job and the new known-issues job; 2 are skipped (automerge and the Pages deploy).

Upstream states re-read with gh api after the rebase:

vllm compare v0.24.0...26eb8720 is diverged (behind 18, ahead 470), and v0.25.0...26eb8720 is behind (behind 106, ahead 0).
vllm#51493 is still open, and so are linux-npu-driver#56 and enterprise-agent-toolkit#9.
kserve#4592 merged 2026-01-16 as 400d2a81. The latest KServe release is v0.21.0 (2026-09-25), and v0.21.0...400d2a81 is behind, so the commit is in v0.21.0. But kserve-openvino appears in neither config/runtimes/kustomization.yaml nor the runtimes chart at v0.21.0 or master. So fixed_in stays empty.
The LiteLLM advisories for 1.83.14 are the same six; GHSA-7hp6 affects later releases (see the review fixes).

Notes for review

New LiteLLM advisories. The task brief named advisories fixed in 1.84.0 and 1.88.6. The advisory database now also lists:
- GHSA-g5ff-637f-6q2m (high, published 2026-10-27, fixed in 1.95.0);
- GHSA-hhww-mrg2-969h (fixed in 1.85.0).
- GHSA-7hp6-4w63-5g45 (high, published 2026-10-27, fixed in 1.100.4 with backports 1.101.3, 1.102.2 and 1.103.1). It affects 1.91.0 through 1.100.3, so it does not affect 1.83.14, but it does affect 1.95.1.
So the first release outside every range is 1.100.4 (on PyPI since 2026-09-29), and affected is a union. Both sites are EAT's stock gateway, so this reports and does not fail.
workarounds holds file paths. Each element is a file that carries the entry's markers. The check matches markers and entries in both directions on (file, id).
Classes before pins/classes lands. Site and marker classes come from pins/classes when it exists, read in its documented <glob> <class> line format with first match winning. Until then the defaults are used: EAT stock paths are eat-stock, bench/ is ledger, oracle/seed/ and console/matrix.v1.json are historical, and everything else is ours.
EXPERIMENTS.md defect, not edited. A ```bash fence opened in the H18 section (around line 7417) is never closed. It swallows the ### H20 pre-registration and ### H20 run headings until H20's provenance block closes it. ledger-freshness detects this and lists H18 under "not machine-readable" rather than reading H20's versions as H18's. H16 and H40 link no run section from their Probe cells. The ledger fix is a ledger PR.
Upstream report still to file. The vLLM v0.32.0 num_speculative_tokens ordering bug has no upstream issue. Nothing was posted outside this repo.

Review fixes (third commit)

Advisory poll. It read only the advisories of the lowest site version, so a patched version inside an advisory that does not affect that version could be chosen, as could a backport below the site version or a prerelease. It now takes the first stable patched version above the site version and reads that candidate's own advisories, moving on until a candidate has none. The live poll found GHSA-7hp6 this way: the recorded fix 1.95.1 was itself vulnerable. The LiteLLM entry now records 1.100.4, and a live poll reads it with no mismatch.
EAT tags. EAT's releases mix release-X.Y.Z and release-X.Y.Z-hotfix. Under docker-suffix those are different flavours, so eat-9's poll saw only hotfix releases and could never find a fix in release-1.1.0 or later. pinver gains an eat scheme (a hotfix sorts after its release), and eat-9 uses it.
Re-measure queue. Two queued rows of one family got the same suggested follow-up id; each suggestion is now reserved before the next is chosen.

Each fix has a test that failed first. Run on e7f4a86: pinver 57, known_issues 44 and ledger_freshness 15 OK (also under Python 3.12); hack/known-issues check ok; claims_check, ledger_drift_check (and its test), claims_check_test 83, ci_wiring_test 72, images-lock check and pins, image_pins_test 43; actionlint 1.7.12 clean.

…ure queue Every upstream bug, behaviour change and advisory k3d-gpu depends on is now an entry in pins/known-issues.toml with its affected range and the first release that fixes it. hack/known-issues check (offline, in CI as the known-issues job) fails when an "ours" pin stays inside a fixed issue's range ("bump <file> to >= <fixed_in>"), when a workaround marker outlives its fix without a retained reason, or when markers and entries disagree. eat-stock sites and ledger references in range are reported, never failed. hack/known-issues poll reads GitHub (read-only) and finds the first stable release containing each fix: the fix commit is compared against release tags walked in version order, a {repo, path, contains} fix reads the file at each tag, and advisories come from the GitHub advisory database. hack/pinver.py is the shared version library: semver, zerover, k8s, llama.cpp builds and releases, calver, PEP 440, docker suffix tags, apt and Hugging Face revisions, with ranges, update types and each scheme's breaking rule. hack/ledger-freshness lists SUPPORTED ledger rows whose recorded component versions are a minor or major behind today's pins, or inside a known issue's range, ordered by ledger-rules rows, then rows docs claims cite (new tests/claims_check.sh --cited), then the rest. It writes nothing. Seeded: vllm-45844 (fixed in 0.25.0), vllm-50801, vllm-pr-51493, linux-npu-driver-56, kserve-4592, eat-9, the LiteLLM 1.83.14 advisories and the vLLM 0.32 num_speculative_tokens ordering bug, with markers at the workaround sites in console/, scripts/k3d-gpu, docs and examples/kserve.

… real-repo tests The ledger re-measure test fixture names cryptoandcoffee/k3d-gpu:0.1.70 as a recorded node image, which the image pin gate (images-lock pins) flagged; it is now allowed in images.unpinned as a test input that is never pulled. The vllm-pr-51493 workaround in console/model_opt.py is one clause of a note string inside an if block, so it carries a single known-issue marker instead of a begin/end pair whose deletion would leave the block empty. The real-repo tests no longer name today's pins: the vLLM CPU pin is moved to v0.24.0 by pattern, and H34's recorded OVMS version is checked by shape, so a version bump that substitutes strings in tests/ does not turn them red.

… EAT tag scheme The advisory poll read only the advisories that affect the lowest site version. A patched version could itself sit inside an advisory that does not affect that version, and a backport below the site version or a prerelease could be chosen as the fix. It now takes the first stable patched version above the site version, reads that candidate's own advisories, and moves on until a candidate has none. That changes the LiteLLM entry. GHSA-7hp6-4w63-5g45 (high, published 2026-10-27) affects 1.91.0 through 1.100.3, so the recorded fix 1.95.1 was itself vulnerable. The first release outside every advisory is 1.100.4; the entry now lists GHSA-7hp6 and the wider affected range. A live poll reads fixed_in 1.100.4 with no mismatch and no new advisories. EAT's releases are one stream of release-X.Y.Z and release-X.Y.Z-hotfix tags. Under docker-suffix a hotfix tag and a plain release are different flavours, so the eat-9 poll saw only hotfix releases and could never find a fix shipped in release-1.1.0 or later, and latest() could not move off a hotfix. pinver gains an eat scheme that orders the two forms in one stream, and eat-9 uses it. The re-measure queue gave two rows of one family the same suggested follow-up id; each suggestion is now taken before the next is chosen.

main added the known-issues job at the end of ci.yml (#256) where this branch adds canary-tests, so the two conflicted and gate_merge.sh's sync stops on a ci.yml conflict. Both jobs are kept as they are: known-issues as main has it, then canary-tests at the end of the file.

88plug mentioned this pull request Oct 29, 2026

eas: k3s/k3d patch overlay that drops each patch once upstream carries the fix #248

Merged

88plug added 2 commits October 29, 2026 17:02

88plug force-pushed the claude/fresh-known-issues branch from c3d495c to 56ce2e4 Compare October 30, 2026 00:10

88plug mentioned this pull request Oct 30, 2026

batch: land #264, #256 #272

Merged

88plug merged commit 7c40003 into main Oct 30, 2026
80 checks passed

88plug added a commit that referenced this pull request Oct 30, 2026

batch: land #264, #256 (#272)

4f1cb68

88plug mentioned this pull request Oct 30, 2026

Pin classes, pin-guard, scoped images-lock, tagged refs and tool pins out of workflows #265

Open

Navigation Menu

Repository navigation

known-issues: fixed-in registry, version schemes and a ledger re-measure queue - #256

known-issues: fixed-in registry, version schemes and a ledger re-measure queue#256
88plug merged 3 commits into
mainfrom
claude/fresh-known-issues

88plug commented Oct 29, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

88plug commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is added

Seeded entries (each upstream state re-read with gh api on 2026-10-29)

Acceptance

Tests run

Second commit

Notes for review

Review fixes (third commit)

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

88plug commented Oct 29, 2026 •

edited

Loading