Skip to content

eas: k3s/k3d patch overlay that drops each patch once upstream carries the fix - #248

Merged
88plug merged 2 commits into
mainfrom
claude/eas-eas-overlay
Oct 30, 2026
Merged

88plug merged 2 commits into
mainfrom
claude/eas-eas-overlay

Conversation

@88plug

@88plug 88plug commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

This is the overlay part of full EAS support. k3d-gpu eas up will use it to patch the intel/enterprise-ai-solutions (EAS) checkout before it installs. Each k3s/k3d patch drops out once upstream carries the fix, and if upstream moves a hunk the run stops before the cluster is touched.

What it does

integrations/eas/overlay.py reads integrations/eas/overlay.json and decides for each patch at one EAS commit:

  • skip: the patch is not needed on this cluster.
    • 0003 is needed only when kubernetes_cpu_policy is nri-balloons, the only case where EAS installs the balloons chart (configs/components.yaml).
    • 0004 is needed only on more than one node.
    • 0005 is docs, so it never applies.
  • drop-merged: a variant reverse-applies (git apply -R --check).
  • drop-fixed: a probe finds an upstream fix done another way, and the knob the generator writes is still in the file the probe matched.
    • If the probe matches but the knob is gone, the row STOPs. That holds even when our patch would still apply, because two wirings of global.platform must not exist side by side.
  • apply: the first variant whose git apply --check passes.
  • STOP: the line names the upstream change that moved the hunk and says to regenerate the patch. The name comes from a row's moved hint when one matches, otherwise from the last commit on the patch's files. The exit code is 3, and nothing is written.

Every decision prints as fact -> choice [source]. With --explain, the source becomes a permalink at the full sha.

The source line is where the hunk lands at that commit, located the way git apply locates it. It is not the patch header's line: on 97f6157, 0002's header says 59, but the hunk lands at 69.

Decisions read the commit through a scratch index, never the working tree. --apply writes the working tree only when no row STOPs and --rev is HEAD:

  • A patch the tree already carries is kept, so a re-run is a no-op.
  • A re-run whose new facts add a patch applies only that patch.
  • A patch's files in any other state cause a STOP before anything is written.

Patches are looked up in integrations/eas/patches first, then in the upstream drafts in integrations/eat/upstream/patches. The drafts stay the only copy.

There is one new patch, integrations/eas/patches/0006-release-1.2-envoy-control-plane-affinity.patch. It is the release-1.2 form of 0006: on that branch the EnvoyProxy task is still in roles/envoy_gateway/tasks/install.yaml, at L191-L219, and EAS #42 moved it to controller.yaml on main. It makes the same change as 0006: affinity Exists, plus nodeSelector: null.

I checked the variant with checks/eas-envoy-reinstall.py on release-1.2 5c27fb3, before and after the patch. A fresh install and a re-install over the unpatched object both give nodeSelector=None with the Exists affinity. release-1.2 pins envoy_gateway_version: "1.7.1".

Rows (overlay.json, checked 2026-10-29)

id needed_when fix probe knob moved hint
0002 always roles/istio/** sets global.platform from a variable istio_platform EAS #61 (headlamp_enabled in the global_config.yaml context)
0003 eas_owns_nri chart_version: in roles/nri_cpu_balloons/tasks/install_advanced.yaml none none
0004 multi_node roles/kubernetes/tasks/install.yaml runs apply_node_topology.yaml none none
0005 never none none none
0006 always a matchExpressions Exists term on the control-plane key, with no exact "" selector (any quoting) left in any file of roles/envoy_gateway; or roles/envoy_gateway/vars/k3s.yaml (the per-distribution layout of EAS #50) selecting a shape other than controlplane_daemonset none EAS #50 (_eg_k8s_provider; #52 carries the same change)

overlay.json also records verified cases: the expected decisions at the two profile pins, for two sets of facts each.

  • main 97f6157d6a72f7173c698c474d4e51d2c79f1fd6.
  • release-1.2 5c27fb39a95eecec41aa4e29a1188984e9c726f3, which is tag v1.2.0-rc.1.

CI checks these cases on byte-identical fixtures. apply-check.sh --overlay checks them on a real clone.

Once integrations/eas/pins.json lands (eas-config), eas_test.sh requires every pinned EAS sha to have a verified entry. That check reads any object with repo ending in enterprise-ai-solutions and a sha, so it does not depend on the pins layout.

Interface for eas-config and cmd_eas:

overlay.py --eas DIR --policy {nri-balloons,kubelet-static,best-effort} --nodes N [--apply] [--json FILE|-] [--explain]
  • Exit codes: 0 when nothing STOPs, 3 on a STOP, 2 on bad input.
  • The JSON decisions carry knob, the generator variable an applied or drop-fixed row needs (istio_platform for 0002).

Tests (written first)

tests/eas_test.sh has section overlay, wired into ci.yml. The suite takes section names, so config and cli can be added beside it. It runs 95 checks on synthetic git repos built from tests/fixtures/eas/trees:

  • The fixtures are the 12 files the patches touch, at 97f6157 and at 5c27fb3, plus EAS's Apache-2.0 LICENSE.
  • They are verbatim copies. SOURCES lists each file's upstream blob sha, and the suite checks every one with git hash-object.

Cases:

  • Base main: 0002 and 0006a apply. 0003 skips for best-effort and kubelet-static and applies for nri-balloons. 0004 skips on one node and applies on two. 0005 skips.
  • Patches pre-applied: all four rows give drop-merged.
  • Fixed another way, knob present: drop-fixed for each of these:
  • Affinity added next to the exact "" selector: STOP. This is not a fix on k3s.
  • Knob renamed (istio_cni_platform): STOP. This also holds when 0002 would still apply.
  • Hunk moved the way feat: opt-in telemetry manifests (RAPL, P/E-core, Intel GPU, DCGM) #50/Helm-adoptable NRI balloons, EAT hybrid profile and reservation patch #52 move it: STOP naming https://github.com/intel/enterprise-ai-solutions/pull/50, and --apply leaves the tree untouched, 0002 included.
  • The feat: k3d-gpu audit and images.lock (console slice S5) #61 context: STOP naming feat: k3d-gpu audit and images.lock (console slice S5) #61.
  • An unknown move: STOP naming the last commit and its subject.
  • release-1.2: 0006 variant b applies. Variant a does not apply there, and neither does 0005.
  • --apply: it changes exactly 0002 and 0006a's files, and a re-run is a no-op. A re-run with nri-balloons adds only 0003. A conflicting local edit gives exit 3 and leaves every other file alone. --rev other than HEAD is refused.
  • Bad input: a bad policy, nodes 0, a directory that is not git, and a missing patch file each exit 2.
  • --check-verified and --watch: both run against a spec pointed at the fixture repo.

I mutation-checked three cases, and each mutation turned the suite red:

Results on this branch, rebased on main 890d1d16:

Check Result
tests/eas_test.sh 95 passed, 0 failed
tests/eat_test.sh 64 passed
tests/ci_wiring_test.sh 70 passed, including tests/eas_test.sh runs in CI
tests/image_pins_test.sh 43 passed; the fixture YAML adds no unpinned image refs
tests/claims_check.sh 26 files clean
tests/ledger_drift_check.py 41 files agree with 81 ledger rows
shellcheck v0.11.0 clean (-S warning, the CI severity; also clean at all severities)
actionlint 1.7.12 the pinned image from hack/ci-images.env passes
ruff E, F, W, B on overlay.py: clean
apply-check.sh (default mode) still passes on main 97f6157: all five patches, and the series with git am

Review fixes (second commit, 41694c1)

A review found three gaps in the auto-drop logic and one in the tests. Each has a case in tests/eas_test.sh that failed before the fix:

  • 0006 Exists probe. It looked for the exact "" selector only in the file it matched, and only with an unquoted key. So an Exists term in another file of roles/envoy_gateway, or a flow-style selector with a quoted key, gave drop-fixed while the EnvoyProxy kept the selector that matches no k3s node. A probe can now carry an absent regex that no file under its glob may match; 0006 uses it for the selector in any quoting, across the role.
  • vars/k3s.yaml probe. It matched any _envoy_gateway_proxy_shape, including controlplane_daemonset, the shape in feat: opt-in telemetry manifests (RAPL, P/E-core, Intel GPU, DCGM) #50 that carries the exact "" selector. It now matches only another shape. A k3s entry that keeps controlplane_daemonset is still the STOP naming feat: opt-in telemetry manifests (RAPL, P/E-core, Intel GPU, DCGM) #50.
  • STOP attribution. A moved hint was named whenever its regex matched anywhere. Once feat: k3d-gpu audit and images.lock (console slice S5) #61 merges, every later 0002 STOP would have named feat: k3d-gpu audit and images.lock (console slice S5) #61. A hint now counts only when it watches a file whose hunks no variant applies (git apply --check --include), and the fallback names the last commit on those files.
  • Stale JSON in the tests. The helper that reads the decisions JSON did not remove it first. A run that wrote none left the previous decisions in place, and two checks passed with --rev or kubelet-static broken outright. It now removes the file before each run.

Mutation check: ignoring absent, unscoping the hint, the old fallback, a broken --rev and a dropped kubelet-static each turn the suite red.

Decisions are unchanged at both pins and at every open EAS PR head (#12 to #75), on a fresh clone.

apply-check.sh --overlay, on a fresh clone, 2026-10-29

integrations/eat/upstream/checks/apply-check.sh --overlay exited 0. It needs the network, so it stays out of CI. Output:

## verified pins (integrations/eas/overlay.json)
ok   main 97f6157 best-effort nodes=1: 0002=apply:a 0003=skip 0004=skip 0005=skip 0006=apply:a
     0002 variant a applies -> apply 0002-eas-istio-global-platform.patch [eas@97f6157 configs/defaults/global_config.yaml:69]
     0003 kubernetes_cpu_policy=best-effort, needed only for nri-balloons -> skip [overlay.json 0003 needed_when=eas_owns_nri]
     0004 nodes=1, needed only on more than one node -> skip [overlay.json 0004 needed_when=multi_node]
     0005 documentation only -> skip [overlay.json 0005 needed_when=never]
     0006 variant a applies -> apply 0006-eas-envoy-control-plane-affinity.patch [eas@97f6157 roles/envoy_gateway/tasks/controller.yaml:154]
ok   main 97f6157 nri-balloons nodes=3: 0002=apply:a 0003=apply:a 0004=apply:a 0005=skip 0006=apply:a
     0002 variant a applies -> apply 0002-eas-istio-global-platform.patch [eas@97f6157 configs/defaults/global_config.yaml:69]
     0003 variant a applies -> apply 0003-eas-pin-nri-balloons-chart.patch [eas@97f6157 roles/nri_cpu_balloons/defaults/main.yaml:27]
     0004 variant a applies -> apply 0004-eas-byo-node-topology-labels.patch [eas@97f6157 docs/customize/node_topology.md:37]
     0005 documentation only -> skip [overlay.json 0005 needed_when=never]
     0006 variant a applies -> apply 0006-eas-envoy-control-plane-affinity.patch [eas@97f6157 roles/envoy_gateway/tasks/controller.yaml:154]
ok   v1.2.0-rc.1 5c27fb3 best-effort nodes=1: 0002=apply:a 0003=skip 0004=skip 0005=skip 0006=apply:b
     0002 variant a applies -> apply 0002-eas-istio-global-platform.patch [eas@5c27fb3 configs/defaults/global_config.yaml:62]
     0003 kubernetes_cpu_policy=best-effort, needed only for nri-balloons -> skip [overlay.json 0003 needed_when=eas_owns_nri]
     0004 nodes=1, needed only on more than one node -> skip [overlay.json 0004 needed_when=multi_node]
     0005 documentation only -> skip [overlay.json 0005 needed_when=never]
     0006 variant a does not apply, variant b does -> apply 0006-release-1.2-envoy-control-plane-affinity.patch [eas@5c27fb3 roles/envoy_gateway/tasks/install.yaml:194]
ok   v1.2.0-rc.1 5c27fb3 nri-balloons nodes=3: 0002=apply:a 0003=apply:a 0004=apply:a 0005=skip 0006=apply:b
     0002 variant a applies -> apply 0002-eas-istio-global-platform.patch [eas@5c27fb3 configs/defaults/global_config.yaml:62]
     0003 variant a applies -> apply 0003-eas-pin-nri-balloons-chart.patch [eas@5c27fb3 roles/nri_cpu_balloons/defaults/main.yaml:27]
     0004 variant a applies -> apply 0004-eas-byo-node-topology-labels.patch [eas@5c27fb3 docs/customize/node_topology.md:37]
     0005 documentation only -> skip [overlay.json 0005 needed_when=never]
     0006 variant a does not apply, variant b does -> apply 0006-release-1.2-envoy-control-plane-affinity.patch [eas@5c27fb3 roles/envoy_gateway/tasks/install.yaml:194]

## watch refs, k3d-gpu facts
# origin/main = eas@97f6157 (EAS main tip), kubernetes_cpu_policy=best-effort, nodes=1
0002 variant a applies -> apply 0002-eas-istio-global-platform.patch [eas@97f6157 configs/defaults/global_config.yaml:69]
0003 kubernetes_cpu_policy=best-effort, needed only for nri-balloons -> skip [overlay.json 0003 needed_when=eas_owns_nri]
0004 nodes=1, needed only on more than one node -> skip [overlay.json 0004 needed_when=multi_node]
0005 documentation only -> skip [overlay.json 0005 needed_when=never]
0006 variant a applies -> apply 0006-eas-envoy-control-plane-affinity.patch [eas@97f6157 roles/envoy_gateway/tasks/controller.yaml:154]
# origin/release-1.2 = eas@5c27fb3 (EAS release-1.2 tip), kubernetes_cpu_policy=best-effort, nodes=1
0002 variant a applies -> apply 0002-eas-istio-global-platform.patch [eas@5c27fb3 configs/defaults/global_config.yaml:62]
0003 kubernetes_cpu_policy=best-effort, needed only for nri-balloons -> skip [overlay.json 0003 needed_when=eas_owns_nri]
0004 nodes=1, needed only on more than one node -> skip [overlay.json 0004 needed_when=multi_node]
0005 documentation only -> skip [overlay.json 0005 needed_when=never]
0006 variant a does not apply, variant b does -> apply 0006-release-1.2-envoy-control-plane-affinity.patch [eas@5c27fb3 roles/envoy_gateway/tasks/install.yaml:194]
# pull/50/head = eas@b7eb331 (open: OpenShift platform layer, moves the EnvoyProxy provider into _eg_k8s_provider), kubernetes_cpu_policy=best-effort, nodes=1
0002 variant a applies -> apply 0002-eas-istio-global-platform.patch [eas@b7eb331 configs/defaults/global_config.yaml:69]
0003 kubernetes_cpu_policy=best-effort, needed only for nri-balloons -> skip [overlay.json 0003 needed_when=eas_owns_nri]
0004 nodes=1, needed only on more than one node -> skip [overlay.json 0004 needed_when=multi_node]
0005 documentation only -> skip [overlay.json 0005 needed_when=never]
0006 no variant applies or reverses and no fix probe matches; the hunk moved in https://github.com/intel/enterprise-ai-solutions/pull/50 (the EnvoyProxy provider moved into the _eg_k8s_provider var, still an exact "" selector (#52 carries the same change)) -> STOP: regenerate 0006 [https://github.com/intel/enterprise-ai-solutions/pull/50]
# pull/52/head = eas@cbaa61a (open: OpenShift ownership labels, stacked on #50), kubernetes_cpu_policy=best-effort, nodes=1
0002 variant a applies -> apply 0002-eas-istio-global-platform.patch [eas@cbaa61a configs/defaults/global_config.yaml:69]
0003 kubernetes_cpu_policy=best-effort, needed only for nri-balloons -> skip [overlay.json 0003 needed_when=eas_owns_nri]
0004 nodes=1, needed only on more than one node -> skip [overlay.json 0004 needed_when=multi_node]
0005 documentation only -> skip [overlay.json 0005 needed_when=never]
0006 no variant applies or reverses and no fix probe matches; the hunk moved in https://github.com/intel/enterprise-ai-solutions/pull/50 (the EnvoyProxy provider moved into the _eg_k8s_provider var, still an exact "" selector (#52 carries the same change)) -> STOP: regenerate 0006 [https://github.com/intel/enterprise-ai-solutions/pull/50]
# pull/61/head = eas@d61b426 (open: Headlamp SSO, changes 0002's global_config.yaml context and 0004/0005), kubernetes_cpu_policy=best-effort, nodes=1
0002 no variant applies or reverses and no fix probe matches; the hunk moved in https://github.com/intel/enterprise-ai-solutions/pull/61 (headlamp_enabled now follows velero_enabled, the context of 0002's global_config.yaml hunk) -> STOP: regenerate 0002 [https://github.com/intel/enterprise-ai-solutions/pull/61]
0003 kubernetes_cpu_policy=best-effort, needed only for nri-balloons -> skip [overlay.json 0003 needed_when=eas_owns_nri]
0004 nodes=1, needed only on more than one node -> skip [overlay.json 0004 needed_when=multi_node]
0005 documentation only -> skip [overlay.json 0005 needed_when=never]
0006 variant a applies -> apply 0006-eas-envoy-control-plane-affinity.patch [eas@d61b426 roles/envoy_gateway/tasks/controller.yaml:154]

The decisions at the pins match the spec: on main, 0002 apply, 0006 apply (variant a), and 0003, 0004 and 0005 skip; release-1.2 is the same with 0006 variant b.

The open PRs show the known future breaks. #50 and #52 STOP 0006 and name #50. #61 STOPs 0002 and names #61.

When #61 merges, a regenerated 0002 is needed. Only the context of 0002's global_config.yaml hunk moves, and that hunk is a comment. eas up on that main would stop with exit 3 before touching the cluster.

Not covered here

Sources:

Nothing was posted to any intel repository.

…s the fix

integrations/eas/overlay.py decides, per row of integrations/eas/overlay.json
and at one EAS commit, whether a k3s/k3d patch is skipped (not needed on this
cluster), dropped because upstream merged it (git apply -R --check), dropped
because upstream fixed it another way (a probe matches and the knob the
generator writes is still there), applied (first variant whose git apply
--check passes), or a STOP that names the upstream change that moved the hunk
and says to regenerate. Every decision prints as `fact -> choice [source]`,
with --explain giving permalinks. Decisions read the commit through a scratch
index, so they do not change once the overlay is applied; --apply writes the
working tree only when nothing STOPs, and a re-run is a no-op.

Patches are looked up in integrations/eas/patches, then in the upstream
drafts, so the drafts stay the only copy. The one new patch is the
release-1.2 form of 0006, where the EnvoyProxy task is still in
roles/envoy_gateway/tasks/install.yaml.

tests/eas_test.sh (section overlay) runs it on synthetic git repos built from
byte-identical copies of the EAS files the patches touch, at main 97f6157
and release-1.2 5c27fb3. apply-check.sh gains --overlay, which checks the
recorded decisions on a real clone and prints them for branch tips and open
EAS PRs #50, #52 and #61.
… STOP names the change that broke the hunk

Three gaps in the auto-drop logic, each found with a synthetic tree on the
fixtures and each now a failing-then-passing case in tests/eas_test.sh:

- The 0006 Exists probe checked for the exact "" control-plane selector only
  in the file it matched, and only with an unquoted key. An Exists term in
  another file of roles/envoy_gateway, or a flow-style selector with a quoted
  key, gave drop-fixed while the EnvoyProxy kept the selector that matches no
  k3s node. A probe can now carry an `absent` regex that no file under its
  glob may match; 0006 uses it for the selector in any quoting, role-wide.
- The vars/k3s.yaml probe for the EAS #50 layout matched any
  _envoy_gateway_proxy_shape, including controlplane_daemonset, the shape that
  carries the exact "" selector. It now matches only another shape; a k3s
  entry that keeps controlplane_daemonset is still the STOP naming #50.
- A STOP named a `moved` hint whenever its regex matched anywhere, and
  otherwise the last commit on any of the patch's files. Once EAS #61 merges,
  every later 0002 STOP would have named #61. A hint now counts only when it
  watches a file whose hunks no variant applies (git apply --check
  --include), and the fallback names the last commit on those files.

The test helper that reads the decisions JSON now removes it before each
run. Before, a run that wrote no JSON left the previous run's decisions in
place, and two checks passed with --rev or kubelet-static broken outright.

Decisions are unchanged at both pins (main 97f6157, release-1.2 5c27fb3) and
at every open EAS pull request head, checked on a fresh clone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant