Skip to content

test(containerlab): Prove the Local kernel handoffs in the lab - #1456

Merged
privateip merged 6 commits into
integration/galactic-workerfrom
feat/issue-1444
Nov 19, 2026
Merged

privateip merged 6 commits into
integration/galactic-workerfrom
feat/issue-1444

Conversation

@privateip

Copy link
Copy Markdown
Collaborator

Summary

The Worker's Local module needs proof that each kernel handoff it will program delivers tenant traffic, enforces policy and stays off the physical wire on the agreed platform.

This adds a TAP-backed guest fixture, prototype programs placed by the existing loaders, and scripted lab checks that prove all three handoffs in native and generic XDP on plain, bond and VLAN devices, both directions, with captures.

Where a specified mechanism failed, the Worker contract now records the failure and the alternative proven here, and the lab pins Cilium's devices as production does, so Cilium no longer strips galactic's filters from tenant TAPs.

Test plan

  • Two lab runs of every cell pass all 343 checks, agree, and match the committed expected results
  • After the Cilium change, the full lab verification passes and the dedicated-placement baseline matches its reference
  • Measurements record MTU, fragment, GSO and checksum behavior for each handoff
  • Lint, build, unit, root and end-to-end checks pass

Fixes #1444

With kube-proxy replacement and no device list, Cilium adopts every
interface with a global address. A galactic-tap host TAP carries one, the
guest's gateway, so Cilium registered the tenant address as a node
address and attached its programs to the TAP with tcx, which deleted
every legacy tc filter there: galactic-cni's egress chain and
tp_deliver. The TAP then forwarded without its traffic policy.

Pin the devices as production deployments pin them to their uplinks. In
a Kind node that is eth0 and the bonds; eth+ would also match the bond
members, whose legacy usid_ingress filters Cilium would delete the same
way.
tests/handoffs/probe holds prototype programs for the three Local
handoffs the Worker qualification plan lists: public receive XDP_PASS
into tc and a tenant TAP, tc into an internal veth pair with XDP on its
peer and the reverse tc receive, and a guard for a pod-to-host routed
link. handoff.c includes the uSID datapath source, so its policy
decisions run the product's tp_capture and tp_decide; the loader swaps
every uSID map for galactic-cni's pin and refuses to create one.

handoff-probe places the programs on a scratch rig with the existing
loaders (xdpdispatch, natattach, attach, egressroutemap) and removes
them. build:ebpf generates the probe, and the unit and root test runs
cover it; a root test loads every program through the verifier against
freshly pinned uSID maps.
ns80-tap puts a TAP-backed guest in each of ns80's VPCs on dfw-worker,
with the same address, and an Envoy stand-in. Each TAP guest takes the
cluster's network as its primary and galactic-tap as a secondary
attachment, so its sandbox starts; a vmm container bridges the host TAP
to the guest's interface. publish-attachments.sh now also publishes an
attachment a pod names as a secondary network.

task handoff:twice deploys the fixture, proves each handoff in every
cell (veth, bond and VLAN devices, native and generic XDP) twice with
captures, measures MTU and offload behavior, removes the fixture, and
compares with handoffs/expected-results.json. Each check covers both
directions, a policy-denied control, isolation between the two VPCs and
the absence of the probes on the node's physical members.
The Local handoffs were proven in the lab, and several mechanisms the
qualification plan named do not work as specified. The contract now
states what the Local module programs and what each handoff requires:
native XDP on the veth peer whatever the host's mode, since generic XDP
skips tc-redirected packets; a VLAN egress through a pushed tag and the
lower device; policy decided on each handoff's own path and marked;
Local pass-through entries; tc order after galactic-cni's filters; no
other tc owner on a tenant TAP; and the MTU, fragment and checksum
limits. The qualification plan lists the supported combinations and
marks the unsupported ones HookUnsupported.
@privateip privateip self-assigned this Nov 19, 2026
@privateip

Copy link
Copy Markdown
Collaborator Author

Evidence for #1444, from the shared lab on 2026-11-19, with the lab host on Linux 7.3 (the platform minimum is 6.8). The scripts and probe in this PR produced every run below. The lab's datapath matched this checkout's uSID maps, which the probe checks before it loads anything.

The tenant TAP fixture works in Kind. Each TAP guest takes the cluster's network as its primary and galactic-tap as a secondary attachment, so its sandbox starts, and a vmm container bridges the host TAP to the guest's tap0 with socat. Both host TAPs had carrier and galactic-cni's egress chain. This is the first time a TAP-backed guest has run in the lab.

Results:

  • Every cell passes all 343 checks in two runs whose results.json are identical, and run 1 matches handoffs/expected-results.json. Each handoff is checked both ways, with a policy-denied control, isolation between VPC 80 and VPC 81 at the same guest address, and zero probe packets, plain or encapsulated, on dfw-worker's four physical bond members.
  • After the Cilium pin, the full task verify and verify:ns80 passed, and baseline:twice matched the Build a reproducible baseline for dedicated placement #1432 reference.
  • Measurements are in the lab doc: a 1500-byte packet reaches a TAP guest and an IPv6 fragment does not; 1 MiB crosses each handoff intact with the sender's checksum offload off; with it on, the pair path corrupts data, and so does a veth pod's own offload (0 of 3 NAT flows answered, 3 of 3 with it off).

What failed as specified, and the alternative the contract adopts (D27):

  • Generic XDP on the veth pair's NAT end never sees a tc-redirected packet (skb_is_redirected in the generic receive path), so on a bnxt_en host the pair fails. The NAT end runs native XDP whatever the host's mode.
  • Native XDP does not attach to a VLAN device. The dispatcher goes on the lower device with tc on the VLAN, and the metadata survives the kernel's untag.
  • A native XDP_REDIRECT from the peer to a VLAN device has no transmit. The peer pushes the tag and redirects to the lower device. XDP_PASS is no alternative: the return comes back through XDP past netfilter, conntrack marks the guest's ACK invalid, and TCP stops after the handshake.
  • tp_deliver decides every unmarked packet on the Private path, which denies a public client. Each handoff decides on its own path, PublicService or PublicNAT, with the datapath's tp_decide, and sets the decided mark.
  • usid_egress sends a tenant's packet for a Local peer to the VPC's NAT default, encapsulated, unless a pass-through entry out-matches it. Local filters run after galactic-cni's priorities 1 and 2.
  • Native XDP does not attach to a veth at MTU 9000, and a redirect does not enforce the TAP's MTU.

Cilium in the lab auto-detected its devices, adopted each host TAP (it carries the guest's gateway address), attached with tcx, and deleted every legacy tc filter on it, the egress chain and tp_deliver included. With the chain gone, a public client reached a guest whose policy denies it. Deployed nodes pin Cilium's devices to their uplinks, so this was the lab diverging from production. The lab now pins eth0 and the bonds, not eth+, which would match the bond members and strip their usid_ingress filters the same way. The contract states the rule: no other tc owner may claim a tenant TAP.

The bnxt_en hardware itself was not available; generic mode was forced on the lab's devices, and #1442 carries the run on the NIC.

Trim the comment on the pinned Cilium devices to the reason for the pin
and point at the lab doc for the evidence, wrapped evenly.
scripts/required-ci.sh generates each eBPF package by name before go
build ./..., so the new tests/handoffs/probe package failed to build
there without its bpf2go output. Generate it as build:ebpf does, and run
its unit tests with the rest.
@privateip
privateip marked this pull request as ready for review November 19, 2026 09:17
@privateip
privateip requested a review from a team as a code owner November 19, 2026 09:17
@privateip
privateip merged commit ca34660 into integration/galactic-worker Nov 19, 2026
28 checks passed
@privateip
privateip deleted the feat/issue-1444 branch November 19, 2026 09:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant