Skip to content

feat(worker): Deliver tenant traffic between handoffs on one node - #1457

Draft
privateip wants to merge 5 commits into
integration/galactic-workerfrom
feat/issue-1435
Draft

privateip wants to merge 5 commits into
integration/galactic-workerfrom
feat/issue-1435

Conversation

@privateip

@privateip privateip commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Worker sorts each destination on this node into Local delivery, but nothing programmed Local delivery, so every Local capability reported that no module was running. This change adds Worker's Local delivery module, which programs the three same-host handoffs the contract specifies and #1444 proved in the lab.

What changes:

  • Public traffic to a VM guest. A gateway's XDP program marks the packet for a guest's TAP. Local delivers it after the guest's traffic policy allows it, and sends the guest's reply back out the device the request came in on.
  • NAT for guests on the same node. Local creates one internal veth pair per VPC and NAT claim. It sends guest flows into the pair, where NAT's program runs, and delivers the replies back to the right guest. Guests in two VPCs may share an address.
  • Envoy sockets. Local creates a link from the VPC to the Envoy pod, only into the pod that made the request. A guard drops any source address the pod was not given. Each pod needs its own claim: while the pod a link serves runs, Worker refuses the claim to any other pod of its owner.

Each handoff decides the guest's policy on its own path and drops anything no claim authorizes. Readiness reports the forward and return paths separately, with a reason for each, and a missing hook withdraws only the direction it carries. Uplink discovery now skips every link Galactic creates for a VPC, so a NAT pair's end can never become an uplink.

Local's programs run after galactic-cni's on shared interfaces and never replace them. Local deletes only the links, programs and routing entries it created. The contract and ownership docs record this. The issue's comments explain two departures from the plan: routed-link addressing, and who steers public traffic into a handoff.

Privileged tests cover every acceptance item on a real kernel, as the comment below lists. The lab run is pending. It checks Local claims on one compute node in two VPCs that share guest addresses. One claim with both a Local and a Fabric destination cannot be shown in the lab: each lab site is its own cluster, so a node's Worker cannot see another site's attachments.

Fixes #1435

Worker resolved destinations on this node to Local but had no module to
program them, so every local.* capability reported ModuleNotRunning. Add
the Local module (internal/worker/local) and register it next to Fabric.

- Programs: local_rx delivers a public packet a dispatcher slot marks
  with galactic_local_meta to the TAP it names; local_link_rx returns a
  local.tap reply out of the device the flow arrived on, or sends a flow
  into the VPC's service-internal veth pair; local_pair_rx delivers the
  pair's return; local_link_guard drops a routed link source the link
  was not given. Each decides the receiver's policy on its own path and
  sets the decided mark; an unauthorized target is dropped, as are IPv6
  fragments and packets above the target's MTU.
- Lifecycle: tc filters sit after galactic-cni's under its interface
  lock. The module creates and deletes the pairs and routed links it
  owns, keeps CNI's interfaces, and reprograms after an ifindex change.
  A routed link's pod end goes only into the owner's authenticated pod.
- Readiness checks the hooks, a live public dispatcher slot, native XDP
  on the NAT end, interface kind, carrier, MTU and failed neighbors,
  forward and return separately.
- Uplink discovery skips every link Galactic creates per VPC.
- API: ClaimStatus gains handoff_interfaces; claims record the owner's
  pod, which a duplicate Request from a new pod updates.

Privileged tests cover every entry path (plain, VLAN and bond, native
and generic XDP), policy denials, untrusted entry, hook removal and
repair, a cold neighbor, ifindex replacement, partial checksums, routed
links and withdrawal.
Record how the Local module implements the handoffs the contract
specifies: its programs and hooks, link and filter ownership, readiness
checks and reasons, trust of public ingress, the handoff_interfaces
status field, and which writer steers public flows into a handoff.

The ownership inventory names the Local module as the sole mutator of
its tc filters, its handoff links and its maps. D28 records the accepted
decisions; D29 proposes routed-link addressing by the claim attachment's
addresses and resolver-written steering entries.
worker-claim requests, observes and releases one claim from inside a lab
pod, as a service agent would, so the lab can check Local and Fabric
programming end to end. worker-fabric.sh now takes the allowlist from
WORKER_ALLOWLIST, so a run can grant a lab pod its capabilities.
- Bring a routed link's host end up only after its guard row and filter
  are in place, and take it down whenever the guard is missing, so a pod
  never reaches the VPC unguarded.
- Mark every new Local link right after creating it and remove the link
  when a later step fails; treat an unmarked link with a Local name as
  Local's to replace, so an interrupted creation does not block a claim.
- Record a routed link's failure on its own claim and keep programming
  the rest of the VPC.
- Keep local_rx on the receiving devices while any TAP is authorized, so
  a restarted Worker planning a scope without local.tap first does not
  detach it.
- Admit a link-local source on a routed link only toward a link-local or
  multicast destination, and the unspecified source only for ICMPv6.
- Drop IPv6 fragments behind extension headers, and check the egress MTU
  on the reply and NAT redirects.
- Report claims whose link names collide instead of overwriting one.
- Pick a pod's lowest PID when opening its network namespace.
@privateip privateip self-assigned this Nov 19, 2026
@privateip

Copy link
Copy Markdown
Collaborator Author

Test evidence at 0de62b6

The privileged tests are in internal/worker/local. They run on a real kernel, with network namespaces, TAP guests at one address in two VPCs, and galactic-cni's own datapath and traffic policy. Stand-ins for the gateway's and NAT's XDP programs (localprog/localfixture) stand in for those services.

Acceptance item Test Result
Frames on every entry path reach the guest, or policy denies them, at both TAPs TestEntryPathsRoot: plain, VLAN and bond receive in native and generic XDP; the NAT flow and its return through the pair; denials on the PublicService and PublicNAT paths pass, 6 cells
Removing a hook withdraws readiness TestHookWithdrawalAndRepairRoot: Local's filters, the dispatcher slot, NAT's XDP (gone, then generic), galactic-cni's chain, MTU. TestRestartKeepsReceiveHookRoot pass
A cold neighbor resolves and delivers TestColdNeighborRoot pass
Delivery recovers after ifindex replacement TestIfindexReplacementRoot pass
An unclaimed or forged packet enters no handoff TestUntrustedEntryRoot: wrong metadata magic, an unclaimed TAP, an unclaimed NAT return, a packet straight to a guest. TestRoutedLinkRoot and TestRoutedLinkFaultsRoot: spoofed and link-local sources pass
Uplink discovery never returns an internal link TestUplinkDiscoveryNeverReturnsInternalLinks, on every discovery path; it fails with the exclusion removed pass

Also covered: a guest's partial checksum through the pair (TestPartialChecksumRoot), the routed link moving with the owner's pod (TestRoutedLinkRoot), withdrawal of everything Local wrote (TestWithdrawalRoot), and the owner pod check (TestProcPodsOpensOnlyTheCallersPod, TestProcPodsOpensThePodRoot).

Repository checks:

  • golangci-lint reports 0 issues.
  • task build, task test:unit and task test:e2e pass.
  • scripts/ci.sh unittest-root: every package this change touches passes.
    • The edgeprog and natprog FIB tests fail the same way on the base commit on this host.
    • The gc, hostgw and ingresssidecar churn tests passed on rerun.
  • yamlfmt cannot run from a git worktree. Taskfile.yaml, the only YAML changed, passes it from a copy.

A routed_link or socket claim's link reaches one pod. A duplicate
Request from another pod of the owner used to move the link, so two pods
holding one claim would move it back and forth on every call.

Worker now refuses a call on such a claim from another pod with
FAILED_PRECONDITION and reason ClaimHeldByAnotherPod while the pod the
link serves still runs, and changes nothing. Once that pod is gone, the
call moves the link to the calling pod, as after a pod replacement.
Claims whose handoff serves no pod never record a move. Worker learns
whether a pod runs from the host's /proc, by the same cgroup check the
API's peer check uses.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant