Skip to content

feat: Make Worker own each node's shared VPC host state - #1447

Merged
privateip merged 5 commits into
integration/galactic-workerfrom
feat/issue-1434
Nov 19, 2026
Merged

privateip merged 5 commits into
integration/galactic-workerfrom
feat/issue-1434

Conversation

@privateip

@privateip privateip commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

CNI, galactic-router and the CNI installer each write a VPC's shared host state on a node without the node's write fence, so Worker cannot become the single authority for that state.

CNI and the installer now make those writes through the fence without waiting for Worker, and router GC leaves VPCs that a Worker owns.

Worker takes over each node and VPC scope, adopts existing state without changing it, keeps the VRF while any claim, tap or edge attachment uses it, and reclaims it after a grace period.

A Worker that dies mid-handover no longer blocks workload attach, and a delayed tap DEL no longer strips a replacement tap's state.

Rollout

This merges into the Worker integration branch, which publishes only branch-scoped images and bundles, so no cluster picks up the new fence mount or write paths until a later decision moves the branch to main.

Test plan

  • Two VPCs whose guests share addresses keep separate tables, routes, SIDs and live sockets through tap replacement, last-tap removal under a service claim, Worker restart and a replayed stale DEL
  • A workload attaches while Worker is down, including mid-handover, and Worker converges on it when it returns
  • Adopting state built the baseline way leaves links, routes, rules, map rows and BGP objects unchanged
  • Lint, build, unit and root unit tests pass

Fixes #1434

Move internal/edge/ownership to internal/ownership. CNI, the installer,
galactic-router and Worker now all write through it, not only
galactic-edge. DefaultDir keeps its path, so no node state moves.

Add Takeover. It advances a scope's generation and names the new owner
under one lock, and the returned ticket requires a snapshot commit.
Worker previously ran Fence and then Adopt as two calls. A Worker that
died between them left the scope transitioning, and every Exclusive
writer was refused until Worker came back. CNI ADD writes shared VPC
state with Exclusive, so workload attach failed while Worker was down.

Worker now uses Takeover both to take over a scope from an earlier
incarnation and to fence a scope under repair when a claim is
released. docs/worker records this as D23. Takeover also completes a
transfer that an earlier Fence left open.

Add ExclusiveIf, an Exclusive write whose check reads the scope's
record under the lock, so a writer can refuse a scope by its owner
with no race. Add WorkerOwnerPrefix and WorkerOwned.

Fix tap-mode CNI DEL so it no longer removes a successor's state. DEL
removed the ifindex-keyed map rows and the router advertisement record
by host interface name before tap.Delete checked the owner stamp. A
predecessor's delayed DEL therefore stripped the rows of the TAP a
replacement container had taken over. tap.Release now checks the stamp,
removes the rows and deletes the tap as one sequence. DEL removes the
router advertisement record only when this container still owns the
tap. The veth path already checked ownership first.
CNI, the galactic-cni installer and galactic-router GC wrote a VPC's
shared host state without the node's VPC write fence. Worker cannot be
the fenced authority for the node and VPC scope (docs/worker
ownership.md, D12) while any of those writes bypass the fence. This
change makes each of them an Exclusive write. Exclusive needs no owner,
so workload attach still never waits for Worker.

- galactic-tap and galactic-veth ADD create the VRF and enslave the
  host link inside an Exclusive write (internal/cni/vpcfence).
- galactic-bgp runs each publish attempt (Argument claim,
  BGPVRFInstance, vrf_table registration and BGPAdvertisement), its
  rollback and the DEL advertisement release under the fence.
- The installer's sidecar return install and advertisement reap run
  under the endpoint's VPC fence. The vrf_table repair in
  SweepEBPFVRFTable writes through a VPCWriter, and the installer's
  writer refuses a scope a galactic-worker incarnation owns.
- maintenance.Guard.Reclaim checks the scope's owner under the lock
  with ExclusiveIf. galactic-router GC therefore keeps the VRF and
  BGPVRFInstance of a Worker-owned VPC, which Worker reclaims itself.
  This is the first step of the two-step GC transfer in ownership.md.

ExclusiveWait retries ErrTransitioning until its context ends, and CNI
bounds that wait at 30 s. The galactic-cni DaemonSet mounts the fence
directory, as galactic-router already did. Without the mount, the
installer's writes would fence against a private directory.
Make Worker the fenced authority for the canonical per-node, per-VPC
host state that CNI attachments and service claims share: the host VRF
and its table, the SID Argument and BGPVRFInstance, and the vrf_table
row (docs/worker ownership.md, D12).

- internal/worker/hostscope is Worker's view of that state. Plan reads
  it without writing. Ensure (nodevpc.Ensure) and Reclaim run inside
  Worker's commit. AttachmentsOf reads the links enslaved to a VPC's
  VRF, with their current ifindex and neighbors, at the time of the
  call.
- Worker discovers each VPC from its host VRF and from the fence
  records a Worker incarnation owns. It takes every scope no other
  writer owns with Claim or Takeover. A snapshot for a VPC with no
  claims writes nothing, so adopting baseline node state leaves it
  unchanged.
- A VPC with claims has its canonical state ensured before the
  delivery modules run. nodevpc.InPlace checks it without locks, so a
  VPC whose state is in place commits nothing.
- A VPC stays in use while Worker holds a claim in it, a link is
  enslaved to its VRF, or an edge attachment uses it. Once it has been
  unused for --host-grace (60 s), Worker reclaims the VRF and the
  BGPVRFInstance as scope owner. gc.ReclaimVPC runs RunGC's evidence for
  that VPC with the fence already held. The grace keeps the state
  across an attachment's replacement.
- A scope left transitioning by an earlier incarnation is completed
  even when its VRF does not exist yet. Otherwise the CNI ADD that
  would create the VRF would wait on it forever.
- galactic-worker subscribes to link events, so a CNI attachment
  joining or leaving a VPC's VRF requeues that VPC.

The privileged tests run under task test:unit-root. In
TestHostScopeTwoVPCsSameGuestIP, CNI ADDs and Envoy-style socket claims
interleave for two VPCs whose guests share IPv4 and IPv6 addresses.
After each step the test checks every VPC's table, guest routes, SID
and a live VRF-bound socket. The steps are: replace a tap; remove the
last tap while a socket claim references the VPC; restart Worker; and
replay a stale DEL. The test then runs one router GC pass, which leaves
the Worker-owned VPC and reclaims an unowned control VRF. Finally,
Worker reclaims the VPC once the claim is released.

TestAttachWithWorkerDown attaches a workload while no Worker runs, and
also into a scope left between Fence and Adopt. The Worker that then
starts converges on what CNI wrote. TestLegacyAdoptionUnchanged builds
node state the way the baseline CNI did and compares links, routes,
rules, every pinned map row and the BGPVRFInstances before and after
Worker adopts them.
@privateip privateip self-assigned this Nov 19, 2026
@privateip

Copy link
Copy Markdown
Collaborator Author

Acceptance evidence for #1434, from the branch head on a privileged development host. task test:unit-root runs every test below as root, and each one passed five times in a row. The test network namespace, bpffs pins and API objects are synthetic.

TestHostScopeTwoVPCsSameGuestIP interleaves tap-mode CNI ADDs with Envoy-style socket claims for two VPCs whose guests both use 198.51.100.2 and 2001:db8:a::/64. After every step it checks each VPC's VRF table, its vrf_table row, its BGPVRFInstance Argument and SID, and that the guest routes go through the VPC's current tap. It also sends a datagram to 2001:db8:ee::1 inside each VRF. The listener opened at the start, bound to that VPC's VRF, must receive it, and the other VPC's listener must not.

  1. Replace one tap. The successor gets a new ifindex, and the table, Argument and SID stay the same. Worker's attachment read reports the new ifindex.
  2. Remove the last tap while the Envoy claim references the VPC. A router GC pass in the same namespace leaves the Worker-owned VPC and reclaims an unowned control VRF in the same pass. Worker keeps the VPC past its grace period.
  3. Restart Worker. The new incarnation takes every scope over with Takeover. Links, routes, rules, every pinned map row and the BGPVRFInstances are byte-for-byte unchanged.
  4. Replay a stale DEL. The successor container takes the tap over first. The predecessor's DEL then leaves the tap, its ifindex_vrf_table row and its route in place.

After step 4 the claim is released, and once the grace passes Worker reclaims the VPC's VRF and BGPVRFInstance as scope owner.

TestAttachWithWorkerDown runs a CNI ADD on a scope owned by a dead Worker incarnation, which succeeds immediately. It runs another on a scope left between Fence and Adopt, which waits until a Worker starts and completes the transfer. The started Worker serves claims on the VRF and Argument that CNI created.

TestLegacyAdoptionUnchanged builds node state the way the baseline CNI did, with no fence. Adoption by Worker leaves the same full snapshot byte-for-byte equal.

TestReleaseLeavesSuccessorTap covers the tap-mode DEL defect on its own. On the base commit, DEL removed the ifindex rows by interface name before it checked ownership.

Checks run:

  • task lint: all four golangci-lint runs report 0 issues. yamlfmt -lint cannot run in a git worktree because .git there is a file. The two changed manifests pass it with the repository config when run outside the worktree, and kubectl kustomize renders both.
  • task build: passes.
  • task test:unit: passes. An earlier run timed out in TestAuthorizationRevoked. Run under load, that existing test hangs on the base commit too: its Observe stream can subscribe after the claim is already gone.
  • task test:unit-root: every package this change touches passes. Failures elsewhere were the edgeprog/natprog FIB-lookup tests, which also fail on the base commit on this host, and two link-churn tests, which pass when rerun alone.
  • task test:e2e: passes on Kind.

Not covered here: Worker's RBAC for BGPVRFInstance writes, and its hostNetwork and fence mounts. That packaging is #1441.

vpcfence.Wait bounds only the retry while ownership of a scope moves.
The scope's flock blocks without a deadline, so a lock holder that never
finishes stalls a CNI ADD past the 30 s wait. Say so in the constant's
comment.
The test env's crash cancelled the Worker and returned while a reconcile
could still be writing fence records. The test's temporary directory
cleanup then raced that write and failed with "directory not empty".
This failed TestClaimLimit in the privileged CI job on two consecutive
runs, and a loaded local run reproduces it on the base commit.

Worker.Wait returns once every goroutine Start started has returned.
crash calls it after cancelling, so an incarnation's late writes land
before the next incarnation starts or the test removes its state.
@privateip
privateip marked this pull request as ready for review November 19, 2026 01:23
@privateip
privateip requested a review from a team as a code owner November 19, 2026 01:23
@privateip
privateip merged commit a639f63 into integration/galactic-worker Nov 19, 2026
28 checks passed
@privateip
privateip deleted the feat/issue-1434 branch November 19, 2026 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant