Repository navigation
feat: Make Worker own each node's shared VPC host state - #1447
Conversation
Move internal/edge/ownership to internal/ownership. CNI, the installer, galactic-router and Worker now all write through it, not only galactic-edge. DefaultDir keeps its path, so no node state moves. Add Takeover. It advances a scope's generation and names the new owner under one lock, and the returned ticket requires a snapshot commit. Worker previously ran Fence and then Adopt as two calls. A Worker that died between them left the scope transitioning, and every Exclusive writer was refused until Worker came back. CNI ADD writes shared VPC state with Exclusive, so workload attach failed while Worker was down. Worker now uses Takeover both to take over a scope from an earlier incarnation and to fence a scope under repair when a claim is released. docs/worker records this as D23. Takeover also completes a transfer that an earlier Fence left open. Add ExclusiveIf, an Exclusive write whose check reads the scope's record under the lock, so a writer can refuse a scope by its owner with no race. Add WorkerOwnerPrefix and WorkerOwned. Fix tap-mode CNI DEL so it no longer removes a successor's state. DEL removed the ifindex-keyed map rows and the router advertisement record by host interface name before tap.Delete checked the owner stamp. A predecessor's delayed DEL therefore stripped the rows of the TAP a replacement container had taken over. tap.Release now checks the stamp, removes the rows and deletes the tap as one sequence. DEL removes the router advertisement record only when this container still owns the tap. The veth path already checked ownership first.
CNI, the galactic-cni installer and galactic-router GC wrote a VPC's shared host state without the node's VPC write fence. Worker cannot be the fenced authority for the node and VPC scope (docs/worker ownership.md, D12) while any of those writes bypass the fence. This change makes each of them an Exclusive write. Exclusive needs no owner, so workload attach still never waits for Worker. - galactic-tap and galactic-veth ADD create the VRF and enslave the host link inside an Exclusive write (internal/cni/vpcfence). - galactic-bgp runs each publish attempt (Argument claim, BGPVRFInstance, vrf_table registration and BGPAdvertisement), its rollback and the DEL advertisement release under the fence. - The installer's sidecar return install and advertisement reap run under the endpoint's VPC fence. The vrf_table repair in SweepEBPFVRFTable writes through a VPCWriter, and the installer's writer refuses a scope a galactic-worker incarnation owns. - maintenance.Guard.Reclaim checks the scope's owner under the lock with ExclusiveIf. galactic-router GC therefore keeps the VRF and BGPVRFInstance of a Worker-owned VPC, which Worker reclaims itself. This is the first step of the two-step GC transfer in ownership.md. ExclusiveWait retries ErrTransitioning until its context ends, and CNI bounds that wait at 30 s. The galactic-cni DaemonSet mounts the fence directory, as galactic-router already did. Without the mount, the installer's writes would fence against a private directory.
Make Worker the fenced authority for the canonical per-node, per-VPC host state that CNI attachments and service claims share: the host VRF and its table, the SID Argument and BGPVRFInstance, and the vrf_table row (docs/worker ownership.md, D12). - internal/worker/hostscope is Worker's view of that state. Plan reads it without writing. Ensure (nodevpc.Ensure) and Reclaim run inside Worker's commit. AttachmentsOf reads the links enslaved to a VPC's VRF, with their current ifindex and neighbors, at the time of the call. - Worker discovers each VPC from its host VRF and from the fence records a Worker incarnation owns. It takes every scope no other writer owns with Claim or Takeover. A snapshot for a VPC with no claims writes nothing, so adopting baseline node state leaves it unchanged. - A VPC with claims has its canonical state ensured before the delivery modules run. nodevpc.InPlace checks it without locks, so a VPC whose state is in place commits nothing. - A VPC stays in use while Worker holds a claim in it, a link is enslaved to its VRF, or an edge attachment uses it. Once it has been unused for --host-grace (60 s), Worker reclaims the VRF and the BGPVRFInstance as scope owner. gc.ReclaimVPC runs RunGC's evidence for that VPC with the fence already held. The grace keeps the state across an attachment's replacement. - A scope left transitioning by an earlier incarnation is completed even when its VRF does not exist yet. Otherwise the CNI ADD that would create the VRF would wait on it forever. - galactic-worker subscribes to link events, so a CNI attachment joining or leaving a VPC's VRF requeues that VPC. The privileged tests run under task test:unit-root. In TestHostScopeTwoVPCsSameGuestIP, CNI ADDs and Envoy-style socket claims interleave for two VPCs whose guests share IPv4 and IPv6 addresses. After each step the test checks every VPC's table, guest routes, SID and a live VRF-bound socket. The steps are: replace a tap; remove the last tap while a socket claim references the VPC; restart Worker; and replay a stale DEL. The test then runs one router GC pass, which leaves the Worker-owned VPC and reclaims an unowned control VRF. Finally, Worker reclaims the VPC once the claim is released. TestAttachWithWorkerDown attaches a workload while no Worker runs, and also into a scope left between Fence and Adopt. The Worker that then starts converges on what CNI wrote. TestLegacyAdoptionUnchanged builds node state the way the baseline CNI did and compares links, routes, rules, every pinned map row and the BGPVRFInstances before and after Worker adopts them.
|
Acceptance evidence for #1434, from the branch head on a privileged development host.
After step 4 the claim is released, and once the grace passes Worker reclaims the VPC's VRF and BGPVRFInstance as scope owner.
Checks run:
Not covered here: Worker's RBAC for BGPVRFInstance writes, and its |
vpcfence.Wait bounds only the retry while ownership of a scope moves. The scope's flock blocks without a deadline, so a lock holder that never finishes stalls a CNI ADD past the 30 s wait. Say so in the constant's comment.
The test env's crash cancelled the Worker and returned while a reconcile could still be writing fence records. The test's temporary directory cleanup then raced that write and failed with "directory not empty". This failed TestClaimLimit in the privileged CI job on two consecutive runs, and a loaded local run reproduces it on the base commit. Worker.Wait returns once every goroutine Start started has returned. crash calls it after cancelling, so an incarnation's late writes land before the next incarnation starts or the test removes its state.
Summary
CNI, galactic-router and the CNI installer each write a VPC's shared host state on a node without the node's write fence, so Worker cannot become the single authority for that state.
CNI and the installer now make those writes through the fence without waiting for Worker, and router GC leaves VPCs that a Worker owns.
Worker takes over each node and VPC scope, adopts existing state without changing it, keeps the VRF while any claim, tap or edge attachment uses it, and reclaims it after a grace period.
A Worker that dies mid-handover no longer blocks workload attach, and a delayed tap DEL no longer strips a replacement tap's state.
Rollout
This merges into the Worker integration branch, which publishes only branch-scoped images and bundles, so no cluster picks up the new fence mount or write paths until a later decision moves the branch to main.
Test plan
Fixes #1434