Repository navigation
feat(worker): Deliver to remote guests through Worker's Fabric module - #1453
Conversation
Galactic Worker's Fabric module becomes the writer of the tenant routes in egress_route_table for every VPC a Worker incarnation owns, and keeps their next hops current itself. The CNI installer's refresh sweep must stop rewriting those entries outside the VPC's fence. - WithoutTenantRoutes makes Refresh re-resolve only egress shard entries and sentinels in the given tables. The installer passes the tables of every host VRF whose VPC scope a Worker incarnation owns, and skips the sweep when it cannot read the fence, as it does for the sidecar's tables. - ResolveNextHop resolves a SID's next hop without soliciting the neighbor. Its errors wrap ErrNoUplinkRoute or ErrNeighborUnresolved, so a caller can tell an unreachable locator from an unresolved neighbor. - LookupRow returns the whole entry a longest-prefix lookup matches, and UnregisterIfSID removes an entry only while it encapsulates toward the given SID, keeping pass-through and shard group entries.
galactic-router learns tenant routes from EVPN, and Galactic Worker's Fabric module is to write them. This adds the handoff and the status the Fabric module reports through. - galactic.worker.v1alpha1.Routes.Sync is a bidirectional stream on the Worker socket, with the Worker service's identity and sessions. Only a caller allowed the new fabric.routes capability may open one. Worker records each request's upserts and withdrawals in an in-memory route table, requeues the VPCs they change, and answers with the request's sequence and the routes it rejected. pkg/workerclient gains SyncRoutes. - DestinationStatus gains forward_ready and return_ready; ready requires both. Modules report them in DestinationState. - A destination's addresses come from its VPCAttachment's spec and are handed to the modules with the resolved destination. - Reconcile also plans the modules for a scope with no claims, so state a module holds for a whole VPC, such as its tenant routes, is committed whether or not a claim exists. - New status reasons for Fabric readiness: DestinationAddressUnknown, ForwardRouteMissing, RouteWithdrawn, RouteMismatch, UnderlayUnreachable, NeighborUnresolved and ReturnPathNotProgrammed.
Worker resolves a destination on another node's locator to Fabric, but no module programmed it, so fabric.srv6 reported ModuleNotRunning. The Fabric module delivers through the existing SRv6 encapsulator and the tenant routes galactic-router learns, and is the single writer of those routes for every VPC scope Worker owns. - Plan writes each tenant route the Routes service recorded into egress_route_table inside the scope's commit, whether or not the scope has claims. It skips a route recorded for another table than the VPC's current one, so a route never lands in another VPC's table, and a route toward this node's own locator, which would loop. It removes an entry only for a withdrawn route, and never touches an entry galactic-router did not hand it. - A route whose underlay next hop does not resolve keeps its entry, so a shorter entry such as the VPC's default toward an egress shard never catches its traffic, and a route that cannot be written is retried without failing the commit for the rest of the scope. A moved next hop is rewritten, and an entry left in a VRF's old table is removed. - Read reports each Fabric destination separately. It is forward ready when the entry each of its addresses matches leads to the destination's own locator and its next hop resolves as written, and return ready when this node decapsulates the VPC's SID into the VPC's table. - A route and neighbor subscription requeues only the VPCs whose routes an underlay event touches. galactic-worker registers the module and serves fabric.srv6 and fabric.routes once the pinned uSID maps are open.
galactic-router wrote its tenant routes into egress_route_table with only an in-process mutex, alongside four other writers. It now writes them under each VPC's write fence and hands them to galactic-worker, whose Fabric module writes the routes of every VPC Worker owns. - Every tenant route write, and each removal after a VRF moves table, runs under the VPC's fence scope with RefuseWorkerOwned. A VPC no Worker owns is written as before; one a Worker incarnation owns is left to Worker and counted as delivered. A withdrawal refused while ownership moves is retried with the pending installs. - With GALACTIC_ROUTER_WORKER_SOCKET set, internal/runtime/workerroutes hands every route the router installs or withdraws to Worker through Routes.Sync, and withdraws every route of a VRF that is removed or moves to another table. It sends every route it holds and its recent withdrawals each time it opens a stream, then each change, and keeps changes while Worker is unreachable. Worker acknowledges a withdrawal before removing the entry, so an acknowledged withdrawal is kept for 10 minutes and resent on each new stream; at most 65536 are kept. - The base DaemonSet sets the socket, mounts /run/galactic-worker and projects a token with audience galactic-worker. Without a Worker on the node the router writes every route itself.
Record the decisions behind the Fabric module and the router's handoff, and describe current behavior where operators look for it. - Decisions D24 to D26: the Routes.Sync handoff behind fabric.routes, separate forward and return readiness, and destination addresses from the VPCAttachment's spec. - The contract gains the tenant route handoff, forward and return readiness, the destination address and tenant route sources with the follow-up for guests that manage their own addressing, the new status reasons, the router's allowlist entry and the underlay loss transition. - The ownership page records what moved for egress_route_table and the CNI writers that still write without the fence. - The router guides document the fenced writes and the Worker socket and token settings.
The dedicated-placement fixture is where Fabric delivery is observed, and Worker has no manifests yet. worker-fabric.sh builds the checkout's galactic-worker and runs it as a process inside dfw-worker and sjc-worker with the cluster's admin kubeconfig and an allowlist granting galactic-router fabric.routes. It prints a node's VPC fence records, a VPC's VRF table and that table's egress_route_table entries, and stops Worker again. Worker never hands a VPC scope back, so the lab must be redeployed after stop. The fixture's two workers are in different clusters, and Worker resolves claim destinations only within its own cluster, so the page says the fixture does not show claim readiness across them.
The Kind node image has no bpftool, so worker-fabric.sh dump could not run it inside the node's mount namespace. It now runs the host's bpftool on the pinned map through the node's /proc root.
…o feat/issue-1436
printf pads a string argument with spaces even with a 0 flag, so table looked up a VRF name with spaces in place of zeros and found none.
bpftool prints a formatted key and value only when the map carries BTF. dump now also decodes the packed egress_route_table key and value from the raw bytes, so it works either way.
…o feat/issue-1436
Lab evidence: #1432 two-worker topology (ns80)The run used head Worker owns the VPCs, and the router hands it the routesBoth ns80 VPC scopes are owned by the Worker on each node, with no commit pending: Both routers log Overlapping guests get per-VPC SIDsThese are the
Each node's own guest subnet stays a CNI pass-through entry, which the Fabric module does not touch. Captures: both VPCs, isolation, NAT and gateway
Withdraw and restore one VPC's routesjc's VPC 80 guest was scaled to 0, then restored, while VPC 81 pinged across sites:
Lab checks with Worker active
Checks that reported FAIL on a wrong patternFour checks in the run script looked for a Not shown hereClaim readiness across the two workers, as the PR description says. RestoreAfter stopping Worker, the run destroyed the lab and deployed it fresh from this branch. Both checks that followed passed, with no allowance: |
Fixes #1436
Worker sent traffic for a guest on another node nowhere: it resolved the guest as remote, but no module programmed the path. This adds the Fabric module. It delivers through the existing SRv6 encapsulation and the tenant routes galactic-router already learns, and becomes the only writer of those routes on a node where Worker runs.
What changes
The contract and router guides describe the new behavior, and the lab gains a script that runs this branch's Worker on the two-worker fixture. Packaging stays with #1441.
Not shown in the lab: claim readiness across two workers
The two-worker fixture places its workers in different clusters, and Worker looks up a claim's destinations only in its own cluster. A claim on one worker toward a guest on the other therefore reports the destination as not found. The lab does not prove claim readiness across two workers. Unit tests and a real-kernel test cover it instead. The cross-cluster gap is tracked separately.
Acceptance
The lab evidence is in the evidence comment on this PR.