Skip to content

feat(worker): Deliver to remote guests through Worker's Fabric module - #1453

Merged
privateip merged 12 commits into
integration/galactic-workerfrom
feat/issue-1436
Nov 19, 2026
Merged

privateip merged 12 commits into
integration/galactic-workerfrom
feat/issue-1436

Conversation

@privateip

@privateip privateip commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1436

Worker sent traffic for a guest on another node nowhere: it resolved the guest as remote, but no module programmed the path. This adds the Fabric module. It delivers through the existing SRv6 encapsulation and the tenant routes galactic-router already learns, and becomes the only writer of those routes on a node where Worker runs.

What changes

  • galactic-router hands its routes to Worker. It streams every tenant route it learns or withdraws to Worker over the Worker socket, behind a new permission only the router holds. Where no Worker owns a VPC, the router writes the route itself, now under the VPC's write lock.
  • Worker writes them. The Fabric module writes the routes of every VPC it owns. It never writes into another VPC's table or back toward its own node, and it removes only what the router withdraws. When a remote node becomes unreachable it keeps the existing route rather than letting traffic fall through to a broader one. It reacts to underlay changes for the affected VPCs only.
  • Status per destination. Each remote destination reports whether traffic can reach it and whether replies can come back. A claim is ready only when both are true, and a withdrawal or an unreachable node changes only the destinations that use it.
  • No second writer. The CNI's periodic route refresh leaves alone the routes Worker owns.

The contract and router guides describe the new behavior, and the lab gains a script that runs this branch's Worker on the two-worker fixture. Packaging stays with #1441.

Not shown in the lab: claim readiness across two workers

The two-worker fixture places its workers in different clusters, and Worker looks up a claim's destinations only in its own cluster. A claim on one worker toward a guest on the other therefore reports the destination as not found. The lab does not prove claim readiness across two workers. Unit tests and a real-kernel test cover it instead. The cross-cluster gap is tracked separately.

Acceptance

The lab evidence is in the evidence comment on this PR.

Criterion Status
Outer SID, underlay egress, inner delivery, policy enforcement, return path Shown in the lab: captures in both VPCs and directions match the #1432 reference, and traffic-policy denials pass with Worker active
Readiness changes only for affected destinations, then recovers Lab: withdrawing one VPC's route removed only that VPC's entry, and restoring it wrote the entry back. Per-destination claim status is covered by unit and real-kernel tests
Unrelated VPCs keep passing traffic Lab: 600 of 600 echoes in the other VPC during the withdrawal and restore
Existing lab SRv6 and EVPN checks, privileged unit tests All lab checks pass with Worker active and after a fresh deploy from this branch. Privileged tests pass for every changed package

Galactic Worker's Fabric module becomes the writer of the tenant routes
in egress_route_table for every VPC a Worker incarnation owns, and keeps
their next hops current itself. The CNI installer's refresh sweep must
stop rewriting those entries outside the VPC's fence.

- WithoutTenantRoutes makes Refresh re-resolve only egress shard entries
  and sentinels in the given tables. The installer passes the tables of
  every host VRF whose VPC scope a Worker incarnation owns, and skips the
  sweep when it cannot read the fence, as it does for the sidecar's
  tables.
- ResolveNextHop resolves a SID's next hop without soliciting the
  neighbor. Its errors wrap ErrNoUplinkRoute or ErrNeighborUnresolved, so
  a caller can tell an unreachable locator from an unresolved neighbor.
- LookupRow returns the whole entry a longest-prefix lookup matches, and
  UnregisterIfSID removes an entry only while it encapsulates toward the
  given SID, keeping pass-through and shard group entries.
galactic-router learns tenant routes from EVPN, and Galactic Worker's
Fabric module is to write them. This adds the handoff and the status the
Fabric module reports through.

- galactic.worker.v1alpha1.Routes.Sync is a bidirectional stream on the
  Worker socket, with the Worker service's identity and sessions. Only a
  caller allowed the new fabric.routes capability may open one. Worker
  records each request's upserts and withdrawals in an in-memory route
  table, requeues the VPCs they change, and answers with the request's
  sequence and the routes it rejected. pkg/workerclient gains
  SyncRoutes.
- DestinationStatus gains forward_ready and return_ready; ready requires
  both. Modules report them in DestinationState.
- A destination's addresses come from its VPCAttachment's spec and are
  handed to the modules with the resolved destination.
- Reconcile also plans the modules for a scope with no claims, so state a
  module holds for a whole VPC, such as its tenant routes, is committed
  whether or not a claim exists.
- New status reasons for Fabric readiness: DestinationAddressUnknown,
  ForwardRouteMissing, RouteWithdrawn, RouteMismatch,
  UnderlayUnreachable, NeighborUnresolved and ReturnPathNotProgrammed.
Worker resolves a destination on another node's locator to Fabric, but
no module programmed it, so fabric.srv6 reported ModuleNotRunning. The
Fabric module delivers through the existing SRv6 encapsulator and the
tenant routes galactic-router learns, and is the single writer of those
routes for every VPC scope Worker owns.

- Plan writes each tenant route the Routes service recorded into
  egress_route_table inside the scope's commit, whether or not the
  scope has claims. It skips a route recorded for another table than
  the VPC's current one, so a route never lands in another VPC's table,
  and a route toward this node's own locator, which would loop. It
  removes an entry only for a withdrawn route, and never touches an
  entry galactic-router did not hand it.
- A route whose underlay next hop does not resolve keeps its entry, so
  a shorter entry such as the VPC's default toward an egress shard never
  catches its traffic, and a route that cannot be written is retried
  without failing the commit for the rest of the scope. A moved next hop
  is rewritten, and an entry left in a VRF's old table is removed.
- Read reports each Fabric destination separately. It is forward ready
  when the entry each of its addresses matches leads to the
  destination's own locator and its next hop resolves as written, and
  return ready when this node decapsulates the VPC's SID into the VPC's
  table.
- A route and neighbor subscription requeues only the VPCs whose routes
  an underlay event touches.

galactic-worker registers the module and serves fabric.srv6 and
fabric.routes once the pinned uSID maps are open.
galactic-router wrote its tenant routes into egress_route_table with
only an in-process mutex, alongside four other writers. It now writes
them under each VPC's write fence and hands them to galactic-worker,
whose Fabric module writes the routes of every VPC Worker owns.

- Every tenant route write, and each removal after a VRF moves table,
  runs under the VPC's fence scope with RefuseWorkerOwned. A VPC no
  Worker owns is written as before; one a Worker incarnation owns is
  left to Worker and counted as delivered. A withdrawal refused while
  ownership moves is retried with the pending installs.
- With GALACTIC_ROUTER_WORKER_SOCKET set, internal/runtime/workerroutes
  hands every route the router installs or withdraws to Worker through
  Routes.Sync, and withdraws every route of a VRF that is removed or
  moves to another table. It sends every route it holds and its recent
  withdrawals each time it opens a stream, then each change, and keeps
  changes while Worker is unreachable. Worker acknowledges a withdrawal
  before removing the entry, so an acknowledged withdrawal is kept for
  10 minutes and resent on each new stream; at most 65536 are kept.
- The base DaemonSet sets the socket, mounts /run/galactic-worker and
  projects a token with audience galactic-worker. Without a Worker on
  the node the router writes every route itself.
Record the decisions behind the Fabric module and the router's
handoff, and describe current behavior where operators look for it.

- Decisions D24 to D26: the Routes.Sync handoff behind fabric.routes,
  separate forward and return readiness, and destination addresses from
  the VPCAttachment's spec.
- The contract gains the tenant route handoff, forward and return
  readiness, the destination address and tenant route sources with the
  follow-up for guests that manage their own addressing, the new status
  reasons, the router's allowlist entry and the underlay loss
  transition.
- The ownership page records what moved for egress_route_table and the
  CNI writers that still write without the fence.
- The router guides document the fenced writes and the Worker socket
  and token settings.
The dedicated-placement fixture is where Fabric delivery is observed,
and Worker has no manifests yet. worker-fabric.sh builds the checkout's
galactic-worker and runs it as a process inside dfw-worker and
sjc-worker with the cluster's admin kubeconfig and an allowlist granting
galactic-router fabric.routes. It prints a node's VPC fence records, a
VPC's VRF table and that table's egress_route_table entries, and stops
Worker again.

Worker never hands a VPC scope back, so the lab must be redeployed after
stop. The fixture's two workers are in different clusters, and Worker
resolves claim destinations only within its own cluster, so the page
says the fixture does not show claim readiness across them.
@privateip privateip self-assigned this Nov 19, 2026
The Kind node image has no bpftool, so worker-fabric.sh dump could not
run it inside the node's mount namespace. It now runs the host's bpftool
on the pinned map through the node's /proc root.
printf pads a string argument with spaces even with a 0 flag, so table
looked up a VRF name with spaces in place of zeros and found none.
bpftool prints a formatted key and value only when the map carries BTF.
dump now also decodes the packed egress_route_table key and value from
the raw bytes, so it works either way.
@privateip

Copy link
Copy Markdown
Collaborator Author

Lab evidence: #1432 two-worker topology (ns80)

The run used head 0ead6681, which includes #1455, under the lab manager. The branch's galactic-router and galactic-cni were rolled out, then this branch's galactic-worker was started on dfw-worker and sjc-worker with scripts/worker-fabric.sh. Each check below ran with Worker active. At the end of the run the lab was destroyed and redeployed from this branch, then verified (see the last section).

Worker owns the VPCs, and the router hands it the routes

Both ns80 VPC scopes are owned by the Worker on each node, with no commit pending:

dfw-worker  vpc-80 owner=galactic-worker:dfw-worker:…  pending=false transitioning=false
dfw-worker  vpc-81 owner=galactic-worker:dfw-worker:…  pending=false transitioning=false
sjc-worker  vpc-80 owner=galactic-worker:sjc-worker:…  pending=false transitioning=false
sjc-worker  vpc-81 owner=galactic-worker:sjc-worker:…  pending=false transitioning=false

Both routers log handing tenant routes to galactic-worker. While the run was active, Worker logged 29 tenant route writes on dfw-worker and 39 on sjc-worker.

Overlapping guests get per-VPC SIDs

These are the egress_route_table entries for the guest subnets, read with worker-fabric.sh dump. Both VPCs use the same addresses. Each VPC encapsulates toward the remote node's locator with its own Argument, so VPC 80 and VPC 81 traffic decapsulate into different tables:

Node VPC Prefix Entry
dfw-worker 80 fd20:80:ff02::1:0:0/96 encap toward 2001:db8:ff02:1001:e006::
dfw-worker 81 fd20:80:ff02::1:0:0/96 encap toward 2001:db8:ff02:1001:e007::
sjc-worker 80 fd20:80:ff01::1:0:0/96 encap toward 2001:db8:ff01:1001:e005::
sjc-worker 81 fd20:80:ff01::1:0:0/96 encap toward 2001:db8:ff01:1001:e006::

Each node's own guest subnet stays a CNI pass-through entry, which the Fabric module does not touch.

Captures: both VPCs, isolation, NAT and gateway

baseline-observe.sh ran with Worker active and reported 108 checks with 0 failed. Its results.json matches the #1432 reference exactly, with an empty diff. The checks cover:

  • ICMPv6 between the dfw and sjc guests in both VPCs, in both directions. Each request and reply was seen at the underlay and on the guest, and the outer header is SRv6 toward the remote locator.
  • The isolation control. Each VPC's dfw guest received its own 5 datagrams and none of the other VPC's.
  • NAT66, NAT64, and gateway ingress over IPv6 and IPv4, including direct server return.

Withdraw and restore one VPC's route

sjc's VPC 80 guest was scaled to 0, then restored, while VPC 81 pinged across sites:

  1. Worker removed VPC 80's entry for fd20:80:ff02::1:0:0/96 7 s after the withdrawal. VPC 81's entry for the same subnet stayed.
  2. VPC 80 no longer reached the withdrawn guest.
  3. After the restore, Worker wrote VPC 80's entry again 9 s later, and verify:ns80 passed.
  4. VPC 81's ping during the whole window: 600 transmitted, 600 received, 0% loss.

Lab checks with Worker active

verify:srv6 and verify:evpn passed. All 27 task verify checks passed, each run on its own, with no allowance used. They include verify:traffic-policy, whose denials are enforced and counted, and verify:scenarios, which covers every tenant.

Checks that reported FAIL on a wrong pattern

Four checks in the run script looked for a /128 host entry. The router advertises the guest's /96, so these checks failed although the entries were right: "VPC A or B routes the sjc guest toward sjc's locator", "Worker removed only VPC A's entry" and "Worker wrote VPC A's entry again". The dumps and Worker's log above show what each check meant to establish. The last of the four also matched Worker's write from before the withdrawal, so it dumped the map before the rewrite.

Not shown here

Claim readiness across the two workers, as the PR description says.

Restore

After stopping Worker, the run destroyed the lab and deployed it fresh from this branch. Both checks that followed passed, with no allowance: verify:ns80, and all 27 task verify checks, each run on its own. The lab manager released the lab with no recovery needed.

@privateip
privateip marked this pull request as ready for review November 19, 2026 06:58
@privateip
privateip requested a review from a team as a code owner November 19, 2026 06:58
@privateip
privateip merged commit ee75981 into integration/galactic-worker Nov 19, 2026
28 checks passed
@privateip
privateip deleted the feat/issue-1436 branch November 19, 2026 06:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant