Skip to content

feat: Add the Galactic Worker service API and reconciliation core - #1446

Merged
privateip merged 6 commits into
integration/galactic-workerfrom
feat/issue-1433
Nov 18, 2026
Merged

privateip merged 6 commits into
integration/galactic-workerfrom
feat/issue-1433

Conversation

@privateip

Copy link
Copy Markdown
Collaborator

Summary

Service agents have no node-local API for tenant delivery, so every later Task in the Worker epic needs one authenticated service to build against.

This adds Galactic Worker: the service and its protobuf API, a client library, the reconciliation core and the module interface that the Local and Fabric delivery Tasks plug into. It also adds an independent example client that uses only the published API.

No delivery module is built in yet, so every capability reports that its module is not running until those Tasks land.

Test plan

  • Lint, build, unit, privileged unit and Kind e2e pass, and the new client target fails on any import beyond the published API
  • A request that reaches another VPC is refused, and routes, rules, links and pinned map rows are identical before and after
  • A release from a caller that does not own the claim is refused and changes nothing
  • The independent client requests, observes, updates and releases a claim using only its allowlisted capability

Fixes #1433

privateip and others added 5 commits November 18, 2026 18:02
Galactic Worker resolves every destination by comparing node locators,
and must read them from the same trusted source as the locator's local
route: the node's BGPRouter Block and Node-ID. The derivation lived
unexported in internal/installer, which Worker should not import.

Move it to internal/nodelocator.Prefix and keep the installer's
nodeLocatorPrefix as a thin wrapper, so its behavior and tests stay
as they are.

Co-Authored-By: Claude <claude@anthropic.com>
Service agents need one published interface to Galactic Worker. This
adds the galactic.worker.v1alpha1 protobuf service from
docs/worker/contract.md (GetCapabilities, Request, Update, Observe and
Release) and the contract's names: capabilities, conditions, refusal
and status reasons, the error domain and the session metadata key.

The proto lives in api/galactic/worker/v1alpha1 because buf's
STANDARD lint requires the directory to match the package. The service
keeps the contract's name, Worker, with SERVICE_SUFFIX ignored.

pkg/workerclient is the client library. It imports only the API and
gRPC, so an independent client can depend on it without Galactic
internals. It dials the unix socket, opens and renews a session, reads
the projected token for every call, drops duplicate and reordered
snapshots by revision, and retries the codes the contract says to
retry, with backoff from 200 ms to 30 s and full jitter.

Co-Authored-By: Claude <claude@anthropic.com>
Galactic Worker is the node-local service agents call to deliver
tenant traffic, and the single writer of shared host state
(docs/worker). This implements its reconciliation core and the
authenticated service, with no delivery module built in: Local
(#1435) and Fabric (#1436) plug into internal/worker/module. Until
then every capability reports ModuleNotRunning.

Calls are authenticated over a unix socket (mode 0660): TokenReview of
the projected token for audience galactic-worker, cached until it
expires; the bound pod must be on this node; the SO_PEERCRED process
must run in that pod, which the cgroup path shows; and a static YAML
allowlist names each ServiceAccount's capabilities.

A mutating call validates the request, authorizes the claim against
trusted sources (the attachment's VPC, node and admission) and records
desired state durably before it returns. A refusal records nothing,
takes no ticket and reaches no module. Every refusal carries a
contract reason and is counted in galactic_worker_refusals_total.

Reconciliation runs per VPC scope from a deduplicating queue, so
duplicate watch events coalesce. It resolves each destination by
locator (own locator Local, any other Fabric, never a fallback), takes
the scope's ticket from the internal/edge/ownership fence and commits
the scope's complete desired state through the modules, then reads the
host back. Desired state is durable; programmed and observed state are
rebuilt from the host, and Ready stays separate from Programmed.

On restart Worker fences and adopts each scope an earlier incarnation
owned, refuses changes there with Resyncing until the snapshot
commits, and never fences a scope another writer owns. A failed commit
leaves the scope resync-required. A release during that repair fences
the scope, so the repair's late commit cannot land. Only the owner
releases a claim, at its current generation. A caller with no live
session for a lease (120 s) has its claims withdrawn, as does a claim
whose attachment loses its authority.

Tests cover concurrent clients and racing callers; duplicate, stale
and out-of-order calls; crashes before programming, mid-apply and
before a release is recorded, then restart; duplicate watch events;
lease expiry; revocation; withdrawal during repair; and unavailable
modules. A privileged test programs real links, routes, rules and a
pinned map, then shows that a cross-VPC request leaves all of them
unchanged.

Co-Authored-By: Claude <claude@anthropic.com>
The contract needs a client that proves the published interface is
enough on its own. examples/worker-client is its own module and imports
only api/galactic/worker/v1alpha1 and pkg/workerclient; its module path
lies outside go.datum.net/galactic, so the compiler refuses internal
imports.

Its lifecycle subcommand checks the negotiated version and its
allowed capabilities, expects a forbidden handoff to be refused,
requests, repeats the request, observes, updates, sends a stale
release, releases and observes the release.

task test:worker-client builds, vets and tests the module and fails if
it imports any other Galactic package; test:unit runs it, so task ci
does too. TestIndependentClient builds the binary and runs its
lifecycle against a real Worker on a unix socket with the real
authentication chain, as a ServiceAccount allowed local.veth_pair
only. It also checks that another process holding the same token is
refused.

Co-Authored-By: Claude <claude@anthropic.com>
Implementing the contract settled six points it left open. The epic
coordinator approved each, and the contract now states them:

- D19: the session ID travels in x-galactic-worker-session; a call
  without a live session is FAILED_PRECONDITION/NoSession, and a
  GetCapabilities with a live session renews it
- D20: shard and backend destinations are in the API but report
  Resolved=False/DestinationKindUnsupported until #1437 and #1438 add
  their resolvers inside Worker
- D21: the peer check maps SO_PEERCRED to a pod by its cgroup path, so
  Worker needs hostPID and the host's /proc, which #1441 packages
- D22: the allowlist is a YAML file passed with --allowlist
- D18 now includes the claim limit, 1024 per caller
- Worker uses internal/edge/ownership in place; #1434 relocates it

The pages also record ClaimNotFound for an Update of a missing claim,
the status reasons the implementation reports, which capability a
claim needs, how Worker compares VPC identity (the base62 identifier
on the attachment's status, which is the VPC's fabric identity in this
cell), and that Worker adopts only scopes an earlier Worker owned.

Co-Authored-By: Claude <claude@anthropic.com>
@privateip privateip self-assigned this Nov 18, 2026
@privateip

Copy link
Copy Markdown
Collaborator Author

Acceptance evidence for #1433, from head cbcdc3de on base 550da21d. Each run happened locally in this branch's worktree.

  • task lint passes: golangci-lint reports 0 issues in the root module and all three example modules. The final yamlfmt -lint step fails in any git worktree, because the .git/** exclude in .yamlfmt cannot stat a .git file. The same yamlfmt lint, with that one exclude removed, passes over the whole tree. This branch changes no YAML except Taskfile.yaml.
  • task build passes and builds bin/galactic-worker.
  • task test:unit passes, including the new task test:worker-client. With a probe import of api/fabric/v1 added to the client, the target fails and names that package.
  • task test:e2e passes on a throwaway Kind cluster.
  • task test:unit-root passes TestCrossVPCRefusalLeavesKernelUnchanged. In the same run, the edgeprog and natprog FIB tests fail identically on the unmodified base 550da21d, so those failures predate this branch. gc and hostgw had two UnderLinkChurn tests time out under the full parallel run, then passed as root on their own on this branch.

Each acceptance bullet maps to these tests:

  • Concurrent clients, duplicate and out-of-order requests, restart with persisted or rediscovered state: TestConcurrentClients, TestDuplicateRequestChangesNothing, TestOutOfOrderRequests, TestRestartWithPersistedState, TestCrashBeforeProgramming, TestCrashMidApply, TestCrashBeforeReleaseRecorded, TestResyncingRefusesChanges and TestDuplicateWatchEvents. The suite passed 30 runs in a row under -race.
  • Foreign-VPC request with zero mutation: TestCrossVPCRefusalLeavesKernelUnchanged runs as root. A claim in VPC A programs a dummy link, a route, a policy rule and a pinned BPF map row in a test namespace. Requests that reach VPC B, which reuses the same guest addresses, are refused with VPCNotAuthorized. The sorted snapshots of links, routes in every table, rules and pinned map rows are identical before and after. Its unprivileged twin, TestCrossVPCRefusedWithoutMutation, also shows the store and fence files unchanged and no module call.
  • Non-owner release: TestNonOwnerRefused checks release, update and request from another caller, with store, fence and host unchanged.
  • Independent client: TestIndependentClient builds the example module's binary and runs its lifecycle against a real Worker on a unix socket. The run uses the real chain of token review, peer credentials and allowlist, with the client allowed only local.veth_pair. Its tap request is refused with CapabilityNotAllowed, and another process holding the same token is refused.

The approved gap decisions are recorded as D19 to D22, plus the claim limit in D18, in docs/worker/README.md and contract.md.

Admission compared locators to decide whether a claim needs
fabric.srv6, ignored lookup errors, and skipped the check while either
locator was unknown. During bring-up a caller without fabric.srv6
could be admitted with a destination on another node, and once the
locators appeared reconciliation programmed it as Fabric without
checking the owner again.

A destination on another node now needs fabric.srv6 whatever the
locators say, and a lookup error refuses with Resyncing. Worker also
records the owner's capabilities with each generation, and
reconciliation leaves unprogrammed any destination whose mode they do
not cover, reporting CapabilityNotAllowed on Programmed.

Co-Authored-By: Claude <claude@anthropic.com>
@privateip

Copy link
Copy Markdown
Collaborator Author

Two independent reviews of the first head both returned merge. The adversarial review found that the fabric.srv6 admission check failed open while a locator was unknown. d573ade6 closes it at admission and again at reconcile, and TestFabricCapabilityFailsClosed fails when either half of the fix is removed. The second pass on d573ade6 returned merge from both reviewers, and all 26 CI checks pass on that head.

@privateip
privateip marked this pull request as ready for review November 18, 2026 23:26
@privateip
privateip requested a review from a team as a code owner November 18, 2026 23:26
@privateip
privateip merged commit 1441530 into integration/galactic-worker Nov 18, 2026
28 checks passed
@privateip
privateip deleted the feat/issue-1433 branch November 18, 2026 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant