Skip to content

Define the Worker API, authority model and integration branch #1431

Closed

Description

@privateip

Parent: Deliver Galactic Worker with Local and Fabric delivery for tenant services

Deliverable

A versioned Worker contract, a single-writer ownership table, the integration-branch workflow and the qualification plan, checked in under docs/worker/ on integration/galactic-worker. It satisfies the contract half of the "Establish contract and baseline" phase exit, and every later Task implements against it.

No Worker or node-local service API exists at the baseline. Today several processes write shared host state through pinned BPF maps, CRDs and file locks. The ownership table starts from these writers:

  • Host VRFs and table IDs: internal/plumbing/vrf (EnsureAndEnslave, Locked, DeleteIfUnused, ranges in vrf.go and reservation.go).
  • Per-node VPC state and arguments: internal/nodevpc (Ensure, ClaimArgument, AllocateArgument), called from internal/cnibgp, internal/edge/agent/vpcstate.go, internal/edge/hostattach, cmd/galactic-edge and internal/gc.
  • Datapath registration: internal/plumbing/ebpf/attachreg (RegisterDatapath, RegisterTenantGateway, RegisterHostLink, SyncHostLinkRoutes, UnregisterInterface).
  • Sidecar host return: internal/installer/sidecarreturn.go and sidecarreap.go.
  • Garbage collection: internal/gc (RunGC, SweepEBPFVRFTable).
  • XDP dispatcher slots and leases: internal/plumbing/ebpf/xdpdispatch.
  • Pin directories /sys/fs/bpf/galactic, galactic-nat, galactic-edge and galactic-xdp.

Work items

  • Define request, observe, update and release semantics keyed by caller, physical node, VPC, attachment UID and generation. Specify capability discovery, errors, retry, idempotency, restart and stale deletion. Choose transport, client identity, version negotiation and the host and pod access model. Name the trusted source for physical locator, authorized VPC, owner and backend SID. Model desired, programmed and observed state separately.
  • Inventory every writer of VRF, table ID, argument, route, BGP advertisement, BPF attachment, pinned map and dispatcher slot, starting from the list above. Define bootstrap with Worker down, adoption of preexisting state, ownership transfer and cleanup after partial application. Service-specific state stays in its service agent.
  • Decide whether "galactic-connector" is a new component or the existing cmd/galactic-edge adapted behind internal/edge/agent.Datapath, and record the name and entry point T10 uses.
  • Record the branch workflow: base SHA, PR target, CI invocation, artifact tags and how Tasks close.
  • Agree the qualification platform: target kernel, NIC, XDP attach modes (internal/plumbing/ebpf/xdpmode), handoffs for Prove the Local kernel handoffs on the agreed platform #1444 to prove, workload, comparison method and numeric performance limits.

Dependencies

None. Can start now.

Acceptance

  • The contract carries a worked state transition for each of these cases: a duplicate request, a stale release, an unauthorized cross-VPC claim, a local locator with a missing TAP, and a withdrawal during repair.
  • The ownership table assigns each resource class exactly one mutation authority and names today's writer and the transfer step for each.
  • Every contract case names its design decision and the Task that owns its test.
  • The connector decision, the platform and numeric limits are recorded. No limit is left as "to be decided".
  • The PR shows ci.yaml and required-ci.yaml running against base integration/galactic-worker. It also links a publish.yaml run for the branch push that published only v0.0.0-integration-galactic-worker* and v0.0.0-<sha> tags, with no v* release and no v0.0.0-main bundle.
  • The delivering PR targets integration/galactic-worker, carries this evidence and says Fixes datum-cloud/galactic#1431. GitHub does not auto-close from a non-default base, so close this issue explicitly when the PR merges and record the PR and merged SHA.

Activity

  1. added theissue type on Nov 18, 2026
  2. privateip commented on Nov 18, 2026

    @privateip
    CollaboratorAuthor

    Verdict: valid. At 11323a99642aecb0e53220c1dbc49f61652f0b52 there is no Worker or node-local mutation API. The only node-local listeners are read-only: fabric-api queries over mTLS, the GoBGP socket behind a method allowlist, and gRPC health. Shared host state has many writers, each with its own coordination:

    • the VRF and table-ID flock, plus tables.json reservations
    • the SID Argument flock
    • the per-interface TC attach flock
    • the XDP dispatcher lock with 120 s slot leases
    • the per-VPC ownership fence in internal/edge/ownership, which only galactic-edge and its maintenance path use today

    Two writers have no cross-process coordination:

    • internal/ingresssidecar/gateway.go:273 writes BGPVRFInstance without the Argument lock. internal/nodevpc/argument.go:91 acknowledges that collision.
    • egress_route_table rows come from five processes with no shared lock: galactic-router, galactic-bgp, the galactic-vrf sidecar, galactic-cni (installer, repair and sweeps) and galactic-edge.

    Approved plan

    The design docs go under docs/worker/ on integration/galactic-worker, starting from branch commit 451eb40ec1ee6238f1e59dfeda8d69e594a4d8a8.

    File Content
    README.md Status, index and decision log
    contract.md Request, observe, update and release semantics; desired, programmed and observed state; errors and retry; versioning; trusted sources; five worked transitions
    ownership.md Writer inventory, one mutation authority per resource class, bootstrap, adoption, transfer and partial-apply cleanup
    connector.md Connector decision and the entry point #1440 uses
    branch-workflow.md Base and start SHAs, PR target, CI, artifact tags, Task closing
    qualification.md Kernel, NIC, XDP modes, handoffs for #1444, workload, comparison method, numeric limits

    Decisions grounded in the current code:

    • Ownership. "One mutation authority" means owning the scope's fence. The plan generalizes internal/edge/ownership (Fence, Adopt, CommitSnapshot, resync-required after a failed commit, Exclusive for GC). While Worker is down, CNI writes under a ticket, so workload attach never depends on Worker being available.
    • API. Protobuf package galactic.worker.v1alpha1. A Capabilities RPC negotiates the version and features.
    • Connector. No new galactic-connector binary. cmd/galactic-edge stays the connector, with a Worker-backed internal/edge/agent.Datapath.
    • Trusted sources. The node locator comes from the node's BGPRouter, VPC identity from NetworkFabricIdentity, attachment authority from VPCAttachment and EdgeAttachment, and the backend slot from the gateway's uSID encoding.

    These are proposals pending confirmation:

    • the transport and client identity: gRPC over a host unix socket, a projected ServiceAccount token checked with TokenReview, peer credentials, and a static allowlist mapping each ServiceAccount to its capabilities
    • the socket access model for the galactic-vrf sidecar, which does not use the host network
    • the NIC model and lab hardware
    • the numeric performance limits

    Each of the five worked transitions names its design decision and the Task that owns its test.

    Validation: task docs:check, task lint and task ci.

  3. privateip commented on Nov 18, 2026

    @privateip
    CollaboratorAuthor

    Delivered by #1445, merged into integration/galactic-worker as 550da21. The contract, ownership table, connector decision, branch workflow and qualification plan are in docs/worker/ on that branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions