Skip to content

Implement Worker reconciliation and its authenticated service API #1433

Closed

Description

@privateip

Parent: Deliver Galactic Worker with Local and Fabric delivery for tenant services

Deliverable

A branch-buildable Worker with its client library, capability registry and deterministic reconciliation, implementing the #1431 contract. It satisfies the "authorized clients use the same API" and "an independent client exercises the contract" parts of the "Implement common foundation" phase exit.

Follow repository conventions:

  • Health gRPC in cmd/galactic-{gateway,nat,router}/root.go.
  • Metrics through go.miloapis.com/instrumentation, as docs/development/conventions.md "Metrics and logging" requires, with a scrape_test.go (TestMetricsScrape) for any /metrics endpoint.
  • The independent-client pattern of examples/looking-glass-client and task test:looking-glass-client.

Work items

  • Authorize ownership and tenant scope. Implement versioned requests, observed generation and per-capability status. Dispatch programming to the delivery modules (T5, T6) through a module interface. Keep observed readiness separate from accepted intent.
  • Serialize conflicting claims and split durable from reconstructible state. Recover from a crash after each side effect, with bounded retry and tolerance for duplicate watch events. Report a reason for each unavailable module. Allow release only by the current owner.
  • Add an independent client as its own module under examples/, with a task target that builds and tests it and fails if it imports Galactic packages beyond the published Worker API.

Dependencies

Blocked by #1431 (contract).

Acceptance

  • task test:unit covers concurrent clients, duplicate and out-of-order requests, and restart with persisted or rediscovered state.
  • A foreign-VPC request is rejected, and a test snapshots the routes, rules, links and pinned maps before and after to show zero mutation.
  • A release from a non-owner is rejected and leaves state unchanged.
  • The independent client requests, observes, updates and releases through the documented interface with only its advertised privileges, and its new task target passes in task ci.
  • The delivering PR targets integration/galactic-worker, carries this evidence and says Fixes datum-cloud/galactic#1433. GitHub does not auto-close from a non-default base, so close this issue explicitly when the PR merges and record the PR and merged SHA.

Activity

  1. added theissue type on Nov 18, 2026
  2. self-assigned this
    on Nov 18, 2026
  3. privateip commented on Nov 18, 2026

    @privateip
    CollaboratorAuthor

    Triage verdict: valid, unblocked. The #1431 contract is merged on integration/galactic-worker (550da21d31d3ef7b99256722b25b7770d4428b0a) under docs/worker/. That branch has no Worker yet: there is no cmd/galactic-worker, no api/worker and no Worker client. The precedents this Task reuses are present:

    • the internal/edge/ownership fence (Claim, Fence, Adopt, CommitSnapshot, Exclusive)
    • the TokenReview shape in internal/edge/masque/authority
    • gRPC health in cmd/galactic-nat/root.go
    • the examples/looking-glass-client module with its import guard in task test:looking-glass-client

    The approved implementation plan follows in the next comment.

  4. privateip commented on Nov 18, 2026

    @privateip
    CollaboratorAuthor

    Approved plan

    Layout

    • api/worker/v1alpha1/worker.proto defines the galactic.worker.v1alpha1.Worker service. Its bindings are generated by task generate:proto.
    • cmd/galactic-worker holds the binary. It serves the API on a unix socket, gRPC health on TCP and /metrics with a TestMetricsScrape.
    • internal/worker/... holds auth, sessions, the durable claim store, reconciliation, the module interface and capability registry, and trusted-source lookups.
    • pkg/workerclient is the client library.
    • examples/worker-client is an independent module. The new task test:worker-client target builds, vets and tests it, and fails if it imports Galactic packages other than api/worker/v1alpha1 and pkg/workerclient. task test:unit runs that target, so task ci does too.

    Auth chain

    1. The unix socket is the only listener for mutating calls.
    2. The projected ServiceAccount token (audience galactic-worker) is checked with TokenReview and cached until it expires.
    3. The token's bound node must be this node.
    4. SO_PEERCRED gives the peer PID, and that PID's cgroup must belong to the token's pod.
    5. A static YAML allowlist maps each ServiceAccount to the capabilities it may use.

    State

    • Desired state is durable: one file per claim, written with an atomic rename and fsync before Request, Update or Release returns.
    • Programmed and observed state are rebuilt from the host through each module's read-back.
    • Each per-VPC scope uses the existing internal/edge/ownership fence. On start, Worker runs Claim, or Fence then Adopt, then CommitSnapshot. Until that snapshot commits, the scope answers UNAVAILABLE/Resyncing.
    • Generation, spec-hash, owner and VPC checks run before any ticket is taken or anything is written.

    Delivery modules (#1435, #1436) implement this interface:

    • Capabilities
    • Plan, which has no side effects
    • Apply, which runs inside the fence commit
    • Read

    This Task uses test doubles for them.

    Contract additions, recorded in docs/worker/ within this PR:

    Tests

    • In task test:unit:
      • concurrent clients
      • duplicate, stale and out-of-order requests
      • a crash injected after each side effect, then restart from the persisted or rediscovered state
      • duplicate watch events
      • lease expiry
      • withdrawal during repair
      • a release from a non-owner, which is refused and changes nothing
    • In task test:unit-root: a cross-VPC request is refused, and snapshots of routes, rules, links and pinned maps taken before and after are identical. An unprivileged twin checks the store and the module only.
    • The independent client requests, observes, updates and releases a claim using only its allowlisted capabilities.

    Risk: the change is additive and no existing writer changes. The proto shape and module interface become binding for #1434 to #1440.

  5. privateip commented on Nov 18, 2026

    @privateip
    CollaboratorAuthor

    Delivered by #1446, merged into integration/galactic-worker as 1441530. The acceptance evidence is in the PR comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions