diff --git a/docs/README.md b/docs/README.md index 90c71250..0b180312 100644 --- a/docs/README.md +++ b/docs/README.md @@ -15,6 +15,7 @@ with the [system overview](overview.md) for components and ownership, or | Inspect metrics and alerts | [Observability](observability/README.md) | | Look up API resources | [API reference index](api/README.md) | | Evaluate a proposal or remaining implementation work | [Designs](designs/README.md) | +| Follow the Galactic Worker integration branch | [Worker contract](worker/README.md) | | Find a dated review or test result | [Archive](archive/README.md) | | Change this documentation | [Documentation maintenance](development/documentation.md) | diff --git a/docs/designs/README.md b/docs/designs/README.md index 32247a4d..e728b820 100644 --- a/docs/designs/README.md +++ b/docs/designs/README.md @@ -7,6 +7,7 @@ These documents explain decisions and planned work. Their status describes imple | [Interface traffic policy](interface-traffic-policy.md) | Implemented design; rollout remains explicit | [Rollout](../traffic-policy/rollout.md) and [node behavior](../traffic-policy/node.md) | | [Galactic Edge](galactic-edge.md) | Partially implemented and subject to qualification | [Provider contract](../edge/provider-contract.md) and its open items | | [Private Service Connect](private-service-connect.md) | Partially implemented; qualification gaps remain | [Private-service routes](../private-services/README.md) | +| [Galactic Worker](../worker/README.md) | Proposed; built on `integration/galactic-worker` | [Branch workflow](../worker/branch-workflow.md) | | [Looking Glass backend service](looking-glass-service.md) | Proposed implementation contract; authentication scaffolding does not enable service execution | [Current fabric-api](../fabric-api/README.md) | | [Shared DNS64](dns64/README.md) | Proposed; end-to-end validation outstanding | [NAT configuration](../nat/configuration.md) for implemented translation | | [Latency-aware routing](latency-aware-routing.md) | Exploratory proposal | [Router configuration](../router/configuration.md) for available settings | diff --git a/docs/worker/README.md b/docs/worker/README.md new file mode 100644 index 00000000..0e5dfa23 --- /dev/null +++ b/docs/worker/README.md @@ -0,0 +1,44 @@ +# Galactic Worker + +Galactic Worker is a proposed node-local service. Service agents use it to +deliver tenant traffic locally or across the fabric, and it becomes the single +writer of shared host state. These pages are the contract for +[epic #1430](https://github.com/datum-cloud/galactic/issues/1430). Every Task of +the epic implements against them on the `integration/galactic-worker` branch. + +Status: proposed. Nothing described here is implemented at the branch start. + +| Page | Contents | +| -------------------------------------- | ----------------------------------------------------------------------------------------- | +| [Service contract](contract.md) | Claims, operations, state model, errors, versions, trusted sources and worked transitions | +| [Host state ownership](ownership.md) | Writer inventory, one authority per resource class, bootstrap, adoption and transfer | +| [Connector decision](connector.md) | Which component is the provider connector, and its entry point | +| [Branch workflow](branch-workflow.md) | Base and start SHAs, PR target, CI, artifact tags and how Tasks close | +| [Qualification plan](qualification.md) | Platform, handoffs, workload, comparison method and limits | + +## Decision log + +"Accepted" means the epic coordinator approved the decision. "Proposed" means +the decision was set while writing this contract and is open to review on its +PR. "Confirmed" means the product owner approved the decision on the date given. + +| ID | Decision | Status | Page | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | ----------------------------------------------------------- | +| D1 | A claim is keyed by caller, node, VPC and attachment UID, and versioned by a client-assigned generation | Proposed | [Contract](contract.md#claims) | +| D2 | Desired, programmed and observed state are kept and reported separately | Proposed | [Contract](contract.md#state-model) | +| D3 | Every mutating call is idempotent per generation; retries use bounded backoff | Proposed | [Contract](contract.md#errors-and-retry) | +| D4 | Only the owner releases a claim, and only at its current generation | Proposed | [Contract](contract.md#operations) | +| D5 | Session lease of 120 s renewed every 10 s, the XDP dispatcher's values | Proposed | [Contract](contract.md#sessions-restart-and-stale-deletion) | +| D6 | gRPC status codes with machine reasons | Proposed | [Contract](contract.md#errors-and-retry) | +| D7 | Protobuf package `galactic.worker.v1alpha1`, negotiated through `GetCapabilities` | Accepted | [Contract](contract.md#version-negotiation) | +| D8 | gRPC over a host unix socket; projected ServiceAccount token with TokenReview, `SO_PEERCRED`, and a static ServiceAccount-to-capability allowlist | Confirmed 2026-11-18 | [Contract](contract.md#transport-identity-and-access) | +| D9 | `hostPath` socket mount, including for the galactic-vrf sidecar, which does not use the host network | Confirmed 2026-11-18 | [Contract](contract.md#transport-identity-and-access) | +| D10 | Trusted sources for locator, VPC, owner, backend SID and policy | Accepted | [Contract](contract.md#trusted-sources) | +| D11 | Own locator resolves to Local, any other to Fabric, with no fallback from Local to Fabric | Proposed | [Contract](contract.md#resolution) | +| D12 | One mutation authority means owning the scope's fence, or being a class's only writer; CNI writes with `Exclusive` | Accepted | [Ownership](ownership.md#what-one-mutation-authority-means) | +| D13 | Bootstrap, adoption and transfer use the `internal/edge/ownership` fence, generalized | Accepted | [Ownership](ownership.md#bootstrap-adoption-and-transfer) | +| D14 | The connector is `cmd/galactic-edge` with a Worker-backed `agent.Datapath` | Accepted | [Connector](connector.md) | +| D15 | Branch workflow with start commit `451eb40` | Accepted | [Branch workflow](branch-workflow.md) | +| D16 | NIC: Broadcom NetXtreme-C/E (`bnxt_en`) in generic mode; native mode on lab veth devices | Confirmed 2026-11-18 | [Qualification](qualification.md#platform) | +| D17 | Comparison against the main base on the same host, using the MASQUE benchmark method | Proposed | [Qualification](qualification.md#comparison-method) | +| D18 | Numeric limits | Confirmed 2026-11-18 | [Qualification](qualification.md#limits) | diff --git a/docs/worker/branch-workflow.md b/docs/worker/branch-workflow.md new file mode 100644 index 00000000..9c5a8fb0 --- /dev/null +++ b/docs/worker/branch-workflow.md @@ -0,0 +1,74 @@ +# Integration branch workflow + +Status: in use (D15). Every Task of +[epic #1430](https://github.com/datum-cloud/galactic/issues/1430) follows it. + +## Branch and base + +| Item | Value | +| ------------ | ------------------------------------------------------------------------------------------------------------ | +| Branch | `integration/galactic-worker` | +| Main base | `11323a99642aecb0e53220c1dbc49f61652f0b52`, the planning baseline and the comparison point for qualification | +| Branch start | `451eb40ec1ee6238f1e59dfeda8d69e594a4d8a8`, a signed empty commit whose parent is the main base | +| PR target | `integration/galactic-worker`, for every implementation and evidence PR | + +### Why the start commit exists + +`publish.yaml` tags images and the bundle with the short commit SHA, +`v0.0.0-`. If the branch started at the main base, its first push would +publish `v0.0.0-11323a9` again. A branch build is amd64 only, so that push +would replace the multi-architecture images main already published under the +same tag. The empty start commit gives the branch its own SHA, and every later +commit on the branch has a SHA that main does not. + +### Upstream changes + +The branch takes changes from `main` only on purpose. Record each one here, +with the main commit and the reason, in the PR that brings it in. + +| Main commit | Brought in by | Reason | +| ----------- | ------------- | ------ | +| None yet | | | + +## CI + +`ci.yaml` and `required-ci.yaml` run on `pull_request` without a base-branch +filter, so they run on every PR to `integration/galactic-worker`. Each PR runs +`task ci` locally before review and reports the CI runs against its base. + +## Artifacts + +`publish.yaml` runs on every branch push. The reusable workflows in +`datum-cloud/actions` (pinned at `v1.27.1`) tag a push to +`integration/galactic-worker` as follows: + +| Tag | Meaning | +| ---------------------------------------------------- | ----------------------------------------------------------------- | +| `v0.0.0-integration-galactic-worker` | Moves with every push to the branch | +| `v0.0.0-integration-galactic-worker-YYYYMMDD-HHmmss` | One per push, from the commit date | +| `v0.0.0-` | One per commit, for example `v0.0.0-451eb40` for the start commit | + +Branch images are amd64 only. The workflow publishes images for every +component except `galactic-debug`, which publishes only from `main` and release +tags, and a `galactic-kustomize` bundle. A branch push never moves +`v0.0.0-main`, which the staging labs follow, and never creates a `v*` release. +Qualification ([#1442](https://github.com/datum-cloud/galactic/issues/1442)) +pins artifacts by `v0.0.0-`. + +Pushing a Task's own branch (for example `docs/issue-1431`) also runs +`publish.yaml` and tags `v0.0.0-`. Those artifacts are for review +only and are never used for qualification. + +The [publish run for the push of the start commit](https://github.com/datum-cloud/galactic/actions/runs/44399031276) +is the evidence that branch pushes publish only the tags above. + +## Closing a Task + +1. The PR targets `integration/galactic-worker`, carries the Task's acceptance + evidence, and says `Fixes datum-cloud/galactic#`. +2. GitHub closes issues only from merges to the default branch, so after the + merge, check the PR's base and merged SHA. +3. Close the Task by hand with a comment naming the PR and the merged SHA. +4. Keep the epic open until the integrated evidence of #1442 is reviewed. + Merging the branch into `main`, publishing a release and rolling out to the + fleet each need a separate decision. diff --git a/docs/worker/connector.md b/docs/worker/connector.md new file mode 100644 index 00000000..425b5a5d --- /dev/null +++ b/docs/worker/connector.md @@ -0,0 +1,44 @@ +# Connector decision + +Status: decided (D14). [#1440](https://github.com/datum-cloud/galactic/issues/1440) +implements it. + +## Decision + +The provider connector is the existing `galactic-edge`. No `galactic-connector` +binary is created. + +| Item | Value | +| ----------- | --------------------------------------------------------------------------------------------- | +| Name | `galactic-edge` | +| Entry point | `cmd/galactic-edge` | +| Seam | `internal/edge/agent.Datapath` (`internal/edge/agent/agent.go:75`) | +| Change | A Worker-backed `Datapath` replaces `agent.PinnedDatapath` (`cmd/galactic-edge/main.go:197`). | +| Conformance | `tests/edgeconformance`, plus the independent provider in `examples/edge-provider` | + +## How it works + +`agent.Datapath` has four methods: `Register`, `Unregister`, `Forget` and +`SyncRoutes`. Today `PinnedDatapath` implements them by writing the pinned maps +directly through `attachreg`. The Worker-backed implementation turns them into +Worker calls instead: + +- `Register` becomes a `Request` (or an `Update` with a new generation) for a + `routed_link` claim on the EdgeAttachment, carrying the host link. +- `SyncRoutes` becomes an `Update` that replaces the claim's destinations. +- `Unregister` and `Forget` become a `Release`. + +galactic-edge keeps everything specific to providers: admission of the +EdgeAttachment, the gates, provider status and the contract with provider pods. +Worker admits a connector claim only while its EdgeAttachment is `Admitted`, and +withdraws it when admission is revoked. + +## Why not a new component + +- galactic-edge already registers host links through `attachreg`, takes the + per-VPC ownership fence that [ownership](ownership.md) generalizes, and is + covered by `tests/edgeconformance`. A new binary would duplicate all three. +- The `Datapath` interface is already the boundary between provider handling + and host programming, so the change stays inside one implementation of it. +- Providers that already ship, including the MASQUE provider, keep their + contract with galactic-edge unchanged. diff --git a/docs/worker/contract.md b/docs/worker/contract.md new file mode 100644 index 00000000..edfbbcf1 --- /dev/null +++ b/docs/worker/contract.md @@ -0,0 +1,199 @@ +# Worker service contract + +Status: proposed, version `v1alpha1`. Nothing in this page is implemented at +the branch start. [#1433](https://github.com/datum-cloud/galactic/issues/1433) +implements the API, and the Tasks named in each section own its tests. Decision +IDs (D1, D2, …) refer to the [decision log](README.md#decision-log). + +Galactic Worker is one process per node. Service agents (NAT, Gateway, the +Envoy galactic-vrf sidecar and the provider connector) ask it to deliver tenant +traffic between their own handoff point and tenant destinations. Worker checks +the request against existing authority, decides whether each destination is +Local (on this node) or Fabric (on another node), programs the host, and reports +what it programmed and what it observed. Packets never pass through Worker: it +programs kernel hooks, maps and routes, and the datapath forwards on its own. + +## Claims + +A **delivery claim** is the unit of the API (D1). + +| Field | Meaning | +| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `caller` | The authenticated client identity (D8). Worker derives it from the connection; a client cannot supply it. | +| `node` | The physical node. A request naming another node is refused with `INVALID_ARGUMENT`. | +| `vpc` | The VPC the claim acts in. It must equal the VPC of the claim's attachment (see [trusted sources](#trusted-sources)). | +| `attachment_uid` | UID of the VPCAttachment or EdgeAttachment that is the service's handoff point in that VPC. | +| `generation` | A client-assigned `uint64`. Every change to a claim carries a larger generation than the last accepted one. | +| `spec` | The handoff kind (`tap`, `veth_pair`, `routed_link`, `socket`), the destinations (an attachment, shard or backend reference each), and the mode preference. | + +The key is `(caller, node, vpc, attachment_uid)`. The handoff resources of one +attachment (its TAP, internal veth pair or routed link) belong to one caller. +A second caller that names the same attachment gets `PERMISSION_DENIED` with +reason `AttachmentClaimedByAnotherCaller`. Fabric state for a VPC is shared: +any number of claims may resolve destinations through the same VPC. + +The mode preference is `auto`, `local` or `fabric`. `auto` lets Worker resolve +each destination. `local` and `fabric` restrict the result; a destination that +cannot resolve to the requested mode reports `Resolved=False` with reason +`ModeUnavailable`. + +## Operations + +The API is a gRPC service, `galactic.worker.v1alpha1.Worker` (D7). + +| RPC | Semantics | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `GetCapabilities` | Version negotiation and capability discovery. Must be the first call of a session. Returns the chosen API version, the session ID and lease, and each capability with its state and reason. | +| `Request` | Creates a claim. Repeating the same generation and spec returns the current status and changes nothing. An existing claim with a different generation gets `FAILED_PRECONDITION`/`ClaimExists`; use `Update`. The same generation with a different spec gets `FAILED_PRECONDITION`/`GenerationConflict`. | +| `Update` | Replaces the spec of an existing claim. A generation smaller than the current one gets `ABORTED`/`StaleGeneration`. The same generation and spec is a no-op; the same generation with a different spec is `GenerationConflict`. | +| `Observe` | Server stream of status snapshots for one claim or for all of the caller's claims. Each snapshot is complete, so a client can drop duplicates and reorderings by `observed_generation`. The stream also renews the session lease. | +| `Release` | Removes a claim. The generation must equal the current generation. An older generation gets `ABORTED`/`StaleGeneration` and a newer one `FAILED_PRECONDITION`/`GenerationConflict`; neither changes anything. A missing claim returns success, so a retried release is safe. A caller that does not own the attachment gets `PERMISSION_DENIED`. | + +`Request`, `Update` and `Release` return once Worker has validated, authorized +and durably recorded the change. They do not wait for programming. The client +learns the outcome from `Observe`. + +### Capabilities + +`GetCapabilities` reports one row per capability: `local.tap`, +`local.veth_pair`, `local.routed_link`, `local.socket` and `fabric.srv6`. Each +row is `Available` or `Unavailable` with a machine reason, for example +`DatapathNotLoaded`, `HookUnsupported` (an XDP mode or device the +[qualified platform](qualification.md) excludes) or `ModuleNotRunning`. A claim +that needs an unavailable capability is accepted and reports `Programmed=False` +with the same reason, so a client can tell a temporary module outage from a +refused request. + +### Version negotiation + +The client sends the range of API versions it supports. Worker picks the +highest version both support, or fails with `FAILED_PRECONDITION` and reason +`UnsupportedVersion`, listing the versions it serves. Within a version, fields +are added only, and clients ignore fields they do not know. A breaking change +is a new package (`v1alpha2`, `v1beta1`), served alongside the old one until +every client on the branch has moved to it (D7). + +## State model + +Worker keeps three states for each claim, and never reports one as another (D2). + +| State | What it holds | Durable | +| ---------- | ------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | +| Desired | The accepted generation and spec, and the owner. | Yes, under `/var/lib/galactic-worker` | +| Programmed | What Worker wrote for each destination: mode, VRF table, SID, map rows, routes, and the read-back of each write. | Rebuilt from the host after a restart | +| Observed | Readiness evidence Worker did not write: the TAP or veth exists and is up, the neighbor resolves, CNI has attached. | Rebuilt from the host and from CNI attachment events | + +Status carries `observed_generation` and these conditions, matching the +EdgeAttachment condition names where they mean the same thing: + +| Condition | True when | +| ------------- | ------------------------------------------------------------------------------------------------------ | +| `Admitted` | The caller is authorized for the VPC, attachment and every destination. | +| `Resolved` | Every destination resolved to Local or Fabric. | +| `Programmed` | Programmed state matches desired state for `observed_generation`, confirmed by read-back. | +| `Ready` | `Programmed` is true and observed state shows every handoff can carry traffic. | +| `Withdrawing` | A release, revocation or lease expiry is removing programmed state. Status names the withdrawal scope. | + +The withdrawal scope names what stops: one destination, one claim, every claim +of a caller, or a whole VPC on the node. Operators read it to judge the reach +of a withdrawal ([#1441](https://github.com/datum-cloud/galactic/issues/1441)). + +### Resolution + +A destination whose locator is this node's locator resolves to Local; any other +locator resolves to Fabric (D11). Worker never falls back from Local to Fabric: +sending a Local destination out of the uplink would hairpin it and change the +return path the service depends on. A Local destination whose handoff is +missing waits, as the [worked transitions](#worked-transitions) show. + +### Trusted sources + +Worker takes these facts only from the sources below, never from the request (D10). + +| Fact | Source | +| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | +| Physical locator | The node's BGPRouter Block and Node-ID, the same source `internal/installer/locatorroute.go` uses for the locator's local route. | +| VPC identity | NetworkFabricIdentity for the VPC (`api/cloud/v1alpha1`), written by the fabric identity allocator. | +| Authorized VPC | `VPCAttachment.spec.vpc` for workload and service attachments, written by NSO; for a connector, an EdgeAttachment that is `Admitted`. | +| Owner | The authenticated caller (D8), recorded with the claim's desired state. | +| Backend SID | The backend node's locator plus the slot the gateway encodes in the uSID, as `docs/gateway/architecture.md` describes. | +| Interface traffic policy | TrafficPolicyBinding, resolved by the cell controller. Worker applies no policy of its own. | + +A claim is cross-VPC when any destination belongs to a VPC other than the +claim's attachment VPC. Worker refuses it. Two VPCs may use the same addresses, +so Worker compares VPC identity, never addresses. + +## Errors and retry + +| gRPC code | Reasons | Client action | +| --------------------- | ------------------------------------------------------------------------------------------ | ----------------------------------------------------------- | +| `INVALID_ARGUMENT` | `MalformedSpec`, `WrongNode` | Fix the request. Do not retry. | +| `UNAUTHENTICATED` | `NoIdentity`, `IdentityExpired` | Refresh the credential, then retry. | +| `PERMISSION_DENIED` | `VPCNotAuthorized`, `AttachmentClaimedByAnotherCaller`, `NotOwner`, `CapabilityNotAllowed` | Do not retry until authority changes. | +| `FAILED_PRECONDITION` | `UnsupportedVersion`, `ClaimExists`, `GenerationConflict` | Re-read with `Observe`, then use the right operation. | +| `ABORTED` | `StaleGeneration` | Re-read with `Observe`; retry only with a newer generation. | +| `UNAVAILABLE` | `Resyncing`, `ShuttingDown` | Retry with backoff. | +| `RESOURCE_EXHAUSTED` | `ClaimLimit`, `TableRangeExhausted` | Retry with backoff; report if it persists. | + +Clients retry with exponential backoff from 200 ms to 30 s with full jitter. +Every mutating call is idempotent for a given generation, so a retry after a +lost response is safe (D3). Every refusal carries a gRPC code and a machine +reason from this table (D6). Worker retries its own programming from 1 s to +60 s and reports the last error as the reason on `Programmed=False`. + +## Sessions, restart and stale deletion + +A session starts with `GetCapabilities` and carries a lease of 120 s, renewed by +an open `Observe` stream or a call at least every 10 s. These are the XDP +dispatcher's slot lease values (`internal/plumbing/ebpf/xdpdispatch`), already +qualified for detecting a dead service agent (D5). + +- **Client restart.** The client opens a new session and repeats `Request` for + each claim with its last generation. Worker answers each as a duplicate. The + owner is the caller identity, not the session, so a restart within the lease + changes no host state. +- **Lease expiry.** When a caller has no live session for 120 s, Worker marks + its claims `Withdrawing` with scope "every claim of the caller" and removes + their programmed state. Desired state is deleted once the read-back confirms + removal. +- **Authority revoked.** When the VPCAttachment is deleted or an EdgeAttachment + loses `Admitted`, Worker sets `Admitted=False` with reason + `AuthorizationRevoked` and withdraws the claim the same way + ([#1440](https://github.com/datum-cloud/galactic/issues/1440) tests + revocation). +- **Worker restart.** Worker reads desired state from disk, then for each scope + fences, adopts and reconciles a full snapshot, as [ownership](ownership.md#bootstrap-adoption-and-transfer) + describes. Programmed and observed state are rebuilt by reading the host. + Forwarding continues while Worker is down, because the datapath does not call + Worker. + +## Transport, identity and access + +D8 and D9 were confirmed on 2026-11-18. + +- **Transport (D8).** gRPC over a unix socket at + `/run/galactic-worker/worker.sock`, mode `0660`, owned by a dedicated group. + No TCP listener serves the mutating API. +- **Client identity (D8).** Each call carries a projected ServiceAccount token + with audience `galactic-worker`, bound to the calling pod. Worker validates it + with TokenReview, caches the result until the token expires, and checks that + the bound pod runs on this node. Worker also reads the peer's credentials + (`SO_PEERCRED`) and refuses a peer that is not a process of that pod. A static + allowlist maps each ServiceAccount to the capabilities it may use, for + example the NAT agent to `local.veth_pair` and `fabric.srv6`. +- **Access model (D9).** Host-network agents (NAT, Gateway, galactic-edge) mount + `/run/galactic-worker` with a `hostPath` volume. The galactic-vrf sidecar runs + in the Envoy pod's network namespace and mounts the same directory; pod + security admission limits that `hostPath` to the sidecar's ServiceAccount. + +## Worked transitions + +Each case names the decision it exercises and the Task that owns its test. + +| Case | Start | Event | Worker action | Result | Decision | Test owner | +| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Duplicate request | Claim at generation 4, `Ready=True`. | The client's response was lost, so it sends `Request` again with generation 4 and the same spec. | Compares generation and spec hash with desired state. Takes no ticket and writes nothing. | `OK` with the current status. Host state is byte-for-byte unchanged. The same generation with a different spec gets `FAILED_PRECONDITION`/`GenerationConflict`. | D1, D3 | [#1433](https://github.com/datum-cloud/galactic/issues/1433) unit test: duplicate and out-of-order requests | +| Stale release | Claim updated from generation 4 to 5, programming in progress. | A delayed `Release` with generation 4 arrives. | Compares the generation with desired state and refuses before taking a ticket. | `ABORTED`/`StaleGeneration`, carrying current generation 5. The claim and host state are unchanged, and programming of generation 5 continues. | D3, D4 | [#1433](https://github.com/datum-cloud/galactic/issues/1433) unit test: release from a stale generation and from a non-owner | +| Unauthorized cross-VPC claim | The caller has a claim on an attachment in VPC A. VPC B uses the same guest addresses. | `Update` adds a destination whose attachment is in VPC B. | Resolves the destination's VPC from its attachment record and its identity from NetworkFabricIdentity, finds B ≠ A, and refuses. Records nothing durable, takes no ticket, logs the refusal and counts it in a metric. | `PERMISSION_DENIED`/`VPCNotAuthorized`. The existing generation keeps running. Routes, rules, links and pinned maps are unchanged. | D1, D10 | [#1433](https://github.com/datum-cloud/galactic/issues/1433) zero-mutation snapshot test; [#1442](https://github.com/datum-cloud/galactic/issues/1442) observes the unauthorized destination in the lab | +| Local locator with a missing TAP | Claim admitted; one destination is a guest on this node. | Resolution finds the node's own locator, but the guest's TAP does not exist (not yet attached, or being replaced). | Resolves the destination to Local and does not fall back to Fabric. Programs the other destinations. Leaves the missing one unprogrammed and watches CNI attachment events. When CNI ADD creates the TAP, programs it on the new ifindex and reads it back. | Until the TAP appears: `Resolved=True`, `Programmed=False` and `Ready=False`, with reason `HandoffInterfaceMissing` naming the destination. Afterwards: `Ready=True`. | D2, D11 | [#1435](https://github.com/datum-cloud/galactic/issues/1435) root test for Local programming; [#1434](https://github.com/datum-cloud/galactic/issues/1434) for CNI attachment observation | +| Withdrawal during repair | A commit failed part-way, so the VPC scope is resync-required and Worker's repair holds a ticket while it prepares a snapshot. | The owner sends `Release` with the current generation. | Records the release durably and answers. Fences the scope, which invalidates the repair's ticket, then adopts it and commits a snapshot computed from desired state without the released claim. The repair's late commit fails with a transitioning or stale error and is dropped. | `Release` returns `OK`. Status shows `Withdrawing` with scope "one claim" until the read-back confirms removal, then `Observe` reports the claim released. No row of the released claim survives the snapshot. | D3, D12, D13 | [#1433](https://github.com/datum-cloud/galactic/issues/1433) crash and interleaving tests; [#1436](https://github.com/datum-cloud/galactic/issues/1436) for Fabric rows | diff --git a/docs/worker/ownership.md b/docs/worker/ownership.md new file mode 100644 index 00000000..5f790e15 --- /dev/null +++ b/docs/worker/ownership.md @@ -0,0 +1,99 @@ +# Host state ownership + +Status: proposed. This page lists every writer of shared node state at the +branch start and assigns each resource class one mutation authority. Line +references are to the base commit `11323a99642aecb0e53220c1dbc49f61652f0b52`. + +## What "one mutation authority" means + +For state that several processes write, the authority is the process that +**owns the fence** of the scope the state belongs to (D12). Only the owner holds tickets +(`Claim`, `Adopt`, `Begin`) and commits with `Commit` or `CommitSnapshot`. +Another process may still write in that scope, but only through the fence, with +`Exclusive`. An `Exclusive` write takes the scope lock, changes no owner, and +makes every outstanding ticket stale, so the owner reads the state again before +its next change. + +This interpretation keeps one property the epic requires: CNI attaches a +workload while Worker is down. CNI never owns a scope. It writes with +`Exclusive` whether or not Worker runs, and Worker adopts that state when it +starts. A write that bypasses the fence is a defect. + +The fence is `internal/edge/ownership`, generalized from its current VPC scope +(`internal/edge/ownership/ownership.go:150-317`). Its state directory must +outlive every writer, as the package documentation requires. + +A resource class has one of two kinds of authority: + +| Kind | Authority | Applies to | +| ------------ | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | +| Fenced | The owner of the node-and-VPC scope's fence | The host VRF and its table ID, the SID Argument, BGPVRFInstance, BGPAdvertisement, `vrf_table`, VPC routes and `egress_route_table` rows | +| Sole mutator | The one process that writes the class | State only one process writes, such as an attachment's own interface, program loading, TC attach and an XDP dispatcher slot | + +A sole mutator needs no fence, because no other process writes its class. If a +second writer appears, the class becomes fenced. + +## Writers and authorities + +| Resource class | Writers at the branch start | Coordination today | Authority after transfer | Transfer step | Task | +| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| Host VRF device and table ID (1 to `0xEFFF`) | `vrf.EnsureAndEnslave` (`internal/plumbing/vrf/vrf.go:116`) from galactic-veth and galactic-tap; `vrf.Add` (`vrf.go:93`) from `nodevpc.Ensure` in galactic-edge; `vrf.DeleteIfUnused` (`vrf.go:302`) from galactic-router GC | flock in `/var/lib/cni/galactic-vrf` (`lock.go:28`), reservations in `tables.json` (`reservation.go:35`) | Worker | CNI writes with `Exclusive`; galactic-edge calls Worker. GC moves into Worker. | [#1434](https://github.com/datum-cloud/galactic/issues/1434) | +| Per-node VPC state, SID Argument and BGPVRFInstance | `nodevpc.Ensure` (`internal/nodevpc/ensure.go:72`) from galactic-edge (`internal/edge/agent/vpcstate.go:30`); `ClaimArgument` (`argument.go:109`) and `AllocateArgument` (`argument.go:239`) from galactic-bgp and galactic-edge; the galactic-vrf sidecar writes BGPVRFInstance directly (`internal/ingresssidecar/gateway.go:273`) | Argument flock (`argument.go:33`); the sidecar's write takes no lock | Worker | Route the sidecar's BGPVRFInstance write through Worker. `argument.go:91` records that a writer outside the lock can still collide. | [#1434](https://github.com/datum-cloud/galactic/issues/1434), [#1439](https://github.com/datum-cloud/galactic/issues/1439) for the sidecar | +| `vrf_table` datapath rows | `attachreg.RegisterDatapath` (`internal/plumbing/ebpf/attachreg/register.go:98`) and `ReleaseArgument` (`register.go:223`) from galactic-bgp and `nodevpc`; galactic-cni repair and the `SweepEBPFVRFTable` sweep (`internal/gc/gc.go:749`) | Per-key convention and sweeps; no lock | Worker | galactic-bgp registers with `Exclusive`; Worker commits its own rows as owner. The sweep runs in Worker. | [#1434](https://github.com/datum-cloud/galactic/issues/1434) | +| `egress_route_table` rows | galactic-router (`internal/runtime/gobgp/vrfroutes.go:34`, `vrfmove.go:30`); galactic-bgp through `attachreg` (`register.go:386`, `register.go:460`); the galactic-vrf sidecar (`internal/ingresssidecar/backend.go:144`, `backend.go:159`); galactic-cni installer, repair and prune (`internal/installer/egressgroup.go:203`, `attachreg/fill.go:123`, `internal/plumbing/ebpf/sidecarmap/prune.go:80`); galactic-edge (`attachreg/hostlink.go:204`) | An in-process mutex in galactic-router only (`vrfroutes.go:81`); no cross-process lock | Worker (Fabric module) | Worker's Fabric module writes the rows. galactic-router keeps learning EVPN routes and hands them to Worker. The other writers call Worker, or use `Exclusive` until their Task moves them. | [#1436](https://github.com/datum-cloud/galactic/issues/1436); [#1439](https://github.com/datum-cloud/galactic/issues/1439) for the sidecar | +| Per-VPC kernel routes and rules | VRF unreachable defaults (`vrf.go`); the locator local route (`internal/installer/locatorroute.go`); sidecar-return routes (`internal/installer/sidecarreturn.go`) | None beyond the sidecar-return range from `0xF000` | Worker | Worker commits them as the scope owner. | [#1434](https://github.com/datum-cloud/galactic/issues/1434) | +| Sidecar return path (tables from `0xF000`) and reaping | `ensureSidecarReturnPath` (`internal/installer/sidecarreturn.go:159`); `deleteIngressAdvertisement` and `pruneAllSidecarRows` (`internal/installer/sidecarreap.go:243`, `sidecarreap.go:264`), all in galactic-cni | Evidence from advertisement annotations and the host veth | Worker | The sidecar requests a `socket` claim; Worker programs the return path and withdraws it on release or lease expiry. | [#1439](https://github.com/datum-cloud/galactic/issues/1439) | +| BGPAdvertisement for a VPC on a node | galactic-bgp; the galactic-vrf sidecar (`gateway.go:300`); edge route publication; galactic-cni reaping (`sidecarreap.go:243`); galactic-router GC (`RunGC`, `gc.go:1265`) | CRD optimistic concurrency | Worker | CNI records its containers with `Exclusive`; Worker writes service claims and owns deletion. | [#1434](https://github.com/datum-cloud/galactic/issues/1434), [#1436](https://github.com/datum-cloud/galactic/issues/1436) | +| GC of per-VPC state | `RunGC` (`internal/gc/gc.go:1265`) in galactic-router; `SweepEBPFVRFTable` (`gc.go:749`) and `SweepEBPFNPTv6Table` (`gc.go:1146`) in galactic-cni | VRF lock; the edge fence for edge scopes | Worker | Each sweep runs in Worker as the scope owner, on the same evidence it uses today. | [#1434](https://github.com/datum-cloud/galactic/issues/1434) | +| Edge host links | `RegisterHostLink`, `UnregisterHostLink` and `SyncHostLinkRoutes` (`attachreg/hostlink.go:50`, `:161`, `:203`) from galactic-edge | The edge VPC fence | Worker | galactic-edge calls Worker through its `Datapath` implementation ([connector](connector.md)). | [#1440](https://github.com/datum-cloud/galactic/issues/1440) | +| Attachment interface and its rows | galactic-veth and galactic-tap create and delete the device; `RegisterTenantGateway` (`register.go:632`) from galactic-bgp; `UnregisterInterface` (`attachreg/unregister.go:33`) from CNI DEL and `internal/installer/staletaps.go` | Device private to one attachment | CNI | None. Worker observes the interface and never deletes it. | [#1434](https://github.com/datum-cloud/galactic/issues/1434) for observation | +| Sidecar pod VRFs (tables from `0x10000`) in the pod | `vrf.AddInRange` (`internal/ingresssidecar/backend.go:95`) in the pod's network namespace | Separate network namespace | galactic-vrf sidecar | None. Service-specific state stays in the service agent. | [#1439](https://github.com/datum-cloud/galactic/issues/1439) | +| BGP and EVPN route origination | galactic-router (GoBGP, from CRDs) | One process | galactic-router | None. | None | +| Pinned programs and maps in `/sys/fs/bpf/galactic` | galactic-cni loads and pins them (`internal/plumbing/ebpf/attach/attach.go:31`) | One loader | galactic-cni (loading only; rows follow their class) | None. | None | +| TC attachment on an interface | galactic-cni (`attach.go:499`) | flock `/run/galactic/tc-.lock` | galactic-cni | None. | None | +| XDP dispatcher slots | galactic-gateway (slots 0 and 1) and galactic-nat (slot 2) through `Fill`, `Clear`, `Release`, `SetRole` and `Renew` (`internal/plumbing/ebpf/xdpdispatch/dispatcher.go:678-881`), 8 slots (`dispatcher.go:60`) | Lock on the pin directory and a 120 s lease per slot | The service that holds the slot | None. Worker reads slot state for capability status only. | None | +| `/sys/fs/bpf/galactic-nat`, `galactic-edge`, `galactic-xdp` | galactic-nat; galactic-gateway; both through the dispatcher | One writer each, plus the dispatcher lock | The owning service | None. | None | +| IPAM, FRR configuration | `internal/cni/ipam`; `cmd/fabric-config-agent` | flock each | Unchanged | None. | None | + +NAT translations and shard sessions, Gateway VIP selection and Maglev, and +Envoy's proxy sockets stay in their service agents. + +## Bootstrap, adoption and transfer + +Each step uses the fence operations that exist today (D13). + +1. **Bootstrap with Worker down.** CNI ADD writes the VPC scope with + `Exclusive`, which needs no owner. Workload attach never waits for Worker. + `Exclusive` returns `ErrTransitioning` only between `Fence` and `Adopt`, and + CNI retries within its own ADD deadline. +2. **Adoption of preexisting state.** When Worker starts, it takes each scope it + finds (from `vrf_table`, the VRF devices and `tables.json`). It uses `Claim` + for an unowned scope, or `Fence` then `Adopt` for one an earlier Worker + incarnation owned. Both return a ticket that requires `CommitSnapshot`, which + Worker runs with its complete desired state read from disk, merged with the + host's current rows. Until that snapshot commits, Worker refuses claim + changes for the scope with `UNAVAILABLE`/`Resyncing`. +3. **Ownership transfer.** A legacy writer moves to Worker in two steps. First it + writes with `Exclusive` on the shared fence instead of writing directly. + Then Worker adopts the scope, and the legacy writer calls Worker instead of + writing. The `Fence` before `Adopt` makes every earlier ticket stale. +4. **Cleanup after partial application.** A commit that fails, or whose process + dies part-way, leaves the scope resync-required. The next owner reconciles + a full snapshot before accepting incremental changes, so a half-applied + change is neither assumed done nor assumed undone. +5. **Loss of the state directory.** Stop every writer, reconcile host state, and + create a new authority, as `internal/edge/ownership` requires. Never restore + the directory while writers run. + +## Defects this transfer removes + +Two writers bypass any cross-process lock at the branch start. Both become +transfer steps of this epic, listed in the table above: + +- The galactic-vrf sidecar writes BGPVRFInstance without the Argument lock + (`internal/ingresssidecar/gateway.go:273`). +- Five processes write `egress_route_table` rows with no shared lock: + galactic-router, galactic-bgp, the galactic-vrf sidecar, galactic-cni and + galactic-edge. Table IDs keep only the sidecar apart from the others + (`internal/plumbing/ebpf/egressroutemap/owner.go`); the four host writers share + the host range from 1 to `0xEFFF` with nothing to separate them. diff --git a/docs/worker/qualification.md b/docs/worker/qualification.md new file mode 100644 index 00000000..4ce6d2d6 --- /dev/null +++ b/docs/worker/qualification.md @@ -0,0 +1,75 @@ +# Qualification plan + +Status: proposed. The NIC and the numeric limits (D16, D18) were confirmed on +2026-11-18. [#1444](https://github.com/datum-cloud/galactic/issues/1444) +proves the handoffs on this platform, and +[#1442](https://github.com/datum-cloud/galactic/issues/1442) runs the +comparison and reports against the limits. + +## Platform + +| Item | Value | +| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Kernel | 6.8 or later, the minimum the interface traffic policy preflight already enforces. Each run records the exact kernel release. | +| Lab | The two-worker Containerlab topology from [#1432](https://github.com/datum-cloud/galactic/issues/1432) (`deploy/containerlab`), reserved through the shared lab procedure. Kind nodes share the lab host's kernel. | +| NIC (D16) | Broadcom NetXtreme-C/E (`bnxt_en` driver), which runs generic XDP under the [selection rule](../operations/xdp-attach-mode.md). A `bnxt_en` NIC anywhere on a host selects generic mode for every interface on it, so native XDP is qualified on a different host without one: the lab's veth devices, and a NIC whose driver supports native XDP when such a host is available. Each run records the NIC model and driver; a hardware combination not run is listed as outstanding in the report. | +| XDP attach modes | Native and generic, as `internal/plumbing/ebpf/xdpmode` selects them. Both modes are qualified. | +| Device types | A plain interface, a bond member and a VLAN device, in each mode. | + +A combination of mode and device type that fails is recorded here as +unsupported, and Worker reports it as `HookUnsupported` in its capabilities. + +## Handoffs #1444 proves + +| Handoff | Capability | Consumer | +| ------------------------------------------------------------------------------------------------------- | ----------------------------------- | --------------------------------------------------------------------------- | +| Public receive `XDP_PASS` into the tenant-side TC program | `local.tap` | Gateway, [#1438](https://github.com/datum-cloud/galactic/issues/1438) | +| TC on one end of the NAT internal veth pair, XDP on its peer, and the reverse TC receive | `local.veth_pair` | NAT, [#1437](https://github.com/datum-cloud/galactic/issues/1437) | +| Envoy pod delivery to a VRF-bound upstream socket (`SO_BINDTODEVICE`) through a pod-to-host routed link | `local.routed_link`, `local.socket` | Envoy sidecar, [#1439](https://github.com/datum-cloud/galactic/issues/1439) | + +Each proof covers both directions, a packet that policy denies, and the +absence of local encapsulation on the physical wire. + +## Workload + +- NAT66 and NAT64, NLB and ALB (Envoy) run at the same time, against two VPCs + that use the same guest addresses. +- Destinations include pods and TAP-backed Unikraft guests, on the local and + the remote worker. +- Throughput uses 1280-byte inner IPv6 packets. A separate run uses 64-byte + packets for packet rate. Traffic is split 50/50 by bytes between TCP and UDP, + evenly in both directions. +- `iperf3` measures throughput and loss; `netperf` `TCP_RR` measures + request-response latency. + +## Comparison method + +This method follows the one the MASQUE contract uses (D17). + +1. Deploy the main base (`11323a9`) and the branch head on the same host and + topology, with artifacts pinned by `v0.0.0-`. +2. For each workload, warm up for 2 minutes, then measure for 10 minutes. + Run each workload three times. +3. Compare the median of the three branch runs with the median of the three + base runs. Report all runs, not only the medians. +4. Record the kernel, CPU model, NIC and driver, XDP mode, MTUs, and image + digests with each result. + +## Limits + +The limits were confirmed on 2026-11-18 (D18). Throughput and latency are judged +against the base on the same host, because absolute numbers depend on the +hardware. + +| Measure | Limit | +| ------------------------------------------------------------- | -------------------------- | +| Throughput, per service path and packet size | At least 95% of the base | +| Added p99 latency | At most 10% above the base | +| Packets delivered to an unauthorized VPC | 0 | +| Worker reconcile time, claim change to `Programmed=True` | p99 under 1 s | +| Re-convergence after a Worker restart, to every claim `Ready` | Under 5 s | +| Worker resident memory | Under 256 MiB | + +CPU per packet, drops and guest contention are measured and reported against +the base without a limit, as the MASQUE benchmark does, because they depend on +the host hardware.