Skip to content

Unify host VRF ownership and CNI attachment observation #1434

Closed

Description

@privateip

Parent: Deliver Galactic Worker with Local and Fabric delivery for tenant services

Deliverable

One authoritative host mapping per physical router identity and VPC, shared by guest (CNI) and service attachments and owned by Worker. It satisfies the "CNI and service reconcilers converge on one VRF" part of the "Implement common foundation" phase exit.

Current writers to fold in:

  • internal/nodevpc (Ensure, ClaimArgument) and internal/plumbing/vrf (EnsureAndEnslave, Locked, DeleteIfUnused).
  • internal/plumbing/ebpf/attachreg (RegisterDatapath, RegisterHostLink, UnregisterInterface).
  • The sidecar host-return writer in internal/installer/sidecarreturn.go and sidecarreap.go.

The CNI entry points are the RunPlugin functions in internal/{cnitap,cnibgp,cniroute}, which handle ADD, CHECK, DEL and STATUS. The TAP lifecycle is in internal/cni/tap/tap.go and host gateway setup in internal/hostgw. CNI has no GC verb. Garbage collection runs in galactic-router through internal/gc.RunGC.

Work items

  • Reconcile CNI argument claims and attach registration with the sidecar host-return writers under Worker ownership. Key by physical router identity and VPC. Retain references for live TAPs and service attachments. Define conflict handling for node tables that are already populated.
  • Integrate ADD, CHECK, DEL, STATUS and internal/gc repair. Preserve runtime ownership of TAPs, tenant policy and host-only ports. Read the current ifindex and guest neighbor at delivery time. Support legacy node state and idempotent adoption.
  • Keep workload attach working when the Worker service API is unavailable, as Define the Worker API, authority model and integration branch #1431 specifies.

Dependencies

Blocked by #1431 (ownership table) and #1433 (Worker reconciliation).

Acceptance

Privileged tests run under task test:unit-root (requireRoot(t)) and interleave CNI and sidecar reconciles for two VPCs with the same guest IP. After each of the following steps the tests verify the correct table, route, SID and live socket:

  1. Replace one TAP.
  2. Remove the last TAP while an Envoy attachment holds a reference.
  3. Restart Worker.
  4. Replay a stale DEL.

Further checks:

  • A workload attach with the Worker API down succeeds and converges once Worker returns.
  • Adopting legacy node state from the baseline leaves existing routes and maps unchanged.
  • The delivering PR targets integration/galactic-worker, carries this evidence and says Fixes datum-cloud/galactic#1434. GitHub does not auto-close from a non-default base, so close this issue explicitly when the PR merges and record the PR and merged SHA.

Activity

  1. added theissue type on Nov 18, 2026
  2. self-assigned this
    on Nov 18, 2026
  3. privateip commented on Nov 18, 2026

    @privateip
    CollaboratorAuthor

    Triage: valid, checked against integration/galactic-worker at 1441530.

    • Worker reconciles only VPC scopes that have claims (internal/worker/reconcile.go, Reconcile). It has no host-scope stage for the VRF, table, Argument, vrf_table or references, and it does not adopt scopes that already exist on the host.
    • Every CNI writer of shared VPC state writes directly, outside the ownership fence: vrf.EnsureAndEnslave in internal/cnitap/ops_add.go and internal/cni/ops_add.go; claimArgument, registerEBPFDatapath and the BGPAdvertisement writes in internal/cnibgp/bgp.go. The same holds for the installer's SweepEBPFVRFTable pass, the sidecar return and reap writers, and router RunGC.
    • One defect beyond the issue text: tap-mode DEL (internal/cnitap/ops_del.go) removes the host interface's ifindex-keyed map rows and radv record by interface name before tap.Delete checks the owner stamp. A replayed DEL from a predecessor container therefore strips the replacement TAP's rows. The veth DEL path checks ownership first. Acceptance step 4 (stale DEL) would fail today.
  4. privateip commented on Nov 18, 2026

    @privateip
    CollaboratorAuthor

    Approved plan.

    Ownership

    • Fenced by Worker (D12): the host VRF and table, the Argument and BGPVRFInstance, vrf_table, per-VPC routes, and GC of those.
    • CNI writes that state with Exclusive on VPCScope(vpc) and never calls Worker:
      • tap and veth ADD: vrf.EnsureAndEnslave
      • galactic-bgp ADD: Argument claim, datapath registration and BGPAdvertisement
      • galactic-bgp DEL: advertisement release
      • installer: the vrf_table sweep, and sidecar return and reap
    • CNI remains the only writer of per-attachment interfaces, tenant gateway and traffic-policy rows, program load and TC attach. Those stay unfenced.

    Worker

    • A host-scope stage runs inside every commit before the delivery modules. It calls nodevpc.Ensure and keeps a reference set per VPC: enslaved links, Worker claims, edge users, and advertisements with live containers.
    • At start, Worker discovers VPC scopes from the VRF links, the table reservations and vrf_table. It claims or takes over each scope with a snapshot commit, which writes nothing when the host state is consistent.
    • Attachment observation uses netlink link events. Modules read the current ifindex and guest neighbour at delivery time.

    GC (two-step transfer)

    • Router RunGC and the installer sweep skip VPCs whose fence owner is a Worker incarnation. For other VPCs they delete under Exclusive.
    • Worker runs the same per-VPC collection as scope owner, so a live service claim keeps the VRF alive.

    Fence

    • internal/edge/ownership moves to internal/ownership. DefaultDir is unchanged.
    • A new atomic Takeover replaces Fence followed by Adopt, so a Worker crash cannot leave a scope transitioning and block CNI ADD while Worker is down. It is recorded as a new decision in docs/worker.
    • The router and cni DaemonSets mount the fence directory.

    Stale DEL

    • Tap-mode DEL removes ifindex rows and radv state only while the TAP carries this container's owner stamp.

    Out of scope here

    Tests (task test:unit-root)

    • Two VPCs with the same guest IPv4 and IPv6 addresses, with CNI ADD, sidecar return and a Worker socket claim interleaved.
    • After each step, the tests check the table, the guest route, the SID and a live socket bound to the VRF. The steps are: replace a TAP; remove the last TAP while the service claim holds a reference; restart Worker; replay a stale DEL.
    • Attach with Worker down, and attach during a takeover; both converge once Worker returns.
    • Adopting baseline node state leaves routes, rules, links and pinned maps byte-for-byte unchanged.
    • Unprivileged tests cover the GC owner-skip and the reference set. task ci runs as well.
  5. privateip commented on Nov 19, 2026

    @privateip
    CollaboratorAuthor

    Delivered by #1447, merged into integration/galactic-worker as a639f63. The acceptance evidence is on the PR, and the privileged root tests pass in CI on the merged head. GitHub does not close issues automatically from a non-default base, so this closes by hand per #1430.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions