Skip to content

Implement Local handoffs and verify tenant-side delivery readiness #1435

Open

Description

@privateip

Parent: Deliver Galactic Worker with Local and Fabric delivery for tenant services

Deliverable

A Local module that exposes the tested same-host handoffs to authorized consumers through Worker. It satisfies the Local half of the "Local and Fabric resolve mixed destinations correctly" phase exit. It implements the handoffs that #1444 proved.

References:

  • Dispatcher: internal/plumbing/ebpf/xdpdispatch (roles RolePublicLB, RoleInternalReturn, RoleEgress, with leases).
  • Tenant TC programs: internal/plumbing/ebpf/attach (AttachDeliver) and internal/plumbing/ebpf/prog/usid.c (tp_deliver).
  • Uplink discovery that must skip internal links: attach/interfaces.go (DetectUplinks), attach/uplinks.go (UplinkIndexes), natattach/uplinks.go (ResolveUplinks), srv6/egress.go (ResolvePublicUplink) and internal/installer/public_uplink.go.

Work items

  • Resolve same-host locality from the physical tenant-router locator and the current attachment. Distinguish a missing local binding from an off-host destination.
  • Provide the public XDP-to-TC handoff, the shared NAT internal veth pair and the per-pod, per-VPC routed link.
  • Separate link lifecycle from program ownership. Check hook presence, dispatcher claim, expected interface role, MTU and neighbor state before reporting ready.
  • Prevent public ingress from asserting trusted tenant context. Exclude internal links from every uplink discovery path listed above.

Dependencies

Blocked by #1431, #1433, #1434 and #1444. The #1432 fixtures are used for lab checks.

Acceptance

  • Privileged tests under task test:unit-root inject frames on every supported entry path and verify guest delivery or a policy denial at both TAPs.
  • Removing a hook withdraws readiness. A cold neighbor resolves and delivers. Delivery recovers after ifindex replacement.
  • An unclaimed packet, or one forged from outside, cannot enter a trusted handoff.
  • Uplink discovery never returns an internal link.
  • The delivering PR targets integration/galactic-worker, carries this evidence and says Fixes datum-cloud/galactic#1435. GitHub does not auto-close from a non-default base, so close this issue explicitly when the PR merges and record the PR and merged SHA.

Activity

  1. added theissue type on Nov 18, 2026
  2. privateip commented on Nov 19, 2026

    @privateip
    CollaboratorAuthor

    Triage: valid. Worker registers only the Fabric module today, so every local.* capability reports ModuleNotRunning. Locality (D11) is already resolved in Worker; this Task adds the module that programs and checks Local delivery.

    Plan

    Layout. A new internal/worker/local module, registered next to Fabric in cmd/galactic-worker. It is backed by internal/plumbing/ebpf/localprog, which holds Local's tc programs and a loader that reuses the uSID maps galactic-cni pins. Trusted sources gain each attachment's hostInterface. Local finds the live link through hostscope.AttachmentsOf, so a Local destination with no bound interface reports HandoffInterfaceMissing, and an off-host destination never reaches Local.

    Handoffs, following the contract's Local handoffs section (D27):

    • local.tap:
      • A tc ingress program on each interface whose dispatcher holds the public role. For a VLAN, tc goes on the VLAN and the dispatcher on its lower device.
      • It accepts only XDP metadata that names a TAP Local has authorized for that receiving interface. It decides on the PublicService path, marks the packet decided and delivers with bpf_redirect_neigh.
      • It checks the TAP's MTU and drops IPv6 fragments (counted).
      • The reply leaves through a TAP filter that runs after galactic-cni's filters.
    • local.veth_pair:
      • Local creates and owns one internal pair per VPC and NAT attachment. The NAT end stays native-XDP-capable.
      • Transmit checksum offload is off on the tenant end, so the kernel completes partial checksums and segments GSO frames before the NAT end's XDP.
      • Tenant-link tc redirects into the pair on the PublicNAT path, and tc on the pair's tenant end decides and delivers the return.
      • The XDP program on the NAT end stays NAT's (Integrate NAT66 and NAT64 with Local and Fabric shard delivery #1437).
    • local.routed_link and local.socket:
      • Local creates the /127 veth from the tenant VRF to the caller's pod. It creates the pod end only after checking that the target process belongs to the authenticated caller pod.
      • A guard drops sources outside the /127.
    • Local destinations get egress_route_table pass-through entries under a dedicated Local owner.

    Lifecycle and readiness. Link lifecycle (CNI for TAPs, Local for the pair and routed link) is separate from program ownership (Local's own tc filters and map rows, keyed by the current ifindex).

    Readiness requires:

    • Local's filter and CNI's chain present;
    • a live dispatcher slot with the expected role, or native XDP on the pair's NAT end;
    • the expected interface kind, VRF and carrier;
    • MTU limits met;
    • a neighbor that is not failed.

    Release removes only what Local owns. A link event, such as CNI replacing a TAP, requeues the VPC and reprograms on the new ifindex.

    Trust.

    • Every Local tc entry decides policy and sets the mark itself; it never trusts an incoming mark or unauthorized metadata.
    • Uplink discovery gains one shared exclusion for Galactic internal links, used by every discovery path the issue lists.

    Probe code. The prototype tc programs and loader from #1444 are promoted into product code. The prototype XDP programs remain test fixtures standing in for Gateway's and NAT's programs.

    Contract updates.

    • ownership.md: Local owns the internal pair and routed link, and is the sole mutator of its own tc filters under galactic-cni's per-interface lock.
    • contract.md: a minimal additive status field naming the handoff interface; Local readiness reasons; IPv6 fragments to a TAP are refused.

    Tests. Privileged tests (task test:unit-root) for each acceptance bullet:

    • inject frames on plain, bond-member and VLAN receiving paths in both XDP modes, and on the NAT return; check delivery or denial at two TAPs with overlapping addresses;
    • hook removal withdraws readiness; a cold neighbor resolves and delivers; delivery recovers after ifindex replacement;
    • forged or unclaimed packets and spoofed routed-link sources are dropped;
    • a caller cannot target another pod's network namespace;
    • uplink discovery never returns an internal link.

    The lab run repeats the #1444 handoff checks with Worker programming the handoffs.

    The PR targets integration/galactic-worker and says Fixes datum-cloud/galactic#1435.

  3. self-assigned this
    on Nov 19, 2026
  4. privateip commented on Nov 19, 2026

    @privateip
    CollaboratorAuthor

    Three changes from the approved plan, found while implementing it:

    1. Routed-link addressing. The plan gave the routed link a /127. Nothing in Worker's trusted sources allocates one, so the pod end now uses the claim attachment's own addresses (D26), at most four. Each address is a /128 route to the host end in the tenant VRF and gets a PassThroughLocal entry, and the guard admits only those addresses plus link-local ones. A pod cannot claim an address its attachment does not record. The contract records this as D29 (Proposed).

    2. Steering public flows into a handoff. With Local's filter after galactic-cni's (as approved), usid_egress hands Local only the traffic that has a pass-through entry. Local writes entries only for prefixes its own handoffs serve, the routed link's addresses. The resolver that selects a handoff writes the entries that steer public flows into it:

      The contract states this boundary (D29).

    3. Lab scope. Each lab site is its own cluster, so one Worker cannot resolve another site's attachments, and a compute node runs no gateway or NAT program. The lab run therefore checks Local programming and readiness on a compute node in both overlapping VPCs, against the real CNI and Cilium:

      • readiness reasons;
      • handoff_interfaces;
      • tc filter order;
      • unchanged uplink discovery;
      • release;
      • the full task verify suite with Worker active.

      The packet-level proof of every entry path is in the privileged tests, which run the real gateway and NAT stand-ins against the real datapath.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions