Skip to content

Prove the Local kernel handoffs on the agreed platform #1444

Closed

Description

@privateip

Parent: Deliver Galactic Worker with Local and Fabric delivery for tenant services

Deliverable

Feasibility evidence for every Local kernel handoff that T1 selects, measured on T1's agreed kernel, NIC and XDP attach modes. Where a handoff fails, this Task also delivers a contract amendment and a proven alternative. It is the gate for the Local module (T5) and part of the "Establish contract and baseline" phase exit ("verified hook and socket feasibility").

Handoffs to prove:

  • Public RX XDP_PASS into the tenant-side TC program.
  • TC on one end of the NAT internal veth pair, XDP on its peer, and the reverse TC receive.
  • Envoy pod delivery to a VRF-bound upstream socket through a pod-to-host routed link. Envoy binds with SO_BINDTODEVICE; see internal/ingresssidecar/gatewayreply.go (ensureGatewayReplyRedirect).

Work items

  • Prototype each handoff with the existing loaders. The dispatcher is internal/plumbing/ebpf/xdpdispatch (xdp_dispatch, slots SlotGatewayLB/SlotGatewayReturn/SlotNAT) with internal/plumbing/ebpf/xdpattach. The TC programs come from internal/plumbing/ebpf/attach (Attach, AttachDeliver), and the NAT attach lives in internal/plumbing/ebpf/natattach.
  • Exercise native and generic XDP as selected by internal/plumbing/ebpf/xdpmode (Select), on bond members and VLAN devices (attach.expandBondSlaves).
  • Provide the tenant TAP fixture and capture point in the Build a reproducible baseline for dedicated placement #1432 topology, moved here from Build a reproducible baseline for dedicated placement #1432. A socat bridge stand-in did not become ready in the Kind lab during Build a reproducible baseline for dedicated placement #1432; its PR records the cause. Use a mechanism that works in that lab, or a host outside it that the PR names.
  • Record MTU limits, GRO/GSO and checksum-offload behavior, and the interface-trust requirements for each handoff.
  • For any handoff that fails, document the failure, amend the T1 contract document in this PR, then prototype and prove the chosen alternative here.

Dependencies

Acceptance

  • In the Build a reproducible baseline for dedicated placement #1432 topology, scripted observations cover each handoff in both directions. Each shows the packet arriving at the tenant TAP or being read from the upstream socket, a policy-denied control being dropped, and no local encapsulation on the physical wire, with a capture as proof.
  • The same checks pass in every attach mode and device type T1 lists. Any combination that isn't supported is recorded in the contract.
  • This Task cannot close on a recorded failure alone. Every handoff T5 consumes needs a passing proof, either of the original mechanism or of the alternative that the contract amendment adopts.
  • The scripts and expected observations are checked in and rerunnable. Lab use follows the shared Containerlab reservation procedure.
  • The delivering PR targets integration/galactic-worker, carries this evidence and says Fixes datum-cloud/galactic#1444. GitHub doesn't auto-close from a non-default base, so close this issue explicitly when the PR merges and record the PR and merged SHA.

Activity

  1. added theissue type on Nov 18, 2026
  2. self-assigned this
    on Nov 19, 2026
  3. privateip commented on Nov 19, 2026

    @privateip
    CollaboratorAuthor

    Plan

    Verdict. Valid. #1431 and #1432 are closed, and nothing on integration/galactic-worker (d967c51) proves the three Local handoffs docs/worker/qualification.md lists. The probe and rig code is test-only. No product loader changes.

    TAP fixture. In #1432 the TAP guest never started because galactic-tap was the pod's default network. Its CNI result names only the host tap (Sandbox: "", internal/cnitap/result.go), so the runtime had no pod interface to read. The fixture now gives the guest pod its normal primary network and adds ns80/vpc-a-tap as a secondary attachment. A privileged sidecar opens the host tap through socat ... netns= and bridges it to a guest-side tap0 in the pod. The overlay is opt-in (resources/tenants/ns80-tap/), so the #1432 baseline reference doesn't change.

    Prototypes. The programs are in tests/handoffs, loaded through the existing loaders: xdpdispatch on a private pin directory, xdpattach, natattach and attach. The real tp_deliver that CNI attaches stays in place. A rig on the compute worker provides a plain veth, an 802.3ad bond (XDP on its members) and a VLAN device, each tested in native and generic XDP mode. The shared edge dispatchers are not touched.

    • local.tap: a dispatcher slot sets metadata and returns XDP_PASS. A TC program on the receiving device redirects to the TAP, where tp_deliver applies policy. The reply leaves through the TAP's TC ingress.
    • local.veth_pair: guest TC redirects into the tenant end of an internal veth pair. XDP on the peer forwards to the device under test. The reverse path uses XDP_REDIRECT into the peer and is received by TC on the tenant end.
    • local.routed_link / local.socket: a pod-side VRF and a routed veth to the host VRF. A socket bound with SO_BINDTODEVICE reaches the guest and reads the reply in both overlapping VPCs. The same connect on an unbound socket is the negative control.

    Evidence. Each handoff is checked in both directions, in each mode and device type:

    • captures at the device under test, the tenant TAP or veth, the pair ends and the routed link
    • a policy-denied control (TCP 9), which must be absent from captures and counted
    • isolation between VPC 80 and VPC 81, which use the same addresses
    • zero SRv6 or probe packets on the physical bond members

    Measurements cover MTU limits, GRO/GSO/TSO, checksum offload and interface-trust requirements.

    Failures. Any mode and device combination that fails gets a HookUnsupported row in qualification.md, matching text in contract.md, and a passing proof of the alternative in the same PR. Two outcomes are expected to need this: native XDP on a VLAN device, and tp_deliver classifying public traffic on the Private path. The Broadcom hardware row stays outstanding because the lab host has no such NIC, and generic mode is forced on the rig; #1442 covers that hardware.

    Outputs.

    • Scripts: deploy/containerlab/scripts/handoff-*.sh
    • Expected observations: deploy/containerlab/handoffs/expected-results.json
    • Docs: deploy/containerlab/docs/handoffs.md
    • Taskfile targets: deploy:ns80-tap, handoff:observe, handoff:measure, handoff:twice

    Lab runs use the shared reservation. Each run stops at a precondition if the deployment doesn't match. Each run ends with no rig or overlay left and the baseline verify-clean.

    Risk. The changes are test-only and do not affect product behavior. The contract changes record limits that #1435 consumes.

    Verification. task ci, task test:unit-root, task handoff:twice and task baseline:twice in the lab.

    The PR will target integration/galactic-worker with Fixes datum-cloud/galactic#1444.

  4. privateip commented on Nov 19, 2026

    @privateip
    CollaboratorAuthor

    Done in #1456, merged into integration/galactic-worker as ca34660.

    All three Local handoffs pass in every cell, both XDP modes, on plain, bond and VLAN devices, in two identical lab runs against the committed expected results. Where a specified mechanism failed, the contract records the failure and the proven alternative (D27), and #1435 builds on those. The evidence is in the PR's evidence comment.

    The bnxt_en hardware run stays with #1442.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions