Skip to content

test: Record a reproducible baseline for dedicated placement - #1451

Merged
privateip merged 2 commits into
integration/galactic-workerfrom
feat/issue-1432
Nov 19, 2026
Merged

privateip merged 2 commits into
integration/galactic-workerfrom
feat/issue-1432

Conversation

@privateip

@privateip privateip commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Later Galactic Worker tasks need a fixed starting point to measure against: today's remote paths on dedicated edge and compute nodes, captured the same way every time.

This adds a lab fixture with two VPCs whose guests share addresses across two compute workers, plus scripted observations of cross-node delivery, NAT66 and NAT64 egress, and gateway ingress over IPv4 and IPv6, each in both directions. A negative control proves a guest never receives traffic sent to its twin in the other VPC, and the run pins and records tool, kernel and image versions and stores a reference result for later runs to compare against.

The TAP capture point moved to #1444, which records why the TAP stand-in failed in Kind.

Evidence scope

The lab evidence is one managed run of two observation passes at this branch's tree, plus an earlier full lab verification with the fixture deployed. Changes after that verification touched only the baseline scripts, not the fixture.

Gateway ingress counts as delivered when each protocol has answered flows, because the lab's own ingress checks lose a TCP flow now and then. With no captures running, one of three IPv4 ingress checks answered 6 of 8 TCP flows. The captures show the request reaching the backend, then the backend's socat responder closing without a reply. That fault predates this change, is tracked in #1452, and is not fixed here.

Test plan

  • Two back-to-back observation runs pass with identical results and match the committed reference
  • Each path delivers in both directions, and the isolation control shows zero cross-VPC deliveries
  • The lab's full verification still passes with the fixture deployed
  • Lint, build, unit and end-to-end tests pass

Fixes #1432

Add ns80, two VPCs (80 and 81) whose guests take identical fixed
addresses on dfw's and sjc's compute workers, with gateway and NAT left
on the dedicated edge nodes. deploy:scenarios deploys it and
verify:scenarios checks placement, the overlap and cross-site ping.

Add task baseline:observe, baseline:twice and baseline:versions. The
observations cover cross-node delivery in both VPCs, NAT66 and NAT64
egress from both overlapping guests at once, gateway ingress over IPv6
and IPv4, and a negative isolation control. Each scenario captures at
the public side, the guests' veth pairs and the remote underlay.
results.json keeps only stable fields, so two runs compare equal;
ports, nonces and translated tuples go to volatile.tsv.

baseline/pins.yaml pins the host tools, the Kind node, FRR, the
remote-host base and the guest image digest, and the run fails on any
mismatch. baseline/reference-results.json is the result of two
identical passing runs.

Recorded limits: inbound packets bypass the host veth through
bpf_redirect_peer, the edge nodes forward in XDP out of tcpdump's view,
and the lab's gateway checks intermittently lose a TCP flow at the
ns60 responder, so those are observations or per-run values. The TAP
capture point moved to #1444.

Fixes #1432
@privateip privateip self-assigned this Nov 19, 2026
@privateip

Copy link
Copy Markdown
Collaborator Author

Acceptance evidence

All lab runs used the shared Containerlab reservation procedure (lab-manager run, with lab-manager recover for the two recoveries). The fixture and baseline were measured on the integration base tree (451eb40, the same tree as 11323a9), at commit 8c16831d.

Acceptance bullet Evidence
Clean lab host, documented task commands, two identical passing runs task destroy, then task deploy (all images rebuilt), then task baseline:twice. Both runs passed all 108 checks (80 with an expectation, 28 recorded observations, 0 failed), their results.json files are identical (sha256 prefix c7b6919f46812241), and run 1 became baseline/reference-results.json.
Each remote path delivers in both directions Cross-node ICMPv6 in VPC 80 and VPC 81: 5 of 5 replies each way, requests seen at both guests and at both sites' edge uplinks on the transit routers. NAT66 and NAT64: both overlapping guests fetched from the external peer at once from one identical source port and got 200, and the peer saw 2 distinct translated tuples, both from shard addresses. Gateway ingress over IPv6 and IPv4: answered TCP and UDP flows, with requests and direct-server-return replies seen at the public side and at the backends.
Negative isolation control 5 UDP probes from each VPC's sjc guest to the shared address fd20:80:ff01::1:0:0. The dfw guest in the sender's VPC received 5, and the dfw guest in the other VPC received 0, in both directions.
Captures at every listed point, pinned versions recorded Each scenario writes one pcap per point and interface: public RX and TX, both veth peers, and the remote underlay. All 11 pin checks matched: containerlab 0.75.0, kind v0.33.0, task 3.51.1, Kind node v1.37.0 in the Dockerfile and on the API server, FRR 10.7.1 in the Dockerfile and on the transit router, remote-host base alpine:3.24, and the guest image digest in both manifests and on the running pods. The host and node kernel (7.3.3), Docker 29.9.3, the Galactic revision and every lab image ID are recorded beside them.
The lab's existing task verify still passes Passed with the fixture deployed, all 36 verify tasks (rc 0), and verify:ns80 is now part of it.
PR target and closing reference Targets integration/galactic-worker and says Fixes datum-cloud/galactic#1432.

Local checks on this commit: task lint (0 golangci-lint issues in all three modules, yamlfmt clean), task build, task test:unit (152 packages ok) and task test:e2e (ok) all pass.

Limits the baseline records

These are recorded as observations without an expectation, or as per-run values, and docs/baseline.md explains each one.

  • Inbound packets skip the host veth. usid_ingress delivers to a veth attachment with bpf_redirect_peer, straight into the guest's side of the pair, so tcpdump on the host side sees only outbound packets. The guest's own eth0 is the evidence for inbound delivery and for the isolation control.
  • Edge nodes forward in XDP. tcpdump on the edge nodes cannot see NAT or gateway traffic (A tenant's TCP session through an egress shard dies after the handshake #565), so the public side is captured on the transit router's end of each edge uplink.
  • The gateway checks intermittently lose a TCP flow. With no captures running, verify-gateway-ingress-ipv4.sh --node dfw-worker2 passed twice and answered 6 of 8 TCP flows once in three back-to-back runs. The captures show the lost flow's request reaching the backend and being acknowledged. The ns60 responder (socat ... fork SYSTEM:) then closes without writing a reply, 1 to 2 ms after the client's FIN, and the close returns over the direct-server-return path. UDP never lost a flow. In the final runs, run 1 answered 6 of 8 TCP flows over IPv6 and 7 of 8 over IPv4, and run 2 answered 8 of 8 over both. So the exact counts and the check's exit status are per-run values, and the stable result is that each protocol had answered flows. This fault predates this change and is not fixed here.

Why the TAP stand-in was dropped (for #1444)

The fixture first included a TAP-backed guest. A pod's sandbox ran galactic-tap, and a host-network helper bridged the host tap to a tap inside its own network namespace with socat. In Kind it failed.

  1. galactic-tap ADD created the host tap G000000080082H and enslaved it to the VPC's VRF G000000080V. The tap stayed NO-CARRIER because nothing ever opened it.
  2. containerd then failed every sandbox for the TAP guest pod in RunPodSandbox with failed to setup network for sandbox ...: failed to find network info for sandbox .... containerd found no interface inside the sandbox in the CNI result. galactic-tap deliberately configures nothing inside the pod, so the result has none, and Kubernetes cannot start a pod whose only network is galactic-tap.
  3. The bridge helper failed as a consequence. Its readiness probe could not open the tapguest network namespace, and its container kept exiting.

The bridge itself works. In a throwaway privileged container, socat 1.8.1.3 (TUN,tun-type=tap,iff-no-pi on both sides, with netns= for the guest side) carried ICMPv6 between a persistent host tap and a tap in a network namespace, and it reattached after a restart. Two details matter for #1444. The guest's gateway lies outside its /96, so the guest needs an on-link host route to it. The guest tap has no carrier until socat holds it, so its routes must be added after the bridge attaches. The tenant TAP capture point now belongs to #1444.

The TAP stand-in moved to #1444, so
baseline-versions.sh no longer has a dfw/tap-guest.yaml to read and
logged a sed error on every run.
@privateip
privateip marked this pull request as ready for review November 19, 2026 03:43
@privateip
privateip requested a review from a team as a code owner November 19, 2026 03:43
@privateip
privateip merged commit d967c51 into integration/galactic-worker Nov 19, 2026
29 checks passed
@privateip
privateip deleted the feat/issue-1432 branch November 19, 2026 03:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant