Skip to content

fix(containerlab): Stop lab VIP backends closing flows without a reply - #1455

Merged
privateip merged 3 commits into
integration/galactic-workerfrom
fix/issue-1452
Nov 19, 2026
Merged

privateip merged 3 commits into
integration/galactic-workerfrom
fix/issue-1452

Conversation

@privateip

@privateip privateip commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The lab's VIP backends sometimes closed a connection without answering, so the gateway ingress checks failed on flows the gateway had delivered. Each backend handed the client's request to a reply command that had usually exited already, and when that hand-off failed first, the backend closed the flow without sending the reply. Backends now discard the request and always send the reply. The baseline guide no longer describes the loss as current.

Test plan

  • Local reproduction loses 34 of 500 TCP and 5 of 900 UDP flows before the fix, and none of 1,100 after
  • A strict full lab verification answers every gateway flow, TCP and UDP, through all four gateway nodes
  • Both gateway ingress checks pass 10 runs in a row with every flow answered, then the lab baseline passes
  • Lint and manifest tests pass

Fixes #1452

The ns60 backends answered each TCP connection and UDP datagram with
socat SYSTEM:"echo ...". socat writes the client's request into the
command's socket, but echo never reads it and has usually exited by
then. The write fails with EPIPE, or the next read with ECONNRESET, and
when that happens before socat has relayed echo's output, socat exits
and closes the flow without replying. The client's half-close is not
the cause, so shut-none and ignoreeof do not help.

Use the dual address SYSTEM:"echo ..."!!/dev/null on both listeners:
socat reads the reply from the command and writes the client's bytes to
/dev/null, so nothing is ever written into an exited command. The reply
text is unchanged.

Fixes #1452
docs/baseline.md and baseline-observe.sh described the ns60 responder's
lost flows as current behavior. Say what the fault was and that the
responder no longer loses flows. The baseline still records only whether
each protocol had answered flows, and its reference is unchanged.
@privateip privateip self-assigned this Nov 19, 2026
@privateip

Copy link
Copy Markdown
Collaborator Author

Root cause and evidence

The ns60 lab backends sometimes closed a TCP connection, and less often dropped a UDP datagram, without replying. This made the gateway ingress checks fail on flows the gateway had delivered correctly.

Each backend answered with socat ... fork SYSTEM:"echo ...". socat writes the client's request into the command's socket, but echo never reads it and has usually exited by then. That write fails with EPIPE, or the next read with ECONNRESET. When the error comes before socat has relayed echo's output, socat exits and closes the flow without a reply. The #1432 captures show this as a close 1 to 2 ms after the client's FIN, because the client sends its request and FIN back to back. The half-close itself is not the cause, so shut-none and ignoreeof do not help.

Both listeners now use socat's dual address SYSTEM:"$reply"'!!'/dev/null. socat reads the reply from the command and writes the client's bytes to /dev/null, so nothing is ever written into an exited command. The reply text is unchanged. A second commit corrects docs/baseline.md and a comment in baseline-observe.sh, which described the loss as current.

Reproduction

Outside the lab, in the backend image (socat 1.8.1.3), with the exact responder and the verify scripts' client command over loopback:

Responder Protocol Flows No reply
SYSTEM:"$reply" (before) TCP 500 34
SYSTEM:"$reply",shut-none TCP 500 36
SYSTEM:"$reply",ignoreeof TCP 500 40
SYSTEM:"$reply" (before) UDP 300 2
SYSTEM:"$reply"'!!'/dev/null (after) TCP 500 0
SYSTEM:"$reply"'!!'/dev/null (after) UDP 600 0

The committed script, extracted from the manifest and run as-is, answered 300 of 300 TCP and 60 of 60 UDP flows. 100 connections closed with a reset, which is how the gateway's health probe closes, plus 50 plain connects, left no stray processes.

Lab evidence

Both runs were under the lab manager, on the running lab with only ns60 reapplied from this branch.

  1. Strict full verification: task deploy:ns60 from dc7a13f5, then task verify:ns80 and task verify with no tolerance. Every gateway flow was answered: TCP and UDP 8/8 through each of the four gateway nodes, in both verify:gateway-ingress and verify:gateway-ingress-ipv4 (64/64 per protocol). Before the fix, the same task verify on the same lab lost 3 TCP flows (no reply) across dfw and sjc, while UDP answered 8/8.
  2. 10x loop: pending. verify:gateway-ingress and verify:gateway-ingress-ipv4, 10 runs each, then the handoff baseline.

Checks

verify-gateway-ingress.sh and verify-gateway-ingress-ipv4.sh already fail on any unanswered flow: each one calls fail, and the script exits 1. They are unchanged.

The dedicated-placement baseline (baseline-observe.sh) still records only whether each protocol had answered flows through the gateway, with exact counts kept as volatile. With this fix in, that bar can be raised to every flow answered. This PR leaves it unchanged, so the committed reference stays valid.

verify-gateway-backend-health.sh stops backend-a's TCP listener to make
it unhealthy, then starts it again with its own copy of the responder
command. That copy still used the plain SYSTEM address, so after the
check backend-a went back to losing replies. Use the same '!!'/dev/null
form as backend.yaml.
@privateip

Copy link
Copy Markdown
Collaborator Author

Lab evidence: 10 runs of each gateway ingress check

One managed run on the shared lab at 313e41d2, with only ns60 changed: task deploy:ns60, then a restart of every ns60 backend. Each backend's live socat processes were checked, and both the TCP and UDP listeners carry !!/dev/null in all six pods. The restart matters because the backend-health check, run by task verify from branches without 313e41d2, restarts backend-a's TCP listener inside the pod with the old command.

Check Runs passed TCP answered UDP answered no reply
verify:gateway-ingress 10/10 320/320 320/320 0
verify:gateway-ingress-ipv4 10/10 320/320 320/320 0

Each run sends 8 flows per protocol through each of the 4 gateway nodes. Every run exited 0 with no FAIL line, including the IPv4 check's same-flow probes and client-side capture.

Handoff after the loop: baseline:twice gave two identical runs matching the committed reference, and verify:ns10, ns20, ns30, ns40 and ns80 passed.

The earlier strict recovery pass, a full task verify with this responder, also answered every gateway flow.

@privateip
privateip marked this pull request as ready for review November 19, 2026 05:35
@privateip
privateip requested a review from a team as a code owner November 19, 2026 05:35
@privateip
privateip merged commit 9517d64 into integration/galactic-worker Nov 19, 2026
28 checks passed
@privateip
privateip deleted the fix/issue-1452 branch November 19, 2026 05:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant