Repository navigation
fix(containerlab): Stop lab VIP backends closing flows without a reply - #1455
Conversation
The ns60 backends answered each TCP connection and UDP datagram with socat SYSTEM:"echo ...". socat writes the client's request into the command's socket, but echo never reads it and has usually exited by then. The write fails with EPIPE, or the next read with ECONNRESET, and when that happens before socat has relayed echo's output, socat exits and closes the flow without replying. The client's half-close is not the cause, so shut-none and ignoreeof do not help. Use the dual address SYSTEM:"echo ..."!!/dev/null on both listeners: socat reads the reply from the command and writes the client's bytes to /dev/null, so nothing is ever written into an exited command. The reply text is unchanged. Fixes #1452
docs/baseline.md and baseline-observe.sh described the ns60 responder's lost flows as current behavior. Say what the fault was and that the responder no longer loses flows. The baseline still records only whether each protocol had answered flows, and its reference is unchanged.
Root cause and evidenceThe Each backend answered with Both listeners now use socat's dual address ReproductionOutside the lab, in the backend image (socat 1.8.1.3), with the exact responder and the verify scripts' client command over loopback:
The committed script, extracted from the manifest and run as-is, answered 300 of 300 TCP and 60 of 60 UDP flows. 100 connections closed with a reset, which is how the gateway's health probe closes, plus 50 plain connects, left no stray processes. Lab evidenceBoth runs were under the lab manager, on the running lab with only
Checks
The dedicated-placement baseline ( |
verify-gateway-backend-health.sh stops backend-a's TCP listener to make it unhealthy, then starts it again with its own copy of the responder command. That copy still used the plain SYSTEM address, so after the check backend-a went back to losing replies. Use the same '!!'/dev/null form as backend.yaml.
Lab evidence: 10 runs of each gateway ingress checkOne managed run on the shared lab at
Each run sends 8 flows per protocol through each of the 4 gateway nodes. Every run exited 0 with no Handoff after the loop: The earlier strict recovery pass, a full |
Summary
The lab's VIP backends sometimes closed a connection without answering, so the gateway ingress checks failed on flows the gateway had delivered. Each backend handed the client's request to a reply command that had usually exited already, and when that hand-off failed first, the backend closed the flow without sending the reply. Backends now discard the request and always send the reply. The baseline guide no longer describes the loss as current.
Test plan
Fixes #1452