You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ns60 lab gateway backends sometimes close TCP flows without replying #1452
The ns60 gateway backends in the containerlab lab sometimes close a TCP connection without replying. This happens on main and on integration/galactic-worker.
Where:deploy/containerlab/resources/tenants/ns60/base/backend.yaml serves port 80 with socat TCP6-LISTEN:80,reuseaddr,fork SYSTEM:"$reply" (lines 77 and 130).
How often: with no captures running, verify-gateway-ingress-ipv4.sh answered 6 of 8 TCP flows in one of three back-to-back runs. UDP never lost a flow.
This makes gateway TCP checks unreliable. #1432 had to accept "answered flows in each protocol" instead of a per-flow count, and #1438 and #1442 need exact gateway delivery evidence.
Expected
Every TCP flow to an ns60 backend gets its reply. The gateway ingress checks can then require every flow to be answered.
Acceptance
The responder writes its reply before it can be torn down by the client's half-close. For example, use socat's shut-none or ignoreeof on the SYSTEM side, or swap in a responder that replies on accept. The PR states the root cause from captures.
Under the containerlab manager, verify:gateway-ingress and verify:gateway-ingress-ipv4 each run 10 times in a row. Every TCP and UDP flow is answered every time, and the PR includes the counts.
If those scripts accept fewer than all flows today, they are tightened to require every flow.
The delivering PR targets integration/galactic-worker, carries this evidence and says Fixes datum-cloud/galactic#1452. GitHub does not auto-close from a non-default base, so close this issue explicitly when the PR merges, and record the PR and merged SHA.
I reproduced the dropped replies outside the lab, using the backend image and the exact responder and client commands from backend.yaml and the verify-gateway-ingress*.sh scripts (socat 1.8.1.3, loopback). The issue's diagnosis needs one correction: the client's half-close is not the cause, so socat's shut-none and ignoreeof options do not help.
Root cause
SYSTEM:"$reply" runs echo, which writes the reply and exits without reading its input. socat still forwards the client's request (probe\n) into the command's socket. By then echo has usually exited, so the write fails with EPIPE, or the next read fails with ECONNRESET. When that error comes before socat has relayed echo's output, socat exits and closes the flow without sending anything. The client sends its request and FIN back to back, which is why the captures show the close 1 to 2 ms after the FIN. The server's log shows the errors:
E write(5, ..., 6): Broken pipe
E read(5, ..., 8192): Connection reset by peer
UDP has the same race, less often: 2 of 300 and 3 of 600 datagrams in the runs below. The lab's few gateway runs had not hit it yet.
Reproduction
Responder
Protocol
Flows
No reply
SYSTEM:"$reply" (today)
TCP
500
34
SYSTEM:"$reply",shut-none
TCP
500
36
SYSTEM:"$reply",ignoreeof
TCP
500
40
SYSTEM:"$reply" (today)
UDP
300
2
SYSTEM:"$reply"'!!'/dev/null (the fix)
TCP
500
0
SYSTEM:"$reply"'!!'/dev/null (the fix)
UDP
600
0
Two UDP alternatives fail outright. A reply command that reads its input first blocks on the client's empty end-of-input datagram, and that child then takes every later flow's datagram. socat -U never reads the datagram, so it is delivered again and again.
Plan (approved)
Responder: use socat's dual address SYSTEM:"$reply"'!!'/dev/null for both the TCP and UDP listeners in both ns60 Deployments, with a comment saying why. socat reads the reply from the command and writes the client's bytes to /dev/null, so nothing is written into an exited command. The reply text is unchanged. Locally, the gateway health probe pattern (connect, then close with linger 0) leaves no stray processes.
Verify scripts: no change. verify-gateway-ingress.sh and verify-gateway-ingress-ipv4.sh already fail on any unanswered flow: each one calls fail, and the script exits 1. The run that answered 6 of 8 TCP flows was a failing run.
Lab evidence: reapply ns60 only, then run verify:gateway-ingress and verify:gateway-ingress-ipv4 ten times each under the lab manager, recording per-run TCP and UDP counts. Then check the handoff baseline (task baseline:twice and the tenant checks).
Delivery: one PR to integration/galactic-worker with Fixes datum-cloud/galactic#1452. I'll close this issue with the PR and merged SHA when it merges.
Fixed by #1455, merged into integration/galactic-worker as 9517d64. GitHub does not auto-close issues from a non-default base, so I'm closing this by hand.
The ns60 backends now answer through socat's dual address SYSTEM:"$reply"'!!'/dev/null, so the client's request is never written into an exited reply command. The backend-health check now restarts the same responder. On the lab, verify:gateway-ingress and verify:gateway-ingress-ipv4 each passed 10 runs in a row. Every flow was answered: 320/320 TCP and 320/320 UDP per check, with no no reply. The handoff baseline passed afterwards. The counts are in #1455 (comment).
Activity
privateip commentedon Nov 19, 2026
Verdict: valid, with a different cause
I reproduced the dropped replies outside the lab, using the backend image and the exact responder and client commands from
backend.yamland theverify-gateway-ingress*.shscripts (socat 1.8.1.3, loopback). The issue's diagnosis needs one correction: the client's half-close is not the cause, so socat'sshut-noneandignoreeofoptions do not help.Root cause
SYSTEM:"$reply"runsecho, which writes the reply and exits without reading its input. socat still forwards the client's request (probe\n) into the command's socket. By thenechohas usually exited, so the write fails withEPIPE, or the next read fails withECONNRESET. When that error comes before socat has relayedecho's output, socat exits and closes the flow without sending anything. The client sends its request and FIN back to back, which is why the captures show the close 1 to 2 ms after the FIN. The server's log shows the errors:UDP has the same race, less often: 2 of 300 and 3 of 600 datagrams in the runs below. The lab's few gateway runs had not hit it yet.
Reproduction
SYSTEM:"$reply"(today)SYSTEM:"$reply",shut-noneSYSTEM:"$reply",ignoreeofSYSTEM:"$reply"(today)SYSTEM:"$reply"'!!'/dev/null(the fix)SYSTEM:"$reply"'!!'/dev/null(the fix)Two UDP alternatives fail outright. A reply command that reads its input first blocks on the client's empty end-of-input datagram, and that child then takes every later flow's datagram. socat
-Unever reads the datagram, so it is delivered again and again.Plan (approved)
SYSTEM:"$reply"'!!'/dev/nullfor both the TCP and UDP listeners in both ns60 Deployments, with a comment saying why. socat reads the reply from the command and writes the client's bytes to/dev/null, so nothing is written into an exited command. The reply text is unchanged. Locally, the gateway health probe pattern (connect, then close with linger 0) leaves no stray processes.verify-gateway-ingress.shandverify-gateway-ingress-ipv4.shalready fail on any unanswered flow: each one callsfail, and the script exits 1. The run that answered 6 of 8 TCP flows was a failing run.verify:gateway-ingressandverify:gateway-ingress-ipv4ten times each under the lab manager, recording per-run TCP and UDP counts. Then check the handoff baseline (task baseline:twiceand the tenant checks).integration/galactic-workerwithFixes datum-cloud/galactic#1452. I'll close this issue with the PR and merged SHA when it merges.privateip commentedon Nov 19, 2026
Fixed by #1455, merged into
integration/galactic-workeras 9517d64. GitHub does not auto-close issues from a non-default base, so I'm closing this by hand.The ns60 backends now answer through socat's dual address
SYSTEM:"$reply"'!!'/dev/null, so the client's request is never written into an exited reply command. The backend-health check now restarts the same responder. On the lab,verify:gateway-ingressandverify:gateway-ingress-ipv4each passed 10 runs in a row. Every flow was answered: 320/320 TCP and 320/320 UDP per check, with nono reply. The handoff baseline passed afterwards. The counts are in #1455 (comment).