Skip to content

ns60 lab gateway backends sometimes close TCP flows without replying #1452

Closed

Description

@privateip

Parent: Deliver Galactic Worker with Local and Fabric delivery for tenant services

Observed

The ns60 gateway backends in the containerlab lab sometimes close a TCP connection without replying. This happens on main and on integration/galactic-worker.

  • Where: deploy/containerlab/resources/tenants/ns60/base/backend.yaml serves port 80 with socat TCP6-LISTEN:80,reuseaddr,fork SYSTEM:"$reply" (lines 77 and 130).
  • What the captures show: while baselining for Build a reproducible baseline for dedicated placement #1432, the responder sometimes closed the connection 1 to 2 ms after the client's FIN without writing the reply.
  • How often: with no captures running, verify-gateway-ingress-ipv4.sh answered 6 of 8 TCP flows in one of three back-to-back runs. UDP never lost a flow.

This makes gateway TCP checks unreliable. #1432 had to accept "answered flows in each protocol" instead of a per-flow count, and #1438 and #1442 need exact gateway delivery evidence.

Expected

Every TCP flow to an ns60 backend gets its reply. The gateway ingress checks can then require every flow to be answered.

Acceptance

  • The responder writes its reply before it can be torn down by the client's half-close. For example, use socat's shut-none or ignoreeof on the SYSTEM side, or swap in a responder that replies on accept. The PR states the root cause from captures.
  • Under the containerlab manager, verify:gateway-ingress and verify:gateway-ingress-ipv4 each run 10 times in a row. Every TCP and UDP flow is answered every time, and the PR includes the counts.
  • If those scripts accept fewer than all flows today, they are tightened to require every flow.
  • The delivering PR targets integration/galactic-worker, carries this evidence and says Fixes datum-cloud/galactic#1452. GitHub does not auto-close from a non-default base, so close this issue explicitly when the PR merges, and record the PR and merged SHA.

Activity

  1. added theissue type on Nov 19, 2026
  2. privateip commented on Nov 19, 2026

    @privateip
    CollaboratorAuthor

    Verdict: valid, with a different cause

    I reproduced the dropped replies outside the lab, using the backend image and the exact responder and client commands from backend.yaml and the verify-gateway-ingress*.sh scripts (socat 1.8.1.3, loopback). The issue's diagnosis needs one correction: the client's half-close is not the cause, so socat's shut-none and ignoreeof options do not help.

    Root cause

    SYSTEM:"$reply" runs echo, which writes the reply and exits without reading its input. socat still forwards the client's request (probe\n) into the command's socket. By then echo has usually exited, so the write fails with EPIPE, or the next read fails with ECONNRESET. When that error comes before socat has relayed echo's output, socat exits and closes the flow without sending anything. The client sends its request and FIN back to back, which is why the captures show the close 1 to 2 ms after the FIN. The server's log shows the errors:

    E write(5, ..., 6): Broken pipe
    E read(5, ..., 8192): Connection reset by peer
    

    UDP has the same race, less often: 2 of 300 and 3 of 600 datagrams in the runs below. The lab's few gateway runs had not hit it yet.

    Reproduction

    Responder Protocol Flows No reply
    SYSTEM:"$reply" (today) TCP 500 34
    SYSTEM:"$reply",shut-none TCP 500 36
    SYSTEM:"$reply",ignoreeof TCP 500 40
    SYSTEM:"$reply" (today) UDP 300 2
    SYSTEM:"$reply"'!!'/dev/null (the fix) TCP 500 0
    SYSTEM:"$reply"'!!'/dev/null (the fix) UDP 600 0

    Two UDP alternatives fail outright. A reply command that reads its input first blocks on the client's empty end-of-input datagram, and that child then takes every later flow's datagram. socat -U never reads the datagram, so it is delivered again and again.

    Plan (approved)

    • Responder: use socat's dual address SYSTEM:"$reply"'!!'/dev/null for both the TCP and UDP listeners in both ns60 Deployments, with a comment saying why. socat reads the reply from the command and writes the client's bytes to /dev/null, so nothing is written into an exited command. The reply text is unchanged. Locally, the gateway health probe pattern (connect, then close with linger 0) leaves no stray processes.
    • Verify scripts: no change. verify-gateway-ingress.sh and verify-gateway-ingress-ipv4.sh already fail on any unanswered flow: each one calls fail, and the script exits 1. The run that answered 6 of 8 TCP flows was a failing run.
    • Lab evidence: reapply ns60 only, then run verify:gateway-ingress and verify:gateway-ingress-ipv4 ten times each under the lab manager, recording per-run TCP and UDP counts. Then check the handoff baseline (task baseline:twice and the tenant checks).
    • Delivery: one PR to integration/galactic-worker with Fixes datum-cloud/galactic#1452. I'll close this issue with the PR and merged SHA when it merges.
  3. self-assigned this
    on Nov 19, 2026
  4. privateip commented on Nov 19, 2026

    @privateip
    CollaboratorAuthor

    Fixed by #1455, merged into integration/galactic-worker as 9517d64. GitHub does not auto-close issues from a non-default base, so I'm closing this by hand.

    The ns60 backends now answer through socat's dual address SYSTEM:"$reply"'!!'/dev/null, so the client's request is never written into an exited reply command. The backend-health check now restarts the same responder. On the lab, verify:gateway-ingress and verify:gateway-ingress-ipv4 each passed 10 runs in a row. Every flow was answered: 320/320 TCP and 320/320 UDP per check, with no no reply. The handoff baseline passed afterwards. The counts are in #1455 (comment).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions