Skip to content

fix(containerlab): Stop lab checks leaking port-forwards that hold the lab - #1450

Merged
privateip merged 1 commit into
integration/galactic-workerfrom
fix/issue-1449
Nov 19, 2026
Merged

privateip merged 1 commit into
integration/galactic-workerfrom
fix/issue-1449

Conversation

@privateip

Copy link
Copy Markdown
Collaborator

The fabric API lab check left one kubectl port-forward running per site after it finished. Under the lab manager those processes inherited the reservation lock, so the shared lab stayed locked until someone killed them by hand. This PR makes the check stop its port-forward whether it passes, fails or is interrupted. It also fixes the NAT active/active lab check, which leaked its background download in the same way.

The cause was the same in both scripts. Each started its background job through a shell function. Bash runs a backgrounded function in a subshell, so the script recorded the subshell's process ID, and cleanup killed only the subshell. Both scripts now start the command directly, kill it and wait for it in cleanup, and the fabric API check turns Ctrl-C and SIGTERM into a normal exit so that cleanup always runs. No other lab script backgrounds a function.

Results on the shared three-site lab, all under one reservation:

Case Revision Exit code Port-forwards left
Pass before this PR 0 3, one per site
Pass this PR 0 0
Forced failure while the port-forward runs this PR 1 0
Ctrl-C while the port-forward runs this PR 130 0

The exit codes are the check script's own; the task runner wrapped both failures as 201. The first comment has the commands and raw output.

After the run released the lab, no port-forward remained and the lab matched its starting snapshot. The manager showed the lab busy, not available, because a queued run from another issue took the lock the same second. None of the processes holding the lock came from this run.

Not covered:

  • The NAT check had only a syntax check, not a lab run.
  • If a check dies during a packet capture, the capture can outlive it by up to its 8 to 30 second timeout. That leak has a time limit, unlike the one verify-fabric-api.sh leaks kubectl port-forwards that hold the lab lock #1449 reported.
  • The full CI suite was not run locally, because its end-to-end test would create Kind clusters next to the shared lab. CI on this PR covers it.

Lint passed locally with 0 issues. The same bug is on main and reaches it when the integration branch merges.

Fixes #1449

verify-fabric-api.sh started its gateway port-forward by backgrounding
the k shell function. Bash runs a backgrounded function in a subshell,
so $! named the subshell and both cleanup paths killed only it, leaving
kubectl port-forward running after the script exited. Under the lab
manager those orphans inherited the reservation lock and held it until
someone killed them.

Start kubectl directly so PF_PID is its own PID, stop it through a
stop_pf helper that kills and waits, and turn INT and TERM into an exit
so an interrupt still runs the EXIT cleanup.

verify-nat-active-active.sh had the same pattern: soak_start backgrounded
the pod_exec function, and cleanup never stopped the download. Run its
docker exec directly, kill and wait for it in cleanup, and clear
SOAK_PID once soak_check has waited for it.

Fixes #1449
@privateip privateip self-assigned this Nov 19, 2026
@privateip

Copy link
Copy Markdown
Collaborator Author

Commands and raw output behind the results table. Evidence wrapper: one lab-manager run reservation. Each case runs task verify:fabric-api from deploy/containerlab, then pgrep -af 'port-forward svc/fabric-api-gateway' two seconds later. PIDs, ports and temp paths are replaced with placeholders.

  1. Base revision, pass: task exit 0. pgrep found:
    <pid> kubectl --kubeconfig <tmp>/dfw.kubeconfig -n galactic-system port-forward svc/fabric-api-gateway <port>:9346
    <pid> kubectl --kubeconfig <tmp>/iad.kubeconfig -n galactic-system port-forward svc/fabric-api-gateway <port>:9346
    <pid> kubectl --kubeconfig <tmp>/sjc.kubeconfig -n galactic-system port-forward svc/fabric-api-gateway <port>:9346
    
    The wrapper then ran pkill -f, and pgrep found nothing.
  2. Fix, pass: task exit 0. pgrep found nothing.
  3. Fix, forced failure: a sleep shim on PATH made sleep 0.2 in the readiness loop exit 1, so set -e aborted the script while the port-forward was live. The script exited 1, task exited 201 (Failed to run task "verify:fabric-api": exit status 1). pgrep found nothing.
  4. Fix, interrupt: started with setsid task verify:fabric-api. Once pgrep saw the port-forward, the wrapper ran kill -INT -- -<pgid>. Task printed Signal received: "interrupt" and exit status 130, then exited 201. pgrep -g <pgid> found nothing left in the group, and the port-forward pgrep found nothing.

Handoff:

  • No verify-* FabricQuery created during the run remains on any site.
  • The ns80 namespace and pods and the clab containers match the snapshot taken before the run.
  • Checked from outside the manager after release, pgrep -af 'port-forward svc/fabric-api-gateway' exited 1 with no output.
  • lab-manager status printed Containerlab manager busy (75). The lock holder was another issue's queued lab-manager run, whose record shows it started at the same second this run logged its clean handoff. None of the processes holding fd 9 came from this run.

Static checks:

  • bash -n passed on both scripts.
  • git diff --check passed.
  • GOLANGCI_LINT_CACHE=$(mktemp -d) task lint, run from a git archive export, exited 0 with 0 issues, and yamlfmt -lint was clean.
  • task ci was not run locally.

@privateip
privateip marked this pull request as ready for review November 19, 2026 03:41
@privateip
privateip requested a review from a team as a code owner November 19, 2026 03:41
@privateip
privateip merged commit aac8a32 into integration/galactic-worker Nov 19, 2026
35 of 36 checks passed
@privateip
privateip deleted the fix/issue-1449 branch November 19, 2026 03:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant