You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
verify-fabric-api.sh leaks kubectl port-forwards that hold the lab lock #1449
deploy/containerlab/scripts/verify-fabric-api.sh leaves one kubectl port-forward svc/fabric-api-gateway running per site after it exits. It does this on main and on integration/galactic-worker.
Line 255 runs k -n galactic-system port-forward ... &, where k is a shell function (line 70). Backgrounding a function starts a subshell, so PF_PID=$! holds the subshell's PID, not kubectl's. Both cleanup paths, kill "${PF_PID}" at line 273 and the EXIT trap at line 38, kill only the subshell. kubectl is reparented to init and keeps running.
Under the containerlab manager, the orphaned kubectl processes inherit the manager's lock descriptor. After a task verify run, three of them (one per site) held the shared lab lock for almost five hours, and every later managed run got "busy" until someone killed them by hand.
Expected
verify:fabric-api leaves no process behind, whether it passes, fails or is interrupted.
Acceptance
The port-forward runs as a direct child whose PID the script records, for example kubectl --kubeconfig "${kc}" ... port-forward ... & instead of the k function, or the script kills its whole process group. Cleanup waits for it to exit.
Under the containerlab manager, after task verify:fabric-api passes, after a forced failure and after an interrupt, pgrep -af 'port-forward svc/fabric-api-gateway' finds nothing and lab-manager status reports Available. The PR records each case.
Every other script under deploy/containerlab/scripts that backgrounds a shell function is checked and fixed the same way, or the PR says there are none.
The delivering PR targets integration/galactic-worker, carries this evidence and says Fixes datum-cloud/galactic#1449 6481250847 I_kwDORgXqtM8AAAABglAKHw. GitHub does not auto-close from a non-default base, so close this issue explicitly when the PR merges, and record the PR and merged SHA. main has the same bug and gets this fix when the integration branch merges, or through a separate backport.
Confirmed on integration/galactic-worker. Line 255 of deploy/containerlab/scripts/verify-fabric-api.sh backgrounds the k shell function, so $! is the PID of a subshell, and both cleanup paths kill only that subshell. The kubectl port-forward it started keeps running.
A sweep of deploy/containerlab/scripts and the lab Taskfile found one more script with the same pattern. verify-nat-active-active.sh backgrounds the pod_exec function in soak_start, and its cleanup never stops that job. A failure between soak_start and soak_check therefore leaves a docker exec running for up to the download's 300-second limit. Every other background job in those files is a direct docker exec or timeout that the script waits for or kills.
Plan
verify-fabric-api.sh
Start the port-forward by calling kubectl --kubeconfig ... directly, so PF_PID is kubectl's own PID.
Add a stop_pf helper that kills the port-forward, waits for it to exit and clears PF_PID. Both the in-loop cleanup and the EXIT trap call it.
Trap INT and TERM and turn them into an exit, so an interrupt always runs the cleanup.
verify-nat-active-active.sh: run the soak download's docker exec directly. Kill it and wait for it in cleanup, and clear SOAK_PID in soak_check.
No change to the containerlab manager.
Verification
All three cases run under one lab reservation: task verify:fabric-api passing, a forced failure while the port-forward is running, and an interrupt (SIGINT to the process group). After each case, pgrep -af 'port-forward svc/fabric-api-gateway' must find nothing. After the run, lab-manager status must report Available. The unfixed revision is run first to show the leak, and those leaked processes are cleaned up in the same reservation. verify-nat-active-active.sh gets a bash -n syntax check only, not a lab run.
Fixed by #1450, merged into integration/galactic-worker as aac8a32.
verify-fabric-api.sh now starts its port-forward directly and stops it on pass, failure and interrupt. On the lab, all three cases left no port-forward behind, and the lab manager was free after the run. #1450 has the exit codes and raw output. verify-nat-active-active.sh had the same leak in its background download and is fixed in #1450 too.
main still has the bug. It gets the fix when integration/galactic-worker merges into main.
Activity
privateip commentedon Nov 19, 2026
Confirmed on
integration/galactic-worker. Line 255 ofdeploy/containerlab/scripts/verify-fabric-api.shbackgrounds thekshell function, so$!is the PID of a subshell, and both cleanup paths kill only that subshell. Thekubectl port-forwardit started keeps running.A sweep of
deploy/containerlab/scriptsand the lab Taskfile found one more script with the same pattern.verify-nat-active-active.shbackgrounds thepod_execfunction insoak_start, and itscleanupnever stops that job. A failure betweensoak_startandsoak_checktherefore leaves adocker execrunning for up to the download's 300-second limit. Every other background job in those files is a directdocker execortimeoutthat the script waits for or kills.Plan
verify-fabric-api.shkubectl --kubeconfig ...directly, soPF_PIDis kubectl's own PID.stop_pfhelper that kills the port-forward, waits for it to exit and clearsPF_PID. Both the in-loop cleanup and theEXITtrap call it.INTandTERMand turn them into anexit, so an interrupt always runs the cleanup.verify-nat-active-active.sh: run the soak download'sdocker execdirectly. Kill it and wait for it incleanup, and clearSOAK_PIDinsoak_check.Verification
All three cases run under one lab reservation:
task verify:fabric-apipassing, a forced failure while the port-forward is running, and an interrupt (SIGINT to the process group). After each case,pgrep -af 'port-forward svc/fabric-api-gateway'must find nothing. After the run,lab-manager statusmust reportAvailable. The unfixed revision is run first to show the leak, and those leaked processes are cleaned up in the same reservation.verify-nat-active-active.shgets abash -nsyntax check only, not a lab run.privateip commentedon Nov 19, 2026
Fixed by #1450, merged into
integration/galactic-workeras aac8a32.verify-fabric-api.shnow starts its port-forward directly and stops it on pass, failure and interrupt. On the lab, all three cases left no port-forward behind, and the lab manager was free after the run. #1450 has the exit codes and raw output.verify-nat-active-active.shhad the same leak in its background download and is fixed in #1450 too.mainstill has the bug. It gets the fix whenintegration/galactic-workermerges intomain.