Skip to content

verify-fabric-api.sh leaks kubectl port-forwards that hold the lab lock #1449

Closed

Description

@privateip

Parent: Deliver Galactic Worker with Local and Fabric delivery for tenant services

Observed

deploy/containerlab/scripts/verify-fabric-api.sh leaves one kubectl port-forward svc/fabric-api-gateway running per site after it exits. It does this on main and on integration/galactic-worker.

Line 255 runs k -n galactic-system port-forward ... &, where k is a shell function (line 70). Backgrounding a function starts a subshell, so PF_PID=$! holds the subshell's PID, not kubectl's. Both cleanup paths, kill "${PF_PID}" at line 273 and the EXIT trap at line 38, kill only the subshell. kubectl is reparented to init and keeps running.

Under the containerlab manager, the orphaned kubectl processes inherit the manager's lock descriptor. After a task verify run, three of them (one per site) held the shared lab lock for almost five hours, and every later managed run got "busy" until someone killed them by hand.

Expected

verify:fabric-api leaves no process behind, whether it passes, fails or is interrupted.

Acceptance

  • The port-forward runs as a direct child whose PID the script records, for example kubectl --kubeconfig "${kc}" ... port-forward ... & instead of the k function, or the script kills its whole process group. Cleanup waits for it to exit.
  • Under the containerlab manager, after task verify:fabric-api passes, after a forced failure and after an interrupt, pgrep -af 'port-forward svc/fabric-api-gateway' finds nothing and lab-manager status reports Available. The PR records each case.
  • Every other script under deploy/containerlab/scripts that backgrounds a shell function is checked and fixed the same way, or the PR says there are none.
  • The delivering PR targets integration/galactic-worker, carries this evidence and says Fixes datum-cloud/galactic#1449 6481250847 I_kwDORgXqtM8AAAABglAKHw. GitHub does not auto-close from a non-default base, so close this issue explicitly when the PR merges, and record the PR and merged SHA. main has the same bug and gets this fix when the integration branch merges, or through a separate backport.

Activity

  1. added theissue type on Nov 19, 2026
  2. privateip commented on Nov 19, 2026

    @privateip
    CollaboratorAuthor

    Confirmed on integration/galactic-worker. Line 255 of deploy/containerlab/scripts/verify-fabric-api.sh backgrounds the k shell function, so $! is the PID of a subshell, and both cleanup paths kill only that subshell. The kubectl port-forward it started keeps running.

    A sweep of deploy/containerlab/scripts and the lab Taskfile found one more script with the same pattern. verify-nat-active-active.sh backgrounds the pod_exec function in soak_start, and its cleanup never stops that job. A failure between soak_start and soak_check therefore leaves a docker exec running for up to the download's 300-second limit. Every other background job in those files is a direct docker exec or timeout that the script waits for or kills.

    Plan

    • verify-fabric-api.sh
      • Start the port-forward by calling kubectl --kubeconfig ... directly, so PF_PID is kubectl's own PID.
      • Add a stop_pf helper that kills the port-forward, waits for it to exit and clears PF_PID. Both the in-loop cleanup and the EXIT trap call it.
      • Trap INT and TERM and turn them into an exit, so an interrupt always runs the cleanup.
    • verify-nat-active-active.sh: run the soak download's docker exec directly. Kill it and wait for it in cleanup, and clear SOAK_PID in soak_check.
    • No change to the containerlab manager.

    Verification

    All three cases run under one lab reservation: task verify:fabric-api passing, a forced failure while the port-forward is running, and an interrupt (SIGINT to the process group). After each case, pgrep -af 'port-forward svc/fabric-api-gateway' must find nothing. After the run, lab-manager status must report Available. The unfixed revision is run first to show the leak, and those leaked processes are cleaned up in the same reservation. verify-nat-active-active.sh gets a bash -n syntax check only, not a lab run.

  3. self-assigned this
    on Nov 19, 2026
  4. privateip commented on Nov 19, 2026

    @privateip
    CollaboratorAuthor

    Fixed by #1450, merged into integration/galactic-worker as aac8a32.

    verify-fabric-api.sh now starts its port-forward directly and stops it on pass, failure and interrupt. On the lab, all three cases left no port-forward behind, and the lab manager was free after the run. #1450 has the exit codes and raw output. verify-nat-active-active.sh had the same leak in its background download and is fixed in #1450 too.

    main still has the bug. It gets the fix when integration/galactic-worker merges into main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions