Skip to content

Shutdown controls fail in Docker and with helper-only sudo permissions #90

Description

@NotSamClemens

Reproduced from sparkDash 1.8.6 source at f035ca243855b3a88c270a9a84a358c2cb1fbcc3, using the repository Dockerfile on ARM64/DGX Spark. These findings were reproduced with mocked process boundaries and shell syntax checks; no machine was powered off during validation.

The documented power-controls setup says to install the helper on each Spark and authorize only that helper. The power-management block in server/index.js has three related problems:

  1. Local shutdown executes inside the container. initiateSparkShutdown() starts sudo -n /usr/local/bin/spark-shutdown directly. The helper is installed on the host, while the production Dockerfile does not install sudo in the container. In the documented container deployment, the invocation needs to enter the host mount namespace through the existing host-proc mount before invoking the host helper.
  2. Remote shutdown generates invalid shell syntax. The array entry ending in & is combined using .join("; "), producing &;. A POSIX shell rejects the command before the helper or authorization check can run.
  3. sudo -n true rejects helper-scoped authorization. A sudoers policy authorizing only /usr/local/bin/spark-shutdown need not authorize true. Check the actual helper's non-destructive authorization contract instead. Our installed helper supports --check, which prints an acknowledgement and exits without scheduling shutdown.

Safe syntax-only reproduction of the second issue, with harmless placeholder commands:

/bin/sh -n -c 'true; nohup true >/dev/null 2>&1 &; sleep 0.3; exit 0'

Observed: exit status 2, Syntax error: ";" unexpected. The -n option parses only. Replacing semicolon joining with newline joining makes the generated background command valid shell syntax.

The minimal local fix we tested preserves the existing shutdown route ordering, acknowledgement behavior and SSH error handling:

  • Build the local invocation as nsenter --mount=${HOST_PROC_PATH || "/host/proc"}/1/ns/mnt -- sudo -n /usr/local/bin/spark-shutdown.
  • Join the remote command lines with newlines.
  • Replace sudo -n true with sudo -n /usr/local/bin/spark-shutdown --check, retaining the existing failure exit status and starting the background action only after that check succeeds.

The helper-specific check requires a documented --check contract; it should not be treated as universally available on arbitrary pre-existing helpers. Our helper implementation supports this contract and was inspected as a file, not executed during these tests.

Validation against the pinned source:

  • Four new behavior tests failed against unchanged upstream code and passed with the fix.
  • The tests exercise the actual power-management block with a mocked spawn boundary; the remote command is interpreted by /bin/sh with all privileged/external commands replaced by inert shell functions and an empty executable search path. They cover default/custom host-proc paths, successful helper-only authorization and failed authorization preventing background execution.
  • Full suite: 320 server tests and 23 frontend tests passed (343 total); TypeScript typecheck passed.
  • The unchanged upstream Dockerfile successfully built the patched source using the repository's Node 22 ARM64 base.

The production change is limited to an import, two shutdown call sites, and a 22-line helper module. A separate test file adds the four regressions. No collectors, model runtimes, API authentication or deployment settings are changed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions