Skip to content

HIL: Ctrl-C terminates the board worker before its lock task logs the failure #4500

Open

Description

@hathach

A Ctrl-C on test/hil/hil_test.py leaves the board's lock record stale instead of ending its task as failed. The
run also writes no report.

Evidence

  • ci.lan, raspberry_pi_pico, Take HIL board locks through hil_lock.board_task #4499's branch with agentrc 48d46bbf. SIGINT went to the run's process group after 4
    tests:
    • The run exited 130 with no report.
    • The lock record stayed stale (cleanupOwed: true, the board worker's pid), and no failed event was logged.
    • All 14 idle pool workers printed KeyboardInterrupt tracebacks. The board worker printed nothing.
  • The next run on the board reported it as previous-cleanup-failure (found-left). Its park logged restored for
    it, so recovery works; only the interrupted run's own record of the failure is lost.

Cause

This is read from the source and is not yet confirmed by a trace:

  • Main's KeyboardInterrupt in drain_pool is not caught by except Exception in main().
  • Leaving with pool: therefore calls Pool.terminate(), which sends SIGTERM to every worker.
  • The board worker has no SIGTERM handler, so it dies before its board_task can end the task and log the failure.
  • _abort_report is skipped for the same reason, which is why there is no report.

Remaining work

  • On KeyboardInterrupt in main(), let in-flight workers end their board task (bounded join, or close() +
    join(timeout)) before terminate(). Write the abort report as the other abort paths do.
  • Confirm on hardware that a Ctrl-C then logs a failed event with the run's receipts and leaves no stale record.

Deferred: the pool shutdown predates #4499 and the recovery path covers it.

Activity

  1. hathach commented on Nov 12, 2026

    @hathach
    OwnerAuthor

    Still reproduces with the verdict collector (#4499 at 9a92941, agentrc ac8a8f65), ci.lan, raspberry_pi_pico:

    • SIGINT to the run's process group after 4 tests: exit 130, record left cleanupOwed with the worker's pid, no failed event.
    • The next run logged it found-left (3acbc91cab1a24b6, "the task ended without its cleanup"), parked, logged restored, exit 0.

    The collector does not change this: the worker is terminated before its task's __exit__ runs, so neither the body nor the collector gets to give a verdict.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Prio 📌follow-upDeferred work from a PR or investigation

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions