[tabedzki]

🤖 AI text below 🤖

Refs tabedzki#6

Problem

The production container (ghcr.io/m-beau/pixelmap) runs for hours and then dies with no traceback. In about 14 h of logs:

• The 60 s Memory usage log fired 20–90 s late 23 times, nearly always right after Selection changed: [] -> [N]. A late log means the single Bokeh event loop was blocked that long.
• During one of those stalls the proxy dropped a user's websocket. That produced Failed sending message as connection was closed followed by 236 AssertionError: assert self._session is not None on queued PATCH-DOC messages.
• RSS sat at about 3.1–3.5 GB, with 50–150 MB swings on each click.

on_electrode_selection ran a full gc.collect() on every tap.

Changes

• Remove the per-click gc.collect().
• New pixelmap/gui/watchdog.py, which only logs and never raises or kills anything:
  • Stall watchdog: the event loop beats a heartbeat; a daemon thread logs [watchdog] event loop blocked for X s with the blocked thread's stack (PIXELMAP_STALL_WARN_S, default 5, 0 disables). It logs again with the total duration when the loop recovers.
  • Memory watch: warns once when RSS crosses PIXELMAP_RSS_WARN_MB and re-arms below 90%, so RSS hovering at the limit doesn't flood Slack.
  • Installed once per process from app_util.main(local=False); thresholds set in docker-compose.yml.
• Regression tests: tests/test_selection_responsiveness.py (no gc on tap, box select or repeated taps; toggle and empty-selection behaviour kept) and tests/test_watchdog.py (threshold edges, report-once-per-stall, recovery and re-arm, stack capture, disabled/0/negative/None configs, idempotent install).

Testing

• The 3 gc regression tests failed before the fix and pass after.
• Full suite: 557 passed.
• Smoke-tested with a real panel serve (very low thresholds): the stall warning with stack, the RSS warning and the recovery line all appear in the server log.

Related

• Production container OOM-killed: Panel leaks every closed session (+ freezes, peaks, broken image build) tabedzki/channelmap_generator#6 has the full investigation. The deaths are OOM kills at the 3500 MB limit, mainly from a Panel bug that leaks closed sessions. That leak is fixed separately in fix: release Panel views leaked by destroyed sessions; pin image Python #65.
• This PR fixes the freezes and adds the warnings. It is independent of fix: release Panel views leaked by destroyed sessions; pin image Python #65, and both merge cleanly onto main together (565 tests pass on the combination).
• Still open: update_electrode_colors rebuilds every column in Python on each click (a performance follow-up).

Assisted-by: ClaudeCode:claude-opus-5-5

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 90.65421% with 10 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@8a792fb). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
pixelmap/gui/watchdog.py	92.38%	8 Missing ⚠️
pixelmap/gui/app_util.py	0.00%	2 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #66   +/-   ##
=======================================
  Coverage        ?   67.57%           
=======================================
  Files           ?       20           
  Lines           ?     3106           
  Branches        ?        0           
=======================================
  Hits            ?     2099           
  Misses          ?     1007           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.