Skip to content

fix(watch): exit when the observer emitter thread dies silently - #879

Open
yzxcj797 wants to merge 1 commit into
tirth8205:mainfrom
yzxcj797:fix/811-watch-observer-death
Open

fix(watch): exit when the observer emitter thread dies silently#879
yzxcj797 wants to merge 1 commit into
tirth8205:mainfrom
yzxcj797:fix/811-watch-observer-death

Conversation

@yzxcj797

Copy link
Copy Markdown
Contributor

Summary

A watchdog emitter reader thread can crash under heavy create/delete churn — the inotify _wd_for_path KeyError when Maven Surefire forks churn target/surefire/, or the Windows ReadDirectoryChangesW variant (both documented in #811) — while the observer thread itself stays alive. The child process keeps passing the daemon's proc.poll()-based health check while delivering no filesystem events: daemon status stays green, built_at_sha freezes, and every subsequent edit/git pull is missed until a manual restart.

The watch loop now checks _observer_threads_alive(observer) — the observer thread and every emitter thread — once per second. On the first dead thread it logs critical and raises SystemExit(1), exiting the child so the daemon's existing restart path recovers watch instead of silently serving a stale graph forever.

This deliberately implements the issue's fix option 2 (exit so poll-based health checks catch it) rather than option 1 (per-subdirectory scheduling): it covers both the Linux and Windows thread variants, and the reproducer requires churn that a Windows-CI test cannot drive. A follow-up can layer schedule-time ignore filtering on top without conflicting.

Testing

Verification boundary: the end-to-end kill (driving a real inotify race under Maven churn) requires the Linux/WSL environment from the report; the unit tests pin the liveness predicate and the exit decision it feeds.

Fixes #811

@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown

code-review-graph review

Overall risk: 0.35 (LOW) — 11 changed function(s)/class(es), 0 affected flow(s), 4 test gap(s)

Risk-scored changes

Risk Level Symbol Location Tested
0.35 low tests/test_incremental.py::TestWatchObserverLiveness._Emitter tests/test_incremental.py:1823 no
0.35 low tests/test_incremental.py::TestWatchObserverLiveness._Shim tests/test_incremental.py:1830 no
0.30 low tests/test_incremental.py::_Emitter.__init__ tests/test_incremental.py:1824 no
0.30 low tests/test_incremental.py::_Emitter.is_alive tests/test_incremental.py:1827 yes
0.25 low code_review_graph/incremental.py::_observer_threads_alive code_review_graph/incremental.py:1540 yes
0.25 low tests/test_incremental.py::TestWatchObserverLiveness._fake_observer tests/test_incremental.py:1822 yes
0.15 low code_review_graph/incremental.py::watch code_review_graph/incremental.py:1555 yes
0.15 low tests/test_incremental.py::TestWatchObserverLiveness tests/test_incremental.py:1818 no
0.15 low tests/test_incremental.py::TestWatchObserverLiveness.test_alive_observer_with_alive_emitter_is_healthy tests/test_incremental.py:1836 (test)
0.15 low tests/test_incremental.py::TestWatchObserverLiveness.test_dead_observer_is_not_healthy tests/test_incremental.py:1841 (test)

Test gaps

  • tests/test_incremental.py::TestWatchObserverLiveness (tests/test_incremental.py:1818)
  • tests/test_incremental.py::TestWatchObserverLiveness._Emitter (tests/test_incremental.py:1823)
  • tests/test_incremental.py::_Emitter.__init__ (tests/test_incremental.py:1824)
  • tests/test_incremental.py::TestWatchObserverLiveness._Shim (tests/test_incremental.py:1830)

Token savings: this graph-backed report used ~31,387 fewer tokens (~93%) than reading every changed file in full (estimated, chars/4 approximation).


Powered by code-review-graph — local-first analysis; no code leaves the CI runner.

A watchdog emitter reader thread can crash under heavy create/delete churn (the inotify _wd_for_path KeyError on Maven target/, or the ReadDirectoryChangesW equivalent on Windows) while the observer thread itself stays alive. The child process then keeps passing the daemon's poll-based health check while delivering no filesystem events, so the graph silently freezes (tirth8205#811).

The watch loop now checks the observer and every emitter thread each second. On death it logs critical and exits the child with SystemExit(1), letting the daemon's existing restart path recover instead of serving a stale built_at_sha indefinitely.
@yzxcj797
yzxcj797 force-pushed the fix/811-watch-observer-death branch from fc4ba2e to 22ec5c3 Compare August 19, 2026 06:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

watch: observer thread silently crashes on Maven target/ dir churn (KeyError in watchdog inotify) — daemon looks alive but stops updating

1 participant