fix(heartbeat): stop restarting the model mid-answer — busy is not down

[AVADSA25]

Chat returned "My reasoning ran on without reaching a final answer" twice in a row on long prompts. It isn't a chat bug — the heartbeat was killing the model while it was answering.

The chain, straight from the audit log

08:03:05  service_down  LLM/Vision  http://localhost:8083/v1/models
08:08:50  service_down  LLM/Vision  http://localhost:8083/v1/models
08:08:58  qwen3.6 restarted  (8 seconds later)   restart_time = 21

Those two timestamps are exactly the two failed replies (10:03 / 10:08 CEST).

Root cause

_check_service() probes the model with a 5-second HTTP timeout. mlx_vlm.server is single-threaded — an 11,216-token prefill at ~800 tok/s blocks it for ~14s, so any probe landing during generation times out. check_services_and_alert() then treated the first failure as "down" (if failures[name] == 1:) and called _try_restart("qwen3.6"), killing the in-flight request. The frontend received reasoning with no final answer and rendered the local-model-glitch notice.

Longer context → longer prefill → higher chance of self-destruction. Big prompts failed most reliably, which is why it looked random.

Nothing crashed: no traceback, no OOM, no jetsam kill, max_memory_restart unset. It was our own graceful pm2 restart, every time.

Fix

Distinguish BUSY from DEAD before restarting. New _is_listening() does a bare TCP connect — a process mid-prefill still has its socket bound; a crashed one doesn't. If the port still accepts, log it as busy, clear the failure counter, skip the restart. Genuine crashes stop listening and still recover through the existing path.

Verification

Same scenario run against both versions (model busy, port listening):

PRE-FIX   restarts issued: ['qwen3.6']     ← the bug
POST-FIX  restarts issued: NONE

• 3 new cases in tests/test_watchdog.py
• test_builtin_down_triggers_single_restart now forces _is_listening False so it keeps exercising a genuinely dead service instead of passing for the wrong reason on a box where :8083 is live
• Full suite: 2740 passed, 78 skipped; ruff clean

🤖 Generated with Claude Code