fix(jobs): job locks are renewed leases that lapse when a pass dies; upgrade stops passes before the rebuild

[joshpanka]

What

A scheduled-job pass killed without unwinding left its job lock held for a full day, and the instance silently stopped running that job. make upgrade triggers exactly this: it recreates the core container with docker compose up --build, and a pass exec'd inside it (by the up-jobs tickers, local-tick, or an operator's cron) is SIGKILLed, which skips SingleFlightCommand's release. Every later pass then logs already running; skipping until the TTL expires or someone runs clear_job_locks.

Seen on a real upgrade to v0.10.0: the poll pass that started at 14:46 UTC was killed at 15:05 when the container was replaced, clear_job_locks --all --dry-run showed feeds.poll_due_feeds HELD with no process in the container, and the host ticker skipped every pass from then on. Nothing was recorded after 14:41.

How

The job lock becomes a short renewed lease, the same pattern poll_feed already uses for its per-feed lease, with a hard ceiling:

• JOB_LOCK_LEASE_SECONDS (default 300) replaces JOB_LOCK_TIMEOUT_SECONDS (86400). It is the liveness window, not a cap on pass length, so it is named as one; apps/core/AGENTS.md now states the rule (*_LEASE_SECONDS is renewed, *_TIMEOUT_SECONDS is a fixed TTL) and names POLL_LOCK_TIMEOUT_SECONDS as the misnamed one to rename with the feed-lease migration. Leaving the old variable set fails startup with a message naming the new ones, the OLLAMA_URL rename's precedent, and so does a lease under 3s or a max hold below the lease (the renewer ticks at a third of the lease floored at 1s, so a smaller lease flaps).
• SingleFlightCommand.execute keeps the lease alive with a LeaseRenewer (new common/lease_renewer.py): a threading.Thread subclass with __enter__ / __exit__, typed against a one-method Renewable Protocol so it is unit-testable against a fake with no cache. It re-stamps the lease every third of the window for as long as the pass runs. A live pass of any length stays locked; a dead holder's renewals stop with it, so the lock lapses within one window however the pass was launched or killed.
• A lost job lease is logged and the pass continues, the opposite of the poll lease's yield; LockLease.renew's docstring and the AGENTS.md bullet now say that the caller decides, and why the two consumers differ. A renewal that raises is retried next tick on a fresh connection: the DB-backed cache raises on a dropped connection rather than returning False, and Django never reconnects a dead connection object, so the thread drops its own thread-local connections and the next tick reopens them. The lease counts as lost only once a whole window has passed unrenewed; a lost lease is logged once and the pass continues, since single-flight is a convenience, not a correctness bound.
• JOB_LOCK_MAX_HOLD_SECONDS (default 86400) is the backstop the old TTL used to provide for a pass that is alive but hung: past it the renewer stops, logs, and the lease lapses. Measured on a real instance from the job logs: poll passes up to 46 min (mean 13 min, 1613 passes), drain passes up to 4 h 02 min (mean 6 s, 95k passes). Every connector and engine call carries a timeout, so a true hang is unlikely; the cap is cheap insurance.

The one consumer that relied on single-flight for correctness gets its own fence. The digest flush gathers a window's PENDING runs with no claim; two flushes of one window would both emit (a duplicate webhook POST, two delivery rows) and both burn an attempt on a transient failure. process_due_digests now takes a per-window digest_flush_lock (DIGEST_FLUSH_LOCK_TIMEOUT_SECONDS, 600) and skips a held window, reported in its summary line. Every other SingleFlightCommand was already overlap-safe by CAS claim, unique constraint, or per-feed lock; emit_telemetry_heartbeat is a plain command that self-throttles.

Upgrade script. scripts/upgrade.sh runs stop_due_jobs before the git checkout, so it runs the old code against the old venv (the source is bind-mounted into the container, and uv run would otherwise re-sync the venv under live passes) and each in-flight pass releases its lock at once. The comment says what the step does and does not buy: a scheduler can still launch a pass into the old container during the build and that one is killed, which the lease bounds to one window and make down-jobs first avoids. No lock clearing after the rebuild, on purpose: a scheduler may already have started a fresh pass in the new container by then, and a blanket clear would free a live lock. A failed stop prints a line instead of being silent.

Docs. clear_job_locks, job_lock, the settings comments, apps/core/AGENTS.md, the README upgrade paragraph (with the down-jobs / up-jobs advice for ticker users) and the up-jobs comments describe the lease and the fence instead of the day-long TTL.

Testing

• common/tests_lease_renewer.py (6 tests, 1.3s) drives LeaseRenewer against a fake lease through its interval test seam (20ms ticks, 150ms lease): the interval rule, its three exits (stopped by the with, lease lost, max hold reached), a renewal that raises once and is retried, and one that keeps raising and counts as lost after one window. Run three times back to back for jitter; counts are asserted as ranges where a tick could move by one.
• common/tests_job_lease.py (5 tests) goes through a SingleFlightCommand on the locmem cache with a 3s lease: the lock outlives its lease while a 5s pass runs and a rival launch still skips; the renewer thread is gone after a normal return and after an exception; a holder taken through job_lock itself and abandoned without unwinding frees after one window (fails under a day-long lease, so it pins the real TTL); a lease cleared under a live pass is logged once and the pass finishes; with a 2s max hold a 6s pass loses its lock while still running.
• common/tests_job_lease_db.py (1 test, TransactionTestCase on the real DatabaseCache): terminates the renewer thread's Postgres backend mid-pass with pg_terminate_backend; the next tick logs renewal failed with the OperationalError, reconnects, and the lock is still held under the same token at the end.
• watches/tests_digest_flush_lock.py (1 test): a window whose flush lock another pass holds is skipped by process_due_digests (runs still PENDING, 1 skipped in the summary) and flushed on the next pass.
• Red/green for each guard: the raising-fake and DB tests fail when the except re-raises; the flush-lock test fails with the fence disabled; the dead-holder test fails under a day-long lease; the main renewal test fails with the renew interval neutered.
• manage.py test common watches feeds: 355 tests OK (the common suite dropped from ~39s to ~24s with the seam). JOB_LOCK_TIMEOUT_SECONDS=1, JOB_LOCK_LEASE_SECONDS=2, and JOB_LOCK_MAX_HOLD_SECONDS=100 each make manage.py check raise the named ImproperlyConfigured. The new stop_due_jobs line run live against a running stack: signaled 0 in-flight job process(es).
• shellcheck -s sh scripts/upgrade.sh, ruff, ty, and all pre-commit hooks clean.
• Not run end to end: scripts/upgrade.sh checks out the latest release tag, so it is not meaningful against a dev checkout.

Notes

• This PR's script changes do not run on the upgrade into this release. make upgrade is sh ./scripts/upgrade.sh, and the shell keeps reading the file it already opened after git checkout swaps the inode (reproduced in sh, bash, dash and zsh). So v0.10.0's script runs the whole upgrade, with no stop step, and a pass in flight at the recreate holds its old-code day-long lock. Nothing in the script can fix that; the commit's BREAKING CHANGE footer carries the operator note into the release notes: make down-jobs before, make up-jobs after, and a one-time clear_job_locks --all if up-jobs reports a lock HELD with nothing running. Worst case for an operator who skips it is a day's lock, released by that command.
• Overlap semantics after a crash are unchanged in kind, only in timing: previously a dead holder blocked its job for a day; now the next pass starts after at most one lease window.
• Named follow-up (the feed-lease migration): put a LeaseRenewer on poll_lock and delete the heartbeat callback threaded through FeedPollOperation, RedditBatchMixin, Connector.poll, sleep_with_heartbeat and five connectors; expose a lost Event so the poll loop keeps its yield-on-loss check at the between-source seam; consider measuring max_hold from the last touch() rather than total hold, with a per-command override; and move the keep-alive into a lock-level helper so stop-then-release ordering lives in one place. POLL_LOCK_TIMEOUT_SECONDS is the remaining renewed lease named as a timeout and would be renamed in the same pass.
• Out of scope: the poll pass on the affected instance takes about 20 minutes per 5-minute interval (two high-volume HN sources plus Reddit's anonymous 429 backoff). Separate throughput question.