fix(watch): capped quiet-period debounce; harness atomic writes, joined drains, stdout drain; #129 handshake control; #318 site (#129, #318, #320, #379) - #382
Conversation
A `workflow_dispatch`-only ubuntu-latest instrument that runs
`cargo test -p mds-cli --test cli_watch` N times (1-200) across two legs,
`default` and `startup-race-probe`, and tallies a pass/fail RATE rather
than aborting on the first red.
It is NOT a gate: dispatch-only means zero check-runs on any PR head, so
it cannot enter branch protection, is not a required context, is not a
release-surface path, and is invisible to scripts/verify-pr-checks.mjs
and all 212 gate specs.
Pins mirror ci.yml/release.yml byte-for-byte (PF-040); per-leg rust-cache
key because the legs build different feature sets (PF-041); no `${{ }}`
inside any `run:` block -- every value crosses via `env:` (PF-045); the
artifact uses `if-no-files-found: error` against an always-written
summary.txt (PF-016).
…es must be invisible
An editor (and the `write_atomic` helper the next commit adds) replaces a file by
writing a sibling temp file and renaming it over the target. That is ONE filesystem
event on the destination — notify 8 surfaces inotify's IN_MOVED_TO as
`Modify(Name(RenameMode::To))` — not the truncate-then-write pair `std::fs::write`
produces. Nothing in the suite pinned that the watcher treats it as a content edit,
nor that the in-flight temp file stays invisible to both watch modes.
Three integration tests in cli_watch.rs:
R1 watch_file_mode_rename_into_place_triggers_rebuild
R2 watch_dir_mode_rename_into_place_triggers_rebuild
R3 watch_dir_mode_write_atomic_temp_file_is_never_compiled
R3 carries its own non-vacuity control inside the same test: a REAL second source
(`u.mds`) is created through the same rename path and must be compiled, so the two
"no temp artefact" assertions cannot pass on a watcher that is simply compiling
nothing.
One unit test in src/output.rs:
collect_mds_files_ignores_write_atomic_temp_names — the `.<name>.tmp-<pid>-<n>`
shape puts the suffix AFTER the `.mds`, so `Path::extension()` is not `mds` and
the shared walker's gate drops it. Second half inverts the name to prove the
first assertion is not passing on an empty walk.
RED observed (cargo nextest run -p mds-cli --test cli_watch):
error[E0432]: unresolved import `common::write_atomic`
--> crates/mds-cli/tests/cli_watch.rs:24:84
|
24 | dup_vars_file_warning, mds_bin, spawn_watch_ready, spawn_watch_unsynchronized, write_atomic,
| ^^^^^^^^^^^^ no `write_atomic` in `common`
error: could not compile `mds-cli` (test "cli_watch") due to 1 previous error
This is the intended RED: the helper does not exist yet. Consequently
`cargo clippy -p mds-cli --all-targets` does not pass at this commit either — it
cannot build the cli_watch test target. The next commit adds the helper and both
clippy variants are clean from there on.
GREEN already, as expected (the walker gate is pre-existing behaviour; the test
pins it):
PASS [0.013s] mds-cli::bin/mds output::tests::collect_mds_files_ignores_write_atomic_temp_names
Summary [0.014s] 1 test run: 1 passed, 137 skipped
Refs #320.
GREEN for the R1-R3 tests added in the previous commit. `std::fs::write` truncates before it writes, so a watcher running at `--debounce 0` can observe — and compile — the 0-byte intermediate. CI run 34366009518 on 2b91850 printed two `Recompiled` lines for a single write for exactly this reason. Every real editor replaces a file by writing a sibling temp and renaming it over the target, which is one event on the destination; `write_atomic` does the same so the suite's writes look like an editor's rather than like a truncate. Temp name shape is load-bearing: `.<name>.tmp-<pid>-<n>` puts the suffix AFTER the name, so `Path::extension()` is the `tmp-…` component and never `mds`. Both `collect_mds_files_inner` (output.rs) and the dir-mode event filter (watch.rs) gate on exactly that extension, so an in-flight temp file is invisible to both. A `debug_assert_ne!` in the helper pins the shape at its source. A process-local `AtomicU64` plus the pid makes the temp name unique across the suite's parallel tests. No fsync — `rename` orders the replacement for every live process, which is all a watcher needs, and the product's own readiness marker is written the same way. GREEN observed (cargo nextest run -p mds-cli --test cli_watch, filtered): PASS [0.590s] (1/3) mds-cli::cli_watch watch_dir_mode_rename_into_place_triggers_rebuild PASS [0.590s] (2/3) mds-cli::cli_watch watch_file_mode_rename_into_place_triggers_rebuild PASS [0.647s] (3/3) mds-cli::cli_watch watch_dir_mode_write_atomic_temp_file_is_never_compiled Summary [0.648s] 3 tests run: 3 passed, 72 skipped No product change was needed: `Modify(Name(RenameMode::To))` already passes `is_content_event` and the destination path is already in the watch set, so both modes saw the rename as a content edit on the first try. cargo fmt --all --check clean; cargo clippy -p mds-cli --all-targets -- -D warnings and the same with --features startup-race-probe both clean. Refs #320.
Mechanical rule, stated here and in the file's doc comment so a reviewer can
reproduce the exact set: convert a `std::fs::write(` call iff it occurs AFTER the
`spawn_ready`/`spawn_unsynchronized` call in the same test fn AND targets a path the
watcher is watching (the `.mds` source, an imported partial, the `--vars` file, an
external dependency). Everything else keeps `std::fs::write`: pre-spawn fixture
writes, `.git` markers, `mds.json`, and output files.
45 conversions. Counts in crates/mds-cli/tests/cli_watch.rs:
std::fs::write( 156 -> 114 (159 -> 114 counting the 3 added by the RED commit)
write_atomic( 0 -> 49 (45 conversions + 4 call sites in R1-R3)
Two post-spawn writes are deliberate exceptions and now say so inline:
- watch_single_status_line_per_rebuild (--debounce 100) — its subject IS the
coalescing of the truncate+write pair; an atomic write would remove the thing
being tested.
- watch_debounce_single_rebuild_from_burst — same reason across a 10-edit burst;
a later phase rewrites this test.
Why this matters: `std::fs::write` truncates first, so at `--debounce 0` the watcher
can see and compile a 0-byte file and then the real content — two rebuilds for one
logical edit. That is the shape behind the i17 over-counts in CI run 34366009518
(left 3 / right 2 at cli_watch.rs:4370). Routing the writes through a rename makes
every watched edit exactly one event, which is also what a real editor does.
GREEN, three consecutive full runs of `cargo nextest run -p mds-cli --test cli_watch`:
Summary [3.910s] 75 tests run: 75 passed, 0 skipped
Summary [3.930s] 75 tests run: 75 passed, 0 skipped
Summary [3.938s] 75 tests run: 75 passed, 0 skipped
macOS cannot reproduce the Linux inotify tearing class this guards against, so this
is green locally and the Linux soak is the instrument.
cargo fmt --all --check clean; both clippy variants clean.
Refs #320.
`StderrTap::bytes` clones the shared buffer with no happens-before edge to the drain
thread's last write. Reaping the child closes its write end and ends the drain loop,
but nothing makes the reader observe that the loop finished, so a snapshot taken
right after `kill` + `wait` can be a truncated prefix. The suite papered over this
with a `thread::sleep` at every such site — 13 of them — and the `JoinHandle` for
the drain thread is discarded at spawn, so joining is not even possible today.
New test: stderr_tap_finish_captures_every_line_the_child_wrote. A dir watcher over
500 sources announces `Compiled to` once per file during the startup batch, and the
readiness handshake fires only after that batch completes, so the expected count is
exact (500) at the moment the child is killed. Written deliberately against the
current `bytes()` path with NO sleep, so the next commit's `finish` has something to
convert.
The `Compiled to` count is its own positive control: a shortfall means a lost tail,
and a count of 0 would mean the watcher compiled nothing rather than that the tap is
sound.
Observed locally (3 consecutive runs):
PASS [0.678s] (1/1) mds-cli::cli_watch stderr_tap_finish_captures_every_line_the_child_wrote
PASS [0.683s] (1/1) ...
PASS [0.696s] (1/1) ...
So this is GREEN on macOS. That is the honest result and it is expected: the field
signature for this defect is Linux — `watch_clear_non_tty_no_ansi_escape` panicking
at cli_watch.rs:520 in CI runs 32954883014 and 32954876042, a kill-then-snapshot site
with the same unjoined-drain shape. macOS pipe timing has not been observed to tear
here. The Linux soak is the instrument; the test pins the property either way.
cargo fmt --all --check clean; both clippy variants clean.
Refs #320.
…s to common (#320) GREEN for stderr_tap_finish_captures_every_line_the_child_wrote. `StderrTap` becomes `PipeTap`, a drained capture of ONE of the child's pipes, with `StderrTap`/`StdoutTap` as aliases so every existing call site keeps reading the way it did. The drain thread's `JoinHandle` — previously discarded at spawn, so joining was impossible — is now kept in the tap behind `Arc<Mutex<Option<_>>>` (the `Option` so `finish` can take it; the `Arc<Mutex<_>>` so `PipeTap` stays `Clone`). New end-of-test read: #[must_use] fn finish(self, child: &mut ChildGuard) -> Vec<u8> #[must_use] fn finish_text(self, child: &mut ChildGuard) -> String Termination is proved rather than bounded: the drain loop exits only at EOF, EOF arrives only when the child's write end closes, and `finish` reaps the child before it joins — so no timeout is needed and none is used. Taking `&mut ChildGuard` puts "reaped before join" in the signature instead of in a comment. `bytes()`/`text()` stay NON-blocking and keep their documented caveat: no happens-before edge to the child's last write. The live-poll sites (`wait_for_stderr_contains_str`, the mid-test snapshots) need exactly that. `ChildGuard` moves from cli_watch.rs to tests/common/mod.rs — it has to live where `finish` can name it — and gains `kill_and_wait` alongside the existing `id` and `wait_status`. It stays `pub struct ChildGuard(pub Child)` so the `child.0` field accesses across the suite are unaffected. cli_build.rs keeps its own private copy; the two are in separate test binaries and cli_watch.rs imports by name, not by glob, so nothing conflicts and cli_build.rs is untouched. GREEN: cargo nextest run -p mds-cli --test cli_watch Summary [3.936s] 76 tests run: 76 passed, 0 skipped cargo fmt --all --check clean; both clippy variants clean. Refs #320.
…320) Every "kill, wait, sleep 50-100ms, then snapshot the tap" site was compensating for the same missing edge: `bytes()` has no happens-before relationship with the drain thread's last write, so the sleep was the only thing making the snapshot usually complete. `finish`/`finish_text` reap the child and then JOIN the drain, so the snapshot is complete by construction and the sleep is not a bound that might be too short on a loaded runner — it is gone. 12 sites converted, thread::sleep count 55 -> 43 (the file now also mentions `thread::sleep` once in a doc comment, so `grep -c` reads 44). Two sites are not plain `kill/sleep/snapshot` and were converted by hand: - watch_clear_non_tty_no_ansi_escape — keeps `finish` (raw bytes, not text): its assertions hunt for raw ESC sequences. This is the site whose Linux field signature is the panic at cli_watch.rs:520 in CI runs 32954883014 and 32954876042 — a kill-then-snapshot with the unjoined drain. - watch_esc_in_initial_compile_error_is_sanitized — was `drop(child)` followed by a snapshot, which reaps the child but can never join the drain. Now holds the guard mutable and uses `finish`. - watch_ctrl_c_prints_stopped_watching — the child has already exited via SIGINT; `finish_text` reaps it again (harmless, `wait` caches the status) and joins. The 13th, in `watch_stdout_no_duplicate_write_on_startup`, drains stdout through a hand-rolled reader thread rather than a tap. It is converted in the commit that adds the stdout tap, where that reader thread is deleted. NOT touched — these are observation windows, not flush waits, and removing them would change what each test observes: - idle-observation sleeps: 400ms after the burst, 1500ms x3 in the no-spurious-recompile / single-status-line tests, 2500ms x2 in the idle_no_recompile_across_ticks pair, 600ms in the 500-file idle test - post-`remove_dir_all` settle waits (200-500ms) in the delete/recreate tests - burst pacing (5ms between rapid edits) - poll granularity inside bounded loops (1ms / 20ms / 50ms in wait_for_file_contains, wait_for_stderr_contains_str, try_wait loops) - the 500ms "give the watcher time to attempt a rebuild" waits in the compile-error tests, where the expectation is that nothing is produced GREEN, three consecutive full runs: Summary [4.000s] 76 tests run: 76 passed, 0 skipped Summary [3.858s] 76 tests run: 76 passed, 0 skipped Summary [3.859s] 76 tests run: 76 passed, 0 skipped cargo fmt --all --check clean; both clippy variants clean. Refs #320.
#320) `spawn_watch_ready` pipes stderr and drains it, but a caller that also passes `.stdout(Stdio::piped())` gets an undrained stdout pipe. `mds watch -o -` publishes the startup output to stdout BEFORE it writes the readiness marker — the marker is emitted after the compile, the arming and the publish — so once the pipe fills (~64 KiB) the child blocks in `write` while the harness sits in the marker poll loop. Neither side can move. Today the suite only gets away with this because its two piped-stdout tests produce a handful of bytes. Any test whose startup output is larger deadlocks, and the failure surfaces as "the watcher never reported readiness" — which reads like a watcher defect rather than a harness one. New test: watch_ready_with_large_piped_stdout_does_not_deadlock. 512 KiB of body, several pipe buffers on both Linux and macOS, so the block is a certainty rather than a timing question. It also asserts the full byte count arrives, so the property stays pinned once the deadlock is fixed. RED observed (cargo nextest run -p mds-cli --test cli_watch -E 'test(large_piped_stdout)'): thread 'watch_ready_with_large_piped_stdout_does_not_deadlock' panicked at crates/mds-cli/tests/common/mod.rs:371:13: mds watch did not signal readiness within 10s; stderr so far was: test result: FAILED. 0 passed; 1 failed; ... finished in 10.01s The panic is the READY_TIMEOUT bound firing after the full 10s with an empty stderr — exactly the shape predicted: the child is blocked before it ever reaches the marker write. cargo fmt --all --check clean. Clippy is clean on the code; the test target builds, it just fails at runtime. Refs #320.
GREEN for watch_ready_with_large_piped_stdout_does_not_deadlock.
`spawn_watch_unsynchronized` now returns `(Child, StderrTap, Option<StdoutTap>)` and
drains a piped stdout the same way it drains stderr. `child.stdout.is_some()` is
exactly "the caller piped stdout" — `Command` inherits stdout by default — so no flag
is needed. `spawn_watch_ready` destructures the triple and returns it unchanged; its
marker poll loop is untouched and now simply runs with both drains live.
Wrappers in cli_watch.rs:
spawn_ready(cmd) -> (ChildGuard, StderrTap)
spawn_ready_piped_stdout(cmd) -> (ChildGuard, StderrTap, StdoutTap)
spawn_unsynchronized(cmd) -> (ChildGuard, StderrTap)
The two non-piped wrappers assert `stdout_tap.is_none()`, so a test that pipes stdout
and reaches for `child.0.stdout` fails with a message naming the right helper instead
of finding a `None` it cannot explain.
`spawn_unsynchronized_piped_stdout` is deliberately NOT added: no unsynchronized test
pipes stdout, so it would be dead code, and `#[allow(dead_code)]` is confined to
tests/common/mod.rs in this crate. `spawn_unsynchronized`'s assertion message says so
and tells the next author to mirror `spawn_ready_piped_stdout`.
Three tests move to the piped-stdout wrapper and their hand-rolled plumbing is
deleted:
- watch_stdout_contains_content_when_o_stdout — its manual read loop over
`child.0.stdout` becomes a poll of `stdout_tap.text()`
- watch_stdout_no_duplicate_write_on_startup — its own 17-line reader thread and
`Arc<Mutex<Vec<u8>>>` are gone, and its post-kill flush sleep with them. This is
the 13th of the 13 sleeps identified in the previous commit.
- watch_ready_with_large_piped_stdout_does_not_deadlock — the RED test
thread::sleep sites: 42 real, plus one mention inside a doc comment, so
`grep -c 'thread::sleep' crates/mds-cli/tests/cli_watch.rs` reads 43 (was 55 before
this phase).
GREEN, three consecutive full runs:
Summary [3.828s] 77 tests run: 77 passed, 0 skipped
Summary [3.822s] 77 tests run: 77 passed, 0 skipped
Summary [3.825s] 77 tests run: 77 passed, 0 skipped
cargo fmt --all --check clean; both clippy variants clean.
Refs #320.
… land The i16–i20 family asserts exact occurrence counts of the duplicate-vars-key warning, and every one of them samples `stderr_tap.text()` immediately after a `wait_for_file_contains`. Those two events are unordered: dir mode emits the warning AFTER the output write (`handle_fs_event_dir`, and the same shape in `liveness_probe_dir`), so the output file being complete says nothing about whether the warning has been written yet. The race has been observed in both directions in CI, which is what rules out "just add a bigger timeout" as a fix: run 34366009518 (main 2b91850) i17 cli_watch.rs:4370 left 3 / right 2 run 34404318888 attempt 1 i17 cli_watch.rs:4370 left 1 / right 2 i18 cli_watch.rs:4448 left 2 / right 1 run 34404318888 attempt 2 i18 cli_watch.rs:4448 left 2 / right 1 New helper next to `wait_for_stderr_contains_str`: fn wait_for_stderr_count(tap, needle, n, timeout) -> String 20ms poll; returns as soon as the count reaches `n`; PANICS on timeout naming the count it actually saw. That last part is the difference from `wait_for_stderr_contains_str`, which RETURNS its text on timeout and so lets the caller's `assert_eq!` report a timeout as though it were a settled answer. That helper is left as-is — changing it is tracked separately as issue I3. This commit adds the helper, wires it into i17's startup assertion (the site with the CI evidence above), and rewrites i17's doc comment: the stale `:1793` / `:1919` / `:2196` line references are replaced with the function names they now live in (`liveness_probe_dir`, `handle_fs_event_dir`, `dir_watch_startup`), which do not drift. The remaining i16–i20 sites migrate in the next commit. RED evidence is the CI race above, not a local run: this does not reproduce deterministically on macOS, where the emit and the sample happen to order correctly. i17 is green here before and after: PASS [0.597s] (1/1) mds-cli::cli_watch i17_dir_watch_vars_file_duplicate_warns_once_per_rebuild cargo fmt --all --check clean; both clippy variants clean. Refs #320, #318.
…(i16–i20)
GREEN for the sampling race described in the previous commit.
Every count assertion in i16–i20 whose expectation is non-zero now runs
`wait_for_stderr_count(&stderr_tap, &expected, N, TIMEOUT)` first, so the assertion
that follows means "never MORE than N" rather than "happened to be N at the instant
the tap was sampled". The final assertion in each test also switches from
`stderr_tap.text()` to `stderr_tap.finish_text(&mut child)`, which reaps the child
and joins the drain — so a warning the child wrote but the drain had not yet copied
is in the snapshot rather than lost.
Sites changed:
i16 startup / after edit 1 / after edit 2 -> wait for 1 / 2 / 3, finish at the end
i17 startup (previous commit) / after edit -> wait for 1 / 2, finish at the end
i18 final assertion -> wait for 1, finish
i19 startup / after self-heal -> wait for 1 / 2, finish at the end
i20 final assertion -> finish (see below)
Two kinds of site do NOT get a wait, because "at least 0 occurrences" is satisfied
immediately and a wait there would be theatre: i18's two zero-count assertions and
i20's pair under `--quiet`. Both already carry a positive control — a
`wait_for_file_contains` proving the rebuild really happened — so the zero is not
vacuous, and the final one in each now uses `finish_text`, the strongest snapshot
available.
Stale source references in the assertion messages and doc comments are replaced with
function names, which do not drift: `:1793` -> `liveness_probe_dir`, `:1919` ->
`handle_fs_event_dir`, `:2196` -> the dedup-baseline second read in
`dir_watch_startup`, `watch.rs:936` -> `rebuild_file`.
RESIDUAL, not closed by this commit and worth naming: i18's over-count exposure
(CI run 34404318888, `:4448` left 2 / right 1) is not a sampling race. i18 writes the
vars file and then the source; in file mode the vars file is watched, so at
`--debounce 0` those two writes can be serviced as two rebuilds, and two rebuilds
legitimately emit two warnings. Waiting for the count does not close that, and
neither does the atomic write — the events are on different paths. The exact count
is left as-is rather than weakened to `>= 1`, since that would drop the
double-emit-per-rebuild guard; the i-family's design is a later phase's problem.
GREEN, i16–i20 eight consecutive times:
5 tests run: 5 passed, 72 skipped (x8, 0.667s-0.882s)
and three consecutive full runs:
Summary [3.818s] 77 tests run: 77 passed, 0 skipped
Summary [3.849s] 77 tests run: 77 passed, 0 skipped
Summary [3.840s] 77 tests run: 77 passed, 0 skipped
macOS orders the emit and the sample correctly, so the race this fixes does not
reproduce locally — green here, and the Linux soak is the instrument.
cargo fmt --all --check clean; both clippy variants clean.
Refs #320, #318.
RED for behavioural reasons only: `drain_debounce` is restructured to return a
typed outcome, but its semantics are unchanged — the deadline is still a FIXED
offset from the first event, never extended, and the `Cap` / `MessageLimit`
exits are wired into the classification with their bounds set beyond reach.
New in watch.rs (temporary bodies, marked TEMPORARY in their doc comments):
enum DebounceEnd { Disabled, Quiet, Cap, MessageLimit, Interrupted, Disconnected }
struct DebounceOutcome { paths, end } + fn interrupted()
fn clamp_debounce(u64) -> Option<Duration> // no clamp yet
fn debounce_cap(Duration) -> Duration // returns u32::MAX seconds
const MAX_DEBOUNCE_MESSAGES: usize = 1_000_000 // unreachable at this size
Both callers move to the new return type: file mode discards the drained paths
(it has already decided relevance and rebuilds its single entry regardless),
dir mode extends `changed` with them after the interrupt check.
Deviation from the plan: MAX_DEBOUNCE_MS / DEBOUNCE_CAP_FACTOR / DEBOUNCE_CAP_FLOOR
are NOT introduced here. Nothing uses them until the real clamp and cap land, and
an unused const is a `dead_code` warning in a non-test build — the repo's
zero-warnings policy leaves no way to carry them through this commit honestly.
MAX_DEBOUNCE_MESSAGES is 1_000_000 rather than usize::MAX because
clippy::absurd_extreme_comparisons denies `>= usize::MAX`.
Observed RED (macOS, `cargo nextest run -p mds-cli`, 7 of 11 selected tests fail):
clamp_debounce_contract
left: Some(18446744073709551.615s) right: Some(60s)
debounce_cap_contract
left: 4294967295s right: 1s ("the floor binds for small windows")
debounce_quiet_period_extends_on_content_events
left: 18 right: 40 paths — the fixed 100ms window closed a third of the way
through a 200ms burst
debounce_cap_ends_a_continuous_stream
left: Quiet right: Cap
debounce_message_limit_bounds_one_window
left: Quiet right: MessageLimit
watch_debounce_single_rebuild_from_burst
left: "---\nname: v8\n---\nBurst v8!\n" right: "Burst v12!\n"
— one Recompiled line, but it compiled v8: the 250ms window expired four
writes before the burst ended
watch_debounce_cap_rebuilds_while_writes_never_stop
Got 14 Recompiled lines (expected 1..=4) from 3s of writes under a 200ms window
Observed GREEN, as expected — these pin behaviour that is already correct:
debounce_zero_is_disabled_and_leaves_the_channel_untouched,
debounce_interrupt_returns_immediately, debounce_access_events_do_not_extend.
`cargo nextest run -p mds-cli --test cli_watch`: 78 tests run, 76 passed,
2 failed — exactly the two integration tests above, so the caller refactor
broke nothing.
cargo fmt --all --check clean; `cargo clippy -p mds-cli --all-targets -- -D warnings`
and the same with --features startup-race-probe both clean.
The integration side: `watch_debounce_single_rebuild_from_burst` keeps its name
(three CI runs and the issue comments cite it) and is rewritten so its burst is
LONGER than the window — 12 plain writes 30ms apart against `--debounce 250` —
with self-diagnosing preconditions (span > 250ms, max gap < 250ms) asserted
before the outcome, so a scheduler artefact cannot masquerade as a product
failure. `--poll-interval` stays at its default, leaving the idle-tick probe
live. The new `watch_debounce_cap_rebuilds_while_writes_never_stop` runs with
`--poll-interval 0`, so the cap is the only mechanism that could rebuild during
a continuous stream.
Refs #379.
`--debounce` was a window that expired at a FIXED offset from the first event. Any save burst longer than the window was split across two or three windows and rebuilt once per window, each compile seeing a different intermediate state of the file — `watch_debounce_single_rebuild_from_burst` failed as `got 3` with three `Recompiled` lines (40ms / 80ms / 3ms) on loaded CI runners, in runs 33996153739, 33976595173 and 33753123463 (11 occurrences across 146 ci.yml runs since 2026-08-26). The size of the burst a user can produce is not a property `debounce_ms` can predict; the size of the GAP between saves is. Now the first relevant event opens the window and every further CONTENT event restarts it, so the window ends when the writing goes quiet. An extendable window with no bound is unbounded, so two bounds come with it: - an absolute cap of `max(10 x window, 1s)` measured from the first event. A file written to continuously would otherwise postpone its own rebuild for as long as the writing lasts — and, because the loop never reaches `TickClock::recv_next` while a window is open, would starve the idle-tick content backstop through a door the absolute tick deadline does not cover. The floor matches the default `--poll-interval`, so it is also the bound on how late the probe can run under a continuous stream. - `MAX_DEBOUNCE_MESSAGES = 10_000` drained messages per window, bounding the window's work and `paths`' memory against a sender faster than the drain. Messages left in the channel are not lost: the next event opens a new window. `deadline <= hard_cap` is an unconditional `assert!`, not a `debug_assert!` — it is the bound's release-build enforcement, and it is pure arithmetic, so a descheduled runner cannot trip it, only a defect can. Asserting on MEASURED elapsed time instead would panic a shipped watcher whenever `recv_timeout` overshoots. `deadline == hard_cap` is a sound `Cap` discriminator because the cap is at least 10x the window, so the initial deadline never equals it. `window * DEBOUNCE_CAP_FACTOR` cannot overflow `Duration`: the clamp caps the window at 60s, so the product is at most 600s. Raw values are clamped to 60s. Observed at 2b91850: `--debounce 18446744073709551615` does NOT panic — `Instant::now() + Duration::from_millis( u64::MAX)` lands ~585 million years out on the i64-second monotonic clocks of macOS and Linux, so the window never closes and the watcher silently never rebuilds (two edits, 13s, no `Recompiled`; SIGINT still exits 0 with `Stopped watching.`). `--debounce 18446744073709551616` is rejected by clap (exit 2). The clamp therefore prevents a silent infinite window, not a crash. Unchanged: `--debounce 0` still means no coalescing and leaves the channel untouched; `Access` events and watch errors still do not extend; relevance is still decided by the caller, not here (an editor's atomic save renames a temp path that is in no watch set — ending the window on it would split the very burst this exists to coalesce). No new output: a cap hit prints nothing. GREEN: the 8 new unit tests and the 2 debounce integration tests all pass; `cargo nextest run -p mds-cli --test cli_watch` 78/78 x3 (4.168s / 5.167s / 4.085s); `cargo nextest run -p mds-cli` 832 tests run: 832 passed. cargo fmt --all --check clean; both clippy variants clean. Two unit tests gained a `tx.clone()` keepalive (quiet-period and access-event): once the window outlives the burst, the sender being dropped ends it as `Disconnected` and the failure names the channel's lifetime instead of the property under test. In production the notify sender lives as long as the watcher. MUTATION TABLE — each a scratch edit, run, then restored and verified with `cmp` against a pristine copy (RESTORED_OK for all eleven): M1 fixed deadline restored (drop the reassignment) debounce_quiet_period_extends_on_content_events left: 17 right: 40 paths watch_debounce_single_rebuild_from_burst left: "---\nname: v8\n---\nBurst v8!\n" right: "...v12..." RED M2 `.min(hard_cap)` dropped watch.rs:742 "debounce deadline escaped its cap: a file written to continuously would postpone its own rebuild (and the idle-tick backstop behind it) for as long as the writing lasts" — the assert fires on the worker thread; drain_bounded then reports "did not return within 3s" RED M3 DEBOUNCE_CAP_FLOOR = 3600s debounce_cap_ends_a_continuous_stream left: Disconnected right: Cap watch_debounce_cap_rebuilds_while_writes_never_stop "a rebuild must happen WHILE the writes are still arriving" RED M4 Access events extend (drop the is_content_event guard) debounce_access_events_do_not_extend "300ms of reads must not extend a 100ms window past ~100ms; got 468.653875ms" RED M5 Msg::Interrupt keeps draining (continue instead of break) debounce_interrupt_returns_immediately left: Disconnected right: Interrupted RED M6 clamp_debounce(0) = Some(1ms) clamp_debounce_contract left: Some(1ms) right: None debounce_zero_is_disabled_... left: Quiet right: Disabled RED M7 message bound removed (100_000_000) debounce_message_limit_bounds_one_window left: Quiet right: MessageLimit RED M8 Cap classification inverted debounce_cap_ends_a_continuous_stream left: Quiet right: Cap RED M9 clamp dropped (`.min` removed) clamp_debounce_contract left: Some(18446744073709551.615s) right: Some(60s) RED M10 dir caller drops `changed.extend(drained.paths)` FINDING — WEAK CONTROL. watch_dir_mode_shared_partial_rebuilds_importers, watch_dir_mode_partial_edit_rebuilds_exactly_n_importers and watch_dir_mode_soak_50_edits_bounded_and_clean_exit all PASS, and so does the whole suite: 78 tests run: 78 passed. The drained paths are redundant with the initial batch whenever the first event of a burst already names every file the burst touches, which is what every dir-mode test does. The line is kept — a burst that starts on one file and continues on another needs it — but nothing in the suite pins it. M11 file caller ignores interrupted() FINDING — WEAK CONTROL. watch_ctrl_c_exits_cleanly, watch_ctrl_c_prints_stopped_watching and watch_file_mode_ctrl_c_during_startup_compile_terminates all PASS, and so does the whole suite: 78 tests run: 78 passed. Cause: all three spawn with `--debounce 0`, so drain_debounce returns Disabled without ever seeing a message, and the outer loop's own interrupt handling is what exits. No ctrl-c test runs with coalescing on. Refs #379.
`--debounce` is no longer "how long to wait after the first event", so the clap
help, the README option block and the CHANGELOG all said the wrong thing.
Rendered help (`cargo run -p mds-cli -- watch --help`):
--debounce <MS>
Quiet period in milliseconds before a rebuild (default 100). Each file
change restarts the window, so a save burst longer than MS still
coalesces into one rebuild; the window is capped at max(10 x MS, 1000)
ms so continuous writes still rebuild. Use 0 to disable coalescing.
Values above 60000 are clamped
[default: 100]
The clap help stays ASCII-only; the README block uses `×` to match the `≥50ms`
in the `--poll-interval` block directly below it. The CHANGELOG bullet goes in a
new `### Changed` section ahead of `### Fixed` (Keep a Changelog order) and
names the known cost: in directory mode every event opens a window, including
events under excluded directories that are only filtered afterwards, so
`npm install` churn can delay a real edit — and the idle tick — by up to the cap.
node scripts/verify-no-control-bytes.mjs: scanned 559 files, 6532099 bytes, clean.
cargo fmt --all --check clean; both clippy variants clean;
`cargo nextest run -p mds-cli` 832 tests run: 832 passed.
Refs #379.
…#129) Surface 1 of #129 was "SIGINT can land before `ctrlc::set_handler` runs". The #323 ordering closed it — `set_handler` precedes `emit_ready_marker` in both watch modes — but nothing in the suite pinned that the handshake is what makes a post-SIGINT `status.success()` deterministic rather than luck. `watch_readiness_handshake_makes_ctrl_c_exit_deterministic` is a two-arm control: same signal, opposite verdicts, 20 iterations. - CONTROL: `spawn_unsynchronized` + SIGINT gated on the `Watching …` line, which `run_watch_file` prints before it creates the watcher and long before the handler is installed -> death by SIGINT. A clean exit in this arm would mean the pre-handler window is no longer being hit, and the treatment arm would then prove nothing. - TREATMENT: `spawn_ready` + immediate SIGINT -> exit 0 and "Stopped watching.". N = 20 is a live discriminator, not a rate bound (the manual Linux soak workflow is the rate instrument). Every wait is bounded; none is a sleep standing in for a synchroniser. `wait_bounded` is a new 1ms-granularity try_wait loop that panics naming the arm. `#[cfg(unix)]`: SIGINT has no Windows analogue. The control fixture reuses the verified shape from `watch_file_mode_ctrl_c_during_startup_compile_terminates` (400 partials, `@define`/`@end`/`@export`, one `@import` each), built once for the whole run. Surface 2 needed no change here: `watch_ctrl_c_prints_stopped_watching` already reads its final stderr through `finish_text` (the post-kill flush sleep was replaced in cffb951), so no sleep gates its assertion. Also documents why i18 expects exactly 1 warning: the warning is gated in `rebuild_file` on an observable output-content change, the fixture never interpolates `x`, so the vars-file rebuild produces byte-identical output and reports nothing — only the `version 3` rebuild is observable. Atomic writes removed the 0-byte intermediate that used to add a second transition. Observed: - new test alone x4: `1 test run: 1 passed, 78 skipped` (0.889 / 0.785 / 0.791 / 0.771s) — 20/20 both arms every run - non-vacuity mutation (control arm `spawn_unsynchronized` -> `spawn_ready`, applied to a scratch copy, restored + `cmp` RESTORED_OK): RED at `cli_watch.rs:4213` `left: None right: Some(2)` with the intended message - `cargo nextest run -p mds-cli --test cli_watch` x3: `79 tests run: 79 passed, 0 skipped` (4.033 / 4.085 / 4.428s) - `cargo fmt --all --check` clean; clippy clean with and without `--features startup-race-probe` Refs #129
…DY (#318) `watch_bare_filename_from_cwd_succeeds` was the last "poll the artifact instead of synchronising on readiness" site outside cli_watch.rs, and the Rust half of #318. It spawned `mds watch hello.mds` from a cwd and polled `hello.md` for up to 10s. Polling is the defect, not the bound: it turns "the startup compile wrote the file" into "something wrote the file eventually", so a startup path that resolved the bare filename only on a later retry or rebuild still passed. A shorter loop would preserve that; only deleting the loop removes it. `run_watch_file` publishes the startup output at the arm-before-publish point, well before `emit_ready_marker`, so once `spawn_watch_ready` returns the file is on disk. It is now read ONCE, directly. Also: the private `ChildGuard` copy is replaced by `common::ChildGuard`, the `Duration`/`Instant` imports are gone with the loop, and `.stderr(Stdio::null())` is dropped in favour of the drained `StderrTap` — `-q` still lets a compile error through, so both failure paths now name their own cause instead of being silent. The issue's line refs (`cli_build.rs:1138-1148`) had drifted; the site was `:1309-1356` on main 2b91850. Observed: - `cargo nextest run -p mds-cli --test cli_build` x3: `46 tests run: 46 passed, 0 skipped` (1.349 / 1.372 / 1.359s) - the test alone: `PASS [0.430s]` (was a 10s-bounded poll) - `cargo fmt --all --check` clean; clippy clean with and without `--features startup-race-probe` Refs #318
Seven `### Internal` bullets under `[Unreleased]`, one per harness or test change on this branch, alongside the existing soak-workflow bullet: atomic writes, joinable drains, stdout drained before the readiness wait, bounded warning-count waits for i16-i20, the #129 handshake control test, and the #318 cli_build.rs site. Observed: `node scripts/verify-no-control-bytes.mjs` clean. Refs #129 #318 #320
`DebounceEnd::Disconnected` was the only variant of the new debounce
outcome with no test. It is not cosmetic: without the early break the
drain busy-spins on `recv_timeout` until the full window elapses, so a
watcher whose sender has been dropped delays its own shutdown by up to
the window (and, with the sender gone, for no possible gain — no further
event can ever arrive).
Positive control (PF-013): replacing the break with `{}` makes the new
test fail `left: Quiet / right: Disconnected` after exactly 5.00s, which
is both the wrong exit reason and the spin it describes.
Refs #379
`PipeTap`'s drain-slot comment claimed "several tests hand a clone to a helper"; no call site clones a tap. Reword to say what the shape is actually for: `Clone` is harness API, and the mutex is what makes a concurrent `finish` from a clone safe. The CHANGELOG's debounce "known cost" was scoped to directory mode. It applies in both: `drain_debounce` extends the deadline on any content event without re-deriving relevance, and file mode watches the entry's parent directory non-recursively, so a sibling scratch write extends a window a real edit has already opened. Dir mode differs only in that an irrelevant event can also *open* one — file mode checks relevance first.
Soak evidence (post-merge, as planned)Measured with the manual Why the measurement is post-merge. Instrument controls first (PF-013 — a gate never observed rejecting anything is not evidence). M14 and M15 were run before the measurements, on a throwaway Runs
Control detailM14 — must-fail control. M15 — empty-artifact control. The alongside the echoed input PRE — what actually failedBoth legs failed on the #326 duplicate-key-warning rebuild assertions, not on the ESC or burst sites:
(Line numbers are VerdictAggregate over both legs: 7 / 40 iterations failed pre-fix → 0 / 40 post-fix. The instrument is proven to reject both a failing suite (M14) and an empty artifact (M15), so the post-fix |
Summary
Closes the C2 watch-reliability cluster: one product defect and three harness defects that
together produced the
cli_watchflake family.mds watch --debouncebecomes a quietperiod with a hard cap (#379) — the fixed-window semantics were the actual cause of the
burst
got 3failures, not the test. The harness gets atomic writes, a joinable pipedrain, and a stdout drain that runs before the readiness wait (#320), and the two
remaining "poll the artifact instead of synchronising" sites are converted (#318). #129's
two surfaces are closed and pinned by a two-arm control test.
This PR also carries the manual
watch-soak.ymlLinux soak instrument from the closedPR #375. That PR was YAML-only and still went red twice — on the very defects fixed
here (i17
:4370, i18:4448) — so it could not passverify-pr-checks.mjsand wasclosed rather than merged. It ships here, on a branch that fixes what reddened it.
17 commits, RED-first throughout: every behaviour claim has an observed failing run
before its fix.
What was wrong
Three distinct mechanisms, read out of the CI transcripts rather than assumed.
1. Fixed-window debounce — a PRODUCT defect.
watch_debounce_single_rebuild_from_burstfailed as
got 3atcli_watch.rs:1222, with three summary linesRecompiled … in 40ms / 80ms / 3ms— runs 33996153739, 33976595173,33753123463 (11 of 146
ci.ymlruns since 2026-08-26). The window expired at a fixedoffset from the first event (
watch.rs:608onmain), so a save burst longer thanthe window split into N rebuilds no matter how the test was written, each compile seeing
a different intermediate state of the file. No harness change can close that.
2.
fs::writetruncate-then-write at--debounce 0— a HARNESS defect.std::fs::writetruncates and then writes, publishing a 0-byte intermediate. Withcoalescing off the watcher compiled that intermediate, which is an extra observable
rebuild and an extra output transition. (The same tearing is reachable by real users with
an in-place editor — filed separately as #380.)
3. Dir-mode warning emitted after the output write, sampled unsynchronised — a HARNESS
defect. The i16–i20 family asserted an exact warning count on a stderr snapshot taken
the instant the output artifact appeared. In directory mode the duplicate-vars warning is
emitted after the output write, so the sample could be one short — or, once a second
rebuild entered the picture, one long:
main2b91850): i17:4370left 3 / right 2:4370left 1 /right 2 and i18
:4448left 2 / right 1; attempt 2 — i18:4448left2 / right 1
4. Unjoined drain thread — a HARNESS defect. Tests killed the child, slept ~100 ms,
then read a shared stderr buffer with no happens-before edge to the drain thread's last
write. Field signature:
watch_clear_non_tty_no_ansi_escape, reported ascli_watch.rs:520in runs 32954883014 / 32954876042 (that line number is fromthe commit those runs were on; the test is
cli_watch.rs:555onmain2b91850, itsunjoined read
:587).Design
D1 —
--debounceis a quiet period with a hard cap (#379). The first relevant eventopens the window; every further content event restarts it. Bounds:
drain_debouncereturns the typedDebounceOutcomeinstead of(BTreeSet<PathBuf>, bool).Decisions and their reasons:
filter — deciding relevance inside it would mean re-deriving
files_of_interestpermessage. Known cost, stated in the CHANGELOG and under Known limitations: in directory
mode
npm installchurn under an excluded directory can delay a real edit by up to thecap.
Accessevents never extend. A reader cannot postpone a writer's rebuild.assert!, notdebug_assert!, ondeadline <= hard_cap. It is the release-buildenforcement of the bound. It is pure arithmetic, so a descheduled runner cannot trip
it — only a defect can.
deadline == hard_capis a soundCapdiscriminator becausethe cap is at least 10× the window, so the initial deadline never equals it.
window * DEBOUNCE_CAP_FACTORcannot overflowDuration: the clamp caps the window at60 s, so the product is at most 600 s.
--debounce 18446744073709551615(
u64::MAX) previously watched forever without ever rebuilding, no panic;u64::MAX + 1is rejected by clap.
printed.
D2 —
common::write_atomic. Temp + rename: one FS event, the way an editor writes.The temp name puts the suffix after the full file name (
.<name>.tmp-<pid>-<seq>) soextension()is never"mds"and the temp is never collected as a source; adebug_assert_ne!pins that.<pid>disambiguates processes, anAtomicU64sequencedisambiguates within one.
D3 — a mechanical migration rule, stated in the file's doc comment so a reviewer can
reproduce the set exactly. Convert a
std::fs::write(call iff it occurs AFTER thespawn_ready/spawn_unsynchronizedcall in the same test fn AND targets a watched path(
.mdssource, imported partial,--varsfile, external dep). 45 sites converted.Untouched: pre-spawn fixture writes,
.gitmarkers,mds.json, output files. Two// DELIBERATE:exceptions whose subject is the truncate+write pair:watch_single_status_line_per_rebuildandwatch_debounce_single_rebuild_from_burst.D4 — joinable drain.
PipeTap::finish(self, &mut ChildGuard)reaps the child, thenjoins the drain thread. Termination is proved, not bounded: the drain loop ends only
at EOF, EOF arrives when the child's write end closes, and the child is reaped first, so
the join cannot hang on a live writer.
bytes()/text()stay non-blocking for thelive-poll sites.
ChildGuardmoved intotests/commonsofinishcan take&mut ChildGuardand put "reaped before join" in the type rather than in a comment.13 of 13 post-kill flush sleeps deleted.
D5 — stdout drained before the readiness wait.
mds watch -o -publishes its startupoutput before it writes the readiness marker, so an undrained pipe filled and blocked
the child while the poller waited for a marker that could never arrive. Both spawn helpers
now return
(Child, StderrTap, Option<StdoutTap>)and drain stdout insidespawn_watch_unsynchronized.child.stdout.is_some()is exactly "the caller pipedstdout", because
Commandinherits by default.D6 — #129 control test.
watch_readiness_handshake_makes_ctrl_c_exit_deterministic:two arms, same signal, opposite verdicts, 20 iterations. CONTROL —
spawn_unsynchronizedwith SIGINT gated on the
Watching …line (printed before the watcher exists and longbefore
ctrlc::set_handler) → death by SIGINT; if this arm ever exits cleanly thetreatment arm proves nothing, and the assertion message says so. TREATMENT —
spawn_readywith an immediate SIGINT → exit 0 and
Stopped watching..N = 20is a livediscriminator, not a rate bound; the soak workflow is the rate instrument.
D7 — #318
cli_build.rssite.watch_bare_filename_from_cwd_succeedssynchronises onthe handshake and reads
hello.mdonce. Polling is the defect, not the bound: itturns "the startup compile wrote the file" into "something wrote the file eventually", so
a shorter loop would have preserved it.
run_watch_filepublishes the startup output wellbefore
emit_ready_marker, so a single direct read is sound.D8 — the burst test was rewritten, not added. Its name is cited by three CI runs and
by the issue, so keeping the name keeps the history legible. The burst is now deliberately
longer than the window (12 writes, 30 ms apart, ≥330 ms against a 250 ms window). A new
watch_debounce_cap_rebuilds_while_writes_never_stoppins the cap.D9 — unit tests for the pure parts (
clamp_debounce,debounce_cap, quiet-periodextension, cap, message limit, interrupt,
Access-does-not-extend), so the properties arepinned without a process.
Why one PR, not two. The D-U5 baseline plan was two PRs — harness first, product
second — with the soak run against the harness-only head as the control. That collapsed
when PR #375's YAML-only head went red twice on i17/i18 (run 34404318888): a "harness
only" PR cannot be merged green while the product defect is live. So the soak control is
taken by ref instead of by PR: pre-fix on
--ref ci/watch-soak-workflow(=a8faafb,soak workflow +
main), post-fix on--ref main— both after merge, becauseworkflow_dispatchrequires the workflow to exist on the dispatched ref.Evidence
RED-first, per commit
fc83b1derror[E0432]: unresolved import common::write_atomicb055957common::write_atomic3 tests run: 3 passed(0.590 / 0.590 / 0.647 s)04b01a6write_atomic75 tests run: 75 passedfb3cd9bStderrTap::bytescan read a truncated bufferwatch_clear_non_tty_no_ansi_escapein runs 32954883014 / 329548760425b343fcPipeTap::finish76 tests run: 76 passedcffb95176 tests run: 76 passed— the 13th (watch_stdout_no_duplicate_write_on_startup) was converted in11c8a43together with the stdout drain; all 13 are gone at the head.f145067spawn_watch_readydeadlocks on a large piped stdoutmds watch did not signal readiness within 10satcommon/mod.rs:371,finished in 10.01s11c8a4377 tests run: 77 passed0b959b257084a8wait_for_stderr_count3373f1ff3fb1a0--debounceis a quiet period with a hard cap3caf629--helprecorded in the commit bodyf78dd7ca27bac9cli_build.rssite46 tests run: 46 passed×3214ff30Commit 11 — observed RED (numbers as printed)
clamp_debounce_contractSome(18446744073709551.615s), rightSome(60s)debounce_cap_contract4294967295s, right1sdebounce_quiet_period_extends_on_content_events18paths, right40debounce_cap_ends_a_continuous_streamQuiet, rightCapdebounce_message_limit_bounds_one_windowQuiet, rightMessageLimitwatch_debounce_single_rebuild_from_burst"---\nname: v8\n---\nBurst v8!\n", right"Burst v12!\n"— oneRecompiledline, but it compiled v8 — the expected string was corrected in the GREEN commit to the observed product output"---\nname: v12\n---\nBurst v12!\n"(frontmatter passes through verbatim); the RED subject, theRecompiled == 1count, was unchanged.watch_debounce_cap_rebuilds_while_writes_never_stopGot 14Recompiledlines (expected1..=4)Already GREEN, as expected (they pin behaviour that was already correct):
debounce_zero_is_disabled_and_leaves_the_channel_untouched,debounce_interrupt_returns_immediately,debounce_access_events_do_not_extend.cli_watchat commit 11:78 tests run: 76 passed, 2 failed— exactly the two newintegration tests, so the caller refactor broke nothing.
Commit 12 — mutation table (M1–M11)
Each mutation was a scratch edit applied with a uniqueness-checked script, run, then
restored from a pristine copy and verified with
cmp(RESTORED_OKfor all eleven).git status --porcelainwasM .gitignoreonly at the end.17right40paths; burst left"---\nname: v8\n---\nBurst v8!\n"right"...v12...".min(hard_cap)droppeddebounce_cap_ends_a_continuous_streamwatch.rs:742panic: "debounce deadline escaped its cap: a file written to continuously would postpone its own rebuild (and the idle-tick backstop behind it) for as long as the writing lasts"; the assert fires on the worker thread anddrain_boundedthen reports "did not return within 3s"DEBOUNCE_CAP_FLOOR = 3600sDisconnectedrightCap; integration "a rebuild must happen WHILE the writes are still arriving"Accessevents extend (drop theis_content_eventguard)debounce_access_events_do_not_extendMsg::Interruptkeeps draining (continue)debounce_interrupt_returns_immediatelyDisconnectedrightInterruptedclamp_debounce(0) = Some(1ms)Some(1ms)rightNone; leftQuietrightDisabled100_000_000)debounce_message_limit_bounds_one_windowQuietrightMessageLimitCapclassification inverteddebounce_cap_ends_a_continuous_streamQuietrightCap.minremoved)clamp_debounce_contractSome(18446744073709551.615s)rightSome(60s)changed.extend(drained.paths)watch_dir_mode_shared_partial_rebuilds_importers,watch_dir_mode_partial_edit_rebuilds_exactly_n_importers,watch_dir_mode_soak_50_edits_bounded_and_clean_exitall PASS; whole suite78 tests run: 78 passedinterrupted()watch_ctrl_c_exits_cleanly,watch_ctrl_c_prints_stopped_watching,watch_file_mode_ctrl_c_during_startup_compile_terminatesall PASS; whole suite78 tests run: 78 passedWhy M10/M11 are weak, with root causes. M10: the drained paths are redundant with the
initial batch whenever the first event of a burst already names every file the burst
touches — which is what every dir-mode test does. The line is load-bearing only for a
burst that STARTS on one file and CONTINUES on another inside one window; nothing in the
suite pins that. M11: all three ctrl-c tests spawn with
--debounce 0, sodrain_debouncereturns
Disabledwithout ever receiving a message and can never observe anInterrupt—no ctrl-c test in the suite runs with coalescing on. Both are recorded rather than
patched: adding either test is a behaviour claim of its own and belongs in its own
RED/GREEN pair, not appended to a GREEN commit.
#129 CI harvest, with its non-vacuity check
ci.ymlsince 2026-08-26: 146 runs, 21 red. 0 of the 21 contain a FAILEDwatch_ctrl_c_prints_stopped_watching. Non-vacuity: the test name appears 13× acrossthose logs, every occurrence
ok— the harvest was reading logs that do mention the test.So #129's two surfaces were latent, not actively firing, in the observed window; they are
closed by construction (the #323 ordering, the joined drain) and now pinned by the control
test, which mutates RED (
left: None right: Some(2)) when the control arm is turned intoa second treatment arm.
JS HMR half of #318 — no follow-up issue warranted
ci.yml, ~300-run window, 2026-07-24 → 2026-09-09: 69 red runs, of which 14 had afailing
JS packages — build & test (ubuntu-latest)job. 0 of those 14 show anyhmr-e2e/waitFor/waitForContentfailure signature; the HMR subtests appear164× in those logs and every occurrence is
ok. Filing a follow-up would be filingagainst a defect no run has exhibited. Deferred, not filed.
R4 — the one RED that reproduced locally
spawn_watch_readygenuinely deadlocked on 512 KiB of piped stdout, failing with the full10 s
READY_TIMEOUTand an empty stderr. Deterministic, on macOS, before the fix.Local gates (this head)
cargo test --doc -p mds-cliis N/A:error: no library targets found in package mds-cli. It is a binary crate with no doctests — not a regression.Snyk
No Snyk code scan ran. The Snyk MCP server failed to start for the entire session
(
ENOENT, missingsnyk-macos-arm64wrapper binary). Thesecurity/snyk (dean0x)checkon this PR is SCA-only ("No manifest changes") — it does not cover the Rust source
changed here. Stated rather than implied. This is CLI test-harness and watch-loop Rust
with no new dependencies.
Harness mutation controls (H1–H13)
Each control is a scratch edit applied with a uniqueness-checked script (exactly one
occurrence or abort), run against only the named tests, restored by
cpfrom apristine copy taken before the pass, and verified with
cmp—RESTORED_OKfor all ofthem — plus
git status --porcelainback toM .gitignoreonly after each. Theharness half of this PR (D2–D5) landed with RED/GREEN pairs but not with per-invariant
mutations; this is that table.
Six are RED, six are GREEN-on-macOS-by-mechanism, two are weak. Every outcome is
recorded, including the ones that did not fail — a control never observed RED proves
nothing about the test it is supposed to validate, so the GREENs carry their cause.
write_atomicas copy+remove: write temp,fs::copy(tmp, path),remove_file(tmp)— norename5 tests run: 5 passed, 74 skipped×3 (0.876 / 0.932 / 1.083 s)std::env::temp_dir()instead of the target's directory3 tests run: 3 passed, 76 skipped(0.967 s)$TMPDIRand the fixtures'tempfile::tempdir()are both on/dev/disk3s5on this machine, so the rename stayed intra-filesystem and could not fail. The cross-device failure this control is meant to provoke is unreachable locally.is_content_eventrejectsModify(Name(_))(extra arm returningfalse)2 tests run: 2 passed, 77 skipped(0.789 s)eprintln!of every kind reachingis_content_event, with R1's final assert forced to dump) shows ONEwrite_atomicrename on macOS producesCreate(File)×4,Modify(Data(Content))×4,Modify(Metadata(Any))×1 andModify(Name(Any))×3 — so dropping theNamevariant still leaves content events behind. Under Linux/inotify the destination arrives asIN_MOVED_TOalone, where this mutation is expected RED..tmp-<pid>-<seq>.<name>(suffix BEFORE the name)collect_mds_files_ignores_write_atomic_temp_namescommon/mod.rs:121:assertion left != right failed: write_atomic temp name must never end in .mds; it would be collected as a source/left: Some("mds")/right: Some("mds"). Theoutput.rsunit test stayed GREEN (1 test run: 1 passed, 833 skipped) — it asserts on literal names, so it is not the instrument for the shape; thedebug_assert_ne!is.PipeTap::finishreturnsself.bytes()WITHOUT joining (join block deleted, kill kept)stderr_tap_finish_captures_every_line_the_child_wrote×3;watch_clear_non_tty_no_ansi_escape×32 tests run: 2 passed, 77 skipped×3 (0.850 / 0.813 / 0.686 s)child.kill_and_wait()deleted fromfinish(join kept)stderr_tap_finish_captures_every_line_the_child_wrotetimeout(1)is not present on macOS and nogtimeout; the bound was a background run plus a bounded monitor rather than an exit-124 wrapper.)spawn_watch_unsynchronizedreturnsNone,spawn_watch_readytakes and drains stdout only once the marker is seenwatch_ready_with_large_piped_stdout_does_not_deadlockFAIL [10.015s],common/mod.rs:386:mds watch did not signal readiness within 10s; stderr so far was:(empty). The fullREADY_TIMEOUT, exactly the pre-fix signature.spawn_ready'sassert!(stdout_tap.is_none())removed3 tests run: 3 passed, 76 skipped(1.951 s)spawn_readywith a pipe, so nothing exercises the guard. It is a misuse barrier for future authors, not a pinned property, and this control cannot be strengthened without adding a test whose subject is the misuse.watch_readiness_handshake_makes_ctrl_c_exit_deterministicspawned withspawn_readycli_watch.rs:4213:left: None/right: Some(2),got ExitStatus(unix_wait_status(0)). Re-confirms the B3 observation on the current head.emit_ready_marker()moved BEFOREctrlc::set_handler(...)inrun_watch_file1 test run: 1 passed, 78 skipped(0.956 s) — the inverted-order window is onetx.clone()plus oneset_handlercall, so 20 iterations do not land in it on macOS.set_handlercli_watch.rs:4243, treatment arm:got ExitStatus(unix_wait_status(2)).watch_bare_filename_from_cwd_succeedsreverted tospawn_watch_unsynchronized+ a 10 s artifact poll1 test run: 1 passed, 45 skipped×3 (0.474 / 0.473 / 0.464 s)wait_for_stderr_countmutated to RETURN the text on timeout instead of panickingcli_watch.rs:4314:expected at least 3 occurrences of "warning: key 'x' is set more than once in vars file …" within 2s; saw 2. (b) RED at the exact-count assert instead —cli_watch.rs:4592:left: 2/right: 3, after 2.525 s.write_atomicreverted to plainstd::fs::write1 test run: 1 passed, 78 skipped×3 (0.480 / 0.503 / 0.494 s)What the GREENs mean. H1, H3, H5, H10, H11 and H13 all mutate invariants whose
failure signature is a Linux/inotify timing or event-shape property; macOS either
cannot produce the signature (H3 — observed, above) or does not produce it at this
window size (H10, rescued by H10b). They are recorded as macOS-GREEN and the Linux soak
is named as their instrument, rather than being presented as passing controls. H2 and H8
are weak for structural reasons stated in their rows, not for want of running them.
Local gates (final head)
Run after every mutation control was restored and verified, on head
b537078(
d4bdd2c+ the docs commit).cargo test --workspaceis CI's command and is includedhere in full — it did not stall locally.
cargo test --doc -p mds-cliremains N/A (no library targets found in package mds-cli) — a binary crate has no doctests. Not a regression.Pending
workflow_dispatchrequires the workflow to existon the dispatched ref, so both runs happen after merge: pre-fix on
--ref ci/watch-soak-workflow(=a8faafb), post-fix on--ref main. Run ids to berecorded here.
Known limitations
events under directories that are filtered only afterwards, so
npm installchurn candelay a real edit — and the idle tick behind it — by up to the cap (
max(10 × window, 1 s)). Documented in the CHANGELOG bullet; not fixed here, because filtering inside thewindow means re-deriving
files_of_interestper message.--debounce 0tearing is still reachable by real users. An in-place editor save canbe compiled as its 0-byte truncate intermediate, leaving a transiently empty output. The
harness stops producing it; the product does not. Filed as
mds watch --debounce 0: an in-place editor save can compile the 0-byte truncate intermediate and leave a transiently empty output #380.wait_for_stderr_contains_strstill returns its partial text on timeout instead ofpanicking, so a caller that does not assert on the returned text silently tolerates a
timeout. Left as-is; filed as cli_watch harness:
wait_for_stderr_contains_strreturns the partial text on timeout, so count assertions built on it are vacuous #381.#[cfg(unix)]) — SIGINT has no Windowsanalogue, so the property is not asserted there.
PipeTap::finish's single-shot-under-clone invariant is documented, not tested. Aclone calling
finishconcurrently blocks on the drain slot and then observes a fullydrained buffer; nothing pins that.
drained paths differ from the first batch, and no ctrl-c test that runs with coalescing
on.
== 1Recompiledcount is cut off early.finish_textreaps thechild immediately after the first
Recompiledline, so the count cannot observe asecond rebuild that would have followed. The discriminating assertion in that test is
the
v12final-content check, not the count — the count alone would pass a watcherthat rebuilt again a moment later.
spawn_ready/spawn_unsynchronizedpanic on stdout misuse AFTER the spawn. Thechild is already running when the assertion fires, and the panic unwinds before a
ChildGuardis constructed, so the misuse path leaks that child until the test processexits. Harness-only, and only on a misuse no test commits (see H8), but the guard is
placed one step later than it should be.
write_atomic's temp-name check is adebug_assert_ne!. It holds in practicebecause tests are debug builds, but it is not a release-build guarantee. The property
it guards is independently pinned by
collect_mds_files_ignores_write_atomic_temp_names— though note from H4 that the unit test uses literal names and so does not fail when
the generator's shape changes; only the
debug_assert_ne!catches that.interpolates
x, so the vars-file rebuild produces byte-identical output and thecontent-changed gate reports nothing. A comment above the write now says so. The
robust redesign (assert the delta across the source edit) is left for a later phase.
Changes
crates/mds-cli/src/watch.rs—drain_debouncereturnsDebounceOutcome { paths, end };quiet-period reset,
debounce_cap,clamp_debounce,MAX_DEBOUNCE_MESSAGES;assert!on the computed deadline; module doc gains a# Coalescingsection and thebounded-loop invariant now names both debounce bounds; unit tests for every pure part.
crates/mds-cli/src/main.rs—--debounceclap help rewritten (quiet period, cap,clamp,
0disables).crates/mds-cli/src/output.rs— unit coverage for the atomic-write shape.README.md—--debounceblock rewritten in the style of the--poll-intervalblockbelow it.
crates/mds-cli/tests/common/mod.rs—write_atomic;ChildGuardmoved here;PipeTap { bytes, text, finish, finish_text }withStderrTap/StdoutTapaliases;both
spawn_watch_*return(Child, StderrTap, Option<StdoutTap>)and drain stdoutbefore the readiness wait.
crates/mds-cli/tests/cli_watch.rs— 45 writes converted towrite_atomicunder adocumented rule (2 documented exceptions); 13 post-kill flush sleeps deleted;
spawn_ready/spawn_ready_piped_stdout/spawn_unsynchronizedwrappers;wait_for_stderr_count; the burst test rewritten and a cap test added; the test: flaky cli_watch.rs::watch_ctrl_c_prints_stopped_watching (timing) #129 two-armcontrol test and its
wait_boundedhelper; i18 comment explaining its exact count.79 tests.
crates/mds-cli/tests/cli_build.rs—watch_bare_filename_from_cwd_succeedssynchronised on the handshake, poll loop and private
ChildGuarddeleted, stderrdrained. 46 tests.
.github/workflows/watch-soak.yml— manualworkflow_dispatchLinux soak instrument(from the closed ci: add a manual Linux watch-soak workflow for the cli_watch flake family (#129, #318, #320) #375). Not a gate, not a required context, not release-surface;
publishes zero check-runs on any PR head.
CHANGELOG.md— one### Changedbullet formds watch --debounceis a fixed window from the first event; a burst longer than the window splits into N rebuilds #379 and seven### Internalbullets.d4bdd2cfix(watch): cover the Disconnected debounce exit (scrutinize)— unit testfor the
DebounceEnd::Disconnectedarm, which no test reached.b537078docs(watch): correct two C2 notes (scrutinize P2)— two documentationcorrections, no behaviour change. (a)
PipeTap's drain-slot comment claimed "severaltests hand a clone to a helper"; no call site clones a tap, so it now states what the
shape is actually for —
Cloneis harness API, and the mutex is what makes aconcurrent
finishfrom a clone safe. (b) the CHANGELOG's debounce "known cost" wasscoped to directory mode; verified against
drain_debounce(extends on any contentevent, relevance never re-derived) and the file-mode arming at
watch.rs:1327/:1422(parent directory,
RecursiveMode::NonRecursive), it applies in both — dir modediffers only in that an irrelevant event can also open a window, since file mode
checks relevance first.
Related Issues
Closes #318
Closes #320
Closes #129
Closes #379
Refs #317 #319 #321 #326 #380 #381
Supersedes #375