Repository navigation
[Bug]: ephemeral lock holder sidecars race directory walks and match *.json globs, causing Windows-only test flakes #6128
Description
Activity
Root cause found - and it is product-level, not just test flakiness. A third instance surfaced while chasing the
KeyError: binding_idclue from the PR #6075 review, and it pins the mechanism precisely.lock_holder_pathinloopx/file_lock.pyis Windows-only in its sidecar shape:def lock_holder_path(path: Path) -> Path: lock_path = _lock_path(path) if os.name == "nt": return lock_path.with_name(f"{lock_path.name}.holder.json") return lock_path
On Windows the holder record is written to a separate file ending in
.json(<locked>.json.lock.holder.json); on POSIX the record is written into the lock file itself, so no.json-suffixed sidecar exists - which is exactly why this class has never been seen on Linux CI.Product crash, third and cleanest instance:
ChatExternalConversations(Lark / native chat) admits each external request underexclusive_file_lock(<requests>/<ref>.json), and the holder sidecar for that lock lands in the same directory._admit_preparedthen callspending(), which is[_read_json(p) for p in sorted(self.root.glob("*.json"))]- it reads the holder sidecar of the request currently being admitted as if it were an external chat request and crashes:loopx/capabilities/native_chat/external_conversations.py:87: KeyError: 'binding_id'Reproduced deterministically on native Windows (zh-CN, cp936, Python 3.12) at tree
07eb7a128viatests/extensions/test_lark_private_manager_returns.py::test_production_pump_returns_to_original_private_source_after_restart(8/8 parametrizations error in fixture setup).commission_evidencein the same module readsrow["binding_id"]overpending()and has the same exposure. This breaks chat admission on Windows wheneverpending()runs while any request file in the directory is locked - i.e., exactly the concurrent-admission case the fence exists to serialize.This upgrades the suggested directions: the cleanest fix is likely
lock_holder_pathreturning a non-.jsonname on Windows (the sidecar is lock metadata, not state), which would fix the test flakes from the original report and this admission crash in one place; product walkers that must stay.json-shaped should still filter the holder name defensively.- added a commit that references this issue
on Oct 11, 2026 Post-fix verification on current main (
8d5a6fc17, which includes the5347064ffix): the reproduction shape from your Minimal reproduction step 2 now comes back clean. While holdingexclusive_cross_runtime_file_lockin a loop,glob("*.json")no longer matches the holder sidecar (it is nowstate.json.lock.holder, no.jsonsuffix), and a concurrent directory walker survived a 20-second run — 6,557 read attempts, zeroFileNotFoundError, zeroPermissionError.tests/test_file_lock.pyon this head: 20 passed, 3 skipped (Windows, Python 3.14.2). The legacy*.lock.holder.jsonread-compat path is a thoughtful touch for upgrade windows.One disclosure, since it belongs on the record: we hit this same family on 2026-10-04 — a bare
glob("*.json")intest_reply_delivery_does_not_replace_receiver_dispositioncounted one real turn file plus two lock sidecars as three items. We sandbox-verified a test-side filter, deferred a small PR until after our #5554 landed, and then failed to follow up — our miss. Your report framed the class properly, and direction 1 was the right call.Thanks for the clean report and the fix.
- added a commit that references this issue
on Oct 12, 2026
Preflight
Issue origin
Observed or reproduced in a real environment
LoopX version or commit
Observed on trees near
641855d9candc2b17b3e(October main); the sidecar mechanism itself is older (loopx/file_lock.py,LOCK_HOLDER_SCHEMA_VERSION = "file_lock_holder_v0").Host or runtime surface
Not host-specific (reproduced on native Windows 10, zh-CN, cp936)
LoopX area
Build, packaging, installer, or CI
Problem
exclusive_cross_runtime_file_lockwrites an ephemeral holder sidecar next to the locked file (lock_holder_path:<locked>.lock.holder.json). Two properties of that sidecar combine into a class of Windows-only test flakes in suites that walk runtime state directories:.json, soglob("*.json")and directory walks that collect JSON state files match it even though it is not state.read_bytes()withFileNotFoundError, or assert against a directory snapshot that still contains it.Two concrete reproductions from reviewing current PRs (both native Windows, cp936):
tests/test_loopx_turn_executor.py::test_reserved_managed_start_recovers_after_death_before_the_first_journal_write- the "no journal written" assertion at the death point usesglob("*.json")and matches the persisted...turns/<hash>.json.lock.holder.json; after a (simulated) crash the holder legitimately survives, so the assertion trips even though no Turn journal was written.tests/test_peer_collaboration.py::test_canonical_stopped_goal_rejects_peer_request_before_any_write- a helper that walks.local/manager-context/entries/**collecting{relative_path: read_bytes()}listed a....lock.lock.holder.jsonand raisedFileNotFoundErrorwhen the sidecar was deleted before the read. Flaky across runs (passes on rerun).Both are test-side failures, but the same walk pattern exists in product-adjacent code paths that enumerate runtime directories, so the class is broader than these two rows.
Minimal reproduction
python -m pytest -q tests/test_peer_collaboration.py::test_canonical_stopped_goal_rejects_peer_request_before_any_write) - repeat until the walker lists a.lock.holder.jsonthat disappears before its read.exclusive_cross_runtime_file_lockover a file in a loop and, from another thread, walk the directory - observe transientFileNotFoundErroron....lock.holder.jsonand*.jsonglob matches.Expected behavior
Directory walks over runtime state should be deterministic: either the holder sidecar is not named like a state file, or walkers (and the tests that assert on walks) skip/ignore it consistently on every platform.
Actual behavior
On Windows the sidecar is caught mid-lifecycle: glob assertions see persisted holders after simulated crashes, and walkers raise
FileNotFoundErroron holders deleted between listing and reading.Sanitized diagnostics
Additional context
Possible directions, maintainer preference welcome:
.json(it is lock metadata, not state), or move holder files into a dedicatedlocks/subdirectory;.lock.holder.jsonand tolerates per-entryFileNotFoundError, and use it in the two affected suites;Happy to implement whichever direction is preferred.