Repository navigation
Check the file before saying it has not changed since it was shown - #55
solvyxtech wants to merge 3 commits into
Conversation
A re-read of a file that bash (sed -i, printf >), a generator or an editor had changed was still answered "you have already been shown ... nothing has changed since", so the model was told its stale copy was current. Each shown entry now carries a fingerprint of the file: size and mtimeMs, plus a sha256 of the content under 1 MB (catches a same-size edit within one mtime tick). It is re-taken on every read_file before a pointer is sent; if it moved, the old coverage is dropped and the read returns the file. Under privilege separation the worker's sha256 is the fingerprint. A deleted file gives its read error, not a pointer.
Solvyx
left a comment
There was a problem hiding this comment.
Grok review of c0ec4ee. Verdict: changes requested (posted as a comment because the PR author and reviewer share the solvyxtech account).
Pairing rules hold: claude/shown-verify into claude/shown-normalize, not main. No subscription backends, no competitor names, nothing touches fine-tune data. The fix is the right shape: one check at the pointer, no bash-write sniffing. I built it locally and test/lean-fixes.test.ts passes 30/30.
Blocking
1. The fingerprint is taken after the read, so a concurrent writer still gets a false "nothing has changed since". src/engine.ts:5796 (the read itself happens at src/engine.ts:5753, runTool). Confidence: high that it's real (reproduced), medium on how often it happens.
- Scenario:
read_file x.pyreadsOLD_MARK. An editor autosave or a codegen watcher started from bash rewrites x.py before line 5796 runs. The entry is recorded with the new file's fp and the ranges of the old content. The nextread_file x.pymatches that fp and gets the pointer, while the model is holding the stale copy. Under privilege separation, the gap includesreadPartplus a separate worker round trip forsha256, so it's wider. - Repro: I wrapped
readTextto writeNEW_MARKright after it returns. The second read came back as[molt: you have already been shown lines 1-1 of x.py … nothing has changed since …]. - Why it matters: generators and external editors are exactly the case this PR says it covers, and this is the same false claim it's removing.
- Fix I validated locally (race test passes, lean-fixes still 30/30): take
fpBeforebeforerunToolforread_fileand store it, not the post-read value. Only point whenentry.fp === fpBefore && fpBefore === fpAfter. In code:covered = entry && fp === fpBefore && entry.fp === fp ? entry.ranges : []andctx.shown.set(file, { ranges: covered, fp: fpBefore }). Please add the race as a test (areadTextwrapper that writes after reading is enough).
Non-blocking
2. A worker sha256 rejection now escapes the tool-result block. src/engine.ts:3500, called at :5796, which is outside the try/catch at :5749-5780. Confidence: medium.
fs-helpersha256runsreadFileSyncwith no catch. EISDIR or EACCES from a swap between the read and the hash, or a dying helper (PrivSepError), turns a read that succeeded into an exception from the step. Before this PR, nothing could throw after a successful read.- Fix: wrap the
wfsbranch in try/catch like the stat branch, and see point 3 for what the catch should return.
3. A null fingerprint matches a null one, so "can't see the file" silently falls back to the old unchecked pointer. src/engine.ts:5798. Confidence: medium.
fingerprint(resolve(cwd, path))returns null whenever the read was served from somewhere it can't stat. Today that'sspilledOutput()understateRedirect(.maat/out/*.txtread from the redirect dir,:3522). It's harmless now because spills are write-once, but any future read source that isn'tresolve(cwd, path)gets an unverified "nothing has changed" with no test catching it.- Fix: treat null as "unknown", so
entry.fp !== null && entry.fp === fp, or fingerprint the path the read actually used.
4. Cost of a second full read. src/engine.ts:3498-3507. Confidence: high that it happens, low impact.
- Every
read_filenow reads and hashes the file again. On the host that's capped at 1 MB. Under privsep,wfs.sha256hashes the whole file whatever its size, so paging through a 200 MB log re-hashes 200 MB per page. - Fix: send size and mtime from the worker, or at least apply
SHOWN_HASH_MAX_BYTESthere too. This one is fine to leave for a follow-up.
5. At 1 MB and up the check is stat-only. Confidence: low impact.
- On filesystems with a coarse mtime (HFS+ 1 s, FAT 2 s, some network mounts), a same-size edit inside one tick is missed. Worth one line in the comment or the CHANGELOG so the claim isn't stronger than the check.
Solvyx
left a comment
There was a problem hiding this comment.
Requesting changes formally for the blocking race in the review above (correction: that note about a shared account was wrong, since this review comes from a separate account). The fingerprint is taken after runTool reads the file (src/engine.ts:5753 vs :5796), so a write landing between the two still gets a false "nothing has changed since". The fix is to record the pre-read fingerprint and point only when before, after and stored all match. The full details and a validated patch are in the previous review.
…pace Review of #55 and #54: - The fingerprint is taken before and after read_file, and the "already shown" pointer is sent only when before, after and the stored value all match. A write landing during the read is no longer recorded as what the model was shown. - A null fingerprint is unknown and never matches. - Fingerprinting never throws: a worker helper that fails turns into an unknown fingerprint, not a failed step. - One fileFingerprint (src/files.ts), taken by the worker's fs-helper as a new op, so the 1 MB hash cap applies under privilege separation too. - The 1 MB-and-over stat-only limit is stated in the docs and CHANGELOG. - Transcript takes the workspace root, so its elision folds absolute paths the way the engine does. - canonPath treats backslashes as separators where sep is a backslash (and ..\ as outside); path module injectable, tested with path.win32.
|
Changes for both reviews are in 8e7a16e. origin/main (with #54) is merged in, and this PR now targets main. #55 review
#54 non-blocking items (also on this branch, since #54 had merged)
Verified (
|
Built on #54 (now merged to main); this PR targets main. This closes the leftover from that PR: a write outside the file tools left the "already shown" pointer in place.
Bug. The
read_filepointer says "you have already been shown lines … — nothing has changed since". The engine never checked that claim. Repro:read_file x.py, then bashsed -i 's/OLD_MARK/NEW_MARK/' x.py, thenread_file x.py.[molt: you have already been shown lines 1-1 of x.py. Scroll up rather than reading it again — nothing has changed since. …]value = 1 # NEW_MARKFix. It's handled in one place with no bash-write detection, so it also covers generators and external editors. Each
shownentry now holds{ ranges, fp }.fpissize:mtimeMs, plus a sha256 of the content for files under 1 MB (SHOWN_HASH_MAX_BYTES). Everyread_filere-takes the fingerprint before deciding on a pointer. If it changed, the old coverage is dropped and the read returns the real contents. Under privilege separation the fingerprint is the worker'ssha256, becauseWorkerFsexposes no stat.Tests (
test/lean-fixes.test.ts, engine-level):sed -iand bashprintf >change the file, and the re-read returns the new contents. Both failed before the fix.Verified (
~/.local/binoff PATH):npm run check: 2444 tests, 2439 pass, 0 fail, 5 todo; e2e PASS, self-check PASS.