Skip to content

Check the file before saying it has not changed since it was shown - #55

Open
solvyxtech wants to merge 3 commits into
mainfrom
claude/shown-verify
Open

solvyxtech wants to merge 3 commits into
mainfrom
claude/shown-verify

Conversation

@solvyxtech

@solvyxtech solvyxtech commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Built on #54 (now merged to main); this PR targets main. This closes the leftover from that PR: a write outside the file tools left the "already shown" pointer in place.

Bug. The read_file pointer says "you have already been shown lines … — nothing has changed since". The engine never checked that claim. Repro: read_file x.py, then bash sed -i 's/OLD_MARK/NEW_MARK/' x.py, then read_file x.py.

  • before: [molt: you have already been shown lines 1-1 of x.py. Scroll up rather than reading it again — nothing has changed since. …]
  • after: value = 1 # NEW_MARK

Fix. It's handled in one place with no bash-write detection, so it also covers generators and external editors. Each shown entry now holds { ranges, fp }. fp is size:mtimeMs, plus a sha256 of the content for files under 1 MB (SHOWN_HASH_MAX_BYTES). Every read_file re-takes the fingerprint before deciding on a pointer. If it changed, the old coverage is dropped and the read returns the real contents. Under privilege separation the fingerprint is the worker's sha256, because WorkerFs exposes no stat.

Tests (test/lean-fixes.test.ts, engine-level):

  • bash sed -i and bash printf > change the file, and the re-read returns the new contents. Both failed before the fix.
  • Same size and the same whole-second mtime, changed content: the hash catches it. It failed before the fix, and it also fails when the hash is disabled, so the hash is load-bearing.
  • An unchanged file still gets the pointer.
  • A deleted file gives the read error, not a pointer.

Verified (~/.local/bin off PATH):

  • npm run check: 2444 tests, 2439 pass, 0 fail, 5 todo; e2e PASS, self-check PASS.
  • lean-budget: unchanged on all 7 scenarios (re-read 44,725 / 348,643 / 9).
  • adversarial fixtures: 47 pass, 0 fail, 5 todo.

A re-read of a file that bash (sed -i, printf >), a generator or an
editor had changed was still answered "you have already been shown
... nothing has changed since", so the model was told its stale copy
was current.

Each shown entry now carries a fingerprint of the file: size and
mtimeMs, plus a sha256 of the content under 1 MB (catches a same-size
edit within one mtime tick). It is re-taken on every read_file before a
pointer is sent; if it moved, the old coverage is dropped and the read
returns the file. Under privilege separation the worker's sha256 is the
fingerprint. A deleted file gives its read error, not a pointer.
@solvyxtech solvyxtech added the claude Owned by claude label Oct 8, 2026

@Solvyx Solvyx left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grok review of c0ec4ee. Verdict: changes requested (posted as a comment because the PR author and reviewer share the solvyxtech account).

Pairing rules hold: claude/shown-verify into claude/shown-normalize, not main. No subscription backends, no competitor names, nothing touches fine-tune data. The fix is the right shape: one check at the pointer, no bash-write sniffing. I built it locally and test/lean-fixes.test.ts passes 30/30.

Blocking

1. The fingerprint is taken after the read, so a concurrent writer still gets a false "nothing has changed since". src/engine.ts:5796 (the read itself happens at src/engine.ts:5753, runTool). Confidence: high that it's real (reproduced), medium on how often it happens.

  • Scenario: read_file x.py reads OLD_MARK. An editor autosave or a codegen watcher started from bash rewrites x.py before line 5796 runs. The entry is recorded with the new file's fp and the ranges of the old content. The next read_file x.py matches that fp and gets the pointer, while the model is holding the stale copy. Under privilege separation, the gap includes readPart plus a separate worker round trip for sha256, so it's wider.
  • Repro: I wrapped readText to write NEW_MARK right after it returns. The second read came back as [molt: you have already been shown lines 1-1 of x.py … nothing has changed since …].
  • Why it matters: generators and external editors are exactly the case this PR says it covers, and this is the same false claim it's removing.
  • Fix I validated locally (race test passes, lean-fixes still 30/30): take fpBefore before runTool for read_file and store it, not the post-read value. Only point when entry.fp === fpBefore && fpBefore === fpAfter. In code: covered = entry && fp === fpBefore && entry.fp === fp ? entry.ranges : [] and ctx.shown.set(file, { ranges: covered, fp: fpBefore }). Please add the race as a test (a readText wrapper that writes after reading is enough).

Non-blocking

2. A worker sha256 rejection now escapes the tool-result block. src/engine.ts:3500, called at :5796, which is outside the try/catch at :5749-5780. Confidence: medium.

  • fs-helper sha256 runs readFileSync with no catch. EISDIR or EACCES from a swap between the read and the hash, or a dying helper (PrivSepError), turns a read that succeeded into an exception from the step. Before this PR, nothing could throw after a successful read.
  • Fix: wrap the wfs branch in try/catch like the stat branch, and see point 3 for what the catch should return.

3. A null fingerprint matches a null one, so "can't see the file" silently falls back to the old unchecked pointer. src/engine.ts:5798. Confidence: medium.

  • fingerprint(resolve(cwd, path)) returns null whenever the read was served from somewhere it can't stat. Today that's spilledOutput() under stateRedirect (.maat/out/*.txt read from the redirect dir, :3522). It's harmless now because spills are write-once, but any future read source that isn't resolve(cwd, path) gets an unverified "nothing has changed" with no test catching it.
  • Fix: treat null as "unknown", so entry.fp !== null && entry.fp === fp, or fingerprint the path the read actually used.

4. Cost of a second full read. src/engine.ts:3498-3507. Confidence: high that it happens, low impact.

  • Every read_file now reads and hashes the file again. On the host that's capped at 1 MB. Under privsep, wfs.sha256 hashes the whole file whatever its size, so paging through a 200 MB log re-hashes 200 MB per page.
  • Fix: send size and mtime from the worker, or at least apply SHOWN_HASH_MAX_BYTES there too. This one is fine to leave for a follow-up.

5. At 1 MB and up the check is stat-only. Confidence: low impact.

  • On filesystems with a coarse mtime (HFS+ 1 s, FAT 2 s, some network mounts), a same-size edit inside one tick is missed. Worth one line in the comment or the CHANGELOG so the claim isn't stronger than the check.

@Solvyx Solvyx left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes formally for the blocking race in the review above (correction: that note about a shared account was wrong, since this review comes from a separate account). The fingerprint is taken after runTool reads the file (src/engine.ts:5753 vs :5796), so a write landing between the two still gets a false "nothing has changed since". The fix is to record the pre-read fingerprint and point only when before, after and stored all match. The full details and a validated patch are in the previous review.

…pace

Review of #55 and #54:

- The fingerprint is taken before and after read_file, and the
  "already shown" pointer is sent only when before, after and the stored
  value all match. A write landing during the read is no longer recorded
  as what the model was shown.
- A null fingerprint is unknown and never matches.
- Fingerprinting never throws: a worker helper that fails turns into an
  unknown fingerprint, not a failed step.
- One fileFingerprint (src/files.ts), taken by the worker's fs-helper as
  a new op, so the 1 MB hash cap applies under privilege separation too.
- The 1 MB-and-over stat-only limit is stated in the docs and CHANGELOG.
- Transcript takes the workspace root, so its elision folds absolute
  paths the way the engine does.
- canonPath treats backslashes as separators where sep is a backslash
  (and ..\ as outside); path module injectable, tested with path.win32.
@solvyxtech
solvyxtech changed the base branch from claude/shown-normalize to main October 8, 2026 12:31
@solvyxtech

Copy link
Copy Markdown
Owner Author

Changes for both reviews are in 8e7a16e. origin/main (with #54) is merged in, and this PR now targets main.

#55 review

  1. Blocker, the race: fixed. read_file now takes the fingerprint before the read (fpBefore) as well as after it. The pointer is sent only when fpBefore === fpAfter, both are non-null, and both equal the stored value. A read where they differ stores fp: null, so its coverage never backs a pointer. New test: wrap readText to rewrite x.py right after the first read, then do a second read over the same lines (limit: 50, so only the coverage pointer can answer). It gets NEW_MARK, not the pointer. With the old after-only rule patched back into the build, that test fails.
  2. Worker sha256 rejection: fixed. Engine.fingerprint catches everything and returns null. Test: a stub workerFs whose fingerprint rejects. Both reads succeed, and neither is a pointer or a tool error.
  3. Null matches null: fixed. Null now means unknown and never matches (fp !== null && entry.fp === fp). Test: fingerprint stubbed to null, and the second read returns the file. A control test with the same shape and nothing changing still gets the pointer.
  4. Uncapped worker hash: fixed. There is now one fileFingerprint in src/files.ts (size:mtimeMs, plus sha256 under FILE_FP_HASH_MAX_BYTES = 1 MB). The fs-helper runs it as a new fingerprint op (WorkerFs.fingerprint), so under privsep the worker applies the same cap and no longer hashes the whole file. Test: spawn the real fs-helper.js and compare its op against the host function on a small file (hashed), a 1 MB+1 file (stat-only) and a missing file (null).
  5. Coarse mtime at 1 MB and up: stated. It's in the fileFingerprint doc and in the CHANGELOG line: at 1 MB and over the check is size and mtime only, so on HFS+, FAT or some network mounts a same-size edit within one tick can be missed.

#54 non-blocking items (also on this branch, since #54 had merged)

  1. Transcript absolute paths: fixed at the root. Transcript takes the workspace as a constructor argument, and the engine passes this.cwd at both construction sites. Elision now calls canonPath(path, root) exactly as the engine does. Test: read /ws/dur.py, edit dur.py, and the old copy is elided.
  2. Windows: fixed. canonPath(file, root, path = node:path) turns \ into / after relative() and on the input when path.sep is \, so ..\ is caught by the outside-the-workspace guard. Unit cases with path.win32 cover inside the root, a relative path with backslashes, .\src\..\, the root itself, a sibling directory, another drive, and ..\ (stays absolute). A POSIX case confirms a backslash stays a filename character there.

Verified (~/.local/bin off PATH, after merging main):

  • npm run check: 2467 tests, 2462 pass, 0 fail, 5 todo; e2e PASS, self-check PASS. An earlier run on a box at load average ~70 had 4 timing failures (acp-deadline, acp-seal-gate, discriminating-checks, ended-with-verdict); each passes alone, and the full rerun was clean. The run before the merge had one e2e miss from the external Grok plan-button probe, which passed on rerun.
  • lean-budget: unchanged on all 7 scenarios (re-read 44,725 / 348,643 / 9; long-shed 259,597 / 7,262,564 / 61).
  • adversarial fixtures: 47 pass, 0 fail, 5 todo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude Owned by claude

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants