[HouMinXi]

Refs #264

Problem

_loads_model_json repairs unescaped inner quotes. When a reply quotes a dict literal without escaping it, the repair can read part of the quote as structure. A finding whose excerpt is cfg = {"a": "b", "c": "d"} parses as:

{"excerpt": "cfg = {\"a\": \"b", "c": "d\"}", ...}

The parse succeeds, the excerpt is cut, and the finding gains a key c. Nothing downstream notices.

Change

Option (A) from the issue: callers say which keys their reply may have.

• llm_invoke takes reply_keys: ReplyKeys | None. ReplyKeys.top lists the top-level keys. ReplyKeys.items[name] lists the keys allowed in each object under that member, whether the member holds one object or a list of them. When the reply is a top-level list, each object in it is checked as a reply, because the runtime parser unwraps a one-object list.
• In _loads_model_json, a parse that only succeeded after repair is checked against reply_keys. If it carries an unlisted key, the repair is thrown away and ReplyKeyRejected is raised, with the original error's message and position. A parse that needed no repair gets no key check, so a model adding its own keys to valid JSON behaves as before. A duplicate key is refused in every parse, with or without keys (see Duplicate keys).
• Any other failure after a repair was tried raises RepairRefused. That covers a repair that still does not parse and the llm: bound the unescaped-quote repair to content quotes #216 quoted-key refusal. RepairRefused is a JSONDecodeError, and ReplyKeyRejected is a subclass of it. A failure where no repair was tried stays a plain JSONDecodeError.
• What happens after the error:
  • API backends: the reply goes to the one JSON-correction retry, or to the truncation path if the JSON was cut off. There is no new retry. All three fallback sites (the API parse, the combined continuation parse and the wider retry) go through _fallback_reply, which picks the scan by error type (see below).
  • CLI backends: a refused repair, a refused key or a duplicate key fails the call with LLMInvokeError, in bare stdout, a type: result envelope, a stream array, or a string or object result (F1). stdout is parsed once. A result string that fails with no repair tried stays a raw string, as before, and the caller's validation rejects it. An object result is key-checked only when the envelope parse needed quote repair; a duplicate key in it is always refused.
• reply_keys is threaded through every parse in the call: _invoke_cli, _invoke_api, _continue_truncated, _retry_with_more_headroom and _repair_missing_excerpts.

Callers that now pass keys:

Caller	Keys
L1 review passes (factories.build_l1_provider), subagent spawn and the test-assertion review (cli)	REVIEW_REPLY_KEYS: top findings, code_excerpts, context_quotes. Findings may carry file, line, severity, description, excerpt and a nested code_excerpts.
Runtime axis (runtime.RuntimeRunner)	RUNTIME_REPLY_KEYS
Real falsifier (falsify_real.RealFalsifier)	_FALSIFY_REPLY_KEYS, including receipt.command and receipt.output
Inline canary review (cli._dispatch_inline_canary)	CANARY_REVIEW_REPLY_KEYS: top findings only. A model may add its own finding keys (title, fix, confidence), which the canary validator ignores, so items are not checked.
Daemon-state questions (daemon_state.DaemonStateRunner)	Q1_REPLY_KEYS: top external_state. Q2Q3_REPLY_KEYS: top conflicts only. Conflict items are not checked, since a model may add its own item keys.

Not covered

Remaining #264 work:

• Required keys. Only unknown keys are checked. A repair that drops or renames a required key is left to each caller's own validation.

• Callers that pass no keys. Four remain, and their behaviour is unchanged:

  • the backend probe (backend.py:1136), which asks for {"ok": true};
  • the spec summary (contract_loader.py:265), which asks for {"summary": ...};
  • contract summarization (cli.py:3284), which takes free text;
  • canary generation (cli.py:3546), which returns mutations.

  None of these replies becomes a finding. Each can opt in later with its own ReplyKeys.

• Deeper levels. Levels below items, and members not listed in items, are not checked. The llm: bound the unescaped-quote repair to content quotes #216 quoted-key guard still applies there. One known case: an extra key in an item of a finding's nested code_excerpts is accepted, and _hoist_nested_excerpts then moves that item to the root.

• Same-level key collisions. A repair whose invented key is also an allowed key at that level (an excerpt that quotes "file": ..., say) passes the check.

• The CLI wrapper envelope. The keys of a type: result envelope are not checked. That JSON comes from the CLI, not the model. A string result is always checked, and an object result for unknown keys only when the envelope needed quote repair. A duplicate key anywhere in the envelope fails the call. A bare stdout with no envelope is checked.

• Callers that pass no keys, on cut or duplicated replies. Correction: the previous description listed two cases here, a cut code_excerpts-first reply that quotes a review and a falsify reply with a quoted second "verdict". Both are now refused without keys too: _loads_model_json refuses a duplicate key in every parse, and the repair of the first case repeats file, start_line, end_line and content.

• Canary and Q2Q3 items. The canary review and Q2Q3 check top-level keys only. A repair inside a finding's or a conflict's string can still invent an item key there, as on main. Checking items would refuse a repaired reply whose model added its own item keys.

• The review-focus line. With a focus spec, the review prompt says "in your response, state whether each area was checked" (factories.py around line 532, cli.py around line 1193) without naming a key. A model that answers with a new top-level key in valid JSON is accepted, since unrepaired JSON is not checked. If that reply also needed quote repair, the extra key is refused and the reply goes to the correction retry. Naming a key (or listing one in REVIEW_REPLY_KEYS) is left to a follow-up.

• The excerpt-repair prompt. _excerpt_repair_prompt echoes parsed["findings"] back to the model. Those findings come from a parse that either needed no repair or passed the key check, so a repaired, unchecked object is never echoed. Values are not checked, only keys.

• Nested code_excerpts. See Deeper levels: items in a finding's nested code_excerpts are not key-checked.

The embedded-object scan

_extract_json_from_text does no quote repair, so it cannot invent a key. It does scan every { in the text, though, and returns the first object that has the reply's keys. After a failed repair, that object can be one quoted inside an excerpt.

In the cross-check reproduction, a P1 finding's excerpt and a code_excerpts item quote x = {"findings": [], "code_excerpts": [...]} unescaped. Correction: an earlier version of this description said the reproduction needed a second literal, ; y = {...}, to trigger the bug. That was wrong. The excerpt alone already moves content, start_line and end_line onto the finding when no keys are passed, and raises ReplyKeyRejected under REVIEW_REPLY_KEYS. The ; y literal was only needed for a variant with an empty code_excerpts.

Other tails after the same excerpt lose the P1 with or without keys, on main too. Examples are ; f("a", "b"), y = ["a","b"] and print("], ", "x"), and the #216 quoted-key refusal on a description of returns, ",", ok, empty. The repair fails, the scan finds the quoted {"findings": [], ...}, and llm_invoke returns a clean review after one call.

_fallback_reply now does this:

• After ReplyKeyRejected, it tries no object.
• After RepairRefused, it tries only the object that opens the reply, after any [ and whitespace (leading_only=True). Every later { may sit inside that object's quoted content. This keeps the falsify salvage: a cut falsify reply with an unescaped quote still yields its verdict.
• After a plain error, it runs the full scan, with the rule below.

I also tried two simpler rules. Skipping the scan on every RepairRefused lost the falsify salvage above. Skipping it whenever the reply starts with { would drop the recovery of an object followed by prose. Correction: an earlier version of this description said that rule broke test_f2_many_invalid_braces_before_real_envelope. That test calls the extractor directly, so it does not cover _fallback_reply. Inside _fallback_reply, only test_failed_repair_still_salvages_the_leading_object fails under that rule. Object-then-prose recovery had no test; test_object_followed_by_prose_is_recovered now covers it.

Errors before any repair

A reply that fails before its first unescaped quote gets no repair, so its error stays plain and the full scan ran unchanged. A cross-check showed that this still returned a quoted object as a clean review after one call, with or without keys, through _invoke_api and llm_invoke:

• a reply that opens with prose (Here is the review: {...}), with or without a f("a", "b") tail;
• a syntax error ahead of the stray quote: Python True, or a trailing comma in an earlier finding;
• a reply whose code_excerpts come first, with an excerpt of x"}], "findings": []}. Here the leading object itself decodes as {"code_excerpts": [...], "findings": []}, and the error is Extra data.

Three checks in _extract_json_from_text now handle these. They replace the unmatched-} count from commits 9 and 10 and the code-mark list from commits 19 and 20.

1. A { that fails to decode can end the scan. The scan ends when that { opens the reply (after any [ and whitespace), or when the text from it to the error position holds a ". In both cases every later { may sit inside it. The falsify salvage is tried on that { first (falsify keys only, with the duplicate and verdict-first checks); if it yields nothing, the scan returns None. A { after prose that fails before reaching any quote ({a, {b}) is skipped as before, so "Note: " + "{a" * 10 before a real envelope still works. After prose, a failed Python dict or JS object literal ({'key': or {key:) is remembered: an object with the reply's keys inside its braces is not returned. A literal that closes before the reply has no effect. I first ended the scan at any failed literal, but that broke existing contract tests for prose with a stray literal before the real envelope.

2. The scan's decoder refuses duplicate keys, with or without reply_keys (_scan_decoder, object_pairs_hook=_refuse_duplicate_keys). A scanned object with a duplicate key ends the scan with None.

3. A matching object is not returned when the text after it refuses it. _text_after_refuses checks the text after an object with the reply's keys. It refuses, in this order, when:

   • the rest opens more { and [ than it closes;
   • a whole object right after it has one of the reply's keys or a duplicate key;
   • after any such whole objects, the text does not start like what follows a whole reply (_prose_may_follow, below);
   • further on, a whole object has one of the reply's keys or a duplicate key, or a " closes into JSON ("}, "], ": or ",) outside any whole object.

   The object's place is leading (it opens the text), list (it opens the text after a [) or prose (text comes before it). _prose_may_follow treats any str.isspace character as whitespace and accepts:

   • the end of the text (not in the list place);
   • after a newline, an unindented line that starts with markdown: a fence, ---, a table |, a footnote [1], a list or quote marker, a heading, or a number and a word. An indented line refuses;
   • a prose word: a letter, perhaps after a bracket, quote or emphasis mark, through to whitespace, prose punctuation or the end. e+1, f(a) and d.k are not words;
   • non-ASCII prose such as CJK text or an emoji, or a dash and a space. Smart double quotes, fullwidth ， and ； and a BOM do not count;
   • ., ! or ? that end the text, with no gap;
   • in the list place, only ] followed by any of the above, or , then [ or {. Nothing else, the end of the text included, since the list is still open;
   • in the prose place, also ] followed by any of the above, or , then [ or {;
   • in the prose place, with no gap: ,, ;, ., :, ! or ? then whitespace, followed by any of the above.

   Everything else refuses: code, a digit, a stray mark, a curly quote.

How the rule was chosen

The corpus and the measurement script are now committed (cr1, cr2 P3-4):

• tests/fixtures/reply_tail_corpus.json: 57 cut tails, 3 whitespace-only tails, 32 valid tails and 4 prose prefixes.
• tests/reply_tail_corpus.py: runs each text as _invoke_api does on a stop reply (_loads_model_json, then _fallback_reply if that raises) and prints the table below. Each REV argument is measured from git show as well.
• tests/test_reply_tail_corpus.py: pins the corpus size and the working-tree counts.

The cases:

• Cut replies: each cut tail follows {"code_excerpts": [...], "findings": []}, as left by an excerpt that quotes "}], "findings": []} and a stream that stops there. Each tail is tried in the three places (171 cases). A returned object loses the P1.
• Whitespace-only tails: nothing, \n and two spaces, in the three places (9 cases).
• Valid replies: a real reply with a P1, followed by each valid tail in the three places, plus the four prose prefixes with and without a trailing line (104 cases). A reply that does not come back with its P1 is refused; on the API path that costs the correction call.

Output of PYTHONPATH=src:. python tests/reply_tail_corpus.py 880b069 071a9ea at this head:

Rule, keys	Cut: P1 lost	Whitespace: P1 lost	Valid: refused
this push, no keys	21 of 171 (leading 8, list 0, prose 13)	6 of 9 (leading 3, list 0, prose 3)	12 of 104 (leading 4, list 4, prose 4)
this push, review keys	8 of 171 (leading 8, list 0, prose 0)	3 of 9 (leading 3, list 0, prose 0)	48 of 104 (leading 4, list 4, prose 40)
880b069 (code marks), either	156 of 171 (leading 44, list 56, prose 56)	9 of 9	14 of 104 (leading 5, list 4, prose 5)
071a9ea (previous push), either	29 of 171 (leading 8, list 8, prose 13)	9 of 9	12 of 104 (leading 4, list 4, prose 4)

With review keys, a prose-led reply gets no scan (F2), so all 40 prose-place valid cases go to the correction retry. Without keys, the scan still reads them as before.

Correction. The previous description had a table of candidate rules (B, C, D and their variants with the brace balance H). Those rows came from scratch code that I did not keep, and the corpus copy I had kept was damaged. I rebuilt the corpus from the tails the description lists. The 880b069 and 071a9ea rows reproduce the counts it gave for them (156 and 14; 29 and 12). The candidate rows cannot be re-run, so they are dropped.

The 8 cut tails lost with keys are done, (see a.py), \n- note, 'a', for k in d:, or y, in d and a lone !, all in the leading place. Without keys, the same 8 are lost after prose too, plus :, ;, . Then, , b) and ; done after prose. In the list place nothing is lost: a list must close with ] or go on with , and another object (cr3 P3-4). The 12 valid cases refused in every mode are four texts in each place: I would return {"findings": []}. and The format was {"findings": [...], ...}. (a later keyed object, or a quote closing into JSON), "Unchecked" means no guard. after a blank line (a double quote does not start prose) and use `d["k"]` instead ("]).

What a refusal costs

_json_correction_prompt now offers the correction retry when the content holds a { anywhere. It used to require content starting with { or [, so a refused prose-led reply failed at once. A BOM, a zero-width space and a Here: prefix before a broken object moved from the no-correction cases to the correction cases in test_json_recovery.py. If the retry is refused too, llm_invoke raises InvalidJSONResponseError. The L1 review passes (factories.py:656) record it as an INFRA, CONFIRMED l1-<pass>-invoke-fail finding and keep the raw response. Falsify (machine.py:2243) marks the finding UNCERTAIN as a protocol violation. Neither retries again. Both fail closed.

Cases that used to return an object and now cost a correction call, then fail closed if the retry is refused too:

• the 12 valid cases above without keys, and with keys any prose-led reply (40 of the 104 valid cases);
• a valid reply followed by a second object with one of the reply's keys, such as a model that corrects itself;
• a reply that opens with a { that does not decode, such as {a .... test_f2_many_invalid_braces_before_real_envelope keeps its intent behind a prose prefix, and the bare {a-led text now asserts None;
• a reply object followed by code.

Cut replies whose code_excerpts come first

A cross-check on 9888e6c found a reply that opens with code_excerpts, quotes "}], "findings": []} in an excerpt, and is cut right after the quote. The leading object decodes whole as an empty review, and the P1 after the cut is lost.

• _continue_truncated now always asks for the rest on length, even when the partial parses whole. A partial can parse whole because a quoted "}], "findings": []} closed it early. A continuation that adds nothing returns the partial: an empty one, or an echo of the partial's end (whitespace, fence marks, closing brackets). A continuation that restates the whole reply is read as that reply when it parses strictly. So a length cut whose partial was already whole costs one call more than before, two in all, and the P1 is kept; the tests assert both. Two existing tests that expected no continuation now expect one (test_a_complete_partial_still_gets_a_continuation and test_complete_json_partial_still_asks_for_a_continuation).
• On stop, the rule above decides.

Results per corpus tail through the stop path, with REVIEW_REPLY_KEYS and without keys. Through llm_invoke with a mocked API that returns the same text again, a refused reply ends in LLMInvokeError after the correction call:

Tail after the cut object	Place	stop, review keys	stop, no keys	length
the other 44 cut tails: code such as ,}, ]}, // done, ; f(a), \u007d, e+1, an indented line, fullwidth ， and ；	all three	refused	refused	continued
:, ;, . Then, , b), ; done	leading, list	refused	refused	continued
the same	prose	refused	empty review returned	continued
done, (see a.py), \n- note, 'a', for k in d:, or y, in d, !	leading	empty review returned	empty review returned	continued
the same	list	refused	refused	continued
the same	prose	refused	empty review returned	continued
nothing, \n, two spaces (the partial parses whole)	leading	empty review returned (it is a whole reply)	the same	continued
the same	list	refused	refused	continued
the same	prose	refused	empty review returned	continued

Python dict and JS object replies that quote a review, bare or after prose, and prose that quotes an example envelope before the real reply, are refused by the extractor and never return the quoted empty review.

Limit. One call can still return a wrong clean review. That needs a reply whose code_excerpts come first, an excerpt that quotes "}], "findings": []} unescaped, and a stream that stops right after the quote while reporting stop. With review keys, that happens only when the object opens the reply and the text left after the quote is one of the 8 prose-like cut tails above or whitespace only (3 cases). Without keys (the four callers listed under Callers that pass no keys), the same tails after prose, and five punctuation tails after prose, are lost too. A reply cut by the output cap reports length and is continued. A whole reply carries its own "}], ... after the quote, which refuses it. Refusing these tails on stop would refuse valid replies that end in prose. test_stop_cut_prose_like_tail_is_a_known_limit pins them.

Correction. The previous description stated this limit for a leading object only, by first character, and its table said prose-led replies and example envelopes fail closed. Both were wrong: the code marks lost the P1 for 56 of 57 tails in a list or after prose, and a prose-led reply got no correction retry and ended in an error after one call.

Trailing commas on Python 3.12

CI runs Python 3.12 (linux-tests.yml). There, ,} fails with "Expecting property name enclosed in double quotes" and ,] with "Expecting value", both at the closer. 3.13 and later report "Illegal trailing comma" at the comma. On 3.12 _error_is_at_quote read the closer as content and escaped an earlier quote. That failed three trailing-comma tests on 3.12 and could turn [{P1 ..., "description": "z",}, {P2}] into a review with only the P2. _error_is_at_quote now treats either message at } or ] as structure when the previous non-space character is ,, so no repair is tried on any version. The tests build the 3.12 and 3.13 errors directly, so they do not depend on the Python that runs them.

Duplicate keys

_loads_model_json parses with object_pairs_hook=_refuse_duplicate_keys, with or without reply_keys. A duplicate key raises DuplicateReplyKey (a ReplyKeyRejected), with the first repair error's position if a repair was tried, and the correction prompt names the key (Duplicate key "verdict" in reply). json keeps the last of two equal keys, so a quoted "verdict": "DISMISSED" after the real one would win, and a cut code_excerpts-first reply could be repaired into an item whose file, start_line, end_line and content come from the quote. Every later parse of reply text uses loads_reply_json, which applies the same hook (F1).

The embedded-object scan now uses the same hook, with or without keys (rule 2 above). A later whole object with a duplicate key also refuses an earlier match. A mutant that lets a duplicate through in a whole object right after the match survives. It is equivalent: that object's { does not start like prose, so _prose_may_follow refuses the text anyway.

The plain JSONDecodeError at the end of _loads_model_json is gone. The loop runs at least once, and every pass that neither returns nor raises sets first_error, so the line could not run. An assert says so.

Falsify salvage

_salvage_truncated_object (reached through the leading_only path) could take a verdict from quoted content in a cut falsify reply. In {"verdict": "CONFIRMED", "reasoning": "the stub returns a", "verdict": "DISMISSED", ... the later duplicate won, and a reasoning-first reply with a quoted verdict was read the same way. This existed before this PR. The salvage now:

• refuses an object with a duplicate key (object_pairs_hook);
• refuses an object that has a verdict that is not its first key. falsify_real.py asks for the verdict first. A salvage with reasoning only and no verdict is still accepted, as test_falsify_real.py expects.

Round 9: review fixes

This push answers the security review (F1-F7, N1, N2; F6 was withdrawn) and the three code reviews. Each item below says what changed and which test or mutant covers it.

Security review

• F1 (High): a refused CLI reply was handed on as raw text, and later parses kept the last of two equal keys. A refused repair, a refused key or a duplicate key on the CLI path now raises LLMInvokeError ("LLM subprocess reply refused -- ..."). That holds for bare stdout, a type: result envelope, a stream array, and a string or object result. Before, the raw text went on to the caller. A caller that reads it again with json.loads would keep the last of two equal keys, so a quoted "findings": [] after the real findings read as a clean review. I searched src for json.loads on reply text and found five sites that parse a reply again. All five now use loads_reply_json, which refuses a duplicate key and raises ValueError, as any other invalid JSON does: reviewer_json.validate_reviewer_json, the canary review parse in canary_gen, the raw-response audit in factories, the canonical form in receipt, and the canary provider in cli. _loads_model_json also refuses a duplicate key in every parse now, with or without keys, repaired or not. tests/test_reply_refusal_e2e.py runs the reported attack (a CLI reply with an escaped newline and a matching excerpt) through the review path. At 071a9ea it ends in PASS. After the fix the call fails closed. API-path tests check that a duplicate key spends the correction retry and that a second duplicate fails the call. N1 (the test-assertion review in cli) goes through the same CLI path and is covered by the same refusal.
• F2: after a plain error, the full scan could still take a quoted object. With reply_keys set, _fallback_reply now tries only the object that opens the reply or opens its list (leading_only). A prose-led reply gets no scan. It goes to the correction retry. Without keys, the full scan with the rules below still runs. This changed one existing test: the embedded case of test_writer_schema_raw_retention.py led with prose and ran one attempt, so it now puts the prose after the reply (816fad8).
• F3: _PROSE_SYMBOL read any non-ASCII character as prose, and the failed-literal tracker counted braces inside strings. _PROSE_SYMBOL is now a whitelist: a dash and a space, a non-ASCII letter, or a CJK full stop, comma or mark that ends the text or comes before whitespace or a letter. Fullwidth brackets and code punctuation, smart and corner quotes, a middle dot, zero-width characters and a BOM do not count. _OpenLiteral skips strings inside a failed literal, so {key: "}", ... stays open.
• F4: no bound on the repair and the scan. The quote repair stops after 256 repairs or on text over 256 KiB, and the scan returns None on text over 256 KiB. Tests time adversarial texts (open lists, open braces, many quotes) of that size. _invoke_cli parses stdout once (test_cli_parses_stdout_once).
• F5: call counts not pinned. The refusal tests now assert the number of model calls: one correction retry and no more, so two calls at most.
• F7: a deeply nested reply raised RecursionError. The parse in _loads_model_json raises RepairRefused, the scan returns None, and loads_reply_json raises ValueError. Each path fails closed.
• N2: the canary path. dispatch_canary_review and the canary provider use loads_reply_json. A duplicate findings or mutations is refused, and the provider falls back to its templates.

cr1

• P2-1: a continuation that only echoes the end of the partial (whitespace, fence marks, closing brackets) now counts as an empty continuation, so the whole partial is returned and the P1 is kept within two calls.
• P3-1: REVIEW_REPLY_KEYS lists the excerpt aliases that reviewer_json reads (line_start, line_end, code).
• P3-2: this description now says "three fallback sites" and no longer counts the partial parse as one.
• P3-3: a test pins the top keys of each key table.
• P3-4: the stale _prose_may_follow docstring is rewritten.
• The corpus and the measurement script are committed (see How the rule was chosen), and the related-test selection command is under Checks.

cr2

• P3-1: _SENTENCE_END is anchored with $, and a . ends a prose word only before whitespace or the end, so . Then, d.keys() and a.b are not prose.
• P3-2: a continuation that restates the whole reply is read as that reply when it parses strictly, and an echo is read as empty (as in cr1 P2-1). The "one more call" wording under Cut replies whose code_excerpts come first is corrected.
• P3-3: same as cr3 P2.
• P3-4: the corpus files are listed under How the rule was chosen.

cr3

• P2: a duplicate key in a CLI object result after an envelope repair was accepted. The envelope parse now refuses a duplicate key anywhere in the envelope, including its result, and the call fails.
• P3-1: the "never checked" wording is gone. The CLI wrapper envelope below says what is checked.
• P3-2: a duplicate key gets its own correction message, Duplicate key "<key>" in reply, instead of the generic one.
• P3-3: probes for M20, a list followed by ] and then more text, are tests.
• P3-4: whitespace-only tails are counted in the limit. In the list place, nothing after the object, or anything other than ] or , and another object or list, now refuses, since the list is still open.
• P3-5: the length tests assert the result, _keeps_the_p1 fails on a missing P1 instead of passing on an exception, and the O(1) wording is gone.
• test_factories.py::TestParallelL1::test_api_pass_starts_are_staggered is timing-sensitive. The mutation run does not count a failure of it as a kill.

Known limits after this push

• A single-finding reply sent as a bare finding object (no findings wrapper) that needs a quote repair is refused, because its keys are not top-level reply keys. It goes to the correction retry.
• With keys, a prose-led reply always costs the correction retry (F2).
• With keys, the stop-cut limit is 8 cut tails and 3 whitespace-only tails, all with the object opening the reply (see Limit).

Commits

1. 9472a50: the change and tests/test_llm_reply_keys.py.
2. 50cbafc: ruff format of the six touched source files, format only. The AST of each file is unchanged. One message string in cli.py is split into two literals to stay under 105 columns. This commit is needed because the lint gate runs ruff format --check on whole touched files, and these files were not formatted on main.
3. b39341b: six existing tests pinned the exact call shape of _invoke_cli, _repair_missing_excerpts or the API retry. Their fakes and expected kwargs now include reply_keys=None. No assertion was loosened.
4. 9d5e030: ruff format of two of those test files, format only, for the same lint-gate reason.
5. af6ed18: the bare-stdout check in _invoke_cli, plus tests for it, for each site that passes reply_keys on, and for the key tables.
6. 310042e: ReplyKeyRejected and no embedded-object scan after it, the object-result check in _invoke_cli, and top-level lists in _unknown_reply_key, with tests.
7. 63fcd76: RepairRefused, _fallback_reply and the leading_only scan, with tests. Also two CLI tests for the object-result check.
8. 6869ce2: removes a leading_only check that the scan guard already covers. A mutant deleting it survived, and no input reaches it.
9. f171539: the enclosed-object rule, the falsify salvage checks, and tests: the budget raise, ReplyKeyRejected as a RepairRefused, object-then-prose recovery, and the plain-error and salvage reproductions.
10. b8858ed: tests that pin the brace count in _unmatched_close_after.
11. bbbdcab: the wiring tests now call each caller with llm_invoke patched and check that it received its key table. They used to search the source text, which a commented-out reply_keys=... still matched.
12. b38c59f: a CLI stream array whose last result event has no result member fell into the bare-stdout re-parse, which returned the whole array instead of the event when keys were set. main returns the event. Fixed, with tests with and without keys.
13. 87fbf3d: rule 1, the first form of rule 2, the unmatched-} count removed, an empty-string unknown key counted (is not None at three sites), and ReplyKeys.items made read-only (MappingProxyType) and left out of the hash.
14. de21bf9: rule 2 as described above, the 3.12 trailing-comma fix, the duplicate-key check and the unreachable raise removed.
15. ac448e4: a long test string wrapped, format only.
16. f698405 and 9888e6c: tests for trailing commas (,}, ,], after a string and with whitespace), each tail case of rule 2, and the CLI object-result check with an empty key.
17. 7078f25: ruff format of tests/test_llm_invoke.py, format only (two blank lines). The file was not formatted on main either.
18. 4468bcc: daemon_state.py builds the static-rules list with extend. Ruff's PERF401 flagged the loop on main, and the lint gate checks whole touched files.
19. 4ec743f: rule 1 at the reply start, rule 2 as described above, the whole-parse rule in _continue_truncated, keys for the inline canary review and both daemon-state questions, and tests. Two contract tests change, as listed under Costs.
20. 452a118, f0ead2f and 880b069: tests for each code mark, a repaired daemon conflict item, a repaired Q1 reply with an unknown key, and three more prose-like tails; the docstring now states the limit by first character.
21. This push, six commits, each with Signed-off-by. I first made them without the sign-off and re-created them with git cherry-pick -s on 880b069 before pushing; nothing on the remote was amended or force-pushed.
    • bffb448: a length cut is always continued. Two tests that expected no continuation now expect one.
    • 3a87d55: the measured rule, the scan's duplicate-key hook, literal tracking after prose, the correction retry for any reply holding a {, CANARY_REVIEW_REPLY_KEYS, top-level-only Q2Q3_REPLY_KEYS, and tests at each _fallback_reply site. Its message gives 30 of 171 P1 lost; the next commit makes it 29.
    • 38720cb: the prose-punctuation check read its position from a slice offset, so it tested the wrong text. A mutant found it. Fixed, with the tests that kill the mutants from that run.
    • 196fd0a: test_per_pass_usage.py::test_factory_recovery_producers_preserve_unknown_usage failed after bffb448. Its transport returned a complete partial on length, which is now continued. The test now cuts the reply after 12 characters and hands the rest to the continuation, and still checks that the usage is Usage(11, 0, 0) with no known fields. I missed this in the narrower runs and found it in the 133-file run below.
    • 0c50687: # noqa: BLE001 with a reason on the canary dispatch's broad catch, which is on main. The lint gate checks whole touched files.
    • 071a9ea: ruff format of test_continuation_failure_contract.py, test_json_recovery.py and test_per_pass_usage.py, format only. They were not formatted on main.
22. This push, eight commits, each with Signed-off-by, on top of 071a9ea; nothing on the remote was amended or force-pushed:
    • 7038ff0: the round 9 source changes (see Round 9: review fixes): CLI refusals raise, loads_reply_json at the five re-parse sites, the duplicate hook in every parse and its correction message, leading_only with keys, the _PROSE_SYMBOL whitelist, strings skipped in _OpenLiteral, the size caps, RecursionError handling, echo and restating continuations, _SENTENCE_END and . in prose words, the list rule, and the excerpt aliases. It also updates test_json_recovery.py and test_llm_reply_keys.py, where the list-place known-limit cases now expect LLMInvokeError.
    • 35ae7b4: tests/test_reply_refusals.py and the CLI end-to-end test tests/test_reply_refusal_e2e.py.
    • be2e6e9: the corpus, its measurement script and tests/test_reply_tail_corpus.py.
    • ad2163a: tests for the three mutants that survived the first run.
    • 32c1d47: drops the json imports in canary_gen and reviewer_json that loads_reply_json left unused (ruff F401).
    • ed92fc1: ruff format of receipt.py, format only. It was not formatted on main, and this branch now touches it.
    • 816fad8: the embedded case in test_writer_schema_raw_retention.py now puts prose only after the reply. With keys, prose before the reply is refused and costs the correction retry (F2), and that test runs one attempt, so its six embedded cases failed. Prose after the reply still gives raw text that is not the parsed JSON, which is what the test checks. The prose-led refusal is tested in test_reply_refusals.py.
    • 1983bbe: ruff format of test_writer_schema_raw_retention.py, format only. It was not formatted on main.

Tests

tests/test_llm_reply_keys.py (528 tests):

• The llm: quote repair silently injects keys when a quoted excerpt contains JSON-like "k": "v", "k2": ... (pre-existing, not caught by #216 key guard) #264 reproduction:
  • Without keys, the repair still cuts the excerpt. This pins the hazard.
  • With REVIEW_REPLY_KEYS, it raises.
• The same shape in the falsify reasoning, the falsify receipt, a runtime finding, a code_excerpts item and a context_quotes item: each raises with its caller's keys.
• Replies that still parse:
  • An unrepaired reply with unknown keys.
  • A repaired reply with only allowed keys: review findings with excerpt and nested code_excerpts, a falsify reply with receipt, a runtime reply with surfaces.
• _unknown_reply_key: each level, with unlisted members and deeper levels left unchecked.
• API backend: the bad reply is followed by a JSON-correction request, and the corrected reply is returned, through both _invoke_api and llm_invoke. Without keys, the repair is kept and no retry happens.
• CLI backend: the inner result and a bare stdout both fall back to the raw string. Without keys, the repair is kept. A bare stdout with only allowed keys still parses.
• Threading: one test per site that passes reply_keys on. Each sends the reproduction through that site and checks that the repair is refused: the partial and the combined parse in _continue_truncated, _retry_with_more_headroom, _invoke_api into each of those two, llm_invoke into _repair_missing_excerpts, and _repair_missing_excerpts into the API backend.
• Quoted fragment: the reproduction above, at the API parse (through _invoke_api and llm_invoke), both continuation parses and the wider retry. Each one retries or refuses instead of returning the quoted empty review. ReplyKeyRejected keeps the original error's message and position.
• Failed repairs: the three tails and the quoted-key refusal each raise RepairRefused and keep the right error position. _invoke_api retries on each, with and without keys, and llm_invoke returns the P1 after the retry. The wider retry and both continuation parses do not return the quoted fragment. A failed repair still salvages a cut falsify reply's leading object, and an error with no repair still gets the full scan.
• CLI object result: a repaired envelope and a repaired stream array whose result carries c fall back to the raw stdout, and keep the repair without keys. An unrepaired envelope with an extra key in its object result is not checked. That includes an envelope with a raw newline inside a string, which json.loads(strict=False) reads without repair. A repaired envelope whose object result has only known keys keeps the result.
• Top-level lists: _unknown_reply_key checks each object in a list. A runtime reply wrapped in a one-object list is refused on the CLI, where _parse_llm_response then rejects the raw text, and on the API, where it spends the correction retry.
• Wiring: each wired caller passes its keys.
• Repair budget: _loads_model_json('{"a"}') raises RepairRefused after the budget runs out, and '"{"findings": []}' tries only the leading object. _fallback_reply on "}{"findings": []}},]][ returns None.
• Error types: ReplyKeyRejected is a subclass of RepairRefused.
• Errors before any repair: the five reproductions above. The extractor skips the quoted object in each, and _invoke_api and llm_invoke either fail closed or retry and return the P1, with and without keys. The code_excerpts-first leading object is checked to hold an empty review. Unit tests cover the brace rule: an object followed by }, } {, "} and "} {"a": 1} is skipped, and one followed by balanced braces is kept.
• Falsify salvage: a duplicate verdict, a reasoning-first cut reply with a quoted verdict, and a duplicate key are refused. A leading verdict is still read.
• Rules 1 and 2: prose-led replies with one to three raw { after the stray quote; a valid reply followed by prose with a lone }, before and after prose, and fenced (kept, one call); five falsify replies that quote a verdict (never DISMISSED); quoted reviews followed by func f() { for {, prose-led, True-led and code_excerpts-first, plus prose-led and True-led replies cut right after the quoted object, with finish reason stop and length (never a review without the P1); '{"x": 1, "ok": True} {"findings": []}}'; and each row of the table above.
• Trailing commas: _error_is_at_quote returns None for both 3.12 messages and the 3.13 one; a } after a stray quote with no comma is still repaired; [P1 with ",}", P2] never comes back as only the P2.
• Duplicate keys: a complete falsify reply with two verdicts and a cut code_excerpts-first reply are refused with keys; the refusal keeps the first repair error's position.
• Empty and frozen keys: a repair that invents an empty key is refused in an object, in a list and in a CLI object result; ReplyKeys is hashable and its items cannot be changed.
• Cut code_excerpts-first replies: each refused tail on stop raises after the retry, with and without keys; each prose-like tail on stop returns the empty review (the limit); every tail on length is continued and keeps the P1; a cut partial that only the scan reads is continued.
• Non-JSON replies and example envelopes: a Python dict reply and a JS object reply that quote a review, and prose that quotes an example envelope before the real reply, never return the quoted empty review.
• Canary and daemon wiring: the inline canary review passes CANARY_REVIEW_REPLY_KEYS, and the daemon runner passes Q1_REPLY_KEYS and Q2Q3_REPLY_KEYS. A duplicate conflicts and a repaired Q1 reply with an unknown key are refused. A repaired conflict item with an invented key is kept, since Q2Q3 now checks the top level only; a test pins this.
• Hazard asserts changed: test_failed_repair_raises_repair_refused (four cases) and test_fragment_repro_hides_an_empty_review used to assert that the full scan returns the quoted empty review. They now decode the quoted object directly to show the hazard, and assert that the extractor returns None.
• This push:
  • each of the 26 code tails refused on stop in all three places and : x in the leading and list places, with and without keys; the prose-like tails and : x after prose pinned as the limit; every tail and the three whole-parse tails continued on length;
  • _text_after_refuses and _prose_may_follow cases for each acceptance branch, each place, the punctuation offset, and ! at the end;
  • a duplicate key in a prose-led review, a prose-led falsify reply, a falsify list, and a review followed by a second reply: the extractor returns None with and without keys, and _invoke_api returns the corrected reply after one correction call;
  • Python dict and JS object replies quoting a review, bare and after four leads, and three literals closed before the reply (plus {a ten times) that do not end the scan;
  • the three _fallback_reply sites (the API parse, the combined continuation parse and the wider retry), each fed a refused repair followed by a keyed empty review (with and without keys) and a duplicate-key reply (with keys). None returns the quoted review, and a test checks that a full scan would have;
  • a partial that parses whole is continued on length, for no tail, and two spaces;
  • the canary keys and Q2Q3_REPLY_KEYS pinned as top-level only.

Mutation run on be2e6e9, then ad2163a (this push). 31 mutants over the round 9 changes, on the 17 files listed for the previous run plus test_reply_refusals.py, test_reply_refusal_e2e.py and test_reply_tail_corpus.py. Python 3.14, a separate tree from git archive, __pycache__ cleared before each mutant, bytecode writing off, no -x and no excluded tests. A mutant counts as killed only if a test other than the timing-sensitive test_api_pass_starts_are_staggered fails. The unmutated tree passed. A sanity mutant (no duplicate-key refusal in the hook) was killed by 40 tests. 28 of 31 were killed in the first run. The three survivors got tests in ad2163a and were killed in a second run on that commit, which repeated the sanity mutant first (killed by 41):

Finding	Mutant	Result	Failing tests
F1	_parse_model_json without the duplicate hook (CLI stdout and envelope, cr3 P2)	killed	32
F1	CLI stdout refusal raised as a plain JSONDecodeError	killed	8
F1	CLI string-result refusal returns the raw text	killed	6
F1	loads_reply_json without the duplicate hook	killed	5
F1	reviewer_json re-parse back to json.loads	killed	1
F1, N2	canary_gen re-parse back to json.loads	killed	1
F1	factories re-parse back to json.loads	killed	1
F1	receipt re-parse back to json.loads	killed	1
F1, N2	cli canary provider back to json.loads	survived, then killed after ad2163a	1
cr3 P3-2	duplicate correction message off	killed	1
cr3 P3-2	duplicate message text changed	killed	21
F2	leading_only ignores reply_keys	killed	16
F3	_PROSE_SYMBOL takes any non-ASCII character	killed	33
F3	_PROSE_SYMBOL takes a fullwidth )	killed	1
F3	_OpenLiteral counts braces inside strings	survived, then killed after ad2163a	1
F4	no repair length cap	killed	1
F4	no repair count cap	killed	1
F4	no scan length cap	killed	1
F7	parse RecursionError not caught	killed	1
F7	scan RecursionError not caught	killed	2
F7	loads_reply_json RecursionError not caught	killed	1
cr1 P2-1	echo continuation not read as empty	killed	4
cr2 P3-2	restating continuation not read as the reply	killed	230
cr3 P3-3 (M20)	text after ] in a list read as prose	killed	3
cr2 P3-1 (M22)	_SENTENCE_END without $	killed	12
cr2 P3-1	. ends a prose word anywhere	survived, then killed after ad2163a	5
cr1 P3-1	excerpt aliases dropped	killed	1
cr1 P3-3	context_quotes dropped from the top keys	killed	2
cr3 P3-4	any text after a list item accepted	killed	80
cr3 P3-4	list rule off	killed	30
cr3 P3-4	] and , [ rules after a prose-led object off	killed	3

F5 (call counts) is a test-only change, and F4's single CLI parse is pinned by test_cli_parses_stdout_once rather than a mutant.

Mutation run on 071a9ea (the previous push). 31 mutants over the new code, on 17 files: test_llm_reply_keys.py, test_llm_invoke.py, test_json_extraction_contract.py, test_json_salvage_contract.py, test_json_recovery.py, test_continuation_failure_contract.py, test_output_budget_retry.py, test_falsify_real.py, test_daemon_state.py, test_per_pass_usage.py, test_canary_gen.py, test_canary_cli.py, test_llm_invoke_usage_contract.py, test_excerpt_repair_contract.py, test_l1_evidence_recovery.py, test_runtime.py and test_factories.py. Python 3.14, a separate worktree, __pycache__ cleared before each mutant, bytecode writing off, no -x and no excluded tests. A sanity mutant (the scan returns a keyed object without the text-after check) was killed by 269 of 1,764 tests. 30 of 31 were killed:

Mutant	Result	Failing tests
length partial returned when it parses whole (P2)	killed	11
scan decoder without the duplicate hook	killed	2
duplicate key skipped, scan goes on	killed	1
duplicate in a whole object right after the match ignored	survived, equivalent (see Duplicate keys)	0
keyed object inside an open literal returned	killed	6
failed literal never recorded	killed	6
any failed literal ends the scan	killed	5
correction gate back to a leading { or [	killed	25
brace-balance check removed	killed	6
keyed whole object right after the match kept	killed	10
whole-object skip removed	killed	3
quote-then-JSON widened to any "	killed	1
quote-then-JSON removed	killed	18
indented line accepted	killed	8
fullwidth marks, BOM and smart quotes count as prose	killed	32
] and , rules only in the list place	killed	3
] and , rules in every place	killed	2
prose punctuation offset read from the slice (the bug fixed in 38720cb)	killed	3
prose punctuation accepts anything after it	killed	3
a prose word needs no ending	killed	15
markdown line starts not accepted	killed	9
ASCII whitespace instead of str.isspace	killed	13
sentence end accepted after a gap	killed	1
list place read as prose	killed	3
prose place read as leading	killed	8
full scan at the continuation site	killed	2
full scan at the wider-retry site	killed	3
full scan at the _invoke_api site	killed	2
canary keys back to REVIEW_REPLY_KEYS	killed	1
canary keys with the review item table	killed	2
Q2Q3_REPLY_KEYS with an item table	killed	2

An earlier run, with -x, on the code before 38720cb found the punctuation-offset bug. 38720cb fixes it and adds tests for the mutants that survived that run.

Correction. An earlier version of this description said the tests killed all 21 hand mutations. That was wrong. A cross-check found 11 survivors: the reply_keys hand-off at seven sites, an extra key in the code_excerpts and context_quotes item tables, receipt dropped from the falsify keys, and surfaces dropped from the runtime keys.

The mutation run is now redone with 38 mutants over the guard, the level walk, each parse and hand-off site, the bare-stdout branch, each key table and each caller. Each mutant runs against the related test files (test_llm_reply_keys.py, test_llm_invoke*, test_json_recovery.py, test_excerpt_repair_*, test_l1_evidence_recovery.py, test_output_budget_retry.py, test_reviewer_json*, test_falsify_real.py, test_runtime.py, test_runtime_response_contract.py, test_factories.py). __pycache__ is cleared before each mutant and bytecode writing is off. A first mutant that adds a failing test was killed, which shows mutants are loaded.

• First pass: 37 of 38 killed. The survivor was excerpt dropped from the finding keys. The allowed-keys test now includes an excerpt.

• test_factories.py::TestParallelL1::test_api_pass_starts_are_staggered failed in two first-pass runs. It passed three times when run alone, so it looks timing-sensitive. Both mutants it hit were also killed by test_llm_reply_keys.py tests.

• Second pass, against test_llm_reply_keys.py alone, for the 16 mutants in question: all killed. Each of the 11 reported survivors is killed by a test written for it.

• Commit 6: 13 mutants over the new code, on the same files and with the same setup, after a sanity mutant that was killed. All 13 were killed. They cover a plain error instead of ReplyKeyRejected, a lost error position, the scan restored at each of the four sites, each condition of the object-result check and its fallback, the bare-list re-parse, and the list walk in _unknown_reply_key. The 38 earlier mutants were not re-run at the shifted line numbers.

• Commit 7: 17 mutants over the new code, on the 18 files, after a sanity mutant that was killed. 15 were killed. They cover each RepairRefused raise made plain, the exc is first_error check, the error position, the ReplyKeyRejected branch in _fallback_reply, leading_only forced on or off, the reply_start guard, [ left out of the strip, the full scan restored at each of the four sites, _unknown_reply_key dropped from the CLI object-result check, and strict=True in _parses_without_repair. Two survived:

  • The RepairRefused raise after the repair budget runs out. Correction: an earlier version of this description called this raise unreachable. That was wrong: '{"a"}' reaches it, and a cross-check found 4,645 of 100,000 random strings that do. Commit 9 adds tests that kill it. Only the plain JSONDecodeError after it is unreachable.
  • The if leading_only: return None after the leading object. The reply_start guard returns None at the next { anyway. Commit 8 removes it.

• Commits 9 and 10: 15 mutants over the new code, on the 28 files below, after a sanity mutant that was killed. All 15 were killed. For the enclosed-object rule, they cover the skip removed, the check made at the object's start, brace matching removed or unguarded, the threshold raised to two, the end-of-text flag set, and ] counted as a close. For the salvage, they cover the verdict-first check removed, forced on every object or keyed on the wrong key, and the duplicate-key check removed or never firing. They also cover the budget raise made plain or with leading_only never set, and ReplyKeyRejected not a RepairRefused. The brace-match and threshold mutants survived the first run and are killed by the tests in commit 10.

• Commits 11-16: 38 mutants on the 28 files below, Python 3.14, __pycache__ cleared before each and bytecode writing off. The first, a sanity mutant that makes every repaired parse return None, was killed by 235 tests.

  • The nine survivors from 6869ce2:
    • R3 (budget raise made plain): killed by test_budget_exhaustion_*.
    • R4 (ReplyKeyRejected based on JSONDecodeError): killed by test_repair_refused_is_the_base_of_reply_key_rejected.
    • R15 (return None to continue for a later { in leading_only mode): survives, equivalent. Every later { also has i != reply_start, so the loop still ends with None.
    • B4 (reply_keys is not None dropped from the bare re-parse): survives, equivalent. With reply_keys=None the re-parse repeats the same pure call that already succeeded on the same text.
    • C1b-C5b (reply_keys=None or omitted in factories.py, the cli.py spawn, the cli.py test-assertion review, runtime.py, falsify_real.py): each killed by its caller's wiring test.
  • Rule 1: off killed (9), always on killed (5). Exclusive error position (text[i : exc.pos]) survives, equivalent: the error lands on a " with no earlier quote in the span only for "Unterminated string" at the first key's opening quote, and then no quote follows, so no later object can carry a key.
  • Rule 2, ten mutants (off, each part of the first-character check, the whole-object skip, the quote flag, } only, ] only, no quote needed, whitespace not skipped): all killed.
  • Trailing commas, four mutants (off, } only, no comma needed, no rstrip): all killed. } only was killed after 9888e6c added the ["a", "b",] case.
  • Duplicate keys, four mutants (hook off, hook always on, first error ignored, plain error): all killed.
  • Salvage return removed, verdict-first check removed, is not None made truthy at the three sites, no MappingProxyType, items hashed: all killed. The CLI truthy mutant survived the first run and is killed by the empty-key CLI test in 9888e6c.

• Commits 17-20: 24 mutants on 18 files (test_llm_reply_keys.py, test_llm_invoke* (five files), test_json_extraction_contract.py, test_json_salvage_contract.py, test_json_recovery.py, test_continuation_failure_contract.py, test_output_budget_retry.py, test_falsify_real.py, test_daemon_state.py, test_canary* (four files) and test_runtime.py), Python 3.14, with the same setup. A sanity mutant that makes the extractor return None was killed by 60 tests. 23 of the 23 others were killed:

  • each of the nine code marks removed in turn;
  • leading forced off, forced on, or without the [ check, and the code-mark check removed;
  • the later-object check removed, or made to fire on any object;
  • rule 1 without the reply-start case;
  • the scan fallback restored in _continue_truncated;
  • reply_keys dropped from the canary review, the Q1 call and the Q2Q3 call; the conflict item table removed or widened; Q1's top keys widened. The Q1 mutant survived the first run and is killed by the Q1 test in f0ead2f.

Local runs at 1983bbe (this push) over 214 related test files, with PYTHONPATH=src:. and bytecode writing off. The files are every tests/**/test_*.py that imports a module this branch changes, plus the test files it changes. tests/eval/corpus/ is left out because it holds fixture repositories, not tests (its test_llm_invoke.py fails to import by design). The selection command:

mods='canary_gen|cli|daemon_state|factories|falsify_real|llm_invoke|receipt|reviewer_json|runtime'
{ rg -l --glob 'tests/**/test_*.py' --glob '!tests/eval/corpus/**' \
     -e "code_forge\.($mods)\b" -e "from code_forge import .*\b($mods)\b" tests
  git diff --name-only ec8cca9 HEAD -- 'tests/test_*.py'; } | sort -u

• Python 3.14: 55 failed, 21039 passed, 263 skipped.
• Python 3.12.15 (CI's version): 55 failed, 21039 passed, 263 skipped. The failing IDs are the same as on 3.14.
• The 55 fail with the same IDs on main (869873f) on 3.14. They are environment failures in test_assertion_outcome_reporting.py (30), test_cli_phase_progress.py (9), test_cli_integration.py (8), test_cli_banner.py (3), test_integration.py (2), test_cli_empty_diff.py (1), test_mutation_ledger_ci_controls.py (1) and test_report_projection_compat.py (1). None is in a file this branch changes. test_gate_check.py, which had two such failures in the previous run, does not import a changed module, so it is not in this selection.
• A first run of this selection found the six embedded failures in test_writer_schema_raw_retention.py fixed by 816fad8. Two test_doctor.py failures in that run came from a src/*.egg-info that a local editable install had left in the worktree (doctor read its pins). With it removed, test_doctor.py passes (121 tests).
• scripts/lint_changed.py --base origin/main passes with ruff 0.17.3 (ruff check, and ruff format --check on 26 files). No added line is over 105 columns.
• git merge-tree against 869873f reports no conflicts. The branch is not rebased.
• Commits after ad2163a change no source behaviour (an unused import, formatting, one test), so the mutation results below stand for this head.

Local runs at 071a9ea (previous push) over 133 related test files (the files this branch touches and the tests that exercise the changed modules), with PYTHONPATH=src:. and bytecode writing off:

• Python 3.14: 54 failed, 17877 passed, 19 skipped.
• Python 3.12 (CI's version): 54 failed, 17877 passed, 19 skipped. The failing IDs are the same as on 3.14.
• The 54 also fail at the merge base ec8cca9 on 3.14, with the same IDs. They are environment failures in test_assertion_outcome_reporting.py (30), test_cli_phase_progress.py (9), test_cli_integration.py (8), test_cli_banner.py (3), test_gate_check.py (2), test_mutation_ledger_ci_controls.py (1) and test_report_projection_compat.py (1). None is in a file this branch changes.
• ruff check and ruff format --check pass on the 20 changed Python files. No added line is over 105 columns.

Local runs at 880b069 over 46 files (the 18 above; the other seven test_falsify_* files; test_excerpt_repair_* (three files); test_reviewer_json* (two files); and test_cli_head_diff_machine_options.py, test_cli_internal_failure_propagation.py, test_cross_repo_identity.py, test_daemon_state_drift.py, test_factories.py, test_gate_check.py, test_index_mutation_admission.py, test_kernel_context_cli.py, test_l1_evidence_recovery.py, test_lens_wiring_ast.py, test_outlet_c_cli.py, test_per_pass_usage.py, test_phase4_ic_t14_routes.py, test_review_uncertainty_boundary.py, test_runtime_response_contract.py and test_schema_corpus.py), with PYTHONPATH=src:., on Python 3.14.9 and 3.12.15:

• This branch: 3 failed, 2411 passed, 2 skipped, on both versions.
• main (4822ac1), without test_llm_reply_keys.py: 3 failed, 2140 passed, 2 skipped, on both versions.
• The failing IDs are the same on both: test_falsify_raise_from.py::test_falsify_except_raise_names_its_cause (needs ruff on PATH) and two test_gate_check.py::TestFailureWaiverEvidence tests that run the system python3, which has no pytest here.
• git merge-tree against 4822ac1 reports no conflicts. The branch is not rebased.

Local runs at 9888e6c over the same 28 files, with PYTHONPATH=src, on Python 3.14.9 (the box venv) and Python 3.12.15 (CI pins 3.12; a fresh venv with the test dependencies and google-auth):

• This branch: 1 failed, 1582 passed, on both versions.
• main (ec8cca9, before it moved to 4822ac1), without test_llm_reply_keys.py: 1 failed, 1381 passed, on both versions.
• The one failure on both is test_falsify_raise_from.py::test_falsify_except_raise_names_its_cause, which needs ruff on PATH.
• tests/test_llm_reply_keys.py from this branch run against b8858ed: 50 failed on 3.14 and 55 on 3.12. Against 87fbf3d: 8 failed on 3.14 and 13 on 3.12 (before the later test commits).

Local runs at commit 10, on this branch and on main (ec8cca9), over 28 files: the 18 below, the other seven test_falsify_* files, test_json_extraction_contract.py, test_json_salvage_contract.py and test_review_uncertainty_boundary.py:

• On this branch: 22 failed, 1455 passed and 31 skipped. On main, without test_llm_reply_keys.py: 22 failed, 1329 passed and 31 skipped. The failing IDs are the same: the 21 Vertex tests below, and test_falsify_raise_from.py::test_falsify_except_raise_names_its_cause, which needs ruff on PATH.

Local runs at commit 8, on this branch and on main (ec8cca9):

• The related test files are test_llm_reply_keys.py, test_llm_invoke* (five files), test_json_recovery.py, test_excerpt_repair_* (three files), test_l1_evidence_recovery.py, test_output_budget_retry.py, test_reviewer_json* (two files), test_falsify_real.py, test_runtime.py, test_runtime_response_contract.py and test_factories.py. That is 18 files. Earlier versions of this description said 16, which was a miscount.
• On this branch: 21 failed, 1200 passed and 31 skipped. Without test_llm_reply_keys.py, both branches give 21 failed, 1114 passed and 31 skipped, with the same failing IDs. (Earlier versions reported 20 failed and 1104 passed for the miscounted set.)
• The 21 failures are all Vertex tests, 20 in test_llm_invoke.py and one in test_llm_invoke_url_scheme.py. They fail on main too, with ModuleNotFoundError: No module named 'google'. The box venv has no Google auth package.
• test_json_salvage_contract.py, test_json_extraction_contract.py and test_review_uncertainty_boundary.py: 62 passed.

Earlier local runs, for commits 1-4:

• A full-suite run on this box was invalid because the disk ran out of inodes partway through. I did not use it.
• I re-ran the 85 test files that had any failure or error in that run, on this branch and on main (ec8cca9), one at a time with a fresh temp dir:
  • On this branch, before commit 3: 178 failed. Nine were new, all of them the call-shape tests fixed in commit 3.
  • On this branch, after commit 4: 169 failed, 5372 passed, 123 skipped.
  • On main: 169 failed, 5372 passed, 123 skipped. The failing test IDs are the same on both.
• test_llm_invoke*, test_json_recovery*, test_cli_integration*, test_falsify_sees_diff*, test_l1_prompt_byte_identity*, test_report_projection_isolation* and test_llm_reply_keys.py: 32 failed and 623 passed on this branch. On main, the same 32 fail and 596 pass. The 27 extra passes are the new tests.

The failures on both branches are in the box environment and exist on main. CI is the reference.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →