Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions .harness/docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1408,6 +1408,8 @@ The section above left the suite parallel and said that a test failing under loa

**One of those constants was widened later, and the reason it is not the defect above is worth reading before the next one is.** Commit `2df481b`, on story-130's branch, raised `SYNC_TIMEOUT` in `tests/test_config_keys_are_obeyed.py` from 1.3 seconds to 5.3 with `SLEEPS_PAST_THE_BOUND` raised to 12 to keep the ordering. The argument is not "the machine was slow" but that the value was the tightest of the three configured-command bounds — five times tighter than the query command's `FILED_QUERY_TIMEOUT` of 6.5 and about half the item command's `ITEM_UPDATE_TIMEOUT` of 2.7 — for no reason any of the three could name, and every site that observes a sync command *landing* needs a shell spawned and finished inside it. What made it bite is that `test_the_pristine_copy_runs_every_proof_green` conducts a nested pytest run inside an outer `-n auto` suite, so the fixture's commands compete with the whole suite for the machine; the two failures it produced were an entry left pending and a sweep count *below* its bound, both being one killed command rather than two defects. Both halves of the bound's proof are pinned exactly as they were, and 12 stays well under the 60-second default, so a harness that stopped reading the key still fails the killed half. **The rule the paragraph above states is intact**: headroom on the *landing* half may be brought into line with its siblings; the killed half, which is what proves the key, is not what was moved.

**One site in that group still observes a landing without routing through the helper, and it is a landing it needs rather than a landing it asks about.** `entry_that_got_to_land` exists so that an entry left pending naming the bound, from a command asked to sleep for nothing, is reported as a slow machine instead of a failure; `test_a_sync_command_that_finishes_inside_the_bound_lands` routes through it. `test_sync_command_is_the_command_an_entry_is_filed_by_running` does not — it asserts `state == outbox.LANDED` directly, though its question is *which command ran*, read off the reference, and landing is only the precondition for asking it. So that test fails, rather than reporting inconclusive, on exactly the machine condition the helper was written for, and it is the one member of the group where the standard the section states is not applied. The repair is the helper, not a wider bound; it is recorded here because a story that may own `tests/test_config_keys_are_obeyed.py` has to make it, and the story that found it could not.

**A precondition is not the question, which is what licenses reporting rather than failing.** A test asking whether a kill reaches a process group needs a child to exist first; a test measuring the gap between a session ending and an approval being conferred needs the session to have written its artifact first. A precondition the machine did not let the run reach says nothing about the code, and failing on it says something false — that the harness is broken when what happened is that the run could not be conducted. This is *not* the skipping-to-pass `.harness/standards/testing.md` forbids, and the standard now carries the distinction rather than leaving the two acts to look alike: a test that reached its question and declined to answer it is the forbidden thing; a test that never reached its question is this one. The census, which refuses a stage whose assertion count fell, is what holds the difference mechanically — no assertion was deleted to make any of these pass.

**Where a load-independent half was trapped behind a load-sensitive one, the test was split rather than converted.** The two configured-bound proofs in `tests/test_config_keys_are_obeyed.py` are now three tests each: the number read off the built transport (or off the resolved query settings) with no command run and no clock consulted, the command asked to sleep past the bound and observed killed with its entry left pending naming the bound, and only then the half where a command with no sleep must land inside the bound. Neither of the first two can skip, and the `KEY_PROOFS` entries for `sync_timeout_seconds` and `filed_query_timeout_seconds` now name the **killed** half — the one that separates the configured value from the harness default by observing what the bound did — so the proof of the key survives whatever the landing half reports.
Expand Down Expand Up @@ -1436,6 +1438,28 @@ The section above left the suite parallel and said that a test failing under loa

**The ripple landed where a nested suite is assembled by copying files.** `tests/conftest.py` now imports from `tests/machine_load.py`, so every module that builds a throwaway tests directory has to copy the helper beside the conftest or the copy cannot import at all — `test_contract_assertions_bite.py`, `test_schema_inventory_location.py` and the two copied-tests tuples in `test_config_keys_are_obeyed.py` each gained the name. That is the recurring shape of giving `tests/conftest.py` a new import: the modules that will redden are the ones enumerating which test files a constructed harness gets, and a plan touching the conftest should expect them rather than discover them. `tests/conftest.py` also inserts its own directory on `sys.path` explicitly rather than relying on pytest having done it, because a ceiling that silently failed to install is exactly the quiet disappearance it exists to prevent.

## A copy of this repository carries what the repository is, not what the runs left in it

A test that needs a repository it may mutate copies this one. `tests/test_git_history_loading_retired.py` does, because its subject is a coordinator edit and the repository the suite is running from is the one being validated. Until story-151 the copy was `shutil.copytree` with `shutil.ignore_patterns(".venv", "__pycache__", ".pytest_cache")`, and **a copytree copies what is on disk while git ignoring a path leaves the path untracked rather than absent** — so the copy carried `.harness/logs`, `.harness/runs` and `.venv310`, the last because the pattern names `.venv` alone. `.harness/stories/story-151.yaml` records what the brief measured: 915 MB and 797 MB in the two harness directories, about 1.75 GB copied in all, and the most expensive fixture in a `--durations=40` run of the whole suite at 19.48 s of module setup.

**The direction is what made it worth fixing rather than tolerating, and it generalises past this fixture.** Both of those directories gain a subdirectory every time a story runs, so the copy gets slower with no line of test code changing. A cost that grows with the harness's own use is a defect even while the number is small.

**The exclusion list is asked of git at run time and is not written down anywhere.** `ignored_paths(repo)` in that module runs `git ls-files --others --ignored --directory --exclude-standard -z` at the repository root and returns the entries with their trailing slashes stripped. `--directory` collapses a wholly-ignored directory to one entry, which is what keeps the answer from being every file under `.harness/runs`; `-z` asks for the NUL-separated form, which git does not quote, so a path carrying an unusual byte arrives as itself. The consequence is the point: a directory added to `.gitignore` later is excluded from the copy with nobody editing this module.

**A pattern list is the wrong instrument here and would have looked right, which is why the module says so where the next author will be standing.** `shutil.ignore_patterns` fnmatches **bare entry names**, so a pattern carrying a slash — `.harness/runs/`, the spelling `.gitignore` actually uses — matches nothing, and an exclusion derived from the text of `.gitignore` would have excluded nothing while reading as though it excluded everything. `copy_repository(destination, source=REPO_ROOT)` passes a callable instead, which joins each entry onto its directory's repository-relative path and compares that against git's answer, because the repository-relative path is the form git reports.

**`.git` comes along, and that is a requirement rather than an oversight.** Every assertion the copy exists for bounds itself at its own story's commit range and cannot resolve one without the history — this module is one of the four declared subject readers in the section above. Eleven megabytes of `.git` is not what made the copy expensive, and the question of whether the module needs history at all is settled: it does.

**What `.git` *is* here decides what running git inside the copy answers about, and a checkout under a linked worktree is not the case a reader assumes.** In a linked worktree `.git` is not a directory but a short pointer file naming an admin directory elsewhere, and the copying — before this story and after it — copies that file verbatim. A linked worktree's admin directory carries no `core.worktree`, so git derives the working tree from where it found the `.git` file: run with its working directory set to the copy, git reports about the copy. That is why an assertion reading git's ignored entries *inside* the copy answers about the copy rather than about the worktree the copy was made from, and why the module's bounded history reads resolve there at all. The cost of the arrangement is that the copy's history is reached *through* the source worktree's admin directory rather than held in the copy, so a copy whose pointer no longer resolved would not be caught by the presence check on `.git` — the nested pytest runs inside the copy would redden instead. That is not a property this story introduced and not one it repaired; it is written here because the next author to reason about what the copy contains will otherwise reason about a directory that is a file.

**One copying implementation, two fixtures, and the second exists because of what the first's consumers do to it.** `repo_copy` calls `copy_repository`; so does `pristine_copy`, which nothing mutates. The standing assertion takes the pristine one deliberately — the subprocess pytest runs inside `repo_copy` write `__pycache__` into it, and `__pycache__` is itself a path the repository ignores, so an assertion about what the *copying* put there would be answered by what the tests did to it afterwards. **A fixture that the tests under it modify cannot be the subject of an assertion about how it was built.**

**The standing assertion is an absence, so it is not left to speak for itself.** `test_a_freshly_made_copy_holds_no_path_the_repository_ignores` requires git, run inside the copy, to report no ignored entry — stated against git rather than against a list of directory names so that it goes on holding for whatever `.gitignore` gains later. Its control then plants a file under `.harness/runs` inside the copy and requires the same reading to report it, so a copy git could not read, or one whose ignore rules had stopped matching, cannot pass looking identical to a correct one. A second test names the two halves as paths — `.git` present, `.harness/logs`, `.harness/runs` and `.venv310` absent — because those are what a reader checks first and neither half is implied by the other.

**That the derivation is a run-time question and not a list is proved on a repository the test builds.** `test_a_directory_newly_ignored_is_excluded_with_no_edit_to_this_module` builds a repository through `conftest.init_repository`, copies it once with the directory unignored and once after `.gitignore` names it, and requires present-then-absent with nothing else differing. Built rather than borrowed for the obvious reason: the only way to vary a `.gitignore` is to own the repository it belongs to, and doing it against this one would mean editing the repository the suite is running in. The directory is written **after** the commit, so it is untracked — a committed directory is not reported as ignored however `.gitignore` is later worded.

**No wall-clock bound was added, and the section above is why.** The subject is what gets copied, not how fast the machine that copies it is. Pruning or retaining `.harness/logs` and `.harness/runs` themselves is a separate question this story did not open; it only stopped a test copy from carrying them.

## The revert check asks one test first

The section above made the suite cheaper to run. This one lets one of the three gates avoid running it. **The revert check establishes a *failure*** — governed edits are permitted iff reverting them makes the suite fail — and one failing test proves a failing suite while no number of passing tests proves a passing suite. That asymmetry is what licenses a subset standing in for the whole here, and it licenses it *nowhere the verdict runs the other way*: the clean-clone check and the coordinator's post-stage suite run both conclude green, so neither may short-circuit and neither was touched. A future story tempted to reuse this mechanism should ask which direction the verdict runs before anything else.
Expand Down Expand Up @@ -2777,6 +2801,7 @@ It is narrower than Chapter 18's **checkpoints**, and the difference is the reas
- **The other three ripples touched contracts, and each stayed exact.** `tests/test_coordinator_contract.py` and `tests/test_escalation_resume.py` both pin the `RunState` field set exactly — which is what those files exist for — so `correction_pass_count` is declared in each with its reason and both comparisons stay exact equality, the same repair `guidance_in_force` took in story-050. `tests/test_plan_commit.py::test_the_argument_list_handed_to_claude_is_what_the_exec_passed` compares today's `l5-plan` argv against the pre-story-023 script's, and the old script resolves no prose partial, so it renders `{{prose_layer}}` as the literal `None` where today's renders the partial's text. The repair subtracts exactly that one substitution and then asserts equality on every other argument and every other byte of the prompt, plus a new assertion that the two argv lists really do differ — **narrowed to what this story changed rather than loosened**, which is the standing ripple rule.
- **`tests/test_harness_layer_extraction.py` is now about which prompts carry *a* shared partial rather than about story-003's extraction**, and its docstring says so. The two partials' reach is asserted off prompts something actually rendered — a real run's rendered stage prompts, and the prompt `scripts/l5-plan` hands a session through a stub `claude` on `PATH` — rather than off the templates, **because a placeholder in a template that nothing resolves is not reach**. That is why the module joins `DECLARED_LIVE_ARTIFACT_READERS` with the shipped prompts as its subject: what it asserts is what this deployment ships, and the planner half drives the real script, which resolves this repository as its own harness root.
- **Why the prose rule is a second partial rather than a paragraph in the first.** The harness-layer partial addresses stages that mutate a tree; the prose rule addresses anything that writes prose a human later reads, which includes the planner — the one template no coordinator renders, and the stage that wrote the first of the stale counts the rule exists against. Folding it into `harness-layer.md` would have forced that partial's reach to widen to a non-stage prompt, which is a change to a distribution three stories rely on being narrow. Two partials with two audiences, resolved through one helper, keeps both reaches decidable by reading one file each. **The constraint that made it cheap: the prose partial carries no placeholder of its own**, so the narrower planner context renders it identically to a stage's.
- **A test that copies this repository excludes what the repository excludes, and asks git which paths those are rather than naming them.** A copytree copies what is on disk, and an ignored path is untracked rather than absent, so a hand-maintained pattern list silently carries the harness's own run and log directories — which grow with every story, making the copy slower with no test code changing. `shutil.ignore_patterns` fnmatches bare entry names and so cannot express a `.gitignore` entry carrying a slash at all. See "A copy of this repository carries what the repository is, not what the runs left in it".
- Verification rules never change between retries; retries narrow scope, they do not restart the workflow.
- Capacity exhaustion (rate limits) is a reason to wait, not to fail; budget ceilings are a reason to stop. Story-063 gave the second half of that a mechanism, and it obeys it literally: a budget stop escalates and nothing on that path retries, sleeps or backs off. See "A run has a cost ceiling".
- **Cost is a peer of the retry ceiling, not a reading of it.** `max_retries` and `max_self_routes` are untouched by story-063 and no ceiling is derived from either. The three bound different quantities — attempts, in-place re-runs, and money — and a run can hit any one of them with the other two unspent.
Expand Down
4 changes: 4 additions & 0 deletions .harness/history/execution-log.jsonl
Original file line number Diff line number Diff line change
Expand Up @@ -82,3 +82,7 @@
{"story_id": "story-148", "status": "completed", "retry_count": 1, "timestamp": "2026-09-14 21:49:01"}
{"story_id": "story-149", "status": "completed", "retry_count": 0, "timestamp": "2026-09-15 09:31:50"}
{"story_id": "story-150", "status": "completed", "retry_count": 0, "timestamp": "2026-09-15 12:47:39"}
{"story_id": "story-151", "status": "escalated", "retry_count": 0, "timestamp": "2026-09-15 15:48:02"}
{"story_id": "story-151", "status": "resumed", "retry_count": 0, "timestamp": "2026-09-16 07:36:26"}
{"story_id": "story-151", "status": "resumed", "retry_count": 0, "timestamp": "2026-09-16 07:36:54"}
{"story_id": "story-151", "status": "completed", "retry_count": 1, "timestamp": "2026-09-16 09:07:08"}
2 changes: 2 additions & 0 deletions .harness/history/inspection-log.jsonl
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,5 @@
{"story_id": "story-148", "mode": "narrow", "cost_usd": 3.7401380000000004, "scope_files": 60, "invocations": 1, "timestamp": "2026-09-14 21:37:12", "findings": 4, "filed": 0, "dropped": 0, "dedupe_ran": true}
{"story_id": "story-149", "mode": "narrow", "cost_usd": 1.749655, "scope_files": 60, "invocations": 1, "timestamp": "2026-09-15 09:20:14", "findings": 1, "filed": 0, "dropped": 0, "dedupe_ran": true}
{"story_id": "story-150", "mode": "narrow", "cost_usd": 2.4072125, "scope_files": 60, "invocations": 1, "timestamp": "2026-09-15 12:37:54", "findings": 1, "filed": 0, "dropped": 0, "dedupe_ran": true}
{"story_id": "story-151", "mode": "narrow", "cost_usd": 2.9910560000000004, "scope_files": 60, "invocations": 1, "timestamp": "2026-09-16 07:55:55", "findings": 1, "filed": 0, "dropped": 0, "dedupe_ran": true}
{"story_id": "story-151", "mode": "narrow", "cost_usd": 3.063886, "scope_files": 60, "invocations": 1, "timestamp": "2026-09-16 08:31:20", "findings": 1, "filed": 0, "dropped": 0, "dedupe_ran": true}
1 change: 1 addition & 0 deletions .harness/history/mandate-log.jsonl
Original file line number Diff line number Diff line change
Expand Up @@ -56,3 +56,4 @@
{"story_id": "story-148", "timestamp": "2026-09-14 20:41:55", "conferred_by": "jerod.wilkerson <30474318+jerodw@users.noreply.github.com>", "source_kind": "human", "recorded_by": "l5-plan", "discarded_session_block": false}
{"story_id": "story-149", "timestamp": "2026-09-15 08:36:13", "conferred_by": "jerod.wilkerson <30474318+jerodw@users.noreply.github.com>", "source_kind": "human", "recorded_by": "l5-plan", "discarded_session_block": false}
{"story_id": "story-150", "timestamp": "2026-09-15 12:12:00", "conferred_by": "jerod.wilkerson <30474318+jerodw@users.noreply.github.com>", "source_kind": "human", "recorded_by": "l5-plan", "discarded_session_block": false}
{"story_id": "story-151", "timestamp": "2026-09-15 14:59:07", "conferred_by": "jerod.wilkerson <30474318+jerodw@users.noreply.github.com>", "source_kind": "human", "recorded_by": "l5-plan", "discarded_session_block": false}
1 change: 1 addition & 0 deletions .harness/history/retry-log.jsonl
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,4 @@
{"story_id": "story-144", "timestamp": "2026-09-14 09:54:36", "stage": "verifier", "retry_decision": "retry", "retry_category": "implementation", "retry_stage": "implementer"}
{"story_id": "story-145", "timestamp": "2026-09-14 13:16:08", "stage": "verifier", "retry_decision": "retry", "retry_category": "implementation", "retry_stage": "implementer"}
{"story_id": "story-147", "timestamp": "2026-09-14 18:26:38", "stage": "verifier", "retry_decision": "retry", "retry_category": "validation", "retry_stage": "tester"}
{"story_id": "story-151", "timestamp": "2026-09-15 15:47:59", "stage": "tester", "retry_stage": "tester"}
Loading
Loading