Repository navigation
feat(evolution): pin invariant rules outside the agent-writable tree — increment of #3518 - #3560
Conversation
…3518) The invariant filter now judges proposals against a pinned rules copy under ~/.hermes/evolution/pinned (override: $EVOLUTION_PINNED_RULES), installed once at deploy time by copying the in-tree seed there. The default load refuses to run when that surface is missing or shadowed by an in-tree copy, so a self-edit that rewrites scripts/evolution_invariant_rules.json cannot weaken the checks that govern it. An explicit path argument still reads as-is. Deferred (next increment): - wire the deploy-time copy into the installer so fresh installs seed the pinned surface automatically - extend the same pinning to the evidence gate and evolution stage contracts Co-Authored-By: Hermes Evolution <evolution@hermes.ai>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
૮ >ﻌ< ა ci reviewran on 96bfa84 — fix(evolution): seed the pinned invariant surface on deploy ❌ Job failuresNo production imports of the tools.lazy_deps stub / No production imports of the tools.lazy_deps stub · View jobJob No production imports of the tools.lazy_deps stub / No production imports of the tools.lazy_deps stub failed. Python lints / Windows footguns (blocking) · View jobJob Python lints / Windows footguns (blocking) failed. Python lints / ruff enforcement (blocking) · View jobJob Python lints / ruff enforcement (blocking) failed. Python tests / Run tests slice 1/8 · View jobJob Python tests / Run tests slice 1/8 failed. Python tests / Run tests slice 2/8 · View jobJob Python tests / Run tests slice 2/8 failed. Python tests / Run tests slice 3/8 · View jobJob Python tests / Run tests slice 3/8 failed. Python tests / Run tests slice 4/8 · View jobJob Python tests / Run tests slice 4/8 failed. Python tests / Run tests slice 5/8 · View jobJob Python tests / Run tests slice 5/8 failed. Python tests / Run tests slice 6/8 · View jobJob Python tests / Run tests slice 6/8 failed. Python tests / Run tests slice 7/8 · View jobJob Python tests / Run tests slice 7/8 failed. Python tests / Run tests slice 8/8 · View jobJob Python tests / Run tests slice 8/8 failed. debug infoCI timingsCI timings · View report · View jobWall time 17m24s vs 17m1s (+2.3%). 13 job(s) slower, 8 faster, 1 unchanged.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b5c4c2c875
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| # Slice 5 (#3518): the rules the filter judges against live OUTSIDE the tree | ||
| # the agent commits to, so a self-edit cannot weaken the checks that govern it. | ||
| _PINNED_RULES_DIR = Path.home() / ".hermes" / "evolution" / "pinned" |
There was a problem hiding this comment.
When HERMES_HOME points to an isolated installation or named profile, this import-time constant still reads rules from the user's default ~/.hermes directory. Consequently, a profile with its own pinned rules either rejects every proposal because the default file is absent or silently evaluates proposals against another profile's policy. Resolve the directory through get_hermes_home() at call time instead of capturing Path.home() at import. This violates the profile-aware path requirement in AGENTS.md.
AGENTS.md reference: AGENTS.md:L140-L145
Useful? React with 👍 / 👎.
| pinned = pinned_rules_path() | ||
| _assert_not_shadowed(pinned) | ||
| if not pinned.is_file(): |
There was a problem hiding this comment.
If EVOLUTION_PINNED_RULES points to scripts/evolution_invariant_rules.json, or to an external symlink targeting that file, the default loader accepts the agent-writable rules unchanged. _assert_not_shadowed() only checks two sibling filenames; it never verifies the location of the file actually being loaded. Both configurations therefore defeat this change's outside-the-tree invariant and allow subsequent in-tree edits to change the judging policy. Resolve the selected path and reject checkout-contained targets before loading it, while retaining the documented explicit-path audit API.
Useful? React with 👍 / 👎.
| @pytest.fixture(autouse=True) | ||
| def _hermetic_pinned_surface(tmp_path, monkeypatch): |
There was a problem hiding this comment.
On a clean test machine without an operator-installed pinned file, this module-local fixture only provisions rules for test_evolution_invariant_filter.py. Existing callers in tests/scripts/test_evolution_harness_gate.py remain unprovisioned: for example, test_run_gate_verdicts expects validated, but run_gate(PROPOSAL, gate_runner=_green) now returns invalid with rules-unavailable before invoking the runner. Move the provisioning into a shared fixture or provision the other consumer tests explicitly so the suite remains hermetic and preserves its existing gate coverage.
Useful? React with 👍 / 👎.
The default load now reads the rules from outside the checkout, so a fresh host failed closed with 'rules-unavailable' and the harness gate rejected every proposal: tests/scripts/test_evolution_harness_gate.py went 12/12 -> 2/12 on this branch. - install_pinned_rules() + '--install-pinned': idempotent deploy-time copy of the in-tree seed; refuses to overwrite a diverged surface without force. - The pinned dir resolves from $HERMES_HOME (host-level, not per-profile), so the surface sits where the rest of the host state does. - The fail-closed error names the exact command to run. - Harness-gate tests simulate a deployed host; installer + resolution have their own tests. tests/scripts -k 'evolution or harness or invariant': 26 failed on main and 26 on this branch, identical ids (pre-existing debt); 46/46 in the two touched modules.
|
Reviewed, amended, and merged. Blocker found on the branch: the default load now resolves the pinned surface, but nothing seeded it. On a host without Pushed
Verification (local, Python 3.14.7):
Merged as an admin squash: the required context Deploy note: each host needs |
Slice 2 of the replay-verified self-edits roadmap (#3515), child issue #3518.
What this slice ships
scripts/evolution_invariant_filter.pyjudged every proposal againstscripts/evolution_invariant_rules.json— a file inside the tree the agent commits to, so a self-edit could rewrite the rules it is judged against. The default load now resolves a pinned copy outside the checkout (~/.hermes/evolution/pinned/, override$EVOLUTION_PINNED_RULES), installed once at deploy time by copying the in-tree seed there. The filter refuses to run when that surface is missing or shadowed by an in-tree copy. An explicit path argument still reads as-is, so existing callers are unchanged.Tests
Checks
lint (blocking PLW1514) ✓, format ✓, pre-PR shard ✓, 198 changed lines.
Deferred (next increment):