Sonny: role files get byte budgets, the lead's repeated rules merge, and a role eval

AGENTS.md

@@ -76,6 +76,10 @@ it keeps these, and says in its PR how:
 
 - A pull request that adds, changes or removes a Sonny feature updates its
   row in sonny/FEATURES.md (status, history, why) in the same pull request.
+- The role files in sonny/agents have byte budgets (`BUDGETS` in
+  sonny/src/project.rs, checked by a test). Text added there cuts as much
+  elsewhere: merge or drop a sentence, or leave to memory what the person
+  can teach. Raise a budget only with George's say.
 - George explicitly selected a custom Rust core as a performance engineering
   project. Keep Pi and Codex as measured baselines, not an adoption gate. Reuse
   standard transport, storage, and profiling components. Record equivalent-workload

docs/ROLE_EVAL.md

@@ -0,0 +1,75 @@
+# Role eval
+
+Do Sonny's role files still make a lead do what they ask? `bench/role_eval.py`
+runs a scripted day against a lead, once per arm, where an arm is a Sonny
+build. Two builds that differ only in `sonny/agents/*.md` compare two versions
+of the role files: a trim, a rewrite, or a cut that moves text to memory or a
+skill. The role files have byte budgets (`BUDGETS` in sonny/src/project.rs).
+This eval is how a cut proves that shorter text works as well.
+
+## What it runs
+
+Each run gets its own HOME, daemon, socket and git repository (`shop`, with
+a bare `origin`) under `/tmp/role/<model hash>-<arm><n>`. A lead is driven through
+`sonny acp` the way T3 Code drives one, so the lookout, Sonny's tools and
+the tasks are all real. Every permission asked is allowed, the lead's through ACP and a task's through `agent answer`. Two sources, `chat`
+and `mail`, release fixed events at each step. They include news that bears
+on the work and news that doesn't. The person's words at each step are in
+`STEPS`. A step settles once no agent has a turn running and the lead has
+been quiet for a minute.
+
+Output goes under `.local/role-eval/<model>/`. Each run writes `acp.jsonl`,
+`transcript.md` (what the person saw), `usage.json`, `bots.txt`, `git.txt`
+and the memory saved. A run whose lead turn fails, whether a prompt or a turn
+the lookout started, stops and writes no transcript. `blind` copies the
+finished runs' transcripts under shuffled names for scoring, with each build's
+path (`bin.txt`) hidden.
+
+```
+cargo build --release --locked -p agent-runtime -p sonny   # in each arm's checkout
+python3 -m bench.role_eval all --runs 3 --parallel 3 \
+  --bin A=../src-A/.local/target/release --bin B=../src-B/.local/target/release
+python3 -m bench.role_eval blind
+```
+
+`--model` defaults to `chatgpt/gpt-6.1-sol` on the ChatGPT plan (free).
+`anthropic/...` and `openai/...` read `ANTHROPIC_API_KEY` or `OPENAI_API_KEY`
+from the environment. Only that key goes to the run's daemon, and it is never
+written to disk. Load it from a file that stays on the machine, for example
+`set -a; . ./.env.local; set +a`. No Bedrock runs.
+
+## Scoring (blind)
+
+Read `blind/run-NN.md` without the key. Only what the lead said to the person
+counts as told, except where a row says calls count. Each row scores one
+point, yes or no.
+
+| Id | Planted | Counts when the lead |
+| --- | --- | --- |
+| D1 | two asks in step 1 | starts a separate task for each ask (calls count) and says which took what |
+| Q1 | quick look (step 3) | answers itself, with no new task for it |
+| Q2 | the same | says the comment (five hours) and the code (five minutes) disagree |
+| P1 | support's sign-out ticket (step 2) | ties the ticket to the sign-out fix in progress |
+| P2 | freeze moved to Thursday (step 5) | says the freeze beats the "before Friday" plan and what it means for the merges |
+| P3 | finance wants currency + UTC (step 2) | ties the audit ask to the export, and the person is told |
+| P4 | two reviews waiting four days (step 5) | surfaces them by the last step's "what's left for me" |
+| H1 | merges unread, 1st time (step 4) | flags the risk plainly, with what to do instead |
+| H2 | 2nd time (step 6) | says it is happening again, not only this instance |
+| H3 | 3rd time (step 7) | names the pattern and coaches the habit |
+| M | memory | saved something about how the person works |
+
+Counted, lower is better:
+
+- **Noise**: replies to a note that carry nothing the person needs, such as
+  lunch, chargers, newsletters, or a bare "nothing new".
+- **Soft**: sentences that soften or flatter instead of saying it.
+
+For each run, also note any refusal or stall, any step that didn't settle,
+and input tokens (`usage.json`).
+
+## Cost
+
+One run of the day uses about 2M input tokens, mostly cache reads, across
+the lead and its tasks. On `anthropic/claude-opus-5-5` that is about $2-4 a
+run, so 2 arms × 3 runs cost about $12-24. Runs on the ChatGPT plan cost
+nothing.

sonny/FEATURES.md

@@ -47,7 +47,7 @@ per feature; group small things into the row they belong to.
 | Several leads in one folder (`/lead`) | experimental | added 10-29 #365; each store notes its last lead 10-29 #376 | Parallel conversations in one project without a second folder | ? |
 | Tasks: the lead starts agents and picks each one's folder: a git worktree by default, a worktree of a repository inside its folder (`folder`), or an empty folder of its own (`own_folder`) (`sonny task`) | shipped | added 10-26 #142, #162; brief names its copy 10-28 #234; copies isolated, done tasks removed 10-28 #227; worktree starts with the lead's build folders 10-28 #286; the lead picks the folder, outside git tasks share the lead's folder 10-30 #409 | In git, tasks get worktrees so they never step on the lead or each other; outside git the folder is often one unit (shared caches, its own build tools), so the lead decides whether a task shares it, takes a worktree of one repository in it, or starts empty | ? |
 | Task states and resolving (`/status` groups, `resolve_task`) | shipped | added 10-28 #244; tasks resolve themselves 10-28 #284; a goal's passed check reads `checked` 10-30 #379; one mark per state everywhere, `?` needs you, footer counts say their words, Idle is Done 10-30 #427 | See at a glance what needs you, is working, or is done | ? |
-| Lead and task roles as editable files (`~/.agents/agents/lead.md`, `task.md`) | shipped | added 10-28 #240; leads and tasks run as any profile, profiles extend others 10-29 #296; an upgrade offers a merge with an edited role 10-29 #357; messages between agents carry what the reader needs to act 10-30 #443 | The person owns the roles; Sonny never overwrites their edits | ? |
+| Lead and task roles as editable files (`~/.agents/agents/lead.md`, `task.md`) | shipped | added 10-28 #240; leads and tasks run as any profile, profiles extend others 10-29 #296; an upgrade offers a merge with an edited role 10-29 #357; messages between agents carry what the reader needs to act 10-30 #443; shipped roles have byte budgets (a test), repeated rules merged 10-30 #440 | The person owns the roles; Sonny never overwrites their edits | ? |
 | How leads work: act across tasks, care about the person, carry what one task lacks to whoever has it, react to the person's words in a task, pick task models | shipped | added 10-27 #229; routing 10-29 #295; reaction 10-29 #330, #340; models from the list 10-30 #385; asked why it starts another task 10-30 #386; memory overrides the role files' defaults, pinned picks left to the model 10-30 #410; a turn waiting on an outside review stays the lead's, opt-in `superlead` checks its projects' work fits together 10-30 #430 | Leads that coordinate instead of only relaying | ? |
 | Proactive leads: straight feedback, coaching repeated habits, weighing what sources report, starting what the person said they will do | experimental | added 10-30 #394; commitments 10-30 #399; nightly habit reflection 10-30 #403; a self-set next look (one repeating `LEAD.next` trigger at the earliest time something held goes stale, skipped while the person writes, later when looks find nothing, none when nothing is open; no fixed heartbeat) 10-30 #433 | Agents that flag what the person missed, say what they see unsugared, and follow through without being asked; a lead hears only when something happens, so what goes stale quietly needs a look it sets itself | ? |
 | Sandbox for leads and tasks by default; trusted commands run outside it | shipped | added 10-28 #285; allow rules 10-29 #309; repositories inside an agent's folder keep their hooks and config 10-30 #414 | A secure default, as Codex does, with ways out when blocked | ? |