Skip to content

Sonny: fleet observation costs grow with what is watched, not the square of the fleet - #538

Merged
lydakis merged 6 commits into
mainfrom
claude/project-thread-je4o8y
Oct 31, 2026
Merged

lydakis merged 6 commits into
mainfrom
claude/project-thread-je4o8y

Conversation

@lydakis

@lydakis lydakis commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

Requested by George · project thread

This PR covers the Sonny-side findings from the external fleet performance review of 1aeb7e5. The daemon-side findings are being built separately, in the order agreed with George.

What changes

  • Lookout keeps one subscriber registry. others() used to clone every other subscriber's name into each lookout's own HashSet, which is S × (S − 1) owned names: about 1.9 GiB of names alone at 10k subscribers. Now there is one shared Arc<HashSet<String>>, and the "not this lead" check is a predicate. When new subscribers appear, Updates::subscribed only looks up the names that lookout already holds news of. At startup, bots are seated through the watcher index instead of every lookout asking about every bot, and a newly placed lookout is added to the index by itself.
  • /status reads every page. It read one 256-agent page without following next_after, so the 257th agent never showed. It now pages through. An agent's latest history items (2 of its 3 calls per refresh) are re-read only once its history head moves, and a failed read is retried on the next refresh. Its turns are still read every time, because a queued turn leaves the bot record unchanged.
  • Sky replay stays bounded. Replay::keep only evicted once 10% of the tape had played, so neither the initial load nor a paused replay was bounded (200k loaded kept 200k; paused plus 200k more kept 280k). It now evicts the oldest tenth past TAPE regardless. If unplayed events go, a cut flag reseats the replay from the new start. The creations it folds are kept separately, matched by name and id, so the final listing doesn't drop them. drain() does at most half a FRAME of work per call. The hour's pair counts are kept as said changes, so hear(), land(), the curves and the stats look a pair up instead of scanning the hour.
  • A lead's layout follows less. Fleet used to follow * from cursor 0 with no kind filter, replay the whole log only to skip it, and decode every token event of every project. It now follows from now (since_ms) with only the 11 kinds it acts on. Where agents currently stand still comes from the listing (reconcile).
  • Draws are capped at 30 a second (Ink's default) for daemon events. Keys still draw immediately.
  • A budget for the processes Sonny runs for agents (sonny.tool_processes, in /config; 0 means twice the CPUs). Each companion has its own: the tool host's tool calls in sonny tool processes, and the lookout's goal checks. A call or check takes a place before it starts and holds it until done. No agent holds more than half, the wait counts against the call's or check's own time, and a wait that is given up leaves nothing behind (sonny/src/budget.rs).
  • sonny/FEATURES.md: history entries on the lookout, /status, full screen, Rex layout, sky, goals and /config rows.

Tests

  • New: status_lists_every_page_and_reads_again_only_the_histories_that_moved (it also covers a failed read being retried), the_hour_s_pair_counts_follow_what_was_said, who_the_tape_let_go_of_stays_in_the_cast_past_the_last_listing, who_the_tape_let_go_of_stays_though_its_name_is_used_again, the replay tape test extended with the reviewer's two cases, and budget tests (one agent takes at most half; no share is left behind, even when a wait is given up).
  • cargo test --bin sonny passes. cargo clippy --workspace --all-targets -D warnings is clean.

Still to come: a limit on auto-opened task windows with LRU close (after #533), browser sessions under a budget, and the perf gate covering sonny/**.

🤖 Generated with Claude Code

https://claude.ai/code/session_013jkdGF9fn2EXiBvzYrQtaA

…are of the fleet

From the fleet performance review of 1aeb7e5:
- Lookout: one subscriber registry every lookout shares, instead of a copy
  of all the other subscribers in each (S x S names); new subscribers drop
  news by looking up only what a lookout holds; bots are seated at start
  through the watcher index, and a new lookout joins the index alone.
- /status: lists every page of agents (the 257th was missing), and an idle
  agent whose record (head, status, turn) has not moved is not read again.
- Sky: the replay tape stays within TAPE while the past loads and while
  paused (unplayed events go too, and the replay is seated again from the
  new start); a frame drains at most half a frame's work; the hour's pair
  counts are kept as messages come, not counted again each frame.
- A lead's layout follows every agent from now, for the event kinds it
  acts on, instead of replaying the whole log from cursor 0 to skip it.
- Draws for the daemon's events are capped at 30 a second (Ink's cap);
  keys still draw at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jkdGF9fn2EXiBvzYrQtaA
@lydakis lydakis self-assigned this Oct 31, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-31T17:55:28.767069Z 07944cf New commits
🔒 Security Review ✅ Completed 2026-10-31T17:10:35.458937Z 971562b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 971562bc2d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sonny/src/main.rs Resolved
Comment thread sonny/src/sky.rs Resolved
- sonny.tool_processes (0 = twice the CPUs): tool calls the tool host
  answers in `sonny tool` processes and goal checks take a place before
  they start and hold it until done, no agent more than half, waiting
  within the call's or check's own time.
- /status reads every agent's turns each refresh again (a queued turn
  moves no field of the bot's record) and keeps only the latest history
  items, by history head.
- A replay keeps who its let-go tape start made apart, so the listing read
  once the past is in does not drop an agent deleted later on the tape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jkdGF9fn2EXiBvzYrQtaA
# Conflicts:
#	sonny/FEATURES.md

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f7a462a8ba

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sonny/src/budget.rs Resolved
Comment thread sonny/src/main.rs Resolved
…o share

/status kept a failed history read as an empty one, so an idle agent's
latest step stayed hidden after a reconnect. The read now says when it
failed, and only a good read is kept.

The process budget's per-agent share is now counted by its holders and
waiters, so an admission that times out before its place comes removes
the share too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jkdGF9fn2EXiBvzYrQtaA

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5d7c3d6901

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sonny/src/sky.rs Resolved
The last listing dropped a folded agent whenever it named someone with
the same name. Folded agents are now matched by name and id, as seek()
tells incarnations apart, so the original stays seated beside its
replacement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jkdGF9fn2EXiBvzYrQtaA

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1b3d208d5f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sonny/src/budget.rs Resolved
The tool host and the lookout are separate processes, each with its own
budget, so together they run at most twice sonny.tool_processes. The
module, the /config line and the feature log now say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jkdGF9fn2EXiBvzYrQtaA
@lydakis
lydakis merged commit b48cdcf into main Oct 31, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants