Daemon companions: commands the daemon keeps running for as long as it runs

bench/socket_client.py

@@ -76,7 +76,7 @@ def close(self):
 
 
 class SocketClient:
-    def __init__(self, binary, path, url, tools, extra=(), settings=None):
+    def __init__(self, binary, path, url, tools, extra=(), settings=None, env=None):
         # Keep AF_UNIX paths short even in deep remote snapshot directories.
         self.directory = tempfile.TemporaryDirectory(prefix='agent-bench-', dir='/tmp')
         self.socket_path = Path(self.directory.name) / 'daemon.sock'
@@ -85,7 +85,7 @@ def __init__(self, binary, path, url, tools, extra=(), settings=None):
         self.process = subprocess.Popen(
             [str(binary), *serve_args(path, url, extra=extra), '--socket', str(self.socket_path)],
             stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
-            start_new_session=True, env=clean_env())
+            start_new_session=True, env=env or clean_env())
         try:
             deadline = time.monotonic() + 5
             while True:

docs/CLI.md

@@ -1,7 +1,7 @@
 # CLI contract
 
 Agent uses flat verbs: `run`, `follow`, `fork`, `interrupt`, `wait`, `ls`,
-`inspect`, `turns`, `usage`, `rm`, `prune`, `approvals`, `answer`, `models`, `start`, `stats`,
+`inspect`, `turns`, `usage`, `rm`, `prune`, `approvals`, `answer`, `models`, `companion`, `start`, `stats`,
 `shutdown`, and `serve`. All named
 bots use the same commands. There is no parent/child command hierarchy.
 
@@ -177,6 +177,40 @@ for both client commands and `serve`. Keep that registry, the implementation,
 and behavior tests aligned when adding an option. No CLI framework or runtime
 dependency is required.
 
+## Companions
+
+A companion is a command the daemon keeps running for as long as it runs:
+a client's helper that agents wait on, such as Sonny's lookout or the
+approver, so it lives as long as the daemon whoever started either. The
+daemon knows nothing of what one is.
+
+```sh
+agent companion add NAME [--cwd DIR] [--env KEY=VALUE]... -- COMMAND [ARGS]...
+agent companion rm NAME
+agent companion ls
+```
+
+`add` keeps the companion in the store under NAME (1 to 64 letters,
+digits, `.`, `_` or `-`, not starting with `.`) and starts it now; every
+later daemon on that store starts it once it listens. The same spec again
+changes nothing (`changed: false`); another spec under the name replaces
+the old one, which is stopped first. `rm` stops and forgets one
+(`removed: false` when there was none); `ls` lists them, as `stats` does
+under `companions`, each with its spec, `running`, `pid`, `started_ms`,
+`restarts` and `last_exit` (`code` and `signal`, or `spawn_failed` with
+why, and `at_ms`). Each runs in its own process group with the daemon's
+environment, its `--env` over that, and `AGENT_STORE`, `AGENT_SOCKET`,
+`AGENT_BIN` and `AGENT_COMPANION` (its name), never a bot shell's marks;
+its stdin is empty and its output goes to `STORE.companions/NAME.log`. One
+that exits starts again after 1 s, the wait doubling up to 60 s while it
+keeps exiting within a minute of starting. The daemon's shutdown stops
+each with SIGTERM to its group, then SIGKILL after 5 s. A companion's own
+sessions keep no daemon from `--idle-exit`, and a daemon on stdio, with no
+socket to reach it on, runs none. The request is `companions` with `add`
+(`name`, `command`, `args`, `env`, `cwd`), `remove` (a name) or neither;
+a sandboxed bot's commands may not make it, since a companion runs as the
+daemon does.
+
 ## Output and status
 
 Default output is machine-readable JSON. `run` and `follow` stream one JSON

docs/LOOKOUT.md

@@ -20,19 +20,23 @@ and the rest, `sonny/src/served.rs`).
 
 ## One lookout a daemon
 
-Sonny starts `sonny lookout --store STORE --socket SOCKET` detached when it
-opens a project, and again after the daemon restarts. A lookout holds
+`sonny lookout --store STORE --socket SOCKET` runs as the daemon's
+companion `sonny-lookout` (CLI.md, Companions): Sonny adds it when it opens
+a project, and the daemon keeps it running for as long as it runs, starts
+it again if it exits, and starts it with every later daemon on that store,
+whether or not a window is open. A lookout holds
 `STORE.lookout.lock` while it runs, so a second one prints
 `{"event":"already_watching"}` and exits, unless another build of Sonny
 started the one running, as an upgrade leaves it: that one is stopped, and
 the new one takes its place, so the tools the new Sonny gives its agents
-are answered. `STORE.lookout.sonny` says which build runs. It watches for every bot with
+are answered; the daemon's companion takes the place of one a window
+started detached, of any build. `STORE.lookout.sonny` says which build runs. It watches for every bot with
 subscriptions on the daemon, whichever Sonny opened it, so a lead hears
 from its tasks even while no window shows that project. It exits
-once the daemon has been unreachable for 30 seconds; the Sonny that starts
-the next daemon starts the next lookout. Its own lines go to
-`STORE.lookout.log`, and each subscriber it watches (`seated`) and each
-note it sends (`woke`) to `STORE.lookout.jsonl`.
+once the daemon has been unreachable for 30 seconds, and the daemon stops
+it as it stops. Its own lines go to `STORE.companions/sonny-lookout.log`,
+and each subscriber it watches (`seated`) and each note it sends (`woke`)
+to `STORE.lookout.jsonl`.
 
 It keeps a cursor in `STORE.lookout.cursor`: the newest event it saw, or
 just before the oldest one whose news it has not told yet. A lookout that

sonny/README.md

@@ -50,7 +50,9 @@ folder under `~/.agent`.
 
 `~/.agent/env` (`KEY=VALUE` lines, readable only by you) holds the provider
 keys, and `AWS_PROFILE` for Bedrock, which Sonny applies over your shell's to
-the daemon, the approver and the lookout. Settings go in
+the daemon it starts; the approver and the lookout run as that daemon's
+companions, with its environment, so they live as long as it does whether
+or not a window is open. Settings go in
 `~/.agent/config.toml`, as Codex keeps them: `provider` (the daemon's
 `--provider` specs, after the shell's `AGENT_PROVIDER`), `model` (a new
 agent's when none is named) and the efforts below. When no provider is named
@@ -176,7 +178,7 @@ notifies, `herdr agent wait` and `herdr agent prompt` reach it by name
 0.9.2 lists Sonny but takes no command to restart it). Sonny reports from a
 pane of the Herdr server on its own machine: one reached by plain `ssh` from a
 Herdr pane is not listed, one in a remote machine Herdr itself connects to is.
-The daemon, lookout and approver Sonny starts run in a session of their own,
+The daemon Sonny starts, and the lookout and approver it keeps, run in a session of their own,
 so closing their pane or stopping Herdr leaves them and their turns running.
 In cmux, which reads OSC 7501 and OSC 777 itself, the workspace's sidebar row
 shows which task is working, waiting on an approval or done, and a
@@ -188,8 +190,9 @@ then cmux offers it to run by hand); quitting Sonny clears it. cmux lets
 only programs started in its tabs reach it, by default; agents' shells run
 in the daemon, not in cmux, so set cmux's Settings > Automation to
 "Automation mode" (your own programs, no ancestry check) for agents to run
-commands in tabs and read them. The lookout Sonny starts in cmux reaches it
-either way; one started outside cmux needs Automation mode too.
+commands in tabs and read them. The lookout, the daemon's companion, reaches it
+either way when the daemon started from a cmux tab; otherwise it needs
+Automation mode too.
 The place a
 project was laid out in is kept in `~/.agent/sonny/places`, so an agent's
 `run_in_pane` and `show_agent` calls go to that Herdr or cmux workspace or

sonny/src/acp.rs

@@ -1500,14 +1500,14 @@ async fn services(
     if let Some(project) = project {
         lookout::subscribe_lead(&agent.options.store, bot, project)?;
     }
-    crate::start_lookout(&agent.options)?;
+    crate::start_lookout(client).await?;
     if approvals::mode(&record) == Some("auto") {
         let model = format!(
             "{}/{}",
             record["provider"].as_str().unwrap_or(""),
             record["model"].as_str().unwrap_or("")
         );
-        crate::serve_auto(client, &agent.options, &model).await?;
+        crate::serve_auto(client, &model).await?;
     }
     Ok(())
 }
@@ -1586,14 +1586,14 @@ async fn setup_lead(
         }
     }
     lookout::subscribe_lead(&agent.options.store, &lead.bot, &lead.project)?;
-    crate::start_lookout(&agent.options)?;
+    crate::start_lookout(client).await?;
     if gating.or(mode) == Some("auto") {
         let model = format!(
             "{}/{}",
             record["provider"].as_str().unwrap_or(""),
             record["model"].as_str().unwrap_or("")
         );
-        crate::serve_auto(client, &agent.options, &model).await?;
+        crate::serve_auto(client, &model).await?;
     }
     Ok(())
 }

sonny/src/lookout/mod.rs

@@ -292,7 +292,7 @@ fn beside(store: &Path, suffix: &str) -> PathBuf {
 
 /// This Sonny, told apart from another build of it: its version and when
 /// its binary was written.
-fn identity() -> String {
+pub(crate) fn identity() -> String {
     let written = std::env::current_exe()
         .and_then(std::fs::metadata)
         .and_then(|meta| meta.modified())
@@ -308,7 +308,9 @@ fn identity() -> String {
 /// when the lookout holding it is this Sonny's own.
 fn replace(store: &Path) -> Result<Option<std::fs::File>, String> {
     let theirs = std::fs::read_to_string(beside(store, ".lookout.sonny")).unwrap_or_default();
-    if theirs == identity() {
+    // The daemon's own companion takes the place of one a window started
+    // before Sonny kept it as one, of this build too.
+    if theirs == identity() && std::env::var_os("AGENT_COMPANION").is_none() {
         return Ok(None);
     }
     let pid: Option<i32> = std::fs::read_to_string(beside(store, ".lookout.lock"))
@@ -343,8 +345,14 @@ fn lock(store: &std::path::Path) -> Result<Option<std::fs::File>, String> {
         .open(&path)
         .map_err(|e| format!("{}: {e}", std::path::Path::new(&path).display()))?;
     // SAFETY: flock on a descriptor `file` owns and keeps open; the lock is
-    // released when the process exits.
-    if unsafe { libc::flock(file.as_raw_fd(), libc::LOCK_EX | libc::LOCK_NB) } == 0 {
+    // released when the process exits. Tried twice, since `running` holds
+    // it for an instant.
+    let take = || unsafe { libc::flock(file.as_raw_fd(), libc::LOCK_EX | libc::LOCK_NB) } == 0;
+    let taken = take() || {
+        std::thread::sleep(Duration::from_millis(50));
+        take()
+    };
+    if taken {
         // Its process, which `/status` asks after.
         use std::io::Write;
         let _ = file.set_len(0);
@@ -1010,14 +1018,24 @@ fn cursor_path(store: &Path) -> PathBuf {
 /// The most of the log `status` reads, from its end.
 const STATUS_BYTES: u64 = 1 << 20;
 
+/// Whether a lookout watches `store`'s daemon: one holds its lock. The pid
+/// the file names is no answer, since after a restart another process may
+/// have it. The lock is tried shared and let go at once; a lookout
+/// starting meanwhile tries again (`lock`).
+pub fn running(store: &std::path::Path) -> bool {
+    use std::os::fd::AsRawFd;
+    let Ok(file) = std::fs::File::open(beside(store, ".lookout.lock")) else {
+        return false;
+    };
+    // SAFETY: flock on a descriptor `file` owns; closing it lets go.
+    let taken = unsafe { libc::flock(file.as_raw_fd(), libc::LOCK_SH | libc::LOCK_NB) } == 0;
+    !taken && std::io::Error::last_os_error().raw_os_error() == Some(libc::EWOULDBLOCK)
+}
+
 /// One line on the lookout for a subscriber, as `/status` shows it:
 /// whether one runs and watches it, and today's notes.
 pub fn status(store: &std::path::Path, lead: &str) -> String {
-    let pid: Option<i32> = std::fs::read_to_string(beside(store, ".lookout.lock"))
-        .ok()
-        .and_then(|pid| pid.trim().parse().ok());
-    // SAFETY: signal 0 sends nothing; it only asks whether the process lives.
-    if !pid.is_some_and(|pid| pid > 0 && unsafe { libc::kill(pid, 0) } == 0) {
+    if !running(store) {
         return "Lookout · not running".to_owned();
     }
     let (mut seated, mut notes) = (false, 0);
@@ -2339,6 +2357,24 @@ mod tests {
         assert!(lookout.updates.due());
     }
 
+    #[test]
+    fn a_lookout_runs_while_one_holds_its_lock_whatever_pid_it_names() {
+        let dir = std::env::temp_dir().join(format!("lookout-running-{}", std::process::id()));
+        std::fs::create_dir_all(&dir).unwrap();
+        let store = dir.join("state.sqlite");
+        assert!(!running(&store));
+        // A lookout gone before a restart, its pid now another process's.
+        std::fs::write(dir.join("state.sqlite.lookout.lock"), "1").unwrap();
+        assert!(!running(&store));
+        let held = lock(&store).unwrap().unwrap();
+        assert!(running(&store));
+        // Asking leaves it held.
+        assert!(lock(&store).unwrap().is_none());
+        drop(held);
+        assert!(!running(&store));
+        let _ = std::fs::remove_dir_all(&dir);
+    }
+
     #[test]
     fn status_says_whether_it_watches_from_the_start() {
         let dir = std::env::temp_dir().join(format!("lookout-status-{}", std::process::id()));