diff --git a/docs/SONNY.md b/docs/SONNY.md index b4bdce762..f183d79d1 100644 --- a/docs/SONNY.md +++ b/docs/SONNY.md @@ -28,6 +28,9 @@ Sonny's lookout answers outside the agent's sandbox, since a trigger is a job of the system's scheduler: `add` (the fields below, as `every`, `cron`, `start {name, model, effort}`, `reply_to`, `if`, `runs`, `turn_budget_tokens`, `replace` and `message`), `remove`, `fire` or `show` NAME, or nothing to list them. +A row whose last fire sent a turn says how that turn ended, as the daemon +has it: `last.turn_status`, and `last.turn_error` when it did not complete, +so an agent sees the failure the person sees in `/background`. It waits for a verdict wherever the agent's other calls do (`/approvals`), since a trigger runs as the person and its `if` command outside any sandbox. For people, scripts and other harnesses, Sonny writes diff --git a/docs/sonny/background-failed.png b/docs/sonny/background-failed.png new file mode 100644 index 000000000..7fae0aa70 Binary files /dev/null and b/docs/sonny/background-failed.png differ diff --git a/docs/sonny/footer-failed.png b/docs/sonny/footer-failed.png new file mode 100644 index 000000000..019c73b48 Binary files /dev/null and b/docs/sonny/footer-failed.png differ diff --git a/sonny/FEATURES.md b/sonny/FEATURES.md index fa1053b8e..19220fa7c 100644 --- a/sonny/FEATURES.md +++ b/sonny/FEATURES.md @@ -110,7 +110,7 @@ per feature; group small things into the row they belong to. | Lookout: tells a lead what its tasks did | shipped | added 10-26 #154, #161; one per daemon 10-27 #180; subscriptions 10-27 #195; subscribers only, leads subscribed 10-27 #212; news within a minute 10-27 #228, #232; a companion of the daemon 10-29 #370; a turn's end carries its reply 10-30 #378; stays light with thousands of bots 10-30 #398; a task's question is news 10-30 #395 | A lead can rest while tasks work and hear the moment they need it | ? | | Lookout judge: a model decides what interrupts the lead (drift, interruptions, look-fors) | removed | added 10-26 #154; judge forks expire 10-27 #203; drift check 10-27 #204; interruptions only 10-27 #215; one judge with the approver 10-27 #217; picks 10-28 #239, #273; removed 10-29 #310 | Removed because the checks felt ad hoc and did not scale; the lookout now forwards everything and only picks the timing | ? | | Lookout sources: pollers the person sets up feed news to agents | experimental | added 10-29 #305 | Agents hear about the world, not only their tasks | ? | -| `/background`: triggers and lookout subscriptions, fire or remove | shipped | added 10-28 #267; in the pane 10-28 #283 | See and control what runs while nobody watches | ? | +| `/background`: triggers and lookout subscriptions, fire or remove | shipped | added 10-28 #267; in the pane 10-28 #283; what failed first, `βœ— N failed` in the footer and the glance, turn failures in the triggers tool 10-30 | See and control what runs while nobody watches; a nightly run that failed while the laptop was offline or a login expired otherwise fails unseen | ? | ## Approvals diff --git a/sonny/README.md b/sonny/README.md index 9fb300e05..6b44e8e74 100644 --- a/sonny/README.md +++ b/sonny/README.md @@ -480,6 +480,18 @@ one of its tasks, with its name (`πŸ‘€ calc.lead`) under the agent's own. day. `↑`/`↓` picks one, `f` fires a trigger now, and `d` pressed twice removes it; the subscription Sonny keeps for a lead's tasks stays. The footer says `↻ 2 in background` while the window's agent has any. + What failed is listed first, with a red `βœ—` and the error: a trigger + whose fire failed (one that ended on it stays listed until `d` removes + it), or whose fire went out and whose turn then failed (as + when the machine was offline or a login expired), and a server the + nightly memory cleanup could not read. Your own agents' failures, such as + the memory cleanup's, show in every window. The footer then says + `βœ— 1 failed` in red, left of the `↻`, and a click on either opens + `/background`. An agent sees the same in its triggers tool, whose rows + say how the turn their last fire sent ended. + + ![/background listing a nightly run whose turn failed offline and a server the cleanup could not read](../docs/sonny/background-failed.png) + ![The footer: βœ— 2 failed in red, left of ↻ 1 in background](../docs/sonny/footer-failed.png) ![/background in the pane](../docs/sonny/pane-background.png) - `/usage` shows, in the pane beside the conversation, @@ -767,8 +779,9 @@ the conversation, from the newest place back: ↑ or Enter steps older, ↓ newer, and Esc stops where you are. The pane's place, when nothing else is open there, is the margin. Its -first row is a glance at what runs for the agent: `↻` for its triggers -and subscriptions (a click opens `/background`), and its context, each +first row is a glance at what runs for the agent: a red `βœ—` for what +failed in the background and `↻` for its triggers and subscriptions (a +click on either opens `/background`), and its context, each left out when there is nothing to say; the footer stops saying them while it shows. Under the composer, the footer counts the other agents at work (cyan `●`) and what needs you (yellow `●`), and a click on the count diff --git a/sonny/src/background.rs b/sonny/src/background.rs index 8fa9dd440..39eb727e8 100644 --- a/sonny/src/background.rs +++ b/sonny/src/background.rs @@ -4,20 +4,28 @@ //! for them (`lookout::subs`). From there a trigger is fired now or //! removed, and a subscription removed; the one Sonny writes for a lead //! stays, since its project opening writes it again. How many an agent -//! has is the `↻` its footer and its `/status` row show. +//! has is the `↻` its footer and its `/status` row show. What failed comes +//! first, and is the `βœ—` beside it: a trigger whose fire failed, or whose +//! fire went out and whose turn then failed, as when the machine was +//! offline or a login expired, the daemon saying how the turn ended; and a +//! server the nightly memory cleanup could not read, by the fact it saves. use crate::{lookout, trigger}; use ratatui::crossterm::event::{KeyCode, KeyEvent, KeyModifiers}; use ratatui::{ - style::Stylize, + style::{Color, Stylize}, text::{Line, Span}, }; use serde_json::Value; +use std::collections::HashMap; use std::path::{Path, PathBuf}; #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum Kind { Trigger, Subscription, + /// A server the nightly memory cleanup could not read: its + /// `unreadable-SERVER` fact. + Server, } #[derive(Clone, Debug, PartialEq)] @@ -41,15 +49,27 @@ pub struct Item { /// Sonny's own: a lead's `tasks` subscription. Never counted, never /// removed. pub own: bool, + /// Why it failed, when its last fire or the turn it sent did, or the + /// server could not be read. + pub failed: Option, + /// The turn a trigger's last fire sent, whose end `turns` tells. + pub handle: Option, } /// Every trigger made on the daemon whose store is `store_id` and every -/// subscription beside `store` that is for an agent `ours` takes. -pub fn read(store: &Path, store_id: Option<&str>, ours: impl Fn(&str) -> bool) -> Vec { +/// subscription beside `store` that is for an agent `ours` takes, each +/// trigger's last fire with how its turn ended where `turns` has it. +pub fn read( + store: &Path, + store_id: Option<&str>, + turns: &HashMap, + ours: impl Fn(&str) -> bool, +) -> Vec { let mut items = Vec::new(); if let (Some(id), Ok(places)) = (store_id, trigger::Places::home()) { - for (row, bots) in trigger::on_store(&places, id) { + for (mut row, bots) in trigger::on_store(&places, id) { if bots.iter().any(|bot| ours(bot)) { + trigger::with_turn(&mut row, turns); items.push(trigger_item(&row, bots)); } } @@ -79,17 +99,123 @@ pub fn read(store: &Path, store_id: Option<&str>, ours: impl Fn(&str) -> bool) - }, reply_to: None, own, + failed: None, + handle: None, }); } } items } +/// The person's own agents, which no project names, as the one the nightly +/// memory cleanup starts: what fails for them is the person's to know in +/// every window. +pub fn personal(bot: &str) -> bool { + !bot.contains('.') +} + +/// Whether a window of `scope` shows `item`: one of its agents is the +/// scope's, or it failed and is the person's own. +pub fn shown(scope: &Scope, item: &Item) -> bool { + item.bots.iter().any(|bot| scope.takes(bot)) + || (item.failed.is_some() && item.bots.iter().all(|bot| personal(bot))) +} + +/// What a window of `scope` lists: its triggers and subscriptions, the +/// person's own that failed, and the servers the nightly memory cleanup +/// could not read, for the person and the project. +pub fn read_scope( + store: &Path, + store_id: Option<&str>, + turns: &HashMap, + scope: &Scope, +) -> Vec { + let mut items = read_around(store, store_id, turns, scope); + items.retain(|item| shown(scope, item)); + items +} + +/// [`read_scope`] before what it does not show is left out: the person's +/// own triggers too, whose turns must be asked about before it is known +/// whether they failed. +pub fn read_around( + store: &Path, + store_id: Option<&str>, + turns: &HashMap, + scope: &Scope, +) -> Vec { + let mut items = read(store, store_id, turns, |bot| { + scope.takes(bot) || personal(bot) + }); + if let Some(home) = std::env::var_os("HOME") { + let root = PathBuf::from(home).join(".agents/memory"); + items.extend(servers(&root, &scope.project)); + } + items +} + +/// The `unreadable-SERVER` facts the nightly memory cleanup saves, the +/// person's and `project`'s, each as the server it names. +fn servers(root: &Path, project: &str) -> Vec { + let view = agent_client::memory::view(root, Some(project)); + let scopes = + std::iter::once(&view["user"]).chain(view["projects"].as_array().into_iter().flatten()); + let mut items = Vec::new(); + for scope in scopes { + for fact in scope["facts"].as_array().into_iter().flatten() { + let name = fact["name"].as_str().unwrap_or_default(); + let Some(server) = name.strip_prefix("unreadable-").filter(|s| !s.is_empty()) else { + continue; + }; + let why = fact["description"].as_str().unwrap_or_default(); + let mut facts = vec![format!("memory fact {name}")]; + if let Some(project) = scope["name"].as_str() { + facts[0].push_str(&format!(" in {project}")); + } + if let Some(at) = fact["modified"].as_u64().filter(|at| *at > 0) { + facts.push(format!("saved {}", ago(now_ms().saturating_sub(at) / 1000))); + } + facts.push("read again each night".into()); + items.push(Item { + kind: Kind::Server, + name: server.to_owned(), + bot: String::new(), + bots: Vec::new(), + when: String::new(), + what: String::new(), + facts, + reply_to: None, + own: false, + failed: Some(match why.is_empty() { + true => "could not be read".into(), + false => first_line(why), + }), + handle: None, + }); + } + } + items +} + +/// How many of `items` failed. +pub fn failed(items: &[Item]) -> usize { + items.iter().filter(|item| item.failed.is_some()).count() +} + +/// The turns `items`' last fires sent whose end `turns` does not have yet. +pub fn unended(items: &[Item], turns: &HashMap) -> Vec { + (items.iter()) + .filter_map(|item| item.handle.clone()) + .filter(|handle| !turns.contains_key(handle)) + .collect() +} + fn trigger_item(row: &Value, bots: Vec) -> Item { let text = |key: &str| row[key].as_str().unwrap_or_default().to_owned(); let mut facts = Vec::new(); let sent = row["sent"].as_u64().unwrap_or(0); let last = &row["last"]; + let mut failed = None; facts.push(match row["runs"].as_u64() { Some(runs) => format!("sent {sent} of {runs}"), None if sent == 0 && last.is_null() => "not fired yet".into(), @@ -101,14 +227,45 @@ fn trigger_item(row: &Value, bots: Vec) -> Item { .map(|at| format!(" {}", ago(now_ms().saturating_sub(at) / 1000))) .unwrap_or_default(); let detail = match last["detail"].as_str() { + Some(detail) if outcome == "failed" => { + failed = Some(first_line(detail)); + String::new() + } Some(detail) if outcome != "sent" => format!(": {}", first_line(detail)), _ => String::new(), }; facts.push(format!("last {outcome}{ago}{detail}")); + match last["turn_status"].as_str() { + Some("completed") | None => {} + // A turn the person stopped, or steered into another, ended as + // they asked. + Some(status) if status != "failed" => facts.push(format!("its turn {status}")), + Some(_) => { + facts.push("its turn failed".into()); + failed = Some(match last["turn_error"].as_str() { + Some(error) if !error.is_empty() => first_line(error), + _ => "its turn failed".into(), + }); + } + } + if outcome == "failed" && failed.is_none() { + failed = Some("failed".into()); + } + // Its turn went well and its answer did not reach `--reply-to`. + if last["reply"]["outcome"] == "failed" && failed.is_none() { + facts.push("its answer was not passed on".into()); + failed = Some(match last["reply"]["detail"].as_str() { + Some(detail) if !detail.is_empty() => first_line(detail), + _ => "its answer was not passed on".into(), + }); + } } if row["missed"] == true { facts.push("missed its time".into()); } + if row["ended"] == true { + facts.push("ended".into()); + } if let Some(gate) = row["if"].as_str() { facts.push(format!("if {}", first_line(gate))); } @@ -122,6 +279,8 @@ fn trigger_item(row: &Value, bots: Vec) -> Item { facts, reply_to: row["reply_to"].as_str().map(str::to_owned), own: false, + failed, + handle: trigger::sent(row), } } @@ -188,6 +347,9 @@ pub struct View { /// The window's own agent, whose lookout line heads the card. bot: String, items: Vec, + /// How the turns triggers' last fires sent ended, as the window last + /// asked the daemon. + turns: HashMap, lookout: String, selected: usize, /// The item the first `d` picked, by kind, agent and name. @@ -203,13 +365,20 @@ pub enum Key { } impl View { - pub fn open(scope: Scope, store: PathBuf, store_id: Option, bot: String) -> Self { + pub fn open( + scope: Scope, + store: PathBuf, + store_id: Option, + bot: String, + turns: HashMap, + ) -> Self { let mut view = Self { scope, store, store_id, bot, items: Vec::new(), + turns, lookout: String::new(), selected: 0, confirm: None, @@ -222,10 +391,13 @@ impl View { /// Read it all again, the pick kept on the same row where it still is. pub fn reload(&mut self) { let picked = self.items.get(self.selected).map(key_of); - let mut items = read(&self.store, self.store_id.as_deref(), |b| { - self.scope.takes(b) - }); - items.sort_by_key(|item| item.kind == Kind::Subscription); + let mut items = read_scope( + &self.store, + self.store_id.as_deref(), + &self.turns, + &self.scope, + ); + items.sort_by_key(|item| (item.failed.is_none(), item.kind != Kind::Trigger)); self.items = items; self.selected = picked .and_then(|k| self.items.iter().position(|i| key_of(i) == k)) @@ -238,6 +410,14 @@ impl View { self.lookout = lookout::status(&self.store, &self.bot); } + /// How the turns ended, as the window has since asked the daemon. + pub fn set_turns(&mut self, turns: &HashMap) { + if *turns != self.turns { + self.turns = turns.clone(); + self.reload(); + } + } + /// The item a removal was asked for is kept: a key went elsewhere. pub fn keep(&mut self) { self.confirm = None; @@ -272,6 +452,10 @@ impl View { "Sonny keeps the lead's tasks subscription; the lookout watches its tasks with it" .into(), ); + } else if item.kind == Kind::Server { + self.notice = Some( + "the nightly cleanup removes it the night the server reads again".into(), + ); } else if confirm == Some(key_of(&item)) { self.remove(&item); } else { @@ -308,6 +492,7 @@ impl View { Kind::Subscription => { lookout::subs::remove(&self.store, &item.bot, &item.name).map(drop) } + Kind::Server => return, }; self.notice = Some(match done { Ok(()) => format!("removed {}", item.name), @@ -337,11 +522,15 @@ impl View { ); return (out, 0); } - for (kind, heading) in [ - (Kind::Trigger, "Triggers"), - (Kind::Subscription, "Lookout subscriptions"), - ] { - let n = self.items.iter().filter(|i| i.kind == kind).count(); + // What failed first, whatever it is; then the rest by kind. + let section = |item: &Item| match (&item.failed, item.kind) { + (Some(_), _) => "Failed", + (None, Kind::Subscription) => "Lookout subscriptions", + (None, _) => "Triggers", + }; + for heading in ["Failed", "Triggers", "Lookout subscriptions"] { + let takes = |item: &Item| section(item) == heading; + let n = self.items.iter().filter(|i| takes(i)).count(); if n == 0 { continue; } @@ -351,7 +540,7 @@ impl View { Span::raw(format!(" Β· {n}")).dim(), ])); for (index, item) in self.items.iter().enumerate() { - if item.kind != kind { + if !takes(item) { continue; } if index == self.selected { @@ -365,13 +554,19 @@ impl View { fn item_lines(&self, item: &Item, picked: bool, width: usize) -> Vec> { let mut out = Vec::new(); + // The pick has a column of its own, as the composer's `β€Ί ` does, + // so it never touches the row's mark; what follows lines up under + // the name. let pick = match picked { - true => Span::raw("β€Ί").bold(), - false => Span::raw(" "), + true => Span::raw("β€Ί ").bold(), + false => Span::raw(" "), }; + // Sonny's one mark for what failed, as `/status` shows a task. let mark = match item.kind { + _ if item.failed.is_some() => Span::raw("βœ— ").fg(Color::Red), Kind::Trigger => Span::raw("↻ ").dim(), Kind::Subscription => Span::raw("β—Ž ").dim(), + Kind::Server => Span::raw(" "), }; let name = match picked { true => Span::raw(item.name.clone()).bold().reversed(), @@ -386,13 +581,24 @@ impl View { Kind::Subscription => { spans.push(Span::raw(format!(" for {}", self.scope.short(&item.bot))).dim()); } + Kind::Server => { + spans.push(Span::raw(" could not be read by the nightly memory cleanup").dim()); + } } out.extend(crate::transcript::wrap( Line::from(spans), width, Span::raw(""), - Span::raw(" "), + Span::raw(INDENT), )); + if let Some(why) = &item.failed { + out.extend(crate::transcript::wrap( + Line::from(why.clone()), + width, + Span::raw(INDENT), + Span::raw(INDENT), + )); + } let mut facts = item.facts.clone(); if let Some(to) = item.reply_to.as_ref().filter(|to| **to != item.bot) { facts.push(format!("answers go to {}", self.scope.short(to))); @@ -401,15 +607,15 @@ impl View { out.extend(crate::transcript::wrap( Line::from(facts.join(" Β· ")).dim(), width, - Span::raw(" "), - Span::raw(" "), + Span::raw(INDENT), + Span::raw(INDENT), )); } if !item.what.is_empty() { out.extend(crate::transcript::capped( Line::from(item.what.clone()).dim(), width, - Span::raw(" β”” ").dim(), + Span::raw(format!("{INDENT}β”” ")).dim(), 1, )); } @@ -435,7 +641,7 @@ impl View { } match self.items.get(self.selected) { Some(item) if item.kind == Kind::Trigger => keys.extend(["f fires now", "d removes"]), - Some(item) if !item.own => keys.push("d removes"), + Some(item) if !item.own && item.kind != Kind::Server => keys.push("d removes"), _ => {} } keys.push("esc to close"); @@ -443,6 +649,10 @@ impl View { } } +/// Where a row's lines after the first start: under its name, past the +/// pick and the mark. +const INDENT: &str = " "; + fn key_of(item: &Item) -> (Kind, String, String) { (item.kind, item.bot.clone(), item.name.clone()) } @@ -489,6 +699,8 @@ mod tests { facts: vec!["sent 2".into()], reply_to: None, own, + failed: None, + handle: None, } } @@ -522,16 +734,190 @@ mod tests { "last": {"outcome": "failed", "fired_ms": fired, "detail": "bot_busy: it works"}}); let item = trigger_item(&row, vec!["calc.lead".into()]); assert_eq!(item.what, "Read the inbox…"); - assert_eq!( - item.facts, - ["sent 4 of 10", "last failed 3h ago: bot_busy: it works"] - ); + assert_eq!(item.facts, ["sent 4 of 10", "last failed 3h ago"]); + assert_eq!(item.failed.as_deref(), Some("bot_busy: it works")); assert_eq!(item.reply_to.as_deref(), Some("calc.lead")); let fresh = trigger_item( &json!({"name": "new", "when": "fire", "bot": "calc.lead", "sent": 0, "last": null}), vec!["calc.lead".into()], ); assert_eq!(fresh.facts, ["not fired yet"]); + assert_eq!(fresh.failed, None); + } + + #[test] + fn a_fire_that_went_out_fails_when_its_turn_did() { + let row = |handle: &str| { + json!({"name": "nightly", "when": "cron 0 5 * * *", "bot": "memory-cleanup", + "message": "Clean up memory", "sent": 3, + "last": {"outcome": "sent", "handle": handle, "fired_ms": now_ms()}}) + }; + let turns = HashMap::from([ + ( + "h1".to_owned(), + json!({"status": "failed", "error": "provider_failed", "detail": "login expired\nsign in again"}), + ), + ("h2".to_owned(), json!({"status": "interrupted"})), + ( + "h3".to_owned(), + json!({"status": "completed", "text": "No changes."}), + ), + ]); + let ended = |handle: &str| { + let mut row = row(handle); + trigger::with_turn(&mut row, &turns); + trigger_item(&row, vec!["memory-cleanup".into()]) + }; + let failed = ended("h1"); + assert_eq!( + failed.failed.as_deref(), + Some("provider_failed: login expired…") + ); + assert_eq!( + failed.facts, + ["sent 3", "last sent 0s ago", "its turn failed"] + ); + assert_eq!(failed.handle.as_deref(), Some("h1")); + // Stopped by the person: how it ended, not a failure. + let stopped = ended("h2"); + assert_eq!(stopped.failed, None); + assert_eq!(stopped.facts[2], "its turn interrupted"); + assert_eq!(ended("h3").failed, None); + // Its answer did not reach the agent it was for. + let mut lost = row("h3"); + lost["reply_to"] = json!("calc.lead"); + lost["last"]["reply"] = json!({"outcome": "failed", "detail": "bot_not_found: calc.lead"}); + trigger::with_turn(&mut lost, &turns); + let lost = trigger_item(&lost, vec!["memory-cleanup".into()]); + assert_eq!(lost.failed.as_deref(), Some("bot_not_found: calc.lead")); + assert_eq!(lost.facts[2], "its answer was not passed on"); + // Not asked yet: the window asks the daemon for it. + let running = ended("h4"); + assert_eq!(running.failed, None); + assert_eq!(unended(&[running, failed], &turns), ["h4"]); + } + + #[test] + fn what_fails_for_the_person_shows_in_every_window_and_a_server_by_its_fact() { + let scope = Scope { + project: "calc".into(), + lead: "calc.lead".into(), + glob: "calc.*".into(), + }; + let mut cleanup = item(Kind::Trigger, "memory", &["memory-cleanup"], false); + assert!(!shown(&scope, &cleanup)); + cleanup.failed = Some("provider_failed".into()); + assert!(shown(&scope, &cleanup)); + let mut other = item(Kind::Trigger, "digest", &["web.lead"], false); + other.failed = Some("bot_busy".into()); + assert!(!shown(&scope, &other)); + assert!(shown( + &scope, + &item(Kind::Trigger, "nightly", &["calc.lead"], false) + )); + + let root = std::env::temp_dir().join(format!("background-servers-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + let fact = |dir: &Path, name: &str, description: &str| { + std::fs::create_dir_all(dir).unwrap(); + std::fs::write( + dir.join(format!("{name}.md")), + format!("---\nname: {name}\ndescription: {description}\ntype: reference\nsource: memory-cleanup\nverified: 2026-10-30\n---\n\nsince 10-29\n"), + ) + .unwrap(); + }; + fact( + &root, + "unreadable-chat", + "chat server: auth_expired since 2026-10-29", + ); + fact(&root, "prefers-short-replies", "George wants short replies"); + fact( + &root.join("projects/calc"), + "unreadable-mail", + "mail server: timeout", + ); + fact( + &root.join("projects/web"), + "unreadable-tracker", + "tracker: 401", + ); + let found = servers(&root, "calc"); + let names: Vec<&str> = found.iter().map(|i| i.name.as_str()).collect(); + assert_eq!(names, ["chat", "mail"]); + assert_eq!( + found[0].failed.as_deref(), + Some("chat server: auth_expired since 2026-10-29") + ); + assert_eq!(found[0].facts[0], "memory fact unreadable-chat"); + assert_eq!(found[1].facts[0], "memory fact unreadable-mail in calc"); + assert!( + found + .iter() + .all(|i| i.kind == Kind::Server && i.bots.is_empty()) + ); + assert_eq!(count(&found, "calc.lead"), 0); + assert_eq!(failed(&found), 2); + let _ = std::fs::remove_dir_all(&root); + } + + #[test] + fn the_card_lists_what_failed_first_marked_with_its_error() { + let dir = std::env::temp_dir().join(format!("background-failed-{}", std::process::id())); + let mut broken = item(Kind::Trigger, "memory", &["memory-cleanup"], false); + broken.failed = Some("provider_failed: login expired".into()); + let mut server = item(Kind::Server, "chat", &["calc.lead"], false); + (server.bot, server.bots, server.what) = (String::new(), Vec::new(), String::new()); + server.facts = vec!["memory fact unreadable-chat".into()]; + server.failed = Some("chat server: auth_expired".into()); + let mut view = View { + scope: Scope { + project: "calc".into(), + lead: "calc.lead".into(), + glob: "calc.*".into(), + }, + store: dir.join("state.sqlite"), + store_id: None, + bot: "calc.lead".into(), + items: vec![ + broken, + server, + item(Kind::Trigger, "nightly", &["calc.lead"], false), + ], + turns: HashMap::new(), + lookout: String::new(), + selected: 1, + confirm: None, + notice: None, + }; + let (card, at) = view.card(100); + let lines: Vec = card.iter().map(|l| l.to_string()).collect(); + assert_eq!(lines[2], "Failed Β· 2"); + assert_eq!(lines[3], " βœ— memory every 1d β†’ memory-cleanup"); + let mark = card[3] + .spans + .iter() + .find(|s| s.content.starts_with('βœ—')) + .unwrap(); + assert_eq!(mark.style.fg, Some(Color::Red)); + assert_eq!(lines[4], " provider_failed: login expired"); + assert_eq!( + card[4].style.add_modifier, + ratatui::style::Modifier::empty() + ); + assert_eq!( + lines[7], + "β€Ί βœ— chat could not be read by the nightly memory cleanup" + ); + assert_eq!(at, 7); + assert_eq!(lines[8], " chat server: auth_expired"); + assert_eq!(lines[9], " memory fact unreadable-chat"); + assert_eq!(lines[11], "Triggers Β· 1"); + assert_eq!(view.footer().to_string(), " ↑↓ pick Β· esc to close"); + let press = |view: &mut View, code| view.key(KeyEvent::new(code, KeyModifiers::NONE)); + press(&mut view, KeyCode::Char('d')); + assert_eq!(view.confirm, None); + assert!(view.notice.as_deref().unwrap().contains("reads again")); } #[test] @@ -553,6 +939,7 @@ mod tests { item(Kind::Trigger, "nightly", &["calc.lead"], false), item(Kind::Subscription, "tasks", &["calc.lead"], true), ], + turns: HashMap::new(), lookout: "Lookout Β· not running".into(), selected: 0, confirm: None, @@ -568,11 +955,11 @@ mod tests { let lines = text(&view); assert_eq!(lines[0], "calc Β· in the background"); assert_eq!(lines[3], "Triggers Β· 1"); - assert_eq!(lines[4], "›↻ nightly every 1d β†’ lead"); - assert_eq!(lines[5], " sent 2"); - assert_eq!(lines[6], " β”” Read preferences.md"); + assert_eq!(lines[4], "β€Ί ↻ nightly every 1d β†’ lead"); + assert_eq!(lines[5], " sent 2"); + assert_eq!(lines[6], " β”” Read preferences.md"); assert_eq!(lines[8], "Lookout subscriptions Β· 1"); - assert_eq!(lines[9], " β—Ž tasks for lead"); + assert_eq!(lines[9], " β—Ž tasks for lead"); let press = |view: &mut View, code| view.key(KeyEvent::new(code, KeyModifiers::NONE)); // Ctrl+D is not d. view.key(KeyEvent::new(KeyCode::Char('d'), KeyModifiers::CONTROL)); diff --git a/sonny/src/main.rs b/sonny/src/main.rs index 7b6bd368d..e4b7b9c41 100644 --- a/sonny/src/main.rs +++ b/sonny/src/main.rs @@ -1961,6 +1961,12 @@ struct App { /// How many triggers and subscriptions run for this agent, as the /// footer's `↻` says, and when they were last counted. behind: (usize, Option), + /// How many of what `/background` lists failed, as the footer's `βœ—` + /// says. + failed: usize, + /// How the turns triggers' last fires sent ended, by handle, as the + /// daemon said: asked once a turn is seen, and again while it runs. + turns: HashMap, /// The turn the person's words last went to from this window: only /// its end is followed by a guess at their next message. yours: Option, @@ -2588,6 +2594,8 @@ impl App { pick: 0, watch: None, behind: (0, None), + failed: 0, + turns: HashMap::new(), yours: None, suggesting: None, suggestion: None, @@ -2854,7 +2862,7 @@ impl App { self.shell_ran().await; self.listed(); self.follow_place().await; - self.count_behind(); + self.count_behind().await; self.read_tasks().await; self.read_running_tasks().await; self.read_references().await; @@ -5930,6 +5938,7 @@ impl App { self.options.store.clone(), self.client.store().map(str::to_owned), self.bot.clone(), + self.turns.clone(), ); self.watch = Some(Watch { project, @@ -6034,8 +6043,9 @@ impl App { } /// Count what runs for this agent in the background again, every ten - /// seconds: files read, no daemon call. - fn count_behind(&mut self) { + /// seconds, and what failed of what `/background` lists: files read, + /// and the daemon asked at once how the turns fires sent ended. + async fn count_behind(&mut self) { if self .behind .1 @@ -6044,10 +6054,28 @@ impl App { { return; } - let items = background::read(&self.options.store, self.client.store(), |bot| { - bot == self.bot - }); + let scope = self.scope(); + let read = |turns: &HashMap| { + background::read_around(&self.options.store, self.client.store(), turns, &scope) + }; + let mut items = read(&self.turns); + let unended = background::unended(&items, &self.turns); + if !unended.is_empty() { + let ended = trigger::ended(&self.client, unended).await; + if !ended.is_empty() { + self.turns.extend(ended); + items = read(&self.turns); + } + } + let fired: std::collections::HashSet<&String> = + items.iter().filter_map(|i| i.handle.as_ref()).collect(); + self.turns.retain(|handle, _| fired.contains(handle)); + items.retain(|item| background::shown(&scope, item)); self.behind = (background::count(&items, &self.bot), Some(Instant::now())); + self.failed = background::failed(&items); + if let Some(view) = (self.watch.as_mut()).and_then(|w| w.background.as_mut()) { + view.set_turns(&self.turns); + } } /// What the lookout does for this window's lead. @@ -6087,9 +6115,12 @@ impl App { lead, glob: pattern, } = self.scope(); - let behind = background::read(&self.options.store, self.client.store(), |bot| { - lookout::subs::matches(&pattern, bot) - }); + let behind = background::read( + &self.options.store, + self.client.store(), + &self.turns, + |bot| lookout::subs::matches(&pattern, bot), + ); let Ok(page) = self .client .request("bots", json!({"name": pattern, "limit": 256})) @@ -7037,6 +7068,7 @@ impl App { margin::Glance { name: self.bot.clone(), behind: self.behind.0, + failed: self.failed, context: self.context, } } @@ -9293,6 +9325,13 @@ impl App { _ if glancing => right, n => format!("↻ {n} in background Β· {right}"), }; + // What failed of it, in red: /background lists it first. + let failed = format!("βœ— {} failed", self.failed); + let right = match self.failed { + 0 => right, + _ if glancing => right, + _ => format!("{failed} Β· {right}"), + }; let right = match self.margin.notes().len() { n if n > 0 && !glancing => format!("✎ {n} Β· {right}"), _ => right, @@ -9317,9 +9356,12 @@ impl App { folder = &folder[folder.chars().next().map_or(1, char::len_utf8)..]; } let place = format!(" Β· {}{folder}", if cut { "…" } else { "" }); - let fill = width.saturating_sub( - left.width() + place.width() + branch.width() + right.width() + counted_width, - ); + // A space at least, so a branch never runs into what follows it. + let fill = width + .saturating_sub( + left.width() + place.width() + branch.width() + right.width() + counted_width, + ) + .max(1); // The branch in green, as shells show it; the agent's name in the // agents' color. let at = left.width() + place.width() + branch.width() + fill; @@ -9336,7 +9378,16 @@ impl App { } match right.rfind(&self.bot) { Some(at) => { - spans.push(Span::raw(right[..at].to_owned())); + let before = &right[..at]; + match before.find(&failed).filter(|_| self.failed > 0) { + Some(from) => { + let to = from + failed.len(); + spans.push(Span::raw(before[..from].to_owned())); + spans.push(Span::raw(failed.clone()).fg(Color::Red).not_dim()); + spans.push(Span::raw(before[to..].to_owned())); + } + None => spans.push(Span::raw(before.to_owned())), + } spans.push(Span::raw(self.bot.clone()).fg(Color::Blue)); let rest = &right[at + self.bot.len()..]; // How full the context is: green, then yellow, then red. @@ -11046,6 +11097,31 @@ mod tests { assert!(!app.footer_line(120).to_string().contains('πŸ“Ž')); } + #[tokio::test] + async fn what_failed_in_the_background_is_red_left_of_what_runs() { + let client = daemon_with(json!([])).await; + let mut app = app(client); + app.behind = (2, Some(Instant::now())); + app.failed = 1; + let line = app.footer_line(140); + let text = line.to_string(); + assert!(text.contains("βœ— 1 failed Β· ↻ 2 in background Β· "), "{text}"); + let mark = line + .spans + .iter() + .find(|s| s.content == "βœ— 1 failed") + .unwrap(); + assert_eq!(mark.style.fg, Some(Color::Red)); + assert!( + mark.style.sub_modifier.contains(Modifier::DIM), + "red, not dimmed" + ); + app.failed = 2; + assert!(app.footer_line(140).to_string().contains("βœ— 2 failed Β· ")); + app.failed = 0; + assert!(!app.footer_line(140).to_string().contains('βœ—')); + } + #[tokio::test] async fn an_emoji_reacts_to_a_reply_without_a_turn_and_again_takes_it_off() { let sent = Arc::new(std::sync::Mutex::new(Vec::new())); diff --git a/sonny/src/margin.rs b/sonny/src/margin.rs index b342932a4..3bc772dd2 100644 --- a/sonny/src/margin.rs +++ b/sonny/src/margin.rs @@ -1164,11 +1164,13 @@ fn is_id(id: &str) -> bool { } /// What runs for the agent, as the margin's first row says it. -#[derive(Default)] +#[derive(Clone, Default)] pub struct Glance { pub name: String, /// Triggers and subscriptions. pub behind: usize, + /// What of `/background` failed. + pub failed: usize, pub context: Option, } @@ -1191,6 +1193,15 @@ impl Glance { spans.push(span); at += width; }; + if self.failed > 0 { + let text = format!("βœ— {}", self.failed); + fact( + &mut spans, + text.clone(), + Span::raw(text).fg(Color::Red), + Some("background"), + ); + } if self.behind > 0 { let text = format!("↻ {}", self.behind); fact( @@ -1777,6 +1788,7 @@ mod tests { let busy = Glance { name: "calc".into(), behind: 3, + failed: 0, context: Some(41_000), }; let (line, buttons) = busy.line(); @@ -1784,6 +1796,18 @@ mod tests { assert!(text.starts_with(" calc Β· ↻ 3 Β· "), "{text}"); let urls: Vec<&str> = buttons.iter().map(|b| b.2.as_str()).collect(); assert_eq!(urls, ["sonny:background"]); + let (failing, failed_buttons) = Glance { + failed: 2, + ..busy.clone() + } + .line(); + assert!( + failing.to_string().starts_with(" calc Β· βœ— 2 Β· ↻ 3 Β· "), + "{failing}" + ); + assert_eq!(failing.spans[3].style.fg, Some(Color::Red)); + assert_eq!(failed_buttons.len(), 2); + assert!(failed_buttons.iter().all(|b| b.2 == "sonny:background")); let slice = |(from, to): (u16, u16)| -> String { text.chars() .skip(from as usize) diff --git a/sonny/src/served.rs b/sonny/src/served.rs index 9ad31ff99..aaea683e7 100644 --- a/sonny/src/served.rs +++ b/sonny/src/served.rs @@ -598,11 +598,43 @@ async fn called(name: &str, arguments: &Value) -> Answer { TRIGGERS => { // A trigger's add runs on a runtime of its own. let arguments = arguments.clone(); - tokio::task::spawn_blocking(move || crate::trigger::tool(&arguments)) + let mut said = tokio::task::spawn_blocking(move || crate::trigger::tool(&arguments)) .await .map_err(|e| ("tool_failed", e.to_string()))? - .map(|said| said.to_string()) - .map_err(|e| ("trigger_refused", e)) + .map_err(|e| ("trigger_refused", e))?; + // How the turns their last fires sent ended, as the person's + // `/background` shows them; without the daemon, as the files say. + // Only a trigger made on this daemon's store: a turn handle names + // a bot and a turn, which another store's daemon has its own of. + if let Ok((_, socket)) = crate::daemon_paths(None, None) + && let Ok((client, _events)) = Client::connect(&socket).await + { + let ours: Vec = match (client.store(), crate::trigger::Places::home()) { + (Some(store), Ok(places)) => crate::trigger::on_store(&places, store) + .into_iter() + .map(|(row, _)| row["name"].clone()) + .collect(), + _ => Vec::new(), + }; + let mut rows: Vec<&mut Value> = if said["triggers"].is_array() { + (said["triggers"].as_array_mut().into_iter().flatten()).collect() + } else if said["last"].is_object() { + vec![&mut said] + } else { + Vec::new() + }; + rows.retain(|row| ours.contains(&row["name"])); + let handles = rows + .iter() + .filter_map(|row| crate::trigger::sent(row)) + .collect(); + let ended = crate::trigger::ended(&client, handles).await; + for row in rows { + crate::trigger::with_turn(row, &ended); + } + client.close().await; + } + Ok(said.to_string()) } _ => Err(("unknown_tool", format!("Sonny answers no tool {name}"))), } diff --git a/sonny/src/trigger.rs b/sonny/src/trigger.rs index f0a1edd40..4ee1bb150 100644 --- a/sonny/src/trigger.rs +++ b/sonny/src/trigger.rs @@ -25,6 +25,7 @@ //! add, list, fire and remove them. use agent_client::Client; use serde_json::{Value, json}; +use std::collections::HashMap; use std::{ path::{Path, PathBuf}, time::Duration, @@ -1599,6 +1600,35 @@ pub fn on_store(places: &Places, store_id: &str) -> Vec<(Value, Vec)> { (row, bots) }) .collect(); + // One that ended keeps its row when it has something to say, as a + // one-shot that did not deliver or one stopped by an error. + for entry in std::fs::read_dir(&places.state) + .into_iter() + .flatten() + .flatten() + { + let file = entry.file_name(); + let Some(name) = file.to_str().and_then(|f| f.strip_suffix(".json")) else { + continue; + }; + if valid_name(name).is_err() + || places.job(name).exists() + || found.iter().any(|(row, _)| row["name"] == name) + { + continue; + } + let Some(mut row) = read_json(&places.last(name)) + .filter(|row| row["name"] == name && row["store_id"] == store_id) + else { + continue; + }; + row["ended"] = json!(true); + let bots: Vec = ["bot", "by", "reply_to"] + .iter() + .filter_map(|key| row[*key].as_str().map(str::to_owned)) + .collect(); + found.push((row, bots)); + } found.sort_by(|a, b| a.0["name"].as_str().cmp(&b.0["name"].as_str())); found } @@ -2988,6 +3018,16 @@ fn write_row(places: &Places, trigger: &Trigger, last: &Value, kept: &Kept) -> R let mut row = trigger.json(&json!({"last": last, "sent": kept.sent, "started_id": kept.started_id})); row["started_id"] = json!(kept.started_id); + // Whose it is once its job is gone: a failure it retired with is + // shown on that daemon's windows only, and on the windows of the + // agent that added it to start its own. + row["store_id"] = json!(trigger.store_id); + if let Target::Start { + by: Some((by, _)), .. + } = &trigger.target + { + row["by"] = json!(by); + } row["head"] = json!(kept.seen.head); row["seen_at"] = json!(kept.seen.at); row["checked"] = kept.checked.clone(); @@ -3426,6 +3466,56 @@ async fn reply(client: &Client, trigger: &Trigger, bot: &str, mut outcome: Value /// What a turn's `wait` result says: its text, else its error. A turn that /// ended saying nothing passes on nothing, not "null". +/// The turn the last fire of `row` sent, by handle, when it sent one. +pub fn sent(row: &Value) -> Option { + let last = &row["last"]; + (last["outcome"] == "sent") + .then(|| last["handle"].as_str().map(str::to_owned)) + .flatten() +} + +/// How the turns of `handles` ended, read from the daemon at once: each +/// one that ended, by handle, as `wait` gives it. A turn still running, +/// or a daemon that does not answer, is left out. +pub async fn ended(client: &Client, handles: Vec) -> HashMap { + let mut found = HashMap::new(); + // As many as one `wait` takes at a time. + for some in handles.chunks(64) { + let Ok(waited) = client + .request("wait", json!({"handles": some, "timeout_ms": 0})) + .await + else { + continue; + }; + found.extend( + (waited["results"].as_object().into_iter().flatten()) + .filter(|(_, result)| result["pending"] != true && result["status"].is_string()) + .map(|(handle, result)| (handle.clone(), result.clone())), + ); + } + found +} + +/// `row`, its `last` taking how the turn its fire sent ended, from +/// `ended`: its `turn_status`, and `turn_error` when it did not complete, +/// so a fire that went out and whose turn then failed, as when the machine +/// was offline or a login expired, reads so wherever its row is shown. +pub fn with_turn(row: &mut Value, ended: &HashMap) { + let Some(result) = sent(row).and_then(|handle| ended.get(&handle)) else { + return; + }; + let status = result["status"].clone(); + if status != "completed" { + let error = answer(&json!({"error": result["error"]})); + row["last"]["turn_error"] = json!(match result["detail"].as_str() { + Some(detail) if !error.is_empty() => format!("{error}: {detail}"), + Some(detail) => detail.to_owned(), + None => error, + }); + } + row["last"]["turn_status"] = status; +} + fn answer(result: &Value) -> String { match (result["text"].as_str(), &result["error"]) { (Some(text), _) if !text.is_empty() => text.to_owned(), @@ -5146,6 +5236,35 @@ mod tests { .unwrap() .starts_with("asks_stuck:") ); + // Its job gone, its failure is still its store's to show, and no + // other's. + let kept = on_store(&w.places, &s.store_id); + assert_eq!(kept.len(), 1); + assert_eq!(kept[0].0["ended"], true); + assert_eq!(kept[0].0["last"]["outcome"], "failed"); + assert_eq!(kept[0].1, [s.bot().to_owned()]); + assert!(on_store(&w.places, "another-store").is_empty()); + // One that starts its agent stays the adder's too. + let started = Trigger { + name: "digest".into(), + target: Target::Start { + name: "digest".into(), + model: "openai/gpt".into(), + effort: None, + by: Some(("p.lead".into(), 7)), + }, + ..trigger() + }; + w.install(&started).unwrap(); + stops(&w.places, &started, "start_failed: no".into(), &|x| { + w.fake.call(x) + }); + let kept = on_store(&w.places, &s.store_id); + let digest = kept + .iter() + .find(|(row, _)| row["name"] == "digest") + .unwrap(); + assert_eq!(digest.1, ["digest".to_owned(), "p.lead".to_owned()]); } #[test]