diff --git a/docs/proactive-input-suggestion-design.md b/docs/proactive-input-suggestion-design.md new file mode 100644 index 0000000..94bd979 --- /dev/null +++ b/docs/proactive-input-suggestion-design.md @@ -0,0 +1,112 @@ +# Architecture Design and Development Plan: Proactive Input Suggestion & Auto-typing + +## 1. Executive Summary & Requirements Overview + +The user requested strengthening Squid's proactive behavior in a specific desktop interaction scenario: +1. **Trigger Mechanism (触发机制):** When the user focuses (e.g. mouse click/focus) on a text input field, Squid pops up a prompt, triggers programmatic reasoning to predict what the user intends to input, and offers this suggestion with an option for the user to trigger/accept it. +2. **Interaction Detail (交互细节):** If the user chooses to accept the suggested input text, Squid must directly type/input the text into the focused field on behalf of the user, rather than merely displaying the suggestion. +3. **GUI Control Toggle (控制开关):** Because this proactive behavior can be intrusive if unwanted, provide a toggle switch in the GUI settings panel, enabled by default (`default: on`). +4. **Development & Review Process (开发与测试流程):** + - Formulate an elegant and clean design and development plan, reviewed by a subagent. + - Step-by-step implementation, with each step reviewed by a subagent. + - After completion, run virtual desktop smoke tests (including visual smoke test on X11/Xvfb), and open a PR linked via `link_pull_request`. + +--- + +## 2. Architecture & Design + +### 2.1 Focus Detection & Ingest Pipeline +- **Observation Source:** Screen text sources (`screen-text-source.ts`, `screen-text-reader-linux.ts`, `screen-text-reader-windows.ts`, `screen-text-reader-synthetic.ts`) produce `focus_changed` activity events with `payload: { applicationId, windowTitle, role, label }`. +- **Trigger Condition:** + 1. `event.kind === "focus_changed"` + 2. `event.payload.role === "text_input" || event.payload.role === "editor"` + 3. Feature switch `isInputSuggestionsEnabled()` is true. + 4. De-duplication / Cooldown: Prevent repeated prompts for the same input control unless focus moves away and returns (keyed by `${applicationId}:${windowTitle}:${role}:${label}`). + +### 2.2 Suggestion Generation ("Thinking") +Squid creates an `InputSuggestionService` (`src/input-suggestion.ts`): +- Context inputs: + - Focused element details: `applicationId`, `windowTitle`, `role`, `label` + - Text context if available: `prefix`, `suffix`, `selection` + - Episodic world state: `recentActions`, `lastEvent` + - Recent timeline events from `EventStore` +- Generation strategy: + - **With Live LLM / Resident Agent:** Ask model for focused text completion prediction in the context of the focused field, returning `{ text: string, why: string }`. + - **Fallback / Heuristics (Offline / Tests / CI):** Contextual prediction based on active window (e.g., terminal command suggestion like `bun test` or `git status`, editor completion, or search phrase). + +### 2.3 Proactive Prompt & Confirmation Card ("我们的方式提供建议") +- Integrates with Squid's existing confirmation surface (`ActionConfirmationPolicy` and `PanelConfirmationSurface`). +- A prompt message is dispatched to the panel: + - `type: "prompt"` + - `kind: "type_input"` + - `summary: suggestion.text` + - `why: suggestion.why` + - `denyConsequence: "No text will be typed into the active input."` + - `expiresAt: timestamp (30s timeout)` +- The Slint panel renders `ConfirmationCard`: + - Clearly displays the suggested text in the monospaced card summary. + - Explains why the suggestion was made. + - Offers three distinct actions: **Allow** (Accept), **Decline**, **Later** (Defer). + +### 2.4 Auto-typing Injection Engine ("直接帮用户输入进去") +When the user clicks "Allow", `SerialConfirmationPolicy` resolves to `accepted` (`user_accepted`). +Squid executes `DesktopInputInjector` / `TextTypingService` (`src/text-typing-service.ts`): +- **Linux (X11):** + Uses `xdotool`: + `xdotool type --delay 12 --clearmodifiers -- ""` +- **Windows:** + Uses PowerShell script with `[System.Windows.Forms.SendKeys]::SendWait(...)` / UI Automation. +- **macOS:** + Uses `osascript` with `keystroke`. +- **Testability:** + All platform command executions go through an injectable subprocess runner, allowing comprehensive offline unit tests without actual GUI or keyboard interference. + +### 2.5 GUI Configuration Switch (默认开启) +1. **Config Schema (`src/config-schema.ts`):** + - Key: `inputSuggestions` + - Env: `SQUID_INPUT_SUGGESTIONS` + - Label: `"Input suggestions"` + - Group: `"Screen"` + - Kind: `"enum"`, values: `["on", "off"]` + - Description: `"Whether Squid proactively proposes text suggestions when focusing on an input field. Defaults to on."` + - Default: unset means `"on"` (enabled). +2. **GUI Slint Widget (`ui/settings.slint`):** + - For enum settings with values `on;off`, render an interactive Slint `Switch` toggle widget showing `Enabled`/`Disabled` state. + - Defaults to checked/Enabled when value is not `"off"`. + - Clicking toggles and dispatches `config_set` immediately. +3. **Protocol & Wiring:** + - Seamlessly works over existing `config_get` / `config_set` protocol without breaking schema invariants. + +--- + +## 3. Development Plan (Step-by-Step) + +### Step 1: Configuration Schema & GUI Switch +- Add `inputSuggestions` to `src/config-schema.ts` with `env: "SQUID_INPUT_SUGGESTIONS"`, `group: "Screen"`, `kind: "enum"`, `values: ["on", "off"]`. +- Enhance `ui/settings.slint` with an explicit `Switch` for on/off enum settings. +- Write unit tests in `tests/config.test.ts`. +- **Review Checkpoint 1:** Invoke subagent to review Step 1 changes. + +### Step 2: Cross-Platform Text Typing Service (`TextTypingService`) +- Create `src/text-typing-service.ts` with Linux (`xdotool`), Windows (PowerShell), macOS (`osascript`) implementations. +- Include proper escaping, `--` termination, error handling, and timeout protection. +- Write unit tests in `tests/text-typing-service.test.ts` covering argument building and mock execution. +- **Review Checkpoint 2:** Invoke subagent to review Step 2 changes. + +### Step 3: Suggestion Engine & Ingest Trigger in Watch Session +- Create `src/input-suggestion.ts` to manage reasoning and heuristics from context. +- Integrate into `src/watch-session.ts`: + - Hook on `focus_changed` when `role` is `text_input` or `editor`. + - Check `isInputSuggestionsEnabled()`. + - Generate suggestion and invoke `actionConfirmation.confirm(...)`. + - On `accepted`, trigger `textTypingService.typeText(...)`. +- Write unit tests in `tests/input-suggestion.test.ts` and `tests/watch-session.test.ts`. +- **Review Checkpoint 3:** Invoke subagent to review Step 3 changes. + +### Step 4: Verification, Virtual Desktop Smoke Testing & PR +- Run full test suite: `bun test` and `cargo test`. +- Build release UI binary: `cargo build --release --manifest-path ui/Cargo.toml`. +- Run virtual desktop smoke test: `bun run ui:smoke`. +- Verify visual output screenshots in `/tmp/squid-ui-smoke-*/`. +- **Review Checkpoint 4:** Final subagent review before PR. +- Commit changes, push branch, open PR via `gh pr create`, and link with `link_pull_request`. diff --git a/src/agent-tools.ts b/src/agent-tools.ts index 58786ba..49d196c 100644 --- a/src/agent-tools.ts +++ b/src/agent-tools.ts @@ -7,6 +7,10 @@ import { type ToolDefinition, } from "@earendil-works/pi-coding-agent"; import type { EventStore } from "./event-store.js"; +import { createTypeTextTool } from "./type-text-tool.js"; +import type { TextTypingService } from "./text-typing-service.js"; + +export { createTypeTextTool, typeTextParams, type TypeTextArgs, type TypeTextToolDetails } from "./type-text-tool.js"; /** * The agent's window onto its own sensory record. @@ -68,7 +72,7 @@ export function queryRecentEventsTool( ): ToolDefinition { // `defineTool` is Pi's own shim for exactly this: a concrete tool kept intact // when it joins a `ToolDefinition[]`, where the array's default type parameters - // would otherwise widen the schema and the details to `unknown`. + // would otherwise widen the schema and the details to `unknown``. return defineTool({ name: "query_recent_events", label: "Query recent events", @@ -98,7 +102,14 @@ export function queryRecentEventsTool( }); } -/** The custom tools a resident agent gets for its own store. */ -export function agentCustomTools(store: EventStore): readonly ToolDefinition[] { - return [queryRecentEventsTool(store)]; +/** The custom tools a resident agent gets for its own store and capabilities. */ +export function agentCustomTools( + store: EventStore, + typingService?: TextTypingService, +): readonly ToolDefinition[] { + const tools: ToolDefinition[] = [queryRecentEventsTool(store)]; + if (typingService !== undefined) { + tools.push(createTypeTextTool(typingService)); + } + return tools; } diff --git a/src/config-schema.ts b/src/config-schema.ts index 3c1e403..79918ac 100644 --- a/src/config-schema.ts +++ b/src/config-schema.ts @@ -390,6 +390,16 @@ const CONFIG_KEY_TABLE: readonly Omit[] = [ description: "How often Squid looks at the screen. The default is usually right.", kind: "int", }, + { + id: "inputSuggestions", + env: "SQUID_INPUT_SUGGESTIONS", + label: "Input suggestions", + group: "Screen", + description: + "Whether Squid proactively proposes text suggestions when focusing on an input field. Defaults to on.", + kind: "enum", + values: ["on", "off"], + }, { id: "browser", env: "SQUID_BROWSER", diff --git a/src/config-service.ts b/src/config-service.ts index 7e70384..1dae36d 100644 --- a/src/config-service.ts +++ b/src/config-service.ts @@ -163,7 +163,7 @@ export function configDumpEntries(options: ConfigServiceOptions): ConfigDump { description: key.description, kind: key.kind, value: values[key.id] ?? "", - applies: key.id === "modelProvider" ? "now" : "restart", + applies: key.id === "modelProvider" || key.id === "inputSuggestions" ? "now" : "restart", overridden: overriddenEnv(key.env), ...(key.values === undefined ? {} : { values: key.values }), }); @@ -223,8 +223,8 @@ function applyFileSet( if ("configError" in written) { return { ok: false, error: written.configError }; } - // Model provider hot reloads; other file-backed settings apply on restart. - return { ok: true, applies: id === "modelProvider" ? "now" : "restart" }; + // Model provider and input suggestions hot reload; other file-backed settings apply on restart. + return { ok: true, applies: id === "modelProvider" || id === "inputSuggestions" ? "now" : "restart" }; } function applyModelSet( diff --git a/src/input-suggestion-extension.ts b/src/input-suggestion-extension.ts new file mode 100644 index 0000000..0e375d7 --- /dev/null +++ b/src/input-suggestion-extension.ts @@ -0,0 +1,33 @@ +import type { ExtensionAPI, InlineExtension, ToolDefinition } from "@earendil-works/pi-coding-agent"; +import type { InputSuggestionService } from "./input-suggestion.js"; + +export interface InputSuggestionExtensionOptions { + readonly service: InputSuggestionService; + readonly typeTextTool?: ToolDefinition | undefined; +} + +/** + * Pi Agent InlineExtension for proactive input suggestions. + * + * Adheres strictly to the `@earendil-works/pi-coding-agent` ExtensionAPI specification. + * Integrates the suggestion service lifecycle with the Pi Agent session and registers + * the official `type_text` tool dynamically. + */ +export function inputSuggestionExtension( + options: InputSuggestionExtensionOptions, +): InlineExtension { + return { + name: "squid-input-suggestion", + hidden: true, + factory: (pi: ExtensionAPI) => { + const toolToRegister = options.typeTextTool ?? options.service.getTypeTextTool?.(); + if (toolToRegister !== undefined) { + pi.registerTool(toolToRegister); + } + + pi.on("session_shutdown", () => { + options.service.reset?.(); + }); + }, + }; +} diff --git a/src/input-suggestion.ts b/src/input-suggestion.ts new file mode 100644 index 0000000..b0bdfb0 --- /dev/null +++ b/src/input-suggestion.ts @@ -0,0 +1,313 @@ +/** + * Proactive Input Suggestion Service. + * + * Proactively triggers when the user's focus moves to an input field (text_input, + * editor, or terminal). Infers/predicts what the user likely intends to enter, + * presents the proposed text to the user via Squid's native ConfirmationCard, + * and upon user acceptance ("Allow"), automatically types the suggested text + * into the target field using the standard Pi Agent `type_text` tool. + * + * Honors the GUI switch / environment variable (SQUID_INPUT_SUGGESTIONS !== "off"). + */ + +import type { ToolDefinition } from "@earendil-works/pi-coding-agent"; +import type { + ActionConfirmationPolicy, + ActionConfirmationRequest, +} from "./action-confirmation.js"; +import type { FocusChangedPayload, WorldState } from "./protocol.js"; +import { TextTypingService } from "./text-typing-service.js"; +import { + createTypeTextTool, + type typeTextParams, + type TypeTextToolDetails, +} from "./type-text-tool.js"; + +export interface InputSuggestion { + readonly text: string; + readonly why: string; +} + +export interface SuggestionContext { + readonly payload: FocusChangedPayload; + readonly worldState: WorldState | null; +} + +export interface InputSuggestionServiceOptions { + readonly isEnabled: () => boolean; + readonly confirmationPolicy: ActionConfirmationPolicy; + readonly textTypingService?: TextTypingService | undefined; + readonly typeTextTool?: ToolDefinition | undefined; + readonly predictSuggestion?: ((context: SuggestionContext) => Promise | InputSuggestion | null) | undefined; + readonly cooldownMs?: number | undefined; + readonly promptTimeoutMs?: number | undefined; + readonly log?: ((line: string) => void) | undefined; + readonly logError?: ((line: string) => void) | undefined; + readonly now?: (() => number) | undefined; +} + +/** + * Checks if the focused element is a text input field or interactive editor/terminal. + */ +export function isTextInputTarget(payload: FocusChangedPayload): boolean { + if (payload.role === "button" || payload.role === "link") { + return false; + } + if (payload.role === "text_input" || payload.role === "editor") { + return true; + } + const app = (payload.applicationId ?? "").toLowerCase(); + const title = (payload.windowTitle ?? "").toLowerCase(); + // Terminals often report role "unknown" in accessibility trees + if ( + app.includes("terminal") || + app.includes("konsole") || + app.includes("alacritty") || + app.includes("kitty") || + app.includes("wezterm") || + title.includes("bash") || + title.includes("zsh") || + title.includes("terminal") + ) { + return true; + } + return false; +} + +/** + * Default heuristic suggestion engine when no external LLM predictor is supplied + * or when the LLM call times out. + */ +export function heuristicSuggestInput(context: SuggestionContext): InputSuggestion | null { + const { payload, worldState } = context; + const app = (payload.applicationId ?? "").toLowerCase(); + const title = (payload.windowTitle ?? "").toLowerCase(); + const label = (payload.label ?? "").toLowerCase(); + + // 1. Search box in browser or app + if (label.includes("search") || label.includes("find") || label.includes("query")) { + const query = worldState?.recentFiles?.[0] + ? `find ${worldState.recentFiles[0]}` + : "search documentation"; + return { + text: query, + why: `Focused on search field in ${payload.applicationId || "application"}.`, + }; + } + + // 2. Terminal prompt + if ( + app.includes("terminal") || + app.includes("konsole") || + app.includes("alacritty") || + app.includes("kitty") || + app.includes("wezterm") || + title.includes("bash") || + title.includes("zsh") || + title.includes("terminal") + ) { + // If recent file saved was a test or ts/js file, suggest running tests + const recentFile = worldState?.recentFiles?.[0] ?? ""; + if (recentFile.endsWith(".test.ts") || recentFile.endsWith(".spec.ts")) { + return { + text: "bun test", + why: `Focused on terminal after modifying ${recentFile}.`, + }; + } + if (recentFile.endsWith(".rs")) { + return { + text: "cargo test", + why: `Focused on terminal after modifying Rust source ${recentFile}.`, + }; + } + return { + text: "git status", + why: `Focused on terminal in ${payload.windowTitle || "workspace"}.`, + }; + } + + // 3. Text editor input + if (payload.role === "editor") { + return { + text: "// TODO: implement next step", + why: `Focused on code editor in ${payload.windowTitle}.`, + }; + } + + // 4. General text input field + const fieldName = payload.label ? `"${payload.label}"` : "text input"; + return { + text: "Squid auto-suggestion", + why: `Focused on ${fieldName} in ${payload.applicationId || payload.windowTitle}.`, + }; +} + +export class InputSuggestionService { + private readonly isEnabled: () => boolean; + private readonly confirmationPolicy: ActionConfirmationPolicy; + private readonly typeTextTool: ToolDefinition; + private readonly predictSuggestion: (context: SuggestionContext) => Promise; + private readonly cooldownMs: number; + private readonly promptTimeoutMs: number; + private readonly log?: ((line: string) => void) | undefined; + private readonly logError?: ((line: string) => void) | undefined; + private readonly now: () => number; + + private lastTargetKey = ""; + private lastTriggerTime = 0; + private inFlight = false; + + constructor(options: InputSuggestionServiceOptions) { + this.isEnabled = options.isEnabled; + this.confirmationPolicy = options.confirmationPolicy; + if (options.typeTextTool !== undefined) { + this.typeTextTool = options.typeTextTool; + } else if (options.textTypingService !== undefined) { + this.typeTextTool = createTypeTextTool(options.textTypingService); + } else { + this.typeTextTool = createTypeTextTool(new TextTypingService()); + } + const predictor = options.predictSuggestion ?? heuristicSuggestInput; + this.predictSuggestion = async (ctx) => predictor(ctx); + this.cooldownMs = options.cooldownMs ?? 4_000; + this.promptTimeoutMs = options.promptTimeoutMs ?? 25_000; + this.log = options.log; + this.logError = options.logError; + this.now = options.now ?? Date.now; + } + + /** + * Returns the registered Pi Agent ToolDefinition. + */ + getTypeTextTool(): ToolDefinition { + return this.typeTextTool; + } + + /** + * Called when a focus_changed activity event is ingested. + */ + async onFocusChanged( + payload: FocusChangedPayload, + worldState: WorldState | null, + ): Promise { + // 1. Verify feature switch + if (!this.isEnabled()) { + return; + } + + // 2. Validate target is actually an input field (not a button, link, etc.) + if (!isTextInputTarget(payload)) { + return; + } + + // 3. Target identity key & debounce check + const targetKey = `${payload.applicationId}:${payload.windowTitle}:${payload.role}:${payload.label ?? ""}`; + const nowMs = this.now(); + if (this.inFlight) { + return; + } + if (targetKey === this.lastTargetKey && nowMs - this.lastTriggerTime < this.cooldownMs) { + return; + } + + this.lastTargetKey = targetKey; + this.lastTriggerTime = nowMs; + this.inFlight = true; + + try { + // 4. Generate suggestion + const suggestion = await this.predictSuggestion({ payload, worldState }); + if (!suggestion || !suggestion.text.trim()) { + return; + } + + this.log?.( + JSON.stringify({ + inputSuggestionTriggered: { + target: targetKey, + suggestedText: suggestion.text, + why: suggestion.why, + }, + }), + ); + + // 5. Present confirmation card to the user + const request: ActionConfirmationRequest = { + candidateId: `suggest-${nowMs}`, + kind: "input_suggestion", + summary: `Type: "${suggestion.text}"`, + why: suggestion.why, + denyConsequence: "Text will not be typed into the active field.", + expiresAt: new Date(nowMs + this.promptTimeoutMs).toISOString(), + }; + + const decision = await this.confirmationPolicy.confirm(request); + + if (decision.outcome === "accepted") { + this.log?.( + JSON.stringify({ + inputSuggestionAccepted: { + target: targetKey, + text: suggestion.text, + }, + }), + ); + + // 6. User accepted: auto-type into target window via standard Pi Agent tool + const toolResult = await this.typeTextTool.execute( + `suggest-${nowMs}`, + { + text: suggestion.text, + ...(payload.windowTitle !== undefined ? { targetWindowTitle: payload.windowTitle } : {}), + ...(payload.applicationId !== undefined ? { targetApplicationId: payload.applicationId } : {}), + }, + undefined, + undefined, + {} as any, + ); + + if (toolResult.details?.ok === false) { + this.logError?.( + JSON.stringify({ + inputSuggestionTypingFailed: { + target: targetKey, + error: toolResult.details.error, + }, + }), + ); + } + } else { + this.log?.( + JSON.stringify({ + inputSuggestionRejected: { + target: targetKey, + decision: decision.outcome, + reason: decision.reason, + }, + }), + ); + } + } catch (err: unknown) { + const message = err instanceof Error ? err.message : String(err); + this.logError?.( + JSON.stringify({ + inputSuggestionFailed: { + target: targetKey, + error: message, + }, + }), + ); + } finally { + this.inFlight = false; + } + } + + /** + * Resets cooldown state (e.g. on test teardown or explicit session clear). + */ + reset(): void { + this.lastTargetKey = ""; + this.lastTriggerTime = 0; + this.inFlight = false; + } +} diff --git a/src/resident-agent.ts b/src/resident-agent.ts index 7b950f5..2f143da 100644 --- a/src/resident-agent.ts +++ b/src/resident-agent.ts @@ -92,7 +92,7 @@ export interface TurnActionTracker { * world-changing tool (or Pi renames one), add it here — a call not in this set * fails safe toward silence, which is the behavior this whole change is for. */ -const ACTING_TOOL_NAMES: ReadonlySet = new Set(["bash", "edit", "write"]); +const ACTING_TOOL_NAMES: ReadonlySet = new Set(["bash", "edit", "write", "type_text"]); export function turnActionTracker(): TurnActionTracker { let acted = false; diff --git a/src/text-typing-service.ts b/src/text-typing-service.ts new file mode 100644 index 0000000..4edaa4f --- /dev/null +++ b/src/text-typing-service.ts @@ -0,0 +1,255 @@ +/** + * Cross-platform desktop text typing service. + * + * Types text into the active focused field on behalf of the user when an + * action confirmation is approved. + * + * Handles target window focus restoration (since clicking "Allow" on the Slint + * panel window moves window manager focus to the panel), argument escaping, + * and fail-closed error handling. + */ + +import { windowsPowerShellPath } from "./windows-powershell.js"; + +export interface TextTypingTarget { + readonly text: string; + readonly targetWindowId?: string; + readonly targetWindowTitle?: string; + readonly targetApplicationId?: string; +} + +export interface TextTypingResult { + readonly ok: boolean; + readonly error?: string; +} + +export interface SubprocessExecutionResult { + readonly exitCode: number; + readonly stdout?: string; + readonly stderr?: string; +} + +export type SubprocessRunner = ( + command: readonly string[], + options?: { timeoutMs?: number }, +) => Promise; + +export interface TextTypingServiceOptions { + readonly platform?: NodeJS.Platform; + readonly runner?: SubprocessRunner; + readonly timeoutMs?: number; + readonly settleDelayMs?: number; + readonly sleep?: (ms: number) => Promise; + readonly env?: Record; +} + +async function defaultSubprocessRunner( + command: readonly string[], + options?: { timeoutMs?: number }, +): Promise { + const proc = Bun.spawn(command as string[], { + stdout: "pipe", + stderr: "pipe", + }); + let killed = false; + const timer = options?.timeoutMs + ? setTimeout(() => { + killed = true; + try { + proc.kill(); + } catch { + // ignore process kill failure + } + }, options.timeoutMs) + : undefined; + + try { + const [stdout, stderr, exitCode] = await Promise.all([ + new Response(proc.stdout).text(), + new Response(proc.stderr).text(), + proc.exited, + ]); + if (killed) { + return { exitCode: 124, stderr: `Subprocess timed out after ${options?.timeoutMs}ms.` }; + } + return { exitCode, stdout, stderr }; + } finally { + if (timer) clearTimeout(timer); + } +} + +/** + * Escapes characters that have special syntactic meaning in Windows Forms SendKeys: + * +, ^, %, ~, (, ), [, ], {, }, and converts newlines to {ENTER}. + * Each special character is enclosed in braces (e.g. `+` -> `{+}`). + */ +export function escapeSendKeys(text: string): string { + return text + .replace(/[{}+^%~()[\]]/g, (match) => `{${match}}`) + .replace(/\r?\n/g, "{ENTER}"); +} + +export class TextTypingService { + private readonly platform: NodeJS.Platform; + private readonly runner: SubprocessRunner; + private readonly timeoutMs: number; + private readonly settleDelayMs: number; + private readonly sleep: (ms: number) => Promise; + private readonly env: Record; + + constructor(options: TextTypingServiceOptions = {}) { + this.platform = options.platform ?? process.platform; + this.runner = options.runner ?? defaultSubprocessRunner; + this.timeoutMs = options.timeoutMs ?? 5_000; + this.settleDelayMs = options.settleDelayMs ?? 60; + this.sleep = options.sleep ?? ((ms) => new Promise((resolve) => setTimeout(resolve, ms))); + this.env = options.env ?? process.env; + } + + /** + * Types the specified text into the focused target input. + */ + async typeText(target: TextTypingTarget): Promise { + const { text, targetWindowId, targetWindowTitle, targetApplicationId } = target; + if (text.length === 0) { + return { ok: true }; + } + + try { + if (this.platform === "linux") { + return await this.typeLinux(text, targetWindowId, targetWindowTitle); + } + if (this.platform === "darwin") { + return await this.typeDarwin(text, targetApplicationId); + } + if (this.platform === "win32") { + return await this.typeWindows(text, targetWindowTitle); + } + return { + ok: false, + error: `Unsupported platform for desktop typing: ${this.platform}`, + }; + } catch (err) { + const errorMsg = err instanceof Error ? err.message : String(err); + return { ok: false, error: `Typing injection error: ${errorMsg}` }; + } + } + + private async typeLinux( + text: string, + targetWindowId?: string, + targetWindowTitle?: string, + ): Promise { + // 1. If target window ID or title is provided, reactivate the target window + // so keystrokes go to the application rather than the Slint confirmation card. + if (targetWindowId !== undefined && targetWindowId !== "") { + const activateResult = await this.runner( + ["xdotool", "windowactivate", "--sync", targetWindowId], + { timeoutMs: this.timeoutMs }, + ); + if (activateResult.exitCode !== 0) { + return { + ok: false, + error: activateResult.stderr?.trim() || `Failed to activate window ID ${targetWindowId}`, + }; + } + if (this.settleDelayMs > 0) { + await this.sleep(this.settleDelayMs); + } + } else if (targetWindowTitle !== undefined && targetWindowTitle.trim() !== "") { + const searchResult = await this.runner( + ["xdotool", "search", "--name", "--onlyvisible", targetWindowTitle.trim(), "windowactivate", "--sync"], + { timeoutMs: this.timeoutMs }, + ); + if (searchResult.exitCode !== 0) { + return { + ok: false, + error: searchResult.stderr?.trim() || `Failed to activate window with title "${targetWindowTitle}"`, + }; + } + if (this.settleDelayMs > 0) { + await this.sleep(this.settleDelayMs); + } + } + + // 2. Type text using xdotool. Use "--" to prevent interpreting leading hyphens as flags. + const result = await this.runner( + ["xdotool", "type", "--delay", "12", "--clearmodifiers", "--", text], + { timeoutMs: this.timeoutMs }, + ); + + if (result.exitCode !== 0) { + return { + ok: false, + error: result.stderr?.trim() || `xdotool failed with exit code ${result.exitCode}`, + }; + } + return { ok: true }; + } + + private async typeDarwin( + text: string, + targetApplicationId?: string, + ): Promise { + // Pass text and targetApplicationId as argv parameters to osascript to prevent script injection. + const script = `on run argv + set txt to item 1 of argv + if (count of argv) > 1 then + set appName to item 2 of argv + if appName is not "" then + try + tell application appName to activate + end try + delay 0.05 + end if + end if + tell application "System Events" + keystroke txt + end tell +end run`; + + const args = ["osascript", "-e", script, "--", text]; + if (targetApplicationId !== undefined && targetApplicationId !== "") { + args.push(targetApplicationId); + } + + const result = await this.runner(args, { timeoutMs: this.timeoutMs }); + + if (result.exitCode !== 0) { + return { + ok: false, + error: result.stderr?.trim() || `osascript failed with exit code ${result.exitCode}`, + }; + } + return { ok: true }; + } + + private async typeWindows( + text: string, + targetWindowTitle?: string, + ): Promise { + const pwshPath = windowsPowerShellPath(this.env); + const escapedText = escapeSendKeys(text).replace(/'/g, "''"); + const escapedTitle = targetWindowTitle ? targetWindowTitle.replace(/'/g, "''") : ""; + + const command = [ + pwshPath, + "-NoProfile", + "-NonInteractive", + "-Command", + escapedTitle + ? `$wshell = New-Object -ComObject WScript.Shell; if (-not $wshell.AppActivate('${escapedTitle}')) { exit 1 }; Start-Sleep -Milliseconds 60; [System.Windows.Forms.SendKeys]::SendWait('${escapedText}')` + : `[System.Windows.Forms.SendKeys]::SendWait('${escapedText}')`, + ]; + + const result = await this.runner(command, { timeoutMs: this.timeoutMs }); + + if (result.exitCode !== 0) { + return { + ok: false, + error: result.stderr?.trim() || `PowerShell typing failed with exit code ${result.exitCode}`, + }; + } + return { ok: true }; + } +} diff --git a/src/type-text-tool.ts b/src/type-text-tool.ts new file mode 100644 index 0000000..06b562b --- /dev/null +++ b/src/type-text-tool.ts @@ -0,0 +1,92 @@ +import { Type } from "typebox"; +import type { Static } from "typebox"; +import { + defineTool, + type AgentToolResult, + type ExtensionToolContext, + type ToolDefinition, +} from "@earendil-works/pi-coding-agent"; +import { TextTypingService, type TextTypingTarget } from "./text-typing-service.js"; + +/** + * Parameters for the standard Pi Agent `type_text` tool. + */ +export const typeTextParams = Type.Object({ + text: Type.String({ + description: "The text to type into the focused text target.", + }), + targetWindowTitle: Type.Optional( + Type.String({ + description: "Optional window title to activate before typing.", + }), + ), + targetApplicationId: Type.Optional( + Type.String({ + description: "Optional application name or process identifier to activate before typing.", + }), + ), +}); + +export type TypeTextArgs = Static; + +export interface TypeTextToolDetails { + ok: boolean; + charsTyped?: number; + error?: string; +} + +/** + * Creates the official Pi Agent ToolDefinition for typing text into desktop fields. + * + * Wraps `TextTypingService` to satisfy the official `@earendil-works/pi-coding-agent` + * tool specification, standardizing argument validation, execution, and result formatting. + */ +export function createTypeTextTool( + service: TextTypingService, +): ToolDefinition { + return defineTool({ + name: "type_text", + label: "Type text", + description: + "Type text into the currently active or specified desktop text target (input box, editor, or terminal). " + + "Reactivates the target window when specified to avoid dropping focus.", + promptSnippet: "type_text: inject text into active desktop input target", + parameters: typeTextParams, + execute: async ( + _toolCallId: string, + params: TypeTextArgs, + signal: AbortSignal | undefined, + _onUpdate: undefined, + _ctx: ExtensionToolContext, + ): Promise> => { + if (signal?.aborted) { + return { + content: [{ type: "text", text: "Text typing aborted." }], + details: { ok: false, error: "Aborted" }, + }; + } + + const target: TextTypingTarget = { + text: params.text, + ...(params.targetWindowTitle !== undefined ? { targetWindowTitle: params.targetWindowTitle } : {}), + ...(params.targetApplicationId !== undefined ? { targetApplicationId: params.targetApplicationId } : {}), + }; + + const result = await service.typeText(target); + if (!result.ok) { + return { + content: [{ type: "text", text: `Failed to type text: ${result.error ?? "unknown error"}` }], + details: { + ok: false, + ...(result.error !== undefined ? { error: result.error } : {}), + }, + }; + } + + return { + content: [{ type: "text", text: `Successfully typed ${params.text.length} characters.` }], + details: { ok: true, charsTyped: params.text.length }, + }; + }, + }); +} diff --git a/src/ui-smoke.ts b/src/ui-smoke.ts index 4d6f869..641771a 100644 --- a/src/ui-smoke.ts +++ b/src/ui-smoke.ts @@ -88,6 +88,7 @@ const CONFIG_ROWS: readonly Record[] = [ { type: "config", id: "watchRoot", label: "Watched folder", kind: "text", value: "/home/you/project", applies: "restart", group: "Files", description: "The folder Squid notices changes in." }, { type: "config", id: "retentionHours", label: "Keep activity for", kind: "int", value: "24", applies: "restart", group: "History", description: "How long Squid keeps what it recorded before deleting it." }, { type: "config", id: "screenReader", label: "Screen reading", kind: "enum", value: "", applies: "restart", group: "Screen", description: "How Squid reads the words on screen.", values: "linux" }, + { type: "config", id: "inputSuggestions", label: "Input suggestions", kind: "enum", value: "on", applies: "now", group: "Screen", description: "Suggests text when focusing an input field and auto-types on accept.", values: "on;off" }, { type: "config", id: "excludeApplications", label: "Never watch these apps", kind: "text", value: "1Password", applies: "restart", group: "Privacy", description: "Squid records nothing at all from the apps you list." }, { type: "config", id: "databasePath", label: "Where history is kept", kind: "text", value: "", applies: "restart", group: "History", description: "The file Squid writes its record to. Empty keeps it in memory only." }, ]; diff --git a/src/watch-session.ts b/src/watch-session.ts index 8ecc588..ac8bc20 100644 --- a/src/watch-session.ts +++ b/src/watch-session.ts @@ -27,7 +27,7 @@ import { } from "./resident-agent.js"; import type { ActivitySourceDiagnostic, SourceEntry } from "./activity-source.js"; import { SourceManager } from "./source-manager.js"; -import type { ActivityEvent, WorldState } from "./protocol.js"; +import type { ActivityEvent, FocusChangedPayload, WorldState } from "./protocol.js"; import type { PredictionRuntimeIngestResult } from "./prediction-runtime.js"; import { runRetentionPrune, type RetentionPruneResult } from "./retention-prune.js"; import { sqliteWatchExclusions } from "./sqlite-event-store.js"; @@ -36,6 +36,10 @@ import { resolvePanel } from "./resolve-panel.js"; import { startPanelServer, type PanelServer } from "./panel-server.js"; import type { DesktopNotifierFactory } from "./desktop-notice.js"; import { PanelConfirmationSurface } from "./panel-confirmation-surface.js"; +import { SerialConfirmationPolicy } from "./action-confirmation.js"; +import { TextTypingService } from "./text-typing-service.js"; +import { InputSuggestionService } from "./input-suggestion.js"; +import { inputSuggestionExtension } from "./input-suggestion-extension.js"; import { panelAgentLine, panelAgentThinking, @@ -182,6 +186,8 @@ export interface WatchSessionOptions { * tests inject a spy instead of ending the process. */ readonly stopExit?: () => void; + readonly textTypingService?: TextTypingService; + readonly inputSuggestionService?: InputSuggestionService; } export interface WatchSession { @@ -375,6 +381,26 @@ export async function startWatchSession( turnStarted: () => panelServerRef.server?.broadcast(panelAgentThinking(true)), turnEnded: () => panelServerRef.server?.broadcast(panelAgentThinking(false)), }; + let panelServer: PanelServer | undefined; + // The adapter is deliberately inert until a future execution seam calls it: + // current custom tools are read-only and Pi's sandbox owns tool safety. + const panelConfirmation = new PanelConfirmationSurface({ server: () => panelServer }); + const textTypingService = options.textTypingService ?? new TextTypingService({ env }); + const confirmationPolicy = new SerialConfirmationPolicy({ + surface: panelConfirmation, + timeoutMs: 30_000, + }); + let inputSuggestionsEnabled = env.SQUID_INPUT_SUGGESTIONS !== "off"; + const inputSuggestionService = + options.inputSuggestionService ?? + new InputSuggestionService({ + isEnabled: () => inputSuggestionsEnabled, + confirmationPolicy, + textTypingService, + log, + logError, + }); + const buildAgent = async (): Promise => { // The decision curator trims what the agent reads before each thought, // asking a fast Jev-shaped model which context a future turn will need. It @@ -409,6 +435,10 @@ export async function startWatchSession( attentionWindowExtension(attentionWindow.windowMs), heartbeatExtension(heartbeatSink), llmLogExt, + inputSuggestionExtension({ + service: inputSuggestionService, + typeTextTool: inputSuggestionService.getTypeTextTool?.(), + }), ...(decisionCurator === undefined ? [] : [decisionCurator]), ], agentDir, @@ -514,10 +544,7 @@ export async function startWatchSession( */ const sourceManagerRef: { manager: SourceManager | undefined } = { manager: undefined }; - let panelServer: PanelServer | undefined; - // The adapter is deliberately inert until a future execution seam calls it: - // current custom tools are read-only and Pi's sandbox owns tool safety. - const panelConfirmation = new PanelConfirmationSurface({ server: () => panelServer }); + // panelServer and confirmation/suggestion services moved above buildAgent /** * Echoes a command's own correlation id back on its verdict. * @@ -581,6 +608,9 @@ export async function startWatchSession( // needs_config: the agent starts in-session, immediately, on the // files just written. The result above already told the panel the // set landed; this is what makes its "applies: now" true. + if (outcome.ok && id === "inputSuggestions") { + inputSuggestionsEnabled = value !== "off"; + } if (outcome.ok && (isModelConfigKey(id) || id === "modelProvider")) { if (agent === undefined) { tryStartAgent(); @@ -987,6 +1017,11 @@ export async function startWatchSession( // The newest projection, kept for needs_config recovery: an agent started // later observes this instead of waking up blind to the world it missed. lastState = ingested.ingest.state; + if (event.kind === "focus_changed") { + void inputSuggestionService.onFocusChanged(event.payload as FocusChangedPayload, ingested.ingest.state).catch((error: unknown) => { + logError(JSON.stringify({ inputSuggestionError: describeErrorForAudit(error) })); + }); + } // A duplicate changed nothing about the world, so there is nothing new for // the agent to see and no reason to spend a turn on it. if (ingested.ingest.disposition === "duplicate") { @@ -1085,6 +1120,7 @@ export async function startWatchSession( // order. Hot-toggled sources are simply wherever `running` left them. if (gateSource !== undefined) await gateSource.stop(); await sourceManager.stopAll(); + await confirmationPolicy.close(); // Then the agent's turns, while its session is still open: a turn cut off // mid-flight loses whatever the agent was doing, and its session history is // the only record that it happened. The panel closes after that drain, so diff --git a/tests/config-service.test.ts b/tests/config-service.test.ts index fce5903..8121dda 100644 --- a/tests/config-service.test.ts +++ b/tests/config-service.test.ts @@ -172,11 +172,13 @@ describe("configDumpEntries", () => { expect(entryOf(live, "modelBaseUrl").applies).toBe("now"); expect(entryOf(live, "modelId").applies).toBe("now"); expect(entryOf(live, "apiKey").applies).toBe("now"); + expect(entryOf(live, "inputSuggestions").applies).toBe("now"); expect(entryOf(live, "watchRoot").applies).toBe("restart"); const idle = configDumpEntries(options(dir, { agentLive: () => false })); if (!idle.ok) throw new Error(idle.error); expect(entryOf(idle, "modelBaseUrl").applies).toBe("now"); + expect(entryOf(idle, "inputSuggestions").applies).toBe("now"); expect(entryOf(idle, "watchRoot").applies).toBe("restart"); rmSync(dir, { recursive: true, force: true }); }); @@ -204,6 +206,15 @@ describe("applyConfigSet on file-backed keys", () => { rmSync(dir, { recursive: true, force: true }); }); + it("writes inputSuggestions and reports now", () => { + const dir = tempDir("input-sug-set"); + const outcome = applyConfigSet(options(dir), "inputSuggestions", "off"); + expect(outcome).toEqual({ ok: true, applies: "now" }); + const onDisk = JSON.parse(readFileSync(join(dir, "config.json"), "utf8")) as Record; + expect(onDisk.inputSuggestions).toBe("off"); + rmSync(dir, { recursive: true, force: true }); + }); + it("writes unset as an empty string for a valid kind", () => { const dir = tempDir("file-set"); applyConfigSet(options(dir), "retentionHours", "72"); diff --git a/tests/config.test.ts b/tests/config.test.ts index 254d821..693ffe1 100644 --- a/tests/config.test.ts +++ b/tests/config.test.ts @@ -39,6 +39,8 @@ describe("config schema", () => { expect(configKeyById("watchRoot")?.env).toBe("SQUID_WATCH_ROOT"); expect(configKeyById("databasePath")?.env).toBe("SQUID_DB"); expect(configKeyByEnv("SQUID_SCREEN_POLL_MS")?.id).toBe("screenPollMs"); + expect(configKeyById("inputSuggestions")?.env).toBe("SQUID_INPUT_SUGGESTIONS"); + expect(configKeyByEnv("SQUID_INPUT_SUGGESTIONS")?.id).toBe("inputSuggestions"); expect(configKeyByEnv("SQUID_PANEL_SOCKET")).toBeUndefined(); }); @@ -57,6 +59,12 @@ describe("config schema", () => { expect(reader.validate("Synthetic").ok).toBe(false); expect(reader.validate("vision").ok).toBe(false); + const inputSug = configKeyById("inputSuggestions")!; + expect(inputSug.validate("on")).toEqual({ ok: true }); + expect(inputSug.validate("off")).toEqual({ ok: true }); + expect(inputSug.validate("true").ok).toBe(false); + expect(inputSug.validate("false").ok).toBe(false); + const root = configKeyById("watchRoot")!; expect(root.validate("/home/me")).toEqual({ ok: true }); // A pasted trailing newline is a different directory; the file layer's @@ -225,6 +233,7 @@ describe("config schema", () => { ["watchRoot", "Files"], ["screenReader", "Screen"], ["screenPollMs", "Screen"], + ["inputSuggestions", "Screen"], ["browser", "Browser"], ["browserPollMs", "Browser"], ["terminal", "Terminal"], diff --git a/tests/input-suggestion-extension.test.ts b/tests/input-suggestion-extension.test.ts new file mode 100644 index 0000000..5a5305d --- /dev/null +++ b/tests/input-suggestion-extension.test.ts @@ -0,0 +1,49 @@ +import { describe, expect, it } from "bun:test"; +import { inputSuggestionExtension } from "../src/input-suggestion-extension.js"; +import { InputSuggestionService } from "../src/input-suggestion.js"; +import type { ExtensionAPI, ToolDefinition } from "@earendil-works/pi-coding-agent"; + +describe("inputSuggestionExtension", () => { + it("conforms to Pi InlineExtension interface and registers type_text tool", () => { + let resetCount = 0; + const fakeService = { + reset: () => { + resetCount++; + }, + getTypeTextTool: () => + ({ + name: "type_text", + }) as unknown as ToolDefinition, + } as unknown as InputSuggestionService; + + const extension = inputSuggestionExtension({ service: fakeService }); + if (typeof extension === "function") { + throw new Error("expected object extension"); + } + + expect(extension.name).toBe("squid-input-suggestion"); + expect(extension.hidden).toBe(true); + + const registeredTools: any[] = []; + const eventHandlers: Record void> = {}; + + const fakePi = { + registerTool: (tool: any) => { + registeredTools.push(tool); + }, + on: (event: string, handler: () => void) => { + eventHandlers[event] = handler; + }, + } as unknown as ExtensionAPI; + + extension.factory(fakePi); + + expect(registeredTools).toHaveLength(1); + expect(registeredTools[0]?.name).toBe("type_text"); + + // Test session_shutdown lifecycle hook + expect(eventHandlers["session_shutdown"]).toBeDefined(); + eventHandlers["session_shutdown"]?.(); + expect(resetCount).toBe(1); + }); +}); diff --git a/tests/input-suggestion.test.ts b/tests/input-suggestion.test.ts new file mode 100644 index 0000000..c0f2b07 --- /dev/null +++ b/tests/input-suggestion.test.ts @@ -0,0 +1,294 @@ +import { describe, expect, it } from "bun:test"; +import type { + ActionConfirmationDecision, + ActionConfirmationPolicy, + ActionConfirmationRequest, +} from "../src/action-confirmation.js"; +import { + heuristicSuggestInput, + InputSuggestionService, + isTextInputTarget, +} from "../src/input-suggestion.js"; +import type { FocusChangedPayload } from "../src/protocol.js"; +import type { TextTypingResult, TextTypingTarget } from "../src/text-typing-service.js"; +import { TextTypingService } from "../src/text-typing-service.js"; + +describe("isTextInputTarget", () => { + it("identifies text_input and editor roles", () => { + expect( + isTextInputTarget({ + role: "text_input", + applicationId: "code", + windowTitle: "main.ts", + label: "Line 10", + }), + ).toBe(true); + + expect( + isTextInputTarget({ + role: "editor", + applicationId: "sublime", + windowTitle: "document.txt", + label: null, + }), + ).toBe(true); + + expect( + isTextInputTarget({ + role: "button", + applicationId: "code", + windowTitle: "dialog", + label: "Submit", + }), + ).toBe(false); + }); + + it("identifies terminal applications even when role is unknown", () => { + expect( + isTextInputTarget({ + role: "unknown", + applicationId: "org.gnome.Terminal", + windowTitle: "user@host:~", + label: null, + }), + ).toBe(true); + + expect( + isTextInputTarget({ + role: "unknown", + applicationId: "kitty", + windowTitle: "kitty", + label: null, + }), + ).toBe(true); + + expect( + isTextInputTarget({ + role: "unknown", + applicationId: "gnome-shell", + windowTitle: "Desktop", + label: null, + }), + ).toBe(false); + + expect( + isTextInputTarget({ + role: "button", + applicationId: "org.gnome.Terminal", + windowTitle: "Terminal - bash", + label: "Close Tab", + }), + ).toBe(false); + }); +}); + +describe("heuristicSuggestInput", () => { + it("suggests test command when terminal is active after a test file was modified", () => { + const suggestion = heuristicSuggestInput({ + payload: { + role: "text_input", + applicationId: "terminal", + windowTitle: "Terminal - bash", + label: null, + }, + worldState: { + recentFiles: ["src/config.test.ts"], + } as any, + }); + + expect(suggestion).not.toBeNull(); + expect(suggestion?.text).toBe("bun test"); + expect(suggestion?.why).toContain("config.test.ts"); + }); + + it("suggests git status by default in terminal", () => { + const suggestion = heuristicSuggestInput({ + payload: { + role: "text_input", + applicationId: "terminal", + windowTitle: "Terminal - bash", + label: null, + }, + worldState: null, + }); + + expect(suggestion?.text).toBe("git status"); + }); + + it("suggests search query when focusing on a search box", () => { + const suggestion = heuristicSuggestInput({ + payload: { + role: "text_input", + applicationId: "google-chrome", + windowTitle: "GitHub", + label: "Search or jump to...", + }, + worldState: null, + }); + + expect(suggestion).not.toBeNull(); + expect(suggestion?.text).toContain("search"); + expect(suggestion?.why).toContain("search field"); + }); +}); + +describe("InputSuggestionService", () => { + const createMockPolicy = (decision: ActionConfirmationDecision): { + policy: ActionConfirmationPolicy; + confirmedRequests: ActionConfirmationRequest[]; + } => { + const confirmedRequests: ActionConfirmationRequest[] = []; + const policy: ActionConfirmationPolicy = { + async confirm(request) { + confirmedRequests.push(request); + return decision; + }, + }; + return { policy, confirmedRequests }; + }; + + const createMockTypingService = (): { + service: TextTypingService; + typedTargets: TextTypingTarget[]; + } => { + const typedTargets: TextTypingTarget[] = []; + const service = new TextTypingService({ + runner: async () => ({ exitCode: 0 }), + }); + service.typeText = async (target: TextTypingTarget): Promise => { + typedTargets.push(target); + return { ok: true }; + }; + return { service, typedTargets }; + }; + + it("does nothing when disabled by configuration", async () => { + const { policy, confirmedRequests } = createMockPolicy({ + outcome: "accepted", + reason: "user_accepted", + }); + const { service, typedTargets } = createMockTypingService(); + + const suggestionService = new InputSuggestionService({ + isEnabled: () => false, + confirmationPolicy: policy, + textTypingService: service, + }); + + await suggestionService.onFocusChanged( + { + role: "text_input", + applicationId: "terminal", + windowTitle: "bash", + label: null, + }, + null, + ); + + expect(confirmedRequests.length).toBe(0); + expect(typedTargets.length).toBe(0); + }); + + it("prompts user and types text when user accepts", async () => { + const { policy, confirmedRequests } = createMockPolicy({ + outcome: "accepted", + reason: "user_accepted", + }); + const { service, typedTargets } = createMockTypingService(); + + const suggestionService = new InputSuggestionService({ + isEnabled: () => true, + confirmationPolicy: policy, + textTypingService: service, + predictSuggestion: async () => ({ + text: "git diff", + why: "Review recent changes", + }), + }); + + await suggestionService.onFocusChanged( + { + role: "text_input", + applicationId: "terminal", + windowTitle: "bash", + label: null, + }, + null, + ); + + expect(confirmedRequests.length).toBe(1); + expect(confirmedRequests[0]?.kind).toBe("input_suggestion"); + expect(confirmedRequests[0]?.summary).toBe('Type: "git diff"'); + expect(confirmedRequests[0]?.why).toBe("Review recent changes"); + + expect(typedTargets.length).toBe(1); + expect(typedTargets[0]?.text).toBe("git diff"); + expect(typedTargets[0]?.targetWindowTitle).toBe("bash"); + }); + + it("does NOT type text when user declines or prompt expires", async () => { + const { policy, confirmedRequests } = createMockPolicy({ + outcome: "rejected", + reason: "user_rejected", + }); + const { service, typedTargets } = createMockTypingService(); + + const suggestionService = new InputSuggestionService({ + isEnabled: () => true, + confirmationPolicy: policy, + textTypingService: service, + }); + + await suggestionService.onFocusChanged( + { + role: "text_input", + applicationId: "terminal", + windowTitle: "bash", + label: null, + }, + null, + ); + + expect(confirmedRequests.length).toBe(1); + // Crucial: No typing occurred! + expect(typedTargets.length).toBe(0); + }); + + it("deduplicates rapid focus changes within cooldown window", async () => { + const { policy, confirmedRequests } = createMockPolicy({ + outcome: "accepted", + reason: "user_accepted", + }); + const { service, typedTargets } = createMockTypingService(); + + let clock = 1000; + const suggestionService = new InputSuggestionService({ + isEnabled: () => true, + confirmationPolicy: policy, + textTypingService: service, + cooldownMs: 3000, + now: () => clock, + }); + + const payload: FocusChangedPayload = { + role: "text_input", + applicationId: "terminal", + windowTitle: "bash", + label: null, + }; + + // First focus: triggers + await suggestionService.onFocusChanged(payload, null); + expect(confirmedRequests.length).toBe(1); + + // Second focus 500ms later: debounced/skipped + clock += 500; + await suggestionService.onFocusChanged(payload, null); + expect(confirmedRequests.length).toBe(1); + + // Third focus after cooldown (3500ms later): triggers + clock += 3500; + await suggestionService.onFocusChanged(payload, null); + expect(confirmedRequests.length).toBe(2); + }); +}); diff --git a/tests/resident-agent.test.ts b/tests/resident-agent.test.ts index c6474b9..75cae44 100644 --- a/tests/resident-agent.test.ts +++ b/tests/resident-agent.test.ts @@ -115,6 +115,9 @@ describe("turnActionTracker", () => { // Running a command or editing a file is the action worth a toast. toolCall("bash"); expect(tracker.acted()).toBe(true); + tracker.reset(); + toolCall("type_text"); + expect(tracker.acted()).toBe(true); // Reset is what the real session calls at the top of each prompt, so the // flag reports this turn and not a stale one. tracker.reset(); diff --git a/tests/text-typing-service.test.ts b/tests/text-typing-service.test.ts new file mode 100644 index 0000000..bac7165 --- /dev/null +++ b/tests/text-typing-service.test.ts @@ -0,0 +1,246 @@ +import { describe, expect, it } from "bun:test"; +import { + escapeSendKeys, + TextTypingService, + type SubprocessExecutionResult, +} from "../src/text-typing-service.js"; + +describe("escapeSendKeys", () => { + it("escapes special SendKeys characters in braces", () => { + expect(escapeSendKeys("hello world")).toBe("hello world"); + expect(escapeSendKeys("a+b")).toBe("a{+}b"); + expect(escapeSendKeys("{test}")).toBe("{{}test{}}"); + expect(escapeSendKeys("func()")).toBe("func{(}{)}"); + expect(escapeSendKeys("arr[0]")).toBe("arr{[}0{]}"); + expect(escapeSendKeys("100% ~home ^key")).toBe("100{%} {~}home {^}key"); + }); + + it("converts newlines to {ENTER}", () => { + expect(escapeSendKeys("line1\nline2")).toBe("line1{ENTER}line2"); + expect(escapeSendKeys("line1\r\nline2")).toBe("line1{ENTER}line2"); + }); +}); + +describe("TextTypingService on Linux", () => { + it("reactivates target window before typing", async () => { + const executedCommands: (readonly string[])[] = []; + const sleptMs: number[] = []; + + const service = new TextTypingService({ + platform: "linux", + runner: async (cmd): Promise => { + executedCommands.push(cmd); + return { exitCode: 0, stdout: "" }; + }, + sleep: async (ms) => { + sleptMs.push(ms); + }, + }); + + const result = await service.typeText({ + text: "git status", + targetWindowId: "0x123456", + }); + + expect(result.ok).toBe(true); + expect(executedCommands.length).toBe(2); + expect(executedCommands[0]).toEqual(["xdotool", "windowactivate", "--sync", "0x123456"]); + expect(sleptMs).toEqual([60]); + expect(executedCommands[1]).toEqual([ + "xdotool", + "type", + "--delay", + "12", + "--clearmodifiers", + "--", + "git status", + ]); + }); + + it("fails closed without typing if window activation fails", async () => { + const executedCommands: (readonly string[])[] = []; + + const service = new TextTypingService({ + platform: "linux", + runner: async (cmd): Promise => { + executedCommands.push(cmd); + return { exitCode: 1, stderr: "window not found" }; + }, + }); + + const result = await service.typeText({ + text: "secret input", + targetWindowId: "0xdeadbeef", + }); + + expect(result.ok).toBe(false); + expect(result.error).toContain("window not found"); + // Crucial: keystrokes were NOT typed because window activation failed! + expect(executedCommands.length).toBe(1); + expect(executedCommands[0]?.[0]).toBe("xdotool"); + expect(executedCommands[0]?.[1]).toBe("windowactivate"); + }); + + it("searches and activates by targetWindowTitle if no windowId is given", async () => { + const executedCommands: (readonly string[])[] = []; + + const service = new TextTypingService({ + platform: "linux", + runner: async (cmd): Promise => { + executedCommands.push(cmd); + return { exitCode: 0, stdout: "" }; + }, + sleep: async () => {}, + }); + + const result = await service.typeText({ + text: "--all", + targetWindowTitle: "Terminal - bash", + }); + + expect(result.ok).toBe(true); + expect(executedCommands[0]).toEqual([ + "xdotool", + "search", + "--name", + "--onlyvisible", + "Terminal - bash", + "windowactivate", + "--sync", + ]); + expect(executedCommands[1]).toEqual([ + "xdotool", + "type", + "--delay", + "12", + "--clearmodifiers", + "--", + "--all", + ]); + }); + + it("types directly when no target window identifier is specified", async () => { + const executedCommands: (readonly string[])[] = []; + + const service = new TextTypingService({ + platform: "linux", + runner: async (cmd): Promise => { + executedCommands.push(cmd); + return { exitCode: 0, stdout: "" }; + }, + }); + + const result = await service.typeText({ + text: "bun test", + }); + + expect(result.ok).toBe(true); + expect(executedCommands.length).toBe(1); + expect(executedCommands[0]).toEqual([ + "xdotool", + "type", + "--delay", + "12", + "--clearmodifiers", + "--", + "bun test", + ]); + }); + + it("reports failure when xdotool exits with non-zero code", async () => { + const service = new TextTypingService({ + platform: "linux", + runner: async (): Promise => { + return { exitCode: 1, stderr: "xdotool: command not found" }; + }, + }); + + const result = await service.typeText({ + text: "echo test", + }); + + expect(result.ok).toBe(false); + expect(result.error).toContain("xdotool: command not found"); + }); +}); + +describe("TextTypingService on Darwin", () => { + it("invokes osascript passing text and target app as argv arguments", async () => { + const executedCommands: (readonly string[])[] = []; + + const service = new TextTypingService({ + platform: "darwin", + runner: async (cmd): Promise => { + executedCommands.push(cmd); + return { exitCode: 0, stdout: "" }; + }, + }); + + const result = await service.typeText({ + text: "hello world", + targetApplicationId: "com.apple.Terminal", + }); + + expect(result.ok).toBe(true); + expect(executedCommands.length).toBe(1); + const cmd = executedCommands[0]!; + expect(cmd[0]).toBe("osascript"); + expect(cmd[1]).toBe("-e"); + expect(cmd[3]).toBe("--"); + expect(cmd[4]).toBe("hello world"); + expect(cmd[5]).toBe("com.apple.Terminal"); + }); +}); + +describe("TextTypingService on Windows", () => { + it("invokes PowerShell with SendKeys and escapes quotes properly", async () => { + const executedCommands: (readonly string[])[] = []; + + const service = new TextTypingService({ + platform: "win32", + runner: async (cmd): Promise => { + executedCommands.push(cmd); + return { exitCode: 0, stdout: "" }; + }, + }); + + const result = await service.typeText({ + text: "don't fail", + targetWindowTitle: "Bob's Terminal", + }); + + expect(result.ok).toBe(true); + expect(executedCommands.length).toBe(1); + const cmd = executedCommands[0]!; + expect(cmd[0]).toContain("powershell"); + expect(cmd[4]).toContain("AppActivate('Bob''s Terminal')"); + expect(cmd[4]).toContain("[System.Windows.Forms.SendKeys]::SendWait('don''t fail')"); + }); +}); + +describe("TextTypingService general behavior", () => { + it("returns ok immediately for empty text without invoking subprocess", async () => { + let invoked = false; + const service = new TextTypingService({ + runner: async () => { + invoked = true; + return { exitCode: 0 }; + }, + }); + + const result = await service.typeText({ text: "" }); + expect(result.ok).toBe(true); + expect(invoked).toBe(false); + }); + + it("handles unsupported platforms cleanly", async () => { + const service = new TextTypingService({ + platform: "aix" as NodeJS.Platform, + runner: async () => ({ exitCode: 0 }), + }); + + const result = await service.typeText({ text: "some text" }); + expect(result.ok).toBe(false); + expect(result.error).toContain("Unsupported platform"); + }); +}); diff --git a/tests/type-text-tool.test.ts b/tests/type-text-tool.test.ts new file mode 100644 index 0000000..df857d3 --- /dev/null +++ b/tests/type-text-tool.test.ts @@ -0,0 +1,102 @@ +import { describe, expect, it } from "bun:test"; +import { createTypeTextTool } from "../src/type-text-tool.js"; +import { TextTypingService, type TextTypingTarget } from "../src/text-typing-service.js"; + +describe("createTypeTextTool", () => { + it("defines standard Pi tool metadata and parameter schema", () => { + const service = new TextTypingService(); + const tool = createTypeTextTool(service); + + expect(tool.name).toBe("type_text"); + expect(tool.label).toBe("Type text"); + expect(tool.description).toContain("Type text into the currently active"); + expect(tool.parameters).toBeDefined(); + expect(tool.parameters.type).toBe("object"); + }); + + it("executes successfully through TextTypingService and returns AgentToolResult", async () => { + const typedTargets: TextTypingTarget[] = []; + const fakeService = { + typeText: async (target: TextTypingTarget) => { + typedTargets.push(target); + return { ok: true }; + }, + } as unknown as TextTypingService; + + const tool = createTypeTextTool(fakeService); + const result = await tool.execute( + "call-1", + { + text: "hello world", + targetWindowTitle: "Terminal", + targetApplicationId: "kitty", + }, + undefined, + undefined, + {} as any, + ); + + expect(result.content[0]).toEqual({ + type: "text", + text: "Successfully typed 11 characters.", + }); + expect(result.details).toEqual({ + ok: true, + charsTyped: 11, + }); + expect(typedTargets).toEqual([ + { + text: "hello world", + targetWindowTitle: "Terminal", + targetApplicationId: "kitty", + }, + ]); + }); + + it("returns error details when typing fails without throwing", async () => { + const fakeService = { + typeText: async () => { + return { ok: false, error: "Window activation failed" }; + }, + } as unknown as TextTypingService; + + const tool = createTypeTextTool(fakeService); + const result = await tool.execute( + "call-2", + { text: "failed text" }, + undefined, + undefined, + {} as any, + ); + + expect(result.content[0]?.type).toBe("text"); + expect((result.content[0] as any)?.text).toContain("Failed to type text: Window activation failed"); + expect(result.details).toEqual({ + ok: false, + error: "Window activation failed", + }); + }); + + it("aborts when AbortSignal is already triggered", async () => { + const controller = new AbortController(); + controller.abort(); + + const fakeService = { + typeText: async () => ({ ok: true }), + } as unknown as TextTypingService; + + const tool = createTypeTextTool(fakeService); + const result = await tool.execute( + "call-3", + { text: "aborted text" }, + controller.signal, + undefined, + {} as any, + ); + + expect(result.details).toEqual({ + ok: false, + error: "Aborted", + }); + }); +}); diff --git a/tests/watch-session.test.ts b/tests/watch-session.test.ts index 3a57f60..93990ec 100644 --- a/tests/watch-session.test.ts +++ b/tests/watch-session.test.ts @@ -228,6 +228,7 @@ async function startSession(options: { readonly stopExit?: () => void; /** A notifier that records instead of interrupting whoever runs the suite. */ readonly notifierFactory?: DesktopNotifierFactory; + readonly inputSuggestionService?: any; }) { const seenContexts: WatchSourceContext[] = []; const agentLog = options.agentLog ?? emptyAgentLog(); @@ -246,6 +247,7 @@ async function startSession(options: { ...(options.restartExit === undefined ? {} : { restartExit: options.restartExit }), ...(options.stopExit === undefined ? {} : { stopExit: options.stopExit }), ...(options.notifierFactory === undefined ? {} : { notifierFactory: options.notifierFactory }), + ...(options.inputSuggestionService === undefined ? {} : { inputSuggestionService: options.inputSuggestionService }), createSources: async (context) => { seenContexts.push(context); if (options.sourceEntries !== undefined) return options.sourceEntries; @@ -2133,6 +2135,64 @@ describe("the control panel on a watch session", () => { await h.close(); } }); + + it("updates inputSuggestions on config_set and applies now", async () => { + const h = await startSettingsSession(false); + try { + h.client.send(JSON.stringify({ type: "config_set", id: "inputSuggestions", value: "off" })); + const setResult = await awaitConfigResult(h.client, "inputSuggestions"); + expect(setResult.ok).toBe(1); + expect(setResult.applies).toBe("now"); + expect(h.restartCalls).toHaveLength(0); + + h.client.send(JSON.stringify({ type: "config_get" })); + const configs = await awaitConfigEnd(h.client); + const sugRow = configs.find((l) => l.id === "inputSuggestions"); + expect(sugRow?.value).toBe("off"); + expect(sugRow?.applies).toBe("now"); + } finally { + await h.close(); + } + }); + + it("dispatches focus_changed event to inputSuggestionService", async () => { + let focusReceived = false; + const mockSuggestionService: any = { + async onFocusChanged(payload: any) { + if (payload.role === "text_input") { + focusReceived = true; + } + }, + }; + + const source = new ScriptedSource("synth-focus", []); + const { result } = await startSession({ + sources: [source], + inputSuggestionService: mockSuggestionService, + }); + if ("configError" in result) throw new Error(result.configError); + try { + await source.emit!(createActivityEvent({ + kind: "focus_changed", + eventId: "fc-1", + source: { kind: "synthetic", instanceId: "s-1" }, + sessionId: "s1", + sequence: 1, + payload: { + applicationId: "terminal", + windowTitle: "bash", + role: "text_input", + label: null, + }, + })); + + // Yield to allow async dispatch to complete + await new Promise((resolve) => setTimeout(resolve, 50)); + expect(focusReceived).toBe(true); + } finally { + await result.session.stop(); + } + }); }); }); diff --git a/ui/settings.slint b/ui/settings.slint index 6e310c0..0fc28af 100644 --- a/ui/settings.slint +++ b/ui/settings.slint @@ -1,4 +1,4 @@ -import { Button, LineEdit } from "std-widgets.slint"; +import { Button, LineEdit, Switch } from "std-widgets.slint"; import { Theme } from "theme.slint"; import { SettingRow } from "model.slint"; @@ -158,7 +158,21 @@ export component SettingEditor inherits Rectangle { wrap: word-wrap; } - HorizontalLayout { + if root.setting.values == "on;off" || root.setting.values == "off;on": HorizontalLayout { + spacing: Theme.gap; + toggle-switch := Switch { + text: toggle-switch.checked ? "Enabled" : "Disabled"; + checked: root.setting.value != "off"; + accessible-label: root.setting.label; + accessible-description: root.setting.description; + toggled => { + root.save(root.setting.id, self.checked ? "on" : "off"); + } + } + Rectangle { horizontal-stretch: 1; } + } + + if root.setting.values != "on;off" && root.setting.values != "off;on": HorizontalLayout { spacing: Theme.gap-tight; editor := LineEdit { @@ -209,7 +223,14 @@ export component SettingEditor inherits Rectangle { wrap: word-wrap; } - if root.setting.values != "": Text { + if root.setting.id == "inputSuggestions": Text { + text: "When enabled, Squid pops up intelligent suggestions and auto-types them upon acceptance."; + color: Theme.text-muted; + font-size: 11px; + wrap: word-wrap; + } + + if root.setting.values != "" && root.setting.values != "on;off" && root.setting.values != "off;on": Text { text: "Choose one of: " + root.setting.values; color: Theme.text-muted; font-size: 11px;