Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions docs/findings/13-acp-agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,3 +191,89 @@ Both agents produced only AG-UI types the SPA already folds (`RUN_*`, `TEXT_MESS
`CLAUDE_CODE_SESSION_ID`, …

Conclusions → [`adr/0013-agents-over-acp.md`](../adr/0013-agents-over-acp.md).

## Addendum 2: host settings, verified with a prompt (#63, same day)

The production adapter (`src/runtime/acp-adapter.ts`, definitions in `acp-agents.ts`) run
through `bun spikes/13-acp-agents.ts isolation <agent> <model> [ignore|include]`. The scratch
repo carries **project** settings: `CLAUDE.md` and `AGENTS.md` with a codeword each, and a skill
in `.claude/skills` and one in `.agents/skills`. One structured step asks the agent which skills
it has and which codewords its instructions hold, then runs `echo $XDG_CONFIG_HOME`,
`gh auth status`, `git config --global --list --show-origin` and `env` through its own shell.
The host has 36 entries in `~/.claude/skills` (49 `SKILL.md` files below them) plus the
`anthropic-skills` plugin, 34 in `~/.agents/skills`, `gh-stack` in `~/.config/opencode/skills`,
and no `~/.claude/CLAUDE.md`. For
the user-level `CLAUDE.md`, Claude ran with `CLAUDE_CONFIG_DIR` pointing at a copy of `~/.claude`
made of symlinks plus a `CLAUDE.md` holding a codeword, so the real `~/.claude` stayed untouched.

### Claude Code: `settingSources: ["project", "local"]` keeps skills and `CLAUDE.md` out

| `hostSettings` | skills listed | host skills among them | user `CLAUDE.md` codeword | project `CLAUDE.md` codeword | project skill |
|---|---|---|---|---|---|
| `include` (SDK default `user, project, local`) | 47 | 21 user + 13 `anthropic-skills:*` | seen | seen | seen |
| `ignore` (`project, local`) | 14 | none | **not seen** | seen | seen |

The other 13 are Claude Code's bundled skills (`simplify`, `loop`, `claude-api`, …). The open
question of the first addendum is settled: `settingSources` keeps the user's skills, plugins and
`CLAUDE.md` out of a run. The residue stays as described there: claude-agent-acp still reads the
user's `model`, `permissions.defaultMode` and `availableModels` itself. Factory always sends a
model and answers every ask, so only an `availableModels` allowlist could still narrow a run.

### opencode: `OPENCODE_CONFIG_DIR` does not hide anything

opencode 1.18.31 (read from the bundled source, then measured) builds its config directories as
`[$XDG_CONFIG_HOME/opencode, project .opencode dirs, ~/.opencode, $OPENCODE_CONFIG_DIR]` and loads
the global config file from the first unconditionally. `OPENCODE_CONFIG_DIR` **adds** a
directory: with it set to an empty dir, `session/new` still offered 715 models including the
host's `omniroute` provider. `XDG_CONFIG_HOME` is read once, at start, into opencode's paths. The
bash tool spawns with `{...process.env, ...shell.env hook output}`. That makes the mechanism:

- `XDG_CONFIG_HOME` → an empty factory-owned dir hides `~/.config/opencode`;
- a factory plugin (`src/runtime/acp-opencode-plugin.ts`, loaded by file URL through
`OPENCODE_CONFIG_CONTENT`) uses opencode's `shell.env` hook to hand the host's
`XDG_CONFIG_HOME` (or `~/.config`) back to every shell command, so `gh` and git keep their
config.

That still left a second leak. opencode also reads **home-level** files of its own accord:
`~/.claude/skills`, `~/.agents/skills`, `~/.claude/CLAUDE.md` and `~/.opencode`. They hang off
its notion of home, which `OPENCODE_TEST_HOME` overrides. The variable changes neither `HOME`
nor the data dir, so provider logins are kept.

| opencode env | skills listed | host skills | project skills | models offered | shell `XDG_CONFIG_HOME` | `gh auth status` | git global config |
|---|---|---|---|---|---|---|---|
| `include` (host's own) | 51 | 48 | both | 715 | unset | logged in | read |
| `XDG_CONFIG_HOME` + plugin | 51 | 48 | both | 39 | `/home/karl/.config` | logged in | read |
| … + `OPENCODE_DISABLE_CLAUDE_CODE_SKILLS=1` | 36 | 34 (`~/.agents`) | `.agents` only | 39 | `/home/karl/.config` | logged in | read |
| … + `OPENCODE_TEST_HOME` → empty dir (**shipped**) | 3 | none | both | 39 | `/home/karl/.config` | logged in | read |

The 3 skills are opencode's built-in `customize-opencode` and the two project skills. In every
variant opencode took the project's `AGENTS.md` codeword; it reads `CLAUDE.md` only when there is
no `AGENTS.md`. The 39 models are the auth-based `opencode` and `opencode-go` providers. Hiding the
host's skill listing shrank opencode's prompt: the `implement` step's cached input went from
~13.9k to ~7.7k tokens.

- `OPENCODE_DISABLE_CLAUDE_CODE` / `_SKILLS` / `_PROMPT` are not an alternative. They also drop the
project's `.claude/skills` and `CLAUDE.md`, and leave `~/.agents/skills` in.
- `OPENCODE_TEST_HOME` is undocumented, a test hook by its name. If opencode drops it, the host's
skills reappear in runs; nothing breaks. Other effects, read from the source: `~/` in a
project's opencode config (permission patterns, `skills.paths`, `instructions`, MCP paths)
resolves against the empty dir; the TUI's path display is not used under ACP.

### Environment scrub

The adapter's agent environment drops `CLAUDECODE` and every `CLAUDE_*` variable except
`CLAUDE_CONFIG_DIR`, `CLAUDE_CODE_OAUTH_TOKEN`, `CLAUDE_CODE_USE_BEDROCK` and
`CLAUDE_CODE_USE_VERTEX` (where the credentials are, or which provider pays). Run from this Claude
Code session: 12 `CLAUDE*` variables in the parent, none passed on. With the parent's
`CLAUDE_EFFORT`, `CLAUDE_CODE_SESSION_ID` and `CLAUDE_CODE_SUBAGENT_MODEL` set to a sentinel, the
sentinel appeared 0 times in either agent's shell. Claude's shell still shows `CLAUDECODE`,
`CLAUDE_CODE_SESSION_ID`, `CLAUDE_EFFORT`, … — those belong to the agent's own Claude Code
session. opencode's shell shows none.

### Availability

`checkAgents(["claude", "opencode"])` reports `opencode` missing from PATH (`Bun.which`) and
Claude not logged in. The second comes from the bundled CLI's `auth status --json`
(`bun …/claude-agent-acp/dist/index.js --cli auth status --json`, ~0.25 s): logged out means no
claude.ai login, no API key source and the first-party backend. On this host both pass. With
`HOME` pointed at an empty dir and opencode off PATH, both are reported.
99 changes: 83 additions & 16 deletions spikes/13-acp-agents.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
/**
* SPIKE (finding 13): run a hello-shaped workflow through factory's real
* runtime (`startRun`, `AgentRuntime`, `agent-step.ts`) on the ACP adapter,
* against opencode and Claude Code, and measure what the ADR needs to know:
* model selection, usage, structured output, permission asks, cancel.
* SPIKE (finding 13), kept as the live check of the production ACP adapter
* (#63): run a hello-shaped workflow through factory's real runtime
* (`startRun`, `AgentRuntime`, `agent-step.ts`) on opencode or Claude Code,
* and measure model selection, usage, structured output, permission asks,
* cancel, and what the agent can see of the host's settings.
*
* bun spikes/13-acp-agents.ts run <claude|opencode> <model>
* bun spikes/13-acp-agents.ts cancel <claude|opencode> <model> [afterMs]
* bun spikes/13-acp-agents.ts bad-model <claude|opencode>
* bun spikes/13-acp-agents.ts isolation <claude|opencode> <model> [ignore|include]
*
* `isolation` asks the agent which skills and instructions it has and runs
* `gh auth status`, git's global config and `env` through its shell, with the
* host's agent settings ignored (default) or included.
*
* Writes `spikes/out/<mode>-<agent>-<ts>/{events,diagnostics}.ndjson`.
*/
Expand All @@ -15,19 +21,22 @@ import { mkdir, mkdtemp, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { RunEvent } from "../src/events";
import { acpAdapter, type AcpDiagnostic } from "../src/runtime/acp-adapter";
import {
ACP_AGENTS,
acpAdapter,
ACP_AGENT_KINDS,
acpAgent,
agentEnv,
type AcpAgentKind,
type AcpDiagnostic,
} from "../src/runtime/acp-adapter";
type HostSettings,
} from "../src/runtime/acp-agents";
import { makeAgentRuntime } from "../src/runtime/agent-runtime";
import { startRun } from "../src/runtime/run";
import { defineWorkflow, Schema } from "../src/workflow";

const [mode = "run", agentArg = "opencode", modelArg, afterArg] = process.argv.slice(2);
const agent = agentArg as AcpAgentKind;
if (!(agent in ACP_AGENTS)) throw new Error(`unknown agent ${agentArg}`);
if (!ACP_AGENT_KINDS.includes(agent)) throw new Error(`unknown agent ${agentArg}`);
const hostSettings: HostSettings = afterArg === "include" ? "include" : "ignore";
const model = mode === "bad-model" ? "no-such-model" : modelArg;
if (model === undefined) throw new Error("model required");

Expand All @@ -46,6 +55,21 @@ async function scratchRepo(): Promise<string> {
"export function add(a: number, b: number): number {\n return a + b;\n}\n",
);
await writeFile(join(dir, "README.md"), "# math\n\nTiny arithmetic helpers.\n");
if (mode === "isolation") {
// Project-level settings, which a run must keep seeing.
await writeFile(join(dir, "CLAUDE.md"), "Project codeword: PROJ-CLAUDE-7731\n");
await writeFile(join(dir, "AGENTS.md"), "Project codeword: PROJ-AGENTS-4410\n");
for (const [root, name] of [
[".claude", "project-claude-probe"],
[".agents", "project-agents-probe"],
] as const) {
await mkdir(join(dir, root, "skills", name), { recursive: true });
await writeFile(
join(dir, root, "skills", name, "SKILL.md"),
`---\nname: ${name}\ndescription: Probe skill; use when asked about probes.\n---\n\nProbe.\n`,
);
}
}
await sh(dir, "git", "init", "-q", "-b", "main");
await sh(dir, "git", "add", ".");
await sh(
Expand Down Expand Up @@ -104,6 +128,40 @@ Make the change. When you are done, stop — do not run git commands and do not
},
});

const Visible = Schema.Struct({
skills: Schema.Array(Schema.String),
codewords: Schema.Array(Schema.String),
xdgConfigHome: Schema.String,
ghAuth: Schema.String,
gitUser: Schema.String,
claudeVars: Schema.String,
parentSentinels: Schema.String,
});

const isolation = defineWorkflow("acp-isolation", {
input: Schema.Struct({ long: Schema.Boolean }),
output: Visible,
agent: { model },
run: async (ctx) => {
const seen = await ctx.agent(
"visible",
`Report what you can see. Do not guess and do not invent anything.

1. skills: the name of every skill available to you, exactly as your skill tool or skill listing names them (all of them).
2. codewords: every "codeword" value that appears in instructions you were given (CLAUDE.md, AGENTS.md or similar), verbatim. Do not search files for them; only what is already in your context.
3. Run these shell commands, each separately, and report their output trimmed:
- xdgConfigHome: \`echo "XDG=$XDG_CONFIG_HOME"\`
- ghAuth: \`gh auth status 2>&1 | head -2\`
- gitUser: \`git config --global --list --show-origin | head -1\`
- claudeVars: \`env | grep -o '^CLAUDE[A-Z_]*' | sort | tr '\\n' ' '\`
- parentSentinels: \`env | grep -c factory-parent-sentinel || true\``,
{ output: Visible },
);
if (seen.output === undefined) throw new Error("no structured output");
return seen.output;
},
});

const stamp = new Date().toISOString().replace(/[:.]/g, "-");
const out = join(import.meta.dir, "out", `${mode}-${agent}-${stamp}`);
await mkdir(out, { recursive: true });
Expand All @@ -112,19 +170,28 @@ const diagnostics: Array<AcpDiagnostic & { step: number }> = [];
const pids: number[] = [];
let step = -1;

const adapter = acpAdapter(ACP_AGENTS[agent], (event) => {
if (event.kind === "spawned") {
step += 1;
pids.push(event.pid);
}
diagnostics.push({ ...event, step });
const definition = acpAgent(agent, { hostSettings });
const leaked = Object.keys(agentEnv(definition)).filter(
(k) => k === "CLAUDECODE" || k.startsWith("CLAUDE_"),
);
console.log(
`parent CLAUDE* vars: ${Object.keys(process.env).filter((k) => k.startsWith("CLAUDE")).length}, passed to the agent: [${leaked.join(", ")}]`,
);
const adapter = acpAdapter(definition, {
onDiagnostic: (event) => {
if (event.kind === "spawned") {
step += 1;
pids.push(event.pid);
}
diagnostics.push({ ...event, step });
},
});

const dir = await scratchRepo();
console.log(`${mode} ${agent} model=${model} dir=${dir}`);
const runtime = makeAgentRuntime(adapter);
const t0 = Date.now();
const handle = startRun(hello, runtime, {
const handle = startRun(mode === "isolation" ? isolation : hello, runtime, {
runId: `spike-${stamp}`,
dir,
input: { long: mode === "cancel" },
Expand Down
Loading
Loading