diff --git a/README.md b/README.md index f4c07002d..ad8f9e048 100644 --- a/README.md +++ b/README.md @@ -243,25 +243,26 @@ show a lightweight runtime, not coding-agent capacity at that scale. The same synthetic conversation work through five harnesses, 32 agents at once, each doing three turns that add 64 KiB of text and stream back 5 KiB -([full screen](docs/HARNESS_MEASUREMENTS.md), 2026-10-28, 4-vCPU Linux VM, -medians of three runs; Agent 0.4.1, daemon source identical to main -`3a9f7df`). +([full screen](docs/HARNESS_MEASUREMENTS.md), 2026-10-30, 4-vCPU Linux VM, +medians of three runs; Agent at main `2182b4a`, the 0.7.0 code). | Harness | Peak memory | CPU time | Turn p99 | | --- | ---: | ---: | ---: | -| Agent | 29 MiB | 0.7 s | 0.61 s | -| Pi 0.85.1 | 159 MiB | 1.9 s | 0.78 s | -| Codex 0.153.1 | 246 MiB | 40.1 s | 10.2 s* | -| opencode 1.18.32 | 958 MiB | 24.7 s | 6.2 s | -| Claude Code 2.1.267 | 6,490 MiB | 42.4 s | 3.5 s* | +| Agent | 29 MiB | 0.7 s | 0.60 s | +| Pi 0.85.1 | 172 MiB | 1.8 s | 0.77 s | +| Codex 0.153.1 | 301 MiB | 35.5 s | 9.0 s* | +| opencode 1.18.32 | 991 MiB | 19.8 s | 4.7 s | +| Claude Code 2.1.267 | 7,348 MiB | 32.7 s | 2.8 s* | Read this as an exploratory screen, not a ranking. The harnesses do different amounts of work: Agent commits every turn to SQLite, opencode keeps its own store, and the others hold conversations in memory; no tools were called. Claude Code runs one process per agent, and its memory is summed across those processes. Turn time includes 0.5 s of scripted streaming. -\* Codex reached only 7 to 9 of the 32 concurrent streams on this machine, and -Claude Code 24 to 29. +\* Codex reached only 8 to 10 of the 32 concurrent streams on this machine, and +Claude Code 26 to 31. On this shared VM the other harnesses' numbers drift by +up to about 20% between screens with the same binaries, so compare harnesses +within one screen, not across screens. The [performance tools](docs/BENCHMARKS.md) reproduce these runs against the same synthetic provider, and [docs/](docs) holds the design record and diff --git a/docs/EVIDENCE.md b/docs/EVIDENCE.md index 404103609..1b70db876 100644 --- a/docs/EVIDENCE.md +++ b/docs/EVIDENCE.md @@ -12,7 +12,7 @@ the automatic approver's own cost, at `81458d2` for the judges compared, at `7a66687` for the finish cost of per-bot settings, by the change that keyed creation for its cost, and at `3ad9e2f`, the change that runs summaries beside the turn's call and adds the Pi Durable -baseline, on `b026767`. +baseline, on `b026767`, and at `2182b4a` for the five-harness rescreen. This is the one place that says what is currently known. The documents it links to keep the method, the raw tables and superseded runs. When a history document's @@ -133,11 +133,14 @@ more turns is not here, because its work changes with its speed; it is under Because every call should run, this says nothing about false allows. [Record](APPROVALS.md#measure-before-building). - **Five harnesses, same synthetic work.** 32 agents, three turns each adding - 64 KiB: Agent 28.7 MiB peak and 0.69 s CPU, Pi 159 MiB and 1.9 s, Codex - 246 MiB and 40.1 s, opencode 958 MiB and 24.7 s, Claude Code 6,490 MiB and - 42.4 s. Exploratory, not a ranking: the harnesses do unequal work, and + 64 KiB: Agent 28.9 MiB peak and 0.65 s CPU, Pi 172 MiB and 1.8 s, Codex + 301 MiB and 35.5 s, opencode 991 MiB and 19.8 s, Claude Code 7,348 MiB and + 32.7 s. Exploratory, not a ranking: the harnesses do unequal work, and Codex and Claude Code did not reach 32 concurrent streams on this host. - Linux VM, Agent 0.4.1 (main `3a9f7df`), 2026-10-28. + Agent is unchanged within host drift from the 2026-10-28 screen (28.7 MiB, + 0.69 s at `3a9f7df`); the other engines, with unchanged binaries, moved by + up to about 20% between screens. Linux VM, main `2182b4a` (the 0.7.0 + code), 2026-10-30. [Record](HARNESS_MEASUREMENTS.md). - **Pi Durable on the durable lifecycle.** 32 named conversations, three turns each, both engines syncing every commit (`synchronous=FULL`, which diff --git a/docs/HARNESS_MEASUREMENTS.md b/docs/HARNESS_MEASUREMENTS.md index e15fca745..15264700f 100644 --- a/docs/HARNESS_MEASUREMENTS.md +++ b/docs/HARNESS_MEASUREMENTS.md @@ -1,7 +1,10 @@ # Five-harness screen -Measured 2026-10-28 UTC with `bench.matrix` plus separate runs of its last -case. **This is an exploratory screen, not a ranking.** The engines differ in +Measured 2026-10-30 UTC at main `2182b4a`, after the daemon changes that go +into 0.7.0, with `bench.matrix` plus separate runs of its last two cases +(`compare` refuses an engine that misses its concurrency, which stops the +matrix; the cases were run by hand with the matrix's own settings and order). +The 2026-10-28 screen at `3a9f7df` is kept [below](#previous-screen-2026-10-28). **This is an exploratory screen, not a ranking.** The engines differ in features, storage, and deployment arrangement (listed below), so the numbers describe what each tested arrangement cost for the same synthetic conversation work. They do not isolate implementation efficiency. See the @@ -11,7 +14,8 @@ conversation work. They do not isolate implementation efficiency. See the ## Conditions - Host: a 4-vCPU Intel Xeon (2.8 GHz) Linux 6.18 cloud VM with 16 GiB RAM, - shared and not otherwise controlled. Earlier screens in these docs ran on a + shared and not otherwise controlled, the same kind as the 2026-10-28 + screen. Earlier screens in these docs ran on a 10-core macOS machine and are not comparable with this one. The 2026-09-23 screen ran on the same kind of VM, but the unchanged engines used 40 to 80% more CPU this time, so compare engines within one screen, not across them. @@ -33,7 +37,7 @@ conversation work. They do not isolate implementation efficiency. See the | Engine | Version | Arrangement | Store | | --- | --- | --- | --- | -| Agent | `agent-runtime 0.4.1`, binary `8dbe6a2f…`, daemon source identical to main `3a9f7df` | one native daemon, stdio JSONL protocol | SQLite, `synchronous=FULL` | +| Agent | main `2182b4a` (0.7.0 code; version string still `agent-runtime 0.6.0`), binary `4685d0c1…` | one native daemon, stdio JSONL protocol | SQLite, `synchronous=FULL` | | Pi | `pi-agent-core` / `pi-ai` 0.85.1 | one Node process, one `Agent` per agent | in memory | | Codex | `codex-cli 0.153.1`, binary `b9315df6…` | one native app-server plus a Node JSON-RPC adapter | ephemeral threads | | opencode | 1.18.32, binary `513f500a…` | one native `opencode serve` plus a Node HTTP adapter | SQLite, WAL, `synchronous=NORMAL` | @@ -48,67 +52,93 @@ differences are in [BENCHMARKS.md](BENCHMARKS.md). | Engine | Peak memory, MiB | CPU s | First chunk p50 | Turn p99 | Streams reached | | --- | ---: | ---: | ---: | ---: | ---: | -| Agent | 28.7 | 0.69 | 82 ms | 608 ms | 32 of 32 | -| Pi | 158.8 | 1.94 | 104 ms | 777 ms | 32 of 32 | -| Codex | 245.5 | 40.06 | 2,391 ms | 10,211 ms | 7–9 of 32 | -| opencode | 957.6 | 24.69 | 3,408 ms | 6,152 ms | 32 of 32 | -| Claude Code | 6,490.2 | 42.43 | 514 ms | 3,489 ms | 24–29 of 32 | - -Codex and Claude Code did not reach the configured concurrency at 32 agents -on this host, in either history size, so their 32-agent rows describe a -smaller overlap than the others and `compare` refuses them. Each of those -runs carries the runner's "configured concurrency was not achieved" warning; -no other run in this screen has a quality warning. In the 2026-09-23 screen -Claude Code reached 32 of 32 and Codex 10 to 18. +| Agent | 28.9 | 0.65 | 74 ms | 599 ms | 32 of 32 | +| Pi | 171.7 | 1.76 | 94 ms | 773 ms | 32 of 32 | +| Codex | 301.2 | 35.50 | 2,009 ms | 8,959 ms | 8–10 of 32 | +| opencode | 990.5 | 19.83 | 2,514 ms | 4,694 ms | 32 of 32 | +| Claude Code | 7,347.5 | 32.66 | 415 ms | 2,751 ms | 26–31 of 32 | + +Codex and Claude Code again did not reach the configured concurrency at 32 +agents, in either history size, so their 32-agent rows describe a smaller +overlap than the others and `compare` refuses them. Codex also reached only +4 to 8 of 8 streams at eight agents with 64 KiB, which stopped the matrix +there. Each of those runs carries the runner's "configured concurrency was +not achieved" warning; no other run in this screen has a quality warning. Median peak memory in MiB by case: | Agents | New text per turn | Agent | Pi | Codex | opencode | Claude Code | | --- | --- | ---: | ---: | ---: | ---: | ---: | -| 1 | 4 KiB | 20.2 | 101.8 | 158.9 | 650.0 | 270.9 | -| 1 | 64 KiB | 21.5 | 99.7 | 160.6 | 654.3 | 271.0 | -| 8 | 4 KiB | 21.2 | 108.4 | 180.0 | 724.9 | 1,708.0 | -| 8 | 64 KiB | 25.2 | 119.2 | 191.0 | 727.0 | 1,711.6 | -| 32 | 4 KiB | 23.9 | 123.8 | 215.1* | 800.4 | 6,528.5* | -| 32 | 64 KiB | 28.7 | 158.8 | 245.5* | 957.6 | 6,490.2* | +| 1 | 4 KiB | 20.7 | 117.4 | 215.7 | 611.6 | 313.8 | +| 1 | 64 KiB | 21.9 | 111.8 | 217.2 | 614.9 | 294.7 | +| 8 | 4 KiB | 21.8 | 121.0 | 237.9 | 738.1 | 1,926.6 | +| 8 | 64 KiB | 25.9 | 132.5 | 248.2* | 769.1 | 1,931.2 | +| 32 | 4 KiB | 24.5 | 137.5 | 276.6* | 843.7 | 7,327.7* | +| 32 | 64 KiB | 28.9 | 171.7 | 301.2* | 990.5 | 7,347.5* | Median observed CPU seconds by case: | Agents | New text per turn | Agent | Pi | Codex | opencode | Claude Code | | --- | --- | ---: | ---: | ---: | ---: | ---: | -| 1 | 4 KiB | 0.43 | 0.93 | 0.52 | 9.86 | 1.15 | -| 1 | 64 KiB | 0.37 | 0.96 | 0.50 | 10.27 | 1.24 | -| 8 | 4 KiB | 0.38 | 1.27 | 7.70 | 14.26 | 9.41 | -| 8 | 64 KiB | 0.41 | 1.18 | 9.30 | 13.12 | 9.32 | -| 32 | 4 KiB | 0.67 | 1.64 | 38.76* | 19.53 | 40.49* | -| 32 | 64 KiB | 0.69 | 1.94 | 40.06* | 24.69 | 42.43* | +| 1 | 4 KiB | 0.34 | 0.86 | 0.43 | 8.90 | 1.04 | +| 1 | 64 KiB | 0.28 | 0.84 | 0.48 | 9.28 | 0.97 | +| 8 | 4 KiB | 0.35 | 1.10 | 9.41 | 12.83 | 8.01 | +| 8 | 64 KiB | 0.44 | 1.05 | 8.83* | 12.95 | 7.56 | +| 32 | 4 KiB | 0.57 | 1.71 | 39.12* | 19.65 | 34.90* | +| 32 | 64 KiB | 0.65 | 1.76 | 35.50* | 19.83 | 32.66* | -\* Codex reached 11–14 (4 KiB) and 7–9 (64 KiB) of 32 concurrent streams, -Claude Code 21–32 and 24–29. +\* Codex reached 4–8 of 8 (8 agents, 64 KiB), 8–12 (4 KiB) and 8–10 (64 KiB) +of 32 concurrent streams, Claude Code 30–32 and 26–31. Every measured run completed all its turns with no invalid provider requests. Full ranges, ready times, connection counts and process counts are in the -ignored captures under `.local/bench/harness-oct28/`. +ignored captures under `.local/bench/harness-oct30-2182b4a/`. + +### Run-to-run drift on this host + +A trial of the same screen about 90 minutes earlier, at main `759c0db` +(before the last daemon changes) with the same pinned engines, measured the +top case at Agent 29.9 MiB and 0.67 s, Pi 155.4 MiB, Codex 247.2 MiB and +35.3 s, opencode 944.4 MiB and Claude Code 6,577.4 MiB. The other engines' +binaries did not change between the two runs, yet their peak memory moved by +5 to 22%, while each run's three repeats agreed within a few percent. Read +differences of that size between screens as host drift, not as a change in +any engine. ## Observations -- Agent's daemon grew 7.2 MiB from 1 to 32 agents at 64 KiB, while also +- Agent's daemon grew 7.0 MiB from 1 to 32 agents at 64 KiB, while also committing every turn to SQLite with full durability, which no other engine - here does at that level. Against the 2026-09-23 screen at `8ebbc44` (22.1 - MiB, 0.63 s) it used 6.6 MiB more at the top case, about 3 MiB of it - already present with one agent, and about 10% more CPU on a host where the - other engines needed 40 to 80% more. -- Pi, a library rather than a coding agent, stayed closest: about 100 MiB of + here does at that level. Against the 2026-10-28 screen at `3a9f7df` (28.7 + MiB, 0.69 s, 608 ms turn p99) the top case is unchanged within this host's + drift, after the 0.7.0 daemon work that touched the session queue, the + service loop's fairness between sessions, waiting requests and approvals. + The 2026-09-23 screen at `8ebbc44` measured 22.1 MiB and 0.63 s. +- Pi, a library rather than a coding agent, stayed closest: about 110 MiB of Node baseline plus 1 to 2 MiB per agent. -- opencode starts at about 650 MiB and used about 10 CPU seconds for a - one-agent run; its first turn includes a lazy start-up of about 2.5 to 3 s. +- opencode starts at about 610 MiB and used about 9 CPU seconds for a + one-agent run; it took about 2.7 s to become ready. - Claude Code's cost is dominated by its arrangement, one CLI process per - agent: about 200 MiB of summed RSS per process (shared pages counted each - time) and about 6 s to get 32 processes ready. -- Codex's CPU rose from 0.5 s at one agent to 8 to 9 s at eight on this host, - where the macOS screen in [RUST_MEASUREMENTS.md](RUST_MEASUREMENTS.md) + agent: about 230 MiB of summed RSS per agent (shared pages counted each + time) and about 5 s to get 32 processes ready. +- Codex's CPU rose from 0.5 s at one agent to about 9 s at eight on this + host, where the macOS screen in [RUST_MEASUREMENTS.md](RUST_MEASUREMENTS.md) recorded 3.2 s at 32 agents. That difference is not investigated here. +## Previous screen (2026-10-28) + +Same host kind, workload and pinned engines; Agent was `agent-runtime 0.4.1`, +binary `8dbe6a2f…`, daemon source identical to main `3a9f7df`. 32 agents, +64 KiB per turn: + +| Engine | Peak memory, MiB | CPU s | First chunk p50 | Turn p99 | Streams reached | +| --- | ---: | ---: | ---: | ---: | ---: | +| Agent | 28.7 | 0.69 | 82 ms | 608 ms | 32 of 32 | +| Pi | 158.8 | 1.94 | 104 ms | 777 ms | 32 of 32 | +| Codex | 245.5 | 40.06 | 2,391 ms | 10,211 ms | 7–9 of 32 | +| opencode | 957.6 | 24.69 | 3,408 ms | 6,152 ms | 32 of 32 | +| Claude Code | 6,490.2 | 42.43 | 514 ms | 3,489 ms | 24–29 of 32 | + ## Not measured Tools, durable resume, historical forks, compaction, cancellation, slow