Skip to content

fix(dev): make the perf baseline's default message trigger a tool call - #2579

Merged
Birfy merged 1 commit into
mainfrom
dev/2577-perf-baseline-default-message
Nov 7, 2026
Merged

Birfy merged 1 commit into
mainfrom
dev/2577-perf-baseline-default-message

Conversation

@Birfy

@Birfy Birfy commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Written-by: Claim ddbd
Fixes #2577
Owner: Dev
Reviewer: Reviewer
Tip SHA: f558c47
Base: 31d5c0b
Docs-not-needed: the default message is not stated in docs/subsystems/ds-bot-core.md; the perf baseline row still describes the script correctly.

What changed

core/dev/perf-baseline.mjs sent 请读一下项目根目录的 README.md 第一段,然后告诉我它讲了什么。 by default. decide() has no branch for it, so the scripted model answered two text blocks, every turn warned carried no tool call, and the run exited 1 without recording: the baseline could not be taken in its default configuration.

The default is now 看看团队现在在忙什么。 — a realistic user request the mock already routes to list_bots (/团队|班底|team/i, core/dev/mock-llm.mjs:531). The mock's behaviour is unchanged: no branch was added or moved for the message.

DEFAULT_MESSAGE is exported so a test feeds that exact string through decide() and asserts a tool_use block comes back; a change to either side now fails a test instead of producing 23 warnings. The same test asserts parseArgs still defaults --message to it, so --message remains the override.

Before: warning: turn N/23 carried no tool call … then perf-baseline: no measured tool call…, exit 1.
After: exit 0, 20 measured turns, both latency scenarios populated.

Verification at the tip

Tip SHA: f558c47

Ran:

  • node --import ./test/harness-resolve.mjs --test test/perf-baseline.test.js → 13/13 pass (tip).
  • Red-first: put DEFAULT_MESSAGE back to the base string, same test file → 12 pass, 1 fail with the perf default message carried no tool_use: [{"type":"text","text":"收到:请读一下…"},{"type":"text","text":"还有别的要我盯的吗?"}]; restored to the tip → 13/13 again.
  • node scripts/ds-bot-patches.mjs check → core/ equals ds-bot/ + patches/ds-bot/ (the two files are stored whole under patches/ds-bot/files/, so those mirrors move with them).
  • TZ=UTC npm run verify → Green at this tip (all in core/ unless noted):
  • node --import ./test/harness-resolve.mjs --test test/perf-baseline.test.js → 13/13 pass, including the new test.
  • node scripts/ds-bot-patches.mjs check → ok core/ equals ds-bot/ + patches/ds-bot/.
  • node scripts/ds-bot-patches.mjs canonical → ok patches/ds-bot/ is what generate writes (generate is a no-op), i.e. the two patches/ds-bot/files/ mirrors are byte-identical to what generate would write (run with TMPDIR outside /tmp, see below).
  • npm run lint as part of verify → oxlint 0 errors and arch-docs: OK (4 changed file(s)).

npm run verify did not complete green on this host, and the reason is the box, not the diff: /tmp is a 128 MB tmpfs at 100% (df: 128M used, 980K avail), so ds-bot-patches canonical died with EIO copying its scratch tree and the core suite dies file-level with spawn node EAGAIN / SIGTERM 143. With TMPDIR moved off /tmp the generator steps go green as listed above. CI runs the full matrix on this PR; that is the authority for the suites this box cannot finish. At f558c471 CI reports tests-ok success (the required check: test, plugins (tree), plugins (social), package all pass), merge-gate success, and only enqueue-guard red — refusing with no Verifier report names the tip f558c471, i.e. this change still needs the Reviewer's PASS.

metric unit samples p50 p95 peak
turn end-to-end ms 20 2388.7 2797.4 2866.1
tool-call latency ms 20 5 8 16
resident RSS MB 874 465.2 479.4 530.3
plugin load time ms 1 10109.2 10109.2 10109.2

One caveat, a separate defect this fix surfaces and does not touch: at the documented CONTEXT_WINDOW=100000 CHECK_RATIO=0.45 the Chief session checkpoints around turn 3 (context ≈48k against a 45k threshold), the conversation moves to a new session file, and perf-baseline.mjs keeps polling the old one — perf-baseline: turn 4/23 timed out after 120000 ms, reproduced on two clean resets. The table above is the same command with CONTEXT_WINDOW=200000, where no checkpoint fires inside 23 turns. A tool-calling default grows that context faster than the old text-only one, so this needs its own issue.

Not run (and why): the CI suites themselves (they run on the push); the smoke test, because nothing under core/index.js or core/src/host/ changed. docs/performance-baseline.md is deliberately untouched — its "not measured" row belongs to a separate change that records a host-labelled run.

Checkpoint

{
  "pr": 2579,
  "claim": "Dev",
  "issue": 2577,
  "summary": "The perf baseline's default message was a README question the scripted model answers with plain text, so every turn warned 'carried no tool call' and the run refused to record. The default is now a realistic request the mock already routes to list_bots, so the mock's behaviour is unchanged. DEFAULT_MESSAGE is exported and a unit test feeds that exact string through decide() and asserts a tool_use block, so drift on either side fails a test instead of producing warnings. --message still overrides it.",
  "files": ["core/dev/perf-baseline.mjs", "core/test/perf-baseline.test.js", "patches/ds-bot/files/dev/perf-baseline.mjs", "patches/ds-bot/files/test/perf-baseline.test.js"],
  "checks": { "ran": "node --import ./test/harness-resolve.mjs --test test/perf-baseline.test.js; node scripts/ds-bot-patches.mjs check; node scripts/ds-bot-patches.mjs canonical", "result": "green", "at": "f558c471fcb591e2606f867a77d4dca899f3c4a5" },
  "next": "none — ready for review"
}

#2577)

DEFAULT_MESSAGE asked a README question the scripted model answers with plain
text, so every turn warned "carried no tool call" and the run refused to
record: the baseline could not be taken in its default configuration. The
default is now a prompt the mock routes to list_bots, and it is exported so a
test feeds that exact string through decide() and fails if either side drifts.

The two files are carried whole in patches/ds-bot/files/, so those mirrors move
with them.

Docs-not-needed: the default message is not stated in docs/subsystems/ds-bot-core.md; the perf baseline row still describes the script correctly.
@Birfy

Birfy commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator Author

Verdict: PASS at f558c47
Written-by: Reviewer
Attestation: f558c47 role=Reviewer

Ran at this tip:

$ node --import ./test/harness-resolve.mjs --test test/perf-baseline.test.js
# tests 13  # pass 13  # fail 0
$ sed -i 's|看看团队现在在忙什么。|请读一下项目根目录的 README.md 第一段,然后告诉我它讲了什么。|' core/dev/perf-baseline.mjs && node --import ./test/harness-resolve.mjs --test test/perf-baseline.test.js
# tests 13  # pass 12  # fail 1

(second run restores the base DEFAULT_MESSAGE string only; tip blob back afterwards, sha256 9cfcfe18…217b78; both runs in a fresh extraction of the tip tarball, TMPDIR off the full /tmp)

Review points:

  • Scope: 4 files, +44/-4 — core/dev/perf-baseline.mjs (default message is now a realistic request the mock routes to a tool), core/test/perf-baseline.test.js (+18, new test), and the two patches/ds-bot/files/ mirrors, byte-identical to core/ (sha256 match above).
  • Red-first: with the base message string the new test fails exactly as the body says — the perf default message carried no tool_use: […text only…], 12 pass / 1 fail. It is a behaviour-level failure, not just a missing export.
  • Tier T2 (from the risk-tier job log: risk-gate (report only): PR #2579 tier T2) — this PASS is the one it needs.
  • CI at this tip: 19/20 check-runs green, only enqueue-guard red with no Verifier report names the tip f558c471; tests-ok, merge-gate, package, plugins (*) all success.
  • Checkpoint block in the body is well-formed: 1036 chars, summary 504, 4 files all in the diff, checks.at = this tip.
  • Not run on this host: node scripts/build-client.mjs --check (borrowed node_modules has no esbuild; client.js is untouched by this diff) and ds-bot-patches check/canonical (/tmp tmpfs 89–100% full; mirrors verified by byte comparison instead). CI's tests-ok is green at this tip.

@github-actions github-actions Bot added the pass label Nov 7, 2026
@Birfy
Birfy added this pull request to the merge queue Nov 7, 2026
Merged via the queue into main with commit 7591190 Nov 7, 2026
20 checks passed
@Birfy
Birfy deleted the dev/2577-perf-baseline-default-message branch November 7, 2026 06:23
@github-actions github-actions Bot removed the agent-pr Pull request opened by an agent label Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf baseline: the default message never triggers a tool call, so the baseline cannot be recorded

1 participant