Repository navigation
fix(dev): make the perf baseline's default message trigger a tool call - #2579
Merged
Merged
Conversation
#2577) DEFAULT_MESSAGE asked a README question the scripted model answers with plain text, so every turn warned "carried no tool call" and the run refused to record: the baseline could not be taken in its default configuration. The default is now a prompt the mock routes to list_bots, and it is exported so a test feeds that exact string through decide() and fails if either side drifts. The two files are carried whole in patches/ds-bot/files/, so those mirrors move with them. Docs-not-needed: the default message is not stated in docs/subsystems/ds-bot-core.md; the perf baseline row still describes the script correctly.
Collaborator
Author
|
Verdict: PASS at f558c47 Ran at this tip: (second run restores the base Review points:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Written-by: Claim ddbd
Fixes #2577
Owner: Dev
Reviewer: Reviewer
Tip SHA: f558c47
Base: 31d5c0b
Docs-not-needed: the default message is not stated in docs/subsystems/ds-bot-core.md; the perf baseline row still describes the script correctly.
What changed
core/dev/perf-baseline.mjssent请读一下项目根目录的 README.md 第一段,然后告诉我它讲了什么。by default.decide()has no branch for it, so the scripted model answered twotextblocks, every turn warnedcarried no tool call, and the run exited 1 without recording: the baseline could not be taken in its default configuration.The default is now
看看团队现在在忙什么。— a realistic user request the mock already routes tolist_bots(/团队|班底|team/i,core/dev/mock-llm.mjs:531). The mock's behaviour is unchanged: no branch was added or moved for the message.DEFAULT_MESSAGEis exported so a test feeds that exact string throughdecide()and asserts atool_useblock comes back; a change to either side now fails a test instead of producing 23 warnings. The same test assertsparseArgsstill defaults--messageto it, so--messageremains the override.Before:
warning: turn N/23 carried no tool call …thenperf-baseline: no measured tool call…, exit 1.After: exit 0, 20 measured turns, both latency scenarios populated.
Verification at the tip
Tip SHA: f558c47
Ran:
node --import ./test/harness-resolve.mjs --test test/perf-baseline.test.js→ 13/13 pass (tip).DEFAULT_MESSAGEback to the base string, same test file → 12 pass, 1 fail withthe perf default message carried no tool_use: [{"type":"text","text":"收到:请读一下…"},{"type":"text","text":"还有别的要我盯的吗?"}]; restored to the tip → 13/13 again.node scripts/ds-bot-patches.mjs check→core/ equals ds-bot/ + patches/ds-bot/(the two files are stored whole underpatches/ds-bot/files/, so those mirrors move with them).TZ=UTC npm run verify→ Green at this tip (all incore/unless noted):node --import ./test/harness-resolve.mjs --test test/perf-baseline.test.js→ 13/13 pass, including the new test.node scripts/ds-bot-patches.mjs check→ok core/ equals ds-bot/ + patches/ds-bot/.node scripts/ds-bot-patches.mjs canonical→ok patches/ds-bot/ is what generate writes (generate is a no-op), i.e. the twopatches/ds-bot/files/mirrors are byte-identical to whatgeneratewould write (run withTMPDIRoutside/tmp, see below).npm run lintas part of verify → oxlint 0 errors andarch-docs: OK (4 changed file(s)).npm run verifydid not complete green on this host, and the reason is the box, not the diff:/tmpis a 128 MB tmpfs at 100% (df: 128M used, 980K avail), sods-bot-patches canonicaldied withEIOcopying its scratch tree and the core suite dies file-level withspawn node EAGAIN/ SIGTERM 143. WithTMPDIRmoved off/tmpthe generator steps go green as listed above. CI runs the full matrix on this PR; that is the authority for the suites this box cannot finish. Atf558c471CI reportstests-oksuccess (the required check:test,plugins (tree),plugins (social),packageall pass),merge-gatesuccess, and onlyenqueue-guardred — refusing withno Verifier report names the tip f558c471, i.e. this change still needs the Reviewer's PASS.MOCK_DELAY_MS=300, harness atruns/i2565/deepseek-harnessper bench: the mock bench cannot run on this host — the only deepseek-harness checkout needs a build, and the host has neither git nor a C compiler #2571):MOCK_DELAY_MS=300 dev/bench.sh perf 3099 8799 reset→ exit 0;PERF_COMMIT=31d5c0b468b41d1f7d8611876fbecf3347be7d8e node dev/perf-baseline.mjs perf 3099 --out /tmp/i2577-perf2.json(no--message, so the new default is what ran) → exit 0, 20 measured turns, nocarried no tool callwarning;stop→ exit 0. Machine lineLinux 5.10.0-60.18.0.50.r865_35.hce2.aarch64; Cortex-A32; 192 cores; 1511.1 GB RAM(this container), run2026-11-07T06:08:37Z, Node v22.23.3:One caveat, a separate defect this fix surfaces and does not touch: at the documented
CONTEXT_WINDOW=100000 CHECK_RATIO=0.45the Chief session checkpoints around turn 3 (context ≈48k against a 45k threshold), the conversation moves to a new session file, andperf-baseline.mjskeeps polling the old one —perf-baseline: turn 4/23 timed out after 120000 ms, reproduced on two clean resets. The table above is the same command withCONTEXT_WINDOW=200000, where no checkpoint fires inside 23 turns. A tool-calling default grows that context faster than the old text-only one, so this needs its own issue.Not run (and why): the CI suites themselves (they run on the push); the smoke test, because nothing under
core/index.jsorcore/src/host/changed.docs/performance-baseline.mdis deliberately untouched — its "not measured" row belongs to a separate change that records a host-labelled run.Checkpoint
{ "pr": 2579, "claim": "Dev", "issue": 2577, "summary": "The perf baseline's default message was a README question the scripted model answers with plain text, so every turn warned 'carried no tool call' and the run refused to record. The default is now a realistic request the mock already routes to list_bots, so the mock's behaviour is unchanged. DEFAULT_MESSAGE is exported and a unit test feeds that exact string through decide() and asserts a tool_use block, so drift on either side fails a test instead of producing warnings. --message still overrides it.", "files": ["core/dev/perf-baseline.mjs", "core/test/perf-baseline.test.js", "patches/ds-bot/files/dev/perf-baseline.mjs", "patches/ds-bot/files/test/perf-baseline.test.js"], "checks": { "ran": "node --import ./test/harness-resolve.mjs --test test/perf-baseline.test.js; node scripts/ds-bot-patches.mjs check; node scripts/ds-bot-patches.mjs canonical", "result": "green", "at": "f558c471fcb591e2606f867a77d4dca899f3c4a5" }, "next": "none — ready for review" }