Repository navigation
Add a measurable performance baseline on the mock bench - #2570
Conversation
Record turn end-to-end, tool-call latency, resident RSS and plugin load time from the bench, so a later slowdown has a number to be compared with. Metrics read the same Session events the activity feed folds; the script starts nothing, prints its table and writes --out JSON. No thresholds: this is a baseline, not a gate. Refs #356
The linter runs over core/dev, and the script no longer reads the directory itself.
The package job holds core/ to ds-bot/ plus patches, so the two new files also live verbatim under patches/ds-bot/files/. `node scripts/ds-bot-patches.mjs check` passes.
|
Verdict: REQUEST_CHANGES at abea5f7 Ran at this tip: Risk tier from the base Blocking — acceptance 3 is not met
The baseline table is what issue #356 exists to produce; a script without numbers closes the issue with no baseline. Two ways forward, either is a fix:
Second defect (small, same pass)
Ready otherwise11/11 tests pass, red-first confirmed (source removed → |
A zero-filled commit made two runs look comparable when neither named the code that was measured. PERF_COMMIT now has to be a real sha on a checkout without .git, the test pins the refusal (red before the fix: 11 pass 1 fail), and the doc names Node 24 as the spec version.
|
Written-by: Dev Addressed the second defect and the Node line; the blocking one is an environment fact, not something I can code around.
Blocking, unchanged: acceptance 3 needs one real run, and this host cannot produce it — no What would unblock it, either way:
New tip |
|
Verdict: REQUEST_CHANGES at c55205c Ran at this tip: Fixed and verified in this tip
Blocking — acceptance 3, and the bench is not bootable here
I checked the blocker rather than taking it: the only The "no pnpm" part of the note is not the blocker — pnpm works here (
This new tip needs its own conclusion; the REQUEST_CHANGES naming |
|
Written-by: Chief Chief decision on the Reviewer's one blocker (acceptance 3), recorded also on #356: The measured run is deferred for this pass. The harness build is impossible on this host — What this PR must do now, in one commit on this same branch:
|
docs/performance-baseline.md now names the real blocker: this host cannot build the deepseek-harness checkout (no git for its version step, no C compiler for the native/system addon), so dev/bench.sh cannot boot a bench. The attempt is tracked in #2571 and the four rows stay unmeasured rather than invented. Fixes #356
|
Written-by: Rework Rework of the single blocker the Reviewer named, per the Chief decision in the previous comment: one doc-only commit on this branch, nothing else touched.
The old REQUEST_CHANGES names |
|
Verdict: PASS at 855b5e7 Ran at this tip: Red first — the added source The narrower red for the commit-naming fix is the one recorded at The one blocker is resolved by a recorded decision
Scope and gates
Carried note, not a blocker
|
Written-by: Dev
Fixes #356
Owner: Dev
Reviewer: hsliuustc0106
Tip SHA: 855b5e7
Base: 5d9e301
Docs: docs/subsystems/ds-bot-core.md updated (core/dev/ is an owned path)
What
core/dev/perf-baseline.mjsmeasures the four metrics fixed in the spec of record(comment 6313125530) on the
mock bench: turn end-to-end p50/p95 (message POST to that turn's
turn/end), tool-call latencyp50/p95 (
tool/calltotool/result), resident RSS peak of the web process, and plugin load time(process start to the
dsh web: httpline). Events are read throughdev/turn-split.mjs, thedecoder the dev tools already use, so a number means the same events the activity feed folds.
How
3 warm-up + 20 measured turns, each one message that the scripted model answers with one tool call.
Percentiles are nearest rank, so every reported number is a real sample. Prints one markdown table
and writes
--outJSON{ machine, commit, node, date, scenarios: [{ metric, unit, samples, p50, p95, peak }] }.Exits non-zero when the bench is down, a turn times out, or no turn produced a tool call; starts no
process of its own.
Docs
docs/performance-baseline.md: one-sentence definition per metric, the exact run command, theoutput shape, and the four numbers to record on each daily tag (#351). No thresholds — a baseline,
not a gate. The manual items (scroll feel, background tab, 500+ message session) are named as not
measured by the script rather than claimed.
Not done here
deepseek-harnesscheckout it would need: there is nogitfor the checkout's version step and no C compiler for thenative/systemaddon, sodev/bench.shcannot boot a bench. The Chief decision that the in-container mock baseline is deferred is recorded on this pull request and on Performance baseline: measure current smoothness (help wanted) #356; the blocker is tracked in bench: the mock bench cannot run on this host — the only deepseek-harness checkout needs a build, and the host has neither git nor a C compiler #2571 with both unlock options.docs/performance-baseline.mdnames that reason next to the table, whose four rows stay—rather than inventing numbers.npm run verify(needscore/dev/harnessinstalled withnpm run harness), the bench,and the smoke test.
Ran at the tip
cd core && node --test test/perf-baseline.test.jsat855b5e70916ee34f6316458c308019f288658d8f(doc-only commit; the script and test blobs are unchanged fromc55205cbc1910152f4dcb18089f8a8d226374f50) — 12 tests, 12 pass; the commit-naming test is red without the fix (11 pass, 1 fail,Missing expected exception).node scripts/ds-bot-patches.mjs check— 10 patches apply,core/equalsds-bot/+patches/.node .github/scripts/testing-layers.mjs --check,check-mojibake.mjs --check,check-pack.mjs,changelog.mjs check— green.Checkpoint
{
"pr": 2570,
"claim": "Dev",
"issue": 356,
"summary": "Adds core/dev/perf-baseline.mjs, which measures four baseline numbers on the mock bench: turn end-to-end p50/p95 (message POST to turn/end), tool-call latency p50/p95, resident RSS peak of the web process, and plugin load time. It reads the same Session events the activity feed folds, prints a markdown table and writes the --out JSON. docs/performance-baseline.md defines each metric, the exact command and the metrics to record per daily tag; core/test/perf-baseline.test.js pins nearest-rank percentiles, the JSON shape and the table row. No thresholds: a baseline, not a gate.",
"files": ["core/dev/perf-baseline.mjs", "core/test/perf-baseline.test.js", "docs/performance-baseline.md", "docs/subsystems/ds-bot-core.md", "patches/ds-bot/files/dev/perf-baseline.mjs", "patches/ds-bot/files/test/perf-baseline.test.js"],
"checks": { "ran": "cd core && node --test test/perf-baseline.test.js (12 pass); node scripts/ds-bot-patches.mjs check", "result": "green", "at": "855b5e70916ee34f6316458c308019f288658d8f" },
"next": "Chief deferred the measured run for this pass (recorded on this PR and #356; blocker tracked in #2571). Docs now record that reason; the doc-only commit needs a fresh PASS at this tip, then enqueue. Safe to rerun the bench on any host that can build the checkout and replace the table."
}