docs: fix stale file references and missing registry entries - #16
Conversation
Correct documentation drift found against the current code: - adding_a_benchmark.md: built-in task sources register and set their loader/domain-hint in src/ksi/benchmarks/sources.py, not tasks/registry.py. - architecture.md: ARC runs natively (no snapshot mount, no ARC MCP server, no direct-ARC adapter); only forum tasks use the direct Anthropic adapter. - glossary.md: add the `command` evaluator and `custom` task source (both used by the quickstart). - adding_an_evaluator.md: Evaluator.evaluate returns dict[str, Any]. - custom_tasks/README.md: add the profile-creation step and Node.js prereq. - scripts/README.md: dataprep/arc_prep live under benchmarks/scripts/.
Code reviewFound 1 issue:
Lines 17 to 19 in cd16b17 Suggested: Everything else in the PR verified as correct against the code: the 🤖 Generated with Claude Code - If this code review was useful, please react with 👍. Otherwise, react with 👎. |
The paths added in cd16b17 (`scripts/dataprep/`, `scripts/arc_prep/`) resolve relative to the repo root, where no such directories exist. The real locations are `benchmarks/scripts/dataprep/` and `benchmarks/scripts/arc_prep/`, as the commit message intended and as benchmarks/docs/BENCHMARK_PREPARE.md spells out. The bare `scripts/dataprep/` form is correct in benchmarks/README.md because it is relative to benchmarks/; it does not carry over here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FhmqhjcboNTbqzZD39d3c
|
Pushed 3d2eab0 to this branch correcting the path to The rest of the PR verified clean against the code, so this should be good to merge from my side. One pre-existing item left alone deliberately, since it is outside this PR's diff: the System Map at |
Fixes documentation drift found by auditing each doc page against the current code. No code changes.
Fixes
src/ksi/benchmarks/sources.py, nottasks/registry.py.arc_load_task), no direct-ARC adapter. Only forum tasks use the direct Anthropic adapter.commandevaluator andcustomtask source (both used by the quickstart).Evaluator.evaluatereturnsdict[str, Any], matching the protocol.cp .env.haiku.template .env.haiku) and the Node.js prerequisite.dataprep/andarc_prep/live underbenchmarks/scripts/.Verification
Doc-relevant tests pass (
test_docs_contracts,test_cli_flags,test_doctor,test_custom_tasks_example).