Skip to content

docs: fix stale file references and missing registry entries - #16

Merged
xuefei-wang merged 2 commits into
mainfrom
docs/fix-doc-drift
Jul 22, 2026
Merged

xuefei-wang merged 2 commits into
mainfrom
docs/fix-doc-drift

Conversation

@lauren-h-yoon

Copy link
Copy Markdown
Contributor

Fixes documentation drift found by auditing each doc page against the current code. No code changes.

Fixes

  • adding_a_benchmark.md — built-in task sources register (and set their loader / domain hint) in src/ksi/benchmarks/sources.py, not tasks/registry.py.
  • architecture.md — ARC now runs natively: no snapshot mount, no ARC MCP server (arc_load_task), no direct-ARC adapter. Only forum tasks use the direct Anthropic adapter.
  • glossary.md — add the command evaluator and custom task source (both used by the quickstart).
  • adding_an_evaluator.md — Evaluator.evaluate returns dict[str, Any], matching the protocol.
  • custom_tasks/README.md — add the profile-creation step (cp .env.haiku.template .env.haiku) and the Node.js prerequisite.
  • scripts/README.md — dataprep/ and arc_prep/ live under benchmarks/scripts/.

Verification

Doc-relevant tests pass (test_docs_contracts, test_cli_flags, test_doctor, test_custom_tasks_example).

Correct documentation drift found against the current code:

- adding_a_benchmark.md: built-in task sources register and set their
  loader/domain-hint in src/ksi/benchmarks/sources.py, not tasks/registry.py.
- architecture.md: ARC runs natively (no snapshot mount, no ARC MCP server,
  no direct-ARC adapter); only forum tasks use the direct Anthropic adapter.
- glossary.md: add the `command` evaluator and `custom` task source (both
  used by the quickstart).
- adding_an_evaluator.md: Evaluator.evaluate returns dict[str, Any].
- custom_tasks/README.md: add the profile-creation step and Node.js prereq.
- scripts/README.md: dataprep/arc_prep live under benchmarks/scripts/.
@xuefei-wang

Copy link
Copy Markdown
Contributor

Code review

Found 1 issue:

  1. scripts/README.md now points at paths that don't exist. The new scripts/dataprep/ and scripts/arc_prep/ resolve to repo-root scripts/, which has no such subdirectories — the real locations are benchmarks/scripts/dataprep/ and benchmarks/scripts/arc_prep/. The line also contradicts its own clause ("live under benchmarks/") and the preceding sentence at line 5. The commit message states the correct target ("scripts/README.md: dataprep/arc_prep live under benchmarks/scripts/"), so the benchmarks/ prefix looks dropped by accident. benchmarks/README.md uses the bare scripts/dataprep/ form correctly because it is relative to benchmarks/; that form does not carry over to this file.

KSI/scripts/README.md

Lines 17 to 19 in cd16b17

Benchmark dataset preparation (`scripts/dataprep/`, `scripts/arc_prep/`) and the run presets
(`run_arc.sh`, `run_polyglot.sh`, `run_swebench_pro.sh`,

Suggested: benchmarks/scripts/dataprep/, benchmarks/scripts/arc_prep/.

Everything else in the PR verified as correct against the code: the src/ksi/benchmarks/sources.py / loaders.py retargeting, the dict[str, Any] evaluator return type, the custom / command registry additions, the ARC-native rewrite, and the Node.js and provider-profile prerequisites.

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

The paths added in cd16b17 (`scripts/dataprep/`, `scripts/arc_prep/`)
resolve relative to the repo root, where no such directories exist. The
real locations are `benchmarks/scripts/dataprep/` and
`benchmarks/scripts/arc_prep/`, as the commit message intended and as
benchmarks/docs/BENCHMARK_PREPARE.md spells out.

The bare `scripts/dataprep/` form is correct in benchmarks/README.md
because it is relative to benchmarks/; it does not carry over here.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019FhmqhjcboNTbqzZD39d3c
@xuefei-wang

Copy link
Copy Markdown
Contributor

Pushed 3d2eab0 to this branch correcting the path to benchmarks/scripts/dataprep/ and benchmarks/scripts/arc_prep/, which is what the original commit message intended. tests/test_docs_contracts.py passes (3 passed).

The rest of the PR verified clean against the code, so this should be good to merge from my side.

One pre-existing item left alone deliberately, since it is outside this PR's diff: the System Map at docs/architecture.md line 38 still labels memory/mcp_server.py as carrying "ARC toolsets", but that file has no ARC tool handlers. Same drift this PR fixes in the prose below it, if you want it in a follow-up.

@xuefei-wang
xuefei-wang merged commit e3182aa into main Jul 22, 2026
3 checks passed
@xuefei-wang
xuefei-wang deleted the docs/fix-doc-drift branch July 22, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants