[shekharprateek]

The bug

Three shipped files told users to run a flag that did not exist. onboard-benchmark/SKILL.md names it at two steps, and benchmarks/dataset/workshop-template.yaml at one — so the repo's guided path for a user bringing their own repository died on an argparse error at its first step, which is exactly where someone is least able to tell a broken tool from their own mistake.

This is the third defect from one partial port

aws-samples commit 6daa1bc took runner_config.py (+85 lines) and the onboard-benchmark skill from upstream PR #193 — without run-swe-headless.py or preflight_check.py:

Declaration ported	Implementation missing	Symptom	Fixed by
strands_agent_runner.py	harness wiring	agent: strands silently ran claude -p	#32
api_key_env in runner_config	all six consumers	gateway token became the literal local	#36
skill text citing --check-repos	the flag itself	argparse hard-fail at step one	this PR

Same shape three times: the declaration arrived without the implementation, so validation passed and docs read correctly while the behaviour was absent.

What --check-repos does

git ls-remote on every repo/ref pair in the dataset before any task starts. The harness clones per task, so an unreachable repo or a missing tag otherwise fails once per task, hours into a batch. It transfers no objects, so it costs seconds, and it's the usual first failure on a private or enterprise repo.

run-e2e-benchmark.sh now runs it ahead of the existing artifact-folder check and refuses to start the batch if it fails.

Ported with it

The upstream file is one piece, so three related things come along:

• _diagnose_git_error turns git's stderr into a cause — missing repo vs missing ref vs untrusted CA vs a credential prompt — so the failure names what to fix instead of printing a porcelain error.
• _assert_inside_root refuses to touch a resolved artifact directory outside the benchmarks root. It's defence-in-depth for --check today, and becomes load-bearing now that _existing also counts a failed attempt that left no artifacts, which widens what --clear will remove.
• --model is no longer required. It's needed by --check and --clear, which resolve a per-model artifact folder, and meaningless to --check-repos, which reads only the dataset. Requiring it would have forced users to invent one.

Verification

Against real endpoints, not mocks:

--dataset dataset/hello-world.yaml --check-repos
  → OK: https://github.com/octocat/Hello-World @ master
  → exit 0

(same dataset, repo swapped for a nonexistent one)
  → UNREACHABLE: ... the repository path does not exist on the server, or the credential...
  → "this would fail once per task, hours into a run"
  → exit 1

Both commands onboard-benchmark documents now run.

635 tests pass — the preflight suite goes from 12 to 27 — no new failures. ruff, ruff format, bandit (0 issues at every severity) clean; shell syntax checked. The one failure, test_judge_common.test_lowercase_preferred_on_collision, fails on main too (assumes a case-sensitive filesystem).

Checked for divergence before porting

main's version of this file had no aws-samples-specific content — the 35 removed lines are all refactors upstream made while adding the feature (docstring, _ARTIFACT_FILENAMES, task-loading extracted into _load_tasks, the --model change). Nothing lost.