[shekharprateek]

Two guardrails this repository states but does not enforce, plus the doc and comment claims that went with them.

1. Run output was committable, and four places said it wasn't

.gitignore covered only the cloned target repos (swe-benchmark-data/*/repo/) and replay.jsonl. Everything a run actually writes — metrics.json, eval.json, patch.diff, the design artifacts and multi-megabyte *-stream.jsonl — was committable by a plain git add -A after a benchmark.

Meanwhile:

File	Claim
docs/repository-structure.md:36	"Gitignored — no run output is committed"
benchmarks/docs/harness-reference.md:473	"The per-task metrics.json is gitignored"
docs/omp-setup.md:83	stream files "stay gitignored"
.claude/skills/onboard-benchmark/SKILL.md	runner.yaml is gitignored

I hit this for real earlier in this migration: a live Strands benchmark run left its artifacts sitting as untracked files that git status offered up for commit.

The tree is now ignored with the directory's README re-included — the only file tracked under it. This is the one place the rule deliberately differs from upstream: upstream re-includes metrics.json, eval.json and run-summary.json so its runs stay inspectable from git, and here those are measured results this repository doesn't publish.

benchmarks/config/runner.yaml is ignored too. It holds the endpoint and, unless api_key_env is used, a literal API key — and two docs plus the onboard skill walk every new user through creating it while stating it is gitignored.

Also ported from upstream: a swe-clone-*/ backstop for a clone_dir pointed inside the repo, the three stray repo-root writes an agent produces when handed a bare relative filename instead of its absolute artifacts_dir, and the .hftoken spelling beside the existing .hf_token.

2. CLAUDE.md did not import AGENTS.md

Claude Code reads CLAUDE.md, not AGENTS.md. So it started with a single Markdown note and never saw this repository's two process guardrails — never commit directly to main, never merge without review — along with the navigation rules and the paths not to read.

CLAUDE.md now imports AGENTS.md, which already carries all of it. Substantive rules stay in one file so the two can't drift.

3. Claims pointing the other way

harness-reference.md described "the committed run-summary.json", and three script docstrings/comments said the same. No run-summary.json is committed here and none was before this change — only the README is tracked under swe-benchmark-data — so that wording was inherited from upstream, where the rollup is committed.

Reworded to describe run-summary.json as the file a team keeps or publishes. No behaviour change: summarize_run.py still scrubs the judge's local temp path, it just no longer gives a reason that doesn't hold here.

Verification

Each case checked with git check-ignore:

benchmarks/swe-benchmark-data/.../metrics.json         IGNORED
benchmarks/swe-benchmark-data/.../strands-stream.jsonl IGNORED
benchmarks/config/runner.yaml                          IGNORED
/github-issue.md                                       IGNORED
benchmarks/swe-benchmark-data/README.md                tracked ✓

git ls-files --ignored --exclude-standard -c is empty, so no file that was tracked became ignored.

642 tests pass, ruff and ruff format clean.

Remaining from the audit

The omp fuzzy model-match guard — the last behavioural defect — then versioning.