[antonwolfy]

Summary

Found while evaluating IntelPython/dpnp#3099 (AGENTS.md guidance) with dokimi: repo scan
marks a scan incomplete but its text output never says so, and --fail-on-fail then exits 1
with no message. dpnp at d8fb283, with a guidance-only manifest modelled on the one in docs/reviewing.md
(globs: ["**/AGENTS.md", "**/CLAUDE.md", ".github/copilot-instructions.md", ".github/instructions/*.md"],
also_scan: ["CONTRIBUTING.md"]; dpnp has no CONTRIBUTING.md), gives on main:

dpnp: 8 guidance files in .../dpnp-d8fb283; adapters: cmake, pytest, pre-commit (detected)
  claims: 252 (ok 242, unresolved 9, fail 1)
  warning: guidance glob matches no file: **/CLAUDE.md
  warning: guidance glob matches no file: .github/copilot-instructions.md
  warning: guidance glob matches no file: .github/instructions/*.md
  warning: guidance/also_scan matches no file: CONTRIBUTING.md
  FAIL dpnp/tests/AGENTS.md:19 [identifier] TestXxx -- `TestXxx` not found in tracked non-doc files

and "incomplete": true in the JSON. The one warning that makes the scan incomplete, the
missing also_scan file, prints exactly like the three harmless glob warnings, and the
person reading it took the glob warnings for the cause. With no failing claim, --fail-on-fail
prints fail 0 and exits 1 with nothing on stderr. The example manifest in
docs/reviewing.md sets this up: copied into a repository without a CONTRIBUTING.md, every scan
is incomplete and the gate can never pass. The rule itself is right and documented (contract
§8: "a file the gate must see belongs in also_scan"); its visibility and the example were not.

The same scan now prints, after the warnings:

  incomplete: guidance/also_scan matches no file: CONTRIBUTING.md (repo scan --fail-on-fail exits 1)

and --fail-on-fail exits 1 with ✗ scan incomplete: guidance/also_scan matches no file: CONTRIBUTING.md
on stderr.

Changes

• repos/l0/__init__.py: _coverage_warnings returns the warnings that make the scan
  incomplete (a missing also_scan file, no guidance file) next to all warnings, and
  scan_repo adds the unreadable guidance files and Bazel build inputs to them. The payload
  gains incomplete_reasons: the same strings as their warnings entries, in the same order,
  [] for a complete scan. incomplete is now bool(incomplete_reasons), the same causes as
  before; warnings is unchanged.
• schemas/repo_readiness.v1.json: incomplete_reasons as an optional array of strings. A
  widening, so the version stays v1 and is not in required: reports already written stay valid.
• commands/repo.py (cmd_repo_scan): the text output prints
  incomplete: <reason>; <reason> (repo scan --fail-on-fail exits 1) after the warnings when the
  scan is incomplete, and nothing new otherwise. The gate moved into _scan_gate (which keeps
  cmd_repo_scan at its complexity baseline): when it exits 1 on an incomplete scan it prints
  ✗ scan incomplete: <reasons> on stderr, in --json mode too. A failed claim on a complete
  scan exits 1 as before, with nothing on stderr: its FAIL lines say why. Exit codes unchanged.
• docs/reviewing.md: the guidance-only example no longer lists also_scan; a paragraph after it
  says what also_scan is for and that a missing file makes the scan incomplete and
  repo scan --fail-on-fail exit 1.
• docs/repo-evaluation-contract.md §8: incomplete_reasons, the incomplete: ... line and the
  gate's ✗ scan incomplete: ... on stderr.
• Tests (tests/test_repo_l0.py): a missing also_scan file is named in incomplete_reasons, in
  the text output's incomplete: line after the warnings, and on stderr when --fail-on-fail
  exits 1; a glob that matches nothing alone gives no incomplete: line, [] and exit 0 under
  --fail-on-fail; both go through --out, so the report validates against the schema. The
  no-guidance case of test_fail_on_fail_follows_the_result_not_the_flag now checks the stderr
  reason, and test_scan_command_writes_a_valid_artifact that a gate failing only on claims
  prints nothing on stderr. test_repo_l0_cases.py and test_repo_l0_parity.py check
  incomplete_reasons for an unreadable guidance file and unreadable Bazel inputs.
• Changelog fragment (Fixed).

Risk

The text output of an incomplete scan gains one line, and --fail-on-fail on an incomplete
scan writes one line to stderr. A complete scan prints what it did before. The JSON and the
--out report gain incomplete_reasons; a consumer that rejects unknown keys of
repo_readiness.v1 would see it, and reports written before still validate. Exit codes are
unchanged.

Testing

On a Linux review host, at 4fa7783 (on main e966422):

• make ci-local: 27 passed, 1 failed, 7 skipped (external-catalog-smoke: needs Docker and
  network; test-latest, docs.yml:deploy, pip-audit.yml:report: job condition false on a PR;
  cli-surface "Package the result": unknown expression; conda-forge-deps: no micromamba;
  semgrep secrets: not installed, runs on the PR). 10188 tests passed, 4 skipped; changed-line
  coverage 100%. The failure is pip-audit: 10 advisories against filelock 3.32.7 (fix
  4.0.9 / 4.2.1), which main's requirements.lock.txt and requirements-ci.lock.txt pin.
  This PR changes no dependency or lock file; ci(deps): document and ignore ten more filelock 3.32.7 advisories #288 documents and ignores those advisories.

• tests/test_repo_l0.py, test_repo_l0_cases.py, test_repo_l0_parity.py, with
  test_docs_code_refs.py, test_growth_ratchet.py, test_audit_test_quality.py and
  test_import_architecture.py: 569 passed; ruff check on the changed files and mypy on the
  package: clean.

• The new and changed tests on main's code: 6 fail (the two new tests, the no-guidance case of
  test_fail_on_fail_follows_the_result_not_the_flag, both cases of
  test_dangling_guidance_symlink_reported, test_unreadable_build_inputs_warn_and_mark_the_scan_incomplete).

• repo scan of this repository with a manifest whose also_scan names a missing file:
  claims: 80 (ok 69, unresolved 11, fail 0), the incomplete: guidance/also_scan matches no file: ... line, ✗ scan incomplete: ... on stderr, exit 1.

• Mutants, each run against the three scan test files; every one fails a test:

  Change	Fails
  the text incomplete: line printed for a complete scan instead	test_a_glob_that_matches_nothing_alone_keeps_the_scan_complete, test_a_missing_also_scan_file_is_named_as_the_cause
  the text incomplete: line dropped	test_a_missing_also_scan_file_is_named_as_the_cause
  the gate's stderr reason dropped	test_a_missing_also_scan_file_is_named_as_the_cause, test_fail_on_fail_follows_the_result_not_the_flag[no guidance]
  the stderr reason printed for a complete scan too	test_scan_command_writes_a_valid_artifact
  the gate ignores incomplete	test_a_missing_also_scan_file_is_named_as_the_cause, test_fail_on_fail_…[no guidance], test_cli_exit_codes
  _coverage_warnings returns no reasons	test_a_missing_also_scan_file_is_named_as_the_cause, test_fail_on_fail_…[no guidance], test_cli_exit_codes
  a glob with no match counted as a reason	test_a_glob_that_matches_nothing_alone_keeps_the_scan_complete, test_a_missing_also_scan_file_is_named_as_the_cause, test_fail_on_fail_…[no guidance]
  unreadable guidance and build files left out of the reasons	17 tests of test_repo_l0_cases.py and test_repo_l0_parity.py

• Merged together on main f0f95a2 with the other PRs from this evaluation (fix(repo): scan says why it is incomplete #295 at 4fa7783, feat(repo): compare records the start context, filters tasks, and flags inconclusive rows #296 at cad52dd, feat(repo): replay grades record each credited hit; repo report --hits lists them #297 at 0a7539a, fix(novelty): a capped run says how far the cap reached #299 at fa453d8), on a workstation: the full suite has 10482 passed and 57 skipped, and the same 5 failures and 1 collection error as main alone there (10456 passed). All six come from that machine's setup: no docker, no hypothesis, and two tests that depend on its git and shell settings.

• Review round at 64fa41a (on main e966422), tests only. test_unreadable_build_inputs_warn_and_mark_the_scan_incomplete (tests/test_repo_l0_parity.py) now adds a guidance glob that matches no file. It asserts that the glob's warning is in warnings and that incomplete_reasons is the two literal build file not scanned: … entries, in order. test_a_missing_also_scan_file_is_named_as_the_cause (tests/test_repo_l0.py) now asserts that stderr is empty for a scan without --fail-on-fail. Mutants, all three caught by the new tests:

  • incomplete_reasons set to a copy of every warning: survived at 4fa7783.
  • The reasons reversed: already caught at 4fa7783.
  • ✗ scan incomplete: … printed to stderr without the flag: survived at 4fa7783.

  make ci-local: 24 passed, 4 failed, 7 skipped. 10187 tests passed, 4 skipped, 1 failed; changed-line coverage 100%. None of the four failures comes from this change:

  • gitleaks download: HTTP 504, so the Scan commits step after it had no binary (exit 127).
  • pip-audit: the same 10 filelock 3.32.7 advisories that main's lock files pin.
  • One test failed: the wall-clock check test_reference_checks.py::test_the_regex_limit_is_shared_by_every_pattern_of_one_check took 1.70 s against a 1.5 s limit. It passed on a rerun of its file (62 passed), and this branch does not touch that test or its code.

• Not run: the dpnp scan itself at this head (the output above is from the evaluation at main).

Revert

A plain revert. Reports written meanwhile keep an incomplete_reasons key that the reverted
repo_readiness.v1 schema (closed, additionalProperties: false) would reject on load; drop the
key or rewrite them.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[antonwolfy]

@exai-bot /review

[exai-bot]

Note

AI-generated review of 4fa7783984bf, requested by @antonwolfy, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
CI on this commit, at review start and at posting: 15 passed, 4 running, 2 skipped.
Model us.anthropic.claude-sonnet-5-5, effort default, 5 of 155 turns, 0.3 of 120 minutes, 124k in / 2k out tokens.

The change adds incomplete_reasons to the scan payload and schema. The text output now prints an incomplete: ... line, and --fail-on-fail prints ✗ scan incomplete: ... on stderr when it exits 1 on an incomplete scan. The reasons are built from the same strings as the matching warnings entries, in the same order, and incomplete is bool(reasons), so it has the same causes as before. Exit codes are unchanged, the schema change only widens v1, and the new tests cover the stderr line, the text line, the JSON field and the glob-only case. I found no defects.

No issues found.

Repository checks

Each ran as the head of this pull request defines it.

Check	From	Result	Note
python -m pytest tests/test_repo_l0.py tests/test_repo_l0_cases.py tests/test_repo_l0_parity.py tests/test_docs_code_refs.py -q -x	AGENTS.md	not run	pytest is not installed in the container and there is no network to install it
ruff check dokimi/commands/repo.py dokimi/repos/l0/__init__.py tests/test_repo_l0.py	.pre-commit-config.yaml	not run	ruff is not installed in the container

Not verified

I did not run the tests or linters, because the container has no pytest or ruff and no network. I relied on reading the diff. The PR description reports make ci-local and the targeted tests as passing.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.47%. Comparing base (e966422) to head (64fa41a).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #295      +/-   ##
==========================================
+ Coverage   93.28%   93.47%   +0.19%     
==========================================
  Files         272      272              
  Lines       35119    35370     +251     
  Branches     5964     6003      +39     
==========================================
+ Hits        32761    33063     +302     
+ Misses       1544     1492      -52     
- Partials      814      815       +1     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/repo.py	96.56% <100.00%> (+0.15%)	⬆️
dokimi/dokimi/repos/l0/__init__.py	100.00% <100.00%> (ø)

... and 23 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[vchamarthi]

✅ OpenCodeReview: Review complete: 0 finding(s) across 9 selected item(s).

Review details

• Commit reviewed: 64fa41a against merge base e966422
• Coverage: 9 files in 3 groups
• Engine: OpenCodeReview v1.13.3, us.anthropic.claude-sonnet-5-5
• Usage: 90k tokens (50k from cache), 21s
• Tool calls: 5 (2 code_search, 2 file_read_diff, 1 file_read)
• Scan-incomplete reasons: implementation and schema: 3 files
• Tests for L0 scan-incomplete reasons: 3 files
• Docs and changelog for scan-incomplete reasons: 3 files

[antonwolfy]

Review round in 64fa41a, tests only. Both OpenCodeReview points held: a mutant survived each of them at 4fa7783, and both are caught now.

• incomplete_reasons is now asserted as literal strings, next to a glob warning that must stay out of it.
• A scan without --fail-on-fail must leave stderr empty.

The reordering half of the first point was already caught; the reply in that thread says how.

[vlad-perevezentsev]

Review at 64fa41a. The change is right: incomplete now comes from the same list as incomplete_reasons, so it has exactly the causes it had before, and the reasons are the matching warnings strings in their order. One small gap with an invalid --static is in the line comment.

What I ran, at 64fa41a:

• make ci-local --offline: 17 checks passed, 0 failed, 13 skipped (network-only); tests 10,135 passed, 57 skipped.
• repo scan of a tree without CONTRIBUTING.md in also_scan, with an invalid --static and --fail-on-fail --json: exit 1, stderr has only the --static errors; incomplete_reasons in the JSON is right.

Checked, no issues

• Same gate as before: the warnings keep the old order (globs, also_scan, no guidance, unreadable guidance, unreadable build inputs), and only the unmatched-glob warnings stay out of incomplete_reasons.
• Schema: incomplete_reasons is optional in repo_readiness.v1, so older reports still validate; schema_compat passes.
• Tests: the new assertions read incomplete_reasons and the stderr line, which do not exist on the base, so they fail there; the two OpenCodeReview points are fixed in 64fa41a as the replies say.
• Docs and changelog: the contract doc, reviewing.md and the Fixed fragment match the code; dropping also_scan: ["CONTRIBUTING.md"] from the guidance-only example removes a manifest that made scans of repos without that file incomplete.

[vlad-perevezentsev]

When the --static document is invalid, an incomplete scan still exits 1 without the ✗ scan incomplete: ... line.

sys.exit(1) for static_errors runs before _scan_gate, so the new stderr line is never printed in that case. Repro on a tree with AGENTS.md, also_scan: ["CONTRIBUTING.md"] and --static bad.json --fail-on-fail --json: exit 1, stderr has only the two ✗ --static output not merged: ... lines, and the missing CONTRIBUTING.md is only in the stdout JSON. The contract doc says that when --fail-on-fail exits 1 on an incomplete scan it prints the reasons on stderr.

Suggestion: print the gate message before the static exit, for example:

    if args.fail_on_fail and payload["incomplete"]:
        print(f"✗ scan incomplete: {'; '.join(payload['incomplete_reasons'])}", file=sys.stderr)
    if static_errors or (args.fail_on_fail and payload["counts"]["fail"]) ...

or pass static_errors into _scan_gate and exit there, plus a test with an invalid --static on an incomplete scan.