Skip to content

ci: fail comment blocks over two lines that a PR adds or edits - #298

Merged
napetrov merged 1 commit into
intel:mainfrom
jharlow-intel:ci/comment-blocks
Oct 30, 2026
Merged

napetrov merged 1 commit into
intel:mainfrom
jharlow-intel:ci/comment-blocks

Conversation

@jharlow-intel

Copy link
Copy Markdown
Contributor

Summary

Adds a CI check for a comment rule: a comment states the constraint or the reason in at most two lines, and links an issue or PR when a reader needs the backstory. It is a new Comment blocks step in ci.yml's static job, beside the changelog-fragment step, so it needs no runner of its own and make ci-local runs it too. It reads only the lines a PR adds or changes: the 493 existing blocks over two lines pass until a PR edits one of them. AGENTS.md, REVIEW.md, the Copilot instructions and CONTRIBUTING.md state the rule.

Changes

  • scripts/check_comment_blocks.py: for each Python, shell, YAML or TOML file, Makefile or Dockerfile that origin/<base>...HEAD adds, changes or renames, finds the runs of full-line # comments and fails a run of three or more lines that holds an added or changed line. One ::error file=...,line=...,endLine=... per block, exit 1; exit 2 when git cannot diff the refs.
    • Python comments come from tokenize, so a # line inside a string or docstring is not one, and a trailing comment does not join a block.
    • Not counted, and ending a block: a shebang on line 1, the SPDX header lines, and tool pragmas (# noqa, # type:, # fmt: and the like). A ruler line (# -----) stays in its block but does not count, so a section banner is one line.
    • Renames are detected, so moving a file does not flag the comments it already had. tests/fixtures/task_formats/ is skipped, as ruff skips it (third-party copies).
  • ci.yml: the Comment blocks step, PR-only and !cancelled() like its neighbours.
  • AGENTS.md: the rule, in the list of what CI holds a PR to. REVIEW.md: a comment that says something the code does not do is a defect; a short comment that tells the backstory instead of the reason is a nit. .github/copilot-instructions.md: the rule inline. CONTRIBUTING.md: the local command.

Risk

  • It changes the house style. Replayed over the last 30 PRs merged to main (each squash commit against its parent), 13 would have failed, on 17 blocks of 3 to 9 lines. All 17 are prose comments, for example the Python 3.13 note at ci.yml:45-50 and the document-count note at benchmarks/spec.v1.yaml:75-80. Two-line comments, 726 of them in the tree, pass.
  • Editing one line of an old long block fails the PR until the block is two lines. That is the intent (shorten a long comment in a file you touch), but it will surprise the first few PRs.
  • No override label. A label would also need an entry in scripts/ci_local.py's expression table, as skip-changelog has; it can follow if a real exception turns up.
  • YAML, shell and TOML are read line by line, so a # line inside a YAML block scalar counts. In workflows that is correct (shell comments in run:); elsewhere it could count text that is not a comment.

Testing

  • tests/test_check_comment_blocks.py, 23 tests, including temp git repos for a new long block, an untouched one, a rename and an undiffable base. Each rule was mutation-checked: allowing two lines, dropping rename detection, the tokenizer, the line split, the exemptions, the ruler rule or the changed-lines filter each fails a test.
  • ruff check . and zizmor --offline --min-severity medium .github/workflows: clean. The check passes on this PR.
  • In an env without the LLM dependencies: test_audit_test_quality.py, test_docs_code_refs.py, test_ci_local.py, test_ci_label_triggers.py, test_text_io_encoding.py, test_docs_links.py and test_growth_ratchet.py pass, bar five growth-ratchet tests that call git init -b, which my git 2.25 does not have. The guidance self-scan reports 0 failures, and pre-commit run --all-files passes.
  • Not run locally: the full pytest suite and the license header check (Docker). mypy does not cover scripts/. CI runs the rest.

No changelog fragment: nothing under dokimi/dokimi/, dokimi/data/ or the install tables changes.

Revert

Revert the commit. It writes no records and changes no schema.

Checklist

  • Tests added/updated; the full pytest suite is left to CI (see Testing)
  • ruff passes; mypy and the schema checks are unaffected (no change under dokimi/dokimi/)
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

A comment says the constraint or the reason in at most two lines and links
an issue or PR for the backstory. scripts/check_comment_blocks.py enforces
the length on the lines a PR adds or changes, as a step of the static job,
so existing long blocks pass until a PR edits them. AGENTS.md, REVIEW.md,
the Copilot instructions and CONTRIBUTING.md carry the rule.

Signed-off-by: Harlow, Jordan <jordan.harlow@intel.com>
@jharlow-intel jharlow-intel self-assigned this Oct 29, 2026
Copilot AI balanced review requested due to automatic review settings October 29, 2026 18:07

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The comment-block CI gate changes have no active review issues requiring resolution.

No open issues 🎉

What changed in this PR

0 new findings; nothing to fix.

This PR adds a CI gate enforcing dokimi’s two-line comment-block rule on comments a PR adds or edits, while leaving untouched existing blocks alone.

Changes:

  • Adds a diff-aware checker with Python tokenization, rename handling, and exemptions for headers, pragmas, and rulers.
  • Integrates the check into the PR-only static CI job and documents the rule and local command.
  • Adds tests covering comment detection, changed-line filtering, renames, and invalid refs.
File Description
REVIEW.md Adds comment accuracy and concise-comment review guidance.
dokimi/​tests/​test_check_comment_blocks.py Tests detection rules and Git-diff behavior.
dokimi/​scripts/​check_comment_blocks.py Implements the changed-comment length gate.
dokimi/​CONTRIBUTING.md Documents enforcement and the local command.
AGENTS.md Adds the comment rule to contributor guidance.
.github/​workflows/​ci.yml Runs the checker in the static job for PRs.
.github/​copilot-instructions.md Adds concise-comment guidance for Copilot.

Review effort: Balanced

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.27%. Comparing base (e966422) to head (3ebbcd7).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #298      +/-   ##
==========================================
- Coverage   93.28%   93.27%   -0.01%     
==========================================
  Files         272      272              
  Lines       35119    35119              
  Branches     5964     5964              
==========================================
- Hits        32761    32759       -2     
- Misses       1544     1546       +2     
  Partials      814      814              

see 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vchamarthi

Copy link
Copy Markdown
Contributor

🔍 OpenCodeReview found 1 issue(s) in this PR.

By severity: 1 low

Severity Category Location Finding
low test test_check_comment_blocks.py:113 Nit: the name says main, but the body calls ccb.findings.
Review details
  • Commit reviewed: 3ebbcd7 against merge base e966422
  • Coverage: 7 files in 3 groups
  • Engine: OpenCodeReview v1.13.3, us.anthropic.claude-sonnet-5-5
  • Usage: 197k tokens (127k from cache), 1m0s
  • Tool calls: 13 (7 code_search, 3 file_find, 2 file_read, 1 code_comment)
  • Comment block check script and its tests: 2 files
  • CI workflow integration for comment block check: 1 file
  • Contributor and reviewer documentation/guidelines updates: 4 files

assert "::error file=a.py,line=2,endLine=4," in capsys.readouterr().out


def test_main_passes_an_untouched_long_block_and_a_renamed_file(tmp_path, monkeypatch):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test · low
Nit: the name says main, but the body calls ccb.findings. No test runs main on a passing PR, so a regression in the exit-0 path would go unnoticed. That includes returning 1 or 2 when found is empty, and dropping the success message. Assert ccb.main(["--base", "base", "--head", "HEAD"]) == 0 here as well.

Suggestion:

Suggested change
def test_main_passes_an_untouched_long_block_and_a_renamed_file(tmp_path, monkeypatch):
assert ccb.findings("base", "HEAD") == []
assert ccb.main(["--base", "base", "--head", "HEAD"]) == 0

@napetrov
napetrov merged commit b930502 into intel:main Oct 30, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants