Skip to content

test(release-verify-rc): grade step 2/3/6 paste recipes by their rules - #1542

Merged
potiuk merged 2 commits into
apache:mainfrom
liwenjie200543:eval-paste-recipe-rules
Oct 6, 2026
Merged

potiuk merged 2 commits into
apache:mainfrom
liwenjie200543:eval-paste-recipe-rules

Conversation

@liwenjie200543

Copy link
Copy Markdown
Contributor

The last three release-verify-rc suites still grade paste_recipe against one reference recipe: Step 2 (signatures), Step 3 (checksums) and Step 6 (binary exclusion). Correct answers fail for quoting the gpg arguments differently, importing the already-downloaded KEYS file, passing -c for --check, or splitting the binary scan across several find calls, while the step only requires properties of the recipe.

Replace the exact paste_recipe in every case with structural checks in a new assertions.json, encoding the output-spec rules: Step 2 - a gpg key import and a gpg --verify of the staged .asc signatures; Step 3 - sha512sum/sha256sum in check mode (-c accepted for --check); Step 6 - a find whose globs cover the eight-entry fixed baseline. Every original reference recipe satisfies them.

Same treatment as the Step 6c suite in PR #1505 and the Step 6b suite in PR #1536.

Generated-by: ZCode (GLM-5.3-Flash)

Summary

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other:

Test plan

  • prek run --all-files passes
  • For Python packages touched: uv run pytest / ruff check / mypy passes
  • For Groovy bridges touched: command-line invocation tested end-to-end
  • For skill changes: eval suite passes for the affected skill
    (PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)
  • For skill behaviour changes: a new or updated eval fixture is included in this PR
    (a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
  • Other:

RFC-AI-0004 compliance

  • HITL — any new mutation is gated on explicit user confirmation
  • Sandbox — no new unrestricted host access; network reach declared in the adapter
  • Vendor neutrality — placeholders (<PROJECT>, <tracker>, <upstream>, <security-list>) used in all skill / tool prose (the check-placeholders prek hook is the mechanical gate)
  • Conversational + correctable — agentic-override path documented if behaviour is adopter-tunable
  • Write-access discipline — no autonomous outbound messages; drafts only, sent on confirmation
  • Privacy LLM — private content does not reach a non-approved LLM; redactor invoked where needed

Linked issues

Notes for reviewers (optional)

The last three release-verify-rc suites still grade `paste_recipe`
against one reference recipe: Step 2 (signatures), Step 3 (checksums)
and Step 6 (binary exclusion). Correct answers fail for quoting the
gpg arguments differently, importing the already-downloaded KEYS file,
passing `-c` for `--check`, or splitting the binary scan across several
find calls, while the step only requires properties of the recipe.

Replace the exact `paste_recipe` in every case with structural checks
in a new `assertions.json`, encoding the output-spec rules: Step 2 - a
gpg key import and a `gpg --verify` of the staged `.asc` signatures;
Step 3 - `sha512sum`/`sha256sum` in check mode (`-c` accepted for
`--check`); Step 6 - a `find` whose globs cover the eight-entry fixed
baseline. Every original reference recipe satisfies them.

Same treatment as the Step 6c suite in PR apache#1505 and the Step 6b suite
in PR apache#1536.

Generated-by: ZCode (GLM-5.3-Flash)
@github-actions github-actions Bot added capability:triage Sweep + classify + propose disposition family:release-management release-* skills labels Oct 6, 2026
…ep 2

has_sig_verify required `--verify` and `.asc` on one line in that order,
so a `for f in *.asc; do gpg --verify "$f"; done` loop or a
backslash-continued `gpg --verify` still failed. Check the verify call
and the `.asc` signature as two separate assertions.

Generated-by: Claude Opus 5

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — this brings Steps 2, 3 and 6 in line with the 6b/6c suites, and every original reference recipe still satisfies the new assertions.

I pushed one small fixup (57cd887): has_sig_verify required --verify and .asc on the same line in that order, so two correct recipes still failed — a for f in *.asc; do gpg --verify "$f"; done loop (the .asc comes before --verify) and a backslash-continued gpg --verify \. It now checks the verify call and the .asc signature as two separate assertions (has_sig_verify + has_asc_signature), so both forms pass.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Magpie handles maintainer review:
Contributing guide.

@potiuk
potiuk merged commit 4de559e into apache:main Oct 6, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capability:triage Sweep + classify + propose disposition family:release-management release-* skills

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants