Repository navigation
test(release-verify-rc): grade step 2/3/6 paste recipes by their rules - #1542
Conversation
The last three release-verify-rc suites still grade `paste_recipe` against one reference recipe: Step 2 (signatures), Step 3 (checksums) and Step 6 (binary exclusion). Correct answers fail for quoting the gpg arguments differently, importing the already-downloaded KEYS file, passing `-c` for `--check`, or splitting the binary scan across several find calls, while the step only requires properties of the recipe. Replace the exact `paste_recipe` in every case with structural checks in a new `assertions.json`, encoding the output-spec rules: Step 2 - a gpg key import and a `gpg --verify` of the staged `.asc` signatures; Step 3 - `sha512sum`/`sha256sum` in check mode (`-c` accepted for `--check`); Step 6 - a `find` whose globs cover the eight-entry fixed baseline. Every original reference recipe satisfies them. Same treatment as the Step 6c suite in PR apache#1505 and the Step 6b suite in PR apache#1536. Generated-by: ZCode (GLM-5.3-Flash)
…ep 2 has_sig_verify required `--verify` and `.asc` on one line in that order, so a `for f in *.asc; do gpg --verify "$f"; done` loop or a backslash-continued `gpg --verify` still failed. Check the verify call and the `.asc` signature as two separate assertions. Generated-by: Claude Opus 5
potiuk
left a comment
There was a problem hiding this comment.
Thanks — this brings Steps 2, 3 and 6 in line with the 6b/6c suites, and every original reference recipe still satisfies the new assertions.
I pushed one small fixup (57cd887): has_sig_verify required --verify and .asc on the same line in that order, so two correct recipes still failed — a for f in *.asc; do gpg --verify "$f"; done loop (the .asc comes before --verify) and a backslash-continued gpg --verify \. It now checks the verify call and the .asc signature as two separate assertions (has_sig_verify + has_asc_signature), so both forms pass.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.More on how Apache Magpie handles maintainer review:
Contributing guide.
The last three release-verify-rc suites still grade
paste_recipeagainst one reference recipe: Step 2 (signatures), Step 3 (checksums) and Step 6 (binary exclusion). Correct answers fail for quoting the gpg arguments differently, importing the already-downloaded KEYS file, passing-cfor--check, or splitting the binary scan across several find calls, while the step only requires properties of the recipe.Replace the exact
paste_recipein every case with structural checks in a newassertions.json, encoding the output-spec rules: Step 2 - a gpg key import and agpg --verifyof the staged.ascsignatures; Step 3 -sha512sum/sha256sumin check mode (-caccepted for--check); Step 6 - afindwhose globs cover the eight-entry fixed baseline. Every original reference recipe satisfies them.Same treatment as the Step 6c suite in PR #1505 and the Step 6b suite in PR #1536.
Generated-by: ZCode (GLM-5.3-Flash)
Summary
Type of change
.claude/skills/<name>/) — eval fixtures updated belowtools/<system>/*.md)tools/*/withpyproject.toml)docs/,README.md,CONTRIBUTING.md)projects/_template/)prek, workflows, validators)Test plan
prek run --all-filespassesuv run pytest/ruff check/mypypasses(
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)(a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
RFC-AI-0004 compliance
<PROJECT>,<tracker>,<upstream>,<security-list>) used in all skill / tool prose (thecheck-placeholdersprek hook is the mechanical gate)Linked issues
Notes for reviewers (optional)