Repository navigation
test(release-verify-rc): grade Step 6b paste recipes by their rules, not one reference text - #1536
Merged
Merged
Conversation
…not one reference text The Step 6b suite flaked 1-2 of 6 cases on `paste_recipe`, on `main` too: the grader compared each candidate recipe with one reference recipe word for word, so correct answers failed for quoting the `gpg --verify` arguments differently or for adding the companion verification lines. Replace the exact `paste_recipe` in every case with structural checks in a new `assertions.json`, encoding the output-spec rule: the recipe invokes `maven-artifact-verify` on the staged directory, sets `--digests sha512` (the digest set every case configures), and passes `--podling` only when the source artefact ships a `DISCLAIMER`. Every original reference recipe satisfies them. Same treatment as the Step 6c suite in apache#1505. Generated-by: Claude Opus 5
github-actions
Bot
added
capability:triage
21 tasks
potiuk
added a commit
that referenced
this pull request
Oct 6, 2026
#1542) * test(release-verify-rc): grade step 2/3/6 paste recipes by their rules The last three release-verify-rc suites still grade `paste_recipe` against one reference recipe: Step 2 (signatures), Step 3 (checksums) and Step 6 (binary exclusion). Correct answers fail for quoting the gpg arguments differently, importing the already-downloaded KEYS file, passing `-c` for `--check`, or splitting the binary scan across several find calls, while the step only requires properties of the recipe. Replace the exact `paste_recipe` in every case with structural checks in a new `assertions.json`, encoding the output-spec rules: Step 2 - a gpg key import and a `gpg --verify` of the staged `.asc` signatures; Step 3 - `sha512sum`/`sha256sum` in check mode (`-c` accepted for `--check`); Step 6 - a `find` whose globs cover the eight-entry fixed baseline. Every original reference recipe satisfies them. Same treatment as the Step 6c suite in PR #1505 and the Step 6b suite in PR #1536. Generated-by: ZCode (GLM-5.3-Flash) * test(release-verify-rc): accept looped and wrapped gpg --verify in step 2 has_sig_verify required `--verify` and `.asc` on one line in that order, so a `for f in *.asc; do gpg --verify "$f"; done` loop or a backslash-continued `gpg --verify` still failed. Check the verify call and the `.asc` signature as two separate assertions. Generated-by: Claude Opus 5 --------- Co-authored-by: Jarek Potiuk <potiuk@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
release-verify-rcStep 6b eval suite flaked 1–2 of 6 cases onpaste_recipe, onmainas well: the grader compared each candidate recipe with one reference recipe word for word, so correct answers failed for quoting thegpg --verifyarguments differently or for adding the companion verification lines.paste_recipevalues are replaced by structural checks in a newassertions.jsonthat encode the output-spec rule: the recipe invokesmaven-artifact-verifyon the staged directory, sets--digests sha512, and passes--podlingonly when the source artefact ships aDISCLAIMER. Every original reference recipe satisfies them.Type of change
.claude/skills/<name>/) — eval fixtures updated belowtools/<system>/*.md)tools/*/withpyproject.toml)docs/,README.md,CONTRIBUTING.md)projects/_template/)prek, workflows, validators)tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/)Test plan
paste_recipesatisfies the newhas_*checks (checked with the runner's ownevaluate_deterministic_assertion)claude -p: 6/6 (a first run was 5/6 only because the grader returned no verdict for the proseobservationsfield — unrelated topaste_recipe)mainbefore this change: 5/6, failingcase-5onpaste_recipeexactnessRFC-AI-0004 compliance
No principle touched — eval fixtures only.
Linked issues
Follow-up to #1505.
Generated-by: Claude Opus 5 (assisted; reviewed by the PR author)
🤖 Generated with Claude Code