gate-of-the-gate: PR-time smoke of promotion-gate.yml's heart - #151
Merged
Conversation
promotion-gate.yml false-negatived twice on v150.0.6 in one hour:
1. First run 127'd on 'hardcoded 7zz not found' — 7zip package only
exists on Ubuntu 24+; ubuntu-latest is 22.04 (p7zip-full, binary 7z)
2. Retry silently returned empty version — plistutil failed silently,
grep matched the binary blob, sed yielded nothing. '' != '150.0.6'
→ quarantined a GOOD artifact
Both would have been caught in PR time in seconds if the gate's own
logic had unit tests. Adds:
- scripts/tests/test_promotion_gate.py — 12 assertions:
- plist extraction against binary + XML fixtures (the failure mode)
- missing-key returns '' not crash
- update-check hygiene grep table
- meta-check that shipped bearstart-autoconfig.js still has all
three hygiene keywords
- 7z-family binary discoverable after apt install
- gate workflow still has all 5 load-bearing steps
- regressions asserted: uses plistlib, NOT grep|sed;
uses detect loop, NOT hardcoded 7zz
- packaging-and-update-tests.yml — new job promotion-gate-heart runs
the tests + installs p7zip-full (mirrors what the real gate does)
on any change to promotion-gate.yml or the test file
Meta-principle encoded in test names: 'gates that catch bug classes
must themselves be exercised, and their regressions must fire in the
PR, not at release publish.'
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
promotion-gate.ymlfalse-negatived twice on v150.0.6 in one hour: (1) hardcoded7zznot on Ubuntu 22.04, (2) plistutil-grep-sed chain returned empty on binary plist. Both quarantined a good artifact. Both would have been caught in PR-time seconds if the gate's own logic had unit tests.Adds
scripts/tests/test_promotion_gate.py— 12 assertions covering plist extraction (binary + XML), missing-key returns '' not crash, update-check hygiene grep, meta-check that shipped autoconfig has all 3 hygiene keywords, 7z-family binary discoverable after apt install, gate workflow still has 5 load-bearing steps, regression assertions (uses plistlib not sed, uses detect-loop not hardcoded 7zz).packaging-and-update-tests.yml— new jobpromotion-gate-heartfires on any change topromotion-gate.ymlor the test file. Installsp7zip-full(mirrors what the real gate does).Adversarial-review checklist
Test plan
python -m pytest scripts/tests/test_promotion_gate.py -q→ 11 passed, 1 skipped (Linux-only 7z test). Runs green locally on macOS.