ci: build the GNU oracle from the pinned parity reference (Emacs 31.0.90) - #352
ci: build the GNU oracle from the pinned parity reference (Emacs 31.0.90)#352tag-und-nacht wants to merge 1 commit into
Conversation
…r release Every GNU-vs-Neomacs comparison in CI spawned the wrong Emacs. Until today the apt emacs-nox 29.3; since 03c946f, Emacs 31.1 from nix-emacs-ci. Neither is the parity reference in parity-reference.toml (31.0.90, emacs-mirror 0ee48ac4df2), which is the revision the lisp tree is synced to and the build every published parity number was measured against. Emacs 31.0.91 and 31.1 removed the `display (min-width ...)` padding from `mode-line-position` (emacs commit 388adcc570b), so against 31.1 the mode-line row of every screen differs: on main, 889 of 942 neomacs-tui-tests fail in CI and locally, and all 20 tui shards have been red on every run. Against a 31.0.90 build the same suite fails 20 tests, all of them environmental (terminfo, HELLO/info buffers, Doom docs). nix-emacs-ci publishes no pretests, so the action now compiles the reference itself: it reads emacs_version and mirror_commit from parity-reference.toml, fetches that commit from emacs-mirror, configures it TTY/batch-only without native compilation (the pinned reference's configuration), installs to /opt/gnu-emacs-oracle and caches the prefix by commit and by the action file's hash. The smoke step compares `emacs-version` with the pin rather than checking that some emacs runs. A new ci.yml job warms the cache alongside the runtime build; the jobs that spawn GNU depend on it, so a cache miss compiles Emacs once instead of once per shard. The 31.1 commit also left three xtask contract tests failing on main: they still asserted `/usr/bin/emacs` and the apt `emacs-nox` package. They now assert the new contract (the /opt/gnu-emacs-oracle path, the action in every GNU-spawning job, no apt emacs in any profile), and a new test ties the action to parity-reference.toml so a re-pin cannot leave CI on a stale oracle. Out of scope, left as follow-ups: an X-capable oracle for the four GUI tests that need one, and the doc-string path relocation that fails the face_documentation_matrix oracle cases on some shards. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JcwM7SRQJGB3E52uwFq9h6
|
A note on cost, and two alternatives, so this can be judged against what upstream is willing to pay for. What this PR costs as written. The build is not repeated per run. The action caches Alternative 1: build the exact commit once and serve it from cachix. The flake already trusts Alternative 2 (complementary): more signal from the suites themselves. Independently of which oracle CI uses, the suites carry stable failures on main (about 66 core tests, 22 display tests, and the Happy to rework the PR either way; the rationale for 31.0.90 rather than 31.1 (the |
Summary
Every GNU-vs-Neomacs comparison in CI has been spawning the wrong GNU Emacs: the apt
emacs-nox29.3 until today, and Emacs 31.1 from nix-emacs-ci since 03c946f. The parity reference inparity-reference.tomlis 31.0.90 (emacs-mirror0ee48ac4df2), the revision the Lisp tree is synced to and the build every published parity number was measured against.Emacs 31.0.91 and 31.1 removed the
display (min-width ...)padding frommode-line-position(emacs commit 388adcc570b, "Finish reverting experiment with proportional font on mode line"). Against 31.1 the mode-line row of every screen therefore differs:neomacs-tui-testson main (942 tests)All 20 tui shards have been red on every CI run for the same reason, so the suite carries no regression signal today.
Change
.github/actions/setup-gnu-emacscompiles the reference itself. It readsemacs_versionandmirror_commitfromparity-reference.toml(the action names no version of its own, socargo run -p xtask -- pin-referencemoves CI too), fetches that commit from emacs-mirror, configures it TTY/batch-only without native compilation to match the pinned reference, installs to/opt/gnu-emacs-oracle, and caches the prefix by commit and by the action file's hash. The smoke step comparesemacs-versionwith the pin instead of checking that some emacs runs.gnu-emacs-oraclejob in ci.yml warms that cache alongside the runtime build. The jobs that spawn GNU (neomacs-tui-tests, the MELPA parity matrix, the prefix-face TUI parity job, the GUI tests, the live MELPA canary) depend on it, so a cache miss compiles Emacs once rather than once per shard. A cache hit is one restore.NEOMACS_MELPA_ORACLE_EMACSandNEOMACS_GUI_TEST_GNU_EMACSpoint at the absolute oracle path./usr/bin/emacsand the aptemacs-nox). They now assert the new contract, and a new test ties the action toparity-reference.tomlso a re-pin cannot leave CI on a stale oracle.docs/building.mdsays which GNU the TUI suite needs locally and why.Verification
cargo nextest run -p xtask -E 'test(ci_) | test(gnu_oracle) | test(linux_ci_setup)': the contract tests that fail on main now pass (the unrelatedCARGO_BUILD_JOBSassertion inci_builds_shared_test_artifacts_on_github_hosted_runnersstill fails on main and is untouched here).actionlintandshellcheckon the action's run blocks are clean;cargo fmt --checkis clean.emacs-31.0.90tag with the same configure flags and ran the full TUI suite against it on main and on fix(layout): end a leaf at its start once the visibility retry budget is spent #351: 20 failures on both, identical sets. The same suite against 31.1 fails 889 on both.Out of scope, noted for follow-up
standard input is not a tty); this build is TTY-only like the pinned reference.face_documentation_matrixoracle cases that fail on some shards withCannot open doc string file /home/ubuntu/work/...: an absolute path from another machine baked into the shared runtime.🤖 Generated with Claude Code
https://claude.ai/code/session_01JcwM7SRQJGB3E52uwFq9h6