Skip to content

ci: build the GNU oracle from the pinned parity reference (Emacs 31.0.90) - #352

Draft
tag-und-nacht wants to merge 1 commit into
eval-exec:mainfrom
tag-und-nacht:ci/gnu-oracle-31.0.90
Draft

ci: build the GNU oracle from the pinned parity reference (Emacs 31.0.90)#352
tag-und-nacht wants to merge 1 commit into
eval-exec:mainfrom
tag-und-nacht:ci/gnu-oracle-31.0.90

Conversation

@tag-und-nacht

Copy link
Copy Markdown
Contributor

Summary

Every GNU-vs-Neomacs comparison in CI has been spawning the wrong GNU Emacs: the apt emacs-nox 29.3 until today, and Emacs 31.1 from nix-emacs-ci since 03c946f. The parity reference in parity-reference.toml is 31.0.90 (emacs-mirror 0ee48ac4df2), the revision the Lisp tree is synced to and the build every published parity number was measured against.

Emacs 31.0.91 and 31.1 removed the display (min-width ...) padding from mode-line-position (emacs commit 388adcc570b, "Finish reverting experiment with proportional font on mode line"). Against 31.1 the mode-line row of every screen therefore differs:

GNU oracle neomacs-tui-tests on main (942 tests)
31.1 (CI today, and locally) 889 fail
31.0.90 (this PR, locally) 20 fail, all environmental

All 20 tui shards have been red on every CI run for the same reason, so the suite carries no regression signal today.

Change

  • .github/actions/setup-gnu-emacs compiles the reference itself. It reads emacs_version and mirror_commit from parity-reference.toml (the action names no version of its own, so cargo run -p xtask -- pin-reference moves CI too), fetches that commit from emacs-mirror, configures it TTY/batch-only without native compilation to match the pinned reference, installs to /opt/gnu-emacs-oracle, and caches the prefix by commit and by the action file's hash. The smoke step compares emacs-version with the pin instead of checking that some emacs runs.
  • A new gnu-emacs-oracle job in ci.yml warms that cache alongside the runtime build. The jobs that spawn GNU (neomacs-tui-tests, the MELPA parity matrix, the prefix-face TUI parity job, the GUI tests, the live MELPA canary) depend on it, so a cache miss compiles Emacs once rather than once per shard. A cache hit is one restore.
  • NEOMACS_MELPA_ORACLE_EMACS and NEOMACS_GUI_TEST_GNU_EMACS point at the absolute oracle path.
  • The 31.1 commit also left three xtask contract tests failing on main (they still asserted /usr/bin/emacs and the apt emacs-nox). They now assert the new contract, and a new test ties the action to parity-reference.toml so a re-pin cannot leave CI on a stale oracle.
  • docs/building.md says which GNU the TUI suite needs locally and why.

Verification

  • cargo nextest run -p xtask -E 'test(ci_) | test(gnu_oracle) | test(linux_ci_setup)': the contract tests that fail on main now pass (the unrelated CARGO_BUILD_JOBS assertion in ci_builds_shared_test_artifacts_on_github_hosted_runners still fails on main and is untouched here).
  • actionlint and shellcheck on the action's run blocks are clean; cargo fmt --check is clean.
  • Locally on Linux x86_64: built the emacs-31.0.90 tag with the same configure flags and ran the full TUI suite against it on main and on fix(layout): end a leaf at its start once the visibility retry budget is spent #351: 20 failures on both, identical sets. The same suite against 31.1 fails 889 on both.
  • The action itself only runs on GitHub; this PR's own CI run is the proof of the build and the cache.

Out of scope, noted for follow-up

  • The four GUI tests that need an X-capable GNU (standard input is not a tty); this build is TTY-only like the pinned reference.
  • The face_documentation_matrix oracle cases that fail on some shards with Cannot open doc string file /home/ubuntu/work/...: an absolute path from another machine baked into the shared runtime.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JcwM7SRQJGB3E52uwFq9h6

…r release

Every GNU-vs-Neomacs comparison in CI spawned the wrong Emacs. Until
today the apt emacs-nox 29.3; since 03c946f, Emacs 31.1 from
nix-emacs-ci. Neither is the parity reference in parity-reference.toml
(31.0.90, emacs-mirror 0ee48ac4df2), which is the revision the lisp tree
is synced to and the build every published parity number was measured
against. Emacs 31.0.91 and 31.1 removed the `display (min-width ...)`
padding from `mode-line-position` (emacs commit 388adcc570b), so against
31.1 the mode-line row of every screen differs: on main, 889 of 942
neomacs-tui-tests fail in CI and locally, and all 20 tui shards have been
red on every run. Against a 31.0.90 build the same suite fails 20 tests,
all of them environmental (terminfo, HELLO/info buffers, Doom docs).

nix-emacs-ci publishes no pretests, so the action now compiles the
reference itself: it reads emacs_version and mirror_commit from
parity-reference.toml, fetches that commit from emacs-mirror, configures
it TTY/batch-only without native compilation (the pinned reference's
configuration), installs to /opt/gnu-emacs-oracle and caches the prefix
by commit and by the action file's hash. The smoke step compares
`emacs-version` with the pin rather than checking that some emacs runs.
A new ci.yml job warms the cache alongside the runtime build; the jobs
that spawn GNU depend on it, so a cache miss compiles Emacs once instead
of once per shard.

The 31.1 commit also left three xtask contract tests failing on main:
they still asserted `/usr/bin/emacs` and the apt `emacs-nox` package.
They now assert the new contract (the /opt/gnu-emacs-oracle path, the
action in every GNU-spawning job, no apt emacs in any profile), and a
new test ties the action to parity-reference.toml so a re-pin cannot
leave CI on a stale oracle.

Out of scope, left as follow-ups: an X-capable oracle for the four GUI
tests that need one, and the doc-string path relocation that fails the
face_documentation_matrix oracle cases on some shards.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JcwM7SRQJGB3E52uwFq9h6
@tag-und-nacht

Copy link
Copy Markdown
Contributor Author

A note on cost, and two alternatives, so this can be judged against what upstream is willing to pay for.

What this PR costs as written. The build is not repeated per run. The action caches /opt/gnu-emacs-oracle with actions/cache, keyed on the pinned commit and on the action file's hash, and the new gnu-emacs-oracle job warms that key once per run while the runtime builds. On this PR's own run the cold build took 5m48s on a 4-vCPU ubuntu-24.04 runner; every later job restored it. A rebuild happens only on a re-pin, an edit to the action, or an eviction. Eviction is the real weakness: the workflows already record that the repo's 10 GiB Actions cache is under pressure and that release caches get evicted by CI-run caches, so the oracle entry (about 150 MB) will be evicted now and then and cost another 6-minute build.

Alternative 1: build the exact commit once and serve it from cachix. The flake already trusts eval-exec.cachix.org, and cachix-release.yml already holds the push token. A gnu-emacs-oracle package in the flake, built from mirror_commit in parity-reference.toml (nixpkgs' emacs with src overridden and native compilation and the window system off), pushed once by a workflow_dispatch job with the token, would make the oracle a substituted download for every CI job and for every developer (nix run .#gnu-emacs-oracle), with no eviction and no per-run compile. It also makes the parity reference bit-reproducible, so parity-reference.toml could pin that derivation's fingerprint rather than one person's local build. The cost is a one-time push and the small flake package. If that is preferred, this PR can switch the action to nix build .#gnu-emacs-oracle and drop the source build; the contract tests and the warm-up job stay the same.

Alternative 2 (complementary): more signal from the suites themselves. Independently of which oracle CI uses, the suites carry stable failures on main (about 66 core tests, 22 display tests, and the face_documentation_matrix oracle cases whose runtime bakes in /home/ubuntu/work/...), so a PR's checks cannot go green and nobody can see a new failure. Two cheap steps would help: have the TUI and GUI harnesses attest and print which GNU they compared against (the neomacs-parity-reference crate already knows how), and keep an explicit, counted list of known-failing tests that the shard workflow compares against, so a new failure fails the job while a known one does not, and a skipped-but-required test is counted rather than silently green.

Happy to rework the PR either way; the rationale for 31.0.90 rather than 31.1 (the mode-line-position min-width change in emacs 388adcc570b) does not depend on how the build is delivered.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant