Skip to content

fix(ci): prebuild dav1d for the Windows test job - #779

Open
Lolle2000la wants to merge 15 commits into
masterfrom
fix/windows-dav1d-prebuilt-ci
Open

Lolle2000la wants to merge 15 commits into
masterfrom
fix/windows-dav1d-prebuilt-ci

Conversation

@Lolle2000la

Copy link
Copy Markdown
Owner

Problem

Test (windows-latest) is red on master and on every PR stacked on it:

LINK : fatal error LNK1181: cannot open input file 'dav1d.lib'

(run 45025059310 / job 137216198289, and master run 45020466209.)

dav1d-sys 0.8.3's build script shells out to git clone (from code.videolan.org), meson setup, ninja and meson install through a helper that only panics when the process cannot be spawned — it never inspects an exit status. A failed clone or compile is therefore silently treated as success: cargo marks the build script fresh, no dav1d.lib is produced, and the link fails. The existing nick-fields/retry around the compile step cannot recover, because the clone it repeats is the part that flakes.

Fix

Build the same static library in its own retried Windows step:

  • dav1d 1.5.0 (the tag dav1d-sys 0.8.3 pins), fetched as a commit-pinned archive from github.com, with the commit's meson.build version asserted;
  • meson/ninja/meson install run under set -e, into a stable prefix ($GITHUB_WORKSPACE/dav1d-static), with the installed lib/dav1d.lib asserted to exist — so a failure is loud and this step is what retries;
  • system-deps is then pointed at that install tree (SYSTEM_DEPS_DAV1D_NO_PKG_CONFIG + _LIB/_LINK=static/_SEARCH_NATIVE/_INCLUDE), so dav1d-sys's build script never runs git at all. The previous SYSTEM_DEPS_DAV1D_BUILD_INTERNAL=always and the half-clone cleanup in on_retry_command are gone with it.

release.yml is deliberately untouched; the same latent risk there is a follow-up once this approach has proven itself on CI.

Verification

  • Locally: cargo build -p dav1d-sys with exactly these SYSTEM_DEPS_* variables emitted rustc-link-search=native=<prefix>/lib, rustc-link-lib=static=dav1d and cargo:include=<prefix>/include, with no git/meson invocation.
  • The archive URL and its version: '1.5.0' were checked; the YAML parses.
  • End-to-end proof is this PR's own Test (windows-latest) job, which now fails loudly with the real meson/ninja/compiler error instead of LNK1181 if anything is still wrong.

dav1d-sys 0.8.3's build script runs git/meson/ninja through a helper that
never inspects an exit status, so a failed clone or compile is silently
treated as success: cargo marks the script fresh, no dav1d.lib appears, and
every later link dies with "LNK1181: cannot open input file 'dav1d.lib'"
(jobs 134899537045 and 137216198289). The existing retry cannot clear it
because the clone it repeats comes from code.videolan.org.

Build the same static library in its own retried step instead: the commit
dav1d-sys pins (1.5.0), fetched from github.com under `set -e`, meson/ninja
into a stable prefix, and the installed dav1d.lib asserted. system-deps is
then pointed at that tree (NO_PKG_CONFIG + LIB/SEARCH_NATIVE/INCLUDE), so
the crate's build script never runs git at all.

Verified locally that the system-deps wiring emits
`rustc-link-search=native=<prefix>/lib`, `rustc-link-lib=static=dav1d` and
`cargo:include=<prefix>/include` without any pkg-config or clone.
The first run of the new step proved the wiring but died in `meson setup`
on all three attempts: git-bash's /usr/bin comes first on PATH, so meson
resolved coreutils' `link.exe` ("Found GNU link.exe instead of MSVC
link.exe") even though msvc-dev-cmd had selected cl.

Reuse the PATH promotion the compile step already does for rustc, as an
in-step export rather than a GITHUB_PATH entry, because bash re-prepends
/usr/bin on startup and only the exported PATH reaches meson's child
processes.

Also from review:
- put the pinned commit in the install prefix, so a pin bump changes
  SYSTEM_DEPS_DAV1D_SEARCH_NATIVE/INCLUDE and re-runs the probe instead of
  being absorbed by a warm cargo cache;
- 30 -> 10 min per attempt, which keeps the Windows job's worst case near
  270 min against GitHub's 360 min ceiling;
- correct the compile-step comment (dav1d is no longer built there), and
  note the new step in AGENTS.md's retried-install rule.
The second run got past meson's linker detection and then failed on all
three attempts in the first object file:

    include/compat/msvc/stdatomic.h(35): error C7780: function-like macro
    '__has_include_next' is not defined

meson wraps the compiler in the first ccache/sccache on PATH when it
resolves CC itself, and the image ships ccache in C:\Strawberry\c\bin;
ccache drives cl through the conforming preprocessor, where dav1d's
`#if defined(__clang__) && __has_include_next(<stdatomic.h>)` guard is an
error rather than being skipped. dav1d 1.5.4 still carries that guard, so
bumping the pin would not have helped.

Name the compiler explicitly (cygpath -w form of the same MSVC directory
already promoted for link.exe): meson then parses the CC entry instead of
probing for a cache, and ninja runs plain cl. Also fail loudly if no MSVC
linker or cl.exe is found, instead of silently falling back to PATH
detection.

This is very likely what the crate's build script has been swallowing all
along: the failure is deterministic, so it fits two consecutive reds on
master better than the flaky clone the retry was written for.
The plain-cl run refutes the ccache theory: with no cache wrapper and no
preprocessor flag on the command line, MSVC 14.52.37029 still rejects
`#if defined(__clang__) && __has_include_next(<stdatomic.h>)` with
"error C7780: function-like macro '__has_include_next' is not defined" in
every object that includes the header (6 of 135 were attempted).

Take the header's own non-clang branch instead. It is the branch MSVC
builds use anyway and is unreachable for plain cl, so only clang-cl
behaviour would change and this job does not use clang-cl. A grep asserts
the exact line before the sed, so a pin bump that moves it fails loudly
rather than silently not applying.

Keep the explicit CC: it takes ccache out of build.ninja, which is what
made the log clear enough to identify this. Correct the comment and the
AGENTS.md clause, both of which claimed the disproved cause.

This is what dav1d-sys's swallowed ninja status was hiding: the failure is
deterministic, which fits master's two consecutive reds far better than
the flaky clone the retry was written for.
The guard patch worked: ninja built all 135 objects and meson install
succeeded on all three attempts. What failed was my own assertion, and its
log line names the real defect:

    Installing src\libdav1d.a to ...\dav1d-static-<sha>\lib

meson names the MSVC static archive libdav1d.a, while `-l dav1d` makes
link.exe look for dav1d.lib. That mismatch is what "LNK1181: cannot open
input file 'dav1d.lib'" was all along -- the crate's build script did
produce a library, just not under the name the link resolves -- and it is
why retrying the clone never helped. The archive is plain COFF under
either name, so copy it to dav1d.lib and keep the assertion on that name.
1.5.5 is the first tag whose compat/msvc/stdatomic.h carries the upstream
fix for MSVC 14.52:

    #ifndef __has_include_next
    #define __has_include_next(x) 0
    #endif

so the local sed that forced the header onto its non-clang branch is no
longer needed and is removed. The header is unchanged from 1.5.0 through
1.5.4, which is why neither the 1.5.0 dav1d-sys 0.8.3 pins nor the 1.5.4
that 0.9.0 moves to would have compiled here.

include/dav1d/version.h is byte-identical between 1.5.0 and 1.5.5, so the
bindings in dav1d-sys 0.8.3 are unaffected by the bump. Verified that
6c7bee68 is the tag's peeled commit and that its archive and
`version: '1.5.5'` are what the step expects.
meson's default naming scheme names MSVC static libraries libfoo.a on
purpose (its FAQ documents the choice), which is why `-l dav1d` could not
find anything. `-Dnamingscheme=platform` gives DEFAULT_STATIC_LIBRARY_NAMES
["windows"] = ('', 'lib'), so meson installs dav1d.lib itself and the step
no longer rewrites the default scheme's output by name.

Fewer assumptions to maintain: one flag instead of a copy keyed on
libdav1d.a, and the assertion now checks meson honoured what was asked.
scripts/bump-appimage.sh already queries dav1d upstream weekly, applies the
14-day soak and opens the chore/appimage-bump PR. Extend it to own the
hand-maintained DAV1D_TAG/DAV1D_COMMIT pair in ci.yml's 'Build dav1d
(Windows)' step, so nobody has to remember it:

* the commit is the tag's peeled commit from
  'git ls-remote --tags <tag> refs/tags/<tag>^{}', not the tag object: the
  step fetches github.com/videolan/dav1d/archive/$DAV1D_COMMIT.tar.gz, and
  an archive URL for an annotated tag's own sha is a 404 (dav1d 1.5.5
  resolves to 6c7bee68..., not to its tag object 0218c8dd...);
* tag and commit are rewritten in one pass over one file, because the step
  asserts the unpacked meson.build carries 'version: <tag>' -- either line
  alone is a tree that cannot build;
* all-or-nothing is kept: the ls-remote is the run's last network call, both
  rewrites are built as temps and validated (one pin line each, the new
  values present, the line count unchanged) before either file is renamed
  into place, so any query/download/hash/ls-remote/validation failure leaves
  every file byte-identical;
* the ci.yml pin resolves independently of versions.env's DAV1D_VERSION (the
  two are deliberately on different releases), forward-only, and only in a
  run that selects dav1d. --check counts it, so a ci.yml-only bump still
  reaches the PR -- which is why appimage-bump.yml's changed-file check and
  'git add' now cover both files.

Also adds --ci-workflow (mirrors --versions-file, for testing the script) and
documents the second pin in the script header, the workflow comments and
AGENTS.md.
-Dnamingscheme=platform does not do what its description suggests for a
plain `library()` call: meson accepts the option (its summary prints
`namingscheme : platform`), then StaticLibrary.determine_default_prefix_and_suffix
overwrites the platform values with the name_prefix/name_suffix fallbacks
unless the target sets them, and the run still installed `libdav1d.a`
(job 137237434306, all three attempts).

So go back to publishing the archive under the name the Rust link asks
for, in the form that does not assume the default scheme's spelling:
locate whichever `.a`/`.lib` was installed and copy it to dav1d.lib,
failing loudly when meson installed no archive at all. The comment
records the dead end so it is not re-attempted.
…push

The previous revision had the bump write DAV1D_TAG/DAV1D_COMMIT into
.github/workflows/ci.yml. That can never work: a push that creates or
modifies a file under .github/workflows/ requires the 'workflows'
permission, GITHUB_TOKEN has no key for it, and the documented bump PAT is
scoped to Contents + Pull requests. The first Monday a dav1d release cleared
the soak, the push would have been rejected server-side: red run, no branch,
no PR.

So move the pin out of the workflow file:

* scripts/ci-pins.env (new, plain file) holds the two values; ci.yml's
  'Build dav1d (Windows)' step sources it right after cd $GITHUB_WORKSPACE
  and keeps everything downstream identical (archive URL, prefix, meson
  build, dav1d.lib, the six SYSTEM_DEPS_* exports). A missing file fails on
  the explicit guard; an unparsable one fails the source under set -e; both
  values are asserted non-empty, so a broken pin file cannot build nothing.
* bump-appimage.sh writes that file instead, with the same guarantees as
  before: peeled commit via 'git ls-remote --tags <tag> <tag>^{}' (never the
  tag object, whose archive URL 404s), forward-only resolution, soak,
  sort -V, --check/--json, and all-or-nothing (the ls-remote is the last
  network call; both rewrites are validated before either mv). The dead
  ci.yml text rewrite and its --ci-workflow flag are gone; --ci-pins is the
  equivalent test escape hatch.
* rewriting a pin line now preserves everything after the value, so
  'DAV1D_TAG=1.5.4   # keep me' keeps its comment (the old ci.yml rewrite
  silently dropped it). The masked comparison of old and new file enforces
  that only the two values differ.
* appimage-bump.yml stages scripts/ci-pins.env, and runs the AppImage gate
  only when packaging/appimage/versions.env actually changed: a pins-only
  bump cannot be judged by building the payload, so it is left to the PR's
  own CI (which builds that pin on Windows) instead of red-ing the run. The
  PAT now only decides whether the bump PR's own CI runs.

Docs (AGENTS.md, packaging/appimage/README.md) say where the pin lives, why
it is a separate file, and that a local --apply can touch it too.
scripts/ci-pins.env is sourced by a bash step in the Windows test job, and
the repository had no .gitattributes, so the runner's core.autocrlf would
check the file out with CRLF: DAV1D_TAG would carry a trailing carriage
return, failing the "this archive is dav1d <tag>" assertion, and
DAV1D_COMMIT would put %0D into the archive URL.

Pin the file to LF rather than stripping CR in the step, so the file stays
readable and the requirement is recorded where the checkout obeys it.
The "why 1.5.5, not the tag dav1d-sys builds" paragraph sat in ci.yml and a
version number in bump-appimage.sh's comment, while the value itself had
moved to scripts/ci-pins.env. The bump only moves the pin forward, so the
first automated bump would have left both describing 1.5.5 against a file
that says something else, with every assertion still passing.

Move the rationale to the pin file, which the rewrite already preserves as
trailing text on the value's line, reword the script comment to point there
instead of naming a version, and correct the header: `#` only starts a
comment when whitespace separates it from the value.

Verified: sourcing the file yields 1.5.5 / 6c7bee68... with the notes in
place, and `bump-appimage.sh --component dav1d` still reads both pins and
reports `keep`.
The Windows release legs still let dav1d-sys build dav1d through its own
build script, which never inspects a child exit status and now fails
deterministically on this toolset: the source it pins (dav1d 1.5.0) does not
compile with MSVC 14.52 -- "error C7780" on compat/msvc/stdatomic.h -- and
meson's classic naming installs libdav1d.a where `-l dav1d` resolves
dav1d.lib. A tagged release would have failed at link time with LNK1181, the
same way the test job did.

Use the prebuild that ci.yml already proves, from the same pin
(scripts/ci-pins.env, so one bump moves both). Two differences on this
path: the workflow restores no cache, so the archive is rebuilt from the
pinned commit on every release (a signed artifact must not depend on a
cached build), and the aarch64 leg must keep its CC=clang-cl for dav1d's
ARM64 .S files, so cl is named only when CC is unset.

release.yml runs on tag pushes only, so this is unverified until the next
release tag; the x64 recipe is the one ci.yml runs green today.
ci.yml's 'Build dav1d (Windows)' step and release.yml's copy -- the same
~100-line script, added to release.yml by the previous commit -- drift apart
the moment one of them is touched. Extract it into
.github/actions/build-dav1d-windows and call it from three places: the x64
test job, release.yml's two Windows legs, and a new dav1d-arm64 job.

Two deltas from ci.yml's body (the one that is green today): the CC rule
from release.yml's copy -- name cl only when CC is unset, so a caller that
exported CC=clang-cl for ARM64 keeps it -- and 20 minutes per attempt
instead of 10, since one attempt downloads the tag archive, configures
meson and compiles the library for whichever architecture the runner is.
Everything else is moved verbatim: the MSVC linker PATH promotion, the
scripts/ci-pins.env guards, the commit-pinned archive, the
version: '$DAV1D_TAG' assertion, meson/ninja/install, publishing the
installed archive as dav1d.lib, and the six SYSTEM_DEPS_* exports.

The arm64 half had never executed anywhere except a v* tag: ci.yml's test
matrix is windows-latest only, and release.yml's aarch64 leg runs only on a
tag, with no fail-fast: false on that matrix -- so a broken clang-cl path
was invisible until a release was cut and then blocked it. The new
dav1d-arm64 job (windows-11-arm) runs on every CI run: checkout, Rust,
msvc-dev-cmd arm64, retried 'pip install meson ninja', CC/CXX=clang-cl,
the shared action, then 'cargo build -p dav1d-sys' to prove the
system-deps handoff on this toolchain. No cache, no libmpv, no workspace
build.

AGENTS.md: the retried-install paragraph now describes the action (three
callers, 20 min per attempt), the dav1d paragraph says the prebuild is a
shared action exercised on both Windows architectures, and CI Routing lists
the new job.
The new arm64 job proved what it was built for on its first run: clang-cl
built dav1d 1.5.5's ARM64 assembly through the shared action and produced
dav1d.lib with no nasm. The failure was the follow-up `cargo build -p
dav1d-sys`, which died linking build scripts:

    = note: "C:\Program Files\Git\usr\bin\link.exe" "/NOLOGO" ...

i.e. the same coreutils-link.exe hazard the composite action promotes PATH
for -- a plain bash step does not. Run that one step under cmd, where the
PATH msvc-dev-cmd set is taken as-is, and keep the comment explaining what
the check is for.

The release path itself is unaffected: the prebuild step is green on ARM64.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant