ci: consolidate retry and failure diagnostics, add swap headroom

[gilescope]

Extracted from #442 (the buildkit upgrade) so the CI flake mitigations can land independently of the bump. Net effect on the workflows is a deletion: two composite actions replace 33 hand-copied blocks.

earth-retry — 16 retry loops become one action

Non-deterministic earthly Canceled failures (different targets each run, no signal received) were already retried, by 16 near-identical inline loops across 10 workflows — ~270 lines that had drifted into 11 distinct shapes, varying in attempt count, sleep, re-bootstrap, and log capture. They are now one composite action.

Attempts are harmonised at 3 everywhere, previously 2 or 3 depending on which copy you landed on. Genuine failures still fail: the final attempt's exit code propagates.

Between attempts the buildkitd container, cache volumes and ~/.earthly/buildkit are dropped. Dropping the container alone isn't enough — earthly's client-side state still references the dead session and the retry fails immediately with no active sessions.

The buildkitd log is now dumped before the reset destroys it at all 16 sites. Exactly one of the previous copies did this; the other 15 discarded the only evidence that made attempt-1 session-loss failures diagnosable.

failure-diagnostics — 17 Buildkit logs steps become one action

Every workflow had a hand-rolled docker logs earthly-buildkitd failure step. Two of them sat immediately beside a failure-diagnostics step that already did strictly more, dumping the same log twice.

All 17 now use the action, which loops four daemon names rather than one, and adds the memory, swap and kernel-OOM context needed to tell an OOM kill from a real failure. LOG_TAIL defaults to 2000 so the consolidation isn't a coverage regression against the copies that dumped the log in full.

The action is written to be usable outside this repo — every input defaults (BINARY: auto probes docker/podman/nerdctl), Linux-only probes are skipped on macOS runners rather than failing noisily, it warns and continues when no engine is present, and it always exits 0 so diagnostics never mask the failure that triggered them. It ships with a README, and docs/ci-integration/ now points users at it.

Also fixes build-earthly.yml's old handler, which logged earth-buildkitd — a container name that has never existed — and so had been silently logging nothing on every failed build.

Memory headroom

Swap setup moves from two duplicated inline blocks into scripts/ci/add-swap.sh:

• Runs detached. mkswap plus ~25G of toolchain deletion costs a minute, and nothing downstream needs to block on it. Output goes to $RUNNER_TEMP/add-swap.log, which failure-diagnostics dumps.
• Picks the largest suitable non-root disk (/mnt, the ~70 GiB ephemeral disk, on GitHub images) instead of /, so the swapfile stops competing with docker and buildkit layers for the root disk. Selection is dynamic — fstype and free space are checked, with / as the fallback.
• stage2-setup now adds swap on the test path too, matching the build path.
• +lint-scripts covers scripts/ci/*.sh, so the script is shellchecked by +lint-all.

Swap is framed as removing memory from the list of suspects, not as a diagnosis. The same signatures in this repo's history have also been a wget flake and a fatal stats decode, so the mix is workload-specific; what makes memory worth pre-empting is that an OOM kill leaves no trace in the job log and so cannot be ruled out after the fact.

Follow-up filed

#795 proposes making this reliable in the engine rather than in CI: earth --retries N --retry-timeout 5m with RUN --no-retry as a poison marker, and a configurable regex list for classifying retryable output. BuildKit already retries 5xx/ECONNRESET/io.EOF with exponential backoff in util/resolver/retryhandler, but only on the registry path — RUN and the HTTP/git sources get nothing. Cross-references moby/buildkit#6429, which covers the per-RUN half that needs layer reset.

Not in this PR

Stays in #442, each for its own review:

• the buildkit image bump (buildkitd-v0.8.17-fix.1 → fix.5) and the actions-setup pin
• the test-group resplit (WAIT blocks, group13/group14)
• the buildkitd-artifact rework — it changes what CI actually tests
• buildkit_max_parallelism 1 — not needed on main

An earlier revision of this branch also made the earthly-next (Tick-Tock) workflow opt-in. The merge of main superseded that: #615 restructured next into a next-tests job under the shared ci-test-suite.yml and deleted the standalone workflow the opt-in change depended on. main's structure was taken; the load-reduction change is not in this PR.

Verification

• yq parses every workflow and composite action; actionlint clean on all changed files.
• shellcheck clean on add-swap.sh and both extracted action scripts, using the repo's pinned alpine shellcheck.
• All 16 retry conversions machine-verified equivalent to the originals — a script re-parsed each site from the git index and the working tree and compared attempt count, sleep, bootstrap command and exact command lines.
• earth-retry exercised directly: succeeds-first-try, fails-then-succeeds, exhausts-attempts-and-propagates-rc, and the multi-line last-command-wins rule that matches the inline set +e semantics it replaces.
• add-swap.sh exercised end-to-end in a privileged container against a real ext4 loop device (fallocate → mkswap → swapon, detach, --wait propagation, idempotent re-run), plus disk-selection rejection paths on a Linux host.
• failure-diagnostics executed on macOS (BSD branch) and in a bare Linux container with no engine present.

Known gaps

• reusable-misc-tests-1.yml previously used 3 attempts where its peers used 2; harmonising to 3 preserves it, but whether remote-cache genuinely needs more than the rest is untested either way.
• Pre-existing, not addressed: reusable-secrets-integrations.yml:59 passes EARTHLY_TOKEN to stage2-setup, which declares no such input (GHA drops it silently). Present on main; adding the input would quiet the lint without establishing whether the token was ever needed.

🤖 Generated with Claude Code

[github-actions]

⚠️ Are we earthbuild yet?

Warning: "earthly" occurrences have increased by 8 (0.2%)

📈 Overall Progress

Branch	Total Count
main	4004
This PR	4012
Difference	+8 (0.2%)

📁 Changes by file type:

File Type	Change
Go files (.go)	➖ No change
Documentation (.md)	❌ +3
Earthfiles	➖ No change

---

Keep up the great work migrating from Earthly to Earthbuild! 🚀

💡 Tips for finding more occurrences

Run locally to see detailed breakdown:

./.github/scripts/count-earthly.sh

Note that the goal is not to reach 0.
There is anticipated to be at least some occurrences of earthly in the source code due to backwards compatibility with config files and language constructs.

[janishorsts]

LGTM.

I have a slight concern about losing the visibility of indeterministic failures when retrying.