bug(macos): macOS portability failures the Intel test lane surfaced

[zackees]

Context

#281 restored the Intel macOS test lane, which had been red for its entire life (14 runs, 14 failures, zero successes). The restored lane is the first thing in this repository to run the test suite on macOS: rust-native's matrix is [ubuntu-latest, windows-latest], macos-arm-tests is gated off by ENABLE_MACOS_RUNNERS, and the two Apple-hosted matrices in ci.yml run only the compiler-policy and screenshot proofs.

So the first green-ish run (35125870261) ran 973 tests on x86_64-apple-darwin and 23 failed. Diagnosing them split the set in two: environment artifacts of a Recovery guest, and genuine macOS portability findings that would fail on any macOS host and had simply never been exercised.

The artifacts are excluded by name in ci/macos-x64/recovery-guest.sh with the evidence recorded inline and in ci/macos-x64/README.md. This issue is for the ones that are real. They are currently excluded from the lane too — otherwise the lane cannot be green — but excluding them is a deferral, not a resolution, and nothing else in CI will catch them.

Findings

1. cap-primitives 4.0.3 panics on a negative st_rdev on macOS — 12 tests

The largest group, and the only one caused by a dependency rather than by a test.

called `Result::unwrap()` on an `Err` value: TryFromIntError(())
  at cap-primitives-4.0.3/src/rustix/fs/metadata_ext.rs:171

Line 171 is:

rdev: u64::try_from(stat.st_rdev).unwrap(),

Two lines above it, the same struct handles the identical problem correctly:

dev: if stat.st_dev < 0 {
    i64::try_from(stat.st_dev).unwrap() as u64
} else {
    u64::try_from(stat.st_dev).unwrap()
},

That guard exists because dev_t is signed on macOS (i32) and unsigned on Linux. The author knew; rdev just never got the same treatment. On Linux the branch is dead, which is exactly why only a macOS run exposes it. rustix's libc backend declares pub st_rdev: c::dev_t, so the conversion is genuinely fallible.

No fixed release exists — cap-primitives 4.0.3 is the newest 4.x on crates.io (the next entries are 3.4.6, a backport on the older line), so this cannot be resolved by a version bump.

Affected: the 12 wasm::worker::output::tests / wasm::worker::tests cases exercised through the capability-based filesystem.

• Report upstream to cap-std/cap-primitives with the code asymmetry above. — done: cap-primitives 4.0.3 panics with TryFromIntError on a negative macOS st_rdev sunfishcode/cap-std#427.
• Decide our mitigation: report-and-wait, pin a patched fork via [patch.crates-io], or drop cap-std from the worker-output path.
• Note this aborts any worker-output operation on a macOS host whose volume reports a negative st_rdev, not just these tests — the impact is not test-only.

2. macOS rejects the TLS test fixtures' certificates — 2 tests

reqwest::Error { kind: Request, source: ... Error { code: -67901,
  message: "The validity period in the certificate exceeds the maximum allowed." } }

macOS's Security framework enforces a maximum TLS certificate validity (825 days) that the fixtures' self-signed certificates exceed. trusted_local_tls_preserves_body and trusted_https_downgrade_is_rejected_before_plaintext_connection are not #[ignore]d, so they run anywhere the suite runs on macOS.

• Regenerate the fixtures with a validity window macOS accepts, or make the tests macOS-aware. — fixed in fix(macos): make three portability findings correct on macOS #284 (820-day validity).

3. macOS canonicalizes /var to /private/var — 1 test

assertion `left == right` failed
  left:  "/private/var/folders/zz/.../T/.tmpvPapPF/target"
  right: "/var/folders/zz/.../T/.tmpvPapPF/target"

context_metadata_and_link_read_do_not_follow_final_symlinks compares a path it built against a canonicalized one. /var is a symlink to /private/var on macOS, so the two differ. Linux and Windows have no equivalent alias, which is why this is macOS-only.

• Compare canonicalized-to-canonicalized, or assert on the symlink-relative property rather than the absolute prefix. — fixed in fix(macos): make three portability findings correct on macOS #284.

4. macOS returns ENOTSUP for the readiness marker's rename — 2 tests

Os { code: 45, kind: Uncategorized, message: "Operation not supported" }

failed_marker_write_is_cleaned_up_and_existing_marker_is_preserved and marker_is_invisible_until_payload_is_complete. Both depend on an atomic no-replace rename for the marker's publish step. Linux's renameat2(RENAME_NOREPLACE) has no drop-in macOS equivalent, and the call is returning ENOTSUP rather than falling back.

This one is worth prioritising: it is the same publish-atomicity property that the wasm::worker output path relies on, so a no-op fallback would be a correctness problem rather than a test-only one.

• Confirm whether the production path reaches the same call, and whether a real Intel/ARM Mac returns ENOTSUP here or the guest's APFS does. — production: no, persist_noclobber has no production caller. The real-hardware half stays open as the checkbox in the correction comment below.
• If production is affected, this is a bug, not a test fix. — n/a.

5. APFS rejects invalid-UTF-8 file names with EILSEQ — 1 test

Os { code: 92, kind: Uncategorized, message: "Illegal byte sequence" }

tree_hash::invalid_utf8_file_names_fail_instead_of_colliding creates a file with invalid UTF-8 in its name and asserts the tree hash fails rather than colliding. On Linux arbitrary bytes are legal; on APFS the name cannot be created at all, so the test fails one step earlier than it asserts. The tree_hash property still needs to hold on macOS, it just cannot be provoked this way.

• Give the test a macOS path (inject the name at the hashing boundary rather than creating it on disk), or #[cfg]-gate the on-disk construction. — fixed in fix(macos): make three portability findings correct on macOS #284 (the test accepts the filesystem refusing the name).

Open question

Findings 2–5 would also fail on aarch64 macOS. We cannot currently tell whether they are Intel-specific, because macos-arm-tests is disabled. Once this lane has a green record it is worth deciding whether to flip ENABLE_MACOS_RUNNERS, which would give a second data point on the same tests.

Not in scope

EXCLUDE_ROOT (the guest runs as root, so a chmod 000 file is still readable), EXCLUDE_TTY (no controlling terminal), and EXCLUDE_VM_TIMING (two cores in a VM) are artifacts of the Recovery environment, not macOS findings, and are excluded for that reason.

🤖 Generated with Claude Code

[zackees]

One classification caveat, so the "not in scope" list is not read as settled.

Of the three tests excluded as EXCLUDE_VM_TIMING, two are unambiguous timing assertions:

• a_handful_of_threads_fills_a_small_ring_long_before_the_window_ends — "the session kept suspending threads for 4552731317 ns with a full ring"
• a_child_bound_to_another_owner_dies_when_that_owner_does — "the child must not outlive its owner: DeadlineElapsed"

The third is not clearly a timing artifact and I would not close it on that basis:

• real_worker_sequential_stress_leaves_no_parent_state — assertion left == right failed with left: ForcedContainment { trigger: Cancelled } vs right: Stopped(Cancelled)

That is a state mismatch, not an elapsed deadline: containment was reached by force rather than by a graceful stop. A slow guest could cause it, but so could a genuine difference in macOS containment semantics — and this is exactly the kind of signal the lane exists to produce. It is excluded from the lane so the lane can be green, but it belongs in the "investigate" column here, not the "artifact" column.

🤖 Generated with Claude Code

[zackees]

Correcting two claims I made in this issue, both mine and both now checked against the code rather than assumed.

1. Finding 4 is NOT production-affecting. I wrote that the readiness marker's rename "is the same publish-atomicity property that the wasm::worker output path relies on, so a no-op fallback would be a correctness problem rather than a test-only one." That is wrong. persist_noclobber has exactly one caller in the repository:

src/platform/fs/temporary.rs:77:  pub fn persist(self) -> PathBuf      (a different method)
tests/support/readiness_marker.rs:23:  .persist_noclobber(path)

There is no production caller. The worker output path uses plain std::fs::rename (replacing) plus symlink_metadata — it never asks for no-clobber. So finding 4 is a test-only issue, and the "prioritise this" framing I gave it was unfounded.

2. Finding 4 is probably not macOS-generic, contrary to the umbrella claim I made that findings 2–5 "would fail on any macOS host." tempfile 3.27's persist_noclobber calls rustix::fs::renameat_with(..., RenameFlags::NOREPLACE) (file/imp/unix.rs:117), which on macOS is renameatx_np with RENAME_EXCL. That is supported on APFS and HFS+, so a real Mac should not return ENOTSUP here. Note also tempfile's error handling: it falls through to a portable path only for NOSYS and INVAL, and Err(e) => return Err(e.into()) — so ENOTSUP is surfaced rather than degraded.

That points at the guest's filesystem (a virtio-backed Recovery volume, likely with a synthetic /tmp) rather than at macOS. I cannot settle it from Linux — it needs one command on real Intel hardware.

Findings 2, 3 and 5 remain macOS-generic on the evidence: the certificate-validity limit is Security-framework policy, /var → /private/var is a symlink present on every macOS install, and APFS enforces UTF-8 file names filesystem-wide. The umbrella sentence should have excluded 4 from that list, and I have moved it to the guest-artifact column pending the hardware check below.

• On a real Intel Mac: renameatx_np with RENAME_EXCL into $TMPDIR — does it return ENOTSUP or succeed?

🤖 Generated with Claude Code

[zackees]

Finding 1's first checkbox — report upstream — is done: sunfishcode/cap-std#427.

I verified the asymmetry by reading the twelve lines rather than restating my earlier analysis, and the report is built on that: src/rustix/fs/metadata_ext.rs:171 does rdev: u64::try_from(stat.st_rdev).unwrap(), while dev a few lines above carries both a sign guard and a comment acknowledging the platform difference ("platforms where it's unsigned since the first branch here will never be taken").

Two things I deliberately did not claim, because I cannot support them:

• No minimal reproduction. I hit this from a Linux host driving a macOS guest and have no Mac to narrow it to a filesystem or file. I said so in the report and framed it as a code asymmetry, which is verifiable by reading, rather than as a characterised trigger.
• No causal link to upstream feat(daemon-identity): owned identity records, explicit-endpoint probe, and verified daemon control #328, which is the same class in the same file (unguarded try_from, TryFromIntError, "assumes nonnegative but that's not always valid"). The dev guard may or may not have come from that fix, so I noted the resemblance without asserting rdev was missed by it.

The other two checkboxes on finding 1 — decide our mitigation (report-and-wait, pin a patched fork via [patch.crates-io], or drop cap-std from the worker-output path) — are a policy call I should not make unilaterally, since a fork carries a maintenance cost and dropping cap-std is a real refactor. Flagging it here for a decision rather than choosing.

Worth restating from the original analysis: the impact is not test-only. It panics inside symlink_metadata, so any cap-std consumer stat'ing such a file fails at runtime on macOS.

🤖 Generated with Claude Code

[zackees]

Pre-answered questions for the #298 burndown items on this issue

Burndown #298 lists four jobs here. Below are the questions an implementer would otherwise stop to ask, each with the answer to use. All of it was checked against main at 2ab662b on 2026-09-17. Follow these answers without asking. If a fact below turns out to be wrong when you do the work, stop and add a comment here instead of guessing.

---

A. Tick the resolved checkboxes (no code)

Which boxes?

• Finding 1, "Report upstream": tick. Done as cap-primitives 4.0.3 panics with TryFromIntError on a negative macOS st_rdev sunfishcode/cap-std#427.
• Finding 1, "Note this aborts any worker-output operation…": tick. It is a note, and the comment above records it.
• Finding 2, "Regenerate the fixtures…": tick. Fixed in fix(macos): make three portability findings correct on macOS #284 (validity is now 820 days, under macOS's 825-day limit).
• Finding 3, "Compare canonicalized-to-canonicalized…": tick. Fixed in fix(macos): make three portability findings correct on macOS #284 (tests/context_file_observation.rs canonicalizes both sides).
• Finding 4, "Confirm whether the production path reaches the same call…": tick, and append — no: persist_noclobber has no production caller (see comment).
• Finding 4, "If production is affected, this is a bug": tick, and append — n/a.
• Finding 5, "Give the test a macOS path…": tick. Fixed in fix(macos): make three portability findings correct on macOS #284 (tests/tree_hash.rs accepts a raw OS error when creating the file).
• Leave unticked: finding 1 "Decide our mitigation". It is still open.

Edit the body or add a comment? Both. Edit the body with gh issue edit 283 --body-file, starting from the current body so nothing else changes. Then post a one-line comment listing what was ticked, because body edits send no notification.

Close #283? No. Still open: the finding 1 mitigation, the finding 4 check on a real Mac, and the containment-state test in item C.

---

B. Fix the stale EXCLUDE_MACOS_RENAME comment

Where exactly? Only ci/macos-x64/recovery-guest.sh:60-61. ci/macos-x64/README.md:87 and :101 were already corrected and are the wording to copy. The header comment at recovery-guest.sh:~49 is also already correct.

Rename the variable to something like EXCLUDE_GUEST_RENAME? No. ci/test_macos_x64_guest.py:175 and the README both use the name. The real-hardware check might still show the cause is macOS after all, and a comment fix shouldn't touch the guard test.

Replacement text: say that the readiness marker's no-clobber publish returns ENOTSUP (45) in this guest; that tempfile asks for renameatx_np(RENAME_EXCL), which APFS/HFS+ support; that this is therefore most likely the guest's virtualized filesystem, not macOS; that it has not been verified on real hardware (#283); and that it is test-only because persist_noclobber has no production caller.

---

C. Move real_worker_sequential_stress_leaves_no_parent_state into its own group

Group name? EXCLUDE_CONTAINMENT_STATE. It has to match the guard's regex ^(EXCLUDE_[A-Z_]+)='.

Group comment? Record the observed mismatch: left: ForcedContainment { trigger: Cancelled } vs right: Stopped(Cancelled). Say it is a state difference, not an elapsed deadline, and is excluded pending investigation of possible macOS containment semantics (#283). Do not call it a guest artifact.

Files to change, all in one PR:

1. ci/macos-x64/recovery-guest.sh: remove the test from EXCLUDE_VM_TIMING. Change that group's comment from "a suspension window and two containment deadlines" to "a suspension window and a containment deadline". Add the new group, and append $EXCLUDE_CONTAINMENT_STATE at the end of EXCLUDED_TESTS=. The guard reads only the first 400 characters after EXCLUDED_TESTS=, and the line stays well under that.
2. ci/test_macos_x64_guest.py:173-179: add "EXCLUDE_CONTAINMENT_STATE" to expected. Keep the total at 19 (test_exclusions_are_named_and_unique), because the test moves between groups and is not added.
3. ci/macos-x64/README.md exclusion table: EXCLUDE_VM_TIMING 3 → 2 with matching wording, plus a new EXCLUDE_CONTAINMENT_STATE | 1 | row that says "investigate, not an artifact". The prose total ("19 further tests") stays the same.

Un-exclude it to get the signal instead? No. The lane is kept green on purpose; investigating the test is separate work.

Same PR as B? Yes. It is the same files and the same issue. Title: ci(macos-x64): correct the rename exclusion's cause and separate the containment-state test. Use Refs #283, not Closes.

How to validate? uv run --no-project -m unittest ci.test_native_proof ci.test_native_proof_jobs ci.test_macos_x64_guest (the same command as ci.yml:313) and bash -n ci/macos-x64/recovery-guest.sh. No macOS lane run is needed: the set of excluded tests does not change.

---

D. Send the rdev fix upstream to cap-std

What change? In cap-primitives/src/rustix/fs/metadata_ext.rs:171 on bytecodealliance/cap-std main, copy the dev guard directly above it:

rdev: if stat.st_rdev < 0 {
    i64::try_from(stat.st_rdev).unwrap() as u64
} else {
    u64::try_from(stat.st_rdev).unwrap()
},

Change the existing comment's first line to "The type of st_dev and st_rdev is dev_t…" rather than adding a second comment. The function already has #[allow(unused_comparisons)] (line 93, rust-lang/rust#115823), so the always-false < 0 check on unsigned platforms needs no new allow.

Is sign-extension the right meaning? Yes. It matches std: library/std/src/os/darwin/fs.rs:106 returns st_rdev as u64, which sign-extends an i32.

Fix other fields at the same time (size, blocks, blksize)? No. Keep the PR to the one confirmed bug; those are non-negative by contract.

Add a regression test? No. A negative st_rdev can't be produced portably without a special filesystem. Say so in the PR body and point to the dev precedent.

Which branch? main. Its cap-primitives version is 4.0.3, and there is no 4.x release branch. In the PR body, ask for a 4.0.4 patch release. Do not open a 3.x backport.

Mechanics? Run gh repo fork bytecodealliance/cap-std --clone (no zackees/cap-std fork exists yet), use branch fix/signed-rdev, commit Handle a negative st_rdev like st_dev, and put Fixes #427 in the PR body. Before pushing, run cargo fmt --check, cargo test -p cap-primitives on Linux, and soldr cargo check -p cap-primitives --target x86_64-apple-darwin (and aarch64-apple-darwin).

Add [patch.crates-io] in kernal-api pointing at the PR? No. That is the undecided mitigation question from finding 1 and is not part of this job.

🤖 Generated with Claude Code

[zackees]

Ticked the checkboxes that are resolved: finding 1's upstream report and impact note; findings 2, 3 and 5 (fixed in #284); finding 4's production question (no production caller). Still open: finding 1's mitigation decision, and the real-Mac renameatx_np check for finding 4. Tracked by #298.

[zackees]

Finding 1: the upstream fix is open as sunfishcode/cap-std#428 (Fixes #427). It copies the st_dev sign guard onto st_rdev; checked locally with fmt, the Linux cap-primitives tests, and macOS x86_64/aarch64 cross-checks. Merging it is up to cap-std's maintainers. The mitigation decision here (wait / [patch.crates-io] / drop cap-std) is still open.