Optimize child process spawns by skipping job server configs - #2827
Merged
Merged
Conversation
rnk
force-pushed
the
jobserver-posix-spawn
branch
from
August 28, 2026 05:50
2d2b264 to
38a715b
Compare
Collaborator
|
Nice to see you here :) |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2827 +/- ##
==========================================
+ Coverage 73.71% 73.73% +0.01%
==========================================
Files 72 72
Lines 37932 37976 +44
==========================================
+ Hits 27963 28002 +39
- Misses 9969 9974 +5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
`AsyncCommand::spawn` calls `jobserver::Client::configure` on every child it
spawns. On Unix that registers a `pre_exec` closure to clear `CLOEXEC` on the
jobserver's two file descriptors -- and the presence of *any* `pre_exec` makes
`std` abandon its `posix_spawn` fast path and fall back to `fork` + `exec`.
That is a bad trade for a server process. `fork` duplicates the parent's page
tables, and sccache's whole job is to be a long-lived process holding a large
cache; the child then throws the copy away microseconds later in `exec`. The
cost scales with how much memory the server has touched, so it grows over the
life of a build.
An empty closure is enough to trigger it:
let mut c = Command::new("/bin/true");
if pre_exec { unsafe { c.pre_exec(|| Ok(())); } }
c.spawn()
without: clone3({flags=CLONE_VM|CLONE_VFORK|CLONE_CLEAR_SIGHAND, ...})
with: clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|...|SIGCHLD)
`CLONE_VM|CLONE_VFORK` shares the address space and copies nothing. The second
form is a real fork, and it is what shows up in a profile as `dup_mmap` ->
`copy_page_range` -> `copy_pte_range`.
Almost nothing sccache spawns can use a jobserver. It reaches a child through
`CARGO_MAKEFLAGS`, and only `rustc` reads it: preprocessors, version probes and
C/C++ compilers all ignore it. So stop sharing by default and let the callers
that spawn `rustc` ask, via `RunCommand::share_jobserver`.
For the local compile, which is one code path shared by every frontend, the
opt-in is a field on `SingleCompileCommand` rather than a builder call. That
makes the compiler ask every frontend the question, so a new one cannot get it
wrong by omission -- and getting it wrong in this direction is what matters,
since a `rustc` without a jobserver spawns as many codegen threads as there
are CPUs, per concurrent `rustc`, which is the oversubscription the jobserver
exists to prevent.
Measured on a 2456-file LLVM build (X86 only, Release, clang, `-j16`, 16
cores). Two scenarios, interleaved rounds:
All compiles are cache hits, with `SCCACHE_DIRECT=false` so the preprocessor
still runs -- this isolates the spawn cost, since it is nearly all the server
does (4 rounds):
| arm | wall | server CPU | of which system |
|--------|------------------|------------------|-----------------|
| before | 40.7 s (+-0.7) | 28.2 s (+-0.4) | 22.8 s |
| after | 33.5 s (+-0.2) | 10.6 s (+-0.1) | 5.8 s |
All compiles are cache misses, so each one both preprocesses and compiles
(2 rounds):
| arm | wall | server CPU | of which system |
|--------|------------------|------------------|-----------------|
| before | 336.5 s | 96.3 s | 45.8 s |
| after | 333.1 s | 70.1 s | 21.2 s |
The saving is almost entirely system time, which is what a page-table copy
costs. Wall clock barely moves on the miss build because it is dominated by the
compiler itself; the win there is 26 s of a core given back, not a faster
build.
`strace` on a 1300-compile miss build confirms the mechanism, and shows why
both spawn sites had to be covered:
| build | fork | posix_spawn |
|------------------|------|-------------|
| before | 2617 | 0 |
| preprocess only | 1319 | 1300 |
| this change | 0 | 2619 |
Under `perf record -p <server>` on the cache-hit build, the address-space
symbols (`copy_pte_range`, `copy_present_ptes`, `zap_pte_range`,
`smp_call_function_many_cond` and friends) go from 4.20% of the server's
samples to 0.05%.
One behaviour change worth naming: the client-side fallback in `commands.rs`,
which runs the compiler itself when the server declines the job, no longer
shares its jobserver either. That jobserver is private to a single short-lived
client process with exactly one child, so it never limited anything across
compiles; the server's jobserver is the one that does real work.
rnk
force-pushed
the
jobserver-posix-spawn
branch
from
August 28, 2026 18:22
38a715b to
3e3d27e
Compare
rnk
marked this pull request as ready for review
August 28, 2026 18:50
lkwilson
pushed a commit
to lkwilson/red
that referenced
this pull request
Sep 23, 2026
This PR contains the following updates: | Package | Update | Change | |---|---|---| | [mozilla/sccache](https://github.com/mozilla/sccache) | minor | `0.17.0` → `0.18.0` | --- ### Release Notes <details> <summary>mozilla/sccache (mozilla/sccache)</summary> ### [`v0.18.0`](https://github.com/mozilla/sccache/releases/tag/v0.18.0) [Compare Source](mozilla/sccache@v0.17.0...v0.18.0) ##### sccache 0.18.0 ##### Summary sccache 0.18.0 is a broad correctness and coverage release: a lot of compiler flags that used to force a cache miss are now understood, several cache-key and multi-level storage bugs are fixed, and the Azure backend gained passwordless authentication. Highlights: - **Cache-key correctness**: the assembler gcc/clang would actually invoke is now part of the cache key, so two toolchains with different binutils can no longer hand each other the wrong object file ([#​2843](mozilla/sccache#2843)). `SCCACHE_BASEDIRS` now also strips base directories from the compiler arguments, so flags like `-ffile-prefix-map=/home/user/project=.` stop tying an entry to one checkout ([#​2840](mozilla/sccache#2840)). - **MSVC/clang flag coverage**: support for `/openmp:llvm`, the `/fsanitize*`, `/fsanitize-coverage*` and `/fno-sanitize*` families, `/feature`, arm64EC and fastfail, `/d20bforceinline`, and a large batch of other flags ([#​2807](mozilla/sccache#2807), [#​2830](mozilla/sccache#2830), [#​2831](mozilla/sccache#2831), [#​2832](mozilla/sccache#2832)), plus more clang CLI options ([#​2834](mozilla/sccache#2834)). gcc now marks flags as `TooHard` when they would require caching something else ([#​2833](mozilla/sccache#2833)). - **CUDA**: `nvcc` dryrun parsing works with CUDA 13.3 ([#​2722](mozilla/sccache#2722)), escaped quotes in Windows dryrun lines are protected before backslash flattening ([#​2811](mozilla/sccache#2811)), and the `--diag-error`/`--diag-suppress`/`--diag-warn` family is accepted ([#​2816](mozilla/sccache#2816)). - **Multi-level cache**: a chain with any writable level is writable again — a single read-only level no longer makes the whole storage read-only ([#​2778](mozilla/sccache#2778)) — and reads are no longer issued twice ([#​2835](mozilla/sccache#2835)). - **Azure**: Microsoft Entra ID (passwordless) authentication for the Azure Blob backend, for storage accounts that disable shared-key access ([#​2802](mozilla/sccache#2802)). - **Distributed compilation**: rlibs are no longer trimmed from crates that also emit a cdylib, which made those jobs fail on the build server ([#​2839](mozilla/sccache#2839)). - **Process handling**: the jobserver is no longer handed to children that can't use it, which restores the `posix_spawn` fast path for child spawns ([#​2827](mozilla/sccache#2827)), and daemonization now uses an allow-list for inherited file descriptors, fixing `make` deadlocks caused by leaked jobserver FDs ([#​2841](mozilla/sccache#2841)). - **MSRV** is now 1.91.0 ([#​2793](mozilla/sccache#2793)). Welcome to 11 new contributors! ##### Features - feat: add Microsoft Entra ID (passwordless) auth for the Azure Blob backend by [@​babrekel](https://github.com/babrekel) in [#​2802](mozilla/sccache#2802) - gcc, clang: make the assembler part of the cache key by [@​glandium](https://github.com/glandium) in [#​2843](mozilla/sccache#2843) - Strip basedirs from the compiler arguments too by [@​avikivity](https://github.com/avikivity) in [#​2840](mozilla/sccache#2840) - cache: allow skipping capability checks by [@​Xuanwo](https://github.com/Xuanwo) in [#​2822](mozilla/sccache#2822) - Optimize child process spawns by skipping job server configs by [@​rnk](https://github.com/rnk) in [#​2827](mozilla/sccache#2827) ##### Compiler support - feat: add support for MSVC's /d20bforceinline. by [@​AJIOB](https://github.com/AJIOB) in [#​2807](mozilla/sccache#2807) - msvc: support arm64EC and fastfail flags by [@​AJIOB](https://github.com/AJIOB) in [#​2830](mozilla/sccache#2830) - msvc: backport flags support by [@​AJIOB](https://github.com/AJIOB) in [#​2831](mozilla/sccache#2831) - msvc: support lots of flags by [@​AJIOB](https://github.com/AJIOB) in [#​2832](mozilla/sccache#2832) - clang: support more CLI options by [@​AJIOB](https://github.com/AJIOB) in [#​2834](mozilla/sccache#2834) - gcc: mark flags as TooHard if need to cache something else by [@​AJIOB](https://github.com/AJIOB) in [#​2833](mozilla/sccache#2833) - nvcc: accept the --diag-error/--diag-suppress/--diag-warn family by [@​Kataglyphis](https://github.com/Kataglyphis) in [#​2816](mozilla/sccache#2816) ##### Fixes - Fix `nvcc` dryrun parsing for CUDA 13.3 by [@​mbrobbel](https://github.com/mbrobbel) in [#​2722](mozilla/sccache#2722) - nvcc (Windows): protect escaped quotes in dryrun lines before flattening backslashes by [@​Kataglyphis](https://github.com/Kataglyphis) in [#​2811](mozilla/sccache#2811) - multilevel: a chain with any writable level is writable by [@​tycho](https://github.com/tycho) in [#​2778](mozilla/sccache#2778) - cache: avoid duplicate multilevel reads by [@​zfaustk](https://github.com/zfaustk) in [#​2835](mozilla/sccache#2835) - Resolving always-false conditional making some compilers miss for preprocessor checks by [@​Joldiges](https://github.com/Joldiges) in [#​2824](mozilla/sccache#2824) - dist: refuse to trim rlibs from crates that also emit a cdylib by [@​gaetschwartz](https://github.com/gaetschwartz) in [#​2839](mozilla/sccache#2839) - daemonize: use an allow-list approach to inherited FDs by [@​schopin-mozilla](https://github.com/schopin-mozilla) in [#​2841](mozilla/sccache#2841) ##### Dependencies - Bump opendal to 0.58.1 and fix fallout (fixes local GCS cache usage) by [@​flip1995](https://github.com/flip1995) in [#​2715](mozilla/sccache#2715) - Update shlex dependency to version 2 by [@​musicinmybrain](https://github.com/musicinmybrain) in [#​2836](mozilla/sccache#2836) - Bump MSRV to 1.91.0 by [@​flip1995](https://github.com/flip1995) in [#​2793](mozilla/sccache#2793) ##### Tests & CI - Fixed hardcoded binary path in test by [@​ranger-ross](https://github.com/ranger-ross) in [#​2790](mozilla/sccache#2790) - ci: dump sccache logs on integration failures by [@​JalenBuildsHub](https://github.com/JalenBuildsHub) in [#​2851](mozilla/sccache#2851) - tests/integration: replace MinIO with Silo by [@​avikivity](https://github.com/avikivity) in [#​2853](mozilla/sccache#2853) ##### Cleanup - Add an agent file by [@​sylvestre](https://github.com/sylvestre) in [#​2812](mozilla/sccache#2812) - chore: fix typo in comment by [@​AJIOB](https://github.com/AJIOB) in [#​2829](mozilla/sccache#2829) - Release 0.18.0 by [@​sylvestre](https://github.com/sylvestre) in [#​2848](mozilla/sccache#2848) ##### New Contributors - [@​mbrobbel](https://github.com/mbrobbel) made their first contribution in [#​2722](mozilla/sccache#2722) - [@​flip1995](https://github.com/flip1995) made their first contribution in [#​2793](mozilla/sccache#2793) - [@​babrekel](https://github.com/babrekel) made their first contribution in [#​2802](mozilla/sccache#2802) - [@​Kataglyphis](https://github.com/Kataglyphis) made their first contribution in [#​2811](mozilla/sccache#2811) - [@​schopin-mozilla](https://github.com/schopin-mozilla) made their first contribution in [#​2841](mozilla/sccache#2841) - [@​Joldiges](https://github.com/Joldiges) made their first contribution in [#​2824](mozilla/sccache#2824) - [@​tycho](https://github.com/tycho) made their first contribution in [#​2778](mozilla/sccache#2778) - [@​JalenBuildsHub](https://github.com/JalenBuildsHub) made their first contribution in [#​2851](mozilla/sccache#2851) - [@​zfaustk](https://github.com/zfaustk) made their first contribution in [#​2835](mozilla/sccache#2835) - [@​gaetschwartz](https://github.com/gaetschwartz) made their first contribution in [#​2839](mozilla/sccache#2839) - [@​rnk](https://github.com/rnk) made their first contribution in [#​2827](mozilla/sccache#2827) **Full Changelog**: <mozilla/sccache@v0.17.0...v0.18.0> </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC40LjUiLCJ1cGRhdGVkSW5WZXIiOiI0NC40LjUiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbXX0=--> --------- Co-authored-by: Renovate Bot <renovate@endsy.me> Reviewed-on: https://gitea.endsy.me/op/red/pulls/30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Configuring the job server so that all child processes can access it causes Rust's subprocess library to switch from
posix_spawntofork, which supports running arbitrary code in a pre-exec context. Fork is, unfortunately, very expensive. You really want to use vfork or posix_spawn, which IIRC uses that under the hood, in order to get fast process launching.This really doesn't matter in the grand scheme of things because sccache direct mode (the defatult) doesn't launch a ton of processes when it gets a cache hit, but if you turn it off, it will launch may pre-processing jobs, and the overhead is observable in a profiler, which is how I (well, Claude) found this:
It seemed like a reasonably small contribution that might make a good first PR.
The code is AI generated, and I'll take another pass to try to simplify it. I think it has the wrong job server default.