Skip to content

Optimize child process spawns by skipping job server configs - #2827

Merged
sylvestre merged 1 commit into
mozilla:mainfrom
rnk:jobserver-posix-spawn
Sep 14, 2026
Merged

sylvestre merged 1 commit into
mozilla:mainfrom
rnk:jobserver-posix-spawn

Conversation

@rnk

@rnk rnk commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Configuring the job server so that all child processes can access it causes Rust's subprocess library to switch from posix_spawn to fork, which supports running arbitrary code in a pre-exec context. Fork is, unfortunately, very expensive. You really want to use vfork or posix_spawn, which IIRC uses that under the hood, in order to get fast process launching.

This really doesn't matter in the grand scheme of things because sccache direct mode (the defatult) doesn't launch a ton of processes when it gets a cache hit, but if you turn it off, it will launch may pre-processing jobs, and the overhead is observable in a profiler, which is how I (well, Claude) found this:

Measured on a 2456-file LLVM rebuild (X86 only, Release, clang, -j16,
16 cores) where every compile is a cache hit, so the preprocessor spawn is
essentially all the server does. Four interleaved rounds per arm:

| arm    | wall             | server CPU       | of which system |
|--------|------------------|------------------|-----------------|
| before | 40.7 s (+-0.7)   | 28.2 s (+-0.4)   | 22.8 s          |
| after  | 33.5 s (+-0.2)   | 10.6 s (+-0.1)   |  5.8 s          |

It seemed like a reasonably small contribution that might make a good first PR.

The code is AI generated, and I'll take another pass to try to simplify it. I think it has the wrong job server default.

@rnk
rnk force-pushed the jobserver-posix-spawn branch from 2d2b264 to 38a715b Compare August 28, 2026 05:50
@sylvestre

Copy link
Copy Markdown
Collaborator

Nice to see you here :)

@codecov-commenter

codecov-commenter commented Aug 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.88889% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.73%. Comparing base (8ab3926) to head (3e3d27e).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/mock_command.rs 85.71% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2827      +/-   ##
==========================================
+ Coverage   73.71%   73.73%   +0.01%     
==========================================
  Files          72       72              
  Lines       37932    37976      +44     
==========================================
+ Hits        27963    28002      +39     
- Misses       9969     9974       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

`AsyncCommand::spawn` calls `jobserver::Client::configure` on every child it
spawns. On Unix that registers a `pre_exec` closure to clear `CLOEXEC` on the
jobserver's two file descriptors -- and the presence of *any* `pre_exec` makes
`std` abandon its `posix_spawn` fast path and fall back to `fork` + `exec`.

That is a bad trade for a server process. `fork` duplicates the parent's page
tables, and sccache's whole job is to be a long-lived process holding a large
cache; the child then throws the copy away microseconds later in `exec`. The
cost scales with how much memory the server has touched, so it grows over the
life of a build.

An empty closure is enough to trigger it:

    let mut c = Command::new("/bin/true");
    if pre_exec { unsafe { c.pre_exec(|| Ok(())); } }
    c.spawn()

    without:  clone3({flags=CLONE_VM|CLONE_VFORK|CLONE_CLEAR_SIGHAND, ...})
    with:     clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|...|SIGCHLD)

`CLONE_VM|CLONE_VFORK` shares the address space and copies nothing. The second
form is a real fork, and it is what shows up in a profile as `dup_mmap` ->
`copy_page_range` -> `copy_pte_range`.

Almost nothing sccache spawns can use a jobserver. It reaches a child through
`CARGO_MAKEFLAGS`, and only `rustc` reads it: preprocessors, version probes and
C/C++ compilers all ignore it. So stop sharing by default and let the callers
that spawn `rustc` ask, via `RunCommand::share_jobserver`.

For the local compile, which is one code path shared by every frontend, the
opt-in is a field on `SingleCompileCommand` rather than a builder call. That
makes the compiler ask every frontend the question, so a new one cannot get it
wrong by omission -- and getting it wrong in this direction is what matters,
since a `rustc` without a jobserver spawns as many codegen threads as there
are CPUs, per concurrent `rustc`, which is the oversubscription the jobserver
exists to prevent.

Measured on a 2456-file LLVM build (X86 only, Release, clang, `-j16`, 16
cores). Two scenarios, interleaved rounds:

All compiles are cache hits, with `SCCACHE_DIRECT=false` so the preprocessor
still runs -- this isolates the spawn cost, since it is nearly all the server
does (4 rounds):

    | arm    | wall             | server CPU       | of which system |
    |--------|------------------|------------------|-----------------|
    | before | 40.7 s (+-0.7)   | 28.2 s (+-0.4)   | 22.8 s          |
    | after  | 33.5 s (+-0.2)   | 10.6 s (+-0.1)   |  5.8 s          |

All compiles are cache misses, so each one both preprocesses and compiles
(2 rounds):

    | arm    | wall             | server CPU       | of which system |
    |--------|------------------|------------------|-----------------|
    | before | 336.5 s          | 96.3 s           | 45.8 s          |
    | after  | 333.1 s          | 70.1 s           | 21.2 s          |

The saving is almost entirely system time, which is what a page-table copy
costs. Wall clock barely moves on the miss build because it is dominated by the
compiler itself; the win there is 26 s of a core given back, not a faster
build.

`strace` on a 1300-compile miss build confirms the mechanism, and shows why
both spawn sites had to be covered:

    | build            | fork | posix_spawn |
    |------------------|------|-------------|
    | before           | 2617 |           0 |
    | preprocess only  | 1319 |        1300 |
    | this change      |    0 |        2619 |

Under `perf record -p <server>` on the cache-hit build, the address-space
symbols (`copy_pte_range`, `copy_present_ptes`, `zap_pte_range`,
`smp_call_function_many_cond` and friends) go from 4.20% of the server's
samples to 0.05%.

One behaviour change worth naming: the client-side fallback in `commands.rs`,
which runs the compiler itself when the server declines the job, no longer
shares its jobserver either. That jobserver is private to a single short-lived
client process with exactly one child, so it never limited anything across
compiles; the server's jobserver is the one that does real work.
@rnk
rnk force-pushed the jobserver-posix-spawn branch from 38a715b to 3e3d27e Compare August 28, 2026 18:22
@rnk
rnk marked this pull request as ready for review August 28, 2026 18:50
@sylvestre
sylvestre merged commit 01c35e6 into mozilla:main Sep 14, 2026
51 checks passed
lkwilson pushed a commit to lkwilson/red that referenced this pull request Sep 23, 2026
This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [mozilla/sccache](https://github.com/mozilla/sccache) | minor | `0.17.0` → `0.18.0` |

---

### Release Notes

<details>
<summary>mozilla/sccache (mozilla/sccache)</summary>

### [`v0.18.0`](https://github.com/mozilla/sccache/releases/tag/v0.18.0)

[Compare Source](mozilla/sccache@v0.17.0...v0.18.0)

##### sccache 0.18.0

##### Summary

sccache 0.18.0 is a broad correctness and coverage release: a lot of compiler flags that used to
force a cache miss are now understood, several cache-key and multi-level storage bugs are fixed,
and the Azure backend gained passwordless authentication.

Highlights:

- **Cache-key correctness**: the assembler gcc/clang would actually invoke is now part of the cache
  key, so two toolchains with different binutils can no longer hand each other the wrong object
  file ([#&#8203;2843](mozilla/sccache#2843)). `SCCACHE_BASEDIRS` now also strips base directories from the compiler arguments,
  so flags like `-ffile-prefix-map=/home/user/project=.` stop tying an entry to one checkout
  ([#&#8203;2840](mozilla/sccache#2840)).
- **MSVC/clang flag coverage**: support for `/openmp:llvm`, the `/fsanitize*`, `/fsanitize-coverage*`
  and `/fno-sanitize*` families, `/feature`, arm64EC and fastfail, `/d20bforceinline`, and a large
  batch of other flags ([#&#8203;2807](mozilla/sccache#2807), [#&#8203;2830](mozilla/sccache#2830), [#&#8203;2831](mozilla/sccache#2831), [#&#8203;2832](mozilla/sccache#2832)), plus more clang CLI options ([#&#8203;2834](mozilla/sccache#2834)). gcc now
  marks flags as `TooHard` when they would require caching something else ([#&#8203;2833](mozilla/sccache#2833)).
- **CUDA**: `nvcc` dryrun parsing works with CUDA 13.3 ([#&#8203;2722](mozilla/sccache#2722)), escaped quotes in Windows dryrun
  lines are protected before backslash flattening ([#&#8203;2811](mozilla/sccache#2811)), and the
  `--diag-error`/`--diag-suppress`/`--diag-warn` family is accepted ([#&#8203;2816](mozilla/sccache#2816)).
- **Multi-level cache**: a chain with any writable level is writable again — a single read-only
  level no longer makes the whole storage read-only ([#&#8203;2778](mozilla/sccache#2778)) — and reads are no longer issued twice
  ([#&#8203;2835](mozilla/sccache#2835)).
- **Azure**: Microsoft Entra ID (passwordless) authentication for the Azure Blob backend, for
  storage accounts that disable shared-key access ([#&#8203;2802](mozilla/sccache#2802)).
- **Distributed compilation**: rlibs are no longer trimmed from crates that also emit a cdylib,
  which made those jobs fail on the build server ([#&#8203;2839](mozilla/sccache#2839)).
- **Process handling**: the jobserver is no longer handed to children that can't use it, which
  restores the `posix_spawn` fast path for child spawns ([#&#8203;2827](mozilla/sccache#2827)), and daemonization now uses an
  allow-list for inherited file descriptors, fixing `make` deadlocks caused by leaked jobserver FDs
  ([#&#8203;2841](mozilla/sccache#2841)).
- **MSRV** is now 1.91.0 ([#&#8203;2793](mozilla/sccache#2793)).

Welcome to 11 new contributors!

##### Features

- feat: add Microsoft Entra ID (passwordless) auth for the Azure Blob backend by [@&#8203;babrekel](https://github.com/babrekel) in [#&#8203;2802](mozilla/sccache#2802)
- gcc, clang: make the assembler part of the cache key by [@&#8203;glandium](https://github.com/glandium) in [#&#8203;2843](mozilla/sccache#2843)
- Strip basedirs from the compiler arguments too by [@&#8203;avikivity](https://github.com/avikivity) in [#&#8203;2840](mozilla/sccache#2840)
- cache: allow skipping capability checks by [@&#8203;Xuanwo](https://github.com/Xuanwo) in [#&#8203;2822](mozilla/sccache#2822)
- Optimize child process spawns by skipping job server configs by [@&#8203;rnk](https://github.com/rnk) in [#&#8203;2827](mozilla/sccache#2827)

##### Compiler support

- feat: add support for MSVC's /d20bforceinline. by [@&#8203;AJIOB](https://github.com/AJIOB) in [#&#8203;2807](mozilla/sccache#2807)
- msvc: support arm64EC and fastfail flags by [@&#8203;AJIOB](https://github.com/AJIOB) in [#&#8203;2830](mozilla/sccache#2830)
- msvc: backport flags support by [@&#8203;AJIOB](https://github.com/AJIOB) in [#&#8203;2831](mozilla/sccache#2831)
- msvc: support lots of flags by [@&#8203;AJIOB](https://github.com/AJIOB) in [#&#8203;2832](mozilla/sccache#2832)
- clang: support more CLI options by [@&#8203;AJIOB](https://github.com/AJIOB) in [#&#8203;2834](mozilla/sccache#2834)
- gcc: mark flags as TooHard if need to cache something else by [@&#8203;AJIOB](https://github.com/AJIOB) in [#&#8203;2833](mozilla/sccache#2833)
- nvcc: accept the --diag-error/--diag-suppress/--diag-warn family by [@&#8203;Kataglyphis](https://github.com/Kataglyphis) in [#&#8203;2816](mozilla/sccache#2816)

##### Fixes

- Fix `nvcc` dryrun parsing for CUDA 13.3 by [@&#8203;mbrobbel](https://github.com/mbrobbel) in [#&#8203;2722](mozilla/sccache#2722)
- nvcc (Windows): protect escaped quotes in dryrun lines before flattening backslashes by [@&#8203;Kataglyphis](https://github.com/Kataglyphis) in [#&#8203;2811](mozilla/sccache#2811)
- multilevel: a chain with any writable level is writable by [@&#8203;tycho](https://github.com/tycho) in [#&#8203;2778](mozilla/sccache#2778)
- cache: avoid duplicate multilevel reads by [@&#8203;zfaustk](https://github.com/zfaustk) in [#&#8203;2835](mozilla/sccache#2835)
- Resolving always-false conditional making some compilers miss for preprocessor checks by [@&#8203;Joldiges](https://github.com/Joldiges) in [#&#8203;2824](mozilla/sccache#2824)
- dist: refuse to trim rlibs from crates that also emit a cdylib by [@&#8203;gaetschwartz](https://github.com/gaetschwartz) in [#&#8203;2839](mozilla/sccache#2839)
- daemonize: use an allow-list approach to inherited FDs by [@&#8203;schopin-mozilla](https://github.com/schopin-mozilla) in [#&#8203;2841](mozilla/sccache#2841)

##### Dependencies

- Bump opendal to 0.58.1 and fix fallout (fixes local GCS cache usage) by [@&#8203;flip1995](https://github.com/flip1995) in [#&#8203;2715](mozilla/sccache#2715)
- Update shlex dependency to version 2 by [@&#8203;musicinmybrain](https://github.com/musicinmybrain) in [#&#8203;2836](mozilla/sccache#2836)
- Bump MSRV to 1.91.0 by [@&#8203;flip1995](https://github.com/flip1995) in [#&#8203;2793](mozilla/sccache#2793)

##### Tests & CI

- Fixed hardcoded binary path in test by [@&#8203;ranger-ross](https://github.com/ranger-ross) in [#&#8203;2790](mozilla/sccache#2790)
- ci: dump sccache logs on integration failures by [@&#8203;JalenBuildsHub](https://github.com/JalenBuildsHub) in [#&#8203;2851](mozilla/sccache#2851)
- tests/integration: replace MinIO with Silo by [@&#8203;avikivity](https://github.com/avikivity) in [#&#8203;2853](mozilla/sccache#2853)

##### Cleanup

- Add an agent file by [@&#8203;sylvestre](https://github.com/sylvestre) in [#&#8203;2812](mozilla/sccache#2812)
- chore: fix typo in comment by [@&#8203;AJIOB](https://github.com/AJIOB) in [#&#8203;2829](mozilla/sccache#2829)
- Release 0.18.0 by [@&#8203;sylvestre](https://github.com/sylvestre) in [#&#8203;2848](mozilla/sccache#2848)

##### New Contributors

- [@&#8203;mbrobbel](https://github.com/mbrobbel) made their first contribution in [#&#8203;2722](mozilla/sccache#2722)
- [@&#8203;flip1995](https://github.com/flip1995) made their first contribution in [#&#8203;2793](mozilla/sccache#2793)
- [@&#8203;babrekel](https://github.com/babrekel) made their first contribution in [#&#8203;2802](mozilla/sccache#2802)
- [@&#8203;Kataglyphis](https://github.com/Kataglyphis) made their first contribution in [#&#8203;2811](mozilla/sccache#2811)
- [@&#8203;schopin-mozilla](https://github.com/schopin-mozilla) made their first contribution in [#&#8203;2841](mozilla/sccache#2841)
- [@&#8203;Joldiges](https://github.com/Joldiges) made their first contribution in [#&#8203;2824](mozilla/sccache#2824)
- [@&#8203;tycho](https://github.com/tycho) made their first contribution in [#&#8203;2778](mozilla/sccache#2778)
- [@&#8203;JalenBuildsHub](https://github.com/JalenBuildsHub) made their first contribution in [#&#8203;2851](mozilla/sccache#2851)
- [@&#8203;zfaustk](https://github.com/zfaustk) made their first contribution in [#&#8203;2835](mozilla/sccache#2835)
- [@&#8203;gaetschwartz](https://github.com/gaetschwartz) made their first contribution in [#&#8203;2839](mozilla/sccache#2839)
- [@&#8203;rnk](https://github.com/rnk) made their first contribution in [#&#8203;2827](mozilla/sccache#2827)

**Full Changelog**: <mozilla/sccache@v0.17.0...v0.18.0>

</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC40LjUiLCJ1cGRhdGVkSW5WZXIiOiI0NC40LjUiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbXX0=-->

---------

Co-authored-by: Renovate Bot <renovate@endsy.me>
Reviewed-on: https://gitea.endsy.me/op/red/pulls/30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants