Add SCCACHE_ALLOW_INCREMENTAL to keep caching the non-incremental parts of a build

README.md

@@ -374,7 +374,7 @@ Known Caveats
 ### Rust
 
 * Crates that invoke the system linker cannot be cached. This includes `bin`, `dylib`, `cdylib`, and `proc-macro` crates. You may be able to improve compilation time of large `bin` crates by converting them to a `lib` crate with a thin `bin` wrapper.
-* Incrementally compiled crates cannot be cached. By default, in the debug profile Cargo will use incremental compilation for workspace members and path dependencies. [You can disable incremental compilation.](https://doc.rust-lang.org/cargo/reference/profiles.html#incremental)
+* Incrementally compiled crates cannot be cached. By default, in the debug profile Cargo will use incremental compilation for workspace members and path dependencies. [You can disable incremental compilation.](https://doc.rust-lang.org/cargo/reference/profiles.html#incremental) sccache exits with an error if `CARGO_INCREMENTAL=1` is set; `SCCACHE_ALLOW_INCREMENTAL=1` lets the build proceed, leaving only those crates uncached.
 
 [More details on Rust caveats](/docs/Rust.md)
 

docs/Configuration.md

@@ -178,6 +178,7 @@ Note that some env variables may need sccache server restart to take effect.
 ### misc
 
 * `SCCACHE_ALLOW_CORE_DUMPS` to enable core dumps by the server
+* `SCCACHE_ALLOW_INCREMENTAL` set to `1` to allow Rust builds with `CARGO_INCREMENTAL=1`; incremental crates go uncached. See [Rust caveats](Rust.md)
 * `SCCACHE_CONF` configuration file path
 * `SCCACHE_BASEDIRS` base directory (or directories) to strip from paths for cache key computation. This is similar to ccache's `CCACHE_BASEDIR` and enables cache hits across different absolute paths when compiling the same source code. Multiple directories can be separated by `;` on Windows hosts and by `:` on any other operating system. When multiple directories are specified, the longest matching prefix is used. Path matching is **case-insensitive** on Windows and **case-sensitive** on other operating systems. Environment variable takes precedence over file configuration. Only absolute paths are supported; relative paths will cause an error and prevent the server from start.
 * `SCCACHE_CACHED_CONF`

docs/Rust.md

@@ -7,7 +7,7 @@ sccache includes support for caching Rust compilation. This includes many caveat
 * Compilation from stdin is not supported, a source file must be provided.
 * Values from `env!` require Rust >= 1.46 to be tracked in caching.
 * Procedural macros that read files from the filesystem may not be cached properly.
-* `rustc`'s incremental compilation needs to be disabled. See [The Cargo Book](https://doc.rust-lang.org/cargo/reference/profiles.html#incremental)
+* `rustc`'s incremental compilation needs to be disabled. See [The Cargo Book](https://doc.rust-lang.org/cargo/reference/profiles.html#incremental). sccache exits with an error when `CARGO_INCREMENTAL=1` is set, unless `SCCACHE_ALLOW_INCREMENTAL=1` is set: then only the incremental crates (workspace members and path dependencies) go uncached.
 * Crates that invoke the system linker cannot be cached. Examples are `bin`, `dylib`, `cdylib`, and `proc-macro` crates.
 
 If you are using Rust 1.18 or later, you can ask cargo to wrap all compilation with sccache by setting `RUSTC_WRAPPER=sccache` in your build environment.

src/commands.rs

@@ -891,13 +891,17 @@ pub fn run_command(cmd: Command) -> Result<i32> {
         } => {
             trace!("Command::Compile {{ {:?}, {:?}, {:?} }}", exe, cmdline, cwd);
 
+            // With the opt-in, `-C incremental=` compiles are left uncached (CannotCache).
+            let allow_incremental = env::var("SCCACHE_ALLOW_INCREMENTAL").as_deref() == Ok("1");
             let incr_env_strs = ["CARGO_BUILD_INCREMENTAL", "CARGO_INCREMENTAL"];
             incr_env_strs
                 .iter()
                 .for_each(|incr_str| match env::var(incr_str) {
-                    Ok(incr_val) if incr_val == "1" => {
+                    Ok(incr_val) if incr_val == "1" && !allow_incremental => {
                         println!(
-                            "sccache: incremental compilation is prohibited: Unset {} to continue.",
+                            "sccache: incremental compilation is prohibited: Unset {} to \
+                             continue, or set SCCACHE_ALLOW_INCREMENTAL=1 to keep caching \
+                             the non-incremental parts of the build.",
                             incr_str
                         );
                         std::process::exit(1);

tests/sccache_cargo.rs

@@ -409,3 +409,77 @@ fn test_rust_cargo_cmd_readonly_preemtive_block() -> Result<()> {
         .try_success()?;
     Ok(())
 }
+
+/// Test that without `SCCACHE_ALLOW_INCREMENTAL`, enabling cargo's incremental
+/// compilation still makes sccache refuse to run.
+#[test]
+#[serial]
+fn test_rust_cargo_incremental_refused_by_default() -> Result<()> {
+    let test_info = SccacheTest::new(None)?;
+    // Clean before setting the env var below: the refusal fires on any rustc
+    // invocation, including the `rustc -vV` probe `cargo clean` makes.
+    cargo_clean(&test_info)?;
+
+    Command::new(CARGO.as_os_str())
+        .args(["build", "--color=never"])
+        .envs(test_info.env.iter().cloned())
+        .env("CARGO_INCREMENTAL", "1")
+        // Explicitly off, so the default is still tested on a machine that opted
+        // in globally (`.env_remove` cannot undo cargo's `[env]` table).
+        .env("SCCACHE_ALLOW_INCREMENTAL", "0")
+        .current_dir(CRATE_DIR.as_os_str())
+        .assert()
+        .try_stderr(predicates::str::contains("incremental compilation is prohibited").from_utf8())?
+        .try_failure()?;
+
+    Ok(())
+}
+
+/// Test that with `SCCACHE_ALLOW_INCREMENTAL` an incremental build succeeds and
+/// splits per invocation: the workspace crates cargo hands `-C incremental=`
+/// bypass the cache, while its registry dependencies keep hitting it.
+#[test]
+#[serial]
+fn test_rust_cargo_incremental_allowed_deps_cached() -> Result<()> {
+    let test_info = SccacheTest::new(None)?;
+
+    // Two incremental builds, each from a clean target.
+    for _ in 0..2 {
+        cargo_clean(&test_info)?;
+        Command::new(CARGO.as_os_str())
+            .args(["build", "--color=never"])
+            .envs(test_info.env.iter().cloned())
+            .env("CARGO_INCREMENTAL", "1")
+            .env("SCCACHE_ALLOW_INCREMENTAL", "1")
+            .current_dir(CRATE_DIR.as_os_str())
+            .assert()
+            .try_success()?;
+    }
+
+    // The workspace lib really went through rustc's incremental machinery.
+    let incremental_dir = test_info.tempdir.path().join("cargo/debug/incremental");
+    let has_mylib_session = std::fs::read_dir(&incremental_dir)?
+        .filter_map(|e| e.ok())
+        .any(|e| e.file_name().to_string_lossy().starts_with("mylib"));
+    assert!(
+        has_mylib_session,
+        "expected an incremental session dir for `mylib` in {incremental_dir:?}"
+    );
+
+    test_info
+        .show_stats()?
+        // itoa: one miss (first build), one hit (second build).
+        .try_stdout(predicates::str::contains(r#""cache_hits":{"counts":{"Rust":1}"#).from_utf8())?
+        .try_stdout(
+            predicates::str::contains(
+                r#""cache_misses":{"counts":{"Rust":1},"adv_counts":{"rust":1}}"#,
+            )
+            .from_utf8(),
+        )?
+        // The workspace lib bypassed the cache in both builds. `mybin` is rejected
+        // on `--crate-type bin` first, which cargo passes ahead of the flag.
+        .try_stdout(predicates::str::contains(r#""incremental":2"#).from_utf8())?
+        .try_success()?;
+
+    Ok(())
+}