sccache-dist: Docker builder hangs indefinitely — `docker cp -` is never sent EOF

[oguzhanmeteozturk]

A build server configured with [builder] type = "docker" never completes a job. The first docker cp - <container>:/ invocation blocks forever, so the build server holds the job until the client times out and falls back to local compilation.

CommandExt::check_piped (src/bin/sccache-dist/build.rs) moves the child's stdin out of the Child and keeps it alive across wait_with_output(). The write end of the pipe therefore stays open while the parent waits for the child to exit, and docker cp - does not exit until it sees EOF on stdin.

Environment

• sccache v0.17.0 (sccache-dist built with --features dist-server)
• Code path unchanged at 67683cc (current main at time of writing)
• Build servers running Docker 27.5.1 (where the hang was observed)
• The docker cp behaviour below re-confirmed on Docker 29.7.2, the current release — it is not specific to an old client
• Affects BuilderType::Docker only; the overlay and pot builders do not use check_piped

Steps to reproduce

1. Run a scheduler and a build server, with the server configured as:

[builder]
type = "docker"

2. From a client with [dist] scheduler_url pointing at the scheduler, compile any single C file:

sccache gcc -c hello.c -o hello.o

Expected

The job is built in the container and the object returned to the client.

Actual

The build server makes no progress. The target container remains in Created, a docker cp process remains alive indefinitely, and no job ever completes. The client eventually reports a distributed failure and compiles locally:

Could not perform distributed compile, falling back to local

Analysis

// src/bin/sccache-dist/build.rs:48
fn check_piped(&mut self, pipe: &mut dyn FnMut(&mut ChildStdin) -> Result<()>) -> Result<()> {
    let mut process = self
        .stdin(Stdio::piped())
        .spawn()
        .context("Failed to start command")?;
    let mut stdin = process
        .stdin
        .take()
        .expect("Requested piped stdin but not present");
    pipe(&mut stdin).context("Failed to pipe input to process")?;
    let output = process
        .wait_with_output()
        .context("Failed to wait for process to return")?;
    check_output(&output)
}

stdin is an owned ChildStdin produced by .take(). It is dropped at the end of the function's scope, which is after wait_with_output() returns.

Child::wait_with_output closes only the stdin still held by the Child:

pub fn wait_with_output(mut self) -> io::Result<Output> {
    drop(self.stdin.take());
    ...
}

Because check_piped already took it, self.stdin is None and that drop is a no-op. The pipe's write end remains open for the duration of the wait.

The closure cannot close it either: its parameter is &mut ChildStdin, so the callee has no way to drop the value. Only check_piped can.

docker cp - requires EOF, not just a complete archive

This can be confirmed without sccache. A complete, valid tar is written, then the write end is held open:

docker run --rm alpine sh -c 'mkdir -p /s && echo hi > /s/f && tar -cf - -C /s f' > /tmp/t.tar
docker run --rm -i alpine tar -tf - < /tmp/t.tar     # archive is valid and complete

docker create --name cptest alpine true

# control — writer closes immediately
time ( cat /tmp/t.tar ) | docker cp - cptest:/          # returns in ~0s

# writer stays open for 20s after the complete archive
time ( cat /tmp/t.tar; sleep 20 ) | docker cp - cptest:/  # returns after 20s

Observed on Docker 29.7.2, timing docker cp itself:

control:    docker cp - cptest:/   0.019 total
held open:  docker cp - cptest:/  20.013 total

docker cp - waits for the writer to close even when it already holds a complete archive. The same result was obtained on an earlier client, so this is long-standing behavior rather than a regression in a particular release.

Affected call sites

Both are in DockerBuilder:

• build.rs:720 — make_image, copying the toolchain tar
  (Failed to copy toolchain tar into container)
• build.rs:771 — perform_build, copying the inputs tar
  (Failed to copy inputs tar into container)

Both call sites already drop() the corresponding reader immediately after the call (drop(toolchain_rdr) at :725, drop(inputs_rdr) at :776); the writer is the one left open.

Suggested fix

Drop stdin before waiting:

pipe(&mut stdin).context("Failed to pipe input to process")?;
drop(stdin);
let output = process
    .wait_with_output()
    .context("Failed to wait for process to return")?;

An alternative consistent with the existing // Should really take a FnOnce/FnBox note is to pass ChildStdin by value into the closure so the callee owns and drops it.

I have this change running and can open a PR if it is wanted.