Expose effective cache access mode in JSON statistics

[seungpyoson]

Problem

With upstream v0.17.0 and one S3 backend, a failed startup write check can leave the server running read-only. The server starts successfully and emits a warning, but --show-stats --stats-format json does not expose the effective access mode. Immediately after startup, automation cannot distinguish that state from a healthy writable cache using the structured statistics.

I reproduced this twice with the official sccache-v0.17.0-aarch64-apple-darwin.tar.gz binary, using a loopback S3 fixture and synthetic credentials. No compiler was invoked.

Requested mode	Startup HTTP observations	Startup exit
READ_WRITE	GET 404, PUT 200	0
READ_WRITE	GET 404, PUT 403	0
READ_ONLY	GET 404, no PUT	0

The complete JSON statistics were identical across all three cases in each run. They contained no effective mode, and cache_write_errors and cache_writes were both zero. Advanced text statistics also omitted the effective mode. The denied case emitted a PermissionDenied warning on stderr; SCCACHE_NO_DAEMON=1 was set to retain diagnostics and allow the fixture to track and stop its own process group.

The startup check deliberately returns ReadOnly after the unsuccessful PUT, and the server applies that mode.

The internal StorageHandshakeInfo already carries cache_mode for client-side storage IPC. Its handler calls storage.check() again and defaults errors to ReadWrite; it is not a snapshot of the installed mode exposed through CLI JSON. This request concerns that CLI reporting gap, not a new mode type or a change to handshake policy.

By source inspection, upstream main at 05aafc82 still omits the mode from ServerInfo. Merged #2822 adds SCCACHE_SKIP_CACHE_CHECK: when enabled, remote initialization returns the configured mode without probing. That change is newer than the tested v0.17.0 binary; main was inspected, not executed.

Requested outcome

Expose the effective access mode installed by the running server in the existing JSON statistics, using the result already selected during initialization. When SCCACHE_SKIP_CACHE_CHECK is enabled, this is the configured mode, not a verified capability. For this single-backend case, consumers should be able to distinguish read/write from read-only without parsing logs or inferring the mode from compile counters.

Preserve the existing startup policy and derive the reported value from the running server. Reading statistics should not trigger an additional remote capability probe. Compile counters should keep their current meaning; a startup probe is a different operation. Effective mode would describe the running configuration, not establish that a probe succeeded, guarantee future writes or establish the token's provider-side permissions.

Please account for existing bincode client/server protocol compatibility when exposing the field. I have not implemented a patch or established its exact change size.

Verification

The healthy-writer, denied-writer and explicit-reader cases above should report their actual effective modes, preserve their existing startup outcomes and perform no extra remote operation when statistics are read. The fixture exercises the official CLI and records the actual HTTP requests. It does not exercise compilation, Linux ARM64 or live cloud storage.

Archive SHA-256, checked against GitHub release metadata: 0c560bfba31aef5bdfb4fb3d2677f6e61d71c5c00952f2a83344f47aa31f00f1.

Related

Related to #2593, which describes startup downgrade following rate limiting. This request concerns effective-mode reporting; recovery policy remains in that discussion. Also related to #2822 and #2132, which cover skipping capability checks; installed mode must remain distinct from evidence that a probe ran or succeeded.

Reproduction

Save the script below as reproduce.py and run it with the unmodified release binary. Use a new output directory and a short writable temporary directory suitable for Unix sockets:

python3 reproduce.py /absolute/path/to/sccache ./results --temporary-root /your/short/temp/directory

The script uses only Python's standard library. It creates isolated configuration and synthetic credentials, starts three loopback cases, records command output and HTTP operations, and stops its own servers. It is intended for macOS/Linux process-group semantics. It never invokes Cargo or a compiler. It creates the supplied temporary-root directory if missing. This is an observation collector, not a regression assertion suite: inspect command results, HTTP records and stderr; the script's exit status alone does not establish the reported behavior.

Executed reproduction script

#!/usr/bin/env python3
"""Observe stock sccache startup against loopback S3; never invoke a compiler."""

import argparse
import json
import os
from pathlib import Path
import signal
import socket
import subprocess
import tempfile
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer


def group_exists(group):
    try:
        os.killpg(group, 0)
        return True
    except ProcessLookupError:
        return False


def run_case(binary, root, name, mode, put_status, temporary_root):
    case = root / name
    case.mkdir()
    home = case / "home"
    home.mkdir()
    config = case / "config.toml"
    config.write_text("server_startup_timeout_ms = 5000\n")
    aws_config = case / "empty-aws-config"
    aws_config.write_text("")
    requests = []

    class Handler(BaseHTTPRequestHandler):
        def log_message(self, *_args):
            pass

        def respond(self, status, body=b""):
            self.send_response(status)
            self.send_header("Content-Type", "application/xml")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)

        def do_GET(self):
            requests.append({"method": "GET", "path": self.path, "status": 404})
            self.respond(404, b"<Error><Code>NoSuchKey</Code></Error>")

        def do_PUT(self):
            self.connection.settimeout(3)
            size = int(self.headers.get("Content-Length", "0"))
            assert size <= 4096, "startup fixture received an unexpected large body"
            data = self.rfile.read(size)
            requests.append({"method": "PUT", "path": self.path, "status": put_status, "body_bytes": len(data)})
            body = b"<Error><Code>AccessDenied</Code></Error>" if put_status == 403 else b""
            self.respond(put_status, body)

    backend = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
    thread = threading.Thread(target=backend.serve_forever)
    thread.start()
    with socket.socket() as reservation:
        reservation.bind(("127.0.0.1", 0))
        port = reservation.getsockname()[1]
    environment = {
        "PATH": "/usr/bin:/bin",
        "HOME": str(home),
        "XDG_CONFIG_HOME": str(home),
        "XDG_CACHE_HOME": str(home),
        "TMPDIR": str(temporary_root),
        "SCCACHE_CONF": str(config),
        "SCCACHE_DIR": str(case / "unused-disk"),
        "SCCACHE_SERVER_PORT": str(port),
        "SCCACHE_NO_DAEMON": "1",
        "SCCACHE_IDLE_TIMEOUT": "30",
        "SCCACHE_BUCKET": "synthetic-cache",
        "SCCACHE_REGION": "auto",
        "SCCACHE_ENDPOINT": f"http://127.0.0.1:{backend.server_port}",
        "SCCACHE_S3_USE_SSL": "false",
        "SCCACHE_S3_KEY_PREFIX": "startup-test/",
        "SCCACHE_S3_RW_MODE": mode,
        "AWS_ACCESS_KEY_ID": "synthetic-access-key",
        "AWS_SECRET_ACCESS_KEY": "synthetic-secret-key",
        "AWS_CONFIG_FILE": str(aws_config),
        "AWS_SHARED_CREDENTIALS_FILE": str(aws_config),
        "AWS_EC2_METADATA_DISABLED": "true",
        "NO_PROXY": "*",
    }
    result = {"case": name, "requested_mode": mode, "put_response": put_status, "commands": {}}
    startup = None

    def command(label, args):
        with (case / f"{label}.stdout").open("w") as stdout, (case / f"{label}.stderr").open("w") as stderr:
            process = subprocess.Popen([str(binary), *args], env=environment, cwd=home,
                                       stdout=stdout, stderr=stderr, start_new_session=True)
            if label == "start":
                nonlocal startup
                startup = process
            try:
                code = process.wait(timeout=12)
            except subprocess.TimeoutExpired:
                os.killpg(process.pid, signal.SIGTERM)
                process.wait(timeout=5)
                raise
        result["commands"][label] = {"args": args, "returncode": code, "pid": process.pid}
        return code

    try:
        command("version", ["--version"])
        if command("start", ["--start-server"]) == 0:
            command("stats", ["--show-stats", "--stats-format", "json"])
            command("advanced", ["--show-adv-stats"])
            command("stop", ["--stop-server"])
    finally:
        if startup is not None:
            until = time.monotonic() + 3
            while group_exists(startup.pid) and time.monotonic() < until:
                time.sleep(0.05)
            if group_exists(startup.pid):
                os.killpg(startup.pid, signal.SIGTERM)
                result["cleanup_sigterm"] = True
                until = time.monotonic() + 3
                while group_exists(startup.pid) and time.monotonic() < until:
                    time.sleep(0.05)
            result["startup_group_exited"] = not group_exists(startup.pid)
        backend.shutdown()
        backend.server_close()
        thread.join(timeout=3)
        result["backend_thread_exited"] = not thread.is_alive()
        result["requests"] = requests
        (case / "result.json").write_text(json.dumps(result, indent=2) + "\n")
    assert result.get("startup_group_exited") and result["backend_thread_exited"], result
    return result


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("binary", type=Path)
    parser.add_argument("output", type=Path)
    parser.add_argument("--temporary-root", type=Path, required=True)
    args = parser.parse_args()
    args.output.mkdir(exist_ok=False)
    args.temporary_root.mkdir(parents=True, exist_ok=True)
    results = []
    with tempfile.TemporaryDirectory(prefix="stock-", dir=args.temporary_root) as temporary:
        for name, mode, status in [("healthy-writer", "READ_WRITE", 200),
                                   ("denied-writer", "READ_WRITE", 403),
                                   ("explicit-reader", "READ_ONLY", 403)]:
            result = run_case(args.binary.resolve(), args.output.resolve(), name, mode, status, Path(temporary))
            results.append(result)
            print(json.dumps(result), flush=True)
    (args.output / "results.json").write_text(json.dumps(results, indent=2) + "\n")