From 375fce686b40f31b841c1f85e7d23f9583898d68 Mon Sep 17 00:00:00 2001 From: Kieran Mann Date: Wed, 29 Jul 2026 16:29:41 -0700 Subject: [PATCH] build: make the compile stages cacheable across machines Exclude .git from the build context and take REVISION as a build arg instead of shelling out to git inside the buildkit-version stage. Three stages mount the build context: buildkit-version, buildctl and buildkitd. A git checkout is not byte-reproducible -- .git/index carries mtimes, logs/HEAD carries timestamps, packed-refs ordering varies -- so the context digest differed from machine to machine and all three missed the layer cache anywhere they had not already been built. runc, the one compile stage that mounts from runc-src rather than the context, was correspondingly the only one that reused a shared cache. Measured downstream in EarthBuild/earthbuild CI, where every test job builds this image in a fresh buildkitd: of the ~136s spent before any test runs, ~96s was the buildctl and buildkitd compile, and it did not improve when a registry cache was added (155s -> 151s with mode=min, 136s with --max-remote-cache; runc started hitting, the other two never did). buildkit-version was the only consumer of .git, so dropping its git usage lets the context exclude .git entirely, which is what makes the two expensive stages cacheable. Also removes a dead assignment: `VERSION=$(git describe ...)` has been unused since #18 changed the ldflags line to use ${RELEASE_VERSION}. REVISION defaults to "unknown". Callers wanting a real revision should pass the buildkit git sha they are pinning -- it is known statically at the call site, which is a better source for it than a git command run inside the build. Co-Authored-By: Claude Opus 5 --- .dockerignore | 3 +++ .earthlyignore | 18 ++++++++++++++++++ Dockerfile | 19 +++++++++++++++++-- Earthfile | 7 ++++++- 4 files changed, 44 insertions(+), 3 deletions(-) create mode 100644 .earthlyignore diff --git a/.dockerignore b/.dockerignore index d07ebf7f5..eda7257f0 100644 --- a/.dockerignore +++ b/.dockerignore @@ -1,3 +1,6 @@ bin .certs .tmp +# See .earthlyignore for why .git is excluded. Kept in sync here so plain +# docker builds get the same cacheable context. +.git diff --git a/.earthlyignore b/.earthlyignore new file mode 100644 index 000000000..5f672f3c4 --- /dev/null +++ b/.earthlyignore @@ -0,0 +1,18 @@ +# .git is deliberately excluded from the build context. +# +# Three Dockerfile stages mount the context (buildkit-version, buildctl, +# buildkitd). A git checkout is not byte-reproducible -- .git/index carries +# mtimes, logs/HEAD carries timestamps, packed-refs ordering varies -- so +# including it made the context digest differ per machine, and those three +# stages missed the layer cache on every machine that had not already built +# them. runc, which mounts from the runc-src stage instead of the context, was +# the only compile stage that reused a shared cache. +# +# Nothing in the build needs .git any more: buildkit-version now takes REVISION +# as a build arg instead of shelling out to git. +# +# NOTE: the Earthfile here is VERSION 0.6, and earthly only falls back to +# .dockerignore from 0.8 (the use-docker-ignore feature). .earthlyignore is +# honoured at every version, so it is what actually takes effect for +# `FROM DOCKERFILE`; .dockerignore is kept in sync for plain docker builds. +.git diff --git a/Dockerfile b/Dockerfile index c42ef1008..01127685e 100644 --- a/Dockerfile +++ b/Dockerfile @@ -85,8 +85,23 @@ ENV GOFLAGS=-mod=vendor FROM buildkit-base AS buildkit-version # TODO: PKG should be inferred from go modules ARG RELEASE_VERSION=v0.0.0+earthlyunknown -RUN --mount=target=. \ - PKG=github.com/moby/buildkit EARTHLY_PKG=github.com/EarthBuild/buildkit VERSION=$(git describe --match 'v[0-9]*' --dirty='.m' --always --tags) REVISION=$(git rev-parse HEAD)$(if ! git diff --no-ext-diff --quiet --exit-code; then echo .m; fi); \ +# REVISION is supplied by the caller rather than derived from git here. +# +# Deriving it needed `--mount=target=.`, which put the whole build context -- +# including .git -- into this stage's cache key. A git checkout is not +# byte-reproducible, so the context digest differed per machine and this stage +# missed the cache everywhere it had not already been built. buildctl and +# buildkitd mount /tmp/.ldflags from here, so they inherited the miss; runc, +# which mounts from runc-src instead of the context, was the only compile stage +# that reused a shared cache. .git is now excluded from the context entirely +# (see .earthlyignore), which is what makes the two expensive compile stages +# cacheable across machines. +# +# The old `VERSION=$(git describe ...)` was already dead: the ldflags line has +# used ${RELEASE_VERSION} since "build: bake version.Package as +# github.com/EarthBuild/buildkit" (#18), so the computed value was discarded. +ARG REVISION=unknown +RUN PKG=github.com/moby/buildkit EARTHLY_PKG=github.com/EarthBuild/buildkit; \ echo "-X ${PKG}/version.Version=${RELEASE_VERSION} -X ${PKG}/version.Revision=${REVISION} -X ${PKG}/version.Package=${EARTHLY_PKG}" | tee /tmp/.ldflags; \ echo -n "${RELEASE_VERSION}" | tee /tmp/.version; diff --git a/Earthfile b/Earthfile index 23fba1839..a7df9e0b1 100644 --- a/Earthfile +++ b/Earthfile @@ -7,7 +7,12 @@ WORKDIR /buildkit # build buildkit image build: ARG RELEASE_VERSION=v0.0.0+earthlyunknown - FROM DOCKERFILE --build-arg RELEASE_VERSION=$RELEASE_VERSION --target buildkit-linux . + # REVISION is baked into `buildkitd --version` output. It used to be read + # from .git inside the buildkit-version stage; see the comment there for why + # that is now passed in instead. Callers that want a real revision should + # pass the pinned buildkit git sha they are building. + ARG REVISION=unknown + FROM DOCKERFILE --build-arg RELEASE_VERSION=$RELEASE_VERSION --build-arg REVISION=$REVISION --target buildkit-linux . code: COPY . .