Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 53 additions & 9 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
name: build

# Four images (bench, fuzz, wasm, tools: images/Dockerfile, one target each),
# each for both architectures. Pull requests build and smoke-test them and run
# core's bench in ci-bench; they never push. Pushes to main and manual runs on
# main also publish ghcr.io/comlineproject/ci-{bench,fuzz,wasm,tools}, with a
# build provenance attestation, and then run core's bench in the published
# ci-bench as a real `container:` job.
# Five images (bench, fuzz, wasm, tools, flatpak: images/Dockerfile, one target
# each), each for both architectures except ci-flatpak (amd64 only: Flathub's
# VSCodium). Pull requests build and smoke-test them, run core's bench in
# ci-bench and the sandbox check in ci-flatpak; they never push. Pushes to main
# and manual runs on main also publish
# ghcr.io/comlineproject/ci-{bench,fuzz,wasm,tools,flatpak}, with a build
# provenance attestation, and then run core's bench in the published ci-bench
# and the sandbox check in the published ci-flatpak as real `container:` jobs.
#
# The images are reproducible: the same inputs give the same digest (the
# "Reproducible builds" section of images/versions.md). A pull request proves it
Expand Down Expand Up @@ -47,8 +49,12 @@ jobs:
strategy:
fail-fast: false
matrix:
image: [bench, fuzz, wasm, tools]
image: [bench, fuzz, wasm, tools, flatpak]
arch: [amd64, arm64]
exclude:
# Flathub's VSCodium is built here for x86_64 only.
- image: flatpak
arch: arm64
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
Expand All @@ -73,6 +79,11 @@ jobs:
run: docker run --rm --network none -v "$PWD/images/smoke.sh:/smoke.sh:ro" ci-${{ matrix.image }}:test bash /smoke.sh ${{ matrix.image }}
- name: Smoke test (uid 1001)
run: docker run --rm --network none --user 1001:1001 -e HOME=/tmp -v "$PWD/images/smoke.sh:/smoke.sh:ro" ci-${{ matrix.image }}:test bash /smoke.sh ${{ matrix.image }}
# Flatpak's sandbox needs more than the smoke test's container gives:
# `--privileged`, which is also what a job's `container:` options say.
- name: Sandbox check (privileged)
if: matrix.image == 'flatpak'
run: docker run --rm --privileged --network none -v "$PWD/images/flatpak-sandbox-check.sh:/check.sh:ro" ci-flatpak:test bash /check.sh
# The same image again as an OCI archive (from the build cache, so it is
# quick): the layers' sizes there are what a pull transfers, and its
# manifest digest is what the registry will hold.
Expand Down Expand Up @@ -170,8 +181,12 @@ jobs:
strategy:
fail-fast: false
matrix:
image: [bench, fuzz, wasm, tools]
image: [bench, fuzz, wasm, tools, flatpak]
arch: [amd64, arm64]
exclude:
# Flathub's VSCodium is built here for x86_64 only.
- image: flatpak
arch: arm64
env:
IMAGE: ghcr.io/comlineproject/ci-${{ matrix.image }}
steps:
Expand Down Expand Up @@ -206,6 +221,11 @@ jobs:
run: |
docker pull "$IMAGE@$DIGEST"
docker run --rm --network none -v "$PWD/images/smoke.sh:/smoke.sh:ro" "$IMAGE@$DIGEST" bash /smoke.sh ${{ matrix.image }}
- name: Sandbox check the pushed image (privileged)
if: matrix.image == 'flatpak'
env:
DIGEST: ${{ steps.build.outputs.digest }}
run: docker run --rm --privileged --network none -v "$PWD/images/flatpak-sandbox-check.sh:/check.sh:ro" "$IMAGE@$DIGEST" bash /check.sh
- name: Record the digest
env:
DIGEST: ${{ steps.build.outputs.digest }}
Expand Down Expand Up @@ -236,7 +256,7 @@ jobs:
strategy:
fail-fast: false
matrix:
image: [bench, fuzz, wasm, tools]
image: [bench, fuzz, wasm, tools, flatpak]
env:
IMAGE: ghcr.io/comlineproject/ci-${{ matrix.image }}
steps:
Expand Down Expand Up @@ -370,3 +390,27 @@ jobs:
- name: Bench
working-directory: core-src
run: sh "$GITHUB_WORKSPACE/images/bench-check.sh"

# The published ci-flatpak as a real `container:` job: the way a consuming
# repository runs its sandbox tests. --privileged, because bubblewrap makes
# namespaces and mounts /proc.
flatpak-container:
name: sandbox in the published ci-flatpak
needs: merge
runs-on: ubuntu-24.04
timeout-minutes: 30
permissions:
contents: read
packages: read
container:
image: ghcr.io/comlineproject/ci-flatpak:sha-${{ github.sha }}
credentials:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
options: --privileged
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Sandbox check
run: bash images/flatpak-sandbox-check.sh
37 changes: 29 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,12 @@ One image per kind of job, each carrying only what those jobs use:
| `ghcr.io/comlineproject/ci-fuzz` | the cargo-fuzz jobs | Rust nightly, cargo-fuzz, gh |
| `ghcr.io/comlineproject/ci-wasm` | the WebAssembly build | Rust stable with wasm32, wasm-bindgen, wasm-pack, binaryen |
| `ghcr.io/comlineproject/ci-tools` | cue, cbindgen and the musl build | Rust stable with musl, cue, cbindgen, musl-tools, make, jq |
| `ghcr.io/comlineproject/ci-flatpak` | the sandbox tests of the VSCodium extension | Flatpak, bubblewrap, xdg-dbus-proxy, a session bus, Flathub's VSCodium and its runtime (amd64 only) |

All are Debian 13 (trixie) slim, by digest, with every apt package pinned to
one snapshot.debian.org date, for `linux/amd64` and `linux/arm64`. They all
have git and a C toolchain. [`images/versions.md`](images/versions.md) lists
one snapshot.debian.org date, for `linux/amd64` and `linux/arm64` (`ci-flatpak`:
`linux/amd64` only, as Flathub's VSCodium is built for x86_64 here). The four
Rust images have git and a C toolchain; `ci-flatpak` has git and no compiler. [`images/versions.md`](images/versions.md) lists
every version and why it is that one; [`docs/base-choice.md`](docs/base-choice.md)
says why Debian.

Expand Down Expand Up @@ -48,6 +50,20 @@ default seccomp profile refuses.
options: --security-opt seccomp=unconfined
```

The flatpak job's container needs `--privileged`: bubblewrap makes namespaces
and mounts `/proc`, which Docker's default seccomp profile and masked paths
refuse. It runs as root, as every job container does, with `XDG_RUNTIME_DIR`
(`/run/user/0`) set; the job's `dbus-run-session -- flatpak run ...` is its own
session bus. VSCodium is installed for the whole system at an exact Flathub
commit, which `/etc/ci-flatpak/vscodium-commit` (and the image's
`org.comlineproject.ci-flatpak.vscodium-commit` label) holds, so a job can log it.

```yaml
container:
image: ghcr.io/comlineproject/ci-flatpak@sha256:<digest>
options: --privileged
```

Renovate bumps the digest: the tags (`latest`, `sha-<short>`) exist for it to
follow. A bump is a pull request in the consuming repository, so it runs that
repository's CI against the new image before it merges.
Expand Down Expand Up @@ -77,18 +93,23 @@ gh attestation verify oci://ghcr.io/comlineproject/ci-bench@sha256:<digest> --ow

[`images/Dockerfile`](images/Dockerfile) is one file with a `base` stage and a
build target per image (`docker build --target bench images`).
[`build.yml`](.github/workflows/build.yml) builds the four images for both
architectures natively (`ubuntu-24.04` and `ubuntu-24.04-arm`). Every build
[`build.yml`](.github/workflows/build.yml) builds the five images for both
architectures natively (`ubuntu-24.04` and `ubuntu-24.04-arm`; `ci-flatpak`
only on the first). Every build
smoke-tests the image ([`images/smoke.sh`](images/smoke.sh), one mode per image:
each tool's version, a build for each target, with no network, as root and as
another uid), prints its compressed size, and `ci-bench` runs core's bench at a
fixed commit. Pull requests stop there. Pushes to `main` and manual runs on
another uid), prints its compressed size, `ci-bench` runs core's bench at a
fixed commit, and `ci-flatpak` runs VSCodium's Electron inside Flatpak's
sandbox in a `--privileged` container ([`images/flatpak-sandbox-check.sh`](images/flatpak-sandbox-check.sh)). Pull requests stop there. Pushes to `main` and manual runs on
`main` push each architecture by digest, smoke-test what was pushed, join the
two into one manifest tagged `sha-<short>` and `latest`, attest it, and run
core's bench in the published `ci-bench` as a real `container:` job.
core's bench in the published `ci-bench` and the sandbox check in the published
`ci-flatpak` as real `container:` jobs.

There is no scheduled rebuild: the apt packages are fixed by date, so a rebuild
would change nothing. To update them, move `DEBIAN_SNAPSHOT` (and the base
digest) in the Dockerfile. To update a tool, change the `ARG` and its hashes in
[`images/Dockerfile`](images/Dockerfile), and its row in
[`images/versions.md`](images/versions.md).
[`images/versions.md`](images/versions.md). `ci-flatpak`'s VSCodium and runtime
are bumped the same way, by commit ([`images/versions.md`](images/versions.md#flatpak)
says how to find them).
59 changes: 58 additions & 1 deletion images/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,11 @@
# fuzz Rust nightly, cargo-fuzz, gh
# wasm Rust stable + wasm32 (browser and WASI), wasm-bindgen, wasm-pack, binaryen
# tools Rust stable + musl, cue, cbindgen, musl-tools, make, jq
# flatpak Flatpak, bubblewrap, a session bus, and Flathub's VSCodium (amd64 only)
#
# docker build --target bench images
#
# All four start from the `base` stage: Debian 13 (trixie) slim, by digest, with
# All start from the `base` stage: Debian 13 (trixie) slim, by digest, with
# apt pointed at a dated snapshot of the archive. images/versions.md says what
# each version is for. Every downloaded binary is checked against the SHA-256
# hash below, per architecture, before it is used; a version bump changes the
Expand Down Expand Up @@ -49,6 +50,15 @@ ARG GH_VERSION=2.103.3
ARG GH_SHA256_AMD64=4c806978ffce1cec70853302a56d70ac6da4f19bae7adca4e1a8ab00633dd91d
ARG GH_SHA256_ARM64=ee32df2323c8847f5fac6cb6b4b69ca75828e9329a8f22f677021f7b3306ce63

# The flatpak image: Flathub's VSCodium and its runtime, by exact commit (the
# Flathub history keeps them). Bump together, in step with what the consumers'
# e2e should run on: images/versions.md says how to find the commits.
ARG FLATHUB_REPO_SHA256=3371dd250e61d9e1633630073fefda153cd4426f72f4afa0c3373ae2e8fea03a
ARG VSCODIUM_REF=app/com.vscodium.codium/x86_64/stable
ARG VSCODIUM_COMMIT=b63788f25cee55b738d5aa6df3b864c237bda087ce0ee43e803e5a7b207a25ab
ARG RUNTIME_REF=runtime/org.freedesktop.Sdk/x86_64/26.08
ARG RUNTIME_COMMIT=a1b744b072d9f55d061e35382547ae80dad8d92abecc1da7dc1818de2daabfd3

# Built from source (no prebuilt binary for both architectures), --locked.
ARG CARGO_FUZZ_VERSION=0.13.2
ARG CBINDGEN_VERSION=0.29.2
Expand Down Expand Up @@ -158,6 +168,13 @@ RUN --mount=type=bind,source=fetch.sh,target=/fetch.sh \
chmod 755 /out/*


FROM dl-base AS dl-flathub
ARG FLATHUB_REPO_SHA256
RUN --mount=type=bind,source=fetch.sh,target=/fetch.sh \
set -eu; mkdir -p /out; \
sh /fetch.sh "https://dl.flathub.org/repo/flathub.flatpakrepo" "$FLATHUB_REPO_SHA256" /out/flathub.flatpakrepo


# ---------------------------------------------------------------------------
# rust-base: base plus a C toolchain (rustc links with cc, build scripts and
# cargo-fuzz's libFuzzer build need it). The toolchain itself is installed per
Expand Down Expand Up @@ -265,3 +282,43 @@ COPY --from=dl-cue /out/ /usr/local/bin/
ENV CI_RUST_STABLE=${RUST_STABLE}
LABEL org.opencontainers.image.source="https://github.com/ComlineProject/ci-images" \
org.opencontainers.image.description="ComlineProject CI: Rust stable with musl, cue and cbindgen (see images/versions.md)"


# ---------------------------------------------------------------------------
# flatpak: the sandbox tests of the VSCodium extension (comline-vscode). Flatpak
# with bubblewrap and xdg-dbus-proxy, a session bus (`dbus-run-session`), procps
# (`pgrep`, which the tests use on the "host"), and Flathub's com.vscodium.codium
# with its runtime as a system installation, at the commits above. Flathub's
# VSCodium is x86_64, so this image is amd64 only.
#
# Run its container with `--privileged`: bubblewrap needs to create namespaces
# and mount /proc, which Docker's default seccomp and masked paths forbid.
# Root is the default user, as for every job container: the workspace is
# mounted from the runner, and flatpak runs as root without a helper.
#
# dbus-run-session -- flatpak run --command=/app/share/codium/com.vscodium.codium \
# --env=ELECTRON_RUN_AS_NODE=1 com.vscodium.codium script.js
FROM base AS flatpak
ARG TARGETARCH
ARG VSCODIUM_REF
ARG VSCODIUM_COMMIT
ARG RUNTIME_REF
ARG RUNTIME_COMMIT
# ostree: to prune what the pinned commits leave behind.
# The machine id apt's postinst writes differs in every build, so it is removed:
# the first dbus-daemon that needs one makes it.
RUN --mount=type=bind,source=apt-install.sh,target=/apt-install.sh \
set -eu; [ "$TARGETARCH" = amd64 ] || { echo "ci-flatpak is amd64 only" >&2; exit 1; }; \
sh /apt-install.sh flatpak bubblewrap xdg-dbus-proxy dbus-daemon procps ostree; \
rm -f /etc/machine-id /var/lib/dbus/machine-id
RUN --mount=type=bind,from=dl-flathub,source=/out,target=/dl \
--mount=type=bind,source=flatpak-install.sh,target=/flatpak-install.sh \
sh /flatpak-install.sh
# flatpak wants a runtime directory, which a container has no login session to
# make.
RUN mkdir -p /run/user/0 && chmod 700 /run/user/0
ENV XDG_RUNTIME_DIR=/run/user/0
LABEL org.opencontainers.image.source="https://github.com/ComlineProject/ci-images" \
org.opencontainers.image.description="ComlineProject CI: Flatpak, bubblewrap, a session bus and Flathub's VSCodium, for the sandbox tests (see images/versions.md)" \
org.comlineproject.ci-flatpak.vscodium-commit="${VSCODIUM_COMMIT}" \
org.comlineproject.ci-flatpak.runtime-commit="${RUNTIME_COMMIT}"
52 changes: 52 additions & 0 deletions images/flatpak-install.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
#!/bin/sh
# flatpak-install.sh: the Flatpak half of the ci-flatpak image. Run at build
# time, as root, with the Flathub remote file at /dl/flathub.flatpakrepo (hash-
# checked) and the ARGs of the Dockerfile in the environment:
#
# VSCODIUM_REF, VSCODIUM_COMMIT the app, and the exact build of it
# RUNTIME_REF, RUNTIME_COMMIT its runtime, and the exact build of it
#
# Installed by the latest Flathub has, then moved to the exact commit with
# `flatpak update --commit` (Flathub keeps the history of a ref), and the
# commits nobody refers to any more are pruned. The commit is what the image
# holds, whatever Flathub's current build is. (ostree's own `pull REF@COMMIT`
# would fetch the commit directly, but Flathub answers 403 to the `ostree`
# client's user agent: only flatpak's is let through.) A SYSTEM installation
# (/var/lib/flatpak), so every uid sees it and nothing depends on a home
# directory. No dependencies or related extensions: the base app is part of the
# app's files already, and GL, locale and codecs are for windows the image never
# opens. What varies from one build to the next (caches, summaries, appstream
# data) is removed: see "Reproducible builds" in images/versions.md.
set -eu

: "${VSCODIUM_REF:?}" "${VSCODIUM_COMMIT:?}" "${RUNTIME_REF:?}" "${RUNTIME_COMMIT:?}"
repo=/var/lib/flatpak/repo

flatpak remote-add --system flathub /dl/flathub.flatpakrepo

install_pinned() {
ref=$1 commit=$2
# NAME/ARCH/BRANCH, which every flatpak command takes.
name=${ref#*/}
flatpak install --system --noninteractive --assumeyes --no-deps --no-related flathub "$name"
flatpak update --system --noninteractive --assumeyes --no-deps --no-related --commit="$commit" "$name"
have=$(flatpak info --system --show-commit "$name")
[ "$have" = "$commit" ] || { echo "installed $ref is $have, not $commit" >&2; exit 1; }
}
install_pinned "$RUNTIME_REF" "$RUNTIME_COMMIT"
install_pinned "$VSCODIUM_REF" "$VSCODIUM_COMMIT"

# The latest builds the installs started from: no ref points at them now.
ostree --repo="$repo" prune --refs-only --depth=0
ostree --repo="$repo" refs

flatpak list --system --columns=ref,active,origin,installation

# What a job logs, and what the smoke test compares to the labels.
mkdir -p /etc/ci-flatpak
printf '%s\n' "$VSCODIUM_COMMIT" > /etc/ci-flatpak/vscodium-commit
printf '%s\n' "$RUNTIME_COMMIT" > /etc/ci-flatpak/runtime-commit

# Not part of the installation, and not the same twice: ostree's cache of the
# remote's summary and its temporary files, flatpak's lock files.
rm -rf "$repo"/tmp/* "$repo"/.lock /var/lib/flatpak/.changed /var/tmp/flatpak-cache-* 2> /dev/null || true
38 changes: 38 additions & 0 deletions images/flatpak-sandbox-check.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#!/usr/bin/env bash
# Flatpak's sandbox, for real, in the ci-flatpak image: what its consumers do.
# Needs the container started with `--privileged` (bubblewrap makes namespaces
# and mounts /proc), as root:
# docker run --rm --privileged -v "$PWD/images/flatpak-sandbox-check.sh:/check.sh:ro" IMAGE bash /check.sh
# It runs VSCodium's Electron as plain Node inside the sandbox, through a
# session bus (`flatpak-spawn --host` talks to the session helper over it), and
# has the sandbox start a program on the "host" (here, the container).
set -euo pipefail

app=com.vscodium.codium
electron=/app/share/codium/com.vscodium.codium
# The runtime directory the image's ENV names; a job may have another HOME.
export XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}
mkdir -p "$XDG_RUNTIME_DIR"
chmod 700 "$XDG_RUNTIME_DIR"

echo "VSCodium commit: $(cat /etc/ci-flatpak/vscodium-commit)"
echo "runtime commit: $(cat /etc/ci-flatpak/runtime-commit)"
echo "uid $(id -u), XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR"

echo "+ Electron, as Node, in the sandbox"
version=$(dbus-run-session -- flatpak run --command="$electron" --env=ELECTRON_RUN_AS_NODE=1 "$app" \
-e 'console.log(process.versions.electron)')
echo "electron $version"
[[ "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+ ]] || { echo "FAILED: no Electron version: $version" >&2; exit 1; }

echo "+ the sandbox is not the container, and reaches it through flatpak-spawn --host"
marker=$(mktemp /usr/local/bin/sandbox-check.XXXXXX)
trap 'rm -f "$marker"' EXIT
dbus-run-session -- flatpak run --command=sh "$app" -c '
set -eu
echo "inside the sandbox: $FLATPAK_ID"
test ! -e "$1" || { echo "FAILED: $1 is visible in the sandbox" >&2; exit 1; }
flatpak-spawn --host test -e "$1"
echo "the host sees it"
' sh "$marker"
echo "sandbox check passed"
Loading