From d7eb93755f456b25cd476fa601070a32b47d9967 Mon Sep 17 00:00:00 2001 From: Thomas Willetal Date: Fri, 24 Jul 2026 20:24:14 +0200 Subject: [PATCH] Update doku --- Readme.md | 421 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 294 insertions(+), 127 deletions(-) diff --git a/Readme.md b/Readme.md index 6ea09a3..62d676b 100644 --- a/Readme.md +++ b/Readme.md @@ -1,187 +1,354 @@ -# packtly-builder +# 1. packtly-builder + +**Reproducible Debian package builds with integrated signing and Aptly publishing.** + +`packtly-builder` is the build component of the **Packtly** platform. It wraps the Debian packaging toolchain (`debuild`, `dpkg-buildpackage`, `lintian`, GPG) inside rootless Podman containers and drives it with a small Python CLI, so a build behaves the same on a laptop, in CI, and on a release machine. + +Pair it with [`packtly-infra`](#related-projects) — which stands up the Aptly repository, HTTP endpoint, and credentials `packtly-builder` publishes to — for a complete, self-hosted APT package pipeline. + +## 1.1. Why Packtly Builder? + +`debuild` builds a package on whatever host you happen to run it on, with whatever toolchain, GPG keys, and dependencies happen to be installed there. `packtly-builder`: + +- Builds inside a pinned, disposable container, so builds are reproducible across machines and CI runners. +- Wraps build → sign → publish as a single command instead of a chain of manual `dpkg-buildpackage`, `debsign`, and `aptly` invocations. +- Understands Aptly: it checks what's already published, skips redundant uploads, and publishes source alongside binaries. +- Needs only Podman, `podman-compose`, and `just` on the host — the packaging toolchain itself never touches the host. + +## 1.2. Contents + +- [1. packtly-builder](#1-packtly-builder) + - [1.1. Why Packtly Builder?](#11-why-packtly-builder) + - [1.2. Contents](#12-contents) + - [1.3. Features](#13-features) + - [1.4. Architecture](#14-architecture) + - [1.4.1. Package Build Workflow](#141-package-build-workflow) + - [1.4.2. What Gets Built?](#142-what-gets-built) + - [1.5. Getting Started](#15-getting-started) + - [1.5.1. Using the Runtime Container](#151-using-the-runtime-container) + - [1.5.2. Multi-Architecture Builds](#152-multi-architecture-builds) + - [1.6. CLI Reference](#16-cli-reference) + - [1.6.1. Usage](#161-usage) + - [1.6.2. Command-line Options](#162-command-line-options) + - [1.6.3. Build Modes](#163-build-modes) + - [1.6.4. Publishing](#164-publishing) + - [1.7. Signing \& Credentials](#17-signing--credentials) + - [1.8. Development](#18-development) + - [1.8.1. Quick Start](#181-quick-start) + - [1.8.2. Container Images](#182-container-images) + - [1.8.3. Project Structure](#183-project-structure) + - [1.8.4. just Targets](#184-just-targets) + - [1.8.5. Dev Container](#185-dev-container) + - [1.8.6. Local Development](#186-local-development) + - [1.9. Versioning](#19-versioning) + - [1.10. Related Projects](#110-related-projects) + +## 1.3. Features + +- Container-native Debian package builds +- Reproducible build environments +- Binary and source package generation +- Multi-architecture support (amd64, arm64, armhf) +- Automatic GPG package signing +- Automatic Aptly publishing +- Debian source package publishing +- VS Code Dev Container + +## 1.4. Architecture + +### 1.4.1. Package Build Workflow -A containerized build system for **building, signing, and publishing Debian packages** to an [Aptly](https://www.aptly.info/) repository — reproducibly inside Podman containers, with no build tooling required on the host. +```mermaid +flowchart LR + A["📁 Debian Source Tree"] + A --> C ---- + subgraph B["📦 packtly-builder"] + C["🔨 Build"] + D["🔑 Sign"] + C --> D + D --> E{"Upload?"} + end -## Table of contents + E -->|No| F["📦 Local Artifacts"] + E -->|Yes| G["📚 Aptly Repository"] -- [Overview](#overview) -- [How it works](#how-it-works) -- [Prerequisites](#prerequisites) -- [Quick start](#quick-start) -- [Container images](#container-images) -- [`just` targets](#just-targets) -- [The `packtly_builder_tooling` CLI](#the-packtly_builder_tooling-cli) -- [Signing keys & credentials](#signing-keys--credentials) -- [Development](#development) -- [Versioning & releases](#versioning--releases) -- [Related projects](#related-projects) + G --> H["💻 Debian"] + G --> I["🤖 CI/CD"] +``` ---- +The build pipeline creates reproducible Debian packages inside isolated containers, signs the resulting artifacts with GPG, and optionally publishes them to an Aptly repository for immediate consumption by Debian systems, CI/CD pipelines, and embedded Linux build systems. -## Overview +### 1.4.2. What Gets Built? -packtly-builder bundles three things: +A build produces one or more of the following artifacts in different architectures (amd64, arm64, armhf): -- **Container images** — a layered set of Podman images based on Debian Trixie that carry the full Debian packaging toolchain (`debhelper`, `devscripts`, `gnupg2`, `git-buildpackage`, …). -- **`packtly_builder_tooling`** — a Python CLI that orchestrates the build → sign → publish pipeline for a single source tree. -- **CI/CD** — a GitLab CI pipeline that builds the images, runs the tooling test suite, and cuts versioned, multi-arch releases. +- Debian binary packages (`.deb`) +- Debian source packages (`.dsc`) +- Original source archives (`.orig.tar.*`) +- Debian packaging archives (`.debian.tar.*`) +- Build metadata (`.changes`, `.buildinfo`) +- GPG-signed release artifacts +- Multi-architecture package builds -## How it works +## 1.5. Getting Started -```mermaid -flowchart LR - A[Debian source tree] --> B[debuild] - B --> C[debsign
GPG signing] - C --> D{--upload?} - D -- yes --> E[Aptly repo] - D -- no --> F[Local artifacts] +### 1.5.1. Using the Runtime Container + +The `packtly-builder` runtime image includes `packtly_builder_tooling` as its ENTRYPOINT, so starting the container is equivalent to invoking the CLI directly. +A typical build is performed with a single `podman run` command that mounts the Debian source tree, GPG signing keys, Aptly credentials, and a log directory into the container: + +```bash +podman run --rm \ + --platform "linux/amd64" \ + -v "$PWD":/workspace:Z \ + -v "$KEYS_DIR/public/repo_signing.key":/opt/keys/gpg/repo_signing.key:Z,ro \ + -v "$KEYS_DIR/private/repo_signing_private.key":/opt/keys/gpg/repo_signing_private.key:Z,ro \ + -v "$KEYS_DIR/private/repo_signing_private_pass":/opt/keys/gpg/repo_signing_private_pass:Z,ro \ + -v "$APTLY_CREDENTIALS_FILE":/run/secrets/aptly-credentials:Z,ro \ + -v "$PWD/logs":/logs:Z \ + -e APTLYHOST=http://localhost:8080 \ + --network=host \ + ghcr.io/packtly/packtly-builder:latest \ + /workspace/debhello-quilt \ + --log-file /logs/build.log \ + --dist trixie-apollo \ + --component main \ + --credentials-file /run/secrets/aptly-credentials ``` -The CLI builds the package with `debuild`, signs the resulting `.changes`/`.dsc` with a configured GPG key, and — when `--upload` is given and an Aptly host is reachable — publishes the artifacts, skipping anything already deployed upstream. +| Mount / flag | Purpose | +| ------------------------------------------------- | -------------------------------------------------------------------------------------- | +| `-v :/workspace` | Mounts the Debian source tree. This path is passed to the CLI as the builddir argument | +| `-v .../repo_signing*.key:/opt/keys/gpg/...` | Mounts the GPG signing key material read-only. | +| `-v :/run/secrets/aptly-credentials` | Mounts the Aptly REST API credentials read-only. | +| `-v :/logs` | Stores build logs generated by --log-file on the host. | +| `-e APTLYHOST=...` | Specifies the Aptly API endpoint. | +| `--network=host` | Allows the container to communicate directly with a locally running Aptly instance. | -## Prerequisites +For a complete example, see [`test/fixtures/build-quilt.sh`](test/fixtures/build-quilt.sh) The script wraps the container invocation into convenient +into convenient `build`, `upload`, `force-upload`, and all actions. The --no-build --upload combination reuses the same runtime image to sign and publish previously built artifacts without rebuilding the package. -On the **host** you only need a container runtime and a task runner: +### 1.5.2. Multi-Architecture Builds -- [Podman](https://podman.io/) and [podman-compose](https://github.com/containers/podman-compose) -- [just](https://github.com/casey/just) -- [yq](https://github.com/mikefarah/yq) (used by some `just` targets) +packtly-builder is published as a multi-architecture container image. The target architecture is selected entirely through Podman's --platform option—there is no separate --arch flag in the CLI. -Everything else (the Debian toolchain, Python, Poetry) lives inside the images. +| Platform | Target architecture | +| ---------- | ------------------- | +| --platform | linux/amd64 | amd64 | +| --platform | linux/arm64 | arm64 | +| --platform | linux/arm/v7 | armhf | -## Quick start +Internally, `packtly_builder_tooling` detects the architecture it is running under (via `platform.machine()`) and configures the build accordingly, including selecting the appropriate Aptly repository endpoints. -```bash -# Build the builder image, run tests, build the wheel, build the runtime image -just all -# …or step by step: -just build-builder # Build the builder image -just test-tooling-keys # Verify the GPG key setup -just test-tooling # Run the Python test suite -just build-tooling # Build the packtly_builder_tooling wheel -just build-runtime # Build the final runtime image +The container itself is built per architecture (see [Container Images](#182-container-images)); which architecture a given `podman run` builds *for* is controlled entirely by Podman's `--platform` flag — the CLI has no separate `--arch` option. It simply detects the architecture it is actually running under (`platform.machine()`) and acts accordingly (e.g. when checking/uploading to the matching Aptly endpoint). + +For example, to build for arm64: +```bash +podman run \ + --platform linux/arm64 \ + ... \ + ghcr.io/packtly/packtly-builder:latest \ + /workspace/debhello-quilt ``` +On an amd64 host, running linux/arm64 or linux/arm/v7 containers requires QEMU user-mode emulation via binfmt_misc (typically provided by the qemu-user-static and binfmt-support packages), allowing foreign-architecture binaries to execute transparently inside the container. + +The reference script [`test/fixtures/build-quilt.sh`](test/fixtures/build-quilt.sh) -List every available target at any time: +exposes the target architecture as its second argument (default: amd64): ```bash -just # equivalent to `just --list` +./build-quilt.sh build arm64 ``` +The script performs a full source package build only for `amd64`, as the architecture-independent source package needs to be created only once. Subsequent architecture builds reuse the same source package and produce only the architecture-specific binary packages. + +In CI, this approach scales naturally to a build matrix with one job per target architecture. Native runners are used where available, while QEMU emulation is only required for `armhf`. The resulting architecture-specific images are finally assembled into a single multi-architecture OCI manifest, allowing `ghcr.io/packtly/packtly-builder:latest` to automatically resolve to the correct image for the host platform. -## Container images +## 1.6. CLI Reference -| Image | Built from | Purpose | -|---|---|---| -| `packtly-builder-base` | Debian Trixie | Debian packaging toolchain (`debhelper`, `devscripts`, `gnupg2`, …) | -| `packtly-builder-builder` | base | + Poetry, `just`, Node, and the tooling venv — used for CI builds & tests | -| `packtly-builder` (runtime) | base | + the installed `packtly_builder_tooling` wheel; entrypoint for real builds | -| `packtly-builder-devcontainer` | builder | Builder image wired up for VS Code Dev Containers | +### 1.6.1. Usage -Container build targets accept an optional **architecture** argument (`amd64` by default, `arm64` also supported), e.g. `just build-builder arm64`. +The runtime image provides the `packtly_builder_tooling` command. -## `just` targets +```text +packtly_builder_tooling [options] +``` -### Build +### 1.6.2. Command-line Options -| Target | Description | -|---|---| -| `build-base [arch]` | Build the base image | -| `build-builder [arch]` | Build the builder image | -| `build-runtime [arch]` | Build the runtime image | -| `build-devcontainer [arch]` | Build the devcontainer image | -| `build-builder-multiarch` | Build the builder image for amd64 + arm64 and assemble a manifest | -| `build-runtime-multiarch` | Build the runtime image for amd64 + arm64 and assemble a manifest | -| `build-tooling` | Build the `packtly_builder_tooling` Python wheel | +| Option | Description | +| ----------------------------------- | ------------------------------------------------------------------------ | +| `builddir` | Path to the Debian build directory (positional, required) | +| `--build-mode {binary,source,full}` | What to build: `binary` (default), `source`, or `full` (source + binary) | +| `--no-build` | Skip the build step (sign/upload existing artifacts) | +| `--aptlyhost URL` | Aptly REST API base URL (falls back to the `APTLYHOST` env var) | +| `--dist NAME` | Aptly publish distribution (e.g. `trixie-apollo`) | +| `--component NAME` | Aptly component (e.g. `main`) | +| `--credentials-file PATH` | Aptly credentials file (default: `/run/secrets/aptly-credentials`) | +| `--upload` | Upload the built package to Aptly after signing | +| `--force-upload` | Upload even if the package already exists upstream | +| `--log-file PATH` | Also write log output to this file | +| `--verbose`, `-v` | Enable debug logging | -### Test +### 1.6.3. Build Modes -| Target | Description | -|---|---| -| `test-tooling [arch]` | Run the full pytest suite inside the builder container | -| `test-tooling-keys [arch]` | Verify GPG key availability before tests | +| Mode | Description | +| -------- | -------------------------------- | +| `binary` | Build binary packages only | +| `source` | Build Debian source package | +| `full` | Build source and binary packages | -### Clean +### 1.6.4. Publishing -| Target | Description | -|---|---| -| `clean-base` | Remove the base image | -| `clean-builder` | Remove the builder image | -| `clean-runtime` | Remove the runtime image | -| `clean-devcontainer` | Remove the devcontainer image | -| `clean-containers` | Remove all container images | -| `clean` | Remove all images **and** built wheel artifacts | +The CLI can optionally: -### Utilities & pipeline +- sign packages with GPG +- upload artifacts to an Aptly repository +- skip packages already published +- force uploads when required -| Target | Description | -|---|---| -| `shell` | Open an interactive bash shell in the builder container | -| `all` | Full pipeline: `build-builder` → `test-tooling` → `build-tooling` → `build-runtime-multiarch` | +--- -## The `packtly_builder_tooling` CLI +## 1.7. Signing & Credentials -The CLI is the entrypoint of the runtime image. It builds, signs, and optionally uploads a single Debian package. +Repository signing keys are mounted into the container at: +```text +/opt/keys/gpg/ ``` -packtly_builder_tooling [options] + +The builder automatically imports the configured signing key and uses it for: + +- Debian package signing +- Source package signing +- Repository uploads + +Aptly API credentials are read from: + +```text +/run/secrets/aptly-credentials ``` -| Option | Description | -|---|---| -| `builddir` | Path to the Debian build directory (positional, required) | -| `--build-mode {binary,source,full}` | What to build: `binary` (default), `source`, or `full` (source + binary) | -| `--no-build` | Skip the build step (sign/upload existing artifacts) | -| `--aptlyhost URL` | Aptly REST API base URL (falls back to the `APTLYHOST` env var) | -| `--dist NAME` | Aptly publish distribution (e.g. `trixie-apollo`) | -| `--component NAME` | Aptly component (e.g. `main`) | -| `--credentials-file PATH` | Aptly credentials file (default: `/run/secrets/aptly-credentials`) | -| `--upload` | Upload the built package to Aptly after signing | -| `--force-upload` | Upload even if the package already exists upstream | -| `--log-file PATH` | Also write log output to this file | -| `--verbose`, `-v` | Enable debug logging | +making the builder suitable for local development and CI/CD environments. + +--- + +## 1.8. Development -### Build modes +`packtly-builder` consists of three major components: -| Mode | `debuild` flag | Produces | -|---|---|---| -| `binary` | `-b` | Binary packages only (no `.orig` tarball required) | -| `source` | `-S` | Source package only | -| `full` | `-F` | Source **and** binary packages | +| Component | Description | +| --------------------------- | ------------------------------------------------------------------------------------- | +| **Container Images** | Layered Podman images containing the complete Debian packaging toolchain. | +| **packtly_builder_tooling** | Python CLI for building, signing, and publishing Debian packages. | +| **CI/CD Pipeline** | Automated image builds, testing, multi-architecture releases, and package publishing. | -## Signing keys & credentials +### 1.8.1. Quick Start -GPG signing keys are read from these fixed paths inside the container: +Build the complete runtime image: +```bash +git clone https://github.com/packtly/packtly-builder.git +cd packtly-builder + +just all ``` -/opt/keys/gpg/repo_signing.key # public key -/opt/keys/gpg/repo_signing_private.key # private key -/opt/keys/gpg/repo_signing_private_pass # passphrase + +Or execute each stage individually: + +```bash +just build-builder +just test-tooling +just build-tooling +just build-runtime ``` -Locally, the `just` targets mount `Keys/gpg/` into `/opt/keys/gpg`, so place your keys there before running builds. Aptly credentials are read from a simple `key = value` file (default `/run/secrets/aptly-credentials`) containing `username` and `password`. +List all available tasks: -## Development +```bash +just +``` -The repository ships a VS Code Dev Container configuration. Open the project in VS Code and choose **Reopen in Container** for a fully configured Debian build environment. +### 1.8.2. Container Images + +| Image | Purpose | +| ------------------------------ | -------------------------------- | +| `packtly-builder-base` | Debian packaging toolchain | +| `packtly-builder-builder` | Build and test environment | +| `packtly-builder` | Runtime image for package builds | +| `packtly-builder-devcontainer` | VS Code Dev Container | + +All images support **amd64** and **arm64**. + +### 1.8.3. Project Structure + +```text +packtly-builder/ +├── tooling/ # Python CLI +├── tests/ # Unit and integration tests +├── container/ # Container definitions +├── Keys/ # Development signing keys +├── scripts/ # Helper scripts +├── justfile # Development workflow +├── Containerfile # Runtime image +└── .github/workflows/ # GitHub Actions +``` -For local development on the tooling without a devcontainer: +### 1.8.4. just Targets + +| Target | Description | +| --------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| `just` / `just list` | List all available recipes | +| `just build-base [arch]` | Build the base toolchain image | +| `just build-builder [arch]` | Build the build/test environment image | +| `just build-runtime [arch]` | Build the runtime image | +| `just build-devcontainer [arch]` | Build the VS Code Dev Container image | +| `just build-builder-multiarch` / `just build-runtime-multiarch` | Build and assemble multi-arch manifests (amd64 + arm64) | +| `just build-tooling` | Build the `packtly_builder_tooling` Python wheel inside the builder container | +| `just test-tooling [arch]` | Run the tooling test suite inside the builder container | +| `just shell` | Drop into an interactive shell in the builder container | +| `just clean-containers` / `just clean` | Remove built images / build artifacts | +| `just lint-robot` | Lint Robot Framework test suites with `robocop` | +| `just all` | Full pipeline: `build-builder` → `test-tooling` → `build-tooling` → `build-runtime-multiarch` | + +### 1.8.5. Dev Container + +The project ships with a complete VS Code Dev Container (`packtly-builder-devcontainer`), pre-configured with Poetry, the Debian packaging toolchain, and the linting/test tools used by CI. Open the repository in VS Code and select **Reopen in Container** to get a ready-to-use development environment without installing anything on the host beyond Podman. + +### 1.8.6. Local Development + +For local development of the `packtly_builder_tooling` CLI: ```bash -cd packtly-builder/tooling -just prepare # Install Poetry dependencies (incl. dev tools) -just test # flake8 + mypy + pytest -just pytest # Run tests only -just mypy # Type-check -just flake8 # Lint +cd tooling + +just prepare +just test +just pytest +just mypy +just flake8 ``` -## Versioning & releases +--- + +## 1.9. Versioning + +Release versions are derived from `CHANGELOG.md` using semantic versioning. -The release version is driven by `CHANGELOG.md` (Keep a Changelog format). At release time, CI reads the latest `## [x.y.z]` entry and applies it as the container image label and Git tag. +CI automatically: + +- builds multi-architecture images +- executes the test suite +- publishes container images +- creates GitHub releases + +--- -## Related projects +## 1.10. Related Projects -- **[packtly-infra](https://github.com/packtly/packtly-infra)** — Infrastructure automation for deploying packtly: a self-hosted Debian package repository based on [Aptly](https://www.aptly.info/) and [nginx](https://nginx.org/), running as a rootless Podman container managed by systemd Quadlets. +| Project | Description | +| ------------------- | --------------------------------------------------------- | +| **Packtly** | Platform overview | +| **packtly-builder** | Container-native Debian package builder (this repository) | +| **packtly-infra** | Deploy and operate an Aptly repository platform |