Cargoship

Cargoship is a Go-based CLI for building, distributing, and applying offline Kubernetes distro packages. It is designed to simplify two core workflows:

1. Package Creation: Building a self-contained, offline Kubernetes distro package containing everything needed for installation in disconnected or highly-regulated environments.
2. Cluster Lifecycle Management: Bootstrapping, upgrading, and managing the cluster on target hosts over SSH using the packaged distribution.

Cargoship bridges the gap between offline distro packaging tools and remote cluster lifecycle managers, supporting OCI image and file packaging, secure publishing to OCI registries, and robust SSH orchestration.

---

Installation

Pre-built binaries

Every release publishes archives for Linux, macOS, and Windows across amd64, arm64, and riscv64. The name carries the OS and the architecture, with amd64 spelled the way uname -m does, so the one for a 64-bit Linux host is cargoship_Linux_x86_64.tar.gz and the arm64 one is cargoship_Linux_arm64.tar.gz:

VERSION=$(curl -fsSL https://api.github.com/repos/colonel-byte/cargoship/releases/latest | grep -o '"tag_name": *"[^"]*"' | cut -d'"' -f4)
curl -fsSLO "https://github.com/colonel-byte/cargoship/releases/download/${VERSION}/cargoship_Linux_x86_64.tar.gz"
tar -xzf cargoship_Linux_x86_64.tar.gz cargoship
install -m 0755 cargoship /usr/local/bin/cargoship

Linux packages

Releases also carry .rpm, .deb, and .apk packages. Each installs the binary, shell completions for bash, fish and zsh, and the colonel_byte.cargoship Ansible collection onto ansible-core's default collection path:

sudo dnf install ./cargoship_*_linux_amd64.rpm    # or: sudo apt install ./cargoship_*_linux_amd64.deb

Container images

podman run --rm ghcr.io/colonel-byte/cargoship:latest version

Image	Contents
ghcr.io/colonel-byte/cargoship	The binary on a minimal base
ghcr.io/colonel-byte/cargoship-ubi	The binary installed from the .rpm, on AlmaLinux 10
ghcr.io/colonel-byte/cargoship-deb	The binary installed from the .deb, on Debian slim
ghcr.io/colonel-byte/cargoship-ansible	The .rpm plus ansible-core, on AlmaLinux 10

The -ubi name is historical: that image was based on Red Hat UBI and is now built on AlmaLinux 10, but the published image name has not changed.

From source

Building needs Go 1.27 or newer.

go install github.com/colonel-byte/cargoship@latest

That reaches the network for the module. To build without it, clone the repository instead: the vendor/ tree is committed, so a clone builds with no network access at all, which is how a management node inside an airlock is staged.

git clone https://github.com/colonel-byte/cargoship.git && cd cargoship
mage build:binary    # or: go run ./magefiles/core build:binary

Verifying a download

Release archives and container images are signed with Cosign, using the key published at cosign.pub. Each archive ships a .sigstore.json bundle beside it, and every release carries a checksums.txt:

cosign verify-blob --key cosign.pub --bundle cargoship_Linux_x86_64.tar.gz.sigstore.json cargoship_Linux_x86_64.tar.gz
cosign verify --key cosign.pub ghcr.io/colonel-byte/cargoship:latest

---

Supported Distributions

Cargoship currently provides native support and integration for the following Kubernetes engines:

• K3s
• RKE2
• Upstream Kubernetes (kubeadm) - installs the pkgs.k8s.io packages and containerd, and applies a CNI the package declares

---

Core Concepts

Cargoship is built around two primary architectural concepts:

1. Offline Distro Packages

A Cargoship package is a single, compressed archive containing all artifacts necessary to spin up or upgrade a Kubernetes cluster in air-gapped networks. It bundles:

• Distro-specific metadata and layout descriptors.
• Required engine configuration templates and system files.
• OCI images and container artifacts.
• Checksums and cryptographic signatures for integrity verification.

The package compilation process loads a local distro definition, gathers all specified assets, and produces a single, verifiable archive.

2. Phase-Based Orchestration

Cluster operations are modeled as ordered sequences of reusable, structured "phases." This makes high-level actions (apply, prepare, reset, kube-config, engine-config-sync) predictable, easier to debug, and simple to extend.

For example, the apply workflow runs these phases, in this order:

1. Connect: Establish secure SSH connections to all target hosts.
2. OS Detection & Fact Gathering: Identify host operating systems, network facts, and the engine version already installed.
3. Validation: Verify that host nodes meet the prerequisites, and stop if the cluster holds a node the config does not.
4. Lock: Take an exclusive host lock, held for the rest of the run, so a second cargoship cannot change the same hosts.
5. Prepare: Install OS packages, configure sysctl and environment, SELinux and fapolicyd rules, /etc/hosts, and the node firewall.
6. Upload: Securely transfer package binaries, RPMs or DEBs, configuration templates, and OCI images to the nodes, then import the images.
7. Bootstrap / Upgrade: Render the engine config, initialize control-plane nodes, apply bundled manifests, join workers, and upgrade any node already running an older version.
8. Engine Config Sync: Reconcile registry, audit, and pod-security config on any node whose config drifted.
9. Kubeconfig Retrieval and Labelling: Fetch the generated admin credentials and label nodes with their profile group.
10. Cleanup & Disconnect: Release the lock, clean up temporary artifacts, and close SSH sessions.

The generated reference in docs/phases/ is the authoritative list, including what each phase does under --dry-run.

---

Usage Examples

Here are the standard workflows for compiling and deploying offline Kubernetes packages with Cargoship.

The commands that change a host - prepare, apply, reset, and engine-config-sync - will not do so without --confirm. Swap it for --dry-run to connect to every host and run the preflight checks for real, reporting what the run would change without changing it.

1. Compile an Offline Package

To build an offline archive containing all required files, binaries, and container images:

# Create a package from the current directory structure
cargoship create .

# Build from a specific path and output to a custom directory
cargoship create ./distro-defs -o ./build/

2. Prepare Target Nodes

Verify and configure OS-level prerequisites (such as kernel modules, firewall ports, fapolicyd rules, and /etc/hosts) across target machines using your cluster configuration inventory:

cargoship prepare ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm

3. Deploy or Upgrade a Cluster

Bootstrap a new cluster or upgrade an existing one from the compiled package:

cargoship apply ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm

4. Fetch the Kubeconfig

Retrieve the admin kubeconfig securely from the primary controller:

cargoship kube-config --config ./cargoship-config.yaml

5. Uninstall or Reset Target Nodes

Stop, uninstall, and completely purge the Kubernetes distro and its state from the target hosts:

cargoship reset --config ./cargoship-config.yaml --distro rke2 --confirm

6. Generate an Inventory from Ansible

Translate an Ansible inventory you already maintain into a cargoship cluster inventory, taking each host's role from its Ansible groups:

cargoship inventory from-ansible ./resolved.json -o ./inventory.yaml

7. Publish, Pull, Sign, and Verify a Package

A package can live in an OCI registry rather than on a disk, and every command that takes a package takes an oci:// reference wherever it takes a path:

# Push a built package to a registry
cargoship publish ./build/cargoship-distro-amd64.tar.zst oci://ghcr.io/my-org

# Pull one back down to a local directory
cargoship pull oci://ghcr.io/my-org/my-package:1.0.0 -o ./build/

# Sign a package with a private key
cargoship sign ./build/cargoship-distro-amd64.tar.zst --signing-key ./private-key.pem

# Verify the signature before anything installs from it
cargoship verify ./build/cargoship-distro-amd64.tar.zst --key ./cosign.pub

An operator who cannot reach the registry from the management node moves the package across the airlock with Skopeo instead, then converts the resulting OCI archive into a native package:

# On the connected side: export the package as an OCI archive
skopeo copy --override-os multi docker://ghcr.io/my-org/my-package:1.0.0 oci-archive:my-package.tar:1.0.0

# On the disconnected side: convert it into a package apply can install
cargoship import ./my-package.tar -o ./build/

import reads an OCI Image Layout, as a .tar or as an unpacked directory; see Importing Archives for the other tools that write one, and why a docker-archive cannot carry a package.

A cosign keypair is the only signing method - keyless (Fulcio/OIDC) signing and verification were removed, because they need network the management node does not have; see #848. The commands that load a package - apply, prepare, engine-config-sync, publish, pull, validate, schema - verify as part of loading, gated on --verify=never|if-possible|always. verify does nothing but check the signature, which is what makes it the one to run before a playbook.

8. Encrypt Credentials in a Config File

Registry credentials in a cluster inventory are encrypted at rest, with Ansible Vault or with age. The vault command group covers both:

# Encrypt one value for a registry's user/pass/token field
cargoship vault encrypt my-registry-password --vault-password-file ./vault-pass.txt

# Encrypt or decrypt every registry credential in a config file, in place
cargoship vault encrypt-file ./cargoship-config.yaml --vault-password-file ./vault-pass.txt
cargoship vault decrypt-file ./cargoship-config.yaml --vault-password-file ./vault-pass.txt

# Encrypt the values already sitting at given YAML paths, naming each one
cargoship vault encrypt-path ./cargoship-config.yaml '.spec.config.registries[0].auth.pass' --vault-password-file ./vault-pass.txt

# Generate an age key pair, and re-wrap an existing file under a new key
cargoship vault keygen
cargoship vault rekey ./cargoship-config.yaml --new-vault-password-file ./new-pass.txt

encrypt-path only encrypts; cargoship decrypts a registry's auth.user, auth.pass, auth.token and tls.ca at apply time, and warns when asked to encrypt a path outside that set. decrypt, decrypt-path, decrypt-file and encrypt-file complete the set; cargoship vault --help lists all eight.

9. Validate a File, or Write Its Schema

# Check an inventory, package definition, or config file against its schema
cargoship validate ./cargoship-config.yaml

# Write the JSON Schema for one of cargoship's own file formats: inventory, package, or config
cargoship schema inventory -o ./inventory.schema.json

# Checksum a file, or one file inside an archive
cargoship sha256sum ./build/cargoship-distro-amd64.tar.zst

10. Record a Run for Audit

# Write a structured record of the run: what ran, against which package, against which fleet,
# and what changed. Written whether the run succeeds or fails, readable only by its owner.
cargoship apply ./build/cargoship-distro-amd64.tar.zst \
  --config ./cluster.yaml --confirm \
  --receipt-path /var/log/cargoship/receipts/apply-20260928-143000.json

# Strip everything that identifies the fleet, ready to attach to a bug report
cargoship redact-receipt ./receipt.json -o ./share.json

The receipt carries the invocation and the cargoship that ran it, the package's digests and signature status, the cluster inventory, and the phase-by-phase ledger with per-host change reasons and diffs. redact-receipt renumbers the hosts host-01, host-02, drops the diff bodies -- an engine configuration diff carries join tokens -- and removes the cluster name, operator and command line, leaving what a maintainer triages from.

A fleet that keeps an audit trail sets the path once in the config file instead, under .distro.receipt_path, where apply, prepare, reset and engine-config-sync all pick it up:

distro:
  # {command} and {timestamp} expand per run, so each receipt lands in its own file
  receipt_path: /var/log/cargoship/receipts/{command}-{timestamp}.json

Those two placeholders are the only ones accepted; anything else is rejected before the run starts, rather than written out literally. --receipt-path overrides the configured value and takes the same placeholders.

---

OpenTofu and Terraform

The same phases run from a declarative configuration through the OpenTofu provider, which lives in colonel-byte/terraform-provider-cargoship and consumes this repository as a library: cargoship_cluster converges a cluster -- install, join and upgrade are one apply, because the engine version already on each host is what tells them apart -- and cargoship_cluster_facts reads a fleet without changing it. It drives cargoship in process, so it still opens every SSH connection itself from the machine OpenTofu runs on.

It grew up here and moved out, so that it could be published under the name the provider registry requires; nothing in cargoship depends on it. See choice-tofu-provider-removal for what moved and what stayed.

---

Ansible Integration

colonel_byte.cargoship runs the workflows above as ordinary playbook tasks: cargoship_apply, cargoship_prepare, cargoship_reset, cargoship_kube_config, and cargoship_engine_config_sync, plus a cluster role that wraps them. Ansible supplies the inventory and runs one task for the whole fleet; cargoship still opens every SSH connection itself, from the management node the package was staged onto.

For driving zarf itself from a playbook, the colonel_byte.zarf collection lives in colonel-byte/ansible-collection-zarf. It grew up here and moved out; nothing in cargoship depends on it.

• Running Cargoship as an Ansible Module - the modules, their parameters, check mode, and what changed means.
• Generating an Inventory from Ansible - the group and host-variable translation, usable with or without the modules.
• The Ansible Container Image - ghcr.io/colonel-byte/cargoship-ansible, which carries the binary and the collections together.

---

Configuration and Schemas

Cargoship relies on strongly-typed YAML definitions to govern its operations:

• Cluster Inventories: Specify SSH configurations, credentials, host roles, profiles, and load-balancer addresses.
• Distro Package Definitions: Map out the required binaries, OCI images, and layout settings.
• Distro Runtime Configs: Configure the underlying distribution engine.

Inventories and package definitions are Kubernetes-style documents under cargoship's own API group:

apiVersion: colonel-byte.dev/v1alpha1
kind: Cluster      # or Distro, for a package definition
metadata:
  name: fleet-cluster

The corresponding JSON schemas are automatically generated from Go structs into schema/, and cargoship schema writes any of them on demand. When authoring configurations in modern editors, refer to these schemas for real-time validation and autocompletion; cargoship validate checks a file against the right one without an editor.

An inventory authoring guide is available in Setting up an inventory.

The zarf.dev group is deprecated

These documents used to declare apiVersion: zarf.dev/v1alpha1 with kind: ZarfCluster and kind: ZarfDistro. That group belongs to the upstream Zarf project, which defines its own kinds under it, so cargoship moved to a group it owns.

Both spellings are accepted, in any combination, and a document using the old one needs no change today:

	Old	New
API group	zarf.dev/v1alpha1	colonel-byte.dev/v1alpha1
Inventory kind	ZarfCluster	Cluster
Package kind	ZarfDistro	Distro

Everything cargoship writes itself - cargoship inventory from-ansible, the Ansible collection, the OpenTofu provider, the generated example packages - now emits the new spelling. No removal date is set for the old one; when one is, it will be announced in a release before it lands.

---

Development and Build Workflows

Task automation is built using Mage, which drives builds, tests and code generation with the host Go toolchain.

Mage Automation

Mage handles tasks including local compilation, e2e test execution, schema updates, and documentation generation. Key generated files include:

• build/cargoship_* (Release binaries)
• docs/commands/* (Cobra command references)
• docs/phases/* (Orchestration phase explanations)
• docs/schema/* (Schema field reference, from the same struct reflection as schema/*.json)
• docs/ansible/<collection>/module_*.md and docs/ansible/<collection>/role_*.md (Ansible collection reference)
• docs/index.md and docs/security.md (this file and .github/SECURITY.md, with their links rewritten for the book)
• docs/SUMMARY.md (mdBook layout manifest)
• schema/*.json (YAML validations)

Continuous Integration (CI) and Releases

GitHub Actions workflows run lint checks, dependency validation, cross-compilation, and end-to-end tests for every pull request. Releases are managed through GoReleaser, automating build signing, verification, and asset publishing.

---

Inspiration

Cargoship draws major design and engineering inspiration from:

• k0sproject/k0sctl - For elegant SSH-based multi-node orchestration and configuration patterns.
• zarf-dev/zarf - For air-gapped image and file packaging and offline-first design.