Branch container images are never deleted from ghcr.io

The "Clean up registry" workflow of NS8 modules fails with Package not found. (HTTP 404), so branch container images are never deleted from ghcr.io (example). The IMAGES_CLEANUP_TOKEN secret can list image versions but cannot delete them.

Each module's workflow switches to the workflow token (GITHUB_TOKEN) with permission to write packages. Do not remove IMAGES_CLEANUP_TOKEN: ns8-core still uses it.

Images of branches with a / in the name are never deleted either: they are published with / replaced by - (e.g. perf/carbon-icons-import becomes perf-carbon-icons-import), but the cleanup action looks for the last part of the name only (carbon-icons-import). The action must replace / with - too.

Pull requests

Cleanup of existing leftover images

This script deletes images of branches deleted before the fix: only versions whose tags all match the branch of a closed pull request that no longer exists, alone or followed by a commit hash (e.g. old piler-server images tagged my-branch and my-branch-1a2b3c4). Releases, latest, untagged versions and anything uncertain are kept.

Log in with a classic token with read:packages and delete:packages, owned by an organization owner: gh auth login --with-token
Dry run and review the list: ./cleanup-branch-images.sh ns8-matrix (no arguments for all repositories)
Delete: ./cleanup-branch-images.sh --delete ns8-matrix
For the nethesis organization, repeat with ORG=nethesis (the token owner must be a nethesis owner), after ci: fix clean registry workflow nethesis/ns8-nethvoice#1048 and ci: clean up branch images from registry nethesis/ns8-nethvoice-proxy#227 are merged: ORG=nethesis ./cleanup-branch-images.sh ns8-nethvoice ns8-nethvoice-proxy

cleanup-branch-images.sh

#!/bin/bash
#
# Delete leftover branch images from ghcr.io for NethServer NS8 modules.
#
# For each repository with a "Clean up registry" workflow, read the image
# names from .github/workflows/clean-registry.yml, then list every version
# of each image package. A version is deleted only when every one of its
# tags is the name of a deleted branch, that is:
#
#   - the head branch of a closed pull request of the repository (or of the
#     repository the package is linked to), with slashes replaced by dashes
#     as publish-branch does, optionally followed by "-" and a commit hash
#     (e.g. old piler-server images tagged "my-branch-1a2b3c4"), and
#   - not an existing branch or git tag of those repositories, and
#   - not latest, main, master, a version-like tag (e.g. 1.2.3, v2.0) or a
#     signature/attestation tag (sha256-...)
#
# Tags pushed by other repositories, and branches that never had a pull
# request, are therefore never deleted. Untagged versions are never touched.
# If any read request fails, the package is skipped: nothing is deleted
# from incomplete data.
#
# Usage:
#   [ORG=nethesis] cleanup-branch-images.sh [--delete] [repo ...]
#
# Without --delete it only prints what it would delete (dry run).
# Without repo arguments it scans every non-archived repo of the org.
# ORG selects the GitHub organization (default: NethServer).
#
# Requirements: bash, gh, jq. The gh token must be a classic PAT with
# read:packages and delete:packages, owned by a user with Admin role on the
# packages (e.g. an org owner). Check with: gh auth status
#
set -euo pipefail

ORG=${ORG:-NethServer}
DELETE=0
if [[ ${1:-} == --delete ]]; then
    DELETE=1
    shift
fi

# Scratch files, removed on exit
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT

# Repositories to scan: from the command line, or all of the org
if (( $# > 0 )); then
    repos=("$@")
else
    gh repo list "$ORG" -L 1000 --no-archived --json name -q '.[].name' | sort > "$tmp/repos"
    repos=()
    while read -r r; do repos+=("$r"); done < "$tmp/repos"
fi

total=0
failed=0

# Print a problem and count it in the final summary
warn() {
    echo "!! $*" >&2
    failed=$((failed + 1))
}

# Save the refs of a repository, once: $tmp/<repo>.live lists existing
# branches and git tags, $tmp/<repo>.gone lists the head branches of
# closed pull requests (fork branches excluded). Branch names get the
# same '/' to '-' replacement publish-branch applies to image tags
fetch_refs() {
    local r=$1
    # Already fetched for a previous image
    [[ -f $tmp/$r.done ]] && return 0
    gh api --paginate "repos/$ORG/$r/branches?per_page=100" -q '.[].name' > "$tmp/$r.branches" || return 1
    gh api --paginate "repos/$ORG/$r/tags?per_page=100" -q '.[].name' > "$tmp/$r.tags" || return 1
    gh api --paginate "repos/$ORG/$r/pulls?state=closed&per_page=100" \
        -q ".[] | select((.head.repo.full_name // \"\" | ascii_downcase) == (\"$ORG/$r\" | ascii_downcase)) | .head.ref" > "$tmp/$r.heads" || return 1
    { tr '/' '-' < "$tmp/$r.branches"; cat "$tmp/$r.tags"; } | sort -u > "$tmp/$r.live"
    tr '/' '-' < "$tmp/$r.heads" | sort -u > "$tmp/$r.gone"
    touch "$tmp/$r.done"
}

for repo in "${repos[@]}"; do
    # Skip repositories without the workflow
    workflow=$(gh api -H "Accept: application/vnd.github.raw" \
        "repos/$ORG/$repo/contents/.github/workflows/clean-registry.yml" 2>/dev/null) || continue
    # Image names from the line: images: "mail mail-dovecot ..."
    images=$(sed -n 's/^ *images: *"\{0,1\}\([^"]*\)"\{0,1\} *$/\1/p' <<<"$workflow")
    if [[ -z $images ]]; then
        warn "$repo: no images found in clean-registry.yml, skipped"
        continue
    fi

    for image in $images; do
        versions_path="orgs/$ORG/packages/container/$image/versions"

        # List the versions before reading the refs: a branch created in
        # the meantime is then seen as existing
        if ! gh api --paginate "$versions_path?per_page=100" > "$tmp/versions"; then
            warn "$repo/$image: cannot list versions, skipped"
            continue
        fi
        if ! linked=$(gh api "orgs/$ORG/packages/container/$image" -q '.repository.name // empty'); then
            warn "$repo/$image: cannot read package, skipped"
            continue
        fi
        # Read refs from the module repository and from the repository the
        # package is linked to, if different (e.g. sogo-server lists kickstart)
        sources=("$repo")
        [[ -n $linked && $linked != "$repo" ]] && sources+=("$linked")
        ok=1
        for r in "${sources[@]}"; do
            fetch_refs "$r" || { ok=0; break; }
        done
        if (( ! ok )); then
            warn "$repo/$image: cannot read branches, tags or pull requests of $r, skipped"
            continue
        fi
        # Merge the refs of all sources into two JSON arrays for jq
        live=$(for r in "${sources[@]}"; do cat "$tmp/$r.live"; done | jq -Rn '[inputs] | unique')
        gone=$(for r in "${sources[@]}"; do cat "$tmp/$r.gone"; done | jq -Rn '[inputs] | unique')

        # Print "<id> <tags>" for every leftover version: a tagged version
        # whose tags are all deleted branches, alone or followed by a commit
        # hash, never a release, latest, main, master or signature tag
        leftovers=$(jq -rs --argjson live "$live" --argjson gone "$gone" '
            add // [] | .[]
            | select(.metadata.container.tags | length > 0)
            | select(all(.metadata.container.tags[];
                . as $t
                | any($gone[]; . as $g
                    | (any($live[]; . == $g) | not)
                    and ($t == $g
                        or ($t | startswith($g + "-") and (.[($g | length) + 1:] | test("^[0-9a-f]{7,40}$")))))
                and (any($live[]; . == $t) | not)
                and (test("^(latest|main|master)$") | not)
                and (test("^v?[0-9]+\\.[0-9]+") | not)
                and (startswith("sha256-") | not)))
            | "\(.id) \(.metadata.container.tags | join(","))"
        ' "$tmp/versions")

        # Delete, or just print in dry run
        while read -r id tags; do
            [[ -n $id ]] || continue
            total=$((total + 1))
            if (( DELETE )); then
                if gh api --silent --input /dev/null -X DELETE "$versions_path/$id"; then
                    echo "deleted  $image:$tags ($id)"
                else
                    warn "failed $image:$tags ($id)"
                fi
            else
                echo "would delete  $image:$tags ($id)"
            fi
        done <<<"$leftovers"
    done
done

echo "== leftover versions: $total, problems: $failed" >&2