Route containerd ns mirror requests to configured registries

README.md

@@ -447,6 +447,65 @@ Or pull images directly:
 docker pull localhost:8080/library/nginx:latest
 ```
 
+#### containerd (Kubernetes, k3s, nerdctl)
+
+containerd mirrors send the original registry host in an `ns` query
+parameter, so one mirror entry can serve Docker Hub and the registries
+configured in `upstream.oci` whose URL has no path. Point containerd's CRI
+plugin at a hosts directory in `/etc/containerd/config.toml` and restart
+containerd. For containerd 2.x:
+
+```toml
+version = 3
+
+[plugins."io.containerd.cri.v1.images".registry]
+  config_path = "/etc/containerd/certs.d"
+```
+
+For containerd 1.x:
+
+```toml
+version = 2
+
+[plugins."io.containerd.grpc.v1.cri".registry]
+  config_path = "/etc/containerd/certs.d"
+```
+
+Then create `/etc/containerd/certs.d/_default/hosts.toml`:
+
+```toml
+[host."http://proxy.example.com:8080"]
+  capabilities = ["pull", "resolve"]
+```
+
+To check that CRI picked up the directory, confirm the setting containerd
+runs with and pull through CRI (on k3s use `k3s crictl`):
+
+```bash
+containerd config dump | grep config_path
+crictl pull docker.io/library/nginx:latest
+```
+
+The proxy logs a `container manifest request` line for the pull and, when
+`access_log.path` is set, an access-log entry. `ctr images pull --hosts-dir`
+is no substitute for this check: it reads the directory itself and succeeds
+even while CRI still has no `config_path`.
+
+`docker.io` and the host of `upstream.oci_default` use the default registry;
+the host of each `upstream.oci` URL uses that named registry, and so do the
+hosts listed for it in `upstream.oci_mirrors`. Pulls for any other registry get
+`404 NAME_UNKNOWN`, and containerd falls back to the registry itself. Existing
+per-registry `hosts.toml` files that point at `/v2/upstream/{name}` with
+`override_path = true` keep working for that upstream's own host and for
+Docker Hub. If the upstream is a mirror of the registry the nodes pull from,
+say an Artifactory remote of `ghcr.io`, list that registry in
+`upstream.oci_mirrors` (`ghcr: ["ghcr.io"]`); otherwise those pulls get the
+404 as well. See [docs/configuration.md](docs/configuration.md) for how
+hosts are matched and which entry wins when two share a host. k3s generates
+the hosts directory itself from `/etc/rancher/k3s/registries.yaml`; `mirrors:
+{"*": {endpoint: ["http://proxy.example.com:8080"]}}` produces the same
+`_default` entry.
+
 ### Helm
 
 Configure each HTTP chart repository with a name, then add the matching proxy

config.example.yaml

@@ -229,6 +229,12 @@ upstream:
   # oci:
   #   ghcr: "https://ghcr.io"
 
+  # Registries a named OCI upstream mirrors, e.g. when ghcr above points at
+  # an Artifactory remote of ghcr.io. containerd mirror requests whose ns
+  # parameter names one of these hosts are served by that upstream.
+  # oci_mirrors:
+  #   ghcr: ["ghcr.io"]
+
   # Named Alpine APK repositories (used by /apk/{name}/).
   # Defaults to {"alpine": "https://dl-cdn.alpinelinux.org/alpine"} when empty;
   # configuring any entry replaces that default.

docs/configuration.md

@@ -290,6 +290,57 @@ mise section in the README for the client-side `url_replacements`.
 while `upstream.oci` selects named registries through the `upstream/{name}/`
 repository prefix. For example, `oci://proxy.example.com/upstream/ghcr/owner/chart`
 uses the `ghcr` registry with `owner/chart` as its repository.
+
+containerd mirror requests carry the original registry host in an `ns` query
+parameter (see the containerd section in the README). The proxy only looks the
+host up and never connects to it: `docker.io`, `index.docker.io`,
+`registry-1.docker.io` and the host of `upstream.oci_default` select the default
+registry, and the host of each `upstream.oci` URL selects that named registry.
+Hosts are compared case-insensitively. The scheme's default port (443 for
+`https`, 80 for `http`) may be spelled out or left out in the image reference;
+any other port must match exactly, so `https://registry.example:80` and
+`https://registry.example` are two different registries. An
+unknown host returns `404 NAME_UNKNOWN`, so containerd falls back to its next
+host. Registry URLs with a path (for example an Artifactory repository path)
+are not reachable through their own host in `ns`, only through
+`upstream/{name}/` or through `upstream.oci_mirrors`. When two
+entries share a host, the proxy logs a warning at startup; the default registry
+wins, otherwise the alphabetically first name. Pulls through `ns`,
+`upstream/{name}/` and unprefixed requests share the same cache entries.
+Requests that combine the `upstream/{name}/` prefix with `ns`, as per-registry
+containerd mirrors with `override_path = true` send them, are routed by the
+prefix. They are accepted only when `ns` is the upstream's own host, a host
+listed for it in `upstream.oci_mirrors`, or Docker Hub (`docker.io` and its
+aliases). Docker Hub repository names have two path components and can never
+start with `upstream/`, so a Docker Hub mirror behind any prefix stays
+reachable. Any other host gets `404 NAME_UNKNOWN`: a containerd `_default`
+mirror sends the same request for an image such as
+`unconfigured.example/upstream/ghcr/owner/app`, and that pull must not be
+answered from the `ghcr` upstream.
+
+When a named upstream mirrors another registry, for example an Artifactory
+remote of `ghcr.io`, say so in `upstream.oci_mirrors`:
+
+```yaml
+upstream:
+  oci:
+    ghcr: "https://artifactory.example.com/artifactory/api/docker/ghcr-remote"
+  oci_mirrors:
+    ghcr: ["ghcr.io"]
+```
+
+Each entry lists bare registry hosts as containerd sends them, optionally with
+a port. A host can be listed for one upstream only. Those hosts select the
+upstream for unprefixed `ns` requests and are accepted as `ns` on its
+`upstream/{name}/` prefix. A host that is also the host of a configured
+registry URL stays with that registry for unprefixed requests; the proxy logs a
+warning at startup.
+
+A host that is accepted on a prefix route gives up image paths of the form
+`<host>/upstream/{name}/...` under a `_default` mirror: with `ghcr: ["ghcr.io"]`,
+a pull of `ghcr.io/upstream/ghcr/owner/app` is answered with `owner/app` from
+the `ghcr` upstream, because containerd sends the same request for it as a
+per-registry mirror does. The same applies to the upstream's own host.
 When the proxy uses plain HTTP (for example `localhost:8080`), pass
 `--plain-http` to Helm OCI commands.
 

internal/config/config.go

@@ -63,6 +63,7 @@ import (
 	"encoding/base64"
 	"encoding/json"
 	"fmt"
+	"net"
 	"net/url"
 	"os"
 	"path/filepath"
@@ -630,6 +631,12 @@ type UpstreamConfig struct {
 	// oci://proxy.example.com/upstream/ghcr/owner/chart.
 	OCI map[string]string `json:"oci" yaml:"oci"`
 
+	// OCIMirrors lists, per upstream.oci name, the registry hosts that
+	// upstream mirrors, for example {"ghcr": ["ghcr.io"]} for an Artifactory
+	// remote of ghcr.io. containerd requests whose ns query parameter names
+	// one of these hosts are served by that upstream.
+	OCIMirrors map[string][]string `json:"oci_mirrors" yaml:"oci_mirrors"`
+
 	// Generic maps names to plain HTTP upstream base URLs, served at
 	// /generic/{name}/. The remaining request path and query string are
 	// appended to the upstream URL. GitHub release asset paths
@@ -687,6 +694,9 @@ func (u *UpstreamConfig) Validate() error {
 	if err := validateNamedUpstreams("upstream.oci", u.OCI); err != nil {
 		return err
 	}
+	if err := validateOCIMirrors(u.OCIMirrors, u.OCI); err != nil {
+		return err
+	}
 	if err := validateNamedUpstreams("upstream.generic", u.Generic); err != nil {
 		return err
 	}
@@ -708,6 +718,56 @@ func (u *UpstreamConfig) Validate() error {
 // paths, which a repository of the same name would shadow.
 var debianReservedRepositoryNames = []string{"pool", "dists"}
 
+// validateOCIMirrors checks that every upstream.oci_mirrors entry belongs to
+// an upstream.oci name and lists bare registry hosts, the form containerd
+// sends in the ns query parameter. A host may belong to one upstream only;
+// otherwise both prefixes would take it while unprefixed requests silently
+// went to one of them.
+func validateOCIMirrors(mirrors map[string][]string, upstreams map[string]string) error {
+	names := make([]string, 0, len(mirrors))
+	for name := range mirrors {
+		names = append(names, name)
+	}
+	sort.Strings(names)
+	owners := make(map[string]string)
+	for _, name := range names {
+		if _, ok := upstreams[name]; !ok {
+			return fmt.Errorf("invalid upstream.oci_mirrors name %q: no upstream.oci entry of that name", name)
+		}
+		for _, host := range mirrors[name] {
+			parsed, err := url.Parse("https://" + host)
+			if host == "" || err != nil || parsed.Host != host {
+				return fmt.Errorf("invalid upstream.oci_mirrors.%s host %q: must be a registry host such as ghcr.io or registry.example:5000", name, host)
+			}
+			key := ociMirrorHostKey(host)
+			if owner, taken := owners[key]; taken && owner != name {
+				return fmt.Errorf("invalid upstream.oci_mirrors.%s host %q: already listed for %s", name, host, owner)
+			}
+			owners[key] = name
+		}
+	}
+	return nil
+}
+
+// ociMirrorHostKey normalizes an upstream.oci_mirrors host the way the
+// container handler builds its ns lookup keys: lowercase, IPv6 literals in
+// brackets, an empty or default https port dropped. Two entries with the same
+// key would select the same route.
+func ociMirrorHostKey(host string) string {
+	name, port, err := net.SplitHostPort(host)
+	if err != nil {
+		name, port = strings.TrimSuffix(strings.TrimPrefix(host, "["), "]"), ""
+	}
+	name = strings.ToLower(name)
+	if strings.Contains(name, ":") {
+		name = "[" + name + "]"
+	}
+	if port == "" || port == "443" {
+		return name
+	}
+	return name + ":" + port
+}
+
 func validateNamedUpstreams(field string, upstreams map[string]string) error {
 	for name, upstreamURL := range upstreams {
 		if name == "" || name == "." || name == ".." || strings.ContainsAny(name, `/\\`) {

internal/config/config_test.go

@@ -1336,6 +1336,84 @@ func TestValidateNamedUpstreams(t *testing.T) {
 			},
 			wantErr: true,
 		},
+		{
+			name: "OCI mirrors for a configured upstream",
+			modify: func(cfg *Config) {
+				cfg.Upstream.OCI = map[string]string{"ghcr": "https://artifactory.example/api/docker/ghcr-remote"}
+				cfg.Upstream.OCIMirrors = map[string][]string{"ghcr": {"ghcr.io", "registry.example:5000"}}
+			},
+		},
+		{
+			name: "OCI mirrors for an unknown upstream",
+			modify: func(cfg *Config) {
+				cfg.Upstream.OCI = map[string]string{"ghcr": "https://ghcr.io"}
+				cfg.Upstream.OCIMirrors = map[string][]string{"quay": {"quay.io"}}
+			},
+			wantErr: true,
+		},
+		{
+			name: "OCI mirror given as URL",
+			modify: func(cfg *Config) {
+				cfg.Upstream.OCI = map[string]string{"ghcr": "https://mirror.example"}
+				cfg.Upstream.OCIMirrors = map[string][]string{"ghcr": {"https://ghcr.io"}}
+			},
+			wantErr: true,
+		},
+		{
+			name: "OCI mirror with a path",
+			modify: func(cfg *Config) {
+				cfg.Upstream.OCI = map[string]string{"ghcr": "https://mirror.example"}
+				cfg.Upstream.OCIMirrors = map[string][]string{"ghcr": {"ghcr.io/owner"}}
+			},
+			wantErr: true,
+		},
+		{
+			name: "OCI mirror host with credentials",
+			modify: func(cfg *Config) {
+				cfg.Upstream.OCI = map[string]string{"ghcr": "https://mirror.example"}
+				cfg.Upstream.OCIMirrors = map[string][]string{"ghcr": {"user@ghcr.io"}}
+			},
+			wantErr: true,
+		},
+		{
+			name: "OCI mirror host listed for two upstreams",
+			modify: func(cfg *Config) {
+				cfg.Upstream.OCI = map[string]string{"art": "https://art.example", "nexus": "https://nexus.example"}
+				cfg.Upstream.OCIMirrors = map[string][]string{"art": {"ghcr.io"}, "nexus": {"GHCR.io:443"}}
+			},
+			wantErr: true,
+		},
+		{
+			name: "OCI mirror host listed for two upstreams with an empty port",
+			modify: func(cfg *Config) {
+				cfg.Upstream.OCI = map[string]string{"art": "https://art.example", "nexus": "https://nexus.example"}
+				cfg.Upstream.OCIMirrors = map[string][]string{"art": {"ghcr.io"}, "nexus": {"ghcr.io:"}}
+			},
+			wantErr: true,
+		},
+		{
+			name: "OCI mirror IPv6 host listed for two upstreams",
+			modify: func(cfg *Config) {
+				cfg.Upstream.OCI = map[string]string{"art": "https://art.example", "nexus": "https://nexus.example"}
+				cfg.Upstream.OCIMirrors = map[string][]string{"art": {"[::1]"}, "nexus": {"[::1]:"}}
+			},
+			wantErr: true,
+		},
+		{
+			name: "OCI mirror hosts that differ only in a non-default port",
+			modify: func(cfg *Config) {
+				cfg.Upstream.OCI = map[string]string{"art": "https://art.example", "nexus": "https://nexus.example"}
+				cfg.Upstream.OCIMirrors = map[string][]string{"art": {"registry.example"}, "nexus": {"registry.example:5000"}}
+			},
+		},
+		{
+			name: "OCI mirror host is empty",
+			modify: func(cfg *Config) {
+				cfg.Upstream.OCI = map[string]string{"ghcr": "https://mirror.example"}
+				cfg.Upstream.OCIMirrors = map[string][]string{"ghcr": {""}}
+			},
+			wantErr: true,
+		},
 		{
 			name: "OCI upstream URL is not absolute",
 			modify: func(cfg *Config) {