Repository navigation
[Bug]: gpu-operator images v26.7.0+, contain an uncompressed layer; v26.7.1+ also switched to OCI manifest types breaks Quay/oc-mirror mirroring #3166
Description
Evidence
1. Manifest-list media type per version
$ for tag in v25.10.0 v26.3.3 v26.7.0 v26.7.1 v26.7.2; do
echo "=== $tag ==="
skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator:$tag | jq -r .mediaType
done
=== v25.10.0 ===
application/vnd.docker.distribution.manifest.list.v2+json
=== v26.3.3 ===
application/vnd.docker.distribution.manifest.list.v2+json
=== v26.7.0 ===
application/vnd.docker.distribution.manifest.list.v2+json
=== v26.7.1 ===
application/vnd.oci.image.index.v1+json
=== v26.7.2 ===
application/vnd.oci.image.index.v1+json
2. Layer media types per version (amd64)
$ for tag in v25.10.0 v26.3.3 v26.7.0 v26.7.1 v26.7.2; do
echo "=== $tag ==="
digest=$(skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator:$tag \
| jq -r '.manifests[] | select(.platform.architecture=="amd64") | .digest')
skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator@$digest \
| jq -r '.layers[].mediaType' | sort | uniq -c
done
=== v25.10.0 ===
36 application/vnd.docker.image.rootfs.diff.tar.gzip
=== v26.3.3 ===
42 application/vnd.docker.image.rootfs.diff.tar.gzip
=== v26.7.0 ===
1 application/vnd.docker.image.rootfs.diff.tar
42 application/vnd.docker.image.rootfs.diff.tar.gzip
=== v26.7.1 ===
1 application/vnd.oci.image.layer.v1.tar
42 application/vnd.oci.image.layer.v1.tar+gzip
=== v26.7.2 ===
1 application/vnd.oci.image.layer.v1.tar
42 application/vnd.oci.image.layer.v1.tar+gzip
3. The uncompressed layer is confirmed at the byte level, not just a label
$ TOKEN=$(curl -s "https://nvcr.io/token?service=nvcr.io&scope=repository:nvidia/gpu-operator:pull" \
| jq -r .token)
$ amd64_digest=$(skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator:v26.7.2 \
| jq -r '.manifests[] | select(.platform.architecture=="amd64") | .digest')
$ layer_digest=$(skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator@$amd64_digest \
| jq -r '.layers[] | select(.mediaType | contains("gzip") | not) | .digest')
$ curl -sL -H "Authorization: Bearer $TOKEN" \
"https://nvcr.io/v2/nvidia/gpu-operator/blobs/$layer_digest" -o layer.bin
$ file -b layer.bin
POSIX tar archive (GNU)
$ tar -tvf layer.bin
etc/
etc/ssl/
etc/ssl/fipsmodule.cnf
etc/ssl/openssl-fips.cnf
etc/ssl/openssl-nofips.cnf
This same layer (containing OpenSSL/FIPS config files) is present,
uncompressed, in v26.7.0, v26.7.1, and v26.7.2. It does not exist at all
in v25.10.0 or v26.3.3.
Impact
Customers mirroring certified operator bundles (e.g. via oc-mirror) to
a disconnected Red Hat Quay registry fail with:
[ERROR] : [Worker] error mirroring image
docker://nvcr.io/nvidia/cloud-native/<image>@sha256:...
error: Manifest list must be converted to type
"application/vnd.docker.distribution.manifest.list.v2+json" ...
"Instructed to preserve digests"
The only workaround today is a manual skopeo copy --format v2s2 --multi-arch=all re-push per image, which doesn't scale for
disconnected/air-gapped operator mirroring workflows.
Fix
- Identify the build step (introduced between v26.3.3 and v26.7.0) that
adds the FIPS/OpenSSL config layer, and ensure it's gzip-compressed
like every other layer. - Revert to publishing Docker v2 Schema2 manifest/manifest-list media
types (as in v25.10.0/v26.3.3/v26.7.0) instead of OCI-only, given the
broader ecosystem's Docker v2s2 compatibility expectations for
certified/catalog content.
Environment
- Images affected:
nvcr.io/nvidia/gpu-operator(and likely other
nvidia/cloud-native/*child images referenced by the certified
gpu-operator-bundle) - Versions: v25.10.0, v26.3.3, v26.7.0, v26.7.1, v26.7.2 (current latest)
- Tools used to reproduce:
skopeo1.22.3,jq,file,tar,curl
Activity
rahulait commentedon Nov 4, 2026
Thanks @nikitabugrovsky , we are looking into this.
For switching manifeststype to OCI:
We believe this happened because buildkit enabled oci-mediatypes=true by default in moby/buildkit#6824 which was in June, 2026
Later, docker engine 29.6.0 picked up this buildkit in its release: https://docs.docker.com/engine/release-notes/29/#2960
Its possible newer images are built on newer version of docker and hence manifest type has changed to OCI.
For uncompressed layer, we are investigating why there is an uncompressed layer in the image.
rahulait commentedon Nov 4, 2026
We are fixing the uncompressed layers in 26.7.3. We do not currently plan a blanket revert to Docker v2 Schema2. The reported schema error specifically rejects an uncompressed Docker layer descriptor (application/vnd.docker.image.rootfs.diff.tar), and changing manifest formats alone would not resolve that. Our planned fix addresses layer compression; we have not established that OCI media types also need to change.