From 22b0939af661768368d3627f404335fc5f7a331b Mon Sep 17 00:00:00 2001 From: Scott McCarty Date: Sun, 30 Aug 2026 17:37:39 -0400 Subject: [PATCH 1/2] docs(mcp-server): correct stale libstdc++ guidance, add distroless gotchas MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The profile claimed the Hummingbird Python runtime does not include libstdc++.so.6. That is no longer true — verified 2026-08-30 in both :latest and :latest-fips (6.0.36). Containerfiles still carrying the copy-forward workaround can drop it. libgomp remains absent from both runtime and builder; PyTorch bundles its own, so torch-based servers are unaffected, but packages expecting the system copy still fail. Adds a symptom-to-fix table for distroless failures. The build-time ones are loud and self-explaining; the dangerous class is third-party native libraries that shell out at import time without checking the result, which fail as a bare SIGSEGV with no traceback and no stderr. Documents that /etc/machine-id must be POPULATED, not empty. The empty "first boot" state does not short-circuit the fallback probe. Measured against onnxruntime 1.29.0 on quay.io/hummingbird/python:latest with telemetry enabled: absent -> SIGSEGV (139) empty -> SIGSEGV (139) populated -> imports cleanly Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013t1uJRVqhkikr52jXxxvck --- profiles/mcp-server.md | 54 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 50 insertions(+), 4 deletions(-) diff --git a/profiles/mcp-server.md b/profiles/mcp-server.md index 98f64eb..3acf9a6 100644 --- a/profiles/mcp-server.md +++ b/profiles/mcp-server.md @@ -160,14 +160,60 @@ ENV PATH="/app/venv/bin:$PATH" ENTRYPOINT ["python", "-m", ""] ``` -### Native Library Gap (libstdc++) +### Native Library Gap (libstdc++, libgomp) -The Hummingbird Python runtime image **does not include libstdc++.so.6**. Any pip package with C++ extensions (numpy, onnxruntime, scikit-learn) will install successfully but crash at import time. +`libstdc++.so.6` **now ships in the Hummingbird Python runtime** (verified 2026-08-30 in both `:latest` and `:latest-fips`, version 6.0.36). The old workaround that copied it forward is no longer needed — delete it if you find it in an existing Containerfile. -**Fix:** Copy `libstdc++` from the Hummingbird builder variant (same ecosystem): +`libgomp.so.1` is still absent from **both** the runtime and the builder variants. PyTorch bundles its own copy under `torch/lib`, so torch-based servers are fine. A package that expects the *system* libgomp (some numpy/scipy/scikit-learn builds) will still fail at import. + +Re-verify rather than trusting this note — the base images move: + +```bash +podman run --rm --entrypoint "" quay.io/hummingbird/python:latest python -c \ + "import glob; print(glob.glob('/usr/lib64/libstdc++.so*'), glob.glob('/usr/lib64/libgomp.so*'))" +``` + +### Distroless Gotchas + +The runtime image has **no `/bin/sh`**. A shell-form `RUN` fails outright at build time, which is loud and easy to fix. The dangerous cases are third-party native libraries that shell out at *import* time and do not check whether it worked — those fail silently with a SIGSEGV, no traceback, and no stderr. + +| Symptom | Cause | Fix | +|---|---|---| +| `RUN` fails: `executable file /bin/sh not found` | Shell-form `RUN` against the distroless runtime | Move every `RUN` into the `-builder` stage; `COPY` results forward | +| Exit 139 / SIGSEGV on `import onnxruntime`, no output | ORT ≥ 1.29 vendors `cpp_client_telemetry`, which falls back to `popen()` for a device ID. `popen()` returns NULL with no shell; the unchecked `pclose(NULL)` crashes. See HUM-6564 / [onnxruntime#32173](https://github.com/microsoft/onnxruntime/issues/32173), fixed by [#32226](https://github.com/microsoft/onnxruntime/pull/32226) but not yet released | `ENV ORT_DISABLE_TELEMETRY=1` **and** a populated `/etc/machine-id` | +| Package installs but crashes on import with a missing symbol | Native C++ extension, missing runtime library | Copy the library from the matching Hummingbird `-builder` — never from Fedora or UBI | +| Model / cache re-downloaded on every start | `HOME` is `/tmp` in the base image and is ephemeral | Pin `HF_HOME` (or equivalent) to a `VOLUME` path | +| Permission denied writing to an app directory | Runtime runs as uid **65532**, gid **0** | Create dirs in the builder, then `chgrp -R 0` and `chmod -R g=u` | + +### `/etc/machine-id` Must Be Populated, Not Empty + +Libraries that probe for a host identity read `/etc/machine-id` and fall through to a shell-based fallback when the value is unusable. The empty file that systemd documents as the "uninitialized / first boot" state **does not** short-circuit that fallback. + +Measured against `onnxruntime` 1.29.0 on `quay.io/hummingbird/python:latest` with telemetry left enabled: + +| `/etc/machine-id` | Result | +|---|---| +| absent | SIGSEGV (139) | +| present, empty | SIGSEGV (139) | +| present, populated | imports cleanly | + +Generate it in the builder stage and copy it in: + +```dockerfile +# builder stage +RUN tr -d - < /proc/sys/kernel/random/uuid > /etc/machine-id.seed +# runtime stage +COPY --from=builder /etc/machine-id.seed /etc/machine-id +``` + +Every container from the image shares this ID. That is acceptable only when telemetry is disabled and nothing uses machine-id for identity — which is the normal case for these servers. Do not use this pattern for anything that needs per-instance identity. + +Set the telemetry kill switches too. Either fix alone prevents the crash; keeping both means a regression in one code path does not take the service down: ```dockerfile -COPY --from=quay.io/hummingbird/python:latest-fips-builder /usr/lib64/libstdc++.so.6* /usr/lib64/ +ENV HF_HUB_DISABLE_TELEMETRY=1 \ + ORT_DISABLE_TELEMETRY=1 \ + DO_NOT_TRACK=1 ``` ### Required Labels From ba2b4e87a983e85744c5d97c05bb0edc2f76dbe3 Mon Sep 17 00:00:00 2001 From: Scott McCarty Date: Tue, 1 Sep 2026 13:06:39 -0400 Subject: [PATCH 2/2] docs(mcp-server): distinguish hi/ and quay.io/hummingbird native lib coverage The two registry paths differ. registry.access.redhat.com/hi/python:3.12 ships both libstdc++ and libgomp; quay.io/hummingbird/python:latest ships libstdc++ only. A Containerfile on the hi/ path that still dnf-installs both and copies them forward is doing redundant work. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013t1uJRVqhkikr52jXxxvck --- profiles/mcp-server.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/profiles/mcp-server.md b/profiles/mcp-server.md index 3acf9a6..296f39a 100644 --- a/profiles/mcp-server.md +++ b/profiles/mcp-server.md @@ -162,9 +162,17 @@ ENTRYPOINT ["python", "-m", ""] ### Native Library Gap (libstdc++, libgomp) -`libstdc++.so.6` **now ships in the Hummingbird Python runtime** (verified 2026-08-30 in both `:latest` and `:latest-fips`, version 6.0.36). The old workaround that copied it forward is no longer needed — delete it if you find it in an existing Containerfile. +`libstdc++.so.6` **now ships in the Hummingbird Python runtime.** The old workaround that copied it forward is no longer needed — delete it if you find it in an existing Containerfile. -`libgomp.so.1` is still absent from **both** the runtime and the builder variants. PyTorch bundles its own copy under `torch/lib`, so torch-based servers are fine. A package that expects the *system* libgomp (some numpy/scipy/scikit-learn builds) will still fail at import. +**The two registry paths are not identical.** Verified 2026-09-01: + +| Runtime image | `libstdc++.so.6` | `libgomp.so.1` | +|---|---|---| +| `registry.access.redhat.com/hi/python:3.12` | present | **present** | +| `quay.io/hummingbird/python:latest` | present | **absent** | +| `quay.io/hummingbird/python:latest-fips` | present | absent | + +So a Containerfile on the `hi/` path that still `dnf install`s `libgomp libstdc++` in the builder and copies them forward is doing redundant work. On the `quay.io/hummingbird` path, `libgomp` genuinely is missing: PyTorch bundles its own under `torch/lib` and is fine, but a package expecting the *system* libgomp (some numpy/scipy/scikit-learn builds) will fail at import. Re-verify rather than trusting this note — the base images move: