Skip to content

[Telemetry] Prevent crashes in shell-less containers - #32226

Merged
bmehta001 merged 3 commits into
mainfrom
bhamehta/fix-chiseled-telemetry-init
Aug 24, 2026
Merged

bmehta001 merged 3 commits into
mainfrom
bhamehta/fix-chiseled-telemetry-init

Conversation

@bmehta001

@bmehta001 bmehta001 commented Aug 24, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fixes #32173.

ONNX Runtime 1.29.0 crashes during OrtEnv initialization in chiseled/distroless Linux images when POSIX telemetry is enabled. These images omit both /etc/machine-id and /bin/sh, so cpp_client_telemetry reaches its device-ID fallback and popen() fails.

The SDK currently constructs std::shared_ptr<FILE>(nullptr, pclose). A shared_ptr created with a custom deleter invokes that deleter even for a null pointer, so teardown calls pclose(nullptr) and segfaults in _IO_new_fclose.

Own the FILE* with a lambda-deleter unique_ptr. Exec() has sole ownership of the pipe, and unique_ptr naturally skips its deleter when popen() returns null, allowing the empty-result fallback to work as intended when no shell is available.

Validation

  • Reproduced the issue with the released Microsoft.ML.OnnxRuntime 1.29.0 package in the filesystem from mcr.microsoft.com/dotnet/aspnet:10.0-noble-chiseled under its default UID.
    • Telemetry enabled: deterministic SIGSEGV at OrtEnv.Instance().
    • ORT_DISABLE_TELEMETRY=1: initialization succeeds.
  • Captured the failing native backtrace: pclose(nullptr) enters _IO_new_fclose(fp=0x0) immediately after the failed popen() fallback.
  • Built the telemetry-enabled Linux shared library with this unique-ownership implementation and warnings treated as errors.
  • Replaced the native library in the same managed repro and chiseled filesystem.
    • Telemetry enabled: initialization succeeds.
    • ORT_DISABLE_TELEMETRY=1: initialization succeeds.

Upstream

The SDK fix is proposed in microsoft/cpp_client_telemetry#1523. ORT should retain this compatibility patch until it pins an SDK release containing that change.

Exec() is used only by the non-Apple legacy device-ID fallback when /etc/machine-id is unavailable. ORT replaces the SDK-generated ID with its own persistent hashed device ID after LogManager initialization, but the SDK probe currently runs before that override. A future SDK option to suppress automatic device-ID discovery would let ORT avoid this unnecessary shell probe entirely; the null-safe cleanup is still required for existing SDK consumers and versions.

cpp_client_telemetry constructs a shared_ptr with pclose as its deleter even when popen fails. Chiseled images have no /bin/sh, so telemetry initialization calls pclose(nullptr) and terminates the process.

Use unique_ptr ownership so a null pipe is not passed to pclose and telemetry can degrade gracefully.

Files changed:
- cmake/patches/cpp_client_telemetry/cpp_client_telemetry.patch: make the POSIX pipe owner null-safe.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d7d2f27a-7339-4585-ad02-9f89ce20ef40
Copilot AI balanced review requested due to automatic review settings August 24, 2026 01:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Prevents telemetry initialization crashes in shell-less Linux containers by making failed popen() ownership null-safe.

Changes:

  • Replaces shared_ptr<FILE> with unique_ptr<FILE> and a pclose deleter.
  • Preserves the existing empty-result fallback when popen() fails.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

bmehta001 and others added 2 commits August 23, 2026 21:46
Keep the existing shared ownership shape while making its custom deleter null-safe. This matches cpp_client_telemetry#1523 and avoids the function-pointer warning that occurred under the Linux warnings-as-errors build.

Files changed:
- cmake/patches/cpp_client_telemetry/cpp_client_telemetry.patch: guard pclose in the shared_ptr deleter.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d7d2f27a-7339-4585-ad02-9f89ce20ef40
The pipe returned by popen has one owner. Matching the upstream unique_ptr fix makes that lifetime explicit and naturally avoids pclose for a null handle.

Files changed:`n- cmake/patches/cpp_client_telemetry/cpp_client_telemetry.patch: use the upstream lambda-deleter unique_ptr.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>`nCopilot-Session: d7d2f27a-7339-4585-ad02-9f89ce20ef40
@bmehta001 bmehta001 self-assigned this Aug 24, 2026
@bmehta001
bmehta001 enabled auto-merge (squash) August 24, 2026 07:37
@bmehta001
bmehta001 merged commit 9aa8fb0 into main Aug 24, 2026
88 of 89 checks passed
@bmehta001
bmehta001 deleted the bhamehta/fix-chiseled-telemetry-init branch August 24, 2026 17:33
bmehta001 added a commit to microsoft/onnxruntime-genai that referenced this pull request Sep 17, 2026
…2563)

## Summary

Ports the relevant telemetry changes from microsoft/onnxruntime#29872,
microsoft/onnxruntime#32226, and microsoft/onnxruntime#32425 into GenAI:

- upgrades `cpp_client_telemetry` to 3.10.240.1, which includes the
null-safe POSIX `popen()` cleanup from
microsoft/cpp_client_telemetry#1523
- moves routine model, generation, adapter, and runtime-error events to
deterministic 1% client-side sampling while keeping `ProcessInfo`
unsampled
- disables SDK tracing at runtime and compiles internal 1DS logging out
of FetchContent builds
- adds bounded container, VM, WSL, cloud-guest, and Android-emulator
classification to `ProcessInfo` without transmitting raw host evidence
- uses the new SDK's canonical CMake options while retaining GenAI's
separate process-exit teardown fixes

GenAI already contained the relevant microsoft/onnxruntime#29872
behavior for default-on supported builds, explicit opt-out, minimized
common context, network-context cleanup, and persistent generated device
IDs. ORT-only work from microsoft/onnxruntime#32425—session/EP lifecycle
consolidation, RuntimePerf/SystemMetrics changes, and EP device
census—does not map to GenAI's event model and is intentionally
excluded.

## Validation

- Repository `lintrunner`: clean
- Linux/WSL Release telemetry build:
  - `onnxruntime-genai`
  - `telemetry_helpers_tests`
  - `telemetry_device_info_tests`
- Linux/WSL focused CTest: 2/2 passed
- Visual Studio 2026 telemetry configure: passed
- Visual Studio 2026 compile could not run locally because the installed
toolchain lacks Spectre-mitigated libraries (`MSB8040`), including for
GoogleTest and vendored dependencies
- Verified the pinned 3.10.240.1 archive SHA-1 and applied the
compatibility patch against a clean extracted archive

---------

Copilot-Session: b72147d1-3937-4b13-940c-33299589af1b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants