Skip to content

Port ONNX Runtime telemetry volume and host classification updates - #2563

Merged
bmehta001 merged 4 commits into
mainfrom
telemetry/port-ort-updates
Sep 17, 2026
Merged

bmehta001 merged 4 commits into
mainfrom
telemetry/port-ort-updates

Conversation

@bmehta001

Copy link
Copy Markdown
Contributor

Summary

Ports the relevant telemetry changes from microsoft/onnxruntime#29872, microsoft/onnxruntime#32226, and microsoft/onnxruntime#32425 into GenAI:

  • upgrades cpp_client_telemetry to 3.10.240.1, which includes the null-safe POSIX popen() cleanup from Guard POSIX pipe cleanup when popen fails cpp_client_telemetry#1523
  • moves routine model, generation, adapter, and runtime-error events to deterministic 1% client-side sampling while keeping ProcessInfo unsampled
  • disables SDK tracing at runtime and compiles internal 1DS logging out of FetchContent builds
  • adds bounded container, VM, WSL, cloud-guest, and Android-emulator classification to ProcessInfo without transmitting raw host evidence
  • uses the new SDK's canonical CMake options while retaining GenAI's separate process-exit teardown fixes

GenAI already contained the relevant microsoft/onnxruntime#29872 behavior for default-on supported builds, explicit opt-out, minimized common context, network-context cleanup, and persistent generated device IDs. ORT-only work from microsoft/onnxruntime#32425—session/EP lifecycle consolidation, RuntimePerf/SystemMetrics changes, and EP device census—does not map to GenAI's event model and is intentionally excluded.

Validation

  • Repository lintrunner: clean
  • Linux/WSL Release telemetry build:
    • onnxruntime-genai
    • telemetry_helpers_tests
    • telemetry_device_info_tests
  • Linux/WSL focused CTest: 2/2 passed
  • Visual Studio 2026 telemetry configure: passed
  • Visual Studio 2026 compile could not run locally because the installed toolchain lacks Spectre-mitigated libraries (MSB8040), including for GoogleTest and vendored dependencies
  • Verified the pinned 3.10.240.1 archive SHA-1 and applied the compatibility patch against a clean extracted archive

Port the relevant ONNX Runtime telemetry updates so GenAI uses the released null-safe 1DS SDK, samples routine lifecycle events at 1%, suppresses SDK diagnostics, and reports coarse container/VM/emulator context without transmitting raw host evidence.

Files changed:
- cmake/deps.txt and cmake/telemetry.cmake: upgrade and configure cpp_client_telemetry 3.10.240.1.
- cmake/patches/cpp_client_telemetry/apply_patch.cmake: retain GenAI teardown fixes and compile SDK logging out.
- src/telemetry/*: add sampling, runtime trace suppression, CA selection, and host classification.
- test/CMakeLists.txt and test/cpp/telemetry_helpers_tests.cpp: cover deterministic sampling and classification.
- docs/Privacy.md: document sampling and coarse environment fields.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b72147d1-3937-4b13-940c-33299589af1b
Copilot AI lite review requested due to automatic review settings September 15, 2026 05:47
@bmehta001
bmehta001 requested a review from a team as a code owner September 15, 2026 05:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two critical build issues remain, and the dependency manifest needs regeneration.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Ports ONNX Runtime telemetry sampling, host classification, SDK updates, and privacy changes into GenAI.

Changes:

  • Upgrades and reconfigures cpp_client_telemetry 3.10.240.1.
  • Adds deterministic 1% sampling and bounded host-environment classification.
  • Adds tests, documentation, CMake integration, and compatibility patches.
File summaries
File Summary Review findings
test/cpp/telemetry_helpers_tests.cpp Adds sampling and classification tests. None
test/CMakeLists.txt Registers telemetry helper tests. Critical (3 votes): duplicate main may cause linker failures.
src/telemetry/telemetry.cpp Configures telemetry and ProcessInfo metadata. None
src/telemetry/telemetry_sampling.h Sets the 1% sampling rate. None
src/telemetry/telemetry_environment.h Implements host classification. Critical (2 votes): Windows max macro can break std::max.
src/telemetry/device_info.h Adds environment metadata fields. None
src/telemetry/device_info.cpp Collects bounded host evidence. None
docs/Privacy.md Documents sampling and classification. None
cmake/telemetry.cmake Updates SDK build configuration. None
cmake/patches/cpp_client_telemetry/apply_patch.cmake Applies SDK compatibility patches. None
cmake/deps.txt Pins cpp_client_telemetry 3.10.240.1. Nit (1 vote): regenerate the component manifest.
Review details

Suppressed comments (1)

cmake/deps.txt:25

  • This changes a dependency in cmake/deps.txt, but the checked-in cgmanifests/generated/cgmanifest.json is not updated and contains no cpp_client_telemetry registration. cmake/deps.txt:12 explicitly requires regenerating the component manifest for dependency changes; please regenerate and commit the resulting manifest so dependency inventory matches the pinned SDK.
cpp_client_telemetry;https://github.com/microsoft/cpp_client_telemetry/archive/refs/tags/v3.10.240.1.zip;96cd290d746b86a31c8e08dd167cf39d390c0134
  • Files reviewed: 11/11 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/telemetry/telemetry_environment.h Outdated
Comment thread test/CMakeLists.txt
Use the macro-safe host-confidence comparison already used by ONNX Runtime so Windows.h cannot rewrite std::max. Keep the standalone telemetry helper executable out of unit_tests to prevent duplicate main symbols. Extend cgmanifest generation for direct Git pins and GitHub release assets so the upgraded telemetry SDK is recorded.

Files changed: cgmanifests/generate_cgmanifest.py, cgmanifests/generated/cgmanifest.json, src/telemetry/telemetry_environment.h, test/CMakeLists.txt

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b72147d1-3937-4b13-940c-33299589af1b

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Address the critical Windows build issue and the moderate portability, coverage, and archive-parsing findings before approval.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (3)

cgmanifests/generate_cgmanifest.py:72

  • Changing this check from re.match to re.fullmatch regresses GitHub commit archives named <sha>.tar.gz: PurePosixPath(...).stem is <sha>.tar, so the commit is no longer recognized and the script instead queries a non-existent tag named <sha>. Normalize the .tar suffix before the full-match (and store the normalized value) so both .tar.gz and .zip commit archives remain supported.
    if tag is None and len(segments) == 5 and re.fullmatch(r"[0-9a-f]{40}", PurePosixPath(segments[4]).stem):
        commit = PurePosixPath(segments[4]).stem

src/telemetry/device_info.cpp:5

  • This include is outside the ORTGENAI_ENABLE_TELEMETRY guard below, so telemetry-off builds now pull in all of telemetry_environment.h (including Windows.h on Windows) even though the following comment says this translation unit is kept platform-header-free in that configuration. Keep the include inside the guard so the default build retains that isolation.
#include "telemetry_environment.h"

src/telemetry/device_info.cpp:249

  • The new runtime probe is not covered by the tests added here: telemetry_helpers_tests exercises only ClassifyHostEnvironment with synthetic evidence, while telemetry_device_info_tests does not assert any of the new DeviceInfo fields. A wrong /proc/DMI path or environment-variable probe would therefore pass CI; add an injectable evidence-collection seam or deterministic integration assertions for the Linux/Android/Apple probes.
TelemetryInternal::HostEnvironmentInfo GetHostEnvironmentInfo() {
  TelemetryInternal::HostEnvironmentEvidence evidence;
#if defined(__linux__) || defined(__ANDROID__)
  evidence.docker_marker = FileExists("/.dockerenv");
  evidence.podman_marker = FileExists("/run/.containerenv");
  evidence.kubernetes = !TelemetryInternal::GetTelemetryEnv("KUBERNETES_SERVICE_HOST").empty();
  • Files reviewed: 12/13 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread src/telemetry/device_info.cpp
Keep platform-only telemetry includes and probe helpers behind their matching feature and OS guards so telemetry-off and Windows /W4 /WX builds remain clean. Normalize compound archive suffixes before identifying commit-pinned dependencies so cgmanifest generation retains tar.gz support.

Files changed: cgmanifests/generate_cgmanifest.py, src/telemetry/device_info.cpp

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b72147d1-3937-4b13-940c-33299589af1b
@bmehta001

Copy link
Copy Markdown
Contributor Author

Addressed the second review pass in c795e87: (1) telemetry_environment.h is now included only when ORTGENAI_ENABLE_TELEMETRY is enabled; (2) Linux/Android probe helpers use the same platform guard as their call sites; and (3) cgmanifest parsing normalizes .tar.gz before commit-SHA detection. For the suppressed probe-coverage suggestion, no additional seam is being added: microsoft/onnxruntime at d6750b398d6c8fce90544ddef954e5016de524a2 uses the same fixed-path GetHostEnvironmentInfo probe (onnxruntime/core/platform/posix/telemetry.cc:751-803), while onnxruntime/test/platform/telemetry_environment_test.cc tests the deterministic classifier and has no probe-I/O tests. GenAI now matches that implementation and test boundary; host-dependent integration assertions would be nondeterministic.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The changes span telemetry behavior, privacy documentation, build integration, and dependency management.

Review details

Suppressed comments (1)

docs/Privacy.md:23

  • The new AdapterActivated event is independently emitted and sampled at the same 1% rate as the other routine events, but this privacy statement omits adapter events. Please include adapters so the documented event-volume policy covers all newly sampled event types.
Routine model, generation, and error events use deterministic 1% client-side sampling. A process information event remains unsampled and includes coarse container, virtual-machine, WSL, or emulator classification when detected. Raw container identifiers, host names, and workload identifiers are not transmitted.
  • Files reviewed: 12/13 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Include adapter activation events in the privacy statement so every routine event governed by deterministic one-percent sampling is documented.

Files changed: docs/Privacy.md

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b72147d1-3937-4b13-940c-33299589af1b

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The broad telemetry, dependency, build, and platform-classification changes require final human review.

Review details
  • Files reviewed: 12/13 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@bmehta001 bmehta001 self-assigned this Sep 15, 2026
@bmehta001
bmehta001 merged commit 03fd474 into main Sep 17, 2026
63 of 66 checks passed
@bmehta001
bmehta001 deleted the telemetry/port-ort-updates branch September 17, 2026 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants