Port ONNX Runtime telemetry volume and host classification updates - #2563
Conversation
Port the relevant ONNX Runtime telemetry updates so GenAI uses the released null-safe 1DS SDK, samples routine lifecycle events at 1%, suppresses SDK diagnostics, and reports coarse container/VM/emulator context without transmitting raw host evidence. Files changed: - cmake/deps.txt and cmake/telemetry.cmake: upgrade and configure cpp_client_telemetry 3.10.240.1. - cmake/patches/cpp_client_telemetry/apply_patch.cmake: retain GenAI teardown fixes and compile SDK logging out. - src/telemetry/*: add sampling, runtime trace suppression, CA selection, and host classification. - test/CMakeLists.txt and test/cpp/telemetry_helpers_tests.cpp: cover deterministic sampling and classification. - docs/Privacy.md: document sampling and coarse environment fields. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b72147d1-3937-4b13-940c-33299589af1b
There was a problem hiding this comment.
🟡 Changes recommended
Two critical build issues remain, and the dependency manifest needs regeneration.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Ports ONNX Runtime telemetry sampling, host classification, SDK updates, and privacy changes into GenAI.
Changes:
- Upgrades and reconfigures
cpp_client_telemetry3.10.240.1. - Adds deterministic 1% sampling and bounded host-environment classification.
- Adds tests, documentation, CMake integration, and compatibility patches.
File summaries
| File | Summary | Review findings |
|---|---|---|
test/cpp/telemetry_helpers_tests.cpp |
Adds sampling and classification tests. | None |
test/CMakeLists.txt |
Registers telemetry helper tests. | Critical (3 votes): duplicate main may cause linker failures. |
src/telemetry/telemetry.cpp |
Configures telemetry and ProcessInfo metadata. |
None |
src/telemetry/telemetry_sampling.h |
Sets the 1% sampling rate. | None |
src/telemetry/telemetry_environment.h |
Implements host classification. | Critical (2 votes): Windows max macro can break std::max. |
src/telemetry/device_info.h |
Adds environment metadata fields. | None |
src/telemetry/device_info.cpp |
Collects bounded host evidence. | None |
docs/Privacy.md |
Documents sampling and classification. | None |
cmake/telemetry.cmake |
Updates SDK build configuration. | None |
cmake/patches/cpp_client_telemetry/apply_patch.cmake |
Applies SDK compatibility patches. | None |
cmake/deps.txt |
Pins cpp_client_telemetry 3.10.240.1. |
Nit (1 vote): regenerate the component manifest. |
Review details
Suppressed comments (1)
cmake/deps.txt:25
- This changes a dependency in
cmake/deps.txt, but the checked-incgmanifests/generated/cgmanifest.jsonis not updated and contains no cpp_client_telemetry registration.cmake/deps.txt:12explicitly requires regenerating the component manifest for dependency changes; please regenerate and commit the resulting manifest so dependency inventory matches the pinned SDK.
cpp_client_telemetry;https://github.com/microsoft/cpp_client_telemetry/archive/refs/tags/v3.10.240.1.zip;96cd290d746b86a31c8e08dd167cf39d390c0134
- Files reviewed: 11/11 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Use the macro-safe host-confidence comparison already used by ONNX Runtime so Windows.h cannot rewrite std::max. Keep the standalone telemetry helper executable out of unit_tests to prevent duplicate main symbols. Extend cgmanifest generation for direct Git pins and GitHub release assets so the upgraded telemetry SDK is recorded. Files changed: cgmanifests/generate_cgmanifest.py, cgmanifests/generated/cgmanifest.json, src/telemetry/telemetry_environment.h, test/CMakeLists.txt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b72147d1-3937-4b13-940c-33299589af1b
There was a problem hiding this comment.
🟡 Changes recommended
Address the critical Windows build issue and the moderate portability, coverage, and archive-parsing findings before approval.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (3)
cgmanifests/generate_cgmanifest.py:72
- Changing this check from
re.matchtore.fullmatchregresses GitHub commit archives named<sha>.tar.gz:PurePosixPath(...).stemis<sha>.tar, so the commit is no longer recognized and the script instead queries a non-existent tag named<sha>. Normalize the.tarsuffix before the full-match (and store the normalized value) so both.tar.gzand.zipcommit archives remain supported.
if tag is None and len(segments) == 5 and re.fullmatch(r"[0-9a-f]{40}", PurePosixPath(segments[4]).stem):
commit = PurePosixPath(segments[4]).stem
src/telemetry/device_info.cpp:5
- This include is outside the
ORTGENAI_ENABLE_TELEMETRYguard below, so telemetry-off builds now pull in all oftelemetry_environment.h(includingWindows.hon Windows) even though the following comment says this translation unit is kept platform-header-free in that configuration. Keep the include inside the guard so the default build retains that isolation.
#include "telemetry_environment.h"
src/telemetry/device_info.cpp:249
- The new runtime probe is not covered by the tests added here:
telemetry_helpers_testsexercises onlyClassifyHostEnvironmentwith synthetic evidence, whiletelemetry_device_info_testsdoes not assert any of the newDeviceInfofields. A wrong/proc/DMI path or environment-variable probe would therefore pass CI; add an injectable evidence-collection seam or deterministic integration assertions for the Linux/Android/Apple probes.
TelemetryInternal::HostEnvironmentInfo GetHostEnvironmentInfo() {
TelemetryInternal::HostEnvironmentEvidence evidence;
#if defined(__linux__) || defined(__ANDROID__)
evidence.docker_marker = FileExists("/.dockerenv");
evidence.podman_marker = FileExists("/run/.containerenv");
evidence.kubernetes = !TelemetryInternal::GetTelemetryEnv("KUBERNETES_SERVICE_HOST").empty();
- Files reviewed: 12/13 changed files
- Comments generated: 1
- Review effort level: Lite
Keep platform-only telemetry includes and probe helpers behind their matching feature and OS guards so telemetry-off and Windows /W4 /WX builds remain clean. Normalize compound archive suffixes before identifying commit-pinned dependencies so cgmanifest generation retains tar.gz support. Files changed: cgmanifests/generate_cgmanifest.py, src/telemetry/device_info.cpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b72147d1-3937-4b13-940c-33299589af1b
|
Addressed the second review pass in c795e87: (1) telemetry_environment.h is now included only when ORTGENAI_ENABLE_TELEMETRY is enabled; (2) Linux/Android probe helpers use the same platform guard as their call sites; and (3) cgmanifest parsing normalizes .tar.gz before commit-SHA detection. For the suppressed probe-coverage suggestion, no additional seam is being added: microsoft/onnxruntime at d6750b398d6c8fce90544ddef954e5016de524a2 uses the same fixed-path GetHostEnvironmentInfo probe (onnxruntime/core/platform/posix/telemetry.cc:751-803), while onnxruntime/test/platform/telemetry_environment_test.cc tests the deterministic classifier and has no probe-I/O tests. GenAI now matches that implementation and test boundary; host-dependent integration assertions would be nondeterministic. |
There was a problem hiding this comment.
🔵 Needs a closer look
The changes span telemetry behavior, privacy documentation, build integration, and dependency management.
Review details
Suppressed comments (1)
docs/Privacy.md:23
- The new
AdapterActivatedevent is independently emitted and sampled at the same 1% rate as the other routine events, but this privacy statement omits adapter events. Please include adapters so the documented event-volume policy covers all newly sampled event types.
Routine model, generation, and error events use deterministic 1% client-side sampling. A process information event remains unsampled and includes coarse container, virtual-machine, WSL, or emulator classification when detected. Raw container identifiers, host names, and workload identifiers are not transmitted.
- Files reviewed: 12/13 changed files
- Comments generated: 0 new
- Review effort level: Lite
Include adapter activation events in the privacy statement so every routine event governed by deterministic one-percent sampling is documented. Files changed: docs/Privacy.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b72147d1-3937-4b13-940c-33299589af1b
Summary
Ports the relevant telemetry changes from microsoft/onnxruntime#29872, microsoft/onnxruntime#32226, and microsoft/onnxruntime#32425 into GenAI:
cpp_client_telemetryto 3.10.240.1, which includes the null-safe POSIXpopen()cleanup from Guard POSIX pipe cleanup when popen fails cpp_client_telemetry#1523ProcessInfounsampledProcessInfowithout transmitting raw host evidenceGenAI already contained the relevant microsoft/onnxruntime#29872 behavior for default-on supported builds, explicit opt-out, minimized common context, network-context cleanup, and persistent generated device IDs. ORT-only work from microsoft/onnxruntime#32425—session/EP lifecycle consolidation, RuntimePerf/SystemMetrics changes, and EP device census—does not map to GenAI's event model and is intentionally excluded.
Validation
lintrunner: cleanonnxruntime-genaitelemetry_helpers_teststelemetry_device_info_testsMSB8040), including for GoogleTest and vendored dependencies