Repository navigation
feat(container-insights): drop duplicate cAdvisor and node-exporter series - #2322
Closed
petruanica wants to merge 2 commits into
Closed
petruanica wants to merge 2 commits into
petruanica wants to merge 2 commits into
Conversation
…ector self-telemetry metrics Agent-side counterpart to the equivalent change in the amazon-cloudwatch-observability Helm chart. The chart and this translator each carry their own copy of the CI pipeline config and must stay in sync. cAdvisor reports most container_* families three times per pod: once per application container, once for the pod cgroup slice, and once for the pause/sandbox container. The latter two carry no `container` with `pod` set, and the slice value is the sum over the pod's containers, so both are duplicates of series already being sent. Add filter/cw_k8s_ci_v0_cadvisor_rollup to cadvisor.yaml, placed ahead of groupbyattrs and k8sattributes so a dropped datapoint never acquires the node and pod label sets. container_network_* is exempt: the sandbox owns the pod's network namespace, so for those eight families the sandbox series is the only series that exists. `container` is tested against both nil and "" because the Prometheus receiver strips empty-valued labels before they reach the pipeline. This was measured on a test cluster: of 19,186 rollup and sandbox datapoints in one sample, all 19,186 arrived with the `container` key absent from the map and none with it present and empty. The nil arm is the one that fires; the "" arm is kept for symmetry with filter/cw_k8s_ci_v0_cadvisor_empty. Also add node_scrape_collector_duration_seconds and node_scrape_collector_success to filter/cw_k8s_ci_v0_scrape_metadata. These are node-exporter's own per-collector scrape timing and status, the same category as the five names that filter already drops. node_textfile_scrape_error is deliberately left in place because the kubernetes-mixin alerting rules reference it. Note that every CI pipeline defines filter/cw_k8s_ci_v0_scrape_metadata under the same name and they are merged into one collector config, so the names have to be added to all ten copies. Updating only node_exporter.yaml left the merged result unchanged, because another pipeline's definition won. A comment to that effect is added to each copy. Golden sample configs regenerated. default_otel_config_aks.yaml, default_otel_config_gke.yaml and combined_v1_v2_eks_config.yaml are hand-edited because verifyToYamlTranslation applies token substitution to those goldens, so writing generated YAML back over them would destroy the placeholders. Verified with AWS_EC2_METADATA_DISABLED=true go test ./translator/... — the whole tocwconfig suite passes. Without that flag several unrelated tests fail on an IMDS lookup and TestOtlpMetricsConfigKubernetes panics, which aborts the test binary and silently skips everything after it; that behaviour is present on main.
petruanica
force-pushed
the
otel-ci-m1-drop-duplicate-series
branch
from
October 5, 2026 13:24
322de2c to
9b077ca
Compare
This was referenced Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Customer-visible change. Queries that aggregate
container_*metrics across a pod without filtering on container name will drop by roughly half once this ships. They were previously summing the per-container series together with the pod cgroup rollup, which is itself the sum over those same containers. The new values are correct and the old ones were double-counted, but the change is visible on any such dashboard or alarm, so it belongs in the release notes. See "Customer-visible changes" below for how to tell an affected query from an unaffected one.Note
Release coordination (working assumption, not yet decided). This PR is written assuming it ships together with the two sibling Container Insights changes in a single major chart release,
v7.0.0. That bundling has not been agreed yet — it is recorded here so reviewers know what the description assumes, and it needs confirming before any of the three merge. If they end up shipping separately, each breaking change still needs its own major bump.Description of the issue
kubelet's cAdvisor endpoint reports most
container_*families three times for every pod: once per application container, once for the pod cgroup slice, and once for the pause/sandbox container. The last two carry nocontainerlabel and the slice value is the sum over the pod's containers, so the Container Insights cadvisor pipeline sends two duplicate copies of every such series.Separately,
filter/cw_k8s_ci_v0_scrape_metadatadrops fivescrape_*self-telemetry names but missesnode_scrape_collector_duration_secondsandnode_scrape_collector_success, which are the same category.Description of changes
Adds
filter/cw_k8s_ci_v0_cadvisor_rolluptocadvisor.yaml, dropping datapoints that havepodset and nocontainer. It runs ahead ofgroupbyattrsandk8sattributesso a dropped datapoint never acquires the node and pod label sets.container_network_*is exempt: the sandbox owns the pod's network namespace, so for those eight families the sandbox series is the only one that exists rather than a duplicate, and dropping it would remove all pod network metrics.The condition tests
containeragainst both nil and"". The Prometheus receiver strips empty-valued labels before they reach the pipeline, so the attribute arrives absent rather than empty — the same reason the existingfilter/cw_k8s_ci_v0_cadvisor_emptyenumerates all four combinations.Adds the two
node_scrape_collector_*names tofilter/cw_k8s_ci_v0_scrape_metadata.node_textfile_scrape_erroris deliberately left in place because the kubernetes-mixin alerting rules read it. Every CI pipeline declares that filter under the same name and they are merged into one collector config, so the names are added to all ten copies — updating onlynode_exporter.yamlleaves the merged result unchanged, because another pipeline's definition wins.Golden sample configs regenerated.
Paired with the equivalent change in
aws-observability/helm-charts.Customer-visible changes
A query is affected if it aggregates a
container_*metric across a pod without constraining the container name — for examplesum by (pod) (rate(container_cpu_usage_seconds_total[5m])). Before this change that sums the per-container series plus the pod cgroup rollup plus the sandbox copy; afterwards it sums only the per-container series, so the displayed value drops by about half. The result is the correct per-pod total; the previous value was inflated.A query is unaffected if it already filters on a non-empty container name, which is the documented way to select application containers, or if it reads
container_network_*, which is exempt.Consumers that specifically want a pod-level aggregate should either sum the per-container series themselves or read the kubeletstats
k8s.pod.*metrics, which report pod scope directly and are unchanged by this PR.The two
node_scrape_collector_*removals are collector self-telemetry and have no dashboard consumer, but they are a removal and so are listed here for completeness.License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Tests
go test ./translator/...passes, including the regeneratedtocwconfiggoldens.New integration coverage in
aws/amazon-cloudwatch-agent-test(aws/amazon-cloudwatch-agent-test#777):TestCadvisorNoPodScopeRollupasserts nocontainer_*datapoint arrives withpodset andcontainerabsent,TestCadvisorNetworkKeepsPodScopeasserts thecontainer_network_*exemption still holds so the filter cannot silently delete all pod network metrics, andTestNodeExporterScrapeSelfTelemetryDroppedcovers the two added filter names.All three were validated on a live EKS cluster, since the claim is about what arrives in CloudWatch rather than what the config renders to. The changed configuration was applied to the running agent and cluster scraper, the pods were restarted, and metrics were given 30 minutes to settle;
test/otel/standard -computeType=EKSthen asserts on datapoints read back from CloudWatch. The suite was also run before the change, because an "attribute is absent" assertion passes trivially against empty data.TestCadvisorNoPodScopeRollupTestCadvisorNetworkKeepsPodScope(guard)TestNodeExporterScrapeSelfTelemetryDroppedNo failures, no skips. The guard passing both before and after is the load-bearing result: it shows the
container_network_*exemption held, so the filter did not silently delete all pod network metrics.Requirements
Before commiting your code, please do the following steps.
make fmtandmake fmt-sh— pass, no files modifiedmake lint— 0 issuesRelated pull requests
PR Checklist
makepasses locally (build, unit tests, lint)Integration Tests
To run integration tests against this PR, add the
ready for testinglabel.