test(otel): cover the cAdvisor rollup filter and node-exporter scrape self-telemetry

[petruanica]

Description of the issue

kubelet's cAdvisor endpoint reports every container_* family except container_network_* three times per pod — once per application container, once for the pod cgroup slice, and once for the pause/sandbox container — and the Container Insights pipelines are gaining a filter that drops the latter two. container_network_* is exempt, because for those eight families the sandbox series is the only one that exists. Neither the removal nor the exemption had integration coverage, and the exemption is the dangerous one: narrowing it would silently delete all pod network telemetry.

Description of changes

Adds three tests to the standard suite, one commit.

TestCadvisorNoPodScopeRollup asserts that no surviving non-network cadvisor series carries an empty datapoint container or a missing k8s.container.name — that is, the pod cgroup slice and the pause/sandbox container are gone. The missing-k8s.container.name clause is the one that fires on containerd; the empty-string clause is defensive, since the Prometheus receiver strips empty-valued labels before the pipeline sees them.

TestCadvisorNetworkKeepsPodScope asserts the other half: for container_network_* the pod-scope series must still be present. If the exemption regex is ever narrowed, this fails loudly instead of the pipeline quietly losing every pod network metric.

TestNodeExporterScrapeSelfTelemetryDropped asserts node_scrape_collector_duration_seconds and node_scrape_collector_success no longer arrive. node_textfile_scrape_error is deliberately absent from the list and must keep flowing, because the kubernetes-mixin NodeTextFileCollectorScrapeError alerting rule reads it.

Worth noting for reviewers: the pre-existing TestCadvisorNoPodSandboxMetrics asserts container != "POD", the dockershim spelling. cAdvisor emits container="" for the containerd sandbox, so that test passes vacuously today. The paired config change is what makes its stated invariant actually hold.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

gofmt clean and go vet -tags integration ./test/otel/standard/ passes; the test binary builds with all three test functions registered.

These are integration tests, so they need a live cluster running the paired configuration change. They will fail against a cluster that has not yet taken it — that is the intended behaviour, and the reason this PR is paired one-to-one with the config change rather than bundled with the other Container Insights test additions.

Related pull requests

Chart change	aws-observability/helm-charts#385
Sibling Container Insights changes	#775, #776

PR Checklist

• Commits are squashed into a logical, reviewable set (one commit for a single change)
• Commits and PR description comply with Amazon internal guidelines
• gofmt clean, go vet -tags integration passes, test binary builds
• All GitHub Actions checks on the PR are passing
• Integration test evidence: N/A — this PR is the integration test coverage. The tests require a cluster running the paired configuration change and will fail until it is deployed, which is intentional.
• New or updated integration test coverage: N/A — this is the corresponding amazon-cloudwatch-agent-test PR
• New functionality has unit tests; bug fixes have a reproducing test — N/A, test-only change
• Config translation changes include updated golden files — N/A, no translator in this repo
• Breaking or customer-visible changes are called out in the PR description — N/A, test-only change

[miconeilaws]

I think the new tests cover the new dropping behaviour... But how are we sure we aren't inadvertently dropping something we don't want to drop? Do you think we have enough coverage to ensure the "Schemas" we want actually flow through?

[petruanica]

Partly. Every drop assertion here is gated on require.NotEmpty, so it can't pass by querying nothing, and the suite's existing positive layer runs on the same data — plus TestCadvisorNetworkKeepsPodScope fails if the ^container_network_ exemption is ever narrowed, which would delete every pod network metric.

But you're right that it isn't a schema contract: ExpectedLabels is a required-subset check, so an attribute nobody listed could vanish and stay green. The fix is a strict per-receiver test asserting the exact expected key set, failing on missing and unexpected — happy to add it as a follow-up.