test(otel): add GPU, Neuron, and EFA DRA-path integration tests

[samehkhalil]

Description of the issue

The awsdevicepodcorrelation processor's Dynamic Resource Allocation (DRA) path had no
integration coverage - only the device-plugin GPU/Neuron/EFA correlation tests existed. There
was nothing exercising devices allocated via a ResourceClaimTemplate end to end (driver ->
ResourceSlice/ResourceClaim -> metric correlation), so regressions on the DRA path could ship
undetected.

Depends on aws-observability/helm-charts#356 (DRA correlation config + resource.k8s.io
RBAC) and the processor change in amazon-contributing/opentelemetry-collector-contrib#631.
These go green in upstream CI once those land; until then they run against a chart/agent image
carrying those changes.

Description of changes

Add integration coverage mirroring the device-plugin tests, each exposing devices via a DRA
driver instead of the device-plugin resource and asserting per-device pod correlation:

• test/otel/multi_efa_dra: EFA via dranet (driver dra.net); efaburn claims one of two
  devices, the other stays unclaimed. Guards the per-device correlation collapse
  (ResourceSlice keying via dra.net/rdmaDevice plus the groupbyattrs split before the
  resource-level promote).
• test/otel/neuron_dra: Neuron via the AWS Neuron DRA driver (DeviceClass neuron.aws.com).
  Single Trainium device (trn1.2xlarge); the claimed device's two cores attribute to the burn
  pod and to no other pod. The Neuron DRA driver supports Trainium only, so this targets
  trn1.2xlarge.
• test/otel/gpu_dra: GPU via the NVIDIA DRA driver (DeviceClass gpu.nvidia.com).
  g4dn.12xlarge (4 GPUs); one claimed GPU correlates to the burn pod, the other three stay
  uncorrelated. Asserts device count, consecutive indices, and all DCGM metrics per device.

New terraform modules under terraform/eks/daemon (otel-multi-efa-dra, otel-neuron-dra,
otel-gpu-dra) install the DRA driver in place of the device plugin and apply a
ResourceClaimTemplate burn workload. All run k8s 1.35 (the processor uses the GA
resource.k8s.io/v1 DRA API, available since 1.34), and are wired into the test case
generator.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

go vet -tags integration is clean for all three packages and the generator. Validated
end-to-end on live EKS 1.35 clusters (all tests pass):

• neuron_dra (trn1.2xlarge): 6/6 pass; the claimed device's 2 cores correlate to the burn pod.
• gpu_dra (g4dn.12xlarge): 5/5 pass (incl. 6 DCGM per-device subtests); 1 of 4 GPUs claimed
  and correlated, the other 3 uncorrelated.
• multi_efa_dra: claimed device correlates to the pod; the second device stays unclaimed.

Test output from the live runs (count=N = correlated-series counts each assertion queried
from CloudWatch):

query returned count=2
--- PASS: TestNeuronDRADeviceCount / CoreCount / CoresPerDevice /
          DeviceIndicesAreIntegers / ClaimedDeviceCorrelation / BurnPodLabels
ok  .../test/otel/neuron_dra   (6/6 pass)

query returned count=1   # claimed GPU on the burn pod
query returned count=4   # GPUs on the node
--- PASS: TestGPUDRADeviceCount / ConsecutiveIndices / ClaimedVsUnclaimedCorrelation /
          BurnPodLabels / AllMetricsPerDevice{GPU_UTIL,MEM_COPY_UTIL,FB_USED,FB_FREE,GPU_TEMP,POWER_USAGE}
ok  .../test/otel/gpu_dra   (5/5 pass)

PR checklist

• Commits are squashed into a logical, reviewable set (one commit for a single change)
• Commits and PR description comply with Amazon internal guidelines
• make passes locally — make simple-lint (checklicense + impi, the Go checks build-check.yml runs) passes on all tracked files; make compile passes. Note these targets do not pass -tags integration, so the added test code (all //go:build integration) is compile-checked with go vet -tags integration on all three DRA packages + the generator (clean).
• All GitHub Actions checks on the PR are passing.
• Integration test evidence: live EKS 1.35 runs, all tests pass (see Tests)
• New or updated integration test coverage: this PR is the coverage; depends on Correlate DRA-allocated GPU/Neuron/EFA devices to pods in OTEL Container Insights aws-observability/helm-charts#356 and feat(awsdevicepodcorrelationprocessor): add DRA device-to-pod correlation with template-claim and ResourceSlice attribute keying amazon-contributing/opentelemetry-collector-contrib#631
• New functionality has unit tests; bug fixes have a reproducing test — these ARE integration tests
• Config translation changes include updated golden files — N/A
• Breaking or customer-visible changes are called out — none (test-only)

[lorespiz]

This checks how many GPUs are correlated, not which one. The DRA keying (the gpu-(\d+) pattern) is what's under test here. If it pointed the pod at the wrong GPU (e.g. the DRA device index not matching DCGM's gpu label), we'd still get "1 claimed, 3 unclaimed" and the test would pass.

Cheap way to pin it: under DRA, gpu_burn only sees its claimed GPU, so the correlated GPU should be the only busy one:

busy := map[string]bool{}
for _, r := range multi {
      if r.Value > 0 {
              busy[r.Labels.Datapoint["gpu"]] = true
      }
}
require.True(t, busy[claimed[0]], "GPU %s is correlated to %s* but idle; the burn pod is on a different GPU", claimed[0], burnPodPrefix)

For an exact check, read the ground truth from the API the way test/otel/gpu/k8s_helpers_test.go already does: burn pod → status.resourceClaimStatuses → ResourceClaim status.allocation.devices.results[].device → ResourceSlice uuid attribute. Then assert that the aDCGM series with that UUID carries the burn pod.

[lorespiz]

Same gap as in gpu_dra: with 2 devices, a keying bug that attaches efaburn to the other EFA still passes "1 claimed / 1 unclaimed". Since the dra.net/rdmaDevice attribute lookup is the main thing this test guards, could we assert the correlated aws.efa.device equals the attribute of the device actually allocated? That's the efaburn ResourceClaim's status.allocation.devices.results[].device, then the matching ResourceSlice device's dra.net/rdmaDevice.