fix(metrics): collapse per-IP destination labels to workload identity to cut TSDB cardinality#280
Merged
blue4209211 merged 2 commits intoJul 6, 2026
Conversation
… to cut TSDB cardinality The destination/actual_destination labels on container_net_tcp_* and all L7 metrics carried raw IP:port. Pod IPs churn and recycle constantly (especially on spot nodes), so every reconnect created a brand-new time series — the dominant driver of VictoriaMetrics cardinality/churn (observed ~440k new series/interval, vmsingle >7GB RSS, vmagent OOM-crashlooping and dropping all scrape targets -> TargetDown storms). Changes: - B1: destination/actual_destination now collapse to the resolved workload identity (namespace/name) for internal destinations, keeping series stable across pod-IP recycling. External FQDNs and the *_workload_* labels are unchanged. Gated by --collapse-internal-destinations (COLLAPSE_INTERNAL_DESTINATIONS, default true) as a kill-switch. Backend queries key on *_workload_* labels, not the raw destination, so this is transparent to consumers. - B3: container_net_latency_seconds destination_ip collapses the same way via DestinationIPLabelValue. - Bugfix: resetAndSetActive now uses Add instead of Set after Reset, so multiple connections that collapse to the same workload label sum instead of overwriting — prevents container_net_tcp_active_connections undercounting. - B5: removed 7 dead metrics.go Net* Descs (live path is TCPMetrics.collect). - Tests for destinationLabelValue / DestinationIPLabelValue covering FQDN, internal-resolved, unresolved, and flag-off branches.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a mechanism to collapse internal IP:port destinations into stable workload identities (namespace/name) when the CollapseInternalDestinations flag is enabled, aiming to reduce TSDB series cardinality and churn. It also cleans up unused metric descriptors. The review feedback highlights a concurrency issue in resetAndSetActive where a lock should be acquired to prevent transient metric drops during scrapes, suggests preserving the port for external unresolved IP destinations in destinationLabelValue, and recommends adding corresponding test coverage.
- resetAndSetActive: hold t.mu.Lock() across Reset+rebuild so a concurrent collect() (RLock) never observes a partially-rebuilt/empty active gauge (transient dips in container_net_tcp_active_connections). Pre-existing race, fixed here. - destinationLabelValue: guard the resolved-workload branch with wl.Name != hp.ip.String() so the IP-echoed-back fallback is treated as unresolved (avoids misleading "external/<ip>"); keep IP:port for unresolved external destinations (low cardinality, port is useful), only drop the port for unresolved private/internal IPs. Consistent with DestinationIPLabelValue. - tests: cover unresolved-external (keep port) and IP-echoed-back cases.
blue4209211
approved these changes
Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Dev VictoriaMetrics was firing
TargetDownconstantly. Root cause: node-agent metric cardinality.The
destination/actual_destinationlabels oncontainer_net_tcp_*and all L7 metrics carried rawIP:port. Pod IPs churn and recycle constantly (especially on spot nodes), so every reconnect created a brand-new time series — the dominant driver of TSDB churn:vm_new_timeseries_created_total)TargetDownstorm/metricspayloads exceeded vmagent's 16MBmaxScrapeSizeVerified the nudgebee backend keys exclusively on the
*_workload_name/namespacelabels, never the rawdestination/actual_destination— so collapsing them is transparent to consumers.Changes
destination/actual_destinationnow emitnamespace/namefor internal destinations instead of churningIP:port. External FQDNs and*_workload_*labels unchanged. Gated by--collapse-internal-destinations/COLLAPSE_INTERNAL_DESTINATIONS(default true) kill-switch.container_net_latency_secondsdestination_ipcollapses the same way viaDestinationIPLabelValue.resetAndSetActiveusesAdd(notSet) afterReset, so connections collapsing to the same workload label sum instead of overwriting — preventscontainer_net_tcp_active_connectionsundercounting.metrics.goNet*Descs (live path isTCPMetrics.collect()).destinationLabelValue/DestinationIPLabelValue.Rollout / rollback
Behind
COLLAPSE_INTERNAL_DESTINATIONS(default on); setfalseto restore raw-IP labels. Expect a large drop in active series andvm_new_timeseries_created_total.