feat: add prometheus alerts for etcd

[jackhodgkiss]

Add alerts for etcd obtained from 1

[gemini-code-assist]

Code Review

This pull request introduces a new Prometheus alert rules file for etcd (etc/kayobe/kolla/config/prometheus/etcd.rules). The review feedback identifies several issues where alert rules aggregate away the instance label (using count, BY without instance, or without (instance)) but still attempt to reference {{ $labels.instance }} in their summary annotations. This would result in empty or missing instance labels in the alerts. The reviewer suggests either removing the instance reference for cluster-wide alerts or including instance in the aggregation clauses to preserve the label.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The alert EtcdInsufficientMembers uses count(etcd_server_id) which aggregates away all labels, including instance. However, the summary annotation references {{ $labels.instance }}. This will result in an empty or missing instance label in the alert summary. Since this is a cluster-wide alert, we should remove the (instance {{ $labels.instance }}) suffix from the summary.

    - alert: EtcdInsufficientMembers
      expr: count(etcd_server_id) % 2 == 0
      for: 0m
      labels:
        severity: critical
      annotations:
        summary: Etcd insufficient Members
        description: "Etcd cluster should have an odd number of members\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

[gemini-code-assist]

The alert EtcdHighNumberOfFailedGRPCRequestsWarning aggregates metrics using BY (grpc_service, grpc_method), which drops the instance label. However, the summary annotation references {{ $labels.instance }}. To fix this and make the alert more specific, we should include instance in the BY clause.

    - alert: EtcdHighNumberOfFailedGRPCRequestsWarning
      expr: sum(rate(grpc_server_handled_total{grpc_code=~"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded"}[1m])) BY (instance, grpc_service, grpc_method) / sum(rate(grpc_server_handled_total[1m])) BY (instance, grpc_service, grpc_method) > 0.01 and sum(rate(grpc_server_handled_total[1m])) BY (instance, grpc_service, grpc_method) > 0
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: Etcd high number of failed GRPC requests warning (instance {{ $labels.instance }})
        description: "More than 1% GRPC request failure detected in Etcd\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

[gemini-code-assist]

The alert EtcdHighNumberOfFailedGRPCRequestsCritical aggregates metrics using BY (grpc_service, grpc_method), which drops the instance label. However, the summary annotation references {{ $labels.instance }}. To fix this and make the alert more specific, we should include instance in the BY clause.

    - alert: EtcdHighNumberOfFailedGRPCRequestsCritical
      expr: sum(rate(grpc_server_handled_total{grpc_code=~"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded"}[1m])) BY (instance, grpc_service, grpc_method) / sum(rate(grpc_server_handled_total[1m])) BY (instance, grpc_service, grpc_method) > 0.05 and sum(rate(grpc_server_handled_total[1m])) BY (instance, grpc_service, grpc_method) > 0
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: Etcd high number of failed GRPC requests critical (instance {{ $labels.instance }})
        description: "More than 5% GRPC request failure detected in Etcd\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

[gemini-code-assist]

The alert EtcdGRPCRequestsSlow aggregates metrics using by (grpc_service, grpc_method, le), which drops the instance label. However, the summary annotation references {{ $labels.instance }}. To fix this and make the alert more specific, we should include instance in the by clause.

    - alert: EtcdGRPCRequestsSlow
      expr: histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{grpc_method!="Defragment", grpc_type="unary"}[1m])) by (instance, grpc_service, grpc_method, le)) > 0.15
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: Etcd GRPC requests slow (instance {{ $labels.instance }})
        description: "GRPC requests slowing down, 99th percentile is over 0.15s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

[gemini-code-assist]

The alert EtcdHighNumberOfFailedHTTPRequestsWarning aggregates metrics using BY (method), which drops the instance label. However, the summary annotation references {{ $labels.instance }}. To fix this and make the alert more specific, we should include instance in the BY clause.

    - alert: EtcdHighNumberOfFailedHTTPRequestsWarning
      expr: sum(rate(etcd_http_failed_total[1m])) BY (instance, method) / sum(rate(etcd_http_received_total[1m])) BY (instance, method) > 0.01 and sum(rate(etcd_http_received_total[1m])) BY (instance, method) > 0
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: Etcd high number of failed HTTP requests warning (instance {{ $labels.instance }})
        description: "More than 1% HTTP failure detected in Etcd\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

[gemini-code-assist]

The alert EtcdHighNumberOfFailedHTTPRequestsCritical aggregates metrics using BY (method), which drops the instance label. However, the summary annotation references {{ $labels.instance }}. To fix this and make the alert more specific, we should include instance in the BY clause.

    - alert: EtcdHighNumberOfFailedHTTPRequestsCritical
      expr: sum(rate(etcd_http_failed_total[1m])) BY (instance, method) / sum(rate(etcd_http_received_total[1m])) BY (instance, method) > 0.05 and sum(rate(etcd_http_received_total[1m])) BY (instance, method) > 0
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: Etcd high number of failed HTTP requests critical (instance {{ $labels.instance }})
        description: "More than 5% HTTP failure detected in Etcd\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

[gemini-code-assist]

The alert EtcdPeerCommunicationFailures aggregates metrics using by (To), which drops the instance label. However, the summary annotation references {{ $labels.instance }}. To fix this and make the alert more specific, we should include instance in the by clause.

    - alert: EtcdPeerCommunicationFailures
      expr: sum(rate(etcd_network_peer_sent_failures_total[1m])) by (instance, To) > 0.01
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: Etcd peer communication failures (instance {{ $labels.instance }})
        description: "Etcd member is experiencing more than 0.01 failed peer sends per second to peer {{ $labels.To }}, which usually indicates the member is unreachable or network-partitioned.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

[gemini-code-assist]

The alert EtcdInsufficientMembersForQuorum aggregates metrics using without (instance), which drops the instance label. However, the summary annotation references {{ $labels.instance }}. Since this is a cluster-wide alert, we should remove the (instance {{ $labels.instance }}) suffix from the summary.

    - alert: EtcdInsufficientMembersForQuorum
      expr: sum(up{job=~".*etcd.*"} == bool 1) without (instance) < ((count(up{job=~".*etcd.*"}) without (instance) + 1) / 2)
      for: 3m
      labels:
        severity: critical
      annotations:
        summary: Etcd insufficient members for quorum
        description: "Only {{ $value }} etcd members are reachable, fewer than required for quorum; the cluster will reject writes until quorum is restored.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"