Skip to content

feat(alerts): warning and critical tiers, recovery values and N-of-M on one rule - #1509

Open
Makisuo wants to merge 20 commits into
mainfrom
worktree-alerting-severity-tiers
Open

Makisuo wants to merge 20 commits into
mainfrom
worktree-alerting-severity-tiers

fix(alerts): decide the critical PagerDuty key from the incident's cu…

2f41cc4
Select commit
Loading
Failed to load commit list.
Maple Review Bot / Maple / review succeeded Oct 16, 2026 in 1m 52s

Confidence 8/10 · No issues found

🟢 Confidence 8/10 · likely safe to merge
The delta is two small, well-reasoned fixes in the tier planner and the PagerDuty key choice, each pinned by a test; I read both and their callers.
quality 100/100 · no findings · tests covered · risk medium · 1 file not read

Fixes the two open findings: the audit counters no longer treat a recovery-band tick as healthy, and PagerDuty picks and closes its critical key from the incident's current tier rather than the queued event's. Safe to merge.

  • countersFromHistory counts only ticks where every tier is clear as healthy (tiers.ts:139)
  • PagerDuty's dedup key and critical-key close now use the incident's current tier (pagerduty.ts:31, pagerduty.ts:112)

Before merge

Fixed since the last review

  • ✅ F3 · countersFromHistory counts a recovery-band tick as healthy
  • ✅ F4 · A warning event dispatched after an escalation closes the critical PagerDuty alert

Production impact

Production traffic of the changed files (last 7 days)
File Calls/day Busiest operations
packages/backend/src/services/errors/issue-hub.ts 11 IssueHub.recordIssueEvent 11/day, 0.0% err
Telemetry this change adds and removes (2)
  • ➕ span name IssueHub.escalateAlertIssue · packages/backend/src/services/errors/issue-hub.ts:377
  • ➕ span name IssueHub.raiseDetectorSeverity · packages/backend/src/services/errors/issue-hub.ts:403
What was checked
  • incidentSeverity is set only for an open incident row; AlertIncidentStatus is open/resolved with holds inside open (alerts.ts:105), so held incidents still suppress the critical close
  • keySeverity demotes only critical→warning, so a critical event retried after a de-escalation pages the base key and cannot re-open the closed critical alert (pagerduty.ts:31)
  • closesCritical excludes test events (pagerduty.ts:112), so a send-test on a two-tier rule still fires only the base-key test trigger
Files not reviewed (1)

The review ended before it read these diffs, so nothing above vouches for them.

  • packages/backend/src/services/alerts/delivery/transports/render.test.ts

2f41cc4 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.