Skip to content

Incident linger: close-side flap grace#371

Merged
passcod merged 7 commits into
mainfrom
claude/alerting-hysteresis-noise-el56zc
Jul 19, 2026
Merged

Incident linger: close-side flap grace#371
passcod merged 7 commits into
mainfrom
claude/alerting-hysteresis-noise-el56zc

Conversation

@passcod

@passcod passcod commented Jul 18, 2026

Copy link
Copy Markdown
Member

🤖 Stacked on #362. Adds the close-side mirror of the open grace: a red check that briefly blips green no longer turns one span of trouble into a resolve + re-open notification pair and a fragmented incident history.

The problem

Grace was one-sided. A never-shipped open is cancelled when its incident closes within slack_open_delay — but the close side had nothing: one green report closed the incident and shipped the resolve, and the next red opened a fresh incident row with fresh grace and a fresh escalation budget. Each blip cost a resolve, a new open, potentially a repeat Critical, and a split history. Inversely, a check flapping faster than the grace never accumulated a continuous red span longer than the window, so it never notified at all — hours of trouble, total silence.

The linger (spec: INC amended)

When an incident's last effective failure leaves because its result recovered, the incident lingers: closing_at is stamped and it stays the target's open incident for the group's slack_close_delay (new column, default 5 minutes; canopy-wide incidents use a fixed 5 minutes). A failure returning within the window — the same check or a member re-graded to failure — clears the stamp and the same incident continues: no resolve was sent, no second open, escalated_at carries over.

A new minute-cadence sweep on the monitor pod closes expired lingers, backdating closed_at to when the failure left (the linger is damping machinery, not part of the incident's span) and then running exactly the old close path: cancel a never-shipped open, or ship the resolve.

Notification interplay: the outbox drainer now refuses to ship an incident_open while its incident lingers, so a single short blip still never notifies. The open's deliver_after is not reset by flapping, so an incident publishes once it is older than grace and currently red — which fixes the fast-flapper silence hole as a side effect (covered by a test).

Operator actions are not flaps: a last failure leaving through resolve, snooze, silence, or monitoring-off closes immediately, with the Slack resolve shipping at once and attributed where known. A zero slack_close_delay opts a group out of lingering entirely.

Surfaces

  • server_groups.slack_close_delay on the group API (create/update) and a "keep incidents open after recovery for N minutes" field on the group edit page, next to the open cooldown (openapi + api-types regenerated, e2e updated).
  • Lingering reads as recovering, not failing: IncidentData.lingering_since (set while open with closing_at stamped) tones lingering incidents info-blue everywhere held incidents are already differentiated — status page group cards get an incident (recovering) chip, the incidents grid cards and group detail banner switch border/icon tone, the server page incident button reads (recovering), and the incident page header gets a "Recovering; last failure cleared …" chip explaining the linger. The held countdown is suppressed while lingering, since the drainer gates the open notice and the posting time would be wrong.

Tests

New incident_linger suite: rejoin-continues-same-incident (one open, one resolve, however many blips), member re-grade ends lingering, operator resolve closes immediately, zero window closes immediately, drainer holds opens while lingering + cancel-at-expiry, and the flapper-publishes-after-grace fix. Existing recovery-path tests updated to expire the linger where they asserted immediate close. Playwright coverage for the recovering indicator on the incident page, server page, and status page (e2e/lingering.spec.ts); full e2e suite green (142).

Full database and statuses suites green; the enrollment / recovery-ceremony failures seen while testing are pre-existing parallel-env flakes (they pass single-threaded and are untouched by this diff).

Design context

The wider noise-reduction exploration (trend aggregates, adaptive windows, flapping state, pattern suggestions, and the off-the-shelf survey) lives in #372, stacked on this PR. This PR is stage 1 of that plan.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AEZZ9HMkKAFqrJzTg3FrE7

claude added 6 commits July 18, 2026 04:23
When an incident's last effective failure recovers, stamp closing_at and
keep the incident open for the group's slack_close_delay (default 5min;
canopy-wide target fixed at 5min) instead of closing outright. A failure
returning within the window clears the stamp and the same incident
continues: no resolve, no second open, no fresh escalation budget. The
monitor pod's new linger sweep closes expired lingers, backdating
closed_at to when the failure left, then runs the usual Slack
cancel-or-resolve.

The outbox drainer holds incident_open rows while their incident lingers,
so a one-off blip still never notifies; conversely the open's
deliver_after keeps counting from the first open, so a fast flapper that
today closes-within-grace forever now publishes once its trouble has
outlived the grace while currently red.

Only report-driven recoveries linger: resolve, snooze, silence, and
monitoring-off are operator actions, not flaps, and close immediately
(Slack resolve attributed where known). Zero window opts a group out.

Spec: INC membership/notification amended. UI: linger minutes field on
group edit, next to the open cooldown.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AEZZ9HMkKAFqrJzTg3FrE7
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AEZZ9HMkKAFqrJzTg3FrE7
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AEZZ9HMkKAFqrJzTg3FrE7
Expose lingering_since on IncidentData (closing_at while open) and tone
lingering incidents info-blue everywhere held incidents are already
differentiated: status page group cards ('incident (recovering)' chip),
the incidents grid cards, the group detail active-incident banner, the
server page incident button, and the incident page header, which gets a
'Recovering; last failure cleared …' chip explaining the linger. The
held countdown is suppressed while lingering — the drainer gates the
open notice, so the posting time would be wrong.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AEZZ9HMkKAFqrJzTg3FrE7
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AEZZ9HMkKAFqrJzTg3FrE7
@passcod

passcod commented Jul 19, 2026

Copy link
Copy Markdown
Member Author

LGTM

Base automatically changed from incident-pipeline-specs to main July 19, 2026 22:12
@passcod
passcod added this pull request to the merge queue Jul 19, 2026
Merged via the queue into main with commit fa904db Jul 19, 2026
7 checks passed
@passcod
passcod deleted the claude/alerting-hysteresis-noise-el56zc branch July 19, 2026 23:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants