Incident linger: close-side flap grace#371
Merged
Merged
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AEZZ9HMkKAFqrJzTg3FrE7
When an incident's last effective failure recovers, stamp closing_at and keep the incident open for the group's slack_close_delay (default 5min; canopy-wide target fixed at 5min) instead of closing outright. A failure returning within the window clears the stamp and the same incident continues: no resolve, no second open, no fresh escalation budget. The monitor pod's new linger sweep closes expired lingers, backdating closed_at to when the failure left, then runs the usual Slack cancel-or-resolve. The outbox drainer holds incident_open rows while their incident lingers, so a one-off blip still never notifies; conversely the open's deliver_after keeps counting from the first open, so a fast flapper that today closes-within-grace forever now publishes once its trouble has outlived the grace while currently red. Only report-driven recoveries linger: resolve, snooze, silence, and monitoring-off are operator actions, not flaps, and close immediately (Slack resolve attributed where known). Zero window opts a group out. Spec: INC membership/notification amended. UI: linger minutes field on group edit, next to the open cooldown. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AEZZ9HMkKAFqrJzTg3FrE7
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AEZZ9HMkKAFqrJzTg3FrE7
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AEZZ9HMkKAFqrJzTg3FrE7
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AEZZ9HMkKAFqrJzTg3FrE7
Expose lingering_since on IncidentData (closing_at while open) and tone
lingering incidents info-blue everywhere held incidents are already
differentiated: status page group cards ('incident (recovering)' chip),
the incidents grid cards, the group detail active-incident banner, the
server page incident button, and the incident page header, which gets a
'Recovering; last failure cleared …' chip explaining the linger. The
held countdown is suppressed while lingering — the drainer gates the
open notice, so the posting time would be wrong.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AEZZ9HMkKAFqrJzTg3FrE7
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AEZZ9HMkKAFqrJzTg3FrE7
Member
Author
|
LGTM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 Stacked on #362. Adds the close-side mirror of the open grace: a red check that briefly blips green no longer turns one span of trouble into a resolve + re-open notification pair and a fragmented incident history.
The problem
Grace was one-sided. A never-shipped open is cancelled when its incident closes within
slack_open_delay— but the close side had nothing: one green report closed the incident and shipped the resolve, and the next red opened a fresh incident row with fresh grace and a fresh escalation budget. Each blip cost a resolve, a new open, potentially a repeat Critical, and a split history. Inversely, a check flapping faster than the grace never accumulated a continuous red span longer than the window, so it never notified at all — hours of trouble, total silence.The linger (spec: INC amended)
When an incident's last effective failure leaves because its result recovered, the incident lingers:
closing_atis stamped and it stays the target's open incident for the group'sslack_close_delay(new column, default 5 minutes; canopy-wide incidents use a fixed 5 minutes). A failure returning within the window — the same check or a member re-graded to failure — clears the stamp and the same incident continues: no resolve was sent, no second open,escalated_atcarries over.A new minute-cadence sweep on the monitor pod closes expired lingers, backdating
closed_atto when the failure left (the linger is damping machinery, not part of the incident's span) and then running exactly the old close path: cancel a never-shipped open, or ship the resolve.Notification interplay: the outbox drainer now refuses to ship an
incident_openwhile its incident lingers, so a single short blip still never notifies. The open'sdeliver_afteris not reset by flapping, so an incident publishes once it is older than grace and currently red — which fixes the fast-flapper silence hole as a side effect (covered by a test).Operator actions are not flaps: a last failure leaving through resolve, snooze, silence, or monitoring-off closes immediately, with the Slack resolve shipping at once and attributed where known. A zero
slack_close_delayopts a group out of lingering entirely.Surfaces
server_groups.slack_close_delayon the group API (create/update) and a "keep incidents open after recovery for N minutes" field on the group edit page, next to the open cooldown (openapi + api-types regenerated, e2e updated).IncidentData.lingering_since(set while open withclosing_atstamped) tones lingering incidents info-blue everywhere held incidents are already differentiated — status page group cards get anincident (recovering)chip, the incidents grid cards and group detail banner switch border/icon tone, the server page incident button reads(recovering), and the incident page header gets a "Recovering; last failure cleared …" chip explaining the linger. The held countdown is suppressed while lingering, since the drainer gates the open notice and the posting time would be wrong.Tests
New
incident_lingersuite: rejoin-continues-same-incident (one open, one resolve, however many blips), member re-grade ends lingering, operator resolve closes immediately, zero window closes immediately, drainer holds opens while lingering + cancel-at-expiry, and the flapper-publishes-after-grace fix. Existing recovery-path tests updated to expire the linger where they asserted immediate close. Playwright coverage for the recovering indicator on the incident page, server page, and status page (e2e/lingering.spec.ts); full e2e suite green (142).Full
databaseand statuses suites green; the enrollment / recovery-ceremony failures seen while testing are pre-existing parallel-env flakes (they pass single-threaded and are untouched by this diff).Design context
The wider noise-reduction exploration (trend aggregates, adaptive windows, flapping state, pattern suggestions, and the off-the-shelf survey) lives in #372, stacked on this PR. This PR is stage 1 of that plan.
🤖 Generated with Claude Code
https://claude.ai/code/session_01AEZZ9HMkKAFqrJzTg3FrE7