feat: buffer connection logs and flush them on failure

[aqandrew]

Implements RFC requirement 13 / DEVEX-669: buffer connection debug logs in memory below the current log level and flush them on a genuine connection failure, so a support bundle captures the detail leading up to the failure without the user having enabled debug logging beforehand.

What this does

• Adds a BufferingLogger decorator (src/logging/logBuffer.ts) that wraps the "Coder" output channel and keeps a bounded, in-memory ring of the entries that sit below the channel's current level, which it would otherwise drop. Only below-level entries are buffered, so nothing already written is duplicated. Entries are formatted (via safeStringify) at record time.
• On a connection failure, flush(reason) re-emits the captured entries into the output channel at the least-verbose level the channel still persists. The first physical line of each entry carries a [buffered] marker with its original ISO timestamp and level; continuation lines carry the bare marker. Capture is best-effort — the channel writes on its own schedule — but a later failure flush replays anything a bundle missed.
  • Collecting a support bundle flushes the buffer (flush("support_bundle", { retain: true })) before the CLI runs, and retains the ring so a later failure flush still has the entries.
  • When the channel is at Off, flush keeps the entries buffered instead of discarding them.
• The buffer is bounded by both entry count and characters (MAX_BUFFERED_CHARS = 2_000_000); trim() enforces both budgets with oldest-eviction, and flush() replays in chunks of 100 to avoid a single oversized write.
• Wires the buffer into ServiceContainer and adds the coder.connectionLogBuffer.size setting (default 1000, capped at 10000, 0 disables). The setting readers live in src/settings/logger.ts and the container reads/watches the size through a single readSize() helper via watchConfigurationChanges. The setting is included in COLLECTED_SETTINGS.
• Flushes only on genuine connection failures, gated by an explicit failure?: boolean on the connection-log reason (not on transient reconnects, a handshake 401, or intentional teardown):
  • a reconnecting WebSocket terminal failure (unrecoverable_close, unrecoverable_http where the status is not 401, or certificate_error);
  • a failure while opening a workspace (canceled build, missing agent, timeout, or CLI/certificate error), funnelled through Remote.closeRemote().
• Connection-failure flushes are funnelled through BufferingLogger.onConnectionFailure(reason, route) (part of ConnectionLogBuffer), so the extension and remote paths flush ${reason} ${route} identically; the socket option defaults to a noop when no observer is supplied.
• The flush reason carries the failing route for attribution; the route is seeded on the socket so even a first-connect failure logs a real route instead of unknown.
• Handshake status is parsed by a shared handshakeStatus(error) (src/websocket/utils.ts) covering both ws (Unexpected server response: <code>) and eventsource (Non-200 status code (<code>)), so a host/port such as 127.0.0.1:4040 is no longer misread as HTTP 404 and the SSE path is handled. CoderApi.is404Error compares against HttpStatusCode.NOT_FOUND.
• Redacts registration_access_token in HTTP body logging alongside the other sensitive fields.
• Documents the behavior, config, hard-kill/OOM loss limitation, and SSH log scope in CONTRIBUTING.md.

WebSocket event fix

OneWayWebSocket now registers open/close/error via DOM-style addEventListener, so close consumers receive a real CloseEvent with .code/.reason. This makes unrecoverable_close reachable in production (message events still use ws.on("message", ...) for JSON parsing). Server-initiated normal closes (1000/1001) now go through scheduleReconnect rather than parking the socket, so the now-unreachable normal_close reason is dropped from the telemetry unions and EVENTS.md.

Scope notes

• Extension SSH debug logs that pass through the shared logger are buffered; the CLI ProxyCommand file logs under coder.proxyLogDirectory are not, since support bundles already collect them from disk.
• The buffer lives in memory, so a hard kill or out-of-memory event loses it (documented).
• Flushing after N consecutive failed reconnect attempts against an unreachable server is deferred to follow-up Flush the connection log buffer after N failed reconnect attempts against an unreachable server #1112.
• Reading the handshake status from the libraries' public event APIs (instead of parsing their error strings, which a real-library test now guards) is deferred to follow-up Read handshake status from public API instead of parsing library error strings #1118.

Testing

• pnpm typecheck, pnpm format:check, and pnpm lint are clean.
• Affected/dependent unit suites pass (logBuffer, settings/logger, formatters, reconnectingWebSocket, oneWayWebSocket, coderApi, workspaceMonitor, workspaceStateMachine, remote, commands.supportBundle, instrumentation/websocket).

Implementation plan & design decisions

Design

• Buffer: bounded by entry count and characters; captures only calls whose severity is below the channel's current level; formats entries at record time; oldest-eviction; live-resizable via config; replays in chunks.
• Flush target (D3): replay into the existing "Coder" output channel at a level that still persists, with a [buffered] marker plus original level/timestamp, chronologically next to the real failure logs. Support bundles already collect the on-disk VS Code logs and also trigger a flush before appending them, so no separate sink is needed. At Off, entries are retained rather than discarded.
• Flush reasons (D4): genuine, surfaced connection failures only, gated by an explicit failure?: boolean — reconnecting-socket terminal failures (except a 401 handshake) and a workspace-open failure funnelled through closeRemote(). Never on transient retrying drops or intentional teardown (manual_disconnect, replaced, dispose/deactivate/reload). The callback carries the failing route.
• SSH scope (D5): buffer extension SSH debug passing through the shared Logger; do not buffer CLI ProxyCommand file logs already handled via coder.proxyLogDirectory.

Decisions

• D1: buffer all below-level session logs.
• D2: bound by entry count (and a character budget to cap memory).
• D3: replay into the existing Coder output channel at a persisted level with [buffered] marker/original level/timestamp; retain at Off; also flush on support-bundle collection.
• D4: flush only on genuine connection failure; not transient, not a 401, and not intentional teardown.
• D5: buffer extension SSH debug through the shared Logger; not CLI ProxyCommand file logs.

---

🤖 Generated with Coder Agents. Reviewed and authored on behalf of @aqandrew.

[linear-code]

DEVEX-669

[EhabY]

Please address the inline comments on remote-client failure wiring, the monitor's failure trigger, and flush suppression. The remaining comments cover simplification, test coverage, and naming.

Review generated with Coder Agents on behalf of @EhabY.

[EhabY]

Two blocking items: the ws close event shape makes the unrecoverable_close flush unreachable in production, and the monitor test cannot fail. The rest covers trigger scope, where the settings reader lives, and trimming the buffer and its tests.

The PR description also needs a refresh: it still mentions the suppression window, the monitor trigger, isConnectionFailure, and four commits where there are twenty.

[EhabY]

Code is solid: every thread from the last round checks out, and typecheck, lint, and the unit suites are clean. Approving, the three suggestions are wording only.

Description: Flush reasons (D4) still lists normal_close, which this PR deletes. The Off bullet still promises the context "survives until logging is turned back on", the phrasing we dropped from flush(), since nothing watches onDidChangeLogLevel.

[aqandrew]

Updated the PR description to remove normal_close from D4's intentional teardown reasons + removed mention of turning logging back on from the Off channel description (per 544d143).

Thanks so much for the thorough reviews @EhabY! 🚀