Skip to content

Trim logs once to 75% of a 3 MB cap and batch stdout capture - #216

Merged
AvdLee merged 2 commits into
mainfrom
fix/log-trim-hysteresis
Sep 24, 2026
Merged

AvdLee merged 2 commits into
mainfrom
fix/log-trim-hysteresis

Conversation

@AvdLee

@AvdLee AvdLee commented Sep 24, 2026

Copy link
Copy Markdown
Owner

Problem

A CPU sample of RocketSim on macOS showed about 10% of all samples on the com.swiftlee.diagnostics.logger queue, almost all of it in LogsWriter.trimIfNecessary. The log file was pinned at the 2 MB cap:

  • There was no hysteresis. Each write trimmed only 10 records, so the next append went over the cap again. That meant a full read, a regex pass, a split/join and an atomic write with fsync on nearly every log line.
  • Blank lines (about 486 KB), legacy HTML sessions (about 203 KB) and 2,107 empty SYSTEM: records could never be trimmed, so trimming evicted the newest structured history instead.
  • Each stdout/stderr line was written separately, including empty lines.

Changes

  • LogsTrimmer: replaced with one linear byte pass (memchr/memmem, no regex and no String split). It drops everything that isn't a DIAGNOSTICS_JSON record (legacy HTML, blank lines, continuation text), drops empty system records, and keeps the newest records within a byte budget. If the oldest kept record isn't a sessionStart, the nearest earlier sessionStart is kept too, so events keep their session metadata in reports.
  • LogsWriter: when an append pushes the file over maximumLogSize, it trims once, down to 75% of the cap, with one atomic write. Appends that stay under the cap never trim. write(_:) accepts a batch of records and returns a WriteOutcome. Trim failures stay nonfatal and are reported through os_log.
  • DiagnosticsLogger: the cap goes from 2 MB to 3 MB (trim target 2.25 MB). Empty and whitespace-only stdout/stderr lines are skipped, and each captured pipe chunk is written with a single append. The writer and the output replay can be injected for tests.
  • Migration: already-oversized files are trimmed on the first write after launch, which is the session-start record written by setup().

Tests

New Swift Testing suites, with no sleeps or polling:

  • LogsTrimmerTests: legacy and blank lines dropped, empty system records dropped, newest records kept in order, sessionStart preservation, marker lookalikes inside messages, returns nil when nothing can be trimmed, oversized record.
  • LogsWriterTrimmingTests: a production-shaped fixture trims to the target in one write and keeps an ordered suffix of the newest records. After that trim, every append under the cap returns .appended until the next crossing. Also covers batched appends and parsing the trimmed file into sessions with metadata.
  • DiagnosticsLoggerStandardOutputTests: blank lines produce no records, and a multi-line chunk results in exactly one write.

All existing XCTest tests pass: 53 XCTest and 16 Swift Testing tests in total.

@AvdLee
AvdLee merged commit e3b6ffc into main Sep 24, 2026
1 check passed
@AvdLee
AvdLee deleted the fix/log-trim-hysteresis branch September 24, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant