Skip to content

Meetings: resample both lanes before combining; repair the wrong-rate meeting.m4a files - #34

Merged
ofcRS merged 6 commits into
mainfrom
fix/combine-sample-rate
Nov 18, 2026
Merged

ofcRS merged 6 commits into
mainfrom
fix/combine-sample-rate

Conversation

@ofcRS

@ofcRS ofcRS commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

The bug

Recorder.combine decoded mic.m4a and system.m4a to mono at each file's own rate, then wrote both channels frame for frame at the microphone's rate. It never resampled. The system lane is 48 kHz.

  • 16 kHz microphone (Bluetooth headset in HFP): the "Them" channel played 3× too slow and meeting.m4a lasted 3× the meeting. History reads the duration from that file.
  • 44.1 kHz microphone (MacBook array): the "Them" channel played ×1.088 too slow.
  • Right length, wrong speed: when the mic lane had at least as many frames as the system lane, the file lasted the right time but the other side was still at the wrong speed.

The lane files were always correct.

The fix

  • Combine: each lane is read as mono and resampled to the higher of the two rates, then the two are interleaved. Downsampling the other side to 16 kHz would cut everything above 8 kHz. Upsampling the mic loses nothing. At 48 kHz stereo the AAC encoder accepts up to 320 kbps, so the 128 kbps default passes the aacSettings clamp.
  • Streamed: each lane is read twice in 16k-frame chunks, first for its loudness and then for the conversion. The old code held both whole channels plus a 2-channel decode in RAM, gigabytes for a long meeting by calculation. The new code peaked at 27 MB RSS on a copy of a long meeting. A cancelled task stops it between chunks.
  • Loudness: one gain per lane from its whole-file RMS, as before. The ±0.97 ceiling is now applied after the rate conversion. Clipped first and resampled after, a boosted lane peaked at 1.086 in a probe; the test measured 0.9999 against the first-round code.
  • Downmix changed: a stereo lane is now the average of its channels. The old code kept the left channel only: AVAudioConverter 2→1 with downmix off (the default) gave 0.5 for L=0.5 / R=0.1 (probed). I found no sign this was deliberate; the pill's system meter already measures both channels. So the system lane's RMS, and its gain, can differ a little from before. ChannelAudio.mixToMono and envelope (the Scribe path) still keep the left channel. This is documented there, not changed.
  • Mark: every combine sets the xattr dev.kleoth.combine = resampled on its output.

Audit: other code that lines up the two lanes

Site Affected? Evidence
ChannelAudio.mixToMono (mono mix sent to Scribe, dictation upload) No It resamples both to the higher rate before summing, and has since 2026-06-01, so no cloud transcript was made from a slowed "Them".
ChannelAudio.envelope No The hop size is hopSeconds × that file's own rate, so envelope index i is a time.
ChannelAttribution (energy-based You/Them mapping) No It indexes by word.start / hopSeconds, on the per-lane files.
LocalTranscriber per channel No It transcribes each lane file on its own.
Screen recorder No The mic is converted to 48 kHz in the tap, SCK is set to 48 kHz, and both are placed by host time.
MeetingPipeline cost Yes (stored data) It probes meeting.m4a, so meta.json cost.audio_duration_secs is stretched. A cloud transcript's transcription_cost is priced on it, while Scribe itself billed the correctly resampled mono mix.
kleoth summarize cost, localtranscribe Yes (stored or printed duration) They probe meeting.m4a. They are right again after the rebuild.
MeetingAudioPlayer Yes (playback) It plays meeting.m4a.

transcript.json (Scribe's own duration is right), word timestamps, summary.* and the Markdown hold no wrong duration.

Repair of meetings already recorded

Detection (MeetingAudioLanes, pure)

  • A readable meeting.m4a below the higher lane's rate is an old file, whatever its length.
  • A mic above the system lane's rate is the one case rate cannot tell: both combines write the mic's rate there. An unmarked file then is old. A file that loses its mark in a copy is only rebuilt once more, never made wrong.
  • A meeting.m4a that exists but does not open or holds no audio (a combine cut short) is rebuilt. A missing one is left to rebuildCombinedAudioIfMissing. That function now also treats an unreadable file as missing and writes through the same safe swap.

Safe write (MeetingAudioRepair.writeCombinedFile)

  • The new file is written beside the old one and checked to last as long as the lanes.
  • It is flushed with F_FULLFSYNC (falling back to fsync) and then rename(2)'d over the old file.
  • The folder's modification date is restored in a defer, so it is kept on failure too.
  • Any failure removes the new file and keeps the old one. The lanes are only read.

Stored length

cost.audio_duration_secs and transcription_cost in meta.json and variants/*/variant.json are corrected only when the stored length equals what the old combine produced. The folder date is kept.

The app's launch pass (RecordingController.repairCombinedAudioAtLaunch)

  • Once per output folder. The state lives in UserDefaults under dev.kleoth.meetingAudioRepair: the finished pass, the pending folders and the failed ones.
  • Cheap first pass. A marked file costs one getxattr. An old file costs three header opens, once.
  • Later launches look only at pending folders. Pending means busy, not downloaded, or a stored length not yet corrected.
  • Failures are remembered and never retried by the app; recombine still repairs them. Progress is saved after each meeting.
  • Cloud folders. A file with SF_DATALESS set, or a ubiquitous item that is .notDownloaded, is never opened. The folder stays pending.
  • Captures. The pass waits while a meeting or a screen recording runs. When one starts, a rebuild under way is cancelled within about 1 s and starts over afterwards. Those captures write continuously and lose what they cannot write. A dictation does not hold the pass up: it lasts seconds and uses one light input, while the repair runs at utility priority.
  • Not on the pipeline queue, so a user's transcription never waits behind it. Never on a demo launch.
  • Concurrency. The meta.json rewrite runs on the main actor, and only while no run owns the folder.

Player

An open meeting page reloads its player when the file on disk was replaced. It compares stat identity: device, inode, size and mtime. It keeps the position and play state. On a contentRevision bump an unaffected meeting costs one stat and keeps playing. A page mounted later loads the new file in onAppear, so it does not rely on onChange at mount.

recombine [--dry-run] <meeting-dir | folder>

It runs the same repair from a shell and looks at every folder it is given, including the app's failed ones. It refuses to write while Kleoth runs (exit 3, with a message), skips files that are not downloaded, and only reads with --dry-run.

Verification

  • Probes:
    • AAC opens a file above 48 kHz and then refuses every write (-66567).
    • AVAudioConverter 2→1 keeps the left channel.
    • Clip-then-upsample overshoots to 1.086.
  • RED (new tests against older code, swapped in temporarily):
    • First round, against the original combine:
      • 16 kHz case: 15.0 s instead of 5.0 s, channel 1 at 146.7 Hz.
      • 44.1 kHz cases: 404.2 Hz, and 5.44 s instead of 5.0.
      • Both rebuild tests fail: the repair's length check refuses the old combine's output and keeps the old file.
    • This round, against the first-round combine:
      • Ceiling test: peak 0.9999 against the 0.97 limit.
      • Mark tests fail.
    • The right-length case is shown by the old rule, not by a run: it required the length to be off, and here it was not.
  • GREEN:
    • swift build && swift test: 812 tests pass.
    • swift build --package-path app: passes.
    • swift test --package-path app: 18 tests in 3 suites pass.
    • Tests use synthetic values only. They cover:
      • rate detection, including an old file of the right length;
      • the mic-above-system gap and the mark;
      • unreadable and empty files;
      • a failed rebuild (it throws halfway or comes out the wrong length): old file kept, no leftovers, date restored;
      • a constant start offset between the lanes kept within 1 ms;
      • the ceiling after resampling;
      • the pass state;
      • triage of a marked file without opening a lane.
  • Copies of real affected meetings (in a temp dir, deleted afterwards): 16 kHz, 44.1 kHz, a short mic lane, and a 48 kHz control.
    • With Kleoth running, recombine refused (exit 3). The repair then ran through a throwaway program calling the same MeetingAudioRepair API.
    • Each rebuilt file is 2 ch / 48 kHz, marked, and exactly as long as its longer lane.
    • Lane checksums are unchanged, folder dates are kept, a second pass finds nothing, and the control is untouched.
    • Channel 1 correlates with system.m4a at r = 1.0000, lag 0, against about 0 for the original.
  • Read-only dry run on the real folder (a marker file showed nothing written): the rate rule finds the same set as the length rule did, so that data holds no right-length old file.

Not verified

  • The app's launch pass was not run: the repair inside the app, the pass state in UserDefaults, the wait during captures, the cancellation, and the player reload. Running it would repair the real data. Only the code under it ran.
  • The cloud-folder check was not run against a real evicted file: I had no File Provider placeholder to test with.
  • No listening test.
  • The packaged smoke test was not run locally (CI runs it).
  • First-launch CPU time in the app was not timed.

The combine copied mic.m4a and system.m4a frame for frame at the
microphone's rate. With a 16 kHz headset the other side played three
times too slow and meeting.m4a lasted three times the meeting (x1.088
with a 44.1 kHz MacBook array). Each lane is now read as mono and
resampled to the higher of the two rates before they are interleaved,
in chunks (two passes: loudness, then conversion), so memory no longer
grows with the meeting.

MeetingAudioLanes (KleothCore) says how long a combined file must last
and how long the old combine made it; MeetingStore can correct the
cost duration stored from the stretched file.
MeetingAudioRepair rebuilds a meeting.m4a only when it matches what the
old combine made of its lanes: written beside the old file, checked to
last as long as the lanes, renamed over it; mic.m4a and system.m4a are
only read, the folder keeps its date, a crash leaves the old file.

The app runs it once per launch on a utility task of its own (not the
pipeline queue), one meeting at a time, skipping a meeting that records
or processes; the stored length goes back on the main actor. Never on a
demo launch. recombine [--dry-run] runs the same repair from a shell.
…ling after resampling

The old combine wrote at the microphone's rate, so a combined file below
the higher lane's rate is an old one whatever its length (a microphone
lane with more frames gave the right length and the wrong speed). Every
combine now marks its file (xattr dev.kleoth.combine), which also tells
the one case rate cannot: a microphone above the system lane's rate.

An existing meeting.m4a that does not open or holds no audio is rebuilt.
The new file is flushed with F_FULLFSYNC before it is renamed over the
old one, the folder date is restored on failure too, and a cancelled
task stops the combine between chunks. The scan opens nothing for a
marked file and never opens a file that is not downloaded.

The loudness gain and its 0.97 ceiling now apply after the rate change:
clipped first and resampled after, a boosted lane overshot past 0.97.
Tests use synthetic values only.
The pass state (UserDefaults dev.kleoth.meetingAudioRepair) keeps a
finished pass per output folder, the meetings left pending and those
whose rebuild failed, so later launches look only at the pending ones
and never retry a failure (recombine still repairs it). The repair waits
while a meeting or a screen recording runs and stops a rebuild under way
when one starts; a dictation does not hold it up.

rebuildCombinedAudioIfMissing treats an unreadable meeting.m4a as
missing and writes through the same safe swap. An open meeting page
reloads its player when the file on disk was replaced, at the same
position. recombine refuses to write while Kleoth runs.
@ofcRS
ofcRS merged commit 1a9c3c3 into main Nov 18, 2026
3 checks passed
@ofcRS ofcRS mentioned this pull request Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant