Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion app/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ Never run `flutterfire configure` — it overwrites prod credentials. Config fil
- Modules: iOS `ios/Runner/PhoneMic/` (self-healing AVAudioEngine capture; interruptions/routes recover natively); Android `android/app/src/main/kotlin/com/friend/ios/phonemic/` (AudioRecord with self-healing rebuild + silencing detection; `PhoneMicForegroundService` keeps background capture alive; batch opus encode via a JNI shim over the plugin-shipped libopus)
- Dart service: `lib/services/mic/native_mic_recorder_service.dart` behind `ServiceManager.phoneMic`; chat memos/speech profile stay on flutter_sound via `ServiceManager.mic`; `MicArbiter` arbitrates the two stacks
- Events carry a Dart-minted session id (`start(mode, sessionId)`); Dart drops foreign-id events, `start()` onto a live session adopts the new id and re-emits state, and `stop()` always forwards to native then tears down locally once
- Two capture modes fixed at `start(mode)`: `stream` (realtime frames → Dart → socket/WAL) and `batch` (Transcribe Later — native opus encode → WAL-compatible `audio_omibatchphone[auto]_…bin`; no frames cross to Dart; liveness = 1Hz `onBatchProgress`). Selection: `CaptureController.streamRecording` (explicit or offline fallback); `omibatchphoneauto` auto-uploads on reconnect
- Two capture modes fixed at `start(mode)`: `stream` (frames → Dart → socket/WAL) and `batch` (Transcribe Later — opus → WAL-compatible `audio_omibatchphone[auto]_…bin`; no frames cross to Dart; liveness = `onBatchProgress`: iOS 4s healthy/1s fault, Android 1Hz; Dart 10s unchanged). Selection: `CaptureController.streamRecording` (explicit/offline); `omibatchphoneauto` uploads on reconnect

On-device speech deadlines and cleanup: [contract](../.github/agent-docs/on-device-speech.md).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -284,8 +284,9 @@ class PhoneMicFlutterApi(private val binaryMessenger: BinaryMessenger, private v
}
}
/**
* Emitted at 1Hz while batch capture runs. Its arrival is the liveness signal
* for the Dart batch watchdog; the value is derived from the number of frames
* Emitted while batch capture runs (iOS: 4s while healthy, 1s on fault or
* foreground; Android: 1Hz). Its arrival is the liveness signal for the Dart
* batch watchdog; the value is derived from the number of frames
* actually written to disk, so mutes and interruptions freeze it while the
* event keeps arriving.
*/
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"kind": "none"}
4 changes: 2 additions & 2 deletions app/ios/Runner/PhoneMic/PhoneMicBatchAudioWriter.swift
Original file line number Diff line number Diff line change
Expand Up @@ -144,8 +144,8 @@ final class PhoneMicBatchAudioWriter: BaseBatchAudioWriter {
closeCurrentLocked(reason)
}

/// Drain the latched storage-full transition. Called by the controller on its
/// 1Hz progress tick (inside `audioQueue.sync`) so at most one onCaptureError
/// Drain the latched storage-full transition. The write path reports the edge
/// immediately; the progress tick is the backup. At most one onCaptureError
/// fires per false->true edge.
func consumeStorageFullTransitionLocked() -> Bool {
let pending = pendingStorageFullReport
Expand Down
103 changes: 90 additions & 13 deletions app/ios/Runner/PhoneMic/PhoneMicController.swift
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,15 @@ final class PhoneMicController {
private static let bringUpRetryDelay: TimeInterval = 0.35
private static let maxStartRetries = 2
private static let resumeTickInterval: TimeInterval = 3.0
private static let progressTickInterval: TimeInterval = 1.0
/// Fault cadence for the batch liveness heartbeat. Interruptions, rebuilds,
/// and the first beat after foreground use this so a gap is still visible
/// inside the Dart 10s batch deadline.
private static let fastProgressTickInterval: TimeInterval = 1.0
/// Healthy `.running` cadence. Strictly under half the 10s Dart stall
/// threshold: even if one beat is lost or coalesced, the next lands at
/// ~8.5s (2x4s + the 500ms leeway) — one skipped tick is not itself a
/// stall. A 5s cadence would put the second beat exactly at the deadline.
private static let healthyProgressTickInterval: TimeInterval = 4.0

private let controlQueue = DispatchQueue(label: "com.omi.phonemic.control", qos: .userInitiated)
private let audioQueue = DispatchQueue(label: "com.omi.phonemic.audio", qos: .userInitiated)
Expand Down Expand Up @@ -53,6 +61,9 @@ final class PhoneMicController {
private var batchWriter: PhoneMicBatchWriting?
private var batchMarker = "omibatchphone"
private var progressTimer: DispatchSourceTimer?
/// Repeating interval currently armed, so a tick can retarget after a
/// state change that did not reschedule itself.
private var progressIntervalArmed: TimeInterval = 0

init(environment: PhoneMicControllerEnvironment) {
self.environment = environment
Expand Down Expand Up @@ -228,11 +239,12 @@ final class PhoneMicController {
state = .running
NSLog("[PhoneMic] running, route %@", environment.describeRoute())
emitter.emitState(.running, sessionId: currentSessionId)
// Batch progress ticker: armed once on first reach of running, kept alive
// Batch progress ticker: armed on first reach of running, kept alive
// across interruptions/rebuilds (its arrival is the Dart liveness signal),
// cancelled only on stop/idle.
if mode == .batch, progressTimer == nil {
armProgressTimer()
// cancelled only on stop/idle. Healthy running uses the slow cadence;
// this call retargets after a fast anomaly window.
if mode == .batch {
applyProgressCadence()
Comment thread
Git-on-my-level marked this conversation as resolved.
}
resolvePendingStarts(.success(()))
if pendingStop {
Expand Down Expand Up @@ -343,6 +355,10 @@ final class PhoneMicController {
if engine?.isRunning != true {
NSLog("[PhoneMic] app active with dead engine, rebuilding")
beginRebuild()
} else if mode == .batch {
// One fast liveness beat, then the healthy cadence. Pulls a
// far-off 4s deadline forward without leaving the 1s poll on.
syncProgressTimer(first: Self.fastProgressTickInterval, repeating: Self.healthyProgressTickInterval)
}
default:
break
Expand Down Expand Up @@ -382,6 +398,7 @@ final class PhoneMicController {
state = .interrupted
emitter.emitState(.interrupted, sessionId: currentSessionId)
armResumeTicker()
applyProgressCadence()
}

/// Probe a resume from .interrupted. Failure is silent by design — during
Expand All @@ -398,6 +415,7 @@ final class PhoneMicController {
private func beginRebuild() {
state = .rebuilding
emitter.emitState(.rebuilding, sessionId: currentSessionId)
applyProgressCadence()
if attemptBringUp() == nil {
enterRunning()
return
Expand Down Expand Up @@ -465,12 +483,23 @@ final class PhoneMicController {
let encoder = batchEncoder
let writer = batchWriter
let marker = batchMarker
return { data, _ in
// The audio bytes stay on this closure. The weak hop is only the
// storage-full fault edge, so the healthy 4s heartbeat cannot delay it.
return { [weak self] data, _ in
guard let encoder, let writer else { return }
let packets = encoder.encode(data)
if !packets.isEmpty {
writer.append(opusPackets: packets, marker: marker)
}
if writer.consumeStorageFullTransitionLocked() {
self?.controlQueue.async { [weak self] in
guard let self, self.state != .idle else { return }
self.emitter.emitError(
code: "batch_storage_full",
message: "storage is low; batch capture paused until space frees up",
sessionId: self.currentSessionId)
}
}
}
}
}
Expand Down Expand Up @@ -519,26 +548,74 @@ final class PhoneMicController {
resumeTicker = nil
}

private func armProgressTimer() {
cancelProgressTimer()
private func desiredProgressInterval() -> TimeInterval {
state == .running ? Self.healthyProgressTickInterval : Self.fastProgressTickInterval
}

/// Arm, or retarget, the batch heartbeat. `repeating` is what subsequent
/// ticks use; `first` can be shorter so an anomaly gets one fast beat
/// before returning to the healthy cadence. Leeway is sized for the
/// soonest fire so a fast recovery beat cannot be coalesced into the
/// looser healthy tolerance.
private func syncProgressTimer(first: TimeInterval, repeating: TimeInterval) {
guard mode == .batch, state != .idle, batchWriter != nil else { return }
let leeway: DispatchTimeInterval = min(first, repeating) >= Self.healthyProgressTickInterval
? .milliseconds(500)
: .milliseconds(50)
if let timer = progressTimer {
timer.schedule(deadline: .now() + first, repeating: repeating, leeway: leeway)
progressIntervalArmed = repeating
return
}
let timer = DispatchSource.makeTimerSource(queue: controlQueue)
timer.schedule(deadline: .now() + Self.progressTickInterval, repeating: Self.progressTickInterval)
timer.schedule(deadline: .now() + first, repeating: repeating, leeway: leeway)
timer.setEventHandler { [weak self] in
self?.emitBatchProgressTick()
guard let self else { return }
self.emitBatchProgressTick()
let desired = self.desiredProgressInterval()
if self.state != .idle, self.progressIntervalArmed != desired {
self.syncProgressTimer(first: desired, repeating: desired)
Comment thread
Git-on-my-level marked this conversation as resolved.
}
}
timer.resume()
progressTimer = timer
progressIntervalArmed = repeating
}

private func applyProgressCadence() {
let interval = desiredProgressInterval()
if progressIntervalArmed != 0 {
// An armed fire's deadline is at most one interval out, so
// re-anchoring it to `now + interval` can only delay the next
// liveness beat. Keep any already-armed deadline — a preserved
// fast fire through recovery, a second fault arriving before that
// fire, or an unchanged healthy window — and let the tick handler
// retarget the repeating cadence after the beat fires. Only a
// fault pulling a far-off healthy deadline in to the fast cadence
// re-schedules. Delaying the beat is what let a late-window
// recovery or a repeated fault push the only liveness event past
// Dart's unchanged 10s stall deadline, restarting a recovered
// session.
if progressIntervalArmed == Self.fastProgressTickInterval
|| progressIntervalArmed == interval {
return
}
}
syncProgressTimer(first: interval, repeating: interval)
}

private func cancelProgressTimer() {
progressTimer?.cancel()
progressTimer = nil
progressIntervalArmed = 0
}

/// 1Hz on controlQueue. Reads the writer's committed frame count and drains any
/// storage-full edge on its own (audio) queue, then emits progress. Emitted every
/// Heartbeat on controlQueue. Reads the writer's committed frame count and drains
/// any storage-full edge the write path did not already report, then emits
/// progress. Healthy running is 4s; interrupted/rebuilding is 1s. Emitted every
/// tick even while interrupted/muted — the value freezes (frames-written derived),
/// but its arrival is the Dart watchdog's liveness signal.
/// but its arrival is the Dart watchdog's liveness signal. The Dart batch stall
/// threshold stays 10s, so the slower healthy beat is not itself a gap.
private func emitBatchProgressTick() {
guard state != .idle, let writer = batchWriter else { return }
let (frames, storageFullEdge): (Int64, Bool) = audioQueue.sync {
Expand Down
3 changes: 2 additions & 1 deletion app/ios/Runner/PhoneMic/PhoneMicEventEmitter.swift
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,9 @@ final class PhoneMicEventEmitter {
}
}

/// Batch-mode 1Hz liveness/progress. Not epoch-gated: unlike frames it carries
/// Batch-mode liveness/progress. Not epoch-gated: unlike frames it carries
/// no audio, and its steady arrival is the Dart watchdog's liveness signal.
/// Healthy iOS capture emits this every 4s; fault states snap back to 1s.
func emitBatchProgress(_ capturedSeconds: Double, sessionId: Int64) {
DispatchQueue.main.async { [weak self] in
self?.sink.onBatchProgress(capturedSeconds: capturedSeconds, sessionId: sessionId)
Expand Down
10 changes: 6 additions & 4 deletions app/ios/Runner/PhoneMic/PhoneMicPigeon.g.swift
Original file line number Diff line number Diff line change
Expand Up @@ -235,8 +235,9 @@ protocol PhoneMicFlutterApiProtocol {
/// Non-fatal runtime failures (capture self-heals): converter_failed,
/// rebuild_failed, resume_failed, media_services_reset, batch_storage_full.
func onCaptureError(code codeArg: String, message messageArg: String, sessionId sessionIdArg: Int64, completion: @escaping (Result<Void, PhoneMicPigeonError>) -> Void)
/// Emitted at 1Hz while batch capture runs. Its arrival is the liveness signal
/// for the Dart batch watchdog; the value is derived from the number of frames
/// Emitted while batch capture runs (iOS: 4s while healthy, 1s on fault or
/// foreground; Android: 1Hz). Its arrival is the liveness signal for the Dart
/// batch watchdog; the value is derived from the number of frames
/// actually written to disk, so mutes and interruptions freeze it while the
/// event keeps arriving.
func onBatchProgress(capturedSeconds capturedSecondsArg: Double, sessionId sessionIdArg: Int64, completion: @escaping (Result<Void, PhoneMicPigeonError>) -> Void)
Expand Down Expand Up @@ -310,8 +311,9 @@ class PhoneMicFlutterApi: PhoneMicFlutterApiProtocol {
}
}
}
/// Emitted at 1Hz while batch capture runs. Its arrival is the liveness signal
/// for the Dart batch watchdog; the value is derived from the number of frames
/// Emitted while batch capture runs (iOS: 4s while healthy, 1s on fault or
/// foreground; Android: 1Hz). Its arrival is the liveness signal for the Dart
/// batch watchdog; the value is derived from the number of frames
/// actually written to disk, so mutes and interruptions freeze it while the
/// event keeps arriving.
func onBatchProgress(capturedSeconds capturedSecondsArg: Double, sessionId sessionIdArg: Int64, completion: @escaping (Result<Void, PhoneMicPigeonError>) -> Void) {
Expand Down
Loading
Loading