Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
795219f
harden(subscription): cover future fields in client decoders
Git-on-my-level Oct 14, 2026
eccc986
feat(clients): show saved content waiting on its allowance
Git-on-my-level Oct 14, 2026
19fd7aa
harden(billing): pin unresolved checkout effects in both ledger modes
Git-on-my-level Oct 14, 2026
9cd0113
feat(web): clarify legacy usage and measure Account billing
Git-on-my-level Oct 14, 2026
96c36df
test(subscription): retain waiting-state UI render evidence
Git-on-my-level Oct 14, 2026
7eb8ec9
harden(app): size waiting memory previews around their actions
Git-on-my-level Oct 14, 2026
babe47e
harden(app): keep waiting-preview fixture analyzer-clean
Git-on-my-level Oct 14, 2026
8551595
test(app): align visual audit with the saved-content waiting state
Git-on-my-level Oct 14, 2026
25fba99
feat(app): ship freemium Apple speech engine with remote rollback
Git-on-my-level Oct 14, 2026
d0c598d
feat(app): bound freemium Whisper decoding to thirty seconds
Git-on-my-level Oct 14, 2026
83e123f
fix(app): keep held memories closed on taps outside See plans
Git-on-my-level Oct 14, 2026
213b459
chore: merge main into subscription clarity lane
Git-on-my-level Oct 14, 2026
e709729
fix: address subscription-clarity review feedback
Git-on-my-level Oct 14, 2026
91e3eea
fix(app): recover local speech failures without losing audio
Git-on-my-level Oct 14, 2026
2bdba45
chore: retain concurrent subscription review fixes
Git-on-my-level Oct 14, 2026
ea215f7
test(app): match reviewed usage action in visual audit
Git-on-my-level Oct 14, 2026
830b41c
test(app): cover local recovery in capture fixtures
Git-on-my-level Oct 14, 2026
bd7c463
fix(app): preserve recording custody and local transcription clocks
Git-on-my-level Oct 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
33 changes: 33 additions & 0 deletions .agent-artifacts/ui-evidence/sub-ship-now/evidence.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"version": 1,
"images": [
{
"file": "001-desktop-macos-saved-waiting-light.png",
"platform": "desktop-macos",
"description": "Actual ConversationStatusBadge(.locked) rendered in light mode with SwiftUI ImageRenderer; isolated synthetic fixture, no account data",
"captured_by": "agent",
"source": "visual-audit"
},
{
"file": "002-desktop-macos-saved-waiting-dark.png",
"platform": "desktop-macos",
"description": "Actual ConversationStatusBadge(.locked) rendered in dark mode with SwiftUI ImageRenderer; isolated synthetic fixture, no account data",
"captured_by": "agent",
"source": "visual-audit"
},
{
"file": "003-mobile-ios-local-speech-permission.png",
"platform": "mobile-ios",
"description": "Actual local speech recovery notice with SF Pro, iOS platform and existing localized capture/custody/actions; synthetic UI fixture",
"captured_by": "agent",
"source": "visual-audit"
},
{
"file": "004-mobile-ios-local-speech-unavailable.png",
"platform": "mobile-ios",
"description": "Actual local speech recovery notice with SF Pro, iOS platform and existing localized capture/custody/actions; synthetic UI fixture",
"captured_by": "agent",
"source": "visual-audit"
}
]
}
8 changes: 8 additions & 0 deletions .github/agent-docs/on-device-speech.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,11 @@
- `app/ios/Runner/SpeechDeadline.swift` owns deadlines for SpeechAnalyzer availability and recognition. A timeout may leave a shared model download running, but must finish recognition cleanup before replying to Dart. Native speech results and timeout callbacks use the main queue for one-shot completion.
- `PurePollingSocket` keeps one provider operation in flight. Do not wrap it in `Future.timeout`: that cannot cancel native/HTTP work and permits duplicate transcription. Providers own timeouts and throw failures so retained audio can be retried; an empty successful result means no speech.
- Native deadline regression: `ruby app/ios/test/speech_deadline_test.rb` (local/CI manifest). Dart regressions (from `app/`): `flutter test test/unit/pure_polling_test.dart test/unit/on_device_transcription_upgrade_test.dart`.

## Mobile freemium engine rollback

The existing iOS speech channel selects SpeechAnalyzer/SpeechTranscriber on supported iOS 26 hardware and locales, reserving and installing its OS-managed asset through AssetInventory. Unsupported hardware/locales, model-install or recognition failures fall back to the existing on-device SFSpeech path. Existing fallback UX/status strings are retained; there is no Parakeet setup in this path.

`mobile-speech-analyzer-disabled` is an inverted PostHog operational kill switch, registered in `config/feature-flags.yaml`. Missing/false/error/offline keeps the improved engine selected; true forces SFSpeech for both availability probing and transcription. It is read per operation through the existing identity/consent-fenced analytics adapter. Users with analytics disabled retain the shipped new-engine default. The flag does not affect cloud STT, overflow, summaries, schedules or plans.

Android's existing Whisper provider splits canonical PCM WAV input into at most 30-second independent plugin calls. The pinned plugin creates/frees a whisper_context for each call; failure propagates so polling retains the complete input for retry.
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"schema_version": 1,
"id": "FC-held-content-tap-bypasses-access-guard",
"violated_contract": "Held content with subscription UI enabled must not open a full-content destination through any row or preview tap. Obscuring the child and guarding only the action button does not guard an enclosing row recognizer.",
"canonical_prevention": "Route the preview action and enclosing row through the same access guard, give the preview a full-area guarded tap recognizer, and test off-button taps on every preview surface with subscription UI enabled and disabled. Preserve interactive row menus and the App Review read-only memory exception.",
"canonical_prevention_artifact": [
"app/test/widgets/waiting_memory_preview_test.dart",
"app/test/widgets/conversation_locked_preview_test.dart"
],
"evidence_prs": [],
"scope_hints": [
"app/lib/ui/components/omi_locked_preview.dart",
"app/lib/pages/memories/widgets/memory_item.dart",
"app/lib/pages/conversations/widgets/conversation_list_item.dart"
],
"status": "open"
}
13 changes: 13 additions & 0 deletions app/android/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -54,3 +54,16 @@ gradle.projectsEvaluated {
tasks.register("clean", Delete) {
delete rootProject.buildDir
}

// The pinned FFI plugin must free model/WAV resources on every error path.
gradle.projectsEvaluated {
def sub = rootProject.subprojects.find { it.name == 'whisper_flutter_new' }
if (sub != null) {
def source = new File(sub.projectDir, '../src/main.cpp')
def patch = new File(rootProject.projectDir, '../scripts/patch_whisper_cleanup.py')
def proc = ['python3', patch.absolutePath, source.canonicalPath].execute(null, rootProject.projectDir)
def out = new StringBuffer(); def err = new StringBuffer()
proc.consumeProcessOutput(out, err); proc.waitFor()
if (proc.exitValue() != 0) throw new GradleException("Whisper cleanup patch failed: ${err} ${out}")
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"change": "On-device fallback uses Apple’s newer speech engine on supported iPhones and decodes Android recordings in bounded chunks for better transcription."
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"change": "Saved conversations and memories waiting on your allowance link to usage without an upgrade prompt"}
3 changes: 3 additions & 0 deletions app/integration_test/visual_audit/scenarios/capture.dart
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import 'package:omi/pages/home/widgets/home_daily_recaps.dart';
import 'package:omi/pages/home/widgets/home_tab_switcher.dart';
import 'package:omi/backend/schema/daily_summary.dart';
import 'package:omi/providers/capture_provider.dart';
import 'package:omi/services/sockets/local_stt_failure.dart';
import 'package:omi/providers/conversation_provider.dart';
import 'package:omi/backend/schema/structured.dart';
import 'package:omi/providers/device_provider.dart';
Expand All @@ -38,6 +39,8 @@ enum AuditLive { idle, pendant, pendantPaused, phone, phonePaused, phoneAfterPen
class AuditCaptureProvider extends ChangeNotifier implements CaptureProvider {
AuditCaptureProvider(this.live);
final AuditLive live;
@override
LocalSttFailure? get localSttFailure => null;

@override
bool get pendantCaptureVerified => true;
Expand Down
11 changes: 7 additions & 4 deletions app/integration_test/visual_audit/scenarios/conversations.dart
Original file line number Diff line number Diff line change
Expand Up @@ -94,10 +94,10 @@ const _twoSourceGroup = CaptureGroup(
final conversationsScenarios = <AuditScenario>[
AuditScenario(
id: 'conversations-locked-preview',
title: 'Locked conversations with an upgrade action',
title: 'Saved conversations waiting on their allowance',
page: _page,
state: 'An unlocked conversation followed by three consecutive locked previews, which share one '
'frosted card and one upgrade action; synthetic titles only',
'frosted card, a waiting status and one usage action; synthetic titles only',
run: (a) async {
final items = [
auditConversation('unlocked', title: 'Design catch-up with Alex'),
Expand All @@ -114,8 +114,11 @@ final conversationsScenarios = <AuditScenario>[
await a.pump(const ConversationsPage(requestInitialLoad: false), providers: _listProviders(items));
expect(find.byType(ConversationListItem), findsNWidgets(4));
expect(find.byType(LockedConversationRun), findsOneWidget);
expect(find.text('Upgrade to Unlimited'), findsOneWidget);
await a.shot('Conversation list with frosted locked previews');
expect(find.text('Pending'), findsOneWidget);
expect(find.text('View Usage'), findsOneWidget);
expect(find.byIcon(Icons.lock_outline), findsNothing);
expect(find.byIcon(Icons.hourglass_empty), findsOneWidget);
await a.shot('Conversation list with saved-content waiting previews');
},
),
AuditScenario(
Expand Down
6 changes: 6 additions & 0 deletions app/ios/Podfile
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,12 @@ else
end

post_install do |installer|
# Fail closed on native dependency drift; patch both copies compiled by the pod.
plugin = File.realpath(File.join(__dir__, '.symlinks/plugins/whisper_flutter_new'))
patch = File.expand_path('../scripts/patch_whisper_cleanup.py', __dir__)
sources = [File.join(plugin, 'src/main.cpp'), File.join(plugin, 'ios/Classes/whisper_flutter_new.cpp')]
raise 'Whisper cleanup patch failed' unless system('python3', patch, *sources)

installer.pods_project.targets.each do |target|
flutter_additional_ios_build_settings(target)
target.build_configurations.each do |config|
Expand Down
126 changes: 75 additions & 51 deletions app/ios/Runner/AppDelegate.swift
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import UIKit

Check warning on line 1 in app/ios/Runner/AppDelegate.swift

View workflow job for this annotation

GitHub Actions / Hygiene

Large changed file

app/ios/Runner/AppDelegate.swift is 1399 lines; consider splitting files over 800 lines.
import Flutter
import UserNotifications
import app_links
Expand Down Expand Up @@ -194,7 +194,7 @@
return launched
}

func didInitializeImplicitFlutterEngine(_ engineBridge: FlutterImplicitEngineBridge) {

Check warning on line 197 in app/ios/Runner/AppDelegate.swift

View workflow job for this annotation

GitHub Actions / Hygiene

Long function

func didInitializeImplicitFlutterEngine(_ engineBridge: FlutterImplicitEngineBri is 393 lines; consider extracting focused helpers over 150 lines.
GeneratedPluginRegistrant.register(with: engineBridge.pluginRegistry)
let messenger = engineBridge.applicationRegistrar.messenger()
FlutterMethodChannel(name: "com.omi/device_health_policy", binaryMessenger: messenger).setMethodCallHandler { call, result in
Expand Down Expand Up @@ -1030,6 +1030,13 @@
/// Installs the model for `locale` if it is not already on the device.
/// Concurrent callers wait on the same download.
static func ensureModel(for locale: Locale) async throws {
// This app keeps one locale reservation; OS-installed assets remain OS-owned.
for previous in await AssetInventory.reservedLocales where previous.identifier(.bcp47) != locale.identifier(.bcp47) {
installTasks[previous.identifier(.bcp47)]?.cancel()
installTasks[previous.identifier(.bcp47)] = nil
_ = await AssetInventory.release(reservedLocale: previous)
}
_ = try await AssetInventory.reserve(locale: locale)
if await isInstalled(locale) { return }
let key = locale.identifier(.bcp47)
let task: Task<Void, Error>
Expand All @@ -1054,34 +1061,32 @@

/// Whether transcription can run for `language`: the locale is supported
/// and its model is installed, or finishes installing within
/// `installWait`. A download still running after that counts as available
/// too; it continues in the background and transcribe() waits for it.
/// Only a failed download (no network, unsupported locale) reports false.
/// `installWait`. A pending install continues, but reports not ready.
/// The caller can offer setup/retry while captured audio stays retained.
static func isAvailable(language: String, installWait: Double = 8) async -> Bool {
guard let locale = await locale(for: language) else {
guard SpeechTranscriber.isAvailable, let locale = await locale(for: language) else {
NSLog("[SpeechAnalyzer] no supported locale for language %@", language)
return false
}
if await isInstalled(locale) { return true }
return await SpeechDeadline.run(seconds: installWait, operation: {
do {
try await ensureModel(for: locale)
return true
return await isInstalled(locale)
} catch {
NSLog("[SpeechAnalyzer] model install failed for %@: %@", locale.identifier(.bcp47), error.localizedDescription)
return false
}
}, onTimeout: {
// The shared download keeps running; only this availability waiter
// has a deadline. A later transcription shares the same download.
true
// The shared install continues, but readiness must not claim success.
false
})
}

/// Transcribes a whole audio file (the Dart side writes 16 kHz mono WAV
/// clips) and returns the text, empty when no speech was recognized.
static func transcribe(fileURL: URL, language: String) async throws -> String {
guard let locale = await locale(for: language) else {
guard SpeechTranscriber.isAvailable, let locale = await locale(for: language) else {
throw TranscriptionError.unsupportedLanguage(language)
}
let installed: Result<Void, Error> = await SpeechDeadline.run(seconds: 20, operation: {
Expand Down Expand Up @@ -1135,53 +1140,72 @@
class SpeechRecognitionHandler: NSObject {

func handle(_ call: FlutterMethodCall, result: @escaping FlutterResult) {
if call.method == "transcribe" {
guard let args = call.arguments as? [String: Any],
let path = args["filePath"] as? String else {
result(FlutterError(code: "INVALID_ARGS", message: "Missing arguments", details: nil))
return
}

let language = args["language"] as? String ?? "en-US"
if #available(iOS 26, *) {
// SpeechAnalyzer needs no Dictation setting; fall back to
// SFSpeechRecognizer only if it cannot handle this clip.
Task {
do {
let text = try await SpeechAnalyzerTranscription.transcribe(
fileURL: URL(fileURLWithPath: path), language: language)
DispatchQueue.main.async { result(text) }
} catch {
NSLog("[SpeechAnalyzer] transcribe failed, falling back to SFSpeechRecognizer: %@", error.localizedDescription)
DispatchQueue.main.async {
self.transcribe(filePath: path, language: language, result: result)
}
let args = call.arguments as? [String: Any] ?? [:]
let language = args["language"] as? String ?? "en-US"
let preferAnalyzer = args["useSpeechAnalyzer"] as? Bool != false
Task { @MainActor in
let coordinator = coordinator(language: language, path: args["filePath"] as? String)
switch call.method {
case "onDeviceAvailable", "readiness", "prepareSpeech":
var readiness = await coordinator.readiness(preferAnalyzer: preferAnalyzer)
if call.method == "prepareSpeech", readiness.status == "permission_required" {
_ = await withCheckedContinuation { continuation in
SFSpeechRecognizer.requestAuthorization { continuation.resume(returning: $0) }
}
readiness = await coordinator.readiness(preferAnalyzer: preferAnalyzer)
}
return
}
transcribe(filePath: path, language: language, result: result)
} else if call.method == "onDeviceAvailable" {
let args = call.arguments as? [String: Any]
let language = args?["language"] as? String ?? "en-US"
if #available(iOS 26, *) {
Task {
if await SpeechAnalyzerTranscription.isAvailable(language: language) {
DispatchQueue.main.async { result(true) }
} else {
DispatchQueue.main.async {
self.probeOnDeviceRecognition(language: language, result: result)
}
}
result(call.method == "onDeviceAvailable" ? readiness.ready : readiness.wire)
case "transcribe":
guard args["filePath"] is String else {
result(FlutterError(code: "INVALID_ARGS", message: "Missing audio file", details: nil)); return
}
return
do { result(try await coordinator.transcribe(preferAnalyzer: preferAnalyzer)) }
catch let error as LocalSpeechUnavailable {
let status = error.readiness.status
result(FlutterError(code: status == "permission_required" || status == "permission_denied" ? "UNAUTHORIZED" : "UNAVAILABLE",
message: "Selected local speech engine needs setup", details: error.readiness.wire))
} catch { result(FlutterError(code: "RECOGNITION_ERROR", message: error.localizedDescription, details: nil)) }
default: result(FlutterMethodNotImplemented)
}
probeOnDeviceRecognition(language: language, result: result)
} else {
result(FlutterMethodNotImplemented)
}
}

@MainActor
private func coordinator(language: String, path: String?) -> SpeechFallbackCoordinator {
SpeechFallbackCoordinator(analyzerReady: {
if #available(iOS 26, *) { return await SpeechAnalyzerTranscription.isAvailable(language: language) }
return false
}, legacyReady: {
let permission = SFSpeechRecognizer.authorizationStatus()
if permission != .authorized {
return LocalSpeechReadiness(engine: "sfspeech", status: permission == .notDetermined ? "permission_required" : "permission_denied")
}
let available: Bool = await withCheckedContinuation { continuation in
self.probeOnDeviceRecognition(language: language) { value in
continuation.resume(returning: value as? Bool ?? false)
}
}
return LocalSpeechReadiness(engine: "sfspeech", status: available ? "ready" : "unavailable")
}, analyze: {
if #available(iOS 26, *), let path {
return try await SpeechAnalyzerTranscription.transcribe(fileURL: URL(fileURLWithPath: path), language: language)
}
throw LocalSpeechUnavailable(readiness: LocalSpeechReadiness(engine: "speech_analyzer", status: "unavailable"))
}, legacy: {
guard let path else { throw LocalSpeechUnavailable(readiness: LocalSpeechReadiness(engine: "sfspeech", status: "unavailable")) }
return try await withCheckedThrowingContinuation { continuation in
self.transcribe(filePath: path, language: language) { value in
if let error = value as? FlutterError {
if error.code == "UNAUTHORIZED" || error.code == "UNAVAILABLE" {
continuation.resume(throwing: LocalSpeechUnavailable(readiness:
LocalSpeechReadiness(engine: "sfspeech", status: error.code == "UNAUTHORIZED" ? "permission_denied" : "unavailable")))
} else { continuation.resume(throwing: NSError(domain: error.code, code: 1)) }
} else { continuation.resume(returning: value as? String ?? "") }
}
}
})
}

/// Resolve the app's language setting to a locale whose recognizer can run
/// on-device. The app passes bare language codes ("en"); on-device assets
/// are installed per full locale ("en-US"), and a recognizer built from a
Expand Down Expand Up @@ -1230,7 +1254,7 @@
return
}
guard let silence = SpeechRecognitionHandler.writeSilentWav(seconds: 0.6) else {
result(true) // Could not build a probe clip; do not block the fallback on that.
result(false) // Unknown readiness is not a working recognizer.
return
}

Expand Down Expand Up @@ -1265,7 +1289,7 @@
}
DispatchQueue.main.asyncAfter(deadline: .now() + 5) {
guard !finished else { return }
finish(true) // Slow but not failing; let the real request decide.
finish(false) // A timed-out probe cannot attest readiness.
}
}

Expand Down
Loading
Loading