feat(app): on-device overflow engines, processing and summaries (flagged off)

[Git-on-my-level]

Summary

Replace the draft's native-engine, power-source, runner and summary stubs with working implementations. Everything remains behind mobile-on-device-overflow (default OFF), including the enhanced iOS freemium fallback. No remote flag is changed. This PR remains a draft for UX audit.

• iOS 26+: supported SpeechAnalyzer/SpeechTranscriber devices/locales use OS-managed assets with reservation, installation progress, word timing, cancellation and native deadlines. No Omi speech-model package download.
• iOS 17–25, or unsupported Apple speech on newer OS: English FluidAudio Parakeet TDT v2, pinned to 19600a485baa4998812e4654b70d2bab8f2c9949, with the 443 MB setup download, Wi-Fi-only cancellation, status and removal. A dynamically loaded iOS-17 framework preserves the app's iOS-15 minimum. Older/other-locale devices keep explicitly lower-quality SFSpeech.
• Android: existing whisper.cpp tiny download (74 MB), independent ~30-second decoding calls with reset contexts. No sherpa model is introduced.
• Existing phone WAL/native batch custody: explicit phone-only records are excluded from automatic and manual cloud-audio drains. Results commit to the account-owned WAL index before appearing in normal conversations. Account changes fence in-flight work; retries use the existing from-segments idempotency contract.
• Charging is the default. The existing native phone-battery channel feeds policy; Transcribe Now uses the same runner. iOS BGProcessingTask requests external power for charging runs. Android WorkManager reuses a live Flutter engine or starts a retained headless entry point, with a startup fence against concurrent WAL writers and an eight-minute worker budget.
• Foundation Models summarizes bounded chunks when available, submitted through the existing hash-bound client_processing display projection (runtime=local). Otherwise Omi Cloud processes transcript text; this does not consume live-STT hours. Keep everything on this phone creates transcript-only local conversations and submits nothing.
• Engine provenance survives ingest through additive stt_provider support. Local summary projections survive refresh only while their transcript digest matches. On-device transcripts have one unlabeled voice; speaker labels/counts and export prefixes are suppressed.
• The free monthly allowance now triggers the flagged at-cap choice, with flag-off behavior covered by tests. Battery percentages were removed from presentation; policy floors are conservative estimates.

Redo with Omi Cloud

POST /v1/conversations/{id}/reprocess regenerates intelligence from the saved transcript; it does not rerun STT. The audio re-transcription button remains hidden for these phone-only recordings rather than calling that endpoint. No new cloud-audio endpoint is added in this PR.

Copy ruling follow-up

The on-device upgrade line is exactly “Upgrade to Pro for speaker separation and higher accuracy”, localized through the repository flow in all 49 locales. Four affected renders were regenerated and inspected.

How to audit

Use a debug dev build with --dart-define=OMI_ON_DEVICE_OVERFLOW=true (for example, add it to the repository's normal flutter run --flavor dev flow). The override is debug-only; the remote flag stays off.

1. Settings → Transcription → On-Device Transcription. On supported iOS 26 inspect the no-Omi-download state and OS asset preparation; on older supported iOS download Parakeet; on Android download tiny.
2. Leave While charging selected. At the monthly cap (free or Plus), choose Transcribe on this phone. Plus can opt into saving live hours before the cap.
3. Record, finalize audio, and inspect the waiting banner. Plug in or press Transcribe Now. Inspect progress, transcript, source provenance, and absence of speaker labels.
4. On an Apple Intelligence device inspect local summary provenance. On unsupported devices inspect the cloud-text-summary explanation. Enable Keep everything on this phone and confirm subsequent results contain transcripts only.
5. Remove an Omi-downloaded model, cancel setup, retry an unavailable cloud-text submission, and verify the original phone audio remains available.
6. Turn the debug override off and verify the existing flow; local phone-only custody remains protected from accidental audio upload.

Internal benchmark evidence

Aggregate results from an internal pendant sample, measured on M5 Max. These are disagreement with cloud, not truth-labelled accuracy: SpeechAnalyzer 44.5%; FluidAudio Parakeet v2 45.5%; sherpa Parakeet int8 44.7%; bounded 30-second whisper.cpp tiny 52.2% (whole-file diagnostic 80.4%); legacy on-device SFSpeech 84.1%. FluidAudio ASR was approximately 443 MB and 0.008 Mac RTF. Sherpa's approximately 2.4 GB Mac RSS excludes it from this Android change.

On-device speaker separation is omitted: FluidAudio 52.9% and sherpa 49.7% both trailed the 54.5% one-speaker control. No transcript text, audio, sample identifiers or private benchmark outputs are included.

Unmeasured and decisions to audit

Phone battery/energy, thermals, phone throughput/memory, model-install recovery on real phones, and OS scheduling/cold-launch acceptance remain unmeasured. OS background execution is best effort. “As you talk” processes finalized existing WAL/batch files; latency and conversation boundaries include their rotation behavior (native pendant batch files currently rotate at 15 minutes). This draft does not claim live-like second-level latency.

The iOS free fallback upgrade stays behind the same flag because model availability, language coverage and phone acceptance still need audit. A local summary can be generated on the phone while transcript text and its display projection are synced to Omi; only Keep everything on this phone prevents that submission. The earlier “nothing left the device” claim was removed.

Verification

• Scoped flutter analyze: no issues for the on-device modules/screens, engine providers and new tests.
• Flutter focused/regression suite: 233 tests passed (runner, selection, custody/account fencing, provenance/hash binding, free trigger, existing freemium/listen, transcript/speaker formatting, phone battery and capture retention).
• Android: ./gradlew :app:testDevDebugUnitTest --tests com.friend.ios.ondevice.OnDeviceBackgroundWorkerTest passed (charging and interval constraints).
• Backend: BACKEND_UNIT_TEST_FILE_LIST=<four-file-list> bash backend/test.sh passed; files: test_transcript_hash.py, test_client_processing_schema.py, test_client_processing_acceptance.py, test_developer_from_segments_idempotency.py.
• Synthetic host smoke using macOS say: current Parakeet bridge returned 18 word timestamps; current Foundation Models summarizer returned a bounded overview; SpeechAnalyzer transcribed the synthetic clip. No private sample was used.
• flutter build ios --debug --no-codesign --flavor dev and flutter build apk --debug --flavor dev: both passed on the final post-merge source; no codesigning or device installation was performed.
• All 22 post-merge render PNGs were visually inspected. SF Pro loaded via FontLoader including FlutterTest; temporary Roboto-family patches restored byte-for-byte. Tracked evidence refreshed below.

Failure-Class: none

Product invariants affected

• INV-DATA-1: the headless engine reuses the exact UI Firebase/profile initialization and canonical API authority; no data-plane routing or identity is changed.
• INV-CAP-1: audio remains in existing durable phone custody while cloud STT is suppressed; runner/account-fence and capture retention tests cover preservation.
• INV-UI-3: the speaker naming rule now explicitly omits on-device labels and counts; shared formatter and transcript tests cover presentation.

UI evidence

• At-cap choice sheet: Transcribe on this phone (Included) or Upgrade to Pro
• Setup step 1: speech model not downloaded, Download Speech Model, Wi-Fi only
• Setup step 1: downloading on Wi-Fi at 62% with time left and Cancel
• Setup step 1: model installed
• Schedule with qualitative battery copy; charging is the default
• Conversations: waiting banner with Transcribe Now, Waiting for charger chips, provenance chips
• Conversations: a run in progress, Transcribing chips
• Local summary provenance; unavailable cloud audio re-transcription action is hidden
• Cloud text summary provenance and transcript-only privacy choice; no audio redo action
• On-Device Transcription settings: live-hours meter and after-hours options
• On-Device Transcription settings: model, waiting audio, Remove Speech Model
• iPhone without an on-device model yet, at cap: audio stays saved, Get More Live Hours
• Conversations waiting banner and chips in light appearance
• Android at cap without a model: finish setup banner, Saved, waiting for setup chip
• Android finish setup banner while the model downloads
• Android summary by Omi Cloud with provenance and keep-everything switch
• iOS 26 setup with OS-managed assets and no Omi model download
• Apple asset installation progress
• Older supported iOS downloads Parakeet v2: 443 MB and Wi-Fi-only option
• Free monthly allowance exhausted: same on-device choice
• On-phone transcription progress
• Legacy Apple native speech with lower-quality warning and no Omi download

Round-2 fallback recovery port

Equivalent fixes from READY #21474: consumed-audio clocks include trailing silence; canonical 16 kHz PCM and <=30-second Whisper calls; unconditional native context/WAV cleanup on rejection and exceptions; permanent input/model failures pause retained retries. Actual selected-engine readiness includes legacy permissions. Rollback with denied/unavailable SFSpeech keeps recording/audio custody active and shows existing localized saved-recording copy with setup/settings and retry actions. The existing WAL owns the retained audio; cloud drains reject phone-only WALs. Locale changes release prior OS asset reservations.

The existing free fallback now shares mobile-speech-analyzer-disabled with READY (false/missing selects the improved engine, true rolls back). Overflow batch engines/Parakeet, runner, schedule and provenance remain behind mobile-on-device-overflow. Batch readiness probes its selected engine; the common fallback core/native coordinator is equivalent to READY, with only the cutover batch and word/provenance extensions retained. No remote flag was changed.

Behavioral validation covers polling speech-then-silence offset, permanent input pause, denied/unavailable rollback, readiness and retained recovery, physical WAL persistence and no cloud-audio drain, composite capture continuity and actionable localized recovery at 200% text. Actual pinned native wrapper passes 600 rejection/error/exception paths under ASan/UBSan without leaked contexts or WAV handles; Swift fallback coordinator/deadline tests pass. Existing overflow runner/policy/UI regression tests run separately. Phone battery and physical-device runtime remain unmeasured.

INV-CAP-1: captured audio remains in the existing account-owned WAL through local engine failure. INV-UI-1 and INV-UI-3: shared Omi controls and existing localization are reused.

Final local evidence: READY focused 209 tests plus 150 custody/recovery and the retained lead regression; overflow 205 focused, 60 runner/schedule/UI, 150 custody/recovery and the retained lead regression. Analyzer ratchets pass without baseline edits; iOS and Android dev debug builds pass on both branches. A fresh current-native SpeechAnalyzer synthetic say host smoke passes with unsupported-locale rejection.

Line-Count-Exception: app/lib/pages/conversation_detail/conversation_detail_provider.dart | 1514 -> 1523 | Existing owner retains overflow plus shared local recovery; behavioral coverage verifies capture and custody.

Line-Count-Exception: app/lib/pages/settings/transcription_settings_page.dart | 1554 -> 1575 | Existing owner retains overflow plus shared local recovery; behavioral coverage verifies capture and custody.

Line-Count-Exception: app/lib/providers/conversation_provider.dart | 2119 -> 2131 | Existing owner retains overflow plus shared local recovery; behavioral coverage verifies capture and custody.

Line-Count-Exception: app/lib/services/capture/capture_controller.dart | 4437 -> 4488 | Existing owner retains overflow plus shared local recovery; behavioral coverage verifies capture and custody.

Line-Count-Exception: app/lib/services/wals/local_wal_sync.dart | 2663 -> 2714 | Existing owner retains overflow plus shared local recovery; behavioral coverage verifies capture and custody.

Line-Count-Exception: backend/routers/developer.py | 2752 -> 2753 | Existing owner retains overflow plus shared local recovery; behavioral coverage verifies capture and custody.

Synchronized current main for the synthetic-merge metadata gate; resolved adjacent capture initialization, flag additions and pipeline documentation by retaining both sides. This retires the previously expected pipeline-doc conflict.

Full-suite follow-up: nullable recovery state was missing from two healthy capture fixtures, producing the Dart CI failures. Both lanes now share the fixture update and real Home-card saved-audio/retry regression. Capture Home plus full visual-audit suite: 171 passed, 2 existing skips on each branch. Final make preflight: 65 selected checks passed. Simulator compile and Android emulator acceptance passed on the engine fix; no production-code change was needed for the fixture failure.

Final integration caution

New main localization additions currently conflict with this draft’s locale additions and suppress fresh pull-request workflows. No cutover merge was performed. Existing Mobile App Checks was dispatched on the exact published head ffbc5c6cd4ece17a67f01700a31acebdea51ba92: run 38596611353 passed, including full Dart tests, Android emulator acceptance and release eligibility. Native compile was scoped out for this test-only commit; the identical engine source passed simulator and Android compile/acceptance on its preceding head, plus both local dev debug builds.

Round-3 engine recovery corrections

• Historical SD-card/flash/ring imports use their own recording metadata, never the current live STT route. Recording-time phone-only custody is persisted separately. A targeted decode/import repair clears pre-fix dev-build stamps on historical device-storage recordings without clearing genuine phone-local capture consent. Behavioral tests import both fresh and already-stamped records while local, change route, exercise refused automatic upload and successful explicit retry.
• Polling overflow advances the source clock by discarded PCM/Opus duration without stretching speech segments. Real polling failure/overflow/recovery tests assert 10 → 11 → 13 seconds at 8 and 16 kHz.
• Explicit custom on-device STT readiness and permission preparation use the configured language, matching the actual provider (including its null-language default). Tests cover both supported/unsupported mismatch directions and the production permission action through its native channel.
• Shared custody field placement is aligned across READY/overflow to prevent duplicate fields during integration. Native engine code is unchanged in this round.

The overlapping different-locale reservation test nit is deferred: safely changing reservation ownership across asynchronous installs requires separate concurrency work. Existing reservation release and native fallback behavior remain unchanged. Phone battery and real-device iOS 15–25 execution remain unmeasured.

Latest round-3 read-only integration rehearsal: shared polling/readiness/language/codec-clock code is byte-identical and native AppDelegate merges cleanly. Expected Git conflicts: Apple provider, Whisper provider, local WAL sync, WAL model, and feature-flag registry. Preserve overflow batch engines/provenance and onDeviceConversation, retaining one shared custody flag, recording-time marker and their JSON entries. The former silently duplicated WAL field problem is gone; the model now requires an explicit conflict resolution. The lead owns integration; no PR was merged.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
omi	🟢 Ready	View Preview	Oct 14, 2026, 6:13 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

[Git-on-my-level]

Closing as fully contained in the ladder-v1 draft #21488: this branch's head 55ce061 is an ancestor of #21488's head 13e4314 (verified with git merge-base --is-ancestor). #21488's CI is green. The ready-PR subset (SpeechAnalyzer fallback plus the mobile-speech-analyzer-disabled kill switch) already shipped in #21474. Overflow UI and engines stay behind mobile-on-device-overflow (off) in #21488.

[github-actions]

Hey @Git-on-my-level 👋

Thank you so much for taking the time to contribute to Omi! We truly appreciate you putting in the effort to submit this pull request.

After careful review, we've decided not to merge this particular PR. Please don't take this personally — we genuinely try to merge as many contributions as possible, but sometimes we have to make tough calls based on:

• Project standards — Ensuring consistency across the codebase
• User needs — Making sure changes align with what our users need
• Code best practices — Maintaining code quality and maintainability
• Project direction — Keeping aligned with our product principles and locked invariants

Before your next PR, please skim:

• PRODUCT.md — product north star
• Product invariants — locked rules (shared chat, memory tiers, agent control plane, integrations, brand)

If this was declined for direction or taste, maintainers should cite an invariant ID or open a proposed one — ask if that citation is missing.

Your contribution is still valuable to us, and we'd love to see you contribute again in the future! If you'd like feedback on how to improve this PR or want to discuss alternative approaches, please don't hesitate to reach out.

Thank you for being part of the Omi community!