Repository navigation
Enforce the transcription cap where cloud STT is dispatched (flag off) - #21866
Conversation
Failure-Class: FC-stt-allowance-not-enforced-at-dispatch
Failure-Class: FC-stt-allowance-not-enforced-at-dispatch
Failure-Class: FC-stt-allowance-not-enforced-at-dispatch
Failure-Class: FC-stt-allowance-not-enforced-at-dispatch
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e26e11ce28
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if stt_cap_dispatch_enabled(): | ||
| if not sync_allowed(uid): | ||
| if deferred_outcome is not None: | ||
| deferred_outcome['cap_held'] = True |
There was a problem hiding this comment.
When allowance passes the pre-VAD check but becomes unavailable before this per-segment check, process_segment returns False without adding an error and only sets deferred_outcome['cap_held']. Neither coordinator consumes that flag: v1 counts the segment as successful and bills the full speech duration, while v2 reports success without held_audio and marks the content ledger completed even though its content is still pending recovery. Return or aggregate a distinct held outcome so both paths publish the held state and avoid settling it as completed content.
AGENTS.md reference: backend/AGENTS.md:L145-L145
Useful? React with 👍 / 👎.
| if cap_held: | ||
| await run_blocking(storage_executor, _cleanup_files, wav_paths) | ||
| wav_paths = [] |
There was a problem hiding this comment.
For an exhausted v2 upload that overlaps audio already committed by live capture or an earlier WAL, this clears the decoded paths before apply_sync_wal_audio_coverage and apply_safety_wal_coverage run. Recovery later performs only VAD plus process_segment, so the held copy bypasses the existing overlap and lineage protections and can create duplicate transcript/conversation content after the allowance resets. Preserve and replay the coverage evidence, or run the coverage stages before retaining the recoverable audio.
AGENTS.md reference: backend/AGENTS.md:L145-L145
Useful? React with 👍 / 👎.
| target_conversation_id=target_conversation_id, | ||
| client_device_id=client_device_id, | ||
| client_platform=client_platform, | ||
| ) |
There was a problem hiding this comment.
When a v2 sync request includes conversation geolocation and is held at the cap, the already-resolved geolocation value is not stored here. cap_hold_recovery._recover_row subsequently calls process_segment without geolocation, so every conversation created from that retained upload permanently loses the location that the ordinary v2 path would persist.
Useful? React with 👍 / 👎.
| async def resume(self): | ||
| if self.raw is None and not self.closed and self.admission.allowed(self.provider): | ||
| raw = await self.factory() | ||
| if raw is None: | ||
| raise RuntimeError('held provider constructor returned no socket') | ||
| self.raw = raw | ||
| for callback in self._open_callbacks: | ||
| callback(raw) |
There was a problem hiding this comment.
When a Modulate target is held during a mid-session cap transition and later resumed, this factory can return another CapSocket: connect_modulate is decorated itself, and its default-target branch calls the separately decorated streaming.process_audio_modulate. Unlike the initial decorator path, resume does not unwrap that result, so every subsequent send passes through LiveAdmission.send twice, double-counting provider metrics and consuming the restored allowance at twice the audio rate.
Useful? React with 👍 / 👎.
| def hold_segment(uid: str, path: str, metadata: dict[str, Any]) -> float: | ||
| audio = Path(path).read_bytes() |
There was a problem hiding this comment.
For a valid large compressed sync upload, the decoded WAV can be hundreds of megabytes or more, and read_bytes() loads it entirely before encrypt_audio_chunk creates another full-size ciphertext buffer for upload. An exhausted request can therefore exhaust a backend-sync instance's memory while trying to create the hold, repeatedly failing the same upload and potentially disrupting concurrent jobs; encrypt and upload bounded chunks instead of materializing the complete WAV and ciphertext in memory.
AGENTS.md reference: backend/AGENTS.md:L202-L205
Useful? React with 👍 / 👎.
| await run_blocking( | ||
| db_executor, | ||
| record_usage, | ||
| uid, | ||
| transcription_seconds=int(seconds), | ||
| speech_seconds=int(seconds), | ||
| idempotency_key=f'cap-hold-{snapshot.id}-{name}', | ||
| usage_metadata={'funding': 'omi', 'provider': 'unknown', 'source': 'sync_v2'}, | ||
| ) | ||
| await run_blocking( | ||
| db_executor, snapshot.reference.update, {'processed_paths': firestore.ArrayUnion([name])} |
There was a problem hiding this comment.
If usage settlement raises after process_segment has already called the paid STT provider and persisted the conversation, execution exits before processed_paths is updated. The next authenticated recovery pass therefore retranscribes the same audio and repeats post-processing; an ambiguous usage write is still idempotent, but the provider dispatch is not. Persist a transcript/content phase checkpoint before attempting usage settlement so settlement retries cannot issue another STT request.
AGENTS.md reference: backend/AGENTS.md:L145-L145
Useful? React with 👍 / 👎.
| ConversationSource(meta['source']), | ||
| False, | ||
| prefs, | ||
| private_cloud_sync_enabled=bool(private), | ||
| data_protection_level=protection, |
There was a problem hiding this comment.
Recovered segments call process_segment without the person_embeddings_cache that both normal sync paths build before transcription. For users with known voice profiles, held audio recovered after a reset or upgrade therefore skips voice-based person matching and persists generic speaker attribution even though the same upload would have identified speakers without the cap hold.
Useful? React with 👍 / 👎.
| if errors or outcome.get('cap_held'): | ||
| return False | ||
| if ok: | ||
| await run_blocking(sync_executor, finalize_recovered_sync_segment, uid, response, bool(private)) | ||
| seconds = await run_blocking(sync_executor, get_wav_duration, segment_path) |
There was a problem hiding this comment.
For a full held capture that VAD splits into multiple pieces, this runs _reprocess_merged_conversations and private-audio finalization inside the per-piece loop. When adjacent pieces merge into the same conversation, the backend repeatedly summarizes and finalizes progressively incomplete versions, unlike ordinary sync which waits for the complete batch; with eager reprocessing enabled this adds repeated LLM work and can expose transient partial enrichment. Accumulate the response across the hold and finalize once after every piece is persisted.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 89120f3d21
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| await run_blocking(db_executor, snapshot.reference.update, {'state': 'completed'}) | ||
| await run_blocking(storage_executor, _erase, uid, snapshot, row) |
There was a problem hiding this comment.
If processing, finalization, or usage settlement for the last segment exceeds the 30-minute lease, another authenticated recovery can claim the same hold with a new token. This unconditional update then lets the stale worker mark the successor's claim completed and _erase its manifest and audio, allowing duplicate provider/content work or disrupting the active recovery. Make the processed/completed transition transactional and conditional on the original lease token still being current.
Useful? React with 👍 / 👎.
| idempotency_key=f'cap-hold-{snapshot.id}-{name}', | ||
| usage_metadata={'funding': 'omi', 'provider': 'unknown', 'source': 'sync_v2'}, |
There was a problem hiding this comment.
For every successfully recovered hold, process_segment records the selected provider in outcome, but usage settlement discards it and hardcodes provider: unknown. The usage-unit projection cannot apply a provider rate to these calls, so recovered STT spend is systematically reported as missing/unknown even though the dispatch result identifies the provider; populate this metadata from the successful outcome.
Useful? React with 👍 / 👎.
| 'credits_exhausted': True, | ||
| 'held_audio': True, |
There was a problem hiding this comment.
When admission fails closed because allowance lookup is temporarily unavailable—for example, a subscription/usage read fails or the launch-date configuration is missing—hold_capture_if_exhausted returns the same boolean as a real quota exhaustion and this response unconditionally reports credits_exhausted: true. Clients can therefore present or act on a quota-exhaustion state during an infrastructure/configuration failure even though the existing precheck deliberately treats transient allowance reasons as non-exhaustion; preserve the denial reason and only set this field for actual exhausted-credit outcomes.
Useful? React with 👍 / 👎.
| cap_held = await run_blocking( | ||
| sync_executor, | ||
| hold_capture_if_exhausted, |
There was a problem hiding this comment.
Both sync coordinators invoke hold_capture_if_exhausted on the 16-worker sync_executor, but that helper performs the full synchronous GCS upload (plus local audio I/O and Firestore work) before returning. When several exhausted users upload large captures or GCS is slow, these holds can occupy every sync worker and starve unrelated sync pipelines; split the hold into async-coordinated DB/storage leaves and run the blob/audio work on storage_executor, which is the repository-designated pool for GCS and audio chunk I/O.
AGENTS.md reference: backend/AGENTS.md:L269-L276
Useful? React with 👍 / 👎.
# Conflicts: # backend/tests/unit/test_calendar_long_range_upcoming.py
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7c0f57fddf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| def sync_allowed(uid: str, seconds: float = 0, provider: Any = 'unknown') -> bool: | ||
| if not enabled(): | ||
| return True | ||
| allowance = funded_allowance(uid) |
There was a problem hiding this comment.
When enforcement is enabled, an exhausted user whose inline v1/v2 sync request carries a validated X-BYOK-Deepgram key is still denied here because funded_allowance forces byok_active=False and omi_funded=True. The downstream prerecorded path would select the user's Deepgram key, so this customer-funded dispatch should not consume Omi allowance; instead, the upload is held until the Omi allowance resets. Determine the actual provider/funding before applying the Omi cap, or preserve the BYOK exemption for Deepgram.
AGENTS.md reference: backend/AGENTS.md:L141-L141
Useful? React with 👍 / 👎.
| partition = { | ||
| key: metadata.get(key) for key in ('timestamp', 'source', 'client_device_id', 'target_conversation_id') | ||
| } |
There was a problem hiding this comment.
When the same WAV is held under two distinct v2 content ledgers—for example, the original batch [A, B] followed by a retry containing only [A]—this identity is unchanged because the partition omits content_id. hold_segment then returns the existing manifest while mark_content_held records its ID on the second ledger; recovery reads the first manifest's content_id, settles and deletes only that hold, and leaves the second ledger permanently held with no backing object. Include ledger ownership in the identity or transactionally attach every ledger owner to the hold.
AGENTS.md reference: backend/AGENTS.md:L145-L145
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c54f5a8e69
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| def defer_segment(uid: str, path: str, metadata: dict, response: dict, lock: Any) -> None: | ||
| """Report a durable late hold to both coordinators, never as silence.""" | ||
| receipt: dict = {} | ||
| hold_segment(uid, path, {**metadata, 'file_name': Path(path).name}, receipt=receipt) |
There was a problem hiding this comment.
When allowance becomes unavailable after the initial precheck, process_segment reaches this call while already running on sync_executor, and hold_segment synchronously hashes the WAV, performs Firestore reads/writes, and uploads the encrypted object. Several large late-held segments can therefore occupy all 16 sync workers and starve unrelated pipelines; route this late-hold path through the async coordinator and the designated DB/storage executors just like the initial hold path.
AGENTS.md reference: backend/AGENTS.md:L274-L276
Useful? React with 👍 / 👎.
| held.update(required - allowed) | ||
| batch = [w for w in batch if w.segment_id not in held] |
There was a problem hiding this comment.
When a long work item spans multiple ASR microbatches and authorization is revoked after an earlier microbatch succeeds, adding its segment ID to held makes the final loop return HeldWork and discard all words already accumulated for that item. The pump then defers the original raw item, so its next attempt retranscribes those successful windows from the beginning, duplicating GPU/provider work and accepted-audio accounting on every such transition; checkpoint the completed windows or resume only the undispatched remainder.
Useful? React with 👍 / 👎.
| allowance = await run_blocking(db_executor, funded_allowance, uid) | ||
| if allowance.managed: | ||
| return False |
There was a problem hiding this comment.
When a v1 held-upload acknowledgment is lost and the client retries after its allowance resets, this early return skips identity lookup even though the prior hold still exists. The authenticated retry also schedules hold recovery, so the recovery worker and the ordinary v1 pipeline can transcribe the same WAV independently and create duplicate conversation content; detect an existing v1 hold before allowing the retry to dispatch, or fence recovery and retry through a shared idempotency record.
Useful? React with 👍 / 👎.
| if receipt['has_content']: | ||
| seconds = int(receipt['seconds']) |
There was a problem hiding this comment.
When a held capture is split into multiple VAD pieces with fractional-second durations, converting every receipt to int separately permanently drops each fractional remainder; coverage-trimmed subsecond pieces are recorded as zero. A capture containing many short pieces can therefore consume materially less monthly allowance than the STT audio actually dispatched and reopen the cap early, so settle an aggregate duration or carry fractional units across the per-piece idempotent writes.
Useful? React with 👍 / 👎.
| if stt_cap_dispatch_enabled() and not subscription_launch_date_str: | ||
| CONFIG_ERRORS.labels(reason='missing').inc() | ||
| logger.error('event=stt_cap_launch_config outcome=invalid reason=missing') | ||
| raise ValueError('missing subscription launch date') |
There was a problem hiding this comment.
When enforcement is enabled with a missing or invalid SUBSCRIPTION_LAUNCH_DATE, this helper now raises, but /v1/users/me/subscription calls it directly while building the usage response, before reaching the exception-safe resolve_transcription_allowance path. The same configuration state that is supposed to conservatively deny funded STT therefore makes the subscription endpoint return a 500 for every ordinary user; return an untrusted-usage sentinel here or confine the exception to the allowance resolver.
Useful? React with 👍 / 👎.
# Conflicts: # backend/diarizer/batch_pipeline.py
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 914beb1905
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| response = _response(receipts) | ||
| if receipts: | ||
| # Ordinary sync finalizes the joined batch, not each VAD piece. | ||
| await run_blocking(sync_executor, finalize_recovered_sync_segment, uid, response, bool(private)) |
There was a problem hiding this comment.
When a private-cloud sync upload is held, its original raw_capture_paths have already been archived, but those paths are neither retained in the hold metadata nor passed to link_archive_captures after the recovered conversations are finalized. Consequently, deleting a recovered conversation cannot discover and delete its raw sync archive because delete_conversation_audio_files relies on the archive's conversation_ids metadata, leaving the original audio stored until account deletion. Persist the archive paths with the hold and link them to the recovered conversation IDs before completing recovery.
Useful? React with 👍 / 👎.
| row = snapshot.to_dict() or {} | ||
| try: | ||
| if row.get('state') == 'completed': | ||
| await run_blocking(storage_executor, _settle_and_erase, uid, snapshot, row) |
There was a problem hiding this comment.
In both completed-hold paths, _settle_and_erase is dispatched on storage_executor, but it performs sync_cap_holds.settle_hold, snapshot.reference.update, and snapshot.reference.delete, all synchronous Firestore operations. A recovery burst therefore occupies storage workers with database latency and defeats the repository's executor isolation; split the GCS deletions from the Firestore settlement and run each leaf on its designated pool.
AGENTS.md reference: backend/AGENTS.md:L269-L276
Useful? React with 👍 / 👎.
| response = _response(receipts) | ||
| if receipts: | ||
| # Ordinary sync finalizes the joined batch, not each VAD piece. | ||
| await run_blocking(sync_executor, finalize_recovered_sync_segment, uid, response, bool(private)) |
There was a problem hiding this comment.
When an audio_omibatchphone_ upload selected through Process Now is held, ordinary v2 processing passes user_selected=True to _reprocess_merged_conversations, which restores rule-discarded conversations and forces their enrichment. Recovery instead calls this helper with its default False, so an explicitly selected recording can remain discarded and unprocessed even though the content ledger is completed and the retained audio is erased. Retain the user-selected marker in hold metadata and pass it through finalization.
Useful? React with 👍 / 👎.
| if suppress_socket_audio(self.raw, data): | ||
| return True |
There was a problem hiding this comment.
When a managed live session runs with AUDIO_TIMELINE_V2 off, a mid-session cap pause returns here before _seconds and session.audio_seconds advance. If allowance becomes available at a later 60-second refresh, the provider resumes on a clock that omits the suppressed interval, so subsequent transcript timestamps are shifted earlier and can be merged into the pre-pause conversation instead of their actual capture time. Advance the legacy receive clocks for suppressed packets without recording provider-accepted spans or usage.
Useful? React with 👍 / 👎.
# Conflicts: # config/feature-flags.yaml
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a8b41192a2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| 'usage_metadata': { | ||
| 'funding': 'omi', | ||
| 'provider': outcome.get('provider') or 'unknown', | ||
| 'source': 'sync_v2', |
There was a problem hiding this comment.
When audio held by the v1 /v1/sync path is recovered, this hardcodes its usage provenance as sync_v2, even though ordinary v1 accounting uses sync_v1. Because hold_capture_if_exhausted serves both protocols, every recovered v1 segment is consequently attributed to the wrong source in the usage-unit projection; derive the source from the retained hold metadata or the presence of a v2 content ID.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5c5de21b1a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| def authorize_dispatch(segments): | ||
| selected = [v for v in leases if v.item.segment_id in segments] | ||
| receipt = self.authorize_compute(selected) if self.authorize_compute is not None else {} |
There was a problem hiding this comment.
When one work item spans multiple ASR microbatches and the user begins with any positive finite allowance, every call here re-reads the same persisted usage because accepted audio is not written to hourly_usage until projection after process_batch finishes. Consequently, the first accepted microbatch cannot reduce authorization for later microbatches, and a long upload can consume far beyond the remaining quota despite the advertised per-microbatch checks. Reserve or locally debit accepted audio before issuing the next authorization request.
AGENTS.md reference: backend/AGENTS.md:L151-L151
Useful? React with 👍 / 👎.
| result.update(failed_segments=0, errors=[], outcome='success') | ||
| updates.update(status='completed', expires_at=now + timedelta(days=LEDGER_RETENTION_DAYS)) |
There was a problem hiding this comment.
When a v2 hold is recovered while its original Redis job is still within the 24-hour reconciliation window, this transition updates only the Firestore content ledger. GET /v2/sync-jobs/{job_id} continues reading the unchanged terminal Redis job, so it keeps returning held_audio: true, zero/stale segment counts, and the pre-recovery memory lists even though recovery has completed. Update the matching Redis job after the final hold settles, or have status reads converge from the completed ledger.
AGENTS.md reference: backend/AGENTS.md:L151-L151
Useful? React with 👍 / 👎.
| source=ConversationSource(binding.source), | ||
| language=binding.language or payload.get('language'), | ||
| is_locked=binding.is_locked, | ||
| is_locked=False if cap_dispatch_enabled() else binding.is_locked, |
There was a problem hiding this comment.
When the rollout flag changes between durable batch computation and later projection, consulting the projector's current environment reclassifies already-produced results. In particular, a result computed with enforcement off and binding.is_locked=True becomes unlocked if projection runs after activation, exposing content that was transcribed under the old locked-content policy; a rollback can conversely lock work that was authorized while enforcement was on. Persist and consult the compute-admission receipt, such as the payload's cap_dispatch provenance, rather than the current flag.
AGENTS.md reference: backend/AGENTS.md:L151-L151
Useful? React with 👍 / 👎.
# Conflicts: # backend/docs/feature-flag-registry.md
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 99c9a95797
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| transcript_segments: List[TranscriptSegment] = [] | ||
| while True: | ||
| phase = 'provider_call' | ||
| if stt_cap_dispatch_enabled() and not sync_allowed(uid): |
There was a problem hiding this comment.
With enforcement enabled and any finite positive balance, every provider invocation in this upload re-reads the same persisted monthly usage: the five-way segment fan-out and all later chunks complete before record_usage runs near finalization. Thus even one remaining second can authorize every segment—and the empty-result retries—in a multi-hour upload, so locally reserve or debit accepted duration before the next invocation rather than relying solely on the unchanged persisted counter.
AGENTS.md reference: backend/AGENTS.md:L151-L151
Useful? React with 👍 / 👎.
| def finalize_recovered_sync_segment(uid: str, response: dict, private: bool) -> None: | ||
| """Use the same enrichment/audio finalization as ordinary sync intake.""" | ||
| _reprocess_merged_conversations(uid, response) | ||
| if private: | ||
| _finalize_sync_audio_files(uid, response) |
There was a problem hiding this comment.
When recovered segments create or update conversations with person-attributed speakers, this recovery finalizer stops after enrichment and optional audio finalization, whereas both ordinary sync coordinators then call schedule_person_voice_learning_retries. After the hold is settled and erased there is no later recovery hook, so a recovered conversation's pending voice-learning work may never run and its audio will not improve the person's voice profile; invoke the same bounded scheduler here after audio finalization.
Useful? React with 👍 / 👎.
| if errors or outcome.get('cap_held'): | ||
| return False |
There was a problem hiding this comment.
If allowance becomes unavailable after _new_dispatch_allowed succeeds but before process_segment reaches its own pre-provider check, cap_recovery=True makes that call return with cap_held without issuing STT. This branch nevertheless leaves the durable receipt in state='dispatched'; every later claim then exits as reconciliation_required, so audio that is known never to have reached a provider is permanently stranded after an ordinary allowance race. Persist a distinguishable not-dispatched state, or roll back this intent when the denial occurs before the first provider call.
AGENTS.md reference: backend/AGENTS.md:L151-L151
Useful? React with 👍 / 👎.
# Conflicts: # web/app/src/components/settings/__tests__/SettingsSubscriptionClarity.test.tsx
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d54c9297bd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -2993,15 +3145,23 @@ def _checkpoint_fenced_conversations(): | |||
| if content_id and failed_segments == 0: | |||
There was a problem hiding this comment.
When a v2 batch both late-holds one segment and records an error for a sibling, failed_segments > 0 bypasses this ledger write even though cap_holds is nonempty. The job is still finalized with held_audio, and the ledger claim is then released as retryable; cap_hold_recovery therefore sees the acknowledged hold as waiting and cannot process it unless the client happens to re-upload. Transfer hold ownership while preserving the sibling failure state so durable recovery can proceed independently.
AGENTS.md reference: backend/AGENTS.md:L145-L145
Useful? React with 👍 / 👎.
| if not admission.allowed(self.host.stt_service): | ||
| admission.omi_selected = True | ||
| self.cap_paused = True | ||
| return True # Receive/persist without consulting provider routing. |
There was a problem hiding this comment.
When a multi-channel session begins already denied, this early return leaves every stt_sockets_multi entry empty. _handle_multi_channel_audio then skips its send block because socket_available is false, so it never reaches suppress_socket_audio; unlike the single-channel _flush_stt_buffer, that path does not call record_cap_audio itself. Consequently all audio suppressed before the first successful refresh is absent from omi_stt_cap_provider_audio_seconds_total, understating cap impact for multi-channel sessions.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c47bc2f9bb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| def _new_dispatch_allowed(uid: str) -> bool: | ||
| if not sync_allowed(uid): | ||
| return False | ||
| restricted, _ = get_hard_restriction_status(uid) | ||
| return not restricted and not is_daily_audio_ceiling_exceeded(uid) |
There was a problem hiding this comment.
When a held capture originated from the backfill lane, both initial and repeated recovery dispatches pass only through this helper, which checks the subscription allowance and the fresh-audio daily ceiling but never calls reserve_backfill_speech. Unlike the ordinary v1/v2 backfill paths, recovered historical audio can therefore bypass the per-user/global daily quotas and backfill dispatch pacing; carry the retained lane and content identity into admission and reserve its VAD duration before invoking STT.
AGENTS.md reference: backend/AGENTS.md:L145-L145
Useful? React with 👍 / 👎.
| if receipt['has_content']: | ||
| seconds = int(receipt['seconds']) | ||
| await run_blocking( | ||
| db_executor, | ||
| record_usage, |
There was a problem hiding this comment.
For an initial pre-VAD hold, both sync coordinators return before their normal record_speech_ms stage, and recovery records only hourly subscription usage here. Consequently recovered fresh speech never enters the realtime + sync_fresh Redis totals used by soft caps and the daily ceiling, while recovered backfill speech is absent from its separate meter; persist whether a hold was already metered and idempotently record the recovered VAD duration under the retained lane.
AGENTS.md reference: backend/AGENTS.md:L145-L145
Useful? React with 👍 / 👎.
| if stt_cap_dispatch_enabled() and cap_recovery and cap_recovery_checkpoint is not None: | ||
| # A held piece gets at most two persisted application dispatch | ||
| # attempts. Reconciliation owns the second, not this local retry. | ||
| if deferred_outcome is not None: | ||
| deferred_outcome['recovery_ambiguous'] = True | ||
| return False |
There was a problem hiding this comment.
When VAD admits noise but prerecorded returns a valid empty result, this branch marks the known return as ambiguous instead of checkpointing silence. The second recovery dispatch follows the same branch, and because an empty result creates no ingest proof, the piece eventually becomes failed_reconciliation and a v2 content ledger remains held indefinitely; preserve the ordinary bounded-empty-retry semantics and checkpoint a confirmed empty result as completed with no content.
Useful? React with 👍 / 👎.
| @property | ||
| def funded_paused(self) -> bool: | ||
| return self.omi_selected and not (self.managed and (self.remaining is None or self.remaining > 0)) |
There was a problem hiding this comment.
If an Omi-funded Soniox or Modulate leg fails over to cloud Deepgram using the request's BYOK key, omi_selected remains permanently true from the earlier leg. Once the Omi allowance later reaches zero, this property reports the whole session as funded-paused even though the current Deepgram socket is still customer-funded and allowed; both STT death monitors and _recover_ingested_audio then skip that live socket, so an upstream close can strand recovery or leave the client listening until the receive timeout. Base the pause decision on the currently serving provider rather than whether any prior leg was Omi-funded.
Useful? React with 👍 / 👎.
This branch was successfully deployed
1 participant
The allowance was computed but Omi-funded STT dispatch did not enforce it. This adds
STT_CAP_DISPATCH_ENFORCEMENT, default false, to pause new funded provider work at exhaustion while preserving capture and retaining sync audio for recovery. Activation is David's product decision; this PR adds no runtime binding or flag flip.Failure-Class: new
The violated contract is the cap being computed but not enforced at dispatch.
FC-stt-allowance-not-enforced-at-dispatchrecords provider, queue, custody and recovery regression guards.UI-Evidence: none -- Client changes are regenerated optional response DTOs; no UI, interactions or rendering change.
Behavior behind the flag
held, without a TTL or completed-transcription proof. Duplicate intake acknowledges custody without another provider call; only all referenced holds settling can complete the ledger.locked_content_recoveryauthenticated-request path, independently of the old monthly unlock receipt/reset flag. Recovery replays WAL coverage/lineage and restores geolocation and speaker context. Before STT or ingestion, a token/lease-fenceddispatchedreceipt records funding, provider/model, dispatch time and attempt. Piece bytes and lineage freeze in encrypted owner-scoped artifacts before dispatch. An explicitprovider_calledsignal clears known non-calls under the same fence. Conversation ingestion atomically commits a proof bound to that hold/piece/lineage. After 30 minutes, ambiguous work uses the proof without STT, or gets one admission-checked second dispatch (attempt=2). A second unresolved attempt becomesfailed_reconciliationand retains audio. Recovery owns the retry, so checkpointed recovery does not perform the ordinary local empty-ASR retry. Provider-internal SDK retries are unchanged. Successful siblings settle/finalize independently with checkpointed progress; failed pieces never publish all-success ledger completion. Reconciled ingests retain raw audio because interrupted playback storage may be incomplete. Terminal retained rows cannot dispatch, and settlement failures remain retryable. Unacknowledged preparing/failed source jobs retain their retry ownership.None/HeldWorkonly on the enforced callback path and does not callfail(). The flag-off adapter keeps current main's[]result and ordinary failure behavior. Held work preserves raw artifacts and refunds the provider-failure attempt. Quarantine recovery rechecks before fetch and its sole provider attempt.What users see when the flag is on — David's activation decision
Existing over-cap paid users would receive held audio instead of transcripts that become locked. Live retention follows existing consent/storage paths; the sync hold is a processing checkpoint. Recovery is bounded and request-driven: WS-only/inactive accounts wait for an authenticated API request. Rollback pauses new hold recovery while retaining its audio. Enabling later must cover listen, sync/backfill, policy and GPU runtimes. No deployment, catalog/price, Stripe or ladder-v1 change is included.
Additive measurement
omi_stt_cap_provider_audio_seconds_total{path,outcome,provider,allowance}records accepted Omi-funded provider-adapter audio versus suppressed live input / held sync quantities.path=live|sync,outcome=accepted|suppressed|held, andallowance=exhausted|otherisolate genuine post-cap acceptance from available/untrusted observations. Provider labels are bounded. GPU results persistcap_dispatch.omi_funded_accepted_provider_audio_secondsand compute admission origin; the counter is also exported by the worker metrics registry.omi_stt_cap_launch_config_errors_total{reason=missing|invalid}distinguishes configuration errors.omi_stt_cap_recovery_reconciliation_total{outcome}countscleared_noncall,reconciled_ingested,redispatched, andfailed; structured logs carry hold id and piece name, with no ids in metric labels. No new provenance emits with the flag off. Held queue attempts can repeat; these are not unique-content counts or an exact vendor invoice ledger.Gate review map
backend/config/stt_cap_dispatch.py:6backend/utils/stt_cap_dispatch.py:79backend/utils/stt_cap_dispatch.py:207backend/routers/listen/receiver.py:1246backend/routers/listen/receiver.py:1861backend/routers/listen/receiver.py:2030backend/utils/stt/live_session.py:1054backend/routers/listen/receiver.py:2010backend/routers/listen/runtime.py:802backend/routers/sync.py:661;backend/routers/sync.py:839backend/utils/sync/pipeline.py:2434;backend/utils/sync/pipeline.py:1353backend/utils/batch_pipeline/admission.py:14backend/diarizer/batch_pipeline.py:366backend/diarizer/batch_pipeline_models.py:86backend/utils/batch_pipeline/quarantine_recovery.py:93backend/utils/sync/cap_hold.py:124;backend/database/sync_cap_holds.py:24backend/utils/sync/cap_hold_recovery.py:85backend/database/sync_cap_recovery.py:13backend/utils/sync/cap_recovery_plan.py:24backend/utils/sync/cap_recovery_audio.py:15backend/utils/stt_cap_dispatch.py:19backend/utils/subscription.py:1593backend/utils/observability/stt_cap_dispatch.py:7Validation
Backend tests run serially under
nice -n 10throughbackend/test.shonly, withBACKEND_PYTEST_WORKERS=1. The final delta run passes 1,276 cases in 50 suites, including the recovery state machine, real provider-call signals, atomic ingest hooks, immutable audio/lineage plans, metric/log outcomes, lifecycle/assignment contracts, both flag states, all earlier dispatch gates and root-meter parity. No timing budget, workflow, marker or skip hatch is weakened.Flag-off evidence: denied-provider matrices delegate to the original constructors/sends with no new allowance IO; the real FastAPI v1 route returns the original exact JSON bytes; the nullable v2 field is omitted when absent. Without held audio, legacy DG settlement receives its original duration argument without new arithmetic; the real partial-batch coordinator regression covers that path. Existing metering suites and canonical OpenAPI / generated Dart, TypeScript and Swift freshness checks pass. Client DTO regeneration adds only the optional held-result field. Current-main calendar fixtures pass unchanged. The parity fixture isolates its synthetic transcript from auxiliary language-model loading. No timing budget, marker, workflow or skip hatch is weakened.
Exact-head CI remains the merge authority. Local validation is synthetic API/provider/persistence-boundary evidence; no production/GCP/provider requests or client/device acceptance was performed. The dispatch/custody contract and limits are in
backend/docs/stt-cap-dispatch.md.Bounded reconciliation state machine (delta review)
The coordinator policy prefers retaining audio over avoiding one duplicate transcription. An ambiguous piece is not permanently sealed:
dispatched(attempt=1)waits thirty minutes, then becomestranscribedfrom the atomic ingest proof or reauthorizes a singledispatched(attempt=2). After a second ambiguous cooldown, proof still wins; absent proof, the piece becomesfailed_reconciliation, retaining raw audio and frozen pieces without an application TTL. Known non-calls clear/revert the intent without spending the additional dispatch. Completed siblings finalize while another piece waits or fails. Usage keeps the samecap-hold-{hold}-{piece}idempotency key.Regression guards cover signed-URL/keyterm/provider-select non-calls, pre-provider and late cap denial, atomic ingest proof with lost/expired/replaced leases, proof-based finalization without STT, exactly one successful or ambiguous redispatch, terminal audio retention, sibling progress, storage/plan failure, immutable piece bytes/lineage, retryable ledger settlement, and all four metric/log outcomes. Real segment-processor and ordinary lifecycle call-shape tests cover flag-off parity. Oversized-retry parity and the duration-parse guard remain intact.
The runtime parity sweep covers live constructors/sends/multichannel/replay/failover/death handling; sync v1/v2 response bytes, null omission and root-meter arguments; GPU retries/failure accounting; queue/quarantine; BYOK; launch configuration; storage/ledger rollback custody; and provider selection. The new proof hooks, immutable plans, provider-call signal and reconciliation emissions are reached only by enforced recovery. No ladder-v1 code, price, activation, deployment or workflow changes are added.
Follow-ups — #21488
No ladder-v1 paths change here. Findings #6 (budget-aware live refresh when
remaining_seconds=None, including remaining admission/reservation races) and #7 (usage-unit projection coverage gaps) remain forlane/sub-ladder-cutover/ #21488. These checks are not distributed quota reservations: already admitted concurrent work can drain across the cap.Line-Count-Exception: backend/routers/listen/receiver.py | 2564 -> 2618 | Integrate the default-off gate at the owning boundary; shared admission, custody and recovery live in small dedicated modules.
Line-Count-Exception: backend/routers/sync.py | 2795 -> 2849 | Integrate the default-off gate at the owning boundary; shared admission, custody and recovery live in small dedicated modules.
Line-Count-Exception: backend/utils/stt/streaming.py | 2401 -> 2405 | Integrate the default-off gate at the owning boundary; shared admission, custody and recovery live in small dedicated modules.
Line-Count-Exception: backend/utils/subscription.py | 1962 -> 1978 | Integrate the default-off gate at the owning boundary; shared admission, custody and recovery live in small dedicated modules.
Line-Count-Exception: backend/utils/sync/pipeline.py | 3213 -> 3385 | Integrate the default-off gate at the owning boundary; shared admission, custody and recovery live in small dedicated modules.
Line-Count-Exception: backend/diarizer/batch_pipeline.py | 484 -> 529 | Integrate the default-off gate at the owning boundary; shared admission, custody and recovery live in small dedicated modules.
Line-Count-Exception: backend/utils/stt/pre_recorded.py | 1285 -> 1293 | Integrate the default-off gate at the owning boundary; shared admission, custody and recovery live in small dedicated modules.
Line-Count-Exception: backend/utils/conversations/lifecycle.py | 980 -> 987 | Integrate the default-off gate at the owning boundary; shared admission, custody and recovery live in small dedicated modules.
🤖 Generated with Codex