diff --git a/docs/log/images/issue-26/ios-simulator.png b/docs/log/images/issue-26/ios-simulator.png new file mode 100644 index 0000000..2ad2547 Binary files /dev/null and b/docs/log/images/issue-26/ios-simulator.png differ diff --git a/docs/log/issue-26-ios-stt-adapter.md b/docs/log/issue-26-ios-stt-adapter.md new file mode 100644 index 0000000..cda4026 --- /dev/null +++ b/docs/log/issue-26-ios-stt-adapter.md @@ -0,0 +1,53 @@ +# Issue #26 iOS SpeechAnalyzer STT 어댑터 + +검증일: 2026-10-27 + +## 구현 범위 + +- `pigeons/stt_api.dart`를 단일 전송 계약으로 두고 Pigeon 29.0.11로 Dart/Swift 코드를 생성했다. +- 공개 Dart 계약(`SttService`, `SttAvailability`, `SttPreparation`, `SttEvent`)과 생성 타입을 `PigeonSttService`에서 분리했다. +- iOS 26+ `SpeechAnalyzer` + `SpeechTranscriber`의 온디바이스 `ko-KR` 지원과 `AssetInventory` 설치 상태를 확인하고, `prepare`에서 필요한 모델 설치를 수행한다. +- 마이크와 음성 인식 권한을 각각 확인한다. 권한이 없으면 오디오 세션을 시작하지 않는다. +- `AVAudioEngine` 입력 포맷과 `bestAvailableAudioFormat`이 다르면 `AVAudioConverter`로 변환한다. 변환기를 만들 수 없으면 `audioFormatMismatch`로 시작을 거부한다. +- volatile/final 결과와 ready/ended/failure/cancelled를 session ID, generation, 단조 증가 sequence와 함께 전달한다. +- 취소·dispose·백그라운드 전환 시 이벤트 발행을 차단하고 audio tap, 입력 stream, 결과 Task, analyzer, audio session을 정리한다. +- 시작 예약을 첫 비동기 작업 전에 잡고 각 비동기 경계 뒤에 취소 여부를 다시 확인해, 준비 중 cancel과 동시 start가 마이크를 고아 상태로 남기지 않게 한다. +- 전화·Siri 등 오디오 interruption과 media-services reset은 재시도 가능한 `interrupted` failure로 세션을 종료한다. +- 실행 중 오디오 변환 실패는 정상 `ended`가 아니라 재시도 불가능한 `audioFormatMismatch` failure로 전달한다. +- 종료된 generation, 중복 sequence, 빈 부분/최종 전사는 Dart 경계에서 폐기한다. +- 원시 오디오와 RMS는 Dart로 전달하지 않고, 오디오 파일을 만들거나 네트워크로 전송하지 않는다. + +Apple 문서상 `SpeechAnalyzer`의 transcriber는 음성을 Apple 서버로 전송하지 않는다. 모델 자산은 `AssetInventory`가 시스템 범위에서 관리한다. + +- +- +- +- + +## 자동 검증 + +| 항목 | 결과 | +|---|---| +| Dart 구조화 계약 | 권한 거부, 모델 미설치, 오디오 포맷 불일치 failure 매핑 테스트 통과 | +| Dart 시작 오류 | Pigeon `PlatformException`을 플랫폼 중립 `SttStartException`으로 변환하는 테스트 통과 | +| generation 방어 | cancel 뒤 도착한 이전 generation 결과와 빈 전사 폐기 테스트 통과 | +| iOS XCTest | iPhone 17 Pro / iOS 26.3.1 대상 `xcodebuild test` 통과 | +| iOS SDK 컴파일 | Xcode 26.2 / iOS Simulator SDK 26.2에서 Debug 빌드 통과 | +| 시뮬레이터 실행 | iPhone 17 Pro / iOS 26.3.1에서 설치·실행 및 기존 화면 렌더링 확인 | + +시뮬레이터 실행 화면: [ios-simulator.png](images/issue-26/ios-simulator.png) + +## 실기기 검증 상태와 절차 + +`xcdevice`에서 물리 iPhone 12 Pro(iOS 18.7.8)가 보였지만 연결 가능 상태가 아니며, 최소 지원 버전 iOS 26도 충족하지 않는다. 따라서 **실기기 오프라인 `ko-KR` 부분/최종 결과는 검증하지 않았고 성공으로 판정하지 않는다.** + +iOS 26+ 실기기가 준비되면 다음을 확인해야 한다. + +1. 앱을 새로 설치하고 마이크·음성 인식 권한의 허용/거부 상태가 각각 구조화 상태로 전달되는지 확인한다. +2. `ko-KR` 모델이 없는 상태에서 `modelDownloadRequired` 또는 설치 진행 상태가 표시되고, 설치 뒤 `ready`가 되는지 확인한다. +3. 네트워크를 끈 상태에서 직접 작성한 더미 문장을 읽고 non-final 결과가 `partial`, 확정 결과가 `finalResult`로 같은 generation에 도착하는지 확인한다. +4. 입력 포맷과 analyzer 포맷이 다른 기기에서 변환 후 결과가 나오며, 변환 불가 시 `audioFormatMismatch`로 종료되는지 확인한다. +5. 낭독 중 홈 화면으로 이동했을 때 마이크 표시가 사라지고 `cancelled` 뒤 늦은 결과가 전달되지 않는지 확인한다. +6. 취소 직후 generation을 증가시켜 재시작하고 이전 generation 결과가 새 소비자에게 전달되지 않는지 확인한다. + +실기기 검증 중 전사문이나 음성은 로그·파일·분석 도구에 남기지 않는다. diff --git a/ios/Runner.xcodeproj/project.pbxproj b/ios/Runner.xcodeproj/project.pbxproj index 89576b6..f476696 100644 --- a/ios/Runner.xcodeproj/project.pbxproj +++ b/ios/Runner.xcodeproj/project.pbxproj @@ -12,6 +12,9 @@ 3B3967161E833CAA004F5970 /* AppFrameworkInfo.plist in Resources */ = {isa = PBXBuildFile; fileRef = 3B3967151E833CAA004F5970 /* AppFrameworkInfo.plist */; }; 74858FAF1ED2DC5600515810 /* AppDelegate.swift in Sources */ = {isa = PBXBuildFile; fileRef = 74858FAE1ED2DC5600515810 /* AppDelegate.swift */; }; 7884E8682EC3CC0700C636F2 /* SceneDelegate.swift in Sources */ = {isa = PBXBuildFile; fileRef = 7884E8672EC3CC0400C636F2 /* SceneDelegate.swift */; }; + A26000010000000000000001 /* SttApi.g.swift in Sources */ = {isa = PBXBuildFile; fileRef = A26000020000000000000001 /* SttApi.g.swift */; }; + A26000030000000000000001 /* IosSpeechAdapter.swift in Sources */ = {isa = PBXBuildFile; fileRef = A26000040000000000000001 /* IosSpeechAdapter.swift */; }; + A26000050000000000000001 /* IosSpeechSession.swift in Sources */ = {isa = PBXBuildFile; fileRef = A26000060000000000000001 /* IosSpeechSession.swift */; }; 78A318202AECB46A00862997 /* FlutterGeneratedPluginSwiftPackage in Frameworks */ = {isa = PBXBuildFile; productRef = 78A3181F2AECB46A00862997 /* FlutterGeneratedPluginSwiftPackage */; }; 97C146FC1CF9000F007C117D /* Main.storyboard in Resources */ = {isa = PBXBuildFile; fileRef = 97C146FA1CF9000F007C117D /* Main.storyboard */; }; 97C146FE1CF9000F007C117D /* Assets.xcassets in Resources */ = {isa = PBXBuildFile; fileRef = 97C146FD1CF9000F007C117D /* Assets.xcassets */; }; @@ -50,6 +53,9 @@ 74858FAD1ED2DC5600515810 /* Runner-Bridging-Header.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = "Runner-Bridging-Header.h"; sourceTree = ""; }; 74858FAE1ED2DC5600515810 /* AppDelegate.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = AppDelegate.swift; sourceTree = ""; }; 7884E8672EC3CC0400C636F2 /* SceneDelegate.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SceneDelegate.swift; sourceTree = ""; }; + A26000020000000000000001 /* SttApi.g.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Generated/SttApi.g.swift; sourceTree = ""; }; + A26000040000000000000001 /* IosSpeechAdapter.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = IosSpeechAdapter.swift; sourceTree = ""; }; + A26000060000000000000001 /* IosSpeechSession.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = IosSpeechSession.swift; sourceTree = ""; }; 78E0A7A72DC9AD7400C4905E /* FlutterGeneratedPluginSwiftPackage */ = {isa = PBXFileReference; lastKnownFileType = wrapper; name = FlutterGeneratedPluginSwiftPackage; path = Flutter/ephemeral/Packages/FlutterGeneratedPluginSwiftPackage; sourceTree = ""; }; 7AFA3C8E1D35360C0083082E /* Release.xcconfig */ = {isa = PBXFileReference; lastKnownFileType = text.xcconfig; name = Release.xcconfig; path = Flutter/Release.xcconfig; sourceTree = ""; }; 9740EEB21CF90195004384FC /* Debug.xcconfig */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.xcconfig; name = Debug.xcconfig; path = Flutter/Debug.xcconfig; sourceTree = ""; }; @@ -123,6 +129,9 @@ 1498D2331E8E89220040F4C2 /* GeneratedPluginRegistrant.m */, 74858FAE1ED2DC5600515810 /* AppDelegate.swift */, 7884E8672EC3CC0400C636F2 /* SceneDelegate.swift */, + A26000020000000000000001 /* SttApi.g.swift */, + A26000040000000000000001 /* IosSpeechAdapter.swift */, + A26000060000000000000001 /* IosSpeechSession.swift */, 74858FAD1ED2DC5600515810 /* Runner-Bridging-Header.h */, ); path = Runner; @@ -282,6 +291,9 @@ buildActionMask = 2147483647; files = ( 74858FAF1ED2DC5600515810 /* AppDelegate.swift in Sources */, + A26000010000000000000001 /* SttApi.g.swift in Sources */, + A26000030000000000000001 /* IosSpeechAdapter.swift in Sources */, + A26000050000000000000001 /* IosSpeechSession.swift in Sources */, 1498D2341E8E89220040F4C2 /* GeneratedPluginRegistrant.m in Sources */, 7884E8682EC3CC0700C636F2 /* SceneDelegate.swift in Sources */, ); diff --git a/ios/Runner/AppDelegate.swift b/ios/Runner/AppDelegate.swift index c30b367..b22739b 100644 --- a/ios/Runner/AppDelegate.swift +++ b/ios/Runner/AppDelegate.swift @@ -3,6 +3,8 @@ import UIKit @main @objc class AppDelegate: FlutterAppDelegate, FlutterImplicitEngineDelegate { + private var speechAdapter: IosSpeechAdapter? + override func application( _ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]? @@ -12,5 +14,11 @@ import UIKit func didInitializeImplicitFlutterEngine(_ engineBridge: FlutterImplicitEngineBridge) { GeneratedPluginRegistrant.register(with: engineBridge.pluginRegistry) + guard let registrar = engineBridge.pluginRegistry.registrar(forPlugin: "IosSpeechAdapter") else { + return + } + let adapter = IosSpeechAdapter(binaryMessenger: registrar.messenger()) + NativeSttHostApiSetup.setUp(binaryMessenger: registrar.messenger(), api: adapter) + speechAdapter = adapter } } diff --git a/ios/Runner/Generated/SttApi.g.swift b/ios/Runner/Generated/SttApi.g.swift new file mode 100644 index 0000000..29b83cd --- /dev/null +++ b/ios/Runner/Generated/SttApi.g.swift @@ -0,0 +1,675 @@ +// Autogenerated from Pigeon (v29.0.11), do not edit directly. +// See also: https://pub.dev/packages/pigeon + +import Foundation + +#if os(iOS) + import Flutter +#elseif os(macOS) + import FlutterMacOS +#else + #error("Unsupported platform.") +#endif + +/// Error class for passing custom error details to Dart side. +final class PigeonError: Error { + let code: String + let message: String? + let details: Sendable? + + init(code: String, message: String?, details: Sendable?) { + self.code = code + self.message = message + self.details = details + } + + var localizedDescription: String { + return + "PigeonError(code: \(code), message: \(message ?? ""), details: \(details ?? ""))" + } +} + +private func wrapResult(_ result: Any?) -> [Any?] { + return [result] +} + +private func wrapError(_ error: Any) -> [Any?] { + if let pigeonError = error as? PigeonError { + return [ + pigeonError.code, + pigeonError.message, + pigeonError.details, + ] + } + if let flutterError = error as? FlutterError { + return [ + flutterError.code, + flutterError.message, + flutterError.details, + ] + } + return [ + "\(error)", + "\(Swift.type(of: error))", + "Stacktrace: \(Thread.callStackSymbols)", + ] +} + +private func createConnectionError(withChannelName channelName: String) -> PigeonError { + return PigeonError(code: "channel-error", message: "Unable to establish connection on channel: '\(channelName)'.", details: "") +} + +enum SttApiPigeonInternal { + static func isNullish(_ value: Any?) -> Bool { + guard let innerValue = value else { + return true + } + + if case Optional.some(Optional.none) = value { + return true + } + + return innerValue is NSNull + } + static func doubleEquals(_ lhs: Double, _ rhs: Double) -> Bool { + return (lhs.isNaN && rhs.isNaN) || lhs == rhs + } + + static func doubleHash(_ value: Double, _ hasher: inout Hasher) { + if value.isNaN { + hasher.combine(0x7FF8000000000000) + } else { + // Normalize -0.0 to 0.0 + hasher.combine(value == 0 ? 0 : value) + } + } + + static func deepEquals(_ lhs: Any?, _ rhs: Any?) -> Bool { + let cleanLhs = nilOrValue(lhs) as Any? + let cleanRhs = nilOrValue(rhs) as Any? + switch (cleanLhs, cleanRhs) { + case (nil, nil): + return true + + case (nil, _), (_, nil): + return false + + case (let lhs as AnyObject, let rhs as AnyObject) where lhs === rhs: + return true + + case is (Void, Void): + return true + + case (let lhsArray, let rhsArray) as ([Double], [Double]): + guard lhsArray.count == rhsArray.count else { return false } + for (index, element) in lhsArray.enumerated() { + if !doubleEquals(element, rhsArray[index]) { + return false + } + } + return true + + case (let lhsArray, let rhsArray) as ([Any?], [Any?]): + guard lhsArray.count == rhsArray.count else { return false } + for (index, element) in lhsArray.enumerated() { + if !deepEquals(element, rhsArray[index]) { + return false + } + } + return true + + case (let lhsDictionary, let rhsDictionary) as ([AnyHashable: Any?], [AnyHashable: Any?]): + guard lhsDictionary.count == rhsDictionary.count else { return false } + for (lhsKey, lhsValue) in lhsDictionary { + var found = false + for (rhsKey, rhsValue) in rhsDictionary { + if deepEquals(lhsKey, rhsKey) { + if deepEquals(lhsValue, rhsValue) { + found = true + break + } else { + return false + } + } + } + if !found { return false } + } + return true + + case (let lhs as Double, let rhs as Double): + return doubleEquals(lhs, rhs) + + case (let lhsHashable, let rhsHashable) as (AnyHashable, AnyHashable): + return lhsHashable == rhsHashable + + default: + return false + } + } + + static func deepHash(value: Any?, hasher: inout Hasher) { + let cleanValue = nilOrValue(value) as Any? + if let cleanValue = cleanValue { + if let doubleValue = cleanValue as? Double { + doubleHash(doubleValue, &hasher) + } else if let valueList = cleanValue as? [Any?] { + for item in valueList { + deepHash(value: item, hasher: &hasher) + } + } else if let valueList = cleanValue as? [Double] { + for item in valueList { + doubleHash(item, &hasher) + } + } else if let valueDict = cleanValue as? [AnyHashable: Any?] { + var result = 0 + for (key, value) in valueDict { + var entryKeyHasher = Hasher() + deepHash(value: key, hasher: &entryKeyHasher) + var entryValueHasher = Hasher() + deepHash(value: value, hasher: &entryValueHasher) + result = result &+ ((entryKeyHasher.finalize() &* 31) ^ entryValueHasher.finalize()) + } + hasher.combine(result) + } else if let hashableValue = cleanValue as? AnyHashable { + hasher.combine(hashableValue) + } else { + hasher.combine(String(describing: cleanValue)) + } + } else { + hasher.combine(0) + } + } + +} + +private func nilOrValue(_ value: Any?) -> T? { + if value is NSNull { return nil } + return value as! T? +} + + +enum NativeSttPermission: Int, CaseIterable { + case notDetermined = 0 + case granted = 1 + case denied = 2 + case restricted = 3 +} + +enum NativeSttModelStatus: Int, CaseIterable { + case installed = 0 + case supported = 1 + case downloading = 2 + case missing = 3 + case unsupported = 4 +} + +enum NativeSttPreparationStatus: Int, CaseIterable { + case ready = 0 + case permissionRequired = 1 + case permissionDenied = 2 + case modelDownloadRequired = 3 + case modelDownloading = 4 + case unsupported = 5 +} + +enum NativeSttEventKind: Int, CaseIterable { + case ready = 0 + case partial = 1 + case finalResult = 2 + case ended = 3 + case failure = 4 + case cancelled = 5 +} + +enum NativeSttFailureCode: Int, CaseIterable { + case permissionDenied = 0 + case onDeviceUnavailable = 1 + case localeUnsupported = 2 + case modelMissing = 3 + case busy = 4 + case noMatch = 5 + case audio = 6 + case audioFormatMismatch = 7 + case interrupted = 8 + case cancelled = 9 + case unknown = 10 +} + +/// Generated class from Pigeon that represents data sent in messages. +struct NativeSttAvailability: Hashable, CustomStringConvertible { + var microphonePermission: NativeSttPermission + var speechPermission: NativeSttPermission + var onDeviceAvailable: Bool + var localeSupported: Bool + var modelStatus: NativeSttModelStatus + + + // swift-format-ignore: AlwaysUseLowerCamelCase + static func fromList(_ pigeonVar_list: [Any?]) -> NativeSttAvailability? { + let microphonePermission = pigeonVar_list[0] as! NativeSttPermission + let speechPermission = pigeonVar_list[1] as! NativeSttPermission + let onDeviceAvailable = pigeonVar_list[2] as! Bool + let localeSupported = pigeonVar_list[3] as! Bool + let modelStatus = pigeonVar_list[4] as! NativeSttModelStatus + + return NativeSttAvailability( + microphonePermission: microphonePermission, + speechPermission: speechPermission, + onDeviceAvailable: onDeviceAvailable, + localeSupported: localeSupported, + modelStatus: modelStatus + ) + } + func toList() -> [Any?] { + return [ + microphonePermission, + speechPermission, + onDeviceAvailable, + localeSupported, + modelStatus, + ] + } + static func == (lhs: NativeSttAvailability, rhs: NativeSttAvailability) -> Bool { + if Swift.type(of: lhs) != Swift.type(of: rhs) { + return false + } + return SttApiPigeonInternal.deepEquals(lhs.microphonePermission, rhs.microphonePermission) && SttApiPigeonInternal.deepEquals(lhs.speechPermission, rhs.speechPermission) && SttApiPigeonInternal.deepEquals(lhs.onDeviceAvailable, rhs.onDeviceAvailable) && SttApiPigeonInternal.deepEquals(lhs.localeSupported, rhs.localeSupported) && SttApiPigeonInternal.deepEquals(lhs.modelStatus, rhs.modelStatus) + } + + func hash(into hasher: inout Hasher) { + hasher.combine("NativeSttAvailability") + SttApiPigeonInternal.deepHash(value: microphonePermission, hasher: &hasher) + SttApiPigeonInternal.deepHash(value: speechPermission, hasher: &hasher) + SttApiPigeonInternal.deepHash(value: onDeviceAvailable, hasher: &hasher) + SttApiPigeonInternal.deepHash(value: localeSupported, hasher: &hasher) + SttApiPigeonInternal.deepHash(value: modelStatus, hasher: &hasher) + } + + public var description: String { + return "NativeSttAvailability(microphonePermission: \(String(describing: microphonePermission)), speechPermission: \(String(describing: speechPermission)), onDeviceAvailable: \(String(describing: onDeviceAvailable)), localeSupported: \(String(describing: localeSupported)), modelStatus: \(String(describing: modelStatus)))" + } +} + +/// Generated class from Pigeon that represents data sent in messages. +struct NativeSttPreparation: Hashable, CustomStringConvertible { + var status: NativeSttPreparationStatus + var failureCode: NativeSttFailureCode? = nil + + + // swift-format-ignore: AlwaysUseLowerCamelCase + static func fromList(_ pigeonVar_list: [Any?]) -> NativeSttPreparation? { + let status = pigeonVar_list[0] as! NativeSttPreparationStatus + let failureCode: NativeSttFailureCode? = nilOrValue(pigeonVar_list[1]) + + return NativeSttPreparation( + status: status, + failureCode: failureCode + ) + } + func toList() -> [Any?] { + return [ + status, + failureCode, + ] + } + static func == (lhs: NativeSttPreparation, rhs: NativeSttPreparation) -> Bool { + if Swift.type(of: lhs) != Swift.type(of: rhs) { + return false + } + return SttApiPigeonInternal.deepEquals(lhs.status, rhs.status) && SttApiPigeonInternal.deepEquals(lhs.failureCode, rhs.failureCode) + } + + func hash(into hasher: inout Hasher) { + hasher.combine("NativeSttPreparation") + SttApiPigeonInternal.deepHash(value: status, hasher: &hasher) + SttApiPigeonInternal.deepHash(value: failureCode, hasher: &hasher) + } + + public var description: String { + return "NativeSttPreparation(status: \(String(describing: status)), failureCode: \(String(describing: failureCode)))" + } +} + +/// Generated class from Pigeon that represents data sent in messages. +struct NativeSttStartRequest: Hashable, CustomStringConvertible { + var sessionId: String + var generation: Int64 + var locale: String + + + // swift-format-ignore: AlwaysUseLowerCamelCase + static func fromList(_ pigeonVar_list: [Any?]) -> NativeSttStartRequest? { + let sessionId = pigeonVar_list[0] as! String + let generation = pigeonVar_list[1] as! Int64 + let locale = pigeonVar_list[2] as! String + + return NativeSttStartRequest( + sessionId: sessionId, + generation: generation, + locale: locale + ) + } + func toList() -> [Any?] { + return [ + sessionId, + generation, + locale, + ] + } + static func == (lhs: NativeSttStartRequest, rhs: NativeSttStartRequest) -> Bool { + if Swift.type(of: lhs) != Swift.type(of: rhs) { + return false + } + return SttApiPigeonInternal.deepEquals(lhs.sessionId, rhs.sessionId) && SttApiPigeonInternal.deepEquals(lhs.generation, rhs.generation) && SttApiPigeonInternal.deepEquals(lhs.locale, rhs.locale) + } + + func hash(into hasher: inout Hasher) { + hasher.combine("NativeSttStartRequest") + SttApiPigeonInternal.deepHash(value: sessionId, hasher: &hasher) + SttApiPigeonInternal.deepHash(value: generation, hasher: &hasher) + SttApiPigeonInternal.deepHash(value: locale, hasher: &hasher) + } + + public var description: String { + return "NativeSttStartRequest(sessionId: \(String(describing: sessionId)), generation: \(String(describing: generation)), locale: \(String(describing: locale)))" + } +} + +/// Generated class from Pigeon that represents data sent in messages. +struct NativeSttEvent: Hashable, CustomStringConvertible { + var sessionId: String + var generation: Int64 + var sequence: Int64 + var kind: NativeSttEventKind + var text: String? = nil + var failureCode: NativeSttFailureCode? = nil + var retryable: Bool? = nil + + + // swift-format-ignore: AlwaysUseLowerCamelCase + static func fromList(_ pigeonVar_list: [Any?]) -> NativeSttEvent? { + let sessionId = pigeonVar_list[0] as! String + let generation = pigeonVar_list[1] as! Int64 + let sequence = pigeonVar_list[2] as! Int64 + let kind = pigeonVar_list[3] as! NativeSttEventKind + let text: String? = nilOrValue(pigeonVar_list[4]) + let failureCode: NativeSttFailureCode? = nilOrValue(pigeonVar_list[5]) + let retryable: Bool? = nilOrValue(pigeonVar_list[6]) + + return NativeSttEvent( + sessionId: sessionId, + generation: generation, + sequence: sequence, + kind: kind, + text: text, + failureCode: failureCode, + retryable: retryable + ) + } + func toList() -> [Any?] { + return [ + sessionId, + generation, + sequence, + kind, + text, + failureCode, + retryable, + ] + } + static func == (lhs: NativeSttEvent, rhs: NativeSttEvent) -> Bool { + if Swift.type(of: lhs) != Swift.type(of: rhs) { + return false + } + return SttApiPigeonInternal.deepEquals(lhs.sessionId, rhs.sessionId) && SttApiPigeonInternal.deepEquals(lhs.generation, rhs.generation) && SttApiPigeonInternal.deepEquals(lhs.sequence, rhs.sequence) && SttApiPigeonInternal.deepEquals(lhs.kind, rhs.kind) && SttApiPigeonInternal.deepEquals(lhs.text, rhs.text) && SttApiPigeonInternal.deepEquals(lhs.failureCode, rhs.failureCode) && SttApiPigeonInternal.deepEquals(lhs.retryable, rhs.retryable) + } + + func hash(into hasher: inout Hasher) { + hasher.combine("NativeSttEvent") + SttApiPigeonInternal.deepHash(value: sessionId, hasher: &hasher) + SttApiPigeonInternal.deepHash(value: generation, hasher: &hasher) + SttApiPigeonInternal.deepHash(value: sequence, hasher: &hasher) + SttApiPigeonInternal.deepHash(value: kind, hasher: &hasher) + SttApiPigeonInternal.deepHash(value: text, hasher: &hasher) + SttApiPigeonInternal.deepHash(value: failureCode, hasher: &hasher) + SttApiPigeonInternal.deepHash(value: retryable, hasher: &hasher) + } + + public var description: String { + return "NativeSttEvent(sessionId: \(String(describing: sessionId)), generation: \(String(describing: generation)), sequence: \(String(describing: sequence)), kind: \(String(describing: kind)), text: \(String(describing: text)), failureCode: \(String(describing: failureCode)), retryable: \(String(describing: retryable)))" + } +} + +private class SttApiPigeonCodecReader: FlutterStandardReader { + override func readValue(ofType type: UInt8) -> Any? { + switch type { + case 129: + let enumResultAsInt: Int? = nilOrValue(self.readValue() as! Int?) + if let enumResultAsInt = enumResultAsInt { + return NativeSttPermission(rawValue: enumResultAsInt) + } + return nil + case 130: + let enumResultAsInt: Int? = nilOrValue(self.readValue() as! Int?) + if let enumResultAsInt = enumResultAsInt { + return NativeSttModelStatus(rawValue: enumResultAsInt) + } + return nil + case 131: + let enumResultAsInt: Int? = nilOrValue(self.readValue() as! Int?) + if let enumResultAsInt = enumResultAsInt { + return NativeSttPreparationStatus(rawValue: enumResultAsInt) + } + return nil + case 132: + let enumResultAsInt: Int? = nilOrValue(self.readValue() as! Int?) + if let enumResultAsInt = enumResultAsInt { + return NativeSttEventKind(rawValue: enumResultAsInt) + } + return nil + case 133: + let enumResultAsInt: Int? = nilOrValue(self.readValue() as! Int?) + if let enumResultAsInt = enumResultAsInt { + return NativeSttFailureCode(rawValue: enumResultAsInt) + } + return nil + case 134: + return NativeSttAvailability.fromList(self.readValue() as! [Any?]) + case 135: + return NativeSttPreparation.fromList(self.readValue() as! [Any?]) + case 136: + return NativeSttStartRequest.fromList(self.readValue() as! [Any?]) + case 137: + return NativeSttEvent.fromList(self.readValue() as! [Any?]) + default: + return super.readValue(ofType: type) + } + } +} + +private class SttApiPigeonCodecWriter: FlutterStandardWriter { + override func writeValue(_ value: Any) { + if let value = value as? NativeSttPermission { + super.writeByte(129) + super.writeValue(value.rawValue) + } else if let value = value as? NativeSttModelStatus { + super.writeByte(130) + super.writeValue(value.rawValue) + } else if let value = value as? NativeSttPreparationStatus { + super.writeByte(131) + super.writeValue(value.rawValue) + } else if let value = value as? NativeSttEventKind { + super.writeByte(132) + super.writeValue(value.rawValue) + } else if let value = value as? NativeSttFailureCode { + super.writeByte(133) + super.writeValue(value.rawValue) + } else if let value = value as? NativeSttAvailability { + super.writeByte(134) + super.writeValue(value.toList()) + } else if let value = value as? NativeSttPreparation { + super.writeByte(135) + super.writeValue(value.toList()) + } else if let value = value as? NativeSttStartRequest { + super.writeByte(136) + super.writeValue(value.toList()) + } else if let value = value as? NativeSttEvent { + super.writeByte(137) + super.writeValue(value.toList()) + } else { + super.writeValue(value) + } + } +} + +private class SttApiPigeonCodecReaderWriter: FlutterStandardReaderWriter { + override func reader(with data: Data) -> FlutterStandardReader { + return SttApiPigeonCodecReader(data: data) + } + + override func writer(with data: NSMutableData) -> FlutterStandardWriter { + return SttApiPigeonCodecWriter(data: data) + } +} + +class SttApiPigeonCodec: FlutterStandardMessageCodec, @unchecked Sendable { + static let shared = SttApiPigeonCodec(readerWriter: SttApiPigeonCodecReaderWriter()) +} + + +/// Generated protocol from Pigeon that represents a handler of messages from Flutter. +protocol NativeSttHostApi { + func checkAvailability(locale: String) async throws -> NativeSttAvailability + func prepare(locale: String) async throws -> NativeSttPreparation + func start(request: NativeSttStartRequest) async throws + func cancel(sessionId: String) async throws + func dispose() async throws +} + +/// Generated setup class from Pigeon to handle messages through the `binaryMessenger`. +class NativeSttHostApiSetup { + static var codec: FlutterStandardMessageCodec { SttApiPigeonCodec.shared } + /// Sets up an instance of `NativeSttHostApi` to handle messages through the `binaryMessenger`. + static func setUp(binaryMessenger: FlutterBinaryMessenger, api: NativeSttHostApi?, messageChannelSuffix: String = "") { + let channelSuffix = messageChannelSuffix.count > 0 ? ".\(messageChannelSuffix)" : "" + let checkAvailabilityChannel = FlutterBasicMessageChannel(name: "dev.flutter.pigeon.maltto.NativeSttHostApi.checkAvailability\(channelSuffix)", binaryMessenger: binaryMessenger, codec: codec) + if let api = api { + checkAvailabilityChannel.setMessageHandler { message, reply in + let args = message as! [Any?] + let localeArg = args[0] as! String + Task { @MainActor in + do { + let result = try await api.checkAvailability(locale: localeArg) + reply(wrapResult(result)) + } catch { + reply(wrapError(error)) + } + } + } + } else { + checkAvailabilityChannel.setMessageHandler(nil) + } + let prepareChannel = FlutterBasicMessageChannel(name: "dev.flutter.pigeon.maltto.NativeSttHostApi.prepare\(channelSuffix)", binaryMessenger: binaryMessenger, codec: codec) + if let api = api { + prepareChannel.setMessageHandler { message, reply in + let args = message as! [Any?] + let localeArg = args[0] as! String + Task { @MainActor in + do { + let result = try await api.prepare(locale: localeArg) + reply(wrapResult(result)) + } catch { + reply(wrapError(error)) + } + } + } + } else { + prepareChannel.setMessageHandler(nil) + } + let startChannel = FlutterBasicMessageChannel(name: "dev.flutter.pigeon.maltto.NativeSttHostApi.start\(channelSuffix)", binaryMessenger: binaryMessenger, codec: codec) + if let api = api { + startChannel.setMessageHandler { message, reply in + let args = message as! [Any?] + let requestArg = args[0] as! NativeSttStartRequest + Task { @MainActor in + do { + try await api.start(request: requestArg) + reply(wrapResult(nil)) + } catch { + reply(wrapError(error)) + } + } + } + } else { + startChannel.setMessageHandler(nil) + } + let cancelChannel = FlutterBasicMessageChannel(name: "dev.flutter.pigeon.maltto.NativeSttHostApi.cancel\(channelSuffix)", binaryMessenger: binaryMessenger, codec: codec) + if let api = api { + cancelChannel.setMessageHandler { message, reply in + let args = message as! [Any?] + let sessionIdArg = args[0] as! String + Task { @MainActor in + do { + try await api.cancel(sessionId: sessionIdArg) + reply(wrapResult(nil)) + } catch { + reply(wrapError(error)) + } + } + } + } else { + cancelChannel.setMessageHandler(nil) + } + let disposeChannel = FlutterBasicMessageChannel(name: "dev.flutter.pigeon.maltto.NativeSttHostApi.dispose\(channelSuffix)", binaryMessenger: binaryMessenger, codec: codec) + if let api = api { + disposeChannel.setMessageHandler { _, reply in + Task { @MainActor in + do { + try await api.dispose() + reply(wrapResult(nil)) + } catch { + reply(wrapError(error)) + } + } + } + } else { + disposeChannel.setMessageHandler(nil) + } + } +} + +/// Generated protocol from Pigeon that represents Flutter messages that can be called from Swift. +protocol NativeSttFlutterApiProtocol { + @MainActor func onEvent(event eventArg: NativeSttEvent) async throws +} +class NativeSttFlutterApi: NativeSttFlutterApiProtocol { + private let binaryMessenger: FlutterBinaryMessenger + private let messageChannelSuffix: String + init(binaryMessenger: FlutterBinaryMessenger, messageChannelSuffix: String = "") { + self.binaryMessenger = binaryMessenger + self.messageChannelSuffix = messageChannelSuffix.count > 0 ? ".\(messageChannelSuffix)" : "" + } + var codec: SttApiPigeonCodec { + return SttApiPigeonCodec.shared + } + @MainActor func onEvent(event eventArg: NativeSttEvent) async throws { + return try await withCheckedThrowingContinuation { continuation in + let channelName: String = "dev.flutter.pigeon.maltto.NativeSttFlutterApi.onEvent\(messageChannelSuffix)" + let channel = FlutterBasicMessageChannel(name: channelName, binaryMessenger: binaryMessenger, codec: codec) + channel.sendMessage([eventArg] as [Any?]) { response in + guard let listResponse = response as? [Any?] else { + continuation.resume(throwing: createConnectionError(withChannelName: channelName)) + return + } + if listResponse.count > 1 { + let code: String = listResponse[0] as! String + let message: String? = nilOrValue(listResponse[1]) + let details: String? = nilOrValue(listResponse[2]) + continuation.resume(throwing: PigeonError(code: code, message: message, details: details)) + } else { + continuation.resume() + } + } + } + } +} diff --git a/ios/Runner/Info.plist b/ios/Runner/Info.plist index f2a5205..30b3ddd 100644 --- a/ios/Runner/Info.plist +++ b/ios/Runner/Info.plist @@ -26,6 +26,10 @@ $(FLUTTER_BUILD_NUMBER) LSRequiresIPhoneOS + NSMicrophoneUsageDescription + 낭독 중 사용자의 목소리를 기기에서 인식하기 위해 마이크를 사용합니다. + NSSpeechRecognitionUsageDescription + 낭독 중 말한 내용을 기기에서 텍스트로 인식하기 위해 사용합니다. UIApplicationSceneManifest UIApplicationSupportsMultipleScenes diff --git a/ios/Runner/IosSpeechAdapter.swift b/ios/Runner/IosSpeechAdapter.swift new file mode 100644 index 0000000..e750efa --- /dev/null +++ b/ios/Runner/IosSpeechAdapter.swift @@ -0,0 +1,302 @@ +import AVFAudio +import Flutter +import Foundation +import Speech +import UIKit + +@MainActor +final class IosSpeechAdapter: NSObject, NativeSttHostApi { + private final class StartReservation { + let sessionId: String + let generation: Int64 + + init(sessionId: String, generation: Int64) { + self.sessionId = sessionId + self.generation = generation + } + } + + private let flutterApi: NativeSttFlutterApi + private var activeSession: IosSpeechSession? + private var pendingStart: StartReservation? + private var observers: [NSObjectProtocol] = [] + + init(binaryMessenger: FlutterBinaryMessenger) { + flutterApi = NativeSttFlutterApi(binaryMessenger: binaryMessenger) + super.init() + observers.append(NotificationCenter.default.addObserver( + forName: UIApplication.didEnterBackgroundNotification, + object: nil, + queue: .main + ) { [weak self] _ in + Task { @MainActor in + await self?.handleBackgroundTransition() + } + }) + observers.append(NotificationCenter.default.addObserver( + forName: AVAudioSession.interruptionNotification, + object: nil, + queue: .main + ) { [weak self] notification in + guard + let rawValue = notification.userInfo?[AVAudioSessionInterruptionTypeKey] as? UInt, + AVAudioSession.InterruptionType(rawValue: rawValue) == .began + else { return } + Task { @MainActor in + await self?.interruptActiveSession() + } + }) + observers.append(NotificationCenter.default.addObserver( + forName: AVAudioSession.mediaServicesWereResetNotification, + object: nil, + queue: .main + ) { [weak self] _ in + Task { @MainActor in + await self?.interruptActiveSession() + } + }) + } + + deinit { + observers.forEach(NotificationCenter.default.removeObserver) + } + + func checkAvailability(locale: String) async throws -> NativeSttAvailability { + let locale = Locale(identifier: locale) + let microphonePermission = Self.microphonePermission() + let speechPermission = Self.speechPermission() + let supportedLocale = await SpeechTranscriber.supportedLocale(equivalentTo: locale) + let isSupported = SpeechTranscriber.isAvailable && supportedLocale != nil + let modelStatus: NativeSttModelStatus + if let supportedLocale, SpeechTranscriber.isAvailable { + let transcriber = Self.makeTranscriber(locale: supportedLocale) + modelStatus = Self.modelStatus(await AssetInventory.status(forModules: [transcriber])) + } else { + modelStatus = .unsupported + } + return NativeSttAvailability( + microphonePermission: microphonePermission, + speechPermission: speechPermission, + onDeviceAvailable: SpeechTranscriber.isAvailable, + localeSupported: isSupported, + modelStatus: modelStatus + ) + } + + func prepare(locale: String) async throws -> NativeSttPreparation { + let microphone = await requestMicrophonePermission() + guard microphone == .granted else { + return NativeSttPreparation( + status: microphone == .notDetermined ? .permissionRequired : .permissionDenied, + failureCode: .permissionDenied + ) + } + let speech = await requestSpeechPermission() + guard speech == .granted else { + return NativeSttPreparation( + status: speech == .notDetermined ? .permissionRequired : .permissionDenied, + failureCode: .permissionDenied + ) + } + guard SpeechTranscriber.isAvailable else { + return NativeSttPreparation(status: .unsupported, failureCode: .onDeviceUnavailable) + } + let requestedLocale = Locale(identifier: locale) + guard let supportedLocale = await SpeechTranscriber.supportedLocale(equivalentTo: requestedLocale) + else { + return NativeSttPreparation(status: .unsupported, failureCode: .localeUnsupported) + } + + let transcriber = Self.makeTranscriber(locale: supportedLocale) + switch await AssetInventory.status(forModules: [transcriber]) { + case .installed: + return NativeSttPreparation(status: .ready, failureCode: nil) + case .unsupported: + return NativeSttPreparation(status: .unsupported, failureCode: .localeUnsupported) + case .downloading: + return NativeSttPreparation(status: .modelDownloading, failureCode: nil) + case .supported: + guard let request = try await AssetInventory.assetInstallationRequest( + supporting: [transcriber] + ) else { + return NativeSttPreparation(status: .modelDownloadRequired, failureCode: .modelMissing) + } + try await request.downloadAndInstall() + let installed = await AssetInventory.status(forModules: [transcriber]) == .installed + return NativeSttPreparation( + status: installed ? .ready : .modelDownloadRequired, + failureCode: installed ? nil : .modelMissing + ) + @unknown default: + return NativeSttPreparation(status: .unsupported, failureCode: .unknown) + } + } + + func start(request: NativeSttStartRequest) async throws { + guard activeSession == nil, pendingStart == nil else { + throw PigeonError(code: "busy", message: "An STT session is already active.", details: nil) + } + let reservation = StartReservation( + sessionId: request.sessionId, + generation: request.generation + ) + pendingStart = reservation + defer { + if pendingStart === reservation { pendingStart = nil } + } + guard Self.microphonePermission() == .granted, Self.speechPermission() == .granted else { + throw PigeonError(code: "permissionDenied", message: "STT permission is not granted.", details: nil) + } + guard SpeechTranscriber.isAvailable else { + throw PigeonError(code: "onDeviceUnavailable", message: "On-device STT is unavailable.", details: nil) + } + let requestedLocale = Locale(identifier: request.locale) + guard let locale = await SpeechTranscriber.supportedLocale(equivalentTo: requestedLocale) else { + try ensurePending(reservation) + throw PigeonError(code: "localeUnsupported", message: "The locale is unsupported.", details: nil) + } + try ensurePending(reservation) + let transcriber = Self.makeTranscriber(locale: locale) + guard await AssetInventory.status(forModules: [transcriber]) == .installed else { + try ensurePending(reservation) + throw PigeonError(code: "modelMissing", message: "The on-device model is not installed.", details: nil) + } + try ensurePending(reservation) + + let session = IosSpeechSession( + sessionId: request.sessionId, + generation: request.generation, + transcriber: transcriber + ) { [weak self] event in + guard let self, self.activeSession?.matches(event: event) == true else { return } + try? await self.flutterApi.onEvent(event: event) + } onFinished: { [weak self] session in + guard let self, self.activeSession === session else { return } + self.activeSession = nil + } + activeSession = session + pendingStart = nil + do { + try await session.start() + } catch { + await session.cancel(emitEvent: false) + if activeSession === session { activeSession = nil } + throw Self.pigeonError(from: error) + } + } + + func cancel(sessionId: String) async throws { + if pendingStart?.sessionId == sessionId { pendingStart = nil } + guard activeSession?.sessionId == sessionId else { return } + await cancelActiveSession(emitCancelled: true) + } + + func dispose() async throws { + pendingStart = nil + await cancelActiveSession(emitCancelled: false) + } + + private func cancelActiveSession(emitCancelled: Bool) async { + guard let session = activeSession else { return } + await session.cancel(emitEvent: emitCancelled) + if activeSession === session { activeSession = nil } + } + + private func interruptActiveSession() async { + pendingStart = nil + guard let session = activeSession else { return } + await session.fail(code: .interrupted, retryable: true) + if activeSession === session { activeSession = nil } + } + + private func handleBackgroundTransition() async { + pendingStart = nil + await cancelActiveSession(emitCancelled: true) + } + + private func ensurePending(_ reservation: StartReservation) throws { + guard pendingStart === reservation else { throw CancellationError() } + } + + private static func makeTranscriber(locale: Locale) -> SpeechTranscriber { + SpeechTranscriber( + locale: locale, + transcriptionOptions: [], + reportingOptions: [.volatileResults, .fastResults], + attributeOptions: [] + ) + } + + private static func microphonePermission() -> NativeSttPermission { + switch AVAudioApplication.shared.recordPermission { + case .undetermined: .notDetermined + case .granted: .granted + case .denied: .denied + @unknown default: .restricted + } + } + + private static func speechPermission() -> NativeSttPermission { + switch SFSpeechRecognizer.authorizationStatus() { + case .notDetermined: .notDetermined + case .authorized: .granted + case .denied: .denied + case .restricted: .restricted + @unknown default: .restricted + } + } + + private func requestMicrophonePermission() async -> NativeSttPermission { + if Self.microphonePermission() != .notDetermined { return Self.microphonePermission() } + _ = await AVAudioApplication.requestRecordPermission() + return Self.microphonePermission() + } + + private func requestSpeechPermission() async -> NativeSttPermission { + if Self.speechPermission() != .notDetermined { return Self.speechPermission() } + await withCheckedContinuation { continuation in + SFSpeechRecognizer.requestAuthorization { _ in continuation.resume() } + } + return Self.speechPermission() + } + + private static func modelStatus(_ status: AssetInventory.Status) -> NativeSttModelStatus { + switch status { + case .installed: .installed + case .supported: .supported + case .downloading: .downloading + case .unsupported: .unsupported + @unknown default: .missing + } + } + + private static func pigeonError(from error: Error) -> PigeonError { + if let error = error as? PigeonError { return error } + if let error = error as? IosSpeechError { + return PigeonError(code: error.code, message: error.message, details: nil) + } + if error is CancellationError { + return PigeonError(code: "cancelled", message: "The STT start was cancelled.", details: nil) + } + return PigeonError(code: "unknown", message: "The STT session failed.", details: nil) + } +} + +enum IosSpeechError: Error { + case audioFormatMismatch + case audio + + var code: String { + switch self { + case .audioFormatMismatch: "audioFormatMismatch" + case .audio: "audio" + } + } + + var message: String { + switch self { + case .audioFormatMismatch: "The microphone format cannot be converted for SpeechAnalyzer." + case .audio: "The audio input could not be started." + } + } +} diff --git a/ios/Runner/IosSpeechSession.swift b/ios/Runner/IosSpeechSession.swift new file mode 100644 index 0000000..0308c4b --- /dev/null +++ b/ios/Runner/IosSpeechSession.swift @@ -0,0 +1,280 @@ +import AVFAudio +import Foundation +import Speech +import Synchronization + +@MainActor +final class IosSpeechSession { + typealias EventHandler = @MainActor (NativeSttEvent) async -> Void + typealias FinishedHandler = @MainActor (IosSpeechSession) -> Void + + let sessionId: String + let generation: Int64 + + private let transcriber: SpeechTranscriber + private let eventHandler: EventHandler + private let finishedHandler: FinishedHandler + private let engine = AVAudioEngine() + private var analyzer: SpeechAnalyzer? + private var inputContinuation: AsyncStream.Continuation? + private var resultsTask: Task? + private var sequence: Int64 = 0 + private var acceptsEvents = true + private var tapInstalled = false + private var cleanupTask: Task? + private let conversionFailed = ThreadSafeFlag() + + init( + sessionId: String, + generation: Int64, + transcriber: SpeechTranscriber, + eventHandler: @escaping EventHandler, + onFinished: @escaping FinishedHandler + ) { + self.sessionId = sessionId + self.generation = generation + self.transcriber = transcriber + self.eventHandler = eventHandler + finishedHandler = onFinished + } + + func matches(event: NativeSttEvent) -> Bool { + acceptsEvents && event.sessionId == sessionId && event.generation == generation + } + + func start() async throws { + let audioSession = AVAudioSession.sharedInstance() + try audioSession.setCategory(.playAndRecord, mode: .spokenAudio, options: [.defaultToSpeaker]) + try audioSession.setActive(true) + + let analyzer = SpeechAnalyzer(modules: [transcriber]) + self.analyzer = analyzer + let inputFormat = engine.inputNode.outputFormat(forBus: 0) + guard inputFormat.sampleRate > 0, inputFormat.channelCount > 0 else { + throw IosSpeechError.audio + } + guard let targetFormat = await SpeechAnalyzer.bestAvailableAudioFormat( + compatibleWith: [transcriber], + considering: inputFormat + ) else { + throw IosSpeechError.audioFormatMismatch + } + try ensureActive() + try await analyzer.prepareToAnalyze(in: targetFormat) + try ensureActive() + + let (inputSequence, continuation) = AsyncStream.makeStream(of: AnalyzerInput.self) + try installTap(inputFormat: inputFormat, targetFormat: targetFormat, continuation: continuation) + inputContinuation = continuation + resultsTask = consumeResults() + + try await analyzer.start(inputSequence: inputSequence) + try ensureActive() + engine.prepare() + do { + try engine.start() + } catch { + throw IosSpeechError.audio + } + await emit(kind: .ready) + } + + func cancel(emitEvent: Bool) async { + if acceptsEvents { + acceptsEvents = false + if emitEvent { await emitUnchecked(kind: .cancelled) } + } + await cleanUp(finalize: false) + } + + func fail(code: NativeSttFailureCode, retryable: Bool) async { + if acceptsEvents { + acceptsEvents = false + await emitUnchecked(kind: .failure, failureCode: code, retryable: retryable) + } + await cleanUp(finalize: false) + finishedHandler(self) + } + + private func consumeResults() -> Task { + Task { [weak self, transcriber] in + do { + for try await result in transcriber.results { + guard let self, self.acceptsEvents else { return } + let text = String(result.text.characters).trimmingCharacters(in: .whitespacesAndNewlines) + guard !text.isEmpty else { continue } + await self.emit(kind: result.isFinal ? .finalResult : .partial, text: text) + } + guard let self, self.acceptsEvents else { return } + self.acceptsEvents = false + if self.conversionFailed.value { + await self.emitUnchecked( + kind: .failure, + failureCode: .audioFormatMismatch, + retryable: false + ) + } else { + await self.emitUnchecked(kind: .ended) + } + await self.cleanUp(finalize: false) + self.finishedHandler(self) + } catch is CancellationError { + return + } catch { + guard let self, self.acceptsEvents else { return } + self.acceptsEvents = false + await self.emitUnchecked(kind: .failure, failureCode: .unknown, retryable: true) + await self.cleanUp(finalize: false) + self.finishedHandler(self) + } + } + } + + private func installTap( + inputFormat: AVAudioFormat, + targetFormat: AVAudioFormat, + continuation: AsyncStream.Continuation + ) throws { + let converter: AVAudioConverter? + if inputFormat == targetFormat { + converter = nil + } else { + guard let made = AVAudioConverter(from: inputFormat, to: targetFormat) else { + throw IosSpeechError.audioFormatMismatch + } + converter = made + } + let converterBox = AudioConverterBox(converter) + let conversionFailed = self.conversionFailed + engine.inputNode.installTap(onBus: 0, bufferSize: 4096, format: inputFormat) { buffer, _ in + if let converter = converterBox.converter { + guard let converted = Self.convert(buffer, with: converter, to: targetFormat) else { + conversionFailed.set() + continuation.finish() + return + } + continuation.yield(AnalyzerInput(buffer: converted)) + } else { + continuation.yield(AnalyzerInput(buffer: buffer)) + } + } + tapInstalled = true + } + + private func emit( + kind: NativeSttEventKind, + text: String? = nil, + failureCode: NativeSttFailureCode? = nil, + retryable: Bool? = nil + ) async { + guard acceptsEvents else { return } + await emitUnchecked( + kind: kind, + text: text, + failureCode: failureCode, + retryable: retryable + ) + } + + private func emitUnchecked( + kind: NativeSttEventKind, + text: String? = nil, + failureCode: NativeSttFailureCode? = nil, + retryable: Bool? = nil + ) async { + sequence += 1 + await eventHandler( + NativeSttEvent( + sessionId: sessionId, + generation: generation, + sequence: sequence, + kind: kind, + text: text, + failureCode: failureCode, + retryable: retryable + ) + ) + } + + private func cleanUp(finalize: Bool) async { + if let cleanupTask { + await cleanupTask.value + return + } + let task = Task { @MainActor [weak self] in + guard let self else { return } + await self.performCleanUp(finalize: finalize) + } + cleanupTask = task + await task.value + } + + private func performCleanUp(finalize: Bool) async { + engine.stop() + if tapInstalled { + engine.inputNode.removeTap(onBus: 0) + tapInstalled = false + } + inputContinuation?.finish() + inputContinuation = nil + let task = resultsTask + resultsTask = nil + task?.cancel() + if let analyzer { + if finalize { + try? await analyzer.finalizeAndFinishThroughEndOfInput() + } else { + await analyzer.cancelAndFinishNow() + } + } + self.analyzer = nil + try? AVAudioSession.sharedInstance().setActive(false, options: .notifyOthersOnDeactivation) + } + + private func ensureActive() throws { + guard acceptsEvents else { throw CancellationError() } + } + + private nonisolated static func convert( + _ buffer: AVAudioPCMBuffer, + with converter: AVAudioConverter, + to format: AVAudioFormat + ) -> AVAudioPCMBuffer? { + let ratio = format.sampleRate / buffer.format.sampleRate + let capacity = AVAudioFrameCount((Double(buffer.frameLength) * ratio).rounded(.up)) + 1 + guard let output = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: capacity) else { return nil } + let source = AudioBufferBox(buffer) + let consumed = Mutex(false) + var conversionError: NSError? + let status = converter.convert(to: output, error: &conversionError) { _, outputStatus in + let wasConsumed = consumed.withLock { value in + defer { value = true } + return value + } + if wasConsumed { + outputStatus.pointee = .noDataNow + return nil + } + outputStatus.pointee = .haveData + return source.buffer + } + guard status != .error, conversionError == nil else { return nil } + return output + } +} + +private struct AudioBufferBox: @unchecked Sendable { + let buffer: AVAudioPCMBuffer + init(_ buffer: AVAudioPCMBuffer) { self.buffer = buffer } +} + +private struct AudioConverterBox: @unchecked Sendable { + let converter: AVAudioConverter? + init(_ converter: AVAudioConverter?) { self.converter = converter } +} + +private final class ThreadSafeFlag: Sendable { + private let storage = Mutex(false) + var value: Bool { storage.withLock { $0 } } + func set() { storage.withLock { $0 = true } } +} diff --git a/lib/src/infrastructure/stt/generated/stt_api.g.dart b/lib/src/infrastructure/stt/generated/stt_api.g.dart new file mode 100644 index 0000000..160ced6 --- /dev/null +++ b/lib/src/infrastructure/stt/generated/stt_api.g.dart @@ -0,0 +1,636 @@ +// Autogenerated from Pigeon (v29.0.11), do not edit directly. +// See also: https://pub.dev/packages/pigeon +// ignore_for_file: unused_import, unused_shown_name +// ignore_for_file: type=lint + +import 'dart:async'; +import 'dart:typed_data' show Float64List, Int32List, Int64List; + +import 'package:flutter/services.dart'; +import 'package:meta/meta.dart' show immutable, protected, visibleForTesting; + +Object? _extractReplyValueOrThrow( + List? replyList, + String channelName, { + required bool isNullValid, +}) { + if (replyList == null) { + throw PlatformException( + code: 'channel-error', + message: 'Unable to establish connection on channel: "$channelName".', + ); + } else if (replyList.length > 1) { + throw PlatformException( + code: replyList[0]! as String, + message: replyList[1] as String?, + details: replyList[2], + ); + } else if (!isNullValid && (replyList.isNotEmpty && replyList[0] == null)) { + throw PlatformException( + code: 'null-error', + message: 'Host platform returned null value for non-null return value.', + ); + } + return replyList.firstOrNull; +} + +List wrapResponse({ + Object? result, + PlatformException? error, + bool empty = false, +}) { + if (empty) { + return []; + } + if (error == null) { + return [result]; + } + return [error.code, error.message, error.details]; +} + +bool _deepEquals(Object? a, Object? b) { + if (identical(a, b)) { + return true; + } + if (a is double && b is double) { + if (a.isNaN && b.isNaN) { + return true; + } + return a == b; + } + if (a is List && b is List) { + return a.length == b.length && + a.indexed.every( + ((int, dynamic) item) => _deepEquals(item.$2, b[item.$1]), + ); + } + if (a is Map && b is Map) { + if (a.length != b.length) { + return false; + } + for (final MapEntry entryA in a.entries) { + bool found = false; + for (final MapEntry entryB in b.entries) { + if (_deepEquals(entryA.key, entryB.key)) { + if (_deepEquals(entryA.value, entryB.value)) { + found = true; + break; + } else { + return false; + } + } + } + if (!found) { + return false; + } + } + return true; + } + return a == b; +} + +int _deepHash(Object? value) { + if (value is List) { + return Object.hashAll(value.map(_deepHash)); + } + if (value is Map) { + int result = 0; + for (final MapEntry entry in value.entries) { + result += (_deepHash(entry.key) * 31) ^ _deepHash(entry.value); + } + return result; + } + if (value is double && value.isNaN) { + // Normalize NaN to a consistent hash. + return 0x7FF8000000000000.hashCode; + } + if (value is double && value == 0.0) { + // Normalize -0.0 to 0.0 so they have the same hash code. + return 0.0.hashCode; + } + return value.hashCode; +} + +enum NativeSttPermission { notDetermined, granted, denied, restricted } + +enum NativeSttModelStatus { + installed, + supported, + downloading, + missing, + unsupported, +} + +enum NativeSttPreparationStatus { + ready, + permissionRequired, + permissionDenied, + modelDownloadRequired, + modelDownloading, + unsupported, +} + +enum NativeSttEventKind { + ready, + partial, + finalResult, + ended, + failure, + cancelled, +} + +enum NativeSttFailureCode { + permissionDenied, + onDeviceUnavailable, + localeUnsupported, + modelMissing, + busy, + noMatch, + audio, + audioFormatMismatch, + interrupted, + cancelled, + unknown, +} + +class NativeSttAvailability { + NativeSttAvailability({ + required this.microphonePermission, + required this.speechPermission, + required this.onDeviceAvailable, + required this.localeSupported, + required this.modelStatus, + }); + + NativeSttPermission microphonePermission; + + NativeSttPermission speechPermission; + + bool onDeviceAvailable; + + bool localeSupported; + + NativeSttModelStatus modelStatus; + + List _toList() { + return [ + microphonePermission, + speechPermission, + onDeviceAvailable, + localeSupported, + modelStatus, + ]; + } + + Object encode() { + return _toList(); + } + + static NativeSttAvailability decode(Object result) { + result as List; + return NativeSttAvailability( + microphonePermission: result[0]! as NativeSttPermission, + speechPermission: result[1]! as NativeSttPermission, + onDeviceAvailable: result[2]! as bool, + localeSupported: result[3]! as bool, + modelStatus: result[4]! as NativeSttModelStatus, + ); + } + + @override + // ignore: avoid_equals_and_hash_code_on_mutable_classes + bool operator ==(Object other) { + if (other is! NativeSttAvailability || other.runtimeType != runtimeType) { + return false; + } + if (identical(this, other)) { + return true; + } + return _deepEquals(microphonePermission, other.microphonePermission) && + _deepEquals(speechPermission, other.speechPermission) && + _deepEquals(onDeviceAvailable, other.onDeviceAvailable) && + _deepEquals(localeSupported, other.localeSupported) && + _deepEquals(modelStatus, other.modelStatus); + } + + @override + // ignore: avoid_equals_and_hash_code_on_mutable_classes + int get hashCode => _deepHash([runtimeType, ..._toList()]); + + @override + String toString() { + return 'NativeSttAvailability(microphonePermission: $microphonePermission, speechPermission: $speechPermission, onDeviceAvailable: $onDeviceAvailable, localeSupported: $localeSupported, modelStatus: $modelStatus)'; + } +} + +class NativeSttPreparation { + NativeSttPreparation({required this.status, this.failureCode}); + + NativeSttPreparationStatus status; + + NativeSttFailureCode? failureCode; + + List _toList() { + return [status, failureCode]; + } + + Object encode() { + return _toList(); + } + + static NativeSttPreparation decode(Object result) { + result as List; + return NativeSttPreparation( + status: result[0]! as NativeSttPreparationStatus, + failureCode: result[1] as NativeSttFailureCode?, + ); + } + + @override + // ignore: avoid_equals_and_hash_code_on_mutable_classes + bool operator ==(Object other) { + if (other is! NativeSttPreparation || other.runtimeType != runtimeType) { + return false; + } + if (identical(this, other)) { + return true; + } + return _deepEquals(status, other.status) && + _deepEquals(failureCode, other.failureCode); + } + + @override + // ignore: avoid_equals_and_hash_code_on_mutable_classes + int get hashCode => _deepHash([runtimeType, ..._toList()]); + + @override + String toString() { + return 'NativeSttPreparation(status: $status, failureCode: $failureCode)'; + } +} + +class NativeSttStartRequest { + NativeSttStartRequest({ + required this.sessionId, + required this.generation, + required this.locale, + }); + + String sessionId; + + int generation; + + String locale; + + List _toList() { + return [sessionId, generation, locale]; + } + + Object encode() { + return _toList(); + } + + static NativeSttStartRequest decode(Object result) { + result as List; + return NativeSttStartRequest( + sessionId: result[0]! as String, + generation: result[1]! as int, + locale: result[2]! as String, + ); + } + + @override + // ignore: avoid_equals_and_hash_code_on_mutable_classes + bool operator ==(Object other) { + if (other is! NativeSttStartRequest || other.runtimeType != runtimeType) { + return false; + } + if (identical(this, other)) { + return true; + } + return _deepEquals(sessionId, other.sessionId) && + _deepEquals(generation, other.generation) && + _deepEquals(locale, other.locale); + } + + @override + // ignore: avoid_equals_and_hash_code_on_mutable_classes + int get hashCode => _deepHash([runtimeType, ..._toList()]); + + @override + String toString() { + return 'NativeSttStartRequest(sessionId: $sessionId, generation: $generation, locale: $locale)'; + } +} + +class NativeSttEvent { + NativeSttEvent({ + required this.sessionId, + required this.generation, + required this.sequence, + required this.kind, + this.text, + this.failureCode, + this.retryable, + }); + + String sessionId; + + int generation; + + int sequence; + + NativeSttEventKind kind; + + String? text; + + NativeSttFailureCode? failureCode; + + bool? retryable; + + List _toList() { + return [ + sessionId, + generation, + sequence, + kind, + text, + failureCode, + retryable, + ]; + } + + Object encode() { + return _toList(); + } + + static NativeSttEvent decode(Object result) { + result as List; + return NativeSttEvent( + sessionId: result[0]! as String, + generation: result[1]! as int, + sequence: result[2]! as int, + kind: result[3]! as NativeSttEventKind, + text: result[4] as String?, + failureCode: result[5] as NativeSttFailureCode?, + retryable: result[6] as bool?, + ); + } + + @override + // ignore: avoid_equals_and_hash_code_on_mutable_classes + bool operator ==(Object other) { + if (other is! NativeSttEvent || other.runtimeType != runtimeType) { + return false; + } + if (identical(this, other)) { + return true; + } + return _deepEquals(sessionId, other.sessionId) && + _deepEquals(generation, other.generation) && + _deepEquals(sequence, other.sequence) && + _deepEquals(kind, other.kind) && + _deepEquals(text, other.text) && + _deepEquals(failureCode, other.failureCode) && + _deepEquals(retryable, other.retryable); + } + + @override + // ignore: avoid_equals_and_hash_code_on_mutable_classes + int get hashCode => _deepHash([runtimeType, ..._toList()]); + + @override + String toString() { + return 'NativeSttEvent(sessionId: $sessionId, generation: $generation, sequence: $sequence, kind: $kind, text: $text, failureCode: $failureCode, retryable: $retryable)'; + } +} + +class _PigeonCodec extends StandardMessageCodec { + const _PigeonCodec(); + @override + void writeValue(WriteBuffer buffer, Object? value) { + if (value is int) { + buffer.putUint8(4); + buffer.putInt64(value); + } else if (value is NativeSttPermission) { + buffer.putUint8(129); + writeValue(buffer, value.index); + } else if (value is NativeSttModelStatus) { + buffer.putUint8(130); + writeValue(buffer, value.index); + } else if (value is NativeSttPreparationStatus) { + buffer.putUint8(131); + writeValue(buffer, value.index); + } else if (value is NativeSttEventKind) { + buffer.putUint8(132); + writeValue(buffer, value.index); + } else if (value is NativeSttFailureCode) { + buffer.putUint8(133); + writeValue(buffer, value.index); + } else if (value is NativeSttAvailability) { + buffer.putUint8(134); + writeValue(buffer, value.encode()); + } else if (value is NativeSttPreparation) { + buffer.putUint8(135); + writeValue(buffer, value.encode()); + } else if (value is NativeSttStartRequest) { + buffer.putUint8(136); + writeValue(buffer, value.encode()); + } else if (value is NativeSttEvent) { + buffer.putUint8(137); + writeValue(buffer, value.encode()); + } else { + super.writeValue(buffer, value); + } + } + + @override + Object? readValueOfType(int type, ReadBuffer buffer) { + switch (type) { + case 129: + final value = readValue(buffer) as int?; + return value == null ? null : NativeSttPermission.values[value]; + case 130: + final value = readValue(buffer) as int?; + return value == null ? null : NativeSttModelStatus.values[value]; + case 131: + final value = readValue(buffer) as int?; + return value == null ? null : NativeSttPreparationStatus.values[value]; + case 132: + final value = readValue(buffer) as int?; + return value == null ? null : NativeSttEventKind.values[value]; + case 133: + final value = readValue(buffer) as int?; + return value == null ? null : NativeSttFailureCode.values[value]; + case 134: + return NativeSttAvailability.decode(readValue(buffer)!); + case 135: + return NativeSttPreparation.decode(readValue(buffer)!); + case 136: + return NativeSttStartRequest.decode(readValue(buffer)!); + case 137: + return NativeSttEvent.decode(readValue(buffer)!); + default: + return super.readValueOfType(type, buffer); + } + } +} + +class NativeSttHostApi { + /// Constructor for [NativeSttHostApi]. The [binaryMessenger] named argument is + /// available for dependency injection. If it is left null, the default + /// BinaryMessenger will be used which routes to the host platform. + NativeSttHostApi({ + BinaryMessenger? binaryMessenger, + String messageChannelSuffix = '', + }) : pigeonVar_binaryMessenger = binaryMessenger, + pigeonVar_messageChannelSuffix = messageChannelSuffix.isNotEmpty + ? '.$messageChannelSuffix' + : ''; + + final BinaryMessenger? pigeonVar_binaryMessenger; + static const MessageCodec pigeonChannelCodec = _PigeonCodec(); + + final String pigeonVar_messageChannelSuffix; + + Future checkAvailability(String locale) async { + final pigeonVar_channelName = + 'dev.flutter.pigeon.maltto.NativeSttHostApi.checkAvailability$pigeonVar_messageChannelSuffix'; + final pigeonVar_channel = BasicMessageChannel( + pigeonVar_channelName, + pigeonChannelCodec, + binaryMessenger: pigeonVar_binaryMessenger, + ); + final Future pigeonVar_sendFuture = pigeonVar_channel.send( + [locale], + ); + final pigeonVar_replyList = await pigeonVar_sendFuture as List?; + + final Object? pigeonVar_replyValue = _extractReplyValueOrThrow( + pigeonVar_replyList, + pigeonVar_channelName, + isNullValid: false, + ); + return pigeonVar_replyValue! as NativeSttAvailability; + } + + Future prepare(String locale) async { + final pigeonVar_channelName = + 'dev.flutter.pigeon.maltto.NativeSttHostApi.prepare$pigeonVar_messageChannelSuffix'; + final pigeonVar_channel = BasicMessageChannel( + pigeonVar_channelName, + pigeonChannelCodec, + binaryMessenger: pigeonVar_binaryMessenger, + ); + final Future pigeonVar_sendFuture = pigeonVar_channel.send( + [locale], + ); + final pigeonVar_replyList = await pigeonVar_sendFuture as List?; + + final Object? pigeonVar_replyValue = _extractReplyValueOrThrow( + pigeonVar_replyList, + pigeonVar_channelName, + isNullValid: false, + ); + return pigeonVar_replyValue! as NativeSttPreparation; + } + + Future start(NativeSttStartRequest request) async { + final pigeonVar_channelName = + 'dev.flutter.pigeon.maltto.NativeSttHostApi.start$pigeonVar_messageChannelSuffix'; + final pigeonVar_channel = BasicMessageChannel( + pigeonVar_channelName, + pigeonChannelCodec, + binaryMessenger: pigeonVar_binaryMessenger, + ); + final Future pigeonVar_sendFuture = pigeonVar_channel.send( + [request], + ); + final pigeonVar_replyList = await pigeonVar_sendFuture as List?; + + _extractReplyValueOrThrow( + pigeonVar_replyList, + pigeonVar_channelName, + isNullValid: true, + ); + } + + Future cancel(String sessionId) async { + final pigeonVar_channelName = + 'dev.flutter.pigeon.maltto.NativeSttHostApi.cancel$pigeonVar_messageChannelSuffix'; + final pigeonVar_channel = BasicMessageChannel( + pigeonVar_channelName, + pigeonChannelCodec, + binaryMessenger: pigeonVar_binaryMessenger, + ); + final Future pigeonVar_sendFuture = pigeonVar_channel.send( + [sessionId], + ); + final pigeonVar_replyList = await pigeonVar_sendFuture as List?; + + _extractReplyValueOrThrow( + pigeonVar_replyList, + pigeonVar_channelName, + isNullValid: true, + ); + } + + Future dispose() async { + final pigeonVar_channelName = + 'dev.flutter.pigeon.maltto.NativeSttHostApi.dispose$pigeonVar_messageChannelSuffix'; + final pigeonVar_channel = BasicMessageChannel( + pigeonVar_channelName, + pigeonChannelCodec, + binaryMessenger: pigeonVar_binaryMessenger, + ); + final Future pigeonVar_sendFuture = pigeonVar_channel.send(null); + final pigeonVar_replyList = await pigeonVar_sendFuture as List?; + + _extractReplyValueOrThrow( + pigeonVar_replyList, + pigeonVar_channelName, + isNullValid: true, + ); + } +} + +abstract class NativeSttFlutterApi { + static const MessageCodec pigeonChannelCodec = _PigeonCodec(); + + void onEvent(NativeSttEvent event); + + static void setUp( + NativeSttFlutterApi? api, { + BinaryMessenger? binaryMessenger, + String messageChannelSuffix = '', + }) { + messageChannelSuffix = messageChannelSuffix.isNotEmpty + ? '.$messageChannelSuffix' + : ''; + { + final pigeonVar_channel = BasicMessageChannel( + 'dev.flutter.pigeon.maltto.NativeSttFlutterApi.onEvent$messageChannelSuffix', + pigeonChannelCodec, + binaryMessenger: binaryMessenger, + ); + if (api == null) { + pigeonVar_channel.setMessageHandler(null); + } else { + pigeonVar_channel.setMessageHandler((Object? message) async { + final List args = message! as List; + final NativeSttEvent arg_event = args[0]! as NativeSttEvent; + try { + api.onEvent(arg_event); + return wrapResponse(empty: true); + } on PlatformException catch (e) { + return wrapResponse(error: e); + } catch (e) { + return wrapResponse( + error: PlatformException(code: 'error', message: e.toString()), + ); + } + }); + } + } + } +} diff --git a/lib/src/infrastructure/stt/pigeon_stt_service.dart b/lib/src/infrastructure/stt/pigeon_stt_service.dart new file mode 100644 index 0000000..e7e9b01 --- /dev/null +++ b/lib/src/infrastructure/stt/pigeon_stt_service.dart @@ -0,0 +1,156 @@ +import 'dart:async'; + +import 'package:flutter/services.dart'; + +import 'generated/stt_api.g.dart' as pigeon; +import 'stt_models.dart'; +import 'stt_service.dart'; + +final class PigeonSttService implements SttService, pigeon.NativeSttFlutterApi { + PigeonSttService({pigeon.NativeSttHostApi? host}) + : _host = host ?? pigeon.NativeSttHostApi() { + pigeon.NativeSttFlutterApi.setUp(this); + } + + final pigeon.NativeSttHostApi _host; + final Map> _controllers = {}; + final Map _activeGenerations = {}; + final Map _lastSequences = {}; + bool _disposed = false; + + @override + Future checkAvailability({required String locale}) async { + _ensureActive(); + return _availabilityFromNative(await _host.checkAvailability(locale)); + } + + @override + Future prepare({required String locale}) async { + _ensureActive(); + final native = await _host.prepare(locale); + return SttPreparation( + status: SttPreparationStatus.values[native.status.index], + failureCode: native.failureCode == null + ? null + : SttFailureCode.values[native.failureCode!.index], + ); + } + + @override + Stream events(String sessionId) { + _ensureActive(); + return _controllers + .putIfAbsent(sessionId, StreamController.broadcast) + .stream; + } + + @override + Future start(SttStartRequest request) async { + _ensureActive(); + if (request.sessionId.isEmpty || request.locale.isEmpty) { + throw ArgumentError('sessionId와 locale은 비어 있을 수 없습니다.'); + } + final previous = _activeGenerations[request.sessionId]; + if (previous != null) { + if (request.generation <= previous) { + throw StateError('generation은 이전 값보다 커야 합니다.'); + } + throw const SttStartException(SttFailureCode.busy); + } + + _activeGenerations[request.sessionId] = request.generation; + _lastSequences[request.sessionId] = -1; + try { + await _host.start( + pigeon.NativeSttStartRequest( + sessionId: request.sessionId, + generation: request.generation, + locale: request.locale, + ), + ); + } on PlatformException catch (error) { + if (_activeGenerations[request.sessionId] == request.generation) { + _activeGenerations.remove(request.sessionId); + _lastSequences.remove(request.sessionId); + } + final code = + SttFailureCode.values.asNameMap()[error.code] ?? + SttFailureCode.unknown; + throw SttStartException(code); + } catch (_) { + if (_activeGenerations[request.sessionId] == request.generation) { + _activeGenerations.remove(request.sessionId); + _lastSequences.remove(request.sessionId); + } + rethrow; + } + } + + @override + Future cancel(String sessionId) async { + if (_disposed) return; + final generation = _activeGenerations[sessionId]; + await _host.cancel(sessionId); + if (_activeGenerations[sessionId] == generation) { + _activeGenerations.remove(sessionId); + _lastSequences.remove(sessionId); + } + } + + @override + void onEvent(pigeon.NativeSttEvent event) { + if (_disposed || _activeGenerations[event.sessionId] != event.generation) { + return; + } + final lastSequence = _lastSequences[event.sessionId] ?? -1; + if (event.sequence <= lastSequence) return; + final text = event.text?.trim(); + if ((event.kind == pigeon.NativeSttEventKind.partial || + event.kind == pigeon.NativeSttEventKind.finalResult) && + (text == null || text.isEmpty)) { + return; + } + _lastSequences[event.sessionId] = event.sequence; + _controllers[event.sessionId]?.add( + SttEvent( + sessionId: event.sessionId, + generation: event.generation, + sequence: event.sequence, + kind: SttEventKind.values[event.kind.index], + text: text, + failureCode: event.failureCode == null + ? null + : SttFailureCode.values[event.failureCode!.index], + retryable: event.retryable, + ), + ); + } + + @override + Future dispose() async { + if (_disposed) return; + _disposed = true; + _activeGenerations.clear(); + _lastSequences.clear(); + pigeon.NativeSttFlutterApi.setUp(null); + await _host.dispose(); + final controllers = _controllers.values.toList(growable: false); + _controllers.clear(); + await Future.wait(controllers.map((controller) => controller.close())); + } + + SttAvailability _availabilityFromNative( + pigeon.NativeSttAvailability native, + ) => SttAvailability( + microphonePermission: + SttPermission.values[native.microphonePermission.index], + speechPermission: SttPermission.values[native.speechPermission.index], + onDeviceAvailable: native.onDeviceAvailable, + localeSupported: native.localeSupported, + modelStatus: SttModelStatus.values[native.modelStatus.index], + ); + + void _ensureActive() { + if (_disposed) throw StateError('SttService가 이미 dispose되었습니다.'); + } +} diff --git a/lib/src/infrastructure/stt/stt_models.dart b/lib/src/infrastructure/stt/stt_models.dart new file mode 100644 index 0000000..2919811 --- /dev/null +++ b/lib/src/infrastructure/stt/stt_models.dart @@ -0,0 +1,92 @@ +enum SttPermission { notDetermined, granted, denied, restricted } + +enum SttModelStatus { installed, supported, downloading, missing, unsupported } + +enum SttPreparationStatus { + ready, + permissionRequired, + permissionDenied, + modelDownloadRequired, + modelDownloading, + unsupported, +} + +enum SttEventKind { ready, partial, finalResult, ended, failure, cancelled } + +enum SttFailureCode { + permissionDenied, + onDeviceUnavailable, + localeUnsupported, + modelMissing, + busy, + noMatch, + audio, + audioFormatMismatch, + interrupted, + cancelled, + unknown, +} + +final class SttStartException implements Exception { + const SttStartException(this.code); + + final SttFailureCode code; + + @override + String toString() => 'SttStartException(${code.name})'; +} + +final class SttAvailability { + const SttAvailability({ + required this.microphonePermission, + required this.speechPermission, + required this.onDeviceAvailable, + required this.localeSupported, + required this.modelStatus, + }); + + final SttPermission microphonePermission; + final SttPermission speechPermission; + final bool onDeviceAvailable; + final bool localeSupported; + final SttModelStatus modelStatus; +} + +final class SttPreparation { + const SttPreparation({required this.status, this.failureCode}); + + final SttPreparationStatus status; + final SttFailureCode? failureCode; +} + +final class SttStartRequest { + const SttStartRequest({ + required this.sessionId, + required this.generation, + required this.locale, + }); + + final String sessionId; + final int generation; + final String locale; +} + +final class SttEvent { + const SttEvent({ + required this.sessionId, + required this.generation, + required this.sequence, + required this.kind, + this.text, + this.failureCode, + this.retryable, + }); + + final String sessionId; + final int generation; + final int sequence; + final SttEventKind kind; + final String? text; + final SttFailureCode? failureCode; + final bool? retryable; +} diff --git a/lib/src/infrastructure/stt/stt_service.dart b/lib/src/infrastructure/stt/stt_service.dart new file mode 100644 index 0000000..9c3434a --- /dev/null +++ b/lib/src/infrastructure/stt/stt_service.dart @@ -0,0 +1,15 @@ +import 'stt_models.dart'; + +abstract interface class SttService { + Future checkAvailability({required String locale}); + + Future prepare({required String locale}); + + Stream events(String sessionId); + + Future start(SttStartRequest request); + + Future cancel(String sessionId); + + Future dispose(); +} diff --git a/pigeons/stt_api.dart b/pigeons/stt_api.dart new file mode 100644 index 0000000..fca38d8 --- /dev/null +++ b/pigeons/stt_api.dart @@ -0,0 +1,129 @@ +import 'package:pigeon/pigeon.dart'; + +@ConfigurePigeon( + PigeonOptions( + dartOut: 'lib/src/infrastructure/stt/generated/stt_api.g.dart', + dartOptions: DartOptions(), + swiftOut: 'ios/Runner/Generated/SttApi.g.swift', + swiftOptions: SwiftOptions(), + ), +) +enum NativeSttPermission { notDetermined, granted, denied, restricted } + +enum NativeSttModelStatus { + installed, + supported, + downloading, + missing, + unsupported, +} + +enum NativeSttPreparationStatus { + ready, + permissionRequired, + permissionDenied, + modelDownloadRequired, + modelDownloading, + unsupported, +} + +enum NativeSttEventKind { + ready, + partial, + finalResult, + ended, + failure, + cancelled, +} + +enum NativeSttFailureCode { + permissionDenied, + onDeviceUnavailable, + localeUnsupported, + modelMissing, + busy, + noMatch, + audio, + audioFormatMismatch, + interrupted, + cancelled, + unknown, +} + +class NativeSttAvailability { + NativeSttAvailability({ + required this.microphonePermission, + required this.speechPermission, + required this.onDeviceAvailable, + required this.localeSupported, + required this.modelStatus, + }); + + NativeSttPermission microphonePermission; + NativeSttPermission speechPermission; + bool onDeviceAvailable; + bool localeSupported; + NativeSttModelStatus modelStatus; +} + +class NativeSttPreparation { + NativeSttPreparation({required this.status, this.failureCode}); + + NativeSttPreparationStatus status; + NativeSttFailureCode? failureCode; +} + +class NativeSttStartRequest { + NativeSttStartRequest({ + required this.sessionId, + required this.generation, + required this.locale, + }); + + String sessionId; + int generation; + String locale; +} + +class NativeSttEvent { + NativeSttEvent({ + required this.sessionId, + required this.generation, + required this.sequence, + required this.kind, + this.text, + this.failureCode, + this.retryable, + }); + + String sessionId; + int generation; + int sequence; + NativeSttEventKind kind; + String? text; + NativeSttFailureCode? failureCode; + bool? retryable; +} + +@HostApi() +abstract class NativeSttHostApi { + @async + NativeSttAvailability checkAvailability(String locale); + + @async + NativeSttPreparation prepare(String locale); + + @async + void start(NativeSttStartRequest request); + + @async + void cancel(String sessionId); + + @async + void dispose(); +} + +@FlutterApi() +abstract class NativeSttFlutterApi { + void onEvent(NativeSttEvent event); +} diff --git a/pubspec.lock b/pubspec.lock index 4fbab7f..65ec6c5 100644 --- a/pubspec.lock +++ b/pubspec.lock @@ -41,6 +41,22 @@ packages: url: "https://pub.dev" source: hosted version: "2.1.2" + built_collection: + dependency: transitive + description: + name: built_collection + sha256: "44922ecebd8e3d0cef743dadd3b54df4e87e2a4c52a29465a1872aa9e192caa8" + url: "https://pub.dev" + source: hosted + version: "5.1.2" + built_value: + dependency: transitive + description: + name: built_value + sha256: f87ea98192116f7093cb214551ce1929caae0681fdba282b3d8b4462adee7bb7 + url: "https://pub.dev" + source: hosted + version: "8.13.0" characters: dependency: transitive description: @@ -73,6 +89,14 @@ packages: url: "https://pub.dev" source: hosted version: "1.1.3" + code_builder: + dependency: transitive + description: + name: code_builder + sha256: "811709c1badd02fc054039a13f66e0db1961865f9a314175f68c3eaeebec786f" + url: "https://pub.dev" + source: hosted + version: "4.13.0" collection: dependency: transitive description: @@ -113,6 +137,14 @@ packages: url: "https://pub.dev" source: hosted version: "1.0.9" + dart_style: + dependency: transitive + description: + name: dart_style + sha256: a0e4f65339b31eb6a9e6a12e14b4682a48d46fbc54a82f4a702396fbdbac63a7 + url: "https://pub.dev" + source: hosted + version: "3.1.14" fake_async: dependency: transitive description: @@ -137,6 +169,14 @@ packages: url: "https://pub.dev" source: hosted version: "7.0.1" + fixnum: + dependency: transitive + description: + name: fixnum + sha256: b6dc7065e46c974bc7c5f143080a6764ec7a4be6da1285ececdc37be96de53be + url: "https://pub.dev" + source: hosted + version: "1.1.1" flutter: dependency: "direct main" description: flutter @@ -171,6 +211,14 @@ packages: url: "https://pub.dev" source: hosted version: "2.2.0" + graphs: + dependency: transitive + description: + name: graphs + sha256: "741bbf84165310a68ff28fe9e727332eef1407342fca52759cb21ad8177bb8d0" + url: "https://pub.dev" + source: hosted + version: "2.3.2" http_multi_server: dependency: transitive description: @@ -291,6 +339,14 @@ packages: url: "https://pub.dev" source: hosted version: "1.9.1" + pigeon: + dependency: "direct dev" + description: + name: pigeon + sha256: "9c4b979de1ed662c8f66a921b7b42ab31c366da5141f3996590c52a8a70baf74" + url: "https://pub.dev" + source: hosted + version: "29.0.11" pool: dependency: transitive description: diff --git a/pubspec.yaml b/pubspec.yaml index 2951149..df5f92b 100644 --- a/pubspec.yaml +++ b/pubspec.yaml @@ -15,6 +15,7 @@ dev_dependencies: flutter_lints: ^6.0.0 flutter_test: sdk: flutter + pigeon: 29.0.11 test: ^1.26.3 flutter: diff --git a/test/infrastructure/stt/pigeon_stt_service_test.dart b/test/infrastructure/stt/pigeon_stt_service_test.dart new file mode 100644 index 0000000..7ce97ce --- /dev/null +++ b/test/infrastructure/stt/pigeon_stt_service_test.dart @@ -0,0 +1,241 @@ +import 'package:flutter/services.dart'; +import 'package:flutter_test/flutter_test.dart'; +import 'package:maltto/src/infrastructure/stt/generated/stt_api.g.dart' + as pigeon; +import 'package:maltto/src/infrastructure/stt/pigeon_stt_service.dart'; +import 'package:maltto/src/infrastructure/stt/stt_models.dart'; + +void main() { + TestWidgetsFlutterBinding.ensureInitialized(); + + const codec = pigeon.NativeSttHostApi.pigeonChannelCodec; + final messenger = + TestDefaultBinaryMessengerBinding.instance.defaultBinaryMessenger; + + BasicMessageChannel hostChannel(String method) => + BasicMessageChannel( + 'dev.flutter.pigeon.maltto.NativeSttHostApi.$method', + codec, + ); + + void setHostHandler( + String method, + Future Function(Object?)? handler, + ) { + messenger.setMockDecodedMessageHandler( + hostChannel(method), + handler, + ); + } + + Future ok(Object? value) async => [value]; + + setUp(() { + setHostHandler('dispose', (_) => ok(null)); + setHostHandler('cancel', (_) => ok(null)); + }); + + tearDown(() { + for (final method in [ + 'checkAvailability', + 'prepare', + 'start', + 'cancel', + 'dispose', + ]) { + setHostHandler(method, null); + } + pigeon.NativeSttFlutterApi.setUp(null); + }); + + test('권한 거부와 모델 미설치 준비 상태를 구조화해 전달한다', () async { + setHostHandler( + 'checkAvailability', + (_) => ok( + pigeon.NativeSttAvailability( + microphonePermission: pigeon.NativeSttPermission.denied, + speechPermission: pigeon.NativeSttPermission.granted, + onDeviceAvailable: true, + localeSupported: true, + modelStatus: pigeon.NativeSttModelStatus.supported, + ), + ), + ); + setHostHandler( + 'prepare', + (_) => ok( + pigeon.NativeSttPreparation( + status: pigeon.NativeSttPreparationStatus.modelDownloadRequired, + failureCode: pigeon.NativeSttFailureCode.modelMissing, + ), + ), + ); + final service = PigeonSttService(); + + final availability = await service.checkAvailability(locale: 'ko-KR'); + final preparation = await service.prepare(locale: 'ko-KR'); + + expect(availability.microphonePermission, SttPermission.denied); + expect(availability.modelStatus, SttModelStatus.supported); + expect(preparation.status, SttPreparationStatus.modelDownloadRequired); + expect(preparation.failureCode, SttFailureCode.modelMissing); + await service.dispose(); + }); + + test('오디오 포맷 불일치를 구조화 failure 이벤트로 전달한다', () async { + setHostHandler('start', (_) => ok(null)); + final service = PigeonSttService(); + final events = []; + final subscription = service.events('session').listen(events.add); + await service.start( + const SttStartRequest( + sessionId: 'session', + generation: 1, + locale: 'ko-KR', + ), + ); + + await messenger.handlePlatformMessage( + 'dev.flutter.pigeon.maltto.NativeSttFlutterApi.onEvent', + codec.encodeMessage([ + pigeon.NativeSttEvent( + sessionId: 'session', + generation: 1, + sequence: 1, + kind: pigeon.NativeSttEventKind.failure, + failureCode: pigeon.NativeSttFailureCode.audioFormatMismatch, + retryable: false, + ), + ]), + (_) {}, + ); + await pumpEventQueue(); + + expect(events.single.failureCode, SttFailureCode.audioFormatMismatch); + await subscription.cancel(); + await service.dispose(); + }); + + test('네이티브 시작 오류를 플랫폼 중립 예외로 변환한다', () async { + setHostHandler( + 'start', + (_) async => [ + 'modelMissing', + 'The on-device model is not installed.', + null, + ], + ); + final service = PigeonSttService(); + + await expectLater( + service.start( + const SttStartRequest( + sessionId: 'session', + generation: 1, + locale: 'ko-KR', + ), + ), + throwsA( + isA().having( + (exception) => exception.code, + 'code', + SttFailureCode.modelMissing, + ), + ), + ); + + await service.dispose(); + }); + + test('실행 중인 session의 새 start 거부가 기존 generation을 보존한다', () async { + setHostHandler('start', (_) => ok(null)); + final service = PigeonSttService(); + final events = []; + final subscription = service.events('session').listen(events.add); + await service.start( + const SttStartRequest( + sessionId: 'session', + generation: 1, + locale: 'ko-KR', + ), + ); + + await expectLater( + service.start( + const SttStartRequest( + sessionId: 'session', + generation: 2, + locale: 'ko-KR', + ), + ), + throwsA( + isA().having( + (exception) => exception.code, + 'code', + SttFailureCode.busy, + ), + ), + ); + await messenger.handlePlatformMessage( + 'dev.flutter.pigeon.maltto.NativeSttFlutterApi.onEvent', + codec.encodeMessage([ + pigeon.NativeSttEvent( + sessionId: 'session', + generation: 1, + sequence: 1, + kind: pigeon.NativeSttEventKind.partial, + text: '직접 쓴 테스트 문장', + ), + ]), + (_) {}, + ); + await pumpEventQueue(); + + expect(events.single.generation, 1); + await subscription.cancel(); + await service.dispose(); + }); + + test('종료된 generation의 늦은 결과와 빈 전사를 버린다', () async { + setHostHandler('start', (_) => ok(null)); + final service = PigeonSttService(); + final events = []; + final subscription = service.events('session').listen(events.add); + await service.start( + const SttStartRequest( + sessionId: 'session', + generation: 1, + locale: 'ko-KR', + ), + ); + await service.cancel('session'); + + for (final event in [ + pigeon.NativeSttEvent( + sessionId: 'session', + generation: 1, + sequence: 2, + kind: pigeon.NativeSttEventKind.finalResult, + text: '늦게 도착한 직접 쓴 문장', + ), + pigeon.NativeSttEvent( + sessionId: 'session', + generation: 2, + sequence: 1, + kind: pigeon.NativeSttEventKind.partial, + text: ' ', + ), + ]) { + await messenger.handlePlatformMessage( + 'dev.flutter.pigeon.maltto.NativeSttFlutterApi.onEvent', + codec.encodeMessage([event]), + (_) {}, + ); + } + await pumpEventQueue(); + + expect(events, isEmpty); + await subscription.cancel(); + await service.dispose(); + }); +}