Repository navigation
feat(agent): add LiveLlmProvider abstraction and Gemini Live WebSocket provider - #69
juliantoledo wants to merge 1 commit into
Conversation
d2c1af9 to
e52f370
Compare
| private val toolConverter: GeminiToolConverter, | ||
| private val settingsRepository: SettingsRepository, | ||
| ) : LiveLlmProvider { | ||
| internal var webSocketConnector: |
There was a problem hiding this comment.
Have a secondary constructor to accept this and the primary one can inject the default without relying on Hilt. Then, mark the secondary constructor with VisibleForTesting to make it clear that swapping connector is for testing purpose
| tools: List<AppFunctionMetadata>, | ||
| block: suspend (LiveLlmSession) -> Unit, | ||
| ) { | ||
| val selectedVoice = settingsRepository.geminiLiveVoice.first() |
There was a problem hiding this comment.
Move this to parameter and make it the caller's responsibility to specify the voice type instead
| "WebSocket closed before setupComplete was received", | ||
| ) | ||
| val firstJson = Json.parseToJsonElement(firstFrameText).jsonObject | ||
| if (!firstJson.containsKey("setupComplete")) { |
There was a problem hiding this comment.
Extract a const val. Apply to all other string that is used to parse Gemini's server response
| val serverContent = serverJson["serverContent"]?.jsonObject | ||
| if (serverContent != null) { | ||
| val interrupted = | ||
| serverContent["interrupted"]?.jsonPrimitive?.booleanOrNull ?: false |
There was a problem hiding this comment.
The logic here goes through each expected key in sequence, does that mean a single serverContent might contain them together (e.g., having both interrupted and inputTranscription at the same time). If only one of them is expected, use "when" statement to branch the logic so it's clear to read
| val name = | ||
| callObj["name"]?.jsonPrimitive?.contentOrNull ?: return@mapNotNull null | ||
| val matchingTool = toolConverter.findToolByName(name, tools) | ||
| val argumentsMap = callObj["args"]?.jsonObject.toArgumentsMap() |
There was a problem hiding this comment.
Please update this section to handle the error in a clear way. Right now there is a lot of scenario that the session will silently let it pass, which would make it difficult to debug if things go wrong (e.g., id not matching)
| return null | ||
| } | ||
|
|
||
| internal fun parseServerMessage( |
There was a problem hiding this comment.
Please audit the internal method in this class to see if they really need to be internal, from the test, it looks like they are not used at all
| every { Log.w(any<String>(), any<String>()) } returns 0 | ||
| every { Log.e(any(), any(), any()) } returns 0 | ||
|
|
||
| settingsRepository = mockk(relaxed = true) |
There was a problem hiding this comment.
Use the FakeSettingRepository instead
| import org.junit.Test | ||
| import java.util.Base64 | ||
|
|
||
| class GeminiLiveProviderImplTest { |
There was a problem hiding this comment.
Missing test cases in GeminiLiveProviderImplTest:
- Frame.Binary server frames: Please add a test verifying receiveNextJsonText() decodes Frame.Binary frames for both setupComplete and events.
- Unexpected first handshake frame (missing setupComplete): Please test that receiving a non-setupComplete JSON frame (e.g., {"error": ...}) throws IllegalStateException("Unexpected setup response: ...").
- WebSocket close() in finally: Please verify that WebSocketSession.close(CloseReason(CloseReason.Codes.NORMAL, ...)) is called when block finishes normally and when block throws/cancels.
- Malformed frame recovery mid-session: Please verify that if a malformed JSON frame arrives in session.events, it is skipped without crashing the flow and subsequent valid frames are still emitted.
| } | ||
|
|
||
| @Test | ||
| fun convertForLiveApi_normalizesEnumsAndStripsFileReferenceFormat() { |
There was a problem hiding this comment.
Move this to a new GeminiLiveToolConverterTest once that part is separated in implementation
Also adding these tests as well
- Integer enums in Live mode: Verify that AppFunctionIntTypeMetadata with enumValues omits "enums" when converted for the Live API (since OpenAPI enum in BidiGenerateContent is only valid on string types).
- android.net.Uri object and reference types in Live mode: Verify that "format": "file_reference" is omitted for android.net.Uri AppFunctionObjectTypeMetadata and AppFunctionReferenceTypeMetadata, not just top-level string parameters ending in Uri.
- selectDistinctTools, findToolByName, and resolveToolName (this should be done in ToolConvertorTest since they will no longer be Gemini specific implementation)
| assertEquals(ServiceTier.PRIORITY, value) | ||
| } | ||
|
|
||
| @Test |
There was a problem hiding this comment.
Corrupted/unknown stored voice fallback: Verify that if DataStore contains an unrecognized string for gemini_live_voice, repository.geminiLiveVoice.first() falls back to GeminiLiveVoice.DEFAULT (ZEPHYR).
Summary
Part 2 of 5 in the Gemini Live voice assistant series (following #65):
LiveLlmProviderabstraction +GeminiLiveProviderImpl)LiveAudioSession+GeminiLiveAudioHelper)LiveAgentOrchestrator)VoiceInteractionServiceand Live UI integrationChanges in this PR
LiveLlmProvider/LiveLlmSession/LiveLlmEvent: Introduce a provider-agnostic bidirectional streaming abstraction for realtime audio, input/output transcriptions, turn completion, interruption events, and tool call requests/responses.LlmProviderFactory/LlmProviderFactoryImpl: Extend the provider factory withgetLiveProvider(name: LlmProviderName): LiveLlmProvider.GeminiLiveProviderImpl&GeminiLiveVoice: Implement Gemini Live over theBidiGenerateContentWebSocket endpoint (models/gemini-3.8-live) using KtorWebSockets, supporting prebuilt studio voice selection (GeminiLiveVoicepersisted inSettingsRepository), realtime PCM audio streaming, input/output audio transcriptions, and tool call/response frames.GeminiToolConverter&GeminiJsonUtils: AddconvertForLiveApi(normalizing OpenAPI schema fields forBidiGenerateContent),selectDistinctTools,findToolByName,resolveToolName, and sharedtoPrimitive()/toArgumentsMap()JSON helpers used across bothGeminiProviderImplandGeminiLiveProviderImpl.GeminiLiveProviderImplTest,GeminiToolConverterTest, andSettingsRepositoryTest.Test Plan
./gradlew spotlessCheck testStandardDebugUnitTest testRetailDebugUnitTest compileStandardDebugAndroidTestKotlin compileStandardDebugScreenshotTestKotlin