[juliantoledo]

Title

feat(agent): add Gemini Live (gemini-3.8-live) voice session support for AppFunctions

Description

Summary

Adds real-time bidirectional voice interaction to the Agent Demo screen using the Gemini Live API (models/gemini-3.8-live), enabling users to invoke Android AppFunctions and internal agent tools by voice while viewing live transcriptions, tool execution status, generated image attachments, and PendingIntent action confirmation cards directly in the active chat thread.

Key Changes

• Gemini Live WebSocket Session (GeminiLiveSessionManager & LiveSessionState):

  • Connects to the Gemini v1beta BidiGenerateContent WebSocket endpoint using Ktor WebSockets (DataModule.provideHttpClient) and the configured Gemini API key from SettingsRepository.
  • Configures "models/gemini-3.8-live" with AUDIO response modality, input/output audio transcriptions, system instructions, and all enabled AppFunction and internal tool declarations.
  • Streams user speech and model audio responses while flushing input and output transcriptions into the active chat thread as USER and ASSISTANT messages.

• Low-Latency PCM Audio I/O (GeminiLiveAudioHelper):

  • Captures 16 kHz mono 16-bit PCM microphone audio (VOICE_COMMUNICATION with AcousticEchoCanceler enabled when available) and plays back 24 kHz mono 16-bit PCM model audio via AudioTrack.
  • Supports immediate playback queue clearing on user interruption and recording pause/resume while executing tool calls.

• AppFunction & Internal Tool Bridge (AgentOrchestrator & GeminiToolConverter):

  • Added AgentOrchestrator.getEnabledTools and AgentOrchestrator.executeLiveToolCalls to execute AppFunctions and internal tools (getCurrentLocation, geocodeAddress, generateImage) during a Live session, updating AgentStatus.InvokingTool, posting generated image attachments and PendingIntent confirmation cards to the thread, and returning functionResponses over the WebSocket.
  • Added GeminiToolConverter.convertForLiveApi to normalize tool schemas (enum field naming and OpenAPI format compatibility) for BidiGenerateContent.

• Agent Demo UI & Permissions (AgentDemoScreen, AgentDemoViewModel, AgentUiState, AndroidManifest.xml):

  • Added a microphone / stop toggle button in the chat input bar with runtime RECORD_AUDIO permission handling and @App scoping support.
  • Added a LiveSessionIndicator status banner when connecting or listening, and updated ModelDropdown in the top bar to display gemini-3.8-live while a Live session is active.
  • Declared INTERNET and RECORD_AUDIO permissions in AndroidManifest.xml.

• Unit Tests:

  • Added GeminiLiveSessionManagerTest, GeminiToolConverterTest.convertForLiveApi_normalizesEnumsAndStripsFileReferenceFormat, and AgentDemoViewModelTest coverage for OnToggleLiveSession.

[mingweiliao]

Thanks for putting this together! Before diving into smaller implementation details, I'd like to align on the architecture around GeminiLiveSessionManager and AgentOrchestrator.

Since a live voice session is a stateful bidirectional stream rather than a DB-driven request/response turn loop, having a dedicated LiveAgentOrchestrator alongside AgentOrchestrator makes sense. However, the current split between GeminiLiveSessionManager, AgentOrchestrator, and the data layer doesn't match how the rest of the project is structured:

1. Tool execution is duplicated inside AgentOrchestrator (executeToolCalls vs. executeLiveToolCalls):

   • Currently, GeminiLiveSessionManager injects AgentOrchestrator just to call getEnabledTools(...) and a newly added executeLiveToolCalls(...) (~220 lines).
   • executeLiveToolCalls duplicates almost the entire tool execution pipeline from executeToolCalls (internal tool execution, remote file URL resolution, URI permission granting, ConvertInputToAppFunctionDataUseCase, and ExecuteAppFunctionUseCase).
   • Instead of having two separate tool execution loops that can easily drift out of sync, we should extract/share a single tool discovery & execution pipeline (e.g., a shared use case/helper or a single executeToolCall entry point) used by both AgentOrchestrator and LiveAgentOrchestrator.

2. The live orchestrator in domain/ should be LLM/provider-agnostic (move Gemini WebSocket & JSON protocol details to data/):

   • In the existing architecture, AgentOrchestrator (domain/) is provider-agnostic—it works with domain models (LlmProvider, LlmInput, LlmResponse, LlmResponsePart.ToolCall) and delegates Ktor network transport, Gemini JSON serialization, system instructions, and GeminiToolConverter usage to GeminiProviderImpl (data/).
   • In this PR, GeminiLiveSessionManager sits in domain/ but is tightly coupled to Gemini: it directly injects Ktor's HttpClient and data-layer GeminiToolConverter, hardcodes the BidiGenerateContent WebSocket URL, builds/parses raw Gemini JSON frames (setup, realtimeInput, toolResponse), and duplicates JsonElement.toPrimitive() and tool-name mapping from GeminiProviderImpl.
   • Just like AgentOrchestrator, LiveAgentOrchestrator in domain/ should remain provider-agnostic (orchestrating session state, audio, transcript persistence, and tool execution), while the Gemini-specific WebSocket transport and JSON/schema conversion move into data/ behind a domain interface.

[mingweiliao]

Another question is around how the UI should work

Right now, all the UI state and lifecycle for the Live session are wired directly into AgentDemoScreen and AgentDemoViewModel (liveSessionState, overriding ModelDropdown with "gemini-3.8-live", rendering LiveSessionIndicator in the chat list, and stopping the session in ViewModel.onCleared()).

Instead of embedding the Live session state inside AgentDemoScreen / AgentDemoViewModel, the experience should work more like Gemini Live:

1. Tapping the Live icon starts a VoiceInteractionService (or VoiceInteractionSession) that takes over:
   • AgentDemoScreen only needs the entry point (tapping the button) to launch/trigger the service.
   • All the logic and UI for interacting with LiveAgentOrchestrator (session state, listening/connecting UI, stopping the session) should live in the VoiceInteractionService / session layer rather than polluting AgentDemoViewModel and AgentDemoScreen.
2. Why this is also needed for AppFunctions execution:
   • When an AppFunction triggers a PendingIntent confirmation that launches a target app's Activity, or when the user switches to the target app (e.g., ChatApp) while talking, MainActivity goes into the background.
   • Without a VoiceInteractionService (or microphone foreground service), Android will silence AudioRecord as soon as MainActivity leaves the foreground, and tying the session to AgentDemoViewModel means navigating away kills the session.

[juliantoledo]

Thank you @mingweiliao, great feedback. I worked on those comments in the last commit.

[mingweiliao]

Taking a step back, this PR has grown to ~5,200 lines across 34 files, and because design changes in one layer ripple across the others, it will become quite messy if we iterate it as it is.

Could we break this down into a series of smaller PRs so we can review and merge each piece in a logical order?

1. Refactor tool discovery and execution: Extract the tool logic out of AgentOrchestrator so it can be shared between chat and live modes.
2. Live LLM provider layer: Introduce the provider-agnostic live streaming abstraction and the Gemini Live implementation.
3. Live session orchestration and audio: Add the live orchestrator and audio capture/playback handling on top of the shared tool logic and live provider. (If audio part is complicated itself, I'll suggest separate these two as well)
4. Voice interaction service and UI: Wire up the Android VoiceInteractionService, the Live UI experience, and the entry point in the app.

The tool refactoring in step 1 looks nearly ready as-is, so we should be able to merge that quickly and then focus on one layer at a time.

[mingweiliao]

This is the review for step 1

[mingweiliao]

Instead of introducing GetEnabledToolsUseCase as a separate class, could we evolve GetAppFunctionsUseCase to accept filter parameters?

// Flatten them here, but it should be a param class
operator fun invoke(
    // Some of the filters are nullable, null meaning don't apply the filter at all
    targetPackageNames: Set<String>? = null,
    isEnabled: Boolean? = null,
    isConnected: Boolean? = null,
    includeInternalTools: Boolean = true,
): Flow<Map<AppFunctionPackageMetadata, List<AppFunctionMetadata>>>

So that there is a single use case to observe AppFunctions and the filter parameter allow them to read what they need.

[mingweiliao]

We should evolve ExecuteAppFunctionUseCase instead of introducing ExecuteToolCallUseCase. That ensures any caller invoking an AppFunction (including DebuggingViewModel) goes through the same execution path, especially for common steps like granting URI permissions and dispatching internal tools.

ExecuteAppFunctionUseCase should still take (function: AppFunctionMetadata, parameters: AppFunctionData) rather than LlmResponsePart.ToolCall. Its only responsibility should be taking AppFunctionData, running common pre-execution processing (like URI permission grants), and returning ExecuteAppFunctionResult. Converting ToolCall arguments into AppFunctionData (such as resolving remote file references via ConvertInputToAppFunctionDataUseCase) and handling caller-specific post-processing (like mapping to ToolOutput or extracting MessageAttachment) should stay in the orchestrators or a dedicated conversion use case if needs to be shared.

[juliantoledo]

thank you, I will break it into smaller PRs shortly.

[juliantoledo]

Closing this umbrella PR (keeping the feature/gemini-live branch intact for reference) to break the changes down into smaller, layer-by-layer PRs as discussed:

1. Refactor tool discovery and execution: refactor(agent): unify AppFunction discovery, input conversion, and execution #65
2. Live LLM provider layer (LiveLlmProvider & GeminiLiveProviderImpl)
3. Live audio capture & playback (LiveAudioSession & GeminiLiveAudioHelper)
4. Live session orchestration (LiveAgentOrchestrator)
5. VoiceInteractionService & UI