fix(voice): refuse unrequested Gemini tool calls and share pointer turns

in push-to-talk, answer a Gemini tool call that arrives with no input, reply
or tool result outstanding with a failure instead of running it, and refuse
every other call in the reply it starts, including one after unprompted
speech; the reply stays tracked so the next input retires it and its late
boundary cannot close the user's turn

in push-to-talk, a caption that arrives after its turn ended no longer opens
a reply, so the next request is not retired with its tool calls discarded
open mic cannot tell either case from a new reply and stays unchanged

check packet ownership after the narration callback before queuing tools,
and raise the per-connection call history to 4096 IDs

move the pointer turn event, context item and announcement into
pointerContext so the OpenAI and Gemini controllers share them, and drop the
Gemini token route's copy of the shared exact-Origin comparison
This commit is contained in:
Sameh Khamis committed 2026-10-19 12:30:54 -07:00
1 parent bd2342dc81
commit 1c727842df
7 files changed
+448 -91

No files matched your search

+1 -1
View File
@@ -620,7 +620,7 @@ Source factories have dedicated `layers/<family>/source` exports. ALPR and earth
Voice controls bind to a protocol-independent session factory. The default WebRTC adapter preserves the existing Realtime connection, push-to-talk, cost controls and radio handoff. Session subscriptions expose state, transcript, action call/progress/result, interruption, completion and turn-metrics events; stopping or replacing a session cancels pending actions. An active session's normal Stop publishes one idle transition, including adapters that emit their own terminal state; a restart triggered by that transition keeps ownership of its new startup state. Alternate adapters can use the same controls and action runner.
The standalone app offers OpenAI and Gemini Live through an explicit provider chooser. Gemini keeps its permanent API key on the local server and mints a short-lived, single-use credential with server-owned instructions and the standalone action catalog. Its native WebSocket adapter supports open microphone and push-to-talk input, streaming audio, captions and the existing action runner. Switching providers ends the old conversation and releases its microphone and connection; only the input mode carries over. Gemini uses bounded shutdown draining and explicit manual restart, without automatic reconnection or a promise of uninterrupted session duration. Hosts that do not opt into the provider facade retain their existing default or custom adapter.
The standalone app offers OpenAI and Gemini Live through an explicit provider chooser. Gemini keeps its permanent API key on the local server and mints a short-lived, single-use credential with server-owned instructions and the standalone action catalog. Its native WebSocket adapter supports open microphone and push-to-talk input, streaming audio, captions and the existing action runner. In push-to-talk, a tool call that arrives with no input, reply or tool result outstanding is answered with a failure and never runs, and neither does any other call in the reply it starts; open-microphone turns start on Gemini's server-side activity detection, which reports no boundary to check. Switching providers ends the old conversation and releases its microphone and connection; only the input mode carries over. Gemini uses bounded shutdown draining and explicit manual restart, without automatic reconnection or a promise of uninterrupted session duration. Hosts that do not opt into the provider facade retain their existing default or custom adapter.
Voice action argument schemas have one portable owner under `src/voice/actionSchemas.js`. The Realtime provider builds 30 tools using separate description-only metadata. Description customization cannot replace argument types, enum values or required fields.