feat(voice): harden Gemini Live provider integration

This commit is contained in:
manjunath committed 2026-10-16 17:09:03 +05:30
1 parent d4492d8f8c
commit 4fde920665
102 files changed
+28528 -1581

No files matched your search

+63
View File
@@ -6,6 +6,11 @@
MobilityData's systems catalogue; AMBici's is CC0. `gbfs.nextbike.net` joins
the proxy allowlist. These are the layer's first stations in Spain.
- Suppress empty and near-silent Gemini push-to-talk submissions without
losing buffered speech onset or release tails. Space holds silence local
playback; only admitted speech retires the current provider/tool turn. Keep narrow-screen toast notices clear of
the voice provider selector and bottom dock.
- Add nine Ontario transit feeds: TTC, MiWay, Hamilton HSR, Durham Region,
YRT/Viva, Guelph, Burlington, Barrie and Kingston, each carrying its own
licence quote and attribution. A feed can now name its own modes, so Toronto's
@@ -130,6 +135,56 @@
get an empty body; anything else that cannot become a response fails that
mirror so the next one is tried.
- Cancel pending voice-owned annotation lookups and deferred outlines on Stop,
interruption or replacement without clearing committed or unrelated marks.
Let stronger Gemini Radio controls start without waiting for older station
lookups; successful controls cancel covered pending work while preserving
committed selection and unrelated sibling calls. “Turn Radio on” enables the
layer without audio; playback requires an explicit play, listen or resume request.
Resume voice after Radio pause/off only when playback interrupted active voice;
manual playback follows the same rule. Provider changes and explicit voice
Stop cancel automatic resume, and manual voice startup pauses Radio.
Show a toast when Radio playback stops active voice. A tuner drag that ends
paused without a new station also resumes the voice Radio interrupted. An
older queued Gemini Radio request cannot start after a later successful
Stop, and deferred annotation outlines outlive their tool call for both
providers until the next input or session boundary.
- Keep Gemini sentence-level voice activity separate from observed reply
generation, so a pause before the first answer does not retire that answer.
Preserve newer submitted input across an older reply's completion and finish
silent native replies without inventing audio or captions. Expiry activity
bookkeeping does not identify server generations or authorize tools.
- Reject malformed Gemini tool-call identities before they can affect turn
policy. Retain accepted and cancelled identities for the socket lifetime;
require an explicit restart before a new batch exceeds the 256-identity budget.
- Refuse voice tool dispatch when authoritative typed input or OpenAI speech
exceeds the normalized 500-character policy limit. Ask for a shorter command
without inferring a provider change; cancel cooperative preparation and withhold
deferred Radio playback while preserving changes that already completed.
- Preserve OpenAI speech-command identity with disabled or failed optional
transcription. Gemini native tool calls use current socket, session, local-turn
and call ownership; unordered input captions are advisory. First-party typed
input retains its authority through deferred delivery and same-turn
continuation until new input or Stop supersedes it. An accepted fresh
`set_voice_provider` call refuses tools that have not started in that local
turn; provider selection remains manual. Narrow provider interception, make
chooser guidance host-aware, hide single-option choosers, preserve useful
bounded failure reasons, and use injected Gemini boundary timers.
Later authoritative input can cancel cooperative generic/CCTV preparation;
committed changes and non-cooperative map-stack work are not undone.
- End OpenAI sessions with an explicit restart message when unmatched response
reservations overflow. Preserve FIFO ownership and reject stale callbacks;
recovery never reconnects automatically or guesses a response's user item.
- Recognize direct voice-provider requests whose finalized input segments carry
sentence punctuation between command and target, such as "Switch to. Gemini."
Provider selection remains manual.
- Snow flakes sway within a fixed range instead of drifting sideways faster the
longer the app runs; the Wind setting still scales the sway.
@@ -372,6 +427,14 @@
Mapillary only and hide while GOOGLE STREET VIEW is lit; its globe marker
is a Google-blue view cone.
- Add Gemini Live as an alternative voice provider using a separate server-side
API key and short-lived browser credentials. It shares the existing microphone,
push-to-talk, result card, map tools and Radio handoff with OpenAI. The manual
provider dropdown reconnects active voice in its prior input mode; spoken or
typed provider-switch requests give fixed dropdown guidance. Gemini owns
device-rate capture, 16 kHz resampling, 24 kHz playback, bounded queues,
interruption cleanup and explicit restart after its connection expires.
- Add an experimental ChatGPT/Codex OAuth option for cloud voice in Provider
Settings. API-key voice remains the default. Local sign-in can start from
the auth button and reports completion, failure, timeout, or expired