feat(voice): harden Gemini Live provider integration

docs/CURRENT-STATE.md

@@ -1,5 +1,75 @@
 # God's Eye View Current State
 
+Authoritative typed input and OpenAI speech longer than 500 characters after
+policy normalization are unverified: tool dispatch asks for a shorter command.
+Later authoritative input can cancel cooperative preparation and the same input's
+deferred Radio playback; completed layer changes or station selection remain
+reported truthfully. This limit refusal does not infer a provider request or
+change providers.
+
+Voice retains authentic user-turn identity when OpenAI input transcription is
+explicitly disabled or fails. Gemini native tool calls use current socket,
+session, local-turn and call ownership; unordered input captions are advisory
+and cannot grant or revoke tool authority. First-party typed input remains
+authoritative through deferred delivery and same-turn continuation until new
+input or Stop supersedes it. An accepted fresh `set_voice_provider` call refuses
+tools that have not started in that local turn without undoing completed work.
+Provider selection remains manual. Built-in embed instructions default to
+host-neutral guidance; standalone opts into its chooser. Hosts with fewer than
+two distinct usable provider options hide the chooser and retain neutral guidance.
+Failed tool output retains bounded, redacted reason data with static follow-up
+instructions. Gemini boundary waits use its injected clock.
+Native Gemini call IDs must be nonempty protocol strings of at most 1,024
+characters, without control characters. Their exact identities stay retired for
+the socket lifetime. At the 256-identity budget, another new identity ends the
+session with a visible restart instruction before any batch action is queued;
+old identities are never evicted to admit replays.
+
+Gemini sentence-level voice activity does not by itself prove a reply is being
+generated. Local activity bookkeeping keeps an expiring session open for a
+possible reply; it does not identify server generations or authorize tools.
+An older retired completion preserves newer submitted input. A native terminal
+completion also settles a silent user turn, while tools and queued audio still
+delay presentation completion. Noise-only activity can wait until the native
+expiry deadline with the microphone muted; no shorter recovery timer is added.
+See [voice ownership](VOICE-OWNERSHIP.md) for deferred-input and protocol limits.
+
+Gemini push-to-talk suppresses empty and near-silent captures before submitting
+context, activity markers or audio. Sustained input energy admits bounded
+pre-roll and the rest of that hold; release still flushes its tail. Physical
+Space holds silence local playback and revoke pending Radio handoff. Only
+admitted speech retires the current provider/tool turn; a silent hold does not
+create a provider reply or a native-boundary wait. This is an energy floor, not speech recognition or a
+guarantee against loud background noise. Open-mic behavior is unchanged.
+On narrow or short screens, transient toast notices are positioned above the
+bottom voice controls rather than covering the provider selector.
+
+Gemini retires old replies before forwarding deferred input. Output refreshes
+the five-second retirement inactivity bound; deferred-input deadlines remain
+absolute. New input, Stop and provider replacement revoke older work. Existing
+map-stack work lacks caller cancellation. Pending annotation lookups and deferred
+outlines now share the voice call's cancellation signal; committed marks and
+unrelated annotation work survive. Stronger Gemini Radio controls start without
+waiting for an older pending selection; successful controls cancel covered work
+without treating siblings as a new turn or erasing committed selection. “Turn
+Radio on” enables its layer without starting audio; playback requires an explicit
+play, listen or resume request. Radio playback suspends active voice; pausing or
+disabling Radio resumes that same session preference only when Radio interrupted
+it. Manual playback follows the same rule, without a playback-duration limit.
+Provider replacement and explicit voice Stop cancel automatic resume; manual
+voice startup pauses Radio. A toast explains when Radio playback stops voice.
+A previously inactive voice session stays off when Radio stops. Local regression checks
+do not establish provider startup or browser/audio acceptance.
+
+OpenAI ends an owned session after more than 16 unmatched response reservations
+with a visible restart-required error. It preserves ordered correlation and
+requires explicit restart rather than dropping old provenance or reconnecting.
+OpenAI provider text arriving after a synchronous action committed cannot
+retroactively prevent that change; later tools remain blocked.
+Provider classification recognizes sentence punctuation at finalized segment
+boundaries while retaining quoted, negated, compound and non-voice-provider
+exclusions.
+
 Vessel sources can report healthy empty coverage using a current, complete
 snapshot with zero raw rows, a healthy transport and a positive last-message
 time. This clears obsolete contacts and settles startup without manufacturing
@@ -5661,3 +5731,13 @@ datacenters, dams and the Nepal locator retain that shared credit while displaye
 
 Military fragments stitch by source identity. Different mapped parcels merge only
 when their footprints touch or contain one another, never across a seam tolerance gap.
+
+### Gemini Live provider integration
+
+Standalone voice supports OpenAI and Gemini; embedded hosts default to OpenAI and opt into Gemini through `voice.supportedProviders`. The provider dropdown uses the existing microphone control and result card. Manual selection reconnects an active session in its prior input mode; push-to-talk stays muted until a fresh Space hold. Voice that was off stays off.
+
+Gemini uses a separate server-side `GEMINI_API_KEY` and one-session ephemeral credentials from `/api/gemini/token`. Device-rate microphone capture is resampled to 16 kHz; playback uses a separate 24 kHz context. Microphone permission precedes token minting. Bounded control queues, retired replies and pending tool actions are invalidated on interruption, Stop, replacement and teardown; input capture identities follow the physical microphone gate and end on Stop, replacement and teardown rather than on every interruption. `goAway` stops new turns and drains the current reply until completion or its supplied deadline, then requires an explicit restart.
+
+Both providers use the shared action schemas, tool-result policy, pointer context and Radio ownership. Successful Radio playback, voice-requested or manual, suspends active voice and shows a toast; pausing, stopping or disabling that Radio resumes the prior voice mode once, with no time limit. Radio-only station, tuner and volume changes keep the lease; a tuner drag that ends paused without a new station also resumes voice. Explicit voice Stop, starting voice, provider replacement and teardown revoke it.
+
+Clear standalone provider-switch requests receive the fixed spoken dropdown guidance; neither provider can change the selected provider through a tool. Ambiguous, quoted, negated, conditional and compound text remains ordinary conversation. The separate API-key setup entry remains available; the core provider does not add a Voice Settings panel or voice actions for opening key setup.

docs/VOICE-OWNERSHIP.md

@@ -5,22 +5,22 @@ overall lifetime. It coordinates startup and shutdown. `realtimeFacade.js`
 preserves existing methods and properties by delegation, without copying state.
 The session adapter, action runner and backend interfaces remain unchanged.
 
-| Owner | State and responsibility |
-| --- | --- |
-| `realtimeConnection.js` | Start generation, request cancellation, peer/data channel, acquired microphone stream and playback element |
-| `realtimeBackend.js` | Token and SDP requests, their separate transports, the abort/timeout scope around both, and client-secret expiry |
-| `realtimeTurns.js` | Active and superseded responses, pending confirmations, deduplicated calls and action cancellation |
-| `realtimeRadio.js` | Speaker ducking, prepared playback, stronger-action reservations and playback observers |
-| `realtimeInput.js` | Physical Space gesture, microphone mute, input/output meters and their frame/listener lifetime |
-| `realtimeCost.js` | Next-session preferences and the current session's model-bound meter |
-| `realtimeViewport.js` | One retained image, bounded deletion identities and capture generation |
-| `realtimeDiagnostics.js` | Bounded error history and sanitized optional diagnostics |
-| `turnMetrics.js` | Per-turn latency and speech spans written to the debug log |
-| `narration.js` | Progress lines and the earcon while tools run; spoken through the turns owner |
-| `pointerContext.js` | Pointer position, dwell pick and the current turn's pointer snapshot (session-scoped) |
-| `referents.js` | Numbered items from the latest result for `referent:n` (session-scoped) |
-| `resultDisplay.js` | What the voice card shows for a result and which items it numbers (read by the card and `referents.js`) |
-| `gevActions.js` runner | Analyst follow-up memory (session: `resetConversation()` on stop, `dispose()` on removal) |
+| Owner                    | State and responsibility                                                                                         |
+| ------------------------ | ---------------------------------------------------------------------------------------------------------------- |
+| `realtimeConnection.js`  | Start generation, request cancellation, peer/data channel, acquired microphone stream and playback element       |
+| `realtimeBackend.js`     | Token and SDP requests, their separate transports, the abort/timeout scope around both, and client-secret expiry |
+| `realtimeTurns.js`       | Active and superseded responses, pending confirmations, deduplicated calls and action cancellation               |
+| `realtimeRadio.js`       | Speaker ducking, prepared playback, stronger-action reservations and playback observers                          |
+| `realtimeInput.js`       | Physical Space gesture, microphone mute, input/output meters and their frame/listener lifetime                   |
+| `realtimeCost.js`        | Next-session preferences and the current session's model-bound meter                                             |
+| `realtimeViewport.js`    | One retained image, bounded deletion identities and capture generation                                           |
+| `realtimeDiagnostics.js` | Bounded error history and sanitized optional diagnostics                                                         |
+| `turnMetrics.js`         | Per-turn latency and speech spans written to the debug log                                                       |
+| `narration.js`           | Progress lines and the earcon while tools run; spoken through the turns owner                                    |
+| `pointerContext.js`      | Pointer position, dwell pick and the current turn's pointer snapshot (session-scoped)                            |
+| `referents.js`           | Numbered items from the latest result for `referent:n` (session-scoped)                                          |
+| `resultDisplay.js`       | What the voice card shows for a result and which items it numbers (read by the card and `referents.js`)          |
+| `gevActions.js` runner   | Analyst follow-up memory (session: `resetConversation()` on stop, `dispose()` on removal)                        |
 
 The backend is replaceable per deployment: `realtimeController.js` accepts it as a
 `backend` option and falls back to `createRealtimeBackend()`. Its two method
@@ -45,11 +45,122 @@ Action and post-capture continuations retain their conversation identity before
 publishing output, queueing a response or changing status.
 
 Behavioral invariants remain in force: a 500 ms Space hold claims push-to-talk;
-short control taps and text entry remain native; a claimed hold while the assistant speaks interrupts it (never during a Radio handoff); a click-started session stays
-open-mic. Every accepted or refused tool call receives its terminal output once.
+short control taps and text entry remain native. A claimed hold silences local playback when the Radio speaker guard permits it;
+admitted speech retires the provider turn and supersedes its tool work. A denied local claim
+preserves queued confirmation audio; a later native Gemini interruption retires
+the old reply and cancels its handoff. A click-started session stays open-mic.
+Gemini PTT waits for two consecutive 10 ms PCM windows above a low energy floor
+before submitting pointer context, activity markers or audio. Up to 400 ms of
+bounded pre-roll preserves speech onset; after admission, tails and pauses pass
+through until release. An empty or near-silent hold submits no new provider turn
+and clears its local pending-input state. It still silences existing local
+playback as an explicit Space hold and revokes a pending Radio handoff. A silent
+hold alone preserves the current provider/tool turn until its natural completion. This energy check is not a speech classifier:
+very quiet input below the floor is suppressed and sustained loud noise can pass.
+Open-mic capture and native automatic activity detection are unchanged.
+Current, uncancelled tool calls receive one terminal output within their owning
+provider conversation. Gemini suppresses calls retired by a session or local-turn
+boundary, or cancelled by the server. OpenAI answers superseded calls with a
+terminal refusal while the same conversation remains owned; callbacks from a
+replaced conversation cannot publish into the new session.
 New typed input supersedes old intent. Radio waits for the spoken confirmation and
-verified muted playback. A live cost meter stays bound to its negotiated model,
-even when the next-session preference changes.
+verified muted playback. Eligible stronger Radio controls begin without waiting
+for an older station lookup; a successful control cancels covered pending work.
+Already committed selection and unrelated sibling calls retain their authority.
+“Turn Radio on” enables the layer without selecting or playing a station;
+playback requires an explicit play, listen or resume request. Radio playback
+suspends an active voice session and remembers it until Radio is paused or
+disabled. This also applies to manual playback; merely enabling the layer does
+not suspend voice. Pausing or disabling Radio leaves a previously inactive
+voice session off. Provider replacement or explicit voice Stop cancels the
+saved resume; manually starting voice pauses Radio. A live cost meter stays
+bound to its negotiated model,
+even when the next-session preference changes. Clear standalone first-party
+provider requests remain ordinary provider turns, so the active OpenAI or Gemini
+model says in its normal voice, “I can’t switch voice providers directly. Hover
+over the Voice panel and use the provider dropdown.” Hosts without at least two
+distinct usable chooser options hide that control and
+instead explain that provider selection is managed by the app. The built-in
+endpoint defaults to this neutral guidance; the standalone host opts into chooser
+guidance. No tool can change the selected provider. Authoritative typed requests and
+recognized OpenAI speech block app actions; Gemini native-call refusal is
+described below. These refusals never change or save provider state, reconnect
+voice or remount a session. Ambiguous, quoted, negated, conditional and compound text
+remains ordinary conversation. Manual selection is the only provider switching
+path.
+
+Gemini native tool calls are bound to the current socket, session epoch, local
+turn and unique call ID. Native input captions are unordered advisory segments:
+they are shown without authoritative request attribution and cannot grant or
+revoke a tool's lifetime. First-party typed input retains its authenticated
+request token through transport pressure, deferred delivery and same-turn
+continuation, until real new input or Stop revokes it. An accepted fresh
+`set_voice_provider` call refuses every not-yet-started tool for the rest of its
+local turn; malformed or replayed calls cannot quarantine a new turn. No tool
+changes or saves the selected provider, reconnects voice or remounts a session.
+
+Gemini's sentence-level `ACTIVITY_START`/`ACTIVITY_END` signals track detected
+speech, not proof that a reply generation exists. Automatic activity alone does
+not retire an unanswered generation; observed server output or an explicit
+submitted push-to-talk/typed request retains the reply fence. Local activity
+owners preserve expiry liveness across an older retired completion, without
+identifying server generations or granting tools authority. The old boundary
+cannot clear a newer submitted request's pending reply. If expiry discards a
+newest deferred request, the ordered retired boundary closes the session rather
+than allowing its superseded forwarded predecessor to run under that request.
+Shorter repeated expiry notices retain that supersession. A native terminal
+completion settles even a silent user turn; pending tools, queued controls and
+audio still delay presentation completion, and no speech or captions are invented.
+
+Noise-only automatic activity may receive no generation. During expiry it can
+wait with the microphone muted until the supplied deadline, capped at ten
+minutes; no shorter no-evidence timer assigns reply identity. Native Live events
+do not provide a VAD-sentence generation ID or guarantee their ordering relative
+to tools. Local activity ownership cannot prove that arbitrary undelimited late
+output belongs to an old generation; tools retain the socket/session/local-turn
+and call-ID policy above.
+
+There is no first-party typed-command UI. Embedders and programmatic `sendTextCommand`
+callers can issue rapid typed requests while open-mic input is active. While a
+typed request is deferred behind an older reply boundary, concurrent microphone
+PCM may be discarded for up to that boundary's deadline. It is never replayed
+under the typed owner's identity. `deferredAudioDrops.nonPttOwner` and
+`deferredAudioDrops.closedActivity` report reason-specific lifetime-cumulative
+counts alongside `droppedAudioChunks`; they do not change admission or recovery.
+
+`sendTextCommand` return values are provider-dependent. Gemini returns `true`
+when it admits the request locally (sent or queued) and `false` for empty input,
+an expiring session or failed local admission. Its empty-input and expiry checks
+precede connection validation; ordinary nonempty input requires an open socket.
+Oversized text and some context or queue-limit violations throw. A queued request
+can still be superseded or fail its bounded wait, which status events report.
+OpenAI returns `undefined` for successful submission or connected empty input,
+`false` when sending the user conversation item fails, and throws when it is not
+connected. Local admission does not guarantee provider delivery or execution.
+
+For authoritative typed input and OpenAI speech, provider classification
+normalizes sentence punctuation at word boundaries, so "Switch to. Gemini."
+still identifies the direct request. Later authoritative segments can cancel
+cooperative pending resources, including generic and CCTV layer initialization.
+Pending annotations and their deferred outlines use the calling voice operation's
+cancellation signal. Stop, interruption and replacement cancel only that work;
+already committed marks and unrelated annotation owners remain. Completed actions
+and map-stack changes without caller cancellation cannot be undone. Gemini
+captions do not supply cancellation authority.
+Gemini retires old replies before forwarding deferred input; continued old output
+refreshes its five-second inactivity bound while deferred-input deadlines stay
+absolute. Stop and replacement invalidate both waits and their continuations.
+OpenAI input with explicitly disabled or failed optional transcription retains
+its authentic speech identity for ordinary tools. Missing, empty or untracked
+input cannot supply tool authority, and provider switching remains manual.
+
+OpenAI response reservations remain ordered: a delayed response must consume its
+own slot. More than 16 unmatched reservations ends the owned session and shows
+"Voice lost turn tracking. Start voice again." A deliberate restart restores a
+fresh transport and turn tracker; it never automatically reconnects or assigns
+old responses to new input. A missing response before overflow remains ambiguous
+with a delayed one, so this does not establish native leak frequency or repair
+every possible FIFO mismatch.
 
 Run the unit suite and both live recorded-audio modes against a configured local
 server: