gods-eye-view/docs/VOICE-OWNERSHIP.md

Realtime voice ownership

realtimeController.js composes the session and owns its status, UI bindings and overall lifetime. It coordinates startup and shutdown. realtimeFacade.js preserves existing methods and properties by delegation, without copying state. The session adapter, action runner and backend interfaces remain unchanged.

Owner	State and responsibility
realtimeConnection.js	Start generation, request cancellation, peer/data channel, acquired microphone stream and playback element
realtimeBackend.js	Token and SDP requests, their separate transports, the abort/timeout scope around both, and client-secret expiry
realtimeTurns.js	Active and superseded responses, pending confirmations, deduplicated calls and action cancellation
realtimeRadio.js	Speaker ducking, prepared playback, stronger-action reservations and playback observers
realtimeInput.js	Physical Space gesture, microphone mute, input/output meters and their frame/listener lifetime
realtimeCost.js	Next-session preferences and the current session's model-bound meter
realtimeViewport.js	One retained image, bounded deletion identities and capture generation
realtimeDiagnostics.js	Bounded error history and sanitized optional diagnostics
turnMetrics.js	Per-turn latency and speech spans written to the debug log
narration.js	Progress lines and the earcon while tools run; spoken through the turns owner
pointerContext.js	Pointer position, dwell pick and the current turn's pointer snapshot (session-scoped)
referents.js	Numbered items from the latest result for referent:n (session-scoped)
resultDisplay.js	What the voice card shows for a result and which items it numbers (read by the card and referents.js)
gevActions.js runner	Analyst follow-up memory (session: resetConversation() on stop, dispose() on removal)

The backend is replaceable per deployment: realtimeController.js accepts it as a backend option and falls back to createRealtimeBackend(). Its two method signatures live with application construction; what belongs here is the ownership. The connection is the backend's only consumer in the application. The connection reads credential metadata for diagnostics and cost accounting, then passes the complete credential unchanged to negotiate. Client-secret expiry is refused inside the backend, not by the connection. realtimeOwners.test.mjs drives whole sessions through a substitute that implements only those two methods.

Pure input policy, preferences and protocol-response policy have separate modules. Owners receive named readers, operations or focused collaborators. Cross-owner cancellation uses operations; response handling does not own Radio's reservations or viewport deletion bookkeeping.

Shutdown first invalidates acquisition and action work, then closes transport, stops audio analysis, releases media and clears conversation/input state. Full removal also detaches UI, keyboard, annotation and Radio observers. Radio takeover retains confirmed playback through that same shutdown path. A late offer, peer callback, viewport capture or cancelled meter frame cannot enter a replacement session. Action and post-capture continuations retain their conversation identity before publishing output, queueing a response or changing status.

Behavioral invariants remain in force: a 500 ms Space hold claims push-to-talk; short control taps and text entry remain native. A claimed hold silences local playback when the Radio speaker guard permits it; admitted speech retires the provider turn and supersedes its tool work. A denied local claim preserves queued confirmation audio; a later native Gemini interruption retires the old reply and cancels its handoff. A click-started session stays open-mic. Gemini PTT waits for two consecutive 10 ms PCM windows above a low energy floor before submitting pointer context, activity markers or audio. Up to 400 ms of bounded pre-roll preserves speech onset; after admission, tails and pauses pass through until release. An empty or near-silent hold submits no new provider turn and clears its local pending-input state. It still silences existing local playback as an explicit Space hold and revokes a pending Radio handoff. A silent hold alone preserves the current provider/tool turn until its natural completion. This energy check is not a speech classifier: very quiet input below the floor is suppressed and sustained loud noise can pass. Open-mic capture and native automatic activity detection are unchanged. Current, uncancelled tool calls receive one terminal output within their owning provider conversation. Gemini suppresses calls retired by a session or local-turn boundary, or cancelled by the server. OpenAI answers superseded calls with a terminal refusal while the same conversation remains owned; callbacks from a replaced conversation cannot publish into the new session. New typed input supersedes old intent. Radio waits for the spoken confirmation and verified muted playback. Eligible stronger Radio controls begin without waiting for an older station lookup; a successful control cancels covered pending work. Already committed selection and unrelated sibling calls retain their authority. “Turn Radio on” enables the layer without selecting or playing a station; playback requires an explicit play, listen or resume request. Radio playback suspends an active voice session and remembers it until Radio is paused or disabled. This also applies to manual playback; merely enabling the layer does not suspend voice. Pausing or disabling Radio leaves a previously inactive voice session off. Provider replacement or explicit voice Stop cancels the saved resume; manually starting voice pauses Radio. A live cost meter stays bound to its negotiated model, even when the next-session preference changes. Clear standalone first-party provider requests remain ordinary provider turns, so the active OpenAI or Gemini model says in its normal voice, “I can’t switch voice providers directly. Hover over the Voice panel and use the provider dropdown.” Hosts without at least two distinct usable chooser options hide that control and instead explain that provider selection is managed by the app. The built-in endpoint defaults to this neutral guidance; the standalone host opts into chooser guidance. No tool can change the selected provider. Authoritative typed requests and recognized OpenAI speech block app actions; Gemini native-call refusal is described below. These refusals never change or save provider state, reconnect voice or remount a session. Ambiguous, quoted, negated, conditional and compound text remains ordinary conversation. Manual selection is the only provider switching path.

Gemini native tool calls are bound to the current socket, session epoch, local turn and unique call ID. Native input captions are unordered advisory segments: they are shown without authoritative request attribution and cannot grant or revoke a tool's lifetime. First-party typed input retains its authenticated request token through transport pressure, deferred delivery and same-turn continuation, until real new input or Stop revokes it. An accepted fresh set_voice_provider call refuses every not-yet-started tool for the rest of its local turn; malformed or replayed calls cannot quarantine a new turn. No tool changes or saves the selected provider, reconnects voice or remounts a session.

Gemini's sentence-level ACTIVITY_START/ACTIVITY_END signals track detected speech, not proof that a reply generation exists. Automatic activity alone does not retire an unanswered generation; observed server output or an explicit submitted push-to-talk/typed request retains the reply fence. Local activity owners preserve expiry liveness across an older retired completion, without identifying server generations or granting tools authority. The old boundary cannot clear a newer submitted request's pending reply. If expiry discards a newest deferred request, the ordered retired boundary closes the session rather than allowing its superseded forwarded predecessor to run under that request. Shorter repeated expiry notices retain that supersession. A native terminal completion settles even a silent user turn; pending tools, queued controls and audio still delay presentation completion, and no speech or captions are invented.

Noise-only automatic activity may receive no generation. During expiry it can wait with the microphone muted until the supplied deadline, capped at ten minutes; no shorter no-evidence timer assigns reply identity. Native Live events do not provide a VAD-sentence generation ID or guarantee their ordering relative to tools. Local activity ownership cannot prove that arbitrary undelimited late output belongs to an old generation; tools retain the socket/session/local-turn and call-ID policy above.

There is no first-party typed-command UI. Embedders and programmatic sendTextCommand callers can issue rapid typed requests while open-mic input is active. While a typed request is deferred behind an older reply boundary, concurrent microphone PCM may be discarded for up to that boundary's deadline. It is never replayed under the typed owner's identity. deferredAudioDrops.nonPttOwner and deferredAudioDrops.closedActivity report reason-specific lifetime-cumulative counts alongside droppedAudioChunks; they do not change admission or recovery.

sendTextCommand return values are provider-dependent. Gemini returns true when it admits the request locally (sent or queued) and false for empty input, an expiring session or failed local admission. Its empty-input and expiry checks precede connection validation; ordinary nonempty input requires an open socket. Oversized text and some context or queue-limit violations throw. A queued request can still be superseded or fail its bounded wait, which status events report. OpenAI returns undefined for successful submission or connected empty input, false when sending the user conversation item fails, and throws when it is not connected. Local admission does not guarantee provider delivery or execution.

For authoritative typed input and OpenAI speech, provider classification normalizes sentence punctuation at word boundaries, so "Switch to. Gemini." still identifies the direct request. Later authoritative segments can cancel cooperative pending resources, including generic and CCTV layer initialization. Pending annotations and their deferred outlines use the calling voice operation's cancellation signal. Stop, interruption and replacement cancel only that work; already committed marks and unrelated annotation owners remain. Completed actions and map-stack changes without caller cancellation cannot be undone. Gemini captions do not supply cancellation authority. Gemini retires old replies before forwarding deferred input; continued old output refreshes its five-second inactivity bound while deferred-input deadlines stay absolute. Stop and replacement invalidate both waits and their continuations. OpenAI input with explicitly disabled or failed optional transcription retains its authentic speech identity for ordinary tools. Missing, empty or untracked input cannot supply tool authority, and provider switching remains manual.

OpenAI response reservations remain ordered: a delayed response must consume its own slot. More than 16 unmatched reservations ends the owned session and shows "Voice lost turn tracking. Start voice again." A deliberate restart restores a fresh transport and turn tracker; it never automatically reconnects or assigns old responses to new input. A missing response before overflow remains ambiguous with a delayed one, so this does not establish native leak frequency or repair every possible FIFO mismatch.

Run the unit suite and both live recorded-audio modes against a configured local server:

node scripts/qa-voice-wav.mjs http://localhost:4173
node scripts/qa-voice-wav.mjs http://localhost:4173 --push-to-talk

The second mode uses the actual Space shortcut, verifies microphone acquisition and release, and checks the same globe/Radio outcome. Tracking and scene-control acceptance exercise voice's shared application operations separately.