Bilawal Sidhu
a19897ad15
- Voice layer manifest (src/voice/layerManifest.js) generates the layer enums, aliases and analyst field hints; a test fails when a catalog layer has no entry. - analyst_query validates fields, operators, units, values and scopes, counts up to 250,000 loaded records per layer, reports complete/partial/unanswered, tells LAYER_OFF / NOT_READY / FEED_UNAVAILABLE apart from zero, refuses mismatched follow-ups and never lets a superseded query replace follow-up memory. Seven more layers expose analyst records. - Tool results carry a code-built say line and a display card; the voice card in the mic control shows captions, plan steps and the result (src/voice/resultDisplay.js is the one adapter). Slow tools narrate their current step out of band; holding Space while the assistant talks cuts it off. Turn spans go to the debug log; caption transcription is metered; the log omits what was said unless GEV_VOICE_LOG_CONTENT=1. - Point and ask: the Space-down pointer snapshot resolves "this", "that one" and "around here" through 'pointer' targets, and "the second one" through referent:n from the last numbered list. - Prompt states each policy once (instructions + tools 48.5k -> 46.7k chars). - Voice benchmark harness (scripts/qa-voice-bench.mjs, scripts/voice-bench/) shares its phrase table and grading with scripts/qa-voice-routing.mjs.
3.5 KiB
Realtime voice ownership
realtimeController.js composes the session and owns its status, UI bindings and
overall lifetime. It coordinates startup and shutdown. realtimeFacade.js
preserves existing methods and properties by delegation, without copying state.
The session adapter, action runner and backend interfaces remain unchanged.
| Owner | State and responsibility |
|---|---|
realtimeConnection.js |
Start generation, request cancellation, peer/data channel, acquired microphone stream and playback element |
realtimeTurns.js |
Active and superseded responses, pending confirmations, deduplicated calls and action cancellation |
realtimeRadio.js |
Speaker ducking, prepared playback, stronger-action reservations and playback observers |
realtimeInput.js |
Physical Space gesture, microphone mute, input/output meters and their frame/listener lifetime |
realtimeCost.js |
Next-session preferences and the current session's model-bound meter |
realtimeViewport.js |
One retained image, bounded deletion identities and capture generation |
realtimeDiagnostics.js |
Bounded error history and sanitized optional diagnostics |
turnMetrics.js |
Per-turn latency and speech spans written to the debug log |
narration.js |
Progress lines and the earcon while tools run; spoken through the turns owner |
pointerContext.js |
Pointer position, dwell pick and the current turn's pointer snapshot (session-scoped) |
referents.js |
Numbered items from the latest result for referent:n (session-scoped) |
resultDisplay.js |
What the voice card shows for a result and which items it numbers (read by the card and referents.js) |
gevActions.js runner |
Analyst follow-up memory (session: resetConversation() on stop, dispose() on removal) |
Pure input policy, preferences and protocol-response policy have separate modules. Owners receive named readers, operations or focused collaborators. Cross-owner cancellation uses operations; response handling does not own Radio's reservations or viewport deletion bookkeeping.
Shutdown first invalidates acquisition and action work, then closes transport, stops audio analysis, releases media and clears conversation/input state. Full removal also detaches UI, keyboard, annotation and Radio observers. Radio takeover retains confirmed playback through that same shutdown path. A late offer, peer callback, viewport capture or cancelled meter frame cannot enter a replacement session. Action and post-capture continuations retain their conversation identity before publishing output, queueing a response or changing status.
Behavioral invariants remain in force: a 500 ms Space hold claims push-to-talk; short control taps and text entry remain native; a claimed hold while the assistant speaks interrupts it (never during a Radio handoff); a click-started session stays open-mic. Every accepted or refused tool call receives its terminal output once. New typed input supersedes old intent. Radio waits for the spoken confirmation and verified muted playback. A live cost meter stays bound to its negotiated model, even when the next-session preference changes.
Run the unit suite and both live recorded-audio modes against a configured local server:
node scripts/qa-voice-wav.mjs http://localhost:4173
node scripts/qa-voice-wav.mjs http://localhost:4173 --push-to-talk
The second mode uses the actual Space shortcut, verifies microphone acquisition and release, and checks the same globe/Radio outcome. Tracking and scene-control acceptance exercise voice's shared application operations separately.