mirror of
https://github.com/bilawalsidhu/gods-eye-view.git
synced 2026-11-04 02:14:01 +00:00
Bilawal Sidhu
a19897ad15
- Voice layer manifest (src/voice/layerManifest.js) generates the layer enums, aliases and analyst field hints; a test fails when a catalog layer has no entry. - analyst_query validates fields, operators, units, values and scopes, counts up to 250,000 loaded records per layer, reports complete/partial/unanswered, tells LAYER_OFF / NOT_READY / FEED_UNAVAILABLE apart from zero, refuses mismatched follow-ups and never lets a superseded query replace follow-up memory. Seven more layers expose analyst records. - Tool results carry a code-built say line and a display card; the voice card in the mic control shows captions, plan steps and the result (src/voice/resultDisplay.js is the one adapter). Slow tools narrate their current step out of band; holding Space while the assistant talks cuts it off. Turn spans go to the debug log; caption transcription is metered; the log omits what was said unless GEV_VOICE_LOG_CONTENT=1. - Point and ask: the Space-down pointer snapshot resolves "this", "that one" and "around here" through 'pointer' targets, and "the second one" through referent:n from the last numbered list. - Prompt states each policy once (instructions + tools 48.5k -> 46.7k chars). - Voice benchmark harness (scripts/qa-voice-bench.mjs, scripts/voice-bench/) shares its phrase table and grading with scripts/qa-voice-routing.mjs.
55 lines
3.5 KiB
Markdown
55 lines
3.5 KiB
Markdown
# Realtime voice ownership
|
|
|
|
`realtimeController.js` composes the session and owns its status, UI bindings and
|
|
overall lifetime. It coordinates startup and shutdown. `realtimeFacade.js`
|
|
preserves existing methods and properties by delegation, without copying state.
|
|
The session adapter, action runner and backend interfaces remain unchanged.
|
|
|
|
| Owner | State and responsibility |
|
|
| --- | --- |
|
|
| `realtimeConnection.js` | Start generation, request cancellation, peer/data channel, acquired microphone stream and playback element |
|
|
| `realtimeTurns.js` | Active and superseded responses, pending confirmations, deduplicated calls and action cancellation |
|
|
| `realtimeRadio.js` | Speaker ducking, prepared playback, stronger-action reservations and playback observers |
|
|
| `realtimeInput.js` | Physical Space gesture, microphone mute, input/output meters and their frame/listener lifetime |
|
|
| `realtimeCost.js` | Next-session preferences and the current session's model-bound meter |
|
|
| `realtimeViewport.js` | One retained image, bounded deletion identities and capture generation |
|
|
| `realtimeDiagnostics.js` | Bounded error history and sanitized optional diagnostics |
|
|
| `turnMetrics.js` | Per-turn latency and speech spans written to the debug log |
|
|
| `narration.js` | Progress lines and the earcon while tools run; spoken through the turns owner |
|
|
| `pointerContext.js` | Pointer position, dwell pick and the current turn's pointer snapshot (session-scoped) |
|
|
| `referents.js` | Numbered items from the latest result for `referent:n` (session-scoped) |
|
|
| `resultDisplay.js` | What the voice card shows for a result and which items it numbers (read by the card and `referents.js`) |
|
|
| `gevActions.js` runner | Analyst follow-up memory (session: `resetConversation()` on stop, `dispose()` on removal) |
|
|
|
|
Pure input policy, preferences and protocol-response policy have separate modules.
|
|
Owners receive named readers, operations or focused collaborators. Cross-owner
|
|
cancellation uses operations; response handling does not own Radio's reservations
|
|
or viewport deletion bookkeeping.
|
|
|
|
Shutdown first invalidates acquisition and action work, then closes transport,
|
|
stops audio analysis, releases media and clears conversation/input state. Full
|
|
removal also detaches UI, keyboard, annotation and Radio observers. Radio takeover
|
|
retains confirmed playback through that same shutdown path. A late offer, peer callback,
|
|
viewport capture or cancelled meter frame cannot enter a replacement session.
|
|
Action and post-capture continuations retain their conversation identity before
|
|
publishing output, queueing a response or changing status.
|
|
|
|
Behavioral invariants remain in force: a 500 ms Space hold claims push-to-talk;
|
|
short control taps and text entry remain native; a claimed hold while the assistant speaks interrupts it (never during a Radio handoff); a click-started session stays
|
|
open-mic. Every accepted or refused tool call receives its terminal output once.
|
|
New typed input supersedes old intent. Radio waits for the spoken confirmation and
|
|
verified muted playback. A live cost meter stays bound to its negotiated model,
|
|
even when the next-session preference changes.
|
|
|
|
Run the unit suite and both live recorded-audio modes against a configured local
|
|
server:
|
|
|
|
```sh
|
|
node scripts/qa-voice-wav.mjs http://localhost:4173
|
|
node scripts/qa-voice-wav.mjs http://localhost:4173 --push-to-talk
|
|
```
|
|
|
|
The second mode uses the actual Space shortcut, verifies microphone acquisition
|
|
and release, and checks the same globe/Radio outcome. Tracking and scene-control
|
|
acceptance exercise voice's shared application operations separately.
|