diff --git a/.env.example b/.env.example index 67a1e74a7..a28f375d4 100644 --- a/.env.example +++ b/.env.example @@ -64,6 +64,13 @@ OPENAI_REALTIME_REASONING_EFFORT=low # Keep only a short conversational window. Current map state is fetched live. OPENAI_REALTIME_CONTEXT_TOKENS=3000 OPENAI_REALTIME_CONTEXT_RETENTION=0.5 +# Voice-card captions of what you said. Billed separately by OpenAI and +# counted in the session cost meter and cap; off disables them. +OPENAI_REALTIME_TRANSCRIBE_MODEL=gpt-4o-mini-transcribe +# The voice debug log (.gev-logs/realtime-conversations.jsonl, rotated at +# 32 MB, never expired) omits transcripts and typed or spoken text by default. +# Set to 1 to keep them while debugging; delete the log when you are done. +GEV_VOICE_LOG_CONTENT= OPENAI_HUD_SUMMARY_MODEL=gpt-5-nano # Optional: opt-in per-IP rate limit for the OpenAI cost endpoints # (/api/realtime/token + /api/openai/hud-summary), in requests per minute. diff --git a/CHANGELOG.md b/CHANGELOG.md index 58103d04d..08c7c47d7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,28 @@ # Changelog +- The voice card shows analyst counts ("At least 250,000 aircraft …") with + their scope, feed window and caveats, and marks partial answers. +- Voice answers are shorter and fill slow work instead of going silent. Slow + or multi-step requests may open with one short spoken plan; instant ones + just happen. Tool results carry a code-built `say` line and a `display` + card: sources and notes stay on screen, a caveat is spoken only when it + changes the answer, and "at least" and partial answers are always said. A card above the mic shows captions, the plan's steps and the + result. When a tool that reports steps (layers, place lookups, outlines, + nearest-aircraft) is still running after 1.5 s its current step is spoken + once; others say "still working" once after 8 s. A soft + tick plays in push-to-talk. Holding Space while the assistant talks cuts it + off. Each turn writes latency and speech figures to the debug log + (`node scripts/voice-turn-spans.mjs`). Captions of what you say are + counted in the voice cost meter; the debug log leaves out what was said + unless `GEV_VOICE_LOG_CONTENT=1`. +- Point and ask: hold Space with the cursor on an aircraft, ship or spot on + the map and say "what's that", "track that one" or "how many flights + around here". A reticle marks the point and the voice card shows what + "this" meant. In open mic it works when you have just moved the cursor + there. "The second one" picks from the last numbered list. Asking about + bare ground up close gives the assistant a small crop of the map around the + point. + - Public Overpass instances are no longer used by default. Street Traffic roads come from TomTom flow tiles, OpenFreeMap vector tiles, or both, chosen on the layer row (TomTom / OSM / Hybrid) or with `?trafficRoads=`. With a @@ -149,6 +172,7 @@ geocode and admin-boundary fallback answers `region-timeout` after 3 s instead of holding the reply. + - Transit and Directions rows repaint as soon as their data lands again: `refreshLayerStats()` now lives on the layer lifecycle, not only on the compatibility facade. `scripts/qa-radio.mjs` uses it instead of a private @@ -182,6 +206,30 @@ (`src/ui/imagerySplit.js`, `src/maps/imageryComparison.js`), and `MapSourceController.subscribe()` reports every settled map switch. +## Unreleased — voice reaches every layer; honest analyst answers + +- Voice can switch on every shipped data layer (the two scene-driven + Bhote Koshi layers stay scene-only), including transit, Recent + Imagery, wind, the weather layers, cyclones and mapped installations, and + understands their spoken names ("buses", "hurricanes", "military bases"). + One voice manifest (`src/voice/layerManifest.js`) generates the tool enums + and aliases, and a unit test fails when a new layer has no entry. +- `analyst_query` counts up to 250,000 loaded records per layer instead of the + first 2,000, and says when a layer went past that (`complete: false`). It + also answers over bikeshare stations, transit vehicles, rocket launches, + cyclones, ALPR cameras, mapped installations, local ADS-B aircraft and fire perimeters; fire + records carry their sensor (VIIRS or MODIS). +- Unknown fields, operators, units, value types, scope kinds and invalid + centres are refused with the allowed values instead of answering zero or widening to anywhere. A + layer that is off, still loading or unavailable is reported as such, not + as a count of zero; a partly answerable query is marked `partial`. + Distance sorts honour `sortDir`, a written unit such as `altitudeM(ft)` + converts, and large rankings no longer sort the whole set. +- Results carry `lat`/`lon` for each item and a `display` object with the + scope, the feed window (earthquakes and fires cover the last 24 h) and any + caveat for the screen. The model is asked to answer in one short phrase. +- The weather and Recent Imagery panels can be opened by voice. + ## Unreleased — local receiver feeds - The Local ADS-B layer also reads local 1090 MHz and 978 MHz UAT decoder diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 22ce1eb3b..92442b023 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -222,7 +222,7 @@ Consult the linked official advisory for safety decisions. Voice and HUD snapshots reuse the existing feedState classifier. Analyst follow-ups retain their original data provenance; current-view results append provenance without replacing legacy fields. HUD context and deterministic telemetry include non-nominal feed state. -Satellite and local infrastructure layers expose on-demand analyst records through their current factory owners. Analyst counts and ranks explicitly cover only bounded examined loaded records (default 2,000 per new layer, core satellite rows before dense extras); omitted records can change nearest/count and satellite distance is ground distance. Existing tools and result fields remain available. +Analyst records come on demand from each layer's `getAnalystRecords`: flights, military, local ADS-B, vessels, satellites, launches, earthquakes, FIRMS (VIIRS and MODIS, with a `sensor` field), fire perimeters, cyclones, transit, bikeshare, ALPR cameras, mapped installations, datacenters and dams. `analyst_query` counts the whole loaded set (up to 250,000 records per layer; past that `coverage.records` reports `returned`, `total` and `truncated: true` and the result carries `complete: false`). Ranking keeps a top-k and scans in 20,000-row slices that yield to the page and stop when superseded. It refuses unknown fields, operators, units that do not convert (`altitudeM(ft)` converts), wrong-typed values, invalid centres, unknown scope kinds and sort fields with `code` and `allowed`, refuses a follow-up with no previous answer (`NO_RESULT_CONTEXT`) or naming other layers than that answer (`FOLLOW_UP_MISMATCH`), and tells `LAYER_OFF`, `NOT_READY` and `FEED_UNAVAILABLE` apart from a real zero; when only some layers can answer the result is `partial` with `unanswered` ids. The Contacts-window path applies the same filters, ranking and item shape to the panel cohort. A superseded query never replaces follow-up memory. Items carry `lat`/`lon`; the feed window (earthquakes and FIRMS: last 24 h), the precise scope and any caveat sit in `display`, which is on-screen detail rather than something the model recites. "In view" is still a radius around the view centre, not the visible footprint. Satellite distance is ground distance. Keyless terrain tiles retry when Re:Earth throttles them. The browser fetches `terrain.reearth.land/cesium-mesh/ellipsoid/{z}/{x}/{y}.terrain` directly; no @@ -297,6 +297,45 @@ replacement session; audio meters release failed initialization and reject revok frames. Delayed action results and post-capture continuations cannot resume a stopped conversation or send output into a replacement. See [voice ownership](VOICE-OWNERSHIP.md). +Voice speech: `src/voice/speech.js` adds `{ say, display, referents?, schedule? }` +to layer toggles, fly_to_location, frame_overhead, select_nearest_aircraft, +get_entity_context and annotate_map (analyst_query plugs in through the +runner's `speechBuilders`). `get_current_view_state` lists enabled layers only. +The voice card nested in the mic control (`voiceCard.js`) shows captions, plan +steps and the result. `resultDisplay.js` is the one adapter from a result to +the card: builder results show as built, and analyst results show a title +with their scope, window and caveats. Caveats are on the card; the model speaks +one only when it changes the answer, and always says lower bounds and partial +answers. `narration.js` speaks one +code-authored line for the current step of a tool still running at 1.5 s +(out of band, never after the result). Tools that report steps: layer +toggles, place lookups, select_nearest_aircraft and annotate_map (finding, +outlining). Other tools get one "still working" line at 8 s. The +earcon plays in push-to-talk only. A claimed Space hold while the assistant +speaks is a barge-in (WebRTC cancel + audio clear), refused while Radio holds +the speaker (`mayVoiceClaimSpeaker()`). Progress lines stay outside the reply +lifecycle and Radio handoff. `turn.span` debug records carry generation and +playback latency, silence, preamble and word-count figures. Caption +transcription (`OPENAI_REALTIME_TRANSCRIBE_MODEL`, default on) is metered into +the session cost and cap. The debug log omits transcripts and text unless +`GEV_VOICE_LOG_CONTENT=1`. + +Point and ask: `pointerContext.js` reads what the cursor is on. A Space hold +is the pointing gesture: its keydown snapshot (released with the key) is sent +as an inert `pointer_context` item with the target, with reticle and chip. +Open-mic speech and typed turns count only a pointer that moved on the map in +the last 5 s and send only the kind (`pointing:"aircraft"`/`"ground"`); the chip appears when a tool +resolves it. Losing the pointer sends one `target:"none"` item. `deixis.js` resolves `'pointer'` in +track_entity/fly_to_location queries, get_entity_context scope, analyst_query +`scope.kind` and annotate_map targets, plus `referent:n` from the latest +numbered result (`referents.js`, cleared at session end). A new result set (an analyst count) replaces +the numbered list even when it is empty. A reticle marks the +point during the turn and the voice card shows "THIS"/"HERE"; both hide in +Clean-UI and recording. A ground-only pointer lookup at local/city scale sends a ringed 512 px crop in +the one retained image slot, dropped if the camera moved or a newer capture or +turn started; screenX/screenY against the crop are mapped back to the canvas. Harness: `qa-voice-routing.mjs --layer deixis`, +`--layer deixis-live`, `--layer behavior --deixis-only`. + The application shell composes focused state owners for navigation, destination lookup/orbit, Cockpit, visual settings, panel layout, aircraft display and layer bindings. Keyboard/display subscriptions have a separate lifetime; existing @@ -363,9 +402,11 @@ Reference feed construction is exported through `sources/reference`; the cable s Source factories have dedicated `layers//source` exports. ALPR and earthquake record normalization and CCTV source policy no longer pull rendering into source consumers. Catalog construction and voice feed reads use their focused owners; compatibility entries remain available. Settings filesystem hardening lives under `server/standalone/`. Import-direction checks complement export ownership checks; source behavior, settings policy and rendering are unchanged. -Voice controls bind to a protocol-independent session factory. The default WebRTC adapter preserves the existing Realtime connection, push-to-talk, cost controls and radio handoff. Session subscriptions expose state, transcript, action call/result, interruption and completion events; stopping or replacing a session cancels pending actions. Alternate adapters can use the same controls and action runner. +Voice controls bind to a protocol-independent session factory. The default WebRTC adapter preserves the existing Realtime connection, push-to-talk, cost controls and radio handoff. Session subscriptions expose state, transcript, action call/progress/result, interruption, completion and turn-metrics events; stopping or replacing a session cancels pending actions. Alternate adapters can use the same controls and action runner. -Voice action argument schemas have one portable owner under `src/voice/actionSchemas.js`. The Realtime provider builds 29 tools, retaining the existing 28 names and argument contracts apart from the three additive analyst-layer enum values using separate description-only metadata. Description customization cannot replace argument types, enum values or required fields. +Voice action argument schemas have one portable owner under `src/voice/actionSchemas.js`. The Realtime provider builds 30 tools using separate description-only metadata. Description customization cannot replace argument types, enum values or required fields. + +`src/voice/layerManifest.js` declares each catalog layer's voice capabilities: spoken aliases, `toggle`, in-view `context`, `track`, and analyst `query` fields with type and unit, or `noQuery` with the reason. The `set_layer_visibility`, `show_data_layers_menu`, `get_entity_context` and `analyst_query` layer enums, the runner's alias table and the analyst field hints are generated from it. `src/voice/layerManifest.test.mjs` fails when a catalog layer is neither listed nor named in `VOICE_OFF_LAYERS` (today only the two scene-driven Bhote Koshi layers). Global Context is listed with `toggle: false`; `set_context_mode` reaches it. `set_panel_open` also opens `weather-panel` and `recent-imagery-panel`. Portable source exports provide Radio Browser station normalization, CCTV feed types and regional records independently of HTTP middleware. Radio normalization accepts an explicit URL policy; the standalone directory retains its existing HTTPS rules. Tile coordinate validation accepts explicit zoom bounds with unchanged traffic defaults. Cache, request and rendering owners remain unchanged. @@ -408,7 +449,10 @@ clears caches and removes map-stack listeners. Standalone setup supplies the existing service clients. HUD summaries, regional brief, cockpit weather, location framing and annotation boundaries receive their service instances explicitly. Compatibility entrypoints retain defaults for direct callers; normal startup does -not change global service slots. Analyst follow-up memory belongs to its voice runner. +not change global service slots. Analyst follow-up memory belongs to its voice +runner and ends with the voice session: stopping voice invalidates writes still in +flight and forgets the last answer, the pointer and the numbered list; removing +voice disposes the runner. Application startup constructs fresh layer instances from explicit source objects. Standalone composition selects the existing providers. Controls, launch orbit @@ -3849,7 +3893,7 @@ and unreachable upstream (502/504) separately from road geometry. - **Reliability**: tool-call dedupe (2.5s window across call/item/args keys); response-create queueing that respects active responses and defers follow-ups when the user starts speaking; per-tool follow-up instructions so the agent confirms only what actually happened (zoom confirms only on `ok=true`). - **Aircraft identity honesty:** “What is this aircraft?” reads callsign, operator, registration, type, and route only from the selected contact context. Missing operator, route, or type enrichment is named explicitly rather than silently omitted or inferred from the callsign. - **Diagnostics**: every client/server event is posted to `/api/realtime/debug-log` and appended to `.gev-logs/realtime-conversations.jsonl` (gitignored) with secret/image redaction; last 30 errors persist in `localStorage` (`gev-realtime-errors`); `window.__gevVoiceCommands.getDiagnostics()` in the console. -- **Counting semantics ("near")** — a CONTRACT; new count-bearing tool work inherits it. Three honest numbers exist for one question: the Contacts cohort (250 km around the subject, what the panel shows), `analyst_query`'s count of _currently-loaded_ records for the requested scope, and the layer-wide loaded total in `coverage.layersQueried[].records`. They diverge legitimately — the flights layer loads by viewport, so after a camera dive the loaded set can hold a fraction of the cohort (field case: panel 42, analyst 8). The contract: +- **Counting semantics ("near")** — a CONTRACT; new count-bearing tool work inherits it. Three honest numbers exist for one question: the Contacts cohort (250 km around the subject, what the panel shows), `analyst_query`'s count of _currently-loaded_ records for the requested scope, and the layer-wide loaded total in `coverage.layersQueried[].total`. They diverge legitimately — the flights layer loads by viewport, so after a camera dive the loaded set can hold a fraction of the cohort (field case: panel 42, analyst 8). The contract: 1. **Contacts ACTIVE** → "near / nearby / how many aircraft" means the **Contacts window** — the panel's numbers, spoken verbatim. Mechanism: `contactsWindow` (`{centeredOn, radiusKm, flights, military, vessels}`), carried by both `analyst_query` and `get_current_view_state`, derived by `contactsWindowFromSnapshot()` from the same snapshot the panel renders so the two cannot drift. A cohort whose feed cannot answer reports `'unknown'`, never a confident zero. 2. **Contacts OFF** → "nearby" means **in view**; "near \" means a radius around that place. A radius query with Contacts active and no explicit centre is centred on the **active contact**, not the camera. 3. **Every count names its scope in words** — "42 in your window", "8 in view", "about 30 within 250 km of Austin" — never a bare number. `analyst_query` returns `scopeLabel` so this is mechanical. Two different numbers with named scopes are not a contradiction. diff --git a/docs/VOICE-OWNERSHIP.md b/docs/VOICE-OWNERSHIP.md index d5553df00..4dac83a51 100644 --- a/docs/VOICE-OWNERSHIP.md +++ b/docs/VOICE-OWNERSHIP.md @@ -14,6 +14,12 @@ The session adapter, action runner and backend interfaces remain unchanged. | `realtimeCost.js` | Next-session preferences and the current session's model-bound meter | | `realtimeViewport.js` | One retained image, bounded deletion identities and capture generation | | `realtimeDiagnostics.js` | Bounded error history and sanitized optional diagnostics | +| `turnMetrics.js` | Per-turn latency and speech spans written to the debug log | +| `narration.js` | Progress lines and the earcon while tools run; spoken through the turns owner | +| `pointerContext.js` | Pointer position, dwell pick and the current turn's pointer snapshot (session-scoped) | +| `referents.js` | Numbered items from the latest result for `referent:n` (session-scoped) | +| `resultDisplay.js` | What the voice card shows for a result and which items it numbers (read by the card and `referents.js`) | +| `gevActions.js` runner | Analyst follow-up memory (session: `resetConversation()` on stop, `dispose()` on removal) | Pure input policy, preferences and protocol-response policy have separate modules. Owners receive named readers, operations or focused collaborators. Cross-owner @@ -29,7 +35,7 @@ Action and post-capture continuations retain their conversation identity before publishing output, queueing a response or changing status. Behavioral invariants remain in force: a 500 ms Space hold claims push-to-talk; -short control taps and text entry remain native; a click-started session stays +short control taps and text entry remain native; a claimed hold while the assistant speaks interrupts it (never during a Radio handoff); a click-started session stays open-mic. Every accepted or refused tool call receives its terminal output once. New typed input supersedes old intent. Radio waits for the spoken confirmation and verified muted playback. A live cost meter stays bound to its negotiated model, diff --git a/scripts/package-boundaries.json b/scripts/package-boundaries.json index 276b9e0db..5e27c4db7 100644 --- a/scripts/package-boundaries.json +++ b/scripts/package-boundaries.json @@ -282,6 +282,7 @@ "src/data/installationFeedback.js", "src/voice/voiceCost.js", "src/voice/actionSchemas.js", + "src/voice/layerManifest.js", "server/providers/openai/toolDescriptions.js", "src/sources/rateLimit.js" ], @@ -649,6 +650,7 @@ "src/ui/styles/command-dock-trays.css", "src/ui/styles/command-dock-sliding.css", "src/ui/styles/voice-cost.css", + "src/ui/styles/voice-card.css", "src/ui/styles/first-run.css", "src/ui/styles/provider-settings.css", "src/ui/styles/weather.css", @@ -1285,7 +1287,19 @@ "src/voice/realtimeFacade.js", "src/voice/realtimeRadio.js", "src/voice/realtimeTurns.js", - "src/voice/realtimeConnection.js" + "src/voice/realtimeConnection.js", + "src/voice/turnMetrics.js", + "src/voice/speechLint.js", + "src/voice/voiceCard.js", + "src/voice/voiceCardPresentation.js", + "src/voice/resultDisplay.js", + "src/voice/referents.js", + "src/voice/layerManifest.js", + "src/voice/speech.js", + "src/voice/narration.js", + "src/voice/earcon.js", + "src/voice/pointerContext.js", + "src/voice/pointerReticle.js" ], "external": [] }, @@ -1385,6 +1399,7 @@ "src/data/aisLiveVessels.js", "src/data/alprCameras.js", "src/data/analystEngine.js", + "src/voice/layerManifest.js", "src/data/bikeshare.js", "src/data/cctv.js", "src/data/cctvCards.js", @@ -1851,6 +1866,14 @@ "src/voice/commands.js", "src/voice/control.js", "src/voice/gevActions.js", + "src/voice/speech.js", + "src/voice/earcon.js", + "src/voice/narration.js", + "src/voice/turnMetrics.js", + "src/voice/speechLint.js", + "src/voice/voiceCard.js", + "src/voice/voiceCardPresentation.js", + "src/voice/resultDisplay.js", "src/voice/gevRealtime.js", "src/voice/layerSummary.js", "src/voice/realtimeBackend.js", @@ -2014,6 +2037,12 @@ "src/maps/imageryHost.js", "src/ui/recentImagery.js", "src/ui/imageryBoxTool.js", + "src/voice/pointerContext.js", + "src/voice/pointerReticle.js", + "src/voice/deixis.js", + "src/voice/referents.js", + "src/voice/pointerPick.js", + "src/data/pickedContext.js", "src/layers/alpr/tileRecords.js", "src/sources/capability.js", "src/sources/openFreeMap.js", @@ -2736,7 +2765,11 @@ }, "voice-action-schemas": { "exports": ["./voice/action-schemas"], - "modules": ["src/voice/actionSchemas.js", "src/hudLayoutPolicy.js"], + "modules": [ + "src/voice/actionSchemas.js", + "src/voice/layerManifest.js", + "src/hudLayoutPolicy.js" + ], "external": [] }, "voice-session": { @@ -2749,7 +2782,14 @@ "modules": [ "src/voice/sessionCommands.js", "src/voice/session.js", - "src/voice/control.js" + "src/voice/control.js", + "src/voice/voiceCard.js", + "src/voice/voiceCardPresentation.js", + "src/voice/resultDisplay.js", + "src/voice/referents.js", + "src/voice/layerManifest.js", + "src/voice/speech.js", + "src/voice/pointerReticle.js" ], "external": [] }, diff --git a/scripts/qa-voice-bench.mjs b/scripts/qa-voice-bench.mjs new file mode 100644 index 000000000..f2f0f973b --- /dev/null +++ b/scripts/qa-voice-bench.mjs @@ -0,0 +1,411 @@ +#!/usr/bin/env node +/** + * qa-voice-bench.mjs — provider-neutral TEXT-mode voice-routing benchmark. + * + * Every provider gets the same prompt: production's realtimeInstructions() + * and GEV_REALTIME_TOOLS (the Realtime adapter uses the exact session body the + * token handler mints). Two suites, reported separately: + * core the qa-voice-routing phrase table, graded with its semantics + * (plus benchmark-only typed-argument checks) + * coverage the newer-layers coverage probe, graded correct / + * honest-refusal / hallucinated-capability / wrong-tool + * dialogue two-turn dialogues in one context; turn 2 depends on turn 1 + * capability area handles / imagery / OSM / point-and-ask phrases (and two + * dialogues) for the tools added after 4b56d0e9 + * Coverage rows are graded twice: v1 (original rubric, 4b56d0e9 tool schema) + * and v2 (rubric for the current tool surface). + * + * Methodology: a fresh context per phrase (new Realtime/Live session, or a + * stateless chat request); stub tool results; continuations until the model + * stops calling tools (max 4). No app page is involved. + * + * Usage: + * node scripts/qa-voice-bench.mjs --provider openai-realtime --model gpt-realtime-2 --reps 2 + * node scripts/qa-voice-bench.mjs --provider gemini-live --model gemini-3.8-live + * node scripts/qa-voice-bench.mjs --provider ollama --model gemma4:e4b + * node scripts/qa-voice-bench.mjs --provider openai-chat --base http://localhost:1234/v1 --model + * node scripts/qa-voice-bench.mjs --summarize --out qa-shots/voice-bench/ + * Common flags: --out (reuse one dir to collect several models), + * --suite core,coverage --only --budget + * --variant