diff --git a/.env.example b/.env.example index bf951495d..2b7ff95ab 100644 --- a/.env.example +++ b/.env.example @@ -67,6 +67,14 @@ OPENAI_REALTIME_REASONING_EFFORT=low # Keep only a short conversational window. Current map state is fetched live. OPENAI_REALTIME_CONTEXT_TOKENS=3000 OPENAI_REALTIME_CONTEXT_RETENTION=0.5 +# Voice-card captions of what you said. Billed separately by OpenAI and +# counted in the session cost meter and cap; off disables them. +OPENAI_REALTIME_TRANSCRIBE_MODEL=gpt-4o-mini-transcribe +# The voice debug log (.gev-logs/realtime-conversations.jsonl, rotated at +# 32 MB, never expired) omits transcripts, typed or spoken text, and tool +# arguments/results (including serialized outputs) by default. Set to 1 to +# keep them while debugging; delete the log when you are done. +GEV_VOICE_LOG_CONTENT= OPENAI_HUD_SUMMARY_MODEL=gpt-5-nano # Per-IP rate limit for the OpenAI cost endpoints # (/api/realtime/token + /api/openai/hud-summary), in requests per minute. diff --git a/CHANGELOG.md b/CHANGELOG.md index 7bf5acbf5..f27b8d911 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,62 @@ # Changelog +- Voice revisits numbered analyst records through current layer snapshots and + preserves resolved pin coordinates. Requested analyst lists and rankings can + speak returned items alongside the exact count and coverage caveats. Failed + actions settle the voice card without reviving cancelled turns. Coverage + benchmarks reject invalid calls before accepting semantic matches. + +- Street Level is included in the voice layer manifest for visibility toggles; + imagery is not exposed as countable analyst records. + +- Voice debug logs omit complete tool-result bodies and serialized function + outputs by default, including location queries and generated spoken replies. + `GEV_VOICE_LOG_CONTENT=1` retains them for explicit content debugging. + +- Unqualified Alps searches use the bundled European range for navigation + and annotations instead of a same-named peak returned by a geocoder. + Coordinates, supplied presets and geographically qualified place names + keep their precedence; annotation proximity and containment guards remain. +- Contacts analyst follow-ups retain the snapshot's distance ranking rather + than switching to the camera. Requested radii and explicit scopes remain + authoritative, and a request for a list still requires an analyst query. + Analyst speech names stale or degraded feeds and withholds authoritative + counts when unavailable. The existing NOTES disclosure preserves complete, + deduplicated coverage and source caveats, including crowded results. +- The voice card shows analyst counts ("At least 250,000 aircraft …") with + their scope, feed window and caveats, and marks partial answers. Analyst + speech uses the same deterministic headline, preserves lower bounds and says + when an answer is partial. Follow-ups retain unanswered-layer coverage, and + keep the original scope label when they narrow a remembered result. + Contacts-window answers read one immutable panel snapshot, mark a 20,000-row + layer cap as a lower bound, and commit that displayed cohort only after + cancellation checks pass. Concurrent analyst calls from one model response + remain independent; a stopped or replaced user turn still cancels them. +- Voice answers are shorter and fill slow work instead of going silent. Slow + or multi-step requests may open with one short spoken plan; instant ones + just happen. Tool results carry a code-built `say` line and a `display` + card: sources and notes stay on screen, a caveat is spoken only when it + changes the answer, and "at least" and partial answers are always said. A card above the mic shows captions, the plan's steps and the + result. When a tool that reports steps (layers, place lookups, outlines, + nearest-aircraft) is still running after 1.5 s its current step is spoken + once; others say "still working" once after 8 s. A soft + tick plays in push-to-talk. Holding Space while the assistant talks cuts it + off. Each turn writes latency and speech figures to the debug log + (`node scripts/voice-turn-spans.mjs`). Captions of what you say are + counted in the voice cost meter; the debug log leaves out what was said + unless `GEV_VOICE_LOG_CONTENT=1`. +- Point and ask: hold Space with the cursor on an aircraft, ship or spot on + the map and say "what's that", "track that one" or "how many flights + around here". A reticle marks the point and the voice card shows what + "this" meant. In open mic it works when you have just moved the cursor + there. "The second one" picks from the last numbered list. Asking about + "the last one" resolves to the last of the five rows actually shown, and + referent-only tracking calls are accepted while empty targets are rejected. + Asking about + bare ground up close gives the assistant a small crop of the map around the + point. A slow render gets one bounded second frame request, while hidden, + stale, moved-camera, invalid and oversized captures still fail closed. + ## [Unreleased] - Keep the globe panel working when an MCP client starts more than one @@ -319,6 +376,32 @@ (`src/ui/imagerySplit.js`, `src/maps/imageryComparison.js`), and `MapSourceController.subscribe()` reports every settled map switch. +## Unreleased β€” voice reaches every layer; honest analyst answers + +- Voice can switch on every shipped data layer (the two scene-driven + Bhote Koshi layers stay scene-only), including transit, Recent + Imagery, wind, the weather layers, cyclones and mapped installations, and + understands their spoken names ("buses", "hurricanes", "military bases"). + One voice manifest (`src/voice/layerManifest.js`) generates the tool enums + and aliases, and a unit test fails when a new layer has no entry. +- `analyst_query` counts up to 250,000 loaded records per layer instead of the + first 2,000, and says when a layer went past that (`complete: false`). It + also answers over bikeshare stations, transit vehicles, rocket launches, + cyclones, ALPR cameras, mapped installations, local ADS-B aircraft and fire perimeters; fire + records carry their sensor (VIIRS or MODIS). +- Unknown fields, operators, units, value types, scope kinds and invalid + centres are refused with the allowed values instead of answering zero or widening to anywhere. A + layer that is off, still loading or unavailable is reported as such, not + as a count of zero; a partly answerable query is marked `partial`. + Distance sorts honour `sortDir`, a written unit such as `altitudeM(ft)` + converts, and large rankings no longer sort the whole set. +- Results carry `lat`/`lon` for each item and a `display` object with the + scope, the feed window (earthquakes and fires cover the last 24 h) and any + caveat for the screen. The model is asked to answer in one short phrase. +- The weather and Recent Imagery panels can be opened by voice. + +## Unreleased β€” local receiver feeds + ### Local receiver feeds - The Local ADS-B layer also reads local 1090 MHz and 978 MHz UAT decoder @@ -461,6 +544,7 @@ Add feed provenance to analyst/view answers and HUD context while retaining existing response fields and runner ownership (Matt Van Horn, #347). Analyst records for loaded satellites, datacenters and dams, with explicit bounded count/rank coverage (Matt Van Horn, #351). + - New Fire Perimeters layer (Events group): live NIFC WFIGS interagency wildfire incident perimeters as ground-clamped polygons with a containment-colored fire line, refreshed every 5 minutes from the public @@ -737,7 +821,6 @@ of current runtime behavior, see [`docs/CURRENT-STATE.md`](docs/CURRENT-STATE.md ### Other changes - - Add ECMWF IFS model selection to Wind (#464, thanks @beneduzi), with model-scoped forecast-step caches, cancellation of replaced requests, and separate issue/valid timestamps. - Add bounded Director feature actions with accessible controls, explicit camera/layer admission and cancellation; restore pack geometry on same-shot seek. Preserve existing scenes and content attribution. @@ -1546,4 +1629,3 @@ represent previously published GitHub Releases. ## [0.1.0] β€” 2026-02-09 - Initial project version. - diff --git a/README.md b/README.md index 1c2f375b5..5a4ab8b24 100644 --- a/README.md +++ b/README.md @@ -245,7 +245,7 @@ Click **GEV MIC**, grant the microphone, and just talk. This is more than a voic - **🎬 Cinematic framing.** _"Show me the planes overhead"_ pulls the camera back, angles it, and frames the live traffic like a director. - **πŸ”’ Honest and secure.** The agent only confirms actions that succeeded. Your `OPENAI_API_KEY` never touches the browser; the client only gets a short-lived session token. -Twenty-nine tools, four jobs β€” the commands below come straight from the product's voice test suite and tool playbook: +Thirty tools, four jobs β€” the commands below come straight from the product's voice test suite and tool playbook: **πŸŽ₯ Direct it** β€” drone-operator camera verbs: @@ -395,7 +395,7 @@ src/ β”œβ”€β”€ hud.js # Intelligence HUD + AI scene summary β”œβ”€β”€ keySetup.js # POWER UP panel β€” in-app provider keys (dev server only) β”œβ”€β”€ mapStackController.js # Basemap switching β€” Google 3D / Esri / OSM / ion stacks -β”œβ”€β”€ voice/ # OpenAI Realtime session + 29 voice tools +β”œβ”€β”€ voice/ # OpenAI Realtime session + 30 voice tools β”œβ”€β”€ layers/ # Layer components β€” weather, wind, cyclones, transit, ALPR, … β”œβ”€β”€ data/ # One module per layer + orchestration + context store β”‚ β”œβ”€β”€ iconOrientation.js # Screen-projected headings + horizon cull diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index d51be986b..730586df9 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -278,7 +278,7 @@ Consult the linked official advisory for safety decisions. Voice and HUD snapshots reuse the existing feedState classifier. Analyst follow-ups retain their original data provenance; current-view results append provenance without replacing legacy fields. HUD context and deterministic telemetry include non-nominal feed state. -Satellite and local infrastructure layers expose on-demand analyst records through their current factory owners. Analyst counts and ranks explicitly cover only bounded examined loaded records (default 2,000 per new layer, core satellite rows before dense extras); omitted records can change nearest/count and satellite distance is ground distance. Existing tools and result fields remain available. +Analyst records come on demand from each layer's `getAnalystRecords`: flights, military, local ADS-B, vessels, satellites, launches, earthquakes, FIRMS (VIIRS and MODIS, with a `sensor` field), fire perimeters, cyclones, transit, bikeshare, ALPR cameras, mapped installations, datacenters and dams. `analyst_query` counts the whole loaded set (up to 250,000 records per layer; past that `coverage.records` reports `returned`, `total` and `truncated: true` and the result carries `complete: false`). Ranking keeps a top-k and scans in 20,000-row slices that yield to the page and stop when superseded. It refuses unknown fields, operators, units that do not convert (`altitudeM(ft)` converts), wrong-typed values, invalid centres, unknown scope kinds and sort fields with `code` and `allowed`, refuses a follow-up with no previous answer (`NO_RESULT_CONTEXT`) or naming other layers than that answer (`FOLLOW_UP_MISMATCH`), and tells `LAYER_OFF`, `NOT_READY` and `FEED_UNAVAILABLE` apart from a real zero; when only some layers can answer the result is `partial` with `unanswered` ids. The Contacts-window path applies the same filters, ranking and item shape to one immutable panel snapshot. Each aircraft layer retains at most 20,000 rows; reaching that bound sets `complete: false` and the spoken/card count becomes β€œAt least”, never an exact claim. A superseded query never replaces follow-up memory, while sibling analyst calls from one assistant response do not cancel one another. Items carry `lat`/`lon`; the feed window (earthquakes and FIRMS: last 24 h), the precise scope and any caveat sit in `display`, which is on-screen detail rather than something the model recites. "In view" is still a radius around the view centre, not the visible footprint. Satellite distance is ground distance. Keyless terrain tiles retry when Re:Earth throttles them. The browser fetches `terrain.reearth.land/cesium-mesh/ellipsoid/{z}/{x}/{y}.terrain` directly; no @@ -353,6 +353,70 @@ replacement session; audio meters release failed initialization and reject revok frames. Delayed action results and post-capture continuations cannot resume a stopped conversation or send output into a replacement. See [voice ownership](VOICE-OWNERSHIP.md). +Contacts nearest-aircraft follow-ups retain the snapshot's distance basis even +after the camera moves. An explicit new scope replaces that basis. Provider +instructions require an analyst query for requested lists and use the requested +radius rather than substituting the panel's fixed window. Analyst speech carries +stale/degraded feed status and does not speak an authoritative unavailable count. +The existing NOTES disclosure deduplicates coverage/source caveats and retains +their full text in at most six rows, grouping analyst overflow in the final row; +ordinary tool-card limits are unchanged. + +Voice speech: `src/voice/speech.js` adds `{ say, display, referents?, schedule? }` +to layer toggles, fly_to_location, frame_overhead, select_nearest_aircraft, +get_entity_context, annotate_map and analyst_query. Analyst speech and the card +share one deterministic count/scope headline; lower bounds use "At least" and +partial answers name the unanswered layers. `get_current_view_state` lists enabled layers only. +The voice card nested in the mic control (`voiceCard.js`) shows captions, plan +steps and the result. `resultDisplay.js` is the one adapter from a result to +the card: builder results show as built, and analyst results show a title +with their scope, window and caveats. Caveats are on the card; the model speaks +one only when it changes the answer, and always says lower bounds and partial +answers. `narration.js` speaks one +code-authored line for the current step of a tool still running at 1.5 s +(out of band, never after the result). Tools that report steps: layer +toggles, place lookups, select_nearest_aircraft and annotate_map (finding, +outlining). Other tools get one "still working" line at 8 s. The +earcon plays in push-to-talk only. A claimed Space hold while the assistant +speaks is a barge-in (WebRTC cancel + audio clear), refused while Radio holds +the speaker (`mayVoiceClaimSpeaker()`). Progress lines stay outside the reply +lifecycle and Radio handoff. `turn.span` debug records carry generation and +playback latency, silence, preamble and word-count figures. Caption +transcription (`OPENAI_REALTIME_TRANSCRIBE_MODEL`, default on) is metered into +the session cost and cap. The debug log omits transcripts, text, tool arguments +and complete tool-result bodies (including serialized function outputs) unless +`GEV_VOICE_LOG_CONTENT=1`. +Tool names, call IDs, protocol event metadata and usage remain available. +Numbered analyst targets resolve from current layer records, including layers +without pick/context lookup; numbered annotations retain their resolved coordinates. +Requested analyst lists and rankings may speak up to three returned items and +values in result order alongside the count and its caveats. A rejected current +action emits a failure result to settle its card; cancelled turns remain silent. +Coverage grading validates calls before accepting an expected capability match. +The voice manifest includes Street Level visibility toggles; its imagery has no +analyst query or entity-context records. + +Point and ask: `pointerContext.js` reads what the cursor is on. A Space hold +is the pointing gesture: its keydown snapshot (released with the key) is sent +as an inert `pointer_context` item with the target, with reticle and chip. +Open-mic speech and typed turns count only a pointer that moved on the map in +the last 5 s and send only the kind (`pointing:"aircraft"`/`"ground"`); the chip appears when a tool +resolves it. Losing the pointer sends one `target:"none"` item. `deixis.js` resolves `'pointer'` in +track_entity/fly_to_location queries, get_entity_context scope, analyst_query +`scope.kind` and annotate_map targets, plus `referent:n` from the latest +numbered result (`referents.js`, cleared at session end). A new result set (an analyst count) replaces +the numbered list even when it is empty. Only the five rows numbered on the +card are resolvable; `-1` means the fifth/last visible row. `track_entity` +accepts either a named query or a referent, while an empty target is rejected. A reticle marks the +point during the turn and the voice card shows "THIS"/"HERE"; both hide in +Clean-UI and recording. A ground-only pointer lookup at local/city scale sends a ringed 512 px crop in +the one retained image slot, dropped if the camera moved or a newer capture or +turn started. The pointer path makes at most two bounded fresh-frame requests so +a slow render loop can recover without accepting a stale frame; a missing camera +identity, expired snapshot, invalid geometry, black frame or oversized payload +still fails closed. screenX/screenY against the crop are mapped back to the canvas. Harness: `qa-voice-routing.mjs --layer deixis`, +`--layer deixis-live`, `--layer behavior --deixis-only`. + The application shell composes focused state owners for navigation, destination lookup/orbit, Cockpit, visual settings, panel layout, aircraft display and layer bindings. Keyboard/display subscriptions have a separate lifetime; existing @@ -419,9 +483,11 @@ Reference feed construction is exported through `sources/reference`; the cable s Source factories have dedicated `layers//source` exports. ALPR and earthquake record normalization and CCTV source policy no longer pull rendering into source consumers. Catalog construction and voice feed reads use their focused owners; compatibility entries remain available. Settings filesystem hardening lives under `server/standalone/`. Import-direction checks complement export ownership checks; source behavior, settings policy and rendering are unchanged. -Voice controls bind to a protocol-independent session factory. The default WebRTC adapter preserves the existing Realtime connection, push-to-talk, cost controls and radio handoff. Session subscriptions expose state, transcript, action call/result, interruption and completion events; stopping or replacing a session cancels pending actions. Alternate adapters can use the same controls and action runner. +Voice controls bind to a protocol-independent session factory. The default WebRTC adapter preserves the existing Realtime connection, push-to-talk, cost controls and radio handoff. Session subscriptions expose state, transcript, action call/progress/result, interruption, completion and turn-metrics events; stopping or replacing a session cancels pending actions. Alternate adapters can use the same controls and action runner. -Voice action argument schemas have one portable owner under `src/voice/actionSchemas.js`. The Realtime provider builds 29 tools, retaining the existing 28 names and argument contracts apart from the three additive analyst-layer enum values using separate description-only metadata. Description customization cannot replace argument types, enum values or required fields. +Voice action argument schemas have one portable owner under `src/voice/actionSchemas.js`. The Realtime provider builds 30 tools using separate description-only metadata. Description customization cannot replace argument types, enum values or required fields. + +`src/voice/layerManifest.js` declares each catalog layer's voice capabilities: spoken aliases, `toggle`, in-view `context`, `track`, and analyst `query` fields with type and unit, or `noQuery` with the reason. The `set_layer_visibility`, `show_data_layers_menu`, `get_entity_context` and `analyst_query` layer enums, the runner's alias table and the analyst field hints are generated from it. `src/voice/layerManifest.test.mjs` fails when a catalog layer is neither listed nor named in `VOICE_OFF_LAYERS` (today only the two scene-driven Bhote Koshi layers). Global Context is listed with `toggle: false`; `set_context_mode` reaches it. `set_panel_open` also opens `weather-panel` and `recent-imagery-panel`. Portable source exports provide Radio Browser station normalization, CCTV feed types and regional records independently of HTTP middleware. Radio normalization accepts an explicit URL policy; the standalone directory retains its existing HTTPS rules. Tile coordinate validation accepts explicit zoom bounds with unchanged traffic defaults. Cache, request and rendering owners remain unchanged. @@ -464,7 +530,18 @@ clears caches and removes map-stack listeners. Standalone setup supplies the existing service clients. HUD summaries, regional brief, cockpit weather, location framing and annotation boundaries receive their service instances explicitly. Compatibility entrypoints retain defaults for direct callers; normal startup does -not change global service slots. Analyst follow-up memory belongs to its voice runner. +not change global service slots. Analyst follow-up memory belongs to its voice +runner and ends with the voice session: stopping voice invalidates writes still in +flight and forgets the last answer, the pointer and the numbered list; removing +voice disposes the runner. + +Analyst result memory is committed only after any Contacts-window projection +finishes and the call is still current. Cancelled or superseded queries cannot +return success or replace follow-up memory; Contacts follow-ups filter the same +cohort that was shown, and partial/unanswered coverage remains attached when a +remembered result is narrowed. +The narrowed answer retains the remembered scope label and detail, including +the Contacts-window attribution, unless the follow-up supplies a new scope. Application startup constructs fresh layer instances from explicit source objects. Standalone composition selects the existing providers. Controls, launch orbit @@ -1474,10 +1551,10 @@ This is the current runtime/source-of-truth snapshot for the project. > samples, `enabling` throughout, panel non-numeric) and across a failing > one. Its `status: 'idle'` is not what saves it β€” the lifecycle is; the > module has no `loading` status at all. -> Any new dependency that can be slow must report `loading` and `lastUpdate` -> honestly for the contract to hold, and must be pinned against the shape its -> own module really returns β€” a fixture that invents a status the module cannot -> emit guards nothing (that is exactly how a hole here survived a green suite). +> Any new dependency that can be slow must report `loading` and `lastUpdate` +> honestly for the contract to hold, and must be pinned against the shape its +> own module really returns β€” a fixture that invents a status the module cannot +> emit guards nothing (that is exactly how a hole here survived a green suite). > - **A held ground snap is dropped when measured ground contradicts it > (2026-08-23):** the WARM hold described below answers on DISTANCE travelled, > which is only a proxy for whether the value still describes the ground. A @@ -2872,7 +2949,7 @@ test:track` 43 tracking invariants Β· headless QA harnesses under > ISS pass prediction, and per-layer data attribution. Gate at close: unit 98/98, build clean, > track 19/19, + five QA harnesses (heading 16/16, sprites 9/9, cctv 5/5, failstate 5/5, > attribution 18/18). New modules: `src/data/{motionModel,aircraftMeta,aircraftClass,aircraftIcons,issPass,routePlausible,dataCredits}.js`. -> The live runtime now declares 29 voice tools; the 17β†’20 count above is retained only as milestone history. +> The live runtime now declares 30 voice tools; the 17β†’20 count above is retained only as milestone history. ## Canonical Docs Order @@ -3947,7 +4024,7 @@ and unreachable upstream (502/504) separately from road geometry. - **Token flow**: browser fetches a short-lived client secret from `/api/realtime/token`; the Vite middleware holds `OPENAI_API_KEY` and posts the full session config (instructions, tool schemas, VAD, truncation) to `api.openai.com/v1/realtime/client_secrets`. SDP exchange goes directly to `api.openai.com/v1/realtime/calls` with the ephemeral token. - **Session defaults** (env-tunable): model `gpt-realtime-2` (or `gpt-realtime-2.1-mini` when the MINI tier is selected β€” see the model-tier entry below), voice `marin`, reasoning effort `low`, semantic VAD with low eagerness, no response interruption, context window truncated to ~3,000 post-instruction tokens with 0.5 retention ratio β€” the conversational window stays short because map state is fetched live per turn. -- **Twenty-nine tools** (argument schemas in `src/voice/actionSchemas.js`, served with their descriptions by `server/providers/openai/tools.js`, executed client-side in `src/voice/gevActions.js`): `fly_to_location`, `select_nearest_aircraft`, `adjust_camera_zoom`, `zoom_to_globe`, `set_layer_visibility`, `show_data_layers_menu`, `set_panel_open`, `set_visual_style`, `get_entity_context`, `get_current_view_state`, `set_hud`, `set_detection`, `set_map_stack`, `set_post_processing`, `control_scene`, `control_cctv`, `set_context_mode`, `control_cockpit`, `control_radio`, `track_entity`, `stop_tracking`, `frame_overhead`, `annotate_map`, `clear_annotations`, `move_camera`, `fly_route`, `analyst_query`, `next_iss_pass`, and `next_satellite_pass`. +- **Thirty tools** (argument schemas in `src/voice/actionSchemas.js`, served with their descriptions by `server/providers/openai/tools.js`, executed client-side in `src/voice/gevActions.js`): `fly_to_location`, `select_nearest_aircraft`, `adjust_camera_zoom`, `zoom_to_globe`, `set_layer_visibility`, `show_data_layers_menu`, `set_panel_open`, `set_visual_style`, `get_entity_context`, `get_current_view_state`, `set_hud`, `set_cyber_sonar`, `set_detection`, `set_map_stack`, `set_post_processing`, `control_scene`, `control_cctv`, `set_context_mode`, `control_cockpit`, `control_radio`, `track_entity`, `stop_tracking`, `frame_overhead`, `annotate_map`, `clear_annotations`, `move_camera`, `fly_route`, `analyst_query`, `next_iss_pass`, and `next_satellite_pass`. > **Reading `npm test` totals:** the count depends on the Node major. The two > GC-bracketed allocation microbenchmarks (`src/data/focusAllocations.test.mjs` @@ -3969,7 +4046,7 @@ and unreachable upstream (502/504) separately from road geometry. - **Analyst β†’ track handoff carries a key, not just a label** (2026-08-21). `analyst_query` items include `icao24`/`mmsi` alongside the display `id`, and contact lookup uses the shared tiered ranking in `src/data/contactMatch.js`: hex exact β†’ callsign exact β†’ registration exact β†’ callsign prefix β†’ registration prefix β†’ callsign substring β†’ registration substring. The tiers keep an exact callsign ahead of a colliding registration regardless of feed order, and registrations compare separator-insensitively (`G-ABCD`/`GABCD`, `05-8152`/`058152`, `N123AB`/`N-123AB`). - **A typed command supersedes the turn it interrupts** (2026-08-21). `sendTextCommand` defers its `response.create` behind an active response instead of colliding with it, marks that response superseded so a late function call from it is refused rather than dispatched, and drops the old turn's queued follow-up. A refused call is still ANSWERED β€” a terminal `{ok:false, superseded:true}` `function_call_output` β€” because an unanswered `function_call` strands a pending call and deadlocks the model; the refusal creates no response of its own. A burst of typed commands coalesces into one response while keeping both conversation items. - **Subject reconciliation across satellite catalog rebuilds** (2026-08-21). A dense↔core toggle or TLE refresh clears and repopulates the catalog, so the published subject is re-resolved against the new satrec; a subject that did NOT survive releases the slot. The per-frame refresh cannot do this itself β€” `_getTrackedFramePosition` returns early once the satellite has no catalog entry β€” so the reconcile runs at rebuild completion. An empty catalog is a rebuild in flight, not a disappearance. - - **One aircraft-proximity engine** (2026-08-22). `collectAircraftProximityWindow` (`src/data/militaryAwareness.js`) is the single computation behind both the Contacts panel window and the voice analyst's entity-centred "how many nearby", so the panel readout and the spoken count for one centre are identical by construction. **Invariant: do not re-derive a proximity count anywhere else.** They diverged before because the panel read live billboard positions (20,000 cap) while the analyst used last-fix coordinates over a 2,000-record slice β€” 111 on screen, 15 spoken. Explicit regions and arbitrary points deliberately keep the general record engine. Entity-centred results carry `window: {engine:'contacts-window', centeredOn, radiusKm, flights, military, aircraft}`. + - **One aircraft-proximity snapshot** (2026-08-22; hardened 2026-09-30). `collectAircraftProximityWindow` (`src/data/militaryAwareness.js`) computes the Contacts panel cohort once. The voice analyst reads the exact retained snapshot and evaluation timestamp instead of re-scanning live layers, so the panel readout and spoken count for one centre cannot cross snapshots. **Invariant: do not re-derive a proximity count anywhere else.** Each layer retains 20,000 rows and samples one row beyond the cap; a saturated cohort reports `complete:false`, `truncated:true`, and is spoken as a lower bound. They diverged before because the panel read live billboard positions while the analyst used last-fix coordinates over a 2,000-record slice β€” 111 on screen, 15 spoken. Explicit regions, arbitrary points, and a radius different from the panel radius deliberately keep the general record engine. Entity-centred results carry `window: {engine:'contacts-window', centeredOn, radiusKm, flights, military, aircraft, complete, evaluatedAt}`. - **Centre precedence for nearby asks**: explicit place in the question > Contacts subject (a selected non-contact entity never silently becomes the centre) > an entity the user names > the current view, said aloud. Contacts active with no subject uses the view rather than reading an empty panel. - **Contact-match ties break on hex ascending.** `track_entity` is a mutation fulfilling "follow that one", and the model's observed answer to a non-ok track result is to retry with guesses rather than ask, so the lookup always commits rather than returning an ambiguity. What it owes the caller is stability: hex is unique and always present, so the same query resolves to the same contact for as long as both are loaded. - **Visual grounding**: at `local` view scale with no structured identity, the client captures the Cesium canvas (≀1200px JPEG, black-frame detection, double-render for freshness) and sends it as `input_image` with a strict "do not invent labels" instruction. @@ -3986,8 +4063,8 @@ and unreachable upstream (502/504) separately from road geometry. - **Reliability**: tool-call dedupe (2.5s window across call/item/args keys); response-create queueing that respects active responses and defers follow-ups when the user starts speaking; per-tool follow-up instructions so the agent confirms only what actually happened (zoom confirms only on `ok=true`). - **Aircraft identity honesty:** β€œWhat is this aircraft?” reads callsign, operator, registration, type, and route only from the selected contact context. Missing operator, route, or type enrichment is named explicitly rather than silently omitted or inferred from the callsign. - **Diagnostics**: every client/server event is posted to `/api/realtime/debug-log` and appended to `.gev-logs/realtime-conversations.jsonl` (gitignored) with secret/image redaction; last 30 errors persist in `localStorage` (`gev-realtime-errors`); `window.__gevVoiceCommands.getDiagnostics()` in the console. -- **Counting semantics ("near")** β€” a CONTRACT; new count-bearing tool work inherits it. Three honest numbers exist for one question: the Contacts cohort (250 km around the subject, what the panel shows), `analyst_query`'s count of _currently-loaded_ records for the requested scope, and the layer-wide loaded total in `coverage.layersQueried[].records`. They diverge legitimately β€” the flights layer loads by viewport, so after a camera dive the loaded set can hold a fraction of the cohort (field case: panel 42, analyst 8). The contract: - 1. **Contacts ACTIVE** β†’ "near / nearby / how many aircraft" means the **Contacts window** β€” the panel's numbers, spoken verbatim. Mechanism: `contactsWindow` (`{centeredOn, radiusKm, flights, military, vessels}`), carried by both `analyst_query` and `get_current_view_state`, derived by `contactsWindowFromSnapshot()` from the same snapshot the panel renders so the two cannot drift. A cohort whose feed cannot answer reports `'unknown'`, never a confident zero. +- **Counting semantics ("near")** β€” a CONTRACT; new count-bearing tool work inherits it. Three honest numbers exist for one question: the Contacts cohort (250 km around the subject, what the panel shows), `analyst_query`'s count of _currently-loaded_ records for the requested scope, and the layer-wide loaded total in `coverage.layersQueried[].total`. They diverge legitimately β€” the flights layer loads by viewport, so after a camera dive the loaded set can hold a fraction of the cohort (field case: panel 42, analyst 8). The contract: + 1. **Contacts ACTIVE** β†’ "near / nearby / how many aircraft" means the **Contacts window** β€” the panel's numbers, spoken verbatim. Mechanism: `contactsWindow` (`{centeredOn, radiusKm, flights, military, vessels, complete}`), carried by both `analyst_query` and `get_current_view_state`, derived by `contactsWindowFromSnapshot()` from the same snapshot the panel renders so the two cannot drift. Voice filtering and follow-up memory use that immutable snapshot too. A cohort whose feed cannot answer reports `'unknown'`, never a confident zero; a cohort at the 20,000-row retention bound reports a lower bound. 2. **Contacts OFF** β†’ "nearby" means **in view**; "near \" means a radius around that place. A radius query with Contacts active and no explicit centre is centred on the **active contact**, not the camera. 3. **Every count names its scope in words** β€” "42 in your window", "8 in view", "about 30 within 250 km of Austin" β€” never a bare number. `analyst_query` returns `scopeLabel` so this is mechanical. Two different numbers with named scopes are not a contradiction. 4. **The loaded-data caveat is stated once when relevant**: counts cover loaded data, and the flights layer loads where you look (appended to `coverage.note` for radius/view scopes over viewport-loaded layers). @@ -4060,6 +4137,15 @@ are omitted rather than framing the wrong part of the globe. retrying. Missing capability never synthesizes an outline from a radius. Manual geometry, bundled neighborhoods, Natural Earth regions and bundled state/province/county outlines remain. +- The shared default place search resolves unqualified `Alps`, `the Alps` + and the `Alps mountains` forms from the bundled European Natural Earth + range before network geocoders. Navigation uses its contained anchor and + region viewport; annotations still require the existing proximity and + outline-containment checks. Coordinates and supplied presets keep priority; + geographic qualifiers, local POI names, other ranges and custom geocoder + arrays retain their existing resolution. A failed Alps pack load reports a + search failure rather than caching an unrelated peak as the range, and the + bundled loader retains its normal retry cooldown. - County queries rank eligible US counties and international admin-1 units together, retaining exact aliases and country/state qualifiers. - Bundled administrative outlines (`src/data/adminBoundaries.js`): states and provinces worldwide (Natural Earth admin-1, @@ -4922,7 +5008,6 @@ Temperature uses stronger fixed βˆ’40..50Β°C colors; the underlying 1Β° forecast numeric inspection values are unchanged. No volumetric cloud height or local rain arrival prediction is claimed. NOAA source limits are documented in DATA_SOURCES. - Traffic source refinements: ALPR admits z9-z12 detail only when the view fits 16 tiles, returning explicit zoom guidance before I/O and allowing the next smaller view to load normally. Outside the US/Canada extracts the row reads β€œNo ALPR data for this diff --git a/docs/VOICE-OWNERSHIP.md b/docs/VOICE-OWNERSHIP.md index d5553df00..4dac83a51 100644 --- a/docs/VOICE-OWNERSHIP.md +++ b/docs/VOICE-OWNERSHIP.md @@ -14,6 +14,12 @@ The session adapter, action runner and backend interfaces remain unchanged. | `realtimeCost.js` | Next-session preferences and the current session's model-bound meter | | `realtimeViewport.js` | One retained image, bounded deletion identities and capture generation | | `realtimeDiagnostics.js` | Bounded error history and sanitized optional diagnostics | +| `turnMetrics.js` | Per-turn latency and speech spans written to the debug log | +| `narration.js` | Progress lines and the earcon while tools run; spoken through the turns owner | +| `pointerContext.js` | Pointer position, dwell pick and the current turn's pointer snapshot (session-scoped) | +| `referents.js` | Numbered items from the latest result for `referent:n` (session-scoped) | +| `resultDisplay.js` | What the voice card shows for a result and which items it numbers (read by the card and `referents.js`) | +| `gevActions.js` runner | Analyst follow-up memory (session: `resetConversation()` on stop, `dispose()` on removal) | Pure input policy, preferences and protocol-response policy have separate modules. Owners receive named readers, operations or focused collaborators. Cross-owner @@ -29,7 +35,7 @@ Action and post-capture continuations retain their conversation identity before publishing output, queueing a response or changing status. Behavioral invariants remain in force: a 500 ms Space hold claims push-to-talk; -short control taps and text entry remain native; a click-started session stays +short control taps and text entry remain native; a claimed hold while the assistant speaks interrupts it (never during a Radio handoff); a click-started session stays open-mic. Every accepted or refused tool call receives its terminal output once. New typed input supersedes old intent. Radio waits for the spoken confirmation and verified muted playback. A live cost meter stays bound to its negotiated model, diff --git a/scripts/package-boundaries.json b/scripts/package-boundaries.json index ce778ee95..2a188b4b0 100644 --- a/scripts/package-boundaries.json +++ b/scripts/package-boundaries.json @@ -311,6 +311,7 @@ "server/providers/common/same-site.js", "src/localRequestGate.mjs", "src/voice/actionSchemas.js", + "src/voice/layerManifest.js", "server/providers/openai/toolDescriptions.js", "src/sources/rateLimit.js" ], @@ -339,12 +340,18 @@ "src/search/geospatial.js", "src/search/http.js", "src/search/defaults.js", + "src/search/alpsGeocoder.js", "src/search/coordinateGeocoder.js", "src/search/coordinateParser.js", "src/search/presetGeocoder.js", "src/search/nominatim.js", "src/nominatimGeocode.js", "src/data/regionalModel.js", + "src/data/naturalEarthRegions.js", + "src/data/bundledJson.js", + "src/data/retryableLoad.js", + "src/data/local_data/natural_earth/regions.json", + "src/data/local_data/natural_earth/marine.json", "src/sources/httpBody.js", "src/sources/nominatim.js" ], @@ -686,6 +693,7 @@ "src/ui/styles/command-dock-trays.css", "src/ui/styles/command-dock-sliding.css", "src/ui/styles/voice-cost.css", + "src/ui/styles/voice-card.css", "src/ui/styles/first-run.css", "src/ui/styles/provider-settings.css", "src/ui/styles/weather.css", @@ -1365,7 +1373,19 @@ "src/voice/realtimeFacade.js", "src/voice/realtimeRadio.js", "src/voice/realtimeTurns.js", - "src/voice/realtimeConnection.js" + "src/voice/realtimeConnection.js", + "src/voice/turnMetrics.js", + "src/voice/speechLint.js", + "src/voice/voiceCard.js", + "src/voice/voiceCardPresentation.js", + "src/voice/resultDisplay.js", + "src/voice/referents.js", + "src/voice/layerManifest.js", + "src/voice/speech.js", + "src/voice/narration.js", + "src/voice/earcon.js", + "src/voice/pointerContext.js", + "src/voice/pointerReticle.js" ], "external": [] }, @@ -1465,6 +1485,7 @@ "src/data/aisLiveVessels.js", "src/data/alprCameras.js", "src/data/analystEngine.js", + "src/voice/layerManifest.js", "src/data/bikeshare.js", "src/data/cctv.js", "src/data/cctvCards.js", @@ -1827,6 +1848,7 @@ "src/scenes/recipes.js", "src/scenes/scenePolicy.js", "src/scopeMask.js", + "src/search/alpsGeocoder.js", "src/search/defaults.js", "src/search/geospatial.js", "src/search/google.js", @@ -1938,6 +1960,14 @@ "src/voice/commands.js", "src/voice/control.js", "src/voice/gevActions.js", + "src/voice/speech.js", + "src/voice/earcon.js", + "src/voice/narration.js", + "src/voice/turnMetrics.js", + "src/voice/speechLint.js", + "src/voice/voiceCard.js", + "src/voice/voiceCardPresentation.js", + "src/voice/resultDisplay.js", "src/voice/gevRealtime.js", "src/voice/layerSummary.js", "src/voice/realtimeBackend.js", @@ -2101,6 +2131,12 @@ "src/maps/imageryHost.js", "src/ui/recentImagery.js", "src/ui/imageryBoxTool.js", + "src/voice/pointerContext.js", + "src/voice/pointerReticle.js", + "src/voice/deixis.js", + "src/voice/referents.js", + "src/voice/pointerPick.js", + "src/data/pickedContext.js", "src/ui/streetLevelControls.js", "src/ui/streetLevelPresentation.js", "src/layers/streetLevel/index.js", @@ -2504,6 +2540,7 @@ "src/overlays/worldOverlayDraw.js", "src/overlays/worldOverlayTokens.js", "src/renderGovernor.js", + "src/search/alpsGeocoder.js", "src/search/defaults.js", "src/search/geospatial.js", "src/search/google.js", @@ -2705,6 +2742,7 @@ "src/layers/firms/anchors.js", "src/locations.js", "src/cameraGroundGuard.js", + "src/search/alpsGeocoder.js", "src/search/defaults.js", "src/search/geospatial.js", "src/search/google.js", @@ -2894,7 +2932,11 @@ }, "voice-action-schemas": { "exports": ["./voice/action-schemas"], - "modules": ["src/voice/actionSchemas.js", "src/hudLayoutPolicy.js"], + "modules": [ + "src/voice/actionSchemas.js", + "src/voice/layerManifest.js", + "src/hudLayoutPolicy.js" + ], "external": [] }, "voice-session": { @@ -2907,7 +2949,14 @@ "modules": [ "src/voice/sessionCommands.js", "src/voice/session.js", - "src/voice/control.js" + "src/voice/control.js", + "src/voice/voiceCard.js", + "src/voice/voiceCardPresentation.js", + "src/voice/resultDisplay.js", + "src/voice/referents.js", + "src/voice/layerManifest.js", + "src/voice/speech.js", + "src/voice/pointerReticle.js" ], "external": [] }, diff --git a/scripts/qa-voice-bench.mjs b/scripts/qa-voice-bench.mjs new file mode 100644 index 000000000..f2f0f973b --- /dev/null +++ b/scripts/qa-voice-bench.mjs @@ -0,0 +1,411 @@ +#!/usr/bin/env node +/** + * qa-voice-bench.mjs β€” provider-neutral TEXT-mode voice-routing benchmark. + * + * Every provider gets the same prompt: production's realtimeInstructions() + * and GEV_REALTIME_TOOLS (the Realtime adapter uses the exact session body the + * token handler mints). Two suites, reported separately: + * core the qa-voice-routing phrase table, graded with its semantics + * (plus benchmark-only typed-argument checks) + * coverage the newer-layers coverage probe, graded correct / + * honest-refusal / hallucinated-capability / wrong-tool + * dialogue two-turn dialogues in one context; turn 2 depends on turn 1 + * capability area handles / imagery / OSM / point-and-ask phrases (and two + * dialogues) for the tools added after 4b56d0e9 + * Coverage rows are graded twice: v1 (original rubric, 4b56d0e9 tool schema) + * and v2 (rubric for the current tool surface). + * + * Methodology: a fresh context per phrase (new Realtime/Live session, or a + * stateless chat request); stub tool results; continuations until the model + * stops calling tools (max 4). No app page is involved. + * + * Usage: + * node scripts/qa-voice-bench.mjs --provider openai-realtime --model gpt-realtime-2 --reps 2 + * node scripts/qa-voice-bench.mjs --provider gemini-live --model gemini-3.8-live + * node scripts/qa-voice-bench.mjs --provider ollama --model gemma4:e4b + * node scripts/qa-voice-bench.mjs --provider openai-chat --base http://localhost:1234/v1 --model + * node scripts/qa-voice-bench.mjs --summarize --out qa-shots/voice-bench/ + * Common flags: --out (reuse one dir to collect several models), + * --suite core,coverage --only --budget + * --variant