feat(voice): voice core — honest layer reach, spoken summary with on-screen card, narration, point-and-ask

- Voice layer manifest (src/voice/layerManifest.js) generates the layer enums,
  aliases and analyst field hints; a test fails when a catalog layer has no entry.
- analyst_query validates fields, operators, units, values and scopes, counts up
  to 250,000 loaded records per layer, reports complete/partial/unanswered, tells
  LAYER_OFF / NOT_READY / FEED_UNAVAILABLE apart from zero, refuses mismatched
  follow-ups and never lets a superseded query replace follow-up memory. Seven
  more layers expose analyst records.
- Tool results carry a code-built say line and a display card; the voice card in
  the mic control shows captions, plan steps and the result
  (src/voice/resultDisplay.js is the one adapter). Slow tools narrate their
  current step out of band; holding Space while the assistant talks cuts it off.
  Turn spans go to the debug log; caption transcription is metered; the log
  omits what was said unless GEV_VOICE_LOG_CONTENT=1.
- Point and ask: the Space-down pointer snapshot resolves "this", "that one" and
  "around here" through 'pointer' targets, and "the second one" through
  referent:n from the last numbered list.
- Prompt states each policy once (instructions + tools 48.5k -> 46.7k chars).
- Voice benchmark harness (scripts/qa-voice-bench.mjs, scripts/voice-bench/)
  shares its phrase table and grading with scripts/qa-voice-routing.mjs.
This commit is contained in:
Bilawal Sidhu committed 2026-09-28 21:19:23 -05:00
1 parent e7707d9a0f
commit a19897ad15
97 files changed
+14343 -755

No files matched your search

+41 -26
View File
@@ -1,3 +1,8 @@
import {
voiceLayerAliasHint,
voiceQueryFieldHint,
} from '../../../src/voice/layerManifest.js';
export const ACTION_DESCRIPTIONS = {
fly_to_location: {
description:
@@ -12,7 +17,11 @@ export const ACTION_DESCRIPTIONS = {
},
query: {
description:
'Plain place search query, e.g. "London", "Eiffel Tower", or "Dubai Marina".',
'Plain place search query, e.g. "London", "Eiffel Tower", or "Dubai Marina". "pointer" = the spot under the cursor.',
$position: 1,
},
referent: {
description: 'Numbered item n from the last result list.',
$position: 1,
},
viewMode: {
@@ -76,13 +85,13 @@ export const ACTION_DESCRIPTIONS = {
$position: 1,
},
set_layer_visibility: {
description: "Enable or disable one registered God's Eye View data layer.",
description:
"Enable or disable one God's Eye View data layer. Every shipped layer is in the enum; never tell the user a listed layer does not exist.",
$position: 1,
parameters: {
properties: {
layerId: {
description:
'Common-name mapping for the non-obvious ids: space mission(s) → rocket-launches; fires/wildfires/active fires → local-firms (NASA FIRMS); ships/vessels/boats → ais-live-vessels; undersea/submarine cables → telegeography-submarine-cables; datacenters → local-datacenters; dams → local-dams; bikes/bike share → bikeshare; street traffic/congestion → traffic; traffic cameras → cctv; internet radio/stations → radio; ALPR/license plate readers/Flock cameras → alpr-cameras; local ADS-B/my receiver/my antenna (aircraft heard by a local RTL-SDR receiver) → local-adsb.',
description: `Spoken names: ${voiceLayerAliasHint()}.`,
$position: 1,
},
},
@@ -154,9 +163,13 @@ export const ACTION_DESCRIPTIONS = {
properties: {
scope: {
description:
'Use auto by default. selected returns the clicked/selected entity; in_view returns visible entities near the screen center.',
'Use auto by default. selected returns the clicked/selected entity; in_view returns visible entities near the screen center; pointer returns what the cursor is on.',
$position: 2,
},
referent: {
description: 'Numbered item n from the last result list.',
$position: 1,
},
layerId: {
description: 'Optional layer filter for visible entity context.',
$position: 2,
@@ -349,12 +362,17 @@ export const ACTION_DESCRIPTIONS = {
properties: {
query: {
description:
'Callsign, ship name, satellite name, ICAO hex, MMSI, or NORAD id.',
'Callsign, ship name, satellite name, ICAO hex, MMSI, or NORAD id. "pointer" = the contact under the cursor.',
$position: 1,
},
referent: {
description:
'Numbered item n from the last result list (-1 = last); query may then be its label.',
$position: 1,
},
layerId: {
description:
'Optional layer hint: flights | military | ais-live-vessels | satellites.',
'Optional: flights, military, ais-live-vessels or satellites. Other layers are not trackable; for fires use query "biggest fire".',
$position: 1,
},
},
@@ -381,7 +399,7 @@ export const ACTION_DESCRIPTIONS = {
},
annotate_map: {
description:
'Draw annotations on the 3D map to visually point out what you are talking about — like sketching on a whiteboard over the world. Use this whenever you mention a specific place, building, campus, boundary, district, or a relationship between two places, so the user can SEE what you mean. Give place NAMES (preferred) or explicit lat/lng; the app resolves them to real-world positions and real building/area outlines — never guess pixel positions. Call this as you begin describing something, and you may mark several places in one call.',
'Mark places on the 3D map as you talk about them (see WHITEBOARD): pins, highlights, footprints and boundaries, arrows, routes and labels. Give place names (or explicit lat/lng); the app resolves real positions and outlines. Several places may go in one call.',
$position: 1,
parameters: {
properties: {
@@ -398,7 +416,7 @@ export const ACTION_DESCRIPTIONS = {
},
target: {
description:
'Place name to resolve, e.g. "Palace of Fine Arts, San Francisco", "the Pentagon", "Presidio of San Francisco". Preferred over coordinates. For a specific monument/statue/feature that sits within a larger landmark, use its OWN name + city ("Tejano Monument, Austin", "Texas African American History Memorial, Austin") — do NOT phrase it as "X at the Texas State Capitol", which makes the geocoder collapse several of them onto the same centroid so they stack on one spot.',
'Place name to resolve ("Palace of Fine Arts, San Francisco"), or "pointer" for the spot under the cursor. A monument inside a larger landmark gets its own name + city ("Tejano Monument, Austin"), never "X at the Texas State Capitol" (that stacks them on one centroid).',
$position: 2,
},
points: {
@@ -450,27 +468,24 @@ export const ACTION_DESCRIPTIONS = {
},
entityKind: {
description:
'What KIND of thing the target IS — a fact, not a style choice: building = one structure; compound = campus/grounds/mall/park; district = neighborhood or area of a city; street = a named road/corridor; point_feature = monument/statue/memorial/plaque/fountain or other small point landmark. Set it whenever you know it — it routes the resolver to the right footprint source (point_feature anchors monuments as precise points instead of adopting a nearby building outline).',
'What the target IS (a fact, not a style): building = one structure; compound = campus/grounds/mall/park; district = neighborhood; street = a named road; point_feature = monument/statue/memorial/fountain, anchored as a precise point. Set it whenever you know it.',
$position: 2,
},
screenX: {
description:
'Fallback only: when you cannot name/geocode the place but can SEE it in the latest viewport screenshot, the normalized horizontal position (0=left, 1=right) of the spot. The app converts it back to a real world point under that pixel.',
'Fallback only, for a spot visible in the latest screenshot that you cannot name: normalized x (0=left, 1=right).',
$position: 3,
},
screenY: {
description:
'Fallback only: normalized vertical position (0=top, 1=bottom) of the spot in the latest viewport screenshot.',
description: 'Fallback only: normalized y (0=top, 1=bottom).',
$position: 3,
},
toScreenX: {
description:
'For type=arrow: normalized x of the arrow destination from the screenshot (pixel fallback).',
description: 'For type=arrow: destination x (pixel fallback).',
$position: 3,
},
toScreenY: {
description:
'For type=arrow: normalized y of the arrow destination from the screenshot (pixel fallback).',
description: 'For type=arrow: destination y (pixel fallback).',
$position: 3,
},
},
@@ -528,18 +543,18 @@ export const ACTION_DESCRIPTIONS = {
},
analyst_query: {
description:
'Answer questions ABOUT the data currently loaded on the map — counts, lists, superlatives, and attribute filters over live layers (flights, military, ships, fires, earthquakes, satellites, datacenters, dams). Examples: "how many flights over Texas", "biggest fire near LA", "which ships are headed to Oakland", "anything above 40,000 feet", "fastest thing in view". For satellites and infrastructure, counts and ranks cover only bounded examined loaded records; omitted records can change nearest/count. Queries ONLY client-side data from ENABLED layers — if the needed layer is off, say so and offer to enable it. For a follow-up about the previous answer\'s set ("which of those is closest?"), set followUp=true and send only the new filters/sort.',
'Counts, lists, superlatives and attribute filters over records loaded by ENABLED layers — "how many flights over Texas", "biggest fire near LA", "ships headed to Oakland", "next rocket launch". complete:false means the count is a floor ("at least") and rankings cover the records examined; partial names unanswered layers. Items carry lat/lon for a follow-up fly_to_location. ok:false codes: LAYER_OFF (offer to enable), NOT_READY (loading), FOLLOW_UP_MISMATCH (ask again without followUp), UNKNOWN_FIELD/UNKNOWN_SCOPE/BAD_VALUE (retry with the allowed values).',
$position: 1,
parameters: {
properties: {
layers: {
description:
'Layers to query. fires/wildfires → local-firms; ships/vessels → ais-live-vessels.',
'Layers to query. fires → local-firms; ships → ais-live-vessels; buses/trains → transit; hurricanes → weather-cyclones; bases → military-installations; my receiver → local-adsb. Units differ per layer, so rank speed within one layer.',
$position: 2,
},
scope: {
description:
'Spatial scope. Default: view (near the camera). Use kind=region for "over Texas"-style asks; kind=anywhere for global questions.',
'Always set it (see WHERE). view = a radius around the view centre ("in view", "nearby"); pointer = "around here/this" while pointing (radius at the cursor); region = "over Texas"; radius with center {lat, lon} = "near <place>"; anywhere = no place ("biggest anywhere", "today"). There is no drawn-area scope yet: say so rather than answering for the view.',
$position: 2,
properties: {
name: {
@@ -548,23 +563,23 @@ export const ACTION_DESCRIPTIONS = {
$position: 1,
},
km: {
description: 'For kind=radius.',
description: 'For kind=radius or pointer.',
$position: 1,
},
},
},
filters: {
description:
'Attribute predicates, ANDed. ALTITUDE IS METERS (40,000 ft = 12192). Fields: altitudeM, speedMps, military, onGround, aircraftClass, callsign, operator, routeOrigin, routeDestination, originCountry (flights); speedKts, shipType, destination (ships); frp, confidence (fires); magnitude, depthKm, place (earthquakes).',
description: `ANDed; values keep their type (numbers, true/false). Altitude is meters (40,000 ft = 12192). Time fields take ISO; ageHours = hours ago. Every layer has id, lat, lon. Fields — ${voiceQueryFieldHint()}.`,
$position: 1,
},
sortBy: {
description: 'Field to rank by, or "distance" for nearest-first.',
description:
'Field to rank by, or "distance" (nearest first unless sortDir=desc).',
$position: 1,
},
followUp: {
description:
'true = re-query the PREVIOUS result set instead of fresh data.',
'true = re-filter the last answer (its layers) instead of fresh data.',
$position: 1,
},
},
@@ -572,7 +587,7 @@ export const ACTION_DESCRIPTIONS = {
},
next_iss_pass: {
description:
"When the user asks when the ISS / the space station will next fly over: returns the next geometric ISS pass with estimated visibility for the current camera location (or an explicit lat/lon) — rise time (ISO + minutes from now), rise compass direction, peak elevation, and duration. Requires the satellites layer to have loaded its catalog at least once this session; if it hasn't, tell the user to enable the satellites layer and try again.",
'Next ISS pass over the camera location (or lat/lon): rise time, direction, peak elevation, duration and estimated visibility. Needs the satellites layer to have loaded once; if not, ask the user to enable it.',
$position: 1,
parameters: {
properties: {