diff --git a/CHANGELOG.md b/CHANGELOG.md index 71e2abd62..a0c0eda35 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,15 @@ # Changelog +- Unqualified Alps searches use the bundled European range for navigation + and annotations instead of a same-named peak returned by a geocoder. + Coordinates, supplied presets and geographically qualified place names + keep their precedence; annotation proximity and containment guards remain. +- Contacts analyst follow-ups retain the snapshot's distance ranking rather + than switching to the camera. Requested radii and explicit scopes remain + authoritative, and a request for a list still requires an analyst query. + Analyst speech names stale or degraded feeds and withholds authoritative + counts when unavailable. The existing NOTES disclosure preserves complete, + deduplicated coverage and source caveats, including crowded results. - The voice card shows analyst counts ("At least 250,000 aircraft …") with their scope, feed window and caveats, and marks partial answers. Analyst speech uses the same deterministic headline, preserves lower bounds and says @@ -31,7 +41,8 @@ referent-only tracking calls are accepted while empty targets are rejected. Asking about bare ground up close gives the assistant a small crop of the map around the - point. + point. A slow render gets one bounded second frame request, while hidden, + stale, moved-camera, invalid and oversized captures still fail closed. ## [Unreleased] @@ -284,7 +295,6 @@ geocode and admin-boundary fallback answers `region-timeout` after 3 s instead of holding the reply. - - Transit and Directions rows repaint as soon as their data lands again: `refreshLayerStats()` now lives on the layer lifecycle, not only on the compatibility facade. `scripts/qa-radio.mjs` uses it instead of a private @@ -486,6 +496,7 @@ Add feed provenance to analyst/view answers and HUD context while retaining existing response fields and runner ownership (Matt Van Horn, #347). Analyst records for loaded satellites, datacenters and dams, with explicit bounded count/rank coverage (Matt Van Horn, #351). + - New Fire Perimeters layer (Events group): live NIFC WFIGS interagency wildfire incident perimeters as ground-clamped polygons with a containment-colored fire line, refreshed every 5 minutes from the public @@ -762,7 +773,6 @@ of current runtime behavior, see [`docs/CURRENT-STATE.md`](docs/CURRENT-STATE.md ### Other changes - - Add ECMWF IFS model selection to Wind (#464, thanks @beneduzi), with model-scoped forecast-step caches, cancellation of replaced requests, and separate issue/valid timestamps. - Add bounded Director feature actions with accessible controls, explicit camera/layer admission and cancellation; restore pack geometry on same-shot seek. Preserve existing scenes and content attribution. @@ -1571,4 +1581,3 @@ represent previously published GitHub Releases. ## [0.1.0] — 2026-02-09 - Initial project version. - diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 3fcf7a38d..52744a0de 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -353,6 +353,15 @@ replacement session; audio meters release failed initialization and reject revok frames. Delayed action results and post-capture continuations cannot resume a stopped conversation or send output into a replacement. See [voice ownership](VOICE-OWNERSHIP.md). +Contacts nearest-aircraft follow-ups retain the snapshot's distance basis even +after the camera moves. An explicit new scope replaces that basis. Provider +instructions require an analyst query for requested lists and use the requested +radius rather than substituting the panel's fixed window. Analyst speech carries +stale/degraded feed status and does not speak an authoritative unavailable count. +The existing NOTES disclosure deduplicates coverage/source caveats and retains +their full text in at most six rows, grouping analyst overflow in the final row; +ordinary tool-card limits are unchanged. + Voice speech: `src/voice/speech.js` adds `{ say, display, referents?, schedule? }` to layer toggles, fly_to_location, frame_overhead, select_nearest_aircraft, get_entity_context, annotate_map and analyst_query. Analyst speech and the card @@ -392,7 +401,10 @@ accepts either a named query or a referent, while an empty target is rejected. A point during the turn and the voice card shows "THIS"/"HERE"; both hide in Clean-UI and recording. A ground-only pointer lookup at local/city scale sends a ringed 512 px crop in the one retained image slot, dropped if the camera moved or a newer capture or -turn started; screenX/screenY against the crop are mapped back to the canvas. Harness: `qa-voice-routing.mjs --layer deixis`, +turn started. The pointer path makes at most two bounded fresh-frame requests so +a slow render loop can recover without accepting a stale frame; a missing camera +identity, expired snapshot, invalid geometry, black frame or oversized payload +still fails closed. screenX/screenY against the crop are mapped back to the canvas. Harness: `qa-voice-routing.mjs --layer deixis`, `--layer deixis-live`, `--layer behavior --deixis-only`. The application shell composes focused state owners for navigation, destination @@ -1529,10 +1541,10 @@ This is the current runtime/source-of-truth snapshot for the project. > samples, `enabling` throughout, panel non-numeric) and across a failing > one. Its `status: 'idle'` is not what saves it — the lifecycle is; the > module has no `loading` status at all. -> Any new dependency that can be slow must report `loading` and `lastUpdate` -> honestly for the contract to hold, and must be pinned against the shape its -> own module really returns — a fixture that invents a status the module cannot -> emit guards nothing (that is exactly how a hole here survived a green suite). +> Any new dependency that can be slow must report `loading` and `lastUpdate` +> honestly for the contract to hold, and must be pinned against the shape its +> own module really returns — a fixture that invents a status the module cannot +> emit guards nothing (that is exactly how a hole here survived a green suite). > - **A held ground snap is dropped when measured ground contradicts it > (2026-08-23):** the WARM hold described below answers on DISTANCE travelled, > which is only a proxy for whether the value still describes the ground. A @@ -2964,27 +2976,27 @@ nearby place—always use station selection. Unqualified “turn on/start the ra requests use Play; a qualified Play-shaped tool call is normalized to Select so its criteria cannot be silently ignored. -| Layer | Source | File | Proxy | Update Interval | -| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------------- | -| Live Flights ✈️ | OpenSky Network; bounded adsb.lol regional fallback | `src/data/flights.js` | `/api/flights` (OAuth + fallback) | 30s | -| Military Flights 🎖️ | adsb.lol /v2/mil | `src/data/militaryFlights.js` | `/api/military` | 15s | -| Live AIS Vessels 🚢 | AISStream websocket | `src/data/aisLiveVessels.js` | `/api/vessels` | 60s (+800ms visibility pass) | -| Mapped Installations ⌖ | OpenFreeMap military areas; optional configured Overpass names and Google Places search | `src/data/militaryInstallations.js` | OpenFreeMap tiles, `/api/military-installations`, `/api/google/text-search` | viewport-driven; transient errors back off, missing capability switches to tiles | -| Earthquakes | USGS | `src/data/earthquakes.js` | — | 60s | -| Satellites | CelesTrak | `src/data/satellites.js` | `/api/celestrak` | 120s | -| Space Missions (30d) | Launch Library 2 + CelesTrak | `src/data/rocketLaunches.js` | `/api/launches` + `/api/celestrak/active` | 5 min | -| Traffic | Selectable TomTom / OpenStreetMap / Hybrid roads; optional TomTom flow (BYOK) | `src/data/traffic.js` | browser-direct OpenFreeMap tiles; `/api/tomtom` for flow | viewport-driven; capped tile cache | -| CCTV | Austin + Caltrans (CA) + TfL London + Ontario 511 + Fintraffic (FI) + DriveBC (BC) + TxDOT (TX) + Estonia (Tallinn, Tarktee) + Live Traffic NSW + Open Calgary Open Data + Street View fallback | `src/data/cctv.js` | `/api/cctv` | 10s (active) | -| Radio | Radio Browser (public-domain station directory) | `src/data/radio.js` | `/api/radio/stations`, `/api/radio/click/:uuid` | 45 min directory refresh | -| Transit 🚌 | Operator GTFS-Realtime VehiclePositions (7 keyless regions, `src/data/transitFeeds.js`) | `src/layers/transit/` via `src/app/layers/transit.js` | `/api/transit` | 15s (poll + delayed playback) | -| Bikeshare 🚲 | GBFS (Lyft + BCycle) | `src/data/bikeshare.js` | `/api/gbfs` | 60s | -| Directions 🧭 | OSRM on FOSSGIS servers (OpenStreetMap) | `src/data/directions.js` | `/api/route` (`steps=1`) | on placement / mode change | -| Datacenters ▣ | OSM extract (bundled) | `src/data/localLayers.js` | — | static | -| Dams ▰ | OpenInfraMap/OSM extract (bundled) | `src/data/localLayers.js` | — | static | -| Submarine Cables ◠ | TeleGeography public map (bundled) | `src/data/telegeographySubmarineCables.js` | — | static | -| FIRMS Active Fires ▲ | NASA FIRMS live (VIIRS ×3 NRT + MODIS NRT, trailing 24h) | `src/data/firmsHeatmap.js` | `/api/firms` (`FIRMS_MAP_KEY`) | 10 min (proxy TTL 30 min) | -| Wind 🌬 | NOAA GFS 10 m wind (keyless, 0.25°→1° grid; animated particles) | `src/data/wind.js` | `/api/wind` | 1 h (forecast cycle) | -| Fire Perimeters 🔥 | NIFC WFIGS current interagency perimeters (keyless, paged past the 2000-record cap); InciWeb catalog + incident page origin and update time checks for verified incident-page links | `src/layers/perimeters/` via `src/app/layers/perimeters.js` | `/api/fire-perimeters` + `/api/fire-perimeters/inciweb/*` | 5 min (server caches: catalog 1 h; publication 30 min) | +| Layer | Source | File | Proxy | Update Interval | +| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | --------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | +| Live Flights ✈️ | OpenSky Network; bounded adsb.lol regional fallback | `src/data/flights.js` | `/api/flights` (OAuth + fallback) | 30s | +| Military Flights 🎖️ | adsb.lol /v2/mil | `src/data/militaryFlights.js` | `/api/military` | 15s | +| Live AIS Vessels 🚢 | AISStream websocket | `src/data/aisLiveVessels.js` | `/api/vessels` | 60s (+800ms visibility pass) | +| Mapped Installations ⌖ | OpenFreeMap military areas; optional configured Overpass names and Google Places search | `src/data/militaryInstallations.js` | OpenFreeMap tiles, `/api/military-installations`, `/api/google/text-search` | viewport-driven; transient errors back off, missing capability switches to tiles | +| Earthquakes | USGS | `src/data/earthquakes.js` | — | 60s | +| Satellites | CelesTrak | `src/data/satellites.js` | `/api/celestrak` | 120s | +| Space Missions (30d) | Launch Library 2 + CelesTrak | `src/data/rocketLaunches.js` | `/api/launches` + `/api/celestrak/active` | 5 min | +| Traffic | Selectable TomTom / OpenStreetMap / Hybrid roads; optional TomTom flow (BYOK) | `src/data/traffic.js` | browser-direct OpenFreeMap tiles; `/api/tomtom` for flow | viewport-driven; capped tile cache | +| CCTV | Austin + Caltrans (CA) + TfL London + Ontario 511 + Fintraffic (FI) + DriveBC (BC) + TxDOT (TX) + Estonia (Tallinn, Tarktee) + Live Traffic NSW + Open Calgary Open Data + Street View fallback | `src/data/cctv.js` | `/api/cctv` | 10s (active) | +| Radio | Radio Browser (public-domain station directory) | `src/data/radio.js` | `/api/radio/stations`, `/api/radio/click/:uuid` | 45 min directory refresh | +| Transit 🚌 | Operator GTFS-Realtime VehiclePositions (7 keyless regions, `src/data/transitFeeds.js`) | `src/layers/transit/` via `src/app/layers/transit.js` | `/api/transit` | 15s (poll + delayed playback) | +| Bikeshare 🚲 | GBFS (Lyft + BCycle) | `src/data/bikeshare.js` | `/api/gbfs` | 60s | +| Directions 🧭 | OSRM on FOSSGIS servers (OpenStreetMap) | `src/data/directions.js` | `/api/route` (`steps=1`) | on placement / mode change | +| Datacenters ▣ | OSM extract (bundled) | `src/data/localLayers.js` | — | static | +| Dams ▰ | OpenInfraMap/OSM extract (bundled) | `src/data/localLayers.js` | — | static | +| Submarine Cables ◠ | TeleGeography public map (bundled) | `src/data/telegeographySubmarineCables.js` | — | static | +| FIRMS Active Fires ▲ | NASA FIRMS live (VIIRS ×3 NRT + MODIS NRT, trailing 24h) | `src/data/firmsHeatmap.js` | `/api/firms` (`FIRMS_MAP_KEY`) | 10 min (proxy TTL 30 min) | +| Wind 🌬 | NOAA GFS 10 m wind (keyless, 0.25°→1° grid; animated particles) | `src/data/wind.js` | `/api/wind` | 1 h (forecast cycle) | +| Fire Perimeters 🔥 | NIFC WFIGS current interagency perimeters (keyless, paged past the 2000-record cap); InciWeb catalog + incident page origin and update time checks for verified incident-page links | `src/layers/perimeters/` via `src/app/layers/perimeters.js` | `/api/fire-perimeters` + `/api/fire-perimeters/inciweb/*` | 5 min (server caches: catalog 1 h; publication 30 min) | Fire Perimeters uses capped, timed server reads with stale-on-error caching and a per-client limit. Unchanged snapshots retain geometry; link checks cancel on disable or selection change, and the row legend shows reported containment. @@ -4047,6 +4059,15 @@ are omitted rather than framing the wrong part of the globe. retrying. Missing capability never synthesizes an outline from a radius. Manual geometry, bundled neighborhoods, Natural Earth regions and bundled state/province/county outlines remain. +- The shared default place search resolves unqualified `Alps`, `the Alps` + and the `Alps mountains` forms from the bundled European Natural Earth + range before network geocoders. Navigation uses its contained anchor and + region viewport; annotations still require the existing proximity and + outline-containment checks. Coordinates and supplied presets keep priority; + geographic qualifiers, local POI names, other ranges and custom geocoder + arrays retain their existing resolution. A failed Alps pack load reports a + search failure rather than caching an unrelated peak as the range, and the + bundled loader retains its normal retry cooldown. - County queries rank eligible US counties and international admin-1 units together, retaining exact aliases and country/state qualifiers. - Bundled administrative outlines (`src/data/adminBoundaries.js`): states and provinces worldwide (Natural Earth admin-1, @@ -4909,7 +4930,6 @@ Temperature uses stronger fixed −40..50°C colors; the underlying 1° forecast numeric inspection values are unchanged. No volumetric cloud height or local rain arrival prediction is claimed. NOAA source limits are documented in DATA_SOURCES. - Traffic source refinements: ALPR admits z9-z12 detail only when the view fits 16 tiles, returning explicit zoom guidance before I/O and allowing the next smaller view to load normally. Outside the US/Canada extracts the row reads “No ALPR data for this diff --git a/scripts/package-boundaries.json b/scripts/package-boundaries.json index 7f5389cbc..4670dbd82 100644 --- a/scripts/package-boundaries.json +++ b/scripts/package-boundaries.json @@ -322,12 +322,18 @@ "src/search/geospatial.js", "src/search/http.js", "src/search/defaults.js", + "src/search/alpsGeocoder.js", "src/search/coordinateGeocoder.js", "src/search/coordinateParser.js", "src/search/presetGeocoder.js", "src/search/nominatim.js", "src/nominatimGeocode.js", "src/data/regionalModel.js", + "src/data/naturalEarthRegions.js", + "src/data/bundledJson.js", + "src/data/retryableLoad.js", + "src/data/local_data/natural_earth/regions.json", + "src/data/local_data/natural_earth/marine.json", "src/sources/httpBody.js", "src/sources/nominatim.js" ], @@ -1784,6 +1790,7 @@ "src/scenes/recipes.js", "src/scenes/scenePolicy.js", "src/scopeMask.js", + "src/search/alpsGeocoder.js", "src/search/defaults.js", "src/search/geospatial.js", "src/search/google.js", @@ -2444,6 +2451,7 @@ "src/overlays/worldOverlayDraw.js", "src/overlays/worldOverlayTokens.js", "src/renderGovernor.js", + "src/search/alpsGeocoder.js", "src/search/defaults.js", "src/search/geospatial.js", "src/search/google.js", @@ -2617,6 +2625,7 @@ "src/layers/firms/anchors.js", "src/locations.js", "src/cameraGroundGuard.js", + "src/search/alpsGeocoder.js", "src/search/defaults.js", "src/search/geospatial.js", "src/search/google.js", diff --git a/scripts/qa-voice-routing.mjs b/scripts/qa-voice-routing.mjs index 15a0e9c0d..f1b2b0ced 100644 --- a/scripts/qa-voice-routing.mjs +++ b/scripts/qa-voice-routing.mjs @@ -37,10 +37,14 @@ import path from 'node:path'; import { fileURLToPath } from 'node:url'; import WebSocket from 'ws'; import puppeteer from 'puppeteer'; +import sharp from 'sharp'; import { ROUTING_PHRASES } from './voice-bench/phrases.mjs'; import { gradeCore, expectLabel } from './voice-bench/grade.mjs'; import { countWords, findHedges } from '../src/voice/speechLint.js'; -import { buildPointerSnapshot, pointerContextItem } from '../src/voice/pointerContext.js'; +import { + buildPointerSnapshot, + pointerContextItem, +} from '../src/voice/pointerContext.js'; const __dirname = path.dirname(fileURLToPath(import.meta.url)); const ROOT = path.resolve(__dirname, '..'); @@ -51,7 +55,11 @@ const CHROME_CANDIDATES = [ '/Applications/Chromium.app/Contents/MacOS/Chromium', ].filter(Boolean); const CHROME_EXECUTABLE = CHROME_CANDIDATES.find((candidate) => { - try { return fs.existsSync(candidate); } catch { return false; } + try { + return fs.existsSync(candidate); + } catch { + return false; + } }); // ── CLI ───────────────────────────────────────────────────── @@ -69,18 +77,66 @@ const HEADFUL = process.argv.includes('--headful'); // real GPU, visible window const SHOT_DIR = getOpt('--shots', null); // point-and-ask evidence screenshots // ── Reporting ─────────────────────────────────────────────── -let pass = 0, fail = 0, skip = 0; +let pass = 0, + fail = 0, + skip = 0; const failures = []; function report(ok, label, detail) { const tag = ok ? '\x1b[32mPASS\x1b[0m' : '\x1b[31mFAIL\x1b[0m'; console.log(` [${tag}] ${label} — ${detail}`); - if (ok) pass += 1; else { fail += 1; failures.push({ label, detail }); } + if (ok) pass += 1; + else { + fail += 1; + failures.push({ label, detail }); + } } function skipped(label, detail) { console.log(` [\x1b[33mSKIP\x1b[0m] ${label} — ${detail}`); skip += 1; } +async function inspectPointerCrop(captured, screenPx, viewport) { + const crop = captured?.dataUrl; + const rect = captured?.rect; + if (!crop || !rect || !screenPx || !viewport?.w || !viewport?.h) return null; + const encoded = crop.split(',')[1]; + if (!encoded) return null; + const { data, info } = await sharp(Buffer.from(encoded, 'base64')) + .removeAlpha() + .raw() + .toBuffer({ resolveWithObject: true }); + const pointX = ((screenPx.x / viewport.w - rect.x) / rect.w) * info.width; + const pointY = ((screenPx.y / viewport.h - rect.y) / rect.h) * info.height; + let ringPixels = 0; + let usablePixels = 0; + for (let y = 0; y < info.height; y += 1) { + for (let x = 0; x < info.width; x += 1) { + const offset = (y * info.width + x) * info.channels; + const red = data[offset]; + const green = data[offset + 1]; + const blue = data[offset + 2]; + const luminance = red * 0.2126 + green * 0.7152 + blue * 0.0722; + if (luminance > 5) usablePixels += 1; + const distance = Math.hypot(x - pointX, y - pointY); + if ( + distance >= 12 && + distance <= 20 && + green > 120 && + blue > 120 && + green - red > 45 && + blue - red > 45 + ) + ringPixels += 1; + } + } + return { + width: info.width, + height: info.height, + ringPixels, + usableRatio: usablePixels / (info.width * info.height), + }; +} + // ════════════════════════════════════════════════════════════ // LAYER 1 — ROUTING // ════════════════════════════════════════════════════════════ @@ -93,11 +149,17 @@ const PHRASES = ROUTING_PHRASES; /** Mint a session via the app's own endpoint. */ async function mintSession() { const res = await fetch(`${APP_URL}/api/realtime/token`, { method: 'POST' }); - if (!res.ok) throw new Error(`token mint failed: ${res.status} ${(await res.text()).slice(0, 120)}`); + if (!res.ok) + throw new Error( + `token mint failed: ${res.status} ${(await res.text()).slice(0, 120)}`, + ); const body = await res.json(); const value = body?.value || body?.client_secret?.value; const model = body?.session?.model || 'gpt-realtime'; - if (!value) throw new Error(`token mint returned no client secret: ${JSON.stringify(body).slice(0, 160)}`); + if (!value) + throw new Error( + `token mint returned no client secret: ${JSON.stringify(body).slice(0, 160)}`, + ); return { value, model }; } @@ -114,29 +176,65 @@ class RoutingSession { connect() { return new Promise((resolve, reject) => { const url = `wss://api.openai.com/v1/realtime?model=${encodeURIComponent(this.model)}`; - const ws = new WebSocket(url, { headers: { Authorization: `Bearer ${this.secret}` } }); - const timer = setTimeout(() => reject(new Error('ws connect timeout')), 15000); - ws.on('open', () => { clearTimeout(timer); this.ws = ws; resolve(); }); - ws.on('error', (e) => { clearTimeout(timer); reject(e); }); + const ws = new WebSocket(url, { + headers: { Authorization: `Bearer ${this.secret}` }, + }); + const timer = setTimeout( + () => reject(new Error('ws connect timeout')), + 15000, + ); + ws.on('open', () => { + clearTimeout(timer); + this.ws = ws; + resolve(); + }); + ws.on('error', (e) => { + clearTimeout(timer); + reject(e); + }); ws.on('message', (raw) => this.onMessage(raw)); }); } onMessage(raw) { let msg; - try { msg = JSON.parse(raw.toString()); } catch { return; } - this.evidence.write(`${JSON.stringify({ at: new Date().toISOString(), dir: 'recv', type: msg.type, item: msg.item?.type, name: msg.item?.name ?? msg.name, error: msg.error?.message })}\n`); + try { + msg = JSON.parse(raw.toString()); + } catch { + return; + } + this.evidence.write( + `${JSON.stringify({ at: new Date().toISOString(), dir: 'recv', type: msg.type, item: msg.item?.type, name: msg.item?.name ?? msg.name, error: msg.error?.message })}\n`, + ); if (!this.pending) return; - if (msg.type === 'response.output_item.done' && msg.item?.type === 'message') { + if ( + msg.type === 'response.output_item.done' && + msg.item?.type === 'message' + ) { // Speech stats only: text said before a call in the same response is a // preamble; text after the last call is the spoken answer. - const text = (msg.item.content || []).map((part) => part?.text ?? part?.transcript ?? '').join(' ').trim(); - if (text) this.pending.messages.push({ text, response: this.pending.continuations, callsBefore: this.pending.calls.length }); + const text = (msg.item.content || []) + .map((part) => part?.text ?? part?.transcript ?? '') + .join(' ') + .trim(); + if (text) + this.pending.messages.push({ + text, + response: this.pending.continuations, + callsBefore: this.pending.calls.length, + }); } - if (msg.type === 'response.output_item.done' && msg.item?.type === 'function_call') { + if ( + msg.type === 'response.output_item.done' && + msg.item?.type === 'function_call' + ) { this.pending.callResponses.push(this.pending.continuations); let parsed = {}; - try { parsed = JSON.parse(msg.item.arguments || '{}'); } catch { /* keep {} */ } + try { + parsed = JSON.parse(msg.item.arguments || '{}'); + } catch { + /* keep {} */ + } this.pending.calls.push({ name: msg.item.name, args: parsed }); // Feed a neutral tool result (item.create is legal mid-response); the // continuation response is requested ONLY after response.done — sending @@ -147,7 +245,9 @@ class RoutingSession { item: { type: 'function_call_output', call_id: msg.item.call_id, - output: JSON.stringify(this.stubResult || { ok: true, note: 'qa-harness stub result' }), + output: JSON.stringify( + this.stubResult || { ok: true, note: 'qa-harness stub result' }, + ), }, }); this.pending.needContinuation = true; @@ -156,7 +256,10 @@ class RoutingSession { if (this.pending.needContinuation && this.pending.continuations < 4) { this.pending.needContinuation = false; this.pending.continuations += 1; - this.send({ type: 'response.create', response: { output_modalities: ['text'] } }); + this.send({ + type: 'response.create', + response: { output_modalities: ['text'] }, + }); return; } const p = this.pending; @@ -172,7 +275,9 @@ class RoutingSession { } send(obj) { - this.evidence.write(`${JSON.stringify({ at: new Date().toISOString(), dir: 'send', type: obj.type })}\n`); + this.evidence.write( + `${JSON.stringify({ at: new Date().toISOString(), dir: 'send', type: obj.type })}\n`, + ); this.ws.send(JSON.stringify(obj)); } @@ -180,42 +285,84 @@ class RoutingSession { sendSystemItem(payload) { this.send({ type: 'conversation.item.create', - item: { type: 'message', role: 'system', content: [{ type: 'input_text', text: JSON.stringify(payload) }] }, + item: { + type: 'message', + role: 'system', + content: [{ type: 'input_text', text: JSON.stringify(payload) }], + }, }); } /** Send one user text turn; resolve with the list of tool calls it produced. */ runTurn(text) { return new Promise((resolve) => { - this.pending = { calls: [], messages: [], callResponses: [], continuations: 0, needContinuation: false, errors: [], resolve }; + this.pending = { + calls: [], + messages: [], + callResponses: [], + continuations: 0, + needContinuation: false, + errors: [], + resolve, + }; const timer = setTimeout(() => { - if (this.pending) { const p = this.pending; this.pending = null; p.resolve(p.calls); } + if (this.pending) { + const p = this.pending; + this.pending = null; + p.resolve(p.calls); + } }, 45000); const origResolve = resolve; - this.pending.resolve = (v) => { clearTimeout(timer); origResolve(v); }; + this.pending.resolve = (v) => { + clearTimeout(timer); + origResolve(v); + }; this.send({ type: 'conversation.item.create', - item: { type: 'message', role: 'user', content: [{ type: 'input_text', text }] }, + item: { + type: 'message', + role: 'user', + content: [{ type: 'input_text', text }], + }, + }); + this.send({ + type: 'response.create', + response: { output_modalities: ['text'] }, }); - this.send({ type: 'response.create', response: { output_modalities: ['text'] } }); }); } - close() { try { this.ws?.close(); } catch { /* noop */ } } + close() { + try { + this.ws?.close(); + } catch { + /* noop */ + } + } } async function runRoutingLayer() { - console.log(`\nLAYER 1 — routing assertions (budget ${TURN_BUDGET} model turns)`); + console.log( + `\nLAYER 1 — routing assertions (budget ${TURN_BUDGET} model turns)`, + ); const logDir = path.join(ROOT, '.gev-logs', 'qa-voice-routing'); fs.mkdirSync(logDir, { recursive: true }); - const evidence = fs.createWriteStream(path.join(logDir, `run-${Date.now()}.jsonl`)); + const evidence = fs.createWriteStream( + path.join(logDir, `run-${Date.now()}.jsonl`), + ); - const list = PHRASES.filter((p) => !ONLY || p.phrase.toLowerCase().includes(ONLY.toLowerCase())); + const list = PHRASES.filter( + (p) => !ONLY || p.phrase.toLowerCase().includes(ONLY.toLowerCase()), + ); let turnsUsed = 0; for (let i = 0; i < list.length; i += PHRASES_PER_SESSION) { if (turnsUsed >= TURN_BUDGET) { - list.slice(i).forEach((p) => skipped(`route: "${p.phrase}"`, 'turn budget exhausted')); + list + .slice(i) + .forEach((p) => + skipped(`route: "${p.phrase}"`, 'turn budget exhausted'), + ); break; } const batch = list.slice(i, i + PHRASES_PER_SESSION); @@ -225,11 +372,20 @@ async function runRoutingLayer() { session = new RoutingSession(value, model, evidence); await session.connect(); } catch (e) { - batch.forEach((p) => report(false, `route: "${p.phrase}"`, `session setup failed: ${e.message}`)); + batch.forEach((p) => + report( + false, + `route: "${p.phrase}"`, + `session setup failed: ${e.message}`, + ), + ); continue; } for (const p of batch) { - if (turnsUsed >= TURN_BUDGET) { skipped(`route: "${p.phrase}"`, 'turn budget exhausted'); continue; } + if (turnsUsed >= TURN_BUDGET) { + skipped(`route: "${p.phrase}"`, 'turn budget exhausted'); + continue; + } turnsUsed += 1; let calls = []; try { @@ -238,9 +394,17 @@ async function runRoutingLayer() { report(false, `route: "${p.phrase}"`, `turn error: ${e.message}`); continue; } - recordSpeech(p.phrase, calls.map((c) => c.name), session.lastTurn); + recordSpeech( + p.phrase, + calls.map((c) => c.name), + session.lastTurn, + ); const grade = gradeCore(p, calls); - report(grade.ok, `route: "${p.phrase}" → ${expectLabel(p)}`, grade.detail); + report( + grade.ok, + `route: "${p.phrase}" → ${expectLabel(p)}`, + grade.detail, + ); } session.close(); } @@ -259,17 +423,39 @@ async function runRoutingLayer() { // start), then the user's words. Asserts the model passes the 'pointer' // sentinel (or referent:n) instead of guessing coordinates or names. -const AIRCRAFT_POINTER = pointerContextItem(buildPointerSnapshot({ - at: 'speech_start', x: 912, y: 388, width: 1500, height: 950, - pick: { - entity: { layerId: 'flights', id: 'a4f2c1', label: 'UPS793', kind: 'aircraft', lat: 30.3121, lon: -97.6013 }, - ground: { lat: 30.3, lon: -97.6 }, - }, -})); -const GROUND_POINTER = pointerContextItem(buildPointerSnapshot({ - at: 'speech_start', x: 640, y: 520, width: 1500, height: 950, - pick: { entity: null, ground: { lat: 30.2849, lon: -97.7341, heightM: 160 } }, -})); +const AIRCRAFT_POINTER = pointerContextItem( + buildPointerSnapshot({ + at: 'speech_start', + x: 912, + y: 388, + width: 1500, + height: 950, + pick: { + entity: { + layerId: 'flights', + id: 'a4f2c1', + label: 'UPS793', + kind: 'aircraft', + lat: 30.3121, + lon: -97.6013, + }, + ground: { lat: 30.3, lon: -97.6 }, + }, + }), +); +const GROUND_POINTER = pointerContextItem( + buildPointerSnapshot({ + at: 'speech_start', + x: 640, + y: 520, + width: 1500, + height: 950, + pick: { + entity: null, + ground: { lat: 30.2849, lon: -97.7341, heightM: 160 }, + }, + }), +); const FRAMED_STUB = { ok: true, action: 'frame_overhead', @@ -286,38 +472,135 @@ const FRAMED_STUB = { const pointerArg = (value) => String(value || '').toLowerCase() === 'pointer'; const DEIXIS_SCENARIOS = [ - { pointer: AIRCRAFT_POINTER, phrase: 'Tell me about this', check: (c) => c.name === 'get_entity_context' && pointerArg(c.args.scope), want: 'get_entity_context{scope:pointer}' }, - { pointer: AIRCRAFT_POINTER, phrase: "What's that plane?", check: (c) => c.name === 'get_entity_context' && pointerArg(c.args.scope), want: 'get_entity_context{scope:pointer}' }, - { pointer: AIRCRAFT_POINTER, phrase: 'Track that one', check: (c) => c.name === 'track_entity' && (pointerArg(c.args.query) || /a4f2c1|ups793/i.test(c.args.query || '')), want: 'track_entity{query:pointer}' }, - { pointer: GROUND_POINTER, phrase: 'How many flights are around here?', check: (c) => c.name === 'analyst_query' && pointerArg(c.args.scope?.kind), want: 'analyst_query{scope.kind:pointer}' }, - { pointer: GROUND_POINTER, phrase: 'Take me there', check: (c) => c.name === 'fly_to_location' && pointerArg(c.args.query), want: 'fly_to_location{query:pointer}' }, - { pointer: GROUND_POINTER, phrase: 'Drop a pin right here', check: (c) => c.name === 'annotate_map' && (c.args.annotations || []).some((a) => pointerArg(a.target)), want: 'annotate_map{target:pointer}' }, - { pointer: GROUND_POINTER, phrase: 'What is this place?', check: (c) => c.name === 'get_entity_context' && pointerArg(c.args.scope), want: 'get_entity_context{scope:pointer}' }, + { + pointer: AIRCRAFT_POINTER, + phrase: 'Tell me about this', + check: (c) => c.name === 'get_entity_context' && pointerArg(c.args.scope), + want: 'get_entity_context{scope:pointer}', + }, + { + pointer: AIRCRAFT_POINTER, + phrase: "What's that plane?", + check: (c) => c.name === 'get_entity_context' && pointerArg(c.args.scope), + want: 'get_entity_context{scope:pointer}', + }, + { + pointer: AIRCRAFT_POINTER, + phrase: 'Track that one', + check: (c) => + c.name === 'track_entity' && + (pointerArg(c.args.query) || /a4f2c1|ups793/i.test(c.args.query || '')), + want: 'track_entity{query:pointer}', + }, + { + pointer: GROUND_POINTER, + phrase: 'How many flights are around here?', + check: (c) => c.name === 'analyst_query' && pointerArg(c.args.scope?.kind), + want: 'analyst_query{scope.kind:pointer}', + }, + { + pointer: GROUND_POINTER, + phrase: 'Take me there', + check: (c) => c.name === 'fly_to_location' && pointerArg(c.args.query), + want: 'fly_to_location{query:pointer}', + }, + { + pointer: GROUND_POINTER, + phrase: 'Drop a pin right here', + check: (c) => + c.name === 'annotate_map' && + (c.args.annotations || []).some((a) => pointerArg(a.target)), + want: 'annotate_map{target:pointer}', + }, + { + pointer: GROUND_POINTER, + phrase: 'What is this place?', + check: (c) => c.name === 'get_entity_context' && pointerArg(c.args.scope), + want: 'get_entity_context{scope:pointer}', + }, // Open mic / typed turns: the model only learns that something is pointed at. - { pointer: { type: 'pointer_context', at: 'speech_start', pointing: 'aircraft' }, phrase: "What's that?", check: (c) => c.name === 'get_entity_context' && pointerArg(c.args.scope), want: 'get_entity_context{scope:pointer} (pointing:aircraft)' }, - { pointer: { type: 'pointer_context', at: 'speech_start', pointing: 'ground' }, phrase: 'How many flights are around here?', check: (c) => c.name === 'analyst_query' && pointerArg(c.args.scope?.kind), want: 'analyst_query{scope.kind:pointer} (pointing:ground)' }, - { pointer: { type: 'pointer_context', at: 'speech_start', pointing: 'aircraft' }, phrase: 'Follow that one', check: (c) => c.name === 'track_entity' && pointerArg(c.args.query), want: 'track_entity{query:pointer} (pointing:aircraft)' }, + { + pointer: { + type: 'pointer_context', + at: 'speech_start', + pointing: 'aircraft', + }, + phrase: "What's that?", + check: (c) => c.name === 'get_entity_context' && pointerArg(c.args.scope), + want: 'get_entity_context{scope:pointer} (pointing:aircraft)', + }, + { + pointer: { + type: 'pointer_context', + at: 'speech_start', + pointing: 'ground', + }, + phrase: 'How many flights are around here?', + check: (c) => c.name === 'analyst_query' && pointerArg(c.args.scope?.kind), + want: 'analyst_query{scope.kind:pointer} (pointing:ground)', + }, + { + pointer: { + type: 'pointer_context', + at: 'speech_start', + pointing: 'aircraft', + }, + phrase: 'Follow that one', + check: (c) => c.name === 'track_entity' && pointerArg(c.args.query), + want: 'track_entity{query:pointer} (pointing:aircraft)', + }, // A named place wins over the pointer. - { pointer: GROUND_POINTER, phrase: 'How many flights are over Texas right now?', check: (c) => c.name === 'analyst_query' && !pointerArg(c.args.scope?.kind), want: 'analyst_query not pointer (named place)' }, + { + pointer: GROUND_POINTER, + phrase: 'How many flights are over Texas right now?', + check: (c) => c.name === 'analyst_query' && !pointerArg(c.args.scope?.kind), + want: 'analyst_query not pointer (named place)', + }, // No pointer_context: the model may still pass the sentinel (observed); the // runner then falls back to scope auto, so only the tool choice is pinned. - { pointer: null, phrase: 'Tell me about this', check: (c) => c.name === 'get_entity_context', want: 'get_entity_context (runner falls back to auto)' }, + { + pointer: null, + phrase: 'Tell me about this', + check: (c) => c.name === 'get_entity_context', + want: 'get_entity_context (runner falls back to auto)', + }, // Referents: a numbered list, then "the second one". - { setup: { phrase: 'Frame the aircraft near us', stub: FRAMED_STUB }, pointer: null, phrase: 'Track the second one', check: (c) => c.name === 'track_entity' && (c.args.referent === 2 || /ups793|a22222/i.test(c.args.query || '')), want: 'track_entity{referent:2}' }, + { + setup: { phrase: 'Frame the aircraft near us', stub: FRAMED_STUB }, + pointer: null, + phrase: 'Track the second one', + check: (c) => + c.name === 'track_entity' && + (c.args.referent === 2 || /ups793|a22222/i.test(c.args.query || '')), + want: 'track_entity{referent:2}', + }, ]; async function runDeixisRoutingLayer() { - console.log(`\nLAYER 1b — deixis routing (budget ${TURN_BUDGET} model turns)`); + console.log( + `\nLAYER 1b — deixis routing (budget ${TURN_BUDGET} model turns)`, + ); const logDir = path.join(ROOT, '.gev-logs', 'qa-voice-routing'); fs.mkdirSync(logDir, { recursive: true }); - const evidence = fs.createWriteStream(path.join(logDir, `deixis-${Date.now()}.jsonl`)); + const evidence = fs.createWriteStream( + path.join(logDir, `deixis-${Date.now()}.jsonl`), + ); let turnsUsed = 0; let referentUse = 0; for (const scenario of DEIXIS_SCENARIOS) { - const where = !scenario.pointer ? 'no pointer' : scenario.pointer.pointing ? `pointing ${scenario.pointer.pointing}` : scenario.pointer.target ? scenario.pointer.target.label : 'ground'; + const where = !scenario.pointer + ? 'no pointer' + : scenario.pointer.pointing + ? `pointing ${scenario.pointer.pointing}` + : scenario.pointer.target + ? scenario.pointer.target.label + : 'ground'; const label = `deixis route: [${where}] "${scenario.phrase}" → ${scenario.want}`; const cost = scenario.setup ? 2 : 1; - if (turnsUsed + cost > TURN_BUDGET) { skipped(label, 'turn budget exhausted'); continue; } + if (turnsUsed + cost > TURN_BUDGET) { + skipped(label, 'turn budget exhausted'); + continue; + } let session; try { const { value, model } = await mintSession(); @@ -339,7 +622,16 @@ async function runDeixisRoutingLayer() { const calls = await session.runTurn(scenario.phrase); const hit = calls.find(scenario.check); if (hit?.args?.referent !== undefined) referentUse += 1; - report(Boolean(hit), label, `called: ${calls.map((c) => `${c.name}${JSON.stringify(c.args)}`).join(' ').slice(0, 220) || '(none)'}`); + report( + Boolean(hit), + label, + `called: ${ + calls + .map((c) => `${c.name}${JSON.stringify(c.args)}`) + .join(' ') + .slice(0, 220) || '(none)' + }`, + ); } catch (e) { report(false, label, `turn error: ${e.message}`); } finally { @@ -347,7 +639,9 @@ async function runDeixisRoutingLayer() { } } evidence.end(); - console.log(` deixis routing turns used: ${turnsUsed}/${TURN_BUDGET}; referent:n used ${referentUse}x`); + console.log( + ` deixis routing turns used: ${turnsUsed}/${TURN_BUDGET}; referent:n used ${referentUse}x`, + ); } // ════════════════════════════════════════════════════════════ @@ -380,19 +674,35 @@ async function runDeixisLiveLayer() { }); const appUrl = new URL(APP_URL); appUrl.searchParams.set('welcome', '0'); - await browser.defaultBrowserContext().overridePermissions(appUrl.origin, ['microphone']); + await browser + .defaultBrowserContext() + .overridePermissions(appUrl.origin, ['microphone']); const page = await browser.newPage(); await page.setViewport({ width: 1500, height: 950 }); const settle = (ms) => new Promise((r) => setTimeout(r, ms)); - const run = (name, args) => page.evaluate( - (n, a) => Promise.resolve(window.__gevVoiceCommands.runner(n, a)).catch((e) => ({ ok: false, error: String(e?.message || e) })), - name, args, - ); + const run = (name, args) => + page.evaluate( + (n, a) => + Promise.resolve(window.__gevVoiceCommands.runner(n, a)).catch((e) => ({ + ok: false, + error: String(e?.message || e), + })), + name, + args, + ); let turnsUsed = 0; try { - await page.goto(appUrl.href, { waitUntil: 'domcontentloaded', timeout: 60000 }); - await page.waitForFunction(() => window.__godsEyeView?.viewer && window.__gevVoiceCommands?.session, { timeout: 120000, polling: 250 }); - await page.evaluate(() => window.__godsEyeView.viewer.camera.cancelFlight()); + await page.goto(appUrl.href, { + waitUntil: 'domcontentloaded', + timeout: 60000, + }); + await page.waitForFunction( + () => window.__godsEyeView?.viewer && window.__gevVoiceCommands?.session, + { timeout: 120000, polling: 250 }, + ); + await page.evaluate(() => + window.__godsEyeView.viewer.camera.cancelFlight(), + ); await run('fly_to_location', { query: 'Austin, Texas' }); await settle(9000); await run('set_layer_visibility', { layerId: 'flights', enabled: true }); @@ -408,7 +718,15 @@ async function runDeixisLiveLayer() { state: event.state || null, args: event.arguments || null, result: event.result - ? { ok: event.result.ok, error: event.result.error || null, say: event.result.say ?? null, resolvedFrom: event.result.resolvedFrom || null, selectedId: event.result.selected?.id || null, scopeLabel: event.result.scopeLabel || null, label: event.result.label || null } + ? { + ok: event.result.ok, + error: event.result.error || null, + say: event.result.say ?? null, + resolvedFrom: event.result.resolvedFrom || null, + selectedId: event.result.selected?.id || null, + scopeLabel: event.result.scopeLabel || null, + label: event.result.label || null, + } : null, text: event.type === 'transcript' && event.final ? event.text : null, role: event.role || null, @@ -416,7 +734,10 @@ async function runDeixisLiveLayer() { }); return window.__gevVoiceCommands.session.start({ pushToTalk: true }); }); - await page.waitForFunction(() => window.__gevVoiceCommands.status === 'listening', { timeout: 45000, polling: 250 }); + await page.waitForFunction( + () => window.__gevVoiceCommands.status === 'listening', + { timeout: 45000, polling: 250 }, + ); // The pointer moves and the text is sent in one step, as when someone // points and types: typed turns count only a pointer moved in the last @@ -424,57 +745,99 @@ async function runDeixisLiveLayer() { const turn = async (text, point = null) => { turnsUsed += 1; const from = await page.evaluate(() => window.__qaEvents.length); - await page.evaluate((t, p) => { - let at = p; - if (p?.aircraftId) { - // Aircraft move: project the contact's live position in this step. - const { viewer, dataManager } = window.__godsEyeView; - // The billboard position, which is what the pick hits. - const found = (dataManager.layers.get('flights')?.module?.getAllPositions?.(3000) || []).find((e) => e.id === p.aircraftId); - viewer.scene.render(); - at = found?.position ? viewer.scene.cartesianToCanvasCoordinates(found.position) : null; - } - if (at) window.__gevVoiceCommands.pointer.moveTo(at.x, at.y); - window.__gevVoiceCommands.sendTextCommand(t); - }, text, point); + await page.evaluate( + (t, p) => { + let at = p; + if (p?.aircraftId) { + // Aircraft move: project the contact's live position in this step. + const { viewer, dataManager } = window.__godsEyeView; + // The billboard position, which is what the pick hits. + const found = ( + dataManager.layers + .get('flights') + ?.module?.getAllPositions?.(3000) || [] + ).find((e) => e.id === p.aircraftId); + viewer.scene.render(); + at = found?.position + ? viewer.scene.cartesianToCanvasCoordinates(found.position) + : null; + } + if (at) window.__gevVoiceCommands.pointer.moveTo(at.x, at.y); + window.__gevVoiceCommands.sendTextCommand(t); + }, + text, + point, + ); // Done when a reply completes after the last tool result (or no tool). const deadline = Date.now() + 90000; let events = []; while (Date.now() < deadline) { await settle(500); events = await page.evaluate((i) => window.__qaEvents.slice(i), from); - const lastResult = events.map((e) => e.type).lastIndexOf('action-result'); + const lastResult = events + .map((e) => e.type) + .lastIndexOf('action-result'); const lastCall = events.map((e) => e.type).lastIndexOf('action-call'); - const answered = events.some((e, i) => e.type === 'transcript' && e.role === 'assistant' && e.text && i > lastResult); + const answered = events.some( + (e, i) => + e.type === 'transcript' && + e.role === 'assistant' && + e.text && + i > lastResult, + ); if (answered && lastResult >= lastCall) break; } return events; }; - const summarize = (events) => events.filter((e) => e.type === 'action-call' || e.type === 'action-result') - .map((e) => (e.type === 'action-call' ? `${e.name}${JSON.stringify(e.args)}` : `→${JSON.stringify(e.result)}`)).join(' ').slice(0, 320); + const summarize = (events) => + events + .filter((e) => e.type === 'action-call' || e.type === 'action-result') + .map((e) => + e.type === 'action-call' + ? `${e.name}${JSON.stringify(e.args)}` + : `→${JSON.stringify(e.result)}`, + ) + .join(' ') + .slice(0, 320); const aircraft = await findPickableAircraft(page); if (!aircraft) { - skipped('deixis live: aircraft scenarios', 'no pickable aircraft on screen'); + skipped( + 'deixis live: aircraft scenarios', + 'no pickable aircraft on screen', + ); } else { - let events = await turn('Tell me about this', { aircraftId: aircraft.id }); - let call = events.find((e) => e.type === 'action-call' && e.name === 'get_entity_context'); - let result = events.find((e) => e.type === 'action-result' && e.name === 'get_entity_context'); + let events = await turn('Tell me about this', { + aircraftId: aircraft.id, + }); + let call = events.find( + (e) => e.type === 'action-call' && e.name === 'get_entity_context', + ); + let result = events.find( + (e) => e.type === 'action-result' && e.name === 'get_entity_context', + ); report( - call?.args?.scope === 'pointer' && result?.result?.selectedId === aircraft.id, + call?.args?.scope === 'pointer' && + result?.result?.selectedId === aircraft.id, `deixis live: [${aircraft.label}] "Tell me about this" → the pointed aircraft`, summarize(events), ); if (SHOT_DIR) { const dir = path.resolve(ROOT, SHOT_DIR); fs.mkdirSync(dir, { recursive: true }); - await page.screenshot({ path: path.join(dir, 'live-tell-me-about-this.png') }); + await page.screenshot({ + path: path.join(dir, 'live-tell-me-about-this.png'), + }); } events = await turn('Track that one', { aircraftId: aircraft.id }); - result = events.find((e) => e.type === 'action-result' && e.name === 'track_entity'); + result = events.find( + (e) => e.type === 'action-result' && e.name === 'track_entity', + ); report( // By the pointer, or by the name the previous answer just gave it. - result?.result?.ok === true && (result.result.resolvedFrom?.source === 'pointer' || result.result.label === aircraft.label), + result?.result?.ok === true && + (result.result.resolvedFrom?.source === 'pointer' || + result.result.label === aircraft.label), `deixis live: [${aircraft.label}] "Track that one" → tracks it`, summarize(events), ); @@ -482,18 +845,35 @@ async function runDeixisLiveLayer() { await settle(3000); } - const size = await page.evaluate(() => ({ w: window.__godsEyeView.viewer.scene.canvas.clientWidth, h: window.__godsEyeView.viewer.scene.canvas.clientHeight })); - const events = await turn('How many flights are around here?', { x: size.w * 0.3, y: size.h * 0.62 }); - const result = events.find((e) => e.type === 'action-result' && e.name === 'analyst_query'); + const size = await page.evaluate(() => ({ + w: window.__godsEyeView.viewer.scene.canvas.clientWidth, + h: window.__godsEyeView.viewer.scene.canvas.clientHeight, + })); + const events = await turn('How many flights are around here?', { + x: size.w * 0.3, + y: size.h * 0.62, + }); + const result = events.find( + (e) => e.type === 'action-result' && e.name === 'analyst_query', + ); report( - result?.result?.resolvedFrom?.source === 'pointer' && /within/i.test(result?.result?.scopeLabel || ''), + result?.result?.resolvedFrom?.source === 'pointer' && + /within/i.test(result?.result?.scopeLabel || ''), 'deixis live: [ground] "How many flights are around here?" → radius at the pointer', summarize(events), ); } catch (e) { - report(false, 'deixis live: layer crashed', String(e?.message || e).slice(0, 200)); + report( + false, + 'deixis live: layer crashed', + String(e?.message || e).slice(0, 200), + ); } finally { - try { await page.evaluate(() => window.__gevVoiceCommands?.session?.stop()); } catch { /* closing */ } + try { + await page.evaluate(() => window.__gevVoiceCommands?.session?.stop()); + } catch { + /* closing */ + } await browser.close(); console.log(` deixis live turns used: ${turnsUsed}`); } @@ -501,17 +881,40 @@ async function runDeixisLiveLayer() { // Speech stats: informational, never pass/fail. Slow tools are the ones whose // latency justifies a spoken plan; instant ones should answer after the call. -const SLOW_TOOLS = new Set(['annotate_map', 'fly_to_location', 'select_nearest_aircraft', 'analyst_query', 'fly_route', 'frame_overhead']); +const SLOW_TOOLS = new Set([ + 'annotate_map', + 'fly_to_location', + 'select_nearest_aircraft', + 'analyst_query', + 'fly_route', + 'frame_overhead', +]); const speechTurns = []; function recordSpeech(phrase, names, turn) { if (!turn) return; - const firstCallResponse = turn.callResponses.length ? Math.min(...turn.callResponses) : null; + const firstCallResponse = turn.callResponses.length + ? Math.min(...turn.callResponses) + : null; const preamble = turn.messages - .filter((m) => firstCallResponse !== null && m.response === firstCallResponse && m.callsBefore === 0) - .map((m) => m.text).join(' '); + .filter( + (m) => + firstCallResponse !== null && + m.response === firstCallResponse && + m.callsBefore === 0, + ) + .map((m) => m.text) + .join(' '); const answer = turn.messages - .filter((m) => !(firstCallResponse !== null && m.response === firstCallResponse && m.callsBefore === 0)) - .map((m) => m.text).join(' '); + .filter( + (m) => + !( + firstCallResponse !== null && + m.response === firstCallResponse && + m.callsBefore === 0 + ), + ) + .map((m) => m.text) + .join(' '); speechTurns.push({ phrase, tools: names, @@ -527,7 +930,10 @@ function summarizeSpeech(logDir) { const toolTurns = speechTurns.filter((t) => t.tools.length); const slow = toolTurns.filter((t) => t.slow); const instant = toolTurns.filter((t) => !t.slow); - const rate = (list) => (list.length ? `${list.filter((t) => t.preamble).length}/${list.length}` : '0/0'); + const rate = (list) => + list.length + ? `${list.filter((t) => t.preamble).length}/${list.length}` + : '0/0'; const words = speechTurns.map((t) => t.answerWords).sort((a, b) => a - b); const summary = { turns: speechTurns.length, @@ -538,10 +944,18 @@ function summarizeSpeech(logDir) { answerWordsTotal: words.reduce((a, b) => a + b, 0), hedgeHits: speechTurns.reduce((sum, t) => sum + t.hedges.length, 0), preambleWordsMax: Math.max(0, ...speechTurns.map((t) => t.preambleWords)), - spokenWordsTotal: speechTurns.reduce((sum, t) => sum + t.answerWords + t.preambleWords, 0), + spokenWordsTotal: speechTurns.reduce( + (sum, t) => sum + t.answerWords + t.preambleWords, + 0, + ), }; - fs.writeFileSync(path.join(logDir, `speech-${Date.now()}.json`), JSON.stringify({ summary, turns: speechTurns }, null, 2)); - console.log(` speech: preamble slow ${summary.preambleSlow}, instant ${summary.preambleInstant}; answer words p50 ${summary.answerWordsP50}, max ${summary.answerWordsMax}, total ${summary.answerWordsTotal}; preamble words max ${summary.preambleWordsMax}; all spoken words ${summary.spokenWordsTotal}; hedge hits ${summary.hedgeHits}`); + fs.writeFileSync( + path.join(logDir, `speech-${Date.now()}.json`), + JSON.stringify({ summary, turns: speechTurns }, null, 2), + ); + console.log( + ` speech: preamble slow ${summary.preambleSlow}, instant ${summary.preambleInstant}; answer words p50 ${summary.answerWordsP50}, max ${summary.answerWordsMax}, total ${summary.answerWordsTotal}; preamble words max ${summary.preambleWordsMax}; all spoken words ${summary.spokenWordsTotal}; hedge hits ${summary.hedgeHits}`, + ); } // ════════════════════════════════════════════════════════════ @@ -563,8 +977,22 @@ function findPickableAircraft(page) { const p = entry.position; if (eye.x * p.x + eye.y * p.y + eye.z * p.z < 6356000 ** 2) continue; const at = viewer.scene.cartesianToCanvasCoordinates(entry.position); - if (!at || at.x < 60 || at.y < 60 || at.x > canvas.clientWidth - 60 || at.y > canvas.clientHeight - 240) continue; - candidates.push({ entry, at, d: Math.hypot(at.x - canvas.clientWidth / 2, at.y - canvas.clientHeight * 0.45) }); + if ( + !at || + at.x < 60 || + at.y < 60 || + at.x > canvas.clientWidth - 60 || + at.y > canvas.clientHeight - 240 + ) + continue; + candidates.push({ + entry, + at, + d: Math.hypot( + at.x - canvas.clientWidth / 2, + at.y - canvas.clientHeight * 0.45, + ), + }); } // Nearest the middle first: clear of the dock and panels. candidates.sort((a, b) => a.d - b.d); @@ -593,67 +1021,131 @@ async function runDeixisBehavior(page, run, settle) { fs.mkdirSync(dir, { recursive: true }); await page.screenshot({ path: path.join(dir, name) }); }; - const beginAt = (x, y, said = null) => page.evaluate((px, py, text) => { - const voice = window.__gevVoiceCommands; - if (text) { - // What the session emits for a spoken turn: a new turn, then the caption. - voice.emitSessionEvent({ type: 'interruption', reason: 'user-speech' }); - voice.emitSessionEvent({ type: 'transcript', role: 'user', text, final: true }); - } - voice.pointer.moveTo(px, py); - // A Space hold: the explicit pointing gesture shows the reticle and chip. - voice.pointer.hold('keydown'); - const snapshot = voice.beginPointerTurn('keydown'); - voice.pointer.releaseHold(); - const chip = document.getElementById('gev-voice-card-pointer'); - const reticle = document.querySelector('.gev-pointer-reticle'); - return { - snapshot, - chipVisible: Boolean(chip && !chip.hidden), - chipKind: document.getElementById('gev-voice-card-pointer-kind')?.textContent || '', - chipText: document.getElementById('gev-voice-card-pointer-text')?.textContent || '', - reticleVisible: Boolean(reticle && !reticle.hidden), - }; - }, x, y, said); - const showResult = (name, result) => page.evaluate((n, res) => { - // The session publishes every runner result to the card this way. - window.__gevVoiceCommands.emitSessionEvent({ type: 'action-call', name: n, arguments: {} }); - window.__gevVoiceCommands.emitSessionEvent({ type: 'action-result', name: n, result: res }); - }, name, result); + const beginAt = (x, y, said = null) => + page.evaluate( + (px, py, text) => { + const voice = window.__gevVoiceCommands; + if (text) { + // What the session emits for a spoken turn: a new turn, then the caption. + voice.emitSessionEvent({ + type: 'interruption', + reason: 'user-speech', + }); + voice.emitSessionEvent({ + type: 'transcript', + role: 'user', + text, + final: true, + }); + } + voice.pointer.moveTo(px, py); + // A Space hold: the explicit pointing gesture shows the reticle and chip. + voice.pointer.hold('keydown'); + const snapshot = voice.beginPointerTurn('keydown'); + voice.pointer.releaseHold(); + const chip = document.getElementById('gev-voice-card-pointer'); + const reticle = document.querySelector('.gev-pointer-reticle'); + return { + snapshot, + chipVisible: Boolean(chip && !chip.hidden), + chipKind: + document.getElementById('gev-voice-card-pointer-kind') + ?.textContent || '', + chipText: + document.getElementById('gev-voice-card-pointer-text') + ?.textContent || '', + reticleVisible: Boolean(reticle && !reticle.hidden), + }; + }, + x, + y, + said, + ); + const showResult = (name, result) => + page.evaluate( + (n, res) => { + // The session publishes every runner result to the card this way. + window.__gevVoiceCommands.emitSessionEvent({ + type: 'action-call', + name: n, + arguments: {}, + }); + window.__gevVoiceCommands.emitSessionEvent({ + type: 'action-result', + name: n, + result: res, + }); + }, + name, + result, + ); // D1 — an aircraft under the pointer: find one whose billboard really picks. const aircraft = await findPickableAircraft(page); if (!aircraft) { - skipped('deixis: pointer on an aircraft', 'no pickable aircraft on screen right now'); + skipped( + 'deixis: pointer on an aircraft', + 'no pickable aircraft on screen right now', + ); } else { const turn = await beginAt(aircraft.x, aircraft.y, 'Tell me about this'); report( - turn.snapshot?.entity?.id === aircraft.id && turn.chipVisible && turn.chipKind === 'THIS' && turn.reticleVisible, + turn.snapshot?.entity?.id === aircraft.id && + turn.chipVisible && + turn.chipKind === 'THIS' && + turn.reticleVisible, 'deixis: pointer on an aircraft resolves "this" with chip and reticle', `entity=${turn.snapshot?.entity?.id}/${turn.snapshot?.entity?.label} expected=${aircraft.id} chip=${turn.chipKind}:${turn.chipText} reticle=${turn.reticleVisible}`, ); await shot('aircraft-reticle-chip.png'); if (SHOT_DIR) { - const clip = { x: Math.max(0, aircraft.x - 90), y: Math.max(0, aircraft.y - 90), width: 180, height: 180 }; - await page.screenshot({ path: path.join(path.resolve(ROOT, SHOT_DIR), 'aircraft-reticle-zoom.png'), clip }); + const clip = { + x: Math.max(0, aircraft.x - 90), + y: Math.max(0, aircraft.y - 90), + width: 180, + height: 180, + }; + await page.screenshot({ + path: path.join( + path.resolve(ROOT, SHOT_DIR), + 'aircraft-reticle-zoom.png', + ), + clip, + }); } const context = await run('get_entity_context', { scope: 'pointer' }); report( - context?.ok === true && context.scope === 'pointer' && context.selected?.id === aircraft.id && Boolean(context.identityLine), + context?.ok === true && + context.scope === 'pointer' && + context.selected?.id === aircraft.id && + Boolean(context.identityLine), 'deixis: "tell me about this" describes the pointed aircraft', `selected=${context?.selected?.id} identityLine=${JSON.stringify(context?.identityLine)}`, ); await showResult('get_entity_context', context); - await page.evaluate((line) => window.__gevVoiceCommands.emitSessionEvent({ type: 'transcript', role: 'assistant', text: line, final: true }), context?.identityLine || ''); + await page.evaluate( + (line) => + window.__gevVoiceCommands.emitSessionEvent({ + type: 'transcript', + role: 'assistant', + text: line, + final: true, + }), + context?.identityLine || '', + ); await settle(400); await shot('result-card.png'); const tracked = await run('track_entity', { query: 'pointer' }); const trackedId = await page.evaluate(() => { - const info = window.__godsEyeView.dataManager.layers.get('flights')?.module?.getTrackedInfo?.(); + const info = window.__godsEyeView.dataManager.layers + .get('flights') + ?.module?.getTrackedInfo?.(); return info?.icao24 || info?.id || null; }); report( - tracked?.ok === true && tracked.resolvedFrom?.source === 'pointer' && (trackedId === aircraft.id || trackedId === null), + tracked?.ok === true && + tracked.resolvedFrom?.source === 'pointer' && + (trackedId === aircraft.id || trackedId === null), 'deixis: "track that one" follows the pointed aircraft', `ok=${tracked?.ok} label=${tracked?.label} trackedId=${trackedId} error=${tracked?.error || ''}`, ); @@ -667,58 +1159,109 @@ async function runDeixisBehavior(page, run, settle) { return { w: canvas.clientWidth, h: canvas.clientHeight }; }); let groundTurn = null; - for (const [fx, fy] of [[0.3, 0.62], [0.7, 0.62], [0.5, 0.7], [0.2, 0.5]]) { + for (const [fx, fy] of [ + [0.3, 0.62], + [0.7, 0.62], + [0.5, 0.7], + [0.2, 0.5], + ]) { groundTurn = await beginAt(size.w * fx, size.h * fy, "What's here?"); if (groundTurn.snapshot?.target === 'ground') break; } report( - groundTurn?.snapshot?.target === 'ground' && groundTurn.chipKind === 'HERE' && /-?\d+\.\d\d, -?\d+\.\d\d/.test(groundTurn.chipText) && groundTurn.reticleVisible, + groundTurn?.snapshot?.target === 'ground' && + groundTurn.chipKind === 'HERE' && + /-?\d+\.\d\d, -?\d+\.\d\d/.test(groundTurn.chipText) && + groundTurn.reticleVisible, 'deixis: pointer on bare ground resolves "here" to coordinates', `target=${groundTurn?.snapshot?.target} chip=${groundTurn?.chipKind}:${groundTurn?.chipText} reticle=${groundTurn?.reticleVisible}`, ); await shot('ground-reticle-chip.png'); // The 512 px crop a ground-only point sends the model (one retained image). - const captured = await page.evaluate((px) => window.__gevVoiceCommands._viewport.capturePointer(px), groundTurn?.snapshot?.screenPx || null); + const captured = await page.evaluate( + (px) => window.__gevVoiceCommands._viewport.capturePointer(px), + groundTurn?.snapshot?.screenPx || null, + ); const crop = captured?.dataUrl || null; + const cropInspection = await inspectPointerCrop( + captured, + groundTurn?.snapshot?.screenPx, + size, + ).catch(() => null); report( - typeof crop === 'string' && crop.startsWith('data:image/jpeg') && crop.length > 4000 && captured.rect?.w > 0, + typeof crop === 'string' && + crop.startsWith('data:image/jpeg') && + crop.length > 4000 && + captured.rect?.w > 0 && + cropInspection?.width === 512 && + cropInspection?.height === 512 && + cropInspection?.ringPixels >= 20 && + cropInspection?.usableRatio >= 0.05, 'deixis: a ground point yields a 512 px crop with the ring', - `bytes~${crop ? Math.round((crop.length * 3) / 4 / 1024) : 0} KB rect=${JSON.stringify(captured?.rect)}`, + `bytes~${crop ? Math.round((crop.length * 3) / 4 / 1024) : 0} KB rect=${JSON.stringify(captured?.rect)} inspection=${JSON.stringify(cropInspection)}`, ); if (SHOT_DIR && crop) { - fs.writeFileSync(path.join(path.resolve(ROOT, SHOT_DIR), 'ground-crop.jpg'), Buffer.from(crop.split(',')[1], 'base64')); + fs.writeFileSync( + path.join(path.resolve(ROOT, SHOT_DIR), 'ground-crop.jpg'), + Buffer.from(crop.split(',')[1], 'base64'), + ); } const where = await run('get_entity_context', { scope: 'pointer' }); report( - where?.ok === true && where.scope === 'pointer' && Number.isFinite(where.pointer?.latitude ?? where.selected?.latitude), + where?.ok === true && + where.scope === 'pointer' && + Number.isFinite(where.pointer?.latitude ?? where.selected?.latitude), 'deixis: "what is here" reads context at the pointer, not the view centre', `scope=${where?.scope} pointer=${JSON.stringify(where?.pointer)}`, ); - const around = await run('analyst_query', { layers: ['flights'], scope: { kind: 'pointer' } }); + const around = await run('analyst_query', { + layers: ['flights'], + scope: { kind: 'pointer' }, + }); report( - around?.ok !== false && around?.resolvedFrom?.source === 'pointer' && /within/i.test(around?.scopeLabel || ''), + around?.ok !== false && + around?.resolvedFrom?.source === 'pointer' && + /within/i.test(around?.scopeLabel || ''), 'deixis: "how many flights around here" counts a radius at the pointer', `count=${around?.count} scope=${around?.scopeLabel} km=${around?.resolvedFrom?.km}`, ); - const pin = await run('annotate_map', { annotations: [{ type: 'pin', target: 'pointer', label: 'Here' }] }); - report(pin?.ok === true && pin.drawn >= 1, 'deixis: "drop a pin here" marks the pointed spot', `drawn=${pin?.drawn} failed=${JSON.stringify(pin?.failedLabels)}`); + const pin = await run('annotate_map', { + annotations: [{ type: 'pin', target: 'pointer', label: 'Here' }], + }); + report( + pin?.ok === true && pin.drawn >= 1, + 'deixis: "drop a pin here" marks the pointed spot', + `drawn=${pin?.drawn} failed=${JSON.stringify(pin?.failedLabels)}`, + ); // D3 — referents: "track the second one" after a numbered list. const framed = await run('frame_overhead', { target: 'flights' }); await settle(5000); const refs = Array.isArray(framed?.referents) ? framed.referents : []; if (refs.length < 2) { - skipped('deixis: "track the second one"', `only ${refs.length} referents in frame`); + skipped( + 'deixis: "track the second one"', + `only ${refs.length} referents in frame`, + ); } else { - const second = await run('track_entity', { query: 'the second one', referent: 2 }); + const second = await run('track_entity', { + query: 'the second one', + referent: 2, + }); report( - second?.ok === true && second.resolvedFrom?.source === 'referent' && second.label === refs[1].label, + second?.ok === true && + second.resolvedFrom?.source === 'referent' && + second.label === refs[1].label, 'deixis: "track the second one" follows referent 2', `label=${second?.label} expected=${refs[1].label}`, ); await run('stop_tracking', {}); const missing = await run('track_entity', { query: 'x', referent: 40 }); - report(missing?.ok === false && /no item 40/.test(missing?.error || ''), 'deixis: an out-of-range referent is a plain refusal', missing?.error); + report( + missing?.ok === false && /no item 40/.test(missing?.error || ''), + 'deixis: an out-of-range referent is a plain refusal', + missing?.error, + ); } // D4 — a stale pointer is never "this". @@ -727,11 +1270,17 @@ async function runDeixisBehavior(page, run, settle) { voice.pointer.element.dispatchEvent(new PointerEvent('pointerleave')); const snapshot = voice.beginPointerTurn('text'); const chip = document.getElementById('gev-voice-card-pointer'); - return { fresh: Boolean(snapshot?.fresh), chipHidden: Boolean(chip?.hidden) }; + return { + fresh: Boolean(snapshot?.fresh), + chipHidden: Boolean(chip?.hidden), + }; }); const refused = await run('track_entity', { query: 'pointer' }); report( - !stale.fresh && stale.chipHidden && refused?.ok === false && /Nothing is under the pointer/.test(refused?.error || ''), + !stale.fresh && + stale.chipHidden && + refused?.ok === false && + /Nothing is under the pointer/.test(refused?.error || ''), 'deixis: a pointer off the canvas is refused, not guessed', `fresh=${stale.fresh} chipHidden=${stale.chipHidden} error=${refused?.error}`, ); @@ -743,7 +1292,9 @@ async function runBehaviorLayer() { const res = await fetch(APP_URL); if (!res.ok) throw new Error(String(res.status)); } catch (e) { - console.error(`\x1b[31mDev server not reachable at ${APP_URL} (${e.message}).\x1b[0m`); + console.error( + `\x1b[31mDev server not reachable at ${APP_URL} (${e.message}).\x1b[0m`, + ); process.exitCode = 1; return; } @@ -765,18 +1316,23 @@ async function runBehaviorLayer() { }); const page = await browser.newPage(); await page.setViewport({ width: 1500, height: 950 }); - page.on('pageerror', (e) => console.log(` [page error] ${String(e).slice(0, 140)}`)); + page.on('pageerror', (e) => + console.log(` [page error] ${String(e).slice(0, 140)}`), + ); // Voice must not reach Overpass unless an operator configured it: count // Overpass queries through the app and any direct request to an Overpass // host (the status probe is not a query). const overpassRequests = []; page.on('request', (request) => { const target = new URL(request.url()); - const viaApp = target.origin === new URL(APP_URL).origin - && target.pathname.startsWith('/api/overpass') - && target.pathname !== '/api/overpass/status'; + const viaApp = + target.origin === new URL(APP_URL).origin && + target.pathname.startsWith('/api/overpass') && + target.pathname !== '/api/overpass/status'; if (viaApp || /overpass/i.test(target.hostname)) - overpassRequests.push(`${request.method()} ${target.origin}${target.pathname}`); + overpassRequests.push( + `${request.method()} ${target.origin}${target.pathname}`, + ); }); try { @@ -784,24 +1340,46 @@ async function runBehaviorLayer() { // screenshots show the map; the full run keeps the historical URL. const behaviorUrl = new URL(APP_URL); if (DEIXIS_ONLY) behaviorUrl.searchParams.set('welcome', '0'); - await page.goto(behaviorUrl.href, { waitUntil: 'domcontentloaded', timeout: 60000 }); + await page.goto(behaviorUrl.href, { + waitUntil: 'domcontentloaded', + timeout: 60000, + }); await page.waitForFunction( - () => window.__godsEyeView?.viewer && window.__gevVoiceCommands?.runner && window.__gevAnnotations, + () => + window.__godsEyeView?.viewer && + window.__gevVoiceCommands?.runner && + window.__gevAnnotations, { timeout: 120000, polling: 250 }, ); // House rule: the intro flight clobbers teleports issued mid-flight. - await page.evaluate(() => window.__godsEyeView.viewer.camera.cancelFlight()); - - const run = (name, args) => page.evaluate( - (n, a) => Promise.resolve(window.__gevVoiceCommands.runner(n, a)) - .catch((e) => ({ ok: false, error: String(e?.message || e), _threw: true })), - name, args, + await page.evaluate(() => + window.__godsEyeView.viewer.camera.cancelFlight(), ); - const camState = () => page.evaluate(() => { - const c = window.__godsEyeView.viewer.camera; - const p = c.positionCartographic; - return { lat: p.latitude * 180 / Math.PI, lon: p.longitude * 180 / Math.PI, altKm: p.height / 1000, pitchDeg: c.pitch * 180 / Math.PI }; - }); + + const run = (name, args) => + page.evaluate( + (n, a) => + Promise.resolve(window.__gevVoiceCommands.runner(n, a)).catch( + (e) => ({ + ok: false, + error: String(e?.message || e), + _threw: true, + }), + ), + name, + args, + ); + const camState = () => + page.evaluate(() => { + const c = window.__godsEyeView.viewer.camera; + const p = c.positionCartographic; + return { + lat: (p.latitude * 180) / Math.PI, + lon: (p.longitude * 180) / Math.PI, + altKm: p.height / 1000, + pitchDeg: (c.pitch * 180) / Math.PI, + }; + }); const settle = (ms) => new Promise((r) => setTimeout(r, ms)); // (1) fly_to_location lands near the target. Annotations follow IMMEDIATELY @@ -811,8 +1389,11 @@ async function runBehaviorLayer() { await settle(9000); let cam = await camState(); const dAustin = Math.hypot(cam.lat - 30.2672, cam.lon + 97.7431); - report(r?.ok !== false && dAustin < 1.2, 'behavior: fly_to_location Austin lands nearby', - `cam=(${cam.lat.toFixed(3)},${cam.lon.toFixed(3)}) Δ=${dAustin.toFixed(2)}° alt=${cam.altKm.toFixed(1)}km`); + report( + r?.ok !== false && dAustin < 1.2, + 'behavior: fly_to_location Austin lands nearby', + `cam=(${cam.lat.toFixed(3)},${cam.lon.toFixed(3)}) Δ=${dAustin.toFixed(2)}° alt=${cam.altKm.toFixed(1)}km`, + ); // Radio voice control selects by semantic category + place without moving // the just-established Austin camera, then preserves explicit volume and @@ -820,48 +1401,90 @@ async function runBehaviorLayer() { // `fly_to_location` may finish its promise just before Cesium drains the // final tween frame. Cancel that already-arrived flight so the assertion // measures Radio's camera ownership, not residual navigation motion. - await page.evaluate(() => window.__godsEyeView.viewer.camera.cancelFlight()); + await page.evaluate(() => + window.__godsEyeView.viewer.camera.cancelFlight(), + ); await settle(250); const radioCameraBefore = await camState(); - r = await run('control_radio', { action: 'select', category: 'news', locationId: 'austin' }); + r = await run('control_radio', { + action: 'select', + category: 'news', + locationId: 'austin', + }); const radioCameraAfter = await camState(); const radioCameraDelta = Math.hypot( radioCameraAfter.lat - radioCameraBefore.lat, radioCameraAfter.lon - radioCameraBefore.lon, ); - report(r?.ok === true && r?.category === 'news' && r?.stationId && radioCameraDelta < 0.001, + report( + r?.ok === true && + r?.category === 'news' && + r?.stationId && + radioCameraDelta < 0.001, 'behavior: Radio selects Austin news without moving the camera', - `stationId=${r?.stationId || 'none'} category=${r?.category} cameraΔ=${radioCameraDelta.toFixed(5)}°`); + `stationId=${r?.stationId || 'none'} category=${r?.category} cameraΔ=${radioCameraDelta.toFixed(5)}°`, + ); r = await run('control_radio', { action: 'volume', volumePct: 30 }); - report(r?.ok === true && r?.volumePct === 30, - 'behavior: Radio voice volume reaches the public player', `volume=${r?.volumePct}`); + report( + r?.ok === true && r?.volumePct === 30, + 'behavior: Radio voice volume reaches the public player', + `volume=${r?.volumePct}`, + ); r = await run('control_radio', { action: 'stop' }); - report(r?.ok === true && r?.audioState === 'stopped', - 'behavior: Radio voice stop releases playback', `audioState=${r?.audioState}`); + report( + r?.ok === true && r?.audioState === 'stopped', + 'behavior: Radio voice stop releases playback', + `audioState=${r?.audioState}`, + ); // (2) two pins near the camera await run('clear_annotations', {}); await settle(400); r = await run('annotate_map', { annotations: [ - { type: 'pin', target: 'Texas State Capitol, Austin', label: 'Capitol' }, + { + type: 'pin', + target: 'Texas State Capitol, Austin', + label: 'Capitol', + }, { type: 'pin', target: 'Zilker Park, Austin', label: 'Zilker' }, ], }); await settle(2500); - let annoCount = await page.evaluate(() => window.__gevAnnotations?.list?.().length ?? window.__gevAnnotations?.count?.() ?? -1); - report((r?.drawn ?? 0) >= 2 && (annoCount >= 2 || annoCount === -1), - 'behavior: two pins drawn near camera', `result.drawn=${r?.drawn} failed=${r?.failed} listCount=${annoCount}`); + let annoCount = await page.evaluate( + () => + window.__gevAnnotations?.list?.().length ?? + window.__gevAnnotations?.count?.() ?? + -1, + ); + report( + (r?.drawn ?? 0) >= 2 && (annoCount >= 2 || annoCount === -1), + 'behavior: two pins drawn near camera', + `result.drawn=${r?.drawn} failed=${r?.failed} listCount=${annoCount}`, + ); // (3) walking route between them (street-following path, sane result) r = await run('annotate_map', { - annotations: [{ type: 'route', points: [{ target: 'Texas State Capitol, Austin' }, { target: 'Zilker Park, Austin' }], mode: 'walking', label: 'Capitol → Zilker walk' }], + annotations: [ + { + type: 'route', + points: [ + { target: 'Texas State Capitol, Austin' }, + { target: 'Zilker Park, Austin' }, + ], + mode: 'walking', + label: 'Capitol → Zilker walk', + }, + ], }); await settle(6000); - const routeItem = (r?.items || []).find((it) => it.type === 'route') || (r?.items || [])[0]; - report((r?.drawn ?? 0) >= 1 && routeItem?.ok !== false, + const routeItem = + (r?.items || []).find((it) => it.type === 'route') || (r?.items || [])[0]; + report( + (r?.drawn ?? 0) >= 1 && routeItem?.ok !== false, 'behavior: walking route Capitol→Zilker draws', - `drawn=${r?.drawn} route=${JSON.stringify(routeItem)?.slice(0, 140)}`); + `drawn=${r?.drawn} route=${JSON.stringify(routeItem)?.slice(0, 140)}`, + ); if (DEIXIS_ONLY) { await run('set_layer_visibility', { layerId: 'flights', enabled: true }); @@ -869,9 +1492,13 @@ async function runBehaviorLayer() { await run('frame_overhead', { target: 'flights' }); await settle(7000); await runDeixisBehavior(page, run, settle); - report(overpassRequests.length === 0, + report( + overpassRequests.length === 0, 'behavior: no Overpass request during the run', - overpassRequests.length ? overpassRequests.slice(0, 3).join(' | ') : 'none'); + overpassRequests.length + ? overpassRequests.slice(0, 3).join(' | ') + : 'none', + ); return; } @@ -883,11 +1510,22 @@ async function runBehaviorLayer() { r = await run('frame_overhead', { target: 'flights' }); await settle(7000); if (r?.ok === false && /not enabled/i.test(r?.error || '')) { - report(false, 'behavior: frame_overhead frames flights', `layer enable did not stick: ${r?.error}`); + report( + false, + 'behavior: frame_overhead frames flights', + `layer enable did not stick: ${r?.error}`, + ); } else if (r?.ok && (r.count === 0 || r.framedCount === 0)) { - skipped('behavior: frame_overhead frames flights', 'no flights in range right now (empty sky) — framing path OK'); + skipped( + 'behavior: frame_overhead frames flights', + 'no flights in range right now (empty sky) — framing path OK', + ); } else { - report(r?.ok === true, 'behavior: frame_overhead frames flights', `result=${JSON.stringify(r)?.slice(0, 140)}`); + report( + r?.ok === true, + 'behavior: frame_overhead frames flights', + `result=${JSON.stringify(r)?.slice(0, 140)}`, + ); } // (4-deixis) Point-and-ask with a synthetic pointer on the live canvas. @@ -919,41 +1557,58 @@ async function runBehaviorLayer() { let currentKind = null; let vesselSelections = 0; viewer.camera.flyToBoundingSphere = function (...args) { - flightStarts.push({ kind: currentKind, trackingReleased: !viewer.trackedEntity }); + flightStarts.push({ + kind: currentKind, + trackingReleased: !viewer.trackedEntity, + }); return original.flyToBoundingSphere.apply(this, args); }; dataManager.isEnabled = function (id) { - if (['local-firms', 'ais-live-vessels', 'flights'].includes(id)) return true; + if (['local-firms', 'ais-live-vessels', 'flights'].includes(id)) + return true; return original.isEnabled.call(this, id); }; fireEntry.module = { ...original.fireModule, getStrongestFire: () => ({ - id: 'qa-synthetic-fire', label: 'QA synthetic fire', - latitude: 37.7749, longitude: -122.4194, frp: 922, + id: 'qa-synthetic-fire', + label: 'QA synthetic fire', + latitude: 37.7749, + longitude: -122.4194, + frp: 922, }), }; vesselEntry.module = { ...original.vesselModule, findByQuery: () => ({ - mmsi: '999000111', name: 'QA synthetic vessel', - latitude: 29.7604, longitude: -95.3698, + mmsi: '999000111', + name: 'QA synthetic vessel', + latitude: 29.7604, + longitude: -95.3698, }), - selectById: () => { vesselSelections += 1; return true; }, + selectById: () => { + vesselSelections += 1; + return true; + }, }; flightsEntry.module = { ...original.flightsModule, - getNearby: () => [{ - id: 'qa-aircraft', - position: viewer.camera.positionWC.clone(), - }], + getNearby: () => [ + { + id: 'qa-aircraft', + position: viewer.camera.positionWC.clone(), + }, + ], }; const results = {}; try { for (const [kind, args] of [ ['fire', { query: 'strongest fire', layerId: 'local-firms' }], - ['vessel', { query: 'QA synthetic vessel', layerId: 'ais-live-vessels' }], + [ + 'vessel', + { query: 'QA synthetic vessel', layerId: 'ais-live-vessels' }, + ], ]) { const sentinel = viewer.entities.add({ id: `qa-prior-${kind}` }); viewer.trackedEntity = sentinel; @@ -962,7 +1617,8 @@ async function runBehaviorLayer() { const result = await runner('track_entity', args); results[kind] = { ok: result?.ok === true, - generationAdvanced: styleManager._navigationGeneration > generationBefore, + generationAdvanced: + styleManager._navigationGeneration > generationBefore, trackingReleased: !viewer.trackedEntity, }; viewer.camera.cancelFlight(); @@ -971,7 +1627,9 @@ async function runBehaviorLayer() { await runner('move_camera', { motion: 'stop' }); const seededMotion = await runner('move_camera', { - motion: 'pan', direction: 'right', mode: 'continuous', + motion: 'pan', + direction: 'right', + mode: 'continuous', }); const cockpitSentinel = viewer.entities.add({ id: 'qa-cockpit-owner' }); viewer.trackedEntity = cockpitSentinel; @@ -986,20 +1644,25 @@ async function runBehaviorLayer() { ['fly_route', { speed: 'fast' }], ['frame_overhead', { target: 'flights' }], ['track_entity', { query: 'strongest fire', layerId: 'local-firms' }], - ['track_entity', { query: 'QA synthetic vessel', layerId: 'ais-live-vessels' }], + [ + 'track_entity', + { query: 'QA synthetic vessel', layerId: 'ais-live-vessels' }, + ], ]) { refused.push((await runner(name, args))?.ok === false); } results.cockpit = { allRefused: refused.every(Boolean), - generationUnchanged: styleManager._navigationGeneration === generationBefore, + generationUnchanged: + styleManager._navigationGeneration === generationBefore, trackingUnchanged: viewer.trackedEntity === cockpitSentinel, noFlight: flightStarts.length === flightsBefore, noSelection: vesselSelections === selectionsBefore, }; styleManager.cockpitView.active = false; - results.cockpit.motionUnchanged = seededMotion?.ok === true - && (await runner('move_camera', { motion: 'stop' }))?.stopped === true; + results.cockpit.motionUnchanged = + seededMotion?.ok === true && + (await runner('move_camera', { motion: 'stop' }))?.stopped === true; viewer.trackedEntity = undefined; viewer.entities.remove(cockpitSentinel); } finally { @@ -1012,66 +1675,132 @@ async function runBehaviorLayer() { } return { results, flightStarts, vesselSelections }; }); - const takeoversPass = ['fire', 'vessel'].every((kind) => { - const result = ownerTransfer?.results?.[kind]; - const flight = ownerTransfer?.flightStarts?.find((candidate) => candidate.kind === kind); - return result?.ok && result?.generationAdvanced && result?.trackingReleased - && flight?.trackingReleased; - }) && ownerTransfer?.vesselSelections === 1; - report(takeoversPass, + const takeoversPass = + ['fire', 'vessel'].every((kind) => { + const result = ownerTransfer?.results?.[kind]; + const flight = ownerTransfer?.flightStarts?.find( + (candidate) => candidate.kind === kind, + ); + return ( + result?.ok && + result?.generationAdvanced && + result?.trackingReleased && + flight?.trackingReleased + ); + }) && ownerTransfer?.vesselSelections === 1; + report( + takeoversPass, 'behavior: synthetic fire/vessel voice targets take camera authority from tracked aircraft', - `syntheticTarget=true result=${JSON.stringify(ownerTransfer)?.slice(0, 240)}`); + `syntheticTarget=true result=${JSON.stringify(ownerTransfer)?.slice(0, 240)}`, + ); const cockpit = ownerTransfer?.results?.cockpit; - report(Boolean(cockpit?.allRefused && cockpit?.generationUnchanged - && cockpit?.trackingUnchanged && cockpit?.noFlight && cockpit?.noSelection - && cockpit?.motionUnchanged), - 'behavior: Cockpit refuses every named voice camera route before mutation', - `result=${JSON.stringify(cockpit)}`); + report( + Boolean( + cockpit?.allRefused && + cockpit?.generationUnchanged && + cockpit?.trackingUnchanged && + cockpit?.noFlight && + cockpit?.noSelection && + cockpit?.motionUnchanged, + ), + 'behavior: Cockpit refuses every named voice camera route before mutation', + `result=${JSON.stringify(cockpit)}`, + ); // (4b) analyst engine end-to-end: count flights over Texas (region ring // via NE-pack/admin machinery), then a follow-up over the same set. // A cold boundary lookup can exceed the resolver's budget; it answers // region-timeout and keeps loading, so asking again must succeed. let texasTimeouts = 0; - r = await run('analyst_query', { layers: ['flights'], scope: { kind: 'region', name: 'Texas' }, limit: 5 }); + r = await run('analyst_query', { + layers: ['flights'], + scope: { kind: 'region', name: 'Texas' }, + limit: 5, + }); while (r?.code === 'region-timeout' && texasTimeouts < 8) { texasTimeouts += 1; await settle(5000); - r = await run('analyst_query', { layers: ['flights'], scope: { kind: 'region', name: 'Texas' }, limit: 5 }); + r = await run('analyst_query', { + layers: ['flights'], + scope: { kind: 'region', name: 'Texas' }, + limit: 5, + }); } - report(r?.ok === true && Number.isFinite(r?.count) && r.count > 0 && String(r?.coverage?.scope || '').includes('Texas'), - 'behavior: analyst counts flights over Texas', `count=${r?.count} scope=${r?.coverage?.scope} timeouts=${texasTimeouts} err=${r?.error || ''}`); - r = await run('analyst_query', { followUp: true, filters: [{ field: 'onGround', op: 'eq', value: false }], sortBy: 'altitudeM', limit: 3 }); - report(r?.ok === true && r?.coverage?.followUp === true, - 'behavior: analyst follow-up re-filters the remembered set', `count=${r?.count} followUp=${r?.coverage?.followUp}`); + report( + r?.ok === true && + Number.isFinite(r?.count) && + r.count > 0 && + String(r?.coverage?.scope || '').includes('Texas'), + 'behavior: analyst counts flights over Texas', + `count=${r?.count} scope=${r?.coverage?.scope} timeouts=${texasTimeouts} err=${r?.error || ''}`, + ); + r = await run('analyst_query', { + followUp: true, + filters: [{ field: 'onGround', op: 'eq', value: false }], + sortBy: 'altitudeM', + limit: 3, + }); + report( + r?.ok === true && r?.coverage?.followUp === true, + 'behavior: analyst follow-up re-filters the remembered set', + `count=${r?.count} followUp=${r?.coverage?.followUp}`, + ); // A marine region resolves from the bundled Natural Earth pack in the page, // without the slower geocode fallback. const gulfStarted = Date.now(); - r = await run('analyst_query', { layers: ['flights'], scope: { kind: 'region', name: 'Gulf of Mexico' }, limit: 5 }); + r = await run('analyst_query', { + layers: ['flights'], + scope: { kind: 'region', name: 'Gulf of Mexico' }, + limit: 5, + }); const gulfMs = Date.now() - gulfStarted; - report(r?.ok === true && String(r?.coverage?.scope || '').includes('Gulf of Mexico') && gulfMs < 3000, - 'behavior: analyst resolves the Gulf of Mexico from the bundled pack', `scope=${r?.coverage?.scope} ms=${gulfMs} err=${r?.error || ''}`); + report( + r?.ok === true && + String(r?.coverage?.scope || '').includes('Gulf of Mexico') && + gulfMs < 3000, + 'behavior: analyst resolves the Gulf of Mexico from the bundled pack', + `scope=${r?.coverage?.scope} ms=${gulfMs} err=${r?.error || ''}`, + ); // (5) zoom_to_globe is ABSOLUTE full-earth (>12,000 km band) r = await run('zoom_to_globe', {}); - await page.waitForFunction( - () => window.__godsEyeView.viewer.camera.positionCartographic.height / 1000 > 12000, - { timeout: 30_000, polling: 250 }, - ).catch(() => {}); + await page + .waitForFunction( + () => + window.__godsEyeView.viewer.camera.positionCartographic.height / + 1000 > + 12000, + { timeout: 30_000, polling: 250 }, + ) + .catch(() => {}); cam = await camState(); - report(cam.altKm > 12000, 'behavior: zoom_to_globe reaches global band', `alt=${Math.round(cam.altKm)}km (want >12000)`); + report( + cam.altKm > 12000, + 'behavior: zoom_to_globe reaches global band', + `alt=${Math.round(cam.altKm)}km (want >12000)`, + ); // (6) natural-region swath: overview of the Alps must CAP the range - r = await run('fly_to_location', { query: 'the Alps', viewMode: 'overview' }); - await page.waitForFunction( - () => window.__godsEyeView.viewer.camera.positionCartographic.height / 1000 < 900, - { timeout: 40_000, polling: 250 }, - ).catch(() => {}); + r = await run('fly_to_location', { + query: 'the Alps', + viewMode: 'overview', + }); + await page + .waitForFunction( + () => + window.__godsEyeView.viewer.camera.positionCartographic.height / + 1000 < + 900, + { timeout: 40_000, polling: 250 }, + ) + .catch(() => {}); cam = await camState(); const swathMode = r?.navigationMode || r?.mode || '(none)'; - report(String(swathMode).includes('swath') || cam.altKm < 900, + report( + String(swathMode).includes('swath') || cam.altKm < 900, 'behavior: Alps overview uses capped swath, not whole-bbox space view', - `navigationMode=${swathMode} alt=${Math.round(cam.altKm)}km (want swath / <900km)`); + `navigationMode=${swathMode} alt=${Math.round(cam.altKm)}km (want swath / <900km)`, + ); // (6b) THE owner field finding: "outline the Alps" must draw the real // range ring (Natural Earth first-rung, offline → resolves in seconds), @@ -1091,39 +1820,63 @@ async function runBehaviorLayer() { return Array.isArray(alps?.ring) && alps.ring.length >= 8; }); } - report((r?.drawn ?? 0) >= 1 && alpsOutline, + report( + (r?.drawn ?? 0) >= 1 && alpsOutline, 'behavior: "outline the Alps" draws the Natural Earth range ring', - `drawn=${r?.drawn} outlineResolved=${alpsOutline}`); + `drawn=${r?.drawn} outlineResolved=${alpsOutline}`, + ); // (6b2) proximity-gate NEGATIVE control at a LOCAL far view (Alps swath, // ~165 km): annotating Austin from here must be an honest rejection. // NOT run at globe scale — the gate is view-scale-aware by design and a // global view legitimately allows marking anywhere. - r = await run('annotate_map', { annotations: [{ type: 'pin', target: 'Zilker Park, Austin', label: 'FarPin' }] }); + r = await run('annotate_map', { + annotations: [ + { type: 'pin', target: 'Zilker Park, Austin', label: 'FarPin' }, + ], + }); await settle(1500); - report((r?.drawn ?? 1) === 0 || (r?.failed ?? 0) >= 1, + report( + (r?.drawn ?? 1) === 0 || (r?.failed ?? 0) >= 1, 'behavior: far-target annotate is proximity-rejected by design', - `drawn=${r?.drawn} failed=${r?.failed}`); + `drawn=${r?.drawn} failed=${r?.failed}`, + ); // Camera-verb scenarios: shed the heavy layers first — headless // SwiftShader drops to ~1 fps with fires+vessels loaded and every // motion assert starves (environment, not product). - await run('set_layer_visibility', { layerId: 'local-firms', enabled: false }); - await run('set_layer_visibility', { layerId: 'ais-live-vessels', enabled: false }); + await run('set_layer_visibility', { + layerId: 'local-firms', + enabled: false, + }); + await run('set_layer_visibility', { + layerId: 'ais-live-vessels', + enabled: false, + }); await settle(1500); // (6c) move_camera orbit ONCE: bounded eased ~30° heading advance. - const heading = () => page.evaluate(() => window.__godsEyeView.viewer.camera.heading * 180 / Math.PI); + const heading = () => + page.evaluate( + () => (window.__godsEyeView.viewer.camera.heading * 180) / Math.PI, + ); let h0 = await heading(); r = await run('move_camera', { motion: 'orbit', mode: 'once' }); await settle(5000); let h1 = await heading(); let dOnce = Math.abs(((h1 - h0 + 540) % 360) - 180); - report(r?.ok === true && dOnce > 10 && dOnce < 45, - 'behavior: orbit once advances ~30° and self-stops', `Δheading=${dOnce.toFixed(1)}° result=${JSON.stringify(r)?.slice(0, 90)}`); + report( + r?.ok === true && dOnce > 10 && dOnce < 45, + 'behavior: orbit once advances ~30° and self-stops', + `Δheading=${dOnce.toFixed(1)}° result=${JSON.stringify(r)?.slice(0, 90)}`, + ); // (6d) continuous orbit runs until stop; stop reports it was active. - r = await run('move_camera', { motion: 'orbit', mode: 'continuous', speed: 'normal' }); + r = await run('move_camera', { + motion: 'orbit', + mode: 'continuous', + speed: 'normal', + }); await settle(2500); h0 = await heading(); // SwiftShader can fall near 1 fps. The motion loop deliberately caps a @@ -1138,22 +1891,36 @@ async function runBehaviorLayer() { await settle(1200); h1 = await heading(); const frozen = Math.abs(((h1 - h0 + 540) % 360) - 180) < 0.5; - report(moving && stopRes?.stopped === true && frozen, - 'behavior: continuous orbit moves, stop freezes it', `moving=${moving} stopped=${stopRes?.stopped} frozen=${frozen}`); + report( + moving && stopRes?.stopped === true && frozen, + 'behavior: continuous orbit moves, stop freezes it', + `moving=${moving} stopped=${stopRes?.stopped} frozen=${frozen}`, + ); // (6e) a navigation tool reclaims the camera from continuous motion. - await run('move_camera', { motion: 'orbit', mode: 'continuous', speed: 'slow' }); + await run('move_camera', { + motion: 'orbit', + mode: 'continuous', + speed: 'slow', + }); await settle(800); await run('zoom_to_globe', {}); await settle(1000); stopRes = await run('move_camera', { motion: 'stop' }); - report(stopRes?.stopped === false, - 'behavior: nav tool interrupts continuous motion', `stop-after-nav reports stopped=${stopRes?.stopped} (want false)`); + report( + stopRes?.stopped === false, + 'behavior: nav tool interrupts continuous motion', + `stop-after-nav reports stopped=${stopRes?.stopped} (want false)`, + ); // (6e3) CHAINED fly+orbit ("take me to X and orbit it"): orbit called // mid-flight must ARM and start once the camera settles (field finding). await run('fly_to_location', { query: 'Texas State Capitol' }); - r = await run('move_camera', { motion: 'orbit', mode: 'continuous', speed: 'normal' }); + r = await run('move_camera', { + motion: 'orbit', + mode: 'continuous', + speed: 'normal', + }); const armedOk = r?.ok === true; // either armed or started, both fine await settle(16000); h0 = await heading(); @@ -1171,9 +1938,11 @@ async function runBehaviorLayer() { // engage AT the Capitol, not wherever the flight happened to be. const camChain = await camState(); const dCapitol = Math.hypot(camChain.lat - 30.2747, camChain.lon + 97.7404); - report(armedOk && chainMoving && dCapitol < 0.1 && camChain.altKm < 30, + report( + armedOk && chainMoving && dCapitol < 0.1 && camChain.altKm < 30, 'behavior: chained fly+orbit arms and starts AT the destination', - `Δheading=${Math.abs(((h1 - h0 + 540) % 360) - 180).toFixed(1)}° cam=(${camChain.lat.toFixed(3)},${camChain.lon.toFixed(3)}) alt=${camChain.altKm.toFixed(1)}km dCapitol=${dCapitol.toFixed(3)}°`); + `Δheading=${Math.abs(((h1 - h0 + 540) % 360) - 180).toFixed(1)}° cam=(${camChain.lat.toFixed(3)},${camChain.lon.toFixed(3)}) alt=${camChain.altKm.toFixed(1)}km dCapitol=${dCapitol.toFixed(3)}°`, + ); // (6e4) zoom during orbit spirals the RADIUS (never a snapped-back no-op). // Wait out any residual arrival descent first so altitude deltas are the @@ -1187,57 +1956,109 @@ async function runBehaviorLayer() { r = await run('adjust_camera_zoom', { direction: 'out', amount: 'medium' }); await settle(2500); let altAfter = (await camState()).altKm; - let stillOrbiting = (await run('move_camera', { motion: 'stop' }))?.stopped === true; - report(r?.orbitRadiusAdjusted === true && altAfter > altBefore * 1.1 && stillOrbiting, + let stillOrbiting = + (await run('move_camera', { motion: 'stop' }))?.stopped === true; + report( + r?.orbitRadiusAdjusted === true && + altAfter > altBefore * 1.1 && + stillOrbiting, 'behavior: zoom during orbit grows the radius while circling', - `alt ${altBefore.toFixed(2)}→${altAfter.toFixed(2)}km adjusted=${r?.orbitRadiusAdjusted} stillOrbiting=${stillOrbiting}`); + `alt ${altBefore.toFixed(2)}→${altAfter.toFixed(2)}km adjusted=${r?.orbitRadiusAdjusted} stillOrbiting=${stillOrbiting}`, + ); // (6e4b) move_camera is explicit navigation: while TRACKING it must first // release the follow owner, then start the requested motion. await run('set_layer_visibility', { layerId: 'flights', enabled: true }); // Track a REAL contact the way the voice model does: nearest via analyst. - const near = await run('analyst_query', { layers: ['flights'], scope: { kind: 'view' }, sortBy: 'distance', limit: 1 }); + const near = await run('analyst_query', { + layers: ['flights'], + scope: { kind: 'view' }, + sortBy: 'distance', + limit: 1, + }); const trackId = near?.items?.[0]?.id; - r = trackId ? await run('track_entity', { query: trackId, layerId: 'flights' }) : { ok: false, error: 'no contacts in view' }; + r = trackId + ? await run('track_entity', { query: trackId, layerId: 'flights' }) + : { ok: false, error: 'no contacts in view' }; if (r?.ok) { - const orbitRes = await run('move_camera', { motion: 'orbit', mode: 'continuous' }); - const trackingReleased = await page.evaluate(() => !window.__godsEyeView.viewer.trackedEntity); + const orbitRes = await run('move_camera', { + motion: 'orbit', + mode: 'continuous', + }); + const trackingReleased = await page.evaluate( + () => !window.__godsEyeView.viewer.trackedEntity, + ); const orbitStop = await run('move_camera', { motion: 'stop' }); - report(orbitRes?.ok === true && trackingReleased && orbitStop?.stopped === true, + report( + orbitRes?.ok === true && + trackingReleased && + orbitStop?.stopped === true, 'behavior: move_camera releases tracking before taking the camera', - `released=${trackingReleased} stopped=${orbitStop?.stopped} result=${JSON.stringify(orbitRes)?.slice(0, 100)}`); + `released=${trackingReleased} stopped=${orbitStop?.stopped} result=${JSON.stringify(orbitRes)?.slice(0, 100)}`, + ); } else { - skipped('behavior: move_camera releases tracking before taking the camera', - `no trackable flight right now (${JSON.stringify(r)?.slice(0, 80)})`); + skipped( + 'behavior: move_camera releases tracking before taking the camera', + `no trackable flight right now (${JSON.stringify(r)?.slice(0, 80)})`, + ); } await settle(1500); // (6e4c) explicit navigation while tracking SUPERSEDES the follow camera // (field finding: "flew there but still tracking — couldn't do anything"). - const near2 = await run('analyst_query', { layers: ['flights'], scope: { kind: 'anywhere' }, sortBy: 'distance', limit: 1 }); + const near2 = await run('analyst_query', { + layers: ['flights'], + scope: { kind: 'anywhere' }, + sortBy: 'distance', + limit: 1, + }); const trackId2 = near2?.items?.[0]?.id; - if (trackId2 && (await run('track_entity', { query: trackId2, layerId: 'flights' }))?.ok) { + if ( + trackId2 && + (await run('track_entity', { query: trackId2, layerId: 'flights' }))?.ok + ) { await settle(2000); await run('fly_to_location', { query: 'Austin, Texas' }); await settle(6000); - const stillTracked = await page.evaluate(() => Boolean(window.__godsEyeView.viewer.trackedEntity)); + const stillTracked = await page.evaluate(() => + Boolean(window.__godsEyeView.viewer.trackedEntity), + ); const camHere = await camState(); - const nearAustin = Math.hypot(camHere.lat - 30.2672, camHere.lon + 97.7431) < 1.5; - report(!stillTracked && nearAustin, + const nearAustin = + Math.hypot(camHere.lat - 30.2672, camHere.lon + 97.7431) < 1.5; + report( + !stillTracked && nearAustin, 'behavior: fly-to while tracking stops the tracking and arrives', - `stillTracked=${stillTracked} cam=(${camHere.lat.toFixed(2)},${camHere.lon.toFixed(2)}) nearAustin=${nearAustin}`); + `stillTracked=${stillTracked} cam=(${camHere.lat.toFixed(2)},${camHere.lon.toFixed(2)}) nearAustin=${nearAustin}`, + ); } else { - skipped('behavior: fly-to while tracking stops the tracking and arrives', 'no trackable contact right now'); + skipped( + 'behavior: fly-to while tracking stops the tracking and arrives', + 'no trackable contact right now', + ); } // (6e5) tilt at the clamp answers honestly instead of silently no-oping. await page.evaluate(() => { const c = window.__godsEyeView.viewer.camera; - c.setView({ orientation: { heading: c.heading, pitch: -5.2 * Math.PI / 180, roll: 0 } }); + c.setView({ + orientation: { + heading: c.heading, + pitch: (-5.2 * Math.PI) / 180, + roll: 0, + }, + }); }); - r = await run('move_camera', { motion: 'tilt', direction: 'up', mode: 'once' }); - report(r?.ok === false && /limit/i.test(r?.error || ''), - 'behavior: tilt at the clamp reports the limit honestly', `result=${JSON.stringify(r)?.slice(0, 120)}`); + r = await run('move_camera', { + motion: 'tilt', + direction: 'up', + mode: 'once', + }); + report( + r?.ok === false && /limit/i.test(r?.error || ''), + 'behavior: tilt at the clamp reports the limit honestly', + `result=${JSON.stringify(r)?.slice(0, 120)}`, + ); // (6f) fly_route dollies along the drawn Capitol→Zilker route. r = await run('fly_route', { speed: 'fast' }); @@ -1245,19 +2066,43 @@ async function runBehaviorLayer() { const camNow = await camState(); const nearRoute = Math.hypot(camNow.lat - 30.27, camNow.lon + 97.755) < 0.2; stopRes = await run('move_camera', { motion: 'stop' }); - report(r?.ok === true && (r?.distanceM ?? 0) > 2000 && (r?.distanceM ?? 0) < 12000 && nearRoute, - 'behavior: fly_route dollies the drawn route', `distanceM=${r?.distanceM} waypoints=${r?.waypoints} cam=(${camNow.lat.toFixed(3)},${camNow.lon.toFixed(3)}) nearRoute=${nearRoute} stopHadMotion=${stopRes?.stopped} err=${r?.error || ''}`); + report( + r?.ok === true && + (r?.distanceM ?? 0) > 2000 && + (r?.distanceM ?? 0) < 12000 && + nearRoute, + 'behavior: fly_route dollies the drawn route', + `distanceM=${r?.distanceM} waypoints=${r?.waypoints} cam=(${camNow.lat.toFixed(3)},${camNow.lon.toFixed(3)}) nearRoute=${nearRoute} stopHadMotion=${stopRes?.stopped} err=${r?.error || ''}`, + ); // (7) clear_annotations empties the board r = await run('clear_annotations', {}); await settle(800); - annoCount = await page.evaluate(() => window.__gevAnnotations?.list?.().length ?? window.__gevAnnotations?.count?.() ?? -1); - report(r?.ok !== false && (annoCount === 0 || annoCount === -1), 'behavior: clear_annotations empties board', `listCount=${annoCount}`); + annoCount = await page.evaluate( + () => + window.__gevAnnotations?.list?.().length ?? + window.__gevAnnotations?.count?.() ?? + -1, + ); + report( + r?.ok !== false && (annoCount === 0 || annoCount === -1), + 'behavior: clear_annotations empties board', + `listCount=${annoCount}`, + ); // (8) get_entity_context returns basemap context r = await run('get_entity_context', {}); - const hasContext = !!(r && (r.basemap || r.scene || r.context || r.viewScale || r.center)); - report(hasContext, 'behavior: get_entity_context returns scene context', `keys=${Object.keys(r || {}).slice(0, 8).join(',')}`); + const hasContext = !!( + r && + (r.basemap || r.scene || r.context || r.viewScale || r.center) + ); + report( + hasContext, + 'behavior: get_entity_context returns scene context', + `keys=${Object.keys(r || {}) + .slice(0, 8) + .join(',')}`, + ); // (9) set_context_mode — happy path and a refusal. // Contacts is an exclusive mode that tears other layers down, so this runs @@ -1267,48 +2112,60 @@ async function runBehaviorLayer() { const contextEntered = await page.evaluate( () => window.__godsEyeView?.styleManager?.getContextModeState?.() || null, ); - report(r?.ok === true && contextEntered?.mode === 'flights', + report( + r?.ok === true && contextEntered?.mode === 'flights', 'behavior: set_context_mode enters Contacts', - `result=${JSON.stringify(r)?.slice(0, 140)} state=${JSON.stringify(contextEntered)?.slice(0, 90)}`); + `result=${JSON.stringify(r)?.slice(0, 140)} state=${JSON.stringify(contextEntered)?.slice(0, 90)}`, + ); // (9a) control_cockpit status reads the real Cockpit surface. r = await run('control_cockpit', { action: 'status' }); - report(r?.ok === true && r?.state && typeof r.state === 'object', + report( + r?.ok === true && r?.state && typeof r.state === 'object', 'behavior: control_cockpit status reports Cockpit state', - `result=${JSON.stringify(r)?.slice(0, 160)}`); + `result=${JSON.stringify(r)?.slice(0, 160)}`, + ); // (9b) Cockpit entry with nothing to fly is an honest failure, never a // silent success that leaves the operator staring at an unchanged map. - const cockpitBefore = await page.evaluate( - () => Boolean(window.__godsEyeView?.styleManager?.cockpitView?.active), + const cockpitBefore = await page.evaluate(() => + Boolean(window.__godsEyeView?.styleManager?.cockpitView?.active), ); r = await run('control_cockpit', { action: 'enter' }); await settle(1200); - const cockpitAfter = await page.evaluate( - () => Boolean(window.__godsEyeView?.styleManager?.cockpitView?.active), + const cockpitAfter = await page.evaluate(() => + Boolean(window.__godsEyeView?.styleManager?.cockpitView?.active), ); // Either it genuinely entered (an aircraft was tracked) or it refused with // a reason — the one thing it must never do is claim success while inert. - report((r?.ok === true && cockpitAfter) || (r?.ok !== true && !!r?.error && !cockpitAfter), + report( + (r?.ok === true && cockpitAfter) || + (r?.ok !== true && !!r?.error && !cockpitAfter), 'behavior: control_cockpit entry is honest about whether it entered', - `ok=${r?.ok} error=${String(r?.error || '').slice(0, 80)} before=${cockpitBefore} after=${cockpitAfter}`); + `ok=${r?.ok} error=${String(r?.error || '').slice(0, 80)} before=${cockpitBefore} after=${cockpitAfter}`, + ); if (cockpitAfter) await run('control_cockpit', { action: 'exit' }); // (9c) An unknown cockpit action is refused by name. r = await run('control_cockpit', { action: 'barrel-roll' }); - report(r?.ok === false && /unknown cockpit action/i.test(r?.error || ''), + report( + r?.ok === false && /unknown cockpit action/i.test(r?.error || ''), 'behavior: control_cockpit refuses an unknown action', - `result=${JSON.stringify(r)?.slice(0, 120)}`); + `result=${JSON.stringify(r)?.slice(0, 120)}`, + ); // (9d) An unavailable context mode is refused, and the live mode survives. r = await run('set_context_mode', { mode: 'orbital-weather' }); const contextAfterRefusal = await page.evaluate( () => window.__godsEyeView?.styleManager?.getContextModeState?.() || null, ); - report(r?.ok === false && /unknown context mode/i.test(r?.error || '') - && contextAfterRefusal?.mode === 'flights', + report( + r?.ok === false && + /unknown context mode/i.test(r?.error || '') && + contextAfterRefusal?.mode === 'flights', 'behavior: set_context_mode refuses an unavailable mode without dropping the live one', - `result=${JSON.stringify(r)?.slice(0, 120)} state=${JSON.stringify(contextAfterRefusal)?.slice(0, 80)}`); + `result=${JSON.stringify(r)?.slice(0, 120)} state=${JSON.stringify(contextAfterRefusal)?.slice(0, 80)}`, + ); // (9c-2) With Contacts up, an aircraft radius query must carry the panel's // own numbers. Field case: analyst said 8 for a 250 km window the panel had @@ -1322,19 +2179,20 @@ async function runBehaviorLayer() { }); const awarenessFlights = await page.evaluate(() => { const snap = window.__godsEyeView?.dataManager?.layers - ?.get('military-awareness')?.module?.getContextSnapshot?.(); + ?.get('military-awareness') + ?.module?.getContextSnapshot?.(); const cohort = snap?.cohorts?.find((c) => c.id === 'flights'); return cohort ? cohort.count : null; }); const windowBlock = r?.contactsWindow || null; report( - Boolean(windowBlock) - && windowBlock.flights === awarenessFlights - && windowBlock.radiusKm === 250 - && typeof windowBlock.centeredOn === 'string' - && /loads by viewport/.test(r?.coverage?.note || '') - // Contract rule 3: the count names its own scope. - && /^within 250 km of /.test(r?.scopeLabel || ''), + Boolean(windowBlock) && + windowBlock.flights === awarenessFlights && + windowBlock.radiusKm === 250 && + typeof windowBlock.centeredOn === 'string' && + /loads by viewport/.test(r?.coverage?.note || '') && + // Contract rule 3: the count names its own scope. + /^within 250 km of /.test(r?.scopeLabel || ''), 'behavior: analyst_query carries the Contacts panel counts and says what it measured', `contactsWindow=${JSON.stringify(windowBlock)} awarenessFlights=${awarenessFlights} analystCount=${r?.count} scopeLabel="${r?.scopeLabel}"`, ); @@ -1343,32 +2201,49 @@ async function runBehaviorLayer() { // Field case: enter{targetLayer:"military"} reported ok:true on a FLIGHTS // subject, so "cockpit in that military helicopter" put the operator in an // airliner. - if (await page.evaluate(() => Boolean(window.__godsEyeView?.styleManager?.cockpitView?.active))) { + if ( + await page.evaluate(() => + Boolean(window.__godsEyeView?.styleManager?.cockpitView?.active), + ) + ) { await run('control_cockpit', { action: 'exit' }); await settle(600); } - r = await run('control_cockpit', { action: 'enter', targetLayer: 'military' }); + r = await run('control_cockpit', { + action: 'enter', + targetLayer: 'military', + }); await settle(1200); const layerAfter = r?.state?.subject?.layerId ?? null; - const cockpitOn = await page.evaluate( - () => Boolean(window.__godsEyeView?.styleManager?.cockpitView?.active), + const cockpitOn = await page.evaluate(() => + Boolean(window.__godsEyeView?.styleManager?.cockpitView?.active), ); report( - (r?.ok === true && layerAfter === 'military' && cockpitOn) - || (r?.ok !== true && !!r?.error && !cockpitOn), + (r?.ok === true && layerAfter === 'military' && cockpitOn) || + (r?.ok !== true && !!r?.error && !cockpitOn), 'behavior: control_cockpit enter honours targetLayer or refuses by name', `ok=${r?.ok} subjectLayer=${layerAfter} active=${cockpitOn} error=${String(r?.error || '').slice(0, 90)}`, ); - if (cockpitOn) { await run('control_cockpit', { action: 'exit' }); await settle(600); } + if (cockpitOn) { + await run('control_cockpit', { action: 'exit' }); + await settle(600); + } // (9d-3) A non-aircraft layer can never be entered — Cockpit flies aircraft. - r = await run('control_cockpit', { action: 'enter', targetLayer: 'ais-live-vessels' }); - const vesselCockpit = await page.evaluate( - () => Boolean(window.__godsEyeView?.styleManager?.cockpitView?.active), + r = await run('control_cockpit', { + action: 'enter', + targetLayer: 'ais-live-vessels', + }); + const vesselCockpit = await page.evaluate(() => + Boolean(window.__godsEyeView?.styleManager?.cockpitView?.active), ); - report(r?.ok === false && /aircraft only/i.test(r?.error || '') && !vesselCockpit, + report( + r?.ok === false && + /aircraft only/i.test(r?.error || '') && + !vesselCockpit, 'behavior: control_cockpit refuses to enter a non-aircraft layer', - `result=${JSON.stringify(r)?.slice(0, 130)} active=${vesselCockpit}`); + `result=${JSON.stringify(r)?.slice(0, 130)} active=${vesselCockpit}`, + ); // (9e) Exit restores the neutral map. r = await run('set_context_mode', { mode: 'off' }); @@ -1376,43 +2251,64 @@ async function runBehaviorLayer() { const contextExited = await page.evaluate( () => window.__godsEyeView?.styleManager?.getContextModeState?.() || null, ); - report(r?.ok !== false && !contextExited?.mode, + report( + r?.ok !== false && !contextExited?.mode, 'behavior: set_context_mode exits back to the neutral map', - `result=${JSON.stringify(r)?.slice(0, 120)} state=${JSON.stringify(contextExited)?.slice(0, 80)}`); + `result=${JSON.stringify(r)?.slice(0, 120)} state=${JSON.stringify(contextExited)?.slice(0, 80)}`, + ); // (9f) With Contacts OFF the entry gate is shut. Entry must refuse with the // reason rather than produce the half-entered state the operator saw: a // plane anchored under the camera with no HUD and no exit control. r = await run('control_cockpit', { action: 'status' }); const gateReport = r?.state || {}; - const cockpitBeforeGate = await page.evaluate( - () => Boolean(window.__godsEyeView?.styleManager?.cockpitView?.active), + const cockpitBeforeGate = await page.evaluate(() => + Boolean(window.__godsEyeView?.styleManager?.cockpitView?.active), + ); + report( + gateReport.active === false && + gateReport.entryAllowed === false && + gateReport.entryBlockedReason === 'contacts-inactive' && + !cockpitBeforeGate, + 'behavior: cockpit status names why entry is blocked with Contacts off', + `state=${JSON.stringify(gateReport)?.slice(0, 150)}`, ); - report(gateReport.active === false && gateReport.entryAllowed === false - && gateReport.entryBlockedReason === 'contacts-inactive' && !cockpitBeforeGate, - 'behavior: cockpit status names why entry is blocked with Contacts off', - `state=${JSON.stringify(gateReport)?.slice(0, 150)}`); // The runner calls controlCockpit directly, so this exercises the app gate // rather than the voice tool's own Contacts bootstrap. const gated = await page.evaluate( - () => window.__godsEyeView?.styleManager?.controlCockpit?.('enter') || null, + () => + window.__godsEyeView?.styleManager?.controlCockpit?.('enter') || null, ); - const cockpitAfterGate = await page.evaluate( - () => Boolean(window.__godsEyeView?.styleManager?.cockpitView?.active), + const cockpitAfterGate = await page.evaluate(() => + Boolean(window.__godsEyeView?.styleManager?.cockpitView?.active), ); - report(gated?.ok === false && /contacts/i.test(gated?.error || '') && !cockpitAfterGate, + report( + gated?.ok === false && + /contacts/i.test(gated?.error || '') && + !cockpitAfterGate, 'behavior: cockpit entry with Contacts off is refused, not half-entered', - `ok=${gated?.ok} error=${String(gated?.error || '').slice(0, 90)} active=${cockpitAfterGate}`); + `ok=${gated?.ok} error=${String(gated?.error || '').slice(0, 90)} active=${cockpitAfterGate}`, + ); const shotDir = path.join(ROOT, 'qa-shots'); fs.mkdirSync(shotDir, { recursive: true }); - await page.screenshot({ path: path.join(shotDir, 'voice-behavior-final.png') }); - report(overpassRequests.length === 0, + await page.screenshot({ + path: path.join(shotDir, 'voice-behavior-final.png'), + }); + report( + overpassRequests.length === 0, 'behavior: no Overpass request during the run', - overpassRequests.length ? overpassRequests.slice(0, 3).join(' | ') : 'none'); + overpassRequests.length + ? overpassRequests.slice(0, 3).join(' | ') + : 'none', + ); } catch (e) { - report(false, 'behavior: layer crashed', String(e?.message || e).slice(0, 200)); + report( + false, + 'behavior: layer crashed', + String(e?.message || e).slice(0, 200), + ); } finally { await browser.close(); } diff --git a/server/providers/openai/instructions.js b/server/providers/openai/instructions.js index 8bb467d9f..2be3365e8 100644 --- a/server/providers/openai/instructions.js +++ b/server/providers/openai/instructions.js @@ -22,7 +22,7 @@ function realtimeInstructions( 'WORLD QUERIES — the query tools (get_weather, get_wind, search_places, places_nearby, plan_route, get_cyclones, get_fire_perimeters, get_bike_share, get_transit_vehicles, get_traffic_flow, situation_brief, military_awareness and the other get_* and find_* tools you have) look up data anywhere in the world without changing the map and without any layer being on. Use them when the user asks about a place they are not looking at or data whose layer is off ("what is the weather in Tokyo", "how do I drive from Austin to Dallas", "any wildfires in California"). For aircraft, ships, earthquakes, active fires, datacenters, dams and satellites overhead, use analyst_query over the layers instead, turning the layer on first when the user wants it. Use analyst_query and get_entity_context for what the app has loaded or what is on screen. Answer from the result summary and data, mention its source when it has one, and never claim the map changed. If the user also wants to SEE it, use the matching app action (fly_to_location, set_layer_visibility) as well.', 'WHERE, for every tool that takes a place or scope, in this order: (1) a place the user names; (2) explicit pointing — "this/that one/here/around here" while pointer_context shows pointing or a target → "pointer"; (3) the Contacts subject while Contacts is active, or the selection when the user refers to it; (4) the current view. A selected datacenter, dam, fire or cable never silently becomes the centre.', 'FEED STATE: when a count\'s feed is stale or degraded, add that one word ("12 flights in view, stale"); never call it live. If unavailable, give no number: say the feed is unavailable. Fallback feeds are valid data; name them only if asked.', - 'COUNTING CONTRACT — what "near" means. (1) While Contacts is ACTIVE, "near / nearby / how many aircraft" means the Contacts window: answer from contactsWindow in the tool result — the exact numbers on the user\'s panel (set_context_mode, analyst_query and get_current_view_state carry it once Contacts settles). For "Open Contacts and tell me how many aircraft are within 250 km", call set_context_mode{mode:"contacts"} first and answer from contactsWindow.aircraft, not from a pre-Contacts analyst query; analyst_query\'s own count measures loaded records and is usually lower — never give it as the window count. The centre follows WHERE; with Contacts active but no subject yet, use the view, never an empty panel. (2) With Contacts OFF, "nearby" means in view; "near " means a radius around that place. (3) EVERY count names its scope in words — "42 in your window", "8 in view", "30 within 250 km of Austin" — never a bare number; analyst_query returns scopeLabel for exactly this. Two numbers with named scopes are not a contradiction; say both if asked. (4) State counts VERBATIM — never estimate, round, or hedge: if a tool returns 46, say 46.', + 'COUNTING CONTRACT — what "near" means. (1) While Contacts is ACTIVE, a bare "near / nearby / how many aircraft" means the settled Contacts subject window: its count is contactsWindow.aircraft. For "Open Contacts and tell me how many aircraft are within 250 km of the active subject", call set_context_mode{mode:"contacts"} first; use its window count only when the requested radius matches contactsWindow.radiusKm. A count does NOT fulfill a requested list, ranking or filter: call analyst_query even when set_context_mode already returned the count. For "within 250 km of the active subject, list the five closest", call analyst_query{layers:["flights","military"],scope:{kind:"radius",km:250},sortBy:"distance",limit:5}, omitting center so the active Contacts subject supplies it. Use the actual requested km for any other radius; never substitute the panel\'s 250 km count for a different radius. Analyst results supply the requested items and filtered count; never invent a list from contactsWindow alone. The centre follows WHERE: explicit view, named place, pointing and explicit coordinates retain their scopes and never become Contacts queries. Never invent center coordinates such as 0,0. With Contacts active but no subject yet, use the view and name that scope. (2) With Contacts OFF, "nearby" means in view; "near " means a radius around that place. (3) EVERY count names its scope in words — "42 in your window", "8 in view", "30 within 250 km of Austin" — never a bare number; analyst_query returns scopeLabel for exactly this. Two numbers with named scopes are not a contradiction; say both if asked. (4) State counts VERBATIM — never estimate, round, or hedge: if a tool returns 46, say 46.', 'While Cockpit is active, navigate with control_cockpit (next/previous, optionally targetLayer or aircraftClass). track_entity and fly_to_location are REFUSED by design while Cockpit owns the camera — that refusal is correct, not an error to retry. To go somewhere else, exit Cockpit first. control_cockpit enter establishes Contacts itself, so do not call set_context_mode before or after it.', 'When the target layer is unknown, OMIT layerId in track_entity so it searches all enabled layers. Passing the wrong layerId ("flights" for a military contact) returns "Nothing matched" even though the contact is loaded.', 'READING THE VIEW: with no selected object or overlay entities, answer from get_entity_context\'s basemap context (tiles source, centre target, reverse-geocoded place, altitude, style, enabled layers); say there is nothing only when that is unavailable too. Prefer, in order: a nearby knownLandmarks name ("Eiffel Tower"); at local zoom nearbyPlaces, place.labels and viewportPlaces visible and street labels; viewportPlaces dominantCountry/Region/Locality over raw coordinates; the majority of viewportSamples over one reverse-geocoded address. Legible labels in an attached screenshot count too, but at global or regional viewScale never name a precise street or city from one centre pixel.', @@ -30,7 +30,7 @@ function realtimeInstructions( 'POINTING: pointer_context says what the cursor is on (pointing:: resolved by the tools; target "none": not pointing). For "this/that/here/that one" with no name, pass "pointer": get_entity_context scope, track_entity/fly_to_location query, analyst_query scope.kind ("around here" is pointer, not view), annotate_map target. "The second/last one" is referent:n from the last numbered list. Without a pointer, ask or follow WHERE.', 'PREAMBLES: for instant commands (layers, styles, HUD, detection, zoom, panels, basemap, radio controls, camera nudges, and lookups such as get_entity_context) call the tool without speaking first. Only for slow or multi-step work (annotate_map, a searched fly_to_location, select_nearest_aircraft, analyst_query over a named place, or two or more tools) you may first say ONE sentence of at most 12 words naming the actual steps ("Finding the Capitol, then outlining its grounds."), then call the tools at once in the same response.', 'When one request contains MULTIPLE changes, call ALL the corresponding tools before the confirmation. Never confirm a partial subset; if one fails, say which.', - 'AFTER TOOLS: speak the result\'s say line once, lightly rephrased at most, never adding to it and never repeating your preamble; if say is null, confirm briefly or continue. Sources, notes and caveats are on the voice card: speak a caveat only when it changes the answer, or when asked why, how sure or which source. Lower bounds ("at least 500") and partial answers are part of the answer — always say them. No filler or hedges ("let me check", "keep in mind", "it seems", "based on the data available"), and never add "approximately" to an exact tool number.', + 'AFTER TOOLS: speak the result\'s say line once, lightly rephrased at most, never adding to it and never repeating your preamble; if say is null, finish any remaining requested tools before confirming. Never add a count, list or action claim that the executed tool results do not support. Sources, notes and caveats are on the voice card: speak a caveat only when it changes the answer, or when asked why, how sure or which source. Lower bounds ("at least 500"), partial answers and stale, degraded or unavailable feed states in say are part of the answer — always preserve them. No filler or hedges ("let me check", "keep in mind", "it seems", "based on the data available"), and never add "approximately" to an exact tool number.', 'For "show/open/turn on" layer requests, enable the matching layer. For "hide/close/turn off", disable it.', // INSTRUCTION-ONLY mapping for the two globe-scale named views. // diff --git a/server/providers/openai/toolDescriptions.js b/server/providers/openai/toolDescriptions.js index cd49cf4ec..450c9da81 100644 --- a/server/providers/openai/toolDescriptions.js +++ b/server/providers/openai/toolDescriptions.js @@ -543,7 +543,7 @@ export const ACTION_DESCRIPTIONS = { }, analyst_query: { description: - 'Counts, lists, superlatives and attribute filters over records loaded by ENABLED layers — "how many flights over Texas", "biggest fire near LA", "ships headed to Oakland", "next rocket launch". complete:false means the count is a floor ("at least") and rankings cover the records examined; partial names unanswered layers. Items carry lat/lon for a follow-up fly_to_location. ok:false codes: LAYER_OFF (offer to enable), NOT_READY (loading), FOLLOW_UP_MISMATCH (ask again without followUp), UNKNOWN_FIELD/UNKNOWN_SCOPE/BAD_VALUE (retry with the allowed values).', + 'Counts, lists, superlatives and attribute filters over records loaded by ENABLED layers — "how many flights over Texas", "biggest fire near LA", "ships headed to Oakland", "next rocket launch". A requested list or ranking requires this tool even after set_context_mode returned a Contacts count; that count alone has no requested items. complete:false means the count is a floor ("at least") and rankings cover the records examined; partial names unanswered layers. Items carry lat/lon for a follow-up fly_to_location. ok:false codes: LAYER_OFF (offer to enable), NOT_READY (loading), FOLLOW_UP_MISMATCH (ask again without followUp), UNKNOWN_FIELD/UNKNOWN_SCOPE/BAD_VALUE (retry with the allowed values).', $position: 1, parameters: { properties: { @@ -554,7 +554,7 @@ export const ACTION_DESCRIPTIONS = { }, scope: { description: - 'Always set it (see WHERE). view = a radius around the view centre ("in view", "nearby"); pointer = "around here/this" while pointing (radius at the cursor); region = "over Texas"; radius with center {lat, lon} = "near "; anywhere = no place ("biggest anywhere", "today"). There is no drawn-area scope yet: say so rather than answering for the view.', + 'Always set it (see WHERE). Explicit "in view" uses view; "nearby" uses view when Contacts is off. Around the active Contacts subject uses radius with the requested km and center omitted, never view: the five closest within 250 km use scope:{kind:"radius",km:250}, sortBy:"distance", limit:5. For another requested radius use its actual km, not 250. pointer = "around here/this" while pointing (radius at the cursor); region = "over Texas"; radius with center {lat, lon} = "near " or explicit coordinates; anywhere = no place ("biggest anywhere", "today"). Preserve explicit view, named place, pointer and explicit center even while Contacts is active; never invent coordinates. There is no drawn-area scope yet: say so rather than answering for the view.', $position: 2, properties: { name: { @@ -563,7 +563,13 @@ export const ACTION_DESCRIPTIONS = { $position: 1, }, km: { - description: 'For kind=radius or pointer.', + description: + 'For kind=radius or pointer: the actual requested kilometers. An active-subject 250 km request uses 250; another requested radius keeps that number.', + $position: 1, + }, + center: { + description: + 'Omit for a radius around the active Contacts subject. Supply only an explicitly requested coordinate or a resolved named-place coordinate; never invent 0,0 or replace an explicit center with the subject.', $position: 1, }, }, diff --git a/src/data/analystEngine.js b/src/data/analystEngine.js index bb0fd2e5b..697469dba 100644 --- a/src/data/analystEngine.js +++ b/src/data/analystEngine.js @@ -698,7 +698,10 @@ export async function selectAnalystRows( const item = { layerKey: key, ...row }; const age = ageOf(key, row); if (age !== null) item[AGE_FIELD] = age; - if (sortBy === 'distance') item.distanceKm = distance(row); + if (sortBy === 'distance') { + const km = distance(row); + item.distanceKm = Number.isFinite(km) ? Math.round(km * 10) / 10 : km; + } return item; }); return { count, items, matched, summary }; @@ -881,6 +884,8 @@ export function createAnalystEngine(providers) { // 2) Spatial scope const rememberedScope = followUp && !spec.scope ? lastResult.scopePresentation || null : null; + const rememberedDistanceOf = + followUp && !spec.scope ? lastResult.distanceOf || null : null; let resolvedScope = null; let scopeNote = rememberedScope?.coverage || 'anywhere'; // Human phrasing for the same scope, so every spoken count can name what it @@ -985,6 +990,7 @@ export function createAnalystEngine(providers) { sortDir: spec.sortDir, limit: spec.limit, layerKeys: layers, + distanceOf: rememberedDistanceOf, ref: spec.sortBy === 'distance' ? resolvedScope?.center || providers.getViewContext() @@ -1070,6 +1076,7 @@ export function createAnalystEngine(providers) { const committedCoverage = memoryOverride?.coverage || result.coverage; lastResult = { matched: memoryOverride?.matched || selection.matched, + distanceOf: memoryOverride?.distanceOf || rememberedDistanceOf, layerKeys: [...(memoryOverride?.layerKeys || layers)], coverage: committedCoverage, unanswered: [ diff --git a/src/firstRunExperience.test.mjs b/src/firstRunExperience.test.mjs index 2d4412cca..c4a98e702 100644 --- a/src/firstRunExperience.test.mjs +++ b/src/firstRunExperience.test.mjs @@ -671,11 +671,12 @@ test('the voice TOOL SCHEMA matches the pinned contract — the mission mapping // field hints), point-and-ask's pointer/referent arguments and the prompt // consolidation (shorter analyst, annotate_map and ISS wording), plus the // referent-only track_entity alternative (kept out of the model-facing - // schema, which may not carry top-level anyOf); the missions still ride existing tools. - assert.equal(block.length, 28831, 'serialized tool schema length drifted'); + // schema, which may not carry top-level anyOf), and Contacts requested-radius + // list descriptions; the missions still ride existing tools. + assert.equal(block.length, 29724, 'serialized tool schema length drifted'); assert.equal( crypto.createHash('sha256').update(block).digest('hex'), - 'cce7eb17b2af9ac34c06bdf59d1d31f4e8ace6a9d695d79111f3e4d49d033f1f', + '5dd41cfbd4f52acce3071dcfbc80acd758e390ad251ca07b5db67094383c6a93', 'the first-run missions must ride EXISTING tools: no schema edit, no cache bust', ); const instructions = fs.readFileSync(new URL('../server/providers/openai/instructions.js', import.meta.url), 'utf8'); diff --git a/src/search/alpsGeocoder.js b/src/search/alpsGeocoder.js new file mode 100644 index 000000000..74fffcd60 --- /dev/null +++ b/src/search/alpsGeocoder.js @@ -0,0 +1,72 @@ +import { findNaturalRegion, pointInRing } from '../data/naturalEarthRegions.js'; + +/** + * The unqualified Alps ask names the European range, not a same-named peak + * or local POI. Keep this correction exact and Alps-only: other ranges and + * geographic qualifiers retain the caller's configured geocoder semantics. + */ +export function createAlpsGeocoder({ lookupRegion = findNaturalRegion } = {}) { + return { + async geocode(query, { signal } = {}) { + signal?.throwIfAborted(); + const name = String(query ?? '') + .trim() + .replace(/\s+/g, ' '); + if (!/^(?:the )?alps(?: mountains)?$/i.test(name)) + return { place: null, answered: true }; + + const region = await lookupRegion('Alps'); + signal?.throwIfAborted(); + if ( + region?.name !== 'Alps' || + region.kind !== 'natural' || + !Array.isArray(region.polygons) || + region.polygons.length === 0 || + !region.polygons.every( + (ring) => + Array.isArray(ring) && + ring.length >= 3 && + ring.every( + (point) => + Array.isArray(point) && + point.length === 2 && + point.every(Number.isFinite) && + Math.abs(point[0]) <= 180 && + Math.abs(point[1]) <= 90, + ), + ) + ) + throw new Error('Bundled Alps geometry unavailable'); + const vertices = region.polygons.flat(); + const west = Math.min(...vertices.map(([lon]) => lon)); + const east = Math.max(...vertices.map(([lon]) => lon)); + const south = Math.min(...vertices.map(([, lat]) => lat)); + const north = Math.max(...vertices.map(([, lat]) => lat)); + const lat = (south + north) / 2; + const lng = (west + east) / 2; + if ( + ![west, east, south, north, lat, lng].every(Number.isFinite) || + !region.polygons.some((ring) => pointInRing(ring, lat, lng)) + ) { + // A broken pack is not permission to cache an unrelated network peak + // as the canonical range. Existing search error/retry handling owns it. + throw new Error('Bundled Alps geometry unavailable'); + } + return { + answered: true, + place: { + lat, + lng, + name: region.name, + label: region.name, + types: ['natural_feature'], + exact: true, + viewport: { + southwest: { lat: south, lng: west }, + northeast: { lat: north, lng: east }, + }, + }, + }; + }, + }; +} diff --git a/src/search/defaults.js b/src/search/defaults.js index 2b0428025..f440ca3ad 100644 --- a/src/search/defaults.js +++ b/src/search/defaults.js @@ -6,9 +6,10 @@ import { createGoogleGeocoder } from './google.js'; import { createPhotonGeocoder } from '../keylessGeocoder.js'; import { createCoordinateGeocoder } from './coordinateGeocoder.js'; import { createPresetGeocoder } from './presetGeocoder.js'; +import { createAlpsGeocoder } from './alpsGeocoder.js'; /** - * Coordinates and bundled names first — both answer offline and with no key — + * Coordinates, bundled names and the unqualified Alps first — all offline — * then Google when configured, then keyless Photon, then the local Nominatim * route as a last resort. Transport stays local to setup. * @@ -37,6 +38,7 @@ export function createDefaultPlaceSearch({ providers: providers.geocode || [ createCoordinateGeocoder(), ...(presets ? [createPresetGeocoder({ presets })] : []), + createAlpsGeocoder(), ...(selected ? [selected] : [ diff --git a/src/ui/styles/voice-card.css b/src/ui/styles/voice-card.css index b1f2e481e..0fbc85595 100644 --- a/src/ui/styles/voice-card.css +++ b/src/ui/styles/voice-card.css @@ -32,6 +32,14 @@ animation: gev-voice-card-in 180ms cubic-bezier(0.2, 0.8, 0.2, 1); } +/* The visible card must clear the dock trays' parent stacking contexts. */ +#command-dock + > #gev-voice-control:not([data-status='error']):has( + .gev-voice-card:not([hidden]) + ) { + z-index: 101; +} + #gev-voice-control .gev-voice-card[hidden], #gev-voice-control .gev-voice-card [hidden], #gev-voice-control[data-status='error'] .gev-voice-card { diff --git a/src/voice/actionSchemas.test.mjs b/src/voice/actionSchemas.test.mjs index 5cc22fd5e..8c3d09b55 100644 --- a/src/voice/actionSchemas.test.mjs +++ b/src/voice/actionSchemas.test.mjs @@ -34,8 +34,9 @@ test('the complete Realtime tool payload pins the manifest-generated layer relea // Re-derived for the voice layer manifest (generated layer enums and // aliases), point-and-ask (pointer sentinels, referent args) and the // consolidated tool wording (each policy stated once); model-facing tools - // omit top-level anyOf (OpenAI rejects it). - '66c955a41e36c6e7d2d255a950993128528288be13946fe4f7dacb05ad2ba291', + // omit top-level anyOf (OpenAI rejects it). Contacts list routing preserves + // requested radius and center authority in descriptions only. + '71c66bb725e6987e56780db1e1a05e17359df6f2fcc4884b7ad46f326eae5f5d', ); }); diff --git a/src/voice/gevActions.js b/src/voice/gevActions.js index 304995c25..01dc87f77 100644 --- a/src/voice/gevActions.js +++ b/src/voice/gevActions.js @@ -3478,6 +3478,9 @@ async function aircraftProximityWindowForQuery( result, isCurrent = () => true, ) { + // Follow-ups re-filter the engine's retained answer. Re-projecting a fresh + // Contacts snapshot here would restore rows a previous filter removed. + if (result?.coverage?.followUp) return null; const spec = normalizeAnalystSpec(args, args?.layers || []); if (!spec) return null; const scope = args?.scope; @@ -3552,8 +3555,10 @@ async function aircraftProximityWindowForQuery( lat, lon, }; + // Rank the retained cohort at its original precision; display rounding + // must not turn nearby contacts into artificial ID-sorted ties. if (Number.isFinite(item.distance)) - distanceKm.set(row, Math.round(item.distance / 100) / 10); + distanceKm.set(row, item.distance / 1000); return row; }); return { key, rows }; @@ -3659,6 +3664,9 @@ async function aircraftProximityWindowForQuery( answer, memory: { matched: selection.matched, + // Follow-up ranking belongs to this retained Contacts evaluation, not + // the camera's current position. Keep the accessor with its row owners. + distanceOf: (row) => distanceKm.get(row), layerKeys: wanted, coverage, unanswered: [...(result?.unanswered || [])], @@ -4791,6 +4799,46 @@ const ANALYST_REFUSAL_FIELDS = Object.freeze([ 'cancelled', ]); +const VIEWPORT_COVERAGE_NOTE = 'the flights layer loads by viewport'; + +/** Resolve omitted follow-up scope from the engine's retained coverage. */ +function effectiveAnalystScopeKind(args, coverage) { + const explicit = String(args.scope?.kind || '').toLowerCase(); + if (explicit) return explicit; + const retained = String(coverage?.scope || '') + .split(':', 1)[0] + .toLowerCase(); + if (retained === 'window') return 'radius'; + if (['anywhere', 'radius', 'region', 'view'].includes(retained)) + return retained; + return args.followUp ? 'anywhere' : 'view'; +} + +/** Add the viewport qualification without replacing retained provenance. */ +function withViewportCoverageNote(coverage) { + if (!coverage) return coverage; + const current = typeof coverage.note === 'string' ? coverage.note : ''; + if (current.includes(VIEWPORT_COVERAGE_NOTE)) return coverage; + const separator = current ? '; ' : ''; + return { + ...coverage, + note: current + ? `${current}${separator}${VIEWPORT_COVERAGE_NOTE}` + : VIEWPORT_COVERAGE_NOTE, + }; +} + +/** Whether this query counts records from a viewport-loaded layer. */ +function isViewportScopedAnalystResult(args, coverage) { + const scopeKind = effectiveAnalystScopeKind(args, coverage); + return ( + (scopeKind === 'radius' || scopeKind === 'view') && + (coverage?.layersQueried || []).some( + (layer) => voiceLayer(layer.layerKey)?.query?.viewportLoaded, + ) + ); +} + async function runAnalystQuery( analystEngine, dataManager, @@ -4820,6 +4868,15 @@ async function runAnalystQuery( candidate, isCurrent, ); + if ( + projected?.answer && + projected?.memory && + isViewportScopedAnalystResult(args, projected.answer.coverage) + ) { + const coverage = withViewportCoverageNote(projected.answer.coverage); + projected.answer.coverage = coverage; + projected.memory.coverage = coverage; + } entityWindow = projected?.answer || null; return projected?.memory || null; }, @@ -4855,14 +4912,12 @@ async function runAnalystQuery( // A radius/view count over a viewport-loaded layer counts what is LOADED, and // the flights layer reloads as the camera moves — so this number can sit well // under the Contacts cohort without either being wrong. - const scopeKind = String(args.scope?.kind || 'view').toLowerCase(); - const viewportScoped = - (scopeKind === 'radius' || scopeKind === 'view') && - (result.coverage?.layersQueried || []).some( - (l) => voiceLayer(l.layerKey)?.query?.viewportLoaded, - ); + const scopeKind = effectiveAnalystScopeKind(args, result.coverage); + const viewportScoped = isViewportScopedAnalystResult(args, result.coverage); if (viewportScoped && result.coverage) { - result.coverage.note = 'the flights layer loads by viewport'; + // The engine's follow-up memory retains this same object, so update its + // note in place after deriving the qualified copy. + result.coverage.note = withViewportCoverageNote(result.coverage).note; } // ENTITY-CENTRED NEARBY: answered by the SAME engine that fills the Contacts // panel, so the spoken number and the panel readout for one centre cannot @@ -4870,13 +4925,6 @@ async function runAnalystQuery( // arbitrary points — only "how many aircraft around " is // unified, because that is the question the panel is already answering. if (entityWindow) { - // The detached Contacts snapshot keeps its own coverage; carry the same - // viewport note so both answer shapes say what they measured. - if (viewportScoped && entityWindow.coverage) - entityWindow.coverage = { - ...entityWindow.coverage, - note: result.coverage?.note, - }; return entityWindow; } diff --git a/src/voice/gevRealtime.js b/src/voice/gevRealtime.js index e63c6af71..58efacad3 100644 --- a/src/voice/gevRealtime.js +++ b/src/voice/gevRealtime.js @@ -15,7 +15,7 @@ const ACTION_NAMES = new Set(GEV_ACTION_SCHEMAS.map((schema) => schema.name)); */ export function withToolCatalog(runner, loadCatalog) { if (typeof loadCatalog !== 'function') return runner; - return async function runGevTool(name, args, options = {}) { + const runGevTool = async function runGevTool(name, args, options = {}) { if (ACTION_NAMES.has(name)) return runner(name, args, options); const catalog = await loadCatalog(); if (!catalog?.get(name)) return runner(name, args, options); @@ -24,6 +24,10 @@ export function withToolCatalog(runner, loadCatalog) { }); return toFunctionOutput(name, result); }; + runGevTool.resetConversation = (...args) => + runner.resetConversation?.(...args); + runGevTool.dispose = (...args) => runner.dispose?.(...args); + return runGevTool; } /** Compose the standalone action runner with the voice controls. */ diff --git a/src/voice/gevRealtime.test.mjs b/src/voice/gevRealtime.test.mjs index dac3adf9a..d59cee759 100644 --- a/src/voice/gevRealtime.test.mjs +++ b/src/voice/gevRealtime.test.mjs @@ -3733,6 +3733,16 @@ test('voice runs app actions itself and other tools through the catalog', async actions.push([name, args]); return { ok: true, action: name }; }; + const resets = []; + const disposals = []; + runner.resetConversation = (...args) => { + resets.push(args); + return 'reset'; + }; + runner.dispose = (...args) => { + disposals.push(args); + return 'disposed'; + }; const calls = []; let loads = 0; const catalog = { @@ -3764,5 +3774,9 @@ test('voice runs app actions itself and other tools through the catalog', async actions.map(([name]) => name), ['zoom_to_globe', 'not_a_tool'], ); + assert.equal(run.resetConversation('stopped'), 'reset'); + assert.equal(run.dispose('removed'), 'disposed'); + assert.deepEqual(resets, [['stopped']]); + assert.deepEqual(disposals, [['removed']]); assert.equal(withToolCatalog(runner, undefined), runner); }); diff --git a/src/voice/pointerCrop.test.mjs b/src/voice/pointerCrop.test.mjs index b8e7fe911..57c8a5983 100644 --- a/src/voice/pointerCrop.test.mjs +++ b/src/voice/pointerCrop.test.mjs @@ -1,6 +1,10 @@ import assert from 'node:assert/strict'; import test from 'node:test'; -import { RealtimeViewport, pointerCropRect } from './realtimeViewport.js'; +import { + RealtimeViewport, + pointerCropRect, + renderFreshPointerFrame, +} from './realtimeViewport.js'; test('crop: a square around the pointer, clamped inside the canvas, in buffer pixels', () => { assert.deepEqual( @@ -48,6 +52,88 @@ test('crop: a square around the pointer, clamped inside the canvas, in buffer pi }), null, ); + assert.equal( + pointerCropRect({ + x: Number.NaN, + y: 1, + cssWidth: 100, + cssHeight: 100, + pixelWidth: 100, + pixelHeight: 100, + }), + null, + 'non-finite pointer geometry fails closed', + ); + assert.equal( + pointerCropRect({ + x: 101, + y: 1, + cssWidth: 100, + cssHeight: 100, + pixelWidth: 100, + pixelHeight: 100, + }), + null, + 'out-of-canvas pointer geometry fails closed', + ); + assert.equal( + pointerCropRect({ + x: 120, + y: 100, + cssWidth: 320, + cssHeight: 240, + pixelWidth: 320, + pixelHeight: 240, + }).output, + 512, + 'the provider image stays 512 px even when the source crop is smaller', + ); +}); + +test('crop: a slow render gets one bounded fresh-frame recovery request', async () => { + const originalDocument = globalThis.document; + let requests = 0; + let listener = null; + try { + globalThis.document = { hidden: false }; + const scene = { + postRender: { + addEventListener(next) { + listener = next; + return () => { + if (listener === next) listener = null; + }; + }, + }, + requestRender() { + requests += 1; + if (requests === 2) queueMicrotask(() => listener?.()); + }, + }; + assert.equal(await renderFreshPointerFrame({ scene }), true); + assert.equal(requests, 2, 'only the pointer path retries once'); + assert.equal(listener, null, 'the successful listener is removed'); + } finally { + globalThis.document = originalDocument; + } +}); + +test('crop: a hidden document never starts the recovery request', async () => { + const originalDocument = globalThis.document; + let requests = 0; + try { + globalThis.document = { hidden: true }; + const scene = { + postRender: { addEventListener: () => () => {} }, + requestRender() { + requests += 1; + }, + }; + assert.equal(await renderFreshPointerFrame({ scene }), false); + assert.equal(requests, 0); + } finally { + globalThis.document = originalDocument; + } }); function viewportFixture({ capturePointer, capture } = {}) { @@ -191,6 +277,15 @@ test('crop: a camera move before or during capture drops it; pixels only name th false, 'moved before capture', ); + state.camera = null; + assert.equal( + await viewport.sendPointerCrop( + { ...ground, cameraKey: null }, + { viewScale: 'local' }, + ), + false, + 'a missing pose identity fails closed', + ); state.camera = 'pose-A'; const pending = viewport.sendPointerCrop( { ...ground }, @@ -202,6 +297,28 @@ test('crop: a camera move before or during capture drops it; pixels only name th assert.equal(sent.length, 0); }); +test('crop: a snapshot that expires during capture is not published', async () => { + let release; + const { viewport, sent, state } = viewportFixture({ + capturePointer: () => + new Promise((resolve) => { + release = () => + resolve({ + dataUrl: 'data:image/jpeg;base64,CROP', + rect: { x: 0, y: 0, w: 1, h: 1 }, + }); + }), + }); + const pending = viewport.sendPointerCrop( + { ...ground }, + { viewScale: 'local' }, + ); + state.clock = 40_000; + release(); + assert.equal(await pending, false); + assert.equal(sent.length, 0); +}); + test('crop: an older capture that finishes last never replaces the newer image', async () => { const releases = []; const { viewport, sent } = viewportFixture({ diff --git a/src/voice/realtimeProtocol.js b/src/voice/realtimeProtocol.js index 3f3186347..14105229c 100644 --- a/src/voice/realtimeProtocol.js +++ b/src/voice/realtimeProtocol.js @@ -143,7 +143,14 @@ export function responseInstructionForToolResult(result) { return 'The map annotations are cleared. Continue naturally; do not announce the clear.'; } if (typeof result?.say === 'string' && result.say) { - return "Speak the say line of this turn's tool results once, in order, lightly rephrased at most; add no qualifiers and don't repeat your preamble."; + return "Speak the say line of this turn's tool results once, in order, lightly rephrased at most. Preserve its lower bounds, partial answers and stale, degraded or unavailable feed states. Add no counts, lists, qualifiers or action claims beyond the executed tool results, and don't repeat your preamble. If a requested list or another action is still unfinished, call the required tool before the final answer."; + } + if ( + result?.ok && + result.action === 'set_context_mode' && + result.contactsWindow + ) { + return 'Contacts activation and its window count do not fulfill a requested list or ranking. Continue any unfinished request with analyst_query: around the active subject use radius with the actual requested km and omit center; preserve explicit view, place, pointer and explicit center scopes. Use the returned items, never invent aircraft, counts or qualifications. Otherwise briefly confirm the completed action from its result.'; } return 'Briefly confirm the completed GEV action once. Do not repeat yourself.'; } diff --git a/src/voice/realtimeViewport.js b/src/voice/realtimeViewport.js index 38b25d6d6..755913906 100644 --- a/src/voice/realtimeViewport.js +++ b/src/voice/realtimeViewport.js @@ -86,13 +86,29 @@ export function pointerCropRect({ pixelHeight, size = POINTER_CROP_SIZE, }) { - if (!cssWidth || !cssHeight || !pixelWidth || !pixelHeight) return null; + const values = [x, y, cssWidth, cssHeight, pixelWidth, pixelHeight, size]; + if ( + values.some((value) => !Number.isFinite(value)) || + cssWidth <= 0 || + cssHeight <= 0 || + pixelWidth <= 0 || + pixelHeight <= 0 || + size <= 0 || + x < 0 || + x > cssWidth || + y < 0 || + y > cssHeight + ) + return null; const side = Math.min(size, cssWidth, cssHeight); const left = Math.min(Math.max(0, x - side / 2), cssWidth - side); const top = Math.min(Math.max(0, y - side / 2), cssHeight - side); const scaleX = pixelWidth / cssWidth; const scaleY = pixelHeight / cssHeight; - const output = Math.round(Math.min(size, side * scaleX)); + // Keep the provider-facing image contract stable even on a narrow canvas. + // drawImage may upscale a smaller source crop, but the retained image and + // pointer-ring coordinates are always expressed in one 512 px square. + const output = Math.round(size); return { sx: Math.round(left * scaleX), sy: Math.round(top * scaleY), @@ -116,7 +132,7 @@ export async function capturePointerCrop(screenPx) { const viewer = window.__godsEyeView?.viewer; const source = viewer?.scene?.canvas; if (!source?.width || !source?.height || !screenPx) return null; - const fresh = await renderFreshCesiumFrame(viewer); + const fresh = await renderFreshPointerFrame(viewer); if (!fresh) return null; const rect = pointerCropRect({ x: screenPx.x, @@ -166,6 +182,16 @@ export async function capturePointerCrop(screenPx) { } } +/** + * Pointer crops get one bounded recovery request for a slow render loop. + * Full-viewport captures intentionally keep their single 400 ms attempt. + */ +export async function renderFreshPointerFrame(viewer) { + if (await renderFreshCesiumFrame(viewer)) return true; + if (typeof document !== 'undefined' && document.hidden) return false; + return renderFreshCesiumFrame(viewer); +} + // Scale (w, h) down so w*h <= maxPixels while preserving aspect ratio. Never // upscales. Both dimensions shrink together, so portrait and landscape are // treated equally (M13). Pure + deterministic → unit-tested (exported below). @@ -345,11 +371,14 @@ export class RealtimeViewport { // Too far out, the crop is a smear: every path shares this guard. if (!POINTER_CROP_SCALES.has(viewScale)) return false; if (snapshot === this.lastPointerCrop) return false; - if (this.now() - (snapshot.capturedAt || 0) > POINTER_CROP_MAX_AGE_MS) - return false; + const capturedAt = Number(snapshot.capturedAt) || 0; + if (this.now() - capturedAt > POINTER_CROP_MAX_AGE_MS) return false; // The snapshot's pixels only name its own camera pose. + const snapshotCameraKey = snapshot.cameraKey; + if (!snapshotCameraKey) return false; const samePose = () => - !snapshot.cameraKey || snapshot.cameraKey === this.readCameraKey(); + snapshotCameraKey === this.readCameraKey() && + this.now() - capturedAt <= POINTER_CROP_MAX_AGE_MS; if (!samePose()) return false; this.lastPointerCrop = snapshot; const sent = await this.sendRetainedImage( diff --git a/src/voice/resultDisplay.js b/src/voice/resultDisplay.js index 8941acd74..7d364de79 100644 --- a/src/voice/resultDisplay.js +++ b/src/voice/resultDisplay.js @@ -28,10 +28,19 @@ const LAYER_NOUNS = Object.freeze({ /** Split "a; b" caveat text into separate notes. */ function notesOf(...values) { - return values - .flatMap((value) => String(value || '').split(/;\s*/)) - .map((text) => text.trim()) - .filter(Boolean); + return [ + ...new Set( + values + .flatMap((value) => String(value || '').split(/;\s*/)) + .map((text) => + text + .replace(/[\p{Cc}\p{Cf}]+/gu, ' ') + .replace(/\s+/g, ' ') + .trim(), + ) + .filter(Boolean), + ), + ]; } export function analystNounFor(layerKeys) { @@ -67,6 +76,19 @@ function analystCard(result) { if (!result?.ok) return null; const display = result.display || {}; const unanswered = Array.isArray(result.unanswered) ? result.unanswered : []; + const provenance = result.feedProvenance || result.coverage?.feedProvenance; + const sources = [ + ...new Set( + [...(provenance?.layers || []), ...(result.coverage?.layersQueried || [])] + .map((layer) => + String(layer?.source || '') + .replace(/[\p{Cc}\p{Cf}]+/gu, ' ') + .replace(/\s+/g, ' ') + .trim(), + ) + .filter(Boolean), + ), + ].map((label) => ({ label })); return { display: { title: analystHeadline(result), @@ -81,12 +103,16 @@ function analystCard(result) { : []), ...(result.partial ? [{ label: 'partial' }] : []), ], + // Coverage qualifications lead the feed detail. The bounded card groups + // overflow in its existing Notes disclosure rather than dropping them. + preserveNotes: true, notes: [ - ...notesOf(display.caveat, result.coverage?.note), + ...notesOf(result.coverage?.note, display.caveat), ...(unanswered.length ? [`Not answered: ${unanswered.join(', ')}`] : []), ], + sources, }, referents: Array.isArray(result.items) ? result.items : [], resultSet: true, diff --git a/src/voice/speech.js b/src/voice/speech.js index ccf35664c..823c65b26 100644 --- a/src/voice/speech.js +++ b/src/voice/speech.js @@ -401,6 +401,18 @@ function annotateMap(result) { function analystQuery(result) { if (!result?.ok || result.cancelled) return null; const headline = spokenLabel(analystHeadline(result), 96); + const state = + result.feedState || + result.feedProvenance?.overall || + result.coverage?.feedProvenance?.overall; + const tag = feedTag(state); + const scope = spokenLabel(result.scopeLabel || '', 96); + const answer = + tag === 'feed unavailable' + ? sentence( + `Feed unavailable${scope ? ` ${scope}` : ''}; no authoritative count${result.complete === false ? ', retained coverage is a lower bound' : ''}`, + ) + : sentence(`${headline}${tag ? `, ${tag}` : ''}`); const unanswered = (Array.isArray(result.unanswered) ? result.unanswered : []) .map((layer) => spokenLabel(layer, 24)) .filter(Boolean); @@ -408,7 +420,7 @@ function analystQuery(result) { ? `Partial; ${listPhrase(unanswered.slice(0, 2))}${unanswered.length > 2 ? ` and ${unanswered.length - 2} more` : ''} not answered.` : 'Partial answer.'; return { - say: `${sentence(headline)}${result.partial ? ` ${missing}` : ''}`, + say: `${answer}${result.partial ? ` ${missing}` : ''}`, }; } diff --git a/src/voice/voiceCardPresentation.js b/src/voice/voiceCardPresentation.js index eb35b8e56..0dead3f6a 100644 --- a/src/voice/voiceCardPresentation.js +++ b/src/voice/voiceCardPresentation.js @@ -242,12 +242,28 @@ export function reduceVoiceCard(state, event) { export function voiceCardView(state) { const running = state.steps.some((step) => step.status === 'running'); const display = state.result?.display || null; - const notes = [ + const allNotes = [ ...(display?.notes || []), ...(display?.sources || []).map((source) => `Source: ${source.label}`), - ] - .map((note) => spokenLabel(note, 96)) - .slice(0, MAX_NOTES); + ]; + // Analyst notes carry count/scope/provenance qualifications. Preserve their + // full sanitized text in at most six existing rows, joining overflow into + // the last row. Other tool cards retain their compact note contract. + const completeNotes = display?.preserveNotes + ? [ + ...new Set( + allNotes.map((note) => spokenLabel(note, Infinity)).filter(Boolean), + ), + ] + : null; + const notes = completeNotes + ? completeNotes.length > MAX_NOTES + ? [ + ...completeNotes.slice(0, MAX_NOTES - 1), + completeNotes.slice(MAX_NOTES - 1).join('; '), + ] + : completeNotes + : allNotes.map((note) => spokenLabel(note, 96)).slice(0, MAX_NOTES); return { visible: Boolean(state.visible), busy: running,