feat(street-level): "show me X on Street View" action, street_level view field and sl= links

show_street_level (voice and typed) flies to a place and opens its nearest
street-level photo from Google Street View, Mapillary, KartaView or
Panoramax; Street View switches to Google 3D first. The view model gains a
street_level field, share links carry sl=<source>,<lat>,<lon>, and
show_in_gods_eye_view can open a photo. A plain sl=google link only arms the
point and waits for OPEN, so it never bills a key without a click.

Embed pages show an open photo in an overlay that Esc closes. On-screen
strings are translated into es, ru, zh-CN and zh-Hant. Covered by unit tests
and three new steps in the Street Level browser suite.
This commit is contained in:
Said Turksever committed 2026-11-03 14:32:02 +00:00
1 parent f3adccd01a
commit 98de41c94a
64 files changed
+4147 -66

No files matched your search

+49 -2
View File
@@ -828,7 +828,7 @@ Voice controls bind to a protocol-independent session factory. The default WebRT
The standalone app offers OpenAI and Gemini Live through an explicit provider chooser. Gemini keeps its permanent API key on the local server and mints a short-lived, single-use credential with server-owned instructions and the standalone action catalog. Its native WebSocket adapter supports open microphone and push-to-talk input, streaming audio, captions and the existing action runner. In push-to-talk, a tool call that arrives with no input, reply or tool result outstanding is answered with a failure and never runs, and neither does any other call in the reply it starts; open-microphone turns start on Gemini's server-side activity detection, which reports no boundary to check. Switching providers ends the old conversation and releases its microphone and connection; only the input mode carries over. Gemini uses bounded shutdown draining and explicit manual restart, without automatic reconnection or a promise of uninterrupted session duration. Hosts that do not opt into the provider facade retain their existing default or custom adapter.
Voice action argument schemas have one portable owner under `src/voice/actionSchemas.js`. The Realtime provider builds 30 tools using separate description-only metadata. Description customization cannot replace argument types, enum values or required fields.
Voice action argument schemas have one portable owner under `src/voice/actionSchemas.js`. The Realtime provider builds 32 tools using separate description-only metadata. Description customization cannot replace argument types, enum values or required fields.
`createGevActionRunner` routes `control_cctv` enable/disable through the shared
layer visibility lifecycle. Results distinguish successful changes from
@@ -4328,6 +4328,53 @@ core's lookups alone. The MCP globe panel allows the meta-catalogue and the
three biggest instances; a picture on another instance shows only in the
app.
**"Show me X on Street View."** The `show_street_level` assistant action
(voice and typed; `src/voice/gevActions.js`) is one call: it checks the
source has its key (a keyless source explains itself and changes nothing, as
a chip does), flies to the place at street framing and waits for arrival (a
name, coordinates, a pointed spot, a numbered item, or nothing for the view
centre), switches to the map the source needs (Google Street View: Google 3D),
lights that source through `setLayerParams` with origin `voice` (one source at
a time, kept in the share link like a chip click), turns the layer on and
calls `openNearest`. `provider` is `google` for "Street View", `mapillary`,
`kartaview` or `panoramax` by name, else the lit source. Stages fail separately (`location`, `map`,
`provider`, `layer`, `key`, `imagery`; an empty `imagery` result names a
narrowing 360°/flat or SINCE filter the source honours, as `filter`) and a
newer turn retires the lookup
(`openNearest`'s `signal` option), so nothing opens after the user moved on.
For a source without a coverage layer the lookup widens to 250 m
(`NEAREST_RADIUS_MAX_M`; Google honours it, Mapillary's API caps at 50 m);
clicks keep 50 m. The layer exposes `requiredMapStack(id)` and
`providerStatus(id)` for callers that must decide before the panel lists a
source, and `coverageLines(id)` (the lines a provider draws now, current and
stale tiles, as `[lon, lat]` parts) for diagnostics and the browser gate,
which aims its line clicks at that geometry, waits for the asynchronous
ground primitives to be ready, and clicks in the same JavaScript turn as the
pick so a rebuilt frame cannot come between them. Views carry the same request as `street_level` (`src/view/index.js`:
source and point, defaulting to where the camera looks; a `google` source
brings `map: photoreal` and refuses another map; not with `follow`), links as
`sl=<source>,<lat>,<lon>`; `show_in_gods_eye_view` accepts it, the globe panel
applies it through the action, and a plain link runs the action once restored
(one shot: the rewritten hash drops `sl`). In embed mode the rail holding the
panel is hidden, so an open photo moves to an overlay on the globe
(`sl-viewer-wrap-embed`) and back when it closes; Esc closes it (in full
screen, Esc leaves full screen first). A caller's abort is
honoured through the viewer's mount and load, not just the lookup, and only
the lookup that still owns the panel may change its spinner or error. A
passive link's photos stand down if anyone navigated (a gesture, a search,
the assistant) while the link waited behind its own marks. A plain
link never loads a photo from a source that bills every load
(`billedPerImage` on the provider; Google): the action runs with
`autoOpen: false`, arms the lookup on the layer (`armNearest`: the link's
point and radius, consumed only by the next OPEN press (`aim:
'screen-centre'`); any other lookup, open, close, disable, its source
switched off or its map leaving drops it), opens the panel and a toast
points at OPEN GOOGLE STREET VIEW. Only the panel page's own link (inline
embed) and posted views open the photo; `?embed=1` on a published link only
arms, without the panel nudge or toast (the rail is hidden in a frame). A link that follows an entity ignores `sl`.
The look-at default applies below 20 km, at a pitch steeper than -5° and
within 10 km ahead; otherwise the camera's own point is used.
Street Level has two browser gates. `npm run qa:street-level -- --url <server>`
runs against real Mapillary and needs `MAPILLARY_CLIENT_TOKEN`. `npm run
qa:street-level:fixtures -- --url <server>` answers every Mapillary request
@@ -5361,7 +5408,7 @@ and unreachable upstream (502/504) separately from road geometry.
- **Token flow**: browser fetches a short-lived client secret from `/api/realtime/token`; the Vite middleware uses `OPENAI_API_KEY` by default, or the signed-in local ChatGPT/Codex OAuth access token when OAuth was explicitly selected in Provider Settings, and posts the full session config (instructions, tool schemas, VAD, truncation) to `api.openai.com/v1/realtime/client_secrets`. Both long-lived credentials stay server-side; OAuth minting is loopback-only. SDP exchange goes directly to `api.openai.com/v1/realtime/calls` with the ephemeral token.
- **Session defaults** (env-tunable): model `gpt-realtime-2` (or `gpt-realtime-2.1-mini` when the MINI tier is selected — see the model-tier entry below), voice `marin`, reasoning effort `low`, semantic VAD with low eagerness, no response interruption, context window truncated to ~3,000 post-instruction tokens with 0.5 retention ratio — the conversational window stays short because map state is fetched live per turn.
- **Thirty tools** (argument schemas in `src/voice/actionSchemas.js`, served with their descriptions by `server/providers/openai/tools.js`, executed client-side in `src/voice/gevActions.js`): `fly_to_location`, `select_nearest_aircraft`, `adjust_camera_zoom`, `zoom_to_globe`, `set_layer_visibility`, `show_data_layers_menu`, `set_panel_open`, `set_visual_style`, `get_entity_context`, `get_current_view_state`, `set_hud`, `set_cyber_sonar`, `set_detection`, `set_map_stack`, `set_post_processing`, `control_scene`, `control_cctv`, `set_context_mode`, `control_cockpit`, `control_radio`, `track_entity`, `stop_tracking`, `frame_overhead`, `annotate_map`, `clear_annotations`, `move_camera`, `fly_route`, `analyst_query`, `next_iss_pass`, and `next_satellite_pass`.
- **Thirty-two tools** (argument schemas in `src/voice/actionSchemas.js`, served with their descriptions by `server/providers/openai/tools.js`, executed client-side in `src/voice/gevActions.js`): `fly_to_location`, `select_nearest_aircraft`, `show_street_level`, `adjust_camera_zoom`, `zoom_to_globe`, `set_layer_visibility`, `show_data_layers_menu`, `set_panel_open`, `set_voice_inactivity_timeout`, `set_visual_style`, `get_entity_context`, `get_current_view_state`, `set_hud`, `set_cyber_sonar`, `set_detection`, `set_map_stack`, `set_post_processing`, `control_scene`, `control_cctv`, `set_context_mode`, `control_cockpit`, `control_radio`, `track_entity`, `stop_tracking`, `frame_overhead`, `annotate_map`, `clear_annotations`, `move_camera`, `fly_route`, `analyst_query`, `next_iss_pass`, and `next_satellite_pass`.
> **Reading `npm test` totals:** the count depends on the Node major. The two
> GC-bracketed allocation microbenchmarks (`src/data/focusAllocations.test.mjs`
+20 -5
View File
@@ -102,11 +102,26 @@ action draws (pins, highlights, areas, arrows, routes and labels at a place
name or coordinates). Links carry them in the `an` parameter, and the app
draws them once the link has been restored.
A view can open street-level photos too: `street_level` names a source
(`google` for Street View, `mapillary`, `kartaview`, `panoramax`, or none for
whichever the panel has lit) and a point, which defaults to the spot the camera looks at. Street View
shows only on the Google 3D map, so a `google` source sets `map` to
`photoreal` and any other map is refused; a view cannot both follow an entity
and open photos. The `street-level` layer is turned on. Links carry it as
`sl=<source>,<lat>,<lon>`; once the link has restored, the app runs its
`show_street_level` action, which flies to the point at street framing and
opens the nearest photo. A plain link with a `google` source stops before the
billed load: the panel is armed and OPEN GOOGLE STREET VIEW is one click away.
A view posted into the embedded app, or the panel page's own opening link,
opens the photo, since it is the client's own request; `?embed=1` on a
published link does not. Mapillary, KartaView and Panoramax links open the
photo at once; none of them bills per image. A link that follows an entity ignores `sl`.
Tools take a view as `VIEW_ARGUMENTS`: an `area` to frame from above, or a
`camera`, plus `layers`, `style`, `map`, `follow` and `annotations`. A
tilted camera over an area looks at its center from behind; camera fields
given with an area override its framing. `resolveViewArguments` turns them
into a view.
`camera`, plus `layers`, `style`, `map`, `follow`, `annotations` and
`street_level`. A tilted camera over an area looks at its center from behind;
camera fields given with an area override its framing. `resolveViewArguments`
turns them into a view.
### The God's Eye View panel
@@ -299,5 +314,5 @@ and the development server's internal routes. See SECURITY.md.
| `situation_brief` | `weather` | Weather, earthquakes, fires, aircraft, ships and cyclones for an area, by section |
| `military_awareness` | `military` | Military and other aircraft, ships and military installations within 250 km of a point, by section |
| `get_hud_caption` | `weather`, `summary` | The app's heads-up display caption for an area |
| `show_in_gods_eye_view` | `app` | A view in God's Eye View: the live panel in clients with MCP Apps, and a link everywhere; takes another answer's view or an area or camera, layers, style, map, marks and something to follow |
| `show_in_gods_eye_view` | `app` | A view in God's Eye View: the live panel in clients with MCP Apps, and a link everywhere; takes another answer's view or an area or camera, layers, style, map, marks, something to follow, or street-level photos (Street View or Mapillary) at a place |
| `panel_request` | `app` | Panel only: loads a path from the app's server for the God's Eye View panel |