mirror of
https://github.com/bilawalsidhu/gods-eye-view.git
synced 2026-11-12 22:32:19 +00:00
Merge upstream main into denis/rail-layer
This commit is contained in:
commit
13818d0b9f
87 files changed
+7438
-144
No files matched your search
+108
-2
@@ -1137,7 +1137,7 @@ Voice controls bind to a protocol-independent session factory. The default WebRT
|
||||
|
||||
The standalone app offers OpenAI and Gemini Live through the **AI Provider** setting, shared by voice and typed chat. Gemini keeps its permanent API key on the local server and mints a short-lived, single-use credential with server-owned instructions and the standalone action catalog. With no Gemini key the token route answers `503` with code `API_KEY_MISSING`, and the voice control says `Add a Gemini API key in Provider Settings` without the microphone-permission hint. Voice status and error messages keep the message as given on the session's `state` event (`sourceDetail`, English catalog text when it is the app's own, with `sourceParams` for its placeholders), beside the worded `detail`, so the voice control words them again after a language change. Typed-chat errors and the chat panel are worded once. Its native WebSocket adapter supports open microphone and push-to-talk input, streaming audio, captions and the existing action runner. In push-to-talk, a tool call that arrives with no input, reply or tool result outstanding is answered with a failure and never runs, and neither does any other call in the reply it starts; open-microphone turns start on Gemini's server-side activity detection, which reports no boundary to check. Switching providers ends the old conversation and releases its microphone and connection; only the input mode carries over. Gemini uses bounded shutdown draining and explicit manual restart, without automatic reconnection or a promise of uninterrupted session duration. Hosts that do not opt into the provider facade retain their existing default or custom adapter.
|
||||
|
||||
Voice action argument schemas have one portable owner under `src/voice/actionSchemas.js`. The Realtime provider builds 33 tools using separate description-only metadata; a session is offered 32 of them unless `OVERPASS_UPSTREAMS` is set, which adds `osm_query` and its one prompt sentence (`server/providers/openai/tools.js`). Description customization cannot replace argument types, enum values or required fields.
|
||||
Voice action argument schemas have one portable owner under `src/voice/actionSchemas.js`. The Realtime provider builds 35 tools using separate description-only metadata; a session is offered 34 of them unless `OVERPASS_UPSTREAMS` is set, which adds `osm_query` and its one prompt sentence (`server/providers/openai/tools.js`). Description customization cannot replace argument types, enum values or required fields.
|
||||
|
||||
Voice areas live in a store owned by the action runner for the page (`src/data/areaStore.js`); the model gets an `areaId`, a rounded bounding box and a summary, never geometry. `resolve_area` (`src/voice/areaActions.js`, `src/data/areaResolver.js`) checks the store, bundled Natural Earth regions, then bundled countries and states by name — a name several prominent units share returns up to four candidates unless the view is inside exactly one — and then the annotation outline ladder (`resolveAnnotationTarget`: bundled counties, Census places and neighborhoods, the guarded `/api/geocode/outline`, and Overpass only when configured). Administrative geometry is stored only when its source identity agrees with the requested unit, by name or by a form the bundled matcher reads as that unit at its level ("State of Texas", "Province of Bagmati", "USA", "UK"; not "Great Britain" or "Korea"); stale, mismatched, synthesized or otherwise unverified administrative boundaries are refused rather than relabelled. Clarification IDs use the bounded `ne:<kind>:<slug>` grammar and are accepted only from the current resolver's candidate memory. A lookup past 8 s answers `AREA_TIMEOUT` and keeps filling the caches. `around=true` asks the same ladder for the grounds enclosing a landmark (Nominatim, then the OpenFreeMap area holding it) and otherwise stores a 250 m buffer marked approximate. `analyst_query` scopes add `area`, `drawn` (newest draw-tool outline) and `annotation` (newest or named outline); region scopes use the same resolver within 3 s. Drawn outlines are a simplified copy (≤1,200 vertices); counting reads the stored geometry (`src/data/areaGeometry.js`). `find_imagery` (`src/voice/imageryActions.js`) sets the Recent Imagery box from an area, the view or the turn's pointer snapshot and pins the best acquisition; any hand change to the panel or explicit layer OFF while it searches ends the request (`DISPLACED`) without pinning or success narration. `osm_query` asks `/api/osm/features` for one curated preset in a bounded box; the route answers `OVERPASS_NOT_CONFIGURED` without an operator Overpass. Results go to the session-only `osm-places` layer and become the analyst's last answer; a list capped at its limit stays incomplete in follow-ups, which keep its "listed the first N" caveat and report counts as floors. An explicit OSM Places OFF while the provider is pending aborts the work and prevents re-enable, result, credit and follow-up-memory commits. The voice answer deadline also aborts the client provider request, so the caller does not leave a timed-out request without a cancellation owner. The server's whole search (22 s across every configured endpoint, with Overpass told 20 s) ends before that 25 s deadline, because a browser that stops waiting cancels the upstream query and leaves nothing cached for a retry.
|
||||
|
||||
@@ -3053,6 +3053,7 @@ to a set time, naming its end as `at` (epoch ms): the row then reads
|
||||
`window_end`. The dev proxy never sets `at`.
|
||||
|
||||
The FIRMS CSV parser rejects impossible acquisition calendar dates before the proxy filters records to the trailing 24 hours, while retaining valid UTC acquisition times.
|
||||
FIRMS ages round to whole hours before choosing a unit, so a 47.6-hour age reads "2d", not "48h"; with detections filtered to 24 hours, this shows mostly in the `STALE · cached {age}` stats line.
|
||||
The `get_active_fires` tool reports acquisition times as UTC ISO timestamps,
|
||||
including zero-padded hours and minutes; missing or invalid dates and times
|
||||
remain `null`, including impossible calendar dates from a custom source.
|
||||
@@ -5258,6 +5259,111 @@ core's lookups alone. The MCP globe panel allows the meta-catalogue and the
|
||||
three biggest instances; a picture on another instance shows only in the
|
||||
app.
|
||||
|
||||
**"Show me X on Street View."** `show_street_level` (voice and typed;
|
||||
`src/voice/gevActions.js`) is one call whose stages fail separately
|
||||
(`location`, `view`, `map`, `provider`, `layer`, `key`, `imagery`):
|
||||
|
||||
- **Key first.** A keyless source explains itself and changes nothing, as a
|
||||
chip does.
|
||||
- **Place.** A name, coordinates, a pointed spot or a numbered item: it flies
|
||||
there at street framing and waits for arrival. With no place it looks at the
|
||||
ground under the screen centre, as OPEN does (`aim: 'screen-centre'`,
|
||||
`armed: false`: sky refuses, and a link's armed point is not used and is kept
|
||||
until a photo opens). Coordinates off the globe are refused.
|
||||
- **Map and source.** It switches to the map the source needs (Google Street
|
||||
View: Google 3D), lights the source through `setLayerParams`, one at a time
|
||||
and kept like a chip click (origin `voice`; a link runs the action with
|
||||
origin `programmatic`, so it never writes the recipient's saved layer state),
|
||||
turns the layer on and calls `openNearest`. `provider` is `google` for
|
||||
"Street View", `mapillary`, `kartaview` or `panoramax` by name, else the lit
|
||||
source.
|
||||
- **Hidden panel.** Cockpit, clean view, recording or scene playback end it at
|
||||
`view`, before the flight and after every await; embed mode's own clean view
|
||||
is exempt, since the photo is lifted over the globe.
|
||||
- **Superseded.** After every await (flight, map switch, layer on, lookup) a
|
||||
newer turn, a shell navigation stamp (`_navigationGeneration`, re-taken where
|
||||
its own flight lands) or a press or wheel on the globe stops it
|
||||
(`superseded: true`). The same checks run before a found photo loads
|
||||
(`proceed`), so nothing billed loads for a request that was given up. A
|
||||
lookup given up leaves the panel as it was when the request kept the photo's
|
||||
source; one that had switched the source off closes that source's photo with
|
||||
it. A photo that opened after either is closed again.
|
||||
- **Replacing a photo.** An open photo stays up while the new request looks
|
||||
(`openNearest` `replace: true`; the viewer is replaced, not closed, so EXPAND
|
||||
stays, also across sources through `holdViewer`, which holds only the action's
|
||||
own switch-off of the photo's source: a source switched by hand meanwhile
|
||||
still stops the load) and closes if nothing opens, so a place with no imagery
|
||||
shows its own error; a link that only arms closes it at once. A replacement
|
||||
from another source drops the old photo's details, and the request's error or
|
||||
armed OPEN outlives the old viewer. A replacement stops a playing Mapillary
|
||||
sequence.
|
||||
- **Outcome.** `noImagery` marks an empty result, `imageryError` a source or
|
||||
viewer failure and `aim: 'sky'` a sky centre (from `street.nearestOutcome`);
|
||||
an empty result names a narrowing 360°/flat or SINCE filter the source
|
||||
honours, as `filter`. Every photo the action opens reveals the panel, also
|
||||
one the user had collapsed.
|
||||
- **Search.** A source without a coverage layer searches up to 250 m
|
||||
(`NEAREST_RADIUS_MAX_M`; Google honours it); clicks keep 50 m. Mapillary
|
||||
asks for 100 photos within 50 m, its API's most (the API returns them
|
||||
unordered and cuts at the limit), and gives up after 12 s, body included
|
||||
(`NEAREST_TIMEOUT_MS`).
|
||||
- **Phone.** The panel sits in the Cameras tab sheet: a photo that opens, or a
|
||||
link that arms, opens the sheet and scrolls the panel into view
|
||||
(`revealPanel`, which does nothing above 720px).
|
||||
|
||||
The layer also exposes `requiredMapStack(id)` and `providerStatus(id)` for
|
||||
callers that decide before the panel lists a source, and `holdViewer()`. A
|
||||
caller's abort is honoured through the viewer's mount and
|
||||
load, and only the lookup that still owns the panel changes its spinner or
|
||||
error.
|
||||
|
||||
Views carry the request as `street_level` (`src/view/index.js`): a source and
|
||||
a point, which defaults to where the camera looks (below 20 km, at a pitch
|
||||
steeper than -5° and within 10 km ahead; else the camera's own point); half a
|
||||
point is not mixed with the camera's: both coordinates come from that default. A `google` source brings `map: photoreal` and refuses
|
||||
another map; `street_level` does not combine with `follow`.
|
||||
`show_in_gods_eye_view`, and the embedded app for a posted view with no
|
||||
point, measure the view's altitude from the terrain under the camera
|
||||
(`groundedLookAt` with the terrain heights service) before taking the look-at
|
||||
point, and keep the view's own estimate without a terrain answer (the embedded
|
||||
app waits 2 s for one). Links carry
|
||||
`sl=<source>,<lat>,<lon>`; the globe panel applies views through the action,
|
||||
and a plain link runs it once restored (one shot: the rewritten hash drops
|
||||
`sl`). A link that follows an entity ignores `sl`.
|
||||
|
||||
- **Billing gate.** A plain link never loads a photo from a source that bills
|
||||
every load (`billedPerImage`; Google): the action runs with
|
||||
`autoOpen: false` and arms the layer (`armNearest`: the link's point and
|
||||
radius, for the next OPEN press only, and only while the ground under the
|
||||
screen centre is within 1 km of it (`ARMED_REACH_M`) or cannot be picked; a
|
||||
sky centre refuses; any other lookup, open, close, disable, the source
|
||||
switched off or its map leaving drops it), opens the panel and shows a toast
|
||||
pointing at OPEN GOOGLE STREET VIEW. A link whose source has no working
|
||||
key here changes nothing and says so in a toast (not in a frame).
|
||||
- **Who may open.** Posted views, and the panel page's own link (inline embed)
|
||||
for a tool result issued in the last 5 minutes, open the photo:
|
||||
`show_in_gods_eye_view` stamps a street-level result `issued_at`, every
|
||||
`gev:view` the panel posts carries `fresh`, and the inline app applies a
|
||||
view without it with `autoOpen: false` (a framing page's posted views still
|
||||
open). `?embed=1` on a published link only arms, without the panel nudge or
|
||||
toast.
|
||||
- **Who owns the camera.** A link's photo step is skipped when a newer link
|
||||
superseded its restore, when its restore flight never landed
|
||||
(`share.camera !== 'applied'`: the visitor dragged or searched), or after
|
||||
any navigation since the generation that flight owned (`navigationToken` on
|
||||
the settled result), also while its layers still load. A link pasted into
|
||||
the open tab runs its step after its restore settles
|
||||
(`gev:share-navigation-restore-settled`).
|
||||
- **Embed.** The rail is hidden, so an open photo moves to an overlay on the
|
||||
globe (`sl-viewer-wrap-embed`) and back when it closes; Esc closes it (in
|
||||
full screen, Esc leaves full screen first). EXPAND stays under embed mode's
|
||||
clean view; cockpit, recording and scene playback hide the overlay. Loading
|
||||
errors render inside the overlay, and inside the viewer while EXPAND is on.
|
||||
A posted view whose photo step fails before the imagery stage closes the
|
||||
earlier photo before its camera flies; one the user overtook (a click on the
|
||||
map, a newer request) leaves their photo and camera alone; one that found a
|
||||
photo it could not load keeps that error.
|
||||
|
||||
Street Level has two browser gates. `npm run qa:street-level -- --url <server>`
|
||||
runs against real Mapillary and needs `MAPILLARY_CLIENT_TOKEN`. `npm run
|
||||
qa:street-level:fixtures -- --url <server>` answers every Mapillary request
|
||||
@@ -6586,7 +6692,7 @@ and unreachable upstream (502/504) separately from road geometry.
|
||||
|
||||
- **Token flow**: browser fetches a short-lived client secret from `/api/realtime/token`; the Vite middleware uses `OPENAI_API_KEY` by default, or the signed-in local ChatGPT/Codex OAuth access token when OAuth was explicitly selected in Provider Settings, and posts the full session config (instructions, tool schemas, VAD, truncation) to `api.openai.com/v1/realtime/client_secrets`. Both long-lived credentials stay server-side; OAuth minting is loopback-only. SDP exchange goes directly to `api.openai.com/v1/realtime/calls` with the ephemeral token. With no OpenAI key the token route answers `503` with code `API_KEY_MISSING`, and the voice control shows `Needs OPENAI_API_KEY — add it in Provider Settings` in the interface language, reworded on a language change, without the microphone-permission hint.
|
||||
- **Session defaults** (env-tunable): model `gpt-realtime-2` (or `gpt-realtime-2.1-mini` when the MINI tier is selected — see the model-tier entry below), voice `marin`, reasoning effort `low`, semantic VAD with low eagerness, no response interruption, context window truncated to ~3,000 post-instruction tokens with 0.5 retention ratio — the conversational window stays short because map state is fetched live per turn.
|
||||
- **Thirty tools** (argument schemas in `src/voice/actionSchemas.js`, served with their descriptions by `server/providers/openai/tools.js`, executed client-side in `src/voice/gevActions.js`): `fly_to_location`, `select_nearest_aircraft`, `adjust_camera_zoom`, `zoom_to_globe`, `set_layer_visibility`, `show_data_layers_menu`, `set_panel_open`, `set_visual_style`, `get_entity_context`, `get_current_view_state`, `set_hud`, `set_cyber_sonar`, `set_detection`, `set_map_stack`, `set_post_processing`, `control_scene`, `control_cctv`, `set_context_mode`, `control_cockpit`, `control_radio`, `track_entity`, `stop_tracking`, `frame_overhead`, `annotate_map`, `clear_annotations`, `move_camera`, `fly_route`, `analyst_query`, `next_iss_pass`, and `next_satellite_pass`.
|
||||
- **Thirty-five tools** (thirty-four in a session without `OVERPASS_UPSTREAMS`, which adds `osm_query`; argument schemas in `src/voice/actionSchemas.js`, served with their descriptions by `server/providers/openai/tools.js`, executed client-side in `src/voice/gevActions.js`): `fly_to_location`, `select_nearest_aircraft`, `show_street_level`, `adjust_camera_zoom`, `zoom_to_globe`, `set_layer_visibility`, `show_data_layers_menu`, `set_panel_open`, `set_voice_inactivity_timeout`, `set_visual_style`, `get_entity_context`, `get_current_view_state`, `set_hud`, `set_cyber_sonar`, `set_detection`, `set_map_stack`, `set_post_processing`, `control_scene`, `control_cctv`, `set_context_mode`, `control_cockpit`, `control_radio`, `track_entity`, `stop_tracking`, `frame_overhead`, `annotate_map`, `clear_annotations`, `move_camera`, `fly_route`, `analyst_query`, `resolve_area`, `find_imagery`, `osm_query`, `next_iss_pass`, and `next_satellite_pass`.
|
||||
|
||||
> **Reading `npm test` totals:** the count depends on the Node major. The two
|
||||
> GC-bracketed allocation microbenchmarks (`src/data/focusAllocations.test.mjs`
|
||||
|
||||
@@ -68,6 +68,16 @@ cannot name still makes a place view. Opening a vessel thing view with a
|
||||
world time waits for the vessel layer's snapshot at that time and selects the
|
||||
vessel once it has it.
|
||||
|
||||
## Street-level photos (`sl`)
|
||||
|
||||
Views built by `show_in_gods_eye_view` or `src/view/index.js` can add
|
||||
`sl=<source>,<lat>,<lon>` to the long form. Once the link has restored, the
|
||||
app opens the nearest photo there, or, for a source that bills every load,
|
||||
arms the panel so OPEN loads it (see `docs/TOOLS.md`). The app's own links do
|
||||
not carry it: the address bar's long form drops `sl` when the camera next
|
||||
moves, and the short form has no field for it. A reload or the app's Copy
|
||||
Link keeps the lit source, the layer and the map, but not the photo request.
|
||||
|
||||
## Payload
|
||||
|
||||
The code is unpadded base64url (`A–Z a–z 0–9 - _`). The decoded bytes are:
|
||||
|
||||
+23
-5
@@ -102,11 +102,29 @@ action draws (pins, highlights, areas, arrows, routes and labels at a place
|
||||
name or coordinates). Links carry them in the `an` parameter, and the app
|
||||
draws them once the link has been restored.
|
||||
|
||||
A view can open street-level photos too: `street_level` names a source
|
||||
(`google` for Street View, `mapillary`, `kartaview`, `panoramax`, or none for
|
||||
the lit one) and a point, by default the spot the camera looks at. Street View
|
||||
shows only on the Google 3D map, so a `google` source sets `map` to
|
||||
`photoreal` and refuses another; a view cannot both follow an entity and open
|
||||
photos. Links carry it as `sl=<source>,<lat>,<lon>`: once restored, the app
|
||||
runs `show_street_level`, which flies there and opens the nearest photo. A
|
||||
plain link with a `google` source only arms the panel for OPEN GOOGLE STREET
|
||||
VIEW; a view posted into the embedded app opens it, and so does the panel
|
||||
page's own opening link, but only for a tool result issued in the last five
|
||||
minutes (an older one, such as a host remounting the panel, only arms);
|
||||
`?embed=1` on a published link does not. Mapillary,
|
||||
KartaView and Panoramax, which do not bill per image, open at once. A link
|
||||
that follows an entity ignores `sl`.
|
||||
|
||||
Tools take a view as `VIEW_ARGUMENTS`: an `area` to frame from above, or a
|
||||
`camera`, plus `layers`, `style`, `map`, `follow` and `annotations`. A
|
||||
tilted camera over an area looks at its center from behind; camera fields
|
||||
given with an area override its framing. `resolveViewArguments` turns them
|
||||
into a view.
|
||||
`camera`, plus `layers`, `style`, `map`, `follow`, `annotations` and
|
||||
`street_level`. A tilted camera over an area looks at its center from behind;
|
||||
camera fields given with an area override its framing. A passed-through
|
||||
view's `street_level` point moves with a new `area` or camera position (a
|
||||
turned camera keeps it). `resolveViewArguments` turns them into a view. The
|
||||
app's own links (the address bar, Copy Link, `/share/<code>`) do not carry
|
||||
`sl`; see `docs/SHARE-LINK-FORMAT.md`.
|
||||
|
||||
### The God's Eye View panel
|
||||
|
||||
@@ -299,5 +317,5 @@ and the development server's internal routes. See SECURITY.md.
|
||||
| `situation_brief` | `weather` | Weather, earthquakes, fires, aircraft, ships and cyclones for an area, by section |
|
||||
| `military_awareness` | `military` | Military and other aircraft, ships and military installations within 250 km of a point, by section |
|
||||
| `get_hud_caption` | `weather`, `summary` | The app's heads-up display caption for an area |
|
||||
| `show_in_gods_eye_view` | `app` | A view in God's Eye View: the live panel in clients with MCP Apps, and a link everywhere; takes another answer's view or an area or camera, layers, style, map, marks and something to follow |
|
||||
| `show_in_gods_eye_view` | `app` | A view in God's Eye View: the live panel in clients with MCP Apps, and a link everywhere; takes another answer's view or an area or camera, layers, style, map, marks, something to follow, or street-level photos (Street View or Mapillary) at a place |
|
||||
| `panel_request` | `app` | Panel only: loads a path from the app's server for the God's Eye View panel |
|
||||
Reference in new issue
Block a user