mirror of
https://github.com/bilawalsidhu/gods-eye-view.git
synced 2026-11-11 05:42:22 +00:00
fix(voice): make viewport escalation diagnostics truthful and paired
Review follow-up for #835. What is captured, sent and kept is unchanged. - Log nothing for tool results other than get_entity_context, so other tools stop spending the shared debug-log budget. - Give every decision and outcome the same contextId and image kind, so overlapping requests pair up. - Log why a crop decision did not send (stale, off ground, off screen, expired, pose changed, already requested) instead of leaving a lone "pointer-ground-crop" that reads as sent. - Split capture-invalidated into session-reset, superseded, channel-lost and frame-stale. - Close the pair with "error" when a step throws, then rethrow. - Document the events in CURRENT-STATE and reword the changelog fragment.
This commit is contained in:
1 parent
15bda43a1e
commit
27823d7d05
4 files changed
+386
-46
No files matched your search
@@ -5774,6 +5774,7 @@ and unreachable upstream (502/504) separately from road geometry.
|
||||
- **Reliability**: tool-call dedupe (2.5s window across call/item/args keys); response-create queueing that respects active responses and defers follow-ups when the user starts speaking; per-tool follow-up instructions so the agent confirms only what actually happened (zoom confirms only on `ok=true`).
|
||||
- **Aircraft identity honesty:** “What is this aircraft?” reads callsign, operator, registration, type, and route only from the selected contact context. Missing operator, route, or type enrichment is named explicitly rather than silently omitted or inferred from the callsign.
|
||||
- **Diagnostics**: every client/server event is posted to `/api/realtime/debug-log` and appended to `.gev-logs/realtime-conversations.jsonl` (gitignored) with secret/image redaction; last 30 errors persist in `localStorage` (`gev-realtime-errors`); `window.__gevVoiceCommands.getDiagnostics()` in the console.
|
||||
- **Visual-context escalation** (OpenAI Realtime): when a `get_entity_context` answer reaches the image step (its tool output was sent and the data channel is open), it logs one `viewport_context.decision` (`contextId`, `send`, `kind` `viewport` or `crop`, `reason`, `viewScale`). A send decision is followed by exactly one outcome with the same `contextId` and `kind`: `viewport_context.result` (`sent`, `data-channel-send-failed`) or `viewport_context.skipped`. Pointer-crop skips are `pointer-not-fresh`, `pointer-not-ground`, `pointer-off-screen`, `pointer-already-requested` (the same snapshot was already sent or is still being captured), `pointer-expired`, `pointer-pose-unknown` and `pose-changed`; dropped captures are `capture-unavailable`, `session-reset`, `superseded` (a newer request or turn), `channel-lost` and `frame-stale`; a step that throws closes the pair with `error` before the caller logs `viewport_context.failed`. `channel-closed` and the crop's `pointer-scale-not-useful` are defensive and not expected from the current caller. Payloads are static codes only, never image data, tool payloads or user text, and other tool results log nothing here. Gemini Live uses a different gate in `geminiSession.js` (no pointer crop, a 1 s throttle), which is not instrumented.
|
||||
- **Counting semantics ("near")** — a CONTRACT; new count-bearing tool work inherits it. Three honest numbers exist for one question: the Contacts cohort (250 km around the subject, what the panel shows), `analyst_query`'s count of _currently-loaded_ records for the requested scope, and the layer-wide loaded total in `coverage.layersQueried[].total`. They diverge legitimately — the flights layer loads by viewport, so after a camera dive the loaded set can hold a fraction of the cohort (field case: panel 42, analyst 8). The contract:
|
||||
1. **Contacts ACTIVE** → "near / nearby / how many aircraft" means the **Contacts window** — the panel's numbers, spoken verbatim. Mechanism: `contactsWindow` (`{centeredOn, radiusKm, flights, military, vessels, complete}`), carried by both `analyst_query` and `get_current_view_state`, derived by `contactsWindowFromSnapshot()` from the same snapshot the panel renders so the two cannot drift. Voice filtering and follow-up memory use that immutable snapshot too. A cohort whose feed cannot answer reports `'unknown'`, never a confident zero; a cohort at the 20,000-row retention bound reports a lower bound.
|
||||
2. **Contacts OFF** → "nearby" means **in view**; "near \<place\>" means a radius around that place. A radius query with Contacts active and no explicit centre is centred on the **active contact**, not the camera.
|
||||
|
||||
Reference in new issue
Block a user