test(voice): pin diagnostics wiring and in-flight crop dedupe; tighten docs

Third review pass on #835. No runtime change.

- Controller test: viewport decisions reach the controller's debug log.
- Policy test: a second request for a snapshot still being captured is
  logged pointer-already-requested and is not captured twice.
- CURRENT-STATE: pointer-not-fresh also covers a missing snapshot,
  already-requested covers an earlier attempt that threw, and the image
  step runs when any tool output of the response was sent.
- Changelog: describe what is recorded rather than a fixed miscount.
This commit is contained in:
manjunath22466 committed 2026-11-06 23:12:20 +05:30
1 parent d0a612786a
commit df37c9360d
4 files changed
+55 -5

No files matched your search

@@ -1,7 +1,7 @@
- The voice debug log now says why OpenAI Realtime did or did not get a
viewport screenshot or pointer crop. When an entity-context answer reaches
the image step, the log records a reason-coded decision and, when an image
was due, one outcome under the same id, so a crop that was dropped because
the pointer went stale or the camera moved is no longer counted as sent.
What is captured, sent and kept is unchanged, and no image data or user text
is logged (kvnloo, manjunath22466, #835).
was due, one outcome under the same id: sent, or why it was dropped (a stale
pointer, a camera move, a newer turn). What is captured, sent and kept is
unchanged, and no image data or user text is logged (kvnloo, manjunath22466,
#835).
+1 -1
View File
@@ -5779,7 +5779,7 @@ and unreachable upstream (502/504) separately from road geometry.
- **Reliability**: tool-call dedupe (2.5s window across call/item/args keys); response-create queueing that respects active responses and defers follow-ups when the user starts speaking; per-tool follow-up instructions so the agent confirms only what actually happened (zoom confirms only on `ok=true`).
- **Aircraft identity honesty:** “What is this aircraft?” reads callsign, operator, registration, type, and route only from the selected contact context. Missing operator, route, or type enrichment is named explicitly rather than silently omitted or inferred from the callsign.
- **Diagnostics**: every client/server event is posted to `/api/realtime/debug-log` and appended to `.gev-logs/realtime-conversations.jsonl` (gitignored) with secret/image redaction; last 30 errors persist in `localStorage` (`gev-realtime-errors`); `window.__gevVoiceCommands.getDiagnostics()` in the console.
- **Visual-context escalation** (OpenAI Realtime): when a response's last tool result is a `get_entity_context` answer and its tool output was sent with the data channel open, it logs one `viewport_context.decision` with `contextId`, `send` and `reason`, plus `viewScale` (absent only for `channel-closed`) and, when `send` is true, `kind` (`viewport` or `crop`). Decision reasons: `structured-identity-sufficient` (structured context made an image unnecessary), `view-scale-not-local`, `structured-identity-missing` (sends a viewport), `pointer-ground-crop` (sends a crop), `pointer-identity-present`, `pointer-missing`, `pointer-scale-not-useful` and `channel-closed`. A send decision is followed by exactly one outcome with the same `contextId` and `kind`: `viewport_context.result` (`sent`, `data-channel-send-failed`), logged as soon as the send returns, or `viewport_context.skipped`. Pointer-crop skips are `pointer-not-fresh`, `pointer-not-ground`, `pointer-off-screen`, `pointer-already-requested` (the same snapshot was already sent, is still being captured, or an earlier capture of it failed), `pointer-expired`, `pointer-pose-unknown` and `pose-changed`; dropped captures are `capture-unavailable`, `session-reset`, `superseded` (a newer request or turn), `channel-lost` and `frame-stale`; a step that throws before an outcome closes the pair with `error`, and the caller then logs `viewport_context.failed`. The `channel-closed` decision and the `channel-closed` and `pointer-scale-not-useful` skip outcomes are defensive and not expected from the current caller. Decision and outcome are separate records, so the debug-log rate limit can drop one half of a pair. Payloads are static codes only, never image data, tool payloads or user text, and other tool results log nothing here. Gemini Live uses a different gate in `geminiSession.js` (no pointer crop, a 1 s throttle), which is not instrumented.
- **Visual-context escalation** (OpenAI Realtime): when a response's last tool result is a `get_entity_context` answer, a tool output from that response was sent and the data channel is open, it logs one `viewport_context.decision` with `contextId`, `send` and `reason`, plus `viewScale` (absent only for `channel-closed`) and, when `send` is true, `kind` (`viewport` or `crop`). Decision reasons: `structured-identity-sufficient` (structured context made an image unnecessary), `view-scale-not-local`, `structured-identity-missing` (sends a viewport), `pointer-ground-crop` (sends a crop), `pointer-identity-present`, `pointer-missing`, `pointer-scale-not-useful` and `channel-closed`. A send decision is followed by exactly one outcome with the same `contextId` and `kind`: `viewport_context.result` (`sent`, `data-channel-send-failed`), logged as soon as the send returns, or `viewport_context.skipped`. Pointer-crop skips are `pointer-not-fresh` (no snapshot, or a stale one), `pointer-not-ground`, `pointer-off-screen`, `pointer-already-requested` (the same snapshot was already sent, is still being captured, or an earlier attempt for it threw), `pointer-expired`, `pointer-pose-unknown` and `pose-changed`; dropped captures are `capture-unavailable`, `session-reset`, `superseded` (a newer request or turn), `channel-lost` and `frame-stale`; a step that throws before an outcome closes the pair with `error`, and the caller then logs `viewport_context.failed`. The `channel-closed` decision and the `channel-closed` and `pointer-scale-not-useful` skip outcomes are defensive and not expected from the current caller. Decision and outcome are separate records, so the debug-log rate limit can drop one half of a pair. Payloads are static codes only, never image data, tool payloads or user text, and other tool results log nothing here. Gemini Live uses a different gate in `geminiSession.js` (no pointer crop, a 1 s throttle), which is not instrumented.
- **Counting semantics ("near")** — a CONTRACT; new count-bearing tool work inherits it. Three honest numbers exist for one question: the Contacts cohort (250 km around the subject, what the panel shows), `analyst_query`'s count of _currently-loaded_ records for the requested scope, and the layer-wide loaded total in `coverage.layersQueried[].total`. They diverge legitimately — the flights layer loads by viewport, so after a camera dive the loaded set can hold a fraction of the cohort (field case: panel 42, analyst 8). The contract:
1. **Contacts ACTIVE** → "near / nearby / how many aircraft" means the **Contacts window** — the panel's numbers, spoken verbatim. Mechanism: `contactsWindow` (`{centeredOn, radiusKm, flights, military, vessels, complete}`), carried by both `analyst_query` and `get_current_view_state`, derived by `contactsWindowFromSnapshot()` from the same snapshot the panel renders so the two cannot drift. Voice filtering and follow-up memory use that immutable snapshot too. A cohort whose feed cannot answer reports `'unknown'`, never a confident zero; a cohort at the 20,000-row retention bound reports a lower bound.
2. **Contacts OFF** → "nearby" means **in view**; "near \<place\>" means a radius around that place. A radius query with Contacts active and no explicit centre is centred on the **active contact**, not the camera.
+26
View File
@@ -3708,3 +3708,29 @@ test('voice runs app actions itself and other tools through the catalog', async
assert.deepEqual(disposals, [['removed']]);
assert.equal(withToolCatalog(runner, undefined), runner);
});
test('the controller forwards viewport escalation diagnostics to its debug log', async () => {
const records = [];
const controller = new GevRealtimeController({
ui: createTestVoiceUi(),
runner: async () => ({ ok: true }),
radioLayer: { subscribePlaybackControls: () => () => {}, setVoiceDucked() {} },
});
controller.debugLog = (event, detail) => records.push({ event, detail });
controller.dc = { readyState: 'open', send() {}, close() {} };
assert.equal(
await controller.sendVisualContextIfUseful({
action: 'get_entity_context',
scene: { basemap: { viewScale: 'regional' } },
}),
false,
);
assert.deepEqual(
records.filter((record) => record.event.startsWith('viewport_context.')),
[{
event: 'viewport_context.decision',
detail: { contextId: 1, send: false, reason: 'view-scale-not-local', viewScale: 'regional' },
}],
);
});
+24
View File
@@ -557,3 +557,27 @@ test('a throw after a real send still logs it as sent', async () => {
[['viewport_context.result', 'sent']],
);
});
test('a second request for a snapshot still being captured is not sent twice', async () => {
const releases = [];
const snapshot = groundSnapshot();
const { viewport, events } = recordingViewport({
readPointer: () => snapshot,
capturePointer: () => new Promise((resolve) => releases.push(resolve)),
});
const first = viewport.sendVisualContextIfUseful(pointerGround());
const second = viewport.sendVisualContextIfUseful(pointerGround());
await Promise.resolve();
for (const release of releases)
release({ dataUrl: 'data:image/jpeg;base64,AA==', rect: {} });
assert.deepEqual([await first, await second], [true, false]);
assert.equal(releases.length, 1);
const outcomes = events.filter((item) => outcomeEvents.has(item.event));
assert.deepEqual(
outcomes.map((item) => [item.detail.contextId, item.detail.reason]),
[
[2, 'pointer-already-requested'],
[1, 'sent'],
],
);
});