From 7688e53424217ce2b807fea06f061b14898ce8a7 Mon Sep 17 00:00:00 2001 From: manjunath Date: Sun, 18 Oct 2026 16:15:06 +0530 Subject: [PATCH] fix(voice): skip replies for rejected typed sends --- docs/CURRENT-STATE.md | 2 +- src/voice/realtimeTurns.js | 2 +- src/voice/realtimeTypedSend.test.mjs | 268 +++++++++++++++++++++++++++ 3 files changed, 270 insertions(+), 2 deletions(-) create mode 100644 src/voice/realtimeTypedSend.test.mjs diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 8cb7b6d17..a1acaa7e5 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -4684,7 +4684,7 @@ and unreachable upstream (502/504) separately from road geometry. - **Every selectable layer writes that one slot** (2026-08-21). The tracking layers — flights, military, satellites — publish selection on their own awareness lane (`gev:awareness-subject-selected`, consumed by the readout card and the Contacts panel) and for the life of the voice tools never wrote the shared slot, so `scope:'selected'` silently answered `in_view` with a contact plainly selected on screen. They now call `selectTrackedSubjectContext` / `refreshTrackedSubjectContext` / `clearTrackedSubjectContext` (`src/data/contextStore.js`) on select / poll / deselect. **Invariants: (a)** that write path must NOT dispatch `gev:entity-selected` — tracking layers already own an event lane and a second one makes two surfaces fight over one subject; **(b)** exactly one record per tracking layer, because a frozen snapshot of a moved contact must never reach the visible-entity scan; **(c)** precedence is recency, not layer — one slot, last selection wins, so clicking an overlay entity supersedes a tracked plane as the voice subject while the plane stays tracked. Satellites refresh on the 1 s propagation beat, not per frame. - **Context-mode vocabulary is symmetric** (2026-08-21). `set_context_mode` accepts `contacts`; the internal id is `flights`. Every model-readable field (`mode`, `entering`, `priorMode`, nested `context`/`contextRollback`, and the transition diagnostic text) is reported in the accepted vocabulary, with the internal id preserved as `Internal`; an absent secondary mode stays `null` rather than claiming to be `off`. Mapping lives in `src/contextModePolicy.js` so UI text and voice payloads cannot drift. Reporting the internal id made the model read `mode:'flights'` as "Contacts is off" and refuse to answer from the `contactsWindow` counts in the same payload. - **Analyst → track handoff carries a key, not just a label** (2026-08-21). `analyst_query` items include `icao24`/`mmsi` alongside the display `id`, and contact lookup uses the shared tiered ranking in `src/data/contactMatch.js`: hex exact → callsign exact → registration exact → callsign prefix → registration prefix → callsign substring → registration substring. The tiers keep an exact callsign ahead of a colliding registration regardless of feed order, and registrations compare separator-insensitively (`G-ABCD`/`GABCD`, `05-8152`/`058152`, `N123AB`/`N-123AB`). - - **A typed command supersedes the turn it interrupts** (2026-08-21). `sendTextCommand` defers its `response.create` behind an active response instead of colliding with it, marks that response superseded so a late function call from it is refused rather than dispatched, and drops the old turn's queued follow-up. A refused call is still ANSWERED — a terminal `{ok:false, superseded:true}` `function_call_output` — because an unanswered `function_call` strands a pending call and deadlocks the model; the refusal creates no response of its own. A burst of typed commands coalesces into one response while keeping both conversation items. + - **A typed command supersedes the turn it interrupts** (2026-08-21; hardened 2026-10-17). `sendTextCommand` defers its `response.create` behind an active response instead of colliding with it, marks that response superseded so a late function call from it is refused rather than dispatched, and drops the old turn's queued follow-up. A refused call is still ANSWERED — a terminal `{ok:false, superseded:true}` `function_call_output` — because an unanswered `function_call` strands a pending call and deadlocks the model; the refusal creates no response of its own. A burst of typed commands coalesces into one response while keeping both conversation items. If the local channel rejects the user item, that command requests no response and adds no deferred answer; earlier accepted deferred input remains queued. Interruption and pointer preparation still happen before the send attempt. - **Subject reconciliation across satellite catalog rebuilds** (2026-08-21). A dense↔core toggle or TLE refresh clears and repopulates the catalog, so the published subject is re-resolved against the new satrec; a subject that did NOT survive releases the slot. The per-frame refresh cannot do this itself — `_getTrackedFramePosition` returns early once the satellite has no catalog entry — so the reconcile runs at rebuild completion. An empty catalog is a rebuild in flight, not a disappearance. - **One aircraft-proximity snapshot** (2026-08-22; hardened 2026-09-30). `collectAircraftProximityWindow` (`src/data/militaryAwareness.js`) computes the Contacts panel cohort once. The voice analyst reads the exact retained snapshot and evaluation timestamp instead of re-scanning live layers, so the panel readout and spoken count for one centre cannot cross snapshots. **Invariant: do not re-derive a proximity count anywhere else.** Each layer retains 20,000 rows and samples one row beyond the cap; a saturated cohort reports `complete:false`, `truncated:true`, and is spoken as a lower bound. They diverged before because the panel read live billboard positions while the analyst used last-fix coordinates over a 2,000-record slice — 111 on screen, 15 spoken. Explicit regions, arbitrary points, and a radius different from the panel radius deliberately keep the general record engine. Entity-centred results carry `window: {engine:'contacts-window', centeredOn, radiusKm, flights, military, aircraft, complete, evaluatedAt}`. - **Centre precedence for nearby asks**: explicit place in the question > Contacts subject (a selected non-contact entity never silently becomes the centre) > an entity the user names > the current view, said aloud. Contacts active with no subject uses the view rather than reading an empty panel. diff --git a/src/voice/realtimeTurns.js b/src/voice/realtimeTurns.js index 6905cc17e..cbc8d5dbc 100644 --- a/src/voice/realtimeTurns.js +++ b/src/voice/realtimeTurns.js @@ -137,7 +137,7 @@ export class RealtimeTurns { content: [{ type: 'input_text', text: cleanText }], }, }; - this.sendRealtimeEvent(itemEvent, 'client.user_text'); + if (!this.sendRealtimeEvent(itemEvent, 'client.user_text')) return; this.requestUserTextResponse(); } diff --git a/src/voice/realtimeTypedSend.test.mjs b/src/voice/realtimeTypedSend.test.mjs new file mode 100644 index 000000000..6a32172c0 --- /dev/null +++ b/src/voice/realtimeTypedSend.test.mjs @@ -0,0 +1,268 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { GevRealtimeController } from './realtimeController.js'; + +function fixture({ pointer = null, onUserSend = () => {} } = {}) { + const sent = []; + const executed = []; + const rejectedTexts = new Set(); + let rejectResponse = false; + const controller = new GevRealtimeController({ + runner: async (name) => { + executed.push(name); + return { ok: true }; + }, + backend: {}, + debugSink: null, + pointer, + ui: { + root: { + dataset: {}, + classList: { remove() {} }, + querySelectorAll: () => [], + }, + status: {}, + detail: {}, + }, + }); + controller.dc = { + readyState: 'open', + send(body) { + const value = JSON.parse(body); + const text = value.item?.role === 'user' && value.item.content?.[0]?.text; + if (value.item?.role === 'user') onUserSend(); + if ( + rejectedTexts.has(text) || + (rejectResponse && value.type === 'response.create') + ) + throw new Error('Synthetic channel rejection'); + sent.push(value); + }, + close() { + this.readyState = 'closed'; + }, + }; + controller.status = 'listening'; + controller._turns.narration = null; + const event = (payload) => + controller.handleRealtimeEvent({ data: JSON.stringify(payload) }); + const created = (id) => event({ type: 'response.created', response: { id } }); + const done = (id) => + event({ + type: 'response.done', + response: { id, status: 'completed', output: [] }, + }); + const creates = () => + sent.filter( + (x) => + x.type === 'response.create' && + x.response?.metadata?.gev !== 'progress', + ); + const texts = () => + sent + .filter((x) => x.item?.role === 'user') + .map((x) => x.item.content[0].text); + return { + controller, + sent, + executed, + rejectedTexts, + event, + created, + done, + creates, + texts, + rejectResponses(value) { + rejectResponse = value; + }, + }; +} + +test('rejected idle text schedules no answer, and a successful successor answers once', async () => { + const f = fixture(); + try { + f.rejectedTexts.add('rejected'); + assert.equal(f.controller.sendTextCommand('rejected'), undefined); + assert.equal(f.creates().length, 0); + assert.equal(f.controller.pendingUserTextResponse, false); + assert.equal(f.controller.responseCreatePending, false); + assert.equal(f.controller.sendTextCommand('successor'), undefined); + assert.deepEqual(f.texts(), ['successor']); + assert.equal(f.creates().length, 1); + await f.created('successor'); + await f.done('successor'); + assert.equal(f.creates().length, 1); + } finally { + f.controller.stop(); + } +}); + +test('rejected text behind an active response cannot queue an answer', async () => { + const f = fixture(); + try { + await f.created('active'); + f.controller.queueResponseCreate('Old tool follow-up'); + assert.equal( + f.controller.pendingResponseInstructions, + 'Old tool follow-up', + ); + const runningTool = new AbortController(); + f.controller._turns.activeToolAbortControllers.add(runningTool); + f.rejectedTexts.add('rejected'); + f.controller.sendTextCommand('rejected'); + assert.equal(f.controller.pendingUserTextResponse, false); + assert.equal(f.controller.pendingResponseInstructions, null); + assert.equal(f.controller._turns.isSupersededResponse('active'), true); + assert.equal(runningTool.signal.aborted, true); + const staleCall = { + type: 'response.function_call_arguments.done', + response_id: 'active', + call_id: 'stale-call', + name: 'fly_to_location', + arguments: '{"query":"Paris"}', + }; + await f.event(staleCall); + await f.event(staleCall); + assert.deepEqual(f.executed, []); + const outputs = f.sent.filter( + (x) => x.item?.type === 'function_call_output', + ); + assert.equal(outputs.length, 1); + assert.equal(outputs[0].item.call_id, 'stale-call'); + assert.equal(JSON.parse(outputs[0].item.output).superseded, true); + await f.done('active'); + assert.equal(f.creates().length, 0); + } finally { + f.controller.stop(); + } +}); + +test('rejected text still prepares narration and pointer context before sending', () => { + const steps = []; + const f = fixture({ + pointer: { + beginTurn() { + steps.push('pointer'); + return null; + }, + clear() {}, + }, + onUserSend() { + steps.push('send'); + }, + }); + f.controller._turns.narration = { + cancel() { + steps.push('cancel'); + }, + userTurnEnded() { + steps.push('user-turn-ended'); + }, + }; + try { + f.rejectedTexts.add('rejected'); + f.controller.sendTextCommand('rejected'); + assert.deepEqual(steps, ['cancel', 'user-turn-ended', 'pointer', 'send']); + assert.equal(f.creates().length, 0); + } finally { + f.controller.stop(); + } +}); + +test('earlier accepted deferred input survives a newer failed send and answers once', async () => { + const f = fixture(); + try { + await f.created('active'); + f.controller.sendTextCommand('accepted'); + f.rejectedTexts.add('rejected'); + f.controller.sendTextCommand('rejected'); + assert.deepEqual(f.texts(), ['accepted']); + assert.equal(f.controller.pendingUserTextResponse, true); + assert.equal(f.creates().length, 0); + await f.done('active'); + assert.equal(f.creates().length, 1); + await f.done('active'); + assert.equal(f.creates().length, 1); + } finally { + f.controller.stop(); + } +}); + +test('rejected text preserves an outstanding create without adding deferred input', async () => { + const f = fixture(); + try { + f.controller.sendTextCommand('accepted'); + f.rejectedTexts.add('rejected'); + f.controller.sendTextCommand('rejected'); + assert.equal(f.controller.responseCreatePending, true); + assert.equal(f.controller.pendingUserTextResponse, false); + await f.created('accepted'); + await f.done('accepted'); + assert.equal(f.creates().length, 1); + } finally { + f.controller.stop(); + } +}); + +for (const acceptedFirst of [false, true]) { + test(`progress completion answers only accepted deferred input (accepted first=${acceptedFirst})`, async () => { + const f = fixture(); + try { + assert.equal( + f.controller._turns.speakProgressLine('Checking the map.'), + true, + ); + await f.event({ + type: 'response.created', + response: { id: 'progress', metadata: { gev: 'progress' } }, + }); + if (acceptedFirst) f.controller.sendTextCommand('accepted'); + f.rejectedTexts.add('rejected'); + f.controller.sendTextCommand('rejected'); + assert.equal(f.controller.pendingUserTextResponse, acceptedFirst); + await f.event({ + type: 'response.done', + response: { + id: 'progress', + status: 'cancelled', + metadata: { gev: 'progress' }, + }, + }); + assert.equal(f.creates().length, acceptedFirst ? 1 : 0); + } finally { + f.controller.stop(); + } + }); +} + +test('accepted text with a rejected response.create releases the pending latch for a successor', async () => { + const f = fixture(); + try { + f.rejectResponses(true); + f.controller.sendTextCommand('accepted without response'); + assert.equal(f.controller.responseCreatePending, false); + assert.equal(f.creates().length, 0); + f.rejectResponses(false); + f.controller.sendTextCommand('successor'); + assert.deepEqual(f.texts(), ['accepted without response', 'successor']); + assert.equal(f.creates().length, 1); + } finally { + f.controller.stop(); + } +}); + +test('successful deferred typed burst still retains both items and coalesces one answer', async () => { + const f = fixture(); + try { + await f.created('active'); + f.controller.sendTextCommand('first'); + f.controller.sendTextCommand('second'); + assert.deepEqual(f.texts(), ['first', 'second']); + assert.equal(f.creates().length, 0); + await f.done('active'); + await f.done('active'); + assert.equal(f.creates().length, 1); + } finally { + f.controller.stop(); + } +});