A voice or chat selection shows its card in the Context panel (#1890)

* fix(voice): a selection steps a docked assistant panel down so Context shows; the panel keeps the last reply

A voice or chat action that selects an entity (select_nearest_aircraft,
track_entity) now minimizes a voice or chat panel docked in the right rail's
lane, so the rail and its Context panel show as after a click. The voice
card inside the panel no longer hides 14 s after its last update: the reply
stays until the next turn, Stop or a dismissal. The floating card keeps its
idle deadline.

* fix(voice): a voice or chat selection opens its card in the Context panel

A successful select_nearest_aircraft or track_entity now selects the entity
in the Context panel's entity card store the same way a click does, so the
panel opens on its details; the docked assistant panel then steps down so it
shows. track_entity reports the id it tracked. Nothing opens in Cockpit,
where Context is hidden.

* fix(voice): a slow selection never replaces a card picked since; Cockpit is read from its own state

The entity card store counts selections and clears. A voice or chat
selection notes that count when it is asked and opens its card only if no
other card was picked or cleared before it finished. Whether Cockpit is
running is read from the style manager's Cockpit state when it has one.

* Add the changelog entry

* fix(voice): the app hands voice its Context card opener

The voice package no longer imports the entity card store. The app builds
the opener (createContextCardOpener) and passes it to the voice commands as
contextCard, which keeps the voice package inside its own boundary.
This commit is contained in:
Sameh Khamis authored and GitHub committed 2026-11-20 22:19:16 -08:00
1 parent fbbe6cd91f
commit 918435ddc7
10 files changed
+419 -14

No files matched your search

@@ -0,0 +1,3 @@
- A voice or chat selection opens its card in the Context panel, as a click
does, and a docked assistant panel steps down so the card shows; the reply
stays until the next one (samehkhamis, #1890).
+7 -1
View File
@@ -1627,7 +1627,7 @@ warning, capped and unknown cost states also appear in the dock heading when
Settings is closed. The existing hard cost cap still stops the session and
announces the reason.
Automatic Voice Preview uses output from the active mounted provider, including late replies that may answer an older request. Manual panel choices remain until genuine local input rearms automatic presentation. Preview stays off when disabled, remains hidden in Cockpit, and retires on Stop or provider replacement. Display events do not grant command authority.
Automatic Voice Preview uses output from the active mounted provider, including late replies that may answer an older request. Manual panel choices remain until genuine local input rearms automatic presentation. Preview stays off when disabled, remains hidden in Cockpit, and retires on Stop or provider replacement. In the panel, the last reply stays until the next turn replaces it; only the voice card floating on a control without the panel hides 14 s after its last update. Display events do not grant command authority.
Typed chat sits beside voice in the dock. Chat replies are silent and never
request microphone access. The first message connects with the selected
@@ -1657,6 +1657,12 @@ and the chat shows a matching message.
Chat and the voice panel sit above the dock or dock into the right rail's lane,
which hides the rail until the panel minimizes or closes, or a rail panel opens.
A voice or chat action that selects something (`select_nearest_aircraft`,
`track_entity`) opens its card in the Context panel as a click on it does
(`createContextCardOpener` in `src/ui/stores/entityCard.js`, handed to voice by the app), except in
Cockpit (as Cockpit's own state reports it) or when another card was picked
while the action ran, which then stays, and steps a panel docked in the lane down to its bar so the rail
shows it; a panel above the dock leaves the rail in view and stays as it is.
Minimized panels always sit as a bar centred above the dock; restoring returns a
docked panel to the lane. A centred panel steps down to its bar when the pointer
reaches the Location or Visual Presets button and rests above an open tray,
+3
View File
@@ -17,6 +17,7 @@ import {
import { initGeoFileImport } from '../ui/geoFileImport.js';
import { useDisplayStore } from '../ui/stores/display.js';
import { applicationPinia } from '../ui/vueRuntime.js';
import { createContextCardOpener } from '../ui/stores/entityCard.js';
import { initImageryBoxTool } from '../ui/imageryBoxTool.js';
import { createRecentImageryPanel } from '../ui/recentImagery.js';
import { createDataLibrary } from '../library/controller.js';
@@ -349,6 +350,8 @@ export function createApplicationTools({
dataManager,
sceneDirector,
annotations,
// A voice or chat selection opens its card in Context, as a click does.
contextCard: createContextCardOpener(applicationPinia()),
});
defer(() => {
voiceCommands.stop({ removeUi: true });
+40
View File
@@ -357,6 +357,13 @@ export const useEntityCardStore = defineStore('gev-entity-card', () => {
* @param {boolean} [options.fly] Also fly the camera to it.
* @returns {boolean} Whether a card could be made.
*/
/**
* Counts card selections and clears, so a selection that was asked for
* before a newer one can tell it came too late.
*/
let generation = 0;
const selectionGeneration = () => generation;
/** Things that move are followed as soon as they are picked. */
const AUTO_FOLLOW = new Set(['flights', 'military', 'satellites']);
/**
@@ -374,6 +381,7 @@ export const useEntityCardStore = defineStore('gev-entity-card', () => {
*/
function select(identity, { fly = false, follow: autoFollow = true } = {}) {
if (!read(identity)) return false;
generation++;
stopLocating();
away.value = false;
reference =
@@ -403,6 +411,7 @@ export const useEntityCardStore = defineStore('gev-entity-card', () => {
}
function clear() {
generation++;
if (following.value) shell.value?.letGo?.(card.value);
following.value = null;
asked = null;
@@ -451,6 +460,7 @@ export const useEntityCardStore = defineStore('gev-entity-card', () => {
locate,
connect,
select,
selectionGeneration,
clear,
refresh,
flyTo,
@@ -477,3 +487,33 @@ export const useEntityCardStore = defineStore('gev-entity-card', () => {
askAboutView,
};
});
/**
* Opens what a voice or chat action selected in the Context panel, as a click
* on it does (`src/ui/entityContextBridge.js` selects the same way): its card
* opens, and the panel with it. The app hands it to voice, which reads
* `generation()` when a selection is asked and passes it back as `since`.
* The action has already selected or followed the item in its own layer, so
* the card does not follow it again (a click on a flight does not either). A
* card picked or cleared since the action was asked stays, and nothing opens
* in Cockpit, where Context is hidden.
* @param {import('pinia').Pinia} pinia
* @returns {{generation: () => number, show: (identity: {layerId: string, id: string|number, label?: string, lat?: number, lon?: number}, options?: {since?: number|null, cockpitActive?: boolean}) => boolean}}
*/
export function createContextCardOpener(pinia) {
const store = () => useEntityCardStore(pinia);
return {
generation: () => store().selectionGeneration(),
show(identity, { since = null, cockpitActive = false } = {}) {
if (!identity?.layerId || identity.id == null || identity.id === '')
return false;
if (cockpitActive) return false;
if (since !== null && store().selectionGeneration() !== since)
return false;
return store().select(
{ ...identity, id: String(identity.id) },
{ follow: false },
);
},
};
}
+10 -4
View File
@@ -3051,8 +3051,15 @@ async function trackEntity(
'track_entity',
() => {
let trackedOk = false;
// The id the layer selects by, which also names it in Context.
const id =
family.kind === 'vessel'
? (found.id ?? found.mmsi)
: family.kind === 'satellite'
? found.noradId
: found.icao24;
if (family.kind === 'vessel') {
trackedOk = !!module.selectById?.(found.id ?? found.mmsi);
trackedOk = !!module.selectById?.(id);
flyToLandmark(viewer, found.latitude, found.longitude, {
range: 6000,
pitch: -45,
@@ -3060,10 +3067,8 @@ async function trackEntity(
buildingHeight: 0,
duration: 2.0,
});
} else if (family.kind === 'satellite') {
trackedOk = !!module.trackById?.(found.noradId, { origin: 'voice' });
} else {
trackedOk = !!module.trackById?.(found.icao24, { origin: 'voice' });
trackedOk = !!module.trackById?.(id, { origin: 'voice' });
}
return {
@@ -3071,6 +3076,7 @@ async function trackEntity(
action: 'track_entity',
layerId: family.layerId,
kind: family.kind,
id: id == null ? null : String(id),
// Aircraft follow the flight layers' label convention (callsign →
// registration → icao24) so the spoken name matches what the UI shows;
// `registration` is absent on vessels/satellites and simply falls
+3
View File
@@ -144,6 +144,9 @@ test('track_entity runner narrates a callsign-less aircraft by its registration'
`${layerId} narration must speak the registration, not the ICAO hex`,
);
assert.equal(trackedId, 'ae1fa4', `${layerId} must still TRACK by icao24`);
// The result names what it tracked, so Context can show it.
assert.equal(result.id, 'ae1fa4');
assert.equal(result.layerId, layerId);
}
});
+231 -3
View File
@@ -5,6 +5,11 @@ import { readFileSync } from 'node:fs';
import { createPinia } from 'pinia';
import { createVoiceControl } from '../control.js';
import { useVoiceStore } from '../../ui/stores/voice.js';
import {
createContextCardOpener,
useEntityCardStore,
} from '../../ui/stores/entityCard.js';
import { applicationPinia } from '../../ui/vueRuntime.js';
import { createGevActionRunner } from '../gevActions.js';
import { createVoiceCommands as bind } from '../sessionCommands.js';
import { createVoiceCommands as compose } from '../commands.js';
@@ -138,6 +143,8 @@ function mounted(t, options = {}) {
createSession: factory(instances, options),
createControl: (control) => createTestVoiceUi({ ...control, pinia }),
panelHost: host,
styleManager: options.styleManager ?? null,
contextCard: createContextCardOpener(applicationPinia()),
nativePanelProvider: options.native
? (options.nativeProvider ?? 'gemini')
: null,
@@ -779,7 +786,7 @@ test('voice timeout change preserves a pending voice reveal', async (t) => {
assert.equal(f.commands.getVoicePreferences().minutes, 0);
});
test('active provider output reveals Preview after done; manual choice wins until genuine input, replays do not reset idle', async (t) => {
test('active provider output reveals Preview after done; manual choice wins until genuine input, replays change nothing', async (t) => {
const f = mounted(t, { native: true, nativeProvider: 'openai', view: true });
await f.commands.session.start();
const emit = f.instances[0].hooks.emit;
@@ -795,10 +802,12 @@ test('active provider output reveals Preview after done; manual choice wins unti
emit({ type: 'completion', responseId: 'r1' });
reply('Late answer');
assert.equal(f.commands.voicePanel.snapshot().panelOpen, true);
const timer = f.commands.voiceCard.hideTimer;
const shown = f.commands.voiceCard.state;
reply('Late answer');
reply('Late', 'r1', { final: false });
assert.equal(f.commands.voiceCard.hideTimer, timer);
assert.equal(f.commands.voiceCard.state, shown);
// The panel's card keeps its reply; it has no idle deadline.
assert.equal(f.commands.voiceCard.hideTimer, null);
assert.equal(f.commands.voiceCard.state.assistant, 'Late answer');
f.commands.voicePanel.close();
emit({
@@ -1444,3 +1453,222 @@ test('teardown inside sendText prevents appending the submitted message', async
0,
);
});
// What the selection actions report, as the real runner does.
const selectionRunner = async (name) =>
name === 'select_nearest_aircraft'
? {
ok: true,
action: name,
layerId: 'flights',
label: 'UAL12',
aircraft: { id: 'a1b2c3', callsign: 'UAL12' },
}
: name === 'track_entity'
? {
ok: true,
action: name,
layerId: 'military',
kind: 'aircraft',
id: 'ae01ce',
label: 'RCH401',
latitude: 42.3,
longitude: -71,
}
: { ok: true };
// The Context panel's entity card, cleared for each test.
function contextCard(t) {
const store = useEntityCardStore(applicationPinia());
store.clear();
t.after(() => store.clear());
return store;
}
// Docks the assistant panels into the right rail's lane for one test.
async function dockIntoRail(t, f) {
t.after(() => globalThis.localStorage.removeItem('gev.panel.docked'));
f.ui.emit('layout-dock');
await f.settle();
await f.settle();
}
test('a voice selection opens its Context card, steps the docked voice panel down and keeps the reply', async (t) => {
const context = contextCard(t);
const f = mounted(t, { view: true, runner: selectionRunner });
await f.settle();
await f.commands.session.start();
f.commands.voicePanel.select('preview');
await f.settle();
await dockIntoRail(t, f);
const panel = $('#gev-voice-panel');
assert.equal(panel.dataset.placement, 'docked');
assert.equal(globalThis.document.body.dataset.assistantDocked, 'true');
const { emit, runAction } = f.instances[0].hooks;
// A failed selection selects nothing, so the panel stays.
emit({
type: 'action-result',
name: 'track_entity',
result: { ok: false, error: 'Nothing matched' },
});
await f.settle();
assert.equal(panel.dataset.minimized, undefined);
assert.equal(context.card, null);
await runAction('select_nearest_aircraft', { location: 'Boston' });
// Context shows the aircraft, as after a click on it.
assert.equal(context.card.layerId, 'flights');
assert.equal(context.card.key, 'flights:a1b2c3');
emit({
type: 'transcript',
role: 'assistant',
text: 'The nearest aircraft is UAL12.',
final: true,
responseId: 'r1',
});
await f.settle();
assert.equal(panel.dataset.minimized, 'true');
// The rail, and its Context panel, show again.
assert.equal(globalThis.document.body.dataset.assistantDocked, undefined);
assert.equal(
$('#gev-voice-mini-assistant').textContent,
'The nearest aircraft is UAL12.',
);
// The reply is not retired on a timer: expanding the bar shows it.
assert.equal(f.commands.voiceCard.hideTimer, null);
f.commands.voicePanel.select('preview');
await f.settle();
assert.equal(panel.dataset.minimized, undefined);
assert.equal(f.commands.voicePanel.snapshot().previewVisible, true);
assert.equal($('#gev-voice-preview-empty').hidden, true);
assert.equal(
f.commands.voiceCard.snapshot().assistant.text,
'The nearest aircraft is UAL12.',
);
// The next turn replaces it.
emit({ type: 'interruption', reason: 'user-speech' });
assert.equal(f.commands.voiceCard.state.assistant, null);
});
test('a voice selection leaves a voice panel above the dock as it is', async (t) => {
const context = contextCard(t);
const f = mounted(t, { view: true, runner: selectionRunner });
await f.settle();
await f.commands.session.start();
f.commands.voicePanel.select('preview');
await f.settle();
assert.equal($('#gev-voice-panel').dataset.placement, 'compact');
await f.instances[0].hooks.runAction('track_entity', { query: 'RCH401' });
await f.settle();
assert.equal(context.card.key, 'military:ae01ce');
assert.equal($('#gev-voice-panel').dataset.minimized, undefined);
});
test('a voice selection in Cockpit leaves the hidden Context card alone', async (t) => {
const context = contextCard(t);
const f = mounted(t, { view: true, runner: selectionRunner });
await f.settle();
await f.commands.session.start();
globalThis.document.body.classList.add('cockpit-mode');
t.after(() => globalThis.document.body.classList.remove('cockpit-mode'));
await f.instances[0].hooks.runAction('track_entity', { query: 'RCH401' });
assert.equal(context.card, null);
});
test('a typed chat selection opens its Context card and steps the docked chat panel down', async (t) => {
const context = contextCard(t);
const f = mounted(t, { view: true, runner: selectionRunner });
typedChat(f);
await f.settle();
$('#gev-chat-button').click();
await f.settle();
await dockIntoRail(t, f);
assert.equal($('#gev-chat-panel').dataset.placement, 'docked');
await f.commands.session.start({ textOnly: true });
await f.instances[0].hooks.runAction('track_entity', { query: 'RCH401' });
await f.settle();
assert.equal(context.card.key, 'military:ae01ce');
assert.equal($('#gev-chat-panel').dataset.minimized, 'true');
assert.equal(f.ui.state.chat.open, true);
assert.equal(globalThis.document.body.dataset.assistantDocked, undefined);
});
test('the voice panel keeps the last reply until the next turn starts', async (t) => {
const f = mounted(t, { view: true });
await f.settle();
await f.commands.session.start();
f.commands.voicePanel.select('preview');
const { emit } = f.instances[0].hooks;
emit({
type: 'transcript',
role: 'assistant',
text: 'Tracking UAL12.',
final: true,
responseId: 'r1',
});
await f.settle();
// Only the card floating on a control without the panel hides when idle.
assert.equal(f.commands.voiceCard.hideTimer, null);
assert.equal(f.commands.voicePanel.snapshot().previewVisible, true);
assert.equal($('#gev-voice-preview-empty').hidden, true);
emit({ type: 'interruption', reason: 'user-speech' });
emit({
type: 'transcript',
role: 'assistant',
text: 'Now following DAL4.',
final: true,
responseId: 'r2',
});
assert.equal(f.commands.voiceCard.state.assistant, 'Now following DAL4.');
});
test('a voice selection in Cockpit, as Cockpit reports it, leaves the Context card alone', async (t) => {
const context = contextCard(t);
let cockpit = true;
const f = mounted(t, {
view: true,
runner: selectionRunner,
styleManager: { getCockpitState: () => ({ active: cockpit }) },
});
await f.settle();
await f.commands.session.start();
// Cockpit's own state decides, not the page's classes.
assert.equal(
globalThis.document.body.classList.contains('cockpit-mode'),
false,
);
await f.instances[0].hooks.runAction('track_entity', { query: 'RCH401' });
assert.equal(context.card, null);
cockpit = false;
await f.instances[0].hooks.runAction('track_entity', { query: 'RCH401' });
assert.equal(context.card.key, 'military:ae01ce');
});
test('a slow voice selection never replaces a card clicked since it was asked', async (t) => {
const context = contextCard(t);
let finish;
const f = mounted(t, {
view: true,
runner: (name) =>
name === 'track_entity'
? new Promise((resolve) => {
finish = () => resolve(selectionRunner(name));
})
: selectionRunner(name),
});
await f.settle();
await f.commands.session.start();
const pending = f.instances[0].hooks.runAction('track_entity', {
query: 'RCH401',
});
await Promise.resolve();
// The person clicks another aircraft while the lookup runs.
context.select({ layerId: 'flights', id: 'click1', label: 'CLICKED' });
const clicked = context.card.key;
finish();
await pending;
assert.equal(context.card.key, clicked);
// A selection with no newer pick still opens its card.
await f.instances[0].hooks.runAction('select_nearest_aircraft', {
location: 'Boston',
});
assert.equal(context.card.key, 'flights:a1b2c3');
});
+84 -2
View File
@@ -11,6 +11,31 @@ import { PointerReticle } from './pointerReticle.js';
import { VOICE_PROVIDERS } from './providerCapabilities.js';
import { t, translateKnown } from '../i18n/index.js';
/**
* Actions whose success selects an entity on the map, and the identity each
* result names it by.
*/
const SELECTION_ACTIONS = new Map([
[
'select_nearest_aircraft',
(result) => ({
layerId: result.layerId,
id: result.aircraft?.id,
label: result.label || result.aircraft?.callsign,
}),
],
[
'track_entity',
(result) => ({
layerId: result.layerId,
id: result.id,
label: result.label,
lat: result.latitude,
lon: result.longitude,
}),
],
]);
function createDefaultReticle(pointer) {
const element = pointer?.element;
if (!element?.ownerDocument) return null;
@@ -38,6 +63,12 @@ export function createVoiceCommands({
onSessionEvent,
canStartVoice,
createCard = (options) => new VoiceCardControls(options),
// Cockpit's own state; Context stays hidden while it runs.
styleManager = null,
// Opens a selection's card in the Context panel, as a click does
// (`createContextCardOpener` in src/ui/stores/entityCard.js); the app
// hands it in. Without one, selections open no card.
contextCard = null,
pointer = null,
referents = null,
createReticle = createDefaultReticle,
@@ -147,10 +178,14 @@ export function createVoiceCommands({
});
});
// Captions, plan and result card nested in the control; it only reads
// provider-neutral session events.
// provider-neutral session events. In the voice panel the card is the
// Preview's content, so the last reply stays until the next turn replaces
// it; only the card floating on the control hides when idle.
const card = createCard({
ui,
...(panelHost ? { onRender: () => panelMount?.panel.sync() } : {}),
...(panelHost
? { onRender: () => panelMount?.panel.sync(), idleMs: null }
: {}),
});
panelMount = panelHost?.mount({
ui,
@@ -279,6 +314,53 @@ export function createVoiceCommands({
}),
]
: [];
// A voice or chat selection shows in the right rail's Context panel, as a
// click's does: its card opens there, and the assistant panel docked over
// the rail steps down to its bar, where the latest reply stays readable. A
// panel above the dock leaves the rail in view, so it stays as it is.
const dockedOverRail = (name) => {
const panel = ui.element(name);
return Boolean(
panel &&
!panel.hidden &&
panel.dataset?.placement === 'docked' &&
!panel.dataset.minimized,
);
};
// The Context card generation each selection was asked at, by call, so a
// slow selection never replaces a card picked since.
const askedAt = new Map();
const callKey = (event) => event.callId ?? event.name;
const cockpitActive = () =>
typeof styleManager?.getCockpitState === 'function'
? styleManager.getCockpitState()?.active === true
: globalThis.document?.body?.classList?.contains('cockpit-mode') === true;
layouts.push(
session.subscribe((event) => {
if (!SELECTION_ACTIONS.has(event.name)) return;
if (event.type === 'action-call') {
askedAt.set(callKey(event), contextCard?.generation?.() ?? null);
return;
}
if (event.type !== 'action-result') return;
const since = askedAt.get(callKey(event)) ?? null;
askedAt.delete(callKey(event));
if (event.result?.ok !== true) return;
contextCard?.show?.(SELECTION_ACTIONS.get(event.name)(event.result), {
since,
cockpitActive: cockpitActive(),
});
if (!panelMount) return;
if (adapter.getStartOptions?.().textOnly) {
if (ui.state.chat.open && dockedOverRail('chatPanel'))
chat?.minimize({ focus: false });
} else if (
panelMount.panel.snapshot().panelOpen &&
dockedOverRail('voicePanel')
)
minimizeVoicePanel();
}),
);
const unbindMiniPreview = panelMount
? bindVoiceMiniPreview({ ui, session })
: null;
+3 -1
View File
@@ -20,7 +20,8 @@ export class VoiceCardControls {
/**
* @param {object} options
* @param {object} options.ui The voice control presenter.
* @param {number} [options.idleMs]
* @param {number|null} [options.idleMs] How long an idle card stays shown;
* null keeps it until the next turn, Stop or a dismissal.
* @param {Function} [options.setTimer]
* @param {Function} [options.clearTimer]
*/
@@ -103,6 +104,7 @@ export class VoiceCardControls {
scheduleIdle() {
this.cancelIdle();
if (this.idleMs === null) return;
this.hideTimer = this.setTimer(() => {
this.hideTimer = null;
if (this.destroyed) return;
+35 -3
View File
@@ -168,8 +168,7 @@ test('markup: the card nests inside the voice control with every element and a p
'the card is part of the voice control',
);
assert.equal(document.querySelectorAll('#gev-voice-card').length, 1);
if (panel)
assert.equal(card.parentElement.id, 'gev-voice-preview-panel');
if (panel) assert.equal(card.parentElement.id, 'gev-voice-preview-panel');
for (const id of [
'gev-voice-card-phase',
'gev-voice-card-dismiss',
@@ -531,7 +530,12 @@ test('additive presentation gates retain the current localized reducer labels',
const steps = card.state.steps;
const result = card.state.result;
const who = () => ui.state.card.user.label;
card.handle({ type: 'transcript', role: 'user', text: 'Same words', attribution: 'unverified' });
card.handle({
type: 'transcript',
role: 'user',
text: 'Same words',
attribution: 'unverified',
});
assert.equal(card.state.userAttribution, 'unverified');
assert.equal(card.snapshot().user.label, 'Habla reciente');
assert.equal(who(), 'Habla reciente');
@@ -615,3 +619,31 @@ test('manual panel and preference compose with the actual card without changing
panel.destroy();
card.destroy();
});
test('a card without an idle deadline keeps its reply until the next turn', () => {
const ui = createTestVoiceUi();
let timers = 0;
const card = new VoiceCardControls({
ui,
idleMs: null,
setTimer() {
timers++;
return timers;
},
clearTimer() {},
});
card.handle({ type: 'interruption' });
card.handle({
type: 'transcript',
role: 'assistant',
text: 'Tracking UAL12.',
final: true,
responseId: 'r1',
});
assert.equal(timers, 0);
assert.equal(card.hideTimer, null);
assert.equal(card.snapshot().visible, true);
card.handle({ type: 'interruption' });
assert.equal(card.snapshot().assistant.visible, false);
card.destroy();
});