diff --git a/changelog.d/1724-stories.md b/changelog.d/1724-stories.md new file mode 100644 index 000000000..f02c33761 --- /dev/null +++ b/changelog.d/1724-stories.md @@ -0,0 +1,8 @@ +- Stories: GEV scenes told as stories on the live globe. A Stories button + opens Explore, a feed of the scenes GEV ships and the person's own (saved + from AUTHORING, or drafts) with a pin each and an idle tour between them; a + story plays with chapters, key moments and its data sources, and Close puts + the person's view, clock and layers back. Play after a pause, a seek or a + paused story link goes on from that exact moment. AUTHORING gains + THUMBNAIL, topic tags and SAVE TO STORIES + (Manjunath, @manjunath22466; Kevin Rajan, @kvnloo; #1724). diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 6d4b2c853..c4eb13b6b 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -660,6 +660,147 @@ owner (open state, filters, the rows they show, Remove all) and the text helpers the view renders, so every label follows the locale. Only the modal open/close and focus moves are imperative. +## Stories — November 13, 2026 + +A Stories button (top right, `#hStories`) opens Explore: a feed of stories +over GEV's own globe with their pins, then a Story screen with a +timeline, chapters and context. Every GEV page loads only what the +closed state needs: the boot shell (`src/stories/boot.js`), the store, +`StoriesView.vue` with the Stories button, first-visit popup and +back-to-story toast, and their styles (`stories-entry.css`). The screens +(`ExploreScreen.vue`, `StoryScreen.vue` and their parts), their styles +(`stories.css`), the story data (`storyMeta.js`, `catalog.js`) and the +controller are separate chunks fetched together on first open, on a +`?stories=` / `?story=` link, or on Back or Forward into Stories; a failed +fetch is reported and the next attempt tries again. Stories keeps its place +in the query, not the share-state hash, with one history entry per screen; +Close steps back over those entries, so Back afterwards does not reopen it, +and share links carry the Stories screen and place. Explore links use +`stories=explore` and optionally `selectedStory`. Player links use `story` +and optionally `storyTime` in milliseconds, opening paused at that time. +GEV's share link keeps +`history.state` when it rewrites the hash, so Stories' entries survive. + +Every story is a GEV scene, played on the person's own globe through +`SceneDirector`, with its labels, embedded media, A/B imagery split (marked +`data-globe-overlay`) and data-pack cards kept while the rest of GEV is +hidden. Explore lists the scenes the person saved to Stories from AUTHORING +first (`SceneDirector.storyScenes()`, refreshed on every `stories-changed`), +then the person's other scenes with a shot as drafts +(`SceneDirector.draftScenes()`, marked DRAFT), then the +scenes GEV ships with a camera path while they are in the project +(`SceneDirector.hasScene`). Zero-shot drafts can be reopened or deleted +from the scene selector in AUTHORING after a reload. A saved scene brings the card image and topics +picked for it in AUTHORING; the topic chips use AUTHORING's tag names and +order. Explore opens at country level +(2,500 km) over the first story and, while idle, flies on to the next story +every few seconds, for a minute after the last touch (never with a story +picked or with reduced motion). The start view and a picked story's place +(at an angle, −60°) are framed in the middle of the map Explore leaves +visible (`.ex-map`: beside the feed on a wide screen, above the sheet on a +phone), not of the whole canvas; a flight that lands after that area has +moved makes one short correction. Stories reaches the globe only +through the renderer's ports (`src/render/rendererOf.js`): its flights, +pose and Close's restore go through the camera port, and after each landing +a continuous-render hold keeps the globe drawing only until the globe port +reports the base settled (at most 10 s). Each story's pin, in the +world overlay's label style (`src/overlays/worldOverlayTokens.js`), is placed +from the query port's projection in the frame port's `postRender`, on every +drawn frame, written straight to the pin's transform, and hidden behind the +globe (`query.isOccluded`); the pins ask for no frames of their own. Card +images are drawn from the keyless Esri World Imagery tiles around the +story's place (`src/stories/thumbs.js`: the same Web Mercator tiles, from +the same host, that GEV's Esri map draws, read as images) unless the scene +has its own; with any other map stack than Esri Satellite +(`mapStackController.getActiveId()`) the card shows its placeholder, and a +card drawn from Esri earlier is dropped. AUTHORING captures thumbnails only +on a ready Esri World Imagery stack and embeds the imagery credit in the +exported image. A shot +tied to an anchor rather than a place pins the whole-globe view. + +A shipped story's context comes from what GEV knows about it +(`src/stories/catalog.js`): what it shows, the place each shot goes, and, +for each layer it turns on, what the layer shows and the data sources +DATA_SOURCES.md lists for it. The Nepal story carries the event pack's own +facts and sources (Vantor imagery and the GeoPera reconstruction, CC BY-NC +4.0), and its key moments are the chapters that show them. Topic chips and +region choices are the ones the listed stories carry. Closing Stories after +a story played puts AUTHORING's own selection back; +Open in GEV leaves the story's scene selected there. Entering Stories +cancels in-flight Authoring work before it captures the view it will +restore. Dragging the timeline previews the time, and releasing +it seeks the globe once. Play after a pause, a seek or a paused link goes on +from that exact moment (`SceneDirector.startScene` with `fromProgress`), even +inside a shot or its final quarter-second: the camera, the hold, the scene +clock and the layers' reveals carry on from there (a clip that moment falls in +starts again from its beginning, and the hold waits for it, with the +timeline held at the shot's end meanwhile). A scene that has actually ended replays from the +beginning. + +Opening Stories claims the camera through the shared navigation policy, so +orbit, a tracked contact or a flight end, and Cockpit refuses with the +policy's own toast ("Exit cockpit to open Stories"). Those owners are not +started again on Close: the camera and the world clock (`src/time/worldClock.js`: +live, or the time it showed, playing on as if never paused) come back, a +follow or orbit does not. While Stories is open it owns the keyboard (`src/stories/keyboard.js`): +a key pressed in Stories stops at its root, and a key pressed anywhere else +stops at the window, so none of GEV's document-level shortcuts act behind it, +except on a playing GEV scene's own on-globe controls (`data-globe-overlay`); +the capture-phase listeners (hold Space to talk, a surface's Tab trap, the +wildlife card's Escape) check `body.stories-mode` themselves. "Open Stories" +works from voice or typed AI chat. Entering Stories ends active or pending +microphone capture and blocks microphone starts while Stories owns the +globe. Stories shows a notice when it turns off an active or connecting +voice session. A typed reply can finish after opening Stories. Globe tool calls +are refused while `stories-mode` is set ("Close Stories first"). + +The address bar names only the story while Stories is open. Stories holds +hash writes (`ShareLinkManager.holdWrites`) and saves a snapshot of the live +globe, including when it opens from a story link without a hash. +Close dismisses the Stories screens and keeps the keyboard and microphone +guards until the previous view is back. The director is waited +on until idle (`SceneDirector.whenIdle`: no run and no pending LOAD, seek or +interaction; at most 15 s). When the 15 s run out, the scene's layer cleanup +still under way is revoked (`SceneDirector.revokeSceneRelease`) before +anything is put back, so it cannot turn off a layer Close restores. The camera and world clock are put back (a clock +nothing moved while Stories was open, with no jump and the same mode and +rate, is left alone), unless a +camera owner took over during the wait (a drag on the globe, a layer's +focus). History steps back over Stories' entries; once that lands (or after +2 s if it never does), the saved share state is re-applied directly +(`ShareLinkManager.restore`, since the share link skips a hash it already +knows) and the share link writes the live view again. Back out of Stories +re-applies it the same way, and Back to Explore from a story re-applies it +once the scene has torn down, with the address still naming only Stories. +"Open in GEV" (GEV scene stories only) closes the +same way but keeps the place and look: nothing is put back, the scene and its +current shot are selected (`SceneDirector.selectShot`) and the Scenes panel +opens for authoring; Back then goes to the page before Stories. A Close or +open made while a traversal is landing runs once it has landed; a second Back +to Explore is ignored. Reopening Stories while a closed GEV scene is still +tearing down keeps the view from before Stories as the one Close restores. +Inside an inline embed Stories is not offered: the URL and history are the +host's. + +The Stories button sits left of whatever holds GEV's top-right corner: the +ACTIVE STYLE readout (`#style-indicator`), or an element marked with +`data-stories-corner` (with or beside the readout; one added after Stories +mounts is found within 20 s). On GEV's phone layout, +where neither shows, it is icon-only under the corner-actions column +(`#corner-actions`), aligned with its right edge. The first-visit popup +is a labelled region that does not take focus. The story page's rail offers +Open in GEV and Share only. Share on Explore reopens the list with the same +shipped story selected. A story link opens paused at the shared +point on its timeline. A scene of the person's own (saved or draft) +shows as theirs ("You") and has no Share: the project lives on this device, +so a link to it would open nothing elsewhere; a link to a story that is not +here opens Explore and says so. GEV's own credit line stays on screen throughout, inside the +Stories map while a scene plays (`creditPlacement` in +`src/stories/format.js`): `stories.css` places the renderer's own credit +display (`#cesium-credits`) and the application's credit line +(`#gev-credits`, `src/ui/creditsView.js`, with its list opening above it) +alike, and exempts both from the rule that hides GEV's chrome. + ## Utilities panel — October 29, 2026 UTILITIES is its own left-side panel, between Data Layers and Scenes, with @@ -1273,7 +1414,7 @@ Native `
- {{ t('EDIT DETAILS') }} - {{ t('SHARE SCENE') }} + {{ t('EDIT DETAILS') }} + {{ t('SHARE SCENE') }}
{{ scene?.progress ?? 0 }}%
diff --git a/src/ui/views/StoriesView.vue b/src/ui/views/StoriesView.vue new file mode 100644 index 000000000..1c3ce552e --- /dev/null +++ b/src/ui/views/StoriesView.vue @@ -0,0 +1,362 @@ + + + diff --git a/src/ui/views/index.js b/src/ui/views/index.js index a85224912..522a8d0c9 100644 --- a/src/ui/views/index.js +++ b/src/ui/views/index.js @@ -10,6 +10,7 @@ import CockpitView from './CockpitView.vue'; import SceneChromeView from './SceneChromeView.vue'; import ProviderSettingsView from './ProviderSettingsView.vue'; import IntelHudView from './IntelHudView.vue'; +import StoriesView from './StoriesView.vue'; /** * Application views by name. A page places each one with a host element, @@ -26,6 +27,7 @@ export const APPLICATION_VIEWS = Object.freeze({ context: ContextView, welcome: WelcomeView, 'intel-hud': IntelHudView, + stories: StoriesView, }); const mounted = new WeakMap(); diff --git a/src/ui/views/panels/ImagerySplitView.vue b/src/ui/views/panels/ImagerySplitView.vue index b25c7cf46..1c2a34e05 100644 --- a/src/ui/views/panels/ImagerySplitView.vue +++ b/src/ui/views/panels/ImagerySplitView.vue @@ -11,6 +11,6 @@ const optional = (value) => value ?? undefined; diff --git a/src/ui/views/stories/ExploreScreen.vue b/src/ui/views/stories/ExploreScreen.vue new file mode 100644 index 000000000..359b65819 --- /dev/null +++ b/src/ui/views/stories/ExploreScreen.vue @@ -0,0 +1,765 @@ + + + diff --git a/src/ui/views/stories/StoriesHeader.vue b/src/ui/views/stories/StoriesHeader.vue new file mode 100644 index 000000000..412f8454d --- /dev/null +++ b/src/ui/views/stories/StoriesHeader.vue @@ -0,0 +1,192 @@ + + + diff --git a/src/ui/views/stories/StoryContext.vue b/src/ui/views/stories/StoryContext.vue new file mode 100644 index 000000000..256d883a7 --- /dev/null +++ b/src/ui/views/stories/StoryContext.vue @@ -0,0 +1,412 @@ + + + diff --git a/src/ui/views/stories/StoryRail.vue b/src/ui/views/stories/StoryRail.vue new file mode 100644 index 000000000..5687a61c9 --- /dev/null +++ b/src/ui/views/stories/StoryRail.vue @@ -0,0 +1,53 @@ + + + diff --git a/src/ui/views/stories/StoryScreen.vue b/src/ui/views/stories/StoryScreen.vue new file mode 100644 index 000000000..ad32f837f --- /dev/null +++ b/src/ui/views/stories/StoryScreen.vue @@ -0,0 +1,614 @@ + + + diff --git a/src/ui/views/stories/StoryTimeline.vue b/src/ui/views/stories/StoryTimeline.vue new file mode 100644 index 000000000..070a6af83 --- /dev/null +++ b/src/ui/views/stories/StoryTimeline.vue @@ -0,0 +1,462 @@ + + + diff --git a/src/ui/wildlifeSightingCard.js b/src/ui/wildlifeSightingCard.js index ce6a24588..7a01b110d 100644 --- a/src/ui/wildlifeSightingCard.js +++ b/src/ui/wildlifeSightingCard.js @@ -203,6 +203,8 @@ export function createWildlifeSightingCard({ const keydown = (event) => { if (!selected.value || event.key !== 'Escape') return; + // Stories owns Escape while it is open (the card is behind it). + if (documentRef.body?.classList?.contains?.('stories-mode')) return; event.preventDefault(); event.stopImmediatePropagation(); layer.clearSightingSelection(); diff --git a/src/voice/actionSchemas.js b/src/voice/actionSchemas.js index 03ae69f8a..2b57090a1 100644 --- a/src/voice/actionSchemas.js +++ b/src/voice/actionSchemas.js @@ -430,6 +430,14 @@ const schemas = [ }, }, }, + { + name: 'open_stories', + parameters: { + type: 'object', + additionalProperties: false, + properties: {}, + }, + }, { name: 'control_scene', parameters: { diff --git a/src/voice/actionSchemas.test.mjs b/src/voice/actionSchemas.test.mjs index c2f084d06..33971d7aa 100644 --- a/src/voice/actionSchemas.test.mjs +++ b/src/voice/actionSchemas.test.mjs @@ -59,7 +59,8 @@ test('the complete Realtime tool payload pins the manifest-generated layer relea // move_camera tilt takes an optional pitchDeg angle, described as // absolute with its -45 default, with steps sent as a direction. // show_street_level (map, source, photo) adds one tool. - '0c27f62062001e6a07241cd301c3d357f55e9b74f66cd2904d16590510337468', + // open_stories adds the Stories Explore entry action. + '5a2deb65bef5db8fa49ee55dd66efb884e7c3ba97dbdcaad3fada087170d2225', ); }); @@ -157,6 +158,7 @@ test('all legacy action arguments are byte-identical after removing the delibera 'find_imagery', 'osm_query', 'show_street_level', + 'open_stories', ].includes(tool.name), ); // Layer enums are generated from the voice layer manifest and pinned by diff --git a/src/voice/geminiSession.js b/src/voice/geminiSession.js index 572db471a..dd8089ed7 100644 --- a/src/voice/geminiSession.js +++ b/src/voice/geminiSession.js @@ -3,6 +3,7 @@ import { t, translateKnown } from '../i18n/index.js'; import { msg } from '../i18n/marker.js'; import { createGeminiAudio } from './geminiAudio.js'; import { RealtimeInput } from './realtimeInput.js'; +import { isStoriesOpen } from './realtimeInputPolicy.js'; import { RealtimeRadio } from './realtimeRadio.js'; import { captureViewportImage } from './realtimeViewport.js'; import { createNarrationScheduler } from './narration.js'; @@ -652,7 +653,13 @@ export class GeminiSessionController { } async start({ pushToTalk = false } = {}) { - if (this.disposed || this.isActive() || this.signal?.aborted) return; + if ( + isStoriesOpen() || + this.disposed || + this.isActive() || + this.signal?.aborted + ) + return; const held = pushToTalk && this.input.pushToTalkKeyHeld; const spaceHeld = this.input.spaceKeyHeld; this.stop({ preserveStatus: true }); diff --git a/src/voice/gevActions.js b/src/voice/gevActions.js index 3343cd243..e2ca226a8 100644 --- a/src/voice/gevActions.js +++ b/src/voice/gevActions.js @@ -8,6 +8,7 @@ import { import { readLayerLifecycleSummary } from './layerSummary.js'; import { SPEECH_BUILDERS, attachVoiceResult } from './speech.js'; import { resolveDeicticArgs } from './deixis.js'; +import { isStoriesOpen } from './realtimeInputPolicy.js'; export { readLayerLifecycleSummary } from './layerSummary.js'; import { defaultGeospatial } from '../search/defaults.js'; import { msg } from '../i18n/index.js'; @@ -294,6 +295,7 @@ export function createGevActionRunner({ styleManager, dataManager, sceneDirector = null, + openStories = null, annotations = null, placeSearch = unavailablePlaceSearch, floorServices = defaultFloorServices, @@ -447,6 +449,56 @@ export function createGevActionRunner({ const current = () => !runOptions.signal?.aborted && (typeof runOptions.isCurrent !== 'function' || runOptions.isCurrent()); + if (name === 'open_stories') { + const denied = { ok: false, action: name, applied: false }; + const cancelled = { + ...denied, + code: 'CANCELLED', + cancelled: true, + error: 'This request was superseded.', + }; + if (!current()) return cancelled; + if (typeof openStories !== 'function') + return { + ...denied, + unavailable: true, + error: 'Stories is unavailable in this host.', + say: 'Stories is unavailable in this host.', + }; + // Opening Stories can stop the voice session that requested it. The + // action was applied even when that session is no longer current. + const opened = await openStories({ isCurrent: current }); + if (opened === true) + return { + ok: true, + action: name, + applied: true, + say: 'Stories is open.', + }; + if (!current()) return cancelled; + if (opened === 'already-open') + return { + ok: true, + action: name, + applied: false, + say: 'Stories is already open.', + }; + return { + ...denied, + error: 'Stories could not be opened.', + say: 'Stories could not be opened.', + }; + } + // Stories owns the camera and scenes. Leave the globe behind it alone. + if (isStoriesOpen()) + return { + ok: false, + action: name, + applied: false, + refused: 'stories-open', + error: 'Stories is open. Close Stories first.', + say: 'Stories is open. Close Stories first to control the globe.', + }; // Step reports for voice narration and the voice card; never fatal. const reportProgress = (step, label = '') => { try { diff --git a/src/voice/gevActions.test.mjs b/src/voice/gevActions.test.mjs index 123bbd8e4..3aa74548c 100644 --- a/src/voice/gevActions.test.mjs +++ b/src/voice/gevActions.test.mjs @@ -258,6 +258,100 @@ test('zoom to globe adopts the shared visible reset route and returns its result assert.equal(calls, 1); }); +test('voice changes nothing behind Stories while it is open', async () => { + globalThis.window = globalThis.window || { + clearTimeout, + setTimeout, + requestIdleCallback: null, + }; + const viewer = { + clock: { onTick: { addEventListener: () => () => {} } }, + scene: { canvas: { addEventListener() {}, removeEventListener() {} } }, + camera: { moveEnd: { addEventListener() {} } }, + }; + let calls = 0; + let storiesVisible = true; + const styleManager = { + async resetToGlobeView() { + calls += 1; + return { ok: true }; + }, + }; + const runner = createGevActionRunner({ + viewer, + styleManager, + dataManager: { layers: new Map(), getAll: () => [] }, + openStories: () => (storiesVisible ? 'already-open' : true), + }); + const classes = new Set(['stories-mode']); + const priorDocument = globalThis.document; + globalThis.document = { + ...(priorDocument ?? {}), + body: { classList: { contains: (c) => classes.has(c) } }, + }; + try { + const refused = await runner('zoom_to_globe'); + assert.equal(refused.ok, false); + assert.equal(refused.refused, 'stories-open'); + const alreadyOpen = await runner('open_stories'); + assert.equal(alreadyOpen.ok, true); + assert.equal(alreadyOpen.applied, false); + assert.equal(calls, 0); + // The input guard still owns the globe while the director settles. + storiesVisible = false; + const reopened = await runner('open_stories'); + assert.equal(reopened.ok, true); + assert.equal(reopened.applied, true); + assert.equal((await runner('zoom_to_globe')).refused, 'stories-open'); + classes.clear(); + await runner('zoom_to_globe'); + assert.equal(calls, 1, 'works again once Stories has closed'); + } finally { + globalThis.document = priorDocument; + } +}); + +test('opening Stories only admits current requests and can stop its own session', async () => { + const { viewer, styleManager } = createVoiceNavigationHarness(); + const first = new AbortController(); + let loaded = Promise.resolve(); + let admitted = 0; + let opened = 0; + const runner = createGevActionRunner({ + viewer, + styleManager, + dataManager: { layers: new Map(), getAll: () => [] }, + async openStories({ isCurrent }) { + admitted += 1; + await loaded; + if (!isCurrent()) return false; + opened += 1; + first.abort(); + return true; + }, + }); + + const applied = await runner('open_stories', {}, { signal: first.signal }); + assert.equal(applied.ok, true); + assert.equal(applied.applied, true); + + const stale = await runner('open_stories', {}, { signal: first.signal }); + assert.equal(stale.cancelled, true); + assert.equal(admitted, 1); + + let finishLoading; + loaded = new Promise((resolve) => { finishLoading = resolve; }); + const second = new AbortController(); + const pending = runner('open_stories', {}, { signal: second.signal }); + assert.equal(admitted, 2); + second.abort(); + finishLoading(); + const canceled = await pending; + assert.equal(canceled.cancelled, true); + assert.equal(canceled.applied, false); + assert.equal(opened, 1); +}); + test('dependent voice navigation waits for the destination viewport to arrive', async () => { globalThis.window = globalThis.window || { clearTimeout, diff --git a/src/voice/gevRealtime.test.mjs b/src/voice/gevRealtime.test.mjs index 54af33e00..505b6759e 100644 --- a/src/voice/gevRealtime.test.mjs +++ b/src/voice/gevRealtime.test.mjs @@ -358,6 +358,34 @@ test('push-to-talk rechecks focus before the hold threshold claims voice', (t) = assert.equal(f.controller.spaceKeyHeld, false); }); +test('opening Stories cancels a pending Space hold even when focus stays put', (t) => { + const f = createPushToTalkFixture(t); + const background = pushToTalkTarget(); + const classes = new Set(); + f.document.body = { classList: { contains: (name) => classes.has(name) } }; + + f.key('keydown', background); + classes.add('stories-mode'); + f.advance(PUSH_TO_TALK_HOLD_DELAY_MS); + assert.deepEqual(f.starts, []); + assert.deepEqual(f.radio, []); + assert.equal(f.controller.spaceKeyHeld, false); + assert.equal(f.timers.size, 0); + + classes.delete('stories-mode'); + f.key('keydown', background); + f.advance(PUSH_TO_TALK_HOLD_DELAY_MS); + assert.equal(f.microphone.enabled, true); + f.key('keyup', background); + assert.equal(f.microphone.enabled, false); + + f.key('keydown', background); + classes.add('stories-mode'); + f.advance(PUSH_TO_TALK_HOLD_DELAY_MS); + assert.equal(f.starts.length, 1); + assert.equal(f.microphone.enabled, false); +}); + test('push-to-talk never claims a control-started hold after focus moves outside', (t) => { const f = createPushToTalkFixture(t); const control = pushToTalkTarget({ tagName: 'BUTTON' }); diff --git a/src/voice/realtimeConnection.js b/src/voice/realtimeConnection.js index 7301ba6ed..94db67134 100644 --- a/src/voice/realtimeConnection.js +++ b/src/voice/realtimeConnection.js @@ -1,4 +1,5 @@ import { t } from '../i18n/index.js'; +import { isStoriesOpen } from './realtimeInputPolicy.js'; const DISCONNECT_GRACE_MS = 6000; @@ -52,7 +53,8 @@ export class RealtimeConnection { return this.readStatus(); } async start({ pushToTalk = false } = {}) { - if (this.isActive() || this.lifetimeSignal?.aborted) return; + if (isStoriesOpen() || this.isActive() || this.lifetimeSignal?.aborted) + return; this.pauseRadioForVoice(); const pushToTalkKeyHeld = pushToTalk && this.input.pushToTalkKeyHeld; const spaceKeyHeld = this.input.spaceKeyHeld; diff --git a/src/voice/realtimeInput.js b/src/voice/realtimeInput.js index 0a43fa6ec..e3e9b3fd5 100644 --- a/src/voice/realtimeInput.js +++ b/src/voice/realtimeInput.js @@ -3,6 +3,7 @@ import { PUSH_TO_TALK_HOLD_DELAY_MS, MICROPHONE_VISUALIZER_GATE, ASSISTANT_VISUALIZER_GATE, + isStoriesOpen, isPushToTalkKey, SPACE_INTERACTIVE_SELECTOR, isInteractiveSpaceTarget, @@ -55,9 +56,11 @@ export class RealtimeInput { bindPushToTalkShortcut() { if (this.shortcutKeyDownHandler) return; this.shortcutKeyDownHandler = (event) => { - // Space types in chat; it never talks while chat is open. + // Space types in chat; it never talks while chat is open, nor while + // Stories owns the keyboard (body.stories-mode). if ( this.ui.state?.chat?.open === true || + isStoriesOpen() || !shouldHandlePushToTalkKeyDown(event) ) return; @@ -93,6 +96,10 @@ export class RealtimeInput { this.pushToTalkHoldTimer = null; if (holdGeneration !== this.pushToTalkHoldGeneration) return; if (!this.spaceKeyHeld || this.pushToTalkKeyHeld) return; + if (isStoriesOpen()) { + this.shortcutBlurHandler(); + return; + } // A pointer or Tab focus change during the delay cancels the original // gesture rather than letting voice claim a key held for another owner. if (document.activeElement !== this.pushToTalkHoldFocusOwner) return; diff --git a/src/voice/realtimeInputPolicy.js b/src/voice/realtimeInputPolicy.js index 39bc29581..bf8e4ac52 100644 --- a/src/voice/realtimeInputPolicy.js +++ b/src/voice/realtimeInputPolicy.js @@ -9,6 +9,13 @@ export const MICROPHONE_VISUALIZER_GATE = 0.12; export const ASSISTANT_VISUALIZER_GATE = 0.04; +/** Stories takes keyboard and microphone ownership away from voice. */ +export function isStoriesOpen() { + return ( + globalThis.document?.body?.classList?.contains?.('stories-mode') ?? false + ); +} + /** * Returns whether a keyboard event represents the hold-Space voice shortcut. * @param {KeyboardEvent|object|null} event diff --git a/src/voice/session.js b/src/voice/session.js index 7c31f17d2..2586d75f6 100644 --- a/src/voice/session.js +++ b/src/voice/session.js @@ -5,7 +5,12 @@ * Events: state, transcript, action-call, progress, action-result, interruption, * completion, turn-metrics. */ -export function createVoiceSession({ createAdapter, runner, signal }) { +export function createVoiceSession({ + createAdapter, + runner, + signal, + canStart, +}) { if (typeof createAdapter !== 'function' || typeof runner !== 'function') throw new TypeError('Voice requires an adapter factory and action runner'); const lifetime = new AbortController(); @@ -146,7 +151,12 @@ export function createVoiceSession({ createAdapter, runner, signal }) { return () => listeners.delete(listener); }, async start(options) { - if (disposed || !['idle', 'error'].includes(state)) return; + if ( + disposed || + !['idle', 'error'].includes(state) || + canStart?.(options) === false + ) + return; const attempt = ++startAttempt; emit({ type: 'state', state: 'connecting' }); if (disposed || attempt !== startAttempt) return; diff --git a/src/voice/session.test.mjs b/src/voice/session.test.mjs index 276952cf4..bec5ed4a2 100644 --- a/src/voice/session.test.mjs +++ b/src/voice/session.test.mjs @@ -429,12 +429,13 @@ test('a late rejection cannot settle a newer turn card', async () => { f.session.destroy(); }); -function startReentrancyFixture() { +function startReentrancyFixture(canStart) { let starts = 0; let stops = 0; const settings = []; const session = createVoiceSession({ runner: async () => ({ ok: true }), + canStart, createAdapter: ({ emit }) => ({ async start(options) { starts++; @@ -528,6 +529,28 @@ test('ordinary start forwards caller settings once and remains stoppable', async f.session.destroy(); }); +test('start admission rejects microphone use before connecting and allows typed chat', async () => { + const f = startReentrancyFixture((options) => options?.textOnly === true); + const states = []; + f.session.subscribe((event) => { + if (event.type === 'state') states.push(event.state); + }); + await f.session.start({ pushToTalk: true }); + assert.equal(f.starts(), 0); + assert.equal(f.session.state, 'idle'); + assert.equal(states.includes('connecting'), false); + + const settings = { textOnly: true }; + await f.session.start(settings); + assert.equal(f.settings.length, 1); + assert.equal(f.settings[0], settings); + assert.equal(f.starts(), 1); + assert.equal(f.session.state, 'listening'); + f.session.stop(); + assert.equal(f.session.state, 'idle'); + f.session.destroy(); +}); + for (const emitsIdle of [false, true]) { for (const outcome of ['normal', 'preserveStatus', 'throw']) { test(`stop preserves custom adapter contract: emitsIdle=${emitsIdle}, ${outcome}`, async (t) => { diff --git a/src/voice/sessionCommands.js b/src/voice/sessionCommands.js index 2add30636..bcf0ba3ab 100644 --- a/src/voice/sessionCommands.js +++ b/src/voice/sessionCommands.js @@ -34,6 +34,7 @@ export function createVoiceCommands({ supportedProviders, onProviderChange, onSessionEvent, + canStartVoice, createCard = (options) => new VoiceCardControls(options), pointer = null, referents = null, @@ -49,6 +50,7 @@ export function createVoiceCommands({ let panelMount; const session = createVoiceSession({ runner, + canStart: (options) => options?.textOnly || canStartVoice?.() !== false, signal, createAdapter: (hooks) => { if (panelHost) actionObserver = panelHost.observeActions(hooks); @@ -284,7 +286,7 @@ export function createVoiceCommands({ void session.start({ pushToTalk: false }); }; const buttonHandler = () => { - if (adapter.ignoreButtonClick?.()) return; + if (canStartVoice?.() === false || adapter.ignoreButtonClick?.()) return; chat?.close(); if (session.isActive() && adapter.getStartOptions?.().textOnly) { session.stop(); diff --git a/src/voice/speech.js b/src/voice/speech.js index 3ecb19114..34fddc1a2 100644 --- a/src/voice/speech.js +++ b/src/voice/speech.js @@ -1070,6 +1070,7 @@ const STEP_TITLES = Object.freeze({ set_detection: msg('Adjust detection overlay'), set_map_stack: msg('Change map stack'), set_post_processing: msg('Adjust post-processing'), + open_stories: msg('Stories'), control_scene: msg('Control scene'), control_cctv: msg('Control CCTV'), control_radio: msg('Control radio'), diff --git a/style.css b/style.css index 7e218d7f8..6d3591643 100644 --- a/style.css +++ b/style.css @@ -55,4 +55,5 @@ @import './src/ui/styles/cockpit-panels.css'; @import './src/ui/styles/ai-chat.css'; @import './src/ui/styles/scope-off-shadows.css'; +@import './src/ui/styles/stories-entry.css'; @import './src/ui/styles/cyber.css';