From af461febae51c176770163c4ec104f8f981287e8 Mon Sep 17 00:00:00 2001 From: manjunath22466 <301209272+manjunath22466@users.noreply.github.com> Date: Mon, 2 Nov 2026 18:39:44 +0530 Subject: [PATCH] feat(voice): add typed AI chat beside voice Chat sits next to the mic in the command dock. Replies are silent and never request the microphone; the first message connects with the selected provider and runs the same map actions as voice through each provider's text API (OpenAI OPENAI_CHAT_MODEL, Gemini GEMINI_CHAT_MODEL via generateContent, which never starts Live). Only one mode runs at a time. - Chat panel rendered by the voice control presenter: conversation, recent and quick-command suggestions, Up/Down history, attachments (map-layer references or small UTF-8 text files, sent only with the message) - Chat and voice panels sit above the dock or dock into the right rail's column; minimized bars sit centred above the dock, and height animates on swaps - A centred panel steps down to its bar for the Location and Visual Presets trays without moving focus, rests above an open tray, and docks right while expanded over one; it returns to the centre once the trays close - Minimized voice previews the last sentence from each side; the empty Voice Preview suggests things to say - Settings opened from chat keep chat's frame and show the shared AI provider and quick commands; voice Settings show Back to voice - Closing chat ends a typed session; the voice panel's close turns voice off - The dock widens its voice control for the chat button and restyles its wings, which show icons only at 900px and below; assistant panels share the left panels' glass, and Cyber keeps its own surfaces - Server text-chat handlers with body, time and per-IP limits (GEV_RATELIMIT_*_CHAT_PER_MIN), documented in SECURITY.md; tests and all eleven locales Co-authored-by: Darren Dignam <22195071+darrendignam@users.noreply.github.com> --- .env.example | 11 + README.md | 2 + SECURITY.md | 3 +- docs/CURRENT-STATE.md | 30 + scripts/format-scope.json | 20 +- scripts/package-boundaries.json | 35 + server/providers/common/textChat.js | 136 ++ server/providers/common/textInstructions.js | 31 + server/providers/common/textTools.js | 54 + server/providers/gemini.js | 5 + server/providers/gemini/chat.js | 172 ++ server/providers/openai.js | 5 + server/providers/openai/chat.js | 139 ++ server/providers/openai/instructions.js | 12 +- src/geminiChatServer.test.mjs | 162 ++ src/geminiComposition.test.mjs | 3 +- src/geminiServer.test.mjs | 5 +- src/i18n/locales/ar.json | 64 +- src/i18n/locales/de.json | 64 +- src/i18n/locales/es.json | 64 +- src/i18n/locales/fr.json | 64 +- src/i18n/locales/it.json | 64 +- src/i18n/locales/ja.json | 64 +- src/i18n/locales/pt.json | 64 +- src/i18n/locales/ru.json | 64 +- src/i18n/locales/uk.json | 64 +- src/i18n/locales/zh-CN.json | 64 +- src/i18n/locales/zh-Hant.json | 64 +- src/i18n/same-as-english.json | 10 + src/textChatServer.test.mjs | 276 ++++ src/ui/stores/voice.js | 22 +- src/ui/styles/ai-chat.css | 1656 +++++++++++++++++++ src/ui/styles/command-dock-sliding.css | 3 +- src/ui/views/panels/ChatPanel.vue | 92 ++ src/ui/views/panels/PanelLayoutButtons.vue | 18 + src/ui/views/panels/VoiceControl.vue | 45 +- src/voice/chatAttachments.js | 137 ++ src/voice/chatControl.js | 387 +++++ src/voice/chatPreferences.js | 113 ++ src/voice/control.js | 81 +- src/voice/geminiSession.js | 32 +- src/voice/geminiTextChat.js | 5 + src/voice/geminiTextChat.test.mjs | 492 ++++++ src/voice/panel/host.test.mjs | 356 +++- src/voice/panel/manualPanel.js | 27 +- src/voice/panel/manualPanel.test.mjs | 9 +- src/voice/panelLayout.js | 384 +++++ src/voice/providerCommands.js | 8 +- src/voice/realtimeInput.js | 7 +- src/voice/realtimeSession.js | 41 +- src/voice/session.test.mjs | 45 + src/voice/sessionCommands.js | 91 +- src/voice/textChat.js | 383 +++++ src/voice/textProtocol.js | 153 ++ src/voice/voiceMiniPreview.js | 68 + src/voice/voiceMiniPreview.test.mjs | 92 ++ style.css | 1 + 57 files changed, 6500 insertions(+), 63 deletions(-) create mode 100644 server/providers/common/textChat.js create mode 100644 server/providers/common/textInstructions.js create mode 100644 server/providers/common/textTools.js create mode 100644 server/providers/gemini/chat.js create mode 100644 server/providers/openai/chat.js create mode 100644 src/geminiChatServer.test.mjs create mode 100644 src/textChatServer.test.mjs create mode 100644 src/ui/styles/ai-chat.css create mode 100644 src/ui/views/panels/ChatPanel.vue create mode 100644 src/ui/views/panels/PanelLayoutButtons.vue create mode 100644 src/voice/chatAttachments.js create mode 100644 src/voice/chatControl.js create mode 100644 src/voice/chatPreferences.js create mode 100644 src/voice/geminiTextChat.js create mode 100644 src/voice/geminiTextChat.test.mjs create mode 100644 src/voice/panelLayout.js create mode 100644 src/voice/textChat.js create mode 100644 src/voice/textProtocol.js create mode 100644 src/voice/voiceMiniPreview.js create mode 100644 src/voice/voiceMiniPreview.test.mjs diff --git a/.env.example b/.env.example index c9700c5be..08b0df8f2 100644 --- a/.env.example +++ b/.env.example @@ -61,6 +61,12 @@ CESIUM_ION_TOKEN= # ChatGPT/Codex OAuth session; the browser still receives only a short-lived # Realtime client secret. Do not prefix with VITE_. OPENAI_API_KEY= +# Typed chat uses Chat Completions, independently of the voice model. It always +# bills OPENAI_API_KEY; GEV_PREFER_CODEX_OAUTH below applies to voice only. +OPENAI_CHAT_MODEL=gpt-4.1-mini +# Typed chat requests (/api/openai/chat) per IP per minute, separate from the +# voice throttle; 0 disables it. Not a billing cap. +# GEV_RATELIMIT_OPENAI_CHAT_PER_MIN=60 # the default # Optional: forbid API-key voice sessions. Select ChatGPT OAuth in Provider # Settings; missing or expired Codex sign-in then fails closed. # GEV_PREFER_CODEX_OAUTH=true @@ -269,3 +275,8 @@ OVERPASS_UPSTREAMS= # frame-ancestors sources such as "https://example.com", or * for any page. # Unset, no page may frame it. Other pages of the app can never be framed. # GEV_EMBED_FRAME_ANCESTORS= + +# Gemini typed chat uses generateContent; voice continues to use Gemini Live. +GEMINI_CHAT_MODEL=gemini-3.5-flash-lite +# Typed chat requests (/api/gemini/chat) per IP per minute; 0 disables it. +# GEV_RATELIMIT_GEMINI_CHAT_PER_MIN=60 # the default diff --git a/README.md b/README.md index 3a4c5fc43..d308e9b15 100644 --- a/README.md +++ b/README.md @@ -327,6 +327,8 @@ The standalone app offers **OpenAI** and **Gemini** in the GEV MIC provider choo Changing provider ends the current conversation. If voice is on, the new provider starts with the same open-mic or push-to-talk mode; a held Space key is not carried over. Gemini uses the same map actions and captions, but has no OpenAI cost meter or MINI tier. When Gemini reports that its session has expired, start voice again for a new conversation. The provider chooser does not change the OpenAI-backed AI HUD summary. +Next to the mic, **CHAT** opens a typed chat with silent replies. It uses the selected provider and the same map actions, and never asks for the microphone. OpenAI chat needs an API key (`OPENAI_CHAT_MODEL`); Gemini chat uses a separate text model (`GEMINI_CHAT_MODEL`). Chat and voice panels sit above the dock or dock to the right. + > Voice can use an **OpenAI API key** or, on the same machine, an opt-in signed-in **ChatGPT/Codex OAuth** session. The API-key path remains the default. Selecting ChatGPT OAuth in Provider Settings reuses an existing local sign-in or starts the Codex browser sign-in flow and selects OAuth when it completes. The OpenAI key still drives the **AI HUD summary**: a terse, five-word intelligence-style readout of the current view that regenerates as you move. The OAuth option is experimental. OpenAI documents ChatGPT plan usage for the [Responses API](https://developers.openai.com/siwc/token-sharing-open-source/models-and-inference); a supported third-party Realtime contract for reusing Codex tokens has not been confirmed. A successful token mint does not establish billing or support terms. diff --git a/SECURITY.md b/SECURITY.md index 4423eead3..b32c5b60f 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -85,6 +85,7 @@ The data proxies under `server/providers/` are written so the browser cannot tur Allowing `maps.googleapis.com` in `script-src` also trusts its JSONP endpoints, which can bypass the script allowlist if an HTML-injection bug exists; this is an accepted limitation of loading the Maps JavaScript API, alongside the existing `'unsafe-eval'` allowance. - **Full Content-Security-Policy.** The dev and preview servers send a real CSP (`script-src 'self' 'unsafe-eval' blob:` plus the YouTube, Facebook and X scripts the Nepal event's embedded media loads, with a matching `frame-src` for their players and posts: no inline script and no other script origin; `blob:` covers the object URLs Cesium's bundled workers bootstrap through in the built app), alongside the existing `X-Frame-Options: DENY` and `frame-ancestors 'none'`. `'unsafe-eval'` stays because Knockout, bundled inside Cesium's widgets package, resolves the global object with `eval` at load time and the globe does not initialize without it; styles permit `'unsafe-inline'` and Google Fonts because Vite's dev client injects `