feat: share the tool catalog with voice

This commit is contained in:
Sameh Khamis committed 2026-10-01 13:52:36 -07:00
1 parent 0953ebb026
commit 1a03e7483d
19 files changed
+288 -15

No files matched your search

+5 -1
View File
@@ -9,7 +9,11 @@
stdio to clients such as Claude Code, reading stdio to clients such as Claude Code, reading
from a running app. Tools are defined from a running app. Tools are defined
once in `gods-eye-view/tools`, reuse the layers' source factories, and are once in `gods-eye-view/tools`, reuse the layers' source factories, and are
exposed through the protocol adapter in `gods-eye-view/tools/mcp`. exposed through the protocol adapter in `gods-eye-view/tools/mcp`. The
catalog also covers weather maps, wind, recent imagery, submarine cables,
license plate reader cameras, datacenters and dams, the Bhote Koshi flood,
military awareness and links that open the app over an area with layers on.
Voice offers the same queries next to its app actions.
- Public Overpass instances are no longer used by default. Street Traffic - Public Overpass instances are no longer used by default. Street Traffic
roads come from TomTom flow tiles, OpenFreeMap vector tiles, or both, chosen roads come from TomTom flow tiles, OpenFreeMap vector tiles, or both, chosen
+3 -2
View File
@@ -20,8 +20,9 @@ perimeters, terrain height, military installations and map features, plus a
combined situation brief, military awareness around a point, the app's heads-up display caption, and a link that combined situation brief, military awareness around a point, the app's heads-up display caption, and a link that
opens the app over an area. Tools reuse the layers' portable source factories, take a opens the app over an area. Tools reuse the layers' portable source factories, take a
shared `area` argument (place name, bounding box, or point and radius) and cap shared `area` argument (place name, bounding box, or point and radius) and cap
lists at 25 rows by default. The app does not import the tools, so its runtime lists at 25 rows by default. Voice offers the same queries next to its app
behavior is unchanged. See [tools and the MCP server](TOOLS.md). actions: the session lists them, and the browser runs them through the same
catalog, loaded on first use. See [tools and the MCP server](TOOLS.md).
## Cyber HUD — September 23, 2026 ## Cyber HUD — September 23, 2026
+28 -3
View File
@@ -1,8 +1,8 @@
# Tools and the MCP server # Tools and the MCP server
Tools answer questions from God's Eye View data for language-model clients. Tools answer questions from God's Eye View data for language-model clients.
They are defined once and exposed through adapters; the Model Context Protocol They are defined once and exposed through adapters: the Model Context Protocol
(MCP) is the first. (MCP) and function calling, which voice uses.
## Layers ## Layers
@@ -11,13 +11,17 @@ They are defined once and exposed through adapters; the Model Context Protocol
| `src/tools/` | Tool definitions, catalog composition, argument validation, shared `area` and result helpers | | `src/tools/` | Tool definitions, catalog composition, argument validation, shared `area` and result helpers |
| `src/tools/queries/` | Queries, one file per domain, reading only portable source contracts | | `src/tools/queries/` | Queries, one file per domain, reading only portable source contracts |
| `src/tools/mcp/` | MCP protocol (JSON-RPC) and a stateless HTTP transport; knows the catalog interface, not queries | | `src/tools/mcp/` | MCP protocol (JSON-RPC) and a stateless HTTP transport; knows the catalog interface, not queries |
| `src/tools/functions.js` | Function-calling adapter: tool records and results for function-calling clients |
| `src/tools/services.js` | The default services: the layers' source factories and request services, given a resolving fetch | | `src/tools/services.js` | The default services: the layers' source factories and request services, given a resolving fetch |
| `server/mcp/` | Node composition: points the sources at a running app's `/api` routes; serves stdio and `/mcp` | | `server/mcp/` | Node composition: points the sources at a running app's `/api` routes; serves stdio and `/mcp` |
| `server/standalone/voiceTools.js`, `src/standalone/toolCatalog.js` | Standalone voice composition: the session's tool list and the browser catalog |
Dependencies point downward only. `gods-eye-view/tools` and Dependencies point downward only. `gods-eye-view/tools` and
`gods-eye-view/tools/mcp` are portable exports: they reach no application, `gods-eye-view/tools/mcp` are portable exports: they reach no application,
rendering, Node, Cesium or browser-global code, which rendering, Node, Cesium or browser-global code, which
`npm run check:boundaries` enforces. Nothing in the application imports them. `npm run check:boundaries` enforces. In the application, only voice reaches
them: `withToolCatalog` in `src/voice/gevRealtime.js` imports the
function-calling adapter, and the standalone entry supplies the catalog.
## Definitions and composition ## Definitions and composition
@@ -77,6 +81,27 @@ returns a `Request`-to-`Response` handler for stateless Streamable HTTP: one
JSON-RPC message per POST, answered with JSON. The host owns routing and any JSON-RPC message per POST, answered with JSON. The host owns routing and any
access control in front of it. access control in front of it.
## Voice
Voice offers the catalog's queries next to its app actions.
`toFunctionTools(tools, { exclude })` turns tools into
`{ type: 'function', name, description, parameters }` records, and
`toFunctionOutput(name, result)` turns a result into
`{ ok, tool, summary, data }`, counting images in `images_omitted` instead of
sending them.
The voice session token endpoint takes its tool list as `realtime.tools`.
`realtimeSessionTools(additional)` appends function tools to the app actions,
skipping names an action already uses, so `next_satellite_pass` stays the
action. The standalone server supplies every core query except
`open_in_gods_eye_view`, since voice runs inside the app.
In the browser, `initGevVoiceCommands({ toolCatalog })` takes a function that
resolves a catalog. App action names go to the action runner; other names the
catalog has go to `catalog.call` with the call's abort signal. The standalone
entry composes the catalog with `createToolServices` over the page's fetch and
loads it the first time voice calls a query.
## Running locally ## Running locally
Start the app (`npm run dev` or `npm run preview`), then register the stdio Start the app (`npm run dev` or `npm run preview`), then register the stdio
+2 -1
View File
@@ -885,5 +885,6 @@
"src/tools/queries/cables.test.mjs", "src/tools/queries/cables.test.mjs",
"src/tools/queries/surveillance.test.mjs", "src/tools/queries/surveillance.test.mjs",
"src/tools/queries/infrastructure.test.mjs", "src/tools/queries/infrastructure.test.mjs",
"src/tools/queries/events.test.mjs" "src/tools/queries/events.test.mjs",
"src/tools/functions.test.mjs"
] ]
+5 -2
View File
@@ -2036,7 +2036,9 @@
"src/data/local_data/osm_military_names/names.json", "src/data/local_data/osm_military_names/names.json",
"src/layers/installations/namedMarkers.js", "src/layers/installations/namedMarkers.js",
"src/data/infrastructureOverlayEntry.js", "src/data/infrastructureOverlayEntry.js",
"src/sources/infrastructureData.js" "src/sources/infrastructureData.js",
"src/tools/functions.js",
"src/voice/actionSchemas.js"
], ],
"external": [ "external": [
"hls.js", "hls.js",
@@ -3119,7 +3121,8 @@
"src/tools/queries/surveillance.js", "src/tools/queries/surveillance.js",
"src/tools/queries/infrastructure.js", "src/tools/queries/infrastructure.js",
"src/tools/queries/events.js", "src/tools/queries/events.js",
"src/data/bhoteKoshiFloodPath.js" "src/data/bhoteKoshiFloodPath.js",
"src/tools/functions.js"
], ],
"external": ["satellite.js"] "external": ["satellite.js"]
}, },
+6 -3
View File
@@ -26,8 +26,11 @@ import { firePerimetersProxy } from './firePerimeters.js';
import { cycloneProxy } from './cyclones.js'; import { cycloneProxy } from './cyclones.js';
import { windProxy } from './wind.js'; import { windProxy } from './wind.js';
/** Construct the local provider plugins in their established order. */ /**
function localProviderPlugins() { * Construct the local provider plugins in their established order.
* `realtime` configures the voice session token endpoint.
*/
function localProviderPlugins({ realtime } = {}) {
return [ return [
openSkyProxy(), openSkyProxy(),
celestrakProxy(), celestrakProxy(),
@@ -49,7 +52,7 @@ function localProviderPlugins() {
adsbLolProxy(), adsbLolProxy(),
aisLiveProxy(), aisLiveProxy(),
trackBackfillProxies(), trackBackfillProxies(),
openAiRealtimeProxy(), openAiRealtimeProxy({ realtime }),
googlePlacesContextProxy(), googlePlacesContextProxy(),
windProxy(), windProxy(),
weatherProxy(), weatherProxy(),
+1
View File
@@ -19,6 +19,7 @@ function realtimeInstructions(
'analyst_query ANSWERS questions; it never moves the camera or starts tracking. For requests to FOLLOW or TRACK a specific aircraft/ship, call track_entity (get_entity_context first when the target is ambiguous), never analyst_query as the final or only action. For "follow/track the nearest aircraft", first call analyst_query with the aircraft layer(s), sortBy=distance, and limit=1, then call track_entity with the returned aircraft identity in the same turn. The lookup alone does not fulfill a follow/track command.', 'analyst_query ANSWERS questions; it never moves the camera or starts tracking. For requests to FOLLOW or TRACK a specific aircraft/ship, call track_entity (get_entity_context first when the target is ambiguous), never analyst_query as the final or only action. For "follow/track the nearest aircraft", first call analyst_query with the aircraft layer(s), sortBy=distance, and limit=1, then call track_entity with the returned aircraft identity in the same turn. The lookup alone does not fulfill a follow/track command.',
'For a request to enable an aircraft layer and SELECT or FIND the nearest/closest aircraft near a named place — for example, "Turn on flights and select the closest aircraft to Austin" — call select_nearest_aircraft once. It atomically turns on the requested aircraft layer first, waits for location arrival, refreshes that layer for the destination viewport, filters out landed/on-ground records, and selects the nearest airborne result. A healthy fallback feed is valid data: report the returned feed source briefly, never call it an enable failure. Do not also call fly_to_location, set_layer_visibility, analyst_query, track_entity, set_context_mode, or control_cockpit for the same request. SELECT/FIND never implies Contacts or Cockpit unless the user explicitly asks for either mode.', 'For a request to enable an aircraft layer and SELECT or FIND the nearest/closest aircraft near a named place — for example, "Turn on flights and select the closest aircraft to Austin" — call select_nearest_aircraft once. It atomically turns on the requested aircraft layer first, waits for location arrival, refreshes that layer for the destination viewport, filters out landed/on-ground records, and selects the nearest airborne result. A healthy fallback feed is valid data: report the returned feed source briefly, never call it an enable failure. Do not also call fly_to_location, set_layer_visibility, analyst_query, track_entity, set_context_mode, or control_cockpit for the same request. SELECT/FIND never implies Contacts or Cockpit unless the user explicitly asks for either mode.',
'For ANALYTICAL questions about layer data — how many / which / fastest / highest / biggest / nearest flights, ships, fires, earthquakes, satellites, datacenters, or dams ("how many flights over Texas", "biggest fire near LA", "which ships are headed to Oakland", "anything above 40,000 feet") — call analyst_query, not get_entity_context. Narrate the count plus two or three notable examples by name, and reflect the result\'s coverage note honestly: the answer covers data loaded by enabled layers, not the whole world. If the needed layer is disabled, say so and offer to enable it. For follow-ups about the same set ("which of THOSE is closest?"), call analyst_query with followUp=true and only the new filter/sort.', 'For ANALYTICAL questions about layer data — how many / which / fastest / highest / biggest / nearest flights, ships, fires, earthquakes, satellites, datacenters, or dams ("how many flights over Texas", "biggest fire near LA", "which ships are headed to Oakland", "anything above 40,000 feet") — call analyst_query, not get_entity_context. Narrate the count plus two or three notable examples by name, and reflect the result\'s coverage note honestly: the answer covers data loaded by enabled layers, not the whole world. If the needed layer is disabled, say so and offer to enable it. For follow-ups about the same set ("which of THOSE is closest?"), call analyst_query with followUp=true and only the new filter/sort.',
'WORLD QUERIES — the query tools (get_weather, aircraft_in_area, vessels_in_area, get_earthquakes, search_places, plan_route, situation_brief, military_awareness and the other get_*, find_* and *_in_area tools) look up data anywhere in the world without changing the map and without any layer being on. Use them when the user asks about a place they are not looking at or data whose layer is off ("what is the weather in Tokyo", "any ships near Singapore", "how do I drive from Austin to Dallas"). Use analyst_query and get_entity_context for what the app has loaded or what is on screen. Answer from the result summary and data, mention its source when it has one, and never claim the map changed. If the user also wants to SEE it, use the matching app action (fly_to_location, set_layer_visibility) as well.',
'FEED STATE CONTRACT — analyst_query and get_current_view_state carry feedProvenance from the same layerFeedState the Data Layers chips use (nominal/loading/degraded/stale/fallback/unavailable, or off). NEVER narrate a count as live or current when feedProvenance.overall is not nominal. Say the feedState in the same breath as the count ("12 flights in view, STALE"). If unavailable, do not invent a count; say the feed is unavailable. Fallback is valid data — name the fallback source, never call it an enable failure. Confirm only what the tool result returned.', 'FEED STATE CONTRACT — analyst_query and get_current_view_state carry feedProvenance from the same layerFeedState the Data Layers chips use (nominal/loading/degraded/stale/fallback/unavailable, or off). NEVER narrate a count as live or current when feedProvenance.overall is not nominal. Say the feedState in the same breath as the count ("12 flights in view, STALE"). If unavailable, do not invent a count; say the feed is unavailable. Fallback is valid data — name the fallback source, never call it an enable failure. Confirm only what the tool result returned.',
'COUNTING CONTRACT — what "near" means. (1) While Contacts is ACTIVE, "near / nearby / how many aircraft" means the Contacts window: answer from contactsWindow in the tool result — those are the exact numbers on the user\'s panel. set_context_mode, analyst_query, and get_current_view_state carry it after Contacts settles. For "Open Contacts and tell me how many aircraft are within 250 km", call set_context_mode{mode:"contacts"} first and answer from contactsWindow.aircraft; do not answer from a pre-Contacts analyst query. analyst_query\'s own count measures currently-loaded records and is usually lower; never give it as the window count. CENTER PRECEDENCE for a nearby/how-many ask, in order: an explicit place in the question ("over Texas", "near Austin") always wins and ignores Contacts state; else the CONTACTS SUBJECT when Contacts is active and has one — a selected datacenter, dam, fire, or cable does NOT silently become the center; else an entity the user explicitly names ("around this datacenter"); else the current view, said aloud ("nothing is selected, so this is the current view"). With Contacts active but NO subject yet, use the view and say so; never read an empty panel. (2) With Contacts OFF, "nearby" means in view; "near <place>" means a radius around that place. (3) EVERY count names its scope in words — "42 in your window", "8 in view", "about 30 within 250 km of Austin" — never a bare number; analyst_query returns scopeLabel for exactly this. Two different numbers with named scopes are not a contradiction; say both if asked. (4) State counts VERBATIM — never estimate, round, or hedge ("a few", "less than a dozen"): if a tool returns 46, say 46. (5) When it matters, add once: counts cover loaded data, and the flights layer loads where you look.', 'COUNTING CONTRACT — what "near" means. (1) While Contacts is ACTIVE, "near / nearby / how many aircraft" means the Contacts window: answer from contactsWindow in the tool result — those are the exact numbers on the user\'s panel. set_context_mode, analyst_query, and get_current_view_state carry it after Contacts settles. For "Open Contacts and tell me how many aircraft are within 250 km", call set_context_mode{mode:"contacts"} first and answer from contactsWindow.aircraft; do not answer from a pre-Contacts analyst query. analyst_query\'s own count measures currently-loaded records and is usually lower; never give it as the window count. CENTER PRECEDENCE for a nearby/how-many ask, in order: an explicit place in the question ("over Texas", "near Austin") always wins and ignores Contacts state; else the CONTACTS SUBJECT when Contacts is active and has one — a selected datacenter, dam, fire, or cable does NOT silently become the center; else an entity the user explicitly names ("around this datacenter"); else the current view, said aloud ("nothing is selected, so this is the current view"). With Contacts active but NO subject yet, use the view and say so; never read an empty panel. (2) With Contacts OFF, "nearby" means in view; "near <place>" means a radius around that place. (3) EVERY count names its scope in words — "42 in your window", "8 in view", "about 30 within 250 km of Austin" — never a bare number; analyst_query returns scopeLabel for exactly this. Two different numbers with named scopes are not a contradiction; say both if asked. (4) State counts VERBATIM — never estimate, round, or hedge ("a few", "less than a dozen"): if a tool returns 46, say 46. (5) When it matters, add once: counts cover loaded data, and the flights layer loads where you look.',
'While Cockpit is active, navigate with control_cockpit (next/previous, optionally targetLayer or aircraftClass). track_entity and fly_to_location are REFUSED by design while Cockpit owns the camera — that refusal is correct, not an error to retry. To go somewhere else, exit Cockpit first. control_cockpit enter establishes Contacts itself, so do not call set_context_mode before or after it.', 'While Cockpit is active, navigate with control_cockpit (next/previous, optionally targetLayer or aircraftClass). track_entity and fly_to_location are REFUSED by design while Cockpit owns the camera — that refusal is correct, not an error to retry. To go somewhere else, exit Cockpit first. control_cockpit enter establishes Contacts itself, so do not call set_context_mode before or after it.',
+2 -1
View File
@@ -20,6 +20,7 @@ function createRealtimeTokenHandler({
fetchImpl = (...args) => fetch(...args), fetchImpl = (...args) => fetch(...args),
resolveApiKey = () => process.env.OPENAI_API_KEY, resolveApiKey = () => process.env.OPENAI_API_KEY,
models = {}, models = {},
tools = GEV_REALTIME_TOOLS,
} = {}) { } = {}) {
return async (req, res) => { return async (req, res) => {
res.setHeader('Cache-Control', 'no-store'); res.setHeader('Cache-Control', 'no-store');
@@ -113,7 +114,7 @@ function createRealtimeTokenHandler({
output: { voice }, output: { voice },
}, },
instructions: realtimeInstructions(annotationGuidance), instructions: realtimeInstructions(annotationGuidance),
tools: GEV_REALTIME_TOOLS, tools,
tool_choice: 'auto', tool_choice: 'auto',
}, },
}; };
+12
View File
@@ -2,3 +2,15 @@ import { createActionTools } from '../../../src/voice/actionSchemas.js';
import { ACTION_DESCRIPTIONS } from './toolDescriptions.js'; import { ACTION_DESCRIPTIONS } from './toolDescriptions.js';
export const GEV_REALTIME_TOOLS = createActionTools(ACTION_DESCRIPTIONS); export const GEV_REALTIME_TOOLS = createActionTools(ACTION_DESCRIPTIONS);
/**
* The session's tools: the app actions, then additional function tools
* whose names no action already uses.
*/
export function realtimeSessionTools(additional = []) {
const names = new Set(GEV_REALTIME_TOOLS.map((tool) => tool.name));
return [
...GEV_REALTIME_TOOLS,
...additional.filter((tool) => !names.has(tool.name)),
];
}
+6 -1
View File
@@ -4,6 +4,7 @@ import { createBrowserViteConfig } from '../../build/vite.js';
import { localProviderPlugins } from '../providers/local.js'; import { localProviderPlugins } from '../providers/local.js';
import { localMcpPlugin } from '../mcp/plugin.js'; import { localMcpPlugin } from '../mcp/plugin.js';
import { apiNotFoundPlugin } from './api-not-found.js'; import { apiNotFoundPlugin } from './api-not-found.js';
import { standaloneVoiceTools } from './voiceTools.js';
const root = fileURLToPath(new URL('../../', import.meta.url)); const root = fileURLToPath(new URL('../../', import.meta.url));
@@ -14,7 +15,11 @@ export default defineConfig(({ command, mode }) => {
if (process.env[key] === undefined) process.env[key] = value; if (process.env[key] === undefined) process.env[key] = value;
} }
return createBrowserViteConfig({ return createBrowserViteConfig({
plugins: [...localProviderPlugins(), localMcpPlugin(), apiNotFoundPlugin()], plugins: [
...localProviderPlugins({ realtime: { tools: standaloneVoiceTools() } }),
localMcpPlugin(),
apiNotFoundPlugin(),
],
googleApiKey: process.env.GOOGLE_MAPS_API_KEY, googleApiKey: process.env.GOOGLE_MAPS_API_KEY,
cesiumToken: process.env.CESIUM_ION_TOKEN, cesiumToken: process.env.CESIUM_ION_TOKEN,
host: process.env.HOST, host: process.env.HOST,
+16
View File
@@ -0,0 +1,16 @@
/**
* The voice session's tools for the standalone server: the app actions plus
* Core's catalog queries, which the browser runs through the same catalog.
* Links to the app are left out, since voice runs inside it.
*/
import { coreTools, toFunctionTools } from '../../src/tools/index.js';
import { realtimeSessionTools } from '../providers/openai/tools.js';
export const VOICE_EXCLUDED_QUERIES = Object.freeze(['open_in_gods_eye_view']);
export function standaloneVoiceTools() {
return realtimeSessionTools(
toFunctionTools(coreTools, { exclude: VOICE_EXCLUDED_QUERIES }),
);
}
+22
View File
@@ -0,0 +1,22 @@
/** The standalone tool catalog for voice, loaded the first time voice needs it. */
let pending = null;
export function loadToolCatalog() {
pending ??= Promise.all([
import('../tools/index.js'),
import('../tools/services.js'),
]).then(([{ composeCatalog, coreTools }, { createToolServices }]) =>
composeCatalog({
tools: coreTools,
services: createToolServices({
fetchImpl: (...args) => globalThis.fetch(...args),
appUrl: globalThis.location.origin,
}),
}),
);
pending.catch(() => {
pending = null;
});
return pending;
}
+2
View File
@@ -1,6 +1,7 @@
import { createAssetDirectorySource } from '../director/packs/source.js'; import { createAssetDirectorySource } from '../director/packs/source.js';
import { createApplicationTools } from '../app/tools.js'; import { createApplicationTools } from '../app/tools.js';
import { startStandaloneChrome } from './startupChrome.js'; import { startStandaloneChrome } from './startupChrome.js';
import { loadToolCatalog } from './toolCatalog.js';
export function createStandaloneTools(options) { export function createStandaloneTools(options) {
return createApplicationTools({ return createApplicationTools({
startChrome: startStandaloneChrome, startChrome: startStandaloneChrome,
@@ -12,5 +13,6 @@ export function createStandaloneTools(options) {
}, },
}, },
...options, ...options,
voice: { toolCatalog: loadToolCatalog, ...options?.voice },
}); });
} }
+33
View File
@@ -21,6 +21,7 @@ import { openAiRealtimeProxy } from 'gods-eye-view/server/providers/openai';
import { keySetupEndpoint } from 'gods-eye-view/server/standalone/key-setup'; import { keySetupEndpoint } from 'gods-eye-view/server/standalone/key-setup';
import { realtimeInstructions } from '../../server/providers/openai/instructions.js'; import { realtimeInstructions } from '../../server/providers/openai/instructions.js';
import { GEV_REALTIME_TOOLS } from '../../server/providers/openai/tools.js'; import { GEV_REALTIME_TOOLS } from '../../server/providers/openai/tools.js';
import { standaloneVoiceTools } from '../../server/standalone/voiceTools.js';
function install(plugin, preview = false) { function install(plugin, preview = false) {
const routes = new Map(); const routes = new Map();
@@ -195,6 +196,38 @@ test('Realtime handler preserves tools and default instructions, isolates suppli
assert.equal(sent[0].session.instructions, sent[2].session.instructions); assert.equal(sent[0].session.instructions, sent[2].session.instructions);
}); });
test('Realtime sessions carry supplied tools, and the standalone voice adds the catalog queries', async (t) => {
env(t, 'OPENAI_API_KEY', 'fixture-upstream-secret');
env(t, 'GEV_RATELIMIT_OPENAI_PER_MIN', undefined);
const sent = [];
t.mock.method(globalThis, 'fetch', async (url, options) => {
sent.push(JSON.parse(options.body));
return Response.json({ value: 'fixture-ephemeral' });
});
const tools = standaloneVoiceTools();
const response = await request(
install(openAiRealtimeProxy({ realtime: { tools } })).get(
'/api/realtime/token',
),
);
assert.equal(response.status, 200);
assert.deepEqual(sent.at(-1).session.tools, tools);
const names = tools.map((tool) => tool.name);
assert.equal(new Set(names).size, names.length);
assert.deepEqual(
tools.slice(0, GEV_REALTIME_TOOLS.length),
GEV_REALTIME_TOOLS,
);
assert.ok(names.includes('get_weather'));
assert.ok(names.includes('military_awareness'));
assert.ok(!names.includes('open_in_gods_eye_view'));
// The action of the same name answers satellite passes.
assert.equal(
tools.findLast((tool) => tool.name === 'next_satellite_pass'),
GEV_REALTIME_TOOLS.find((tool) => tool.name === 'next_satellite_pass'),
);
});
test('debug logging resolves each supplied application directory independently', async (t) => { test('debug logging resolves each supplied application directory independently', async (t) => {
const first = root(t), const first = root(t),
second = root(t); second = root(t);
+28
View File
@@ -0,0 +1,28 @@
/** Function-calling adapter for a tool catalog: tool records and results. */
/** Tools as `{ type: 'function', name, description, parameters }` records. */
export function toFunctionTools(tools, { exclude = [] } = {}) {
const skipped = new Set(exclude);
return tools
.filter((tool) => !skipped.has(tool.name))
.map((tool) => ({
type: 'function',
name: tool.name,
description: tool.description,
parameters: structuredClone(tool.inputSchema),
}));
}
/**
* A tool result as one JSON function output. Images are counted rather than
* sent, since function outputs carry text.
*/
export function toFunctionOutput(name, result) {
return {
ok: true,
tool: name,
summary: result.summary,
data: result.data,
...(result.images?.length ? { images_omitted: result.images.length } : {}),
};
}
+50
View File
@@ -0,0 +1,50 @@
import assert from 'node:assert/strict';
import test from 'node:test';
import { coreTools, toFunctionOutput, toFunctionTools } from './index.js';
test('catalog tools become independent function records', () => {
const records = toFunctionTools(coreTools, {
exclude: ['open_in_gods_eye_view'],
});
assert.equal(records.length, coreTools.length - 1);
assert.ok(!records.some((tool) => tool.name === 'open_in_gods_eye_view'));
const weather = records.find((tool) => tool.name === 'get_weather');
const source = coreTools.find((tool) => tool.name === 'get_weather');
assert.deepEqual(Object.keys(weather), [
'type',
'name',
'description',
'parameters',
]);
assert.equal(weather.type, 'function');
assert.equal(weather.description, source.description);
assert.deepEqual(weather.parameters, source.inputSchema);
assert.notEqual(weather.parameters, source.inputSchema);
weather.parameters.properties.location.description = 'changed';
assert.notEqual(
source.inputSchema.properties.location.description,
'changed',
);
assert.ok(records.every((tool) => tool.parameters.type === 'object'));
});
test('results become function outputs without image data', () => {
assert.deepEqual(
toFunctionOutput('get_wind', { summary: 'Calm.', data: { calm: true } }),
{ ok: true, tool: 'get_wind', summary: 'Calm.', data: { calm: true } },
);
assert.deepEqual(
toFunctionOutput('get_weather_map', {
summary: 'Radar.',
data: {},
images: [{ mimeType: 'image/png', data: 'AAAA' }],
}),
{
ok: true,
tool: 'get_weather_map',
summary: 'Radar.',
data: {},
images_omitted: 1,
},
);
});
+1
View File
@@ -67,6 +67,7 @@ export {
distanceKm, distanceKm,
} from './area.js'; } from './area.js';
export { LIMIT_SCHEMA, DEFAULT_LIMIT, MAX_LIMIT, capRows } from './results.js'; export { LIMIT_SCHEMA, DEFAULT_LIMIT, MAX_LIMIT, capRows } from './results.js';
export { toFunctionOutput, toFunctionTools } from './functions.js';
export { export {
createGeocodePlaceService, createGeocodePlaceService,
createPlaceSearchService, createPlaceSearchService,
+25 -1
View File
@@ -1,11 +1,35 @@
import { toFunctionOutput } from '../tools/functions.js';
import { GEV_ACTION_SCHEMAS } from './actionSchemas.js';
import { createGevActionRunner } from './gevActions.js'; import { createGevActionRunner } from './gevActions.js';
import { createVoiceCommands } from './commands.js'; import { createVoiceCommands } from './commands.js';
export * from './realtimeController.js'; export * from './realtimeController.js';
const ACTION_NAMES = new Set(GEV_ACTION_SCHEMAS.map((schema) => schema.name));
/**
* Run app actions through `runner` and every other tool the catalog has
* through the catalog. `loadCatalog` resolves the catalog when first needed.
*/
export function withToolCatalog(runner, loadCatalog) {
if (typeof loadCatalog !== 'function') return runner;
return async function runGevTool(name, args, options = {}) {
if (ACTION_NAMES.has(name)) return runner(name, args, options);
const catalog = await loadCatalog();
if (!catalog?.get(name)) return runner(name, args, options);
const result = await catalog.call(name, args ?? {}, {
signal: options.signal,
});
return toFunctionOutput(name, result);
};
}
/** Compose the standalone action runner with the voice controls. */ /** Compose the standalone action runner with the voice controls. */
export function initGevVoiceCommands(options) { export function initGevVoiceCommands(options) {
return createVoiceCommands({ return createVoiceCommands({
...options, ...options,
runner: createGevActionRunner(options), runner: withToolCatalog(
createGevActionRunner(options),
options.toolCatalog,
),
}); });
} }
+41
View File
@@ -31,6 +31,7 @@ import {
readStoredVoiceLimits, readStoredVoiceLimits,
writeStoredVoiceTier, writeStoredVoiceTier,
writeStoredVoiceLimits, writeStoredVoiceLimits,
withToolCatalog,
} from './gevRealtime.js'; } from './gevRealtime.js';
import { createVoiceCostTracker } from './voiceCost.js'; import { createVoiceCostTracker } from './voiceCost.js';
@@ -3681,3 +3682,43 @@ test('a genuinely different refused call still gets its own output', async () =>
const testPlaceSearch = () => createStandalonePlaceSearch({ resolveApiKey: () => globalThis.window?.__GOOGLE_MAPS_API_KEY__ }); const testPlaceSearch = () => createStandalonePlaceSearch({ resolveApiKey: () => globalThis.window?.__GOOGLE_MAPS_API_KEY__ });
function createGevActionRunner(options) { return createActionRunner({ placeSearch: testPlaceSearch(), ...options }); } function createGevActionRunner(options) { return createActionRunner({ placeSearch: testPlaceSearch(), ...options }); }
function controlRadio(viewer, manager, args, options) { return runControlRadio(viewer, manager, args, { placeSearch: testPlaceSearch(), ...options }); } function controlRadio(viewer, manager, args, options) { return runControlRadio(viewer, manager, args, { placeSearch: testPlaceSearch(), ...options }); }
test('voice runs app actions itself and other tools through the catalog', async () => {
const actions = [];
const runner = async (name, args) => {
actions.push([name, args]);
return { ok: true, action: name };
};
const calls = [];
let loads = 0;
const catalog = {
get: (name) => (name === 'get_wind' ? { name } : undefined),
async call(name, args, options) {
calls.push([name, args, options.signal]);
return { summary: 'Calm.', data: { calm: true } };
},
};
const run = withToolCatalog(runner, async () => {
loads += 1;
return catalog;
});
const signal = new AbortController().signal;
assert.deepEqual(await run('zoom_to_globe', {}), {
ok: true,
action: 'zoom_to_globe',
});
assert.equal(loads, 0);
assert.deepEqual(
await run('get_wind', { location: { place: 'Oslo' } }, { signal }),
{ ok: true, tool: 'get_wind', summary: 'Calm.', data: { calm: true } },
);
assert.deepEqual(calls, [
['get_wind', { location: { place: 'Oslo' } }, signal],
]);
await run('not_a_tool', {});
assert.deepEqual(
actions.map(([name]) => name),
['zoom_to_globe', 'not_a_tool'],
);
assert.equal(withToolCatalog(runner, undefined), runner);
});