From feba5af0b98026c924decbc21737622125810bb5 Mon Sep 17 00:00:00 2001 From: Tom-Neverwinter <3943634+Tom-Neverwinter@users.noreply.github.com> Date: Sat, 29 Aug 2026 08:05:56 -0500 Subject: [PATCH 01/21] Add OpenSeaMap / OpenSnowMap overlay layers (#10) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two independently toggleable open-data raster overlays — OpenSeaMap sea marks (buoys, lighthouses, ports) and OpenSnowMap ski pistes — drawn as Cesium.ImageryLayers directly on the Cesium globe surface. Same "small array of descriptors" shape as MAP_STACKS in mapStackController.js. Visible on the osm/bing-aerial/bing-labels map stacks and invisible (not broken — the layer's stats say so in plain text) on the default photoreal stack, since Google Photorealistic 3D Tiles render with the globe hidden. OpenRailwayMap and OpenInfraMap, also requested in #10, are deliberately NOT included in this pass: OpenRailwayMap's public tile server enforces a Referer allowlist scoped to its own site (confirmed by direct request — every other Referer, including none, 403s), and OpenInfraMap serves MapLibre GL vector tiles, not raster XYZ — that would need a genuine vector-tile rendering pipeline, not a Cesium.ImageryLayer drop-in. Worth its own dedicated pass. Co-Authored-By: Claude Sonnet 5 --- DATA_SOURCES.md | 3 + src/data/dataCredits.js | 12 +++ src/data/layerState.js | 9 ++ src/data/layerState.test.mjs | 4 +- src/data/osmOverlays.js | 191 ++++++++++++++++++++++++++++++++++ src/data/osmOverlays.test.mjs | 41 ++++++++ src/standalone/data.js | 2 + src/voice/gevActions.js | 10 ++ 8 files changed, 270 insertions(+), 2 deletions(-) create mode 100644 src/data/osmOverlays.js create mode 100644 src/data/osmOverlays.test.mjs diff --git a/DATA_SOURCES.md b/DATA_SOURCES.md index faa351dca..7f5cdbdad 100644 --- a/DATA_SOURCES.md +++ b/DATA_SOURCES.md @@ -37,10 +37,13 @@ How to read this: | **GBFS (Lyft / BCycle)** | Bikeshare availability | Per-feed (attribution-only) | Credit the operator (e.g. Austin BCycle) + its `license_url` | | **Radio Browser** | Geolocated internet-radio station directory and station-level tags | Public-domain directory data under PDDL 1.0; individual broadcaster stream terms apply | "Radio Browser" plus a link to the selected broadcaster | | **Re:Earth Terrain** (Mapterhorn) | Terrain (keyless globe stacks — OSM etc. — + `/api/terrain/heights` ellipsoidal-height lookups) | Terrain mesh: CC BY 4.0; geoid: EGM2008 (NGA, public domain) | "Terrain (keyless globe stacks): Re:Earth Terrain / Mapterhorn (CC BY 4.0) / EGM2008 (NGA)" | +| **OpenSeaMap** (`tiles.openseamap.org`) | Sea marks raster overlay (buoys, lighthouses, ports) — one of two Open Map Overlays (#10) | Map data ODbL 1.0 via OpenStreetMap | "© OpenSeaMap contributors" | +| **OpenSnowMap** (`opensnowmap.org`) | Ski piste raster overlay — one of two Open Map Overlays (#10) | Map data ODbL 1.0 (OpenStreetMap) + SRTM; map style CC BY-SA | "Map data: © OpenStreetMap contributors, SRTM \| Map style: © OpenSnowMap.org (CC-BY-SA)" | ### Notes on the live sources - **Google Maps Platform.** You supply your own API key and are bound by [Google's ToS](https://cloud.google.com/maps-platform/terms). Google Maps Content (tiles, geocodes, places) **may not be cached, stored, rehosted, or committed** — this app only ever uses it live, which is the compliant pattern. The "Google" attribution is displayed on the globe and must stay visible. Restrict your key (see [SECURITY.md](SECURITY.md)). +- **Open Map Overlays (#10).** Two independently toggleable keyless raster overlays — OpenSeaMap sea marks and OpenSnowMap ski pistes — drawn as `Cesium.ImageryLayer`s directly on the Cesium globe surface. Like every globe-surface source, they render on the `osm`/`bing-aerial`/`bing-labels` map stacks and are invisible on the default `photoreal` stack (Google Photorealistic 3D Tiles render with the globe hidden; see `mapStackController.js`) — the layer's stats surface that as plain text instead of looking broken. OpenRailwayMap and OpenInfraMap, also requested in #10, are not included: OpenRailwayMap's public tile server enforces a Referer allowlist restricted to its own site (confirmed by direct request — any other Referer, including none, 403s), and OpenInfraMap serves MapLibre vector tiles rather than raster XYZ tiles, which would need a genuine vector-tile rendering pipeline, not an imagery-layer drop-in. - **OpenSky Network.** Its license is **non-commercial**, and operational use of the REST API in a live product can require a prior written agreement with OpenSky — even for non-profit/government use. If you deploy this commercially, contact OpenSky for your own terms. The flights layer is a toggle and runs anonymously by default. - **adsb.lol flight fallback.** When OpenSky is unavailable and no last-good OpenSky response exists, the server requests a cached, capped 250 nm adsb.lol point snapshot around the current camera subpoint. This is regional observed context, not worldwide completeness; provenance is exposed in the Flights stats/context row. Military ICAOs remain reconciled through the existing dedicated military registry rather than duplicated. - **Launch Library 2.** `/api/launches` makes a server-side rolling-30-day query against the supported v2.3 detailed launch endpoint, caches successful responses for 15 minutes in memory and on disk, and serves the last successful response during a throttle or transient outage. Anonymous access is limited to 15 calls/hour; deployments can provide `LL2_API_TOKEN` for authenticated access. The Space Devs' published terms permit using and sharing the API data in any form, ask users not to forward it without adding value, disclaim complete accuracy, and encourage—but do not require—attribution. This app keeps a courtesy credit. Payload and stage/recovery records are shown only when supplied. Failed launches expose their source status and never receive fallback orbit geometry or a live/estimated marker. LL2 supplies launch context and event timing, not continuous ascent telemetry or live orbital state. diff --git a/src/data/dataCredits.js b/src/data/dataCredits.js index b35da9e00..5cd3d1b66 100644 --- a/src/data/dataCredits.js +++ b/src/data/dataCredits.js @@ -66,6 +66,18 @@ export const DATA_CREDITS = [ key: 'usgs', html: 'Earthquakes: Data courtesy of the U.S. Geological Survey', }, + { + key: 'openseamap', + html: + 'Sea marks overlay: ' + + '© OpenSeaMap contributors', + }, + { + key: 'opensnowmap', + html: + 'Ski pistes overlay: Map data © OpenStreetMap contributors, SRTM | Map style © ' + + 'OpenSnowMap.org (CC BY-SA)', + }, { key: 'overpass', html: diff --git a/src/data/layerState.js b/src/data/layerState.js index eb68c14e7..c209634f6 100644 --- a/src/data/layerState.js +++ b/src/data/layerState.js @@ -223,6 +223,14 @@ const OPTION_GROUPS = Object.freeze({ booleanOption('showProjection', 'p', true), booleanOption('autoHop', 'a', false), ]), + // Two independently toggleable open-data raster overlays (#10) sharing one + // owner — same "one master row, several booleans" shape as `cctv`/`radio` + // above, not a new mechanism. Both default ON: switching the row on with + // nothing yet chosen would show nothing and read as broken. + osmOverlays: Object.freeze([ + booleanOption('seamap', 's', true), + booleanOption('snowmap', 'n', true), + ]), radio: Object.freeze([ Object.freeze({ key: 'filter', @@ -286,6 +294,7 @@ export const LAYER_STATE_REGISTRY = Object.freeze([ Object.freeze({ id: 'military', token: 'm', disposition: 'enabled+mirrored-options', optionOwner: 'flights' }), Object.freeze({ id: 'military-awareness', token: 'g', disposition: 'enabled-only' }), Object.freeze({ id: 'military-installations', token: 'i', disposition: 'enabled-only' }), + Object.freeze({ id: 'osm-overlays', token: 'j', disposition: 'enabled+options', optionOwner: 'osmOverlays' }), Object.freeze({ id: 'radio', token: 'r', disposition: 'enabled+options', optionOwner: 'radio' }), Object.freeze({ id: 'rocket-launches', token: 'x', disposition: 'enabled-only' }), Object.freeze({ id: 'satellites', token: 's', disposition: 'enabled+options', optionOwner: 'satellites' }), diff --git a/src/data/layerState.test.mjs b/src/data/layerState.test.mjs index 2c282e75e..2cb4ed2bf 100644 --- a/src/data/layerState.test.mjs +++ b/src/data/layerState.test.mjs @@ -155,8 +155,8 @@ function encode(state) { test('production registry is exact, canonical, and rejects incomplete contracts', async () => { assert.equal(validateLayerStateRegistry(), true); - assert.equal(REGISTERED_LAYER_IDS.length, 16); - assert.equal(new Set(REGISTERED_LAYER_IDS).size, 16); + assert.equal(REGISTERED_LAYER_IDS.length, 17); + assert.equal(new Set(REGISTERED_LAYER_IDS).size, 17); assert.deepEqual(REGISTERED_LAYER_IDS, [...REGISTERED_LAYER_IDS].sort()); assert.throws( () => validateLayerStateRegistry([...LAYER_STATE_REGISTRY, LAYER_STATE_REGISTRY[0]]), diff --git a/src/data/osmOverlays.js b/src/data/osmOverlays.js new file mode 100644 index 000000000..7183813fe --- /dev/null +++ b/src/data/osmOverlays.js @@ -0,0 +1,191 @@ +import * as Cesium from 'cesium'; + +/** + * @file Open-data raster overlays drawn on the Cesium GLOBE surface — + * OpenSeaMap sea marks and OpenSnowMap ski pistes (#10). Two independently + * toggleable overlays sharing one config-driven module, same "small array of + * descriptors" shape as `MAP_STACKS` in mapStackController.js. + * + * OpenRailwayMap and OpenInfraMap, also named in #10, are deliberately NOT + * included in this pass: + * - OpenRailwayMap's public tile server (tiles.openrailwaymap.org) enforces + * a Referer allowlist scoped to openrailwaymap.org itself — verified by + * direct request: a Referer of `https://www.openrailwaymap.org/` returns + * the tile, every other Referer (including none) returns 403, regardless + * of User-Agent. A browser tab on this app sends ITS OWN origin as + * Referer, so the tiles would 403 in practice, and embedding them here + * would run against their stated tile-usage policy either way + * (CONTRIBUTING.md: "Don't add scraping of sources whose terms forbid it"). + * - OpenInfraMap serves MapLibre GL vector tiles (MVT) from its own + * infrastructure, not simple raster XYZ tiles — adding it means a genuine + * vector-tile rendering pipeline, not a `Cesium.ImageryLayer` drop-in. + * Worth its own dedicated pass; out of scope here. + * + * Like every non-photoreal-stack raster source, these overlays draw on the + * Cesium globe (`viewer.scene.globe`), not on the Google Photorealistic 3D + * Tiles mesh (see `mapStackController.js`: the photoreal stack renders with + * `globe.show = false`, and a hidden globe requests and draws no imagery). + * They're visible on the `osm`/`bing-aerial`/`bing-labels` map stacks and + * invisible — not broken, just nothing to draw on — on the default + * `photoreal` stack. `getStats().error` surfaces that in plain language + * whenever an enabled overlay is currently showing nothing for this reason. + * @module data/osmOverlays + */ + +export const OSM_OVERLAYS_LAYER_ID = 'osm-overlays'; + +/** @type {Array<{key:string,label:string,icon:string,urlTemplate:string,credit:string,maximumLevel:number}>} */ +export const OSM_OVERLAYS = Object.freeze([ + Object.freeze({ + key: 'seamap', + label: 'Sea Marks', + icon: '⚓', + // Confirmed live 2026-08-25: https://tiles.openseamap.org/seamark/{z}/{x}/{y}.png + urlTemplate: 'https://tiles.openseamap.org/seamark/{z}/{x}/{y}.png', + credit: '© OpenSeaMap contributors', + maximumLevel: 18, + }), + Object.freeze({ + key: 'snowmap', + label: 'Ski Pistes', + icon: '⛷', + // Confirmed live 2026-08-25: https://www.opensnowmap.org/pistes/{z}/{x}/{y}.png + urlTemplate: 'https://www.opensnowmap.org/pistes/{z}/{x}/{y}.png', + credit: 'Map data: © OpenStreetMap contributors, SRTM | Map style: © OpenSnowMap.org (CC-BY-SA)', + maximumLevel: 18, + }), +]); + +const DEFAULT_PARAMS = Object.freeze( + Object.fromEntries(OSM_OVERLAYS.map((overlay) => [overlay.key, true])), +); + +/** Reason an enabled, chosen overlay is currently showing nothing, or null. */ +export function osmOverlaysHiddenReason(enabled, params, globeShown) { + if (!enabled) return null; + const anyChosen = OSM_OVERLAYS.some((overlay) => Boolean(params?.[overlay.key])); + if (!anyChosen) return null; + if (globeShown) return null; + return 'Overlays draw on the globe surface — switch the map stack to OSM or Bing (not Google 3D) to see them'; +} + +/** + * Sanitize a candidate params object against the two known overlay keys. + * Unknown keys are ignored; a non-boolean value for a known key leaves that + * key's current value untouched rather than coercing it (mirrors the + * `undefined` = "not requested" convention `satellites.js`/`radio.js` use). + * @param {object} current — the module's live `{seamap, snowmap}` state + * @param {object} candidate + * @returns {{next: object, changed: boolean}} + */ +export function applyOsmOverlayParams(current, candidate = {}) { + const next = { ...current }; + let changed = false; + for (const overlay of OSM_OVERLAYS) { + if (!Object.hasOwn(candidate, overlay.key)) continue; + const value = candidate[overlay.key]; + if (typeof value !== 'boolean') continue; + if (next[overlay.key] !== value) changed = true; + next[overlay.key] = value; + } + return { next, changed }; +} + +const state = { + viewer: null, + enabled: false, + params: { ...DEFAULT_PARAMS }, + imageryLayers: new Map(), +}; + +function syncImageryLayerVisibility() { + for (const overlay of OSM_OVERLAYS) { + const layer = state.imageryLayers.get(overlay.key); + if (layer) layer.show = state.enabled && Boolean(state.params[overlay.key]); + } +} + +const osmOverlaysLayer = { + id: OSM_OVERLAYS_LAYER_ID, + name: 'Open Map Overlays', + icon: '🗺️', + source: 'OpenSeaMap / OpenSnowMap', + updateInterval: 0, + init(viewer) { + state.viewer = viewer; + for (const overlay of OSM_OVERLAYS) { + const provider = new Cesium.UrlTemplateImageryProvider({ + url: overlay.urlTemplate, + credit: overlay.credit, + maximumLevel: overlay.maximumLevel, + }); + const layer = new Cesium.ImageryLayer(provider); + layer.show = false; + // Appended (no index) so it draws on top of the base map-stack imagery + // layer mapStackController.js keeps at index 0. + viewer.imageryLayers.add(layer); + state.imageryLayers.set(overlay.key, layer); + } + }, + enable() { + state.enabled = true; + syncImageryLayerVisibility(); + }, + disable() { + state.enabled = false; + syncImageryLayerVisibility(); + }, + update() { + return true; + }, + setParams(params = {}) { + const { next, changed } = applyOsmOverlayParams(state.params, params); + if (changed) { + state.params = next; + syncImageryLayerVisibility(); + } + return true; + }, + getParams() { + return { ...state.params }; + }, + getRowControls() { + if (!state.enabled) return { chips: [], legend: [] }; + const globeShown = state.viewer?.scene?.globe?.show !== false; + return { + chips: OSM_OVERLAYS.map((overlay) => { + const active = Boolean(state.params[overlay.key]); + return { + id: overlay.key, + label: `${overlay.icon} ${overlay.label}`, + active, + title: globeShown + ? overlay.credit + : `${overlay.credit} — needs the OSM or Bing map stack to render, not Google 3D`, + params: { [overlay.key]: !active }, + }; + }), + legend: [], + }; + }, + destroy(viewer) { + this.disable(); + for (const layer of state.imageryLayers.values()) { + if (viewer) viewer.imageryLayers.remove(layer, true); + } + state.imageryLayers.clear(); + state.viewer = null; + state.params = { ...DEFAULT_PARAMS }; + }, + getStats() { + const globeShown = state.viewer?.scene?.globe?.show !== false; + return { + count: state.enabled + ? OSM_OVERLAYS.filter((overlay) => state.params[overlay.key]).length + : 0, + error: osmOverlaysHiddenReason(state.enabled, state.params, globeShown), + }; + }, +}; + +export default osmOverlaysLayer; diff --git a/src/data/osmOverlays.test.mjs b/src/data/osmOverlays.test.mjs new file mode 100644 index 000000000..d4dcbe691 --- /dev/null +++ b/src/data/osmOverlays.test.mjs @@ -0,0 +1,41 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { + OSM_OVERLAYS, + applyOsmOverlayParams, + osmOverlaysHiddenReason, +} from './osmOverlays.js'; + +test('OSM_OVERLAYS: exactly the two shipped overlays, each with a tile template and credit', () => { + assert.deepEqual(OSM_OVERLAYS.map((o) => o.key), ['seamap', 'snowmap']); + for (const overlay of OSM_OVERLAYS) { + assert.match(overlay.urlTemplate, /\{z\}.*\{x\}.*\{y\}/); + assert.ok(overlay.credit); + } +}); + +test('osmOverlaysHiddenReason: null when disabled, nothing chosen, or the globe is shown', () => { + assert.equal(osmOverlaysHiddenReason(false, { seamap: true, snowmap: true }, false), null); + assert.equal(osmOverlaysHiddenReason(true, { seamap: false, snowmap: false }, false), null); + assert.equal(osmOverlaysHiddenReason(true, { seamap: true, snowmap: false }, true), null); +}); + +test('osmOverlaysHiddenReason: explains itself when enabled + chosen but the photoreal globe is hidden', () => { + const reason = osmOverlaysHiddenReason(true, { seamap: true, snowmap: false }, false); + assert.ok(reason && /globe/i.test(reason)); +}); + +test('applyOsmOverlayParams: flips only known keys with boolean values, reports whether anything changed', () => { + const current = { seamap: true, snowmap: true }; + const a = applyOsmOverlayParams(current, { seamap: false }); + assert.deepEqual(a.next, { seamap: false, snowmap: true }); + assert.equal(a.changed, true); + + const b = applyOsmOverlayParams(current, { seamap: true }); + assert.deepEqual(b.next, current); + assert.equal(b.changed, false); + + const c = applyOsmOverlayParams(current, { unknownKey: false, snowmap: 'nope' }); + assert.deepEqual(c.next, current); + assert.equal(c.changed, false); +}); diff --git a/src/standalone/data.js b/src/standalone/data.js index 1c3e20366..a7359c584 100644 --- a/src/standalone/data.js +++ b/src/standalone/data.js @@ -10,6 +10,7 @@ import radioLayer from '../data/radio.js'; import bikeshareLayer from '../data/bikeshare.js'; import aisLiveVesselsLayer from '../data/aisLiveVessels.js'; import militaryInstallationsLayer from '../data/militaryInstallations.js'; +import osmOverlaysLayer from '../data/osmOverlays.js'; import militaryAwarenessLayer from '../data/militaryAwareness.js'; import localDataLayers from '../data/localLayers.js'; import { LAYER_STATE_REGISTRY } from '../data/layerState.js'; @@ -44,6 +45,7 @@ export function createStandaloneData({ dataManager.register(bikeshareLayer); dataManager.register(aisLiveVesselsLayer); dataManager.register(militaryInstallationsLayer); + dataManager.register(osmOverlaysLayer); dataManager.register(militaryAwarenessLayer); militaryAwarenessLayer.attachDataManager(dataManager); for (const layer of localDataLayers) { diff --git a/src/voice/gevActions.js b/src/voice/gevActions.js index a56c840e7..3ab9d149e 100644 --- a/src/voice/gevActions.js +++ b/src/voice/gevActions.js @@ -176,6 +176,16 @@ const LAYER_ALIASES = new Map([ ['firms', 'local-firms'], ['fires', 'local-firms'], ['active fires', 'local-firms'], + ['map overlays', 'osm-overlays'], + ['open map overlays', 'osm-overlays'], + ['sea marks', 'osm-overlays'], + ['seamarks', 'osm-overlays'], + ['nautical charts', 'osm-overlays'], + ['openseamap', 'osm-overlays'], + ['ski pistes', 'osm-overlays'], + ['ski slopes', 'osm-overlays'], + ['snow map', 'osm-overlays'], + ['opensnowmap', 'osm-overlays'], ]); const CITY_ALIASES = new Map([ From 14d7ddf06bccf85e738f99e1ef762098ec6e9bf8 Mon Sep 17 00:00:00 2001 From: m3ayush Date: Sun, 18 Oct 2026 12:19:25 +0530 Subject: [PATCH 02/21] feat(radio): add configurable priority-country coverage --- .env.example | 5 ++ DATA_SOURCES.md | 2 + docs/CURRENT-STATE.md | 27 +++++++ server/providers/radio.js | 8 +- server/providers/radio/catalog.js | 115 +++++++++++++++++++++++++--- src/data/radioProxy.test.mjs | 120 ++++++++++++++++++++++++++++++ 6 files changed, 263 insertions(+), 14 deletions(-) diff --git a/.env.example b/.env.example index a2f8afa24..fdfd8bf48 100644 --- a/.env.example +++ b/.env.example @@ -248,3 +248,8 @@ OVERPASS_UPSTREAMS= # frame-ancestors sources such as "https://example.com", or * for any page. # Unset, no page may frame it. Other pages of the app can never be framed. # GEV_EMBED_FRAME_ANCESTORS= + +# Optional keyless Radio Browser coverage: up to 8 ISO alpha-2 countries. +# Each reserves up to 40 stations within the existing 750-station directory. +# Restart after changing. Leave blank for the existing global selection. +# RADIO_PRIORITY_COUNTRIES=IN,PL diff --git a/DATA_SOURCES.md b/DATA_SOURCES.md index d930bcb8f..0b012e812 100644 --- a/DATA_SOURCES.md +++ b/DATA_SOURCES.md @@ -170,6 +170,8 @@ The optional Local ADS-B layer and the broadcast-FM receiver use radio you recei - **City of Toronto RESCU cameras.** Keyless. The camera list is the City's public `tmcearthcameras.json` on `opendata.toronto.ca` (dataset "Traffic Cameras", Transportation Services), served as JSONP and unwrapped with an anchored pattern. This is the municipal network the Ontario 511 pack does not carry: downtown arterials plus the Gardiner, DVP, Allen and Lake Shore. The list publishes no frame URL, so frame URLs are BUILT from the documented `CameraImages/loc####.jpg` convention and a strictly-digits camera number, pinning the frame proxy to the portal origin by construction; the list fetch refuses redirects and bounds the body it buffers. Frames are fetched at request time and never stored or redistributed. The dataset publishes no camera facing: the `D1`-`D4` fields are the directions of the *comparison* images (static reference stills captured in 2016 under a different path), not a bearing for the live pan-tilt-zoom frame, so headings use the shared id-hash fallback at low confidence and are corrected with the in-app calibration gizmo. A camera that is temporarily out of service still answers 200 with a byte-identical "Image Unavailable" placeholder rather than an error (about 6% of the network at any moment), so such a camera reads as healthy; the frame it shows is the agency's, not a synthesized one. The licence requires the attribution statement above. Disable with `CCTV_TORONTO_ENABLED=0`. - **Camera frame content.** Public camera frames can contain people, vehicles and license plates, and the camera layer does nothing with them beyond putting them on the map. A successful upstream response is relayed as the provider served it: nothing in the camera pipeline blurs, redacts, enhances or restores it, and nothing there runs face, plate or object detection or otherwise recognises or classifies what is in the picture. (The ALPR layer described below maps where plate-reader cameras are, from locations contributors tagged in OpenStreetMap; it reads nothing from any camera frame.) When an upstream has no frame for a camera, what you see instead is a Google Street View still or the app's own placeholder card; the panel's source badge names which of them you are looking at, and a refresh that fails after a frame has arrived leaves the last good picture on screen rather than replacing it. The browser resamples each frame for display: onto a fixed 1920x1080 surface for the camera's projection plane in the scene, and onto a 192x108 canvas for the small ambient card. That scaling is the only change made to the picture, and the only pixels the camera layer reads back are a 64x36 downscale hashed to tell whether the picture has changed. Live streams are piped through; the still path reads the image into memory to pass it on — TxDOT wraps its JPEG in a JSON envelope, which is decoded there — and no frame is written to disk (the disk cache, `.gev-cache/`, holds feed JSON for other layers and never camera imagery). Outside that pipeline, the voice assistant is the one feature that sends imagery anywhere: when it answers a question about what is in view it may attach a downscaled screenshot of the rendered scene — including any camera plane on screen — to the request it sends to the voice model. Frames are fetched only from server-registered camera URLs — a client cannot hand the proxy a URL (see [SECURITY.md](SECURITY.md)). What the app does stays inside the line README draws: no named-person search, no face recognition, no tracking individuals. - **Transit (GTFS-Realtime).** The proxy requests only registered operator feeds, validates redirects, bounds responses, and uses conditional requests, short snapshot caching and failure backoff. All sixteen registered regions are live-enabled; each can be disabled independently. Vehicles use delayed playback between reports, with report age and waiting states shown separately. Session-local selected trails are available for every feed. MBTA additionally retains up to 15 minutes of recently observed positions in the running proxy, so available history can survive a browser reload; server restart clears it. History reads never fetch upstream, and other feeds have no proxy retention. Nothing is archived to disk or collected in the background. Operator text credits are included as a courtesy and do not imply endorsement. Heights are aligned to work with Google 3D tiles; subway markers are shown at street level because these feeds do not supply depth. The TTC feed is BusTime, the surface system: it carries buses and streetcars and no subway at all. About four in ten of its vehicles report no route at all (unassigned or deadheading), and those are shown with the feed default rather than a route-established mode. It sends no ETag or Last-Modified, so its conditional requests fall through to the ordinary snapshot cache. Three Golden Horseshoe neighbours run alongside it — MiWay, Hamilton HSR and Durham Region Transit — each bus-only and each polled only near its own region. Durham Region Transit publishes no bearing on any vehicle, so those sprites take their heading from observed movement like any other feed that omits it. Guelph, Kingston and Barrie extend the same pattern eastward and north. Hamilton, Guelph, Burlington and YRT publish route ids that are not the number on the vehicle, so those entries carry a route-name table taken from each operator’s own published GTFS; any id missing from a table renders unchanged rather than blank. A label is never invented: YRT runs a route `9101` that appears nowhere in its published routes, so it stays as the feed sends it. Burlington’s is the one that cannot be shortcut: `351` is route 1 while `3510` is route 10, so the table is a real join rather than a trimmed prefix. GO Transit / Metrolinx is deliberately absent: its REALTIME feed requires an API key issued by manual review, which the keyless admission rule excludes (its static schedule download is keyless, but carries no vehicle positions). Barrie and YRT/Viva each invite acceptance of their licence through a form, and each licence also states that using the data is itself acceptance; this deployment relies on that clause, which is recorded in both entries. Grand River Transit in Waterloo is absent for a different reason: it answers, but its feed header was months stale and it publishes no per-vehicle timestamps, so it would render vehicles that look live and are not. Ottawa’s OC Transpo falls to the same keyless admission rule as GO Transit — its realtime feed requires an API key. London Transit and Thunder Bay Transit publish vehicle positions over plain HTTP with no HTTPS endpoint available. Every other Ontario system — Brampton, Oakville, Niagara, Milton, Windsor, Greater Sudbury, UP Express and a dozen smaller operators — publishes static GTFS schedules only, with no realtime vehicle positions to register. Between the nine included and these exclusions, every Ontario realtime vehicle-position feed catalogued in the Mobility Database is accounted for. +- **Radio country coverage (optional).** Set `RADIO_PRIORITY_COUNTRIES=IN,PL` in the local `.env` and restart to include stations beyond the global click charts. The operator may choose up to eight unique ISO alpha-2 country codes; no key is needed. Each country adds one bounded query (at most 200 rows) and reserves up to 40 unique, usable stations inside the existing 750-station catalog. Country seeds precede specialist-tag seeds and global popularity fill; unused slots return to the shared pool. This is a coverage preference, not an exhaustive national directory or a guarantee of stations in every city. Countries with no eligible HTTPS MP3/AAC geolocated stations remain empty. An unset list preserves global selection. Country requests share the directory cache and three-worker pool; a failed country query cannot replace a healthy snapshot. Failed/degraded refreshes back off for one minute, with at most three mirrors per query. + - **Radio Browser.** `/api/radio/stations` discovers official API mirrors, makes bounded and coalesced healthy/geolocated HTTPS-station queries, caches the normalized public-domain directory for 45 minutes, and may serve the last good catalog for up to seven days during an outage. Refreshes must meet minimum accepted-query and station coverage before replacing a warm catalog; schema-valid responses whose rows all fail the product's health policy do not count as successful queries. A usable partial cold catalog is explicitly `DEGRADED`, and malformed or empty successful payloads are rejected atomically. Every directory and click-count request rejects redirects, validates all resolved addresses as globally routable (including reserved/documentation IPv4 and special/non-global IPv6 exclusions), and pins the TLS connection to a validated address. Only MP3/AAC non-HLS directory rows with public HTTPS stream targets are returned; favicons are intentionally omitted. Pressing play connects one browser audio element directly to the selected broadcaster and calls the directory's click counter through known-ID-only `POST /api/radio/click/:uuid`. GEV never proxies, caches, records, bundles, or redistributes audio. Radio Browser supplies station-level tags, not dependable current-song or upcoming-program metadata, so Radio filtering never claims either. Direct playback exposes the listener's IP address to the broadcaster, whose own stream terms apply. - **TomTom Traffic.** Optional and BYOK: without `TOMTOM_API_KEY` the traffic layer runs its built-in simulation and no TomTom data (or attribution) appears. With a key, flow vector tiles are fetched through the server-side `/api/tomtom` proxy (120 s cache + a daily tile-budget governor — `TOMTOM_DAILY_TILE_BUDGET`, default 6,000 — sized so a 31-day month stays inside TomTom's 200K monthly free allowance; still a configurable application safety ceiling, not a guarantee; TomTom's [current pricing](https://docs.tomtom.com/pricing/) lists 200K free tile requests per month) and the "Traffic flow data © TomTom" credit is registered in the Data attribution popover the moment live mode activates. TomTom data is served live and cached only transiently (≤120 s TTL under `.gev-cache/`, gitignored) — it is not bundled or redistributed. One 23 KB point-in-time tile snapshot is committed as a decode-test fixture (`src/data/fixtures/`, © TomTom, never served to the app). - **Routing (OSRM / FOSSGIS).** `/api/route` proxies the public OSRM instances FOSSGIS e.V. runs for the OpenStreetMap community (car, bike, foot profiles). They are keyless and intended for fair, interactive use — the proxy rate-limits clients (60/min, 200/min total), caps route span and response size, caches each route for 10 minutes, and identifies itself in the User-Agent. Heavy or automated use must run its own OSRM. Routes are computed from OpenStreetMap data (ODbL); the Directions layer and voice routes never redistribute them. A route the router cannot find is reported as such; the Directions layer draws nothing in that case, and voice routes fall back to a clearly labeled straight line. diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 9562dc9f5..014bfd849 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -5807,3 +5807,30 @@ datacenters, dams and the Nepal locator retain that shared credit while displaye Military fragments stitch by source identity. Different mapped parcels merge only when their footprints touch or contain one another, never across a seam tolerance gap. + + +### Radio priority-country coverage + +Optional `RADIO_PRIORITY_COUNTRIES` is a server-owned comma-separated list of +up to eight ISO alpha-2 codes (for example `IN,PL`). Restart after changing it. +It needs no API key and adds no POWER UP entry. An empty list preserves the +existing global/tag-based directory. Invalid configuration fails at startup. + +Each country adds one Radio Browser search capped at 200 rows, with existing +HTTPS/geolocation/codec validation and an additional matching-country check. +Up to 40 unique accepted stations per country are seeded before the specialist +tag queries and global popularity fill, within the unchanged 750-station cap. +No country gets exhaustive coverage; an empty valid country response is allowed. +The response's `coverage.priorityCountries` reports accepted query status and +selected counts when configured. The existing broad/specialist health policy +still applies separately; a failed country query marks the refresh degraded +and cannot evict a healthy cached catalog. Queries share the existing three-worker +pool, single-flight refresh and 45-minute cache. At most three mirrors are tried +per query; failed or partial refreshes retry after one minute. The seven-day +last-good expiry and known-station-only click reporting remain in force. + +Validation: `node --test src/data/radioProxy.test.mjs` covers configuration, +country quotas under global popularity pressure, duplicate records, concurrency, +cache reuse, country mismatch/oversize/empty responses, stale fallback and retry +backoff. `node scripts/qa-radio.mjs --url http://localhost:4173` covers the existing +750-station browser presentation and playback contract. diff --git a/server/providers/radio.js b/server/providers/radio.js index 5dd4b196f..983e61d09 100644 --- a/server/providers/radio.js +++ b/server/providers/radio.js @@ -7,9 +7,13 @@ export { publicRadioHttpsUrl, } from './radio/stations.js'; export function radioBrowserProxy() { - const middleware = createRadioProxyMiddleware(); + let middleware; + const initialize = () => + (middleware ||= createRadioProxyMiddleware({ + priorityCountries: process.env.RADIO_PRIORITY_COUNTRIES || '', + })); const install = (server) => { - server.middlewares.use('/api/radio', middleware); + server.middlewares.use('/api/radio', initialize()); }; return { name: 'radio-browser-proxy', diff --git a/server/providers/radio/catalog.js b/server/providers/radio/catalog.js index 36a3eb730..fa1b7f6be 100644 --- a/server/providers/radio/catalog.js +++ b/server/providers/radio/catalog.js @@ -1,3 +1,4 @@ +import { normalizeRadioCountryInput } from '../../../src/data/radioCountry.js'; import { lookup as lookupDns } from 'node:dns/promises'; import { randomUUID } from 'node:crypto'; @@ -26,6 +27,32 @@ import { RADIO_UUID_RE, RADIO_FALLBACK_MIRRORS, } from './constants.js'; +const PRIORITY_COUNTRY_LIMIT = 8; +const COUNTRY_QUERY_LIMIT = 200; +const COUNTRY_RESERVED_STATIONS = 40; +const REFRESH_RETRY_MS = 60_000; +const MAX_MIRROR_ATTEMPTS = 3; + +/** Validate an operator-owned list; malformed configuration must not silently widen queries. */ +export function parseRadioPriorityCountries(value = '') { + const entries = Array.isArray(value) ? value : String(value).split(','); + const countries = []; + for (const entry of entries) { + const code = String(entry).trim().toUpperCase(); + if (!code) continue; + if (!/^[A-Z]{2}$/.test(code) || !normalizeRadioCountryInput(code).valid) + throw new TypeError( + 'RADIO_PRIORITY_COUNTRIES requires ISO alpha-2 country codes', + ); + if (!countries.includes(code)) countries.push(code); + } + if (countries.length > PRIORITY_COUNTRY_LIMIT) + throw new RangeError( + 'RADIO_PRIORITY_COUNTRIES supports at most eight countries', + ); + return countries; +} + export async function mapRadioConcurrent(values, concurrency, mapper) { const results = new Array(values.length); let cursor = 0; @@ -48,7 +75,9 @@ export function createRadioProxyMiddleware({ fetchImpl = null, lookupImpl = lookupDns, now = Date.now, + priorityCountries = [], } = {}) { + const countries = parseRadioPriorityCountries(priorityCountries); let mirrorCache = { origins: [...RADIO_FALLBACK_MIRRORS], cachedAt: 0 }; let mirrorPromise = null; let catalogCache = null; @@ -60,6 +89,9 @@ export function createRadioProxyMiddleware({ const catalogInstance = randomUUID(); let servedStationIds = new Set(); let refreshPromise = null; + let retryAfter = 0; + let partialCatalog = null; + let refreshError = null; async function fetchJson(url, maxBytes = RADIO_RESPONSE_MAX_BYTES) { const destination = radioProxyDestination(url); @@ -138,7 +170,7 @@ export function createRadioProxyMiddleware({ async function fetchPath(pathname) { let lastError = null; - for (const origin of await mirrors()) { + for (const origin of (await mirrors()).slice(0, MAX_MIRROR_ATTEMPTS)) { try { return await fetchJson(`${origin}${pathname}`); } catch (error) { @@ -159,20 +191,27 @@ export function createRadioProxyMiddleware({ 'aviation', 'marine', 'traffic', - ]; + ].map((tag) => ({ tag })); + const baseQueryCount = queries.length; + queries.push(...countries.map((country) => ({ country }))); const outcomes = await mapRadioConcurrent( queries, 3, - async (tag, index) => { + async ({ tag, country }, index) => { const params = new URLSearchParams({ has_geo_info: 'true', is_https: 'true', hidebroken: 'true', order: 'clickcount', reverse: 'true', - limit: index === 0 ? '1800' : '220', + limit: country + ? String(COUNTRY_QUERY_LIMIT) + : index === 0 + ? '1800' + : '220', }); if (tag) params.set('tag', tag); + if (country) params.set('countrycode', country); try { const rows = await fetchPath(`/json/stations/search?${params}`); if (!Array.isArray(rows)) @@ -194,7 +233,12 @@ export function createRadioProxyMiddleware({ ); const stations = rows .map(normalizeRadioBrowserStation) - .filter(Boolean); + .filter( + (station) => + station && (!country || station.countryCode === country), + ); + if (country && rows.length > COUNTRY_QUERY_LIMIT) + throw new Error('Radio Browser exceeded the country query limit'); const requestedTag = cleanRadioText(tag, 80) .toLocaleLowerCase() .replace(/[_-]+/g, ' ') @@ -213,7 +257,9 @@ export function createRadioProxyMiddleware({ // Query coverage is based on accepted rows, not merely a payload that // happens to match the upstream schema. Specialist responses must // also contain an accepted station tagged for the requested category. - succeeded: stations.length > 0 && requestedTagCovered, + succeeded: country + ? rows.length === 0 || stations.length > 0 + : stations.length > 0 && requestedTagCovered, stations, }; } catch { @@ -235,9 +281,24 @@ export function createRadioProxyMiddleware({ seen.add(station.id); selected.push(station); }; + // Reserve country slots first; low-click regional stations cannot lose all + // representation to the global charts. Keep room for all eight specialist + // seeds (8 * 45) and global fill even at the eight-country limit (8 * 40). + for (const rows of resultSets.slice(baseQueryCount)) { + const ranked = [...rows].sort( + (a, b) => b.clickCount - a.clickCount || a.id.localeCompare(b.id), + ); + let added = 0; + for (const station of ranked) { + if (seen.has(station.id)) continue; + take(station); + if (++added >= COUNTRY_RESERVED_STATIONS) break; + } + } // Seed specialist station-tag queries before popularity fill so operational // categories remain represented even when global click charts skew musical. - for (const rows of resultSets.slice(1)) rows.slice(0, 45).forEach(take); + for (const rows of resultSets.slice(1, baseQueryCount)) + rows.slice(0, 45).forEach(take); resultSets .flat() .sort( @@ -245,12 +306,14 @@ export function createRadioProxyMiddleware({ ) .forEach(take); const timestamp = now(); - const successfulQueries = outcomes.filter( - (outcome) => outcome.succeeded, - ).length; + const successfulQueries = outcomes + .slice(0, baseQueryCount) + .filter((outcome) => outcome.succeeded).length; const broadQueryHealthy = outcomes[0].succeeded && outcomes[0].stations.length > 0; const healthReasons = []; + if (outcomes.slice(baseQueryCount).some((outcome) => !outcome.succeeded)) + healthReasons.push('priority-country-query-unhealthy'); if (!broadQueryHealthy) healthReasons.push('broad-query-unhealthy'); if (successfulQueries < RADIO_CATALOG_MIN_SUCCESSFUL_QUERIES) healthReasons.push('query-coverage-below-policy'); @@ -259,7 +322,18 @@ export function createRadioProxyMiddleware({ const degraded = healthReasons.length > 0; const coverage = { successfulQueries, - totalQueries: queries.length, + totalQueries: baseQueryCount, + ...(countries.length + ? { + priorityCountries: countries.map((country, index) => ({ + countryCode: country, + succeeded: outcomes[baseQueryCount + index].succeeded, + stationCount: selected.filter( + (station) => station.countryCode === country, + ).length, + })), + } + : {}), stationCount: selected.length, healthyStationMinimum: RADIO_CATALOG_HEALTHY_MIN_STATIONS, }; @@ -310,7 +384,24 @@ export function createRadioProxyMiddleware({ return { ...catalogCache, stale: false }; } if (!refreshPromise) { - refreshPromise = refreshCatalog().finally(() => { + refreshPromise = (async () => { + if (now() < retryAfter) { + if (refreshError) throw refreshError; + return partialCatalog; + } + try { + const catalog = await refreshCatalog(); + refreshError = null; + partialCatalog = catalog.degraded ? catalog : null; + retryAfter = catalog.degraded ? now() + REFRESH_RETRY_MS : 0; + return catalog; + } catch (error) { + refreshError = error; + partialCatalog = null; + retryAfter = now() + REFRESH_RETRY_MS; + throw error; + } + })().finally(() => { refreshPromise = null; }); } diff --git a/src/data/radioProxy.test.mjs b/src/data/radioProxy.test.mjs index 60463205b..a5ac3dee7 100644 --- a/src/data/radioProxy.test.mjs +++ b/src/data/radioProxy.test.mjs @@ -533,3 +533,123 @@ test('a mirror answering an invalid reason phrase rejects instead of crashing', const crashed = mockMirrorStatus(t, 200, 'OK\x7f'); await assert.rejects(Promise.race([fetchMirror(), crashed]), TypeError); }); + +test('priority-country configuration is bounded, normalized and validated before requests', async () => { + const { parseRadioPriorityCountries } = await import('../../server/providers/radio/catalog.js'); + assert.deepEqual(parseRadioPriorityCountries(' in, PL,in, '), ['IN', 'PL']); + assert.deepEqual(parseRadioPriorityCountries(''), []); + for (const value of ['ZZ', 'India', 'IN&limit=9999', 'US,CA,IN,PL,DE,FR,GB,JP,AU']) { + assert.throws(() => createProxy({ priorityCountries: value }), /RADIO_PRIORITY_COUNTRIES/); + } +}); + +test('low-click country stations survive a full global catalog, with bounded cached queries and deduplication', async () => { + const calls = []; + let active = 0; + let peak = 0; + const middleware = createProxy({ + priorityCountries: 'IN,PL', + fetchImpl: async (url) => { + const parsed = new URL(url); + calls.push(parsed); + if (parsed.pathname === '/json/servers') return responseJson([{ name: 'de1.api.radio-browser.info' }]); + if (parsed.pathname.startsWith('/json/url/')) return responseJson({ ok: true }); + peak = Math.max(peak, ++active); + await new Promise((resolve) => setTimeout(resolve, 1)); + active--; + const country = parsed.searchParams.get('countrycode'); + if (country) { + assert.equal(parsed.searchParams.get('limit'), '200'); + assert.equal(parsed.searchParams.get('is_https'), 'true'); + const rows = catalogRows(country === 'IN' ? '10000000' : '20000000', 'community', 60) + .map((row) => ({ ...row, countrycode: country, clickcount: 0 })); + return responseJson([rows[0], ...rows]); + } + return responseJson(catalogRows('30000000', queryTag(url) || 'news', 900)); + }, + }); + const responses = await Promise.all([invoke(middleware, '/stations'), invoke(middleware, '/stations')]); + const body = JSON.parse(responses[0].body); + assert.equal(body.degraded, false); + assert.equal(body.stations.length, 750); + assert.equal(new Set(body.stations.map((row) => row.id)).size, 750); + for (const country of ['IN', 'PL']) + assert.ok(body.stations.filter((row) => row.countryCode === country).length >= 40); + assert.ok(body.stations.some((row) => row.countryCode === 'US')); + assert.equal(calls.filter((url) => url.pathname.includes('/stations/')).length, 11); + assert.ok(peak <= 3); + const count = calls.length; + await invoke(middleware, '/stations'); + assert.equal(calls.length, count, 'country queries share the 45-minute cache'); + const id = body.stations.find((row) => row.countryCode === 'IN').id; + assert.equal((await invoke(middleware, `/click/${id}`, 'POST')).status, 204); +}); + +test('an unavailable country retains the last healthy snapshot instead of silently losing its stations', async () => { + let clock = 4_000_000; + let fail = false; + const middleware = createProxy({ + priorityCountries: ['IN'], now: () => clock, + fetchImpl: async (url) => { + if (String(url).includes('/json/servers')) return responseJson([{ name: 'de1.api.radio-browser.info' }]); + if (new URL(url).searchParams.has('countrycode')) { + if (fail) throw new Error('offline'); + return responseJson([station({ countrycode: 'IN', country: 'India', tags: 'community' })]); + } + return responseJson(healthyRowsForQuery(url)); + }, + }); + const first = JSON.parse((await invoke(middleware, '/stations')).body); + assert.equal(first.degraded, false); + clock += 46 * 60 * 1000; + fail = true; + const stale = JSON.parse((await invoke(middleware, '/stations')).body); + assert.equal(stale.stale, true); + assert.match(stale.degradedReason, /priority-country-query-unhealthy/); + assert.deepEqual(stale.stations, first.stations); + assert.equal(stale.acceptedGeneration, first.acceptedGeneration); + clock += 8 * 24 * 60 * 60 * 1000; + assert.equal((await invoke(middleware, '/stations')).status, 503); +}); + +test('country discovery rejects wrong-country and oversized responses, but accepts an honest empty country', async () => { + for (const mode of ['wrong', 'oversized', 'empty']) { + const middleware = createProxy({ + priorityCountries: ['IN'], + fetchImpl: async (url) => { + if (String(url).includes('/json/servers')) return responseJson([{ name: 'de1.api.radio-browser.info' }]); + if (new URL(url).searchParams.has('countrycode')) return responseJson( + mode === 'empty' ? [] : mode === 'wrong' ? [station()] : + catalogRows('10000000', 'community', 201).map((row) => ({ ...row, countrycode: 'IN' })), + ); + return responseJson(healthyRowsForQuery(url)); + }, + }); + const body = JSON.parse((await invoke(middleware, '/stations')).body); + assert.equal(body.degraded, mode !== 'empty'); + assert.equal(body.coverage.priorityCountries[0].stationCount, 0); + assert.equal(body.coverage.priorityCountries[0].succeeded, mode === 'empty'); + } +}); + + +test('failed cold country refreshes are coalesced and back off instead of hammering mirrors', async () => { + let clock = 9_000_000; + let calls = 0; + const middleware = createProxy({ + priorityCountries: 'IN', now: () => clock, + fetchImpl: async (url) => { + calls++; + if (String(url).includes('/json/servers')) return responseJson([{ name: 'de1.api.radio-browser.info' }]); + throw new Error('offline'); + }, + }); + assert.equal((await invoke(middleware, '/stations')).status, 503); + const initial = calls; + assert.ok(initial <= 31, 'ten queries try at most three mirrors, plus discovery'); + assert.equal((await invoke(middleware, '/stations')).status, 503); + assert.equal(calls, initial); + clock += 60_001; + assert.equal((await invoke(middleware, '/stations')).status, 503); + assert.ok(calls > initial); +}); From cbfcaff8df4a015c091faa03afb186adb74a1c03 Mon Sep 17 00:00:00 2001 From: m3ayush Date: Sun, 18 Oct 2026 15:06:13 +0530 Subject: [PATCH 03/21] fix(radio): isolate country coverage from catalog health --- DATA_SOURCES.md | 4 +- changelog.d/1206-radio-priority-countries.md | 5 + docs/CURRENT-STATE.md | 67 ++++---- server/providers/radio/catalog.js | 10 +- src/data/radioProxy.test.mjs | 154 ++++++++++++------- 5 files changed, 145 insertions(+), 95 deletions(-) create mode 100644 changelog.d/1206-radio-priority-countries.md diff --git a/DATA_SOURCES.md b/DATA_SOURCES.md index 0b012e812..563f28544 100644 --- a/DATA_SOURCES.md +++ b/DATA_SOURCES.md @@ -170,9 +170,9 @@ The optional Local ADS-B layer and the broadcast-FM receiver use radio you recei - **City of Toronto RESCU cameras.** Keyless. The camera list is the City's public `tmcearthcameras.json` on `opendata.toronto.ca` (dataset "Traffic Cameras", Transportation Services), served as JSONP and unwrapped with an anchored pattern. This is the municipal network the Ontario 511 pack does not carry: downtown arterials plus the Gardiner, DVP, Allen and Lake Shore. The list publishes no frame URL, so frame URLs are BUILT from the documented `CameraImages/loc####.jpg` convention and a strictly-digits camera number, pinning the frame proxy to the portal origin by construction; the list fetch refuses redirects and bounds the body it buffers. Frames are fetched at request time and never stored or redistributed. The dataset publishes no camera facing: the `D1`-`D4` fields are the directions of the *comparison* images (static reference stills captured in 2016 under a different path), not a bearing for the live pan-tilt-zoom frame, so headings use the shared id-hash fallback at low confidence and are corrected with the in-app calibration gizmo. A camera that is temporarily out of service still answers 200 with a byte-identical "Image Unavailable" placeholder rather than an error (about 6% of the network at any moment), so such a camera reads as healthy; the frame it shows is the agency's, not a synthesized one. The licence requires the attribution statement above. Disable with `CCTV_TORONTO_ENABLED=0`. - **Camera frame content.** Public camera frames can contain people, vehicles and license plates, and the camera layer does nothing with them beyond putting them on the map. A successful upstream response is relayed as the provider served it: nothing in the camera pipeline blurs, redacts, enhances or restores it, and nothing there runs face, plate or object detection or otherwise recognises or classifies what is in the picture. (The ALPR layer described below maps where plate-reader cameras are, from locations contributors tagged in OpenStreetMap; it reads nothing from any camera frame.) When an upstream has no frame for a camera, what you see instead is a Google Street View still or the app's own placeholder card; the panel's source badge names which of them you are looking at, and a refresh that fails after a frame has arrived leaves the last good picture on screen rather than replacing it. The browser resamples each frame for display: onto a fixed 1920x1080 surface for the camera's projection plane in the scene, and onto a 192x108 canvas for the small ambient card. That scaling is the only change made to the picture, and the only pixels the camera layer reads back are a 64x36 downscale hashed to tell whether the picture has changed. Live streams are piped through; the still path reads the image into memory to pass it on — TxDOT wraps its JPEG in a JSON envelope, which is decoded there — and no frame is written to disk (the disk cache, `.gev-cache/`, holds feed JSON for other layers and never camera imagery). Outside that pipeline, the voice assistant is the one feature that sends imagery anywhere: when it answers a question about what is in view it may attach a downscaled screenshot of the rendered scene — including any camera plane on screen — to the request it sends to the voice model. Frames are fetched only from server-registered camera URLs — a client cannot hand the proxy a URL (see [SECURITY.md](SECURITY.md)). What the app does stays inside the line README draws: no named-person search, no face recognition, no tracking individuals. - **Transit (GTFS-Realtime).** The proxy requests only registered operator feeds, validates redirects, bounds responses, and uses conditional requests, short snapshot caching and failure backoff. All sixteen registered regions are live-enabled; each can be disabled independently. Vehicles use delayed playback between reports, with report age and waiting states shown separately. Session-local selected trails are available for every feed. MBTA additionally retains up to 15 minutes of recently observed positions in the running proxy, so available history can survive a browser reload; server restart clears it. History reads never fetch upstream, and other feeds have no proxy retention. Nothing is archived to disk or collected in the background. Operator text credits are included as a courtesy and do not imply endorsement. Heights are aligned to work with Google 3D tiles; subway markers are shown at street level because these feeds do not supply depth. The TTC feed is BusTime, the surface system: it carries buses and streetcars and no subway at all. About four in ten of its vehicles report no route at all (unassigned or deadheading), and those are shown with the feed default rather than a route-established mode. It sends no ETag or Last-Modified, so its conditional requests fall through to the ordinary snapshot cache. Three Golden Horseshoe neighbours run alongside it — MiWay, Hamilton HSR and Durham Region Transit — each bus-only and each polled only near its own region. Durham Region Transit publishes no bearing on any vehicle, so those sprites take their heading from observed movement like any other feed that omits it. Guelph, Kingston and Barrie extend the same pattern eastward and north. Hamilton, Guelph, Burlington and YRT publish route ids that are not the number on the vehicle, so those entries carry a route-name table taken from each operator’s own published GTFS; any id missing from a table renders unchanged rather than blank. A label is never invented: YRT runs a route `9101` that appears nowhere in its published routes, so it stays as the feed sends it. Burlington’s is the one that cannot be shortcut: `351` is route 1 while `3510` is route 10, so the table is a real join rather than a trimmed prefix. GO Transit / Metrolinx is deliberately absent: its REALTIME feed requires an API key issued by manual review, which the keyless admission rule excludes (its static schedule download is keyless, but carries no vehicle positions). Barrie and YRT/Viva each invite acceptance of their licence through a form, and each licence also states that using the data is itself acceptance; this deployment relies on that clause, which is recorded in both entries. Grand River Transit in Waterloo is absent for a different reason: it answers, but its feed header was months stale and it publishes no per-vehicle timestamps, so it would render vehicles that look live and are not. Ottawa’s OC Transpo falls to the same keyless admission rule as GO Transit — its realtime feed requires an API key. London Transit and Thunder Bay Transit publish vehicle positions over plain HTTP with no HTTPS endpoint available. Every other Ontario system — Brampton, Oakville, Niagara, Milton, Windsor, Greater Sudbury, UP Express and a dozen smaller operators — publishes static GTFS schedules only, with no realtime vehicle positions to register. Between the nine included and these exclusions, every Ontario realtime vehicle-position feed catalogued in the Mobility Database is accounted for. -- **Radio country coverage (optional).** Set `RADIO_PRIORITY_COUNTRIES=IN,PL` in the local `.env` and restart to include stations beyond the global click charts. The operator may choose up to eight unique ISO alpha-2 country codes; no key is needed. Each country adds one bounded query (at most 200 rows) and reserves up to 40 unique, usable stations inside the existing 750-station catalog. Country seeds precede specialist-tag seeds and global popularity fill; unused slots return to the shared pool. This is a coverage preference, not an exhaustive national directory or a guarantee of stations in every city. Countries with no eligible HTTPS MP3/AAC geolocated stations remain empty. An unset list preserves global selection. Country requests share the directory cache and three-worker pool; a failed country query cannot replace a healthy snapshot. Failed/degraded refreshes back off for one minute, with at most three mirrors per query. +- **Radio country coverage (optional).** Set `RADIO_PRIORITY_COUNTRIES=IN,PL` in the local `.env` and restart to include stations beyond the global click charts. The operator may choose up to eight unique ISO alpha-2 country codes; no key is needed. Each country adds one bounded query (at most 200 rows) and reserves up to 40 unique, usable stations inside the existing 750-station catalog. Country seeds precede specialist-tag seeds and global popularity fill; unused slots return to the shared pool. This is a coverage preference, not an exhaustive national directory or a guarantee of stations in every city. Countries with no eligible HTTPS MP3/AAC geolocated stations remain empty. An unset list preserves global selection. Country requests share the directory cache and three-worker pool; failed country queries are reported in coverage without blocking a healthy global refresh. Failed/degraded refreshes back off for one minute, with at most three mirrors per query. -- **Radio Browser.** `/api/radio/stations` discovers official API mirrors, makes bounded and coalesced healthy/geolocated HTTPS-station queries, caches the normalized public-domain directory for 45 minutes, and may serve the last good catalog for up to seven days during an outage. Refreshes must meet minimum accepted-query and station coverage before replacing a warm catalog; schema-valid responses whose rows all fail the product's health policy do not count as successful queries. A usable partial cold catalog is explicitly `DEGRADED`, and malformed or empty successful payloads are rejected atomically. Every directory and click-count request rejects redirects, validates all resolved addresses as globally routable (including reserved/documentation IPv4 and special/non-global IPv6 exclusions), and pins the TLS connection to a validated address. Only MP3/AAC non-HLS directory rows with public HTTPS stream targets are returned; favicons are intentionally omitted. Pressing play connects one browser audio element directly to the selected broadcaster and calls the directory's click counter through known-ID-only `POST /api/radio/click/:uuid`. GEV never proxies, caches, records, bundles, or redistributes audio. Radio Browser supplies station-level tags, not dependable current-song or upcoming-program metadata, so Radio filtering never claims either. Direct playback exposes the listener's IP address to the broadcaster, whose own stream terms apply. +- **Radio Browser.** In every installation, including when `RADIO_PRIORITY_COUNTRIES` is unset, each request tries at most three mirror candidates (later fallback candidates are skipped), and failed or partial catalog refreshes back off for 60 seconds. A cold-start failure therefore continues returning 503 during that delay even if upstream recovers. `/api/radio/stations` discovers official API mirrors, makes bounded and coalesced healthy/geolocated HTTPS-station queries, caches the normalized public-domain directory for 45 minutes, and may serve the last good catalog for up to seven days during an outage. Refreshes must meet minimum accepted-query and station coverage before replacing a warm catalog; broad/specialist responses whose rows all fail the product's health policy do not count as successful queries. Optional priority-country outcomes are advisory: valid, in-limit responses with no eligible stations succeed as empty results, while failed or malformed country queries report unsuccessful coverage without degrading the global catalog. A usable partial cold catalog is explicitly `DEGRADED`, and malformed or empty successful payloads are rejected atomically. Every directory and click-count request rejects redirects, validates all resolved addresses as globally routable (including reserved/documentation IPv4 and special/non-global IPv6 exclusions), and pins the TLS connection to a validated address. Only MP3/AAC non-HLS directory rows with public HTTPS stream targets are returned; favicons are intentionally omitted. Pressing play connects one browser audio element directly to the selected broadcaster and calls the directory's click counter through known-ID-only `POST /api/radio/click/:uuid`. GEV never proxies, caches, records, bundles, or redistributes audio. Radio Browser supplies station-level tags, not dependable current-song or upcoming-program metadata, so Radio filtering never claims either. Direct playback exposes the listener's IP address to the broadcaster, whose own stream terms apply. - **TomTom Traffic.** Optional and BYOK: without `TOMTOM_API_KEY` the traffic layer runs its built-in simulation and no TomTom data (or attribution) appears. With a key, flow vector tiles are fetched through the server-side `/api/tomtom` proxy (120 s cache + a daily tile-budget governor — `TOMTOM_DAILY_TILE_BUDGET`, default 6,000 — sized so a 31-day month stays inside TomTom's 200K monthly free allowance; still a configurable application safety ceiling, not a guarantee; TomTom's [current pricing](https://docs.tomtom.com/pricing/) lists 200K free tile requests per month) and the "Traffic flow data © TomTom" credit is registered in the Data attribution popover the moment live mode activates. TomTom data is served live and cached only transiently (≤120 s TTL under `.gev-cache/`, gitignored) — it is not bundled or redistributed. One 23 KB point-in-time tile snapshot is committed as a decode-test fixture (`src/data/fixtures/`, © TomTom, never served to the app). - **Routing (OSRM / FOSSGIS).** `/api/route` proxies the public OSRM instances FOSSGIS e.V. runs for the OpenStreetMap community (car, bike, foot profiles). They are keyless and intended for fair, interactive use — the proxy rate-limits clients (60/min, 200/min total), caps route span and response size, caches each route for 10 minutes, and identifies itself in the User-Agent. Heavy or automated use must run its own OSRM. Routes are computed from OpenStreetMap data (ODbL); the Directions layer and voice routes never redistribute them. A route the router cannot find is reported as such; the Directions layer draws nothing in that case, and voice routes fall back to a clearly labeled straight line. - **Re:Earth Terrain.** Keyless (no API key). Used two ways: (1) `src/mapStackController.js` swaps in a `Cesium.CesiumTerrainProvider` pointed at Re:Earth's `cesium-mesh/ellipsoid` quantized-mesh endpoint for globe stacks without a Cesium ion token (e.g. OSM), replacing a flat `EllipsoidTerrainProvider`; falls back to the flat provider if the endpoint can't be reached. (2) The server-side `/api/terrain/heights` proxy (disk-cached, serve-stale) resolves per-point ellipsoidal ground height for entity placement. Both are best-effort with a keyless-safe fallback (bundled EGM96 geoid math) if Re:Earth is unreachable. diff --git a/changelog.d/1206-radio-priority-countries.md b/changelog.d/1206-radio-priority-countries.md new file mode 100644 index 000000000..75536344e --- /dev/null +++ b/changelog.d/1206-radio-priority-countries.md @@ -0,0 +1,5 @@ +- Add optional keyless Radio priority-country coverage with up to 40 stations + per configured country inside the existing 750-station cap. Country query + failures do not block a healthy global catalog. Bound mirror attempts to three + and retry failed or partial refreshes after 60 seconds on all installations, + including those without priority countries configured (m3ayush, #1206). diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 014bfd849..5abc85750 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -4066,8 +4066,9 @@ programmatic activation do not trigger this reveal. Radio directory admission is atomic on both sides of the proxy boundary. A refresh is healthy only when it reaches the minimum accepted-query and station -coverage; schema-valid responses with zero normalized stations count as failed -queries rather than inflating refresh health. Each specialist query also needs +coverage from broad/specialist queries; their schema-valid responses with zero +normalized stations count as failed queries rather than inflating refresh health. +Optional priority-country outcomes do not gate catalog health. Each specialist query also needs an accepted station whose normalized tags match that requested category; rows tagged only for another category remain usable catalog data but do not earn specialist health credit. A partial cold result remains @@ -5516,6 +5517,41 @@ Existing catalog validation, category filters, tuning and voice playback behavio remain unchanged. Audio connects directly to the broadcaster after an explicit play action; the source does not relay or record streams. +### Radio priority-country coverage + +Optional `RADIO_PRIORITY_COUNTRIES` is a server-owned comma-separated list of +up to eight ISO alpha-2 codes (for example `IN,PL`). Restart after changing it. +It needs no API key and adds no POWER UP entry. An empty list preserves the +existing global/tag-based directory. Invalid configuration fails at startup. + +Each country adds one Radio Browser search capped at 200 rows, with existing +HTTPS/geolocation/codec validation and an additional matching-country check. +Up to 40 unique accepted stations per country are seeded before the specialist +tag queries and global popularity fill, within the unchanged 750-station cap. +No country gets exhaustive coverage. A valid, in-limit country response with +zero eligible stations succeeds with zero selected stations. +The response's `coverage.priorityCountries` reports accepted query status and +selected counts when configured. The existing broad/specialist health policy +still applies separately. Country outcomes are advisory: failed, malformed or +oversized country queries report unsuccessful coverage but cannot degrade or +block admission of an otherwise healthy global catalog. Queries share the existing +three-worker pool, single-flight refresh and 45-minute cache. + +For every installation, including those with no priority countries configured, +at most three mirrors are tried per request; fallback mirrors beyond the first +three candidates are not attempted. Failed or partial catalog refreshes retry +after one minute, so a cold-start failure can continue returning 503 for that +minute even if upstream recovers sooner. Healthy catalogs retain the 45-minute +cache. The seven-day last-good expiry and known-station-only click reporting +remain in force. + +Validation: `node --test src/data/radioProxy.test.mjs` covers configuration, +country quotas under global popularity pressure, duplicate records, concurrency, +cache reuse, country mismatch/oversize/empty/ineligible responses, failure isolation on cold +and warm refreshes beyond seven days, and retry/backoff recovery both with and +without configured countries. `node scripts/qa-radio.mjs --url http://localhost:4173` covers the existing +750-station browser presentation and playback contract. + ## Local RTL-SDR and Local ADS-B The Radio panel holds a Local RTL-SDR card below internet radio. One browser @@ -5807,30 +5843,3 @@ datacenters, dams and the Nepal locator retain that shared credit while displaye Military fragments stitch by source identity. Different mapped parcels merge only when their footprints touch or contain one another, never across a seam tolerance gap. - - -### Radio priority-country coverage - -Optional `RADIO_PRIORITY_COUNTRIES` is a server-owned comma-separated list of -up to eight ISO alpha-2 codes (for example `IN,PL`). Restart after changing it. -It needs no API key and adds no POWER UP entry. An empty list preserves the -existing global/tag-based directory. Invalid configuration fails at startup. - -Each country adds one Radio Browser search capped at 200 rows, with existing -HTTPS/geolocation/codec validation and an additional matching-country check. -Up to 40 unique accepted stations per country are seeded before the specialist -tag queries and global popularity fill, within the unchanged 750-station cap. -No country gets exhaustive coverage; an empty valid country response is allowed. -The response's `coverage.priorityCountries` reports accepted query status and -selected counts when configured. The existing broad/specialist health policy -still applies separately; a failed country query marks the refresh degraded -and cannot evict a healthy cached catalog. Queries share the existing three-worker -pool, single-flight refresh and 45-minute cache. At most three mirrors are tried -per query; failed or partial refreshes retry after one minute. The seven-day -last-good expiry and known-station-only click reporting remain in force. - -Validation: `node --test src/data/radioProxy.test.mjs` covers configuration, -country quotas under global popularity pressure, duplicate records, concurrency, -cache reuse, country mismatch/oversize/empty responses, stale fallback and retry -backoff. `node scripts/qa-radio.mjs --url http://localhost:4173` covers the existing -750-station browser presentation and playback contract. diff --git a/server/providers/radio/catalog.js b/server/providers/radio/catalog.js index fa1b7f6be..d26eab6e6 100644 --- a/server/providers/radio/catalog.js +++ b/server/providers/radio/catalog.js @@ -254,11 +254,11 @@ export function createRadioProxyMiddleware({ ), ); return { - // Query coverage is based on accepted rows, not merely a payload that - // happens to match the upstream schema. Specialist responses must - // also contain an accepted station tagged for the requested category. + // A valid country response can contain no playable stations. Country + // coverage is advisory; only broad/specialist queries gate health. + // Specialist responses need an accepted station with matching tags. succeeded: country - ? rows.length === 0 || stations.length > 0 + ? true : stations.length > 0 && requestedTagCovered, stations, }; @@ -312,8 +312,6 @@ export function createRadioProxyMiddleware({ const broadQueryHealthy = outcomes[0].succeeded && outcomes[0].stations.length > 0; const healthReasons = []; - if (outcomes.slice(baseQueryCount).some((outcome) => !outcome.succeeded)) - healthReasons.push('priority-country-query-unhealthy'); if (!broadQueryHealthy) healthReasons.push('broad-query-unhealthy'); if (successfulQueries < RADIO_CATALOG_MIN_SUCCESSFUL_QUERIES) healthReasons.push('query-coverage-below-policy'); diff --git a/src/data/radioProxy.test.mjs b/src/data/radioProxy.test.mjs index a5ac3dee7..01b1305f8 100644 --- a/src/data/radioProxy.test.mjs +++ b/src/data/radioProxy.test.mjs @@ -585,71 +585,109 @@ test('low-click country stations survive a full global catalog, with bounded cac assert.equal((await invoke(middleware, `/click/${id}`, 'POST')).status, 204); }); -test('an unavailable country retains the last healthy snapshot instead of silently losing its stations', async () => { - let clock = 4_000_000; - let fail = false; - const middleware = createProxy({ - priorityCountries: ['IN'], now: () => clock, - fetchImpl: async (url) => { - if (String(url).includes('/json/servers')) return responseJson([{ name: 'de1.api.radio-browser.info' }]); - if (new URL(url).searchParams.has('countrycode')) { - if (fail) throw new Error('offline'); - return responseJson([station({ countrycode: 'IN', country: 'India', tags: 'community' })]); - } - return responseJson(healthyRowsForQuery(url)); - }, - }); - const first = JSON.parse((await invoke(middleware, '/stations')).body); - assert.equal(first.degraded, false); - clock += 46 * 60 * 1000; - fail = true; - const stale = JSON.parse((await invoke(middleware, '/stations')).body); - assert.equal(stale.stale, true); - assert.match(stale.degradedReason, /priority-country-query-unhealthy/); - assert.deepEqual(stale.stations, first.stations); - assert.equal(stale.acceptedGeneration, first.acceptedGeneration); - clock += 8 * 24 * 60 * 60 * 1000; - assert.equal((await invoke(middleware, '/stations')).status, 503); -}); - -test('country discovery rejects wrong-country and oversized responses, but accepts an honest empty country', async () => { - for (const mode of ['wrong', 'oversized', 'empty']) { +test('country failures never block cold or warm global catalog admission, even after seven days', async () => { + for (const coldFailure of [false, true]) { + let clock = 4_000_000; + let fail = coldFailure; + let calls = 0; const middleware = createProxy({ - priorityCountries: ['IN'], + priorityCountries: ['IN'], now: () => clock, fetchImpl: async (url) => { + calls++; if (String(url).includes('/json/servers')) return responseJson([{ name: 'de1.api.radio-browser.info' }]); - if (new URL(url).searchParams.has('countrycode')) return responseJson( - mode === 'empty' ? [] : mode === 'wrong' ? [station()] : - catalogRows('10000000', 'community', 201).map((row) => ({ ...row, countrycode: 'IN' })), - ); + if (new URL(url).searchParams.has('countrycode')) { + if (fail) throw new Error('offline'); + return responseJson([station({ countrycode: 'IN', country: 'India', tags: 'community' })]); + } return responseJson(healthyRowsForQuery(url)); }, }); - const body = JSON.parse((await invoke(middleware, '/stations')).body); - assert.equal(body.degraded, mode !== 'empty'); - assert.equal(body.coverage.priorityCountries[0].stationCount, 0); - assert.equal(body.coverage.priorityCountries[0].succeeded, mode === 'empty'); + let generation = 0; + for (const elapsed of [0, 46 * 60 * 1000, 8 * 24 * 60 * 60 * 1000]) { + clock += elapsed; + if (elapsed) fail = true; + const response = await invoke(middleware, '/stations'); + const body = JSON.parse(response.body); + assert.equal(response.status, 200); + assert.equal(body.stale, false); + assert.equal(body.degraded, false); + assert.ok(body.acceptedGeneration > generation); + generation = body.acceptedGeneration; + assert.equal(body.coverage.priorityCountries[0].succeeded, !fail); + assert.equal(body.coverage.priorityCountries[0].stationCount, fail ? 0 : 1); + const count = calls; + clock += 60_001; + assert.deepEqual(JSON.parse((await invoke(middleware, '/stations')).body), body); + assert.equal(calls, count, 'country failure does not shorten the healthy cache'); + } } }); - -test('failed cold country refreshes are coalesced and back off instead of hammering mirrors', async () => { - let clock = 9_000_000; - let calls = 0; - const middleware = createProxy({ - priorityCountries: 'IN', now: () => clock, - fetchImpl: async (url) => { - calls++; - if (String(url).includes('/json/servers')) return responseJson([{ name: 'de1.api.radio-browser.info' }]); - throw new Error('offline'); - }, - }); - assert.equal((await invoke(middleware, '/stations')).status, 503); - const initial = calls; - assert.ok(initial <= 31, 'ten queries try at most three mirrors, plus discovery'); - assert.equal((await invoke(middleware, '/stations')).status, 503); - assert.equal(calls, initial); - clock += 60_001; - assert.equal((await invoke(middleware, '/stations')).status, 503); - assert.ok(calls > initial); +test('valid countries with no eligible stations are accepted and cached; malformed country responses stay isolated', async () => { + for (const mode of ['wrong', 'oversized', 'empty', 'ineligible', 'malformed']) { + let clock = 4_000_000; + let calls = 0; + const middleware = createProxy({ + priorityCountries: ['IN'], now: () => clock, + fetchImpl: async (url) => { + calls++; + if (String(url).includes('/json/servers')) return responseJson([{ name: 'de1.api.radio-browser.info' }]); + if (new URL(url).searchParams.has('countrycode')) { + const rows = { + empty: [], wrong: [station()], malformed: [{ name: 'missing fields' }], + oversized: catalogRows('10000000', 'community', 201).map((row) => ({ ...row, countrycode: 'IN' })), + ineligible: [{ hls: 1 }, { codec: 'OGG' }, { codec: 'OPUS' }, { url_resolved: 'https://127.0.0.1/live.mp3' }] + .map((overrides) => station({ countrycode: 'IN', ...overrides })), + }; + return responseJson(rows[mode]); + } + return responseJson(healthyRowsForQuery(url)); + }, + }); + const response = await invoke(middleware, '/stations'); + const body = JSON.parse(response.body); + assert.equal(response.status, 200, mode); + assert.equal(body.degraded, false, mode); + assert.ok(body.acceptedGeneration > 0, mode); + assert.equal(body.coverage.priorityCountries[0].stationCount, 0, mode); + assert.equal(body.coverage.priorityCountries[0].succeeded, !['oversized', 'malformed'].includes(mode), mode); + const count = calls; + clock += 44 * 60 * 1000; + assert.deepEqual(JSON.parse((await invoke(middleware, '/stations')).body), body); + assert.equal(calls, count, `${mode} country response preserves the 45-minute cache`); + } }); + +for (const priorityCountries of ['', 'IN']) { + test(`cold failures coalesce, bound mirrors and back off with priority countries ${priorityCountries || 'unset'}`, async () => { + let clock = Date.UTC(2026, 0, 1); + let calls = 0; + let offline = true; + const middleware = createProxy({ + priorityCountries, now: () => clock, + fetchImpl: async (url) => { + calls++; + if (String(url).includes('/json/servers')) return responseJson([{ name: 'de1.api.radio-browser.info' }]); + if (offline) throw new Error('offline'); + if (new URL(url).searchParams.has('countrycode')) return responseJson([]); + return responseJson(healthyRowsForQuery(url)); + }, + }); + const responses = await Promise.all([invoke(middleware, '/stations'), invoke(middleware, '/stations')]); + assert.ok(responses.every((response) => response.status === 503)); + const initial = calls; + assert.equal(initial, 1 + (priorityCountries ? 10 : 9) * 3, 'one discovery and at most three mirrors per query'); + offline = false; + clock += 59_999; + assert.equal((await invoke(middleware, '/stations')).status, 503); + assert.equal(calls, initial); + clock += 2; + const recovered = await invoke(middleware, '/stations'); + assert.equal(recovered.status, 200); + assert.equal(JSON.parse(recovered.body).degraded, false); + assert.ok(calls > initial); + const afterRecovery = calls; + await invoke(middleware, '/stations'); + assert.equal(calls, afterRecovery); + }); +} From 55f979c128c0e979c430ad5f7d82c358e30c84b4 Mon Sep 17 00:00:00 2001 From: Said Turksever Date: Wed, 21 Oct 2026 14:59:53 +0100 Subject: [PATCH 04/21] fix(overpass): keyless gate, loopback admission headroom and per-phase arrival budgets Address gekh's review of 69a87fe2: - P1: /api/overpass now uses admitSameSiteKeyless, like /api/terrain/heights, so an instance behind the README's reverse proxy keeps its Overpass-backed layers; cross-site shapes are still refused and forwarding headers no longer launder them. SECURITY.md, the gate header and CURRENT-STATE say so. - P2: admission keys IPv6 peers by /64 and folds IPv4-mapped peers (clientNetworkKey), and loopback has its own 300/min window outside the 900/min global, so a few LAN peers cannot lock the owner's browser out. - P2 (track): the settle and reveal frames each get their own 500 ms budget, measured from moveEnd and from the settle frame respectively, so a ~300 ms SwiftShader frame no longer trips the reveal deadline while both stay inside the 1 s idle refresh. - Smaller: drop the dead settings-block check and stale probe comment in query.js, clamp whitespace-tolerant [ timeout : N ], and add tests for the global window, the /64 keying, loopback headroom, proxied admission and cache hits spending admission but not upstream budget. Co-Authored-By: Claude Fable 5.1 --- SECURITY.md | 2 +- .../1264-overpass-request-admission.md | 4 +- docs/CURRENT-STATE.md | 11 +- scripts/format-scope.json | 1 + scripts/track-regression.mjs | 8 +- server/providers/common/rate-limit.js | 44 ++++++ .../providers/common/rateLimitKeys.test.mjs | 37 +++++ server/providers/common/same-site.js | 2 +- server/providers/overpass.js | 48 ++++-- server/providers/overpass/query.js | 16 +- .../overpass/requestAdmission.test.mjs | 145 +++++++++++++++++- src/localRequestGate.mjs | 17 +- src/tooling/trackArrivalObservation.mjs | 32 +++- src/tooling/trackArrivalObservation.test.mjs | 39 ++++- 14 files changed, 357 insertions(+), 49 deletions(-) create mode 100644 server/providers/common/rateLimitKeys.test.mjs diff --git a/SECURITY.md b/SECURITY.md index 30203ed94..cfbffe3cb 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -73,7 +73,7 @@ The data proxies under `server/providers/` are written so the browser cannot tur - **Sanitized errors** — internal error details are not echoed back to clients. - **Coalesced OAuth refresh** and cached successful responses only (OpenSky). - **Redacted debug logging.** The voice debug log (`.gev-logs/`, gitignored) strips API keys, bearer tokens, client secrets, and image data URLs before writing. -- **Cross-site gate on cost/log endpoints.** The cost-bearing (`/api/realtime/token`, `/api/openai/hud-summary`, `/api/google/nearby-places`, `/api/google/text-search`), quota-bearing (`/api/mapillary/*`, with a per-IP limit on tiles, and `/api/tomtom/*`, backed by its persistent daily tile budget) and log (`/api/realtime/debug-log`) endpoints refuse cross-site browser requests: a foreign or opaque `Origin`, a `Sec-Fetch-Site` other than `same-origin`/`none` (this is what blocks an ``/navigation that carries no `Origin`), or any reverse-proxy/CDN forwarding header returns `403`. Non-browser loopback tools (the repo QA harnesses POST the token endpoint with no `Origin`) and the explicit `HOST=0.0.0.0` LAN opt-in keep working — the gate deliberately does not require a loopback remote address; that stricter requirement belongs only to the credential panel (`admitKeySetupRequest`). The Host check (`allowedHosts`, applied by `hostCheckPlugin` ahead of every route, since Vite's own check runs only after plugin middleware) rejects a rebinding `Host` before any handler runs, in every mode, and the gate is defense in depth behind it: the LAN opt-in keeps that check and accepts only IP addresses, `localhost` and names listed in `GEV_ALLOWED_HOSTS`. **Limit:** a name you list there is trusted like the app's own address, so list only names you control. The keyless terrain route (`/api/terrain/heights`) uses the same cross-site checks, minus the forwarding-header refusal: it guards only the per-client quota and point cache, not a key or a bill, and the map needs it to render behind a reverse proxy. +- **Cross-site gate on cost/log endpoints.** The cost-bearing (`/api/realtime/token`, `/api/openai/hud-summary`, `/api/google/nearby-places`, `/api/google/text-search`), quota-bearing (`/api/mapillary/*`, with a per-IP limit on tiles, and `/api/tomtom/*`, backed by its persistent daily tile budget) and log (`/api/realtime/debug-log`) endpoints refuse cross-site browser requests: a foreign or opaque `Origin`, a `Sec-Fetch-Site` other than `same-origin`/`none` (this is what blocks an ``/navigation that carries no `Origin`), or any reverse-proxy/CDN forwarding header returns `403`. Non-browser loopback tools (the repo QA harnesses POST the token endpoint with no `Origin`) and the explicit `HOST=0.0.0.0` LAN opt-in keep working — the gate deliberately does not require a loopback remote address; that stricter requirement belongs only to the credential panel (`admitKeySetupRequest`). The Host check (`allowedHosts`, applied by `hostCheckPlugin` ahead of every route, since Vite's own check runs only after plugin middleware) rejects a rebinding `Host` before any handler runs, in every mode, and the gate is defense in depth behind it: the LAN opt-in keeps that check and accepts only IP addresses, `localhost` and names listed in `GEV_ALLOWED_HOSTS`. **Limit:** a name you list there is trusted like the app's own address, so list only names you control. The keyless map routes (`/api/terrain/heights` and `/api/overpass`) use the same cross-site checks, minus the forwarding-header refusal: they guard only per-client quotas and caches, not a key or a bill, and the map needs both to render behind a reverse proxy. `/api/overpass` also admits at most 300 POSTs a minute per peer (IPv6 peers keyed by /64) and 900 a minute across all non-loopback peers before it reads a body; loopback has its own per-peer window, so LAN peers cannot lock the owner out. - **Full Content-Security-Policy.** The dev and preview servers send a real CSP (`script-src 'self' 'unsafe-eval' blob:` plus the YouTube, Facebook and X scripts the Nepal event's embedded media loads and the Maps JavaScript API that Street View runs on, with a matching `frame-src` for their players and posts: no inline script and no other script origin; `blob:` covers the object URLs Cesium's bundled workers bootstrap through in the built app), alongside the existing `X-Frame-Options: DENY` and `frame-ancestors 'none'`. `'unsafe-eval'` stays because Knockout, bundled inside Cesium's widgets package, resolves the global object with `eval` at load time and the globe does not initialize without it; styles permit `'unsafe-inline'` and Google Fonts because Vite's dev client injects `