From ffb332472d00e21aceb6d24e4f3ae97c8dbcff78 Mon Sep 17 00:00:00 2001 From: Matt Van Horn Date: Mon, 12 Oct 2026 19:35:29 +0000 Subject: [PATCH 01/48] feat(layers): add FAA drone restrictions airspace layer Show US prohibited and restricted special-use airspace from the keyless FAA SUA FeatureServer as an off-by-default Infrastructure layer, with share-link token 3 and an informational-only disclaimer. --- CHANGELOG.md | 8 + DATA_SOURCES.md | 2 + README.md | 3 +- docs/CURRENT-STATE.md | 25 +- package.json | 2 + scripts/format-scope.json | 11 + scripts/package-boundaries.json | 32 ++ server/providers/droneAirspace.js | 135 ++++++++ server/providers/local.js | 2 + src/app/constructCatalog.js | 4 + src/app/constructCatalog.test.mjs | 5 +- src/app/layers/droneAirspace.js | 22 ++ src/app/sourceComposition.js | 1 + src/app/sourceComposition.test.mjs | 2 +- src/data/dataCredits.js | 7 + src/data/droneAirspaceProxy.test.mjs | 146 +++++++++ src/data/layerState.js | 5 + src/data/layerState.test.mjs | 34 +- src/data/layerStateTokenLedger.test.mjs | 62 ++-- src/data/layerStateTokenReservations.json | 1 + src/firstRunExperience.test.mjs | 5 +- src/layers/droneAirspace/cards.js | 93 ++++++ src/layers/droneAirspace/cards.test.mjs | 74 +++++ src/layers/droneAirspace/index.js | 326 ++++++++++++++++++++ src/layers/droneAirspace/ownership.test.mjs | 161 ++++++++++ src/layers/droneAirspace/records.js | 97 ++++++ src/layers/droneAirspace/records.test.mjs | 95 ++++++ src/layers/droneAirspace/source.js | 23 ++ src/layers/droneAirspace/source.test.mjs | 105 +++++++ src/sources/reference.js | 2 + src/sources/reference.test.mjs | 3 + src/tooling/previewServing.test.mjs | 1 + src/ui/layerPanel.js | 1 + src/voice/layerManifest.js | 24 ++ src/voice/layerManifest.test.mjs | 128 ++++++-- 35 files changed, 1573 insertions(+), 74 deletions(-) create mode 100644 server/providers/droneAirspace.js create mode 100644 src/app/layers/droneAirspace.js create mode 100644 src/data/droneAirspaceProxy.test.mjs create mode 100644 src/layers/droneAirspace/cards.js create mode 100644 src/layers/droneAirspace/cards.test.mjs create mode 100644 src/layers/droneAirspace/index.js create mode 100644 src/layers/droneAirspace/ownership.test.mjs create mode 100644 src/layers/droneAirspace/records.js create mode 100644 src/layers/droneAirspace/records.test.mjs create mode 100644 src/layers/droneAirspace/source.js create mode 100644 src/layers/droneAirspace/source.test.mjs diff --git a/CHANGELOG.md b/CHANGELOG.md index b24ac95ed..90402cad5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # Changelog +- Add a Drone Restrictions layer for FAA prohibited (P) and restricted (R) + special-use airspace. The keyless `/api/drone-airspace` proxy pages the + public SUA FeatureServer, keeps a one-hour snapshot, and serves stale data + when a refresh fails. Zones draw as ground polygons (red prohibited, orange + restricted), selection shows name, altitudes, times of use and agency, and + the row legend states the layer is informational only. Off by default; + share-link token `3`. + - Treat explicitly current, complete empty vessel coverage as a successful update, clearing obsolete contacts while retaining stale or incomplete snapshots. diff --git a/DATA_SOURCES.md b/DATA_SOURCES.md index 0b8eac358..2aa957e2b 100644 --- a/DATA_SOURCES.md +++ b/DATA_SOURCES.md @@ -35,6 +35,7 @@ How to read this: | **NASA GIBS** (WMTS tiles + Worldview Snapshots) and **NASA CMR** (granule catalog) | Recent Imagery layer: HLS S30/L30 (Sentinel-2, Landsat 8/9) and VIIRS true-colour tiles draped on the map, per-day thumbnails and PNG exports; the HLS granule catalog (dates, cloud, footprints) for a selected box | U.S. public domain (NASA data); keyless, browser-direct (CORS `*`); no bulk use — one 30-day catalog query per box, tiles bounded to the box, six tile requests in flight | "We acknowledge the use of imagery provided by services from NASA's Global Imagery Browse Services (GIBS), part of NASA's Earth Science Data and Information System (ESDIS)." — [gibs.earthdata.nasa.gov](https://gibs.earthdata.nasa.gov), [HLS](https://lpdaac.usgs.gov/products/hlss30v002/) | | **NIFC WFIGS** (Wildland Fire Interagency Geospatial Services, ArcGIS feature service) | Fire Perimeters layer: current interagency wildfire incident perimeters, containment, and incident facts | U.S. public domain (interagency wildland-fire data published through the [NIFC Open Data portal](https://data-nifc.opendata.arcgis.com/)); fetched keyless through `/api/fire-perimeters` (5-minute server cache) with `exceededTransferLimit` paging | "Wildfire perimeters: National Interagency Fire Center (WFIGS)" | | **InciWeb** (inciweb.wildfire.gov) | Per-incident "InciWeb ↗" card links to official incident pages (narratives, evacuation/closure notices) | U.S. government public incident information. Links are matched best-effort by incident name and state, with incident page origin and update times checked on selection. Same-origin `/api/fire-perimeters/inciweb` routes cache the catalog for 1 hour and publication metadata for 30 minutes; linked pages remain InciWeb content | "Incident information: InciWeb (inciweb.wildfire.gov)" | +| **FAA Special Use Airspace** (`services6.arcgis.com` FeatureServer `Special_Use_Airspace`) | Drone Restrictions layer: US prohibited (P) and restricted (R) special-use airspace polygons, altitude limits, times of use, and controlling agency | U.S. public domain (FAA aeronautical data published through the [FAA AIS Open Data portal](https://adds-faa.opendata.arcgis.com/)); fetched keyless through `/api/drone-airspace` (1-hour server cache). Informational only — not a substitute for official UAS flight-planning tools | "Drone restrictions: FAA Special Use Airspace" | | **OpenFreeMap / OpenMapTiles** | Vector tiles | OpenStreetMap data under ODbL 1.0 | OpenFreeMap © OpenMapTiles | | **Overture Maps Foundation** | Military area names pack | [ODbL 1.0](https://opendatacommons.org/licenses/odbl/1-0/) | Overture Maps Foundation | | **TomTom Traffic API** (flow vector tiles) | Traffic roads and live congestion (optional, BYOK) | [TomTom for Developers terms](https://docs.tomtom.com/legal/terms-and-conditions) (proprietary, your own key; free tier currently 200K tile requests/month — see [current pricing](https://docs.tomtom.com/pricing/)) | "Traffic flow data © TomTom" — registered when live mode activates | @@ -98,6 +99,7 @@ The optional Local ADS-B layer and the broadcast-FM receiver use radio you recei - **Routing (OSRM / FOSSGIS).** `/api/route` proxies the public OSRM instances FOSSGIS e.V. runs for the OpenStreetMap community (car, bike, foot profiles). They are keyless and intended for fair, interactive use — the proxy rate-limits clients (60/min, 200/min total), caps route span and response size, caches each route for 10 minutes, and identifies itself in the User-Agent. Heavy or automated use must run its own OSRM. Routes are computed from OpenStreetMap data (ODbL); the Directions layer and voice routes never redistribute them. A route the router cannot find is reported as such; the Directions layer draws nothing in that case, and voice routes fall back to a clearly labeled straight line. - **Re:Earth Terrain.** Keyless (no API key). Used two ways: (1) `src/mapStackController.js` swaps in a `Cesium.CesiumTerrainProvider` pointed at Re:Earth's `cesium-mesh/ellipsoid` quantized-mesh endpoint for globe stacks without a Cesium ion token (e.g. OSM), replacing a flat `EllipsoidTerrainProvider`; falls back to the flat provider if the endpoint can't be reached. (2) The server-side `/api/terrain/heights` proxy (disk-cached, serve-stale) resolves per-point ellipsoidal ground height for entity placement. Both are best-effort with a keyless-safe fallback (bundled EGM96 geoid math) if Re:Earth is unreachable. - **OSRM on the FOSSGIS routing servers.** Keyless (no API key). Every route is requested by a person: clicking A and B in the Directions layer, or a voice "route from A to B". The server-side `/api/route` proxy sends an identifying `User-Agent`, pins the upstream host, accepts 2-12 coordinates, rejects any leg over 600 km or a total over 2,500 km, caps the buffered response at 8 MB, times the upstream out at 12 s, rate-limits per client, caches each route for 10 minutes, and coalesces identical requests that are in flight at the same time. Turn-by-turn maneuvers are asked for only when a caller wants them (`steps=1`), so the callers that do not read them send the smaller request. The "fix the map" link the policy asks for is in the Data attribution popover beside the OpenStreetMap credit. +- **FAA Special Use Airspace.** Keyless. `/api/drone-airspace` queries the public FAA AIS FeatureServer for prohibited (P) and restricted (R) zones only — not MOAs, TFRs or other SUA types — with simplified geometry, a body cap, paging, a one-hour cache and stale-on-error. The layer is US coverage, informational only, and is not a substitute for B4UFLY, LAANC or a preflight briefing. Data is fetched at request time and never stored or redistributed. US government public domain; courtesy attribution is registered in the Data attribution popover. - **Mapped installations.** OpenFreeMap military polygons use a bundled Overture/OSM name index; wide views show named points. Optional operator-configured Overpass and explicit Google Places searches keep their own source paths. Mapped data does not establish current activity or complete coverage. - **Place search.** Named-place fly-to uses Google Geocoding when a Maps key is configured, then Photon's public instance, then `/api/geocode` (Nominatim search) when neither answers. `/api/geocode` uses the shared Nominatim forward-search queue at no more than one request per second, sends an identifying `User-Agent` and `Referer`, caches answers for five minutes, shares one upstream call between identical searches already in flight, bounds the queue so a burst is refused rather than held, and drops a queued search whose caller has given up. It asks for one result per search and is not used for systematic or bulk queries. - **Cockpit regional briefing.** `/api/regional-brief` rounds aircraft coordinates into 0.1° cache cells, caches results for five minutes, and resolves physical regions from bundled public-domain Natural Earth polygons without reverse-geocoding requests. Google News RSS is queried with the resolved locality/region first; GDELT is used only when that RSS query fails or is empty. Google's published Google News terms restrict that source to personal, noncommercial use, so commercial deployments must disable/replace it or obtain separate permission; GDELT permits commercial dataset use with citation. The Data attribution popover identifies the active headline sources; article links retain publisher attribution. Headlines are location-query matches, not verified incidents, risk rankings, or evidence that a location is safe. Empty, partial, stale, and unavailable source states remain distinct. Open-Meteo supplies current conditions independently of the news source. `WX OFF` disables cockpit weather rendering only; the Local Info briefing still fetches its source-backed weather values and displays the required linked Open-Meteo credit. diff --git a/README.md b/README.md index 5d97dfb2f..6b8043e5a 100644 --- a/README.md +++ b/README.md @@ -289,7 +289,7 @@ _Ask for radio near anywhere and the globe starts broadcasting — every station ## 🛰️ What's on the Globe -Twenty layers and map sources. **Seventeen have a keyless path.** Some offer additional capabilities with a provider key. (🟢 no key · 🟡 free key · 🔴 metered.) +Twenty-one layers and map sources. **Eighteen have a keyless path.** Some offer additional capabilities with a provider key. (🟢 no key · 🟡 free key · 🔴 metered.) | Layer | What you get | Source | Auth | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------------------- | @@ -310,6 +310,7 @@ Twenty layers and map sources. **Seventeen have a keyless path.** Some offer add | 🔥 **Active Fires** | Live NASA FIRMS detections, trailing 24h | NASA FIRMS | 🟡 | | 🚀 **Space Missions** | Rolling 30-day launches with payload, stage, and recovery detail | Launch Library 2 | 🟢 (🟡 optional token raises the allowance) | | 🎖️ **Mapped Installations** | Viewport-bounded military-site context from community mapping — incomplete by nature, and labeled that way | OpenStreetMap | 🟢 | +| 🚫 **Drone Restrictions** | FAA prohibited (P) and restricted (R) special-use airspace as ground polygons, with name, altitudes and times of use on select — informational only, not for flight planning | FAA SUA | 🟢 | | 🌬️ **Wind** | Animated 10 m forecast wind with an optional speed, temperature or pressure color field and a reading at map center | NOAA GFS / ECMWF IFS | 🟢 | | 🌦️ **Observed Weather** | Rain radar (contiguous US), Satellite clouds (GOES regional or global infrared, Clouds only or Full image) and Lightning density (Americas and Pacific) on one history timeline | NOAA nowCOAST | 🟢 | | 🌀 **Cyclones** | Advisory positions, forecast tracks and uncertainty cones for the Atlantic and eastern/central North Pacific | NOAA NHC / CPHC | 🟢 | diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 26d56e66b..2fc40751a 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1,5 +1,19 @@ # God's Eye View Current State +## Drone Restrictions — FAA special-use airspace + +The Infrastructure group includes **Drone Restrictions** (`drone-airspace`, +share-link token `3`), off by default. `/api/drone-airspace` pages the public +FAA Special Use Airspace FeatureServer for `TYPE_CODE` P (prohibited) and R +(restricted), simplifies geometry, caches for an hour, and serves the last +good snapshot as stale when a refresh fails. The globe draws ground-clamped +outlines with a translucent fill: red for prohibited, orange for restricted. +Selecting a zone shows name, altitudes, times of use and controlling agency, +plus a notice that the layer is informational only and not a substitute for +official UAS flight-planning tools. Pick ownership is unregistered so sibling +layer cards still work over a zone. Voice can toggle the layer and query the +loaded records. See the live-layers table below. + Vessel sources can report healthy empty coverage using a current, complete snapshot with zero raw rows, a healthy transport and a positive last-message time. This clears obsolete contacts and settles startup without manufacturing @@ -335,7 +349,7 @@ Consult the linked official advisory for safety decisions. Voice and HUD snapshots reuse the existing feedState classifier. Analyst follow-ups retain their original data provenance; current-view results append provenance without replacing legacy fields. HUD context and deterministic telemetry include non-nominal feed state. -Analyst records come on demand from each layer's `getAnalystRecords`: flights, military, local ADS-B, vessels, satellites, launches, earthquakes, FIRMS (VIIRS and MODIS, with a `sensor` field), fire perimeters, cyclones, transit, bikeshare, ALPR cameras, mapped installations, datacenters and dams. `analyst_query` counts the whole loaded set (up to 250,000 records per layer; past that `coverage.records` reports `returned`, `total` and `truncated: true` and the result carries `complete: false`). Ranking keeps a top-k and scans in 20,000-row slices that yield to the page and stop when superseded. It refuses unknown fields, operators, units that do not convert (`altitudeM(ft)` converts), wrong-typed values, invalid centres, unknown scope kinds and sort fields with `code` and `allowed`, refuses a follow-up with no previous answer (`NO_RESULT_CONTEXT`) or naming other layers than that answer (`FOLLOW_UP_MISMATCH`), and tells `LAYER_OFF`, `NOT_READY` and `FEED_UNAVAILABLE` apart from a real zero; when only some layers can answer the result is `partial` with `unanswered` ids. The Contacts-window path applies the same filters, ranking and item shape to one immutable panel snapshot. Each aircraft layer retains at most 20,000 rows; reaching that bound sets `complete: false` and the spoken/card count becomes “At least”, never an exact claim. A superseded query never replaces follow-up memory, while sibling analyst calls from one assistant response do not cancel one another. Items carry `lat`/`lon`; the feed window (earthquakes and FIRMS: last 24 h), the precise scope and any caveat sit in `display`, which is on-screen detail rather than something the model recites. "In view" is still a radius around the view centre, not the visible footprint. Satellite distance is ground distance. +Analyst records come on demand from each layer's `getAnalystRecords`: flights, military, local ADS-B, vessels, satellites, launches, earthquakes, FIRMS (VIIRS and MODIS, with a `sensor` field), fire perimeters, drone restrictions, cyclones, transit, bikeshare, ALPR cameras, mapped installations, datacenters and dams. `analyst_query` counts the whole loaded set (up to 250,000 records per layer; past that `coverage.records` reports `returned`, `total` and `truncated: true` and the result carries `complete: false`). Ranking keeps a top-k and scans in 20,000-row slices that yield to the page and stop when superseded. It refuses unknown fields, operators, units that do not convert (`altitudeM(ft)` converts), wrong-typed values, invalid centres, unknown scope kinds and sort fields with `code` and `allowed`, refuses a follow-up with no previous answer (`NO_RESULT_CONTEXT`) or naming other layers than that answer (`FOLLOW_UP_MISMATCH`), and tells `LAYER_OFF`, `NOT_READY` and `FEED_UNAVAILABLE` apart from a real zero; when only some layers can answer the result is `partial` with `unanswered` ids. The Contacts-window path applies the same filters, ranking and item shape to one immutable panel snapshot. Each aircraft layer retains at most 20,000 rows; reaching that bound sets `complete: false` and the spoken/card count becomes “At least”, never an exact claim. A superseded query never replaces follow-up memory, while sibling analyst calls from one assistant response do not cancel one another. Items carry `lat`/`lon`; the feed window (earthquakes and FIRMS: last 24 h), the precise scope and any caveat sit in `display`, which is on-screen detail rather than something the model recites. "In view" is still a radius around the view centre, not the visible footprint. Satellite distance is ground distance. Keyless terrain tiles retry when Re:Earth throttles them. The browser fetches `terrain.reearth.land/cesium-mesh/ellipsoid/{z}/{x}/{y}.terrain` directly; no @@ -3070,9 +3084,18 @@ its criteria cannot be silently ignored. | Street Level 📷 | Street-level imagery; Mapillary is the first provider (vector tiles, Graph API, MapillaryJS) | `src/layers/streetLevel/` via `src/app/layers/streetLevel.js` | `/api/mapillary/status`, `/api/mapillary/tiles/coverage/{z}/{x}/{y}` (`MAPILLARY_CLIENT_TOKEN`) | camera-driven (320 ms debounce, ≤9 tiles, cached 24 h) | | Wind 🌬 | NOAA GFS 10 m wind (keyless, 0.25°→1° grid; animated particles) | `src/data/wind.js` | `/api/wind` | 1 h (forecast cycle) | | Fire Perimeters 🔥 | NIFC WFIGS current interagency perimeters (keyless, paged past the 2000-record cap); InciWeb catalog + incident page origin and update time checks for verified incident-page links | `src/layers/perimeters/` via `src/app/layers/perimeters.js` | `/api/fire-perimeters` + `/api/fire-perimeters/inciweb/*` | 5 min (server caches: catalog 1 h; publication 30 min) | +| Drone Restrictions 🚫 | FAA Special Use Airspace prohibited (P) and restricted (R) zones (keyless FeatureServer, simplified geometry) | `src/layers/droneAirspace/` via `src/app/layers/droneAirspace.js` | `/api/drone-airspace` | 1 h (server cache 1 h; stale-on-error) | Fire Perimeters uses capped, timed server reads with stale-on-error caching and a per-client limit. Unchanged snapshots retain geometry; link checks cancel on disable or selection change, and the row legend shows reported containment. +Drone Restrictions is off by default (share-link token `3`) and sits in +Infrastructure. The proxy pages `TYPE_CODE IN ('P','R')` from the FAA SUA +FeatureServer, caps the body, and reuses a one-hour snapshot, serving the last +good payload as stale when a refresh fails. Client-supplied URLs are rejected. +Polygons keep holes; pick ownership is deliberately unregistered so sibling +cards still work over a zone. The row legend and selected card state that the +layer is informational only and not a substitute for B4UFLY / LAANC. + Street Level is a collapsible right-rail panel (`#street-level-panel`, layer token `0`, option owner `street-level`, panel `ui` token `t`) that starts collapsed. It opens itself only when the user, voice or a tool switches the diff --git a/package.json b/package.json index d9bf6cf8a..ca8a972ce 100644 --- a/package.json +++ b/package.json @@ -165,6 +165,7 @@ "./layers/recent-imagery": "./src/layers/recentImagery/index.js", "./layers/recent-imagery/source": "./src/layers/recentImagery/source.js", "./layers/perimeters": "./src/layers/perimeters/index.js", + "./layers/drone-airspace": "./src/layers/droneAirspace/index.js", "./sources/firms-csv": "./src/data/firmsCsv.js", "./layers/firms": "./src/layers/firms/index.js", "./layers/satellites": "./src/layers/satellites/index.js", @@ -238,6 +239,7 @@ "./layers/cctv/source": "./src/layers/cctv/source.js", "./layers/earthquakes/source": "./src/layers/earthquakes/source.js", "./layers/perimeters/source": "./src/layers/perimeters/source.js", + "./layers/drone-airspace/source": "./src/layers/droneAirspace/source.js", "./layers/firms/source": "./src/layers/firms/source.js", "./layers/installations/source": "./src/layers/installations/source.js", "./layers/launches/source": "./src/layers/launches/source.js", diff --git a/scripts/format-scope.json b/scripts/format-scope.json index 6b5c9b544..0d72c1096 100644 --- a/scripts/format-scope.json +++ b/scripts/format-scope.json @@ -371,6 +371,14 @@ "src/layers/perimeters/cards.test.mjs", "src/layers/perimeters/inciweb.test.mjs", "src/layers/perimeters/ownership.test.mjs", + "src/layers/droneAirspace/index.js", + "src/layers/droneAirspace/records.js", + "src/layers/droneAirspace/source.js", + "src/layers/droneAirspace/cards.js", + "src/layers/droneAirspace/records.test.mjs", + "src/layers/droneAirspace/source.test.mjs", + "src/layers/droneAirspace/cards.test.mjs", + "src/layers/droneAirspace/ownership.test.mjs", "src/data/firmsHeatmap.js", "src/data/fireAnchors.js", "src/layers/firms/anchors.js", @@ -643,6 +651,7 @@ "src/app/layers/radio.js", "src/app/layers/earthquakes.js", "src/app/layers/perimeters.js", + "src/app/layers/droneAirspace.js", "src/app/layers/cctv.js", "src/app/layers/firms.js", "src/app/layers/satellites.js", @@ -824,8 +833,10 @@ "src/layers/weather/imageryOrder.test.mjs", "server/providers/cyclones.js", "server/providers/firePerimeters.js", + "server/providers/droneAirspace.js", "src/data/cyclonesProxy.test.mjs", "src/data/firePerimetersProxy.test.mjs", + "src/data/droneAirspaceProxy.test.mjs", "src/layers/cyclones/index.js", "src/layers/cyclones/rendering.js", "src/layers/cyclones/labels.js", diff --git a/scripts/package-boundaries.json b/scripts/package-boundaries.json index 497be5b08..ed1fb5c4d 100644 --- a/scripts/package-boundaries.json +++ b/scripts/package-boundaries.json @@ -2023,6 +2023,11 @@ "src/layers/perimeters/records.js", "src/layers/perimeters/cards.js", "src/layers/perimeters/inciweb.js", + "src/app/layers/droneAirspace.js", + "src/layers/droneAirspace/index.js", + "src/layers/droneAirspace/source.js", + "src/layers/droneAirspace/records.js", + "src/layers/droneAirspace/cards.js", "src/layers/cctv/sourcePolicy.js", "src/sources/reference.js", "src/search/nominatim.js", @@ -2602,6 +2607,11 @@ "src/layers/perimeters/records.js", "src/layers/perimeters/cards.js", "src/layers/perimeters/inciweb.js", + "src/app/layers/droneAirspace.js", + "src/layers/droneAirspace/index.js", + "src/layers/droneAirspace/source.js", + "src/layers/droneAirspace/records.js", + "src/layers/droneAirspace/cards.js", "src/layers/cctv/sourcePolicy.js", "src/search/nominatim.js", "src/nominatimGeocode.js", @@ -3097,6 +3107,8 @@ "src/sources/httpBody.js", "src/layers/perimeters/source.js", "src/layers/perimeters/records.js", + "src/layers/droneAirspace/source.js", + "src/layers/droneAirspace/records.js", "src/layers/submarineCables/bundledSource.js" ], "external": [] @@ -3317,6 +3329,26 @@ ], "external": [] }, + "drone-airspace": { + "exports": ["./layers/drone-airspace"], + "modules": [ + "src/layers/droneAirspace/index.js", + "src/sources/httpBody.js", + "src/layers/droneAirspace/source.js", + "src/layers/droneAirspace/records.js", + "src/layers/droneAirspace/cards.js" + ], + "external": ["cesium"] + }, + "drone-airspace-source": { + "exports": ["./layers/drone-airspace/source"], + "modules": [ + "src/layers/droneAirspace/records.js", + "src/sources/httpBody.js", + "src/layers/droneAirspace/source.js" + ], + "external": [] + }, "tools": { "exports": ["./tools"], "modules": [ diff --git a/server/providers/droneAirspace.js b/server/providers/droneAirspace.js new file mode 100644 index 000000000..3a1905fc9 --- /dev/null +++ b/server/providers/droneAirspace.js @@ -0,0 +1,135 @@ +import { normalizeDroneAirspaceSnapshot } from '../../src/layers/droneAirspace/records.js'; +import { readResponseJsonCapped, coalesceProxyRequest } from './common/http.js'; +import { makeRateLimiter, clientKey } from './common/rate-limit.js'; + +// FAA Special Use Airspace (public, keyless). TYPE_CODE P (prohibited) and +// R (restricted) only — the small, always-in-force slice of the national +// SUA table. maxAllowableOffset trades ~100 m of boundary fidelity for a +// payload small enough to refresh (~330 KB for the current P+R set). +const API_URL = + 'https://services6.arcgis.com/ssFJjBXIUyZDrSYZ/arcgis/rest/services/' + + 'Special_Use_Airspace/FeatureServer/0/query?' + + new URLSearchParams({ + where: "TYPE_CODE IN ('P','R')", + outFields: [ + 'NAME', + 'TYPE_CODE', + 'UPPER_DESC', + 'LOWER_DESC', + 'TIMESOFUSE', + 'CONT_AGENT', + 'CITY', + 'STATE', + 'OBJECTID', + ].join(','), + maxAllowableOffset: '0.001', + outSR: '4326', + f: 'geojson', + }).toString(); + +const MAX_PAGES = 5; +const MIB = 1024 * 1024; +const SNAPSHOT_TTL_MS = 3_600_000; + +const exceededTransferLimit = (payload) => + payload?.exceededTransferLimit === true || + payload?.properties?.exceededTransferLimit === true; + +/** Fixed-origin, bounded FAA SUA routes for dev and preview. */ +export function droneAirspaceProxy({ + fetchImpl = (...args) => globalThis.fetch(...args), + now = () => Date.now(), +} = {}) { + const cache = new Map(); + const inFlight = new Map(); + const allow = makeRateLimiter({ windowMs: 60_000, max: 60, globalMax: 1200 }); + + async function upstream(url, cap, timeout) { + const signal = AbortSignal.timeout(timeout); + const response = await fetchImpl(url, { + signal, + redirect: 'error', + }); + if (!response.ok) { + await response.body?.cancel(); + throw new Error('upstream_unavailable'); + } + return readResponseJsonCapped(response, cap, signal); + } + + async function fetchZones() { + const features = []; + for (let page = 0; page < MAX_PAGES; page++) { + const url = + page === 0 ? API_URL : `${API_URL}&resultOffset=${features.length}`; + const payload = await upstream(url, 8 * MIB, 30_000); + if (!Array.isArray(payload?.features)) + throw new Error('invalid_snapshot'); + features.push(...payload.features); + if (!exceededTransferLimit(payload) || !payload.features.length) break; + } + const rows = normalizeDroneAirspaceSnapshot({ features }); + if (!rows) throw new Error('invalid_snapshot'); + return { fetchedAt: now(), rows }; + } + + async function acquire(key, ttl, load) { + const previous = cache.get(key); + if (previous && now() - previous.savedAt < ttl) + return { value: previous.value, stale: false }; + if (!inFlight.has(key) && inFlight.size >= 256) + throw Object.assign(new Error('busy'), { status: 429 }); + try { + const { promise } = coalesceProxyRequest(inFlight, key, async () => { + const value = await load(); + cache.set(key, { value, savedAt: now() }); + return value; + }); + return { value: await promise, stale: false }; + } catch (error) { + if (previous) return { value: previous.value, stale: true }; + throw error; + } + } + + async function handler(req, res) { + const json = (status, value, stale = false) => { + if (res.destroyed) return; + res.writeHead(status, { + 'Content-Type': 'application/json', + 'Cache-Control': 'no-store', + ...(status === 405 ? { Allow: 'GET' } : {}), + ...(status === 429 ? { 'Retry-After': '60' } : {}), + ...(stale ? { 'X-Data-Stale': 'true' } : {}), + }); + res.end(JSON.stringify(value)); + }; + if (req.method !== 'GET') return json(405, { error: 'method_not_allowed' }); + const path = (req.url || '/').split('?')[0]; + if (path !== '/' && path !== '') + return json(404, { error: 'unknown_route' }); + if (!allow(clientKey(req))) return json(429, { error: 'rate_limited' }); + try { + const { value, stale } = await acquire( + 'zones', + SNAPSHOT_TTL_MS, + fetchZones, + ); + json(200, stale ? { ...value, stale: true } : value, stale); + } catch (error) { + json(error.status === 429 ? 429 : 502, { + error: 'drone_airspace_unavailable', + }); + } + } + + return { + name: 'drone-airspace', + configureServer({ middlewares }) { + middlewares.use('/api/drone-airspace', handler); + }, + configurePreviewServer({ middlewares }) { + middlewares.use('/api/drone-airspace', handler); + }, + }; +} diff --git a/server/providers/local.js b/server/providers/local.js index 9acbade31..e976455ed 100644 --- a/server/providers/local.js +++ b/server/providers/local.js @@ -24,6 +24,7 @@ import { mapillaryProxy } from './mapillary.js'; import { keySetupEndpoint } from '../standalone/key-setup.js'; import { weatherProxy } from './weather.js'; import { firePerimetersProxy } from './firePerimeters.js'; +import { droneAirspaceProxy } from './droneAirspace.js'; import { cycloneProxy } from './cyclones.js'; import { windProxy } from './wind.js'; @@ -60,6 +61,7 @@ function localProviderPlugins({ realtime } = {}) { cycloneProxy(), mapillaryProxy(), firePerimetersProxy(), + droneAirspaceProxy(), keySetupEndpoint(), ]; } diff --git a/src/app/constructCatalog.js b/src/app/constructCatalog.js index 80315fe85..fe36dd76a 100644 --- a/src/app/constructCatalog.js +++ b/src/app/constructCatalog.js @@ -25,6 +25,7 @@ import { createApplicationAwareness } from './layers/militaryAwareness.js'; import { createApplicationFirms } from './layers/firms.js'; import { createApplicationEarthquakes } from './layers/earthquakes.js'; import { createApplicationFirePerimeters } from './layers/perimeters.js'; +import { createApplicationDroneAirspace } from './layers/droneAirspace.js'; import { createApplicationCables } from './layers/submarineCables.js'; import { createApplicationStreetLevel } from './layers/streetLevel.js'; import { createInfrastructureLayers } from '../data/infrastructure.js'; @@ -120,6 +121,9 @@ export function createApplicationCatalog({ createApplicationFirePerimeters({ source: sources['fire-perimeters'], }), + createApplicationDroneAirspace({ + source: sources['drone-airspace'], + }), createApplicationAlpr({ surface, source: sources.alpr }), createApplicationStreetLevel({ surface, diff --git a/src/app/constructCatalog.test.mjs b/src/app/constructCatalog.test.mjs index f42e82c7c..e298580ad 100644 --- a/src/app/constructCatalog.test.mjs +++ b/src/app/constructCatalog.test.mjs @@ -40,7 +40,7 @@ test('catalogs construct distinct layers and classification from their supplied signal: b.signal, surface: fixtureSurface(b.signal), }); - assert.equal(first.layers.length, 30); + assert.equal(first.layers.length, 31); assert.ok(first.get('local-adsb'), 'Local ADS-B is registered'); assert.deepEqual( first.metadata.find(({ id }) => id === 'local-adsb'), @@ -60,6 +60,7 @@ test('catalogs construct distinct layers and classification from their supplied '2026-09-21T12:00:00.000Z', ); assert.ok(first.get('fire-perimeters')); + assert.ok(first.get('drone-airspace')); assert.ok(first.get('transit')); const order = first.layers.map(({ id }) => id); assert.deepEqual( @@ -153,7 +154,7 @@ test('Mapillary can be omitted or replaced without changing catalog membership', surface: fixtureSurface(lifetime.signal), }; const catalog = createApplicationCatalog(options); - assert.equal(catalog.layers.length, 30); + assert.equal(catalog.layers.length, 31); assert.deepEqual(catalog.get('street-level').providerIds, []); assert.equal( catalogControlServices(catalog).streetLevelLayer, diff --git a/src/app/layers/droneAirspace.js b/src/app/layers/droneAirspace.js new file mode 100644 index 000000000..f3472fec4 --- /dev/null +++ b/src/app/layers/droneAirspace.js @@ -0,0 +1,22 @@ +import * as Cesium from 'cesium'; +import { createDroneAirspaceLayer } from '../../layers/droneAirspace/index.js'; +import * as picking from '../../data/pickRegistry.js'; +import * as overlays from '../../overlays/worldOverlay.js'; +import { isPointerFree } from '../../data/inputOwnership.js'; + +/** Wire FAA special-use airspace into the application catalog. */ +export function createApplicationDroneAirspace(options) { + return createDroneAirspaceLayer({ + overlayHost: { + setEntries: overlays.setOverlayEntries, + setVisible: overlays.setOverlaySourceVisible, + clearSource: overlays.clearOverlaySource, + hitTest: overlays.hitTestWorldOverlay, + }, + screenSpaceEventHandlerFactory: (viewer) => + new Cesium.ScreenSpaceEventHandler(viewer.scene.canvas), + picking, + pointer: { isPointerFree }, + ...options, + }); +} diff --git a/src/app/sourceComposition.js b/src/app/sourceComposition.js index 14c864ac5..c9f79e407 100644 --- a/src/app/sourceComposition.js +++ b/src/app/sourceComposition.js @@ -46,6 +46,7 @@ export const CATALOG_SOURCE_CONTRACTS = Object.freeze({ cyclones: contract(['weather-cyclones'], ['getSnapshot']), earthquakes: contract(['earthquakes'], ['getSnapshot']), 'fire-perimeters': contract(['fire-perimeters'], ['getSnapshot']), + 'drone-airspace': contract(['drone-airspace'], ['getSnapshot']), cables: contract(['telegeography-submarine-cables'], ['fetch']), transit: contract(['transit'], ['requestSnapshot', 'getHistory']), }); diff --git a/src/app/sourceComposition.test.mjs b/src/app/sourceComposition.test.mjs index 44a7c8fa0..cf8646e25 100644 --- a/src/app/sourceComposition.test.mjs +++ b/src/app/sourceComposition.test.mjs @@ -37,7 +37,7 @@ test('every acquisition source can be omitted without losing catalog membership' test('all acquisition sources may be absent, and enable fails before initializing or fetching', async (t) => { const actual = catalog({}, t); - assert.equal(actual.layers.length, 30); + assert.equal(actual.layers.length, 31); const manager = new LayerLifecycle( {}, { getSourceAvailability: actual.getSourceAvailability }, diff --git a/src/data/dataCredits.js b/src/data/dataCredits.js index 2967f0a2b..2e4a8148e 100644 --- a/src/data/dataCredits.js +++ b/src/data/dataCredits.js @@ -160,6 +160,13 @@ export const DATA_CREDITS = [ ' · Incident information: ' + 'InciWeb', }, + { + key: 'faa-sua', + html: + 'Drone restrictions: ' + + 'FAA Special Use Airspace' + + ' (prohibited and restricted zones; informational only)', + }, OSM_CREDIT, OPENMAPTILES_CREDIT, { diff --git a/src/data/droneAirspaceProxy.test.mjs b/src/data/droneAirspaceProxy.test.mjs new file mode 100644 index 000000000..52eb0dcc8 --- /dev/null +++ b/src/data/droneAirspaceProxy.test.mjs @@ -0,0 +1,146 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { droneAirspaceProxy } from '../../server/providers/droneAirspace.js'; + +const feature = (id = 310, type = 'P') => ({ + properties: { + NAME: `${type}-${id}`, + TYPE_CODE: type, + OBJECTID: id, + UPPER_DESC: 'TNI', + }, + geometry: { + type: 'Polygon', + coordinates: [ + [ + [-77, 38], + [-76, 38], + [-76, 39], + [-77, 38], + ], + ], + }, +}); + +function install(options = {}, hook = 'configureServer') { + let handler; + const plugin = droneAirspaceProxy(options); + assert.equal(typeof plugin.configureServer, 'function'); + assert.equal(typeof plugin.configurePreviewServer, 'function'); + plugin[hook]({ + middlewares: { + use(path, callback) { + assert.equal(path, '/api/drone-airspace'); + handler = callback; + }, + }, + }); + return async (url = '/', method = 'GET', peer = 'local') => { + const res = { + writeHead(status, headers) { + this.status = status; + this.headers = headers; + }, + end(body) { + this.body = JSON.parse(body); + }, + }; + await handler({ url, method, socket: { remoteAddress: peer } }, res); + return res; + }; +} + +for (const hook of ['configureServer', 'configurePreviewServer']) { + test(`${hook}: paging stops at five pages and normalizes the combined snapshot`, async () => { + const calls = []; + const request = install( + { + now: () => 1234, + fetchImpl: async (url, options) => { + calls.push(new URL(url)); + assert.ok(options.signal instanceof AbortSignal); + assert.equal(options.redirect, 'error'); + return Response.json({ + features: [feature(calls.length)], + exceededTransferLimit: true, + }); + }, + }, + hook, + ); + const res = await request('/?url=https://invalid.example&resultOffset=999'); + assert.equal(res.status, 200); + assert.equal(res.body.fetchedAt, 1234); + assert.equal(res.body.rows.length, 5); + assert.equal(calls.length, 5); + assert.equal(calls[0].searchParams.get('where'), "TYPE_CODE IN ('P','R')"); + assert.equal(calls[0].searchParams.get('f'), 'geojson'); + assert.equal(calls[0].searchParams.has('resultOffset'), false); + assert.equal(calls[1].searchParams.get('resultOffset'), '1'); + }); +} + +test('a cached snapshot is reused until TTL, then refreshed', async () => { + let now = 1000; + let fetches = 0; + const request = install({ + now: () => now, + fetchImpl: async () => { + fetches += 1; + return Response.json({ features: [feature(fetches)] }); + }, + }); + const first = await request(); + now += 3_599_000; + const cached = await request(); + now += 2000; + const refreshed = await request(); + assert.equal(first.body.rows[0].stableId, '1'); + assert.equal(cached.body.rows[0].stableId, '1'); + assert.equal(refreshed.body.rows[0].stableId, '2'); + assert.equal(fetches, 2); +}); + +test('a failed refresh serves the last good snapshot as stale', async () => { + let now = 1000; + let fail = false; + const request = install({ + now: () => now, + fetchImpl: async () => { + if (fail) return new Response('nope', { status: 503 }); + return Response.json({ features: [feature(7)] }); + }, + }); + const first = await request(); + fail = true; + now += 3_600_001; + const stale = await request(); + assert.equal(first.status, 200); + assert.equal(stale.status, 200); + assert.equal(stale.headers['X-Data-Stale'], 'true'); + assert.equal(stale.body.stale, true); + assert.equal(stale.body.rows[0].stableId, '7'); +}); + +test('method, unknown routes and rate limits are rejected', async () => { + const request = install({ + fetchImpl: async () => Response.json({ features: [feature()] }), + }); + assert.equal((await request('/', 'POST')).status, 405); + assert.equal((await request('/nope')).status, 404); + const limited = install({ + fetchImpl: async () => Response.json({ features: [feature()] }), + }); + let last; + for (let i = 0; i < 61; i++) last = await limited('/', 'GET', '10.0.0.9'); + assert.equal(last.status, 429); +}); + +test('a cold upstream failure is 502, not an empty snapshot', async () => { + const request = install({ + fetchImpl: async () => new Response('nope', { status: 503 }), + }); + const res = await request(); + assert.equal(res.status, 502); + assert.equal(res.body.error, 'drone_airspace_unavailable'); +}); diff --git a/src/data/layerState.js b/src/data/layerState.js index 373eac20d..bc5c660ca 100644 --- a/src/data/layerState.js +++ b/src/data/layerState.js @@ -595,6 +595,11 @@ export const LAYER_STATE_REGISTRY = Object.freeze([ optionOwner: 'cctv', }), Object.freeze({ id: 'directions', token: 'n', disposition: 'enabled-only' }), + Object.freeze({ + id: 'drone-airspace', + token: '3', + disposition: 'enabled-only', + }), Object.freeze({ id: 'earthquakes', token: 'e', disposition: 'enabled-only' }), Object.freeze({ id: 'fire-perimeters', diff --git a/src/data/layerState.test.mjs b/src/data/layerState.test.mjs index 08120a604..0fbdcf000 100644 --- a/src/data/layerState.test.mjs +++ b/src/data/layerState.test.mjs @@ -202,8 +202,8 @@ function encode(state) { test('production registry is exact, canonical, and rejects incomplete contracts', async () => { assert.equal(validateLayerStateRegistry(), true); - assert.equal(REGISTERED_LAYER_IDS.length, 29); - assert.equal(new Set(REGISTERED_LAYER_IDS).size, 29); + assert.equal(REGISTERED_LAYER_IDS.length, 30); + assert.equal(new Set(REGISTERED_LAYER_IDS).size, 30); assert.ok(REGISTERED_LAYER_IDS.includes('transit')); assert.deepEqual(REGISTERED_LAYER_IDS, [...REGISTERED_LAYER_IDS].sort()); assert.deepEqual(LEGACY_LAYER_STATE_TOKENS, { @@ -242,7 +242,7 @@ test('production registry is exact, canonical, and rejects incomplete contracts' for (const [id, token] of Object.entries(LEGACY_LAYER_STATE_TOKENS)) { assert.equal(LAYER_STATE_TOKEN_RESERVATIONS[id], token); } - assert.equal(nextLayerStateToken(), '3'); + assert.equal(nextLayerStateToken(), '4'); assert.equal( nextLayerStateToken({ ...LAYER_STATE_TOKEN_RESERVATIONS, @@ -307,8 +307,8 @@ test('production registry is exact, canonical, and rejects incomplete contracts' assert.equal( validateLayerStateAllocations(LAYER_STATE_TOKEN_RESERVATIONS, { ...LAYER_STATE_TOKEN_RESERVATIONS, - future: '3', - next: '4', + future: '4', + next: '5', }), true, ); @@ -318,7 +318,7 @@ test('production registry is exact, canonical, and rejects incomplete contracts' ...LAYER_STATE_TOKEN_RESERVATIONS, future: '00', }), - /next free token 3/, + /next free token 4/, ); const beforeLastDigit = { ...digitsExhausted }; delete beforeLastDigit['prior-9']; @@ -341,10 +341,10 @@ test('production registry is exact, canonical, and rejects incomplete contracts' assert.throws( () => validateLayerStateAllocations( - { ...LAYER_STATE_TOKEN_RESERVATIONS, merged: '3' }, - { ...LAYER_STATE_TOKEN_RESERVATIONS, merged: '3', competing: '3' }, + { ...LAYER_STATE_TOKEN_RESERVATIONS, merged: '4' }, + { ...LAYER_STATE_TOKEN_RESERVATIONS, merged: '4', competing: '4' }, ), - /next free token 4/, + /next free token 5/, ); assert.throws( () => validateLayerStateAllocations({ future: '00' }, { future: '01' }), @@ -2639,6 +2639,22 @@ test('the recent-imagery split is share-link only: never stored locally, and a s ); }); +test('drone restrictions uses digit 3 without colliding with fire perimeters', () => { + const decoded = decodeLayerStateParams(new URLSearchParams('v=2&l=3.2')); + assert.deepEqual(decoded.enabledLayerIds, [ + 'drone-airspace', + 'fire-perimeters', + ]); + assert.equal( + LAYER_STATE_REGISTRY.find(({ id }) => id === 'drone-airspace').token, + '3', + ); + assert.deepEqual( + decodeLayerStateParams(new URLSearchParams(encode(decoded))), + decoded, + ); +}); + test('fire perimeters uses digit 2 without colliding with wind or recent imagery', () => { const decoded = decodeLayerStateParams(new URLSearchParams('v=2&l=2.k.1')); assert.deepEqual(decoded.enabledLayerIds, [ diff --git a/src/data/layerStateTokenLedger.test.mjs b/src/data/layerStateTokenLedger.test.mjs index 5a02d85cf..56b2e1582 100644 --- a/src/data/layerStateTokenLedger.test.mjs +++ b/src/data/layerStateTokenLedger.test.mjs @@ -128,10 +128,14 @@ test('reservation ledger is complete, pinned, and rejects duplicate or malformed { ...parseLayerStateTokenReservations(reservationRows) }, { ...LAYER_STATE_TOKEN_RESERVATIONS }, ); - assert.equal(Object.keys(LAYER_STATE_TOKEN_RESERVATIONS).length, 29); + assert.equal(Object.keys(LAYER_STATE_TOKEN_RESERVATIONS).length, 30); assert.deepEqual( { ...LAYER_STATE_TOKEN_RESERVATIONS }, - { ...LEGACY_LAYER_STATE_TOKENS, 'street-level': '0' }, + { + ...LEGACY_LAYER_STATE_TOKENS, + 'street-level': '0', + 'drone-airspace': '3', + }, ); assert.throws( () => parseLayerStateTokenReservations(reservationRows.slice(1)), @@ -151,9 +155,9 @@ test('reservation ledger is complete, pinned, and rejects duplicate or malformed /Duplicate layer-state token reservation/, ); assert.equal( - parseLayerStateTokenReservations([...reservationRows, ['future', '3']]) + parseLayerStateTokenReservations([...reservationRows, ['future', '4']]) .future, - '3', + '4', ); assert.throws( () => @@ -171,14 +175,14 @@ test('reservation ledger is complete, pinned, and rejects duplicate or malformed test('published retired digits and competing PRs advance the merge-time allocation', () => { withPublishedBase( - { rows: [...reservationRows, ['retired-layer', '3']] }, + { rows: [...reservationRows, ['retired-layer', '4']] }, (cwd) => { const published = readPublishedLayerStateReservations('HEAD', cwd); - assert.equal(nextLayerStateToken(published), '4'); + assert.equal(nextLayerStateToken(published), '5'); assert.equal( validateLayerStateAllocations(published, { ...published, - newcomer: '4', + newcomer: '5', }), true, ); @@ -186,15 +190,15 @@ test('published retired digits and competing PRs advance the merge-time allocati () => validateLayerStateAllocations(published, { ...published, - competing: '3', + competing: '4', }), - /next free token 4/, + /next free token 5/, ); assert.throws( () => validateLayerStateAllocations(published, { ...LAYER_STATE_TOKEN_RESERVATIONS, - newcomer: '3', + newcomer: '4', }), /changed or removed: retired-layer/, ); @@ -202,8 +206,8 @@ test('published retired digits and competing PRs advance the merge-time allocati ); }); -test('PR B manually replaces provisional 3 with 4 after PR A publishes 3', () => { - withPublishedBase({ rows: [...reservationRows, ['pr-a', '3']] }, (cwd) => { +test('PR B manually replaces provisional 4 with 5 after PR A publishes 4', () => { + withPublishedBase({ rows: [...reservationRows, ['pr-a', '4']] }, (cwd) => { mkdirSync(path.join(cwd, 'scripts')); const fixtureChecker = path.join( cwd, @@ -224,8 +228,8 @@ test('PR B manually replaces provisional 3 with 4 after PR A publishes 3', () => writeCodecFixture( cwd, - [...reservationRows, ['pr-b', '3']], - [{ id: 'pr-b', token: '3' }], + [...reservationRows, ['pr-b', '4']], + [{ id: 'pr-b', token: '4' }], ); const beforeRebase = runCheck(); assert.equal(beforeRebase.status, 1); @@ -233,10 +237,10 @@ test('PR B manually replaces provisional 3 with 4 after PR A publishes 3', () => writeCodecFixture( cwd, - [...reservationRows, ['pr-a', '3'], ['pr-b', '3']], + [...reservationRows, ['pr-a', '4'], ['pr-b', '4']], [ - { id: 'pr-a', token: '3' }, - { id: 'pr-b', token: '3' }, + { id: 'pr-a', token: '4' }, + { id: 'pr-b', token: '4' }, ], ); const staleRebase = runCheck(); @@ -244,16 +248,16 @@ test('PR B manually replaces provisional 3 with 4 after PR A publishes 3', () => assert.match(staleRebase.stderr, /Duplicate layer-state token reservation/); const published = readPublishedLayerStateReservations('HEAD', cwd); - assert.equal(nextLayerStateToken(published), '4'); - const validBaselineRows = [...reservationRows, ['pr-a', '3']]; - writeCodecFixture(cwd, validBaselineRows, [{ id: 'pr-a', token: '3' }]); + assert.equal(nextLayerStateToken(published), '5'); + const validBaselineRows = [...reservationRows, ['pr-a', '4']]; + writeCodecFixture(cwd, validBaselineRows, [{ id: 'pr-a', token: '4' }]); const helper = spawnSync( process.execPath, [realpathSync(fixtureNext), 'pr-b'], { cwd, encoding: 'utf8' }, ); assert.equal(helper.status, 0, helper.stderr); - assert.match(helper.stdout, /pr-b: 4/); + assert.match(helper.stdout, /pr-b: 5/); assert.deepEqual( JSON.parse(readFileSync(path.join(cwd, LEDGER_PATH), 'utf8')), validBaselineRows, @@ -261,20 +265,20 @@ test('PR B manually replaces provisional 3 with 4 after PR A publishes 3', () => ); writeCodecFixture( cwd, - [...reservationRows, ['pr-a', '3'], ['pr-b', '4']], + [...reservationRows, ['pr-a', '4'], ['pr-b', '5']], [ - { id: 'pr-a', token: '3' }, - { id: 'pr-b', token: '4' }, + { id: 'pr-a', token: '4' }, + { id: 'pr-b', token: '5' }, ], ); const corrected = runCheck(); assert.equal(corrected.status, 0, corrected.stderr); - assert.match(corrected.stdout, /30 published, 1 new/); + assert.match(corrected.stdout, /31 published, 1 new/); }); }); test('allocation batches cross the last digit and base-36 pair boundaries in order', () => { - const digitRows = [...'3456789'].map((digit) => [`retired-${digit}`, digit]); + const digitRows = [...'456789'].map((digit) => [`retired-${digit}`, digit]); const allDigits = parseLayerStateTokenReservations([ ...reservationRows, ...digitRows, @@ -326,7 +330,7 @@ test('allocation batches cross the last digit and base-36 pair boundaries in ord test('an isolated valid two-character fixture round-trips an l field beyond the old 64-character cap', async () => { const cwd = mkdtempSync(path.join(tmpdir(), 'gev-layer-token-width-')); try { - const priorDigits = [...'3456789'].map((digit) => [ + const priorDigits = [...'456789'].map((digit) => [ `qa-prior-digit-${digit}`, digit, ]); @@ -363,7 +367,7 @@ test('an isolated valid two-character fixture round-trips an l field beyond the const params = new URLSearchParams([['v', '2']]); codec.encodeLayerStateParams(params, state); assert.ok(params.get('l').length > 64, 'fixture must cross the old cap'); - assert.equal(params.get('l').length, 69); + assert.equal(params.get('l').length, 71); const restored = codec.decodeLayerStateParams(params); assert.deepEqual(restored?.enabledLayerIds, expectedLayerIds); assert.deepEqual(restored?.options, state.options); @@ -381,7 +385,7 @@ test('checker reads a complete future base ledger and rejects retired-token reus }, (cwd) => { const published = readPublishedLayerStateReservations('HEAD', cwd); - assert.equal(Object.keys(published).length, 30); + assert.equal(Object.keys(published).length, 31); assert.equal(published['retired-layer'], '00'); assert.throws( () => diff --git a/src/data/layerStateTokenReservations.json b/src/data/layerStateTokenReservations.json index eb49eed27..6024f5248 100644 --- a/src/data/layerStateTokenReservations.json +++ b/src/data/layerStateTokenReservations.json @@ -6,6 +6,7 @@ ["bikeshare", "b"], ["cctv", "c"], ["directions", "n"], + ["drone-airspace", "3"], ["earthquakes", "e"], ["fire-perimeters", "2"], ["flights", "f"], diff --git a/src/firstRunExperience.test.mjs b/src/firstRunExperience.test.mjs index 49711163f..cd8efce77 100644 --- a/src/firstRunExperience.test.mjs +++ b/src/firstRunExperience.test.mjs @@ -674,10 +674,11 @@ test('the voice TOOL SCHEMA matches the pinned contract — the mission mapping // schema, which may not carry top-level anyOf), and Contacts requested-radius // list descriptions; the missions still ride existing tools. // Street Level adds toggle enum values and the generated alias hint. - assert.equal(block.length, 29802, 'serialized tool schema length drifted'); + // Drone Restrictions adds toggle/query enum values and spoken aliases. + assert.equal(block.length, 29988, 'serialized tool schema length drifted'); assert.equal( crypto.createHash('sha256').update(block).digest('hex'), - 'b316ca2e67848eb565b6847ac044a19d1465d8ef8252a5dfbd7d947a2a36d5ed', + 'e3605286e8b91c07f29b687cb3cf0e9400fef92225fbc7efeaa35451274df571', 'the first-run missions must ride EXISTING tools: no schema edit, no cache bust', ); const instructions = fs.readFileSync(new URL('../server/providers/openai/instructions.js', import.meta.url), 'utf8'); diff --git a/src/layers/droneAirspace/cards.js b/src/layers/droneAirspace/cards.js new file mode 100644 index 000000000..eb6cc6433 --- /dev/null +++ b/src/layers/droneAirspace/cards.js @@ -0,0 +1,93 @@ +/** Card model for one selected special-use airspace zone. Pure — no Cesium types. */ + +export const DRONE_AIRSPACE_OVERLAY_SOURCE_ID = 'drone-airspace'; + +/** Fill/line CSS color for a zone by TYPE_CODE. */ +export function droneAirspaceAccent(typeCode) { + return typeCode === 'P' ? '#ff3b30' : '#ff7a00'; +} + +/** Shoelace area (degree², sign dropped) — relative sizes only. */ +function ringArea(ring) { + let sum = 0; + for (let i = 0; i < ring.length - 1; i++) { + const [x1, y1] = ring[i]; + const [x2, y2] = ring[i + 1]; + sum += x1 * y2 - x2 * y1; + } + return Math.abs(sum / 2); +} + +/** + * Anchor point for the zone card: the centroid of the largest polygon's + * outer ring. Degrees, so callers own the Cartesian conversion. + * @param {Array>>} polygons + * @returns {{lon: number, lat: number}} + */ +export function droneAirspaceAnchorDegrees(polygons) { + let best = null; + let bestArea = -1; + for (const rings of polygons) { + const outer = rings[0]; + const area = ringArea(outer); + if (area > bestArea) { + bestArea = area; + best = outer; + } + } + let lon = 0; + let lat = 0; + const count = best.length - 1; + for (let i = 0; i < count; i++) { + lon += best[i][0]; + lat += best[i][1]; + } + return { lon: lon / count, lat: lat / count }; +} + +function altitudeLine(row) { + const lower = row.lowerDesc; + const upper = row.upperDesc; + if (lower && upper) return `${lower} – ${upper}`; + if (upper) return `up to ${upper}`; + if (lower) return `from ${lower}`; + return null; +} + +function placeLine(row) { + if (row.city && row.state) return `${row.city}, ${row.state}`; + return row.city || row.state || null; +} + +/** + * Build the overlay-host entry for one selected zone. The caller supplies + * `position` (Cartesian) separately — this model stays JSON-safe for tests. + * @param {Object} row - Normalized zone row (see records.js). + * @returns {Object} World-overlay entry without `position`. + */ +export function buildDroneAirspaceCard(row) { + const facts = [row.type || row.typeCode].filter(Boolean); + const altitude = altitudeLine(row); + if (altitude) facts.push(altitude); + if (row.timesOfUse) facts.push(row.timesOfUse); + + const details = [facts.join(' · ')]; + const place = placeLine(row); + if (place) details.push(place); + if (row.agency) details.push(row.agency); + details.push('Informational only · not for flight planning'); + + const title = `${(row.type || 'SUA').toUpperCase()} · ${row.name || 'Unnamed zone'}`; + return { + id: `drone-airspace-card:${row.stableId}`, + selected: true, + interactive: false, + title, + details, + accent: droneAirspaceAccent(row.typeCode), + priority: Number.MAX_SAFE_INTEGER, + gapPx: 15, + verticalOnly: true, + placement: 'above', + }; +} diff --git a/src/layers/droneAirspace/cards.test.mjs b/src/layers/droneAirspace/cards.test.mjs new file mode 100644 index 000000000..e8d58c37c --- /dev/null +++ b/src/layers/droneAirspace/cards.test.mjs @@ -0,0 +1,74 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { + droneAirspaceAnchorDegrees, + droneAirspaceAccent, + buildDroneAirspaceCard, +} from './cards.js'; + +const square = [ + [-77.2, 38.8], + [-77.0, 38.8], + [-77.0, 39.0], + [-77.2, 39.0], + [-77.2, 38.8], +]; +const tiny = [ + [-76.01, 38.0], + [-76.0, 38.0], + [-76.0, 38.01], + [-76.01, 38.0], +]; + +test('the card anchors at the centroid of the largest polygon', () => { + const anchor = droneAirspaceAnchorDegrees([[tiny], [square]]); + assert.ok(Math.abs(anchor.lon - -77.1) < 1e-9); + assert.ok(Math.abs(anchor.lat - 38.9) < 1e-9); +}); + +test('a prohibited zone card names the type, altitudes and the disclaimer', () => { + const card = buildDroneAirspaceCard({ + stableId: '310', + name: 'P-56', + typeCode: 'P', + type: 'Prohibited', + lowerDesc: 'SFC', + upperDesc: 'TNI', + timesOfUse: 'CONTINUOUS', + agency: 'WASHINGTON CENTER', + city: 'WASHINGTON', + state: 'DC', + }); + assert.equal(card.id, 'drone-airspace-card:310'); + assert.equal(card.title, 'PROHIBITED · P-56'); + assert.deepEqual(card.details, [ + 'Prohibited · SFC – TNI · CONTINUOUS', + 'WASHINGTON, DC', + 'WASHINGTON CENTER', + 'Informational only · not for flight planning', + ]); + assert.equal(card.accent, droneAirspaceAccent('P')); + assert.equal(card.interactive, false); + assert.equal(card.selected, true); +}); + +test('missing attributes degrade to available facts instead of placeholders', () => { + const card = buildDroneAirspaceCard({ + stableId: 'x', + name: null, + typeCode: 'R', + type: 'Restricted', + lowerDesc: null, + upperDesc: null, + timesOfUse: null, + agency: null, + city: null, + state: null, + }); + assert.equal(card.title, 'RESTRICTED · Unnamed zone'); + assert.deepEqual(card.details, [ + 'Restricted', + 'Informational only · not for flight planning', + ]); + assert.equal(card.accent, droneAirspaceAccent('R')); +}); diff --git a/src/layers/droneAirspace/index.js b/src/layers/droneAirspace/index.js new file mode 100644 index 000000000..d7b4a40c8 --- /dev/null +++ b/src/layers/droneAirspace/index.js @@ -0,0 +1,326 @@ +import * as Cesium from 'cesium'; +import { + DRONE_AIRSPACE_OVERLAY_SOURCE_ID, + droneAirspaceAnchorDegrees, + buildDroneAirspaceCard, + droneAirspaceAccent, +} from './cards.js'; +export { normalizeDroneAirspaceSnapshot } from './records.js'; +export { createDroneAirspaceSource } from './source.js'; +export * from './cards.js'; + +/** Fill/line color for a zone by TYPE_CODE. */ +export function droneAirspaceColor(typeCode) { + return Cesium.Color.fromCssColorString(droneAirspaceAccent(typeCode)); +} + +const ringPositions = (ring) => + ring.map(([lon, lat]) => Cesium.Cartesian3.fromDegrees(lon, lat)); + +const PICK_PREFIX = 'drone-airspace:'; +const CARD_HOST_OPTIONS = Object.freeze({ + cohortLimit: 1, + collisionCapacity: 1, + moving: false, +}); + +/** Own one drone-restriction display, its refresh lifecycle, and click selection. */ +export function createDroneAirspaceLayer({ + source, + overlayHost = null, + screenSpaceEventHandlerFactory = null, + picking = null, + pointer = null, +} = {}) { + if (typeof source?.getSnapshot !== 'function') + throw new TypeError('Drone airspace requires a snapshot source'); + let _viewer = null; + let _request = null; + let _snapshotSignature = null; + let _dataSource = null; + let _count = 0; + let _lastUpdate = null; + let _lastError = null; + let _enabled = false; + let _clickHandler = null; + let _selectedId = null; + let _selectedCardId = null; + /** @type {Map} */ + const _rowById = new Map(); + + const canSelect = () => + overlayHost && screenSpaceEventHandlerFactory && picking; + + function publishSelectedCard() { + if (!canSelect()) return; + const row = _selectedId ? _rowById.get(_selectedId) : null; + if (!row) { + _selectedId = null; + _selectedCardId = null; + overlayHost.setEntries( + DRONE_AIRSPACE_OVERLAY_SOURCE_ID, + [], + CARD_HOST_OPTIONS, + ); + return; + } + const card = { + ...buildDroneAirspaceCard(row), + position: Cesium.Cartesian3.fromDegrees(row.anchor.lon, row.anchor.lat), + }; + _selectedCardId = card.id; + overlayHost.setEntries( + DRONE_AIRSPACE_OVERLAY_SOURCE_ID, + [card], + CARD_HOST_OPTIONS, + ); + } + + /** Resolve a scene pick to one of this layer's zone ids, or null. */ + function pickedZoneId(picked) { + const pickId = picking.resolvePickId(picked); + if (typeof pickId !== 'string' || !pickId.startsWith(PICK_PREFIX)) + return null; + const zoneId = pickId.slice(PICK_PREFIX.length).split(':')[0]; + return _rowById.has(zoneId) ? zoneId : null; + } + + function installClickHandler() { + if (!canSelect() || _clickHandler || !_viewer) return; + // Deliberately NOT registered in the pick-ownership registry: sibling + // layers yield to owned picks before their own overlay-card hit-tests, + // so claiming these large ground polygons would make FIRMS/vessel/CCTV + // cards inert anywhere over a zone. Unowned, our picks read as + // "empty space" to siblings — the same contract as fire perimeters. + _clickHandler = screenSpaceEventHandlerFactory(_viewer); + _clickHandler.setInputAction((click) => { + if (pointer && !pointer.isPointerFree()) return; + const cardHit = overlayHost.hitTest?.( + click.position?.x, + click.position?.y, + { sourceId: DRONE_AIRSPACE_OVERLAY_SOURCE_ID }, + ); + if (cardHit && cardHit.entryId === _selectedCardId) return; + const picked = _viewer.scene.pick(click.position); + const zoneId = picked ? pickedZoneId(picked) : null; + if (zoneId) { + _selectedId = zoneId; + publishSelectedCard(); + return; + } + if (picked) { + const pickId = picking.resolvePickId(picked); + if (pickId && picking.isOwnedByOtherLayer(layer.id, pickId)) return; + } + if (_selectedId) { + _selectedId = null; + publishSelectedCard(); + } + }, Cesium.ScreenSpaceEventType.LEFT_CLICK); + } + + function removeClickHandler() { + if (_clickHandler) { + _clickHandler.destroy(); + _clickHandler = null; + } + } + + function clearSelection() { + _selectedId = null; + _selectedCardId = null; + if (overlayHost) { + overlayHost.clearSource(DRONE_AIRSPACE_OVERLAY_SOURCE_ID); + overlayHost.setVisible?.(DRONE_AIRSPACE_OVERLAY_SOURCE_ID, false); + } + } + + const layer = { + id: 'drone-airspace', + name: 'Drone Restrictions', + icon: '🚫', + source: 'FAA SUA', + updateInterval: 3600000, + + init(viewer) { + if (_viewer) + throw new Error('Drone airspace layer is already initialized'); + _viewer = viewer; + _dataSource = new Cesium.CustomDataSource('drone-airspace'); + _dataSource.show = false; + viewer.dataSources.add(_dataSource); + _count = 0; + _lastUpdate = null; + _lastError = null; + _enabled = false; + console.log('[Data:DroneAirspace] Initialized'); + }, + + enable() { + _enabled = true; + if (_dataSource) _dataSource.show = true; + overlayHost?.setVisible?.(DRONE_AIRSPACE_OVERLAY_SOURCE_ID, true); + installClickHandler(); + }, + + disable() { + _request?.abort(); + _request = null; + _enabled = false; + if (_dataSource) _dataSource.show = false; + removeClickHandler(); + clearSelection(); + }, + + async update() { + if (!_enabled || !_dataSource) return false; + _request?.abort(); + const request = new AbortController(); + _request = request; + try { + const rows = await source.getSnapshot({ signal: request.signal }); + if (request.signal.aborted || _request !== request || !_enabled) + return false; + + const signature = JSON.stringify( + rows + .map(({ polygons, ...facts }) => facts) + .sort((a, b) => a.stableId.localeCompare(b.stableId)), + ); + if (signature === _snapshotSignature) { + if (_selectedId) publishSelectedCard(); + _lastUpdate = Date.now(); + _lastError = null; + return true; + } + const nextEntities = []; + _rowById.clear(); + for (const row of rows) { + const color = droneAirspaceColor(row.typeCode); + for (const [index, rings] of row.polygons.entries()) { + const [outer, ...holes] = rings; + const outerPositions = ringPositions(outer); + nextEntities.push( + new Cesium.Entity({ + id: `drone-airspace:${row.stableId}:${index}`, + polygon: { + hierarchy: new Cesium.PolygonHierarchy( + outerPositions, + holes.map( + (hole) => + new Cesium.PolygonHierarchy(ringPositions(hole)), + ), + ), + material: new Cesium.ColorMaterialProperty( + color.withAlpha(0.25), + ), + }, + polyline: { + positions: outerPositions, + clampToGround: true, + width: 2, + material: new Cesium.ColorMaterialProperty( + color.withAlpha(0.9), + ), + }, + }), + ); + } + const { polygons, ...facts } = row; + _rowById.set(row.stableId, { + ...facts, + anchor: droneAirspaceAnchorDegrees(polygons), + }); + } + + _dataSource.entities.removeAll(); + for (const entity of nextEntities) _dataSource.entities.add(entity); + _snapshotSignature = signature; + if (_selectedId) publishSelectedCard(); + _count = rows.length; + _lastUpdate = Date.now(); + _lastError = null; + console.log( + `[Data:DroneAirspace] Updated: ${_count} zones, ${nextEntities.length} polygons`, + ); + return true; + } catch (e) { + if (request.signal.aborted || _request !== request || !_enabled) + return false; + console.warn('[Data:DroneAirspace] Fetch error:', e); + _lastError = e?.message || 'Drone airspace source unavailable'; + return false; + } finally { + if (_request === request) _request = null; + } + }, + + destroy(viewer = _viewer) { + _request?.abort(); + _request = null; + removeClickHandler(); + clearSelection(); + _rowById.clear(); + _snapshotSignature = null; + _viewer = null; + _enabled = false; + if (_dataSource) { + viewer.dataSources.remove(_dataSource, true); + _dataSource = null; + } + _count = 0; + _lastUpdate = null; + _lastError = null; + }, + + getAnalystRecords(maxCount = 2000) { + if (!_dataSource || !_dataSource.show) return []; + const limit = Number.isFinite(maxCount) + ? Math.max(1, Math.floor(maxCount)) + : 2000; + const result = []; + for (const row of _rowById.values()) { + if (result.length >= limit) break; + const { anchor, stableId, ...facts } = row; + result.push({ + id: stableId, + ...facts, + lat: anchor.lat, + lon: anchor.lon, + }); + } + return result; + }, + + getRowControls() { + const bands = [ + { label: 'Prohibited (P)', color: droneAirspaceAccent('P') }, + { label: 'Restricted (R)', color: droneAirspaceAccent('R') }, + ]; + return { + chips: [], + legend: bands.map((band, index) => ({ + ...band, + count: [..._rowById.values()].filter( + (row) => droneAirspaceAccent(row.typeCode) === band.color, + ).length, + ...(index === 0 + ? { + blurb: + 'FAA special-use airspace. Informational only — not a substitute for official UAS flight-planning tools (B4UFLY / LAANC).', + } + : {}), + })), + }; + }, + + getStats() { + return { + count: _count, + lastUpdate: _lastUpdate, + error: _lastError, + }; + }, + }; + return layer; +} diff --git a/src/layers/droneAirspace/ownership.test.mjs b/src/layers/droneAirspace/ownership.test.mjs new file mode 100644 index 000000000..ae10184f2 --- /dev/null +++ b/src/layers/droneAirspace/ownership.test.mjs @@ -0,0 +1,161 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { createDroneAirspaceLayer } from './index.js'; + +function harness(source, { pick = () => null, cardHit = () => null } = {}) { + const sources = []; + const overlay = { entries: new Map(), visible: null }; + const clicks = { handler: null, destroyed: 0 }; + const owners = new Map(); + const viewer = { + scene: { pick }, + dataSources: { + add(value) { + sources.push(value); + }, + remove(value) { + sources.splice(sources.indexOf(value), 1); + }, + }, + }; + const layer = createDroneAirspaceLayer({ + source, + overlayHost: { + setEntries(sourceId, entries) { + overlay.entries.set(sourceId, entries); + }, + setVisible(sourceId, visible) { + overlay.visible = visible; + }, + clearSource(sourceId) { + overlay.entries.delete(sourceId); + }, + hitTest: (x, y, options) => cardHit(x, y, options), + }, + screenSpaceEventHandlerFactory: () => ({ + setInputAction(callback) { + clicks.handler = callback; + }, + destroy() { + clicks.destroyed += 1; + clicks.handler = null; + }, + }), + picking: { + resolvePickId: (picked) => picked?.id ?? null, + isOwnedByOtherLayer: (layerId, pickedId) => + String(pickedId).startsWith('other-layer:'), + registerPickOwner: (layerId, predicate) => owners.set(layerId, predicate), + unregisterPickOwner: (layerId) => owners.delete(layerId), + }, + pointer: { isPointerFree: () => true }, + }); + layer.init(viewer); + layer.enable(viewer); + return { layer, viewer, sources, overlay, clicks, owners }; +} + +const ring = [ + [-77.04, 38.89], + [-77.03, 38.89], + [-77.03, 38.9], + [-77.04, 38.89], +]; +const row = { + stableId: '310', + name: 'P-56', + typeCode: 'P', + type: 'Prohibited', + lowerDesc: 'SFC', + upperDesc: 'TNI', + timesOfUse: 'CONTINUOUS', + agency: null, + city: 'WASHINGTON', + state: 'DC', + polygons: [[ring]], +}; + +test('each zone polygon renders as one filled entity with its outline', async () => { + const h = harness({ + getSnapshot: async () => [ + row, + { ...row, stableId: '400', typeCode: 'R', polygons: [[ring], [ring]] }, + ], + }); + assert.equal(await h.layer.update(h.viewer), true); + const entities = h.sources[0].entities.values; + assert.equal(entities.length, 3); + assert.equal(entities[0].id, 'drone-airspace:310:0'); + assert.ok(entities[0].polygon, 'zone fill missing'); + assert.ok(entities[0].polyline, 'zone outline missing'); + assert.equal(h.layer.getStats().count, 2); +}); + +test('clicking a zone publishes its card; empty space clears it', async () => { + let pickResult = null; + const h = harness( + { getSnapshot: async () => [row] }, + { pick: () => pickResult }, + ); + await h.layer.update(h.viewer); + assert.equal(typeof h.clicks.handler, 'function', 'click handler missing'); + assert.equal(h.owners.size, 0); + + pickResult = { id: 'drone-airspace:310:0' }; + h.clicks.handler({ position: { x: 10, y: 10 } }); + const entries = h.overlay.entries.get('drone-airspace'); + assert.equal(entries.length, 1); + assert.equal(entries[0].id, 'drone-airspace-card:310'); + assert.equal(entries[0].title, 'PROHIBITED · P-56'); + assert.ok(entries[0].position, 'card must carry a world anchor'); + + pickResult = { id: 'other-layer:aircraft-1' }; + h.clicks.handler({ position: { x: 10, y: 10 } }); + assert.equal( + h.overlay.entries.get('drone-airspace')?.length, + 1, + 'sibling-layer picks must not clear the selection', + ); + + pickResult = null; + h.clicks.handler({ position: { x: 10, y: 10 } }); + assert.equal(h.overlay.entries.get('drone-airspace')?.length ?? 0, 0); +}); + +test('late refresh cannot publish after disable or destroy', async () => { + for (const action of ['disable', 'destroy']) { + let resolve, signal; + const h = harness({ + getSnapshot(options) { + signal = options.signal; + return new Promise((done) => { + resolve = done; + }); + }, + }); + const pending = h.layer.update(h.viewer); + h.layer[action](h.viewer); + assert.equal(signal.aborted, true); + if (action === 'disable') h.layer.enable(h.viewer); + resolve([row]); + assert.equal(await pending, false); + assert.equal(h.layer.getStats().count, 0); + h.layer.destroy(h.viewer); + } +}); + +test('the row legend names prohibited and restricted with the disclaimer', async () => { + const h = harness({ + getSnapshot: async () => [ + row, + { ...row, stableId: '400', typeCode: 'R', type: 'Restricted' }, + ], + }); + await h.layer.update(h.viewer); + const legend = h.layer.getRowControls().legend; + assert.equal(legend[0].label, 'Prohibited (P)'); + assert.equal(legend[1].label, 'Restricted (R)'); + assert.equal(legend[0].count, 1); + assert.equal(legend[1].count, 1); + assert.match(legend[0].blurb, /not a substitute/); +}); diff --git a/src/layers/droneAirspace/records.js b/src/layers/droneAirspace/records.js new file mode 100644 index 000000000..c1d5f73e2 --- /dev/null +++ b/src/layers/droneAirspace/records.js @@ -0,0 +1,97 @@ +/** + * Normalize an FAA Special Use Airspace feed, skipping individually invalid + * features. Only a payload that is not a feature collection at all rejects + * the whole snapshot: a single degenerate polygon must never blank the layer. + */ + +const TYPE_LABELS = Object.freeze({ + P: 'Prohibited', + R: 'Restricted', +}); + +const finiteOrNull = (value) => (Number.isFinite(value) ? value : null); +const textOrNull = (value) => { + if (typeof value !== 'string') return null; + const text = value.trim(); + return text ? text : null; +}; + +function validRing(ring) { + if (!Array.isArray(ring) || ring.length < 4) return false; + for (const position of ring) { + if (!Array.isArray(position) || position.length < 2) return false; + const [lon, lat] = position; + if (!Number.isFinite(lon) || Math.abs(lon) > 180) return false; + if (!Number.isFinite(lat) || Math.abs(lat) > 90) return false; + } + return true; +} + +/** Normalize Polygon/MultiPolygon geometry to an array of polygons (each an + * array of rings). Returns null on malformed geometry, [] when empty. */ +function normalizePolygons(geometry) { + if (!geometry || typeof geometry !== 'object') return null; + let polygons; + if (geometry.type === 'Polygon') polygons = [geometry.coordinates]; + else if (geometry.type === 'MultiPolygon') polygons = geometry.coordinates; + else return null; + if (!Array.isArray(polygons)) return null; + const result = []; + for (const rings of polygons) { + if (!Array.isArray(rings)) return null; + if (!rings.length) continue; + if (!rings.every(validRing)) return null; + result.push(rings); + } + return result; +} + +/** Human label for a TYPE_CODE, or null when the code is not in the pack. */ +export function droneAirspaceTypeLabel(typeCode) { + return TYPE_LABELS[typeCode] || null; +} + +export function normalizeDroneAirspaceSnapshot(geojson) { + if (!Array.isArray(geojson?.features)) return null; + const rows = []; + const ids = new Set(); + for (const feature of geojson.features) { + const properties = feature?.properties; + if ( + !properties || + typeof properties !== 'object' || + Array.isArray(properties) + ) + continue; + const typeCode = textOrNull(properties.TYPE_CODE); + const type = droneAirspaceTypeLabel(typeCode); + if (!type) continue; + const polygons = normalizePolygons(feature.geometry); + if (polygons === null || !polygons.length) continue; + const objectId = finiteOrNull(Number(properties.OBJECTID)); + const featureId = + feature.id == null || feature.id === '' ? null : String(feature.id); + const stableId = + objectId != null + ? String(objectId) + : featureId && !ids.has(featureId) + ? featureId + : null; + if (stableId == null || ids.has(stableId)) continue; + ids.add(stableId); + rows.push({ + stableId, + name: textOrNull(properties.NAME), + typeCode, + type, + lowerDesc: textOrNull(properties.LOWER_DESC), + upperDesc: textOrNull(properties.UPPER_DESC), + timesOfUse: textOrNull(properties.TIMESOFUSE), + agency: textOrNull(properties.CONT_AGENT), + city: textOrNull(properties.CITY), + state: textOrNull(properties.STATE), + polygons, + }); + } + return rows; +} diff --git a/src/layers/droneAirspace/records.test.mjs b/src/layers/droneAirspace/records.test.mjs new file mode 100644 index 000000000..4c8c1a215 --- /dev/null +++ b/src/layers/droneAirspace/records.test.mjs @@ -0,0 +1,95 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { + droneAirspaceTypeLabel, + normalizeDroneAirspaceSnapshot, +} from './records.js'; + +const ring = [ + [-77.04, 38.89], + [-77.03, 38.89], + [-77.03, 38.9], + [-77.04, 38.89], +]; + +function feature(overrides = {}) { + return { + id: 310, + geometry: { type: 'Polygon', coordinates: [ring] }, + properties: { + NAME: 'P-56', + TYPE_CODE: 'P', + UPPER_DESC: 'TNI', + LOWER_DESC: 'SFC', + TIMESOFUSE: 'CONTINUOUS', + CONT_AGENT: 'WASHINGTON CENTER', + CITY: 'WASHINGTON', + STATE: 'DC', + OBJECTID: 310, + ...overrides.properties, + }, + ...overrides, + }; +} + +test('a valid snapshot yields one row per prohibited or restricted zone', () => { + const rows = normalizeDroneAirspaceSnapshot({ + features: [ + feature(), + feature({ + id: 400, + properties: { + NAME: 'R-2508', + TYPE_CODE: 'R', + OBJECTID: 400, + }, + }), + ], + }); + assert.equal(rows.length, 2); + assert.deepEqual(rows[0], { + stableId: '310', + name: 'P-56', + typeCode: 'P', + type: 'Prohibited', + lowerDesc: 'SFC', + upperDesc: 'TNI', + timesOfUse: 'CONTINUOUS', + agency: 'WASHINGTON CENTER', + city: 'WASHINGTON', + state: 'DC', + polygons: [[ring]], + }); + assert.equal(rows[1].type, 'Restricted'); + assert.equal(rows[1].typeCode, 'R'); +}); + +test('MOA, warning and advisory codes are dropped, as are bad geometries', () => { + const rows = normalizeDroneAirspaceSnapshot({ + features: [ + feature({ properties: { TYPE_CODE: 'MOA', OBJECTID: 1 } }), + feature({ properties: { TYPE_CODE: 'W', OBJECTID: 2 } }), + feature({ properties: { TYPE_CODE: 'A', OBJECTID: 3 } }), + feature({ geometry: null, properties: { OBJECTID: 4 } }), + feature({ + geometry: { type: 'Point', coordinates: [-77, 38] }, + properties: { OBJECTID: 5 }, + }), + feature(), + ], + }); + assert.equal(rows.length, 1); + assert.equal(rows[0].name, 'P-56'); +}); + +test('duplicate ids and a missing collection reject or skip as documented', () => { + const dup = normalizeDroneAirspaceSnapshot({ + features: [feature(), feature({ id: 311 })], + }); + assert.equal(dup.length, 1); + assert.equal(normalizeDroneAirspaceSnapshot({}), null); + assert.equal(normalizeDroneAirspaceSnapshot(null), null); + assert.equal(droneAirspaceTypeLabel('P'), 'Prohibited'); + assert.equal(droneAirspaceTypeLabel('R'), 'Restricted'); + assert.equal(droneAirspaceTypeLabel('MOA'), null); +}); diff --git a/src/layers/droneAirspace/source.js b/src/layers/droneAirspace/source.js new file mode 100644 index 000000000..17e67b3ae --- /dev/null +++ b/src/layers/droneAirspace/source.js @@ -0,0 +1,23 @@ +import { readResponseJsonCapped } from '../../sources/httpBody.js'; + +/** Request a normalized snapshot through the bounded, same-origin FAA proxy. */ +export function createDroneAirspaceSource({ + fetchImpl = (...args) => globalThis.fetch(...args), +} = {}) { + return { + async getSnapshot({ signal } = {}) { + signal?.throwIfAborted(); + const response = await fetchImpl('/api/drone-airspace', { signal }); + if (!response.ok) throw new Error(`FAA SUA HTTP ${response.status}`); + const payload = await readResponseJsonCapped( + response, + 80 * 1024 * 1024, + signal, + ); + signal?.throwIfAborted(); + if (!Array.isArray(payload?.rows)) + throw new Error('Malformed drone airspace snapshot'); + return payload.rows; + }, + }; +} diff --git a/src/layers/droneAirspace/source.test.mjs b/src/layers/droneAirspace/source.test.mjs new file mode 100644 index 000000000..160fb1789 --- /dev/null +++ b/src/layers/droneAirspace/source.test.mjs @@ -0,0 +1,105 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { createDroneAirspaceSource } from './source.js'; +import { normalizeDroneAirspaceSnapshot } from './records.js'; +import { droneAirspaceProxy } from '../../../server/providers/droneAirspace.js'; + +const ring = [ + [-77.04, 38.89], + [-77.03, 38.89], + [-77.03, 38.9], + [-77.04, 38.89], +]; +const validPayload = { + features: [ + { + id: 310, + geometry: { type: 'Polygon', coordinates: [ring] }, + properties: { + NAME: 'P-56', + TYPE_CODE: 'P', + OBJECTID: 310, + }, + }, + ], +}; + +test('a successful response yields normalized zone rows', async () => { + let requested; + const source = createDroneAirspaceSource({ + fetchImpl: async (url) => { + requested = String(url); + return Response.json({ + rows: normalizeDroneAirspaceSnapshot(validPayload), + }); + }, + }); + const rows = await source.getSnapshot(); + assert.equal(rows.length, 1); + assert.equal(rows[0].stableId, '310'); + assert.equal(requested, '/api/drone-airspace'); +}); + +test('a truncated response pages until the feed is complete', async () => { + const pageRing = (id) => ({ + id, + geometry: { type: 'Polygon', coordinates: [ring] }, + properties: { NAME: `P-${id}`, TYPE_CODE: 'P', OBJECTID: id }, + }); + const requests = []; + let handler; + const plugin = droneAirspaceProxy({ + fetchImpl: async (url) => { + requests.push(new URL(String(url)).searchParams.get('resultOffset')); + const page = requests.length; + return Response.json({ + features: [pageRing(page)], + properties: { exceededTransferLimit: page < 3 }, + }); + }, + }); + plugin.configureServer({ + middlewares: { + use: (_path, callback) => { + handler = callback; + }, + }, + }); + let payload; + await handler( + { url: '/', method: 'GET' }, + { + writeHead(status) { + assert.equal(status, 200); + }, + end(body) { + payload = JSON.parse(body); + }, + }, + ); + const rows = payload.rows; + assert.deepEqual( + rows.map((row) => row.stableId), + ['1', '2', '3'], + ); + assert.deepEqual(requests, [null, '1', '2']); +}); + +test('an upstream failure surfaces its HTTP status', async () => { + const source = createDroneAirspaceSource({ + fetchImpl: async () => ({ ok: false, status: 503 }), + }); + await assert.rejects(source.getSnapshot(), /FAA SUA HTTP 503/); +}); + +test('a malformed successful response is never accepted as an empty snapshot', async () => { + for (const payload of [{}, { rows: null }, { rows: {} }]) { + const source = createDroneAirspaceSource({ + fetchImpl: async () => Response.json(payload), + }); + await assert.rejects( + source.getSnapshot(), + /Malformed drone airspace snapshot/, + ); + } +}); diff --git a/src/sources/reference.js b/src/sources/reference.js index 4bdf7e9ac..302e15303 100644 --- a/src/sources/reference.js +++ b/src/sources/reference.js @@ -1,5 +1,6 @@ import { createUsgsEarthquakeSource } from '../layers/earthquakes/source.js'; import { createWfigsPerimeterSource } from '../layers/perimeters/source.js'; +import { createDroneAirspaceSource } from '../layers/droneAirspace/source.js'; import { createBundledCableSource } from '../layers/submarineCables/bundledSource.js'; /** Construct the existing reference feeds independently of application setup. */ @@ -7,6 +8,7 @@ export function createReferenceSources() { return { earthquakes: createUsgsEarthquakeSource(), 'fire-perimeters': createWfigsPerimeterSource(), + 'drone-airspace': createDroneAirspaceSource(), cables: createBundledCableSource(), }; } diff --git a/src/sources/reference.test.mjs b/src/sources/reference.test.mjs index 902a72c83..735b5c72e 100644 --- a/src/sources/reference.test.mjs +++ b/src/sources/reference.test.mjs @@ -15,13 +15,16 @@ test('reference factories retain compatibility without starting acquisition or s assert.deepEqual(Object.keys(first), [ 'earthquakes', 'fire-perimeters', + 'drone-airspace', 'cables', ]); assert.notEqual(first.earthquakes, second.earthquakes); assert.notEqual(first['fire-perimeters'], second['fire-perimeters']); + assert.notEqual(first['drone-airspace'], second['drone-airspace']); assert.notEqual(first.cables, second.cables); assert.equal(typeof first.earthquakes.getSnapshot, 'function'); assert.equal(typeof first['fire-perimeters'].getSnapshot, 'function'); + assert.equal(typeof first['drone-airspace'].getSnapshot, 'function'); assert.equal(typeof first.cables.fetch, 'function'); assert.equal(requests, 0); }); diff --git a/src/tooling/previewServing.test.mjs b/src/tooling/previewServing.test.mjs index d6eb80e20..b88b4e1f8 100644 --- a/src/tooling/previewServing.test.mjs +++ b/src/tooling/previewServing.test.mjs @@ -10,6 +10,7 @@ import { makeFixtureRoot } from './fixtureRoot.mjs'; test('data providers have both hooks; credential editing stays development-only', () => { const plugins = localProviderPlugins(); assert.ok(plugins.some(({ name }) => name === 'fire-perimeters')); + assert.ok(plugins.some(({ name }) => name === 'drone-airspace')); for (const plugin of plugins) { if (plugin.name === 'gev-key-setup') { assert.equal(plugin.configurePreviewServer, undefined); diff --git a/src/ui/layerPanel.js b/src/ui/layerPanel.js index e4c240f23..ecbf42cca 100644 --- a/src/ui/layerPanel.js +++ b/src/ui/layerPanel.js @@ -39,6 +39,7 @@ const PANEL_GROUPS = [ ids: [ 'alpr-cameras', 'military-installations', + 'drone-airspace', 'local-datacenters', 'telegeography-submarine-cables', 'local-dams', diff --git a/src/voice/layerManifest.js b/src/voice/layerManifest.js index 1e348ee41..facc90e32 100644 --- a/src/voice/layerManifest.js +++ b/src/voice/layerManifest.js @@ -327,6 +327,30 @@ const layers = [ context: true, query: { fields: { name: ['t'], class: ['t'], kind: ['t'] } }, }, + { + id: 'drone-airspace', + aliases: [ + 'drone restrictions', + 'drone airspace', + 'no-fly zones', + 'no fly zones', + 'drones', + ], + context: false, + query: { + fields: { + name: ['t'], + typeCode: ['t'], + type: ['t'], + lowerDesc: ['t'], + upperDesc: ['t'], + timesOfUse: ['t'], + agency: ['t'], + city: ['t'], + state: ['t'], + }, + }, + }, { id: 'local-datacenters', aliases: ['datacenters', 'data centers', 'data centres'], diff --git a/src/voice/layerManifest.test.mjs b/src/voice/layerManifest.test.mjs index d0222b934..98e3889bf 100644 --- a/src/voice/layerManifest.test.mjs +++ b/src/voice/layerManifest.test.mjs @@ -20,14 +20,28 @@ test('every catalog layer is in the voice manifest or explicitly voice-off', () const missing = catalogIds.filter( (id) => !manifestIds.includes(id) && !Object.hasOwn(VOICE_OFF_LAYERS, id), ); - assert.deepEqual(missing, [], `add a voice manifest entry for: ${missing.join(', ')}`); + assert.deepEqual( + missing, + [], + `add a voice manifest entry for: ${missing.join(', ')}`, + ); for (const id of Object.keys(VOICE_OFF_LAYERS)) { - assert.ok(catalogIds.includes(id), `${id} is voice-off but not in the catalog`); - assert.ok(!manifestIds.includes(id), `${id} is both voice-off and in the manifest`); + assert.ok( + catalogIds.includes(id), + `${id} is voice-off but not in the catalog`, + ); + assert.ok( + !manifestIds.includes(id), + `${id} is both voice-off and in the manifest`, + ); } const unknown = manifestIds.filter((id) => !catalogIds.includes(id)); assert.deepEqual(unknown, [], 'the manifest names only catalog layers'); - assert.equal(new Set(manifestIds).size, manifestIds.length, 'one entry per layer'); + assert.equal( + new Set(manifestIds).size, + manifestIds.length, + 'one entry per layer', + ); }); test('each layer declares a query or the reason it has none', () => { @@ -41,7 +55,8 @@ test('each layer declares a query or the reason it has none', () => { }); test('tool enums are generated from the manifest', () => { - const toggle = tool('set_layer_visibility').parameters.properties.layerId.enum; + const toggle = tool('set_layer_visibility').parameters.properties.layerId + .enum; assert.deepEqual(toggle, [...VOICE_TOGGLE_LAYER_IDS]); for (const id of [ 'transit', @@ -57,9 +72,13 @@ test('tool enums are generated from the manifest', () => { 'rocket-launches', 'bikeshare', 'local-adsb', + 'drone-airspace', ]) assert.ok(toggle.includes(id), `${id} is voice-toggleable`); - assert.ok(!toggle.includes('military-awareness'), 'Global Context is reached through set_context_mode'); + assert.ok( + !toggle.includes('military-awareness'), + 'Global Context is reached through set_context_mode', + ); assert.deepEqual( tool('show_data_layers_menu').parameters.properties.layerId.enum, [...VOICE_TOGGLE_LAYER_IDS], @@ -68,47 +87,98 @@ test('tool enums are generated from the manifest', () => { tool('get_entity_context').parameters.properties.layerId.enum, [...VOICE_CONTEXT_LAYER_IDS], ); - assert.ok(!VOICE_CONTEXT_LAYER_IDS.includes('telegeography-submarine-cables'), 'cables register no context records'); + assert.ok( + !VOICE_CONTEXT_LAYER_IDS.includes('telegeography-submarine-cables'), + 'cables register no context records', + ); assert.deepEqual( tool('analyst_query').parameters.properties.layers.items.enum, [...VOICE_QUERY_LAYER_IDS], ); assert.deepEqual(Object.keys(ANALYST_LAYERS), [...VOICE_QUERY_LAYER_IDS]); const panels = tool('set_panel_open').parameters.properties.panelId.enum; - assert.ok(panels.includes('weather-panel') && panels.includes('recent-imagery-panel')); + assert.ok( + panels.includes('weather-panel') && panels.includes('recent-imagery-panel'), + ); }); test('spoken aliases resolve to real layers, keep every legacy phrase and never collide', () => { const aliases = voiceLayerAliasMap(); for (const id of aliases.values()) assert.ok(manifestIds.includes(id), id); const legacy = { - planes: 'flights', aircraft: 'flights', 'military flights': 'military', quakes: 'earthquakes', - 'space mission': 'rocket-launches', 'space missions': 'rocket-launches', missions: 'rocket-launches', - 'street traffic': 'traffic', cameras: 'cctv', 'internet radio': 'radio', 'radio stations': 'radio', - bikes: 'bikeshare', ais: 'ais-live-vessels', ships: 'ais-live-vessels', vessels: 'ais-live-vessels', - 'live vessels': 'ais-live-vessels', datacenters: 'local-datacenters', 'data centers': 'local-datacenters', - 'data centres': 'local-datacenters', dams: 'local-dams', 'submarine cables': 'telegeography-submarine-cables', - cables: 'telegeography-submarine-cables', telegeography: 'telegeography-submarine-cables', - firms: 'local-firms', fires: 'local-firms', 'active fires': 'local-firms', alpr: 'alpr-cameras', - 'alpr cameras': 'alpr-cameras', 'flock cameras': 'alpr-cameras', 'license plate readers': 'alpr-cameras', - 'license plate cameras': 'alpr-cameras', 'plate readers': 'alpr-cameras', 'local-adsb': 'local-adsb', - 'local adsb': 'local-adsb', 'local ads-b': 'local-adsb', 'my receiver': 'local-adsb', - 'my antenna': 'local-adsb', 'my sdr': 'local-adsb', + planes: 'flights', + aircraft: 'flights', + 'military flights': 'military', + quakes: 'earthquakes', + 'space mission': 'rocket-launches', + 'space missions': 'rocket-launches', + missions: 'rocket-launches', + 'street traffic': 'traffic', + cameras: 'cctv', + 'internet radio': 'radio', + 'radio stations': 'radio', + bikes: 'bikeshare', + ais: 'ais-live-vessels', + ships: 'ais-live-vessels', + vessels: 'ais-live-vessels', + 'live vessels': 'ais-live-vessels', + datacenters: 'local-datacenters', + 'data centers': 'local-datacenters', + 'data centres': 'local-datacenters', + dams: 'local-dams', + 'submarine cables': 'telegeography-submarine-cables', + cables: 'telegeography-submarine-cables', + telegeography: 'telegeography-submarine-cables', + firms: 'local-firms', + fires: 'local-firms', + 'active fires': 'local-firms', + alpr: 'alpr-cameras', + 'alpr cameras': 'alpr-cameras', + 'flock cameras': 'alpr-cameras', + 'license plate readers': 'alpr-cameras', + 'license plate cameras': 'alpr-cameras', + 'plate readers': 'alpr-cameras', + 'local-adsb': 'local-adsb', + 'local adsb': 'local-adsb', + 'local ads-b': 'local-adsb', + 'my receiver': 'local-adsb', + 'my antenna': 'local-adsb', + 'my sdr': 'local-adsb', }; - for (const [phrase, id] of Object.entries(legacy)) assert.equal(aliases.get(phrase), id, phrase); + for (const [phrase, id] of Object.entries(legacy)) + assert.equal(aliases.get(phrase), id, phrase); const added = { - buses: 'transit', trains: 'transit', hurricanes: 'weather-cyclones', cyclones: 'weather-cyclones', - 'weather radar': 'weather-radar', lightning: 'weather-lightning', 'military bases': 'military-installations', - 'recent imagery': 'recent-imagery', 'bike share': 'bikeshare', 'rocket launches': 'rocket-launches', - 'street level': 'street-level', mapillary: 'street-level', - wind: 'wind', directions: 'directions', 'fire perimeters': 'fire-perimeters', - 'wildfire perimeters': 'fire-perimeters', 'burn areas': 'fire-perimeters', + buses: 'transit', + trains: 'transit', + hurricanes: 'weather-cyclones', + cyclones: 'weather-cyclones', + 'weather radar': 'weather-radar', + lightning: 'weather-lightning', + 'military bases': 'military-installations', + 'recent imagery': 'recent-imagery', + 'bike share': 'bikeshare', + 'rocket launches': 'rocket-launches', + 'street level': 'street-level', + mapillary: 'street-level', + wind: 'wind', + directions: 'directions', + 'fire perimeters': 'fire-perimeters', + 'wildfire perimeters': 'fire-perimeters', + 'burn areas': 'fire-perimeters', + 'drone restrictions': 'drone-airspace', + 'drone airspace': 'drone-airspace', + 'no-fly zones': 'drone-airspace', + drones: 'drone-airspace', }; - for (const [phrase, id] of Object.entries(added)) assert.equal(aliases.get(phrase), id, phrase); + for (const [phrase, id] of Object.entries(added)) + assert.equal(aliases.get(phrase), id, phrase); const claimed = new Map(); for (const entry of VOICE_LAYER_MANIFEST) for (const alias of entry.aliases) { - assert.ok(!claimed.has(alias), `"${alias}" claimed by ${claimed.get(alias)} and ${entry.id}`); + assert.ok( + !claimed.has(alias), + `"${alias}" claimed by ${claimed.get(alias)} and ${entry.id}`, + ); claimed.set(alias, entry.id); } }); From a275c2c63047964de9b2ba5241d7e6cb450edb40 Mon Sep 17 00:00:00 2001 From: Matt Van Horn Date: Mon, 12 Oct 2026 19:39:15 +0000 Subject: [PATCH 02/48] test: pin voice schema and README layer count for drone airspace --- src/tooling/transitQa.test.mjs | 2 +- src/voice/actionSchemas.test.mjs | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/src/tooling/transitQa.test.mjs b/src/tooling/transitQa.test.mjs index 2ec277748..4e79bffbf 100644 --- a/src/tooling/transitQa.test.mjs +++ b/src/tooling/transitQa.test.mjs @@ -675,7 +675,7 @@ test('public transit documentation uses courtesy attribution and the approved gr ); assert.ok( readme.includes( - 'Twenty layers and map sources. **Seventeen have a keyless path.**', + 'Twenty-one layers and map sources. **Eighteen have a keyless path.**', ), ); assert.ok( diff --git a/src/voice/actionSchemas.test.mjs b/src/voice/actionSchemas.test.mjs index d3b8e8a90..a62c292f7 100644 --- a/src/voice/actionSchemas.test.mjs +++ b/src/voice/actionSchemas.test.mjs @@ -37,7 +37,8 @@ test('the complete Realtime tool payload pins the manifest-generated layer relea // omit top-level anyOf (OpenAI rejects it). Contacts list routing preserves // requested radius and center authority in descriptions only. // Street Level adds its toggle enum values and generated alias hint. - '9eb113454c5af5e32e27ffcdc7919d0e4600ba1d204f8f175e071152be41ad0f', + // Drone Restrictions adds toggle/query enum values and spoken aliases. + '3ceef7975c1437eacb2c602bf9728829b299532b7dd95534cf7d9eb47115fa54', ); }); From 052435473e878deb59b91b4bb0a009474f6d6feb Mon Sep 17 00:00:00 2001 From: Matt Van Horn Date: Wed, 14 Oct 2026 01:51:25 +0000 Subject: [PATCH 03/48] feat(cctv): add Houston TranStar camera pack Register the keyless TranStar snapshot catalog as a live CCTV source, synthesize frame URLs from validated numeric camera ids, and keep the nearest cameras to Houston's metro cores under the existing catalog ceiling. --- .env.example | 3 + CHANGELOG.md | 11 + DATA_SOURCES.md | 2 + README.md | 2 +- docs/CURRENT-STATE.md | 19 +- scripts/format-scope.json | 1 + server/providers/cctv/catalog.js | 6 + server/providers/cctv/constants.js | 31 +++ server/providers/cctv/normalize.js | 12 + server/providers/cctv/sources.js | 143 +++++++++++ src/data/cctvHouston.test.mjs | 379 +++++++++++++++++++++++++++++ src/data/dataCredits.js | 6 + 12 files changed, 613 insertions(+), 2 deletions(-) create mode 100644 src/data/cctvHouston.test.mjs diff --git a/.env.example b/.env.example index a2f8afa24..265377bf7 100644 --- a/.env.example +++ b/.env.example @@ -237,6 +237,9 @@ AISSTREAM_API_KEY= # CCTV_TORONTO_LIST_URL= # override the Toronto RESCU camera list URL # CCTV_LITHUANIA_ENABLED= # off by default; set 1 to load Via Lietuva / Eismoinfo road cameras (photo licence unconfirmed; see DATA_SOURCES.md) # CCTV_LITHUANIA_MAX_SOURCES=400 # cap Lithuanian cameras; no API key required +# CCTV_HOUSTON_ENABLED=1 # set 0 to disable the Houston TranStar cameras +# CCTV_HOUSTON_MAX_SOURCES=250 # cap cameras loaded from the Houston TranStar catalog (nearest metro cores first) +# CCTV_HOUSTON_CAMERAS_URL= # override the Houston TranStar catalog URL # CCTV_MAX_SOURCES=4000 # catalog-wide ceiling, shared round-robin across packs # CCTV_PREFER_AUSTIN=1 # prioritize Austin cameras in the registry # CCTV_FORCE_AUSTIN= # force live default sources when a source-pack JSON is configured diff --git a/CHANGELOG.md b/CHANGELOG.md index 98d32365f..b4de32107 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,16 @@ # Changelog +- Add Houston traffic cameras from Houston TranStar as a CCTV pack. The camera + list is one keyless JSON request to + `traffic.houstontranstar.org/data/layers/cctvSnapshots_json.js`; frames are + stills synthesized from each numeric camera id on `www.houstontranstar.org`, + so a catalog field cannot steer the proxy off-host. Cameras without a + snapshot id, and cameras outside Greater Houston, are dropped. By default + the 250 cameras nearest downtown, Uptown, the Medical Center, IAH and + Galveston load. `CCTV_HOUSTON_ENABLED=0` turns the pack off and + `CCTV_HOUSTON_MAX_SOURCES` changes the cap. The catalog-wide ceiling is + unchanged; extra cameras share it round-robin with the other packs. + - The tracked-aircraft descriptor now publishes `verticalRateMps`, on both the civil and military layers, so a consumer reading `getTrackedInfo()` no longer has to reach past it into the raw poll record for climb/descent. Reported, diff --git a/DATA_SOURCES.md b/DATA_SOURCES.md index 0e04fb39b..913cad6d1 100644 --- a/DATA_SOURCES.md +++ b/DATA_SOURCES.md @@ -65,6 +65,7 @@ How to read this: | **Statens vegvesen** (catalog `ogckart-sn1.atlas.vegvesen.no`, frames `kamera.atlas.vegvesen.no`, HLS `kamera.vegvesen.no`) | CCTV camera catalog + frames + live HLS video, Norway road cameras | [NLOD 2.0](https://data.norge.no/nlod/en/2.0) — attribution REQUIRED | "Contains data under the Norwegian licence for Open Government data (NLOD) distributed by Statens vegvesen" | | **NYC DOT Traffic Management Center** (`webcams.nyctmc.org`) | CCTV camera catalog + frames, New York City five boroughs | NYC DOT requires developers to sign a [data-sharing agreement](https://www.nyc.gov/html/dot/html/about/datafeeds.shtml) for the camera feed. **Off by default**; enable only under your own agreement | "NYC DOT Traffic Management Center" + NYC DOT's required application disclaimer (in the Data attribution popover) | | **City of Toronto Open Data** (`opendata.toronto.ca`) | CCTV camera catalog + frames, Toronto city streets | [Open Government Licence – Toronto](https://www.toronto.ca/city-government/data-research-maps/open-data/open-data-licence/) — attribution REQUIRED | "Contains information licensed under the Open Government Licence – Toronto" | +| **Houston TranStar** (`traffic.houstontranstar.org`, frames `www.houstontranstar.org`) | CCTV camera catalog + frames, Greater Houston | Public Houston TranStar traffic camera stills | "Houston TranStar" (courtesy) | | **GBFS (Lyft / BCycle / Social Bicycles / PBSC)** | Bikeshare availability | Per-feed (attribution-only) | Credit the operator (e.g. Austin BCycle) + its `license_url` | | **Bike Share Toronto** (`toronto.publicbikesystem.net`) | Bikeshare station availability, Toronto | Operator GBFS feed. No licence is published: the City's open-data package declares none and the feed has no `license_url`. Attribution is given to the operator. | "Bike Share Toronto (Toronto Parking Authority)" | | **Bike Itaú / Tembici** (`*.publicbikesystem.net`, seven Brazilian cities, Santiago and Bogotá) | Bikeshare station availability: São Paulo, Rio de Janeiro, Salvador, Recife, Porto Alegre, Brasília, Curitiba, Santiago de Chile and Bogotá | Operator GBFS feeds. No licence is published: the feeds' `system_information.json` carries no `license_url`, operator or attribution fields. Attribution is given to the operator. | "Bike Itaú (Tembici)"; "Tembici Bogotá" | @@ -166,6 +167,7 @@ The optional Local ADS-B layer and the broadcast-FM receiver use radio you recei - **Statens vegvesen road cameras.** Keyless. The camera list is one GeoJSON request to the OGC API Features view of the official DATEX 3.1 CCTV site table (`ogckart-sn1.atlas.vegvesen.no/ogc/features/v1/collections/datex_3_1:CctvSimple/items`), so no DATEX account is needed; frames are stills on `kamera.atlas.vegvesen.no/api/images/`, and only that exact path per camera is registered. Cameras the feed reports as faulty are dropped. About 135 working cameras also publish live HLS (`kamera.vegvesen.no/public//manifest.m3u8`, MPEG-TS); those play as live video through the bounded in-memory HLS relay, with the still as the snapshot fallback, and only that exact manifest path per camera is registered. `CCTV_VEGVESEN_VIDEO=0` keeps every camera on stills. `orientationDescription` names the place a camera faces, not a bearing, so it goes into the label and headings use the id-hash fallback at low confidence. By default every live camera loads (~850); `CCTV_VEGVESEN_MAX_SOURCES` (8–1000) lowers the cap, keeping the cameras nearest Oslo, Bergen, Stavanger, Trondheim, Kristiansand, Bodø and Tromsø and `CCTV_VEGVESEN_ENABLED=0` turns the pack off. Frames are fetched at request time and never stored or redistributed. The NLOD attribution is registered in the Data attribution popover. - **NYC DOT Traffic Management Center cameras.** Off by default. NYC DOT's [developer data feeds page](https://www.nyc.gov/html/dot/html/about/datafeeds.shtml) asks developers who want the camera feed to contact `tmcdot@dot.nyc.gov` and sign a data-sharing agreement. This project does not hold one that covers downstream installations, so the pack loads only when an operator who holds their own agreement sets `CCTV_NYC_ENABLED=1`; with the flag unset, empty or any other value, the server makes no request to NYC DOT. The endpoint needs no key. The camera list is the public `webcams.nyctmc.org/api/cameras` catalog (~980 cameras across the five boroughs); frames are stills at `webcams.nyctmc.org/api/cameras//image`, synthesized from a validated UUID rather than taken from the payload, and only that host is registered. Cameras the feed reports as offline are dropped. The catalog publishes no camera facing, so headings use the shared id-hash fallback at low confidence and are corrected with the in-app calibration gizmo. Once enabled, the 250 cameras nearest the borough cores load; `CCTV_NYC_MAX_SOURCES` (8–400) changes the cap. Frames are fetched at request time and never stored or redistributed. NYC DOT asks every application that uses its data to carry its disclaimer; it is registered verbatim in the Data attribution popover. - **City of Toronto RESCU cameras.** Keyless. The camera list is the City's public `tmcearthcameras.json` on `opendata.toronto.ca` (dataset "Traffic Cameras", Transportation Services), served as JSONP and unwrapped with an anchored pattern. This is the municipal network the Ontario 511 pack does not carry: downtown arterials plus the Gardiner, DVP, Allen and Lake Shore. The list publishes no frame URL, so frame URLs are BUILT from the documented `CameraImages/loc####.jpg` convention and a strictly-digits camera number, pinning the frame proxy to the portal origin by construction; the list fetch refuses redirects and bounds the body it buffers. Frames are fetched at request time and never stored or redistributed. The dataset publishes no camera facing: the `D1`-`D4` fields are the directions of the *comparison* images (static reference stills captured in 2016 under a different path), not a bearing for the live pan-tilt-zoom frame, so headings use the shared id-hash fallback at low confidence and are corrected with the in-app calibration gizmo. A camera that is temporarily out of service still answers 200 with a byte-identical "Image Unavailable" placeholder rather than an error (about 6% of the network at any moment), so such a camera reads as healthy; the frame it shows is the agency's, not a synthesized one. The licence requires the attribution statement above. Disable with `CCTV_TORONTO_ENABLED=0`. +- **Houston TranStar cameras.** Keyless. The camera list is the public `traffic.houstontranstar.org/data/layers/cctvSnapshots_json.js` catalog (~1,000 cameras across Greater Houston); frames are stills at `www.houstontranstar.org/snapshots/cctv/.jpg`, synthesized from a validated numeric id rather than taken from the payload, and only that host is registered. Cameras whose catalog id is `0` (a different, non-snapshot class) and cameras outside Greater Houston are dropped. Dedicated `dir` values (North/South/East/West) become high-confidence headings; `None` uses the shared id-hash fallback at low confidence. By default the 250 cameras nearest downtown, Uptown, the Medical Center, IAH and Galveston load; `CCTV_HOUSTON_MAX_SOURCES` (8–400) changes the cap and `CCTV_HOUSTON_ENABLED=0` turns the pack off. Frames are fetched at request time and never stored or redistributed. - **Camera frame content.** Public camera frames can contain people, vehicles and license plates, and the camera layer does nothing with them beyond putting them on the map. A successful upstream response is relayed as the provider served it: nothing in the camera pipeline blurs, redacts, enhances or restores it, and nothing there runs face, plate or object detection or otherwise recognises or classifies what is in the picture. (The ALPR layer described below maps where plate-reader cameras are, from locations contributors tagged in OpenStreetMap; it reads nothing from any camera frame.) When an upstream has no frame for a camera, what you see instead is a Google Street View still or the app's own placeholder card; the panel's source badge names which of them you are looking at, and a refresh that fails after a frame has arrived leaves the last good picture on screen rather than replacing it. The browser resamples each frame for display: onto a fixed 1920x1080 surface for the camera's projection plane in the scene, and onto a 192x108 canvas for the small ambient card. That scaling is the only change made to the picture, and the only pixels the camera layer reads back are a 64x36 downscale hashed to tell whether the picture has changed. Live streams are piped through; the still path reads the image into memory to pass it on — TxDOT wraps its JPEG in a JSON envelope, which is decoded there — and no frame is written to disk (the disk cache, `.gev-cache/`, holds feed JSON for other layers and never camera imagery). Outside that pipeline, the voice assistant is the one feature that sends imagery anywhere: when it answers a question about what is in view it may attach a downscaled screenshot of the rendered scene — including any camera plane on screen — to the request it sends to the voice model. Frames are fetched only from server-registered camera URLs — a client cannot hand the proxy a URL (see [SECURITY.md](SECURITY.md)). What the app does stays inside the line README draws: no named-person search, no face recognition, no tracking individuals. - **Transit (GTFS-Realtime).** The proxy requests only registered operator feeds, validates redirects, bounds responses, and uses conditional requests, short snapshot caching and failure backoff. All sixteen registered regions are live-enabled; each can be disabled independently. Vehicles use delayed playback between reports, with report age and waiting states shown separately. Session-local selected trails are available for every feed. MBTA additionally retains up to 15 minutes of recently observed positions in the running proxy, so available history can survive a browser reload; server restart clears it. History reads never fetch upstream, and other feeds have no proxy retention. Nothing is archived to disk or collected in the background. Operator text credits are included as a courtesy and do not imply endorsement. Heights are aligned to work with Google 3D tiles; subway markers are shown at street level because these feeds do not supply depth. The TTC feed is BusTime, the surface system: it carries buses and streetcars and no subway at all. About four in ten of its vehicles report no route at all (unassigned or deadheading), and those are shown with the feed default rather than a route-established mode. It sends no ETag or Last-Modified, so its conditional requests fall through to the ordinary snapshot cache. Three Golden Horseshoe neighbours run alongside it — MiWay, Hamilton HSR and Durham Region Transit — each bus-only and each polled only near its own region. Durham Region Transit publishes no bearing on any vehicle, so those sprites take their heading from observed movement like any other feed that omits it. Guelph, Kingston and Barrie extend the same pattern eastward and north. Hamilton, Guelph, Burlington and YRT publish route ids that are not the number on the vehicle, so those entries carry a route-name table taken from each operator’s own published GTFS; any id missing from a table renders unchanged rather than blank. A label is never invented: YRT runs a route `9101` that appears nowhere in its published routes, so it stays as the feed sends it. Burlington’s is the one that cannot be shortcut: `351` is route 1 while `3510` is route 10, so the table is a real join rather than a trimmed prefix. GO Transit / Metrolinx is deliberately absent: its REALTIME feed requires an API key issued by manual review, which the keyless admission rule excludes (its static schedule download is keyless, but carries no vehicle positions). Barrie and YRT/Viva each invite acceptance of their licence through a form, and each licence also states that using the data is itself acceptance; this deployment relies on that clause, which is recorded in both entries. Grand River Transit in Waterloo is absent for a different reason: it answers, but its feed header was months stale and it publishes no per-vehicle timestamps, so it would render vehicles that look live and are not. Ottawa’s OC Transpo falls to the same keyless admission rule as GO Transit — its realtime feed requires an API key. London Transit and Thunder Bay Transit publish vehicle positions over plain HTTP with no HTTPS endpoint available. Every other Ontario system — Brampton, Oakville, Niagara, Milton, Windsor, Greater Sudbury, UP Express and a dozen smaller operators — publishes static GTFS schedules only, with no realtime vehicle positions to register. Between the nine included and these exclusions, every Ontario realtime vehicle-position feed catalogued in the Mobility Database is accounted for. - **Radio Browser.** `/api/radio/stations` discovers official API mirrors, makes bounded and coalesced healthy/geolocated HTTPS-station queries, caches the normalized public-domain directory for 45 minutes, and may serve the last good catalog for up to seven days during an outage. Refreshes must meet minimum accepted-query and station coverage before replacing a warm catalog; schema-valid responses whose rows all fail the product's health policy do not count as successful queries. A usable partial cold catalog is explicitly `DEGRADED`, and malformed or empty successful payloads are rejected atomically. Every directory and click-count request rejects redirects, validates all resolved addresses as globally routable (including reserved/documentation IPv4 and special/non-global IPv6 exclusions), and pins the TLS connection to a validated address. Only MP3/AAC non-HLS directory rows with public HTTPS stream targets are returned; favicons are intentionally omitted. Pressing play connects one browser audio element directly to the selected broadcaster and calls the directory's click counter through known-ID-only `POST /api/radio/click/:uuid`. GEV never proxies, caches, records, bundles, or redistributes audio. Radio Browser supplies station-level tags, not dependable current-song or upcoming-program metadata, so Radio filtering never claims either. Direct playback exposes the listener's IP address to the broadcaster, whose own stream terms apply. diff --git a/README.md b/README.md index c49c53ece..867f14824 100644 --- a/README.md +++ b/README.md @@ -377,7 +377,7 @@ Twenty layers and map sources. **Eighteen have a keyless path.** Some offer addi | 🌍 **Earthquakes** | Global seismic activity, last 24h | USGS | 🟢 | | ◍ **Air quality** | Live AQHI on the published 1–10+ health-risk scale, with a Weather-panel card showing the band legend and the highest readings in view. **Canada only** — ECCC is the first registered provider | ECCC MSC GeoMet | 🟢 | | 🚗 **Traffic** | Simulated vehicles on OSM roads. With TomTom, live flow speeds drive the simulation and congestion colors below ~8 km; individual vehicle positions are not live observations | TomTom + OSM | 🟢 simulation · 🟡 live flow speeds | -| 📹 **CCTV Mesh** | ~4,000 public cameras (shared cap) projected _into_ the 3D space — Austin · Texas (TxDOT) · California (Caltrans) · London (TfL) · Ontario (511) · Finland (Fintraffic) · British Columbia (DriveBC) · Estonia (Tallinn, Tarktee) · Delaware (DelDOT live video) · New South Wales (Live Traffic NSW) · Calgary · Norway (Statens vegvesen) · Toronto · King County (WA) · Washington (WSDOT) · Catalonia (Servei Català de Trànsit), plus opt-in New York City (NYC DOT TMC, needs your own NYC DOT agreement) and opt-in Lithuania (Via Lietuva / Eismoinfo, photo licence unconfirmed). Positions are published; poses are estimated priors **you calibrate by dragging a gizmo on the camera itself** | City APIs | 🟢 | +| 📹 **CCTV Mesh** | ~4,000 public cameras (shared cap) projected _into_ the 3D space — Austin · Texas (TxDOT) · California (Caltrans) · London (TfL) · Ontario (511) · Finland (Fintraffic) · British Columbia (DriveBC) · Estonia (Tallinn, Tarktee) · Delaware (DelDOT live video) · New South Wales (Live Traffic NSW) · Calgary · Norway (Statens vegvesen) · Toronto · King County (WA) · Washington (WSDOT) · Catalonia (Servei Català de Trànsit) · Houston (TranStar), plus opt-in New York City (NYC DOT TMC, needs your own NYC DOT agreement) and opt-in Lithuania (Via Lietuva / Eismoinfo, photo licence unconfirmed). Positions are published; poses are estimated priors **you calibrate by dragging a gizmo on the camera itself** | City APIs | 🟢 | | 📷 **Mapped ALPR Cameras** | License-plate-reader camera locations tagged by OpenStreetMap contributors, loaded one city-sized view at a time, with **SHOW NEAREST**. Locations and tags only: no plate data, no video | OpenStreetMap (incl. DeFlock mapping) | 🟢 | | 📷 **Street Level** | Street-level photos behind one panel, one chip per provider: coverage on the streets, image cones, an embedded viewer and **FOLLOW**. Mapillary, Google Street View, KartaView and Panoramax | Mapillary · Google · KartaView · Panoramax | 🟢 KartaView, Panoramax · 🟡 Mapillary token · 🔴 Google Maps key | | 📻 **Radio** | Geolocated world radio with an **analog tuner** — drag the needle across up to 750 stations and the globe flies to each broadcaster | Radio Browser / broadcasters | 🟢 | diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 8cb7b6d17..99cfc3234 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1072,6 +1072,23 @@ By default the 250 cameras nearest downtown Vancouver and Victoria load. The Open Government Licence – British Columbia attribution is registered in the Data attribution popover. +## Houston TranStar CCTV source pack + +The CCTV catalog adds Greater Houston traffic cameras from Houston TranStar +alongside the other live packs. The server reads the keyless camera list at +`https://traffic.houstontranstar.org/data/layers/cctvSnapshots_json.js` and +keeps cameras with a numeric snapshot id inside the metro bounding box. Frame +URLs are built from each camera id under +`https://www.houstontranstar.org/snapshots/cctv/.jpg`; they are never taken +from the payload. Dedicated `dir` values become high-confidence headings; +`None` uses the shared id-hash fallback at low confidence. + +By default the 250 cameras nearest downtown, Uptown, the Medical Center, IAH +and Galveston load. `CCTV_HOUSTON_MAX_SOURCES` sets the pack cap (8–400) and +`CCTV_HOUSTON_ENABLED=0` turns the pack off. The catalog-wide ceiling stays at +4,000 and is shared round-robin, so adding this pack does not silently drop +another region. + ## Location control ownership City/POI rows, search/reset bindings, location readouts and the orbit indicator @@ -3366,7 +3383,7 @@ its criteria cannot be silently ignored. | Satellites | CelesTrak | `src/data/satellites.js` | `/api/celestrak` | 120s | | Space Missions (30d) | Launch Library 2 + CelesTrak | `src/data/rocketLaunches.js` | `/api/launches` + `/api/celestrak/active` | 5 min | | Traffic | Selectable TomTom / OpenStreetMap / Hybrid roads; optional TomTom flow (BYOK) | `src/data/traffic.js` | browser-direct OpenFreeMap tiles; `/api/tomtom` for flow | viewport-driven; capped tile cache | -| CCTV | Austin + Caltrans (CA) + TfL London + Ontario 511 + Fintraffic (FI) + DriveBC (BC) + TxDOT (TX) + Estonia (Tallinn, Tarktee) + Live Traffic NSW + Open Calgary Open Data + Statens vegvesen (NO, stills + live HLS) + King County (WA) Open Data + WSDOT (WA) Open Data + NYC DOT TMC (opt-in) + City of Toronto RESCU + Catalonia (Servei Català de Trànsit) + Street View fallback | `src/data/cctv.js` | `/api/cctv` | 10s (active) | +| CCTV | Austin + Caltrans (CA) + TfL London + Ontario 511 + Fintraffic (FI) + DriveBC (BC) + TxDOT (TX) + Estonia (Tallinn, Tarktee) + Live Traffic NSW + Open Calgary Open Data + Statens vegvesen (NO, stills + live HLS) + King County (WA) Open Data + WSDOT (WA) Open Data + NYC DOT TMC (opt-in) + City of Toronto RESCU + Catalonia (Servei Català de Trànsit) + Houston TranStar + Street View fallback | `src/data/cctv.js` | `/api/cctv` | 10s (active) | | Radio | Radio Browser (public-domain station directory) | `src/data/radio.js` | `/api/radio/stations`, `/api/radio/click/:uuid` | 45 min directory refresh | | Transit 🚌 | Operator GTFS-Realtime VehiclePositions (7 keyless regions, `src/data/transitFeeds.js`) | `src/layers/transit/` via `src/app/layers/transit.js` | `/api/transit` | 15s (poll + delayed playback) | | Bikeshare 🚲 | GBFS (Lyft + BCycle + Social Bicycles + PBSC incl. Bike Share Toronto, Bike Itaú (Brazil) and Bicing (Barcelona) + Urban Sharing, Entur, JCDecaux NO + nextbike (AMBici)) | `src/data/bikeshare.js` | `/api/gbfs` | 60s | diff --git a/scripts/format-scope.json b/scripts/format-scope.json index 8bcc4c04d..84002f2cf 100644 --- a/scripts/format-scope.json +++ b/scripts/format-scope.json @@ -552,6 +552,7 @@ "src/data/cctvCataloniaSource.test.mjs", "src/data/cctvVegvesen.test.mjs", "src/data/cctvNyc.test.mjs", + "src/data/cctvHouston.test.mjs", "src/data/cctvNswSource.test.mjs", "src/data/cctvTxdotSource.test.mjs", "src/layers/cctv/source.test.mjs", diff --git a/server/providers/cctv/catalog.js b/server/providers/cctv/catalog.js index 75e58d2a6..0ed194660 100644 --- a/server/providers/cctv/catalog.js +++ b/server/providers/cctv/catalog.js @@ -25,6 +25,7 @@ import { loadVegvesenSourcesFromOpenData, loadNycSourcesFromOpenData, loadTorontoSourcesFromOpenData, + loadHoustonSourcesFromOpenData, } from './sources.js'; /** Env kill switch: unset or anything but "0" means enabled. */ @@ -144,6 +145,11 @@ const LIVE_PACKS = [ enabled: () => envOptIn('CCTV_LITHUANIA_ENABLED'), load: loadLithuaniaSources, }, + { + name: 'houston', + enabled: () => envEnabled('CCTV_HOUSTON_ENABLED'), + load: loadHoustonSourcesFromOpenData, + }, ]; /** * Load CCTV sources from a local JSON file (CCTV_SOURCES_FILE env or default). diff --git a/server/providers/cctv/constants.js b/server/providers/cctv/constants.js index 20b963770..43d804963 100644 --- a/server/providers/cctv/constants.js +++ b/server/providers/cctv/constants.js @@ -347,6 +347,37 @@ export const NYC_ANCHORS = [ { lat: 40.5795, lon: -74.1502 }, // Staten Island ]; +/** Houston TranStar cameras: one keyless JSON list for the metro (~1,000 + * cameras with a numeric id). Frames are stills synthesized from that id on + * the city's snapshot host — never taken from the payload. The catalog is + * served as `application/javascript`; asking for `application/json` gets HTTP + * 406, so the loader accepts any media type. */ +export const HOUSTON_CAMERAS_URL = + 'https://traffic.houstontranstar.org/data/layers/cctvSnapshots_json.js'; +/** The only origin Houston camera frames may come from. The catalog's `url` + * is a `showWin(...)` viewer helper and is ignored; the registered path is + * always this origin plus `/snapshots/cctv/.jpg`. */ +export const HOUSTON_IMAGE_ORIGIN = 'https://www.houstontranstar.org/'; +export const HOUSTON_IMAGE_URL = (id) => + `${HOUSTON_IMAGE_ORIGIN}snapshots/cctv/${id}.jpg`; +/** Numeric TranStar snapshot ids: 3–5 digits, no leading zero. Id `0` is a + * different (non-snapshot) camera class in the same list. */ +export const HOUSTON_CAMERA_ID_RE = /^[1-9]\d{2,4}$/; +export const DEFAULT_HOUSTON_MAX_SOURCES = 250; +export const HOUSTON_MAX_CATALOG_BYTES = 4 * 1024 * 1024; +/** Ground-elevation prior in metres. The TranStar payload carries none, and + * Houston sits a few metres above sea level. */ +export const HOUSTON_GROUND_ELEVATION_M = 15; +/** Downtown Houston (Main @ Texas): the primary prioritization anchor. */ +export const HOUSTON_DOWNTOWN = { lat: 29.7604, lon: -95.3698 }; +export const HOUSTON_ANCHORS = [ + { lat: 29.7604, lon: -95.3698 }, // Downtown + { lat: 29.739, lon: -95.461 }, // Uptown / Galleria + { lat: 29.71, lon: -95.398 }, // Texas Medical Center + { lat: 29.9902, lon: -95.3368 }, // IAH + { lat: 29.3013, lon: -94.7977 }, // Galveston +]; + /** DelDOT CCTV: one keyless statewide JSON catalog; live video via RTMP-over-HTTP (rtmpt:80). */ export const DELDOT_CCTV_URL = 'https://tmc.deldot.gov/json/videocamera.json'; export const DEFAULT_DELDOT_MAX_SOURCES = 300; diff --git a/server/providers/cctv/normalize.js b/server/providers/cctv/normalize.js index 2d4e16e11..1b0e79a0c 100644 --- a/server/providers/cctv/normalize.js +++ b/server/providers/cctv/normalize.js @@ -388,6 +388,18 @@ export function isLikelyNycCoordinate(lat, lon) { ); } +/** Greater Houston (the Woodlands to Galveston, Katy to Baytown). Farther + * TranStar partners (Dallas, Columbus) are out of this city pack. */ +export function isLikelyHoustonCoordinate(lat, lon) { + return ( + isPlausibleLatLon(lat, lon) && + lat >= 29.2 && + lat <= 30.35 && + lon >= -96.05 && + lon <= -94.7 + ); +} + /** Mainland Norway, with slack for the coast and the Finnmark border. */ export function isLikelyNorwayCoordinate(lat, lon) { return ( diff --git a/server/providers/cctv/sources.js b/server/providers/cctv/sources.js index c3d11e752..895fe4c4d 100644 --- a/server/providers/cctv/sources.js +++ b/server/providers/cctv/sources.js @@ -96,6 +96,13 @@ import { NYC_MAX_CATALOG_BYTES, NYC_GROUND_ELEVATION_M, NYC_ANCHORS, + HOUSTON_CAMERAS_URL, + HOUSTON_IMAGE_URL, + HOUSTON_CAMERA_ID_RE, + DEFAULT_HOUSTON_MAX_SOURCES, + HOUSTON_MAX_CATALOG_BYTES, + HOUSTON_GROUND_ELEVATION_M, + HOUSTON_ANCHORS, } from './constants.js'; import { toFiniteNumber, @@ -116,6 +123,7 @@ import { isLikelyCataloniaCoordinate, decodeNumericEntities, isLikelyNycCoordinate, + isLikelyHoustonCoordinate, isLikelyNorwayCoordinate, isLikelyTorontoCoordinate, cameraDisplayCode, @@ -2784,3 +2792,138 @@ export async function loadTorontoSourcesFromOpenData() { return []; } } + +/** + * Label for one Houston TranStar camera: the interchange an operator + * recognises ("10 EAST @ SAN JACINTO"). + * + * @param {object} record - Raw TranStar camera row. + * @param {string} cameraId - Derived stable id. + * @returns {string} + */ +export function houstonCameraName(record, cameraId) { + const name = String(record?.location ?? '').trim(); + if (name) return name; + const short = String(cameraId || '').replace(/^houston-/, ''); + return short ? `Houston Camera ${short}` : 'Houston Camera'; +} + +/** + * One Houston TranStar camera row -> one catalog source, or null. + * + * Frame URLs are synthesized from a validated numeric id on the pinned + * snapshot origin. The payload's `url` is a `showWin(...)` viewer helper + * and is ignored so a catalog edit cannot steer the frame proxy off-host. + * Dedicated `dir` values (North/South/East/West) become high-confidence + * headings; `None`/blank fall back to the shared id-hash heading. + * + * @param {object} record - Raw TranStar camera row. + * @returns {?object} + */ +export function houstonCameraToSource(record) { + if (!record || typeof record !== 'object') return null; + + const lat = toFiniteNumber(record.lat); + const lon = toFiniteNumber(record.lng); + if (!isLikelyHoustonCoordinate(lat, lon)) return null; + + const rawId = String(record.id ?? '').trim(); + if (!HOUSTON_CAMERA_ID_RE.test(rawId)) return null; + const cameraId = `houston-${rawId}`; + const imageUrl = HOUSTON_IMAGE_URL(rawId); + const name = houstonCameraName(record, cameraId); + const extractedHeading = directionToHeading(record.dir, true); + const hasHeading = Number.isFinite(extractedHeading); + + return { + id: cameraId, + name, + city: 'Houston', + cityId: 'houston', + provider: 'Houston TranStar', + lat, + lon, + headingDeg: hasHeading ? extractedHeading : fallbackHeadingFromId(cameraId), + headingConfidence: hasHeading ? 'high' : 'low', + pitchDeg: hasHeading ? -24 : -18, + fovDeg: hasHeading ? 56 : 44, + rangeM: hasHeading ? 210 : 145, + mountHeightM: hasHeading ? 10 : 8, + groundElevationM: HOUSTON_GROUND_ELEVATION_M, + feedType: 'image', + url: imageUrl, + snapshotUrl: imageUrl, + sourceKind: 'houston-transtar', + license: 'Public Houston TranStar traffic camera', + code: cameraDisplayCode(name.toUpperCase()), + }; +} + +/** + * Fetch Houston TranStar cameras from the keyless `cctvSnapshots_json.js` + * catalog. Frames are stills on `www.houstontranstar.org`, registered from + * a validated numeric id rather than from the payload URL. The catalog host + * answers `Accept: application/json` with HTTP 406, so this request accepts + * any media type. + * + * @returns {Promise>} Normalized camera source objects. + */ +export async function loadHoustonSourcesFromOpenData() { + try { + const endpoint = + process.env.CCTV_HOUSTON_CAMERAS_URL || HOUSTON_CAMERAS_URL; + const resp = await fetch(endpoint, { + headers: { Accept: '*/*' }, + redirect: 'manual', + signal: AbortSignal.timeout(CCTV_SOURCE_FETCH_TIMEOUT_MS), + }); + const discard = async () => { + try { + await resp.body?.cancel(); + } catch { + /* no-op */ + } + return []; + }; + if (resp.status >= 300 && resp.status < 400) { + console.warn( + '[CCTV] Houston catalog redirected; redirects are not followed', + ); + return discard(); + } + if (!resp.ok) { + console.warn('[CCTV] Houston camera download failed:', resp.status); + return discard(); + } + const payload = await readResponseJsonCapped( + resp, + HOUSTON_MAX_CATALOG_BYTES, + ); + const rows = Array.isArray(payload?.cameras) ? payload.cameras : []; + const cameras = []; + const seen = new Set(); + for (const record of rows) { + const camera = houstonCameraToSource(record); + if (!camera || seen.has(camera.id)) continue; + seen.add(camera.id); + cameras.push(camera); + } + const maxRaw = Number( + process.env.CCTV_HOUSTON_MAX_SOURCES || DEFAULT_HOUSTON_MAX_SOURCES, + ); + const maxCount = Number.isFinite(maxRaw) + ? Math.max(8, Math.min(400, Math.floor(maxRaw))) + : DEFAULT_HOUSTON_MAX_SOURCES; + const prioritized = prioritizeSources(cameras, maxCount, HOUSTON_ANCHORS); + console.log( + `[CCTV] Loaded Houston camera sources: ${cameras.length} (using nearest ${prioritized.length})`, + ); + return prioritized; + } catch (error) { + console.warn( + '[CCTV] Houston camera download error:', + error?.message || error, + ); + return []; + } +} diff --git a/src/data/cctvHouston.test.mjs b/src/data/cctvHouston.test.mjs new file mode 100644 index 000000000..97de95a84 --- /dev/null +++ b/src/data/cctvHouston.test.mjs @@ -0,0 +1,379 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { + houstonCameraName, + houstonCameraToSource, + loadHoustonSourcesFromOpenData, +} from '../../server/providers/cctv/sources.js'; +import { + DEFAULT_CCTV_MAX_SOURCES, + DEFAULT_HOUSTON_MAX_SOURCES, + HOUSTON_CAMERAS_URL, + HOUSTON_GROUND_ELEVATION_M, + HOUSTON_IMAGE_ORIGIN, + HOUSTON_IMAGE_URL, + HOUSTON_MAX_CATALOG_BYTES, +} from '../../server/providers/cctv/constants.js'; +import { + CAMERA_CODE_MAX_CHARS, + isLikelyHoustonCoordinate, +} from '../../server/providers/cctv/normalize.js'; +import { createCctvCatalog } from '../../server/providers/cctv/catalog.js'; +import { directionToHeading } from './directionText.js'; + +/** + * A response whose body is a live stream, plus a flag that flips when the + * stream is cancelled. A rejection path that returns without cancelling holds + * the transport open, so the flag is what the refusal tests actually assert. + */ +const streamingResponse = (init = {}) => { + const state = { cancelled: false }; + const body = new ReadableStream({ + start(controller) { + controller.enqueue(new TextEncoder().encode('{')); + }, + cancel() { + state.cancelled = true; + }, + }); + return { response: new Response(body, init), state }; +}; + +const DOWNTOWN_ID = '1002'; +const GALLERIA_ID = '2001'; + +/** One TranStar camera row, shaped like the live `cctvSnapshots_json.js` payload. */ +const row = (overrides = {}) => ({ + location: '10 EAST @ SAN JACINTO', + id: DOWNTOWN_ID, + lat: '29.76824', + lng: '-95.355711', + dir: 'South', + fc: '6', + fp: '300', + url: "showWin('/layers/gc.aspx?cam=1002&loc=IH-10_East_at_SAN_JACINTO')", + ...overrides, +}); + +test('a Houston row maps to a source on the pinned HTTPS frame host', () => { + const source = houstonCameraToSource(row()); + assert.equal(source.id, `houston-${DOWNTOWN_ID}`); + assert.equal(source.name, '10 EAST @ SAN JACINTO'); + assert.equal(source.city, 'Houston'); + assert.equal(source.cityId, 'houston'); + assert.equal(source.provider, 'Houston TranStar'); + assert.equal(source.lat, 29.76824); + assert.equal(source.lon, -95.355711); + assert.equal(source.feedType, 'image'); + assert.equal(source.sourceKind, 'houston-transtar'); + assert.equal(source.license, 'Public Houston TranStar traffic camera'); + assert.equal(source.groundElevationM, HOUSTON_GROUND_ELEVATION_M); + assert.equal(source.url, HOUSTON_IMAGE_URL(DOWNTOWN_ID)); + assert.equal(source.url, source.snapshotUrl); + assert.ok(source.url.startsWith(HOUSTON_IMAGE_ORIGIN)); + assert.equal(source.headingConfidence, 'high'); + assert.equal(source.headingDeg, 180); +}); + +test('frame URLs are synthesized from the numeric id; the payload URL is ignored', () => { + const source = houstonCameraToSource( + row({ url: 'https://evil.example/cameras/x/image' }), + ); + assert.equal(source.url, HOUSTON_IMAGE_URL(DOWNTOWN_ID)); + assert.equal(source.snapshotUrl, source.url); +}); + +test('zero ids, off-city geometry and unusable ids are dropped', () => { + assert.equal(houstonCameraToSource(row({ id: '0' })), null); + assert.equal(houstonCameraToSource(row({ id: '00' })), null); + assert.equal(houstonCameraToSource(row({ id: '../etc/passwd' })), null); + assert.equal(houstonCameraToSource(row({ id: '12' })), null); + assert.equal(houstonCameraToSource(row({ id: '1002/../x' })), null); + // Dallas: a plausible lat/lon, but not a Houston camera. + assert.equal( + houstonCameraToSource(row({ lat: '32.7767', lng: '-96.797' })), + null, + ); + assert.equal(houstonCameraToSource(row({ lat: '0', lng: '0' })), null); + assert.equal(houstonCameraToSource(null), null); + // 3-digit and 5-digit snapshot ids stay. + assert.equal(houstonCameraToSource(row({ id: '102' })).id, 'houston-102'); + assert.equal(houstonCameraToSource(row({ id: '13711' })).id, 'houston-13711'); +}); + +test('bare cardinal dirs are high-confidence; None uses the id-hash fallback', () => { + assert.equal(directionToHeading('South', true), 180); + assert.equal(directionToHeading('None', true), NaN); + + const south = houstonCameraToSource(row({ dir: 'South' })); + assert.equal(south.headingConfidence, 'high'); + assert.equal(south.headingDeg, 180); + + const none = houstonCameraToSource(row({ dir: 'None' })); + assert.equal(none.headingConfidence, 'low'); + assert.ok(Number.isFinite(none.headingDeg)); + assert.equal( + none.headingDeg, + houstonCameraToSource(row({ dir: '' })).headingDeg, + ); +}); + +test('a nameless row still gets a label', () => { + assert.equal( + houstonCameraName({ location: ' 45 GULF @ WEST DALLAS ' }, 'houston-102'), + '45 GULF @ WEST DALLAS', + ); + assert.equal(houstonCameraName({}, 'houston-102'), 'Houston Camera 102'); + assert.equal(houstonCameraName({}, ''), 'Houston Camera'); +}); + +test('the unselected label is the interchange, trimmed to the code width', () => { + assert.equal(houstonCameraToSource(row()).code, '10 EAST @ SAN JACINTO'); + const long = houstonCameraToSource( + row({ + location: + 'IH-10 Katy Managed Lanes Eastbound at a very long interchange name that must truncate', + }), + ); + assert.equal(long.code.length, CAMERA_CODE_MAX_CHARS); + assert.ok(long.code.endsWith('…')); +}); + +test('the loader reads the keyless catalog and collapses duplicate ids', async (t) => { + t.mock.method(console, 'log', () => {}); + const requested = []; + t.mock.method(globalThis, 'fetch', async (url, init) => { + requested.push([String(url), init.headers.Accept, init.redirect]); + return Response.json({ + cameras: [ + row(), + row({ location: '10 EAST @ SAN JACINTO (duplicate)' }), + row({ + id: GALLERIA_ID, + location: '610 WEST LOOP @ SAN FELIPE', + lat: '29.749', + lng: '-95.461', + dir: 'North', + }), + row({ url: 'https://evil.example/x.jpg', id: '0' }), + ], + }); + }); + const cameras = await loadHoustonSourcesFromOpenData(); + assert.deepEqual(requested, [[HOUSTON_CAMERAS_URL, '*/*', 'manual']]); + assert.deepEqual( + cameras.map((camera) => camera.id), + [`houston-${GALLERIA_ID}`, `houston-${DOWNTOWN_ID}`], + 'nearest-to-anchor first: Uptown sits closer to its pin than downtown does', + ); + assert.equal(cameras[1].name, '10 EAST @ SAN JACINTO'); +}); + +test('an upstream failure yields an empty pack and releases the response', async (t) => { + t.mock.method(console, 'warn', () => {}); + const failed = streamingResponse({ status: 503 }); + t.mock.method(globalThis, 'fetch', async () => failed.response); + assert.deepEqual(await loadHoustonSourcesFromOpenData(), []); + assert.equal(failed.state.cancelled, true, 'the failed body is cancelled'); + + t.mock.restoreAll(); + t.mock.method(console, 'warn', () => {}); + t.mock.method(globalThis, 'fetch', async () => { + throw new Error('network down'); + }); + assert.deepEqual(await loadHoustonSourcesFromOpenData(), []); +}); + +test('the catalog fetch refuses redirects and oversized bodies', async (t) => { + t.mock.method(console, 'warn', () => {}); + const seen = []; + const redirected = streamingResponse({ + status: 302, + headers: { location: 'https://evil.example/cameras.js' }, + }); + t.mock.method(globalThis, 'fetch', async (url, init) => { + seen.push([String(url), init.redirect]); + return redirected.response; + }); + assert.deepEqual(await loadHoustonSourcesFromOpenData(), []); + assert.deepEqual(seen, [[HOUSTON_CAMERAS_URL, 'manual']]); + assert.equal( + redirected.state.cancelled, + true, + 'the redirect body is cancelled', + ); + + t.mock.restoreAll(); + t.mock.method(console, 'warn', () => {}); + t.mock.method(globalThis, 'fetch', async () => { + const body = JSON.stringify({ cameras: [row()] }); + return new Response(body, { + headers: { + 'Content-Type': 'application/javascript', + 'content-length': String(HOUSTON_MAX_CATALOG_BYTES + 1), + }, + }); + }); + assert.deepEqual(await loadHoustonSourcesFromOpenData(), []); + + t.mock.restoreAll(); + t.mock.method(console, 'warn', () => {}); + t.mock.method(globalThis, 'fetch', async () => { + const oversized = 'x'.repeat(HOUSTON_MAX_CATALOG_BYTES + 1024); + return new Response( + new ReadableStream({ + start(controller) { + controller.enqueue(new TextEncoder().encode(oversized)); + controller.close(); + }, + }), + { headers: { 'Content-Type': 'application/javascript' } }, + ); + }); + assert.deepEqual(await loadHoustonSourcesFromOpenData(), []); +}); + +test('a reduced pack cap keeps cameras nearest the metro anchors', async (t) => { + t.mock.method(console, 'log', () => {}); + const saved = process.env.CCTV_HOUSTON_MAX_SOURCES; + t.after(() => { + if (saved === undefined) delete process.env.CCTV_HOUSTON_MAX_SOURCES; + else process.env.CCTV_HOUSTON_MAX_SOURCES = saved; + }); + process.env.CCTV_HOUSTON_MAX_SOURCES = '8'; + const rows = [ + ...[29.32, 29.322, 29.324, 29.326, 29.328, 29.33].map((lat, i) => + row({ + id: String(3000 + i), + lat: String(lat), + lng: '-94.80', + location: `Galveston ${i}`, + dir: 'East', + }), + ), + ...[29.76, 29.759, 29.758, 29.757, 29.756, 29.755].map((lat, i) => + row({ + id: String(4000 + i), + lat: String(lat), + lng: '-95.3698', + location: `Downtown ${i}`, + dir: 'North', + }), + ), + ]; + t.mock.method(globalThis, 'fetch', async () => + Response.json({ cameras: rows }), + ); + const cameras = await loadHoustonSourcesFromOpenData(); + assert.equal(cameras.length, 8); + assert.deepEqual( + cameras.map((camera) => camera.name.slice(0, 8)), + [ + 'Downtown', + 'Downtown', + 'Downtown', + 'Downtown', + 'Downtown', + 'Downtown', + 'Galvesto', + 'Galvesto', + ], + 'downtown fills first; the Galveston anchor still keeps a share', + ); +}); + +/** + * Serve the Houston catalog to the Houston endpoint and an empty payload to + * every other pack, so one catalog refresh exercises the registration without + * reaching the network. + */ +const runCatalogWithMockedUpstreams = async (t) => { + const requested = []; + t.mock.method(console, 'log', () => {}); + t.mock.method(console, 'warn', () => {}); + t.mock.method(globalThis, 'fetch', async (url) => { + const href = String(url); + requested.push(href); + if (href === HOUSTON_CAMERAS_URL) { + return Response.json({ + cameras: [ + row(), + row({ + id: GALLERIA_ID, + location: '610 WEST LOOP @ SAN FELIPE', + lat: '29.749', + lng: '-95.461', + dir: 'West', + }), + ], + }); + } + return Response.json([]); + }); + const sources = await createCctvCatalog({ sourceRoot: '/nonexistent' })(); + return { requested, sources }; +}; + +test('the Houston lane is wired into the catalog and its loader runs', async (t) => { + const saved = { ...process.env }; + try { + delete process.env.CCTV_SOURCES_FILE; + delete process.env.CCTV_SOURCES_JSON; + delete process.env.CCTV_HOUSTON_ENABLED; + const { requested, sources } = await runCatalogWithMockedUpstreams(t); + assert.ok( + requested.includes(HOUSTON_CAMERAS_URL), + 'the catalog refresh invokes the Houston loader', + ); + assert.deepEqual( + sources.filter((s) => s.cityId === 'houston').map((s) => s.id), + [`houston-${GALLERIA_ID}`, `houston-${DOWNTOWN_ID}`], + 'Houston cameras reach the served catalog through the registered lane', + ); + } finally { + for (const key of Object.keys(process.env)) { + if (!(key in saved)) delete process.env[key]; + } + Object.assign(process.env, saved); + } +}); + +test('CCTV_HOUSTON_ENABLED=0 keeps the lane from being loaded at all', async (t) => { + const saved = { ...process.env }; + try { + delete process.env.CCTV_SOURCES_FILE; + delete process.env.CCTV_SOURCES_JSON; + process.env.CCTV_HOUSTON_ENABLED = '0'; + const { requested, sources } = await runCatalogWithMockedUpstreams(t); + assert.equal( + requested.includes(HOUSTON_CAMERAS_URL), + false, + 'the disabled lane never reaches its upstream', + ); + assert.deepEqual( + sources.filter((s) => s.cityId === 'houston'), + [], + ); + } finally { + for (const key of Object.keys(process.env)) { + if (!(key in saved)) delete process.env[key]; + } + Object.assign(process.env, saved); + } +}); + +test('the shipped catalog ceiling is not raised to make room for this pack', () => { + assert.equal(DEFAULT_CCTV_MAX_SOURCES, 4000); + assert.equal(DEFAULT_HOUSTON_MAX_SOURCES, 250); +}); + +test('isLikelyHoustonCoordinate spans the metro and rejects the rest', () => { + assert.equal(isLikelyHoustonCoordinate(29.7604, -95.3698), true); // Downtown + assert.equal(isLikelyHoustonCoordinate(29.3013, -94.7977), true); // Galveston + assert.equal(isLikelyHoustonCoordinate(29.9902, -95.3368), true); // IAH + assert.equal(isLikelyHoustonCoordinate(32.7767, -96.797), false); // Dallas + assert.equal(isLikelyHoustonCoordinate(30.2672, -97.7431), false); // Austin + assert.equal(isLikelyHoustonCoordinate(0, 0), false); + assert.equal(isLikelyHoustonCoordinate(NaN, -95.37), false); +}); diff --git a/src/data/dataCredits.js b/src/data/dataCredits.js index c13c7dd94..b74e8c061 100644 --- a/src/data/dataCredits.js +++ b/src/data/dataCredits.js @@ -300,6 +300,12 @@ export const DATA_CREDITS = [ 'Traffic cameras (Toronto): contains information licensed under the ' + 'Open Government Licence – Toronto', }, + { + key: 'houston-cctv', + html: + 'CCTV cameras & frames (Houston): ' + + 'Houston TranStar', + }, { key: 'gbfs', html: From 01a6734a5620a8ad4668f7a1ce0d65040f2f8bf7 Mon Sep 17 00:00:00 2001 From: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Date: Fri, 16 Oct 2026 18:22:41 -0700 Subject: [PATCH 04/48] fix(cctv): make the Houston TranStar pack opt-in TranStar's disclaimer prohibits copying, distribution and external framing of specific images without TxDOT's prior written permission, so the pack no longer loads by default. It now uses envOptIn and loads only when CCTV_HOUSTON_ENABLED=1, the same pattern as NYC DOT and Via Lietuva. - DATA_SOURCES.md: replace the licence column with the disclaimer's terms and a link, and document the opt-in - Ambient stills for Houston refresh every 3 minutes, matching TranStar's stated snapshot cadence - README, CHANGELOG, CURRENT-STATE and .env.example describe the pack as opt-in; the README camera count no longer includes Houston - Tests cover the default-off and =0 cases and the opted-in catalog path --- .env.example | 2 +- CHANGELOG.md | 11 --------- DATA_SOURCES.md | 4 +-- README.md | 2 +- changelog.d/1125-houston-cctv-pack.md | 13 ++++++++++ docs/CURRENT-STATE.md | 14 ++++++----- server/providers/cctv/catalog.js | 2 +- server/providers/cctv/sources.js | 3 ++- src/data/cctvHouston.test.mjs | 35 ++++++++++++++++----------- src/data/cctvLod.js | 2 ++ src/data/cctvLod.test.mjs | 1 + 11 files changed, 52 insertions(+), 37 deletions(-) create mode 100644 changelog.d/1125-houston-cctv-pack.md diff --git a/.env.example b/.env.example index 265377bf7..d176acf40 100644 --- a/.env.example +++ b/.env.example @@ -237,7 +237,7 @@ AISSTREAM_API_KEY= # CCTV_TORONTO_LIST_URL= # override the Toronto RESCU camera list URL # CCTV_LITHUANIA_ENABLED= # off by default; set 1 to load Via Lietuva / Eismoinfo road cameras (photo licence unconfirmed; see DATA_SOURCES.md) # CCTV_LITHUANIA_MAX_SOURCES=400 # cap Lithuanian cameras; no API key required -# CCTV_HOUSTON_ENABLED=1 # set 0 to disable the Houston TranStar cameras +# CCTV_HOUSTON_ENABLED= # set to 1 to opt in; off by default (TranStar terms) # CCTV_HOUSTON_MAX_SOURCES=250 # cap cameras loaded from the Houston TranStar catalog (nearest metro cores first) # CCTV_HOUSTON_CAMERAS_URL= # override the Houston TranStar catalog URL # CCTV_MAX_SOURCES=4000 # catalog-wide ceiling, shared round-robin across packs diff --git a/CHANGELOG.md b/CHANGELOG.md index b4de32107..98d32365f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,16 +1,5 @@ # Changelog -- Add Houston traffic cameras from Houston TranStar as a CCTV pack. The camera - list is one keyless JSON request to - `traffic.houstontranstar.org/data/layers/cctvSnapshots_json.js`; frames are - stills synthesized from each numeric camera id on `www.houstontranstar.org`, - so a catalog field cannot steer the proxy off-host. Cameras without a - snapshot id, and cameras outside Greater Houston, are dropped. By default - the 250 cameras nearest downtown, Uptown, the Medical Center, IAH and - Galveston load. `CCTV_HOUSTON_ENABLED=0` turns the pack off and - `CCTV_HOUSTON_MAX_SOURCES` changes the cap. The catalog-wide ceiling is - unchanged; extra cameras share it round-robin with the other packs. - - The tracked-aircraft descriptor now publishes `verticalRateMps`, on both the civil and military layers, so a consumer reading `getTrackedInfo()` no longer has to reach past it into the raw poll record for climb/descent. Reported, diff --git a/DATA_SOURCES.md b/DATA_SOURCES.md index 913cad6d1..92f623deb 100644 --- a/DATA_SOURCES.md +++ b/DATA_SOURCES.md @@ -65,7 +65,7 @@ How to read this: | **Statens vegvesen** (catalog `ogckart-sn1.atlas.vegvesen.no`, frames `kamera.atlas.vegvesen.no`, HLS `kamera.vegvesen.no`) | CCTV camera catalog + frames + live HLS video, Norway road cameras | [NLOD 2.0](https://data.norge.no/nlod/en/2.0) — attribution REQUIRED | "Contains data under the Norwegian licence for Open Government data (NLOD) distributed by Statens vegvesen" | | **NYC DOT Traffic Management Center** (`webcams.nyctmc.org`) | CCTV camera catalog + frames, New York City five boroughs | NYC DOT requires developers to sign a [data-sharing agreement](https://www.nyc.gov/html/dot/html/about/datafeeds.shtml) for the camera feed. **Off by default**; enable only under your own agreement | "NYC DOT Traffic Management Center" + NYC DOT's required application disclaimer (in the Data attribution popover) | | **City of Toronto Open Data** (`opendata.toronto.ca`) | CCTV camera catalog + frames, Toronto city streets | [Open Government Licence – Toronto](https://www.toronto.ca/city-government/data-research-maps/open-data/open-data-licence/) — attribution REQUIRED | "Contains information licensed under the Open Government Licence – Toronto" | -| **Houston TranStar** (`traffic.houstontranstar.org`, frames `www.houstontranstar.org`) | CCTV camera catalog + frames, Greater Houston | Public Houston TranStar traffic camera stills | "Houston TranStar" (courtesy) | +| **Houston TranStar** (`traffic.houstontranstar.org`, frames `www.houstontranstar.org`) | CCTV camera catalog + frames, Greater Houston | No licence. The [TranStar disclaimer](https://traffic.houstontranstar.org/disclaimer.aspx) (Content Usage) prohibits copying or distribution, and external framing or linking to specific images, without TxDOT's prior written permission. Off unless `CCTV_HOUSTON_ENABLED=1` | "Houston TranStar" (courtesy) | | **GBFS (Lyft / BCycle / Social Bicycles / PBSC)** | Bikeshare availability | Per-feed (attribution-only) | Credit the operator (e.g. Austin BCycle) + its `license_url` | | **Bike Share Toronto** (`toronto.publicbikesystem.net`) | Bikeshare station availability, Toronto | Operator GBFS feed. No licence is published: the City's open-data package declares none and the feed has no `license_url`. Attribution is given to the operator. | "Bike Share Toronto (Toronto Parking Authority)" | | **Bike Itaú / Tembici** (`*.publicbikesystem.net`, seven Brazilian cities, Santiago and Bogotá) | Bikeshare station availability: São Paulo, Rio de Janeiro, Salvador, Recife, Porto Alegre, Brasília, Curitiba, Santiago de Chile and Bogotá | Operator GBFS feeds. No licence is published: the feeds' `system_information.json` carries no `license_url`, operator or attribution fields. Attribution is given to the operator. | "Bike Itaú (Tembici)"; "Tembici Bogotá" | @@ -167,7 +167,7 @@ The optional Local ADS-B layer and the broadcast-FM receiver use radio you recei - **Statens vegvesen road cameras.** Keyless. The camera list is one GeoJSON request to the OGC API Features view of the official DATEX 3.1 CCTV site table (`ogckart-sn1.atlas.vegvesen.no/ogc/features/v1/collections/datex_3_1:CctvSimple/items`), so no DATEX account is needed; frames are stills on `kamera.atlas.vegvesen.no/api/images/`, and only that exact path per camera is registered. Cameras the feed reports as faulty are dropped. About 135 working cameras also publish live HLS (`kamera.vegvesen.no/public//manifest.m3u8`, MPEG-TS); those play as live video through the bounded in-memory HLS relay, with the still as the snapshot fallback, and only that exact manifest path per camera is registered. `CCTV_VEGVESEN_VIDEO=0` keeps every camera on stills. `orientationDescription` names the place a camera faces, not a bearing, so it goes into the label and headings use the id-hash fallback at low confidence. By default every live camera loads (~850); `CCTV_VEGVESEN_MAX_SOURCES` (8–1000) lowers the cap, keeping the cameras nearest Oslo, Bergen, Stavanger, Trondheim, Kristiansand, Bodø and Tromsø and `CCTV_VEGVESEN_ENABLED=0` turns the pack off. Frames are fetched at request time and never stored or redistributed. The NLOD attribution is registered in the Data attribution popover. - **NYC DOT Traffic Management Center cameras.** Off by default. NYC DOT's [developer data feeds page](https://www.nyc.gov/html/dot/html/about/datafeeds.shtml) asks developers who want the camera feed to contact `tmcdot@dot.nyc.gov` and sign a data-sharing agreement. This project does not hold one that covers downstream installations, so the pack loads only when an operator who holds their own agreement sets `CCTV_NYC_ENABLED=1`; with the flag unset, empty or any other value, the server makes no request to NYC DOT. The endpoint needs no key. The camera list is the public `webcams.nyctmc.org/api/cameras` catalog (~980 cameras across the five boroughs); frames are stills at `webcams.nyctmc.org/api/cameras//image`, synthesized from a validated UUID rather than taken from the payload, and only that host is registered. Cameras the feed reports as offline are dropped. The catalog publishes no camera facing, so headings use the shared id-hash fallback at low confidence and are corrected with the in-app calibration gizmo. Once enabled, the 250 cameras nearest the borough cores load; `CCTV_NYC_MAX_SOURCES` (8–400) changes the cap. Frames are fetched at request time and never stored or redistributed. NYC DOT asks every application that uses its data to carry its disclaimer; it is registered verbatim in the Data attribution popover. - **City of Toronto RESCU cameras.** Keyless. The camera list is the City's public `tmcearthcameras.json` on `opendata.toronto.ca` (dataset "Traffic Cameras", Transportation Services), served as JSONP and unwrapped with an anchored pattern. This is the municipal network the Ontario 511 pack does not carry: downtown arterials plus the Gardiner, DVP, Allen and Lake Shore. The list publishes no frame URL, so frame URLs are BUILT from the documented `CameraImages/loc####.jpg` convention and a strictly-digits camera number, pinning the frame proxy to the portal origin by construction; the list fetch refuses redirects and bounds the body it buffers. Frames are fetched at request time and never stored or redistributed. The dataset publishes no camera facing: the `D1`-`D4` fields are the directions of the *comparison* images (static reference stills captured in 2016 under a different path), not a bearing for the live pan-tilt-zoom frame, so headings use the shared id-hash fallback at low confidence and are corrected with the in-app calibration gizmo. A camera that is temporarily out of service still answers 200 with a byte-identical "Image Unavailable" placeholder rather than an error (about 6% of the network at any moment), so such a camera reads as healthy; the frame it shows is the agency's, not a synthesized one. The licence requires the attribution statement above. Disable with `CCTV_TORONTO_ENABLED=0`. -- **Houston TranStar cameras.** Keyless. The camera list is the public `traffic.houstontranstar.org/data/layers/cctvSnapshots_json.js` catalog (~1,000 cameras across Greater Houston); frames are stills at `www.houstontranstar.org/snapshots/cctv/.jpg`, synthesized from a validated numeric id rather than taken from the payload, and only that host is registered. Cameras whose catalog id is `0` (a different, non-snapshot class) and cameras outside Greater Houston are dropped. Dedicated `dir` values (North/South/East/West) become high-confidence headings; `None` uses the shared id-hash fallback at low confidence. By default the 250 cameras nearest downtown, Uptown, the Medical Center, IAH and Galveston load; `CCTV_HOUSTON_MAX_SOURCES` (8–400) changes the cap and `CCTV_HOUSTON_ENABLED=0` turns the pack off. Frames are fetched at request time and never stored or redistributed. +- **Houston TranStar cameras.** Keyless. The camera list is the public `traffic.houstontranstar.org/data/layers/cctvSnapshots_json.js` catalog (~1,000 cameras across Greater Houston); frames are stills at `www.houstontranstar.org/snapshots/cctv/.jpg`, synthesized from a validated numeric id rather than taken from the payload, and only that host is registered. Cameras whose catalog id is `0` (a different, non-snapshot class) and cameras outside Greater Houston are dropped. Dedicated `dir` values (North/South/East/West) become high-confidence headings; `None` uses the shared id-hash fallback at low confidence. TranStar's disclaimer is not a licence to ship the stills on by default (its FAQ grants permission-free use only for the RSS feeds), so the pack stays off unless `CCTV_HOUSTON_ENABLED=1`. Opting in loads the 250 cameras nearest downtown, Uptown, the Medical Center, IAH and Galveston; `CCTV_HOUSTON_MAX_SOURCES` (8–400) changes the cap. Snapshots update about every 3 minutes, which is also the ambient still cadence. Frames are fetched at request time and never stored or redistributed. - **Camera frame content.** Public camera frames can contain people, vehicles and license plates, and the camera layer does nothing with them beyond putting them on the map. A successful upstream response is relayed as the provider served it: nothing in the camera pipeline blurs, redacts, enhances or restores it, and nothing there runs face, plate or object detection or otherwise recognises or classifies what is in the picture. (The ALPR layer described below maps where plate-reader cameras are, from locations contributors tagged in OpenStreetMap; it reads nothing from any camera frame.) When an upstream has no frame for a camera, what you see instead is a Google Street View still or the app's own placeholder card; the panel's source badge names which of them you are looking at, and a refresh that fails after a frame has arrived leaves the last good picture on screen rather than replacing it. The browser resamples each frame for display: onto a fixed 1920x1080 surface for the camera's projection plane in the scene, and onto a 192x108 canvas for the small ambient card. That scaling is the only change made to the picture, and the only pixels the camera layer reads back are a 64x36 downscale hashed to tell whether the picture has changed. Live streams are piped through; the still path reads the image into memory to pass it on — TxDOT wraps its JPEG in a JSON envelope, which is decoded there — and no frame is written to disk (the disk cache, `.gev-cache/`, holds feed JSON for other layers and never camera imagery). Outside that pipeline, the voice assistant is the one feature that sends imagery anywhere: when it answers a question about what is in view it may attach a downscaled screenshot of the rendered scene — including any camera plane on screen — to the request it sends to the voice model. Frames are fetched only from server-registered camera URLs — a client cannot hand the proxy a URL (see [SECURITY.md](SECURITY.md)). What the app does stays inside the line README draws: no named-person search, no face recognition, no tracking individuals. - **Transit (GTFS-Realtime).** The proxy requests only registered operator feeds, validates redirects, bounds responses, and uses conditional requests, short snapshot caching and failure backoff. All sixteen registered regions are live-enabled; each can be disabled independently. Vehicles use delayed playback between reports, with report age and waiting states shown separately. Session-local selected trails are available for every feed. MBTA additionally retains up to 15 minutes of recently observed positions in the running proxy, so available history can survive a browser reload; server restart clears it. History reads never fetch upstream, and other feeds have no proxy retention. Nothing is archived to disk or collected in the background. Operator text credits are included as a courtesy and do not imply endorsement. Heights are aligned to work with Google 3D tiles; subway markers are shown at street level because these feeds do not supply depth. The TTC feed is BusTime, the surface system: it carries buses and streetcars and no subway at all. About four in ten of its vehicles report no route at all (unassigned or deadheading), and those are shown with the feed default rather than a route-established mode. It sends no ETag or Last-Modified, so its conditional requests fall through to the ordinary snapshot cache. Three Golden Horseshoe neighbours run alongside it — MiWay, Hamilton HSR and Durham Region Transit — each bus-only and each polled only near its own region. Durham Region Transit publishes no bearing on any vehicle, so those sprites take their heading from observed movement like any other feed that omits it. Guelph, Kingston and Barrie extend the same pattern eastward and north. Hamilton, Guelph, Burlington and YRT publish route ids that are not the number on the vehicle, so those entries carry a route-name table taken from each operator’s own published GTFS; any id missing from a table renders unchanged rather than blank. A label is never invented: YRT runs a route `9101` that appears nowhere in its published routes, so it stays as the feed sends it. Burlington’s is the one that cannot be shortcut: `351` is route 1 while `3510` is route 10, so the table is a real join rather than a trimmed prefix. GO Transit / Metrolinx is deliberately absent: its REALTIME feed requires an API key issued by manual review, which the keyless admission rule excludes (its static schedule download is keyless, but carries no vehicle positions). Barrie and YRT/Viva each invite acceptance of their licence through a form, and each licence also states that using the data is itself acceptance; this deployment relies on that clause, which is recorded in both entries. Grand River Transit in Waterloo is absent for a different reason: it answers, but its feed header was months stale and it publishes no per-vehicle timestamps, so it would render vehicles that look live and are not. Ottawa’s OC Transpo falls to the same keyless admission rule as GO Transit — its realtime feed requires an API key. London Transit and Thunder Bay Transit publish vehicle positions over plain HTTP with no HTTPS endpoint available. Every other Ontario system — Brampton, Oakville, Niagara, Milton, Windsor, Greater Sudbury, UP Express and a dozen smaller operators — publishes static GTFS schedules only, with no realtime vehicle positions to register. Between the nine included and these exclusions, every Ontario realtime vehicle-position feed catalogued in the Mobility Database is accounted for. - **Radio Browser.** `/api/radio/stations` discovers official API mirrors, makes bounded and coalesced healthy/geolocated HTTPS-station queries, caches the normalized public-domain directory for 45 minutes, and may serve the last good catalog for up to seven days during an outage. Refreshes must meet minimum accepted-query and station coverage before replacing a warm catalog; schema-valid responses whose rows all fail the product's health policy do not count as successful queries. A usable partial cold catalog is explicitly `DEGRADED`, and malformed or empty successful payloads are rejected atomically. Every directory and click-count request rejects redirects, validates all resolved addresses as globally routable (including reserved/documentation IPv4 and special/non-global IPv6 exclusions), and pins the TLS connection to a validated address. Only MP3/AAC non-HLS directory rows with public HTTPS stream targets are returned; favicons are intentionally omitted. Pressing play connects one browser audio element directly to the selected broadcaster and calls the directory's click counter through known-ID-only `POST /api/radio/click/:uuid`. GEV never proxies, caches, records, bundles, or redistributes audio. Radio Browser supplies station-level tags, not dependable current-song or upcoming-program metadata, so Radio filtering never claims either. Direct playback exposes the listener's IP address to the broadcaster, whose own stream terms apply. diff --git a/README.md b/README.md index 867f14824..6c2baec79 100644 --- a/README.md +++ b/README.md @@ -377,7 +377,7 @@ Twenty layers and map sources. **Eighteen have a keyless path.** Some offer addi | 🌍 **Earthquakes** | Global seismic activity, last 24h | USGS | 🟢 | | ◍ **Air quality** | Live AQHI on the published 1–10+ health-risk scale, with a Weather-panel card showing the band legend and the highest readings in view. **Canada only** — ECCC is the first registered provider | ECCC MSC GeoMet | 🟢 | | 🚗 **Traffic** | Simulated vehicles on OSM roads. With TomTom, live flow speeds drive the simulation and congestion colors below ~8 km; individual vehicle positions are not live observations | TomTom + OSM | 🟢 simulation · 🟡 live flow speeds | -| 📹 **CCTV Mesh** | ~4,000 public cameras (shared cap) projected _into_ the 3D space — Austin · Texas (TxDOT) · California (Caltrans) · London (TfL) · Ontario (511) · Finland (Fintraffic) · British Columbia (DriveBC) · Estonia (Tallinn, Tarktee) · Delaware (DelDOT live video) · New South Wales (Live Traffic NSW) · Calgary · Norway (Statens vegvesen) · Toronto · King County (WA) · Washington (WSDOT) · Catalonia (Servei Català de Trànsit) · Houston (TranStar), plus opt-in New York City (NYC DOT TMC, needs your own NYC DOT agreement) and opt-in Lithuania (Via Lietuva / Eismoinfo, photo licence unconfirmed). Positions are published; poses are estimated priors **you calibrate by dragging a gizmo on the camera itself** | City APIs | 🟢 | +| 📹 **CCTV Mesh** | ~4,000 public cameras (shared cap) projected _into_ the 3D space — Austin · Texas (TxDOT) · California (Caltrans) · London (TfL) · Ontario (511) · Finland (Fintraffic) · British Columbia (DriveBC) · Estonia (Tallinn, Tarktee) · Delaware (DelDOT live video) · New South Wales (Live Traffic NSW) · Calgary · Norway (Statens vegvesen) · Toronto · King County (WA) · Washington (WSDOT) · Catalonia (Servei Català de Trànsit), plus opt-in New York City (NYC DOT TMC, needs your own NYC DOT agreement), opt-in Lithuania (Via Lietuva / Eismoinfo, photo licence unconfirmed) and opt-in Houston (TranStar, off unless CCTV_HOUSTON_ENABLED=1). Positions are published; poses are estimated priors **you calibrate by dragging a gizmo on the camera itself** | City APIs | 🟢 | | 📷 **Mapped ALPR Cameras** | License-plate-reader camera locations tagged by OpenStreetMap contributors, loaded one city-sized view at a time, with **SHOW NEAREST**. Locations and tags only: no plate data, no video | OpenStreetMap (incl. DeFlock mapping) | 🟢 | | 📷 **Street Level** | Street-level photos behind one panel, one chip per provider: coverage on the streets, image cones, an embedded viewer and **FOLLOW**. Mapillary, Google Street View, KartaView and Panoramax | Mapillary · Google · KartaView · Panoramax | 🟢 KartaView, Panoramax · 🟡 Mapillary token · 🔴 Google Maps key | | 📻 **Radio** | Geolocated world radio with an **analog tuner** — drag the needle across up to 750 stations and the globe flies to each broadcaster | Radio Browser / broadcasters | 🟢 | diff --git a/changelog.d/1125-houston-cctv-pack.md b/changelog.d/1125-houston-cctv-pack.md new file mode 100644 index 000000000..dc573993f --- /dev/null +++ b/changelog.d/1125-houston-cctv-pack.md @@ -0,0 +1,13 @@ +- Add Houston traffic cameras from Houston TranStar as a CCTV pack. The camera + list is one keyless JSON request to + `traffic.houstontranstar.org/data/layers/cctvSnapshots_json.js`; frames are + stills synthesized from each numeric camera id on `www.houstontranstar.org`, + so a catalog field cannot steer the proxy off-host. Cameras without a + snapshot id, and cameras outside Greater Houston, are dropped. TranStar's + disclaimer prohibits redistribution and external framing without TxDOT's + written permission, so the pack is off unless `CCTV_HOUSTON_ENABLED=1`. + Opting in loads the 250 cameras nearest downtown, Uptown, the Medical + Center, IAH and Galveston; `CCTV_HOUSTON_MAX_SOURCES` changes that cap. + Snapshots update about every 3 minutes. The catalog-wide ceiling is + unchanged; extra cameras share it round-robin with the other packs + (mvanhorn, #1125). diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 99cfc3234..0c8fe6261 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1083,11 +1083,13 @@ URLs are built from each camera id under from the payload. Dedicated `dir` values become high-confidence headings; `None` uses the shared id-hash fallback at low confidence. -By default the 250 cameras nearest downtown, Uptown, the Medical Center, IAH -and Galveston load. `CCTV_HOUSTON_MAX_SOURCES` sets the pack cap (8–400) and -`CCTV_HOUSTON_ENABLED=0` turns the pack off. The catalog-wide ceiling stays at -4,000 and is shared round-robin, so adding this pack does not silently drop -another region. +TranStar's disclaimer prohibits redistribution and external framing of specific +images without TxDOT's written permission, so the pack is off unless +`CCTV_HOUSTON_ENABLED=1`. Opting in loads the 250 cameras nearest downtown, +Uptown, the Medical Center, IAH and Galveston. `CCTV_HOUSTON_MAX_SOURCES` sets +the pack cap (8–400). Snapshots update about every 3 minutes. The catalog-wide +ceiling stays at 4,000 and is shared round-robin, so adding this pack does not +silently drop another region. ## Location control ownership @@ -3383,7 +3385,7 @@ its criteria cannot be silently ignored. | Satellites | CelesTrak | `src/data/satellites.js` | `/api/celestrak` | 120s | | Space Missions (30d) | Launch Library 2 + CelesTrak | `src/data/rocketLaunches.js` | `/api/launches` + `/api/celestrak/active` | 5 min | | Traffic | Selectable TomTom / OpenStreetMap / Hybrid roads; optional TomTom flow (BYOK) | `src/data/traffic.js` | browser-direct OpenFreeMap tiles; `/api/tomtom` for flow | viewport-driven; capped tile cache | -| CCTV | Austin + Caltrans (CA) + TfL London + Ontario 511 + Fintraffic (FI) + DriveBC (BC) + TxDOT (TX) + Estonia (Tallinn, Tarktee) + Live Traffic NSW + Open Calgary Open Data + Statens vegvesen (NO, stills + live HLS) + King County (WA) Open Data + WSDOT (WA) Open Data + NYC DOT TMC (opt-in) + City of Toronto RESCU + Catalonia (Servei Català de Trànsit) + Houston TranStar + Street View fallback | `src/data/cctv.js` | `/api/cctv` | 10s (active) | +| CCTV | Austin + Caltrans (CA) + TfL London + Ontario 511 + Fintraffic (FI) + DriveBC (BC) + TxDOT (TX) + Estonia (Tallinn, Tarktee) + Live Traffic NSW + Open Calgary Open Data + Statens vegvesen (NO, stills + live HLS) + King County (WA) Open Data + WSDOT (WA) Open Data + NYC DOT TMC (opt-in) + City of Toronto RESCU + Catalonia (Servei Català de Trànsit) + Houston TranStar (opt-in) + Street View fallback | `src/data/cctv.js` | `/api/cctv` | 10s (active) | | Radio | Radio Browser (public-domain station directory) | `src/data/radio.js` | `/api/radio/stations`, `/api/radio/click/:uuid` | 45 min directory refresh | | Transit 🚌 | Operator GTFS-Realtime VehiclePositions (7 keyless regions, `src/data/transitFeeds.js`) | `src/layers/transit/` via `src/app/layers/transit.js` | `/api/transit` | 15s (poll + delayed playback) | | Bikeshare 🚲 | GBFS (Lyft + BCycle + Social Bicycles + PBSC incl. Bike Share Toronto, Bike Itaú (Brazil) and Bicing (Barcelona) + Urban Sharing, Entur, JCDecaux NO + nextbike (AMBici)) | `src/data/bikeshare.js` | `/api/gbfs` | 60s | diff --git a/server/providers/cctv/catalog.js b/server/providers/cctv/catalog.js index 0ed194660..2268b45ee 100644 --- a/server/providers/cctv/catalog.js +++ b/server/providers/cctv/catalog.js @@ -147,7 +147,7 @@ const LIVE_PACKS = [ }, { name: 'houston', - enabled: () => envEnabled('CCTV_HOUSTON_ENABLED'), + enabled: () => envOptIn('CCTV_HOUSTON_ENABLED'), load: loadHoustonSourcesFromOpenData, }, ]; diff --git a/server/providers/cctv/sources.js b/server/providers/cctv/sources.js index 895fe4c4d..f51d6bf35 100644 --- a/server/providers/cctv/sources.js +++ b/server/providers/cctv/sources.js @@ -2854,7 +2854,8 @@ export function houstonCameraToSource(record) { url: imageUrl, snapshotUrl: imageUrl, sourceKind: 'houston-transtar', - license: 'Public Houston TranStar traffic camera', + license: + 'TranStar disclaimer — redistribution and external framing need TxDOT written permission', code: cameraDisplayCode(name.toUpperCase()), }; } diff --git a/src/data/cctvHouston.test.mjs b/src/data/cctvHouston.test.mjs index 97de95a84..54aace016 100644 --- a/src/data/cctvHouston.test.mjs +++ b/src/data/cctvHouston.test.mjs @@ -67,7 +67,10 @@ test('a Houston row maps to a source on the pinned HTTPS frame host', () => { assert.equal(source.lon, -95.355711); assert.equal(source.feedType, 'image'); assert.equal(source.sourceKind, 'houston-transtar'); - assert.equal(source.license, 'Public Houston TranStar traffic camera'); + assert.equal( + source.license, + 'TranStar disclaimer — redistribution and external framing need TxDOT written permission', + ); assert.equal(source.groundElevationM, HOUSTON_GROUND_ELEVATION_M); assert.equal(source.url, HOUSTON_IMAGE_URL(DOWNTOWN_ID)); assert.equal(source.url, source.snapshotUrl); @@ -290,6 +293,7 @@ test('a reduced pack cap keeps cameras nearest the metro anchors', async (t) => */ const runCatalogWithMockedUpstreams = async (t) => { const requested = []; + t.mock.restoreAll(); t.mock.method(console, 'log', () => {}); t.mock.method(console, 'warn', () => {}); t.mock.method(globalThis, 'fetch', async (url) => { @@ -320,7 +324,7 @@ test('the Houston lane is wired into the catalog and its loader runs', async (t) try { delete process.env.CCTV_SOURCES_FILE; delete process.env.CCTV_SOURCES_JSON; - delete process.env.CCTV_HOUSTON_ENABLED; + process.env.CCTV_HOUSTON_ENABLED = '1'; const { requested, sources } = await runCatalogWithMockedUpstreams(t); assert.ok( requested.includes(HOUSTON_CAMERAS_URL), @@ -339,22 +343,25 @@ test('the Houston lane is wired into the catalog and its loader runs', async (t) } }); -test('CCTV_HOUSTON_ENABLED=0 keeps the lane from being loaded at all', async (t) => { +test('Houston stays unloaded unless CCTV_HOUSTON_ENABLED=1', async (t) => { const saved = { ...process.env }; try { delete process.env.CCTV_SOURCES_FILE; delete process.env.CCTV_SOURCES_JSON; - process.env.CCTV_HOUSTON_ENABLED = '0'; - const { requested, sources } = await runCatalogWithMockedUpstreams(t); - assert.equal( - requested.includes(HOUSTON_CAMERAS_URL), - false, - 'the disabled lane never reaches its upstream', - ); - assert.deepEqual( - sources.filter((s) => s.cityId === 'houston'), - [], - ); + for (const value of [undefined, '0']) { + if (value === undefined) delete process.env.CCTV_HOUSTON_ENABLED; + else process.env.CCTV_HOUSTON_ENABLED = value; + const { requested, sources } = await runCatalogWithMockedUpstreams(t); + assert.equal( + requested.includes(HOUSTON_CAMERAS_URL), + false, + 'the pack reaches TranStar only after an explicit opt-in', + ); + assert.deepEqual( + sources.filter((s) => s.cityId === 'houston'), + [], + ); + } } finally { for (const key of Object.keys(process.env)) { if (!(key in saved)) delete process.env[key]; diff --git a/src/data/cctvLod.js b/src/data/cctvLod.js index 44b2e8ba3..d91e3b4b3 100644 --- a/src/data/cctvLod.js +++ b/src/data/cctvLod.js @@ -34,6 +34,8 @@ const PROVIDER_STATIC_REFRESH_MS = Object.freeze({ 'ontario 511': 3 * 60 * 1000, // TxDOT publishes roughly once a minute; 3 min matches the other highway packs. txdot: 3 * 60 * 1000, + // TranStar says snapshots update about every 3 minutes. + 'houston transtar': 3 * 60 * 1000, // Digitraffic publishes a new weathercam frame on each station's // collectionInterval, 600 s for every station sampled; polling faster only // re-fetches the same JPEG. diff --git a/src/data/cctvLod.test.mjs b/src/data/cctvLod.test.mjs index 4f5982496..55821ae43 100644 --- a/src/data/cctvLod.test.mjs +++ b/src/data/cctvLod.test.mjs @@ -639,6 +639,7 @@ test('staticFrameRefreshMs follows known pack cadences and bounds explicit value assert.equal(staticFrameRefreshMs({ provider: 'Caltrans' }), 180_000); assert.equal(staticFrameRefreshMs({ provider: 'Ontario 511' }), 180_000); assert.equal(staticFrameRefreshMs({ provider: 'TxDOT' }), 180_000); + assert.equal(staticFrameRefreshMs({ provider: 'Houston TranStar' }), 180_000); // Digitraffic publishes one weathercam frame per 600 s collection interval. assert.equal(staticFrameRefreshMs({ provider: 'Fintraffic' }), 600_000); assert.equal( From 7a39d2d29b4ecd186b0c80c408b488d304c189cc Mon Sep 17 00:00:00 2001 From: quanticsoul4772 <71096528+quanticsoul4772@users.noreply.github.com> Date: Thu, 29 Oct 2026 06:51:42 -0700 Subject: [PATCH 05/48] fix(traffic): give paired TomTom sides of one street a two-way share of dots TomTom draws a two-way street whose directions differ as two one_side lines on one centreline, and each got a one-way road's full share of simulated vehicles, so the street drew twice the dots of the same street as one full-coverage line or a two-way OSM road. A one_side line with an opposite-direction one_side line within 5 m of at least two of three points along it now gets a two-way direction's half share. On the decoded Austin flow fixture every one_side line is paired, and road length weighted by direction share falls from 553 to 417 km. Pairing a 16-tile area takes about 41 ms per road pass. Refs #1455. Co-Authored-By: Claude Opus 5.5 --- docs/CURRENT-STATE.md | 5 ++ src/layers/traffic/roadModes.js | 77 ++++++++++++++++++++++++--- src/layers/traffic/roadModes.test.mjs | 47 ++++++++++++++++ 3 files changed, 121 insertions(+), 8 deletions(-) diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 766823550..e882ef822 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -5897,6 +5897,11 @@ easier to meet (detection is now on more often), but does not create it. TomTom draws only flow-tile lines with their own flow; roads without flow are not drawn and the status says so. A `full`-coverage flow line carries both travel directions, a `one_side` line the direction it is drawn in. + When an opposite-direction `one_side` line runs within 5 m of at least two + of three points a quarter, half and three quarters along a `one_side` line, + the two are one street's directions and each + gets a two-way direction's half share of dots, so the street is as dense as + one `full` line (#1455). OSM draws OpenFreeMap roads with flow matched onto them (below). Hybrid draws eligible TomTom motor-road lines and adds OpenFreeMap roads only where no eligible TomTom line runs within 35 m in the same travel direction (30 degrees), tested every diff --git a/src/layers/traffic/roadModes.js b/src/layers/traffic/roadModes.js index 05b7ba4f7..e9220b635 100644 --- a/src/layers/traffic/roadModes.js +++ b/src/layers/traffic/roadModes.js @@ -35,6 +35,14 @@ const MIN_FILL_M = 40; * Same-class overlap keeps the full 35 m matcher radius. */ const CLASS_MISMATCH_RADIUS_M = 15; +/** + * Opposite-direction TomTom lines this close share one street: TomTom draws + * both directions of a two-way street a metre or two apart, and a divided + * carriageway's sides further apart. + */ +const PAIRED_SIDE_RADIUS_M = 5; +/** Arc-length positions sampled along a `one_side` line for its partner. */ +const PAIRED_SIDE_SAMPLES = [0.25, 0.5, 0.75]; const CONTROLLED_ACCESS_FLOW = new Set(['Motorway', 'International road']); const CONTROLLED_ACCESS_OSM = new Set(['motorway', 'trunk']); @@ -85,20 +93,65 @@ export function isDrivableFlowSegment(segment) { ); } +/** + * Share of a `one_side` line's sample points (PAIRED_SIDE_SAMPLES) with an + * opposite-direction TomTom line beside them: TomTom draws a two-way street + * whose directions differ as two `one_side` lines on one centreline, while a + * divided carriageway's other side is further away than PAIRED_SIDE_RADIUS_M. + * @returns {number} Paired share in [0, 1]. + */ +function pairedSampleShare(segment, index) { + const xs = []; + const ys = []; + const cum = [0]; + for (const [lon, lat] of segment.coords) { + const [x, y] = index.project(lon, lat); + const i = xs.push(x) - 1; + ys.push(y); + if (i) cum.push(cum[i - 1] + Math.hypot(x - xs[i - 1], y - ys[i - 1])); + } + const length = cum[cum.length - 1]; + if (!(length > 0)) return 0; + let paired = 0; + let i = 1; + for (const share of PAIRED_SIDE_SAMPLES) { + const target = share * length; + while (i < cum.length - 1 && cum[i] < target) i++; + const dx = xs[i] - xs[i - 1]; + const dy = ys[i] - ys[i - 1]; + const t = (target - cum[i - 1]) / (cum[i] - cum[i - 1] || 1); + const reverse = (Math.atan2(-dx, -dy) * 180) / Math.PI; + if ( + index.nearest( + xs[i - 1] + dx * t, + ys[i - 1] + dy * t, + reverse, + () => PAIRED_SIDE_RADIUS_M, + ).best + ) + paired++; + } + return paired / PAIRED_SIDE_SAMPLES.length; +} + /** * Convert decoded TomTom flow lines into road records that carry their own * flow. A line with `coverage: 'full'` describes both travel directions; - * `one_side` lines describe the direction they are drawn in. + * `one_side` lines describe the direction they are drawn in. A `one_side` + * line paired with the opposite direction on the same street gets that + * direction's half share of dots, as a two-way road's direction does. * @param {Array<{coords:number[][], trafficLevel:number, roadType:string, closure:boolean, coverage?:string}>} segments * @returns {Array} Road records (`directFlow: true`). */ export function flowSegmentsToRoads(segments) { - return (segments || []) - .filter( - (segment) => - segment?.coords?.length >= 2 && isDrivableFlowSegment(segment), - ) - .map((segment) => ({ + const drivable = (segments || []).filter( + (segment) => segment?.coords?.length >= 2 && isDrivableFlowSegment(segment), + ); + const sides = createFlowSegmentIndex( + drivable.filter((segment) => segment.coverage !== 'full'), + ); + return drivable.map((segment) => { + const road = { coordinates: segment.coords, type: FLOW_ROAD_TYPES[segment.roadType], drivable: true, @@ -110,7 +163,15 @@ export function flowSegmentsToRoads(segments) { oneway: segment.coverage === 'full' ? 0 : 1, flow: { level: segment.trafficLevel, closure: segment.closure === true }, directFlow: true, - })); + }; + if ( + segment.coverage !== 'full' && + sides && + pairedSampleShare(segment, sides) >= 0.5 + ) + road.densityWeight = 0.5; + return road; + }); } /** Cut a projected polyline to the arc-length interval [from, to]. */ diff --git a/src/layers/traffic/roadModes.test.mjs b/src/layers/traffic/roadModes.test.mjs index 1d1cd4986..465d41496 100644 --- a/src/layers/traffic/roadModes.test.mjs +++ b/src/layers/traffic/roadModes.test.mjs @@ -116,6 +116,53 @@ test('a full-coverage TomTom line is two-way and covers both OSM directions', () assert.equal(roads.length, 1); assert.equal(roads[0].directFlow, true); }); +test("opposite TomTom sides of one street share a two-way street's dots", () => { + // About 1 m and 20 m north of `coordinates`, drawn westbound. + const back = (dLat) => ({ + ...flow, + coords: coordinates.map(([lon, lat]) => [lon, lat + dLat]).reverse(), + }); + const weights = (segments) => + flowSegmentsToRoads(segments).map((road) => road.densityWeight ?? 1); + assert.deepEqual(weights([flow, back(0.00001)]), [0.5, 0.5]); + assert.deepEqual(weights([flow]), [1], 'a lone side keeps the one-way share'); + assert.deepEqual( + weights([flow, back(0.00018)]), + [1, 1], + 'a divided carriageway keeps one-way shares', + ); + assert.deepEqual( + weights([ + flow, + { + ...flow, + coords: coordinates.map(([lon, lat]) => [lon, lat + 0.00001]), + }, + ]), + [1, 1], + 'same-direction lines are not a pair', + ); + assert.deepEqual( + weights([{ ...flow, coverage: 'full' }, back(0.00001)]), + [1, 1], + 'a full-coverage line is not a side, and pairs with nothing', + ); +}); +test('a TomTom side paired along only part of its length keeps its share', () => { + // Westbound partners about 1 m north, from the start of `flow` to 30% and + // to 60% of its length: one or two of its three sample points. + const partner = (toLon) => ({ + ...flow, + coords: [ + [toLon, 30.26701], + [-97.744, 30.26701], + ], + }); + const weightOfFlow = (other) => + flowSegmentsToRoads([flow, other])[0].densityWeight ?? 1; + assert.equal(weightOfFlow(partner(-97.7434)), 1); + assert.equal(weightOfFlow(partner(-97.7428)), 0.5); +}); test('bearing and distance reject nearby crossing and separate OSM roads', () => { const parallel = { ...main, From 1cd4dde26df2f4bf33909fbf3894696399c8a339 Mon Sep 17 00:00:00 2001 From: quanticsoul4772 <71096528+quanticsoul4772@users.noreply.github.com> Date: Sun, 1 Nov 2026 06:30:45 -0800 Subject: [PATCH 06/48] docs(changelog): credit #1495 in the fragment Co-Authored-By: Claude Opus 5.5 --- changelog.d/1495-traffic-paired-sides.md | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 changelog.d/1495-traffic-paired-sides.md diff --git a/changelog.d/1495-traffic-paired-sides.md b/changelog.d/1495-traffic-paired-sides.md new file mode 100644 index 000000000..4007daf8e --- /dev/null +++ b/changelog.d/1495-traffic-paired-sides.md @@ -0,0 +1,4 @@ +- Street Traffic gives the two TomTom `one_side` lines of one two-way street + a two-way direction's half share of simulated vehicles each, so the street + is no longer drawn twice as dense as the same street from OpenStreetMap + (quanticsoul4772, #1495). From 9caa0947b69489cc6975248245152074f8e275f6 Mon Sep 17 00:00:00 2001 From: quanticsoul4772 <71096528+quanticsoul4772@users.noreply.github.com> Date: Sun, 1 Nov 2026 07:43:22 -0800 Subject: [PATCH 07/48] fix(traffic): keep a TomTom side from pairing with its own pieces A one_side line that doubles back within 5 m of itself, such as a turnaround drawn as one line, matched its own return leg and took a two-way half share with no partner. Flow index pieces now carry their source line, and a radiusFor of 0 leaves a piece out of nearest(), so pairing skips the line's own pieces. Co-Authored-By: Claude Opus 5.5 --- src/data/flowMatch.js | 6 ++++-- src/layers/traffic/roadModes.js | 3 ++- src/layers/traffic/roadModes.test.mjs | 8 ++++++++ 3 files changed, 14 insertions(+), 3 deletions(-) diff --git a/src/data/flowMatch.js b/src/data/flowMatch.js index c050f6917..cc4fe2280 100644 --- a/src/data/flowMatch.js +++ b/src/data/flowMatch.js @@ -139,6 +139,7 @@ export function createFlowSegmentIndex(flowSegments) { roadType, level, closure, + flow, }; const key = cellKey( Math.floor((seg.ax + seg.bx) / 2 / CELL_SIZE_M), @@ -158,7 +159,8 @@ export function createFlowSegmentIndex(flowSegments) { const radius2 = MATCH_RADIUS_M * MATCH_RADIUS_M; /** * Nearest direction-compatible segment within 35 m of a projected point. - * `radiusFor(segment)` may tighten the radius per segment (never widen it). + * `radiusFor(segment)` may tighten the radius per segment (never widen it); + * 0 leaves the segment out. */ function nearest(px, py, bearing, radiusFor = null) { const cx = Math.floor(px / CELL_SIZE_M); @@ -177,7 +179,7 @@ export function createFlowSegmentIndex(flowSegments) { candidate = true; // within radius, bearing not yet checked if (radiusFor) { const radius = Math.min(MATCH_RADIUS_M, radiusFor(seg)); - if (d2 > radius * radius) continue; + if (!(radius > 0) || d2 > radius * radius) continue; } const diff = bearingDiffDeg(seg.bearing, bearing); if ( diff --git a/src/layers/traffic/roadModes.js b/src/layers/traffic/roadModes.js index e9220b635..a6eecc5b5 100644 --- a/src/layers/traffic/roadModes.js +++ b/src/layers/traffic/roadModes.js @@ -126,7 +126,8 @@ function pairedSampleShare(segment, index) { xs[i - 1] + dx * t, ys[i - 1] + dy * t, reverse, - () => PAIRED_SIDE_RADIUS_M, + // A line that doubles back beside itself is not its own pair. + (piece) => (piece.flow === segment ? 0 : PAIRED_SIDE_RADIUS_M), ).best ) paired++; diff --git a/src/layers/traffic/roadModes.test.mjs b/src/layers/traffic/roadModes.test.mjs index 465d41496..406dc934b 100644 --- a/src/layers/traffic/roadModes.test.mjs +++ b/src/layers/traffic/roadModes.test.mjs @@ -163,6 +163,14 @@ test('a TomTom side paired along only part of its length keeps its share', () => assert.equal(weightOfFlow(partner(-97.7434)), 1); assert.equal(weightOfFlow(partner(-97.7428)), 0.5); }); +test('a TomTom side that doubles back beside itself is not its own pair', () => { + // East along `coordinates`, then back west about 1 m north. + const hairpin = { + ...flow, + coords: [...coordinates, [-97.742, 30.26701], [-97.744, 30.26701]], + }; + assert.equal(flowSegmentsToRoads([hairpin])[0].densityWeight ?? 1, 1); +}); test('bearing and distance reject nearby crossing and separate OSM roads', () => { const parallel = { ...main, From af461febae51c176770163c4ec104f8f981287e8 Mon Sep 17 00:00:00 2001 From: manjunath22466 <301209272+manjunath22466@users.noreply.github.com> Date: Mon, 2 Nov 2026 18:39:44 +0530 Subject: [PATCH 08/48] feat(voice): add typed AI chat beside voice Chat sits next to the mic in the command dock. Replies are silent and never request the microphone; the first message connects with the selected provider and runs the same map actions as voice through each provider's text API (OpenAI OPENAI_CHAT_MODEL, Gemini GEMINI_CHAT_MODEL via generateContent, which never starts Live). Only one mode runs at a time. - Chat panel rendered by the voice control presenter: conversation, recent and quick-command suggestions, Up/Down history, attachments (map-layer references or small UTF-8 text files, sent only with the message) - Chat and voice panels sit above the dock or dock into the right rail's column; minimized bars sit centred above the dock, and height animates on swaps - A centred panel steps down to its bar for the Location and Visual Presets trays without moving focus, rests above an open tray, and docks right while expanded over one; it returns to the centre once the trays close - Minimized voice previews the last sentence from each side; the empty Voice Preview suggests things to say - Settings opened from chat keep chat's frame and show the shared AI provider and quick commands; voice Settings show Back to voice - Closing chat ends a typed session; the voice panel's close turns voice off - The dock widens its voice control for the chat button and restyles its wings, which show icons only at 900px and below; assistant panels share the left panels' glass, and Cyber keeps its own surfaces - Server text-chat handlers with body, time and per-IP limits (GEV_RATELIMIT_*_CHAT_PER_MIN), documented in SECURITY.md; tests and all eleven locales Co-authored-by: Darren Dignam <22195071+darrendignam@users.noreply.github.com> --- .env.example | 11 + README.md | 2 + SECURITY.md | 3 +- docs/CURRENT-STATE.md | 30 + scripts/format-scope.json | 20 +- scripts/package-boundaries.json | 35 + server/providers/common/textChat.js | 136 ++ server/providers/common/textInstructions.js | 31 + server/providers/common/textTools.js | 54 + server/providers/gemini.js | 5 + server/providers/gemini/chat.js | 172 ++ server/providers/openai.js | 5 + server/providers/openai/chat.js | 139 ++ server/providers/openai/instructions.js | 12 +- src/geminiChatServer.test.mjs | 162 ++ src/geminiComposition.test.mjs | 3 +- src/geminiServer.test.mjs | 5 +- src/i18n/locales/ar.json | 64 +- src/i18n/locales/de.json | 64 +- src/i18n/locales/es.json | 64 +- src/i18n/locales/fr.json | 64 +- src/i18n/locales/it.json | 64 +- src/i18n/locales/ja.json | 64 +- src/i18n/locales/pt.json | 64 +- src/i18n/locales/ru.json | 64 +- src/i18n/locales/uk.json | 64 +- src/i18n/locales/zh-CN.json | 64 +- src/i18n/locales/zh-Hant.json | 64 +- src/i18n/same-as-english.json | 10 + src/textChatServer.test.mjs | 276 ++++ src/ui/stores/voice.js | 22 +- src/ui/styles/ai-chat.css | 1656 +++++++++++++++++++ src/ui/styles/command-dock-sliding.css | 3 +- src/ui/views/panels/ChatPanel.vue | 92 ++ src/ui/views/panels/PanelLayoutButtons.vue | 18 + src/ui/views/panels/VoiceControl.vue | 45 +- src/voice/chatAttachments.js | 137 ++ src/voice/chatControl.js | 387 +++++ src/voice/chatPreferences.js | 113 ++ src/voice/control.js | 81 +- src/voice/geminiSession.js | 32 +- src/voice/geminiTextChat.js | 5 + src/voice/geminiTextChat.test.mjs | 492 ++++++ src/voice/panel/host.test.mjs | 356 +++- src/voice/panel/manualPanel.js | 27 +- src/voice/panel/manualPanel.test.mjs | 9 +- src/voice/panelLayout.js | 384 +++++ src/voice/providerCommands.js | 8 +- src/voice/realtimeInput.js | 7 +- src/voice/realtimeSession.js | 41 +- src/voice/session.test.mjs | 45 + src/voice/sessionCommands.js | 91 +- src/voice/textChat.js | 383 +++++ src/voice/textProtocol.js | 153 ++ src/voice/voiceMiniPreview.js | 68 + src/voice/voiceMiniPreview.test.mjs | 92 ++ style.css | 1 + 57 files changed, 6500 insertions(+), 63 deletions(-) create mode 100644 server/providers/common/textChat.js create mode 100644 server/providers/common/textInstructions.js create mode 100644 server/providers/common/textTools.js create mode 100644 server/providers/gemini/chat.js create mode 100644 server/providers/openai/chat.js create mode 100644 src/geminiChatServer.test.mjs create mode 100644 src/textChatServer.test.mjs create mode 100644 src/ui/styles/ai-chat.css create mode 100644 src/ui/views/panels/ChatPanel.vue create mode 100644 src/ui/views/panels/PanelLayoutButtons.vue create mode 100644 src/voice/chatAttachments.js create mode 100644 src/voice/chatControl.js create mode 100644 src/voice/chatPreferences.js create mode 100644 src/voice/geminiTextChat.js create mode 100644 src/voice/geminiTextChat.test.mjs create mode 100644 src/voice/panelLayout.js create mode 100644 src/voice/textChat.js create mode 100644 src/voice/textProtocol.js create mode 100644 src/voice/voiceMiniPreview.js create mode 100644 src/voice/voiceMiniPreview.test.mjs diff --git a/.env.example b/.env.example index c9700c5be..08b0df8f2 100644 --- a/.env.example +++ b/.env.example @@ -61,6 +61,12 @@ CESIUM_ION_TOKEN= # ChatGPT/Codex OAuth session; the browser still receives only a short-lived # Realtime client secret. Do not prefix with VITE_. OPENAI_API_KEY= +# Typed chat uses Chat Completions, independently of the voice model. It always +# bills OPENAI_API_KEY; GEV_PREFER_CODEX_OAUTH below applies to voice only. +OPENAI_CHAT_MODEL=gpt-4.1-mini +# Typed chat requests (/api/openai/chat) per IP per minute, separate from the +# voice throttle; 0 disables it. Not a billing cap. +# GEV_RATELIMIT_OPENAI_CHAT_PER_MIN=60 # the default # Optional: forbid API-key voice sessions. Select ChatGPT OAuth in Provider # Settings; missing or expired Codex sign-in then fails closed. # GEV_PREFER_CODEX_OAUTH=true @@ -269,3 +275,8 @@ OVERPASS_UPSTREAMS= # frame-ancestors sources such as "https://example.com", or * for any page. # Unset, no page may frame it. Other pages of the app can never be framed. # GEV_EMBED_FRAME_ANCESTORS= + +# Gemini typed chat uses generateContent; voice continues to use Gemini Live. +GEMINI_CHAT_MODEL=gemini-3.5-flash-lite +# Typed chat requests (/api/gemini/chat) per IP per minute; 0 disables it. +# GEV_RATELIMIT_GEMINI_CHAT_PER_MIN=60 # the default diff --git a/README.md b/README.md index 3a4c5fc43..d308e9b15 100644 --- a/README.md +++ b/README.md @@ -327,6 +327,8 @@ The standalone app offers **OpenAI** and **Gemini** in the GEV MIC provider choo Changing provider ends the current conversation. If voice is on, the new provider starts with the same open-mic or push-to-talk mode; a held Space key is not carried over. Gemini uses the same map actions and captions, but has no OpenAI cost meter or MINI tier. When Gemini reports that its session has expired, start voice again for a new conversation. The provider chooser does not change the OpenAI-backed AI HUD summary. +Next to the mic, **CHAT** opens a typed chat with silent replies. It uses the selected provider and the same map actions, and never asks for the microphone. OpenAI chat needs an API key (`OPENAI_CHAT_MODEL`); Gemini chat uses a separate text model (`GEMINI_CHAT_MODEL`). Chat and voice panels sit above the dock or dock to the right. + > Voice can use an **OpenAI API key** or, on the same machine, an opt-in signed-in **ChatGPT/Codex OAuth** session. The API-key path remains the default. Selecting ChatGPT OAuth in Provider Settings reuses an existing local sign-in or starts the Codex browser sign-in flow and selects OAuth when it completes. The OpenAI key still drives the **AI HUD summary**: a terse, five-word intelligence-style readout of the current view that regenerates as you move. The OAuth option is experimental. OpenAI documents ChatGPT plan usage for the [Responses API](https://developers.openai.com/siwc/token-sharing-open-source/models-and-inference); a supported third-party Realtime contract for reusing Codex tokens has not been confirmed. A successful token mint does not establish billing or support terms. diff --git a/SECURITY.md b/SECURITY.md index 4423eead3..b32c5b60f 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -85,6 +85,7 @@ The data proxies under `server/providers/` are written so the browser cannot tur Allowing `maps.googleapis.com` in `script-src` also trusts its JSONP endpoints, which can bypass the script allowlist if an HTML-injection bug exists; this is an accepted limitation of loading the Maps JavaScript API, alongside the existing `'unsafe-eval'` allowance. - **Full Content-Security-Policy.** The dev and preview servers send a real CSP (`script-src 'self' 'unsafe-eval' blob:` plus the YouTube, Facebook and X scripts the Nepal event's embedded media loads, with a matching `frame-src` for their players and posts: no inline script and no other script origin; `blob:` covers the object URLs Cesium's bundled workers bootstrap through in the built app), alongside the existing `X-Frame-Options: DENY` and `frame-ancestors 'none'`. `'unsafe-eval'` stays because Knockout, bundled inside Cesium's widgets package, resolves the global object with `eval` at load time and the globe does not initialize without it; styles permit `'unsafe-inline'` and Google Fonts because Vite's dev client injects `