diff --git a/DATA_SOURCES.md b/DATA_SOURCES.md index b1439230e..73eaf8a1f 100644 --- a/DATA_SOURCES.md +++ b/DATA_SOURCES.md @@ -55,6 +55,7 @@ How to read this: | **Fintraffic / Digitraffic (weathercams)** | CCTV camera catalog + frames, Finland | [CC BY 4.0](https://www.digitraffic.fi/en/terms-of-service/) — attribution REQUIRED | "Fintraffic / digitraffic.fi, license CC BY 4.0" | | **DriveBC** (Government of British Columbia) | CCTV camera catalog + frames, British Columbia highways | [Open Government Licence – British Columbia](https://www2.gov.bc.ca/gov/content/data/open-data/open-government-licence-bc) — attribution REQUIRED | "DriveBC. Contains information licensed under the Open Government Licence – British Columbia" | | **Open Calgary** (`data.calgary.ca`) | CCTV camera catalog + frames, Calgary | [Open Government Licence – City of Calgary](https://data.calgary.ca/stories/s/Open-Calgary-Terms-of-Use/u45n-7awa) — attribution REQUIRED | "Contains information licensed under the Open Government Licence – City of Calgary" | +| **WSDOT** (`data.wsdot.wa.gov`) | CCTV camera catalog + frames, Washington State highways | [WSDOT travel information disclaimer](https://wsdot.wa.gov/about/policies/travel-information-disclaimer) — public "low volume" feed; catalog cached, frames on WSDOT's own image host only | "Washington State Department of Transportation (Travel Center)" (courtesy) | | **GBFS (Lyft / BCycle)** | Bikeshare availability | Per-feed (attribution-only) | Credit the operator (e.g. Austin BCycle) + its `license_url` | | **MBTA / MassDOT** (cdn.mbta.com) | Transit layer live vehicles and up to 15 minutes of recently observed positions, Greater Boston | MBTA / MassDOT open realtime vehicle data, used for the live view and up to fifteen minutes of recently observed positions; published for developer use and not against their use. | Attribution to MBTA / MassDOT (courtesy text) | | **CapMetro** (data.texas.gov) | Transit layer live vehicles, Austin TX | CapMetro open realtime vehicle data, used for the live view; published for developer use and not against their use. | Attribution to CapMetro (courtesy text) | @@ -89,6 +90,7 @@ The optional Local ADS-B layer and the broadcast-FM receiver use radio you recei - **Warendorf webcam.** The Stadt Warendorf Marktplatz webcam, registered from the curated `config/cctv_sources.warendorf.json` (the Kreis Warendorf registration-office cameras from the contributing PR are indoor waiting rooms and were left out); only the municipal image host is proxied, frames are fetched at request time and never stored or redistributed. Disable with `CCTV_WARENDORF_ENABLED=0`. - **Live Traffic NSW cameras.** The camera list is the keyless public `data.livetraffic.com/cameras/traffic-cam.json` feed (Transport for NSW open data, CC BY 4.0); frames are stills on `webcams.transport.nsw.gov.au`, and only that host is registered. That image host answers non-browser clients with an HTML placeholder, so the proxy identifies as a browser for that one host and as itself everywhere else. Frames are fetched at request time and never stored or redistributed. Disable with `CCTV_NSW_ENABLED=0`. - **Open Calgary traffic cameras.** Keyless. The camera list is the public Socrata dataset `k7p9-kppz` on `data.calgary.ca`; frames are stills on the city's own `trafficcam.calgary.ca` host, and only that host is registered. Most rows publish an `http://` frame URL and the host redirects those to HTTPS, so URLs are upgraded and pinned to that origin before registration. Frames are fetched at request time and never stored or redistributed. The dataset publishes no camera facing: the `quadrant` field and the quadrant suffix on each camera name are Calgary's address grid, not a bearing, so headings use the shared id-hash fallback at low confidence and are corrected with the in-app calibration gizmo. The licence requires the attribution statement above. Disable with `CCTV_CALGARY_ENABLED=0`. +- **WSDOT highway cameras.** Keyless. The camera list is the Esri JSON layer `data.wsdot.wa.gov/travelcenter/Cameras.json` — the same layer WSDOT's own Travel Center map consumes (~1,700 point features, EPSG:3857 geometry converted to WGS84 on load; the body carries Windows-1252 bytes and is decoded as such). Only frames on WSDOT's own `images.wsdot.wa.gov` host are registered — the catalog also lists ~70 partner cameras (Oregon DOT, City of Seattle, lodges, airports) whose own terms apply, and those are skipped rather than proxied. WSDOT's layer metadata marks the feed for "low volume" use, which the 15-minute catalog cache and per-request frame fetches respect; frames are never stored or redistributed. `CompassDirection` letters (N/E/S/W) give a high-confidence heading; the majority `B` ("both directions") rows carry no single facing and use the shared id-hash fallback at low confidence. By default the 250 cameras nearest Seattle and Spokane load; `CCTV_WSDOT_MAX_SOURCES` changes the cap and `CCTV_WSDOT_ENABLED=0` turns the pack off. A keyed WSDOT Traveler Information API serves the same population for higher-volume use and is deliberately not required here. - **Camera frame content.** Public camera frames can contain people, vehicles and license plates, and the camera layer does nothing with them beyond putting them on the map. A successful upstream response is relayed as the provider served it: nothing in the camera pipeline blurs, redacts, enhances or restores it, and nothing there runs face, plate or object detection or otherwise recognises or classifies what is in the picture. (The ALPR layer described below maps where plate-reader cameras are, from locations contributors tagged in OpenStreetMap; it reads nothing from any camera frame.) When an upstream has no frame for a camera, what you see instead is a Google Street View still or the app's own placeholder card; the panel's source badge names which of them you are looking at, and a refresh that fails after a frame has arrived leaves the last good picture on screen rather than replacing it. The browser resamples each frame for display: onto a fixed 1920x1080 surface for the camera's projection plane in the scene, and onto a 192x108 canvas for the small ambient card. That scaling is the only change made to the picture, and the only pixels the camera layer reads back are a 64x36 downscale hashed to tell whether the picture has changed. Live streams are piped through; the still path reads the image into memory to pass it on — TxDOT wraps its JPEG in a JSON envelope, which is decoded there — and no frame is written to disk (the disk cache, `.gev-cache/`, holds feed JSON for other layers and never camera imagery). Outside that pipeline, the voice assistant is the one feature that sends imagery anywhere: when it answers a question about what is in view it may attach a downscaled screenshot of the rendered scene — including any camera plane on screen — to the request it sends to the voice model. Frames are fetched only from server-registered camera URLs — a client cannot hand the proxy a URL (see [SECURITY.md](SECURITY.md)). What the app does stays inside the line README draws: no named-person search, no face recognition, no tracking individuals. - **Transit (GTFS-Realtime).** The proxy requests only registered operator feeds, validates redirects, bounds responses, and uses conditional requests, short snapshot caching and failure backoff. All seven registered regions are live-enabled; each can be disabled independently. Vehicles use delayed playback between reports, with report age and waiting states shown separately. Session-local selected trails are available for every feed. MBTA additionally retains up to 15 minutes of recently observed positions in the running proxy, so available history can survive a browser reload; server restart clears it. History reads never fetch upstream, and other feeds have no proxy retention. Nothing is archived to disk or collected in the background. Operator text credits are included as a courtesy and do not imply endorsement. Heights are aligned to work with Google 3D tiles; subway markers are shown at street level because these feeds do not supply depth. - **Radio Browser.** `/api/radio/stations` discovers official API mirrors, makes bounded and coalesced healthy/geolocated HTTPS-station queries, caches the normalized public-domain directory for 45 minutes, and may serve the last good catalog for up to seven days during an outage. Refreshes must meet minimum accepted-query and station coverage before replacing a warm catalog; schema-valid responses whose rows all fail the product's health policy do not count as successful queries. A usable partial cold catalog is explicitly `DEGRADED`, and malformed or empty successful payloads are rejected atomically. Every directory and click-count request rejects redirects, validates all resolved addresses as globally routable (including reserved/documentation IPv4 and special/non-global IPv6 exclusions), and pins the TLS connection to a validated address. Only MP3/AAC non-HLS directory rows with public HTTPS stream targets are returned; favicons are intentionally omitted. Pressing play connects one browser audio element directly to the selected broadcaster and calls the directory's click counter through known-ID-only `POST /api/radio/click/:uuid`. GEV never proxies, caches, records, bundles, or redistributes audio. Radio Browser supplies station-level tags, not dependable current-song or upcoming-program metadata, so Radio filtering never claims either. Direct playback exposes the listener's IP address to the broadcaster, whose own stream terms apply. diff --git a/server/providers/cctv/catalog.js b/server/providers/cctv/catalog.js index ae0ab9afa..3241cacd5 100644 --- a/server/providers/cctv/catalog.js +++ b/server/providers/cctv/catalog.js @@ -18,6 +18,7 @@ import { loadNswSourcesFromOpenData, loadCalgarySourcesFromOpenData, loadDelDOTSourcesFromOpenData, + loadWsdotSourcesFromOpenData, } from './sources.js'; /** Env kill switch: unset or anything but "0" means enabled. */ @@ -92,6 +93,11 @@ const LIVE_PACKS = [ enabled: () => envEnabled('CCTV_DELDOT_ENABLED'), load: loadDelDOTSourcesFromOpenData, }, + { + name: 'wsdot', + enabled: () => envEnabled('CCTV_WSDOT_ENABLED'), + load: loadWsdotSourcesFromOpenData, + }, ]; /** * Load CCTV sources from a local JSON file (CCTV_SOURCES_FILE env or default). diff --git a/server/providers/cctv/constants.js b/server/providers/cctv/constants.js index da0f2fcda..3ab8ce9cb 100644 --- a/server/providers/cctv/constants.js +++ b/server/providers/cctv/constants.js @@ -244,6 +244,24 @@ export const DELDOT_ANCHORS = [ { lat: 39.1582, lon: -75.5244 }, // Dover (Kent) { lat: 38.6906, lon: -75.3877 }, // Georgetown (Sussex) ]; + +/** WSDOT Travel Center cameras (Washington State): keyless Esri JSON on the + * official open-data host — ~1,700 point features in EPSG:3857 (Web Mercator), + * ETag/Last-Modified served. WSDOT's metadata marks the feed "low volume" use, + * which the 15-minute catalog cache respects. */ +export const WSDOT_CAMERAS_URL = + 'https://data.wsdot.wa.gov/travelcenter/Cameras.json'; +/** Only frames on WSDOT's own image host are registered; the catalog also + * lists ~70 partner cameras (Oregon DOT, City of Seattle, lodges, airports) + * whose own terms apply. */ +export const WSDOT_IMAGE_ORIGIN = 'https://images.wsdot.wa.gov/'; +export const DEFAULT_WSDOT_MAX_SOURCES = 250; +/** Prioritization anchors: downtown Seattle and Spokane. */ +export const WSDOT_ANCHORS = [ + { lat: 47.6062, lon: -122.3321 }, // Seattle + { lat: 47.6588, lon: -117.426 }, // Spokane +]; + /** Camera CATALOGS change rarely; 15 min keeps multi-megabyte upstream list refetches (Austin rows.json + 4 Caltrans districts + TfL + Ontario 511) infrequent. Frames are fetched per-request and are unaffected. */ export const CCTV_SOURCE_CACHE_MS = 15 * 60 * 1000; /** Per-provider catalog-fetch timeout. Bounds the worst-case refresh so one diff --git a/server/providers/cctv/sources.js b/server/providers/cctv/sources.js index ad8a883da..1124b187f 100644 --- a/server/providers/cctv/sources.js +++ b/server/providers/cctv/sources.js @@ -59,6 +59,10 @@ import { DELDOT_CCTV_URL, DEFAULT_DELDOT_MAX_SOURCES, DELDOT_ANCHORS, + WSDOT_CAMERAS_URL, + WSDOT_IMAGE_ORIGIN, + DEFAULT_WSDOT_MAX_SOURCES, + WSDOT_ANCHORS, } from './constants.js'; import { toFiniteNumber, @@ -1701,3 +1705,133 @@ export async function loadDelDOTSourcesFromOpenData() { return []; } } + +/** WSDOT compass letters as headings in degrees. `B` (both directions), `O` + * (other) and null carry no single facing and fall through to the id hash. */ +const WSDOT_COMPASS_HEADINGS = Object.freeze({ + N: 0, + E: 90, + S: 180, + W: 270, +}); + +/** + * Converts an EPSG:3857 (Web Mercator) coordinate pair to WGS84 degrees. + * The WSDOT camera layer publishes point geometry in wkid 102100. + * @param {number} x - Web Mercator metres east. + * @param {number} y - Web Mercator metres north. + * @returns {{lat: number, lon: number}} WGS84 degrees. + */ +export function webMercatorToWgs84(x, y) { + const R = 6378137; + const lon = (x / R) * (180 / Math.PI); + const lat = (2 * Math.atan(Math.exp(y / R)) - Math.PI / 2) * (180 / Math.PI); + return { lat, lon }; +} + +/** Rough Washington State bounding box (with the Columbia River border cams). */ +function isLikelyWashingtonCoordinate(lat, lon) { + return ( + Number.isFinite(lat) && + Number.isFinite(lon) && + lat >= 45.3 && + lat <= 49.3 && + lon >= -125 && + lon <= -116.5 + ); +} + +/** + * Fetch WSDOT highway cameras (Washington State). Keyless: one Esri JSON + * layer on WSDOT's official open-data host (the same layer the WSDOT Travel + * Center map consumes; a keyed Traveler Information API serves the identical + * population for higher-volume use). Only cameras with a positive integer id, + * finite in-state coordinates, and a frame on WSDOT's own image host are + * kept — the catalog's ~70 partner cameras (Oregon DOT, City of Seattle, + * lodges, airports) are skipped rather than proxied under someone else's + * terms. `CompassDirection` letters give a high-confidence heading; `B` + * (both directions, the majority) and `O`/null fall back to the id hash. + * The feed carries Windows-1252 bytes (en-dashes in titles) with no charset + * declaration, so the body is decoded explicitly before parsing. + * + * @returns {Promise>} Normalized camera source objects. + */ +export async function loadWsdotSourcesFromOpenData() { + try { + const resp = await fetch(WSDOT_CAMERAS_URL, { + headers: { Accept: 'application/json' }, + signal: AbortSignal.timeout(CCTV_SOURCE_FETCH_TIMEOUT_MS), + }); + if (!resp.ok) { + console.warn('[CCTV] WSDOT camera download failed:', resp.status); + return []; + } + const payload = JSON.parse( + new TextDecoder('windows-1252').decode(await resp.arrayBuffer()), + ); + const rows = Array.isArray(payload?.features) ? payload.features : []; + + const cameras = []; + for (const row of rows) { + const attrs = row?.attributes; + const rawId = attrs?.CameraID; + if (!Number.isSafeInteger(rawId) || rawId <= 0) continue; + const imageUrl = String(attrs.ImageURL || '').trim(); + if (!imageUrl.startsWith(WSDOT_IMAGE_ORIGIN)) continue; + const x = row?.geometry?.x; + const y = row?.geometry?.y; + if (!Number.isFinite(x) || !Number.isFinite(y)) continue; + const { lat, lon } = webMercatorToWgs84(x, y); + if (!isLikelyWashingtonCoordinate(lat, lon)) continue; + + const cameraId = `wsdot-${rawId}`; + const heading = + WSDOT_COMPASS_HEADINGS[ + String(attrs.CompassDirection || '') + .trim() + .toUpperCase() + ]; + const hasHeading = Number.isFinite(heading); + cameras.push({ + id: cameraId, + name: String(attrs.CameraTitle || '').trim() || `WSDOT camera ${rawId}`, + city: 'Washington', + cityId: 'washington-state', + provider: 'WSDOT', + lat, + lon, + headingDeg: hasHeading ? heading : fallbackHeadingFromId(cameraId), + headingConfidence: hasHeading ? 'high' : 'low', + // Same two pose personalities as the other packs: raw priors that the + // client's ground snap and manual calibration refine. + pitchDeg: hasHeading ? -24 : -18, + fovDeg: hasHeading ? 56 : 44, + rangeM: hasHeading ? 210 : 145, + mountHeightM: hasHeading ? 10 : 8, + feedType: 'image', + url: imageUrl, + snapshotUrl: imageUrl, + sourceKind: 'wsdot-open-data', + license: 'WSDOT Travel Center camera data (public low-volume feed)', + }); + } + + const maxRaw = Number( + process.env.CCTV_WSDOT_MAX_SOURCES || DEFAULT_WSDOT_MAX_SOURCES, + ); + const maxCount = Number.isFinite(maxRaw) + ? Math.max(8, Math.min(1200, Math.floor(maxRaw))) + : DEFAULT_WSDOT_MAX_SOURCES; + const prioritized = prioritizeSources(cameras, maxCount, WSDOT_ANCHORS); + console.log( + `[CCTV] Loaded WSDOT camera sources: ${cameras.length} published (using nearest ${prioritized.length})`, + ); + return prioritized; + } catch (error) { + console.warn( + '[CCTV] WSDOT camera download error:', + error?.message || error, + ); + return []; + } +} diff --git a/src/data/cctvWsdotSource.test.mjs b/src/data/cctvWsdotSource.test.mjs new file mode 100644 index 000000000..0459f3c72 --- /dev/null +++ b/src/data/cctvWsdotSource.test.mjs @@ -0,0 +1,167 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { + DEFAULT_WSDOT_MAX_SOURCES, + WSDOT_CAMERAS_URL, +} from '../../server/providers/cctv/constants.js'; +import { + loadWsdotSourcesFromOpenData, + webMercatorToWgs84, +} from '../../server/providers/cctv/sources.js'; + +/** Set (or, for `undefined`, delete) environment variables for one test. */ +function withEnv(t, env) { + for (const [name, value] of Object.entries(env)) { + const previous = process.env[name]; + t.after(() => { + if (previous === undefined) delete process.env[name]; + else process.env[name] = previous; + }); + if (value === undefined) delete process.env[name]; + else process.env[name] = value; + } +} + +/** Silence the loaders' progress and failure logging. */ +function quiet(t) { + t.mock.method(console, 'log', () => {}); + t.mock.method(console, 'warn', () => {}); +} + +/** Web Mercator metres for a WGS84 coordinate (inverse of the loader's math). */ +function mercator(lat, lon) { + const R = 6378137; + return { + x: (lon * Math.PI * R) / 180, + y: R * Math.log(Math.tan(Math.PI / 4 + (lat * Math.PI) / 360)), + }; +} + +/** A `Cameras.json` point feature carrying the fields the loader reads. */ +function wsdotFeature(id, lat, lon, overrides = {}) { + const { attributes = {}, geometry } = overrides; + return { + attributes: { + CameraID: id, + CameraTitle: `Camera ${id}`, + CompassDirection: 'N', + ImageURL: `https://images.wsdot.wa.gov/nw/${id}.jpg`, + ...attributes, + }, + geometry: geometry === undefined ? mercator(lat, lon) : geometry, + }; +} + +/** Serialize a payload for the mocked fetch as raw Windows-1252 bytes. */ +function cp1252Response(payload) { + const text = JSON.stringify(payload); + const bytes = new Uint8Array(text.length); + for (let i = 0; i < text.length; i += 1) { + const code = text.charCodeAt(i); + // U+2013 EN DASH is 0x96 in Windows-1252 — the byte the live feed ships. + bytes[i] = code === 0x2013 ? 0x96 : code; + } + return new Response(bytes, { + headers: { 'Content-Type': 'application/json' }, + }); +} + +test('webMercatorToWgs84 recovers WGS84 degrees from layer geometry', () => { + // Real feature from the live layer: I-5 at the Interstate Bridge. + const { lat, lon } = webMercatorToWgs84(-13656007.102, 5719738.545); + assert.ok(Math.abs(lat - 45.62) < 0.05, `lat ${lat}`); + assert.ok(Math.abs(lon - -122.675) < 0.05, `lon ${lon}`); +}); + +test('WSDOT loader keeps official-host cameras and maps compass directions', async (t) => { + quiet(t); + withEnv(t, { CCTV_WSDOT_MAX_SOURCES: undefined }); + const requested = []; + t.mock.method(globalThis, 'fetch', async (url) => { + requested.push(String(url)); + return cp1252Response({ + features: [ + wsdotFeature(1001, 47.6097, -122.3331, { + attributes: { + // The 0x2013 below is serialized as the feed's 0x96 byte. + CameraTitle: 'I-5 – Seattle, looking south', + CompassDirection: 'S', + }, + }), + wsdotFeature(9059, 47.66, -117.43, { + attributes: { CompassDirection: 'B' }, + }), + // Partner-hosted frame (Oregon DOT): skipped, never proxied. + wsdotFeature(1002, 45.62, -122.675, { + attributes: { + ImageURL: 'https://www.tripcheck.com/RoadCams/cams/bridge.jpg', + }, + }), + // Bad rows: non-integer id, missing geometry, out-of-state point. + wsdotFeature('1003', 47.6, -122.3), + wsdotFeature(1004, 47.6, -122.3, { geometry: null }), + wsdotFeature(1005, 34.05, -118.24), + ], + }); + }); + + const cameras = await loadWsdotSourcesFromOpenData(); + + assert.deepEqual(requested, [WSDOT_CAMERAS_URL]); + // Exactly the two valid official-host cameras survive; the partner-hosted + // frame and the malformed rows are dropped. + assert.deepEqual(cameras.map((camera) => camera.id).sort(), [ + 'wsdot-1001', + 'wsdot-9059', + ]); + + const seattle = cameras.find((camera) => camera.id === 'wsdot-1001'); + assert.equal(seattle.name, 'I-5 – Seattle, looking south'); + assert.equal(seattle.provider, 'WSDOT'); + assert.equal(seattle.url, 'https://images.wsdot.wa.gov/nw/1001.jpg'); + assert.equal(seattle.snapshotUrl, seattle.url); + assert.equal(seattle.feedType, 'image'); + assert.equal(seattle.headingDeg, 180); + assert.equal(seattle.headingConfidence, 'high'); + assert.equal(seattle.fovDeg, 56); + assert.ok(Math.abs(seattle.lat - 47.6097) < 0.001); + assert.ok(Math.abs(seattle.lon - -122.3331) < 0.001); + + // "Both directions" carries no single facing: id-hash fallback at low + // confidence, with the low-confidence pose personality. + const spokane = cameras.find((camera) => camera.id === 'wsdot-9059'); + assert.equal(spokane.headingConfidence, 'low'); + assert.equal(spokane.fovDeg, 44); + assert.ok(spokane.headingDeg >= 0 && spokane.headingDeg < 360); +}); + +test('WSDOT loader honors the max-sources cap', async (t) => { + quiet(t); + withEnv(t, { CCTV_WSDOT_MAX_SOURCES: '8' }); + const features = []; + for (let i = 0; i < 24; i += 1) { + features.push(wsdotFeature(2000 + i, 47.0 + i * 0.05, -122.9 + i * 0.05)); + } + t.mock.method(globalThis, 'fetch', async () => cp1252Response({ features })); + + const cameras = await loadWsdotSourcesFromOpenData(); + assert.equal(cameras.length, 8); +}); + +test('WSDOT loader defaults its cap and survives upstream failure', async (t) => { + quiet(t); + withEnv(t, { CCTV_WSDOT_MAX_SOURCES: undefined }); + assert.equal(DEFAULT_WSDOT_MAX_SOURCES, 250); + + t.mock.method( + globalThis, + 'fetch', + async () => new Response('', { status: 503 }), + ); + assert.deepEqual(await loadWsdotSourcesFromOpenData(), []); + + t.mock.method(globalThis, 'fetch', async () => { + throw new Error('network down'); + }); + assert.deepEqual(await loadWsdotSourcesFromOpenData(), []); +}); diff --git a/src/data/dataCredits.js b/src/data/dataCredits.js index ab9213c26..ea7adeb78 100644 --- a/src/data/dataCredits.js +++ b/src/data/dataCredits.js @@ -293,6 +293,16 @@ export const DATA_CREDITS = [ 'Open Government Licence – British Columbia. ' + 'Some cameras are supplied by partners (TransLink, the City of Vancouver, the City of Surrey, Parks Canada and others); each names its provider in the CCTV panel.', }, + { + key: 'wsdot-cctv', + html: + 'CCTV cameras & frames (Washington State): ' + + 'Washington State Department of Transportation ' + + '(Travel Center), ' + + 'used per the ' + + 'WSDOT travel information disclaimer ' + + '(low-volume public feed; catalog cached, frames proxied on demand)', + }, { key: 'tallinn-cctv', html: