From 8cd0cc82f55dcba3b70e36bbcb853597cbbe497d Mon Sep 17 00:00:00 2001 From: Sameh Khamis Date: Tue, 3 Nov 2026 15:02:30 -0800 Subject: [PATCH] fix(cctv): read TranStar dir as the camera's side of the road, at low confidence TranStar's `dir` field is the side of the freeway a camera sits on, not the way it faces. TranStar's own map script (translateCameraDirection) shows it as "Traffic closest to the camera is traveling": South is eastbound, North westbound, West southbound and anything else northbound. The live catalog agrees: I-10 cameras are North/South and I-45 cameras East/West. Treating `dir` as a high-confidence facing pointed most Houston cameras across or away from the road they watch, and dropped the "estimated" marking. Map it to the travel direction instead (S 90, N 270, W 180, E 0) as a low-confidence prior with the low-confidence pose, and keep the id-hash fallback for None or unknown values. --- DATA_SOURCES.md | 2 +- changelog.d/1125-houston-cctv-pack.md | 2 ++ docs/CURRENT-STATE.md | 9 ++++-- server/providers/cctv/sources.js | 43 ++++++++++++++++++++------- src/data/cctvHouston.test.mjs | 33 +++++++++++++------- 5 files changed, 66 insertions(+), 23 deletions(-) diff --git a/DATA_SOURCES.md b/DATA_SOURCES.md index b9f180a1f..3f26bc945 100644 --- a/DATA_SOURCES.md +++ b/DATA_SOURCES.md @@ -182,7 +182,7 @@ The optional Local ADS-B layer and the broadcast-FM receiver use radio you recei - **Kartverket ground heights for Statens vegvesen cameras.** The camera feed's points are 2D, so the loader asks Kartverket's keyless point-height API (`ws.geonorge.no/hoydedata/v1/punkt`, CC BY 4.0) for the terrain height at each camera. Requests carry 20 points: the API accepts 50, but above ~20 it answers the extra points from contour lines (`hoydekurver`, metres to ~20 m off) or not at all. Eight requests run at a time, about 36 for the whole country: a cold lookup took 6–13 s in testing (single requests 0.6–9.5 s). Answers are matched by the coordinates the API echoes and recorded only for points that request asked about. Over water the API returns the sea-floor depth (`dybdekurver` or `Havflate`), which is replaced by 0 m, the water surface; for the ~14 bridge cameras that is below the deck, but much closer than the sea floor or the flat prior. Heights are cached in memory by position for the life of the server, and points with no height (a few ferry quays and bridges, or a point a 200 answer leaves out) for 6 h, so the 15-minute catalog refresh only asks about new or moved cameras. The first failing request (a 408, 429 or 5xx, a network error, a request over 10 s, or an unreadable or oversized body) stops the lookup, with no retries, and no new lookup starts until the backoff ends (Retry-After when sent, else 30 min, longer than the refresh; bounded to 1 min–1 h). Any other 4xx marks that request's points as having no height and the lookup carries on. The whole lookup has a 30 s deadline; a lookup cut short by it resumes on the next refresh without a backoff. The lane waits at most 12 s, counted from the start of the loader, and only when more than 10% of the cameras have no answer yet; a few new cameras or expired misses are looked up in the background. A slower lookup keeps filling the cache, and the cameras it had not reached keep the flat prior until the next 15-minute refresh (about 190 of 865 on one slow first run). Only coordinates are sent, with the same User-Agent as the other CCTV requests. `CCTV_VEGVESEN_HEIGHTS=0` turns the lookup off. The Kartverket credit, with a note that heights over water are set to 0 m and heights are rounded to 0.1 m, is shown beside the Statens vegvesen entry in the Data attribution popover. - **NYC DOT Traffic Management Center cameras.** Off by default. NYC DOT's [developer data feeds page](https://www.nyc.gov/html/dot/html/about/datafeeds.shtml) asks developers who want the camera feed to contact `tmcdot@dot.nyc.gov` and sign a data-sharing agreement. This project does not hold one that covers downstream installations, so the pack loads only when an operator who holds their own agreement sets `CCTV_NYC_ENABLED=1`; with the flag unset, empty or any other value, the server makes no request to NYC DOT. The endpoint needs no key. The camera list is the public `webcams.nyctmc.org/api/cameras` catalog (~980 cameras across the five boroughs); frames are stills at `webcams.nyctmc.org/api/cameras//image`, synthesized from a validated UUID rather than taken from the payload, and only that host is registered. Cameras the feed reports as offline are dropped. The catalog publishes no camera facing, so headings use the shared id-hash fallback at low confidence and are corrected with the in-app calibration gizmo. Once enabled, the 250 cameras nearest the borough cores load; `CCTV_NYC_MAX_SOURCES` (8–400) changes the cap. Frames are fetched at request time and never stored or redistributed. NYC DOT asks every application that uses its data to carry its disclaimer; it is registered verbatim in the Data attribution popover. - **City of Toronto RESCU cameras.** Keyless. The camera list is the City's public `tmcearthcameras.json` on `opendata.toronto.ca` (dataset "Traffic Cameras", Transportation Services), served as JSONP and unwrapped with an anchored pattern. This is the municipal network the Ontario 511 pack does not carry: downtown arterials plus the Gardiner, DVP, Allen and Lake Shore. The list publishes no frame URL, so frame URLs are BUILT from the documented `CameraImages/loc####.jpg` convention and a strictly-digits camera number, pinning the frame proxy to the portal origin by construction; the list fetch refuses redirects and bounds the body it buffers. Frames are fetched at request time and never stored or redistributed. The dataset publishes no camera facing: the `D1`-`D4` fields are the directions of the *comparison* images (static reference stills captured in 2016 under a different path), not a bearing for the live pan-tilt-zoom frame, so headings use the shared id-hash fallback at low confidence and are corrected with the in-app calibration gizmo. A camera that is temporarily out of service still answers 200 with a byte-identical "Image Unavailable" placeholder rather than an error (about 6% of the network at any moment), so such a camera reads as healthy; the frame it shows is the agency's, not a synthesized one. The licence requires the attribution statement above. Disable with `CCTV_TORONTO_ENABLED=0`. -- **Houston TranStar cameras.** Keyless. The camera list is the public `traffic.houstontranstar.org/data/layers/cctvSnapshots_json.js` catalog (~1,000 cameras across Greater Houston); frames are stills at `www.houstontranstar.org/snapshots/cctv/.jpg`, synthesized from a validated numeric id rather than taken from the payload, and only that host is registered. Cameras whose catalog id is `0` (a different, non-snapshot class) and cameras outside Greater Houston are dropped. Dedicated `dir` values (North/South/East/West) become high-confidence headings; `None` uses the shared id-hash fallback at low confidence. TranStar's disclaimer is not a licence to ship the stills on by default (its FAQ grants permission-free use only for the RSS feeds), so the pack stays off unless `CCTV_HOUSTON_ENABLED=1`. Opting in loads the 250 cameras nearest downtown, Uptown, the Medical Center, IAH and Galveston; `CCTV_HOUSTON_MAX_SOURCES` (8–400) changes the cap. Snapshots update about every 3 minutes, which is also the ambient still cadence. Frames are fetched at request time and never stored or redistributed. +- **Houston TranStar cameras.** Keyless. The camera list is the public `traffic.houstontranstar.org/data/layers/cctvSnapshots_json.js` catalog (~1,000 cameras across Greater Houston); frames are stills at `www.houstontranstar.org/snapshots/cctv/.jpg`, synthesized from a validated numeric id rather than taken from the payload, and only that host is registered. Cameras whose catalog id is `0` (a different, non-snapshot class) and cameras outside Greater Houston are dropped. TranStar's `dir` field is the side of the freeway the camera sits on, not the way it faces: TranStar's own map shows it as "Traffic closest to the camera is traveling …" (South → eastbound, North → westbound, West → southbound, East → northbound). That travel direction (90°, 270°, 180°, 0°) seeds the heading as a low-confidence prior with the low-confidence pose; `None` or an unknown value uses the shared id-hash fallback at low confidence. Headings are corrected with the in-app calibration gizmo. TranStar's disclaimer is not a licence to ship the stills on by default (its FAQ grants permission-free use only for the RSS feeds), so the pack stays off unless `CCTV_HOUSTON_ENABLED=1`. Opting in loads the 250 cameras nearest downtown, Uptown, the Medical Center, IAH and Galveston; `CCTV_HOUSTON_MAX_SOURCES` (8–400) changes the cap. Snapshots update about every 3 minutes, which is also the ambient still cadence. Frames are fetched at request time and never stored or redistributed. - **Autostrada del Brennero (A22) webcams.** Keyless. The camera list is one request to the Open Data Hub tourism API (`tourism.api.opendatahub.com/v1/WebcamInfo?source=a22`), run by NOI Techpark in South Tyrol, which republishes Autobrennero's own list of 13 motorway webcams with coordinates and kilometre markers. Frames are the operator's stills on `www.autobrennero.it/WebCamImg/km.jpg`, refreshed about once a minute, and only the exact image for each camera's kilometre is registered. The feed names no bearing, so headings use the id-hash fallback at low confidence. The road climbs from about 20 m in the Po valley to about 1,370 m at the Brenner Pass, so each camera's ground elevation comes from a per-kilometre table sampled from the Copernicus GLO-90 DEM (via Open-Meteo's elevation API); a camera not in the table uses a 200 m prior. The Open Data Hub tags every record CC0 (`ClosedData: false`, Autobrennero as licence holder), but that tag is the aggregator's metadata and Autobrennero publishes no terms of its own for the images, so they are credited to the operator as a courtesy and are not redistributed. Frames are fetched at request time and never stored. Disable with `CCTV_A22_ENABLED=0`. - **Camera frame content.** Public camera frames can contain people, vehicles and license plates, and the camera layer does nothing with them beyond putting them on the map. A successful upstream response is relayed as the provider served it: nothing in the camera pipeline blurs, redacts, enhances or restores it, and nothing there runs face, plate or object detection or otherwise recognises or classifies what is in the picture. (The ALPR layer described below maps where plate-reader cameras are, from locations contributors tagged in OpenStreetMap; it reads nothing from any camera frame.) When an upstream has no frame for a camera, what you see instead is a Google Street View still or the app's own placeholder card; the panel's source badge names which of them you are looking at, and a refresh that fails after a frame has arrived leaves the last good picture on screen rather than replacing it. The browser resamples each frame for display: onto a fixed 1920x1080 surface for the camera's projection plane in the scene, and onto a 192x108 canvas for the small ambient card. That scaling is the only change made to the picture, and the only pixels the camera layer reads back are a 64x36 downscale hashed to tell whether the picture has changed. Live streams are piped through; the still path reads the image into memory to pass it on — TxDOT wraps its JPEG in a JSON envelope, which is decoded there — and no frame is written to disk (the disk cache, `.gev-cache/`, holds feed JSON for other layers and never camera imagery). Outside that pipeline, the voice assistant is the one feature that sends imagery anywhere: when it answers a question about what is in view it may attach a downscaled screenshot of the rendered scene — including any camera plane on screen — to the request it sends to the voice model. Frames are fetched only from server-registered camera URLs — a client cannot hand the proxy a URL (see [SECURITY.md](SECURITY.md)). What the app does stays inside the line README draws: no named-person search, no face recognition, no tracking individuals. - **Transit (GTFS-Realtime).** The proxy requests only registered operator feeds, validates redirects, bounds responses, and uses conditional requests, short snapshot caching and failure backoff. All sixteen registered regions are live-enabled; each can be disabled independently. Vehicles use delayed playback between reports, with report age and waiting states shown separately. Session-local selected trails are available for every feed. MBTA additionally retains up to 15 minutes of recently observed positions in the running proxy, so available history can survive a browser reload; server restart clears it. History reads never fetch upstream, and other feeds have no proxy retention. Nothing is archived to disk or collected in the background. Operator text credits are included as a courtesy and do not imply endorsement. Heights are aligned to work with Google 3D tiles; subway markers are shown at street level because these feeds do not supply depth. The TTC feed is BusTime, the surface system: it carries buses and streetcars and no subway at all. About four in ten of its vehicles report no route at all (unassigned or deadheading), and those are shown with the feed default rather than a route-established mode. It sends no ETag or Last-Modified, so its conditional requests fall through to the ordinary snapshot cache. Three Golden Horseshoe neighbours run alongside it — MiWay, Hamilton HSR and Durham Region Transit — each bus-only and each polled only near its own region. Durham Region Transit publishes no bearing on any vehicle, so those sprites take their heading from observed movement like any other feed that omits it. Guelph, Kingston and Barrie extend the same pattern eastward and north. Hamilton, Guelph, Burlington and YRT publish route ids that are not the number on the vehicle, so those entries carry a route-name table taken from each operator’s own published GTFS; any id missing from a table renders unchanged rather than blank. A label is never invented: YRT runs a route `9101` that appears nowhere in its published routes, so it stays as the feed sends it. Burlington’s is the one that cannot be shortcut: `351` is route 1 while `3510` is route 10, so the table is a real join rather than a trimmed prefix. GO Transit / Metrolinx is deliberately absent: its REALTIME feed requires an API key issued by manual review, which the keyless admission rule excludes (its static schedule download is keyless, but carries no vehicle positions). Barrie and YRT/Viva each invite acceptance of their licence through a form, and each licence also states that using the data is itself acceptance; this deployment relies on that clause, which is recorded in both entries. Grand River Transit in Waterloo is absent for a different reason: it answers, but its feed header was months stale and it publishes no per-vehicle timestamps, so it would render vehicles that look live and are not. Ottawa’s OC Transpo falls to the same keyless admission rule as GO Transit — its realtime feed requires an API key. London Transit and Thunder Bay Transit publish vehicle positions over plain HTTP with no HTTPS endpoint available. Every other Ontario system — Brampton, Oakville, Niagara, Milton, Windsor, Greater Sudbury, UP Express and a dozen smaller operators — publishes static GTFS schedules only, with no realtime vehicle positions to register. Between the nine included and these exclusions, every Ontario realtime vehicle-position feed catalogued in the Mobility Database is accounted for. diff --git a/changelog.d/1125-houston-cctv-pack.md b/changelog.d/1125-houston-cctv-pack.md index dc573993f..499fab516 100644 --- a/changelog.d/1125-houston-cctv-pack.md +++ b/changelog.d/1125-houston-cctv-pack.md @@ -4,6 +4,8 @@ stills synthesized from each numeric camera id on `www.houstontranstar.org`, so a catalog field cannot steer the proxy off-host. Cameras without a snapshot id, and cameras outside Greater Houston, are dropped. TranStar's + `dir` field (the side of the freeway the camera sits on) seeds a + low-confidence heading along the nearest lanes' travel direction. TranStar's disclaimer prohibits redistribution and external framing without TxDOT's written permission, so the pack is off unless `CCTV_HOUSTON_ENABLED=1`. Opting in loads the 250 cameras nearest downtown, Uptown, the Medical diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 253db42d5..7a31997dc 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1448,8 +1448,13 @@ alongside the other live packs. The server reads the keyless camera list at keeps cameras with a numeric snapshot id inside the metro bounding box. Frame URLs are built from each camera id under `https://www.houstontranstar.org/snapshots/cctv/.jpg`; they are never taken -from the payload. Dedicated `dir` values become high-confidence headings; -`None` uses the shared id-hash fallback at low confidence. +from the payload. TranStar's `dir` is the side of the freeway the camera sits +on, not the way it faces; TranStar's own map reads it as the direction of the +nearest traffic (South → eastbound, North → westbound, West → southbound, +East → northbound). That travel direction seeds a low-confidence heading with +the low-confidence pose, and `None` or an unknown value uses the shared id-hash +fallback; both stay marked as estimated. +Gate: `src/data/cctvHouston.test.mjs`. TranStar's disclaimer prohibits redistribution and external framing of specific images without TxDOT's written permission, so the pack is off unless diff --git a/server/providers/cctv/sources.js b/server/providers/cctv/sources.js index 013c69da3..1ae4de637 100644 --- a/server/providers/cctv/sources.js +++ b/server/providers/cctv/sources.js @@ -3795,14 +3795,31 @@ export function houstonCameraName(record, cameraId) { return short ? `Houston Camera ${short}` : 'Houston Camera'; } +/** + * TranStar `dir` (the camera's side of the freeway) -> the travel direction + * of the nearest lanes, as TranStar's map script `translateCameraDirection` + * reads it. Only these four values are mapped; anything else is unknown. + */ +const HOUSTON_TRAVEL_HEADING_BY_SIDE = Object.freeze({ + south: 90, + north: 270, + west: 180, + east: 0, +}); + /** * One Houston TranStar camera row -> one catalog source, or null. * * Frame URLs are synthesized from a validated numeric id on the pinned * snapshot origin. The payload's `url` is a `showWin(...)` viewer helper * and is ignored so a catalog edit cannot steer the frame proxy off-host. - * Dedicated `dir` values (North/South/East/West) become high-confidence - * headings; `None`/blank fall back to the shared id-hash heading. + * TranStar's `dir` is the side of the freeway the camera sits on, not the + * way it faces: TranStar's own map labels it "Traffic closest to the camera + * is traveling …" (South→Eastbound, North→Westbound, West→Southbound, + * East→Northbound). That travel direction is only a low-confidence prior + * for the view, so the heading stays estimated with the low-confidence + * pose; `None`, blank or unknown values fall back to the shared id-hash + * heading. * * @param {object} record - Raw TranStar camera row. * @returns {?object} @@ -3819,8 +3836,12 @@ export function houstonCameraToSource(record) { const cameraId = `houston-${rawId}`; const imageUrl = HOUSTON_IMAGE_URL(rawId); const name = houstonCameraName(record, cameraId); - const extractedHeading = directionToHeading(record.dir, true); - const hasHeading = Number.isFinite(extractedHeading); + const travelHeading = + HOUSTON_TRAVEL_HEADING_BY_SIDE[ + String(record.dir ?? '') + .trim() + .toLowerCase() + ]; return { id: cameraId, @@ -3830,12 +3851,14 @@ export function houstonCameraToSource(record) { provider: 'Houston TranStar', lat, lon, - headingDeg: hasHeading ? extractedHeading : fallbackHeadingFromId(cameraId), - headingConfidence: hasHeading ? 'high' : 'low', - pitchDeg: hasHeading ? -24 : -18, - fovDeg: hasHeading ? 56 : 44, - rangeM: hasHeading ? 210 : 145, - mountHeightM: hasHeading ? 10 : 8, + headingDeg: Number.isFinite(travelHeading) + ? travelHeading + : fallbackHeadingFromId(cameraId), + headingConfidence: 'low', + pitchDeg: -18, + fovDeg: 44, + rangeM: 145, + mountHeightM: 8, groundElevationM: HOUSTON_GROUND_ELEVATION_M, feedType: 'image', url: imageUrl, diff --git a/src/data/cctvHouston.test.mjs b/src/data/cctvHouston.test.mjs index 54aace016..6af6ccdbf 100644 --- a/src/data/cctvHouston.test.mjs +++ b/src/data/cctvHouston.test.mjs @@ -20,7 +20,6 @@ import { isLikelyHoustonCoordinate, } from '../../server/providers/cctv/normalize.js'; import { createCctvCatalog } from '../../server/providers/cctv/catalog.js'; -import { directionToHeading } from './directionText.js'; /** * A response whose body is a live stream, plus a flag that flips when the @@ -75,8 +74,9 @@ test('a Houston row maps to a source on the pinned HTTPS frame host', () => { assert.equal(source.url, HOUSTON_IMAGE_URL(DOWNTOWN_ID)); assert.equal(source.url, source.snapshotUrl); assert.ok(source.url.startsWith(HOUSTON_IMAGE_ORIGIN)); - assert.equal(source.headingConfidence, 'high'); - assert.equal(source.headingDeg, 180); + // dir South = the camera sits south of I-10; the nearest lanes run east. + assert.equal(source.headingConfidence, 'low'); + assert.equal(source.headingDeg, 90); }); test('frame URLs are synthesized from the numeric id; the payload URL is ignored', () => { @@ -105,13 +105,19 @@ test('zero ids, off-city geometry and unusable ids are dropped', () => { assert.equal(houstonCameraToSource(row({ id: '13711' })).id, 'houston-13711'); }); -test('bare cardinal dirs are high-confidence; None uses the id-hash fallback', () => { - assert.equal(directionToHeading('South', true), 180); - assert.equal(directionToHeading('None', true), NaN); - - const south = houstonCameraToSource(row({ dir: 'South' })); - assert.equal(south.headingConfidence, 'high'); - assert.equal(south.headingDeg, 180); +test('dir is the side of the road: it gives a low-confidence travel heading', () => { + // TranStar's map reads dir as "Traffic closest to the camera is traveling": + // South -> Eastbound, North -> Westbound, West -> Southbound, East -> Northbound. + const expected = { South: 90, North: 270, West: 180, East: 0, south: 90 }; + for (const [dir, heading] of Object.entries(expected)) { + const source = houstonCameraToSource(row({ dir })); + assert.equal(source.headingDeg, heading, `dir ${dir}`); + assert.equal(source.headingConfidence, 'low', `dir ${dir}`); + assert.equal(source.pitchDeg, -18, `dir ${dir}`); + assert.equal(source.fovDeg, 44, `dir ${dir}`); + assert.equal(source.rangeM, 145, `dir ${dir}`); + assert.equal(source.mountHeightM, 8, `dir ${dir}`); + } const none = houstonCameraToSource(row({ dir: 'None' })); assert.equal(none.headingConfidence, 'low'); @@ -120,6 +126,13 @@ test('bare cardinal dirs are high-confidence; None uses the id-hash fallback', ( none.headingDeg, houstonCameraToSource(row({ dir: '' })).headingDeg, ); + for (const dir of ['Northeast', 'constructor', '__proto__', undefined]) { + assert.equal( + houstonCameraToSource(row({ dir })).headingDeg, + none.headingDeg, + `dir ${dir} falls back to the id hash`, + ); + } }); test('a nameless row still gets a label', () => {