Merge main through #1634 into the renderer seam

Brings in the CCTV opening camera, the Iceland, Giroc HLS, Taiwan, Quebec,
Ottawa and world camera packs, the world clock and timeline, the zoom-out
limit and registered-only layer state, ported onto the renderer contract:
the sky follows the world time through the frame port's setTime in
preUpdate, the Quebec overlay reads the eye, projects and redraws through
the camera, query and frame ports, launches hide pending missions through
their draw records, and mission orbit rings turn on the scene clock.
This commit is contained in:
Sameh Khamis committed 2026-11-10 00:31:23 -08:00
commit 2b60531c3c
170 files changed
+44096 -319

No files matched your search

+155
View File
@@ -0,0 +1,155 @@
# CCTV integration workflow
Use this workflow when adding a camera provider or repairing an existing feed.
The CCTV catalog comes through the local server; media normally does too. Keep
provider-specific URL policy on the server, including any validated browser-only
media URL permitted by the exception below.
## 1. Check the source before coding
- Identify the catalog endpoint and the actual frame or live-media endpoint.
They may be different hosts or use different formats.
- Read the provider's license, terms, attribution, rate limits, and access
requirements. Publicly reachable data is not automatically licensed for
reuse. Record whether the app may display it and whether any use is
restricted.
- Prefer a public, keyless feed. If credentials are required, keep them in a
server-only environment variable; never put a secret in browser code or a
Vite-exposed variable. A pack that needs a key is opt-in by that key: with
the variable unset it makes no request and logs once (Ontario 511 and
Ottawa). Keep the key out of catalog URLs and add it only at fetch time. Use
the provider's documented API, not a website endpoint that skips its key. If
access terms require an agreement or remain unconfirmed, make the pack
opt-in and ensure it makes no request by default.
- Confirm the feed can be fetched from the app's Node server environment. A URL
opening in a browser does not prove the server can reach it, and browser CORS
behavior does not diagnose a server-side fetch.
- Treat camera directories and aggregators such as [TW Live](https://tw.live/)
as discovery aids. Trace each listing to the original agency or camera owner;
their data and media terms govern reuse. For example, SkylineWebcams says
only camera hosts may embed its live feed and its terms restrict copying or
retransmitting footage, so do not proxy its stream without written
authorization ([FAQ](https://www.skylinewebcams.com/support/faq.html),
[terms](https://www.skylinewebcams.com/en/terms-of-use.html)).
- Decide whether the provider supplies still snapshots or continuous video.
Prefer stills unless live video is necessary and supported by the existing
proxy path.
## 2. Add and register the provider pack
Use the smallest existing provider as a template. Current owners are:
- `server/providers/cctv/sources.js` for standard open-data loaders.
- `server/providers/cctv/<provider>.js` for a provider with its own parser,
fallback, or more involved URL policy.
- `server/providers/cctv/catalog.js` to import the loader and add it to
`LIVE_PACKS`. Each pack is independently gated and the shared catalog cap
allocates space across packs.
- `server/providers/cctv/constants.js` for provider endpoints, caps, and
timeouts.
Return normalized camera records with stable unique IDs, readable names,
`city`/`cityId`, provider attribution, valid `lat`/`lon`, and the correct
`feedType`. Include camera heading only when the source actually provides a
bearing; descriptive direction text is not a numeric heading. Mark uncertain
heading data with the existing low-confidence conventions. Set `sourceKind` so
health and source diagnostics identify the provider. Apply a bounded per-pack
camera limit and geographic filtering where appropriate; avoid one provider
silently consuming the whole shared catalog cap.
Camera coordinates need their own verification. Prefer the original provider's
camera coordinates, then corroborate them against its location description,
route/mile marker, or official map. An aggregator's map dot, a town/county name,
or a geocoded attraction name is not a precise camera position. Do not pin a
camera to a city center or nearby landmark; if its actual location cannot be
verified, leave it out until it can. For manually curated coordinates, record
the evidence in the provider data or an adjacent comment. For example, TW Live
camera pages identify the originating agency; use that agency's catalog for
the camera point when available.
If the source becomes opt-in, use the catalog's opt-in gate (`envOptIn`) and
document the flag and reason in `DATA_SOURCES.md`. Otherwise use the normal
enabled-by-default flag pattern and document how to disable the pack.
## 3. Keep frame and media requests server-registered
The browser should request the local `/api/cctv/frame/:id` or
`/api/cctv/media/:id` endpoint. Do not pass an arbitrary upstream URL from the
browser, and do not fetch an unvalidated URL supplied by a catalog record.
Every provider must have a server-side rule that pins frame and media URLs to
the expected scheme, host, and path; use HTTPS where the provider supports it.
Use `pinnedCameraUrl()` for URLs that belong under a fixed provider directory,
or a stricter provider-specific validator when hosts or paths vary. Register
any new permitted image/media origin in the existing
`server/providers/cctv` media policy.
Catalog and media loaders must also:
- refuse unsafe schemes, credentials, unexpected ports, redirects outside the
registered origin/path, and malformed identifiers;
- set request deadlines and cap catalog and buffered frame bodies;
- parse only the expected format and validate coordinates before returning a
camera;
- handle offline cameras and partial provider failures without failing the
entire merged catalog; and
- avoid persistent frame storage unless the provider's terms explicitly allow
it and the design has been reviewed.
For live media, prefer the existing proxy and supported feed type (`mjpeg`,
`mp4`, `webm`, or `hls`). A direct browser-only video is a narrow fallback for
a source whose terms allow embedding when the browser can play it but the app
server cannot fetch it. Unconfirmed embedding terms make it opt-in like any
other pack, and the browser loads it only after an explicit play action, never
on selection, because the provider sees the viewer's IP address. Build its URL
from a validated provider ID on the server; never accept a browser-supplied
media URL. Render it in a native DOM
`<video>` without `crossOrigin`, and do not read its pixels into canvas, WebGL,
or a Cesium texture. If a map preview is needed, a second video request may be
paused as a DOM overlay at the camera location; label it as an approximate
still and scale it with map distance. If terms disallow embedding, use a source
link instead. Read [SECURITY.md](../SECURITY.md) before changing URL
validation, redirects, or proxy behavior.
When an image provider sets a minimum refresh interval, enforce it at the
server frame boundary as well as in the client. A bounded in-memory cache and
single-flight request per camera prevent projection, panel, and re-selection
paths from multiplying upstream requests; do not persist frames unless the
provider's terms explicitly allow it.
## 4. Add focused regression coverage
Add provider tests under `src/data/` following the existing CCTV test naming
and fixtures. Cover at least:
- representative valid catalog rows and their normalized output;
- missing or invalid IDs, coordinates, and frame URLs;
- duplicate records and the pack's cap/filter behavior;
- unsafe host/path, credentials, and redirect cases;
- timeout, oversized, malformed, and unavailable upstream responses; and
- the frame or live-media content type the provider actually serves.
Use `src/data/cctvProxy.test.mjs` for shared proxy/media behavior and
`src/data/cctvCatalogCap.test.mjs` for shared catalog allocation. For a
browser-only source, verify native playback, denied canvas pixel access, the
paused second-player overlay, and its cleanup separately from proxy tests. Run
the focused provider and proxy tests, then the standard checks from
[CONTRIBUTING.md](../CONTRIBUTING.md) that apply to the change. For runtime
verification, inspect `/api/cctv/sources`, `/api/cctv/health`, and the camera's
`/api/cctv/frame/:id` or `/api/cctv/stream/:id` response locally. Confirm a
successful frame is from the upstream camera by checking its content type and
`X-CCTV-Source`; a fallback image is not proof that the upstream feed works.
## 5. Update source documentation and submit
- Add the provider, endpoint, terms/license, required attribution, key status,
caching behavior, and any restrictions to `DATA_SOURCES.md`.
- Update the CCTV source section in `docs/CURRENT-STATE.md` when runtime
behavior, flags, caps, or fallback behavior changes.
- Register in-app attribution through the existing source/provider credit
fields; update `src/data/dataCredits.js` when the source needs a layer-level
credit.
- Add the required `changelog.d/<pr-number>-<slug>.md` entry for a runtime
change, and include the source, license, attribution, key/runtime-fetch
details, and verification evidence in the PR description.
- For a new layer, also follow the camera/source requirements in the
[new-layer PR checklist](../CONTRIBUTING.md#adding-a-new-layer).
+124 -26
View File
@@ -1641,6 +1641,32 @@ default renderer's entity clustering grouped them (`geo/screenCluster.js`,
changes. A caller handing the layer a Cesium viewer, with no
renderer bound, loads the `./render/cesium/unbound` export once first.
## Giroc / Webcam Romania CCTV source pack
`config/cctv_sources.romania.json` is an optional single-camera pack for Giroc,
Timiș, Romania. To use it, set `CCTV_SOURCES_FILE=config/cctv_sources.romania.json`
in the local `.env` and restart the server. Set `CCTV_FORCE_AUSTIN=1` as well to
retain the built-in live packs; despite its historical name, this flag applies
to all live packs. The default catalog is unchanged.
The camera uses Webcam Romania's public HLS stream and JPEG snapshot, fetched
at runtime without an API key. Its position, 45.70028947266781, 21.233675914004227,
was corrected by the contributor rather than copied from the directory's
inaccurate pin. Heading, pitch, field of view, mounting height and ground
elevation are uncalibrated priors; the camera retains the RAW PRIOR badge.
See `DATA_SOURCES.md` for attribution and provider terms. Webcam Romania's
terms grant no reuse licence for the video and name no camera operator; the
provider's sanctioned route is its iframe embed. Keep this pack to local use
and off hosted deployments unless the operator grants permission.
The existing HLS puller resolves same-origin relative master/media references
and serves local leased playlists and segments. It downloads complete segments
without forwarding a browser Range header. Its per-segment streamed-byte cap
is 8 MiB: Giroc's roughly ten-second 2702×1520 segments can exceed the former
4 MiB cap (one measured 6,437,120 bytes). Retained data remains capped at
24 MiB and 12 segments per session, with two sessions maximum. Redirect
refusal, origin pinning, timeouts and lease expiry are unchanged.
## NYC DOT TMC CCTV source pack
The CCTV catalog can add New York City traffic cameras from the NYC DOT
@@ -1656,9 +1682,11 @@ from the payload. There is no heading in the feed, so gizmos use the shared
id-hash fallback at low confidence.
Once enabled, the 250 cameras nearest the five borough cores load.
`CCTV_NYC_MAX_SOURCES` sets the pack cap (8–400). The catalog-wide ceiling stays at 4,000 and is shared
round-robin: no region is dropped outright, but when the packs together exceed
the ceiling, adding a pack trims cameras from the largest packs.
`CCTV_NYC_MAX_SOURCES` sets the pack cap (8–400). The default catalog limit is
7,350: 4,000 shared round-robin slots, plus room above them for the 1,120
Québec locations and 2,230 World image sources. Lower explicit
`CCTV_MAX_SOURCES` values are honored; reservations only use the capacity
above the 4,000 shared slots.
## City of Toronto RESCU CCTV source pack
@@ -1674,8 +1702,9 @@ cameras return a 200 "Image Unavailable" placeholder frame instead of an error.
By default all ~336 cameras load, nearest downtown first.
`CCTV_TORONTO_MAX_SOURCES` sets the pack cap (8–500) and
`CCTV_TORONTO_ENABLED=0` turns the pack off. The pack shares the 4,000
catalog-wide ceiling round-robin with the other packs.
`CCTV_TORONTO_ENABLED=0` turns the pack off. The pack shares the default
7,350-source catalog limit with the other packs; Québec and the World image
pack have reserved slots and the remaining sources use round-robin allocation.
## Lithuania CCTV source pack
@@ -1695,24 +1724,8 @@ video; camera poses use the existing low-confidence heading prior.
## Taiwan CCTV source pack
The CCTV catalog includes Taiwan national freeway cameras from the Freeway
Bureau, MOTC. One keyless request reads the daily MOTC 2.0 `CCTV.xml` table
(~1,800 cameras); the request refuses redirects, has a 15-second deadline and a
6 MiB response cap, and uses the shared 15-minute cache and fair catalog
allocation. A failed download serves the last good table (up to a day old),
or an empty pack, and is not retried for an hour: the catalog host does not
answer from some networks outside Taiwan, and without the backoff every refresh
there would wait out the connect timeout. The table
publishes MJPEG stream URLs; each is mapped to the same camera server's
single-JPEG path for the same camera number, and only those exact HTTPS pairs
on the four `freeway.gov.tw` camera servers are registered with the frame
proxy. Camera keys come from the server and camera number, because published
CCTV IDs contain Chinese place names and a few repeat across cameras. Poses use
the low-confidence heading prior. `CCTV_TAIWAN_ENABLED=0` disables the pack;
`CCTV_TAIWAN_MAX_SOURCES` sets its cap (default 600, bounded to 1–2000), and
`CCTV_TAIWAN_URL` overrides the table URL.
Gate: `src/data/cctvTaiwan.test.mjs`.
The CCTV catalog includes Taiwan national freeway cameras from the Freeway Bureau, MOTC. The app first reads the daily MOTC 2.0 CCTV.xml table with a keyless HTTPS request, a 15-second deadline, a 6 MiB response cap, and the shared 15-minute cache. If that catalog is unreachable and no last-good source list exists, the loader uses the Highway Bureau’s public provincial-road CCTV XML feed at https://cctv-maintain.thb.gov.tw/opendataCCTVs.xml. The fallback is capped by the same Taiwan source limit and hourly retry backoff; its MJPEG feeds are restricted to the eight cctv-ss01 through cctv-ss08.thb.gov.tw HTTPS servers. The frame proxy returns one bounded JPEG from each stream, not an unbounded video response. The Highway Bureau open-data page states XML downloads are free and asks users to attribute the source; the app credits Highway Bureau, MOTC. These are provincial-road cameras, not a replacement for the national freeway cameras. CCTV_TAIWAN_ENABLED=0 disables both feeds; CCTV_TAIWAN_MAX_SOURCES sets the shared cap (default 600, bounded to 1–2000), and CCTV_TAIWAN_URL overrides the Freeway Bureau table URL.
Gate: src/data/cctvTaiwan.test.mjs and src/data/cctvProxy.test.mjs.
## DriveBC CCTV source pack
The CCTV catalog adds DriveBC highway cameras for British Columbia alongside the
@@ -4467,7 +4480,7 @@ its criteria cannot be silently ignored.
| Satellites | CelesTrak | `src/data/satellites.js` | `/api/celestrak` | 120s |
| Space Missions (30d) | Launch Library 2 + CelesTrak | `src/data/rocketLaunches.js` | `/api/launches` + `/api/celestrak/active` | 5 min |
| Traffic | Selectable TomTom / OpenStreetMap / Hybrid roads; optional TomTom flow (BYOK) | `src/data/traffic.js` | browser-direct OpenFreeMap tiles; `/api/tomtom` for flow | viewport-driven; capped tile cache |
| CCTV | Austin + Caltrans (CA) + TfL London + Ontario 511 + Fintraffic (FI) + DriveBC (BC) + TxDOT (TX) + Estonia (Tallinn, Tarktee) + Live Traffic NSW + Open Calgary Open Data + Statens vegvesen (NO, stills + live HLS, Kartverket ground heights) + King County (WA) Open Data + WSDOT (WA) Open Data + NYC DOT TMC (opt-in) + City of Toronto RESCU + Catalonia (Servei Català de Trànsit) + Hong Kong Transport Department + FL511 (FL, opt-in) + Freeway Bureau, MOTC (Taiwan) + Houston TranStar (opt-in) + Autostrada del Brennero (IT, A22 stills) + Street View fallback | `src/data/cctv.js` | `/api/cctv` | 10s (active) |
| CCTV | Austin + Caltrans (CA) + TfL London + Ontario 511 + Fintraffic (FI) + DriveBC (BC) + TxDOT (TX) + Estonia (Tallinn, Tarktee) + Live Traffic NSW + Open Calgary Open Data + Statens vegvesen (NO, stills + live HLS, Kartverket ground heights) + King County (WA) Open Data + WSDOT (WA) Open Data + NYC DOT TMC (opt-in) + City of Toronto RESCU + Catalonia (Servei Català de Trànsit) + Iceland (Vegagerðin / IRCA) + Hong Kong Transport Department + FL511 (FL, opt-in) + Freeway Bureau, MOTC (Taiwan) + Houston TranStar (opt-in) + Autostrada del Brennero (IT, A22 stills) + Street View fallback | `src/data/cctv.js` | `/api/cctv` | 10s (active) |
| Radio | Radio Browser (public-domain station directory) | `src/data/radio.js` | `/api/radio/stations`, `/api/radio/click/:uuid` | 45 min directory refresh |
| Transit 🚌 | Operator GTFS-Realtime VehiclePositions (7 keyless regions, `src/data/transitFeeds.js`) | `src/layers/transit/` via `src/app/layers/transit.js` | `/api/transit` | 15s (poll + delayed playback) |
| Bikeshare 🚲 | GBFS (Lyft + BCycle + Social Bicycles + PBSC incl. Bike Share Toronto, Bike Itaú (Brazil), Bicing (Barcelona) and seven other Spanish cities + Urban Sharing, Entur, JCDecaux (Lillestrøm, Valencia, Seville) + nextbike (AMBici)) | `src/data/bikeshare.js` | `/api/gbfs` | 60s |
@@ -5548,8 +5561,10 @@ silently demoting every later lookup for the session.
kill switch `CCTV_FINTRAFFIC_ENABLED=0`) are the fourth pack: one keyless GeoJSON station list
covering the whole country, where one _preset_ (a station's fixed view) is one camera — 806
GATHERING stations carry 2,256 in-collection presets, prioritized to 300 against seven anchors
on the main road spine. `CCTV_MAX_SOURCES` is a 4,000 catalog-wide ceiling shared round-robin
across packs, so a lower global cap thins every region instead of starving the last pack.
on the main road spine. `CCTV_MAX_SOURCES` defaults to 7,350: 4,000 shared
round-robin slots plus 3,350 reserved Québec and World image slots.
Reservations only use capacity above the 4,000 shared slots, so at 4,000 or
less every pack, Québec and World included, shares the cap round-robin.
~1,100 cameras total, all RAW PRIOR poses, stills-first. The catalog lives for 15 minutes; once
it expires, `/api/cctv` keeps answering from it while one background refresh reloads every
pack, so only the very first load waits for the slowest pack (a live first load took about
@@ -5738,6 +5753,13 @@ and unreachable upstream (502/504) separately from road geometry.
explicit-empty form); unknown option tokens are ignored. Restoration
settles independently per layer so one failed or unavailable source cannot
block its siblings.
- An application that registers fewer layers (an embedder with a smaller
catalog) keeps only its own: `LayerStateCoordinator` drops enabled-layer ids
the application did not register before it normalizes a share, saved state or
explicit change, so such a layer is never restored, written to local storage
or carried into a link this application shares. Options that depend on a
dropped layer (a tracked selection) go with it. An application with every
layer registered is unchanged.
- Layer tokens are permanent public compatibility identifiers. The reservation
JSON ledger at `src/data/layerStateTokenReservations.json` pins all existing
one-character assignments (all letters plus `0`, `1` and
@@ -7260,3 +7282,79 @@ the container healthy. Checks time out after 10 seconds, with a 30-second
startup grace period and three consecutive failures before reporting unhealthy.
This reports app HTTP availability, not individual provider health, and does not
configure automatic restarts.
## Québec CCTV integration
The regular CCTV source loader reads `public/quebec-cameras.json` by default
when `CCTV_QUEBEC_ENABLED` is not `0`. The pack contains 1,120 records: 680
MTQ highway cameras, 426 Montréal OSM camera locations, and 14 scenic webcams.
The default catalog limit is 7,350: it reserves all 1,120 Québec locations and
2,230 World image feeds, and keeps 4,000 slots for the existing round-robin
camera packs. Reservations only use capacity above those 4,000 slots: a lower
explicit `CCTV_MAX_SOURCES` thins every pack round-robin, Québec and World
included, instead of taking the shared slots.
The Québec 511 video is opt-in (`CCTV_QUEBEC_VIDEO_ENABLED=1`) because Québec
511's terms for embedding its clips on another site are unconfirmed; the camera
locations themselves are CC BY 4.0. With the flag unset, MTQ cameras show
their location, credit and a link to the Québec 511 viewer page. With it set,
selecting an MTQ camera opens a floating panel with a Play button and a notice
that Québec 511 sees the viewer's IP address; the browser contacts Québec 511
only after Play, never on selection alone (layer enable, auto-hop, a scene or
voice), and a different camera starts unplayed. After Play, the panel plays
the official MP4 in a native browser player and requests a second video for the
paused map preview. The official feed is a five-second clip refreshed by
Québec 511 every 2–4 minutes, not continuous live video; the browser player
reloads every two minutes while the tab is visible. The
map preview is a DOM overlay attached to the camera's coordinates and scales
down with camera distance. Neither video is read into a canvas or Cesium texture:
Québec 511 does not grant localhost pixel access, and its MP4 is not available
to the app's server proxy. Montréal and scenic entries are regular CCTV camera
locations with per-camera source-page links, but no direct frame URLs; those
show a location label and open the official/operator page. MTQ positions carry
CC BY 4.0 attribution, Montréal positions are attributed to OpenStreetMap
contributors under ODbL, and scenic rows retain their position accuracy and
provenance. Québec cameras load through the normal CCTV layer.
## City of Ottawa CCTV source pack
The CCTV catalog can add City of Ottawa traffic cameras, including the MTO
cameras the city carries. Camera images come from the city's Traffic Camera
Images API (`traffic.ottawa.ca/opendata/camera`), which needs a certificate
from the city's registration form and lets one certificate and user id fetch a
camera at most once a minute. Without `OTTAWA_CAMERA_CERTIFICATE` the pack is
skipped with no request and one log line. The camera list is the documented
keyless `traffic.ottawa.ca/map/service/camera` JSON. Catalog URLs carry no
key: the frame proxy adds the certificate and the fixed user id `gev` at fetch
time, only for a numeric camera number inside the pinned `/opendata/` path,
and serves each camera from one shared server-side frame per minute. Inside
that minute a camera with no cached frame gets the placeholder, never the
Street View fallback. The feed publishes no bearing, so headings use the
id-hash fallback at low confidence. `CCTV_OTTAWA_MAX_SOURCES` caps the pack
(default 250, nearest downtown) and `CCTV_OTTAWA_ENABLED=0` turns it off.
## World camera image feeds
The default `world-cameras` pack reads only the checked-in, curated
`config/cctv_sources.world-image-cameras.json` file. It adds 1,176 Oregon
TripCheck and 1,054 MLIT river cameras. Duplicate feeds are excluded because
DriveBC, Ottawa, Québec and Iceland (Vegagerðin) already have app packs. NZTA's
249 rows are excluded because they were collected from a Journey Planner web
cache whose current terms prohibit automated scraping; an authorized API feed
or permission is required before adding them. The wider discovery export is
not shipped with the app.
The app validates and pins each still-image URL to its provider directory
before the server proxy uses it. Oregon frames refresh no faster
than every five minutes; MLIT river frames refresh every ten minutes to reduce
requests. Images are not persisted; the proxy reuses frames in a bounded 32 MB
in-memory cache until their provider cadence. Set
`CCTV_WORLD_CAMERAS_ENABLED=0` to disable the pack.
Only the curated image-backed rows are bundled. Camera images are fetched from
their registered providers at request time and are not included in this data
file. The additive `/api/cctv/sources` metadata includes provider source links,
browser-only media links, refresh cadence, and location provenance where
available. When MLIT serves its recognized adjustment image, `/api/cctv/frame`
returns a God’s Eye maintenance card and `/api/cctv/health` reports the camera
as degraded with the provider-adjustment reason.
+5
View File
@@ -119,6 +119,7 @@ probability of about 255 in 256 (many of them already fail as base64url).
| 25 | `scopeFeatherPct` | uint | 100 | 0–200 |
| 26 | `scopeTerminusPct` | uint | none | 94–100; absent is the adaptive terminus |
| 27 | `styleParams` | LEN | style | packed `(token, value × 100)` pairs |
| 28 | `worldTime` | epoch | none (live) | zigzag seconds since 2026-01-01T00:00:00Z |
Field 1 absent means `thing` when field 13 is present and `place` otherwise,
so v1 never writes it; a later view kind writes its code.
@@ -132,6 +133,10 @@ style also extends the pinned style-defaults table.
`hudVariant` code 1 (`operator`) is retired: the app no longer offers that
layout and restores it as `tactical`. The code stays reserved.
Field 28 is the world time the link was made at, written only when it was away
from live. A link without it opens live. Unlike field 8 it may lie ahead of the
link's creation (a forecast moment), and it opens paused at that time.
Flag bits (field 9): 0 `bloom`, 1 `sharpen`, 2 `hudVisible`, 3
`celestialRing` (each defaulting by style), 4 `scopeEnabled` (default on).