Pin camera sources by parsed URL

Camera catalog image URLs were pinned by string prefix, so a URL that only
began with the agency's directory before dot segments or an encoded separator
were resolved still counted as inside it. The pin now parses the URL and
requires the same scheme, the exact host, no credentials, the default port and
a normalized path inside the pinned directory. Redirects on a pinned host must
stay inside the same directory.

Document the response format checks and API response headers alongside the
existing server hardening notes.

SECURITY.md

@@ -69,6 +69,8 @@ The data proxies under `server/providers/` are written so the browser cannot tur
 - **No verbatim client headers upstream.** The CCTV media route is the one route that relays a request header (`Range`, for video seeking). It is parsed and canonicalized before it is forwarded: one `bytes=` range only, with every accepted form — explicit span, open-ended and suffix — bounded to the same 64 MiB ceiling the relay applies to a declared response body. Multi-range, malformed, inverted and non-`bytes` values are dropped and the request proceeds without a `Range`, as RFC 7233 §3.1 prescribes. Bounding the request bounds what is asked for: a response that declares no length — live streamed media, or a chunked body from an upstream that ignores the `Range` — has no ceiling. An upstream request the browser has stopped waiting for is cancelled rather than left running, whether the viewer leaves before the headers arrive or during the body.
 - **Transit fetches registered feeds only.** `/api/transit/vehicles/<id>` resolves the id against `src/data/transitFeeds.js`; the browser never supplies a URL, and a feed that is registered but disabled does not resolve at all. Redirects are followed manually and each hop is validated against the feed's own https origin before it is requested, so an off-origin or downgraded hop is refused rather than contacted. Bodies are capped at 8 MB, the protobuf is decoded server-side under entity-count and string-length ceilings, a differential feed is refused, and snapshots live 15 s in memory with no disk cache. A per-feed admission limiter and a failure cooldown ladder bound what this process can ask of any operator.
 - **Local receiver feeds are operator-configured and local-only.** `/api/local-receivers/aircraft` reads only the `aircraft.json` URLs in the server's `LOCAL_RECEIVER_FEEDS`; the browser never supplies an address. Each host must pass the shared tap address rule (`src/data/tapAddress.js`: loopback, RFC1918, `localhost`, `*.local`; no link-local, IPv6 or other names), the scheme must be http(s), the path must end in `aircraft.json`, and credentials, queries and fragments are refused. Invalid entries are logged at startup and never fetched. Reads use a 2 s timeout, refuse redirects, cap bodies at 2 MB and share one read per second; responses name each feed by band only and never carry upstream error text.
+- **Validated response formats.** Proxied bodies are relayed only in the format the route serves. CCTV frames and the Street View fallback accept JPEG, PNG, GIF, WebP and AVIF only, sent under their canonical type; any other type falls through to the next fallback. Live CCTV media must match its feed kind (`video/mp4`, `video/webm`, or `multipart/x-mixed-replace` with a boundary) before it is relayed or the camera counts as healthy, and its redirects stay on the registered origin. GBFS bodies must parse as JSON and are re-serialized as `application/json; charset=utf-8`. Camera catalog URLs are pinned by parsed URL — scheme, exact host, no credentials, default port, and a normalized path inside the agency's image directory — and a redirect on a pinned host must stay inside the same directory. Every `/api/cctv` route uses the cross-site gate below.
+- **Consistent API response headers.** Every `/api` response, including errors, partial content, Host refusals and unknown routes, carries `Content-Security-Policy: default-src 'none'; sandbox` and `X-Content-Type-Options: nosniff` (`apiResponseHeadersPlugin` in `build/vite.js`, installed ahead of the Host check). A route that sets its own policy adds it as a further policy; it never replaces this one.
 - **Response-size caps and timeouts** on proxied responses.
 - **Sanitized errors** — internal error details are not echoed back to clients.
 - **Coalesced OAuth refresh** and cached successful responses only (OpenSky).

changelog.d/0000-provider-response-hardening.md

@@ -0,0 +1 @@
+- Validate provider response formats, strengthen camera source checks, and apply consistent API response protections (samehkhamis).

docs/CURRENT-STATE.md

@@ -30,6 +30,15 @@ time. This clears obsolete contacts and settles startup without manufacturing
 position timestamps. Stale, incomplete and malformed empty results retain the
 existing recovery behavior.
 
+Every `/api` response on the dev and preview servers carries
+`Content-Security-Policy: default-src 'none'; sandbox` and
+`X-Content-Type-Options: nosniff`. `apiResponseHeadersPlugin` sets them ahead of
+the Host check, so errors, partial responses, Host refusals and unknown routes
+are covered too; a route's own policy is sent as an additional policy, never in
+place of it. The app document's policy is unchanged. The GBFS proxy relays only
+bodies that parse as JSON, re-serialized as `application/json; charset=utf-8`;
+anything else answers 502.
+
 Catalog source contracts and their consuming layers are declared in
 `src/app/sourceComposition.js`. Omitted or null acquisition sources preserve layer
 registration and serialization, report unavailable, and are refused by the existing
@@ -812,6 +821,13 @@ standalone entry supplies application-owned scene, ground and activation service
 Each layer owns its state and visibility listener; destruction cancels source
 reads and pending initialization. Malformed health responses retain prior health.
 Existing catalog fallback, camera poses, frame pacing and coverage controls remain.
+CCTV frames and the Street View fallback accept only JPEG, PNG, GIF, WebP and
+AVIF, sent under the canonical type; any other type falls through to the next
+fallback. Live media must match its feed kind (`video/mp4`, `video/webm`, or
+`multipart/x-mixed-replace` with a boundary) before health reports it connected,
+and follows redirects only on the camera's origin. Catalog image URLs, and
+redirects on those hosts, are pinned by parsed URL to the agency's image
+directory. Every `/api/cctv` route uses the same-site gate.
 
 Traffic owns its rendered-surface observer only while enabled, removes it on disable/destroy, and caches surface identity without allocating on unchanged frames.
 

scripts/format-scope.json

@@ -121,6 +121,7 @@
   "server/providers/cctv/normalize.js",
   "src/data/cctvMissingNumbers.test.mjs",
   "server/providers/cctv/sources.js",
+  "src/data/cctvSourcePins.test.mjs",
   "server/providers/radio.js",
   "server/providers/radio/catalog.js",
   "server/providers/radio/constants.js",

server/providers/cctv/media.js

@@ -1,13 +1,21 @@
 import { Readable } from 'node:stream';
-import { hashSeed, escapeXml } from './normalize.js';
+import { hashSeed, escapeXml, pinnedCameraUrl } from './normalize.js';
 import {
+  CALGARY_IMAGE_ORIGIN,
   CCTV_FRAME_FETCH_TIMEOUT_MS,
   CCTV_FRAME_MAX_BODY_BYTES,
   CCTV_MEDIA_FETCH_TIMEOUT_MS,
   CCTV_MEDIA_IDLE_TIMEOUT_MS,
   CCTV_MEDIA_MAX_BODY_BYTES,
+  HK_IMAGE_ORIGIN,
+  KING_COUNTY_IMAGE_ORIGIN,
   NSW_IMAGE_ORIGIN,
   NSW_IMAGE_USER_AGENT,
+  TALLINN_IMAGE_ORIGIN,
+  TARKTEE_IMAGE_ORIGIN,
+  TFL_IMAGE_ORIGIN,
+  WARENDORF_IMAGE_ORIGINS,
+  WSDOT_IMAGE_ORIGIN,
 } from './constants.js';
 /**
  * Generate a synthetic SVG billboard image for a CCTV camera placeholder.
@@ -504,9 +512,40 @@ export async function fetchTxdotSnapshot(
 /** Redirect hops the frame path will follow, and only within the same host. */
 const MAX_SAME_HOST_REDIRECTS = 2;
 
+/**
+ * Directories camera catalogs are pinned to, by host. A redirect on one of
+ * these hosts must stay inside the pinned directory too, so a same-origin
+ * redirect cannot leave it (another bucket on a shared storage host).
+ */
+const CCTV_PINNED_PREFIXES_BY_HOST = new Map();
+for (const prefix of [
+  TFL_IMAGE_ORIGIN,
+  TALLINN_IMAGE_ORIGIN,
+  TARKTEE_IMAGE_ORIGIN,
+  ...WARENDORF_IMAGE_ORIGINS,
+  NSW_IMAGE_ORIGIN,
+  CALGARY_IMAGE_ORIGIN,
+  HK_IMAGE_ORIGIN,
+  KING_COUNTY_IMAGE_ORIGIN,
+  WSDOT_IMAGE_ORIGIN,
+]) {
+  const host = new URL(prefix).hostname;
+  CCTV_PINNED_PREFIXES_BY_HOST.set(host, [
+    ...(CCTV_PINNED_PREFIXES_BY_HOST.get(host) || []),
+    prefix,
+  ]);
+}
+
+/** Whether a redirect target stays inside its host's pinned directory. */
+function withinPinnedPrefix(url) {
+  const prefixes = CCTV_PINNED_PREFIXES_BY_HOST.get(url.hostname);
+  return !prefixes || prefixes.some((p) => pinnedCameraUrl(url.href, p));
+}
+
 /**
  * Fetch a registered frame URL following redirects ONLY within the original
- * origin (scheme, host and port; at most MAX_SAME_HOST_REDIRECTS hops).
+ * origin (scheme, host and port; at most MAX_SAME_HOST_REDIRECTS hops) and,
+ * on a pinned camera host, within its pinned directory.
  * Default redirect-following would let an upstream steer a host-pinned
  * request, and its host-specific headers, to any origin, another port, or a
  * plaintext downgrade.
@@ -547,7 +586,7 @@ export async function fetchWithinHost(url, init, fetchImpl = fetch) {
     } catch {
       return null;
     }
-    if (next.origin !== origin) return null;
+    if (next.origin !== origin || !withinPinnedPrefix(next)) return null;
     current = next;
   }
   return null;

server/providers/cctv/normalize.js

@@ -659,3 +659,36 @@ export function isLikelyTorontoCoordinate(lat, lon) {
     lon <= -79.1
   );
 }
+
+/**
+ * A camera URL under a pinned directory, by parsed URL rather than string
+ * prefix: the same scheme and exact host as `prefix`, no credentials, the
+ * default port, and a normalized path inside the prefix's directory. Dot
+ * segments (including encoded ones) are resolved before the path is compared,
+ * and an encoded slash or backslash is refused.
+ *
+ * @param {string} raw - Camera URL from a catalog or redirect.
+ * @param {string} prefix - Pinned origin and directory, ending in `/`.
+ * @returns {string|null} The normalized URL, or null when it is not pinned.
+ */
+export function pinnedCameraUrl(raw, prefix) {
+  let url;
+  let pin;
+  try {
+    url = new URL(String(raw ?? '').trim());
+    pin = new URL(prefix);
+  } catch {
+    return null;
+  }
+  if (
+    url.protocol !== pin.protocol ||
+    url.hostname !== pin.hostname ||
+    url.username ||
+    url.password ||
+    url.port ||
+    !url.pathname.startsWith(pin.pathname) ||
+    /%2f|%5c|\\/i.test(url.pathname)
+  )
+    return null;
+  return url.href;
+}

server/providers/cctv/sources.js

@@ -149,6 +149,7 @@ import {
   cameraDisplayCode,
   rowArrayToObject,
   prioritizeSources,
+  pinnedCameraUrl,
 } from './normalize.js';
 import { directionToHeading } from '../../../src/data/directionText.js';
 import {
@@ -310,9 +311,12 @@ export async function loadCaltransSourcesFromOpenData() {
       const lon = toFiniteNumber(loc.longitude);
       if (!Number.isFinite(lat) || !Number.isFinite(lon)) continue;
 
-      const imageUrl = String(cctv.imageData?.static?.currentImageURL || '');
       // Official-host pin (see JSDoc). Also drops records with no still image.
-      if (!imageUrl.startsWith('https://cwwp2.dot.ca.gov/')) continue;
+      const imageUrl = pinnedCameraUrl(
+        cctv.imageData?.static?.currentImageURL,
+        'https://cwwp2.dot.ca.gov/',
+      );
+      if (!imageUrl) continue;
 
       const locationName = String(loc.locationName || '').trim();
       // Leading token of locationName is the stable camera code ("TV102 -- I-580 : …").
@@ -417,8 +421,8 @@ export async function loadTflSourcesFromOpenData() {
       const lat = toFiniteNumber(place?.lat);
       const lon = toFiniteNumber(place?.lon);
       if (!Number.isFinite(lat) || !Number.isFinite(lon)) continue;
-      const imageUrl = String(props.imageUrl || '');
-      if (!imageUrl.startsWith(TFL_IMAGE_ORIGIN)) continue; // official-bucket pin
+      const imageUrl = pinnedCameraUrl(props.imageUrl, TFL_IMAGE_ORIGIN);
+      if (!imageUrl) continue; // official-bucket pin
 
       // "JamCams_00002.00865" → "tfl-00002.00865" (provider-stable id).
       const rawId = String(place?.id || '').replace(/^JamCams_/, '');
@@ -1127,13 +1131,15 @@ export function loadTallinnSourcesFromCatalog({
     // Rough Estonia/Tallinn metro sanity (allows nearby suburbs already in the pack).
     if (lat < 59.2 || lat > 59.7 || lon < 24.3 || lon > 25.4) continue;
 
-    const imageUrl =
+    const imageUrl = pinnedCameraUrl(
       typeof item.url === 'string'
-        ? item.url.trim()
+        ? item.url
         : typeof item.snapshotUrl === 'string'
-          ? item.snapshotUrl.trim()
-          : '';
-    if (!imageUrl.startsWith(TALLINN_IMAGE_ORIGIN)) continue;
+          ? item.snapshotUrl
+          : '',
+      TALLINN_IMAGE_ORIGIN,
+    );
+    if (!imageUrl) continue;
 
     const extractedHeading = toFiniteNumber(item.headingDeg, NaN);
     const hasHeading = Number.isFinite(extractedHeading);
@@ -1235,8 +1241,8 @@ export function parseTarkteeDatexImages(xml) {
       body,
     );
     if (!refMatch || !urlMatch) continue;
-    const url = urlMatch[1].trim();
-    if (!url.startsWith(TARKTEE_IMAGE_ORIGIN)) continue;
+    const url = pinnedCameraUrl(urlMatch[1], TARKTEE_IMAGE_ORIGIN);
+    if (!url) continue;
     out.set(refMatch[1], url);
   }
   return out;
@@ -1422,14 +1428,16 @@ export function loadWarendorfSourcesFromCatalog({
   for (const item of rows) {
     if (!item || typeof item !== 'object') continue;
     const id = typeof item.id === 'string' ? item.id.trim() : '';
-    const url =
+    const raw =
       typeof item.url === 'string'
-        ? item.url.trim()
+        ? item.url
         : typeof item.snapshotUrl === 'string'
-          ? item.snapshotUrl.trim()
+          ? item.snapshotUrl
           : '';
-    if (!id || !WARENDORF_IMAGE_ORIGINS.some((o) => url.startsWith(o)))
-      continue;
+    const url = WARENDORF_IMAGE_ORIGINS.map((o) =>
+      pinnedCameraUrl(raw, o),
+    ).find(Boolean);
+    if (!id || !url) continue;
     const lat = typeof item.lat === 'number' ? item.lat : NaN;
     const lon = typeof item.lon === 'number' ? item.lon : NaN;
     if (!isPlausibleLatLon(lat, lon)) continue;
@@ -1483,8 +1491,8 @@ export function nswCameraToSource(feature) {
   const lat = typeof coords?.[1] === 'number' ? coords[1] : NaN;
   if (!isLikelyNswCoordinate(lat, lon)) return null;
   const props = feature?.properties || {};
-  const url = String(props.href || '').trim();
-  if (!url.startsWith(NSW_IMAGE_ORIGIN)) return null;
+  const url = pinnedCameraUrl(props.href, NSW_IMAGE_ORIGIN);
+  if (!url) return null;
   // "N-E" -> "NE" for the compass lookup.
   const direction = String(props.direction || '')
     .trim()
@@ -1578,8 +1586,7 @@ export function normalizeCalgaryImageUrl(raw) {
   }
   if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') return null;
   parsed.protocol = 'https:';
-  const upgraded = parsed.toString();
-  return upgraded.startsWith(CALGARY_IMAGE_ORIGIN) ? upgraded : null;
+  return pinnedCameraUrl(parsed.toString(), CALGARY_IMAGE_ORIGIN);
 }
 
 /**
@@ -2085,8 +2092,7 @@ export function normalizeHkImageUrl(raw) {
   }
   if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') return null;
   parsed.protocol = 'https:';
-  const upgraded = parsed.toString();
-  return upgraded.startsWith(HK_IMAGE_ORIGIN) ? upgraded : null;
+  return pinnedCameraUrl(parsed.toString(), HK_IMAGE_ORIGIN);
 }
 
 /**
@@ -2423,10 +2429,16 @@ export async function loadKingCountySourcesFromOpenData() {
       // county image origin, so they are skipped.
       if (owner === 'WSDOT') continue;
       // The handler is served over HTTPS; the feed still writes http://.
-      const imageUrl = String(attrs.ImageURL || '')
-        .trim()
-        .replace(/^http:\/\/info\.kingcounty\.gov\//, KING_COUNTY_IMAGE_ORIGIN);
-      if (!imageUrl.startsWith(KING_COUNTY_IMAGE_ORIGIN)) continue;
+      const imageUrl = pinnedCameraUrl(
+        String(attrs.ImageURL || '')
+          .trim()
+          .replace(
+            /^http:\/\/info\.kingcounty\.gov\//,
+            KING_COUNTY_IMAGE_ORIGIN,
+          ),
+        KING_COUNTY_IMAGE_ORIGIN,
+      );
+      if (!imageUrl) continue;
       const [lon, lat] = Array.isArray(row?.geometry?.coordinates)
         ? row.geometry.coordinates
         : [];
@@ -2586,8 +2598,8 @@ export async function loadWsdotSourcesFromOpenData() {
       const attrs = row?.attributes;
       const rawId = attrs?.CameraID;
       if (!Number.isSafeInteger(rawId) || rawId <= 0) continue;
-      const imageUrl = String(attrs.ImageURL || '').trim();
-      if (!imageUrl.startsWith(WSDOT_IMAGE_ORIGIN)) continue;
+      const imageUrl = pinnedCameraUrl(attrs.ImageURL, WSDOT_IMAGE_ORIGIN);
+      if (!imageUrl) continue;
       const x = row?.geometry?.x;
       const y = row?.geometry?.y;
       if (!Number.isFinite(x) || !Number.isFinite(y)) continue;

src/data/cctvSourcePins.test.mjs

@@ -0,0 +1,106 @@
+// Camera catalogs are pinned to their agency's image directory. The pin is a
+// parsed-URL check, not a string prefix, so a path that only starts with the
+// directory before dot segments, an encoded separator, credentials or another
+// port are resolved never counts as inside it, and a redirect on a pinned host
+// cannot leave the directory either.
+//
+// Run with: npm test   (node --test)
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import { loadTflSourcesFromOpenData } from '../../server/providers/cctv/sources.js';
+import { fetchWithinHost } from '../../server/providers/cctv/media.js';
+
+const BUCKET = 'https://s3-eu-west-1.amazonaws.com/jamcams.tfl.gov.uk/';
+
+/** One available JamCam whose frame is at `imageUrl`. */
+const jamCam = (n, imageUrl) => ({
+  id: `JamCams_00001.0000${n}`,
+  commonName: `JamCam ${n}`,
+  lat: 51.5072,
+  lon: -0.1276,
+  additionalProperties: [
+    { key: 'available', value: 'true' },
+    { key: 'imageUrl', value: imageUrl },
+  ],
+});
+
+test('TfL JamCams keep only frames inside the official bucket, by parsed URL', async (t) => {
+  t.mock.method(console, 'log', () => {});
+  t.mock.method(console, 'warn', () => {});
+  const accepted = [
+    [`${BUCKET}00001.00001.jpg`, `${BUCKET}00001.00001.jpg`],
+    [`${BUCKET}00001.00002.jpg?t=1`, `${BUCKET}00001.00002.jpg?t=1`],
+    [
+      'https://s3-eu-west-1.amazonaws.com:443/jamcams.tfl.gov.uk/00001.00003.jpg',
+      `${BUCKET}00001.00003.jpg`,
+    ],
+  ];
+  const rejected = [
+    `${BUCKET}../other/x.jpg`,
+    `${BUCKET}%2e%2e/other/x.jpg`,
+    `${BUCKET}%2E%2E/other/x.jpg`,
+    `${BUCKET}.%2e/other/x.jpg`,
+    `${BUCKET}..%2fother/x.jpg`,
+    `${BUCKET}..%5Cother/x.jpg`,
+    `${BUCKET}..\\other\\x.jpg`,
+    'https://s3-eu-west-1.amazonaws.com/jamcams.tfl.gov.uk.other/x.jpg',
+    'https://s3-eu-west-1.amazonaws.com/jamcams.tfl.gov.uk-other/x.jpg',
+    'https://s3-eu-west-1.amazonaws.com/other/jamcams.tfl.gov.uk/x.jpg',
+    'https://user:[email protected]/jamcams.tfl.gov.uk/x.jpg',
+    'https://s3-eu-west-1.amazonaws.com:8443/jamcams.tfl.gov.uk/x.jpg',
+    'http://s3-eu-west-1.amazonaws.com/jamcams.tfl.gov.uk/x.jpg',
+    'https://s3-eu-west-1.amazonaws.com.other.example/jamcams.tfl.gov.uk/x.jpg',
+    'not a url',
+  ];
+  t.mock.method(globalThis, 'fetch', async () =>
+    Response.json(
+      [...accepted.map(([url]) => url), ...rejected].map((url, i) =>
+        jamCam(i + 1, url),
+      ),
+    ),
+  );
+  const cameras = await loadTflSourcesFromOpenData();
+  assert.deepEqual(
+    cameras.map((camera) => camera.url),
+    accepted.map(([, normalized]) => normalized),
+  );
+});
+
+test('a redirect on a pinned camera host stays inside the pinned directory', async () => {
+  const redirectTo = (location) => async (url) =>
+    url === `${BUCKET}00001.00001.jpg`
+      ? new Response(null, { status: 302, headers: { location } })
+      : new Response('frame', { headers: { 'content-type': 'image/jpeg' } });
+  for (const location of [
+    '/jamcams.tfl.gov.uk.other/x.jpg',
+    '/other/x.jpg',
+    '/jamcams.tfl.gov.uk/../other/x.jpg',
+    '/jamcams.tfl.gov.uk/%2e%2e/other/x.jpg',
+    '/jamcams.tfl.gov.uk/..%2fother/x.jpg',
+  ])
+    assert.equal(
+      await fetchWithinHost(
+        `${BUCKET}00001.00001.jpg`,
+        {},
+        redirectTo(location),
+      ),
+      null,
+      location,
+    );
+  const followed = await fetchWithinHost(
+    `${BUCKET}00001.00001.jpg`,
+    {},
+    redirectTo('/jamcams.tfl.gov.uk/00001.00002.jpg'),
+  );
+  assert.equal(await followed.text(), 'frame');
+  // A host with no pinned directory keeps the same-origin rule alone.
+  const unpinned = await fetchWithinHost(
+    'https://cams.example/a/frame.jpg',
+    {},
+    async (url) =>
+      url === 'https://cams.example/a/frame.jpg'
+        ? new Response(null, { status: 302, headers: { location: '/b/x.jpg' } })
+        : new Response('frame'),
+  );
+  assert.equal(await unpinned.text(), 'frame');
+});