From 11e1c9d26d879110ac87a24d4394e805f74ca714 Mon Sep 17 00:00:00 2001 From: German Khokhlov Date: Thu, 19 Nov 2026 19:06:53 +0400 Subject: [PATCH] CCTV Street View fallback: the browser's pose at the catalog position The fallback keeps the catalog position but takes heading, field of view and pitch from the request again, as before, clamped as before. A catalog camera with no pose (every world-cameras row on main) otherwise asked Google for heading 0, pitch 0, FOV 80 while the browser draws it at the id-hash heading, pitch -17 and FOV 74, and a calibrated camera lost its calibration. Cost and the per-IP budget are unchanged. - The Street View budget is built per proxy, so tests no longer depend on GEV_RATELIMIT_GOOGLE_PER_MIN in the shell; the fixture pins it. - New tests: a pose-less catalog camera, and the per-IP budget (no call without a key, the cap, a separate budget per address). - The client no longer sends lat/lon on /frame. - CURRENT-STATE, the changelog fragment and SECURITY.md say the fallback has its own per-client limit, and that unlisted (seed) cameras get the placeholder. --- SECURITY.md | 4 +- .../1602-cctv-streetview-fallback-catalog.md | 10 +- docs/CURRENT-STATE.md | 11 +- server/providers/cctv.js | 51 ++++--- src/data/cctvMediaFeedType.test.mjs | 48 +++++- src/data/cctvStreetViewPose.test.mjs | 141 ++++++++++++++++++ src/layers/cctv/source.js | 2 - src/layers/cctv/source.test.mjs | 4 +- 8 files changed, 235 insertions(+), 36 deletions(-) create mode 100644 src/data/cctvStreetViewPose.test.mjs diff --git a/SECURITY.md b/SECURITY.md index 7b894c346f..fb2890e174 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -68,7 +68,7 @@ browser-key fallback for existing single-key setups. The data proxies under `server/providers/` are written so the browser cannot turn the server into an open relay: -- **No arbitrary-URL fetching.** The CCTV frame proxy fetches only server-registered camera/frame URLs — clients cannot pass an upstream URL to fetch (SSRF mitigation). Other proxies target fixed upstream hosts. +- **No arbitrary-URL fetching.** The CCTV frame proxy fetches only server-registered camera/frame URLs — clients cannot pass an upstream URL to fetch (SSRF mitigation). The key-bearing Street View fallback likewise answers only for a registered camera, always at its catalog position: a request sets only the view direction, field of view (20–120°) and pitch (−40–20°), so it cannot buy Street View of a place of its choosing. Other proxies target fixed upstream hosts. - **Live HLS is bounded and origin-pinned.** Registered HTTP(S) HLS sources use a Node puller that rejects redirects and off-origin playlist references, bounds playlists and segment bodies while streaming, and aborts timed-out or released sessions. At most two sessions retain 12 segments/24 MiB each, plus bounded download buffers. Stores are memory-only and idle sessions expire after 15 seconds. RTMP/ffmpeg execution is not enabled. Locally configured source URLs remain an operator trust boundary. - **Radio is not an audio relay.** `/api/radio/stations` contacts only allowlisted Radio Browser HTTPS hosts and paths, rejects redirects, rejects any hostname with a loopback/private/link-local/metadata/non-public A or AAAA result, and pins each TLS connection to a validated address. It returns normalized public HTTPS stream URLs; `/api/radio/click/:uuid` applies the same destination policy and accepts only station IDs from the current bounded catalog. The browser then connects directly to the broadcaster after an explicit playback action, so the broadcaster sees the listener's IP address. GEV never proxies, caches, records, or redistributes audio. - **No verbatim client headers upstream.** The CCTV media route is the one route that relays a request header (`Range`, for video seeking). It is parsed and canonicalized before it is forwarded: one `bytes=` range only, with every accepted form — explicit span, open-ended and suffix — bounded to the same 64 MiB ceiling the relay applies to a declared response body. Multi-range, malformed, inverted and non-`bytes` values are dropped and the request proceeds without a `Range`, as RFC 7233 §3.1 prescribes. Bounding the request bounds what is asked for: a response that declares no length — live streamed media, or a chunked body from an upstream that ignores the `Range` — has no ceiling. An upstream request the browser has stopped waiting for is cancelled rather than left running, whether the viewer leaves before the headers arrive or during the body. @@ -95,7 +95,7 @@ The dev server is a **key broker**: every server-side key above is spendable by - **Local-only by default.** `./scripts/dev-fresh.sh` (and the Vite config itself) bind to `localhost`, so only your machine can reach the server — and only local names are accepted (`allowedHosts` stays restricted, which also blunts DNS-rebinding tricks). Binding to all interfaces keeps that restriction: IP addresses and `localhost` work, and LAN hostnames must be listed in `GEV_ALLOWED_HOSTS` (for example `GEV_ALLOWED_HOSTS=globe.lan`); suffix and wildcard entries are ignored. - **LAN exposure is an explicit opt-in**: `HOST=0.0.0.0 ./scripts/dev-fresh.sh`. The launcher prints a prominent warning plus your LAN URL. Understand what opting in means: **every device on that network can drive the proxies and spend your OpenAI / Google / OpenSky / AISStream / TomTom / FIRMS quota** for as long as the server runs. Do this only on networks you trust. -- **App-level throttles (on by default):** `GEV_RATELIMIT_OPENAI_PER_MIN` (default `30`) and `GEV_RATELIMIT_GOOGLE_PER_MIN` (default `120`) — the values the Pinokio build already ships — cap the cost-bearing endpoints per client IP per minute (over-limit requests receive a sanitized `429`). Typed chat keeps its own per-IP window, set by `GEV_RATELIMIT_OPENAI_CHAT_PER_MIN` and `GEV_RATELIMIT_GEMINI_CHAT_PER_MIN`; left unset, each takes its provider's resolved cap (`GEV_RATELIMIT_OPENAI_PER_MIN`, or `GEV_RATELIMIT_GEMINI_PER_MIN`, default `30`), so lowering a provider throttle lowers chat too, while an explicit chat value, including `0`, wins. Both defaults sit well above what the app itself generates; set either to exactly `0` to run unthrottled. They are **per-IP, process-local, in-memory guards** — they reset on restart, a caller with several addresses gets a bucket per address, and they are **not billing caps**. +- **App-level throttles (on by default):** `GEV_RATELIMIT_OPENAI_PER_MIN` (default `30`) and `GEV_RATELIMIT_GOOGLE_PER_MIN` (default `120`) — the values the Pinokio build already ships — cap the cost-bearing endpoints per client IP per minute (over-limit requests receive a sanitized `429`). The CCTV Street View fallback has its own per-IP window on `GEV_RATELIMIT_GOOGLE_PER_MIN`, separate from the Places and Geocoding one; over it, a frame gets the placeholder instead of a `429`. Typed chat keeps its own per-IP window, set by `GEV_RATELIMIT_OPENAI_CHAT_PER_MIN` and `GEV_RATELIMIT_GEMINI_CHAT_PER_MIN`; left unset, each takes its provider's resolved cap (`GEV_RATELIMIT_OPENAI_PER_MIN`, or `GEV_RATELIMIT_GEMINI_PER_MIN`, default `30`), so lowering a provider throttle lowers chat too, while an explicit chat value, including `0`, wins. Both defaults sit well above what the app itself generates; set either to exactly `0` to run unthrottled. They are **per-IP, process-local, in-memory guards** — they reset on restart, a caller with several addresses gets a bucket per address, and they are **not billing caps**. - **Docker trusts one address, and only because the port is on loopback.** Docker delivers the host's own browser to the container from the container network's gateway, never from `127.0.0.1`, so a loopback-only Provider diff --git a/changelog.d/1602-cctv-streetview-fallback-catalog.md b/changelog.d/1602-cctv-streetview-fallback-catalog.md index 04574b9142..194923b264 100644 --- a/changelog.d/1602-cctv-streetview-fallback-catalog.md +++ b/changelog.d/1602-cctv-streetview-fallback-catalog.md @@ -1,5 +1,5 @@ -- The CCTV Street View fallback serves only catalog cameras, at their catalog - position, heading, field of view and pitch, and ignores coordinates a - request supplies. It spends from the per-client Google budget - (`GEV_RATELIMIT_GOOGLE_PER_MIN`, default 120), so the Google key can no - longer be used as an unmetered Street View service (asturksever, #1602). +- The CCTV Street View fallback serves only catalog cameras, always at their + catalog position, and ignores coordinates a request supplies. It has its own + per-client limit on the Google setting (`GEV_RATELIMIT_GOOGLE_PER_MIN`, + default 120), so the Google key can no longer be used as an unmetered Street + View service (asturksever, #1602). diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 019e25704e..d7f60479ed 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1277,9 +1277,14 @@ redirects on those hosts, are pinned by parsed URL to the agency's image directory. Every `/api/cctv` route refuses cross-site browser requests but accepts forwarding headers, so cameras work behind a reverse proxy; a forwarded request skips the key-bearing Street View fallback and gets the placeholder. The -Street View fallback serves only catalog cameras, at the catalog position, -heading, field of view and pitch, never coordinates a request supplies, and it -shares the per-client Google budget (`GEV_RATELIMIT_GOOGLE_PER_MIN`, default 120). +Street View fallback serves only catalog cameras, always at the catalog +position; coordinates a request supplies are ignored. Heading, field of view +and pitch come from the request, the pose the browser draws with calibration +included, and fall back to the catalog pose. A camera the server does not list, +such as a bundled seed camera the browser shows when the server catalog is +empty, gets the placeholder. The fallback has its own per-client limit on the +Google setting (`GEV_RATELIMIT_GOOGLE_PER_MIN`, default 120, `0` disables it); +a request over it gets the placeholder. The CCTV layer draws only the cameras on or near the screen. Zoomed out, it keeps one camera per ground cell sized to the view, preferring a healthy feed, diff --git a/server/providers/cctv.js b/server/providers/cctv.js index e8e18ad51e..3eb94dbb2f 100644 --- a/server/providers/cctv.js +++ b/server/providers/cctv.js @@ -21,19 +21,6 @@ import { createHlsPuller } from './cctv/stream.js'; import { googleServerApiKey } from './places/google-key.js'; import { clientKey, makeCostRateLimiter } from './common/rate-limit.js'; export { CCTV_FRAME_FETCH_TIMEOUT_MS, fetchCctvImageFromUpstream }; -let _streetViewRateLimiter; -/** - * Per-IP budget for the Street View fallback, on the same setting as the - * other Google routes (GEV_RATELIMIT_GOOGLE_PER_MIN; 0 disables it). - */ -function streetViewRateLimiter() { - if (_streetViewRateLimiter === undefined) - _streetViewRateLimiter = makeCostRateLimiter( - process.env.GEV_RATELIMIT_GOOGLE_PER_MIN, - 120, - ); - return _streetViewRateLimiter; -} /** * Vite plugin: CCTV camera proxy with source registry, frame/media serving, @@ -96,6 +83,22 @@ export function cctvProxy({ sourceRoot = process.cwd() } = {}) { }; }; + /** + * Per-IP budget for the Street View fallback, one per proxy, on the same + * setting as the other Google routes (GEV_RATELIMIT_GOOGLE_PER_MIN; 0 + * disables it). Built on first use, after the standalone env has loaded. + * undefined = not built yet; null = the explicit 0 opt-out. + */ + let streetViewBudget; + const streetViewRateLimiter = () => { + if (streetViewBudget === undefined) + streetViewBudget = makeCostRateLimiter( + process.env.GEV_RATELIMIT_GOOGLE_PER_MIN, + 120, + ); + return streetViewBudget; + }; + /** * Fetch a Google Street View static image as a fallback frame. Server-side * call, never reaches the browser — prefers GOOGLE_MAPS_SERVER_API_KEY @@ -524,6 +527,13 @@ export function cctvProxy({ sourceRoot = process.cwd() } = {}) { const source = sourceById.get(cameraId); const label = url.searchParams.get('label') || source?.name || cameraId; const city = url.searchParams.get('city') || source?.city || ''; + // The view follows the pose the browser draws (calibration included); + // the position never comes from the query (see the Street View fallback). + const heading = Number( + url.searchParams.get('heading') || source?.headingDeg, + ); + const fov = Number(url.searchParams.get('fov') || source?.fovDeg); + const pitch = Number(url.searchParams.get('pitch') || source?.pitchDeg); // Only use server-registered upstream URLs — never accept client-supplied URLs // (prevents SSRF via ?upstream= query parameter) @@ -557,8 +567,9 @@ export function cctvProxy({ sourceRoot = process.cwd() } = {}) { // request (whose client this server cannot see) gets the placeholder, // as if no key were configured. Any forwarding header counts, even an // empty one. - // It is also only for catalog cameras, framed from the catalog (never - // a spot the query names), within the per-IP Google budget. + // It is also only for catalog cameras, at the catalog position (never a + // spot the query names; the query sets only the view), within the per-IP + // Google budget. const forwarded = PROXY_SIGNALS.some( (name) => req.headers?.[name] !== undefined, ); @@ -571,11 +582,11 @@ export function cctvProxy({ sourceRoot = process.cwd() } = {}) { (googleBudget && !googleBudget(clientKey(req))) ? null : await streetViewFallback({ - lat: Number(source.lat), - lon: Number(source.lon), - heading: Number(source.headingDeg), - fov: Number(source.fovDeg), - pitch: Number(source.pitchDeg ?? -10), + lat: source.lat, + lon: source.lon, + heading, + fov, + pitch, }); if (sv?.ok) { setHealth(cameraId, { diff --git a/src/data/cctvMediaFeedType.test.mjs b/src/data/cctvMediaFeedType.test.mjs index 0523c95f39..191b3015ed 100644 --- a/src/data/cctvMediaFeedType.test.mjs +++ b/src/data/cctvMediaFeedType.test.mjs @@ -111,6 +111,7 @@ function mount(t, respond) { austin: process.env.CCTV_FORCE_AUSTIN, serverKey: process.env.GOOGLE_MAPS_SERVER_API_KEY, browserKey: process.env.GOOGLE_MAPS_API_KEY, + rateLimit: process.env.GEV_RATELIMIT_GOOGLE_PER_MIN, }; process.env.CCTV_SOURCES_JSON = JSON.stringify([ STILL, @@ -125,6 +126,8 @@ function mount(t, respond) { // than a network call this file does not model. delete process.env.GOOGLE_MAPS_SERVER_API_KEY; delete process.env.GOOGLE_MAPS_API_KEY; + // The default Google budget, whatever the shell sets. + delete process.env.GEV_RATELIMIT_GOOGLE_PER_MIN; const nativeFetch = globalThis.fetch; const requests = []; globalThis.fetch = async (url, init = {}) => { @@ -139,6 +142,7 @@ function mount(t, respond) { ['CCTV_FORCE_AUSTIN', before.austin], ['GOOGLE_MAPS_SERVER_API_KEY', before.serverKey], ['GOOGLE_MAPS_API_KEY', before.browserKey], + ['GEV_RATELIMIT_GOOGLE_PER_MIN', before.rateLimit], ]) { if (value === undefined) delete process.env[name]; else process.env[name] = value; @@ -155,9 +159,9 @@ function mount(t, respond) { }); assert.ok(handler, 'the CCTV provider must mount a middleware'); - const call = async (path, headers = {}) => { + const call = async (path, headers = {}, socket) => { const { res, done } = recordingResponse(); - await handler({ url: path, headers, method: 'GET', on() {} }, res); + await handler({ url: path, headers, method: 'GET', on() {}, socket }, res); await done; return res; }; @@ -554,4 +558,44 @@ test('the Street View fallback frames catalog cameras only, from the catalog pos assert.equal(steered.headers['X-CCTV-Source'], 'streetview'); const location = new URL(streetView()[0].url).searchParams.get('location'); assert.equal(location, `${STILL.lat},${STILL.lon}`); + assert.equal(new URL(streetView()[0].url).searchParams.get('heading'), '90'); +}); + +test('the Street View fallback spends a per-IP Google budget, and only with a key', async (t) => { + const app = mount(t, ({ url }) => + url.startsWith('https://maps.googleapis.com/') + ? trackedUpstream({ + headers: { 'content-type': 'image/jpeg' }, + body: 'street-view', + }).response + : upstreamResponse({ status: 503 }), + ); + // Read when the first keyed fallback builds the limiter, so after mount(). + process.env.GEV_RATELIMIT_GOOGLE_PER_MIN = '2'; + const streetViewCalls = () => + app.requests.filter(({ url }) => + url.startsWith('https://maps.googleapis.com/'), + ).length; + const first = { remoteAddress: '203.0.113.7' }; + const frameSource = async (socket) => + (await app.call(`/frame/${STILL.id}`, {}, socket)).headers['X-CCTV-Source']; + + // Without a key nothing is spent. + for (let i = 0; i < 3; i++) + assert.equal(await frameSource(first), 'synthetic'); + assert.equal(streetViewCalls(), 0); + + process.env.GOOGLE_MAPS_SERVER_API_KEY = 'fixture-key'; + assert.deepEqual( + [await frameSource(first), await frameSource(first), await frameSource(first)], + ['streetview', 'streetview', 'synthetic'], + ); + assert.equal(streetViewCalls(), 2, 'no Google call over the budget'); + + // Another address has a budget of its own. + assert.equal( + await frameSource({ remoteAddress: '203.0.113.8' }), + 'streetview', + ); + assert.equal(streetViewCalls(), 3); }); diff --git a/src/data/cctvStreetViewPose.test.mjs b/src/data/cctvStreetViewPose.test.mjs new file mode 100644 index 0000000000..584fe8629d --- /dev/null +++ b/src/data/cctvStreetViewPose.test.mjs @@ -0,0 +1,141 @@ +// When a catalog camera has no pose, the browser sends /frame a heading derived +// from the camera ID, a pitch of -17°, and a field of view of 74°. Street View +// uses that pose at the camera's catalog coordinates. +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { cctvProxy } from '../../server/providers/cctv.js'; + +// Same shape as a world-cameras row: an image URL, position, and no +// headingDeg / pitchDeg / fovDeg. +const POSELESS = { + id: 'world-or-fixture', + name: 'Pose-less fixture', + lat: 44.05, + lon: -123.09, + feedType: 'image', + url: 'https://cams.upstream.invalid/still.jpg', +}; + +/** The browser's id-hash heading (src/layers/cctv/model.js headingFromId). */ +function browserHeadingFromId(id) { + let acc = 0; + for (let i = 0; i < id.length; i++) acc = (acc * 33 + id.charCodeAt(i)) >>> 0; + return ((((acc % 16) * 22.5) % 360) + 360) % 360; +} + +function response({ status = 200, headers = {}, body = '' }) { + return new Response(status === 204 ? null : String(body), { + status, + headers, + }); +} + +function recording() { + const chunks = []; + let finished; + const done = new Promise((resolve) => (finished = resolve)); + const res = { + statusCode: 0, + headers: {}, + writableEnded: false, + writeHead(status, headers) { + res.statusCode = status; + res.headers = { ...res.headers, ...headers }; + }, + setHeader(name, value) { + res.headers[name] = value; + }, + write(chunk) { + chunks.push(Buffer.from(chunk)); + return true; + }, + end(chunk) { + if (chunk) chunks.push(Buffer.from(chunk)); + res.writableEnded = true; + res.body = Buffer.concat(chunks).toString('utf8'); + finished(); + }, + on() {}, + once() {}, + emit() {}, + destroy() {}, + }; + return { res, done }; +} + +test('the Street View fallback frames a pose-less catalog camera the way the browser draws it', async (t) => { + const saved = Object.fromEntries( + [ + 'CCTV_SOURCES_JSON', + 'CCTV_SOURCES_FILE', + 'CCTV_FORCE_AUSTIN', + 'GOOGLE_MAPS_SERVER_API_KEY', + 'GOOGLE_MAPS_API_KEY', + 'GEV_RATELIMIT_GOOGLE_PER_MIN', + ].map((name) => [name, process.env[name]]), + ); + process.env.CCTV_SOURCES_JSON = JSON.stringify([POSELESS]); + process.env.CCTV_SOURCES_FILE = 'absent-source-file.json'; + process.env.CCTV_FORCE_AUSTIN = '0'; + process.env.GOOGLE_MAPS_SERVER_API_KEY = 'fixture-key'; + delete process.env.GOOGLE_MAPS_API_KEY; + delete process.env.GEV_RATELIMIT_GOOGLE_PER_MIN; + const nativeFetch = globalThis.fetch; + const streetView = []; + globalThis.fetch = async (url) => { + const href = String(url); + if (href.startsWith('https://maps.googleapis.com/')) { + streetView.push(new URL(href)); + return response({ + headers: { 'content-type': 'image/jpeg' }, + body: 'street-view', + }); + } + return response({ status: 503 }); // the camera's upstream is down + }; + t.after(() => { + globalThis.fetch = nativeFetch; + for (const [name, value] of Object.entries(saved)) { + if (value === undefined) delete process.env[name]; + else process.env[name] = value; + } + }); + + let handler; + cctvProxy().configureServer({ + middlewares: { use: (_route, fn) => (handler = fn) }, + }); + + // What src/layers/cctv/source.js sends for this camera with no calibration. + const heading = Math.round(browserHeadingFromId(POSELESS.id)); + const params = new URLSearchParams({ + label: POSELESS.name, + city: '', + lat: POSELESS.lat.toFixed(6), + lon: POSELESS.lon.toFixed(6), + heading: String(heading), + fov: '74', + pitch: '-17', + ts: '1', + }); + const { res, done } = recording(); + await handler( + { + url: `/frame/${POSELESS.id}?${params}`, + headers: {}, + method: 'GET', + on() {}, + }, + res, + ); + await done; + + assert.equal(res.headers['X-CCTV-Source'], 'streetview'); + assert.equal(streetView.length, 1); + const sv = streetView[0].searchParams; + assert.equal(sv.get('location'), `${POSELESS.lat},${POSELESS.lon}`); + assert.equal(Number(sv.get('heading')), heading, 'heading the browser draws'); + assert.equal(Number(sv.get('fov')), 74, 'FOV the browser draws'); + assert.equal(Number(sv.get('pitch')), -17, 'pitch the browser draws'); +}); diff --git a/src/layers/cctv/source.js b/src/layers/cctv/source.js index 3b628954b4..78fd5cb194 100644 --- a/src/layers/cctv/source.js +++ b/src/layers/cctv/source.js @@ -22,8 +22,6 @@ function frameUrlFor(camera, refreshMs = ACTIVE_FRAME_REFRESH_MS) { const params = new URLSearchParams({ label: camera.name, city: camera.city, - lat: camera.lat.toFixed(6), - lon: camera.lon.toFixed(6), heading: String(Math.round(camera.headingDeg)), fov: String(Math.round(camera.fovDeg)), pitch: String(Math.round(camera.pitchDeg || -10)), diff --git a/src/layers/cctv/source.test.mjs b/src/layers/cctv/source.test.mjs index a4b276d213..61e36c3321 100644 --- a/src/layers/cctv/source.test.mjs +++ b/src/layers/cctv/source.test.mjs @@ -81,8 +81,8 @@ test('frame and media URLs preserve registered camera identity and encoded metad ); assert.equal(frame.searchParams.get('label'), camera.name); assert.equal(frame.searchParams.get('city'), camera.city); - assert.equal(frame.searchParams.get('lat'), '30.267000'); - assert.equal(frame.searchParams.get('lon'), '-97.744000'); + assert.equal(frame.searchParams.has('lat'), false); + assert.equal(frame.searchParams.has('lon'), false); assert.equal(frame.searchParams.get('heading'), '45'); assert.equal(frame.searchParams.get('pitch'), '-12'); assert.deepEqual([...media.searchParams.keys()], ['ts']);