Compare commits

...
Author SHA1 Message Date
German Khokhlov 01368b95bf Merge remote-tracking branch 'origin/main' into fix/cctv-streetview-fallback-catalog
# Conflicts:
#	SECURITY.md
2026-11-19 19:12:56 +04:00
German Khokhlov 11e1c9d26d CCTV Street View fallback: the browser's pose at the catalog position
The fallback keeps the catalog position but takes heading, field of view
and pitch from the request again, as before, clamped as before. A catalog
camera with no pose (every world-cameras row on main) otherwise asked
Google for heading 0, pitch 0, FOV 80 while the browser draws it at the
id-hash heading, pitch -17 and FOV 74, and a calibrated camera lost its
calibration. Cost and the per-IP budget are unchanged.

- The Street View budget is built per proxy, so tests no longer depend on
  GEV_RATELIMIT_GOOGLE_PER_MIN in the shell; the fixture pins it.
- New tests: a pose-less catalog camera, and the per-IP budget (no call
  without a key, the cap, a separate budget per address).
- The client no longer sends lat/lon on /frame.
- CURRENT-STATE, the changelog fragment and SECURITY.md say the fallback
  has its own per-client limit, and that unlisted (seed) cameras get the
  placeholder.
2026-11-19 19:06:53 +04:00
Said Turksever f579b23f52 Changelog: 1602 fragment 2026-11-06 11:59:32 +00:00
Said Turksever f0d3db3164 Docs: the CCTV Street View fallback serves catalog cameras only, inside the Google budget 2026-11-06 11:57:58 +00:00
Said Turksever 078a6c0db0 CCTV Street View fallback: own per-IP Google budget inside the cctv package boundary 2026-11-06 11:51:34 +00:00
Said Turksever 7af0453f34 CCTV Street View fallback: catalog cameras only, at their catalog position, inside the Google budget 2026-11-06 11:51:34 +00:00
9 changed files with 267 additions and 15 deletions

No files matched your search

+2 -2
View File
@@ -68,7 +68,7 @@ browser-key fallback for existing single-key setups.
The data proxies under `server/providers/` are written so the browser cannot turn the server into an open relay:
- **No arbitrary-URL fetching.** The CCTV frame proxy fetches only server-registered camera/frame URLs — clients cannot pass an upstream URL to fetch (SSRF mitigation). Other proxies target fixed upstream hosts.
- **No arbitrary-URL fetching.** The CCTV frame proxy fetches only server-registered camera/frame URLs — clients cannot pass an upstream URL to fetch (SSRF mitigation). The key-bearing Street View fallback likewise answers only for a registered camera, always at its catalog position: a request sets only the view direction, field of view (20–120°) and pitch (−40–20°), so it cannot buy Street View of a place of its choosing. Other proxies target fixed upstream hosts.
- **Live HLS is bounded and origin-pinned.** Registered HTTP(S) HLS sources use a Node puller that rejects redirects and off-origin playlist references, bounds playlists to 256 KiB and segment bodies to 8 MiB while streaming, and aborts timed-out or released sessions. At most two sessions retain 12 segments/24 MiB each, plus bounded download buffers. Stores are memory-only and idle sessions expire after 15 seconds. RTMP/ffmpeg execution is not enabled. Locally configured source URLs remain an operator trust boundary.
- **Radio is not an audio relay.** `/api/radio/stations` contacts only allowlisted Radio Browser HTTPS hosts and paths, rejects redirects, rejects any hostname with a loopback/private/link-local/metadata/non-public A or AAAA result, and pins each TLS connection to a validated address. It returns normalized public HTTPS stream URLs; `/api/radio/click/:uuid` applies the same destination policy and accepts only station IDs from the current bounded catalog. The browser then connects directly to the broadcaster after an explicit playback action, so the broadcaster sees the listener's IP address. GEV never proxies, caches, records, or redistributes audio.
- **Québec 511 video plays in the browser, opt-in and after Play.** With `CCTV_QUEBEC_VIDEO_ENABLED=1`, `/api/cctv/sources` lists, for each MTQ camera, one URL built server-side from its numeric id on `https://www.quebec511.info/Carte/Fenetres/camera.ashx`; the overlay accepts no other host, path or format. The browser loads that clip only after the viewer presses Play for the selected camera, so Québec 511 sees the viewer's IP address, and the panel says so before Play. Selecting a camera by click, auto-hop, a scene or voice never loads it, and switching cameras stops playback. GEV never proxies, caches or reads the video's pixels. With the flag unset (the default) the browser makes no request to Québec 511.
@@ -98,7 +98,7 @@ The dev server is a **key broker**: every server-side key above is spendable by
- **Local-only by default.** `./scripts/dev-fresh.sh` (and the Vite config itself) bind to `localhost`, so only your machine can reach the server — and only local names are accepted (`allowedHosts` stays restricted, which also blunts DNS-rebinding tricks). Binding to all interfaces keeps that restriction: IP addresses and `localhost` work, and LAN hostnames must be listed in `GEV_ALLOWED_HOSTS` (for example `GEV_ALLOWED_HOSTS=globe.lan`); suffix and wildcard entries are ignored.
- **LAN exposure is an explicit opt-in**: `HOST=0.0.0.0 ./scripts/dev-fresh.sh`. The launcher prints a prominent warning plus your LAN URL. Understand what opting in means: **every device on that network can drive the proxies and spend your OpenAI / Google / OpenSky / AISStream / TomTom / FIRMS quota** for as long as the server runs. Do this only on networks you trust.
- **App-level throttles (on by default):** `GEV_RATELIMIT_OPENAI_PER_MIN` (default `30`) and `GEV_RATELIMIT_GOOGLE_PER_MIN` (default `120`) — the values the Pinokio build already ships — cap the cost-bearing endpoints per client IP per minute (over-limit requests receive a sanitized `429`). Typed chat keeps its own per-IP window, set by `GEV_RATELIMIT_OPENAI_CHAT_PER_MIN` and `GEV_RATELIMIT_GEMINI_CHAT_PER_MIN`; left unset, each takes its provider's resolved cap (`GEV_RATELIMIT_OPENAI_PER_MIN`, or `GEV_RATELIMIT_GEMINI_PER_MIN`, default `30`), so lowering a provider throttle lowers chat too, while an explicit chat value, including `0`, wins. Both defaults sit well above what the app itself generates; set either to exactly `0` to run unthrottled. They are **per-IP, process-local, in-memory guards** — they reset on restart, a caller with several addresses gets a bucket per address, and they are **not billing caps**.
- **App-level throttles (on by default):** `GEV_RATELIMIT_OPENAI_PER_MIN` (default `30`) and `GEV_RATELIMIT_GOOGLE_PER_MIN` (default `120`) — the values the Pinokio build already ships — cap the cost-bearing endpoints per client IP per minute (over-limit requests receive a sanitized `429`). The CCTV Street View fallback has its own per-IP window on `GEV_RATELIMIT_GOOGLE_PER_MIN`, separate from the Places and Geocoding one; over it, a frame gets the placeholder instead of a `429`. Typed chat keeps its own per-IP window, set by `GEV_RATELIMIT_OPENAI_CHAT_PER_MIN` and `GEV_RATELIMIT_GEMINI_CHAT_PER_MIN`; left unset, each takes its provider's resolved cap (`GEV_RATELIMIT_OPENAI_PER_MIN`, or `GEV_RATELIMIT_GEMINI_PER_MIN`, default `30`), so lowering a provider throttle lowers chat too, while an explicit chat value, including `0`, wins. Both defaults sit well above what the app itself generates; set either to exactly `0` to run unthrottled. They are **per-IP, process-local, in-memory guards** — they reset on restart, a caller with several addresses gets a bucket per address, and they are **not billing caps**.
- **Docker trusts one address, and only because the port is on loopback.**
Docker delivers the host's own browser to the container from the container
network's gateway, never from `127.0.0.1`, so a loopback-only Provider
@@ -0,0 +1,5 @@
- The CCTV Street View fallback serves only catalog cameras, always at their
catalog position, and ignores coordinates a request supplies. It has its own
per-client limit on the Google setting (`GEV_RATELIMIT_GOOGLE_PER_MIN`,
default 120), so the Google key can no longer be used as an unmetered Street
View service (asturksever, #1602).
+9 -1
View File
@@ -1989,7 +1989,15 @@ and follows redirects only on the camera's origin. Catalog image URLs, and
redirects on those hosts, are pinned by parsed URL to the agency's image
directory. Every `/api/cctv` route refuses cross-site browser requests but
accepts forwarding headers, so cameras work behind a reverse proxy; a forwarded
request skips the key-bearing Street View fallback and gets the placeholder.
request skips the key-bearing Street View fallback and gets the placeholder. The
Street View fallback serves only catalog cameras, always at the catalog
position; coordinates a request supplies are ignored. Heading, field of view
and pitch come from the request, the pose the browser draws with calibration
included, and fall back to the catalog pose. A camera the server does not list,
such as a bundled seed camera the browser shows when the server catalog is
empty, gets the placeholder. The fallback has its own per-client limit on the
Google setting (`GEV_RATELIMIT_GOOGLE_PER_MIN`, default 120, `0` disables it);
a request over it gets the placeholder.
The CCTV layer draws only the cameras on or near the screen. Zoomed out, it
keeps one camera per ground cell sized to the view, preferring a healthy feed,
+3 -1
View File
@@ -581,7 +581,9 @@
"src/data/cctvFootprint.js",
"src/sources/cctvTypes.js",
"src/sources/httpBody.js",
"src/i18n/marker.js"
"src/i18n/marker.js",
"server/providers/common/rate-limit.js",
"src/sources/rateLimit.js"
],
"external": []
},
+32 -5
View File
@@ -22,7 +22,9 @@ import { createHlsPuller } from './cctv/stream.js';
import { isMlitRiverAdjustmentImage } from './cctv/worldCameras.js';
import { ottawaAuthorizedFrameUrl } from './cctv/ottawa.js';
import { googleServerApiKey } from './places/google-key.js';
import { clientKey, makeCostRateLimiter } from './common/rate-limit.js';
export { CCTV_FRAME_FETCH_TIMEOUT_MS, fetchCctvImageFromUpstream };
/**
* Vite plugin: CCTV camera proxy with source registry, frame/media serving,
* fallback chain (upstream -> Street View -> synthetic SVG), and health tracking.
@@ -198,6 +200,22 @@ export function cctvProxy({ sourceRoot = process.cwd() } = {}) {
};
};
/**
* Per-IP budget for the Street View fallback, one per proxy, on the same
* setting as the other Google routes (GEV_RATELIMIT_GOOGLE_PER_MIN; 0
* disables it). Built on first use, after the standalone env has loaded.
* undefined = not built yet; null = the explicit 0 opt-out.
*/
let streetViewBudget;
const streetViewRateLimiter = () => {
if (streetViewBudget === undefined)
streetViewBudget = makeCostRateLimiter(
process.env.GEV_RATELIMIT_GOOGLE_PER_MIN,
120,
);
return streetViewBudget;
};
/**
* Fetch a Google Street View static image as a fallback frame. Server-side
* call, never reaches the browser — prefers GOOGLE_MAPS_SERVER_API_KEY
@@ -635,8 +653,8 @@ export function cctvProxy({ sourceRoot = process.cwd() } = {}) {
const source = sourceById.get(cameraId);
const label = url.searchParams.get('label') || source?.name || cameraId;
const city = url.searchParams.get('city') || source?.city || '';
const lat = Number(url.searchParams.get('lat') || source?.lat);
const lon = Number(url.searchParams.get('lon') || source?.lon);
// The view follows the pose the browser draws (calibration included);
// the position never comes from the query (see the Street View fallback).
const heading = Number(
url.searchParams.get('heading') || source?.headingDeg,
);
@@ -731,14 +749,23 @@ export function cctvProxy({ sourceRoot = process.cwd() } = {}) {
// request (whose client this server cannot see) gets the placeholder,
// as if no key were configured. Any forwarding header counts, even an
// empty one.
// It is also only for catalog cameras, at the catalog position (never a
// spot the query names; the query sets only the view), within the per-IP
// Google budget.
const forwarded = PROXY_SIGNALS.some(
(name) => req.headers?.[name] !== undefined,
);
const sv = forwarded
const googleBudget = googleServerApiKey()
? streetViewRateLimiter()
: null;
const sv =
forwarded ||
!source ||
(googleBudget && !googleBudget(clientKey(req)))
? null
: await streetViewFallback({
lat,
lon,
lat: source.lat,
lon: source.lon,
heading,
fov,
pitch,
+73 -2
View File
@@ -111,6 +111,7 @@ function mount(t, respond) {
austin: process.env.CCTV_FORCE_AUSTIN,
serverKey: process.env.GOOGLE_MAPS_SERVER_API_KEY,
browserKey: process.env.GOOGLE_MAPS_API_KEY,
rateLimit: process.env.GEV_RATELIMIT_GOOGLE_PER_MIN,
};
process.env.CCTV_SOURCES_JSON = JSON.stringify([
STILL,
@@ -125,6 +126,8 @@ function mount(t, respond) {
// than a network call this file does not model.
delete process.env.GOOGLE_MAPS_SERVER_API_KEY;
delete process.env.GOOGLE_MAPS_API_KEY;
// The default Google budget, whatever the shell sets.
delete process.env.GEV_RATELIMIT_GOOGLE_PER_MIN;
const nativeFetch = globalThis.fetch;
const requests = [];
globalThis.fetch = async (url, init = {}) => {
@@ -139,6 +142,7 @@ function mount(t, respond) {
['CCTV_FORCE_AUSTIN', before.austin],
['GOOGLE_MAPS_SERVER_API_KEY', before.serverKey],
['GOOGLE_MAPS_API_KEY', before.browserKey],
['GEV_RATELIMIT_GOOGLE_PER_MIN', before.rateLimit],
]) {
if (value === undefined) delete process.env[name];
else process.env[name] = value;
@@ -155,9 +159,9 @@ function mount(t, respond) {
});
assert.ok(handler, 'the CCTV provider must mount a middleware');
const call = async (path, headers = {}) => {
const call = async (path, headers = {}, socket) => {
const { res, done } = recordingResponse();
await handler({ url: path, headers, method: 'GET', on() {} }, res);
await handler({ url: path, headers, method: 'GET', on() {}, socket }, res);
await done;
return res;
};
@@ -528,3 +532,70 @@ test('a forwarded same-origin request reaches every CCTV route but not the Stree
assert.equal(direct.body, 'street-view');
assert.equal(streetViewCalls(), 1);
});
test('the Street View fallback frames catalog cameras only, from the catalog position', async (t) => {
const app = mount(t, ({ url }) =>
url.startsWith('https://maps.googleapis.com/')
? trackedUpstream({
headers: { 'content-type': 'image/jpeg' },
body: 'street-view',
}).response
: upstreamResponse({ status: 503 }),
);
process.env.GOOGLE_MAPS_SERVER_API_KEY = 'fixture-key';
const streetView = () =>
app.requests.filter(({ url }) =>
url.startsWith('https://maps.googleapis.com/'),
);
const unknown = await app.call('/frame/not-a-camera?lat=48.85&lon=2.29');
assert.equal(unknown.headers['X-CCTV-Source'], 'synthetic');
assert.deepEqual(streetView(), [], 'no Google call for an unknown id');
const steered = await app.call(
`/frame/${STILL.id}?lat=48.85&lon=2.29&heading=90`,
);
assert.equal(steered.headers['X-CCTV-Source'], 'streetview');
const location = new URL(streetView()[0].url).searchParams.get('location');
assert.equal(location, `${STILL.lat},${STILL.lon}`);
assert.equal(new URL(streetView()[0].url).searchParams.get('heading'), '90');
});
test('the Street View fallback spends a per-IP Google budget, and only with a key', async (t) => {
const app = mount(t, ({ url }) =>
url.startsWith('https://maps.googleapis.com/')
? trackedUpstream({
headers: { 'content-type': 'image/jpeg' },
body: 'street-view',
}).response
: upstreamResponse({ status: 503 }),
);
// Read when the first keyed fallback builds the limiter, so after mount().
process.env.GEV_RATELIMIT_GOOGLE_PER_MIN = '2';
const streetViewCalls = () =>
app.requests.filter(({ url }) =>
url.startsWith('https://maps.googleapis.com/'),
).length;
const first = { remoteAddress: '203.0.113.7' };
const frameSource = async (socket) =>
(await app.call(`/frame/${STILL.id}`, {}, socket)).headers['X-CCTV-Source'];
// Without a key nothing is spent.
for (let i = 0; i < 3; i++)
assert.equal(await frameSource(first), 'synthetic');
assert.equal(streetViewCalls(), 0);
process.env.GOOGLE_MAPS_SERVER_API_KEY = 'fixture-key';
assert.deepEqual(
[await frameSource(first), await frameSource(first), await frameSource(first)],
['streetview', 'streetview', 'synthetic'],
);
assert.equal(streetViewCalls(), 2, 'no Google call over the budget');
// Another address has a budget of its own.
assert.equal(
await frameSource({ remoteAddress: '203.0.113.8' }),
'streetview',
);
assert.equal(streetViewCalls(), 3);
});
+141
View File
@@ -0,0 +1,141 @@
// When a catalog camera has no pose, the browser sends /frame a heading derived
// from the camera ID, a pitch of -17°, and a field of view of 74°. Street View
// uses that pose at the camera's catalog coordinates.
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { cctvProxy } from '../../server/providers/cctv.js';
// Same shape as a world-cameras row: an image URL, position, and no
// headingDeg / pitchDeg / fovDeg.
const POSELESS = {
id: 'world-or-fixture',
name: 'Pose-less fixture',
lat: 44.05,
lon: -123.09,
feedType: 'image',
url: 'https://cams.upstream.invalid/still.jpg',
};
/** The browser's id-hash heading (src/layers/cctv/model.js headingFromId). */
function browserHeadingFromId(id) {
let acc = 0;
for (let i = 0; i < id.length; i++) acc = (acc * 33 + id.charCodeAt(i)) >>> 0;
return ((((acc % 16) * 22.5) % 360) + 360) % 360;
}
function response({ status = 200, headers = {}, body = '' }) {
return new Response(status === 204 ? null : String(body), {
status,
headers,
});
}
function recording() {
const chunks = [];
let finished;
const done = new Promise((resolve) => (finished = resolve));
const res = {
statusCode: 0,
headers: {},
writableEnded: false,
writeHead(status, headers) {
res.statusCode = status;
res.headers = { ...res.headers, ...headers };
},
setHeader(name, value) {
res.headers[name] = value;
},
write(chunk) {
chunks.push(Buffer.from(chunk));
return true;
},
end(chunk) {
if (chunk) chunks.push(Buffer.from(chunk));
res.writableEnded = true;
res.body = Buffer.concat(chunks).toString('utf8');
finished();
},
on() {},
once() {},
emit() {},
destroy() {},
};
return { res, done };
}
test('the Street View fallback frames a pose-less catalog camera the way the browser draws it', async (t) => {
const saved = Object.fromEntries(
[
'CCTV_SOURCES_JSON',
'CCTV_SOURCES_FILE',
'CCTV_FORCE_AUSTIN',
'GOOGLE_MAPS_SERVER_API_KEY',
'GOOGLE_MAPS_API_KEY',
'GEV_RATELIMIT_GOOGLE_PER_MIN',
].map((name) => [name, process.env[name]]),
);
process.env.CCTV_SOURCES_JSON = JSON.stringify([POSELESS]);
process.env.CCTV_SOURCES_FILE = 'absent-source-file.json';
process.env.CCTV_FORCE_AUSTIN = '0';
process.env.GOOGLE_MAPS_SERVER_API_KEY = 'fixture-key';
delete process.env.GOOGLE_MAPS_API_KEY;
delete process.env.GEV_RATELIMIT_GOOGLE_PER_MIN;
const nativeFetch = globalThis.fetch;
const streetView = [];
globalThis.fetch = async (url) => {
const href = String(url);
if (href.startsWith('https://maps.googleapis.com/')) {
streetView.push(new URL(href));
return response({
headers: { 'content-type': 'image/jpeg' },
body: 'street-view',
});
}
return response({ status: 503 }); // the camera's upstream is down
};
t.after(() => {
globalThis.fetch = nativeFetch;
for (const [name, value] of Object.entries(saved)) {
if (value === undefined) delete process.env[name];
else process.env[name] = value;
}
});
let handler;
cctvProxy().configureServer({
middlewares: { use: (_route, fn) => (handler = fn) },
});
// What src/layers/cctv/source.js sends for this camera with no calibration.
const heading = Math.round(browserHeadingFromId(POSELESS.id));
const params = new URLSearchParams({
label: POSELESS.name,
city: '',
lat: POSELESS.lat.toFixed(6),
lon: POSELESS.lon.toFixed(6),
heading: String(heading),
fov: '74',
pitch: '-17',
ts: '1',
});
const { res, done } = recording();
await handler(
{
url: `/frame/${POSELESS.id}?${params}`,
headers: {},
method: 'GET',
on() {},
},
res,
);
await done;
assert.equal(res.headers['X-CCTV-Source'], 'streetview');
assert.equal(streetView.length, 1);
const sv = streetView[0].searchParams;
assert.equal(sv.get('location'), `${POSELESS.lat},${POSELESS.lon}`);
assert.equal(Number(sv.get('heading')), heading, 'heading the browser draws');
assert.equal(Number(sv.get('fov')), 74, 'FOV the browser draws');
assert.equal(Number(sv.get('pitch')), -17, 'pitch the browser draws');
});
-2
View File
@@ -22,8 +22,6 @@ function frameUrlFor(camera, refreshMs = ACTIVE_FRAME_REFRESH_MS) {
const params = new URLSearchParams({
label: camera.name,
city: camera.city,
lat: camera.lat.toFixed(6),
lon: camera.lon.toFixed(6),
heading: String(Math.round(camera.headingDeg)),
fov: String(Math.round(camera.fovDeg)),
pitch: String(Math.round(camera.pitchDeg || -10)),
+2 -2
View File
@@ -82,8 +82,8 @@ test('frame and media URLs preserve registered camera identity and encoded metad
);
assert.equal(frame.searchParams.get('label'), camera.name);
assert.equal(frame.searchParams.get('city'), camera.city);
assert.equal(frame.searchParams.get('lat'), '30.267000');
assert.equal(frame.searchParams.get('lon'), '-97.744000');
assert.equal(frame.searchParams.has('lat'), false);
assert.equal(frame.searchParams.has('lon'), false);
assert.equal(frame.searchParams.get('heading'), '45');
assert.equal(frame.searchParams.get('pitch'), '-12');
assert.deepEqual([...media.searchParams.keys()], ['ts']);