CCTV Street View budget: key the client before the camera wait, share the default

- The /frame handler reads the client's network key (IPv6 /64) before it
  awaits the camera, so a client that drops the connection is still
  charged to its own window instead of the shared fallback key.
- The budget is charged only after the Google key and finite catalog
  coordinates are confirmed, and the key is read once.
- Places and the CCTV window share one default (GOOGLE_DEFAULT_PER_MIN).
- Tests: the catalog-only test checks heading/fov/pitch from the query at
  the catalog location; the budget test covers unknown ids, forwarded
  requests and /64 grouping; a real-socket test covers a dropped
  connection. The duplicate pose test is removed.
- .env.example, SECURITY.md, CURRENT-STATE and the changelog fragment
  describe the separate per-IP window, the catalog position and the
  placeholder for seed cameras.
This commit is contained in:
German Khokhlov committed 2026-11-20 11:31:14 +04:00
1 parent 01368b95bf
commit 01d4a3f454
10 files changed
+175 -201

No files matched your search

+2
View File
@@ -37,6 +37,8 @@ GOOGLE_MAPS_API_KEY=
GOOGLE_MAPS_SERVER_API_KEY=
# Per-IP rate limit for the Google Places cost endpoints
# (/api/google/nearby-places + /api/google/text-search), in requests per minute.
# The CCTV Street View fallback counts in its own window with the same cap;
# over it, a frame gets the placeholder image instead of a 429.
# DEFAULTS TO 120, the value the Pinokio build already ships. Raise or lower it
# with a positive integer; 60 is a sensible tightening whenever HOST is not
# localhost. Set it to exactly 0 to run unthrottled. An unreadable value (a
+1 -1
View File
@@ -68,7 +68,7 @@ browser-key fallback for existing single-key setups.
The data proxies under `server/providers/` are written so the browser cannot turn the server into an open relay:
- **No arbitrary-URL fetching.** The CCTV frame proxy fetches only server-registered camera/frame URLs — clients cannot pass an upstream URL to fetch (SSRF mitigation). The key-bearing Street View fallback likewise answers only for a registered camera, always at its catalog position: a request sets only the view direction, field of view (20–120°) and pitch (−40–20°), so it cannot buy Street View of a place of its choosing. Other proxies target fixed upstream hosts.
- **No arbitrary-URL fetching.** The CCTV frame proxy fetches only server-registered camera/frame URLs — clients cannot pass an upstream URL to fetch (SSRF mitigation). The key-bearing Street View fallback likewise answers only for a registered camera, at its catalog position; a request sets only the heading, field of view and pitch, never the location. Other proxies target fixed upstream hosts.
- **Live HLS is bounded and origin-pinned.** Registered HTTP(S) HLS sources use a Node puller that rejects redirects and off-origin playlist references, bounds playlists to 256 KiB and segment bodies to 8 MiB while streaming, and aborts timed-out or released sessions. At most two sessions retain 12 segments/24 MiB each, plus bounded download buffers. Stores are memory-only and idle sessions expire after 15 seconds. RTMP/ffmpeg execution is not enabled. Locally configured source URLs remain an operator trust boundary.
- **Radio is not an audio relay.** `/api/radio/stations` contacts only allowlisted Radio Browser HTTPS hosts and paths, rejects redirects, rejects any hostname with a loopback/private/link-local/metadata/non-public A or AAAA result, and pins each TLS connection to a validated address. It returns normalized public HTTPS stream URLs; `/api/radio/click/:uuid` applies the same destination policy and accepts only station IDs from the current bounded catalog. The browser then connects directly to the broadcaster after an explicit playback action, so the broadcaster sees the listener's IP address. GEV never proxies, caches, records, or redistributes audio.
- **Québec 511 video plays in the browser, opt-in and after Play.** With `CCTV_QUEBEC_VIDEO_ENABLED=1`, `/api/cctv/sources` lists, for each MTQ camera, one URL built server-side from its numeric id on `https://www.quebec511.info/Carte/Fenetres/camera.ashx`; the overlay accepts no other host, path or format. The browser loads that clip only after the viewer presses Play for the selected camera, so Québec 511 sees the viewer's IP address, and the panel says so before Play. Selecting a camera by click, auto-hop, a scene or voice never loads it, and switching cameras stops playback. GEV never proxies, caches or reads the video's pixels. With the flag unset (the default) the browser makes no request to Québec 511.
@@ -1,5 +1,5 @@
- The CCTV Street View fallback serves only catalog cameras, always at their
catalog position, and ignores coordinates a request supplies. It has its own
per-client limit on the Google setting (`GEV_RATELIMIT_GOOGLE_PER_MIN`,
default 120), so the Google key can no longer be used as an unmetered Street
View service (asturksever, #1602).
- The CCTV Street View fallback answers only for cameras the server lists, at
their catalog position; a request sets only the view. Bundled seed cameras
get the placeholder. A per-IP limit on `GEV_RATELIMIT_GOOGLE_PER_MIN`
(default 120) keeps the Google key from serving as an unmetered Street View
service (asturksever, #1602).
+9 -8
View File
@@ -1990,14 +1990,12 @@ redirects on those hosts, are pinned by parsed URL to the agency's image
directory. Every `/api/cctv` route refuses cross-site browser requests but
accepts forwarding headers, so cameras work behind a reverse proxy; a forwarded
request skips the key-bearing Street View fallback and gets the placeholder. The
Street View fallback serves only catalog cameras, always at the catalog
position; coordinates a request supplies are ignored. Heading, field of view
and pitch come from the request, the pose the browser draws with calibration
included, and fall back to the catalog pose. A camera the server does not list,
such as a bundled seed camera the browser shows when the server catalog is
empty, gets the placeholder. The fallback has its own per-client limit on the
Google setting (`GEV_RATELIMIT_GOOGLE_PER_MIN`, default 120, `0` disables it);
a request over it gets the placeholder.
fallback answers only for cameras the server lists, at their catalog position,
so position calibration never reaches it. Heading, field of view (clamped to
20–120°) and pitch (clamped to −40–20°) come from the request, else from the
catalog pose. A bundled seed camera, shown when the server catalog is empty,
gets the placeholder. The fallback counts in its own per-IP Google window (see
the cost-bearing proxies below).
The CCTV layer draws only the cameras on or near the screen. Zoomed out, it
keeps one camera per ground cell sized to the view, preferring a healthy feed,
@@ -3529,6 +3527,9 @@ unaffected. `GEV_RATELIMIT_OPENAI_PER_MIN` and `GEV_RATELIMIT_GOOGLE_PER_MIN` ov
those; exactly `0` disables the limiter, while a value that cannot be read as a
number falls back to the default rather than to unlimited. Over-limit requests
receive a sanitized `429` with `Retry-After: 5` and never reach the provider.
The CCTV Street View fallback counts in a separate window with the Google cap.
An IPv6 /64 counts as one client. Over the cap, a frame gets the placeholder
image instead of a `429`.
Typed chat (`/api/openai/chat`, `/api/gemini/chat`) counts in its own window,
set by `GEV_RATELIMIT_OPENAI_CHAT_PER_MIN` and `GEV_RATELIMIT_GEMINI_CHAT_PER_MIN`.
Left unset, each takes its provider's resolved cap — `GEV_RATELIMIT_OPENAI_PER_MIN`
+26 -30
View File
@@ -21,8 +21,11 @@ import { PROXY_SIGNALS } from '../../src/localRequestGate.mjs';
import { createHlsPuller } from './cctv/stream.js';
import { isMlitRiverAdjustmentImage } from './cctv/worldCameras.js';
import { ottawaAuthorizedFrameUrl } from './cctv/ottawa.js';
import { googleServerApiKey } from './places/google-key.js';
import { clientKey, makeCostRateLimiter } from './common/rate-limit.js';
import {
GOOGLE_DEFAULT_PER_MIN,
googleServerApiKey,
} from './places/google-key.js';
import { clientNetworkKey, makeCostRateLimiter } from './common/rate-limit.js';
export { CCTV_FRAME_FETCH_TIMEOUT_MS, fetchCctvImageFromUpstream };
/**
@@ -200,21 +203,7 @@ export function cctvProxy({ sourceRoot = process.cwd() } = {}) {
};
};
/**
* Per-IP budget for the Street View fallback, one per proxy, on the same
* setting as the other Google routes (GEV_RATELIMIT_GOOGLE_PER_MIN; 0
* disables it). Built on first use, after the standalone env has loaded.
* undefined = not built yet; null = the explicit 0 opt-out.
*/
let streetViewBudget;
const streetViewRateLimiter = () => {
if (streetViewBudget === undefined)
streetViewBudget = makeCostRateLimiter(
process.env.GEV_RATELIMIT_GOOGLE_PER_MIN,
120,
);
return streetViewBudget;
};
let streetViewBudget; // undefined = not built yet; null = the explicit 0 opt-out
/**
* Fetch a Google Street View static image as a fallback frame. Server-side
@@ -222,11 +211,23 @@ export function cctvProxy({ sourceRoot = process.cwd() } = {}) {
* (#33: a key scoped to Street View Static/Places, restricted by server IP
* rather than HTTP referrer) and falls back to the browser-exposed
* GOOGLE_MAPS_API_KEY for setups that haven't split the two yet.
* Each IP (or IPv6 /64) has its own GEV_RATELIMIT_GOOGLE_PER_MIN window,
* separate from Places. Setting it to 0 disables the limit.
*/
const streetViewFallback = async ({ lat, lon, heading, fov, pitch }) => {
const streetViewFallback = async (
client,
{ lat, lon, heading, fov, pitch },
) => {
const streetViewKey = googleServerApiKey();
if (!streetViewKey || !Number.isFinite(lat) || !Number.isFinite(lon))
return null;
// Built on first use, after the standalone env has loaded.
if (streetViewBudget === undefined)
streetViewBudget = makeCostRateLimiter(
process.env.GEV_RATELIMIT_GOOGLE_PER_MIN,
GOOGLE_DEFAULT_PER_MIN,
);
if (streetViewBudget && !streetViewBudget(client)) return null;
try {
const sv = new URL('https://maps.googleapis.com/maps/api/streetview');
sv.searchParams.set('size', '960x540');
@@ -282,6 +283,8 @@ export function cctvProxy({ sourceRoot = process.cwd() } = {}) {
// keyless gate keeps the routes reachable behind a reverse proxy; the
// key-bearing Street View fallback below has its own check.
if (admitSameSiteKeyless(req, res)) return;
// Read before any await: a socket the client has closed has no address.
const client = clientNetworkKey(req);
const sources = await getCctvSources();
const sourceById = new Map(
sources.map((source) => [source.id, source]),
@@ -653,8 +656,7 @@ export function cctvProxy({ sourceRoot = process.cwd() } = {}) {
const source = sourceById.get(cameraId);
const label = url.searchParams.get('label') || source?.name || cameraId;
const city = url.searchParams.get('city') || source?.city || '';
// The view follows the pose the browser draws (calibration included);
// the position never comes from the query (see the Street View fallback).
// The query sets only the view, the pose the browser draws.
const heading = Number(
url.searchParams.get('heading') || source?.headingDeg,
);
@@ -749,21 +751,15 @@ export function cctvProxy({ sourceRoot = process.cwd() } = {}) {
// request (whose client this server cannot see) gets the placeholder,
// as if no key were configured. Any forwarding header counts, even an
// empty one.
// It is also only for catalog cameras, at the catalog position (never a
// spot the query names; the query sets only the view), within the per-IP
// Google budget.
// It also answers only for catalog cameras, at the catalog position;
// the query never sets the location.
const forwarded = PROXY_SIGNALS.some(
(name) => req.headers?.[name] !== undefined,
);
const googleBudget = googleServerApiKey()
? streetViewRateLimiter()
: null;
const sv =
forwarded ||
!source ||
(googleBudget && !googleBudget(clientKey(req)))
forwarded || !source
? null
: await streetViewFallback({
: await streetViewFallback(client, {
lat: source.lat,
lon: source.lon,
heading,
+8
View File
@@ -1,5 +1,13 @@
import { resolveGoogleServerKey } from '../../../scripts/google-server-key.mjs';
/**
* Default GEV_RATELIMIT_GOOGLE_PER_MIN for the Places and Geocoding window
* and the separate CCTV Street View window, in requests per minute per IP.
* The Pinokio build sets the same default in pinokio/_ENVIRONMENT.
* `.env.example` suggests 60 for hosts other than localhost.
*/
export const GOOGLE_DEFAULT_PER_MIN = 120;
/**
* Optional Google place context is an empty capability when no key is present,
* not a server outage. Returning 200 keeps a deliberately keyless session out
+2 -11
View File
@@ -1,4 +1,5 @@
import {
GOOGLE_DEFAULT_PER_MIN,
googleServerApiKey,
keylessGooglePlacesResponse,
keylessGoogleGeocodeResponse,
@@ -15,17 +16,7 @@ import {
// undefined = not built yet; null = the explicit 0 opt-out; fn = active limiter
let _googleRateLimiter;
/**
* Requests/min/IP applied when GEV_RATELIMIT_GOOGLE_PER_MIN is unset. This is
* the value the Pinokio build already ships (pinokio/_ENVIRONMENT), so the
* packaged app keeps behaving exactly as it does today and only an
* unconfigured server changes — from unlimited to what the product already
* runs with. Places calls are user-driven (a search, a nearby lookup, an
* installation probe) rather than polled, so this sits far above what the app
* generates and bites only a caller enumerating the endpoint. `.env.example`
* suggests a tighter 60 when the host is not localhost; that still applies.
*/
export const GOOGLE_DEFAULT_PER_MIN = 120;
export { GOOGLE_DEFAULT_PER_MIN };
/** Google cost endpoints (nearby-places, text-search, geocoding). Null only when set to 0. */
function googleRateLimiter() {
+35 -5
View File
@@ -553,12 +553,14 @@ test('the Street View fallback frames catalog cameras only, from the catalog pos
assert.deepEqual(streetView(), [], 'no Google call for an unknown id');
const steered = await app.call(
`/frame/${STILL.id}?lat=48.85&lon=2.29&heading=90`,
`/frame/${STILL.id}?lat=48.85&lon=2.29&heading=90&fov=74&pitch=-17`,
);
assert.equal(steered.headers['X-CCTV-Source'], 'streetview');
const location = new URL(streetView()[0].url).searchParams.get('location');
assert.equal(location, `${STILL.lat},${STILL.lon}`);
assert.equal(new URL(streetView()[0].url).searchParams.get('heading'), '90');
const sv = new URL(streetView()[0].url).searchParams;
assert.equal(sv.get('location'), `${STILL.lat},${STILL.lon}`);
assert.equal(sv.get('heading'), '90');
assert.equal(sv.get('fov'), '74');
assert.equal(sv.get('pitch'), '-17');
});
test('the Street View fallback spends a per-IP Google budget, and only with a key', async (t) => {
@@ -586,8 +588,27 @@ test('the Street View fallback spends a per-IP Google budget, and only with a ke
assert.equal(streetViewCalls(), 0);
process.env.GOOGLE_MAPS_SERVER_API_KEY = 'fixture-key';
// Neither an unknown id nor a forwarded request spends the window.
assert.equal(
(await app.call('/frame/not-a-camera', {}, first)).headers['X-CCTV-Source'],
'synthetic',
);
assert.equal(
(
await app.call(
`/frame/${STILL.id}`,
{ 'x-forwarded-for': '203.0.113.9' },
first,
)
).headers['X-CCTV-Source'],
'synthetic',
);
assert.deepEqual(
[await frameSource(first), await frameSource(first), await frameSource(first)],
[
await frameSource(first),
await frameSource(first),
await frameSource(first),
],
['streetview', 'streetview', 'synthetic'],
);
assert.equal(streetViewCalls(), 2, 'no Google call over the budget');
@@ -598,4 +619,13 @@ test('the Street View fallback spends a per-IP Google budget, and only with a ke
'streetview',
);
assert.equal(streetViewCalls(), 3);
// Addresses within one IPv6 /64 share a budget.
const v6 = (remoteAddress) => frameSource({ remoteAddress });
assert.deepEqual(
[await v6('2001:db8::1'), await v6('2001:db8::2'), await v6('2001:db8::3')],
['streetview', 'streetview', 'synthetic'],
);
assert.equal(await v6('2001:db8:0:1::1'), 'streetview', 'another /64');
assert.equal(streetViewCalls(), 6);
});
@@ -0,0 +1,87 @@
// Real sockets: a client that closes its connection while the camera upstream
// is still pending must not get a Street View call outside its own budget.
import { test } from 'node:test';
import assert from 'node:assert/strict';
import http from 'node:http';
import { cctvProxy } from '../../server/providers/cctv.js';
const CAM = {
id: 'still-fixture',
name: 'Still fixture',
lat: 30.27,
lon: -97.74,
feedType: 'image',
url: 'https://cams.upstream.invalid/still.jpg',
};
test('a client that disconnects mid-fetch stays inside its own Street View budget', async (t) => {
const saved = { ...process.env };
process.env.CCTV_SOURCES_JSON = JSON.stringify([CAM]);
process.env.CCTV_SOURCES_FILE = 'absent-source-file.json';
process.env.CCTV_FORCE_AUSTIN = '0';
process.env.GOOGLE_MAPS_SERVER_API_KEY = 'fixture-key';
delete process.env.GOOGLE_MAPS_API_KEY;
process.env.GEV_RATELIMIT_GOOGLE_PER_MIN = '1';
const nativeFetch = globalThis.fetch;
let streetViewCalls = 0;
globalThis.fetch = async (url) => {
if (String(url).startsWith('https://maps.googleapis.com/')) {
streetViewCalls += 1;
return new Response('street-view', {
headers: { 'content-type': 'image/jpeg' },
});
}
await new Promise((r) => setTimeout(r, 400));
return new Response(null, { status: 503 });
};
let handler;
cctvProxy().configureServer({
middlewares: { use: (_r, fn) => (handler = fn) },
});
const server = http.createServer((req, res) => {
req.url = req.url.replace(/^\/api\/cctv/, '');
handler(req, res, () => res.end());
});
await new Promise((r) => server.listen(0, '127.0.0.1', r));
t.after(() => {
server.close();
globalThis.fetch = nativeFetch;
for (const k of Object.keys(process.env))
if (!(k in saved)) delete process.env[k];
Object.assign(process.env, saved);
});
const { port } = server.address();
const get = (abortAfterMs) =>
new Promise((resolve) => {
const req = http.get(
{
host: '127.0.0.1',
port,
path: `/api/cctv/frame/${CAM.id}`,
headers: { host: `localhost:${port}` },
agent: false,
},
(res) => {
res.resume();
res.on('end', () => resolve(res.headers['x-cctv-source']));
},
);
req.on('error', () => resolve('aborted'));
if (abortAfterMs)
setTimeout(() => {
req.destroy();
resolve('aborted');
}, abortAfterMs);
});
assert.equal(await get(), 'streetview');
assert.equal(await get(), 'synthetic', 'the cap of one is spent');
assert.equal(streetViewCalls, 1);
await get(50);
await new Promise((r) => setTimeout(r, 800));
assert.equal(
streetViewCalls,
1,
'an aborted request spent the key outside its own budget',
);
});
-141
View File
@@ -1,141 +0,0 @@
// When a catalog camera has no pose, the browser sends /frame a heading derived
// from the camera ID, a pitch of -17°, and a field of view of 74°. Street View
// uses that pose at the camera's catalog coordinates.
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { cctvProxy } from '../../server/providers/cctv.js';
// Same shape as a world-cameras row: an image URL, position, and no
// headingDeg / pitchDeg / fovDeg.
const POSELESS = {
id: 'world-or-fixture',
name: 'Pose-less fixture',
lat: 44.05,
lon: -123.09,
feedType: 'image',
url: 'https://cams.upstream.invalid/still.jpg',
};
/** The browser's id-hash heading (src/layers/cctv/model.js headingFromId). */
function browserHeadingFromId(id) {
let acc = 0;
for (let i = 0; i < id.length; i++) acc = (acc * 33 + id.charCodeAt(i)) >>> 0;
return ((((acc % 16) * 22.5) % 360) + 360) % 360;
}
function response({ status = 200, headers = {}, body = '' }) {
return new Response(status === 204 ? null : String(body), {
status,
headers,
});
}
function recording() {
const chunks = [];
let finished;
const done = new Promise((resolve) => (finished = resolve));
const res = {
statusCode: 0,
headers: {},
writableEnded: false,
writeHead(status, headers) {
res.statusCode = status;
res.headers = { ...res.headers, ...headers };
},
setHeader(name, value) {
res.headers[name] = value;
},
write(chunk) {
chunks.push(Buffer.from(chunk));
return true;
},
end(chunk) {
if (chunk) chunks.push(Buffer.from(chunk));
res.writableEnded = true;
res.body = Buffer.concat(chunks).toString('utf8');
finished();
},
on() {},
once() {},
emit() {},
destroy() {},
};
return { res, done };
}
test('the Street View fallback frames a pose-less catalog camera the way the browser draws it', async (t) => {
const saved = Object.fromEntries(
[
'CCTV_SOURCES_JSON',
'CCTV_SOURCES_FILE',
'CCTV_FORCE_AUSTIN',
'GOOGLE_MAPS_SERVER_API_KEY',
'GOOGLE_MAPS_API_KEY',
'GEV_RATELIMIT_GOOGLE_PER_MIN',
].map((name) => [name, process.env[name]]),
);
process.env.CCTV_SOURCES_JSON = JSON.stringify([POSELESS]);
process.env.CCTV_SOURCES_FILE = 'absent-source-file.json';
process.env.CCTV_FORCE_AUSTIN = '0';
process.env.GOOGLE_MAPS_SERVER_API_KEY = 'fixture-key';
delete process.env.GOOGLE_MAPS_API_KEY;
delete process.env.GEV_RATELIMIT_GOOGLE_PER_MIN;
const nativeFetch = globalThis.fetch;
const streetView = [];
globalThis.fetch = async (url) => {
const href = String(url);
if (href.startsWith('https://maps.googleapis.com/')) {
streetView.push(new URL(href));
return response({
headers: { 'content-type': 'image/jpeg' },
body: 'street-view',
});
}
return response({ status: 503 }); // the camera's upstream is down
};
t.after(() => {
globalThis.fetch = nativeFetch;
for (const [name, value] of Object.entries(saved)) {
if (value === undefined) delete process.env[name];
else process.env[name] = value;
}
});
let handler;
cctvProxy().configureServer({
middlewares: { use: (_route, fn) => (handler = fn) },
});
// What src/layers/cctv/source.js sends for this camera with no calibration.
const heading = Math.round(browserHeadingFromId(POSELESS.id));
const params = new URLSearchParams({
label: POSELESS.name,
city: '',
lat: POSELESS.lat.toFixed(6),
lon: POSELESS.lon.toFixed(6),
heading: String(heading),
fov: '74',
pitch: '-17',
ts: '1',
});
const { res, done } = recording();
await handler(
{
url: `/frame/${POSELESS.id}?${params}`,
headers: {},
method: 'GET',
on() {},
},
res,
);
await done;
assert.equal(res.headers['X-CCTV-Source'], 'streetview');
assert.equal(streetView.length, 1);
const sv = streetView[0].searchParams;
assert.equal(sv.get('location'), `${POSELESS.lat},${POSELESS.lon}`);
assert.equal(Number(sv.get('heading')), heading, 'heading the browser draws');
assert.equal(Number(sv.get('fov')), 74, 'FOV the browser draws');
assert.equal(Number(sv.get('pitch')), -17, 'pitch the browser draws');
});