mirror of
https://github.com/bilawalsidhu/gods-eye-view.git
synced 2026-11-04 10:24:02 +00:00
fix(street-level): harden the Mapillary tile proxy
From gekh's review of #768, each with a test that fails before: - The routes refuse cross-site requests (sameSiteGated, as the paid proxies do) and the tile route has a generous per-IP limit (600/min), so another page cannot burn the token's quota or trip the shared 429 hold. - At most six upstream fetches run at once; a queued request that is abandoned never fetches, nor does one abandoned during the disk read. - A body past the 48 MB cap aborts the upstream request as it streams. - Empty tiles are charged against the memory and disk budgets. - Each disk write uses its own temp file and a retrim is skipped while a write of that tile is pending, so concurrent hits cannot extend a tile's life or fail a rename. - Upstream refusals other than 401/403/429 answer 502. - The tile route's success path is tested end to end (200 with the image layer stripped and the right headers, 204 when empty, an abandoned response aborts upstream); the teardown race that made this file flaky is gone (240 parallel runs, no failure); the QA-constant tautologies are replaced by a check against the real provider registry; the boundary group lists the modules the provider actually imports. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
5f98b871e8
commit
d70f468781
8 files changed
+709
-94
No files matched your search
@@ -19,6 +19,8 @@
|
||||
|
||||
- Street Level polish from review: whole-earth coverage dots on the far side of the globe no longer show through it; a 360° or flat filter keeps a dense tile's matching lines instead of losing them to the per-tile cap; lines that lost their ground heights are cast again rather than left draped; a selection made while lines rebuild is applied once they are ready; hovering picks at most every 120 ms; Esc shrinks the expanded viewer wherever focus is, and closing a photo leaves focus in the panel; switching the layer off and on during the first photo load no longer leaves the viewer stuck. Portable panels lift only after a real mouse or pen drag, never on a click or a touch swipe. The MapillaryJS stylesheet now loads with the viewer instead of on every page (about 85 KB less render-blocking CSS). The tile proxy's disk cache is swept of expired tiles and capped at 1 GiB, keeps a cached tile's real age in memory, and refuses zoom levels the app never requests.
|
||||
|
||||
- Street Level, second review round: the panel no longer reopens itself on every page load (only a user, voice or tool switch-on or an opening photo opens it, and those automatic opens are never stored); a floating panel is pulled back on-screen when its content grows or the window shrinks, and its header can never be dropped under the command dock or the voice pill; the expanded photo hides in Clean View, recording and the cockpit; closing a photo stops sequence playback; a failed sequence load can be retried and leaves no stale image cones. The tile proxy refuses cross-site requests and has a per-IP limit, runs at most six upstream fetches at once, stops a response past its size cap as it streams, counts empty tiles against its caches and answers other upstream refusals with 502. The panel's collapsed state now travels in share links (`ui` token `t`).
|
||||
|
||||
- Add Norway's road cameras from Statens vegvesen as a CCTV pack. The camera
|
||||
list is one keyless GeoJSON request to the agency's OGC API view of its
|
||||
DATEX 3.1 CCTV table, so no DATEX account is needed. Every working camera
|
||||
|
||||
+1
-1
@@ -72,7 +72,7 @@ The data proxies under `server/providers/` are written so the browser cannot tur
|
||||
- **Sanitized errors** — internal error details are not echoed back to clients.
|
||||
- **Coalesced OAuth refresh** and cached successful responses only (OpenSky).
|
||||
- **Redacted debug logging.** The voice debug log (`.gev-logs/`, gitignored) strips API keys, bearer tokens, client secrets, and image data URLs before writing.
|
||||
- **Cross-site gate on cost/log endpoints.** The cost-bearing (`/api/realtime/token`, `/api/openai/hud-summary`, `/api/google/nearby-places`, `/api/google/text-search`) and log (`/api/realtime/debug-log`) endpoints refuse cross-site browser requests: a foreign or opaque `Origin`, a `Sec-Fetch-Site` other than `same-origin`/`none` (this is what blocks an `<img>`/navigation that carries no `Origin`), or any reverse-proxy/CDN forwarding header returns `403`. Non-browser loopback tools (the repo QA harnesses POST the token endpoint with no `Origin`) and the explicit `HOST=0.0.0.0` LAN opt-in keep working — the gate deliberately does not require a loopback remote address; that stricter requirement belongs only to the credential panel (`admitKeySetupRequest`). The Host check (`allowedHosts`, applied by `hostCheckPlugin` ahead of every route, since Vite's own check runs only after plugin middleware) rejects a rebinding `Host` before any handler runs, in every mode, and the gate is defense in depth behind it: the LAN opt-in keeps that check and accepts only IP addresses, `localhost` and names listed in `GEV_ALLOWED_HOSTS`. **Limit:** a name you list there is trusted like the app's own address, so list only names you control.
|
||||
- **Cross-site gate on cost/log endpoints.** The cost-bearing (`/api/realtime/token`, `/api/openai/hud-summary`, `/api/google/nearby-places`, `/api/google/text-search`), quota-bearing (`/api/mapillary/status`, `/api/mapillary/tiles`, whose tile route also has a per-IP limit) and log (`/api/realtime/debug-log`) endpoints refuse cross-site browser requests: a foreign or opaque `Origin`, a `Sec-Fetch-Site` other than `same-origin`/`none` (this is what blocks an `<img>`/navigation that carries no `Origin`), or any reverse-proxy/CDN forwarding header returns `403`. Non-browser loopback tools (the repo QA harnesses POST the token endpoint with no `Origin`) and the explicit `HOST=0.0.0.0` LAN opt-in keep working — the gate deliberately does not require a loopback remote address; that stricter requirement belongs only to the credential panel (`admitKeySetupRequest`). The Host check (`allowedHosts`, applied by `hostCheckPlugin` ahead of every route, since Vite's own check runs only after plugin middleware) rejects a rebinding `Host` before any handler runs, in every mode, and the gate is defense in depth behind it: the LAN opt-in keeps that check and accepts only IP addresses, `localhost` and names listed in `GEV_ALLOWED_HOSTS`. **Limit:** a name you list there is trusted like the app's own address, so list only names you control.
|
||||
- **Full Content-Security-Policy.** The dev and preview servers send a real CSP (`script-src 'self' 'unsafe-eval' blob:` plus the YouTube, Facebook and X scripts the Nepal event's embedded media loads, with a matching `frame-src` for their players and posts: no inline script and no other script origin; `blob:` covers the object URLs Cesium's bundled workers bootstrap through in the built app), alongside the existing `X-Frame-Options: DENY` and `frame-ancestors 'none'`. `'unsafe-eval'` stays because Knockout, bundled inside Cesium's widgets package, resolves the global object with `eval` at load time and the globe does not initialize without it; styles permit `'unsafe-inline'` and Google Fonts because Vite's dev client injects `<style>` elements.
|
||||
|
||||
## Network exposure — the operator threat model
|
||||
|
||||
@@ -192,10 +192,10 @@
|
||||
"server/providers/mapillary/constants.js",
|
||||
"server/providers/mapillary/tiles.js",
|
||||
"server/providers/mapillary/trim.js",
|
||||
"src/layers/streetLevel/tileMath.js",
|
||||
"server/providers/common/request.js",
|
||||
"src/sources/rateLimit.js",
|
||||
"server/providers/common/rate-limit.js"
|
||||
"server/providers/common/same-site.js",
|
||||
"src/localRequestGate.mjs",
|
||||
"server/providers/common/rate-limit.js",
|
||||
"src/sources/rateLimit.js"
|
||||
],
|
||||
"external": ["pbf"]
|
||||
},
|
||||
|
||||
@@ -39,13 +39,40 @@ export const TILE_DISK_TTL_MS = 24 * 60 * 60 * 1000;
|
||||
*/
|
||||
export const TILE_DISK_SWEEP_INTERVAL_MS = 15 * 60 * 1000;
|
||||
export const TILE_DISK_MAX_BYTES = 1024 * 1024 * 1024;
|
||||
/**
|
||||
* What each cached file costs against TILE_DISK_MAX_BYTES on top of its size:
|
||||
* one filesystem block and its directory entry. An empty (no coverage) tile is
|
||||
* 0 bytes but still a file, so without this ocean tiles would never count.
|
||||
*/
|
||||
export const TILE_DISK_FILE_OVERHEAD_BYTES = 4096;
|
||||
|
||||
/** A z14 image tile over a dense city is ~11 MB; anything past this is wrong. */
|
||||
export const TILE_MAX_BYTES = 48 * 1024 * 1024;
|
||||
|
||||
/** In-memory tile cache budget (bytes) and upstream fetch timeout. */
|
||||
export const TILE_MEMORY_BUDGET_BYTES = 96 * 1024 * 1024;
|
||||
/**
|
||||
* What each in-memory entry costs against the budget on top of its bytes (the
|
||||
* key, the map entry and the Buffer object), so empty tiles count too.
|
||||
*/
|
||||
export const TILE_MEMORY_ENTRY_OVERHEAD_BYTES = 1024;
|
||||
export const TILE_FETCH_TIMEOUT_MS = 60_000;
|
||||
/**
|
||||
* Upstream tile fetches running at once; further misses queue for a slot. A
|
||||
* view asks for at most 25 tiles, so a few slots keep it quick while a burst
|
||||
* of distinct misses cannot open thousands of buffered requests at once.
|
||||
*/
|
||||
export const TILE_UPSTREAM_CONCURRENCY = 6;
|
||||
/**
|
||||
* Tile requests per client IP per minute (cache hits included). The app asks
|
||||
* only for tiles it has not loaded yet, at most 9 street + 16 overview tiles
|
||||
* per view after a 320 ms camera debounce (src/layers/streetLevel/providers/
|
||||
* mapillary/policy.js), and the browser keeps each answer for an hour. 600 is
|
||||
* 24 entirely new views a minute, one every 2.5 s, which a person flying the
|
||||
* camera does not reach; it still stops one page from draining the shared
|
||||
* token into Mapillary rate limits that would hold every miss for everyone.
|
||||
*/
|
||||
export const TILE_ROUTE_MAX_PER_MIN = 600;
|
||||
/**
|
||||
* After Mapillary rejects the token (401/403), tile misses are answered from
|
||||
* that verdict for this long instead of asking again on every camera move.
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
import { sameSiteGated } from '../common/same-site.js';
|
||||
import { makeRateLimiter, clientKey } from '../common/rate-limit.js';
|
||||
import { fetchTile, TileRequestError, TileUpstreamError } from './tiles.js';
|
||||
import { mapillaryToken } from './constants.js';
|
||||
import { mapillaryToken, TILE_ROUTE_MAX_PER_MIN } from './constants.js';
|
||||
|
||||
function sendJson(res, status, payload) {
|
||||
res.statusCode = status;
|
||||
@@ -16,9 +18,16 @@ function handleStatus(req, res) {
|
||||
}
|
||||
|
||||
/** GET /api/mapillary/tiles/coverage/{z}/{x}/{y} — cached protobuf tile. */
|
||||
async function handleTile(req, res) {
|
||||
async function handleTile(req, res, allow) {
|
||||
if (req.method !== 'GET')
|
||||
return sendJson(res, 405, { error: 'Method not allowed' });
|
||||
if (!allow(clientKey(req))) {
|
||||
res.setHeader('Retry-After', '5');
|
||||
return sendJson(res, 429, {
|
||||
error: 'Too many tile requests',
|
||||
retryAfter: 5,
|
||||
});
|
||||
}
|
||||
const match = /^\/(coverage)\/(\d{1,2})\/(\d{1,6})\/(\d{1,6})$/.exec(
|
||||
(req.url || '').split('?')[0],
|
||||
);
|
||||
@@ -61,16 +70,30 @@ async function handleTile(req, res) {
|
||||
retryAfter,
|
||||
});
|
||||
}
|
||||
return sendJson(res, error.status >= 500 ? 502 : error.status, {
|
||||
error: error.message,
|
||||
});
|
||||
// Any other refusal is Mapillary's fault, not the client's: an
|
||||
// upstream 400 must not read as a malformed request to this proxy.
|
||||
return sendJson(res, 502, { error: error.message });
|
||||
}
|
||||
sendJson(res, 502, { error: error?.message || 'Tile fetch failed' });
|
||||
}
|
||||
}
|
||||
|
||||
/** Attach every Mapillary route to a connect-style middleware stack. */
|
||||
/**
|
||||
* Attach every Mapillary route to a connect-style middleware stack. Both
|
||||
* refuse cross-site browser requests (server/providers/common/same-site.js):
|
||||
* another page could otherwise spend the user's token, and one Mapillary 429
|
||||
* it provokes holds every tile miss. Tiles are also rate-limited per client
|
||||
* IP (TILE_ROUTE_MAX_PER_MIN).
|
||||
*/
|
||||
export function installMapillaryRoutes(middlewares) {
|
||||
middlewares.use('/api/mapillary/status', handleStatus);
|
||||
middlewares.use('/api/mapillary/tiles', handleTile);
|
||||
const allow = makeRateLimiter({
|
||||
windowMs: 60_000,
|
||||
max: TILE_ROUTE_MAX_PER_MIN,
|
||||
globalMax: TILE_ROUTE_MAX_PER_MIN * 4,
|
||||
});
|
||||
middlewares.use('/api/mapillary/status', sameSiteGated(handleStatus));
|
||||
middlewares.use(
|
||||
'/api/mapillary/tiles',
|
||||
sameSiteGated((req, res) => handleTile(req, res, allow)),
|
||||
);
|
||||
}
|
||||
@@ -8,9 +8,12 @@ import {
|
||||
TILE_DISK_TTL_MS,
|
||||
TILE_DISK_SWEEP_INTERVAL_MS,
|
||||
TILE_DISK_MAX_BYTES,
|
||||
TILE_DISK_FILE_OVERHEAD_BYTES,
|
||||
TILE_MAX_BYTES,
|
||||
TILE_MEMORY_BUDGET_BYTES,
|
||||
TILE_MEMORY_ENTRY_OVERHEAD_BYTES,
|
||||
TILE_FETCH_TIMEOUT_MS,
|
||||
TILE_UPSTREAM_CONCURRENCY,
|
||||
TILE_KEY_REJECTED_HOLD_MS,
|
||||
TILE_RATE_LIMIT_HOLD_MS,
|
||||
TILE_RATE_LIMIT_MAX_HOLD_MS,
|
||||
@@ -108,7 +111,9 @@ export function normalizeTileAddress({ layer, z, x, y }) {
|
||||
|
||||
/** @type {Map<string, {bytes: Buffer, at: number}>} insertion-ordered LRU */
|
||||
const _memory = new Map();
|
||||
/** Charged bytes: every entry's size plus a fixed overhead (empty tiles too). */
|
||||
let _memoryBytes = 0;
|
||||
const memoryCost = (bytes) => bytes.length + TILE_MEMORY_ENTRY_OVERHEAD_BYTES;
|
||||
/**
|
||||
* Upstream fetches in progress, shared by every request for the same tile.
|
||||
* A flight owns its controller; callers only count as waiters, and the last
|
||||
@@ -124,7 +129,7 @@ function memoryGet(key) {
|
||||
// serving stale (or empty) coverage until eviction.
|
||||
if (Date.now() - hit.at > TILE_DISK_TTL_MS) {
|
||||
_memory.delete(key);
|
||||
_memoryBytes -= hit.bytes.length;
|
||||
_memoryBytes -= memoryCost(hit.bytes);
|
||||
return null;
|
||||
}
|
||||
// Re-insert to mark as most recently used.
|
||||
@@ -135,15 +140,15 @@ function memoryGet(key) {
|
||||
|
||||
/** `at` is when Mapillary served the bytes (a disk tile's mtime). */
|
||||
function memoryPut(key, bytes, at = Date.now()) {
|
||||
if (bytes.length > TILE_MEMORY_BUDGET_BYTES / 2) return;
|
||||
if (memoryCost(bytes) > TILE_MEMORY_BUDGET_BYTES / 2) return;
|
||||
const existing = _memory.get(key);
|
||||
if (existing) _memoryBytes -= existing.bytes.length;
|
||||
if (existing) _memoryBytes -= memoryCost(existing.bytes);
|
||||
_memory.set(key, { bytes, at });
|
||||
_memoryBytes += bytes.length;
|
||||
_memoryBytes += memoryCost(bytes);
|
||||
while (_memoryBytes > TILE_MEMORY_BUDGET_BYTES && _memory.size) {
|
||||
const [oldest, entry] = _memory.entries().next().value;
|
||||
_memory.delete(oldest);
|
||||
_memoryBytes -= entry.bytes.length;
|
||||
_memoryBytes -= memoryCost(entry.bytes);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -166,28 +171,68 @@ async function readDisk(address) {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Background disk work (tile writes and sweeps) not yet settled. Requests
|
||||
* never wait on it; tests settle it before removing a cache directory.
|
||||
* @type {Set<Promise<void>>}
|
||||
*/
|
||||
const _background = new Set();
|
||||
|
||||
/** Track a background task until it settles; it never rejects. */
|
||||
function track(task) {
|
||||
const settled = task
|
||||
.catch(() => {})
|
||||
.finally(() => _background.delete(settled));
|
||||
_background.add(settled);
|
||||
return settled;
|
||||
}
|
||||
|
||||
/**
|
||||
* Disk writes in progress, by cache file: the sequence number of the newest.
|
||||
* @type {Map<string, number>}
|
||||
*/
|
||||
const _diskWrites = new Map();
|
||||
let _diskWriteSeq = 0;
|
||||
|
||||
/** Whether a write of this tile's cache file is still in progress. */
|
||||
function diskWritePending(address) {
|
||||
return _diskWrites.has(diskPath(address));
|
||||
}
|
||||
|
||||
/**
|
||||
* Write a tile in the background. `at` dates a rewrite of an existing tile
|
||||
* with its original fetch time, so the rewrite does not extend its life.
|
||||
* Each write has its own temporary file, and a write that a newer one of the
|
||||
* same tile overtook steps aside instead of renaming over it.
|
||||
*/
|
||||
function writeDisk(address, bytes, at = null) {
|
||||
// The sweep after it covers this write's cache root, even if it has moved.
|
||||
const root = _diskDir;
|
||||
const file = diskPath(address, root);
|
||||
fsp
|
||||
.mkdir(path.dirname(file), { recursive: true })
|
||||
.then(() => fsp.writeFile(`${file}.tmp`, bytes))
|
||||
.then(
|
||||
() => at != null && fsp.utimes(`${file}.tmp`, new Date(), new Date(at)),
|
||||
)
|
||||
.then(() => fsp.rename(`${file}.tmp`, file))
|
||||
.then(() => scheduleSweep(root))
|
||||
.catch((error) =>
|
||||
console.warn(
|
||||
'[Mapillary Proxy] tile cache write failed:',
|
||||
error?.message || error,
|
||||
),
|
||||
);
|
||||
const seq = ++_diskWriteSeq;
|
||||
const tmp = `${file}.${process.pid}-${seq}.tmp`;
|
||||
const newest = () => _diskWrites.get(file) === seq;
|
||||
_diskWrites.set(file, seq);
|
||||
return track(
|
||||
fsp
|
||||
.mkdir(path.dirname(file), { recursive: true })
|
||||
.then(() => fsp.writeFile(tmp, bytes))
|
||||
.then(() => at != null && fsp.utimes(tmp, new Date(), new Date(at)))
|
||||
.then(() => {
|
||||
if (!newest()) return fsp.rm(tmp, { force: true });
|
||||
return fsp.rename(tmp, file).then(() => scheduleSweep(root));
|
||||
})
|
||||
.catch(async (error) => {
|
||||
await fsp.rm(tmp, { force: true }).catch(() => {});
|
||||
console.warn(
|
||||
'[Mapillary Proxy] tile cache write failed:',
|
||||
error?.message || error,
|
||||
);
|
||||
})
|
||||
.finally(() => {
|
||||
if (newest()) _diskWrites.delete(file);
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
let _lastSweepAt = 0;
|
||||
@@ -199,7 +244,7 @@ function scheduleSweep(root) {
|
||||
if (_sweeping || Date.now() - _lastSweepAt < TILE_DISK_SWEEP_INTERVAL_MS)
|
||||
return;
|
||||
_lastSweepAt = Date.now();
|
||||
_sweeping = sweepTileDisk({ root })
|
||||
const sweep = sweepTileDisk({ root })
|
||||
.catch((error) => {
|
||||
if (_sweepWarned) return;
|
||||
_sweepWarned = true;
|
||||
@@ -209,8 +254,10 @@ function scheduleSweep(root) {
|
||||
);
|
||||
})
|
||||
.finally(() => {
|
||||
_sweeping = null;
|
||||
if (_sweeping === tracked) _sweeping = null;
|
||||
});
|
||||
const tracked = track(sweep);
|
||||
_sweeping = tracked;
|
||||
}
|
||||
|
||||
/** Every file under `dir` with its size and mtime; a missing dir is empty. */
|
||||
@@ -237,8 +284,10 @@ async function listCacheFiles(dir) {
|
||||
|
||||
/**
|
||||
* Bound the disk cache: delete tiles past TILE_DISK_TTL_MS, then the oldest
|
||||
* until the rest fit in `maxBytes`. Runs in the background after tile writes.
|
||||
* @returns {Promise<{removed: number, bytes: number}>} files deleted, bytes kept
|
||||
* until the rest fit in `maxBytes`. Each file is charged its size plus
|
||||
* TILE_DISK_FILE_OVERHEAD_BYTES, so empty tiles count toward the cap.
|
||||
* Runs in the background after tile writes.
|
||||
* @returns {Promise<{removed: number, bytes: number}>} files deleted, charged bytes kept
|
||||
*/
|
||||
export async function sweepTileDisk({
|
||||
root = _diskDir,
|
||||
@@ -246,12 +295,13 @@ export async function sweepTileDisk({
|
||||
} = {}) {
|
||||
const now = Date.now();
|
||||
const files = (await listCacheFiles(root)).sort((a, b) => a.at - b.at);
|
||||
let bytes = files.reduce((sum, { size }) => sum + size, 0);
|
||||
const cost = (size) => size + TILE_DISK_FILE_OVERHEAD_BYTES;
|
||||
let bytes = files.reduce((sum, { size }) => sum + cost(size), 0);
|
||||
let removed = 0;
|
||||
for (const { file, size, at } of files) {
|
||||
if (now - at <= TILE_DISK_TTL_MS && bytes <= maxBytes) break;
|
||||
await fsp.rm(file, { force: true });
|
||||
bytes -= size;
|
||||
bytes -= cost(size);
|
||||
removed++;
|
||||
}
|
||||
return { removed, bytes };
|
||||
@@ -264,8 +314,10 @@ async function fetchUpstream(address, signal) {
|
||||
if (held) throw held;
|
||||
const url = `${MAPILLARY_TILE_HOST}/${address.upstream}/2/${address.z}/${address.x}/${address.y}?access_token=${encodeURIComponent(token)}`;
|
||||
const timeout = AbortSignal.timeout(TILE_FETCH_TIMEOUT_MS);
|
||||
// Aborts the request itself once its body runs past the size cap.
|
||||
const oversize = new AbortController();
|
||||
const response = await fetch(url, {
|
||||
signal: signal ? AbortSignal.any([signal, timeout]) : timeout,
|
||||
signal: AbortSignal.any([signal, timeout, oversize.signal].filter(Boolean)),
|
||||
headers: { Accept: 'application/x-protobuf' },
|
||||
});
|
||||
if (response.status === 404 || response.status === 204) {
|
||||
@@ -293,10 +345,70 @@ async function fetchUpstream(address, signal) {
|
||||
await response.body?.cancel().catch(() => {});
|
||||
throw new TileUpstreamError(502, 'Mapillary tile exceeds size cap');
|
||||
}
|
||||
const bytes = Buffer.from(await response.arrayBuffer());
|
||||
if (bytes.length > TILE_MAX_BYTES)
|
||||
throw new TileUpstreamError(502, 'Mapillary tile exceeds size cap');
|
||||
return bytes;
|
||||
return readCapped(response, () => oversize.abort());
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a tile body, counting as it arrives: a chunked or compressed response
|
||||
* has no Content-Length to check first, so the request is aborted as soon as
|
||||
* the running total passes TILE_MAX_BYTES rather than after buffering it all.
|
||||
*/
|
||||
async function readCapped(response, abortRequest) {
|
||||
if (!response.body) return Buffer.alloc(0);
|
||||
const reader = response.body.getReader();
|
||||
const chunks = [];
|
||||
let total = 0;
|
||||
for (;;) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
total += value.byteLength;
|
||||
if (total > TILE_MAX_BYTES) {
|
||||
abortRequest();
|
||||
await reader.cancel().catch(() => {});
|
||||
throw new TileUpstreamError(502, 'Mapillary tile exceeds size cap');
|
||||
}
|
||||
chunks.push(value);
|
||||
}
|
||||
return Buffer.concat(chunks, total);
|
||||
}
|
||||
|
||||
/**
|
||||
* Upstream fetch slots: at most TILE_UPSTREAM_CONCURRENCY run at once, and
|
||||
* further flights wait in order. Replaced whole by the test reset.
|
||||
* @type {{active: number, queue: Array<() => void>}}
|
||||
*/
|
||||
let _slots = { active: 0, queue: [] };
|
||||
|
||||
/**
|
||||
* Wait for an upstream fetch slot. Resolves with its release function. A
|
||||
* flight aborted while it waits leaves the queue and never fetches.
|
||||
* @param {AbortSignal} signal
|
||||
* @returns {Promise<() => void>}
|
||||
*/
|
||||
function acquireUpstreamSlot(signal) {
|
||||
const slots = _slots;
|
||||
return new Promise((resolve, reject) => {
|
||||
if (signal.aborted) return reject(signal.reason);
|
||||
const onAbort = () => {
|
||||
const index = slots.queue.indexOf(grant);
|
||||
if (index !== -1) slots.queue.splice(index, 1);
|
||||
reject(signal.reason);
|
||||
};
|
||||
function grant() {
|
||||
signal.removeEventListener('abort', onAbort);
|
||||
slots.active++;
|
||||
let released = false;
|
||||
resolve(() => {
|
||||
if (released) return;
|
||||
released = true;
|
||||
slots.active--;
|
||||
slots.queue.shift()?.();
|
||||
});
|
||||
}
|
||||
if (slots.active < TILE_UPSTREAM_CONCURRENCY) return grant();
|
||||
signal.addEventListener('abort', onAbort, { once: true });
|
||||
slots.queue.push(grant);
|
||||
});
|
||||
}
|
||||
|
||||
/** Strip the layers this proxy never serves for the address's layer spec. */
|
||||
@@ -328,11 +440,15 @@ export async function fetchTile(request, { signal } = {}) {
|
||||
if (disk) {
|
||||
// Older cache files may still hold the untrimmed tile: trim and replace,
|
||||
// keeping the original fetch time so the tile still expires on schedule.
|
||||
// Concurrent hits on one such tile rewrite it once.
|
||||
const bytes = trim(address, disk.bytes);
|
||||
if (bytes !== disk.bytes) writeDisk(address, bytes, disk.at);
|
||||
if (bytes !== disk.bytes && !diskWritePending(address))
|
||||
writeDisk(address, bytes, disk.at);
|
||||
memoryPut(address.key, bytes, disk.at);
|
||||
return { bytes, source: 'disk', address };
|
||||
}
|
||||
// A caller gone while the disk was read must not start a fetch nobody joins.
|
||||
signal?.throwIfAborted();
|
||||
let flight = _inFlight.get(address.key);
|
||||
const joined = Boolean(flight) && !flight.controller.signal.aborted;
|
||||
if (!joined) flight = startFlight(address);
|
||||
@@ -340,11 +456,17 @@ export async function fetchTile(request, { signal } = {}) {
|
||||
return { bytes, source: joined ? 'inflight' : 'upstream', address };
|
||||
}
|
||||
|
||||
/** Start the one upstream fetch for a tile, owned by the flight itself. */
|
||||
/**
|
||||
* Start the one upstream fetch for a tile, owned by the flight itself. It
|
||||
* waits for an upstream slot first; the slot is held until the body is read.
|
||||
*/
|
||||
function startFlight(address) {
|
||||
const controller = new AbortController();
|
||||
const flight = { controller, waiters: 0, promise: null };
|
||||
flight.promise = fetchUpstream(address, controller.signal)
|
||||
flight.promise = acquireUpstreamSlot(controller.signal)
|
||||
.then((release) =>
|
||||
fetchUpstream(address, controller.signal).finally(release),
|
||||
)
|
||||
.then((raw) => trim(address, raw))
|
||||
.then((bytes) => {
|
||||
memoryPut(address.key, bytes);
|
||||
@@ -399,6 +521,20 @@ export function _resetTileMemoryForTest() {
|
||||
_memoryBytes = 0;
|
||||
_inFlight.clear();
|
||||
_upstreamHold = null;
|
||||
_slots = { active: 0, queue: [] };
|
||||
}
|
||||
|
||||
/** Test seam: how many tiles memory holds and the bytes charged for them. */
|
||||
export function _tileMemoryForTest() {
|
||||
return { entries: _memory.size, bytes: _memoryBytes };
|
||||
}
|
||||
|
||||
/**
|
||||
* Test seam: wait until every background tile write and sweep has settled,
|
||||
* including sweeps those writes start, so a test may remove its cache dir.
|
||||
*/
|
||||
export async function _settleTileWritesForTest() {
|
||||
while (_background.size) await Promise.allSettled([..._background]);
|
||||
}
|
||||
|
||||
/** Test seam: keep the disk cache in `dir` (null restores the default). */
|
||||
|
||||
@@ -10,12 +10,20 @@ import {
|
||||
stripTileLayers,
|
||||
} from '../../server/providers/mapillary/trim.js';
|
||||
import * as tiles from '../../server/providers/mapillary/tiles.js';
|
||||
import {
|
||||
TILE_DISK_FILE_OVERHEAD_BYTES,
|
||||
TILE_MAX_BYTES,
|
||||
TILE_MEMORY_ENTRY_OVERHEAD_BYTES,
|
||||
TILE_ROUTE_MAX_PER_MIN,
|
||||
TILE_UPSTREAM_CONCURRENCY,
|
||||
} from '../../server/providers/mapillary/constants.js';
|
||||
import {
|
||||
fetchTile,
|
||||
normalizeTileAddress,
|
||||
TileRequestError,
|
||||
_resetTileMemoryForTest,
|
||||
_setTileCacheDirForTest,
|
||||
_settleTileWritesForTest,
|
||||
} from '../../server/providers/mapillary/tiles.js';
|
||||
|
||||
// Every tile these tests cache lands in a throwaway directory, never in the
|
||||
@@ -23,8 +31,12 @@ import {
|
||||
const cacheDir = await fsp.mkdtemp(path.join(os.tmpdir(), 'gev-mly-tiles-'));
|
||||
_setTileCacheDirForTest(cacheDir);
|
||||
// The directory is not switched back: a background write still landing must
|
||||
// never sweep the real cache.
|
||||
after(() => fsp.rm(cacheDir, { recursive: true, force: true }));
|
||||
// never sweep the real cache. Background writes and sweeps settle first, so
|
||||
// none lands in the directory while it is being removed.
|
||||
after(async () => {
|
||||
await tiles._settleTileWritesForTest();
|
||||
await fsp.rm(cacheDir, { recursive: true, force: true });
|
||||
});
|
||||
const tileFile = ({ z, x, y }, root = cacheDir) =>
|
||||
path.join(root, 'coverage', String(z), `${x}-${y}.pbf`);
|
||||
const HOUR = 60 * 60 * 1000;
|
||||
@@ -52,7 +64,11 @@ const exists = (file) =>
|
||||
() => false,
|
||||
);
|
||||
|
||||
/** Mount the plugin and return a caller keyed by route. */
|
||||
/**
|
||||
* Mount the plugin and return callers keyed by route. `start` returns at once
|
||||
* with the response, its `close` trigger and the handler's `done` promise;
|
||||
* `call` waits for the handler.
|
||||
*/
|
||||
function install(plugin, mode = 'configureServer') {
|
||||
const routes = new Map();
|
||||
plugin[mode]({
|
||||
@@ -62,10 +78,17 @@ function install(plugin, mode = 'configureServer') {
|
||||
},
|
||||
},
|
||||
});
|
||||
const call = async (route, url = '/', method = 'GET', body) => {
|
||||
const start = (
|
||||
route,
|
||||
url = '/',
|
||||
method = 'GET',
|
||||
body,
|
||||
requestHeaders = {},
|
||||
) => {
|
||||
const handler = routes.get(route);
|
||||
assert.ok(handler, `route ${route} is mounted`);
|
||||
const headers = {};
|
||||
const resListeners = {};
|
||||
const res = {
|
||||
statusCode: 200,
|
||||
headersSent: false,
|
||||
@@ -84,13 +107,15 @@ function install(plugin, mode = 'configureServer') {
|
||||
this.body = payload;
|
||||
this.writableEnded = true;
|
||||
},
|
||||
on() {},
|
||||
on(event, fn) {
|
||||
resListeners[event] = fn;
|
||||
},
|
||||
};
|
||||
const listeners = {};
|
||||
const req = {
|
||||
url,
|
||||
method,
|
||||
headers: {},
|
||||
headers: requestHeaders,
|
||||
on(event, fn) {
|
||||
listeners[event] = fn;
|
||||
},
|
||||
@@ -98,10 +123,15 @@ function install(plugin, mode = 'configureServer') {
|
||||
if (body) yield Buffer.from(body);
|
||||
},
|
||||
};
|
||||
await handler(req, res);
|
||||
const done = Promise.resolve(handler(req, res));
|
||||
return { res, headers, done, close: () => resListeners.close?.() };
|
||||
};
|
||||
const call = async (...args) => {
|
||||
const { res, headers, done } = start(...args);
|
||||
await done;
|
||||
return { ...res, headers };
|
||||
};
|
||||
return { routes, call };
|
||||
return { routes, call, start };
|
||||
}
|
||||
|
||||
const json = (res) => JSON.parse(String(res.body));
|
||||
@@ -170,8 +200,11 @@ async function withUpstream(answer, run) {
|
||||
return answer(calls.length);
|
||||
};
|
||||
try {
|
||||
await run({ calls, call: install(mapillaryProxy()).call });
|
||||
const { call, start } = install(mapillaryProxy());
|
||||
await run({ calls, call, start });
|
||||
} finally {
|
||||
// No background write or sweep (dated by a mocked clock) outlives the test.
|
||||
await tiles._settleTileWritesForTest();
|
||||
globalThis.fetch = savedFetch;
|
||||
if (savedToken === undefined) delete process.env.MAPILLARY_CLIENT_TOKEN;
|
||||
else process.env.MAPILLARY_CLIENT_TOKEN = savedToken;
|
||||
@@ -339,15 +372,15 @@ test('a tile held in memory past the 24 h TTL is fetched again', async (t) => {
|
||||
const tile = { layer: 'coverage', z: 14, x: 3, y: 3 };
|
||||
await withDeferredUpstream([tile], async ({ upstream, settle }) => {
|
||||
const first = fetchTile(tile);
|
||||
await settle();
|
||||
await waitFor(() => upstream.length === 1, 'the upstream fetch');
|
||||
upstream[0].release();
|
||||
assert.equal((await first).source, 'upstream');
|
||||
assert.equal((await fetchTile(tile)).source, 'memory');
|
||||
const now = Date.now();
|
||||
t.mock.method(Date, 'now', () => now + 25 * 60 * 60 * 1000);
|
||||
const again = fetchTile(tile);
|
||||
await settle();
|
||||
assert.equal(upstream.length, 2, 'neither cache answers a stale tile');
|
||||
// Neither cache answers a stale tile: Mapillary is asked again.
|
||||
await waitFor(() => upstream.length === 2, 'a second upstream fetch');
|
||||
upstream[1].release();
|
||||
assert.equal((await again).source, 'upstream');
|
||||
});
|
||||
@@ -370,10 +403,8 @@ test('a disk tile keeps its age in memory and through a retrim rewrite (review P
|
||||
assert.deepEqual(listTileLayers(first.bytes), ['sequence']);
|
||||
// The untrimmed file is replaced by the trimmed tile, still dated when
|
||||
// Mapillary served it.
|
||||
await waitFor(
|
||||
async () => (await fsp.stat(file)).size === first.bytes.length,
|
||||
'the trimmed rewrite',
|
||||
);
|
||||
await tiles._settleTileWritesForTest();
|
||||
assert.equal((await fsp.stat(file)).size, first.bytes.length);
|
||||
assert.ok(
|
||||
Math.abs((await fsp.stat(file)).mtimeMs - writtenAt) < 1000,
|
||||
'the rewrite keeps the original mtime',
|
||||
@@ -383,11 +414,8 @@ test('a disk tile keeps its age in memory and through a retrim rewrite (review P
|
||||
const later = await fetchTile(address);
|
||||
assert.equal(later.source, 'upstream', '25 h old: no cache serves it');
|
||||
assert.equal(calls.length, 1);
|
||||
// Let the refetched tile land before the next test moves the cache.
|
||||
await waitFor(
|
||||
async () => (await fsp.stat(file)).mtimeMs > writtenAt + HOUR,
|
||||
'the refetched tile write',
|
||||
);
|
||||
await tiles._settleTileWritesForTest();
|
||||
assert.ok((await fsp.stat(file)).mtimeMs > writtenAt + HOUR);
|
||||
},
|
||||
);
|
||||
});
|
||||
@@ -399,6 +427,7 @@ async function withCacheDir(run) {
|
||||
try {
|
||||
await run(dir);
|
||||
} finally {
|
||||
await tiles._settleTileWritesForTest();
|
||||
tiles._setTileCacheDirForTest(cacheDir);
|
||||
await fsp.rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
@@ -425,8 +454,9 @@ test('the disk sweep evicts the oldest tiles past the size cap (review P2)', asy
|
||||
});
|
||||
for (const { file, age } of files)
|
||||
await writeAged(file, Buffer.alloc(100, age), age * HOUR);
|
||||
const result = await tiles.sweepTileDisk({ maxBytes: 250 });
|
||||
assert.deepEqual(result, { removed: 1, bytes: 200 });
|
||||
const charged = 100 + TILE_DISK_FILE_OVERHEAD_BYTES;
|
||||
const result = await tiles.sweepTileDisk({ maxBytes: 2.5 * charged });
|
||||
assert.deepEqual(result, { removed: 1, bytes: 2 * charged });
|
||||
assert.equal(await exists(files[0].file), false, 'the oldest goes first');
|
||||
assert.equal(await exists(files[1].file), true);
|
||||
assert.equal(await exists(files[2].file), true);
|
||||
@@ -500,6 +530,7 @@ async function withDeferredUpstream(tiles, run) {
|
||||
try {
|
||||
await run({ upstream, settle, body });
|
||||
} finally {
|
||||
await _settleTileWritesForTest();
|
||||
globalThis.fetch = savedFetch;
|
||||
if (savedToken === undefined) delete process.env.MAPILLARY_CLIENT_TOKEN;
|
||||
else process.env.MAPILLARY_CLIENT_TOKEN = savedToken;
|
||||
@@ -513,7 +544,7 @@ test('a joined tile request survives the first caller abandoning it', async () =
|
||||
const first = new AbortController();
|
||||
const second = new AbortController();
|
||||
const a = fetchTile(tile, { signal: first.signal });
|
||||
await settle();
|
||||
await waitFor(() => upstream.length === 1, 'the upstream fetch');
|
||||
const b = fetchTile(tile, { signal: second.signal });
|
||||
await settle();
|
||||
assert.equal(upstream.length, 1, 'one upstream request for both');
|
||||
@@ -534,7 +565,7 @@ test('a joined caller that aborts leaves at once and the first still gets the ti
|
||||
const first = new AbortController();
|
||||
const second = new AbortController();
|
||||
const a = fetchTile(tile, { signal: first.signal });
|
||||
await settle();
|
||||
await waitFor(() => upstream.length === 1, 'the upstream fetch');
|
||||
const b = fetchTile(tile, { signal: second.signal });
|
||||
await settle();
|
||||
second.abort();
|
||||
@@ -551,7 +582,7 @@ test('the upstream tile fetch is cancelled once every waiter has left', async ()
|
||||
const first = new AbortController();
|
||||
const second = new AbortController();
|
||||
const a = fetchTile(tile, { signal: first.signal });
|
||||
await settle();
|
||||
await waitFor(() => upstream.length === 1, 'the upstream fetch');
|
||||
const b = fetchTile(tile, { signal: second.signal });
|
||||
await settle();
|
||||
first.abort();
|
||||
@@ -562,13 +593,419 @@ test('the upstream tile fetch is cancelled once every waiter has left', async ()
|
||||
assert.equal(upstream[0].aborted, true, 'the last waiter cancels it');
|
||||
// A later request for the same tile starts a fresh fetch.
|
||||
const again = fetchTile(tile);
|
||||
await settle();
|
||||
assert.equal(upstream.length, 2);
|
||||
await waitFor(() => upstream.length === 2, 'a fresh fetch');
|
||||
upstream[1].release();
|
||||
assert.equal((await again).source, 'upstream');
|
||||
});
|
||||
});
|
||||
|
||||
// ── Route: success path, refusals, gate and limiter (review gekh) ──
|
||||
|
||||
/** Every header value and the body, as text, for a token-leak check. */
|
||||
const leakText = (res) =>
|
||||
[
|
||||
...Object.values(res.headers).map(String),
|
||||
Buffer.isBuffer(res.body) ? res.body.toString('latin1') : String(res.body),
|
||||
].join('\n');
|
||||
|
||||
test('the tile route serves a trimmed protobuf tile, then from memory (review gekh P0)', async () => {
|
||||
const address = { layer: 'coverage', z: 14, x: 9, y: 9 };
|
||||
await fsp.rm(tileFile(address), { force: true });
|
||||
const upstream = tile([
|
||||
{ name: 'sequence' },
|
||||
{ name: 'image', payload: Buffer.alloc(4000, 1) },
|
||||
]);
|
||||
await withUpstream(
|
||||
() =>
|
||||
new Response(upstream, {
|
||||
status: 200,
|
||||
headers: { 'content-type': 'application/x-protobuf' },
|
||||
}),
|
||||
async ({ calls, call }) => {
|
||||
const res = await call('/api/mapillary/tiles', '/coverage/14/9/9');
|
||||
assert.equal(res.statusCode, 200);
|
||||
assert.equal(res.headers['content-type'], 'application/x-protobuf');
|
||||
assert.equal(res.headers['cache-control'], 'public, max-age=3600');
|
||||
assert.equal(res.headers['x-gev-cache'], 'upstream');
|
||||
assert.ok(Buffer.isBuffer(res.body));
|
||||
assert.deepEqual(listTileLayers(res.body), ['sequence']);
|
||||
assert.deepEqual(res.body, tile([{ name: 'sequence' }]));
|
||||
assert.doesNotMatch(leakText(res), /MLY\|/);
|
||||
const again = await call('/api/mapillary/tiles', '/coverage/14/9/9');
|
||||
assert.equal(again.statusCode, 200);
|
||||
assert.equal(again.headers['x-gev-cache'], 'memory');
|
||||
assert.deepEqual(again.body, res.body);
|
||||
assert.equal(calls.length, 1);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
for (const status of [204, 404])
|
||||
test(`an empty upstream tile (${status}) is a 204 with no body (review gekh P0)`, async () => {
|
||||
const address = { layer: 'coverage', z: 14, x: 10, y: status };
|
||||
await fsp.rm(tileFile(address), { force: true });
|
||||
await withUpstream(
|
||||
() => new Response(null, { status }),
|
||||
async ({ call }) => {
|
||||
const res = await call(
|
||||
'/api/mapillary/tiles',
|
||||
`/coverage/14/10/${status}`,
|
||||
);
|
||||
assert.equal(res.statusCode, 204);
|
||||
assert.equal(res.body, undefined);
|
||||
assert.equal(res.headers['content-type'], 'application/x-protobuf');
|
||||
assert.equal(res.headers['x-gev-cache'], 'upstream');
|
||||
assert.doesNotMatch(leakText(res), /MLY\|/);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test('closing the response mid-flight aborts the upstream fetch (review gekh P0)', async () => {
|
||||
const address = { layer: 'coverage', z: 14, x: 11, y: 11 };
|
||||
await withDeferredUpstream([address], async ({ upstream }) => {
|
||||
const { start } = install(mapillaryProxy());
|
||||
const pending = start('/api/mapillary/tiles', '/coverage/14/11/11');
|
||||
await waitFor(() => upstream.length === 1, 'the upstream fetch');
|
||||
assert.equal(upstream[0].aborted, false);
|
||||
pending.close();
|
||||
await pending.done;
|
||||
assert.equal(upstream[0].aborted, true, 'the last waiter left');
|
||||
assert.equal(pending.res.writableEnded, false, 'nothing is written');
|
||||
});
|
||||
});
|
||||
|
||||
for (const status of [400, 410, 422])
|
||||
test(`an upstream ${status} is a 502, not the client's fault (review gekh)`, async () => {
|
||||
await withUpstream(
|
||||
() => new Response('{}', { status }),
|
||||
async ({ call }) => {
|
||||
const res = await call('/api/mapillary/tiles', '/coverage/14/12/12');
|
||||
assert.equal(res.statusCode, 502);
|
||||
assert.deepEqual(json(res), {
|
||||
error: `Mapillary tiles HTTP ${status}`,
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test('cross-site requests are refused on both routes; the app itself passes (review gekh)', async () => {
|
||||
await withUpstream(
|
||||
() => new Response(tile([{ name: 'sequence' }]), { status: 200 }),
|
||||
async ({ calls, call }) => {
|
||||
const host = 'localhost:5173';
|
||||
const refused = [
|
||||
{ host, 'sec-fetch-site': 'cross-site' }, // <img>, navigation
|
||||
{ host, 'sec-fetch-site': 'same-site' },
|
||||
{ host, origin: 'https://evil.example' }, // fetch from another page
|
||||
{ host, origin: 'null' }, // sandboxed frame
|
||||
{ host, 'x-forwarded-for': '203.0.113.9' }, // through a proxy
|
||||
];
|
||||
for (const headers of refused)
|
||||
for (const [route, url] of [
|
||||
['/api/mapillary/status', '/'],
|
||||
['/api/mapillary/tiles', '/coverage/14/13/13'],
|
||||
]) {
|
||||
const res = await call(route, url, 'GET', undefined, headers);
|
||||
assert.equal(
|
||||
res.statusCode,
|
||||
403,
|
||||
`${route} ${JSON.stringify(headers)}`,
|
||||
);
|
||||
}
|
||||
assert.equal(calls.length, 0, 'Mapillary is never asked');
|
||||
const app = {
|
||||
host,
|
||||
origin: `http://${host}`,
|
||||
'sec-fetch-site': 'same-origin',
|
||||
};
|
||||
const status = await call(
|
||||
'/api/mapillary/status',
|
||||
'/',
|
||||
'GET',
|
||||
undefined,
|
||||
app,
|
||||
);
|
||||
assert.equal(status.statusCode, 200);
|
||||
assert.deepEqual(json(status), { configured: true });
|
||||
const res = await call(
|
||||
'/api/mapillary/tiles',
|
||||
'/coverage/14/13/14',
|
||||
'GET',
|
||||
undefined,
|
||||
{ host, 'sec-fetch-site': 'same-origin' }, // a same-origin GET has no Origin
|
||||
);
|
||||
assert.notEqual(res.statusCode, 403);
|
||||
assert.equal(calls.length, 1);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test('the tile route answers 429 past its per-IP budget (review gekh)', async () => {
|
||||
const saved = process.env.MAPILLARY_CLIENT_TOKEN;
|
||||
delete process.env.MAPILLARY_CLIENT_TOKEN;
|
||||
try {
|
||||
const { call } = install(mapillaryProxy());
|
||||
for (let i = 0; i < TILE_ROUTE_MAX_PER_MIN; i++) {
|
||||
const res = await call('/api/mapillary/tiles', '/coverage/14/1/2');
|
||||
assert.equal(res.statusCode, 503, `request ${i + 1} is within budget`);
|
||||
}
|
||||
const over = await call('/api/mapillary/tiles', '/coverage/14/1/2');
|
||||
assert.equal(over.statusCode, 429);
|
||||
assert.equal(over.headers['retry-after'], '5');
|
||||
assert.deepEqual(json(over), {
|
||||
error: 'Too many tile requests',
|
||||
retryAfter: 5,
|
||||
});
|
||||
const status = await call('/api/mapillary/status');
|
||||
assert.equal(status.statusCode, 200, 'the status route is not limited');
|
||||
} finally {
|
||||
if (saved === undefined) delete process.env.MAPILLARY_CLIENT_TOKEN;
|
||||
else process.env.MAPILLARY_CLIENT_TOKEN = saved;
|
||||
}
|
||||
});
|
||||
|
||||
// ── Tile cache and upstream bounds (review gekh) ──
|
||||
|
||||
test('concurrent disk hits on an untrimmed tile rewrite it once, keeping its age (review gekh)', async (t) => {
|
||||
const address = { layer: 'coverage', z: 14, x: 14, y: 14 };
|
||||
const file = tileFile(address);
|
||||
await writeAged(
|
||||
file,
|
||||
tile([
|
||||
{ name: 'sequence' },
|
||||
{ name: 'image', payload: Buffer.alloc(4000, 1) },
|
||||
]),
|
||||
23 * HOUR,
|
||||
);
|
||||
const writtenAt = (await fsp.stat(file)).mtimeMs;
|
||||
const writeFile = fsp.writeFile;
|
||||
const writes = [];
|
||||
t.mock.method(fsp, 'writeFile', (target, ...rest) => {
|
||||
if (String(target).startsWith(file)) writes.push(String(target));
|
||||
return writeFile.call(fsp, target, ...rest);
|
||||
});
|
||||
const warn = t.mock.method(console, 'warn', () => {});
|
||||
await withUpstream(
|
||||
() => assert.fail('a disk hit never asks Mapillary'),
|
||||
async () => {
|
||||
const hits = await Promise.all([1, 2, 3].map(() => fetchTile(address)));
|
||||
for (const hit of hits) {
|
||||
assert.equal(hit.source, 'disk');
|
||||
assert.deepEqual(listTileLayers(hit.bytes), ['sequence']);
|
||||
}
|
||||
await tiles._settleTileWritesForTest();
|
||||
},
|
||||
);
|
||||
assert.equal(writes.length, 1, 'one rewrite for three hits');
|
||||
assert.equal(warn.mock.callCount(), 0);
|
||||
const stat = await fsp.stat(file);
|
||||
assert.equal(stat.size, tile([{ name: 'sequence' }]).length);
|
||||
assert.ok(Math.abs(stat.mtimeMs - writtenAt) < 1000, 'no fresh 24 h of life');
|
||||
});
|
||||
|
||||
test('a rewrite overtaken by a fresh tile write steps aside cleanly (review gekh)', async (t) => {
|
||||
const address = { layer: 'coverage', z: 14, x: 15, y: 15 };
|
||||
const file = tileFile(address);
|
||||
await writeAged(
|
||||
file,
|
||||
tile([
|
||||
{ name: 'sequence' },
|
||||
{ name: 'image', payload: Buffer.alloc(4000, 1) },
|
||||
]),
|
||||
23 * HOUR,
|
||||
);
|
||||
const writtenAt = (await fsp.stat(file)).mtimeMs;
|
||||
// Hold the retrim's re-dating step until the fresh write has landed.
|
||||
const utimes = fsp.utimes;
|
||||
let held = null;
|
||||
let release;
|
||||
const gate = new Promise((resolve) => (release = resolve));
|
||||
t.mock.method(fsp, 'utimes', async (target, ...rest) => {
|
||||
if (String(target).startsWith(file)) {
|
||||
held = String(target);
|
||||
await gate;
|
||||
}
|
||||
return utimes.call(fsp, target, ...rest);
|
||||
});
|
||||
const warn = t.mock.method(console, 'warn', () => {});
|
||||
const fresh = tile([{ name: 'sequence' }, { name: 'overview' }]);
|
||||
await withUpstream(
|
||||
() => new Response(fresh, { status: 200 }),
|
||||
async ({ calls }) => {
|
||||
assert.equal((await fetchTile(address)).source, 'disk');
|
||||
await waitFor(() => held, 'the retrim write');
|
||||
// Two hours on, the disk copy is past its 24 h: Mapillary is asked.
|
||||
_resetTileMemoryForTest();
|
||||
const now = Date.now();
|
||||
t.mock.method(Date, 'now', () => now + 2 * HOUR);
|
||||
assert.equal((await fetchTile(address)).source, 'upstream');
|
||||
assert.equal(calls.length, 1);
|
||||
await waitFor(
|
||||
async () => (await fsp.stat(file)).mtimeMs > writtenAt + HOUR,
|
||||
'the fresh tile write',
|
||||
);
|
||||
release();
|
||||
await tiles._settleTileWritesForTest();
|
||||
},
|
||||
);
|
||||
assert.equal(warn.mock.callCount(), 0, 'no ENOENT from a shared temp file');
|
||||
assert.deepEqual(await fsp.readFile(file), fresh, 'the fresh tile stays');
|
||||
assert.ok((await fsp.stat(file)).mtimeMs > writtenAt + HOUR);
|
||||
const left = await fsp.readdir(path.dirname(file));
|
||||
assert.deepEqual(
|
||||
left.filter((name) => name.endsWith('.tmp')),
|
||||
[],
|
||||
'no temporary file is left behind',
|
||||
);
|
||||
});
|
||||
|
||||
test('at most a few upstream fetches run at once; a queued abort never fetches (review gekh)', async () => {
|
||||
const addresses = Array.from(
|
||||
{ length: TILE_UPSTREAM_CONCURRENCY + 2 },
|
||||
(_, i) => ({
|
||||
layer: 'coverage',
|
||||
z: 13,
|
||||
x: 100 + i,
|
||||
y: 100,
|
||||
}),
|
||||
);
|
||||
await withDeferredUpstream(addresses, async ({ upstream, settle }) => {
|
||||
const controllers = addresses.map(() => new AbortController());
|
||||
const begin = (i) =>
|
||||
fetchTile(addresses[i], { signal: controllers[i].signal });
|
||||
// Fill every slot first, so the two extra misses are the ones that wait.
|
||||
const results = [];
|
||||
for (let i = 0; i < TILE_UPSTREAM_CONCURRENCY; i++) results.push(begin(i));
|
||||
await waitFor(
|
||||
() => upstream.length === TILE_UPSTREAM_CONCURRENCY,
|
||||
'every slot in use',
|
||||
);
|
||||
const queued = TILE_UPSTREAM_CONCURRENCY;
|
||||
const aborted = queued + 1;
|
||||
results.push(begin(queued), begin(aborted));
|
||||
await settle();
|
||||
assert.equal(upstream.length, TILE_UPSTREAM_CONCURRENCY, 'the rest wait');
|
||||
controllers[aborted].abort();
|
||||
await assert.rejects(results[aborted], { name: 'AbortError' });
|
||||
upstream[0].release();
|
||||
await waitFor(
|
||||
() => upstream.length === TILE_UPSTREAM_CONCURRENCY + 1,
|
||||
'the queued miss taking the freed slot',
|
||||
);
|
||||
assert.match(upstream.at(-1).url, new RegExp(`/13/${100 + queued}/100\\?`));
|
||||
for (const call of upstream) call.release();
|
||||
for (const result of results.slice(0, aborted))
|
||||
assert.equal((await result).source, 'upstream');
|
||||
await settle();
|
||||
assert.equal(
|
||||
upstream.length,
|
||||
TILE_UPSTREAM_CONCURRENCY + 1,
|
||||
'the aborted miss never reached Mapillary',
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
test('a caller gone while the disk cache is read starts no upstream fetch (review gekh)', async () => {
|
||||
const address = { layer: 'coverage', z: 13, x: 120, y: 100 };
|
||||
await withDeferredUpstream([address], async ({ upstream, settle }) => {
|
||||
const controller = new AbortController();
|
||||
const pending = fetchTile(address, { signal: controller.signal });
|
||||
controller.abort(); // fetchTile is still reading the disk cache
|
||||
await assert.rejects(pending, { name: 'AbortError' });
|
||||
await settle();
|
||||
assert.equal(upstream.length, 0, 'Mapillary is not asked');
|
||||
});
|
||||
});
|
||||
|
||||
test('empty tiles are charged against the memory budget (review gekh)', async () => {
|
||||
const addresses = [16, 17, 18].map((x) => ({
|
||||
layer: 'coverage',
|
||||
z: 14,
|
||||
x,
|
||||
y: 16,
|
||||
}));
|
||||
for (const address of addresses)
|
||||
await fsp.rm(tileFile(address), { force: true });
|
||||
await withUpstream(
|
||||
() => new Response(null, { status: 204 }),
|
||||
async () => {
|
||||
for (const address of addresses)
|
||||
assert.equal((await fetchTile(address)).bytes.length, 0);
|
||||
assert.deepEqual(tiles._tileMemoryForTest(), {
|
||||
entries: 3,
|
||||
bytes: 3 * TILE_MEMORY_ENTRY_OVERHEAD_BYTES,
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test('empty tile files count toward the disk cap (review gekh)', async () => {
|
||||
await withCacheDir(async (dir) => {
|
||||
const files = [3, 2, 1].map((age) => ({
|
||||
file: tileFile({ z: 13, x: age, y: age }, dir),
|
||||
age,
|
||||
}));
|
||||
for (const { file, age } of files)
|
||||
await writeAged(file, Buffer.alloc(0), age * HOUR);
|
||||
const result = await tiles.sweepTileDisk({
|
||||
maxBytes: 2 * TILE_DISK_FILE_OVERHEAD_BYTES,
|
||||
});
|
||||
assert.deepEqual(result, {
|
||||
removed: 1,
|
||||
bytes: 2 * TILE_DISK_FILE_OVERHEAD_BYTES,
|
||||
});
|
||||
assert.equal(await exists(files[0].file), false, 'the oldest goes first');
|
||||
assert.equal(await exists(files[2].file), true);
|
||||
});
|
||||
});
|
||||
|
||||
test('a body past the size cap aborts the upstream request as it streams (review gekh)', async () => {
|
||||
const address = { layer: 'coverage', z: 14, x: 19, y: 19 };
|
||||
await fsp.rm(tileFile(address), { force: true });
|
||||
const MB = 1024 * 1024;
|
||||
const chunk = new Uint8Array(MB);
|
||||
let pulled = 0;
|
||||
let cancelled = false;
|
||||
let upstreamSignal = null;
|
||||
const savedFetch = globalThis.fetch;
|
||||
const savedToken = process.env.MAPILLARY_CLIENT_TOKEN;
|
||||
process.env.MAPILLARY_CLIENT_TOKEN = 'MLY|test|token';
|
||||
_resetTileMemoryForTest();
|
||||
// Chunked, no Content-Length: only a running count can see it is too big.
|
||||
globalThis.fetch = async (_url, { signal } = {}) => {
|
||||
upstreamSignal = signal;
|
||||
const body = new ReadableStream({
|
||||
pull(controller) {
|
||||
controller.enqueue(chunk);
|
||||
pulled += chunk.byteLength;
|
||||
if (pulled >= TILE_MAX_BYTES + 8 * MB) controller.close();
|
||||
},
|
||||
cancel() {
|
||||
cancelled = true;
|
||||
},
|
||||
});
|
||||
return new Response(body, { status: 200 });
|
||||
};
|
||||
try {
|
||||
await assert.rejects(fetchTile(address), {
|
||||
name: 'TileUpstreamError',
|
||||
status: 502,
|
||||
message: 'Mapillary tile exceeds size cap',
|
||||
});
|
||||
assert.equal(upstreamSignal.aborted, true, 'the request is aborted');
|
||||
assert.equal(cancelled, true, 'the body is cancelled');
|
||||
assert.ok(
|
||||
pulled <= TILE_MAX_BYTES + 2 * MB,
|
||||
`stopped after ${pulled} bytes`,
|
||||
);
|
||||
} finally {
|
||||
globalThis.fetch = savedFetch;
|
||||
if (savedToken === undefined) delete process.env.MAPILLARY_CLIENT_TOKEN;
|
||||
else process.env.MAPILLARY_CLIENT_TOKEN = savedToken;
|
||||
_resetTileMemoryForTest();
|
||||
}
|
||||
});
|
||||
|
||||
// ── Tile trimming (relocated from server/providers/mapillary/trim.test.mjs) ──
|
||||
/** Build a minimal MVT: layers with a name, a version and one opaque feature. */
|
||||
function tile(layers) {
|
||||
|
||||
@@ -1,34 +1,24 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import test from 'node:test';
|
||||
import { createApplicationStreetLevel } from '../app/layers/streetLevel.js';
|
||||
import {
|
||||
fixtureTile,
|
||||
EXPECTED_PROVIDERS,
|
||||
isCollapsed,
|
||||
RAIL_ORDER,
|
||||
VIEWPORTS,
|
||||
} from '../../scripts/qa-street-level.mjs';
|
||||
|
||||
test('the QA harness exercises both review viewports', () => {
|
||||
assert.deepEqual(
|
||||
VIEWPORTS.map((v) => `${v.width}x${v.height}`),
|
||||
['1440x900', '1280x800'],
|
||||
);
|
||||
test('the harness expects exactly the providers the app registers, in chip order', () => {
|
||||
// Any method a provider asks its source for is a no-op: only ids matter.
|
||||
const source = new Proxy({}, { get: () => () => {} });
|
||||
const layer = createApplicationStreetLevel({
|
||||
surface: null,
|
||||
sources: { mapillary: source },
|
||||
});
|
||||
assert.deepEqual([...EXPECTED_PROVIDERS], [...layer.providerIds]);
|
||||
});
|
||||
|
||||
test('the harness expects exactly the registered providers', () => {
|
||||
assert.deepEqual(EXPECTED_PROVIDERS, ['mapillary']);
|
||||
});
|
||||
|
||||
test('rail order puts Street Level between CCTV and Context', () => {
|
||||
assert.deepEqual(RAIL_ORDER, [
|
||||
'pp-toggles',
|
||||
'cctv-panel',
|
||||
'weather-panel',
|
||||
'recent-imagery-panel',
|
||||
'street-level-panel',
|
||||
'global-context-panel',
|
||||
]);
|
||||
test('a panel reads as collapsed only with the collapsed class', () => {
|
||||
assert.equal(isCollapsed(['panel-collapsible', 'collapsed']), true);
|
||||
assert.equal(isCollapsed(['panel-collapsible']), false);
|
||||
});
|
||||
|
||||
Reference in new issue
Block a user