diff --git a/CHANGELOG.md b/CHANGELOG.md index 09e0bf056..8e0c5f754 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,58 @@ # Changelog +- CCTV cameras whose bearing is a guess now say so. Packs mark bearings derived + from a hash of the camera id as `headingConfidence: 'low'`, but nothing read the + flag, so roughly 70% of a default catalog rendered like surveyed facings. The HUD + now reads `HDG n° (ESTIMATED)` and the coverage wireframe draws dashed; manual + calibrations and curated poses are never marked estimated (bassem chagra, #643). + +- Report which upstream declined a Street Traffic road load. The layer row now + reads `Overpass rate-limited`, `Overpass timed out`, or + `Overpass refused the road query (HTTP 406)` instead of a general + "Road data temporarily unavailable", so a reader is not sent to check a + TomTom key when the public OpenStreetMap mirrors are the side that failed. + The proxy's own 502 (every mirror unreachable) and 503 (local limiter busy) + read `Overpass mirrors unreachable` and `Overpass temporarily unavailable`. + Failures the layer cannot classify keep the general line (daikaginza, #665). + +- Bound the client terrain-height cache at 20 000 entries with least-recently-used eviction, so a long session no longer retains every coordinate it ever resolved. Consumer reads promote their entry and a batch still reports every point it resolved (Pedro Lobato, #594). + +- Release CCTV media streams whose upstream falls silent after answering. The + 15-second media deadline covered only the wait for response headers, so a + camera that replied and then stopped sending held both the proxy connection + and its upstream socket open for as long as the camera host allowed; the + declared `Content-Length` ceiling was the only body bound, and a chunked or + length-less body had none. A 30-second idle deadline now bounds the gap + between upstream chunks and releases a body that has gone silent. It is + rescheduled while the response is still waiting to drain, so a viewer on a + slow link is not mistaken for a dead camera; live feeds are unaffected + (Ethan Stoner, #688). + +- Aircraft track backfill (`/api/opensky-track`, `/api/adsblol/trace`) now + answers 502 when the upstream body exceeds the 5 MB cap, instead of a 200 + whose error body the client read as an empty track. The failure is cached + like other upstream errors, so retries inside the 60 s window do not spend + OpenSky credits (Raushankumar0720, #720, #722). + +- Saving a key from Provider Settings works again on Macs where Nix or + Homebrew coreutils sit ahead of `/bin` on `PATH`. The credential hardener + now spawns Apple's `/bin/chmod -N` by absolute path; GNU `chmod` has no `-N`, + so the ACL strip failed closed and every save was refused (Arthur Bogaart, #694). + +- Render on iPad and iPhone instead of stopping with "An error occurred while + rendering." Cesium's per-vertex model atmosphere binds shader `out` + parameters directly to varyings, which Apple's Metal/ANGLE backend cannot + link, so the program failed and the render loop was torn down. The stage is + now kept out of the pipeline on affected devices by clearing + `scene.fog.renderable`, which leaves `fog.enabled` — and the fog density that + drives 3D Tiles refinement — untouched. Detection is a WebGL2 link probe of + the same pattern, so a future driver fix restores the effect with no code + change, with iOS/iPadOS detection as a backstop. Sky atmosphere and the + ground-atmosphere fragment path route through locals and are unaffected; + affected devices lose distance fog on 3D tiles and on the globe basemaps. + `src/app/atmosphereCompat.test.mjs` pins the probe, the platform matrix and + the `renderable`-not-`enabled` choice (KnottyDyes, #705). + - Make share-link layer tokens durable allocations instead of ad hoc picks. Existing one-character mappings are pinned permanently, while new layers use the next free single-character digit, then two-character base-36 tokens. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1957b66dd..cdfbb6547 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -91,6 +91,18 @@ this checks that every published assignment remains reserved and that new assignments consume free digits before the two-character tokens, in order. Pull-request CI runs the same check against `main`. +The helper reads the local ledger; it does not run during a rebase or edit files. +If another PR publishes your provisional token first, keep every published +registry entry and ledger reservation from the refreshed `main`. Temporarily +remove only your still-unpublished layer's registry entry and ledger row so the +local registry is valid again, then rerun `npm run layer-token:next -- `. +For example, if PR A publishes `0` while PR B still provisionally uses `0`, +PR B removes only its own provisional entry, reruns the helper, and receives +`3`. Put `3` on PR B's registry entry and ledger row, update its pinned count +and token tests, then rerun `npm run layer-token:check -- --base-ref origin/main`. +An invalid duplicate ledger can fail during module import before the helper +prints anything; never fix that by renaming or deleting PR A's published token. + ## Good first contributions The highest-leverage places to jump in: diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index de6c4ce20..8a6cde193 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -701,6 +701,25 @@ snapshots, lifecycle actions and row descriptors. Remount and teardown remove listeners and row subscriptions; obsolete completions do not repaint old rows. The clear control presents busy state while its existing action owns the transaction. +## Model atmosphere on Apple Metal + +Cesium's per-vertex model atmosphere is kept out of the pipeline on devices +whose driver cannot link it. `AtmosphereStageVS` binds shader `out` parameters +directly to varyings, which ANGLE's Metal backend rejects at link time, tearing +down the render loop on iPadOS and iOS. `ModelSceneGraph.configurePipeline` +attaches that stage only when `fog.enabled && fog.renderable`, so the viewer +clears `scene.fog.renderable` and leaves `fog.enabled` — and the fog density +that drives 3D Tiles screen-space-error scaling — intact. Sky atmosphere and the +ground-atmosphere fragment path route through locals and are unaffected. +Affected devices lose distance fog on 3D tiles and on globe basemaps (Esri, +Bing, keyless), since the globe fog shader uses the same `renderable` flag. +The probe releases its throwaway WebGL2 context immediately. + +Detection is a WebGL2 link probe of the same out-parameter/varying pattern, so a +future driver fix restores the effect with no code change; iOS/iPadOS platform +detection is the backstop for when no probe context can be created. Applied +during viewer construction, before any tile builds a draw command. + ## Map Source control ownership Map Source controls own chip listeners, source-state subscriptions and selection @@ -794,10 +813,12 @@ their `configured: false` response. CCTV media waits at most 15 seconds for upstream response headers and returns 504 on timeout. Its timer stops when headers arrive, so live bodies can continue -streaming; body idle deadlines are separate from this header deadline. Error -responses are cancelled. Buffered snapshots have a 16 MiB streaming cap; an -oversized image remains an upstream miss and uses the normal fallback chain. -The existing declared media size ceiling remains 64 MiB. +streaming; a separate 30-second idle deadline then bounds the gap between +upstream chunks and releases a body that has gone silent. A response that is +still waiting to drain reschedules that deadline, so a slow client does not read +as a stalled upstream. Error responses are cancelled. Buffered snapshots have a +16 MiB streaming cap; an oversized image remains an upstream miss and uses the +normal fallback chain. The existing declared media size ceiling remains 64 MiB. ## GBFS upstream bounds @@ -858,6 +879,22 @@ under `gods-eye-view/server/providers/`. Portable terrain mechanics, traffic til math and GBFS source rules are available under `gods-eye-view/sources/`. The browser layers and their rendering remain in their existing modules. +## Terrain height cache bound + +The client terrain-height resolver's in-memory cache is bounded at 20 000 +entries (`DEFAULT_MAX_CACHE_ENTRIES`, overridable per instance through +`createTerrainHeights({ maxCacheEntries })`). Coordinate rounding makes repeated +visits to one place share a key but does not bound how many distinct places a +session visits, and the cache previously had no eviction, so a long session +retained every coordinate it ever resolved. Eviction is least-recently-used: +a consumer read promotes its entry, so the cell being watched is not dropped for +having been resolved early. The bound sits far above a city-scale session, so +normal use never evicts and issues no additional proxy requests. A batch's +results are assembled from what that call resolved, so eviction during a large +batch cannot report a just-resolved point as unresolved. Re:Earth and +geoid-fallback entry semantics, the fallback cooldown and the abort-time flush +are unchanged. + ## Landmark annotation identity When a landmark geocode contains only address components, annotations retain @@ -3347,6 +3384,12 @@ silently demoting every later lookup for the session. - **The DENSE chip reports the dense LOAD, not the catalog param.** The param flips synchronously while the Starlink shell takes seconds to arrive over a chunked load, and CelesTrak 502s that feed regularly. So the chip reads `DENSE ···` (busy, disabled) while loading, ACTIVE only once dense points are actually on screen, and `DENSE ✕` with the reason on hover when the load fails — a failure also reverts `catalog` to `core`, drops any partial chunk, and leaves the chip clickable to retry. A load is judged by points added, not by HTTP status: a 200 carrying an empty body, a passed-through HTML error page, or only TLEs the core catalog already owns fails with the same revert semantics as a 502. Any explicit request for `core` clears a latched error even when the mode does not change, so a Space Missions restore of an already-core snapshot never leaves the user with a failure they did not cause. Because the load settles asynchronously, the layer pushes a re-render through the optional `setRowControlsListener()` hook; nothing else would repaint that row before the 5-minute catalog refresh, so the count and legend would otherwise sit stale. - **A dependency owner takes the row with it.** Space Missions borrows this layer for TLE lookup with `showPoints:false`; while points are hidden the layer returns empty row controls, so the legend never describes an empty sky and the chip cannot accept a write that the owner's restore would silently revert. - The detection-overlay record cache (`_detectionObjects`) is cleared with the catalog on every rebuild: it stamps id/class at creation only, and a rebuild can re-tag a satellite when a partial CelesTrak outage changes which group wins dedupe. +- **CCTV estimated bearings:** a camera whose pack marks `headingConfidence: 'low'` (the id-hash + `fallbackHeadingFromId` bearing) shows `HDG n° (ESTIMATED)` in the HUD and draws its coverage + wireframe dashed; colours, widths and active/idle emphasis are unchanged. A manual calibration + (`calSource: 'manual'`) or curated pose (`poseSource: 'curated'`) is never presented as + estimated, matching the CAL badge. Public camera state carries `headingConfidence` and + `headingEstimated` (`src/layers/cctv/headingConfidence.js`, #639). - **FIRMS**: no ground clamping (zero 3D-tiles height sampling), ≤18 screen-decluttered ambient labels, click-to-inspect detail card, 2.5k/3k sprite budgets viewport-clipped by FRP. - **CCTV v2 foundation:** a pitched frustum wireframe (4 corner rays + far-cap rectangle) with a monitor plane at the frustum's @@ -3446,6 +3489,7 @@ silently demoting every later lookup for the session. - HTTP refusals such as 406 now rotate alongside the existing network, rate-limit, and runtime-error cases. A refusal from one mirror no longer prevents reaching healthy alternatives or persists under the seven-day road/month-long boundary cache TTLs. Concurrent identical queries share one mirror sequence; if it fails, both the initiating and joined callers can use the same last-good data. - A refusal every mirror agrees on is still reported with the first mirror's status and body, so a genuinely malformed query says what upstream said — but only after every mirror has had the chance to answer it. `fetchOverpassPayload` takes injectable endpoints and fetch so the rotation is tested without a live mirror (`src/overpassProxy.test.mjs`). - Every mirror is asked with a User-Agent that names the application, its version and the project address (`OVERPASS_USER_AGENT` in `server/providers/overpass/constants.js`), which is what the OSM API usage policy asks for. Mirrors may refuse a client they cannot identify; such a refusal is never treated as data and costs the fan-out that mirror, so the header is what keeps the list at full strength. +- When the fan-out has nothing left to offer, the traffic row says which upstream declined and how: `Overpass rate-limited`, `Overpass timed out`, or `Overpass refused the road query (HTTP 406)`; the proxy's own 502 (every mirror unreachable) and 503 (local limiter busy) read `Overpass mirrors unreachable` and `Overpass temporarily unavailable`. Street Traffic draws on two upstreams — OpenStreetMap for geometry, TomTom for flow — so a row that only said "road data unavailable" sent readers to check a TomTom key that was never the problem (#661). `roadRequestError` in `src/layers/traffic/source.js` owns the mapping and tags the error with its status. A refusal whose code is unreadable — the layer takes its source by injection, so an adapter may not supply one — reports `Overpass temporarily unavailable` rather than printing `HTTP undefined`, and anything the layer did not classify at all (a dropped connection, a malformed snapshot) still reads `Road data temporarily unavailable`, because a platform exception is not a sentence to put on a panel. ### Share-link v2 layer state (August 2026) diff --git a/scripts/format-scope.json b/scripts/format-scope.json index f83966e72..ef44df03f 100644 --- a/scripts/format-scope.json +++ b/scripts/format-scope.json @@ -36,6 +36,7 @@ "src/app/mapLifecycle.test.mjs", "src/app/startupCamera.test.mjs", "src/app/viewer.js", + "src/app/atmosphereCompat.test.mjs", "src/standalone/application.js", "src/standalone/controls.js", "src/standalone/data.js", @@ -637,6 +638,7 @@ "src/data/surfaceServices.js", "src/annotations/resolver.js", "src/services/terrainHeights.js", + "src/services/terrainHeights.test.mjs", "src/services/groundFloor.js", "src/services/meshFloorSampler.js", "src/services/groundSnap.js", diff --git a/scripts/package-boundaries.json b/scripts/package-boundaries.json index 1971d3dda..f6e63cc0f 100644 --- a/scripts/package-boundaries.json +++ b/scripts/package-boundaries.json @@ -20,7 +20,7 @@ }, "viewer": { "exports": ["./application/viewer"], - "modules": ["src/app/viewer.js"], + "modules": ["src/app/viewer.js", "src/app/atmosphereCompat.js"], "external": ["cesium"] }, "vite-build": { @@ -1076,6 +1076,7 @@ "src/layers/cctv/geometry.js", "src/layers/cctv/geometryQueue.js", "src/layers/cctv/ground.js", + "src/layers/cctv/headingConfidence.js", "src/layers/cctv/health.js", "src/layers/cctv/hover.js", "src/layers/cctv/index.js", @@ -1318,6 +1319,7 @@ "src/app/surfaceServices.js", "src/app/tools.js", "src/app/viewer.js", + "src/app/atmosphereCompat.js", "src/bloom.js", "src/camera.js", "src/cameraVerbs.js", @@ -1493,6 +1495,7 @@ "src/layers/cctv/geometry.js", "src/layers/cctv/geometryQueue.js", "src/layers/cctv/ground.js", + "src/layers/cctv/headingConfidence.js", "src/layers/cctv/health.js", "src/layers/cctv/hover.js", "src/layers/cctv/index.js", @@ -2133,6 +2136,7 @@ "src/layers/cctv/geometry.js", "src/layers/cctv/geometryQueue.js", "src/layers/cctv/ground.js", + "src/layers/cctv/headingConfidence.js", "src/layers/cctv/health.js", "src/layers/cctv/hover.js", "src/layers/cctv/index.js", diff --git a/server/providers/aircraft/tracks.js b/server/providers/aircraft/tracks.js index 7d9501628..2c7b78a78 100644 --- a/server/providers/aircraft/tracks.js +++ b/server/providers/aircraft/tracks.js @@ -45,7 +45,9 @@ export function trackBackfillProxies() { RESPONSE_CAP_BYTES, ); let body; + let status = upstream.status; if (tooLarge) { + status = 502; body = JSON.stringify({ error: 'Upstream track response too large' }); } else if (!upstream.ok) { // Sanitize upstream error surface; status code is signal enough @@ -53,8 +55,10 @@ export function trackBackfillProxies() { } else { body = text; } - cachePut(key, { at: Date.now(), status: upstream.status, body }); - res.statusCode = upstream.status; + // Cache the 502 like any upstream error: /tracks/all costs OpenSky + // credits, so a retry inside the window must not re-download the body. + cachePut(key, { at: Date.now(), status, body }); + res.statusCode = status; res.setHeader('Content-Type', 'application/json; charset=utf-8'); res.setHeader('Cache-Control', 'no-store'); res.end(body); diff --git a/server/providers/cctv/constants.js b/server/providers/cctv/constants.js index 13578d0b8..da0f2fcda 100644 --- a/server/providers/cctv/constants.js +++ b/server/providers/cctv/constants.js @@ -261,5 +261,10 @@ export const CCTV_FRAME_MAX_BODY_BYTES = 16 * 1024 * 1024; /** Deadline for upstream response headers; live bodies keep streaming afterward. */ export const CCTV_MEDIA_FETCH_TIMEOUT_MS = 15 * 1000; +/** Silence a live body may carry before the relay gives up on it. Twice the + * header deadline, because a camera that is merely slow between frames is far + * more common than one that has died mid-stream, and a viewer would rather + * wait than be dropped. */ +export const CCTV_MEDIA_IDLE_TIMEOUT_MS = 30 * 1000; /** Declared size ceiling for fixed media responses. */ export const CCTV_MEDIA_MAX_BODY_BYTES = 64 * 1024 * 1024; diff --git a/server/providers/cctv/media.js b/server/providers/cctv/media.js index d652f7150..549e6cf06 100644 --- a/server/providers/cctv/media.js +++ b/server/providers/cctv/media.js @@ -4,6 +4,7 @@ import { CCTV_FRAME_FETCH_TIMEOUT_MS, CCTV_FRAME_MAX_BODY_BYTES, CCTV_MEDIA_FETCH_TIMEOUT_MS, + CCTV_MEDIA_IDLE_TIMEOUT_MS, CCTV_MEDIA_MAX_BODY_BYTES, NSW_IMAGE_ORIGIN, NSW_IMAGE_USER_AGENT, @@ -103,11 +104,17 @@ export function toReadable(body) { * @param {Response} upstream - fetch() Response object. * @param {object} [opts] * @param {string} [opts.sourceHeader='upstream'] - Value for X-CCTV-Source header. + * @param {number} [opts.idleTimeoutMs=CCTV_MEDIA_IDLE_TIMEOUT_MS] - Silence + * allowed between upstream chunks before the stream is released. Injectable + * only to keep the deadline unit-testable. */ export async function proxyMediaResponse( res, upstream, - { sourceHeader = 'upstream' } = {}, + { + sourceHeader = 'upstream', + idleTimeoutMs = CCTV_MEDIA_IDLE_TIMEOUT_MS, + } = {}, ) { const contentType = upstream.headers.get('content-type') || 'application/octet-stream'; @@ -153,7 +160,20 @@ export async function proxyMediaResponse( return; } + // The header deadline only covers the wait for a response line. Past that an + // upstream can hold the connection open and send nothing at all, and the + // relay would wait on it for as long as the camera host cared to. The + // deadline below measures the gap between upstream chunks rather than the + // life of the stream, so a feed that keeps delivering keeps its connection. + let idleTimer = null; + const clearIdleDeadline = () => { + if (!idleTimer) return; + clearTimeout(idleTimer); + idleTimer = null; + }; + stream.on('error', () => { + clearIdleDeadline(); if (!res.writableEnded) res.end(); }); @@ -164,6 +184,7 @@ export async function proxyMediaResponse( const releaseUpstream = () => { if (released) return; released = true; + clearIdleDeadline(); stream.unpipe(res); // Destroying the Node stream cancels the web body it wraps; the direct // cancel covers a body that was never wrapped, and rejects harmlessly when @@ -176,14 +197,37 @@ export async function proxyMediaResponse( /* already closed */ } }; + const armIdleDeadline = () => { + clearIdleDeadline(); + idleTimer = setTimeout(onIdleDeadline, idleTimeoutMs); + idleTimer.unref?.(); + }; + const onIdleDeadline = () => { + // A viewer who cannot keep up pauses the pipe, and no upstream bytes arrive + // while it is paused. That is a slow client rather than a dead camera, so + // it gets the deadline again instead of a teardown. + if (res.writableNeedDrain) { + armIdleDeadline(); + return; + } + releaseUpstream(); + if (!res.writableEnded) res.end(); + }; res.once('close', () => { + clearIdleDeadline(); if (!res.writableEnded) releaseUpstream(); }); res.once('error', releaseUpstream); stream.once('end', () => { + clearIdleDeadline(); released = true; }); + armIdleDeadline(); stream.pipe(res); + // Attached after the pipe because a data listener resumes the stream, and + // flowing before the destination is attached would spill chunks nobody + // forwards. Only bytes from upstream renew the deadline. + stream.on('data', armIdleDeadline); } /** diff --git a/server/standalone/key-setup-hardening.mjs b/server/standalone/key-setup-hardening.mjs index 002ccabdc..78440948c 100644 --- a/server/standalone/key-setup-hardening.mjs +++ b/server/standalone/key-setup-hardening.mjs @@ -123,7 +123,11 @@ export function hardenCredentialFile( if (platform !== 'win32') { try { if (platform === 'darwin') { - const aclRemoval = spawn('chmod', ['-N', filepath], { + // `-N` (strip the ACL) is Apple-only. Spawn /bin/chmod by absolute + // path, never by name: a Nix/Homebrew coreutils profile puts GNU chmod + // first on PATH, and GNU chmod rejects `-N` — which was treated as a + // hardening failure and refused every save on such machines. + const aclRemoval = spawn('/bin/chmod', ['-N', filepath], { stdio: 'ignore', }); if (!commandCompletedSuccessfully(aclRemoval)) return false; diff --git a/src/app/atmosphereCompat.js b/src/app/atmosphereCompat.js new file mode 100644 index 000000000..1bd23c8ea --- /dev/null +++ b/src/app/atmosphereCompat.js @@ -0,0 +1,189 @@ +/** + * Workaround for Apple's Metal/ANGLE shader translator rejecting Cesium's + * per-vertex model atmosphere. + * + * Cesium's `AtmosphereStageVS` calls a function whose `out` parameters are + * bound DIRECTLY to varyings: + * + * czm_computeGroundAtmosphereScattering( + * v_positionWC, lightDirection, + * v_atmosphereRayleighColor, v_atmosphereMieColor, v_atmosphereOpacity); + * + * ANGLE's Metal backend wraps each `out` argument in an `ANGLE_out(...)` + * helper that takes a `thread T &`. Vertex-output struct members live in + * `__metal_generic`, which cannot bind to `thread`, so the program fails to + * LINK (not compile) and Cesium tears the render loop down: + * + * RuntimeError: Program failed to link. ... MSL compilation error: + * reference to type 'thread float3' could not bind to an lvalue of type + * '__metal_generic float3' + * + * This is the only shader in Cesium that binds varyings to `out` params -- + * SkyAtmosphere and the ground-atmosphere fragment path both route through + * locals first, so they are unaffected and stay on. + * + * Cesium attaches that stage in `ModelSceneGraph.configurePipeline` under a + * single condition: + * + * const fogRenderable = frameState.fog.enabled && frameState.fog.renderable; + * if (fogRenderable) modelPipelineStages.push(AtmospherePipelineStage); + * + * so clearing `scene.fog.renderable` keeps the broken stage out of the + * pipeline entirely -- the program is never built, so it cannot fail to link. + * + * We clear `renderable` rather than `enabled` deliberately: + * - `Fog.update` still computes fog density when only `renderable` is off, + * preserving the distance-based screen-space-error scaling that governs + * 3D Tiles refinement. Clearing `enabled` returns early and forfeits it. + * - `visualSettings.js` saves and restores `scene.fog.enabled` around IR + * boost. Parking our fix on `enabled` would let that restore resurrect + * the broken shader mid-session; `renderable` is orthogonal to it. + * + * @module app/atmosphereCompat + */ + +/** Minimal vertex shader reproducing the exact out-param-to-varying bind. */ +const PROBE_VERTEX_SHADER = `#version 300 es +precision highp float; +in vec3 a_position; +out vec3 v_probe; +void writeOut(out vec3 value) { value = vec3(1.0); } +void main() { + writeOut(v_probe); + gl_Position = vec4(a_position, 1.0); +}`; + +const PROBE_FRAGMENT_SHADER = `#version 300 es +precision highp float; +in vec3 v_probe; +out vec4 fragColor; +void main() { fragColor = vec4(v_probe, 1.0); }`; + +/** + * True when the user agent is iOS or iPadOS. + * + * iPadOS 13+ defaults to desktop-mode browsing and reports `MacIntel` with no + * iPad token, so it is identified by touch points instead -- real Macs report + * `maxTouchPoints` of 0, including Apple Silicon ones that also claim + * `MacIntel`. Brave/Chrome/Firefox on iOS are all WebKit underneath and share + * the same Metal backend, so matching the platform rather than the browser is + * what we want. + */ +export function isAppleMobilePlatform(navigatorLike = globalThis.navigator) { + if (!navigatorLike) return false; + const ua = String(navigatorLike.userAgent || ''); + if (/\b(iPad|iPhone|iPod)\b/.test(ua)) return true; + // Desktop-mode iPadOS presents as a Mac. What separates it from a real Mac + // is touch: Macs report 0 touch points, including Apple Silicon ones that + // also claim `MacIntel`. `navigator.platform` is deprecated and may be + // empty, so the UA's Macintosh token is accepted as an equal signal rather + // than relying on platform alone. + const platform = String(navigatorLike.platform || ''); + const macLike = + platform === 'MacIntel' || + platform === 'MacARM' || + /\bMacintosh\b/.test(ua); + const touchPoints = Number(navigatorLike.maxTouchPoints) || 0; + return macLike && touchPoints > 1; +} + +/** + * Link the probe program and report whether the driver rejects the pattern. + * + * Returns `false` when WebGL2 is unavailable so that a missing context is + * never mistaken for a broken driver -- Cesium has its own handling for that, + * and the caller still falls back to platform detection. + */ +export function probeOutParamVaryingLinkFailure(gl) { + if (!gl) return false; + const vs = gl.createShader(gl.VERTEX_SHADER); + const fs = gl.createShader(gl.FRAGMENT_SHADER); + const program = gl.createProgram(); + if (!vs || !fs || !program) return false; + try { + gl.shaderSource(vs, PROBE_VERTEX_SHADER); + gl.compileShader(vs); + gl.shaderSource(fs, PROBE_FRAGMENT_SHADER); + gl.compileShader(fs); + // A compile failure here is not the bug we are probing for: the driver + // rejects this pattern at LINK time, after both stages compile cleanly. + if ( + !gl.getShaderParameter(vs, gl.COMPILE_STATUS) || + !gl.getShaderParameter(fs, gl.COMPILE_STATUS) + ) + return false; + gl.attachShader(program, vs); + gl.attachShader(program, fs); + gl.linkProgram(program); + return !gl.getProgramParameter(program, gl.LINK_STATUS); + } catch { + return false; + } finally { + gl.deleteShader(vs); + gl.deleteShader(fs); + gl.deleteProgram(program); + } +} + +/** Build a throwaway WebGL2 context for the probe, or null if unavailable. */ +function defaultProbeContext() { + try { + const canvas = + typeof OffscreenCanvas === 'function' + ? new OffscreenCanvas(1, 1) + : globalThis.document?.createElement('canvas'); + return canvas?.getContext('webgl2') ?? null; + } catch { + return null; + } +} + +/** + * Decide whether the per-vertex model atmosphere must be kept out of the + * pipeline on this device. + * + * The probe is authoritative when it fires, so a future driver fix silently + * restores the effect without a code change. Platform detection is the + * backstop for the case where no context can be created -- on Apple mobile a + * link failure is fatal and unrecoverable, so we would rather lose fog than + * gamble the whole render loop on the probe having run. + */ +export function shouldDisableModelAtmosphere({ + navigatorLike = globalThis.navigator, + createProbeContext = defaultProbeContext, +} = {}) { + const gl = createProbeContext(); + if (gl) { + let linkFails; + try { + linkFails = probeOutParamVaryingLinkFailure(gl); + } finally { + // Browsers cap live WebGL contexts (Safari especially); release the + // throwaway probe context now rather than waiting for GC. + try { + gl.getExtension?.('WEBGL_lose_context')?.loseContext(); + } catch { + // A context that cannot be released is left to GC. + } + } + if (linkFails) return true; + // Probe ran and the driver accepted the pattern. Apple mobile is still + // quarantined: the probe program is far simpler than Cesium's, and a + // false negative there costs the entire scene. + return isAppleMobilePlatform(navigatorLike); + } + return isAppleMobilePlatform(navigatorLike); +} + +/** + * Keep Cesium's broken model-atmosphere stage out of the pipeline. + * + * Returns true when the workaround was applied. Safe to call on any scene; + * scenes without a fog object are left untouched. + */ +export function applyModelAtmosphereWorkaround(scene, options = {}) { + if (!scene?.fog) return false; + if (!shouldDisableModelAtmosphere(options)) return false; + scene.fog.renderable = false; + return true; +} diff --git a/src/app/atmosphereCompat.test.mjs b/src/app/atmosphereCompat.test.mjs new file mode 100644 index 000000000..f1c1d43bf --- /dev/null +++ b/src/app/atmosphereCompat.test.mjs @@ -0,0 +1,221 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { + isAppleMobilePlatform, + probeOutParamVaryingLinkFailure, + shouldDisableModelAtmosphere, + applyModelAtmosphereWorkaround, +} from './atmosphereCompat.js'; + +const IPAD_OS12 = { + userAgent: + 'Mozilla/5.0 (iPad; CPU OS 12_5 like Mac OS X) AppleWebKit/605.1.15', + platform: 'iPad', + maxTouchPoints: 5, +}; +// iPadOS 13+ desktop mode: no iPad token, reports as a Mac. +const IPAD_DESKTOP_MODE = { + userAgent: + 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15', + platform: 'MacIntel', + maxTouchPoints: 5, +}; +const MAC_DESKTOP = { + userAgent: + 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15', + platform: 'MacIntel', + maxTouchPoints: 0, +}; +const LINUX_DESKTOP = { + userAgent: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/141', + platform: 'Linux x86_64', + maxTouchPoints: 0, +}; + +/** Minimal GL double. `linkOk: false` mimics the Metal translator's refusal. */ +function fakeGl({ linkOk = true, compileOk = true, throwOn = null } = {}) { + const deleted = { shaders: 0, programs: 0 }; + return { + deleted, + VERTEX_SHADER: 1, + FRAGMENT_SHADER: 2, + COMPILE_STATUS: 3, + LINK_STATUS: 4, + createShader: () => ({}), + createProgram: () => ({}), + shaderSource() { + if (throwOn === 'shaderSource') throw new Error('context lost'); + }, + compileShader() {}, + attachShader() {}, + linkProgram() {}, + getShaderParameter: () => compileOk, + getProgramParameter: () => linkOk, + deleteShader: () => (deleted.shaders += 1), + deleteProgram: () => (deleted.programs += 1), + }; +} + +test('identifies iPadOS in both legacy and desktop-mode guises', () => { + assert.equal(isAppleMobilePlatform(IPAD_OS12), true); + assert.equal(isAppleMobilePlatform(IPAD_DESKTOP_MODE), true); +}); + +test('does not mistake a real Mac or a Linux desktop for Apple mobile', () => { + // Regression guard: the desktop-mode check keys on touch points, and a Mac + // reports the same MacIntel platform string with zero of them. + assert.equal(isAppleMobilePlatform(MAC_DESKTOP), false); + assert.equal(isAppleMobilePlatform(LINUX_DESKTOP), false); +}); + +test('treats a missing navigator as not-Apple rather than throwing', () => { + assert.equal(isAppleMobilePlatform(undefined), false); + assert.equal(isAppleMobilePlatform(null), false); +}); + +test('detects desktop-mode iPadOS without navigator.platform', () => { + // `navigator.platform` is deprecated and can be absent or stripped; the UA's + // Macintosh token plus touch points has to carry the detection on its own. + assert.equal( + isAppleMobilePlatform({ + userAgent: + 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15', + maxTouchPoints: 5, + }), + true, + ); + // Same UA, no touch: a real Mac, which must stay untouched. + assert.equal( + isAppleMobilePlatform({ + userAgent: + 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15', + maxTouchPoints: 0, + }), + false, + ); +}); + +test('probe reports failure only when the program fails to link', () => { + assert.equal( + probeOutParamVaryingLinkFailure(fakeGl({ linkOk: false })), + true, + ); + assert.equal( + probeOutParamVaryingLinkFailure(fakeGl({ linkOk: true })), + false, + ); +}); + +test('a compile failure is not reported as the link bug', () => { + // The Metal defect surfaces at link time; both stages compile cleanly. + // Treating a compile error as the bug would disable fog for the wrong reason. + assert.equal( + probeOutParamVaryingLinkFailure( + fakeGl({ compileOk: false, linkOk: false }), + ), + false, + ); +}); + +test('probe releases its GL objects on both paths', () => { + const ok = fakeGl({ linkOk: true }); + probeOutParamVaryingLinkFailure(ok); + assert.equal(ok.deleted.shaders, 2); + assert.equal(ok.deleted.programs, 1); + + const thrown = fakeGl({ throwOn: 'shaderSource' }); + probeOutParamVaryingLinkFailure(thrown); + assert.equal(thrown.deleted.shaders, 2, 'shaders freed even when GL throws'); + assert.equal(thrown.deleted.programs, 1); +}); + +test('probe treats a null context as healthy, not broken', () => { + assert.equal(probeOutParamVaryingLinkFailure(null), false); +}); + +test('a failing probe disables the stage on any platform', () => { + assert.equal( + shouldDisableModelAtmosphere({ + navigatorLike: LINUX_DESKTOP, + createProbeContext: () => fakeGl({ linkOk: false }), + }), + true, + ); +}); + +test('a passing probe still quarantines Apple mobile but frees desktops', () => { + const createProbeContext = () => fakeGl({ linkOk: true }); + assert.equal( + shouldDisableModelAtmosphere({ + navigatorLike: IPAD_DESKTOP_MODE, + createProbeContext, + }), + true, + ); + assert.equal( + shouldDisableModelAtmosphere({ + navigatorLike: LINUX_DESKTOP, + createProbeContext, + }), + false, + ); +}); + +test('falls back to platform detection when no context can be made', () => { + const createProbeContext = () => null; + assert.equal( + shouldDisableModelAtmosphere({ + navigatorLike: IPAD_OS12, + createProbeContext, + }), + true, + ); + assert.equal( + shouldDisableModelAtmosphere({ + navigatorLike: MAC_DESKTOP, + createProbeContext, + }), + false, + ); +}); + +test('applies to the scene by clearing renderable, never enabled', () => { + // `enabled` must survive: Fog.update short-circuits when it is false, which + // forfeits the density used for 3D Tiles screen-space-error scaling, and + // visualSettings' IR boost saves/restores that same flag. + const scene = { fog: { enabled: true, renderable: true } }; + const applied = applyModelAtmosphereWorkaround(scene, { + navigatorLike: IPAD_DESKTOP_MODE, + createProbeContext: () => null, + }); + assert.equal(applied, true); + assert.equal(scene.fog.renderable, false); + assert.equal(scene.fog.enabled, true); +}); + +test('leaves unaffected devices completely untouched', () => { + const scene = { fog: { enabled: true, renderable: true } }; + const applied = applyModelAtmosphereWorkaround(scene, { + navigatorLike: LINUX_DESKTOP, + createProbeContext: () => fakeGl({ linkOk: true }), + }); + assert.equal(applied, false); + assert.equal(scene.fog.renderable, true); +}); + +test('tolerates a scene with no fog object', () => { + assert.equal(applyModelAtmosphereWorkaround({}, {}), false); + assert.equal(applyModelAtmosphereWorkaround(null, {}), false); +}); + +test('releases the throwaway probe context after probing', () => { + let lost = 0; + const gl = fakeGl({ linkOk: true }); + gl.getExtension = (name) => + name === 'WEBGL_lose_context' ? { loseContext: () => (lost += 1) } : null; + shouldDisableModelAtmosphere({ + navigatorLike: LINUX_DESKTOP, + createProbeContext: () => gl, + }); + assert.equal(lost, 1, 'probe context must not linger until GC'); +}); diff --git a/src/app/viewer.js b/src/app/viewer.js index ec58f52dc..9b6e7f631 100644 --- a/src/app/viewer.js +++ b/src/app/viewer.js @@ -1,4 +1,5 @@ import * as Cesium from 'cesium'; +import { applyModelAtmosphereWorkaround } from './atmosphereCompat.js'; const PINCH_ZOOM_MULTIPLIER = 8; const MAX_PINCH_PIXEL_DELTA = 120; @@ -124,6 +125,10 @@ export function createApplicationViewer({ container, creditContainer }) { }); try { viewer.targetFrameRate = 60; + // Before any tile builds a draw command: Cesium's per-vertex model + // atmosphere fails to LINK on Apple's Metal backend and kills the + // render loop. See app/atmosphereCompat.js. + applyModelAtmosphereWorkaround(viewer.scene); viewer.scene.globe.show = false; viewer.scene.skyAtmosphere.show = true; viewer.scene.skyAtmosphere.atmosphereLightIntensity = 18; diff --git a/src/data/cctvMediaRange.test.mjs b/src/data/cctvMediaRange.test.mjs index 3e71d3e17..9f279bdc7 100644 --- a/src/data/cctvMediaRange.test.mjs +++ b/src/data/cctvMediaRange.test.mjs @@ -8,8 +8,9 @@ import { test } from 'node:test'; import assert from 'node:assert/strict'; import http from 'node:http'; import net from 'node:net'; -import { Readable } from 'node:stream'; +import { Readable, Writable } from 'node:stream'; import { cctvProxy } from '../../server/providers/cctv.js'; +import { proxyMediaResponse } from '../../server/providers/cctv/media.js'; import { sanitizeCctvRangeHeader } from '../../server/providers/cctv/range.js'; import { CCTV_MEDIA_MAX_BODY_BYTES as CAP } from '../../server/providers/cctv/constants.js'; @@ -586,3 +587,117 @@ test('two sequential seeks are served as two partial responses', async (t) => { assert.equal(first.body, 'chunk-0'); assert.equal(second.body, 'chunk-100'); }); + +// --------------------------------------------------------------------------- +// The body idle deadline. proxyMediaResponse is driven directly here because +// the deadline is only injectable through its options; the route always passes +// the production value. +// --------------------------------------------------------------------------- + +/** + * A response to pipe into, built on a real Writable so the backpressure the + * proxy reads is the stream's own rather than a flag the test sets. With + * `drains: false` the bytes are accepted and never flushed, which is what a + * viewer on a slow link looks like from the server side. + */ +function pipeTarget({ drains = true } = {}) { + const res = new Writable({ + // One byte is enough to leave a target that never empties its buffer + // permanently in need of a drain. + highWaterMark: drains ? undefined : 1, + write(_chunk, _encoding, callback) { + if (drains) callback(); + }, + }); + res.writeHead = (status, headers) => { + res.statusCode = status; + res.headers = headers || {}; + }; + return res; +} + +/** + * An upstream that answers with headers, sends one chunk, and then either + * keeps producing every `everyMs` or goes silent. `released` reports the body's + * own cleanup: a cancelled fetch body lands in the web stream's cancel hook. + */ +function mediaUpstream({ everyMs = 0 } = {}) { + const state = { produced: 0, released: false }; + let ticker = null; + const body = new ReadableStream({ + start(controller) { + const send = () => { + state.produced += 1; + controller.enqueue(new Uint8Array([0xff, 0xd8, 0xff, 0xe0])); + }; + send(); + if (everyMs > 0) { + ticker = setInterval(send, everyMs); + ticker.unref(); + } + }, + cancel() { + state.released = true; + if (ticker) clearInterval(ticker); + }, + }); + state.upstream = { + ok: true, + status: 200, + headers: { + get: (name) => + String(name).toLowerCase() === 'content-type' ? 'video/mp4' : null, + }, + body, + arrayBuffer: async () => Buffer.alloc(0), + }; + return state; +} + +test('an upstream that goes silent after its headers is released at the idle deadline', async (t) => { + const feed = mediaUpstream(); + const res = pipeTarget(); + t.after(() => res.destroy()); + await proxyMediaResponse(res, feed.upstream, { idleTimeoutMs: 40 }); + + await new Promise((resolve) => setTimeout(resolve, 20)); + assert.equal(feed.released, false, 'released before the deadline was due'); + await new Promise((resolve) => setTimeout(resolve, 120)); + // Without the deadline this stream is held open against the camera host for + // as long as that host will keep the socket, with no bytes ever arriving. + assert.equal(feed.released, true, 'the silent upstream was never released'); + assert.equal(res.writableEnded, true, 'the client was left waiting'); +}); + +test('a live feed that keeps producing is not cut off by the idle deadline', async (t) => { + const feed = mediaUpstream({ everyMs: 10 }); + const res = pipeTarget(); + t.after(() => res.destroy()); + await proxyMediaResponse(res, feed.upstream, { idleTimeoutMs: 40 }); + + await new Promise((resolve) => setTimeout(resolve, 250)); + assert.equal(feed.released, false, 'a healthy feed was torn down'); + assert.equal(res.writableEnded, false, 'a healthy feed was ended early'); + assert.ok( + feed.produced > 5, + `the feed stopped producing (${feed.produced} chunks)`, + ); +}); + +test('a client that cannot keep up is not mistaken for a stalled upstream', async (t) => { + // Nothing arrives from upstream while the pipe is paused, which looks exactly + // like silence unless the response is asked whether it is still draining. + const feed = mediaUpstream({ everyMs: 10 }); + const res = pipeTarget({ drains: false }); + t.after(() => res.destroy()); + await proxyMediaResponse(res, feed.upstream, { idleTimeoutMs: 30 }); + + await new Promise((resolve) => setTimeout(resolve, 200)); + assert.equal( + res.writableNeedDrain, + true, + 'the client was not the bottleneck', + ); + assert.equal(feed.released, false, 'a slow viewer tore down a healthy feed'); + assert.equal(res.writableEnded, false, 'a slow viewer ended the response'); +}); diff --git a/src/data/layerState.test.mjs b/src/data/layerState.test.mjs index af2cd6526..1b504a439 100644 --- a/src/data/layerState.test.mjs +++ b/src/data/layerState.test.mjs @@ -456,6 +456,25 @@ test('v2 codec distinguishes absent from empty and keeps canonical deterministic ); }); +test('all production layers and options round-trip through a v2 share URL', () => { + const allEnabled = normalizeLayerState({ + enabledLayerIds: REGISTERED_LAYER_IDS, + options: { + cctv: { coverageMode: 'viewshed', showProjection: false, autoHop: true }, + flights: { models3d: true, models3dMode: 'all' }, + radio: { filter: 'news', volume: 0.37 }, + }, + }); + const shareUrl = new URL('https://example.invalid/'); + shareUrl.hash = encode(allEnabled); + const params = new URLSearchParams(shareUrl.hash.slice(1)); + const restored = decodeLayerStateParams(params); + + assert.equal(params.get('l')?.split('.').length, REGISTERED_LAYER_IDS.length); + assert.deepEqual(restored?.enabledLayerIds, REGISTERED_LAYER_IDS); + assert.deepEqual(restored?.options, allEnabled.options); +}); + test('unknown enabled-layer tokens reject the payload instead of becoming an empty set', () => { assert.equal( decodeLayerStateParams(new URLSearchParams('v=2&l=unknown')), diff --git a/src/data/layerStateTokenLedger.test.mjs b/src/data/layerStateTokenLedger.test.mjs index 2cbefbace..7b0a9263c 100644 --- a/src/data/layerStateTokenLedger.test.mjs +++ b/src/data/layerStateTokenLedger.test.mjs @@ -4,12 +4,13 @@ import { mkdtempSync, mkdirSync, readFileSync, + realpathSync, rmSync, writeFileSync, } from 'node:fs'; import { tmpdir } from 'node:os'; import path from 'node:path'; -import { fileURLToPath } from 'node:url'; +import { fileURLToPath, pathToFileURL } from 'node:url'; import test from 'node:test'; import { LEGACY_LAYER_STATE_TOKENS, @@ -27,9 +28,15 @@ import { const LEDGER_PATH = 'src/data/layerStateTokenReservations.json'; const SOURCE_PATH = 'src/data/layerState.js'; +const CODEC_SOURCE_PATH = fileURLToPath( + new URL('./layerState.js', import.meta.url), +); const CHECKER_PATH = fileURLToPath( new URL('../../scripts/check-layer-state-tokens.mjs', import.meta.url), ); +const NEXT_PATH = fileURLToPath( + new URL('../../scripts/next-layer-state-token.mjs', import.meta.url), +); const reservationRows = JSON.parse( readFileSync( new URL('./layerStateTokenReservations.json', import.meta.url), @@ -90,6 +97,26 @@ function withPublishedBase( } } +function writeCodecFixture(cwd, rows, entries) { + const source = readFileSync(CODEC_SOURCE_PATH, 'utf8'); + const registryEnd = ']);\n\nexport const REGISTERED_LAYER_IDS'; + assert.equal(source.split(registryEnd).length, 2); + const additions = entries + .map( + ({ id, token }) => + ` Object.freeze({ id: '${id}', token: '${token}', disposition: 'enabled-only' }),`, + ) + .join('\n'); + mkdirSync(path.join(cwd, 'src/data'), { recursive: true }); + writeFileSync(path.join(cwd, 'package.json'), '{"type":"module"}'); + writeFileSync( + path.join(cwd, SOURCE_PATH), + source.replace(registryEnd, `${additions}\n${registryEnd}`), + ); + writeFileSync(path.join(cwd, LEDGER_PATH), JSON.stringify(rows)); + return path.join(cwd, SOURCE_PATH); +} + test('reservation ledger is complete, pinned, and rejects duplicate or malformed rows', () => { assert.deepEqual( { ...parseLayerStateTokenReservations(reservationRows) }, @@ -169,6 +196,77 @@ test('published retired digits and competing PRs advance the merge-time allocati ); }); +test('PR B manually replaces provisional 0 with 3 after PR A publishes 0', () => { + withPublishedBase({ rows: [...reservationRows, ['pr-a', '0']] }, (cwd) => { + mkdirSync(path.join(cwd, 'scripts')); + const fixtureChecker = path.join( + cwd, + 'scripts/check-layer-state-tokens.mjs', + ); + writeFileSync(fixtureChecker, readFileSync(CHECKER_PATH, 'utf8')); + const fixtureNext = path.join(cwd, 'scripts/next-layer-state-token.mjs'); + writeFileSync(fixtureNext, readFileSync(NEXT_PATH, 'utf8')); + const runCheck = () => + spawnSync( + process.execPath, + [realpathSync(fixtureChecker), '--base-ref', 'HEAD'], + { + cwd, + encoding: 'utf8', + }, + ); + + writeCodecFixture( + cwd, + [...reservationRows, ['pr-b', '0']], + [{ id: 'pr-b', token: '0' }], + ); + const beforeRebase = runCheck(); + assert.equal(beforeRebase.status, 1); + assert.match(beforeRebase.stderr, /changed or removed: pr-a/); + + writeCodecFixture( + cwd, + [...reservationRows, ['pr-a', '0'], ['pr-b', '0']], + [ + { id: 'pr-a', token: '0' }, + { id: 'pr-b', token: '0' }, + ], + ); + const staleRebase = runCheck(); + assert.equal(staleRebase.status, 1); + assert.match(staleRebase.stderr, /Duplicate layer-state token reservation/); + + const published = readPublishedLayerStateReservations('HEAD', cwd); + assert.equal(nextLayerStateToken(published), '3'); + const validBaselineRows = [...reservationRows, ['pr-a', '0']]; + writeCodecFixture(cwd, validBaselineRows, [{ id: 'pr-a', token: '0' }]); + const helper = spawnSync( + process.execPath, + [realpathSync(fixtureNext), 'pr-b'], + { cwd, encoding: 'utf8' }, + ); + assert.equal(helper.status, 0, helper.stderr); + assert.match(helper.stdout, /pr-b: 3/); + assert.deepEqual( + JSON.parse(readFileSync(path.join(cwd, LEDGER_PATH), 'utf8')), + validBaselineRows, + 'the helper reports a token without rewriting the ledger', + ); + writeCodecFixture( + cwd, + [...reservationRows, ['pr-a', '0'], ['pr-b', '3']], + [ + { id: 'pr-a', token: '0' }, + { id: 'pr-b', token: '3' }, + ], + ); + const corrected = runCheck(); + assert.equal(corrected.status, 0, corrected.stderr); + assert.match(corrected.stdout, /29 published, 1 new/); + }); +}); + test('allocation batches cross the last digit and base-36 pair boundaries in order', () => { const digitRows = [...'03456789'].map((digit) => [`retired-${digit}`, digit]); const allDigits = parseLayerStateTokenReservations([ @@ -219,6 +317,55 @@ test('allocation batches cross the last digit and base-36 pair boundaries in ord ); }); +test('an isolated valid two-character fixture round-trips an l field beyond the old 64-character cap', async () => { + const cwd = mkdtempSync(path.join(tmpdir(), 'gev-layer-token-width-')); + try { + const priorDigits = [...'03456789'].map((digit) => [ + `qa-prior-digit-${digit}`, + digit, + ]); + const fixtureLayers = ['00', '01', '02', '03'].map((token) => ({ + id: `qa-layer-${token}`, + token, + })); + const fixtureRows = [ + ...reservationRows, + ...priorDigits, + ...fixtureLayers.map(({ id, token }) => [id, token]), + ]; + const fixtureModule = writeCodecFixture(cwd, fixtureRows, fixtureLayers); + const codec = await import(pathToFileURL(fixtureModule)); + assert.equal(codec.validateLayerStateRegistry(), true); + assert.equal( + nextLayerStateToken( + parseLayerStateTokenReservations([...reservationRows, ...priorDigits]), + ), + '00', + ); + assert.equal( + validateLayerStateAllocations( + LAYER_STATE_TOKEN_RESERVATIONS, + codec.LAYER_STATE_TOKEN_RESERVATIONS, + ), + true, + ); + + const expectedLayerIds = codec.REGISTERED_LAYER_IDS; + const state = codec.normalizeLayerState({ + enabledLayerIds: expectedLayerIds, + }); + const params = new URLSearchParams([['v', '2']]); + codec.encodeLayerStateParams(params, state); + assert.ok(params.get('l').length > 64, 'fixture must cross the old cap'); + assert.equal(params.get('l').length, 67); + const restored = codec.decodeLayerStateParams(params); + assert.deepEqual(restored?.enabledLayerIds, expectedLayerIds); + assert.deepEqual(restored?.options, state.options); + } finally { + rmSync(cwd, { recursive: true, force: true }); + } +}); + test('checker reads a complete future base ledger and rejects retired-token reuse', () => { withPublishedBase( { diff --git a/src/keySetupHardening.test.mjs b/src/keySetupHardening.test.mjs index 2578be37d..c519576e3 100644 --- a/src/keySetupHardening.test.mjs +++ b/src/keySetupHardening.test.mjs @@ -55,6 +55,21 @@ test('macOS ACL removal failure stops before chmod and fails closed', () => { assert.deepEqual(fileSystem.calls, [], 'mode bits must not disguise an ACL-removal failure'); }); +test('macOS ACL removal spawns Apple /bin/chmod by absolute path, not via PATH', () => { + const spawned = []; + hardenCredentialFile(FILE, { + platform: 'darwin', + fileSystem: fileSystemWithMode(0o100600), + spawn(command, args) { + spawned.push([command, args]); + return { status: 0, signal: null }; + }, + }); + // A coreutils-first PATH (Nix home-manager, Homebrew) resolves bare `chmod` + // to GNU chmod, which has no `-N` and would fail every save. + assert.deepEqual(spawned, [['/bin/chmod', ['-N', FILE]]]); +}); + test('POSIX hardening verifies the resulting 0600 mode', () => { const goodFs = fileSystemWithMode(0o100600); assert.equal(hardenCredentialFile(FILE, { diff --git a/src/layers/cctv/headingConfidence.js b/src/layers/cctv/headingConfidence.js new file mode 100644 index 000000000..a42297e28 --- /dev/null +++ b/src/layers/cctv/headingConfidence.js @@ -0,0 +1,47 @@ +/** + * @module headingConfidence + * + * Consumers for the catalog's per-camera `headingConfidence` flag (#639). + * + * Packs whose feed publishes no facing get a synthetic bearing from + * `fallbackHeadingFromId` — a hash of the id string, with no geographic + * input — and are marked `headingConfidence: 'low'`. Until now nothing read + * the flag, so a guessed bearing rendered exactly like a surveyed one. These + * helpers make the guess visible: the HUD heading token gains an + * `(ESTIMATED)` tag and the coverage wireframe draws dashed instead of solid. + * + * A human-vouched pose always wins over the pack flag: a manually saved + * calibration (`calSource:'manual'`) or a hand-authored catalog entry + * (`poseSource:'curated'`) is never presented as estimated, mirroring the + * CAL badge states in calibration.js. + */ + +/** + * Whether a camera's bearing is a synthetic guess that should present as + * provisional. + * @param {{headingConfidence?: string|null, calSource?: string|null, + * poseSource?: string|null}} camera - Catalog camera (or public state). + * @returns {boolean} + */ +export function isHeadingEstimated(camera) { + if (!camera) return false; + if (camera.calSource === 'manual') return false; + if (camera.poseSource === 'curated') return false; + return ( + String(camera.headingConfidence || '') + .trim() + .toLowerCase() === 'low' + ); +} + +/** + * HUD heading token: `HDG 194°`, tagged `(ESTIMATED)` when the bearing is + * synthetic — so a hash can never read as a surveyed facing. + * @param {{headingDeg?: number, headingConfidence?: string|null, + * calSource?: string|null, poseSource?: string|null}} camera + * @returns {string} + */ +export function headingHudToken(camera) { + const hdg = Math.round(Number(camera?.headingDeg) || 0); + return `HDG ${hdg}°${isHeadingEstimated(camera) ? ' (ESTIMATED)' : ''}`; +} diff --git a/src/layers/cctv/headingConfidence.test.mjs b/src/layers/cctv/headingConfidence.test.mjs new file mode 100644 index 000000000..513bf0d1b --- /dev/null +++ b/src/layers/cctv/headingConfidence.test.mjs @@ -0,0 +1,60 @@ +// src/layers/cctv/headingConfidence.test.mjs +// Bearing-provenance consumers (#639): a pack's headingConfidence:'low' +// (a hashed synthetic bearing) must present as estimated — unless a human +// vouched for the pose via manual calibration or a curated catalog entry. +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { isHeadingEstimated, headingHudToken } from './headingConfidence.js'; + +test('low confidence marks the bearing estimated', () => { + assert.equal(isHeadingEstimated({ headingConfidence: 'low' }), true); +}); + +test('confidence matching is trimmed and case-insensitive', () => { + assert.equal(isHeadingEstimated({ headingConfidence: ' LOW ' }), true); + assert.equal(isHeadingEstimated({ headingConfidence: 'Low' }), true); +}); + +test('medium, high, missing, and absent cameras are not estimated', () => { + assert.equal(isHeadingEstimated({ headingConfidence: 'medium' }), false); + assert.equal(isHeadingEstimated({ headingConfidence: 'high' }), false); + assert.equal(isHeadingEstimated({}), false); + assert.equal(isHeadingEstimated(null), false); + assert.equal(isHeadingEstimated(undefined), false); +}); + +test('a manually saved calibration overrides a low pack flag', () => { + assert.equal( + isHeadingEstimated({ headingConfidence: 'low', calSource: 'manual' }), + false, + ); +}); + +test('a curated catalog entry overrides a low pack flag', () => { + assert.equal( + isHeadingEstimated({ headingConfidence: 'low', poseSource: 'curated' }), + false, + ); +}); + +test('HUD token tags a synthetic bearing and rounds the degrees', () => { + assert.equal( + headingHudToken({ headingDeg: 194.4, headingConfidence: 'low' }), + 'HDG 194° (ESTIMATED)', + ); +}); + +test('HUD token stays untagged for surveyed and calibrated bearings', () => { + assert.equal( + headingHudToken({ headingDeg: 67.5, headingConfidence: 'high' }), + 'HDG 68°', + ); + assert.equal( + headingHudToken({ + headingDeg: 225, + headingConfidence: 'low', + calSource: 'manual', + }), + 'HDG 225°', + ); +}); diff --git a/src/layers/cctv/presentation.js b/src/layers/cctv/presentation.js index 1ee54df50..f8f83fcd1 100644 --- a/src/layers/cctv/presentation.js +++ b/src/layers/cctv/presentation.js @@ -1,5 +1,6 @@ import { CCTV_AMBIENT_CARD_MAX } from '../../data/cctvLod.js'; import { ACTIVE_FRAME_REFRESH_MS, IDLE_FRAME_REFRESH_MS } from './policy.js'; +import { headingHudToken, isHeadingEstimated } from './headingConfidence.js'; export function createPresentation({ state: layerState, @@ -35,7 +36,9 @@ export function createPresentation({ return [ `${active.camera.city.toUpperCase()} CCTV`, `${active.camera.name.toUpperCase()}`, - `HDG ${Math.round(active.camera.headingDeg)}°`, + // A synthetic bearing (headingConfidence 'low', no human calibration) + // is tagged so a hashed guess never reads as a surveyed facing (#639). + headingHudToken(active.camera), `FOV ${Math.round(active.camera.fovDeg)}°`, `COVERAGE ${area.toFixed(2)}km²`, overlapCount > 0 ? `OVERLAP ${overlapCount} cams` : 'ISOLATED VIEW', @@ -76,6 +79,11 @@ export function createPresentation({ lat: camera.lat, lon: camera.lon, headingDeg: camera.headingDeg, + // Bearing provenance (#639): the pack's confidence flag plus the derived + // "is this a synthetic guess" bit (calibration-aware), so UI consumers + // never have to re-derive it. + headingConfidence: camera.headingConfidence || null, + headingEstimated: isHeadingEstimated(camera), pitchDeg: camera.pitchDeg, fovDeg: camera.fovDeg, rangeM: camera.rangeM, diff --git a/src/layers/cctv/rendering.js b/src/layers/cctv/rendering.js index faac69eb8..6601d938b 100644 --- a/src/layers/cctv/rendering.js +++ b/src/layers/cctv/rendering.js @@ -1,5 +1,6 @@ import * as Cesium from 'cesium'; import { horizonOccluder } from '../../data/iconOrientation.js'; +import { isHeadingEstimated } from './headingConfidence.js'; import { ACTIVE_CAMERA_COLOR, IDLE_CAMERA_COLOR, @@ -233,6 +234,15 @@ export function createRendering({ ); const inVisibleSet = coverageVisible.has(record.camera.id); + // A synthetic bearing (headingConfidence 'low', no human calibration) + // draws its wireframe dashed so a guessed facing is visibly provisional + // rather than rendering identically to a surveyed one (#639). Colors, + // widths, and the active/idle emphasis are unchanged. + const bearingEstimated = isHeadingEstimated(record.camera); + const lineMaterial = (color) => + bearingEstimated + ? new Cesium.PolylineDashMaterialProperty({ color }) + : color; for (const entity of record.coverageEntities || []) { // The frustum wireframe is part of the projection representation — // force it on for the active camera and let it read through geometry @@ -247,13 +257,15 @@ export function createRendering({ // active camera keeps its width/alpha emphasis in both schemes. const hue = viewshedOn ? record.viewshedColors : null; if (entity._coverageRole === 'cap') { - entity.polyline.material = hue - ? isActive - ? hue.lineActive - : hue.line - : isActive - ? ACTIVE_COVERAGE_CENTER - : IDLE_COVERAGE_CENTER_MUTED; + entity.polyline.material = lineMaterial( + hue + ? isActive + ? hue.lineActive + : hue.line + : isActive + ? ACTIVE_COVERAGE_CENTER + : IDLE_COVERAGE_CENTER_MUTED, + ); entity.polyline.width = isActive ? 2.2 : 1.0; entity.polyline.depthFailMaterial = planeShowing ? hue @@ -261,13 +273,15 @@ export function createRendering({ : ACTIVE_COVERAGE_CENTER_DEPTHFAIL : undefined; } else { - entity.polyline.material = hue - ? isActive - ? hue.lineActive - : hue.line.withAlpha(0.6) - : isActive - ? ACTIVE_COVERAGE_EDGE - : IDLE_COVERAGE_EDGE_MUTED; + entity.polyline.material = lineMaterial( + hue + ? isActive + ? hue.lineActive + : hue.line.withAlpha(0.6) + : isActive + ? ACTIVE_COVERAGE_EDGE + : IDLE_COVERAGE_EDGE_MUTED, + ); entity.polyline.width = isActive ? 1.8 : 0.9; entity.polyline.depthFailMaterial = planeShowing ? hue diff --git a/src/layers/traffic/ingestion.js b/src/layers/traffic/ingestion.js index de5b56d9f..904ceca04 100644 --- a/src/layers/traffic/ingestion.js +++ b/src/layers/traffic/ingestion.js @@ -4,6 +4,7 @@ import { TILE_CACHE_MAX_ENTRIES, FAST_FETCH_ALTITUDE, } from './policy.js'; +import { RoadRequestError, roadRequestError } from './source.js'; export function createIngestion({ state: layerState, @@ -67,7 +68,7 @@ export function createIngestion({ ); if (!response.ok) { - throw new Error(`Overpass API returned ${response.status}`); + throw roadRequestError(response.status); } if (!state) { @@ -307,8 +308,16 @@ export function createIngestion({ renderedSomething = true; } catch (e) { if (e?.name === 'AbortError') return; - if (generation === layerState._loadGeneration && !renderedSomething) - layerState._roadError = 'Road data temporarily unavailable'; + if (generation === layerState._loadGeneration && !renderedSomething) { + // A refusal we classified carries its own line; anything else — a + // dropped connection, a malformed snapshot — keeps the general one, + // because `e.message` from the platform is not a sentence anyone + // should have to read off a panel. + layerState._roadError = + e instanceof RoadRequestError + ? e.message + : 'Road data temporarily unavailable'; + } console.warn('[Data:Traffic] Fetch error:', e); } finally { if (generation === layerState._loadGeneration) { diff --git a/src/layers/traffic/navigation.test.mjs b/src/layers/traffic/navigation.test.mjs index e7bd4cc62..c07ae6ad9 100644 --- a/src/layers/traffic/navigation.test.mjs +++ b/src/layers/traffic/navigation.test.mjs @@ -216,3 +216,39 @@ test('parked failures back off and disabling cancels the scheduled retry', async assert.equal(calls, 2); assert.equal(layer.getStats().error, null); }); + +test('a declined road request reaches the row as the reason, not as a shrug', async (t) => { + // Issue #661: with every public Overpass mirror refusing a network, the row + // read "Road data temporarily unavailable" while the console already held + // "Overpass API returned 406". The reporter went looking for a bad TomTom + // key — the other half of this layer, and never the problem. The status the + // code already has must survive the trip to the panel. + let status = 406; + const { layer, viewer, tick } = setup(t, async () => ({ ok: false, status })); + layer.enable(viewer); + await tick(400); + assert.equal( + layer.getStats().error, + 'Overpass refused the road query (HTTP 406)', + ); + + status = 429; + for (let i = 0; i < 20; i++) await tick(1500); + assert.equal( + layer.getStats().error, + 'Overpass rate-limited', + 'a rate limit is a different instruction to the reader than a refusal', + ); +}); + +test('an unclassified road failure keeps the general line', async (t) => { + // The complement of the test above, and the reason this is not just + // `e.message`: a dropped socket throws whatever the platform felt like + // saying, and "Failed to fetch" on a panel row helps nobody. + const { layer, viewer, tick } = setup(t, async () => { + throw new Error('socket hang up'); + }); + layer.enable(viewer); + await tick(400); + assert.equal(layer.getStats().error, 'Road data temporarily unavailable'); +}); diff --git a/src/layers/traffic/source.js b/src/layers/traffic/source.js index d499a4b7b..b2c992b12 100644 --- a/src/layers/traffic/source.js +++ b/src/layers/traffic/source.js @@ -15,6 +15,68 @@ function buildOverpassQuery( return `[out:json][timeout:${timeoutSec}];(way["highway"~"${regex}"](${south},${west},${north},${east}););out geom qt;`; } +/** + * A road request upstream declined, carrying a line fit to show an operator. + * + * The status travels beside the message instead of only inside it, so a caller + * can branch on the code without parsing English — the same shape + * `LiveSourceError` uses in `src/sources/live/contract.js`. + */ +export class RoadRequestError extends Error { + constructor(message, { status = null } = {}) { + super(message); + this.name = 'RoadRequestError'; + this.status = status; + } +} + +/** + * Name an Overpass refusal in the words the traffic row will print. + * + * This layer has two upstreams — OpenStreetMap for road geometry and TomTom + * for flow — and only one of them can be down at a time. A row that says + * nothing more than "road data unavailable" sends the reader to check their + * TomTom key, which is the wrong half of the layer and costs them the + * afternoon. So the status upstream actually returned is reported: 406 from a + * public mirror is not a configuration problem the reader can fix, and saying + * so is the difference between a dead end and a next step. + * + * The first two lines match `alpr`'s Overpass source word for word; this is + * the same vocabulary, not a new one. What the status adds is *which* upstream + * and *how* it declined — the part a single-source layer never needs to say. + * + * Only what the browser can see is claimed. The proxy rotates mirrors behind + * `/api/overpass`, so a refusal arriving here means the proxy had nothing + * better to offer; how many mirrors it tried is not something this side knows, + * and the message does not pretend otherwise. + * @param {number} status - HTTP status the proxy returned; a code that is + * not a finite number is treated as absent rather than printed. + * @returns {RoadRequestError} + */ +export function roadRequestError(status) { + const code = Number.isFinite(status) ? status : null; + const message = + code === 429 + ? 'Overpass rate-limited' + : code === 504 + ? 'Overpass timed out' + : // Our own proxy answers these, not a mirror: 502 when every mirror + // failed at the network level, 503 from its local concurrency + // limiter before any mirror was asked. Neither is a refusal. + code === 502 + ? 'Overpass mirrors unreachable' + : code === 503 + ? 'Overpass temporarily unavailable' + : // A source is injected, so a caller's adapter may hand back a + // refusal with no status on it. "HTTP undefined" on a panel row is + // worse than not naming a number, so an unreadable code falls back + // to what `alpr` says when it cannot be more specific either. + code === null + ? 'Overpass temporarily unavailable' + : `Overpass refused the road query (HTTP ${code})`; + return new RoadRequestError(message, { status: code }); +} + /** Supply road responses, flow availability and one decoded flow cache. */ export function createTrafficSource({ fetchImpl = (...args) => globalThis.fetch(...args), diff --git a/src/layers/traffic/source.test.mjs b/src/layers/traffic/source.test.mjs index 8047ddbbd..b492223eb 100644 --- a/src/layers/traffic/source.test.mjs +++ b/src/layers/traffic/source.test.mjs @@ -1,7 +1,11 @@ import assert from 'node:assert/strict'; import test from 'node:test'; import { readFileSync } from 'node:fs'; -import { createTrafficSource } from './source.js'; +import { + RoadRequestError, + createTrafficSource, + roadRequestError, +} from './source.js'; import { clampBoundsAroundCenter } from '../../data/trafficBounds.js'; const bounds = { south: 30.267, west: -97.744, north: 30.268, east: -97.743 }; const fixture = readFileSync( @@ -188,3 +192,45 @@ test('road sources decode direction and coordinates before scene construction', ], }); }); + +test('an Overpass refusal is named by status, and keeps the status beside the words', () => { + // The panel prints this string verbatim, so the mapping is part of the + // layer's contract with the reader, not an implementation detail. + assert.equal(roadRequestError(429).message, 'Overpass rate-limited'); + assert.equal(roadRequestError(504).message, 'Overpass timed out'); + assert.equal( + roadRequestError(406).message, + 'Overpass refused the road query (HTTP 406)', + ); + assert.equal( + roadRequestError(500).message, + 'Overpass refused the road query (HTTP 500)', + ); + + // 502 and 503 come from the local proxy (mirrors unreachable, local + // limiter busy), so they must not read as a mirror refusing the query. + assert.equal(roadRequestError(502).message, 'Overpass mirrors unreachable'); + assert.equal( + roadRequestError(503).message, + 'Overpass temporarily unavailable', + ); + assert.equal(roadRequestError(503).status, 503); + + // An injected source may refuse without a readable code; a row must never + // print "HTTP undefined" at a reader. + for (const missing of [undefined, null, NaN, 'four-oh-six']) { + const vague = roadRequestError(missing); + assert.equal(vague.message, 'Overpass temporarily unavailable'); + assert.equal(vague.status, null, 'an unreadable code is absent, not zero'); + } + + const refused = roadRequestError(406); + assert.ok(refused instanceof RoadRequestError); + assert.ok(refused instanceof Error); + assert.equal(refused.name, 'RoadRequestError'); + assert.equal( + refused.status, + 406, + 'a caller must be able to branch on the code without parsing English', + ); +}); diff --git a/src/services/terrainHeights.js b/src/services/terrainHeights.js index 4366d632a..747eda145 100644 --- a/src/services/terrainHeights.js +++ b/src/services/terrainHeights.js @@ -1,7 +1,19 @@ import { ensureGeoidReady, geoidHeight } from '../data/geoid.js'; -/** Construct an instance-owned terrainHeights service with explicit dependencies. */ -export function createTerrainHeights({ source, signal }) { +/** Default live-entry ceiling for the in-memory cache (see CACHE_MAX_ENTRIES). */ +export const DEFAULT_MAX_CACHE_ENTRIES = 20_000; + +/** + * Construct an instance-owned terrainHeights service with explicit dependencies. + * `maxCacheEntries` overrides the cache bound; it exists so the bound is + * testable without minting twenty thousand coordinates, mirroring the injectable + * `maxBytes` on the GBFS reader. + */ +export function createTerrainHeights({ + source, + signal, + maxCacheEntries = DEFAULT_MAX_CACHE_ENTRIES, +}) { if (typeof source?.getHeights !== 'function') throw new TypeError('Terrain heights require a source'); signal?.throwIfAborted(); @@ -45,15 +57,71 @@ export function createTerrainHeights({ source, signal }) { /** Avoid repeatedly hitting a known-failing proxy from warm fallback reads. */ const GEOID_FALLBACK_COOLDOWN_MS = 60_000; + /** + * Live entry ceiling for the cache below. The rounding in point 2 of the file + * header makes repeated visits to one place share a key; it does nothing to + * bound the number of DISTINCT places a session visits, and this cache had no + * eviction, so a long session kept every coordinate it ever resolved for the + * lifetime of the page. + * + * This is a bound, not a budget. At the ~111 m floor grid a city-scale + * session stays well under it, so eviction is the pathological case rather + * than the normal one and a well-behaved session sees no extra network. Sized + * above the sibling caches (adsb.lol points 80, track history 200, TomTom + * tiles 256) because a miss here costs a proxy round trip, not just RAM. + */ + const CACHE_MAX_ENTRIES = + Number.isFinite(maxCacheEntries) && maxCacheEntries > 0 + ? Math.floor(maxCacheEntries) + : DEFAULT_MAX_CACHE_ENTRIES; + /** * In-memory cache: `"lat.toFixed(5),lon.toFixed(5)"` -> `{ellipsoid, source}`. * Module-scoped (not exported) — the only reads are through * `cachedEllipsoidalGround` and the internal lookup in - * `resolveEllipsoidalGround`. + * `resolveEllipsoidalGround`. Bounded at CACHE_MAX_ENTRIES; write through + * `cacheStore` and read a consumer-facing hit through `cacheTouch` so the + * eviction order stays least-recently-used. * @type {Map} */ const cache = new Map(); + /** + * Store an entry and drop the coldest ones once past the cap. `Map` iterates + * in insertion order, so deleting the first key evicts the least recently + * used provided every write and every consumer read re-inserts its key. + * + * Recency rather than plain insertion order because revisits here are + * spatial: a session comes back to cells near where it already was, so the + * cell it is parked on must not be evicted merely for having been resolved + * early. + * @param {string} key + * @param {{ellipsoid: number, source: 'reearth'|'geoid-fallback', retryAt?: number}} entry + */ + function cacheStore(key, entry) { + cache.delete(key); + cache.set(key, entry); + while (cache.size > CACHE_MAX_ENTRIES) { + const coldest = cache.keys().next().value; + if (coldest === undefined) break; + cache.delete(coldest); + } + } + + /** + * Read an entry and promote it to newest, so a coordinate a consumer keeps + * asking about survives eviction. Returns undefined on a miss. + * @param {string} key + * @returns {{ellipsoid: number, source: 'reearth'|'geoid-fallback', retryAt?: number}|undefined} + */ + function cacheTouch(key) { + const entry = cache.get(key); + if (entry === undefined) return undefined; + cache.delete(key); + cache.set(key, entry); + return entry; + } + /** * Builds the rounded cache key shared between the in-memory cache and the * `points` string sent to the proxy (see file header, point 2). @@ -74,7 +142,7 @@ export function createTerrainHeights({ source, signal }) { * @returns {number|null} */ function cachedEllipsoidalGround(lat, lon) { - const entry = cache.get(cacheKey(lat, lon)); + const entry = cacheTouch(cacheKey(lat, lon)); return entry ? entry.ellipsoid : null; } @@ -91,7 +159,7 @@ export function createTerrainHeights({ source, signal }) { * @returns {number|null} */ function cachedRealEllipsoidalGround(lat, lon) { - const entry = cache.get(cacheKey(lat, lon)); + const entry = cacheTouch(cacheKey(lat, lon)); return entry && entry.source === 'reearth' ? entry.ellipsoid : null; } @@ -213,6 +281,12 @@ export function createTerrainHeights({ source, signal }) { uncached.push(item); } + // What this call resolved, so the assembly below can report a point even + // if a later chunk's writes pushed it past CACHE_MAX_ENTRIES. Without it a + // batch big enough to evict its own earlier chunks would report points as + // unresolved that it had in fact just resolved. + const resolvedThisCall = new Map(); + // Resolve the network path in sequential <=CHUNK_SIZE chunks. Each chunk's // failure is isolated to that chunk's points (geoid fallback), so a single // bad chunk doesn't lose results for the rest of a large batch. @@ -230,7 +304,9 @@ export function createTerrainHeights({ source, signal }) { // one contact frozen at the geoid while its neighbors resolved). // An omitted point now caches nothing and retries on the next warm. if (Number.isFinite(ellipsoid)) { - cache.set(item.key, { ellipsoid, source: 'reearth' }); + const entry = { ellipsoid, source: 'reearth' }; + resolvedThisCall.set(item.key, entry); + cacheStore(item.key, entry); } } } catch { @@ -247,11 +323,13 @@ export function createTerrainHeights({ source, signal }) { item.lon, item.sourceOrthometricM, ); - cache.set(item.key, { + const entry = { ellipsoid, source: 'geoid-fallback', retryAt: Date.now() + GEOID_FALLBACK_COOLDOWN_MS, - }); + }; + resolvedThisCall.set(item.key, entry); + cacheStore(item.key, entry); } } } @@ -260,7 +338,7 @@ export function createTerrainHeights({ source, signal }) { // the upstream omitted has NO entry (round 6 — deliberately uncached so it // retries later): report it unresolved instead of throwing. return work.map((item) => { - const entry = cache.get(item.key); + const entry = cache.get(item.key) || resolvedThisCall.get(item.key); return entry ? { ellipsoid: entry.ellipsoid, source: entry.source } : { ellipsoid: null, source: 'unresolved' }; diff --git a/src/services/terrainHeights.test.mjs b/src/services/terrainHeights.test.mjs new file mode 100644 index 000000000..c9884e89c --- /dev/null +++ b/src/services/terrainHeights.test.mjs @@ -0,0 +1,130 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { + createTerrainHeights, + DEFAULT_MAX_CACHE_ENTRIES, +} from './terrainHeights.js'; + +/** + * A terrain source that always resolves, so these tests exercise the cache and + * never the geoid-fallback path. Heights are derived from the coordinate so an + * assertion can name the value it expects. + */ +function stubSource() { + const batches = []; + return { + batches, + async getHeights(chunk) { + batches.push(chunk.map((item) => item.key)); + return chunk.map((item) => ({ ellipsoid: heightAt(item.lon) })); + }, + }; +} + +/** Distinct 5-decimal coordinates along the equator: 0.001, 0.002, … */ +const lonAt = (i) => i / 1000; +const heightAt = (lon) => 1000 + Math.round(lon * 1000); +const pointAt = (i) => ({ lat: 0, lon: lonAt(i) }); + +test('the default cache bound is exported so callers can reason about it', () => { + assert.equal(DEFAULT_MAX_CACHE_ENTRIES, 20_000); +}); + +test('the in-memory cache stops growing once it reaches its entry bound', async () => { + const terrain = createTerrainHeights({ + source: stubSource(), + maxCacheEntries: 4, + }); + + // Six distinct coordinates through a cache that holds four. + for (let i = 0; i < 6; i += 1) { + await terrain.resolveEllipsoidalGround([pointAt(i)]); + } + + // The two coldest are gone. Before the bound existed the cache kept every + // coordinate a session ever resolved, so both of these still answered. + assert.equal(terrain.cachedEllipsoidalGround(0, lonAt(0)), null); + assert.equal(terrain.cachedEllipsoidalGround(0, lonAt(1)), null); + + // The four most recent are untouched — a bound, not a flush. + for (let i = 2; i < 6; i += 1) { + assert.equal( + terrain.cachedEllipsoidalGround(0, lonAt(i)), + heightAt(lonAt(i)), + `coordinate ${i} stayed warm`, + ); + } +}); + +test('a coordinate a consumer keeps reading outlives an equally old one it ignores', async () => { + const terrain = createTerrainHeights({ + source: stubSource(), + maxCacheEntries: 4, + }); + for (let i = 0; i < 4; i += 1) { + await terrain.resolveEllipsoidalGround([pointAt(i)]); + } + + // Read coordinate 0 and ignore coordinate 1. They are otherwise the same age, + // so insertion order alone could not tell them apart. + assert.equal( + terrain.cachedEllipsoidalGround(0, lonAt(0)), + heightAt(lonAt(0)), + ); + + // Two fresh coordinates push the cache two entries past its bound. + for (let i = 4; i < 6; i += 1) { + await terrain.resolveEllipsoidalGround([pointAt(i)]); + } + + assert.equal( + terrain.cachedEllipsoidalGround(0, lonAt(0)), + heightAt(lonAt(0)), + 'the cell being read survived eviction', + ); + assert.equal( + terrain.cachedEllipsoidalGround(0, lonAt(1)), + null, + 'the same-age cell nobody read was evicted instead', + ); +}); + +test('a batch larger than the cache bound still reports every point it resolved', async () => { + const terrain = createTerrainHeights({ + source: stubSource(), + maxCacheEntries: 4, + }); + const batch = Array.from({ length: 10 }, (_, i) => pointAt(i)); + + const results = await terrain.resolveEllipsoidalGround(batch); + + // Eviction inside the call must not turn a point this batch resolved into an + // unresolved one: the results are assembled from what the call produced, not + // from whatever survived in the cache by the time it finished. + assert.equal(results.length, batch.length); + results.forEach((result, i) => { + assert.equal(result.source, 'reearth', `point ${i} reported resolved`); + assert.equal(result.ellipsoid, heightAt(lonAt(i))); + }); + + // The cache still honours the bound afterwards. + const warm = batch.filter( + (point) => terrain.cachedEllipsoidalGround(point.lat, point.lon) !== null, + ); + assert.equal(warm.length, 4); +}); + +test('an out-of-range cache bound falls back to the default instead of disabling the cache', async () => { + for (const maxCacheEntries of [0, -1, Number.NaN, undefined]) { + const terrain = createTerrainHeights({ + source: stubSource(), + maxCacheEntries, + }); + await terrain.resolveEllipsoidalGround([pointAt(1)]); + assert.equal( + terrain.cachedEllipsoidalGround(0, lonAt(1)), + heightAt(lonAt(1)), + `maxCacheEntries=${String(maxCacheEntries)} kept a working cache`, + ); + } +}); diff --git a/src/tooling/liveProviders.test.mjs b/src/tooling/liveProviders.test.mjs index 250924b42..38953eff9 100644 --- a/src/tooling/liveProviders.test.mjs +++ b/src/tooling/liveProviders.test.mjs @@ -353,3 +353,31 @@ test('military cooldown bounds untrusted Retry-After and defaults server errors' assert.equal(Number(result.headers['retry-after']), seconds); } }); + +test('track backfill proxy returns 502 on an oversized upstream body and caches it as an error', async (t) => { + const tracks = install(providers.trackBackfillProxies(), true); + let callCount = 0; + t.mock.method(globalThis, 'fetch', async () => { + callCount++; + return { + ok: true, + status: 200, + headers: new Map(), + body: (async function* () { + yield Buffer.alloc(6 * 1024 * 1024, 'x'); + })(), + }; + }); + + const res1 = await tracks('/api/opensky-track', '?icao24=def456'); + assert.equal(res1.statusCode, 502); + assert.deepEqual(JSON.parse(res1.body), { + error: 'Upstream track response too large', + }); + + // Cached as a 502 (never a 200): a retry inside the window neither + // reads as an empty track nor spends OpenSky credits on another download. + const res2 = await tracks('/api/opensky-track', '?icao24=def456'); + assert.equal(res2.statusCode, 502); + assert.equal(callCount, 1); +});