mirror of
https://github.com/bilawalsidhu/gods-eye-view.git
synced 2026-11-22 11:12:18 +00:00
Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
72d9f91ad6 | ||
|
|
fda54f6611 | ||
|
|
0760733780 | ||
|
|
6b4b9e43c8 | ||
|
|
c16bfa5abd | ||
|
|
22bdf14be1 | ||
|
|
5aed9ce064 | ||
|
|
96847d5856 | ||
|
|
5eb6515f44 |
No files matched your search
@@ -1,2 +0,0 @@
|
||||
- `/api/firms` keeps its response bytes and sends them gzipped instead of
|
||||
serialising the whole fire set for every request (mazy1998, #1711).
|
||||
@@ -3297,15 +3297,6 @@ The `get_active_fires` tool reports acquisition times as UTC ISO timestamps,
|
||||
including zero-padded hours and minutes; missing or invalid dates and times
|
||||
remain `null`, including impossible calendar dates from a custom source.
|
||||
|
||||
`/api/firms` keeps its last response body as bytes and reuses it for requests
|
||||
that would get the same payload: the same cache entry and `stale` flag, while
|
||||
the clock stays inside the span over which the trailing-24-hour filter keeps
|
||||
the same detections. That span ends when the oldest kept acquisition minute
|
||||
turns 24 hours old, so requests minutes apart are still serialised afresh. A
|
||||
client that sends `Accept-Encoding: gzip` gets the body gzipped (compressed
|
||||
once per kept body, `Content-Encoding: gzip`); every response carries
|
||||
`Vary: Accept-Encoding`. The JSON is unchanged.
|
||||
|
||||
The FIRMS proxy records source success after appending its rows. If aggregation
|
||||
throws, that source reports failure without a contradictory success entry.
|
||||
The existing per-record append continues to support large feeds; sequential
|
||||
|
||||
@@ -0,0 +1,322 @@
|
||||
# Slowdown audit, November 2026
|
||||
|
||||
A sweep of `main` (`c02ede3bb`, 13 November 2026) for bugs that freeze the page
|
||||
or hold the frame rate down. This page lists what was found, why it happens and
|
||||
how to fix it, and names the pull request that carries each fix.
|
||||
|
||||
[PERFORMANCE.md](PERFORMANCE.md) is the measured baseline of a healthy build;
|
||||
this page is the list of things that fall outside it.
|
||||
|
||||
## How it was measured
|
||||
|
||||
- Keyless dev server, headless Chrome at 1440 x 900 on Apple silicon (Metal
|
||||
through ANGLE), default globe map.
|
||||
- Per scenario: load, a spell at rest, scripted drags and wheel zooms, then a
|
||||
settle. Long tasks were counted with a `PerformanceObserver`, the hot function
|
||||
named from a CPU profile, and `globe.getHeight`, `scene.sampleHeight`,
|
||||
`scene.pick` and friends wrapped to count calls and time.
|
||||
- A long task is one main-thread task over 50 ms. With no layers on, the
|
||||
baseline is two at load (longest about 240 ms) and none afterwards, with zero
|
||||
frames rendered while the camera is still.
|
||||
- Four profiling sessions ran on one machine at the same time, so absolute
|
||||
timings carry some noise. Ratios against the baseline are what to read.
|
||||
|
||||
Nothing here was measured on Google Photorealistic 3D Tiles, on a discrete or
|
||||
integrated PC GPU, or with keyed feeds (AISStream, FIRMS, TomTom). Those rows
|
||||
are marked "read in code".
|
||||
|
||||
## Findings and fixes
|
||||
|
||||
Worst first. "Fix" names the pull request that carries the fix.
|
||||
|
||||
### Measured
|
||||
|
||||
| # | Where | What happens | Fix |
|
||||
| ---: | --- | --- | --- |
|
||||
| 1 | Bike Share, local data layers, Transit, Rail | 18 s freeze switching Bike Share on over New York (2,519 stations); 2.7 s on Data Centers; multi-second freezes with Transit while zooming | #1688, #1689 |
|
||||
| 2 | Shipwrecks | 3 to 5 s freeze when the markers are built over loaded terrain; a share link with the layer on held the loading screen for 22.9 s in one run | #1703 |
|
||||
| 3 | Camera | Rendering stops until reload: at a whole-globe view looking straight down, pan a few times, then scroll one notch | #1704 |
|
||||
| 4 | Street Traffic | Zooming at 250 m over Manhattan: 25 long tasks in 24 s, longest 746 ms, 6.6 s in total | #1705 |
|
||||
| 5 | ALPR cameras | Depth samples inside the overlay paint, repeated on every re-enable: 240 samples and about 1 s across eight toggles. A first-sample freeze of up to 1.9 s was seen in the sweep but not on a second day | #1706 |
|
||||
| 6 | Wind | First zoom after load: up to 14 long tasks (longest 414 ms). Re-measured as one-off shader links, not a standing cost: see below | none proposed |
|
||||
| 7 | Submarine cables | About 2 s of long tasks on load, heap briefly at 1 GB, about 0.5 s again on each re-enable | #1707 (first step), #1722 (batched primitives) |
|
||||
| 8 | Live Flights at whole-globe view with detection labels | Fine on the test machine; at 4x CPU throttle, 8 fps with 143 long tasks in 15 s | #1708 (fleet pass), #1719 (overlay) |
|
||||
| 9 | Radio | 7 to 10 long tasks when the directory loads, longest 447 ms | #1709 |
|
||||
| 10 | Pointer pick | Not reproduced on a second look: see below | none needed |
|
||||
| 11 | CCTV | A still-image camera holds continuous rendering at 60 frames a second although its picture changes about every 10 s | #1710 |
|
||||
|
||||
### Read in code
|
||||
|
||||
| # | Where | What happens | Fix |
|
||||
| ---: | --- | --- | --- |
|
||||
| 12 | `/api/firms` | The whole fire set is filtered, serialised and sent uncompressed on every request; 15 to 55 MB estimated in a heavy season | #1711 |
|
||||
| 13 | AIS store | Both vessel maps are walked on every position message, and sorted whole once past the cap | #1712 |
|
||||
| 14 | Satellite clouds on the 3D Tiles map | A CPU pass over a 4096 x 2048 image, 90 to 180 ms measured on a synthetic canvas, twice in a row | #1713 (canvas path), #1721 (shader) |
|
||||
| 15 | NVG and FLIR styles | Over 100 texture reads per pixel, and 60 renders a second while the style is on. Measured afterwards: the reads cost 0.4 ms a frame at 1440 x 900 and 1.2 ms at 2560 x 1440 | #1718 |
|
||||
| 16 | Wind server | Full 0.25 degree GRIB fields decoded on the event loop. Measured afterwards: 68 ms blocked per warm refresh, about 250 ms on the first, about 400 ms when all six keys refresh together | #1717 |
|
||||
| 17 | Imported geo files and other per-feature ground clamps | Measured afterwards: imported files do not freeze; see below | none needed |
|
||||
|
||||
## Cause and fix, per finding
|
||||
|
||||
### 1. Terrain height reads
|
||||
|
||||
`globe.getHeight` casts a ray against the mesh of the tile under the point, and
|
||||
`scene.sampleHeight` renders a depth pass. Transit asked for every trail fix of
|
||||
every vehicle several times per poll; Bike Share sampled every station in one
|
||||
go. The fix keeps heights per point in one shared reader with a read budget
|
||||
(`src/services/terrainSurface.js`) and spreads the depth samples across frames.
|
||||
|
||||
### 2. Shipwrecks
|
||||
|
||||
`rebuild()` in `src/layers/shipwrecks/index.js` adds 7,958 billboards with
|
||||
`heightReference: CLAMP_TO_GROUND`. Each constructor runs
|
||||
`Billboard._updateClamping`, which intersects a ray with the terrain mesh,
|
||||
synchronously: 4.8 of the 4.9 s. The cost is only paid when terrain is loaded
|
||||
at the time the markers are built, so a cold start with the layer in the URL
|
||||
sometimes escapes it.
|
||||
|
||||
Fix: do not clamp. Wrecks are at sea and the markers already ignore the depth
|
||||
test, so place them on the ellipsoid with `HeightReference.NONE`.
|
||||
|
||||
### 3. Camera: rendering stops after pan then zoom
|
||||
|
||||
Pure Cesium; no application code is involved. Above 1,000 km,
|
||||
`ScreenSpaceCameraController` `handleZoom` rotates the camera about the cross
|
||||
product of the picked point's normal and the screen centre's normal whenever
|
||||
their dot product is between 0 and 1. With the pointer on the exact centre pixel
|
||||
the two are the same vector, yet `n · n` often rounds to `0.9999999999999999`,
|
||||
so the check passes with a zero axis. `Camera.rotate` then throws
|
||||
`DeveloperError: normalized result is not a number` from
|
||||
`Scene.initializeFrame`, outside the scene's own try/catch, so
|
||||
`scene.renderError` never fires and Cesium stops its render loop for good. The
|
||||
pans in the first reproduction only moved the camera to a pose where the
|
||||
rounding goes that way.
|
||||
|
||||
Fix: make `camera.rotate` ignore a zero or non-finite axis, and recover from any
|
||||
stopped loop by restoring the last drawn camera pose and restarting it a bounded
|
||||
number of times before showing Cesium's error panel.
|
||||
|
||||
### 4. Street Traffic road loading
|
||||
|
||||
Three costs in one task after the detail tiles arrive, in
|
||||
`src/layers/traffic/source.js`:
|
||||
|
||||
- `outsideDetailTiles` clips every coarse z12 road against every loaded detail
|
||||
tile (up to 16) for four regions.
|
||||
- The composed set is then handed on with `replace: true`, so
|
||||
`ingestion.js` parses every road again although each tile was parsed as it
|
||||
streamed.
|
||||
- `parseRoads` (`model.js`) asks `cachedGroundFloor` once per coordinate, and
|
||||
each call builds several string keys.
|
||||
|
||||
Fix: clip once per coarse tile against the detail tiles' union box and slice the
|
||||
work across frames (or do it in the vector-tile worker); reuse the roads parsed
|
||||
while streaming and parse only the clipped coarse remainder; look the floor up
|
||||
once per floor cell within a parse.
|
||||
|
||||
This is separate from the one-viewport-tile-at-a-time loading recorded in
|
||||
[KNOWN-ISSUES.md](KNOWN-ISSUES.md).
|
||||
|
||||
### 5. ALPR cameras
|
||||
|
||||
`refineAnchors` and `anchorFor` in `src/layers/alpr/overlay.js` call
|
||||
`scene.sampleHeight` from the overlay's paint, four per paint. Each is a depth
|
||||
render and a `readPixels`; the first one also compiles the pick shader. Results
|
||||
are kept on the entity, so a rebuild or a re-enable samples everything again.
|
||||
|
||||
Fix: on a terrain globe read `globe.getHeight` through the shared terrain reader
|
||||
instead of sampling; keep the answers in a map keyed by camera that outlives the
|
||||
entities; move what sampling remains out of the paint into a queue of one per
|
||||
frame.
|
||||
|
||||
### 6. Wind
|
||||
|
||||
The first reading blamed the layer's 916,476 vertices for slow depth reads
|
||||
during zoom. Measuring again did not support that: depth reads cost the same
|
||||
with and without Wind, hiding every wind primitive left pick time unchanged, and
|
||||
the stalls appear on the first zoom only, as shader programs link (seven
|
||||
programs, about 0.5 s, long tasks up to 166 ms). Later zooms had none.
|
||||
|
||||
The roughly 30 renders a second are the animation itself. It is already capped
|
||||
at 30 Hz and stops when paused, hidden, out of view or under reduced motion;
|
||||
lowering the rate would change how the layer looks. `disable()` releases the
|
||||
baked geometry on purpose, at the price of one task of about 130 ms on
|
||||
re-enable.
|
||||
|
||||
A third look, timing every program link on a preview build, settled it: Wind
|
||||
owns two shader programs, both link during load in 9 to 18 ms, and none ever
|
||||
linked during a zoom. The programs that do link on a zoom are Cesium's own
|
||||
globe-surface variants, with Wind on or off, and Cesium links them
|
||||
synchronously. No fix is proposed.
|
||||
|
||||
### 7. Submarine cables
|
||||
|
||||
`src/layers/submarineCables/ingestion.js` loads both collections through
|
||||
`Cesium.GeoJsonDataSource` with `clampToGround`. Switching the layer on is one
|
||||
task of about 1.4 s, and 1.2 s of it is Cesium reading the terrain height for
|
||||
the 1,917 ground-clamped landing markers, all created in the first frame,
|
||||
including the ones behind the globe.
|
||||
|
||||
Fix, first step: start the landing markers hidden and let the existing horizon
|
||||
sweep show the visible ones a few dozen per frame.
|
||||
|
||||
Fix, second step: draw from batched primitives in place of the entity data
|
||||
sources: one ground polyline primitive for the lines, one billboard collection
|
||||
for the landing markers, one point collection and one static polyline primitive
|
||||
for the reference stems, all kept while the layer is off. That removes the
|
||||
visualiser tasks, the stall on switch-off and on re-enable, and about 900 MB of
|
||||
heap. The first switch-on still links shaders and builds the ground-line
|
||||
geometry once.
|
||||
|
||||
### 8. Live Flights at whole-globe view
|
||||
|
||||
Two costs that scale with the 7,500 aircraft of a full poll:
|
||||
|
||||
- The fleet pass (`src/layers/flights/rendering.js`, every 80 ms) dead-reckons
|
||||
each visible aircraft and asks `cachedGroundFloor` for it, including aircraft
|
||||
at cruise altitude that can never touch the floor. Each lookup formats numbers
|
||||
to strings several times and rewrites a cache entry: 40% of the busy time.
|
||||
- The detection overlay (`src/data/detection.js` `_drawOverlay`) collects every
|
||||
detectable object and strokes every bracket on every frame.
|
||||
|
||||
Fix for the first: skip the floor path for a contact clearly above any terrain,
|
||||
make the floor-cell key a number, and stop promoting the cache entry on every
|
||||
read.
|
||||
|
||||
The second turned out smaller than it looked. The bracket strokes are already
|
||||
one path per colour, and collecting the objects is about 3 ms of a 56 ms frame
|
||||
at 4x throttle once the first fix is in. What costs is the per-object loop in
|
||||
`_drawOverlay` (projection, occlusion, alpha, twelve path calls per bracket).
|
||||
|
||||
Fix for the second: keep what a sweep built and reuse it on the next paint when
|
||||
the view and every contact are exactly what it was built from, compared value
|
||||
by value so no layer has to announce a change. Between two fleet passes most
|
||||
paints qualify, which cuts frame work by a fifth to a third on a still camera
|
||||
at full speed. At 4x throttle a frame outlasts a fleet pass, no paint
|
||||
qualifies, and the gain is a few milliseconds; the fleet pass is what
|
||||
dominates there.
|
||||
|
||||
### 9. Radio
|
||||
|
||||
`stationTags()` in `src/layers/radio/categories.js` splits and normalises a
|
||||
station's tags on every `hasTag` call, and `buildRadioCategories` runs that for
|
||||
every station against every category, twice.
|
||||
|
||||
Fix: normalise once per station at ingest and count categories from that.
|
||||
|
||||
### 10. Pointer pick
|
||||
|
||||
Three sessions saw a single `scene.pick` of 207 to 342 ms during a gesture and
|
||||
attributed it to `pickPointer` in `src/voice/pointerPick.js`. Timing every pick
|
||||
across pointer rests, drags and wheel bursts did not bear that out: of 78 picks
|
||||
with CCTV on over London all but two took 3 to 8 ms, and with five heavy layers
|
||||
on, 81 picks took 5 to 30 ms apart from the first (135 ms). `pickPointer` picks
|
||||
at most once per pointer rest.
|
||||
|
||||
The slow pick is the first pick after a layer appears, once per page load, most
|
||||
likely the pick shader compiling, and its stack is CCTV's own hover pick
|
||||
(`src/layers/cctv/hover.js`). No fix is proposed: removing the dwell pick would
|
||||
only move the one-off cost to the next click.
|
||||
|
||||
### 11. CCTV still-image cameras
|
||||
|
||||
`src/layers/cctv/projection.js` holds continuous rendering for as long as any
|
||||
projected camera is active and redraws the projection every frame.
|
||||
|
||||
Fix: hold only for video. For a still image, request one render when a new image
|
||||
lands or while a focus fade runs.
|
||||
|
||||
### 12. `/api/firms`
|
||||
|
||||
`server/providers/firms.js` builds the payload and calls `JSON.stringify` per
|
||||
request, with no kept body and no compression, and the client refetches every
|
||||
ten minutes.
|
||||
|
||||
Fix: keep the response bytes and their gzip and serve those, as the OpenSky
|
||||
provider does. The route filters to the trailing 24 hours at serve time, so the
|
||||
bytes can only be reused while that filter keeps the same records (about a
|
||||
minute at a time): bursts and several clients gain most, and every response is
|
||||
about seven times smaller.
|
||||
|
||||
### 13. AIS store
|
||||
|
||||
`server/providers/vessels/ais-store.js` calls `pruneAisStreamCache()` for every
|
||||
message; the static-data map is never pruned.
|
||||
|
||||
Fix: prune on a timer, evict in insertion order, and drop a vessel's static
|
||||
entry with the vessel.
|
||||
|
||||
### 14. Satellite clouds on the 3D Tiles map
|
||||
|
||||
`src/layers/weather/infraredImage.js` runs `getImageData`, a per-pixel power
|
||||
curve and `putImageData` over the whole canvas.
|
||||
|
||||
Fix: a 256-entry lookup table applied in place for the canvas path, which
|
||||
halves it; then, for the shell on the 3D Tiles map, the same curve in its
|
||||
fragment shader from a strength uniform, so the image is never read back and a
|
||||
strength change builds no new image.
|
||||
|
||||
### 15. NVG and FLIR styles
|
||||
|
||||
`src/styles/surveillance.js` loops 11 x 11 texture reads per pixel and
|
||||
`src/styles/thermal.js` totals 106. Timed on the GPU, those loops cost about
|
||||
0.4 ms a frame at 1440 x 900 and 1.2 ms at 2560 x 1440 on the test machine:
|
||||
real, but small there. The standing cost is that a style with animated grain
|
||||
holds continuous rendering at 60 frames a second.
|
||||
|
||||
Fix: split the bloom into a row pass and a column pass, which gives the same
|
||||
picture from a fifth to a half of the reads, and step the grain's clock at
|
||||
15 Hz, asking for one frame per step instead of holding the render loop. The
|
||||
grain then changes 15 times a second, which is visible in motion.
|
||||
|
||||
Switching on any style stage at all costs more than the bloom did (1.8 ms a
|
||||
frame at 1440 x 900 for the one-line noir style). That was not looked into.
|
||||
|
||||
### 16. Wind server
|
||||
|
||||
`server/providers/wind/decode.js` decodes about a million doubles per message
|
||||
synchronously, hourly, for six keys.
|
||||
|
||||
Fix: decode on a worker thread, and keep the sampled U and V of the last
|
||||
forecast file per model so that choosing an overlay fetches only its field.
|
||||
|
||||
### 17. Per-feature ground clamps
|
||||
|
||||
`heightReference: CLAMP_TO_GROUND` per feature is also set in
|
||||
`installations/rendering.js`, `alpr/presentation.js`, `ghostMaps/index.js`,
|
||||
`earthquakes/index.js`, `airQuality/index.js`, `news/index.js` and
|
||||
`osmPlaces/index.js`. Their counts were small in these runs (30 to 1,500) and
|
||||
none froze.
|
||||
|
||||
Imported geo files were the one unbounded case, and measuring them cleared
|
||||
them: 20,000 generated points (the import cap) finish in about 1.5 s with a
|
||||
longest task near 0.5 s, because `settleGroundPins` in
|
||||
`src/data/geoFileImport.js` already places pins with `HeightReference.NONE` and
|
||||
resolves heights 256 at a time with a yield between batches. No fix is needed.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
- Start-up of a production build: viewer ready in about 0.5 s, loading screen
|
||||
gone at 1.6 s, three long tasks totalling about 300 ms.
|
||||
- Idle: no frames rendered with no layers on; every render hold was released
|
||||
when its layer went off.
|
||||
- Leaks: all 41 layers toggled three times and ten of them eight times; no
|
||||
timer, scene listener, primitive or data source grew per cycle. The heap
|
||||
levelled off near 320 MB over 7.5 minutes of toggling eleven layers.
|
||||
- At most one long task after load: Military, Satellites (including 11,903
|
||||
Starlink points), Earthquakes, News, Wildlife, GNSS, Drone Airspace, Military
|
||||
Installations, Fire Perimeters, rain radar, lightning, and satellite clouds on
|
||||
the globe map.
|
||||
- Live Flights at city scale with the same 7,500 aircraft loaded: 60 fps, no
|
||||
long tasks.
|
||||
|
||||
## Not covered
|
||||
|
||||
- Google Photorealistic 3D Tiles: nothing was run there. Depth sampling is the
|
||||
only height source on that map, so findings 4, 5 and 17 may be worse.
|
||||
- Live Vessels, FIRMS with data, TomTom traffic, Local ADS-B.
|
||||
- Frame rate on anything but one Apple silicon machine.
|
||||
- One all-layers run over Manhattan grew the heap from 346 to 638 MB in 60 s;
|
||||
whether that is retained was not established.
|
||||
- One Satellites load showed a 5.2 s long task once and never again in four more
|
||||
loads.
|
||||
@@ -6,6 +6,9 @@ specification and should not be used to predict performance on untested systems.
|
||||
The original capture artifacts are not included here, so this page records
|
||||
results rather than defining a runnable benchmark.
|
||||
|
||||
For slowdowns found since, with their causes and fixes, see
|
||||
[PERFORMANCE-AUDIT-2026-11.md](PERFORMANCE-AUDIT-2026-11.md).
|
||||
|
||||
## Test context
|
||||
|
||||
The baseline was captured on 22 August 2026 with these conditions:
|
||||
|
||||
@@ -376,7 +376,6 @@
|
||||
"modules": [
|
||||
"server/providers/firms.js",
|
||||
"server/providers/common/http.js",
|
||||
"server/providers/common/compress.js",
|
||||
"src/sources/httpBody.js",
|
||||
"src/i18n/marker.js",
|
||||
"src/data/firmsCsv.js"
|
||||
|
||||
+10
-58
@@ -3,12 +3,7 @@ import https from 'node:https';
|
||||
import { promises as fsp } from 'node:fs';
|
||||
import { Readable } from 'node:stream';
|
||||
|
||||
import {
|
||||
filterTrailing24h,
|
||||
parseFirmsCsv,
|
||||
trailing24hWindow,
|
||||
} from '../../src/data/firmsCsv.js';
|
||||
import { acceptsGzip, encodedBody, gzipBody } from './common/compress.js';
|
||||
import { filterTrailing24h, parseFirmsCsv } from '../../src/data/firmsCsv.js';
|
||||
import { readResponseTextCapped } from './common/http.js';
|
||||
|
||||
const NULL_BODY_STATUSES = new Set([204, 205, 304]);
|
||||
@@ -184,11 +179,6 @@ export function firmsProxy() {
|
||||
let statusCache = null;
|
||||
/** @type {?Promise<?{used: number, limit: number}>} */
|
||||
let statusInflight = null;
|
||||
/**
|
||||
* The last /api/firms payload as bytes, with its gzip. One slot.
|
||||
* @type {?{entry: object, stale: boolean, from: number, until: number, body: Buffer, gzip: ?Promise<?Buffer>}}
|
||||
*/
|
||||
let served = null;
|
||||
|
||||
const mapKey = () => String(process.env.FIRMS_MAP_KEY || '').trim();
|
||||
|
||||
@@ -269,8 +259,12 @@ export function firmsProxy() {
|
||||
return { at: now, sources, fires };
|
||||
}
|
||||
|
||||
/** Cache entry and the fires to serve from it → response payload. */
|
||||
function buildPayload(entry, stale, fires) {
|
||||
/**
|
||||
* Cache entry → response payload. Fires are RE-filtered to the trailing
|
||||
* 24 h at serve time so a stale cache never serves >24h-old detections.
|
||||
*/
|
||||
function buildPayload(entry, stale) {
|
||||
const fires = filterTrailing24h(entry.fires, Date.now());
|
||||
return {
|
||||
fetchedAt: entry.at,
|
||||
stale,
|
||||
@@ -281,32 +275,6 @@ export function firmsProxy() {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The payload as the bytes to send. Serialising the whole fire set is tens
|
||||
* of MB in a heavy season, so it is done once and kept, not once per request.
|
||||
*
|
||||
* Fires are RE-filtered to the trailing 24 h at serve time so a stale cache
|
||||
* never serves >24h-old detections. The kept bytes are therefore reused only
|
||||
* for the same entry and stale flag while the clock is inside the span over
|
||||
* which that filter keeps the same records: a request never gets a body that
|
||||
* serialising for it would not have produced.
|
||||
*/
|
||||
function payloadBody(entry, stale) {
|
||||
const now = Date.now();
|
||||
if (
|
||||
served?.entry === entry &&
|
||||
served.stale === stale &&
|
||||
now >= served.from &&
|
||||
now <= served.until
|
||||
) {
|
||||
return served;
|
||||
}
|
||||
const { kept, from, until } = trailing24hWindow(entry.fires, now);
|
||||
const body = Buffer.from(JSON.stringify(buildPayload(entry, stale, kept)));
|
||||
served = { entry, stale, from, until, body, gzip: null };
|
||||
return served;
|
||||
}
|
||||
|
||||
/** mapkey_status transactions, cached 5 min, best-effort (null on failure). */
|
||||
function getTransactions(key) {
|
||||
const now = Date.now();
|
||||
@@ -359,22 +327,6 @@ export function firmsProxy() {
|
||||
});
|
||||
res.end(JSON.stringify(obj));
|
||||
};
|
||||
const sendPayload = async (entry, stale) => {
|
||||
const kept = payloadBody(entry, stale);
|
||||
// Compressed once per kept body, on the first request that accepts
|
||||
// gzip, so an identity response never pays or waits for it.
|
||||
const gzip = acceptsGzip(req)
|
||||
? await (kept.gzip ??= gzipBody(kept.body).catch(() => null))
|
||||
: null;
|
||||
const encoded = encodedBody(req, kept.body, gzip);
|
||||
if (res.headersSent) return;
|
||||
res.writeHead(200, {
|
||||
'Content-Type': 'application/json',
|
||||
'Cache-Control': 'no-store',
|
||||
...encoded.headers,
|
||||
});
|
||||
res.end(encoded.body);
|
||||
};
|
||||
try {
|
||||
const subPath = String(req.url || '').split('?')[0];
|
||||
const key = mapKey();
|
||||
@@ -411,7 +363,7 @@ export function firmsProxy() {
|
||||
|
||||
const entry = mem;
|
||||
if (entry && Date.now() - entry.at < TTL_MS) {
|
||||
await sendPayload(entry, false);
|
||||
sendJson(200, buildPayload(entry, false));
|
||||
return;
|
||||
}
|
||||
// Stale or missing → refresh, single-flight (concurrent requests
|
||||
@@ -437,9 +389,9 @@ export function firmsProxy() {
|
||||
const pending = inflight;
|
||||
const fresh = await pending;
|
||||
if (fresh) {
|
||||
await sendPayload(fresh, false);
|
||||
sendJson(200, buildPayload(fresh, false));
|
||||
} else if (entry) {
|
||||
await sendPayload(entry, true); // upstream down — stale beats empty
|
||||
sendJson(200, buildPayload(entry, true)); // upstream down — stale beats empty
|
||||
} else {
|
||||
sendJson(502, {
|
||||
error: 'firms fetch failed and no cache available',
|
||||
|
||||
+2
-27
@@ -182,44 +182,19 @@ export function acquisitionMsUtc(acqDate, acqTime) {
|
||||
* @returns {Array<Object>} Filtered records (original objects, order preserved).
|
||||
*/
|
||||
export function filterTrailing24h(records, nowMs) {
|
||||
return trailing24hWindow(records, nowMs).kept;
|
||||
}
|
||||
|
||||
/**
|
||||
* {@link filterTrailing24h} with the span of `nowMs` values, inclusive, over
|
||||
* which it keeps exactly these records. A caller holding something built from
|
||||
* `kept` can reuse it while its clock stays inside `from..until`.
|
||||
* @param {Array<{acqDate: string, acqTime: string|number}>} records - Parser records.
|
||||
* @param {number} nowMs - Reference epoch milliseconds.
|
||||
* @returns {{kept: Array<Object>, from: number, until: number}} Span in epoch
|
||||
* ms; NaN (never reusable) for unusable input.
|
||||
*/
|
||||
export function trailing24hWindow(records, nowMs) {
|
||||
if (!Array.isArray(records) || !Number.isFinite(nowMs))
|
||||
return { kept: [], from: NaN, until: NaN };
|
||||
if (!Array.isArray(records) || !Number.isFinite(nowMs)) return [];
|
||||
const oldest = nowMs - WINDOW_MS;
|
||||
const newest = nowMs + FORWARD_SLACK_MS;
|
||||
let from = -Infinity;
|
||||
let until = Infinity;
|
||||
const memo = new Map();
|
||||
const kept = records.filter((record) => {
|
||||
return records.filter((record) => {
|
||||
const key = `${record.acqDate}:${record.acqTime}`;
|
||||
let ms = memo.get(key);
|
||||
if (ms === undefined) {
|
||||
ms = acquisitionMsUtc(record.acqDate, record.acqTime);
|
||||
memo.set(key, ms);
|
||||
// Each distinct time narrows the span to the clock values that leave
|
||||
// it on the same side of the window.
|
||||
if (ms < oldest) from = Math.max(from, ms + WINDOW_MS + 1);
|
||||
else if (ms > newest) until = Math.min(until, ms - FORWARD_SLACK_MS - 1);
|
||||
else if (Number.isFinite(ms)) {
|
||||
from = Math.max(from, ms - FORWARD_SLACK_MS);
|
||||
until = Math.min(until, ms + WINDOW_MS);
|
||||
}
|
||||
}
|
||||
return Number.isFinite(ms) && ms >= oldest && ms <= newest;
|
||||
});
|
||||
return { kept, from, until };
|
||||
}
|
||||
|
||||
/** Trimmed string cell at index, '' for missing columns. */
|
||||
|
||||
@@ -7,15 +7,11 @@ import dnsLookup from 'node:dns';
|
||||
import https from 'node:https';
|
||||
import net from 'node:net';
|
||||
import { Readable } from 'node:stream';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { gunzipSync } from 'node:zlib';
|
||||
import { filterTrailing24h, parseFirmsCsv, trailing24hWindow } from './firmsCsv.js';
|
||||
import { filterTrailing24h, parseFirmsCsv } from './firmsCsv.js';
|
||||
import {
|
||||
describeFetchError,
|
||||
fetchFirms,
|
||||
fetchIpv4,
|
||||
firmsProxy,
|
||||
isAddressSelectionFailure,
|
||||
} from '../../server/providers/firms.js';
|
||||
|
||||
@@ -320,119 +316,3 @@ test('FIRMS requests only the satellites NASA still delivers', () => {
|
||||
// a MAP_KEY transaction.
|
||||
assert.ok(!SOURCES.includes('VIIRS_SNPP_NRT'));
|
||||
});
|
||||
|
||||
// The /api/firms handler itself, over a disk cache seeded in a temp directory:
|
||||
// no server, no upstream, and a placeholder MAP_KEY that is never sent anywhere.
|
||||
const TTL_MS = 30 * 60_000;
|
||||
const fire = (acqDate, acqTime, lat = 1) => ({ lat, lon: 2, frp: 4.5, confidence: 'n', brightness: 330.1,
|
||||
brightnessTi5: 290.2, daynight: 'D', acqDate, acqTime, satellite: 'N20', instrument: 'VIIRS' });
|
||||
const SERVED_SOURCES = SOURCES.map((source, index) => ({ source, count: index === 0 ? 20_001 : 0, ok: true }));
|
||||
// 20,000 detections an hour old and one that turns 24 h old five minutes after NOW.
|
||||
const SERVED_FIRES = [fire('2026-09-10', '1205'), ...Array.from({ length: 20_000 }, (_, i) => fire('2026-09-11', '1100', i / 1000))];
|
||||
const LEAVES_AT = NOW + 5 * 60_000;
|
||||
/** What the handler sent before it kept a body: one JSON.stringify of the payload per request. */
|
||||
const unfixedBody = (at, stale, now) => {
|
||||
const fires = filterTrailing24h(SERVED_FIRES, now);
|
||||
return JSON.stringify({ fetchedAt: at, stale, ttlMs: TTL_MS, sources: SERVED_SOURCES, count: fires.length, fires });
|
||||
};
|
||||
|
||||
function createFirmsHandler(t, at) {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'firms-proxy-'));
|
||||
fs.mkdirSync(path.join(dir, '.gev-cache'));
|
||||
fs.writeFileSync(path.join(dir, '.gev-cache', 'firms.json'), JSON.stringify({ at, sources: SERVED_SOURCES, fires: SERVED_FIRES }));
|
||||
const cwd = process.cwd();
|
||||
const key = process.env.FIRMS_MAP_KEY;
|
||||
const realFetch = globalThis.fetch;
|
||||
process.env.FIRMS_MAP_KEY = 'unit-test-placeholder';
|
||||
// A refresh must never reach NASA from a test: every upstream call fails.
|
||||
globalThis.fetch = async () => { throw new Error('upstream down'); };
|
||||
t.mock.method(console, 'warn', () => {});
|
||||
t.mock.timers.enable({ apis: ['Date'], now: NOW });
|
||||
t.after(() => {
|
||||
globalThis.fetch = realFetch;
|
||||
if (key === undefined) delete process.env.FIRMS_MAP_KEY; else process.env.FIRMS_MAP_KEY = key;
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
let handler;
|
||||
process.chdir(dir); // the proxy resolves its cache directory when created
|
||||
try {
|
||||
firmsProxy().configureServer({ middlewares: { use: (route, fn) => { handler = fn; } } });
|
||||
} finally {
|
||||
process.chdir(cwd);
|
||||
}
|
||||
return (acceptEncoding) => new Promise((resolve, reject) => {
|
||||
const res = {
|
||||
headersSent: false,
|
||||
writeHead(status, headers) { this.headersSent = true; this.status = status; this.headers = headers; },
|
||||
end(body) { resolve({ status: this.status, headers: this.headers, body }); },
|
||||
};
|
||||
handler({ url: '/', headers: acceptEncoding === undefined ? {} : { 'accept-encoding': acceptEncoding } }, res).catch(reject);
|
||||
});
|
||||
}
|
||||
|
||||
test('/api/firms serialises once for requests that share a payload', async (t) => {
|
||||
const request = createFirmsHandler(t, NOW);
|
||||
const first = await request();
|
||||
t.mock.timers.setTime(LEAVES_AT); // exactly 24 h old is still inside the window
|
||||
const second = await request();
|
||||
assert.equal(first.status, 200);
|
||||
assert.ok(Buffer.isBuffer(first.body), 'the body is kept as bytes');
|
||||
assert.equal(second.body, first.body, 'the second request sends the kept bytes, not a new serialisation');
|
||||
assert.equal(first.body.toString(), unfixedBody(NOW, false, NOW), 'byte-identical to the per-request serialisation');
|
||||
assert.equal(JSON.parse(first.body).count, 20_001);
|
||||
});
|
||||
|
||||
test('/api/firms sends gzip only to a client that accepts it', async (t) => {
|
||||
const request = createFirmsHandler(t, NOW);
|
||||
const identity = await request();
|
||||
const gzipped = await request('gzip, deflate, br');
|
||||
const refused = await request('gzip;q=0, br');
|
||||
assert.deepEqual(identity.headers,
|
||||
{ 'Content-Type': 'application/json', 'Cache-Control': 'no-store', Vary: 'Accept-Encoding' });
|
||||
assert.deepEqual(gzipped.headers, { 'Content-Type': 'application/json', 'Cache-Control': 'no-store',
|
||||
'Content-Encoding': 'gzip', Vary: 'Accept-Encoding' });
|
||||
assert.deepEqual(refused.headers, identity.headers);
|
||||
assert.equal(refused.body, identity.body);
|
||||
assert.ok(gzipped.body.byteLength < identity.body.byteLength / 4, 'the fire set compresses');
|
||||
assert.equal(gunzipSync(gzipped.body).toString(), unfixedBody(NOW, false, NOW), 'decodes to the identity bytes');
|
||||
assert.equal((await request('gzip')).body, gzipped.body, 'compressed once');
|
||||
});
|
||||
|
||||
test('/api/firms never reuses a body the request would not have produced', async (t) => {
|
||||
const request = createFirmsHandler(t, NOW - TTL_MS + 6 * 60_000);
|
||||
const at = NOW - TTL_MS + 6 * 60_000;
|
||||
const before = await request();
|
||||
t.mock.timers.setTime(LEAVES_AT + 1); // one detection is now older than 24 h
|
||||
const after = await request('gzip');
|
||||
assert.equal(before.body.toString(), unfixedBody(at, false, NOW));
|
||||
assert.equal(gunzipSync(after.body).toString(), unfixedBody(at, false, LEAVES_AT + 1));
|
||||
assert.equal(JSON.parse(gunzipSync(after.body)).count, 20_000);
|
||||
t.mock.timers.setTime(NOW + 6 * 60_000); // the entry reaches its TTL and the refresh fails
|
||||
const stale = await request();
|
||||
assert.equal(stale.body.toString(), unfixedBody(at, true, NOW + 6 * 60_000));
|
||||
assert.equal((await request()).body, stale.body, 'the stale body is kept too');
|
||||
});
|
||||
|
||||
test('trailing24hWindow spans the clock values that keep the same records', () => {
|
||||
const records = [fire('2026-09-10', '1205'), fire('2026-09-10', '1300'), fire('2026-09-11', '1100'),
|
||||
fire('2026-09-11', '1430'), fire('2026-09-11', '1600'), fire('bad-date', '1006')];
|
||||
const edges = records.map((r) => Date.parse(`${r.acqDate}T${r.acqTime.slice(0, 2)}:${r.acqTime.slice(2)}Z`))
|
||||
.filter(Number.isFinite).flatMap((ms) => [ms - 2 * 3600_000, ms + 24 * 3600_000]);
|
||||
assert.equal(edges.length, 10);
|
||||
const clocks = [NOW, ...edges.flatMap((ms) => [ms - 1, ms, ms + 1])];
|
||||
for (const now of clocks) {
|
||||
const { kept, from, until } = trailing24hWindow(records, now);
|
||||
assert.deepEqual(kept, filterTrailing24h(records, now));
|
||||
assert.ok(from <= now && now <= until, 'the span holds its own clock');
|
||||
for (const other of clocks) {
|
||||
const then = filterTrailing24h(records, other);
|
||||
const same = then.length === kept.length && then.every((r, i) => r === kept[i]);
|
||||
// Before the first record and after the last both keep nothing; the span covers one side.
|
||||
const bothEmpty = then.length === 0 && kept.length === 0;
|
||||
if (other >= from && other <= until) assert.ok(same, `${other} is inside the span of ${now}`);
|
||||
else assert.ok(!same || bothEmpty, `${other} is outside the span of ${now}`);
|
||||
}
|
||||
}
|
||||
assert.deepEqual(trailing24hWindow([], NOW), { kept: [], from: -Infinity, until: Infinity });
|
||||
assert.deepEqual(trailing24hWindow(records, NaN), { kept: [], from: NaN, until: NaN });
|
||||
});
|
||||
Reference in new issue
Block a user