mirror of
https://github.com/bilawalsidhu/gods-eye-view.git
synced 2026-11-09 21:02:19 +00:00
Merge pull request #1552 from IMBrandao/fix/firms-drop-snpp
fix(firms): stop requesting the retired Suomi NPP source
This commit is contained in:
7 files changed
+38
-23
No files matched your search
+4
-3
@@ -324,11 +324,12 @@ Bundled OSM-derived data is distributed under [ODbL 1.0](https://opendatacommons
|
||||
> We acknowledge the use of data and/or imagery from NASA's Fire Information for Resource Management System (FIRMS) (https://earthdata.nasa.gov/firms), part of NASA's Earth Observing System Data and Information System (EOSDIS).
|
||||
|
||||
FIRMS active fires are **fetched live at runtime** (CC0 / U.S. public domain data): the
|
||||
`/api/firms` server-side proxy merges the three VIIRS NRT sources (NOAA-20, NOAA-21,
|
||||
Suomi-NPP) and MODIS NRT (Terra + Aqua), clamped to the trailing 24 h, cached 30 min to respect the shared MAP_KEY
|
||||
`/api/firms` server-side proxy merges the two VIIRS NRT sources (NOAA-20, NOAA-21)
|
||||
and MODIS NRT (Terra + Aqua), clamped to the trailing 24 h, cached 30 min to respect the shared MAP_KEY
|
||||
transaction quota. Requires a free `FIRMS_MAP_KEY`
|
||||
(https://firms.modaps.eosdis.nasa.gov/api/map_key/); the layer is empty without it.
|
||||
The former bundled 2026-05-25 snapshot was removed 2026-07-16.
|
||||
The former bundled 2026-05-25 snapshot was removed 2026-07-16. Suomi NPP (`VIIRS_SNPP_NRT`)
|
||||
is not requested: NASA stopped delivering it on 2026-11-02.
|
||||
|
||||
### UCS Satellite Database (`ucs_satellites/`)
|
||||
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
- Stop requesting Suomi NPP in Active Fires. NASA stopped delivering its data
|
||||
on November 2, 2026, so `VIIRS_SNPP_NRT` returned no rows while still costing
|
||||
a MAP_KEY transaction on every refresh; the layer now merges VIIRS NOAA-20,
|
||||
VIIRS NOAA-21 and MODIS (IMBrandao, #1552).
|
||||
@@ -2859,6 +2859,9 @@ reporting are unchanged.
|
||||
|
||||
## FIRMS source status
|
||||
|
||||
The FIRMS proxy requests VIIRS NOAA-20, VIIRS NOAA-21 and MODIS NRT (Terra and
|
||||
Aqua). It skips VIIRS Suomi NPP, which NASA stopped delivering on 2026-11-02.
|
||||
|
||||
The FIRMS CSV parser rejects impossible acquisition calendar dates before the proxy filters records to the trailing 24 hours, while retaining valid UTC acquisition times.
|
||||
The `get_active_fires` tool reports acquisition times as UTC ISO timestamps,
|
||||
including zero-padded hours and minutes; missing or invalid dates and times
|
||||
@@ -4680,7 +4683,7 @@ its criteria cannot be silently ignored.
|
||||
| Datacenters ▣ | OSM extract (bundled) | `src/data/infrastructure.js` | — | static |
|
||||
| Dams ▰ | OpenInfraMap/OSM extract (bundled) | `src/data/infrastructure.js` | — | static |
|
||||
| Submarine Cables ◠ | TeleGeography public map (bundled) | `src/data/telegeographySubmarineCables.js` | — | static |
|
||||
| FIRMS Active Fires ▲ | NASA FIRMS live (VIIRS ×3 NRT + MODIS NRT, trailing 24h) | `src/data/firmsHeatmap.js` | `/api/firms` (`FIRMS_MAP_KEY`) | 10 min (proxy TTL 30 min) |
|
||||
| FIRMS Active Fires ▲ | NASA FIRMS live (VIIRS ×2 NRT + MODIS NRT, trailing 24h) | `src/data/firmsHeatmap.js` | `/api/firms` (`FIRMS_MAP_KEY`) | 10 min (proxy TTL 30 min) |
|
||||
| Street Level 📷 | Mapillary street-level imagery (vector tiles, Graph API, MapillaryJS) | `src/layers/streetLevel/` via `src/app/layers/streetLevel.js` | `/api/mapillary/status`, `/api/mapillary/tiles/coverage/{z}/{x}/{y}` (`MAPILLARY_CLIENT_TOKEN`) | camera-driven (320 ms debounce, ≤9 tiles, server memory cache 24 h) |
|
||||
| Wind 🌬 | NOAA GFS 10 m wind (keyless, 0.25°→1° grid; animated particles) | `src/data/wind.js` | `/api/wind` | 1 h (forecast cycle) |
|
||||
| GNSS Integrity 📡 (`gnss-integrity`) | adsb.lol point snapshot, low navigation accuracy binned per 0.5° cell (GEV NIC/NACp threshold + gpsjam.org formula and bands, keyless, ODbL) | `src/layers/gnss/` via `src/app/layers/gnss.js` | `/api/gnss-integrity` | camera-driven (600 ms debounce, minimum 2.5 s between reads) + 60s poll (server cache 60 s; 30 min rolling window) |
|
||||
|
||||
@@ -131,9 +131,9 @@ export async function fetchFirms(
|
||||
* NASA FIRMS live active-fire proxy with a memory + disk cache.
|
||||
* Upstream: https://firms.modaps.eosdis.nasa.gov/api/area/csv/{KEY}/{SOURCE}/world/2
|
||||
*
|
||||
* Merges three VIIRS NRT sources (NOAA-20, NOAA-21, Suomi-NPP — independent
|
||||
* satellites, no cross-source dedup) plus MODIS_NRT (combined Terra+Aqua,
|
||||
* ~1 km, numeric 0-100 confidence), fetched sequentially (quota courtesy)
|
||||
* Merges two VIIRS NRT sources (NOAA-20, NOAA-21 — independent satellites, no
|
||||
* cross-source dedup) plus MODIS_NRT (combined Terra+Aqua, ~1 km, numeric
|
||||
* 0-100 confidence), fetched sequentially (quota courtesy)
|
||||
* with `days=2` (`days=1` means "current UTC day", nearly empty just after
|
||||
* 00:00Z) and clamps to the trailing 24 h via src/data/firmsCsv.js. FIRMS
|
||||
* quota is 5,000 transactions / 10 min per MAP_KEY, so the cache is the point:
|
||||
@@ -141,6 +141,8 @@ export async function fetchFirms(
|
||||
* fresh-enough disk cache (.gev-cache/firms.json) prevents ANY upstream
|
||||
* fetch across dev-server restarts. Pattern mirrors celestrakProxy.
|
||||
*
|
||||
* VIIRS_SNPP_NRT is left out: NASA stopped delivering Suomi NPP on 2026-11-02.
|
||||
*
|
||||
* Routes:
|
||||
* GET /api/firms → {fetchedAt, stale, ttlMs, sources, count, fires}
|
||||
* GET /api/firms/status → {hasKey, lastFetch, count, stale, ttlMs, transactions}
|
||||
@@ -153,12 +155,7 @@ export async function fetchFirms(
|
||||
export function firmsProxy() {
|
||||
const TTL_MS = 30 * 60_000;
|
||||
const STATUS_TTL_MS = 5 * 60_000;
|
||||
const SOURCES = [
|
||||
'VIIRS_NOAA20_NRT',
|
||||
'VIIRS_NOAA21_NRT',
|
||||
'VIIRS_SNPP_NRT',
|
||||
'MODIS_NRT',
|
||||
];
|
||||
const SOURCES = ['VIIRS_NOAA20_NRT', 'VIIRS_NOAA21_NRT', 'MODIS_NRT'];
|
||||
/**
|
||||
* Per-source body cap. A VIIRS world/2 pull is ~130k rows x ~100 B, about
|
||||
* 13 MB per satellite in heavy fire season (MODIS runs smaller), so this is a
|
||||
|
||||
@@ -120,9 +120,10 @@ test('fire detection key survives a refetch that renumbers the index', () => {
|
||||
});
|
||||
|
||||
test('two satellites over the same pixel at the same time stay distinct', () => {
|
||||
// The proxy merges SNPP + NOAA-20 + NOAA-21 without dedup and their orbits
|
||||
// overlap, so position and time alone are NOT unique. Collapsing them let a
|
||||
// restored selection land on the other satellite's record.
|
||||
// The proxy merges NOAA-20 + NOAA-21 without dedup and their orbits overlap,
|
||||
// so position and time alone are NOT unique. Collapsing them let a restored
|
||||
// selection land on the other satellite's record. Archived Suomi NPP (`N`)
|
||||
// rows must stay distinct too.
|
||||
const base = makeFire({ satellite: 'N20' });
|
||||
const twin = makeFire({ satellite: 'N21' });
|
||||
const suomi = makeFire({ satellite: 'N' });
|
||||
|
||||
@@ -21,7 +21,10 @@ assert.notEqual(start, -1, 'FIRMS refresh function exists');
|
||||
const end = config.indexOf('\n }', start);
|
||||
assert.notEqual(end, -1, 'FIRMS refresh function closes');
|
||||
const refreshSource = config.slice(start, end + 4);
|
||||
const SOURCES = ['VIIRS_NOAA20_NRT', 'VIIRS_NOAA21_NRT', 'VIIRS_SNPP_NRT', 'MODIS_NRT'];
|
||||
// Read the list from production so these tests follow it.
|
||||
const listed = config.match(/const SOURCES = \[([^\]]*)\];/);
|
||||
assert.ok(listed, 'FIRMS source list exists');
|
||||
const SOURCES = [...listed[1].matchAll(/'([A-Z0-9_]+)'/g)].map((m) => m[1]);
|
||||
const NOW = Date.UTC(2026, 8, 11, 12);
|
||||
const recent = { acqDate: '2026-09-11', acqTime: '1100' };
|
||||
|
||||
@@ -54,7 +57,7 @@ test('FIRMS retains large sources in order and filters expired rows', async () =
|
||||
assert.equal(result.fires[199_999], large.at(-1));
|
||||
assert.equal(result.fires.at(-1), last);
|
||||
assert.deepEqual(result.sources, SOURCES.map((source, index) => ({
|
||||
source, count: [200_000, 1, 0, 0][index], ok: true,
|
||||
source, count: [200_000, 1, 0][index], ok: true,
|
||||
})));
|
||||
});
|
||||
|
||||
@@ -63,7 +66,7 @@ test('FIRMS keeps successful sources when another upstream fails', async () => {
|
||||
if (source === SOURCES[1]) throw new Error('upstream unavailable');
|
||||
return [recent];
|
||||
})('fixture');
|
||||
assert.equal(result.fires.length, 3);
|
||||
assert.equal(result.fires.length, SOURCES.length - 1);
|
||||
assert.deepEqual(result.sources, SOURCES.map((source, index) => ({
|
||||
source, count: index === 1 ? 0 : 1, ok: index !== 1,
|
||||
})));
|
||||
@@ -77,7 +80,7 @@ test('FIRMS reports one failure if consuming a source throws before append', asy
|
||||
// Fault injection for aggregation; ordinary parsed CSV returns an array.
|
||||
return { length: 1, [Symbol.iterator]() { throw new Error('aggregation failed'); } };
|
||||
})('fixture');
|
||||
assert.equal(result.fires.length, 3);
|
||||
assert.equal(result.fires.length, SOURCES.length - 1);
|
||||
assert.deepEqual(result.sources, SOURCES.map((source, index) => ({
|
||||
source, count: index === 0 ? 0 : 1, ok: index !== 0,
|
||||
})));
|
||||
@@ -107,7 +110,7 @@ function wrappedAddressSelectionFailure() {
|
||||
{ name: 'ConnectTimeoutError', code: 'UND_ERR_CONNECT_TIMEOUT', cause: inner });
|
||||
return new TypeError('fetch failed', { cause });
|
||||
}
|
||||
const URL_WITH_KEY = 'https://firms.modaps.eosdis.nasa.gov/api/area/csv/SECRET/VIIRS_SNPP_NRT/world/2';
|
||||
const URL_WITH_KEY = 'https://firms.modaps.eosdis.nasa.gov/api/area/csv/SECRET/VIIRS_NOAA20_NRT/world/2';
|
||||
|
||||
test('FIRMS failures log the socket cause, never the keyed URL', () => {
|
||||
const described = describeFetchError(addressSelectionFailure());
|
||||
@@ -307,3 +310,9 @@ test('every FIRMS upstream request goes through the IPv4 fallback', async () =>
|
||||
assert.ok(calls[0][2] instanceof AbortSignal);
|
||||
assert.deepEqual(calls[1], ['read', response, calls[0][2]], 'the body read shares the request deadline');
|
||||
});
|
||||
|
||||
test('FIRMS requests only the satellites NASA still delivers', () => {
|
||||
// NASA stopped delivering Suomi NPP on 2026-11-02; asking for it only spends
|
||||
// a MAP_KEY transaction.
|
||||
assert.ok(!SOURCES.includes('VIIRS_SNPP_NRT'));
|
||||
});
|
||||
@@ -313,9 +313,9 @@ test('FIRMS retains a large successful source during partial failure and filters
|
||||
const first = json(await request());
|
||||
assert.equal(first.count, 130001);
|
||||
assert.equal(first.sources.filter((s) => s.ok).length, 1);
|
||||
assert.equal(calls, 4);
|
||||
assert.equal(calls, 3);
|
||||
assert.equal(json(await request()).count, 130001);
|
||||
assert.equal(calls, 4);
|
||||
assert.equal(calls, 3);
|
||||
assert.deepEqual(json(await request('/status')).transactions, {
|
||||
used: 3,
|
||||
limit: 5000,
|
||||
|
||||
Reference in new issue
Block a user