mirror of
https://github.com/bilawalsidhu/gods-eye-view.git
synced 2026-11-10 13:22:18 +00:00
fix(firms): stop requesting the retired Suomi NPP source
NASA stopped delivering Suomi NPP data on 2026-11-02 (FIRMS news page). With a live MAP_KEY on 2026-11-04, VIIRS_SNPP_NRT came back ok with zero rows while NOAA-20, NOAA-21 and MODIS returned 57k, 49k and 11k, so the source only spent one transaction per refresh. The proxy now requests NOAA-20, NOAA-21 and MODIS NRT. The "N" satellite label stays, so cached or archived Suomi NPP rows still read SNPP. Tests count sources from the list instead of assuming four, and a regression test keeps the retired source out of the production list. Refs #849 Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
ac28a11e41
commit
84c1fef8d3
5 files changed
+31
-17
No files matched your search
+4
-3
@@ -273,11 +273,12 @@ Bundled OSM-derived data is distributed under [ODbL 1.0](https://opendatacommons
|
|||||||
> We acknowledge the use of data and/or imagery from NASA's Fire Information for Resource Management System (FIRMS) (https://earthdata.nasa.gov/firms), part of NASA's Earth Observing System Data and Information System (EOSDIS).
|
> We acknowledge the use of data and/or imagery from NASA's Fire Information for Resource Management System (FIRMS) (https://earthdata.nasa.gov/firms), part of NASA's Earth Observing System Data and Information System (EOSDIS).
|
||||||
|
|
||||||
FIRMS active fires are **fetched live at runtime** (CC0 / U.S. public domain data): the
|
FIRMS active fires are **fetched live at runtime** (CC0 / U.S. public domain data): the
|
||||||
`/api/firms` server-side proxy merges the three VIIRS NRT sources (NOAA-20, NOAA-21,
|
`/api/firms` server-side proxy merges the two VIIRS NRT sources (NOAA-20, NOAA-21)
|
||||||
Suomi-NPP) and MODIS NRT (Terra + Aqua), clamped to the trailing 24 h, cached 30 min to respect the shared MAP_KEY
|
and MODIS NRT (Terra + Aqua), clamped to the trailing 24 h, cached 30 min to respect the shared MAP_KEY
|
||||||
transaction quota. Requires a free `FIRMS_MAP_KEY`
|
transaction quota. Requires a free `FIRMS_MAP_KEY`
|
||||||
(https://firms.modaps.eosdis.nasa.gov/api/map_key/); the layer is empty without it.
|
(https://firms.modaps.eosdis.nasa.gov/api/map_key/); the layer is empty without it.
|
||||||
The former bundled 2026-05-25 snapshot was removed 2026-07-16.
|
The former bundled 2026-05-25 snapshot was removed 2026-07-16. Suomi NPP (`VIIRS_SNPP_NRT`)
|
||||||
|
is no longer requested: NASA stopped delivering its data on 2026-11-02.
|
||||||
|
|
||||||
### Natural Earth physical regions (`natural_earth/`)
|
### Natural Earth physical regions (`natural_earth/`)
|
||||||
|
|
||||||
|
|||||||
@@ -2325,6 +2325,11 @@ accessible name remains "Don't show this again". Control behavior is unchanged.
|
|||||||
|
|
||||||
## FIRMS source status
|
## FIRMS source status
|
||||||
|
|
||||||
|
The FIRMS proxy requests VIIRS NOAA-20, VIIRS NOAA-21 and MODIS NRT (Terra and
|
||||||
|
Aqua). It no longer requests VIIRS Suomi NPP: NASA stopped delivering Suomi NPP
|
||||||
|
data on 2026-11-02, and the source returned no rows while still costing one
|
||||||
|
MAP_KEY transaction per refresh.
|
||||||
|
|
||||||
The FIRMS CSV parser rejects impossible acquisition calendar dates before the proxy filters records to the trailing 24 hours, while retaining valid UTC acquisition times.
|
The FIRMS CSV parser rejects impossible acquisition calendar dates before the proxy filters records to the trailing 24 hours, while retaining valid UTC acquisition times.
|
||||||
The `get_active_fires` tool reports acquisition times as UTC ISO timestamps,
|
The `get_active_fires` tool reports acquisition times as UTC ISO timestamps,
|
||||||
including zero-padded hours and minutes; missing or invalid dates and times
|
including zero-padded hours and minutes; missing or invalid dates and times
|
||||||
|
|||||||
@@ -131,8 +131,8 @@ export async function fetchFirms(
|
|||||||
* NASA FIRMS live active-fire proxy with a memory + disk cache.
|
* NASA FIRMS live active-fire proxy with a memory + disk cache.
|
||||||
* Upstream: https://firms.modaps.eosdis.nasa.gov/api/area/csv/{KEY}/{SOURCE}/world/2
|
* Upstream: https://firms.modaps.eosdis.nasa.gov/api/area/csv/{KEY}/{SOURCE}/world/2
|
||||||
*
|
*
|
||||||
* Merges three VIIRS NRT sources (NOAA-20, NOAA-21, Suomi-NPP — independent
|
* Merges two VIIRS NRT sources (NOAA-20, NOAA-21 — independent satellites, no
|
||||||
* satellites, no cross-source dedup) plus MODIS_NRT (combined Terra+Aqua,
|
* cross-source dedup) plus MODIS_NRT (combined Terra+Aqua,
|
||||||
* ~1 km, numeric 0-100 confidence), fetched sequentially (quota courtesy)
|
* ~1 km, numeric 0-100 confidence), fetched sequentially (quota courtesy)
|
||||||
* with `days=2` (`days=1` means "current UTC day", nearly empty just after
|
* with `days=2` (`days=1` means "current UTC day", nearly empty just after
|
||||||
* 00:00Z) and clamps to the trailing 24 h via src/data/firmsCsv.js. FIRMS
|
* 00:00Z) and clamps to the trailing 24 h via src/data/firmsCsv.js. FIRMS
|
||||||
@@ -141,6 +141,9 @@ export async function fetchFirms(
|
|||||||
* fresh-enough disk cache (.gev-cache/firms.json) prevents ANY upstream
|
* fresh-enough disk cache (.gev-cache/firms.json) prevents ANY upstream
|
||||||
* fetch across dev-server restarts. Pattern mirrors celestrakProxy.
|
* fetch across dev-server restarts. Pattern mirrors celestrakProxy.
|
||||||
*
|
*
|
||||||
|
* VIIRS_SNPP_NRT is not requested: NASA stopped delivering Suomi NPP data on
|
||||||
|
* 2026-11-02, so that source only spent a transaction per refresh (#849).
|
||||||
|
*
|
||||||
* Routes:
|
* Routes:
|
||||||
* GET /api/firms → {fetchedAt, stale, ttlMs, sources, count, fires}
|
* GET /api/firms → {fetchedAt, stale, ttlMs, sources, count, fires}
|
||||||
* GET /api/firms/status → {hasKey, lastFetch, count, stale, ttlMs, transactions}
|
* GET /api/firms/status → {hasKey, lastFetch, count, stale, ttlMs, transactions}
|
||||||
@@ -153,12 +156,7 @@ export async function fetchFirms(
|
|||||||
export function firmsProxy() {
|
export function firmsProxy() {
|
||||||
const TTL_MS = 30 * 60_000;
|
const TTL_MS = 30 * 60_000;
|
||||||
const STATUS_TTL_MS = 5 * 60_000;
|
const STATUS_TTL_MS = 5 * 60_000;
|
||||||
const SOURCES = [
|
const SOURCES = ['VIIRS_NOAA20_NRT', 'VIIRS_NOAA21_NRT', 'MODIS_NRT'];
|
||||||
'VIIRS_NOAA20_NRT',
|
|
||||||
'VIIRS_NOAA21_NRT',
|
|
||||||
'VIIRS_SNPP_NRT',
|
|
||||||
'MODIS_NRT',
|
|
||||||
];
|
|
||||||
/**
|
/**
|
||||||
* Per-source body cap. A VIIRS world/2 pull is ~130k rows x ~100 B, about
|
* Per-source body cap. A VIIRS world/2 pull is ~130k rows x ~100 B, about
|
||||||
* 13 MB per satellite in heavy fire season (MODIS runs smaller), so this is a
|
* 13 MB per satellite in heavy fire season (MODIS runs smaller), so this is a
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ assert.notEqual(start, -1, 'FIRMS refresh function exists');
|
|||||||
const end = config.indexOf('\n }', start);
|
const end = config.indexOf('\n }', start);
|
||||||
assert.notEqual(end, -1, 'FIRMS refresh function closes');
|
assert.notEqual(end, -1, 'FIRMS refresh function closes');
|
||||||
const refreshSource = config.slice(start, end + 4);
|
const refreshSource = config.slice(start, end + 4);
|
||||||
const SOURCES = ['VIIRS_NOAA20_NRT', 'VIIRS_NOAA21_NRT', 'VIIRS_SNPP_NRT', 'MODIS_NRT'];
|
const SOURCES = ['VIIRS_NOAA20_NRT', 'VIIRS_NOAA21_NRT', 'MODIS_NRT'];
|
||||||
const NOW = Date.UTC(2026, 8, 11, 12);
|
const NOW = Date.UTC(2026, 8, 11, 12);
|
||||||
const recent = { acqDate: '2026-09-11', acqTime: '1100' };
|
const recent = { acqDate: '2026-09-11', acqTime: '1100' };
|
||||||
|
|
||||||
@@ -63,7 +63,7 @@ test('FIRMS keeps successful sources when another upstream fails', async () => {
|
|||||||
if (source === SOURCES[1]) throw new Error('upstream unavailable');
|
if (source === SOURCES[1]) throw new Error('upstream unavailable');
|
||||||
return [recent];
|
return [recent];
|
||||||
})('fixture');
|
})('fixture');
|
||||||
assert.equal(result.fires.length, 3);
|
assert.equal(result.fires.length, SOURCES.length - 1);
|
||||||
assert.deepEqual(result.sources, SOURCES.map((source, index) => ({
|
assert.deepEqual(result.sources, SOURCES.map((source, index) => ({
|
||||||
source, count: index === 1 ? 0 : 1, ok: index !== 1,
|
source, count: index === 1 ? 0 : 1, ok: index !== 1,
|
||||||
})));
|
})));
|
||||||
@@ -77,7 +77,7 @@ test('FIRMS reports one failure if consuming a source throws before append', asy
|
|||||||
// Fault injection for aggregation; ordinary parsed CSV returns an array.
|
// Fault injection for aggregation; ordinary parsed CSV returns an array.
|
||||||
return { length: 1, [Symbol.iterator]() { throw new Error('aggregation failed'); } };
|
return { length: 1, [Symbol.iterator]() { throw new Error('aggregation failed'); } };
|
||||||
})('fixture');
|
})('fixture');
|
||||||
assert.equal(result.fires.length, 3);
|
assert.equal(result.fires.length, SOURCES.length - 1);
|
||||||
assert.deepEqual(result.sources, SOURCES.map((source, index) => ({
|
assert.deepEqual(result.sources, SOURCES.map((source, index) => ({
|
||||||
source, count: index === 0 ? 0 : 1, ok: index !== 0,
|
source, count: index === 0 ? 0 : 1, ok: index !== 0,
|
||||||
})));
|
})));
|
||||||
@@ -107,7 +107,7 @@ function wrappedAddressSelectionFailure() {
|
|||||||
{ name: 'ConnectTimeoutError', code: 'UND_ERR_CONNECT_TIMEOUT', cause: inner });
|
{ name: 'ConnectTimeoutError', code: 'UND_ERR_CONNECT_TIMEOUT', cause: inner });
|
||||||
return new TypeError('fetch failed', { cause });
|
return new TypeError('fetch failed', { cause });
|
||||||
}
|
}
|
||||||
const URL_WITH_KEY = 'https://firms.modaps.eosdis.nasa.gov/api/area/csv/SECRET/VIIRS_SNPP_NRT/world/2';
|
const URL_WITH_KEY = 'https://firms.modaps.eosdis.nasa.gov/api/area/csv/SECRET/VIIRS_NOAA20_NRT/world/2';
|
||||||
|
|
||||||
test('FIRMS failures log the socket cause, never the keyed URL', () => {
|
test('FIRMS failures log the socket cause, never the keyed URL', () => {
|
||||||
const described = describeFetchError(addressSelectionFailure());
|
const described = describeFetchError(addressSelectionFailure());
|
||||||
@@ -307,3 +307,13 @@ test('every FIRMS upstream request goes through the IPv4 fallback', async () =>
|
|||||||
assert.ok(calls[0][2] instanceof AbortSignal);
|
assert.ok(calls[0][2] instanceof AbortSignal);
|
||||||
assert.deepEqual(calls[1], ['read', response, calls[0][2]], 'the body read shares the request deadline');
|
assert.deepEqual(calls[1], ['read', response, calls[0][2]], 'the body read shares the request deadline');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('FIRMS requests only the satellites NASA still delivers', () => {
|
||||||
|
const listed = config.match(/const SOURCES = \[([^\]]*)\];/);
|
||||||
|
assert.ok(listed, 'FIRMS source list exists');
|
||||||
|
const production = [...listed[1].matchAll(/'([A-Z0-9_]+)'/g)].map((m) => m[1]);
|
||||||
|
assert.deepEqual(production, SOURCES, 'the test mirrors the production source list');
|
||||||
|
// NASA stopped delivering Suomi NPP data on 2026-11-02 (#849): requesting it
|
||||||
|
// returns no rows and still spends a MAP_KEY transaction on every refresh.
|
||||||
|
assert.ok(!production.includes('VIIRS_SNPP_NRT'));
|
||||||
|
});
|
||||||
@@ -313,9 +313,9 @@ test('FIRMS retains a large successful source during partial failure and filters
|
|||||||
const first = json(await request());
|
const first = json(await request());
|
||||||
assert.equal(first.count, 130001);
|
assert.equal(first.count, 130001);
|
||||||
assert.equal(first.sources.filter((s) => s.ok).length, 1);
|
assert.equal(first.sources.filter((s) => s.ok).length, 1);
|
||||||
assert.equal(calls, 4);
|
assert.equal(calls, first.sources.length);
|
||||||
assert.equal(json(await request()).count, 130001);
|
assert.equal(json(await request()).count, 130001);
|
||||||
assert.equal(calls, 4);
|
assert.equal(calls, first.sources.length);
|
||||||
assert.deepEqual(json(await request('/status')).transactions, {
|
assert.deepEqual(json(await request('/status')).transactions, {
|
||||||
used: 3,
|
used: 3,
|
||||||
limit: 5000,
|
limit: 5000,
|
||||||
|
|||||||
Reference in new issue
Block a user