benmichael/gods-eye-view
1
0 Fork 0
mirror of https://github.com/bilawalsidhu/gods-eye-view.git synced 2026-11-15 15:52:21 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1552 from IMBrandao/fix/firms-drop-snpp

Browse Source 
fix(firms): stop requesting the retired Suomi NPP source
This commit is contained in:
German Khokhlov authored and GitHub committed 2026-11-09 21:52:09 +04:00
2 parents 581ef7eef4 fce5ee6381
commit 10784fa9c9
7 files changed
+38 -23
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

DATA_SOURCES.md
+4 -3
View File
@@ -324,11 +324,12 @@ Bundled OSM-derived data is distributed under [ODbL 1.0](https://opendatacommons
> We acknowledge the use of data and/or imagery from NASA's Fire Information for Resource Management System (FIRMS) (https://earthdata.nasa.gov/firms), part of NASA's Earth Observing System Data and Information System (EOSDIS). > We acknowledge the use of data and/or imagery from NASA's Fire Information for Resource Management System (FIRMS) (https://earthdata.nasa.gov/firms), part of NASA's Earth Observing System Data and Information System (EOSDIS).
FIRMS active fires are **fetched live at runtime** (CC0 / U.S. public domain data): the FIRMS active fires are **fetched live at runtime** (CC0 / U.S. public domain data): the
`/api/firms` server-side proxy merges the three VIIRS NRT sources (NOAA-20, NOAA-21, `/api/firms` server-side proxy merges the two VIIRS NRT sources (NOAA-20, NOAA-21)
Suomi-NPP) and MODIS NRT (Terra + Aqua), clamped to the trailing 24 h, cached 30 min to respect the shared MAP_KEY and MODIS NRT (Terra + Aqua), clamped to the trailing 24 h, cached 30 min to respect the shared MAP_KEY
transaction quota. Requires a free `FIRMS_MAP_KEY` transaction quota. Requires a free `FIRMS_MAP_KEY`
(https://firms.modaps.eosdis.nasa.gov/api/map_key/); the layer is empty without it. (https://firms.modaps.eosdis.nasa.gov/api/map_key/); the layer is empty without it.
The former bundled 2026-05-25 snapshot was removed 2026-07-16. The former bundled 2026-05-25 snapshot was removed 2026-07-16. Suomi NPP (`VIIRS_SNPP_NRT`)
is not requested: NASA stopped delivering it on 2026-11-02.
### UCS Satellite Database (`ucs_satellites/`) ### UCS Satellite Database (`ucs_satellites/`)
changelog.d/1552-firms-drop-snpp.md
+4
View File
@@ -0,0 +1,4 @@
- Stop requesting Suomi NPP in Active Fires. NASA stopped delivering its data
on November 2, 2026, so `VIIRS_SNPP_NRT` returned no rows while still costing
a MAP_KEY transaction on every refresh; the layer now merges VIIRS NOAA-20,
VIIRS NOAA-21 and MODIS (IMBrandao, #1552).
docs/CURRENT-STATE.md
+4 -1
View File
@@ -2859,6 +2859,9 @@ reporting are unchanged.
## FIRMS source status ## FIRMS source status
The FIRMS proxy requests VIIRS NOAA-20, VIIRS NOAA-21 and MODIS NRT (Terra and
Aqua). It skips VIIRS Suomi NPP, which NASA stopped delivering on 2026-11-02.
The FIRMS CSV parser rejects impossible acquisition calendar dates before the proxy filters records to the trailing 24 hours, while retaining valid UTC acquisition times. The FIRMS CSV parser rejects impossible acquisition calendar dates before the proxy filters records to the trailing 24 hours, while retaining valid UTC acquisition times.
The `get_active_fires` tool reports acquisition times as UTC ISO timestamps, The `get_active_fires` tool reports acquisition times as UTC ISO timestamps,
including zero-padded hours and minutes; missing or invalid dates and times including zero-padded hours and minutes; missing or invalid dates and times
@@ -4680,7 +4683,7 @@ its criteria cannot be silently ignored.
| Datacenters ▣ | OSM extract (bundled) | `src/data/infrastructure.js` | — | static | | Datacenters ▣ | OSM extract (bundled) | `src/data/infrastructure.js` | — | static |
| Dams ▰ | OpenInfraMap/OSM extract (bundled) | `src/data/infrastructure.js` | — | static | | Dams ▰ | OpenInfraMap/OSM extract (bundled) | `src/data/infrastructure.js` | — | static |
| Submarine Cables ◠ | TeleGeography public map (bundled) | `src/data/telegeographySubmarineCables.js` | — | static | | Submarine Cables ◠ | TeleGeography public map (bundled) | `src/data/telegeographySubmarineCables.js` | — | static |
| FIRMS Active Fires ▲ | NASA FIRMS live (VIIRS ×3 NRT + MODIS NRT, trailing 24h) | `src/data/firmsHeatmap.js` | `/api/firms` (`FIRMS_MAP_KEY`) | 10 min (proxy TTL 30 min) | | FIRMS Active Fires ▲ | NASA FIRMS live (VIIRS ×2 NRT + MODIS NRT, trailing 24h) | `src/data/firmsHeatmap.js` | `/api/firms` (`FIRMS_MAP_KEY`) | 10 min (proxy TTL 30 min) |
| Street Level 📷 | Mapillary street-level imagery (vector tiles, Graph API, MapillaryJS) | `src/layers/streetLevel/` via `src/app/layers/streetLevel.js` | `/api/mapillary/status`, `/api/mapillary/tiles/coverage/{z}/{x}/{y}` (`MAPILLARY_CLIENT_TOKEN`) | camera-driven (320 ms debounce, ≤9 tiles, server memory cache 24 h) | | Street Level 📷 | Mapillary street-level imagery (vector tiles, Graph API, MapillaryJS) | `src/layers/streetLevel/` via `src/app/layers/streetLevel.js` | `/api/mapillary/status`, `/api/mapillary/tiles/coverage/{z}/{x}/{y}` (`MAPILLARY_CLIENT_TOKEN`) | camera-driven (320 ms debounce, ≤9 tiles, server memory cache 24 h) |
| Wind 🌬 | NOAA GFS 10 m wind (keyless, 0.25°→1° grid; animated particles) | `src/data/wind.js` | `/api/wind` | 1 h (forecast cycle) | | Wind 🌬 | NOAA GFS 10 m wind (keyless, 0.25°→1° grid; animated particles) | `src/data/wind.js` | `/api/wind` | 1 h (forecast cycle) |
| GNSS Integrity 📡 (`gnss-integrity`) | adsb.lol point snapshot, low navigation accuracy binned per 0.5° cell (GEV NIC/NACp threshold + gpsjam.org formula and bands, keyless, ODbL) | `src/layers/gnss/` via `src/app/layers/gnss.js` | `/api/gnss-integrity` | camera-driven (600 ms debounce, minimum 2.5 s between reads) + 60s poll (server cache 60 s; 30 min rolling window) | | GNSS Integrity 📡 (`gnss-integrity`) | adsb.lol point snapshot, low navigation accuracy binned per 0.5° cell (GEV NIC/NACp threshold + gpsjam.org formula and bands, keyless, ODbL) | `src/layers/gnss/` via `src/app/layers/gnss.js` | `/api/gnss-integrity` | camera-driven (600 ms debounce, minimum 2.5 s between reads) + 60s poll (server cache 60 s; 30 min rolling window) |
server/providers/firms.js
+6 -9
View File
@@ -131,9 +131,9 @@ export async function fetchFirms(
* NASA FIRMS live active-fire proxy with a memory + disk cache. * NASA FIRMS live active-fire proxy with a memory + disk cache.
* Upstream: https://firms.modaps.eosdis.nasa.gov/api/area/csv/{KEY}/{SOURCE}/world/2 * Upstream: https://firms.modaps.eosdis.nasa.gov/api/area/csv/{KEY}/{SOURCE}/world/2
* *
* Merges three VIIRS NRT sources (NOAA-20, NOAA-21, Suomi-NPP — independent * Merges two VIIRS NRT sources (NOAA-20, NOAA-21 — independent satellites, no
* satellites, no cross-source dedup) plus MODIS_NRT (combined Terra+Aqua, * cross-source dedup) plus MODIS_NRT (combined Terra+Aqua, ~1 km, numeric
* ~1 km, numeric 0-100 confidence), fetched sequentially (quota courtesy) * 0-100 confidence), fetched sequentially (quota courtesy)
* with `days=2` (`days=1` means "current UTC day", nearly empty just after * with `days=2` (`days=1` means "current UTC day", nearly empty just after
* 00:00Z) and clamps to the trailing 24 h via src/data/firmsCsv.js. FIRMS * 00:00Z) and clamps to the trailing 24 h via src/data/firmsCsv.js. FIRMS
* quota is 5,000 transactions / 10 min per MAP_KEY, so the cache is the point: * quota is 5,000 transactions / 10 min per MAP_KEY, so the cache is the point:
@@ -141,6 +141,8 @@ export async function fetchFirms(
* fresh-enough disk cache (.gev-cache/firms.json) prevents ANY upstream * fresh-enough disk cache (.gev-cache/firms.json) prevents ANY upstream
* fetch across dev-server restarts. Pattern mirrors celestrakProxy. * fetch across dev-server restarts. Pattern mirrors celestrakProxy.
* *
* VIIRS_SNPP_NRT is left out: NASA stopped delivering Suomi NPP on 2026-11-02.
*
* Routes: * Routes:
* GET /api/firms → {fetchedAt, stale, ttlMs, sources, count, fires} * GET /api/firms → {fetchedAt, stale, ttlMs, sources, count, fires}
* GET /api/firms/status → {hasKey, lastFetch, count, stale, ttlMs, transactions} * GET /api/firms/status → {hasKey, lastFetch, count, stale, ttlMs, transactions}
@@ -153,12 +155,7 @@ export async function fetchFirms(
export function firmsProxy() { export function firmsProxy() {
const TTL_MS = 30 * 60_000; const TTL_MS = 30 * 60_000;
const STATUS_TTL_MS = 5 * 60_000; const STATUS_TTL_MS = 5 * 60_000;
const SOURCES = [ const SOURCES = ['VIIRS_NOAA20_NRT', 'VIIRS_NOAA21_NRT', 'MODIS_NRT'];
'VIIRS_NOAA20_NRT',
'VIIRS_NOAA21_NRT',
'VIIRS_SNPP_NRT',
'MODIS_NRT',
];
/** /**
* Per-source body cap. A VIIRS world/2 pull is ~130k rows x ~100 B, about * Per-source body cap. A VIIRS world/2 pull is ~130k rows x ~100 B, about
* 13 MB per satellite in heavy fire season (MODIS runs smaller), so this is a * 13 MB per satellite in heavy fire season (MODIS runs smaller), so this is a
src/data/firmsInteraction.test.mjs
+4 -3
View File
@@ -120,9 +120,10 @@ test('fire detection key survives a refetch that renumbers the index', () => {
}); });
test('two satellites over the same pixel at the same time stay distinct', () => { test('two satellites over the same pixel at the same time stay distinct', () => {
// The proxy merges SNPP + NOAA-20 + NOAA-21 without dedup and their orbits // The proxy merges NOAA-20 + NOAA-21 without dedup and their orbits overlap,
// overlap, so position and time alone are NOT unique. Collapsing them let a // so position and time alone are NOT unique. Collapsing them let a restored
// restored selection land on the other satellite's record. // selection land on the other satellite's record. Archived Suomi NPP (`N`)
// rows must stay distinct too.
const base = makeFire({ satellite: 'N20' }); const base = makeFire({ satellite: 'N20' });
const twin = makeFire({ satellite: 'N21' }); const twin = makeFire({ satellite: 'N21' });
const suomi = makeFire({ satellite: 'N' }); const suomi = makeFire({ satellite: 'N' });
src/data/firmsProxy.test.mjs
+14 -5
View File
@@ -21,7 +21,10 @@ assert.notEqual(start, -1, 'FIRMS refresh function exists');
const end = config.indexOf('\n }', start); const end = config.indexOf('\n }', start);
assert.notEqual(end, -1, 'FIRMS refresh function closes'); assert.notEqual(end, -1, 'FIRMS refresh function closes');
const refreshSource = config.slice(start, end + 4); const refreshSource = config.slice(start, end + 4);
const SOURCES = ['VIIRS_NOAA20_NRT', 'VIIRS_NOAA21_NRT', 'VIIRS_SNPP_NRT', 'MODIS_NRT']; // Read the list from production so these tests follow it.
const listed = config.match(/const SOURCES = \[([^\]]*)\];/);
assert.ok(listed, 'FIRMS source list exists');
const SOURCES = [...listed[1].matchAll(/'([A-Z0-9_]+)'/g)].map((m) => m[1]);
const NOW = Date.UTC(2026, 8, 11, 12); const NOW = Date.UTC(2026, 8, 11, 12);
const recent = { acqDate: '2026-09-11', acqTime: '1100' }; const recent = { acqDate: '2026-09-11', acqTime: '1100' };
@@ -54,7 +57,7 @@ test('FIRMS retains large sources in order and filters expired rows', async () =
assert.equal(result.fires[199_999], large.at(-1)); assert.equal(result.fires[199_999], large.at(-1));
assert.equal(result.fires.at(-1), last); assert.equal(result.fires.at(-1), last);
assert.deepEqual(result.sources, SOURCES.map((source, index) => ({ assert.deepEqual(result.sources, SOURCES.map((source, index) => ({
source, count: [200_000, 1, 0, 0][index], ok: true, source, count: [200_000, 1, 0][index], ok: true,
}))); })));
}); });
@@ -63,7 +66,7 @@ test('FIRMS keeps successful sources when another upstream fails', async () => {
if (source === SOURCES[1]) throw new Error('upstream unavailable'); if (source === SOURCES[1]) throw new Error('upstream unavailable');
return [recent]; return [recent];
})('fixture'); })('fixture');
assert.equal(result.fires.length, 3); assert.equal(result.fires.length, SOURCES.length - 1);
assert.deepEqual(result.sources, SOURCES.map((source, index) => ({ assert.deepEqual(result.sources, SOURCES.map((source, index) => ({
source, count: index === 1 ? 0 : 1, ok: index !== 1, source, count: index === 1 ? 0 : 1, ok: index !== 1,
}))); })));
@@ -77,7 +80,7 @@ test('FIRMS reports one failure if consuming a source throws before append', asy
// Fault injection for aggregation; ordinary parsed CSV returns an array. // Fault injection for aggregation; ordinary parsed CSV returns an array.
return { length: 1, [Symbol.iterator]() { throw new Error('aggregation failed'); } }; return { length: 1, [Symbol.iterator]() { throw new Error('aggregation failed'); } };
})('fixture'); })('fixture');
assert.equal(result.fires.length, 3); assert.equal(result.fires.length, SOURCES.length - 1);
assert.deepEqual(result.sources, SOURCES.map((source, index) => ({ assert.deepEqual(result.sources, SOURCES.map((source, index) => ({
source, count: index === 0 ? 0 : 1, ok: index !== 0, source, count: index === 0 ? 0 : 1, ok: index !== 0,
}))); })));
@@ -107,7 +110,7 @@ function wrappedAddressSelectionFailure() {
{ name: 'ConnectTimeoutError', code: 'UND_ERR_CONNECT_TIMEOUT', cause: inner }); { name: 'ConnectTimeoutError', code: 'UND_ERR_CONNECT_TIMEOUT', cause: inner });
return new TypeError('fetch failed', { cause }); return new TypeError('fetch failed', { cause });
} }
const URL_WITH_KEY = 'https://firms.modaps.eosdis.nasa.gov/api/area/csv/SECRET/VIIRS_SNPP_NRT/world/2'; const URL_WITH_KEY = 'https://firms.modaps.eosdis.nasa.gov/api/area/csv/SECRET/VIIRS_NOAA20_NRT/world/2';
test('FIRMS failures log the socket cause, never the keyed URL', () => { test('FIRMS failures log the socket cause, never the keyed URL', () => {
const described = describeFetchError(addressSelectionFailure()); const described = describeFetchError(addressSelectionFailure());
@@ -307,3 +310,9 @@ test('every FIRMS upstream request goes through the IPv4 fallback', async () =>
assert.ok(calls[0][2] instanceof AbortSignal); assert.ok(calls[0][2] instanceof AbortSignal);
assert.deepEqual(calls[1], ['read', response, calls[0][2]], 'the body read shares the request deadline'); assert.deepEqual(calls[1], ['read', response, calls[0][2]], 'the body read shares the request deadline');
}); });
test('FIRMS requests only the satellites NASA still delivers', () => {
// NASA stopped delivering Suomi NPP on 2026-11-02; asking for it only spends
// a MAP_KEY transaction.
assert.ok(!SOURCES.includes('VIIRS_SNPP_NRT'));
});
src/tooling/environmentProviders.test.mjs
+2 -2
View File
@@ -313,9 +313,9 @@ test('FIRMS retains a large successful source during partial failure and filters
const first = json(await request()); const first = json(await request());
assert.equal(first.count, 130001); assert.equal(first.count, 130001);
assert.equal(first.sources.filter((s) => s.ok).length, 1); assert.equal(first.sources.filter((s) => s.ok).length, 1);
assert.equal(calls, 4); assert.equal(calls, 3);
assert.equal(json(await request()).count, 130001); assert.equal(json(await request()).count, 130001);
assert.equal(calls, 4); assert.equal(calls, 3);
assert.deepEqual(json(await request('/status')).transactions, { assert.deepEqual(json(await request('/status')).transactions, {
used: 3, used: 3,
limit: 5000, limit: 5000,