mirror of
https://github.com/bilawalsidhu/gods-eye-view.git
synced 2026-11-04 02:14:01 +00:00
Merge branch 'main' into feat/aqhi
# Conflicts: # CHANGELOG.md
This commit is contained in:
Sameh Khamis committed
commit
ac13acfe3d
27 files changed
+1100
-87
No files matched your search
@@ -1,5 +1,36 @@
|
||||
# Changelog
|
||||
|
||||
- Stop one retired bikeshare feed from reporting the whole layer as failed.
|
||||
City feeds are fetched independently, but failure was recorded in a single
|
||||
flag that only a successful *activation* cleared — and an already-loaded city
|
||||
never activates again. A dead feed re-entering camera range therefore latched
|
||||
LOAD FAILED permanently over a map still drawing every other city's stations.
|
||||
Failure is now tracked per city: a city that fails beside others still
|
||||
drawing marks the layer DEGRADED, polls that all fail over previously drawn
|
||||
points read STALE without advancing freshness, and LOAD FAILED is reported
|
||||
only when nothing is left to draw. A city that recovers clears its own
|
||||
failure, a failure is dropped once its city leaves range or the camera rises
|
||||
above the activation altitude, and late results from superseded requests are
|
||||
ignored.
|
||||
|
||||
Repointed the three feeds whose operators had moved: Reddy Bikeshare (Buffalo)
|
||||
to Social Bicycles, Tugo (Tucson) and Trinity Metro (Fort Worth) to PBSC.
|
||||
Dropped CoGo (Columbus) and Broward B-cycle (Fort Lauderdale), whose systems
|
||||
have wound down — CoGo's feed reports no stations and last moved a bike in
|
||||
March 2025. The three repointed feeds were verified live.
|
||||
|
||||
- Add Bike Share Toronto (1,069 stations) to the bikeshare registry. The system
|
||||
publishes the same PBSC GBFS v2 feeds as the Austin, Honolulu and Chattanooga
|
||||
entries, so it needs no new plumbing; a registry test now holds every city's
|
||||
feed URLs against the proxy host and path allowlists, so a city the proxy
|
||||
would refuse fails the suite instead of rendering an empty layer.
|
||||
|
||||
- GBFS v3 station names use English translations when available, then the first
|
||||
usable translation or short name. Empty or malformed names no longer appear as
|
||||
`[object Object]` or interrupt station loading; legacy string names remain
|
||||
supported in the globe and bike-share tools. Thanks to
|
||||
[Kristóf Kékesi](https://github.com/KristofKekesi) (#61).
|
||||
|
||||
- Overpass and military-installation disk caches no longer serve answers from a
|
||||
previously configured `OVERPASS_UPSTREAMS` instance after it is repointed.
|
||||
|
||||
@@ -160,6 +191,17 @@
|
||||
provenance cannot describe an aged feed as live. Voice can switch it on and
|
||||
query readings by value, band, station and observation time.
|
||||
|
||||
- Bound six upstream body reads (FIRMS CSV, CelesTrak TLE groups, adsb.lol
|
||||
`/v2/mil`, TomTom flow tiles, OpenSky `states/all` and adsbdb lookups) with
|
||||
streaming size caps that cancel the body at the cap or the route's deadline;
|
||||
the adsb.lol military poll gains a deadline of its own. An oversized OpenSky
|
||||
snapshot is not retried, and an OpenSky 429 starts its cooldown before the
|
||||
body is read.
|
||||
|
||||
- Keep later satellites when a TLE catalog entry is missing an orbital line.
|
||||
The parser skips the incomplete set and resumes at the next complete named
|
||||
set, instead of losing the rest of the group (#906).
|
||||
|
||||
- Satellite clouds keep Clouds only / Full as quick endpoints and add a continuous
|
||||
cloud-filter slider: 0% is Full, 100% is the existing brightness key, and
|
||||
intermediate values blend the keyed alpha. The slider commits on release to
|
||||
|
||||
+2
-1
@@ -61,7 +61,8 @@ How to read this:
|
||||
| **Open Calgary** (`data.calgary.ca`) | CCTV camera catalog + frames, Calgary | [Open Government Licence – City of Calgary](https://data.calgary.ca/stories/s/Open-Calgary-Terms-of-Use/u45n-7awa) — attribution REQUIRED | "Contains information licensed under the Open Government Licence – City of Calgary" |
|
||||
| **Statens vegvesen** (catalog `ogckart-sn1.atlas.vegvesen.no`, frames `kamera.atlas.vegvesen.no`, HLS `kamera.vegvesen.no`) | CCTV camera catalog + frames + live HLS video, Norway road cameras | [NLOD 2.0](https://data.norge.no/nlod/en/2.0) — attribution REQUIRED | "Contains data under the Norwegian licence for Open Government data (NLOD) distributed by Statens vegvesen" |
|
||||
| **NYC DOT Traffic Management Center** (`webcams.nyctmc.org`) | CCTV camera catalog + frames, New York City five boroughs | NYC DOT requires developers to sign a [data-sharing agreement](https://www.nyc.gov/html/dot/html/about/datafeeds.shtml) for the camera feed. **Off by default**; enable only under your own agreement | "NYC DOT Traffic Management Center" + NYC DOT's required application disclaimer (in the Data attribution popover) |
|
||||
| **GBFS (Lyft / BCycle)** | Bikeshare availability | Per-feed (attribution-only) | Credit the operator (e.g. Austin BCycle) + its `license_url` |
|
||||
| **GBFS (Lyft / BCycle / Social Bicycles / PBSC)** | Bikeshare availability | Per-feed (attribution-only) | Credit the operator (e.g. Austin BCycle) + its `license_url` |
|
||||
| **Bike Share Toronto** (`toronto.publicbikesystem.net`) | Bikeshare station availability, Toronto | Operator GBFS feed. No licence is published: the City's open-data package declares none and the feed has no `license_url`. Attribution is given to the operator. | "Bike Share Toronto (Toronto Parking Authority)" |
|
||||
| **Oslo / Bergen / Trondheim Bysykkel** (Urban Sharing, `gbfs.urbansharing.com`) | Bikeshare availability, Oslo, Bergen and Trondheim city bikes. Keyless GBFS; the proxy sends the `Client-Identifier` header the operators ask for | [NLOD 2.0](https://data.norge.no/nlod/en/2.0) — attribution REQUIRED ([Oslo](https://oslobysykkel.no/en/open-data/realtime), [Bergen](https://bergenbysykkel.no/en/apne-data/sanntid), [Trondheim](https://trondheimbysykkel.no/en/open-data/realtime)) | "Contains data under the Norwegian licence for Open Government data (NLOD) made available by Oslo Bysykkel, Bergen Bysykkel and Trondheim Bysykkel" |
|
||||
| **Kolumbus / Farte Bysykkel** (Entur Mobility, `api.entur.io/mobility/v2/gbfs`) | Bikeshare availability, Kolumbus Bysykkel (Stavanger/Nord-Jæren) and Farte Bysykkel (Skien/Porsgrunn). Keyless GBFS; the proxy sends the `ET-Client-Name` header Entur requires and is limited to Entur's GBFS paths | [NLOD](https://data.norge.no/nlod/en/2.0) per the [Entur Mobility API docs](https://developer.entur.org/pages-mobility-docs-mobility-v2) — attribution REQUIRED | "Contains data under the Norwegian licence for Open Government data (NLOD) made available by Entur (Kolumbus Bysykkel, Farte Bysykkel)" |
|
||||
| **JCDecaux Cyclocity** (`api.cyclocity.fr`) | Bikeshare availability, Bysykkel Lillestrøm. Keyless GBFS; the proxy is limited to JCDecaux contract paths | [Licence Ouverte / Etalab](https://developer.jcdecaux.com/files/Open-Licence-fr.pdf) (the feed's `license_url`) — attribution REQUIRED | "Bysykkel Lillestrøm: JCDecaux, Licence Ouverte" |
|
||||
|
||||
+23
-1
@@ -1192,6 +1192,28 @@ Sharing asks of clients of the Oslo, Bergen and Trondheim Bysykkel feeds
|
||||
their GBFS tree: `api.entur.io` to `/mobility/v2/gbfs/` and `api.cyclocity.fr`
|
||||
(Bysykkel Lillestrøm) to `/contracts/`.
|
||||
|
||||
## Upstream body bounds
|
||||
|
||||
FIRMS CSV, CelesTrak TLE groups, adsb.lol military responses, TomTom flow
|
||||
tiles, OpenSky states/all and adsbdb lookups now read their bodies through the
|
||||
shared streaming readers in `src/sources/httpBody.js`, so a declared or observed body
|
||||
over the cap is refused with `RESPONSE_TOO_LARGE` and cancelled rather than
|
||||
buffered. TomTom flow tiles use `readResponseBytesCapped` alongside
|
||||
GTFS-Realtime. Each read takes the route's abort signal, so a body still
|
||||
streaming when the deadline fires is cancelled. Caps are bounds, not budgets, sized well
|
||||
above measured payloads: FIRMS 64 MiB per satellite source, CelesTrak 16 MiB
|
||||
per group, OpenSky `states/all` 8 MiB, adsb.lol `/v2/mil` 8 MiB, TomTom tiles
|
||||
8 MiB and adsbdb lookups 1 MiB. Where a route already had a deadline (FIRMS
|
||||
60 s, CelesTrak 20 s, TomTom 15 s, adsbdb 8 s, and each 10 s OpenSky
|
||||
`states/all` attempt) it already bounded the body read, because aborting a
|
||||
fetch signal also aborts its body; the capped reader takes the same signal.
|
||||
The adsb.lol military poll, which had no deadline, now waits at most 12
|
||||
seconds for headers and body together. An oversized OpenSky snapshot is not
|
||||
retried, and an OpenSky 429 starts its cooldown from the response headers and
|
||||
cancels the body unread. A rejected or timed-out body is an upstream miss and
|
||||
follows each route's existing error and cached-data paths. FIRMS map-key
|
||||
status, the OpenSky token exchange and CCTV media keep their current handling.
|
||||
|
||||
## Remaining local service modules
|
||||
|
||||
Overpass query validation, geometry simplification, disk caching and upstream
|
||||
@@ -3173,7 +3195,7 @@ its criteria cannot be silently ignored.
|
||||
| CCTV | Austin + Caltrans (CA) + TfL London + Ontario 511 + Fintraffic (FI) + DriveBC (BC) + TxDOT (TX) + Estonia (Tallinn, Tarktee) + Live Traffic NSW + Open Calgary Open Data + Statens vegvesen (NO, stills + live HLS) + NYC DOT TMC (opt-in) + Street View fallback | `src/data/cctv.js` | `/api/cctv` | 10s (active) |
|
||||
| Radio | Radio Browser (public-domain station directory) | `src/data/radio.js` | `/api/radio/stations`, `/api/radio/click/:uuid` | 45 min directory refresh |
|
||||
| Transit 🚌 | Operator GTFS-Realtime VehiclePositions (7 keyless regions, `src/data/transitFeeds.js`) | `src/layers/transit/` via `src/app/layers/transit.js` | `/api/transit` | 15s (poll + delayed playback) |
|
||||
| Bikeshare 🚲 | GBFS (Lyft + BCycle + Urban Sharing, Entur, JCDecaux NO) | `src/data/bikeshare.js` | `/api/gbfs` | 60s |
|
||||
| Bikeshare 🚲 | GBFS (Lyft + BCycle + Social Bicycles + PBSC incl. Bike Share Toronto + Urban Sharing, Entur, JCDecaux NO) | `src/data/bikeshare.js` | `/api/gbfs` | 60s |
|
||||
| Directions 🧭 | OSRM on FOSSGIS servers (OpenStreetMap) | `src/data/directions.js` | `/api/route` (`steps=1`) | on placement / mode change |
|
||||
| Datacenters ▣ | OSM extract (bundled) | `src/data/localLayers.js` | — | static |
|
||||
| Dams ▰ | OpenInfraMap/OSM extract (bundled) | `src/data/localLayers.js` | — | static |
|
||||
|
||||
@@ -740,6 +740,7 @@
|
||||
"src/search/nominatim.js",
|
||||
"src/search/nominatim.test.mjs",
|
||||
"src/sources/httpBody.js",
|
||||
"src/sources/httpBody.test.mjs",
|
||||
"src/sources/overpass.js",
|
||||
"src/sources/nominatim.js",
|
||||
"server/providers/transit.js",
|
||||
|
||||
@@ -123,7 +123,12 @@
|
||||
"traffic-provider": {
|
||||
"runtime": "node",
|
||||
"exports": ["./server/providers/traffic"],
|
||||
"modules": ["server/providers/traffic.js", "src/data/tomtomTiles.js"],
|
||||
"modules": [
|
||||
"server/providers/traffic.js",
|
||||
"server/providers/common/http.js",
|
||||
"src/sources/httpBody.js",
|
||||
"src/data/tomtomTiles.js"
|
||||
],
|
||||
"external": []
|
||||
},
|
||||
"traffic-source": {
|
||||
@@ -139,7 +144,12 @@
|
||||
"firms-provider": {
|
||||
"runtime": "node",
|
||||
"exports": ["./server/providers/firms"],
|
||||
"modules": ["server/providers/firms.js", "src/data/firmsCsv.js"],
|
||||
"modules": [
|
||||
"server/providers/firms.js",
|
||||
"server/providers/common/http.js",
|
||||
"src/sources/httpBody.js",
|
||||
"src/data/firmsCsv.js"
|
||||
],
|
||||
"external": []
|
||||
},
|
||||
"gbfs-provider": {
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { readResponseTextCapped } from '../common/http.js';
|
||||
|
||||
/**
|
||||
* Vite plugin: adsb.lol military aircraft proxy with 12 s response cache.
|
||||
*
|
||||
@@ -25,6 +27,10 @@ export function adsbLolProxy() {
|
||||
let _cooldownStatus = 0;
|
||||
/** Response cache TTL (ms). */
|
||||
const CACHE_MS = 12000;
|
||||
/** Upstream deadline (ms); the fetch previously had none and could hang a poll. */
|
||||
const UPSTREAM_TIMEOUT_MS = 12000;
|
||||
/** /v2/mil measures ~67 KB; a relayed error page is bounded by the same cap. */
|
||||
const ADSBLOL_MIL_MAX_RESPONSE_BYTES = 8 * 1024 * 1024;
|
||||
/** Cooldown after a 429 when upstream sends no usable Retry-After (ms). */
|
||||
const RATE_LIMIT_COOLDOWN_MS = 30000;
|
||||
/** Cooldown after a 5xx (ms). */
|
||||
@@ -98,11 +104,17 @@ export function adsbLolProxy() {
|
||||
);
|
||||
return;
|
||||
}
|
||||
const signal = AbortSignal.timeout(UPSTREAM_TIMEOUT_MS);
|
||||
const upstream = await fetch('https://api.adsb.lol/v2/mil', {
|
||||
headers: { 'User-Agent': 'gods-eye-view-adsblol-proxy/1.0' },
|
||||
signal,
|
||||
});
|
||||
if (upstream.ok) {
|
||||
const body = await upstream.text();
|
||||
const body = await readResponseTextCapped(
|
||||
upstream,
|
||||
ADSBLOL_MIL_MAX_RESPONSE_BYTES,
|
||||
signal,
|
||||
);
|
||||
_cache = body;
|
||||
_cacheAt = Date.now();
|
||||
_cooldownUntil = 0;
|
||||
@@ -129,7 +141,11 @@ export function adsbLolProxy() {
|
||||
return;
|
||||
}
|
||||
}
|
||||
const body = await upstream.text();
|
||||
const body = await readResponseTextCapped(
|
||||
upstream,
|
||||
ADSBLOL_MIL_MAX_RESPONSE_BYTES,
|
||||
signal,
|
||||
);
|
||||
serve(
|
||||
res,
|
||||
upstream.status,
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
import path from 'node:path';
|
||||
import { promises as fsp } from 'node:fs';
|
||||
import { readResponseJsonCapped } from '../common/http.js';
|
||||
|
||||
/** Route and aircraft lookups are a few hundred bytes; 1 MiB is a hard bound. */
|
||||
const ADSBDB_MAX_RESPONSE_BYTES = 1024 * 1024;
|
||||
/**
|
||||
* adsbdb.com enrichment proxy: callsign → route (airline + origin/destination
|
||||
* airports) and hex → aircraft type/registration. Free community API — cached
|
||||
@@ -80,12 +84,16 @@ export function adsbdbProxy() {
|
||||
kind === 'route'
|
||||
? `https://api.adsbdb.com/v0/callsign/${encodeURIComponent(key)}`
|
||||
: `https://api.adsbdb.com/v0/aircraft/${encodeURIComponent(key)}`;
|
||||
const res = await fetch(url, { signal: AbortSignal.timeout(8000) });
|
||||
const signal = AbortSignal.timeout(8000);
|
||||
const res = await fetch(url, { signal });
|
||||
if (res.ok) {
|
||||
const payload = await readResponseJsonCapped(
|
||||
res,
|
||||
ADSBDB_MAX_RESPONSE_BYTES,
|
||||
signal,
|
||||
);
|
||||
const data =
|
||||
kind === 'route'
|
||||
? parseRoute(await res.json())
|
||||
: parseAircraft(await res.json());
|
||||
kind === 'route' ? parseRoute(payload) : parseAircraft(payload);
|
||||
store[key] = { at: Date.now(), data }; // data may be null — negative cache
|
||||
dirty = true;
|
||||
return data;
|
||||
|
||||
@@ -2,6 +2,7 @@ import { normalizeAdsbLolPointResponse } from '../../../src/data/adsbLolFallback
|
||||
import {
|
||||
coalesceProxyRequest,
|
||||
readResponseJsonCapped,
|
||||
readResponseTextCapped,
|
||||
} from '../common/http.js';
|
||||
import { requiredFiniteQueryNumber } from '../common/query.js';
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -29,11 +30,34 @@ const OPENSKY_CACHE_MS = 9000;
|
||||
const OPENSKY_ATTEMPT_TIMEOUT_MS = 10_000;
|
||||
const OPENSKY_STATES_URL =
|
||||
'https://opensky-network.org/api/states/all?extended=1';
|
||||
/** states/all?extended=1 measures ~0.8 MB anonymously; bounded, not budgeted. */
|
||||
const OPENSKY_STATES_MAX_RESPONSE_BYTES = 8 * 1024 * 1024;
|
||||
|
||||
/**
|
||||
* Credit-governor cooldown for a 429: OpenSky's retry-after bounded to
|
||||
* 30 s … 30 min, or 2 min when the header is absent.
|
||||
* @param {Response} upstream Rate-limited response.
|
||||
* @returns {number} Cooldown (ms).
|
||||
*/
|
||||
function openSkyRateLimitCooldownMs(upstream) {
|
||||
// headers.get() gives null when absent, and Number(null) is 0, not NaN.
|
||||
const header = upstream.headers.get('x-rate-limit-retry-after-seconds');
|
||||
const retryAfterSec =
|
||||
header == null || header.trim() === '' ? NaN : Number(header);
|
||||
return Math.min(
|
||||
Math.max(
|
||||
Number.isFinite(retryAfterSec) ? retryAfterSec * 1000 : 120_000,
|
||||
30_000,
|
||||
),
|
||||
30 * 60_000,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch and read the global snapshot, giving each attempt
|
||||
* OPENSKY_ATTEMPT_TIMEOUT_MS and retrying once when an attempt times out or
|
||||
* the connection fails. A second failure throws, which the proxy answers
|
||||
* the connection fails. An oversized body is not retried: it would fail again
|
||||
* and spend more credits. A second failure throws, which the proxy answers
|
||||
* from its stale cache or the regional fallback.
|
||||
* @param {Record<string, string>} headers Request headers.
|
||||
* @returns {Promise<{upstream: Response, body: string}>}
|
||||
@@ -41,13 +65,27 @@ const OPENSKY_STATES_URL =
|
||||
async function fetchOpenSkyStates(headers) {
|
||||
for (let attempt = 1; ; attempt++) {
|
||||
try {
|
||||
const upstream = await fetch(OPENSKY_STATES_URL, {
|
||||
headers,
|
||||
signal: AbortSignal.timeout(OPENSKY_ATTEMPT_TIMEOUT_MS),
|
||||
});
|
||||
return { upstream, body: await upstream.text() };
|
||||
const signal = AbortSignal.timeout(OPENSKY_ATTEMPT_TIMEOUT_MS);
|
||||
const upstream = await fetch(OPENSKY_STATES_URL, { headers, signal });
|
||||
if (upstream.status === 429) {
|
||||
// Start the cooldown from the headers and drop the rejected body
|
||||
// unread, so a stalled or oversized 429 body cannot skip it.
|
||||
_openskyCooldownUntil =
|
||||
Date.now() + openSkyRateLimitCooldownMs(upstream);
|
||||
void upstream.body?.cancel().catch(() => {});
|
||||
return {
|
||||
upstream,
|
||||
body: JSON.stringify({ error: 'OpenSky rate limited.' }),
|
||||
};
|
||||
}
|
||||
const body = await readResponseTextCapped(
|
||||
upstream,
|
||||
OPENSKY_STATES_MAX_RESPONSE_BYTES,
|
||||
signal,
|
||||
);
|
||||
return { upstream, body };
|
||||
} catch (error) {
|
||||
if (attempt >= 2) throw error;
|
||||
if (attempt >= 2 || error?.code === 'RESPONSE_TOO_LARGE') throw error;
|
||||
console.warn(
|
||||
'[OpenSky] global snapshot attempt failed, retrying:',
|
||||
error?.message || error,
|
||||
@@ -544,19 +582,9 @@ export function openSkyProxy() {
|
||||
}
|
||||
if (upstream.status === 429) {
|
||||
reason = 'rate_limited';
|
||||
// Credit governor: honor OpenSky's retry-after (bounded 30 s … 30 min;
|
||||
// 2 min when the header is absent) — no upstream attempts until then.
|
||||
const retryAfterSec = Number(
|
||||
upstream.headers.get('x-rate-limit-retry-after-seconds'),
|
||||
);
|
||||
const cooldownMs = Math.min(
|
||||
Math.max(
|
||||
Number.isFinite(retryAfterSec) ? retryAfterSec * 1000 : 120_000,
|
||||
30_000,
|
||||
),
|
||||
30 * 60_000,
|
||||
);
|
||||
_openskyCooldownUntil = now + cooldownMs;
|
||||
// Credit governor: fetchOpenSkyStates already started the cooldown
|
||||
// from OpenSky's retry-after — no upstream attempts until then.
|
||||
const cooldownMs = openSkyRateLimitCooldownMs(upstream);
|
||||
// Serve the last-good body instead of the 429 when we have one —
|
||||
// the layer keeps rendering (STALE-cued) instead of dying.
|
||||
if (_openskyCacheBody && _openskyCacheStatus === 200) {
|
||||
|
||||
@@ -2,6 +2,7 @@ import path from 'node:path';
|
||||
import { promises as fsp } from 'node:fs';
|
||||
|
||||
import { filterTrailing24h, parseFirmsCsv } from '../../src/data/firmsCsv.js';
|
||||
import { readResponseTextCapped } from './common/http.js';
|
||||
|
||||
/**
|
||||
* NASA FIRMS live active-fire proxy with a memory + disk cache.
|
||||
@@ -35,6 +36,12 @@ export function firmsProxy() {
|
||||
'VIIRS_SNPP_NRT',
|
||||
'MODIS_NRT',
|
||||
];
|
||||
/**
|
||||
* Per-source body cap. A VIIRS world/2 pull is ~130k rows x ~100 B, about
|
||||
* 13 MB per satellite in heavy fire season (MODIS runs smaller), so this is a
|
||||
* bound, not a budget.
|
||||
*/
|
||||
const FIRMS_MAX_RESPONSE_BYTES = 64 * 1024 * 1024;
|
||||
const CACHE_DIR = path.join(process.cwd(), '.gev-cache');
|
||||
const CACHE_PATH = path.join(CACHE_DIR, 'firms.json');
|
||||
|
||||
@@ -83,9 +90,12 @@ export function firmsProxy() {
|
||||
*/
|
||||
async function fetchSource(key, source) {
|
||||
const url = `https://firms.modaps.eosdis.nasa.gov/api/area/csv/${encodeURIComponent(key)}/${source}/world/2`;
|
||||
const res = await fetch(url, { signal: AbortSignal.timeout(60_000) });
|
||||
const signal = AbortSignal.timeout(60_000);
|
||||
const res = await fetch(url, { signal });
|
||||
if (!res.ok) throw new Error(`HTTP ${res.status}`);
|
||||
const records = parseFirmsCsv(await res.text());
|
||||
const records = parseFirmsCsv(
|
||||
await readResponseTextCapped(res, FIRMS_MAX_RESPONSE_BYTES, signal),
|
||||
);
|
||||
if (records === null) throw new Error('non-CSV upstream response');
|
||||
return records;
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import path from 'node:path';
|
||||
import { promises as fsp } from 'node:fs';
|
||||
import { celestrakTleUrl } from '../../../src/data/spaceProviderRequests.js';
|
||||
import { readResponseTextCapped } from '../common/http.js';
|
||||
|
||||
/**
|
||||
* Vite plugin: CelesTrak TLE proxy.
|
||||
@@ -48,10 +49,14 @@ export function celestrakProxy() {
|
||||
}
|
||||
}
|
||||
|
||||
/** The largest group (`active`) measures ~2.7 MB; this bounds a runaway body. */
|
||||
const CELESTRAK_MAX_RESPONSE_BYTES = 16 * 1024 * 1024;
|
||||
|
||||
async function fetchUpstream(group) {
|
||||
const url = celestrakTleUrl(group);
|
||||
const signal = AbortSignal.timeout(20000);
|
||||
const res = await fetch(url.toString(), {
|
||||
signal: AbortSignal.timeout(20000),
|
||||
signal,
|
||||
// CelesTrak 403s bulk groups (e.g. `active`) unless the request carries a
|
||||
// descriptive User-Agent with a contact point.
|
||||
headers: {
|
||||
@@ -60,7 +65,11 @@ export function celestrakProxy() {
|
||||
},
|
||||
});
|
||||
if (!res.ok) throw new Error(`HTTP ${res.status}`);
|
||||
const body = await res.text();
|
||||
const body = await readResponseTextCapped(
|
||||
res,
|
||||
CELESTRAK_MAX_RESPONSE_BYTES,
|
||||
signal,
|
||||
);
|
||||
// An upstream error page parses to zero TLEs — treat as failure, keep cache.
|
||||
if (!/^1 /m.test(body)) throw new Error('no TLE lines in response');
|
||||
return { at: Date.now(), body };
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import path from 'node:path';
|
||||
import { promises as fsp } from 'node:fs';
|
||||
|
||||
import { readResponseBytesCapped } from './common/http.js';
|
||||
import {
|
||||
isValidTileCoord as isValidTomTomTile,
|
||||
utcDayKey as tomtomUtcDayKey,
|
||||
@@ -48,6 +49,8 @@ export function tomtomProxy() {
|
||||
const DEFAULT_DAILY_BUDGET = 6000;
|
||||
const MEM_MAX_ENTRIES = 256;
|
||||
const UPSTREAM_TIMEOUT_MS = 15000;
|
||||
/** Flow tiles are well under 1 MB; this bounds a runaway body, not a budget. */
|
||||
const TOMTOM_TILE_MAX_RESPONSE_BYTES = 8 * 1024 * 1024;
|
||||
|
||||
/** @type {Map<string, {at:number, buf:Buffer}>} tile key `z/x/y` -> cached tile (kept past TTL for serve-stale). */
|
||||
const mem = new Map();
|
||||
@@ -143,11 +146,15 @@ export function tomtomProxy() {
|
||||
'https://api.tomtom.com/traffic/map/4/tile/flow/relative/' +
|
||||
`${z}/${x}/${y}.pbf?key=${encodeURIComponent(process.env.TOMTOM_API_KEY)}`;
|
||||
recordUpstreamFetch(); // attempts count — upstream bills the request either way
|
||||
const res = await fetch(url, {
|
||||
signal: AbortSignal.timeout(UPSTREAM_TIMEOUT_MS),
|
||||
});
|
||||
const signal = AbortSignal.timeout(UPSTREAM_TIMEOUT_MS);
|
||||
const res = await fetch(url, { signal });
|
||||
if (!res.ok) throw new Error(`HTTP ${res.status}`);
|
||||
const buf = Buffer.from(await res.arrayBuffer());
|
||||
const bytes = await readResponseBytesCapped(
|
||||
res,
|
||||
TOMTOM_TILE_MAX_RESPONSE_BYTES,
|
||||
signal,
|
||||
);
|
||||
const buf = Buffer.from(bytes.buffer, bytes.byteOffset, bytes.byteLength);
|
||||
if (buf.length === 0) throw new Error('empty tile body');
|
||||
return buf;
|
||||
}
|
||||
|
||||
@@ -270,7 +270,9 @@ export const DATA_CREDITS = [
|
||||
},
|
||||
{
|
||||
key: 'gbfs',
|
||||
html: 'Bikeshare availability: GBFS operator feeds (e.g. Austin BCycle)',
|
||||
html:
|
||||
'Bikeshare availability: GBFS operator feeds ' +
|
||||
'(e.g. Austin BCycle; Bike Share Toronto, operated by the Toronto Parking Authority)',
|
||||
},
|
||||
{
|
||||
key: 'urbansharing-gbfs',
|
||||
|
||||
@@ -4,7 +4,7 @@ const GBFS_ALLOWED_HOSTS = new Set([
|
||||
'gbfs.bluebikes.com',
|
||||
'gbfs.bcycle.com',
|
||||
'gbfs.biketownpdx.com',
|
||||
'gbfs.cogobikeshare.com',
|
||||
'reddybikeshare.socialbicycles.com',
|
||||
'austin.publicbikesystem.net',
|
||||
'hon.publicbikesystem.net',
|
||||
'chat.publicbikesystem.net',
|
||||
|
||||
Vendored
+16
-2
@@ -56,7 +56,7 @@ export function createControls({ state: layerState, services, parts, source }) {
|
||||
|
||||
/**
|
||||
* Return current layer statistics for the UI status display.
|
||||
* @returns {{ count: number, lastUpdate: number|null, loading: boolean, loadingLabel?: string, error?: string }}
|
||||
* @returns {{ count: number, lastUpdate: number|null, loading: boolean, loadingLabel?: string, error?: string, degraded?: boolean, stale?: boolean }}
|
||||
*/
|
||||
getStats() {
|
||||
const stats = {
|
||||
@@ -70,7 +70,21 @@ export function createControls({ state: layerState, services, parts, source }) {
|
||||
? `syncing ${layerState._activeCityIds.size} city feeds...`
|
||||
: 'scanning nearby systems...';
|
||||
}
|
||||
if (layerState._error) stats.error = layerState._error;
|
||||
// A failed city is only the layer's failure when it left nothing on the
|
||||
// map. While other cities still render, the partial failure reads
|
||||
// DEGRADED; when every in-range city failed and only previously drawn
|
||||
// points remain, they read STALE.
|
||||
if (layerState._failedCityIds.size > 0) {
|
||||
if (layerState._stationRenderMap.size === 0) {
|
||||
stats.error = 'GBFS fetch error';
|
||||
} else {
|
||||
stats.degraded = true;
|
||||
const allFailed = Array.from(layerState._activeCityIds).every(
|
||||
(cityId) => layerState._failedCityIds.has(cityId),
|
||||
);
|
||||
if (allFailed) stats.stale = true;
|
||||
}
|
||||
}
|
||||
return stats;
|
||||
},
|
||||
};
|
||||
|
||||
@@ -0,0 +1,376 @@
|
||||
// BIKESHARE FAILURE REPORTING — one dead city is not a dead layer.
|
||||
//
|
||||
// City feeds are fetched independently, but the layer reported failure through
|
||||
// a single flag. A city whose operator had retired its feed set that flag on
|
||||
// every proximity check, and the only code that cleared it ran on a successful
|
||||
// *activation* — which an already-loaded city never repeats. The result was a
|
||||
// permanent LOAD FAILED over a map drawing a thousand live stations.
|
||||
//
|
||||
// These cases pin the rule that replaced it: the layer reports failure only
|
||||
// when it has nothing to show, and a city that comes back clears its own
|
||||
// failure rather than waiting for a neighbour to succeed.
|
||||
//
|
||||
// Run with: npm test
|
||||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { createState } from './state.js';
|
||||
import { createControls } from './controls.js';
|
||||
|
||||
function makeLayer() {
|
||||
const services = {
|
||||
overlays: {
|
||||
setOverlayEntries() {},
|
||||
setOverlaySourceVisible() {},
|
||||
clearOverlaySource() {},
|
||||
},
|
||||
};
|
||||
const state = createState({ services });
|
||||
const parts = { queries: { collectDetectableStations: () => [] } };
|
||||
const { methods } = createControls({ state, services, parts, source: {} });
|
||||
return { state, methods };
|
||||
}
|
||||
|
||||
/** Stand in for a rendered station without reaching into the renderer. */
|
||||
function renderStations(state, count) {
|
||||
for (let i = 0; i < count; i += 1) state._stationRenderMap.set(`s${i}`, {});
|
||||
state._count = state._stationRenderMap.size;
|
||||
}
|
||||
|
||||
test('a failed city does not fail a layer that is drawing stations', () => {
|
||||
const { state, methods } = makeLayer();
|
||||
renderStations(state, 1069);
|
||||
state._failedCityIds.add('buffalo-reddy');
|
||||
|
||||
const stats = methods.getStats();
|
||||
assert.equal(stats.count, 1069);
|
||||
assert.equal(
|
||||
stats.error,
|
||||
undefined,
|
||||
'a layer drawing 1069 stations reported LOAD FAILED',
|
||||
);
|
||||
});
|
||||
|
||||
test('the layer reports failure when a failed city left it with nothing', () => {
|
||||
const { state, methods } = makeLayer();
|
||||
state._failedCityIds.add('buffalo-reddy');
|
||||
|
||||
const stats = methods.getStats();
|
||||
assert.equal(stats.count, 0);
|
||||
assert.equal(stats.error, 'GBFS fetch error');
|
||||
});
|
||||
|
||||
test('a layer with nothing to draw and nothing failed is not an error', () => {
|
||||
const { state, methods } = makeLayer();
|
||||
|
||||
// Out of range of every city is empty, not broken.
|
||||
assert.equal(methods.getStats().error, undefined);
|
||||
});
|
||||
|
||||
test('a recovered city clears its own failure', () => {
|
||||
const { state, methods } = makeLayer();
|
||||
state._failedCityIds.add('buffalo-reddy');
|
||||
assert.equal(methods.getStats().error, 'GBFS fetch error');
|
||||
|
||||
state._failedCityIds.delete('buffalo-reddy');
|
||||
assert.equal(
|
||||
methods.getStats().error,
|
||||
undefined,
|
||||
'the failure outlived the city that caused it',
|
||||
);
|
||||
});
|
||||
|
||||
// The cases above pin how getStats() reads failure state. The cases below
|
||||
// drive the real proximity check, status poll and lifecycle against a fake
|
||||
// GBFS source, so the failure state they read is the state the layer builds.
|
||||
|
||||
import * as Cesium from 'cesium';
|
||||
import { CITY_BY_ID } from './registry.js';
|
||||
import { createIngestion } from './ingestion.js';
|
||||
import { createModel } from './model.js';
|
||||
import { createViewport } from './viewport.js';
|
||||
import { createRendering } from './rendering.js';
|
||||
import { createLifecycle } from './lifecycle.js';
|
||||
|
||||
const TORONTO = 'toronto-bike-share';
|
||||
const BUFFALO = 'buffalo-reddy';
|
||||
// Within range of both Toronto and Buffalo.
|
||||
const BETWEEN = { lat: 43.25, lon: -79.1 };
|
||||
// Within range of Toronto only.
|
||||
const TORONTO_ONLY = { lat: 43.7, lon: -79.6 };
|
||||
// Within range of Buffalo only.
|
||||
const BUFFALO_ONLY = { lat: 42.5, lon: -78.0 };
|
||||
const LOW_M = 10_000;
|
||||
|
||||
// disable() detaches a keydown listener; node has no document.
|
||||
globalThis.document ??= {
|
||||
addEventListener() {},
|
||||
removeEventListener() {},
|
||||
};
|
||||
|
||||
/** A GBFS source whose per-city answers each test can change. */
|
||||
function makeSource() {
|
||||
const cityOf = (url) => {
|
||||
for (const [id, city] of CITY_BY_ID) {
|
||||
if (url === city.stationInformationUrl) return { id, kind: 'info' };
|
||||
if (url === city.stationStatusUrl) return { id, kind: 'status' };
|
||||
}
|
||||
throw new Error(`unexpected GBFS url ${url}`);
|
||||
};
|
||||
const source = {
|
||||
dead: new Set(),
|
||||
deadStatus: new Set(),
|
||||
/** Optional per-call override: (cityId, kind) => Promise | undefined. */
|
||||
hook: null,
|
||||
async getStations(url, { signal } = {}) {
|
||||
const { id, kind } = cityOf(url);
|
||||
const hooked = source.hook?.(id, kind);
|
||||
if (hooked) return hooked;
|
||||
await new Promise((resolve) => setTimeout(resolve, 1));
|
||||
signal?.throwIfAborted();
|
||||
if (source.dead.has(id)) throw new Error('GBFS HTTP 404');
|
||||
if (kind === 'status' && source.deadStatus.has(id))
|
||||
throw new Error('GBFS HTTP 503');
|
||||
const city = CITY_BY_ID.get(id);
|
||||
if (kind === 'info') {
|
||||
return {
|
||||
data: {
|
||||
stations: [
|
||||
{
|
||||
station_id: `${id}-1`,
|
||||
name: `${id} dock`,
|
||||
lat: city.centerLat,
|
||||
lon: city.centerLon,
|
||||
capacity: 10,
|
||||
},
|
||||
],
|
||||
},
|
||||
};
|
||||
}
|
||||
return {
|
||||
data: {
|
||||
stations: [
|
||||
{
|
||||
station_id: `${id}-1`,
|
||||
num_bikes_available: 3,
|
||||
num_docks_available: 7,
|
||||
is_installed: true,
|
||||
is_renting: true,
|
||||
is_returning: true,
|
||||
},
|
||||
],
|
||||
},
|
||||
};
|
||||
},
|
||||
};
|
||||
return source;
|
||||
}
|
||||
|
||||
/** Assemble the layer from its real parts around a camera the test moves. */
|
||||
function makeLiveLayer() {
|
||||
const source = makeSource();
|
||||
const services = {
|
||||
render: { governorRequestRender() {} },
|
||||
overlays: {
|
||||
setOverlayEntries() {},
|
||||
setOverlaySourceVisible() {},
|
||||
clearOverlaySource() {},
|
||||
},
|
||||
sprites: { registerSpriteCollection() {}, restoreSpriteOrder() {} },
|
||||
picking: { registerPickOwner() {}, unregisterPickOwner() {} },
|
||||
};
|
||||
const state = createState({ services });
|
||||
const parts = {};
|
||||
const context = { state, services, parts, source };
|
||||
parts.ingestion = createIngestion(context);
|
||||
parts.model = createModel(context);
|
||||
parts.viewport = createViewport(context);
|
||||
parts.rendering = createRendering(context);
|
||||
// Selection owns a DOM click handler; nothing here selects a station.
|
||||
parts.selection = {
|
||||
_installClickHandler() {},
|
||||
_clearSelection() {},
|
||||
_onKeyDown() {},
|
||||
};
|
||||
parts.queries = { collectDetectableStations: () => [] };
|
||||
parts.lifecycle = createLifecycle(context);
|
||||
const { methods: controls } = createControls(context);
|
||||
|
||||
const camera = { ...BETWEEN, height: LOW_M };
|
||||
const viewer = {
|
||||
camera: {
|
||||
get positionCartographic() {
|
||||
return Cesium.Cartographic.fromDegrees(
|
||||
camera.lon,
|
||||
camera.lat,
|
||||
camera.height,
|
||||
);
|
||||
},
|
||||
computeViewRectangle: () => undefined,
|
||||
changed: { addEventListener() {}, removeEventListener() {} },
|
||||
},
|
||||
scene: { globe: {}, primitives: { add() {}, remove() {} } },
|
||||
};
|
||||
parts.lifecycle.methods.init(viewer);
|
||||
|
||||
return {
|
||||
state,
|
||||
source,
|
||||
stats: () => controls.getStats(),
|
||||
moveTo(place, height = LOW_M) {
|
||||
Object.assign(camera, place, { height });
|
||||
return parts.viewport.runProximityCheck();
|
||||
},
|
||||
async enable() {
|
||||
parts.lifecycle.methods.enable(viewer);
|
||||
await settle();
|
||||
},
|
||||
disable: () => parts.lifecycle.methods.disable(viewer),
|
||||
poll: () => parts.ingestion.methods.update(),
|
||||
};
|
||||
}
|
||||
|
||||
/** Let fire-and-forget activation started by enable() finish. */
|
||||
async function settle() {
|
||||
for (let i = 0; i < 5; i += 1)
|
||||
await new Promise((resolve) => setTimeout(resolve, 2));
|
||||
}
|
||||
|
||||
function deferred() {
|
||||
let resolve, reject;
|
||||
const promise = new Promise((res, rej) => {
|
||||
resolve = res;
|
||||
reject = rej;
|
||||
});
|
||||
return { promise, resolve, reject };
|
||||
}
|
||||
|
||||
test('a city that fails beside one that renders reads DEGRADED, not healthy', async () => {
|
||||
const layer = makeLiveLayer();
|
||||
layer.source.dead.add(BUFFALO);
|
||||
await layer.enable();
|
||||
|
||||
const stats = layer.stats();
|
||||
assert.equal(stats.count, 1, 'Toronto should still draw');
|
||||
assert.equal(stats.error, undefined);
|
||||
assert.equal(stats.degraded, true, 'the Buffalo failure was hidden');
|
||||
assert.equal(stats.stale, undefined);
|
||||
});
|
||||
|
||||
test('a failed city that comes back clears its own failure', async () => {
|
||||
const layer = makeLiveLayer();
|
||||
layer.source.dead.add(BUFFALO);
|
||||
await layer.enable();
|
||||
assert.equal(layer.stats().degraded, true);
|
||||
|
||||
layer.source.dead.clear();
|
||||
await layer.moveTo(BETWEEN);
|
||||
const stats = layer.stats();
|
||||
assert.equal(stats.count, 2);
|
||||
assert.equal(stats.degraded, undefined);
|
||||
assert.equal(stats.error, undefined);
|
||||
});
|
||||
|
||||
test('a status poll that recovers clears the failure the last poll set', async () => {
|
||||
const layer = makeLiveLayer();
|
||||
await layer.moveTo(TORONTO_ONLY);
|
||||
await layer.enable();
|
||||
layer.source.deadStatus.add(TORONTO);
|
||||
await layer.poll();
|
||||
assert.equal(layer.stats().degraded, true);
|
||||
|
||||
layer.source.deadStatus.clear();
|
||||
await layer.poll();
|
||||
assert.equal(layer.stats().degraded, undefined);
|
||||
assert.equal(layer.stats().stale, undefined);
|
||||
});
|
||||
|
||||
test('a failure does not outlive its city leaving range', async () => {
|
||||
const layer = makeLiveLayer();
|
||||
layer.source.dead.add(BUFFALO);
|
||||
await layer.enable();
|
||||
await layer.moveTo(BUFFALO_ONLY);
|
||||
assert.equal(layer.stats().error, 'GBFS fetch error');
|
||||
|
||||
// Low over the open Atlantic: nothing in range is empty, not broken.
|
||||
await layer.moveTo({ lat: 30, lon: -40 });
|
||||
const stats = layer.stats();
|
||||
assert.equal(stats.count, 0);
|
||||
assert.equal(stats.error, undefined, 'LOAD FAILED over empty ocean');
|
||||
assert.equal(stats.degraded, undefined);
|
||||
});
|
||||
|
||||
test('zooming out above the altitude gate drops a city failure', async () => {
|
||||
const layer = makeLiveLayer();
|
||||
layer.source.dead.add(BUFFALO);
|
||||
await layer.enable();
|
||||
await layer.moveTo(BUFFALO_ONLY);
|
||||
assert.equal(layer.stats().error, 'GBFS fetch error');
|
||||
|
||||
await layer.moveTo(BUFFALO_ONLY, 200_000);
|
||||
const stats = layer.stats();
|
||||
assert.equal(stats.count, 0);
|
||||
assert.equal(stats.error, undefined, 'LOAD FAILED above the gate');
|
||||
});
|
||||
|
||||
test('a late rejection from before a disable/re-enable is ignored', async () => {
|
||||
const layer = makeLiveLayer();
|
||||
await layer.moveTo(BUFFALO_ONLY);
|
||||
const first = deferred();
|
||||
// The first info request hangs and ignores its abort signal.
|
||||
layer.source.hook = (id, kind) => {
|
||||
if (id !== BUFFALO || kind !== 'info') return undefined;
|
||||
layer.source.hook = null;
|
||||
return first.promise;
|
||||
};
|
||||
await layer.enable();
|
||||
layer.disable();
|
||||
await layer.enable();
|
||||
assert.equal(layer.stats().count, 1, 're-enable should draw Buffalo');
|
||||
|
||||
first.reject(new Error('GBFS HTTP 500'));
|
||||
await settle();
|
||||
const stats = layer.stats();
|
||||
assert.equal(stats.count, 1, 'a stale rejection tore down live points');
|
||||
assert.equal(stats.error, undefined);
|
||||
assert.equal(stats.degraded, undefined);
|
||||
});
|
||||
|
||||
test('a late status poll rejection from before a disable/re-enable is ignored', async () => {
|
||||
const layer = makeLiveLayer();
|
||||
await layer.moveTo(TORONTO_ONLY);
|
||||
await layer.enable();
|
||||
const first = deferred();
|
||||
layer.source.hook = (id, kind) => {
|
||||
if (id !== TORONTO || kind !== 'status') return undefined;
|
||||
layer.source.hook = null;
|
||||
return first.promise;
|
||||
};
|
||||
const poll = layer.poll();
|
||||
layer.disable();
|
||||
await layer.enable();
|
||||
|
||||
first.reject(new Error('GBFS HTTP 500'));
|
||||
await poll;
|
||||
const stats = layer.stats();
|
||||
assert.equal(stats.count, 1);
|
||||
assert.equal(stats.degraded, undefined);
|
||||
assert.equal(stats.error, undefined);
|
||||
});
|
||||
|
||||
test('every status poll failing over cached points reads STALE, not fresh', async () => {
|
||||
const layer = makeLiveLayer();
|
||||
await layer.moveTo(TORONTO_ONLY);
|
||||
await layer.enable();
|
||||
const loadedAt = layer.stats().lastUpdate;
|
||||
assert.ok(loadedAt, 'activation should record freshness');
|
||||
|
||||
layer.source.deadStatus.add(TORONTO);
|
||||
await new Promise((resolve) => setTimeout(resolve, 5));
|
||||
await layer.poll();
|
||||
const stats = layer.stats();
|
||||
assert.equal(stats.count, 1, 'the cached points stay drawn');
|
||||
assert.equal(stats.error, undefined);
|
||||
assert.equal(stats.degraded, true);
|
||||
assert.equal(stats.stale, true);
|
||||
assert.equal(stats.lastUpdate, loadedAt, 'a failed poll advanced freshness');
|
||||
});
|
||||
@@ -171,30 +171,34 @@ export function createIngestion({
|
||||
|
||||
const generation = layerState._proximityGeneration;
|
||||
const cityIds = Array.from(layerState._activeCityIds);
|
||||
const isCurrent = (cityId) =>
|
||||
layerState._enabled &&
|
||||
layerState._activeCityIds.has(cityId) &&
|
||||
generation === layerState._proximityGeneration;
|
||||
let refreshed = false;
|
||||
await Promise.all(
|
||||
cityIds.map(async (cityId) => {
|
||||
try {
|
||||
const statusMap = await loadCityStationStatus(cityId, generation);
|
||||
if (
|
||||
!layerState._enabled ||
|
||||
!layerState._activeCityIds.has(cityId) ||
|
||||
generation !== layerState._proximityGeneration
|
||||
)
|
||||
return;
|
||||
if (!isCurrent(cityId)) return;
|
||||
parts.rendering.applyStatusToPoints(cityId, statusMap);
|
||||
layerState._failedCityIds.delete(cityId);
|
||||
refreshed = true;
|
||||
} catch (error) {
|
||||
if (error?.name === 'AbortError') return;
|
||||
if (error?.name === 'AbortError' || !isCurrent(cityId)) return;
|
||||
console.warn(
|
||||
`[Data:Bikeshare] ${cityId} status update error:`,
|
||||
error,
|
||||
);
|
||||
layerState._error = 'GBFS status update failed';
|
||||
layerState._failedCityIds.add(cityId);
|
||||
}
|
||||
}),
|
||||
);
|
||||
|
||||
layerState._count = layerState._stationRenderMap.size;
|
||||
if (layerState._count > 0) layerState._lastUpdate = Date.now();
|
||||
// Freshness advances only on a poll that actually refreshed a city.
|
||||
if (refreshed && layerState._count > 0)
|
||||
layerState._lastUpdate = Date.now();
|
||||
},
|
||||
};
|
||||
|
||||
|
||||
@@ -51,7 +51,7 @@ export function createLifecycle({
|
||||
layerState._lastUpdate = null;
|
||||
layerState._loading = false;
|
||||
layerState._loadingOps = 0;
|
||||
layerState._error = null;
|
||||
layerState._failedCityIds.clear();
|
||||
layerState._limitWarned = false;
|
||||
|
||||
layerState._overlayHost.setVisible(
|
||||
@@ -75,7 +75,7 @@ export function createLifecycle({
|
||||
*/
|
||||
enable(viewer) {
|
||||
layerState._enabled = true;
|
||||
layerState._error = null;
|
||||
layerState._failedCityIds.clear();
|
||||
layerState._pointCollection.show = true;
|
||||
layerState._overlayHost.setVisible(
|
||||
BIKESHARE_SELECTED_OVERLAY_SOURCE_ID,
|
||||
|
||||
@@ -234,18 +234,6 @@ const RAW_GBFS_CITY_REGISTRY = [
|
||||
'https://hon.publicbikesystem.net/customer/gbfs/v2/en/station_status.json',
|
||||
provider: 'Biki',
|
||||
},
|
||||
{
|
||||
id: 'columbus-cogo',
|
||||
city: 'Columbus, OH',
|
||||
centerLat: 39.9612,
|
||||
centerLon: -82.9988,
|
||||
loadRadiusKm: 90,
|
||||
stationInformationUrl:
|
||||
'https://gbfs.cogobikeshare.com/gbfs/2.3/en/station_information.json',
|
||||
stationStatusUrl:
|
||||
'https://gbfs.cogobikeshare.com/gbfs/2.3/en/station_status.json',
|
||||
provider: 'CoGo',
|
||||
},
|
||||
{
|
||||
id: 'chattanooga-bikechatt',
|
||||
city: 'Chattanooga, TN',
|
||||
@@ -258,6 +246,18 @@ const RAW_GBFS_CITY_REGISTRY = [
|
||||
'https://chat.publicbikesystem.net/customer/gbfs/v2/en/station_status.json',
|
||||
provider: 'Bike Chattanooga',
|
||||
},
|
||||
{
|
||||
id: 'toronto-bike-share',
|
||||
city: 'Toronto, ON',
|
||||
centerLat: 43.6532,
|
||||
centerLon: -79.3832,
|
||||
loadRadiusKm: 90,
|
||||
stationInformationUrl:
|
||||
'https://toronto.publicbikesystem.net/customer/gbfs/v2/en/station_information.json',
|
||||
stationStatusUrl:
|
||||
'https://toronto.publicbikesystem.net/customer/gbfs/v2/en/station_status.json',
|
||||
provider: 'Bike Share Toronto',
|
||||
},
|
||||
bcycleEntry({
|
||||
id: 'boulder-bcycle',
|
||||
city: 'Boulder, CO',
|
||||
@@ -325,14 +325,6 @@ const RAW_GBFS_CITY_REGISTRY = [
|
||||
systemId: 'bcycle_pacersbikeshare',
|
||||
provider: 'Pacers Bikeshare',
|
||||
}),
|
||||
bcycleEntry({
|
||||
id: 'fort-lauderdale-broward',
|
||||
city: 'Fort Lauderdale, FL',
|
||||
centerLat: 26.1224,
|
||||
centerLon: -80.1373,
|
||||
systemId: 'bcycle_broward',
|
||||
provider: 'Broward B-cycle',
|
||||
}),
|
||||
bcycleEntry({
|
||||
id: 'memphis-bcycle',
|
||||
city: 'Memphis, TN',
|
||||
@@ -347,22 +339,32 @@ const RAW_GBFS_CITY_REGISTRY = [
|
||||
centerLon: -93.625,
|
||||
systemId: 'bcycle_desmoines',
|
||||
}),
|
||||
bcycleEntry({
|
||||
{
|
||||
// Tugo moved off BCycle to PBSC; the bcycle_tugo feed is gone (404).
|
||||
id: 'tucson-tugo',
|
||||
city: 'Tucson, AZ',
|
||||
centerLat: 32.2226,
|
||||
centerLon: -110.9747,
|
||||
systemId: 'bcycle_tugo',
|
||||
loadRadiusKm: 100,
|
||||
stationInformationUrl:
|
||||
'https://tucson.publicbikesystem.net/customer/gbfs/v2/en/station_information.json',
|
||||
stationStatusUrl:
|
||||
'https://tucson.publicbikesystem.net/customer/gbfs/v2/en/station_status.json',
|
||||
provider: 'Tugo',
|
||||
}),
|
||||
bcycleEntry({
|
||||
},
|
||||
{
|
||||
// Trinity Metro moved off BCycle to PBSC; the bcycle_fortworth feed is gone.
|
||||
id: 'fort-worth-trinity',
|
||||
city: 'Fort Worth, TX',
|
||||
centerLat: 32.7555,
|
||||
centerLon: -97.3308,
|
||||
systemId: 'bcycle_fortworth',
|
||||
loadRadiusKm: 100,
|
||||
stationInformationUrl:
|
||||
'https://fortworth.publicbikesystem.net/customer/gbfs/v2/en/station_information.json',
|
||||
stationStatusUrl:
|
||||
'https://fortworth.publicbikesystem.net/customer/gbfs/v2/en/station_status.json',
|
||||
provider: 'Trinity Metro',
|
||||
}),
|
||||
},
|
||||
bcycleEntry({
|
||||
id: 'omaha-heartland',
|
||||
city: 'Omaha, NE',
|
||||
@@ -386,14 +388,20 @@ const RAW_GBFS_CITY_REGISTRY = [
|
||||
centerLon: -82.394,
|
||||
systemId: 'bcycle_greenville',
|
||||
}),
|
||||
bcycleEntry({
|
||||
{
|
||||
// Reddy left BCycle for Social Bicycles; the bcycle_reddy feed is gone
|
||||
// (404). This is the endpoint the MobilityData GBFS registry lists.
|
||||
id: 'buffalo-reddy',
|
||||
city: 'Buffalo, NY',
|
||||
centerLat: 42.8864,
|
||||
centerLon: -78.8784,
|
||||
systemId: 'bcycle_reddy',
|
||||
loadRadiusKm: 100,
|
||||
stationInformationUrl:
|
||||
'https://reddybikeshare.socialbicycles.com/opendata/station_information.json',
|
||||
stationStatusUrl:
|
||||
'https://reddybikeshare.socialbicycles.com/opendata/station_status.json',
|
||||
provider: 'Reddy Bikeshare',
|
||||
}),
|
||||
},
|
||||
bcycleEntry({
|
||||
id: 'las-vegas-rtc-bike-share',
|
||||
city: 'Las Vegas, NV',
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
// BIKESHARE REGISTRY — a city entry is only useful if the proxy will serve it.
|
||||
//
|
||||
// A registry entry and the GBFS proxy allowlist are edited in different files,
|
||||
// so a city can look correctly registered while every one of its requests is
|
||||
// rejected at the proxy. These cases hold the two together: each entry is
|
||||
// checked against the same host and path predicates the proxy enforces, so a
|
||||
// new city that the proxy would refuse fails here rather than silently
|
||||
// rendering an empty layer.
|
||||
//
|
||||
// Run with: npm test
|
||||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { GBFS_CITY_REGISTRY, CITY_BY_ID } from './registry.js';
|
||||
import { isAllowedGbfsHost, isAllowedGbfsPath } from '../../data/gbfsSource.js';
|
||||
|
||||
test('every registered city is reachable through the GBFS proxy allowlist', () => {
|
||||
for (const city of GBFS_CITY_REGISTRY) {
|
||||
for (const raw of [city.stationInformationUrl, city.stationStatusUrl]) {
|
||||
const url = new URL(raw);
|
||||
assert.equal(
|
||||
isAllowedGbfsHost(url.hostname),
|
||||
true,
|
||||
`${city.id}: host ${url.hostname} is not allowlisted`,
|
||||
);
|
||||
assert.equal(
|
||||
isAllowedGbfsPath(url.pathname, url.hostname),
|
||||
true,
|
||||
`${city.id}: path ${url.pathname} is not allowlisted`,
|
||||
);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('Bike Share Toronto is registered against the PBSC v2 feeds', () => {
|
||||
const toronto = CITY_BY_ID.get('toronto-bike-share');
|
||||
assert.ok(toronto, 'toronto-bike-share is missing from the registry');
|
||||
assert.equal(toronto.city, 'Toronto, ON');
|
||||
assert.equal(toronto.provider, 'Bike Share Toronto');
|
||||
assert.equal(
|
||||
toronto.stationInformationUrl,
|
||||
'https://toronto.publicbikesystem.net/customer/gbfs/v2/en/station_information.json',
|
||||
);
|
||||
assert.equal(
|
||||
toronto.stationStatusUrl,
|
||||
'https://toronto.publicbikesystem.net/customer/gbfs/v2/en/station_status.json',
|
||||
);
|
||||
});
|
||||
|
||||
test('the Toronto center sits inside the system service area', () => {
|
||||
const toronto = CITY_BY_ID.get('toronto-bike-share');
|
||||
// Bounding box of the 1,069 live stations, sampled from the published
|
||||
// station_information feed: 43.588..43.813 N, -79.603..-79.123 W.
|
||||
assert.ok(toronto.centerLat > 43.588 && toronto.centerLat < 43.813);
|
||||
assert.ok(toronto.centerLon > -79.603 && toronto.centerLon < -79.123);
|
||||
// The system spans roughly 37 km east-west, so the load radius has to clear
|
||||
// half that from the center or downtown pans lose the outer stations.
|
||||
assert.ok(toronto.loadRadiusKm >= 25);
|
||||
});
|
||||
@@ -99,9 +99,17 @@ export function createState({ services }) {
|
||||
|
||||
layerState._loadingOps = 0;
|
||||
|
||||
/** Most recent error message string, or null. */
|
||||
/**
|
||||
* @type {Set<string>} City ids whose most recent fetch failed.
|
||||
*
|
||||
* Per city rather than one layer-wide flag: feeds are fetched independently,
|
||||
* so a retired operator must not mark a layer that other cities are still
|
||||
* populating as failed. Entries are dropped when their city leaves range.
|
||||
* The layer reports failure only when these cost it everything it had to
|
||||
* draw, and degraded otherwise — see `getStats()` in controls.js.
|
||||
*/
|
||||
|
||||
layerState._error = null;
|
||||
layerState._failedCityIds = new Set();
|
||||
|
||||
/** Whether the MAX_TOTAL_POINTS cap warning has already been logged. */
|
||||
|
||||
|
||||
@@ -122,6 +122,9 @@ export function createViewport({ state: layerState, services, parts, source }) {
|
||||
const cityIds = Array.from(layerState._activeCityIds);
|
||||
for (const cityId of cityIds) deactivateCity(cityId);
|
||||
layerState._activeCityIds.clear();
|
||||
// A failure belongs to a city in range; none are once everything is off.
|
||||
layerState._failedCityIds.clear();
|
||||
layerState._count = 0;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -168,11 +171,19 @@ export function createViewport({ state: layerState, services, parts, source }) {
|
||||
return;
|
||||
parts.rendering.applyStatusToPoints(cityId, statusMap);
|
||||
layerState._lastUpdate = Date.now();
|
||||
layerState._error = null;
|
||||
layerState._failedCityIds.delete(cityId);
|
||||
} catch (error) {
|
||||
if (error?.name === 'AbortError') return;
|
||||
// A rejection from superseded work must not mark (or tear down) a city
|
||||
// that a newer proximity check or re-enable now owns.
|
||||
if (
|
||||
!layerState._enabled ||
|
||||
!layerState._activeCityIds.has(cityId) ||
|
||||
generation !== layerState._proximityGeneration
|
||||
)
|
||||
return;
|
||||
console.warn(`[Data:Bikeshare] ${cityId} activate error:`, error);
|
||||
layerState._error = 'GBFS fetch error';
|
||||
layerState._failedCityIds.add(cityId);
|
||||
deactivateCity(cityId);
|
||||
layerState._activeCityIds.delete(cityId);
|
||||
} finally {
|
||||
@@ -208,6 +219,10 @@ export function createViewport({ state: layerState, services, parts, source }) {
|
||||
deactivateCity(cityId);
|
||||
}
|
||||
}
|
||||
// A failure never outlives its city's range.
|
||||
for (const cityId of layerState._failedCityIds) {
|
||||
if (!nextActive.has(cityId)) layerState._failedCityIds.delete(cityId);
|
||||
}
|
||||
|
||||
layerState._activeCityIds = nextActive;
|
||||
if (layerState._activeCityIds.size === 0) {
|
||||
|
||||
@@ -3586,6 +3586,7 @@ test('actual Traffic, Bikeshare and Transit lifecycles retain sensitivity in eve
|
||||
_pointCollection: {},
|
||||
_overlayHost: { setVisible: noop },
|
||||
_cityRuntime: new Map(),
|
||||
_failedCityIds: new Set(),
|
||||
},
|
||||
services: {
|
||||
sprites: { restoreSpriteOrder: noop },
|
||||
|
||||
@@ -55,7 +55,9 @@ export async function readResponseJsonCapped(response, maxBytes, signal) {
|
||||
|
||||
/**
|
||||
* Read a fetch() Response body as bytes with the same hard cap as
|
||||
* readResponseTextCapped — for protobuf upstreams (GTFS-Realtime).
|
||||
* readResponseTextCapped — for protobuf and tile upstreams (GTFS-Realtime,
|
||||
* TomTom flow). Same early Content-Length rejection, running cap while
|
||||
* streaming and cancellation on abort, so callers can treat the two alike.
|
||||
* Throws { code:'RESPONSE_TOO_LARGE' }.
|
||||
*/
|
||||
export async function readResponseBytesCapped(response, maxBytes, signal) {
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { readResponseBytesCapped } from './httpBody.js';
|
||||
|
||||
/**
|
||||
* A Response that streams `chunk` forever until the reader cancels it. The
|
||||
* zero high-water mark keeps the stream from pulling eagerly at construction,
|
||||
* so `pulls()` counts only what a reader actually asked for.
|
||||
*/
|
||||
function endlessResponse(chunk, onCancel, headers = {}) {
|
||||
let pulls = 0;
|
||||
const stream = new ReadableStream(
|
||||
{
|
||||
pull(controller) {
|
||||
pulls += 1;
|
||||
controller.enqueue(chunk);
|
||||
},
|
||||
cancel: onCancel,
|
||||
},
|
||||
{ highWaterMark: 0 },
|
||||
);
|
||||
return { response: new Response(stream, { headers }), pulls: () => pulls };
|
||||
}
|
||||
|
||||
function chunkedResponse(chunks, headers = {}) {
|
||||
const stream = new ReadableStream(
|
||||
{
|
||||
start(controller) {
|
||||
for (const chunk of chunks) controller.enqueue(chunk);
|
||||
controller.close();
|
||||
},
|
||||
},
|
||||
{ highWaterMark: 0 },
|
||||
);
|
||||
return new Response(stream, { headers });
|
||||
}
|
||||
|
||||
const bytes = (n, fill) => new Uint8Array(n).fill(fill);
|
||||
|
||||
test('returns the exact bytes of a body that fits, untouched by text decoding', async () => {
|
||||
// 0xFF and 0xFE are invalid UTF-8 lead bytes; a text path would mangle them.
|
||||
const payload = Uint8Array.from([0xff, 0xfe, 0x00, 0x7f, 0x80, 0x01]);
|
||||
const buf = await readResponseBytesCapped(chunkedResponse([payload]), 1024);
|
||||
assert.ok(buf instanceof Uint8Array);
|
||||
assert.deepEqual([...buf], [...payload]);
|
||||
});
|
||||
|
||||
test('joins chunks in order and reports the total length', async () => {
|
||||
const buf = await readResponseBytesCapped(
|
||||
chunkedResponse([bytes(3, 1), bytes(2, 2), bytes(1, 3)]),
|
||||
1024,
|
||||
);
|
||||
assert.deepEqual([...buf], [1, 1, 1, 2, 2, 3]);
|
||||
});
|
||||
|
||||
test('refuses on a declared Content-Length without pulling the body', async () => {
|
||||
let cancelled = false;
|
||||
const { response, pulls } = endlessResponse(
|
||||
bytes(512, 0),
|
||||
() => {
|
||||
cancelled = true;
|
||||
},
|
||||
{ 'content-length': '4096' },
|
||||
);
|
||||
await assert.rejects(readResponseBytesCapped(response, 1024), {
|
||||
code: 'RESPONSE_TOO_LARGE',
|
||||
});
|
||||
assert.equal(pulls(), 0);
|
||||
assert.ok(cancelled);
|
||||
});
|
||||
|
||||
test('refuses a chunked body the moment it passes the cap and stops pulling', async () => {
|
||||
let cancelled = false;
|
||||
const { response, pulls } = endlessResponse(bytes(512, 0), () => {
|
||||
cancelled = true;
|
||||
});
|
||||
await assert.rejects(readResponseBytesCapped(response, 1024), {
|
||||
code: 'RESPONSE_TOO_LARGE',
|
||||
});
|
||||
// Three 512-byte chunks is the first total over 1024; an uncapped read of
|
||||
// this never-ending stream would not terminate at all.
|
||||
assert.ok(pulls() <= 3, `pulled ${pulls()} chunks`);
|
||||
assert.ok(cancelled);
|
||||
});
|
||||
|
||||
test('accepts a body exactly at the cap', async () => {
|
||||
const buf = await readResponseBytesCapped(
|
||||
chunkedResponse([bytes(512, 7), bytes(512, 7)]),
|
||||
1024,
|
||||
);
|
||||
assert.equal(buf.byteLength, 1024);
|
||||
});
|
||||
|
||||
test('cancels a body still streaming when the signal aborts, and never returns a partial one', async () => {
|
||||
const controller = new AbortController();
|
||||
let cancelled = false;
|
||||
let pulls = 0;
|
||||
const stream = new ReadableStream(
|
||||
{
|
||||
pull(ctrl) {
|
||||
pulls += 1;
|
||||
ctrl.enqueue(bytes(64, 0));
|
||||
if (pulls === 2) controller.abort();
|
||||
},
|
||||
cancel() {
|
||||
cancelled = true;
|
||||
},
|
||||
},
|
||||
{ highWaterMark: 0 },
|
||||
);
|
||||
await assert.rejects(
|
||||
readResponseBytesCapped(new Response(stream), 1 << 20, controller.signal),
|
||||
);
|
||||
assert.ok(cancelled);
|
||||
assert.ok(pulls <= 3, `pulled ${pulls} chunks after abort`);
|
||||
});
|
||||
|
||||
test('refuses immediately when the signal is already aborted', async () => {
|
||||
const controller = new AbortController();
|
||||
controller.abort();
|
||||
const { response, pulls } = endlessResponse(bytes(8, 0), () => {});
|
||||
await assert.rejects(
|
||||
readResponseBytesCapped(response, 1024, controller.signal),
|
||||
);
|
||||
assert.equal(pulls(), 0);
|
||||
});
|
||||
|
||||
test('falls back to arrayBuffer() for a body with no stream reader', async () => {
|
||||
const response = new Response(bytes(16, 9));
|
||||
const buf = await readResponseBytesCapped(response, 1024);
|
||||
assert.equal(buf.byteLength, 16);
|
||||
assert.equal(buf[0], 9);
|
||||
});
|
||||
@@ -53,6 +53,29 @@ function isolate(t, env = {}) {
|
||||
}
|
||||
const json = (res) => JSON.parse(res.body);
|
||||
|
||||
/** A response that streams 1 MiB chunks of `text` well past every cap, then ends. */
|
||||
function oversizedResponse(text) {
|
||||
const state = { pulled: 0, cancelled: false };
|
||||
const chunk = new TextEncoder().encode(
|
||||
text.repeat(Math.ceil((1 << 20) / text.length)),
|
||||
);
|
||||
const response = new Response(
|
||||
new ReadableStream(
|
||||
{
|
||||
pull(controller) {
|
||||
if (++state.pulled > 40) controller.close();
|
||||
else controller.enqueue(chunk);
|
||||
},
|
||||
cancel() {
|
||||
state.cancelled = true;
|
||||
},
|
||||
},
|
||||
{ highWaterMark: 0 },
|
||||
),
|
||||
);
|
||||
return { response, state };
|
||||
}
|
||||
|
||||
test('standalone composition mounts every extracted provider exactly once without acquisition', (t) => {
|
||||
t.mock.method(globalThis, 'fetch', () => {
|
||||
throw Error('construction must not fetch');
|
||||
@@ -168,6 +191,29 @@ test('traffic middleware preserves keyless mode, caching, stale budget fallback
|
||||
assert.equal(calls, 2);
|
||||
});
|
||||
|
||||
test('traffic middleware answers 502 to an oversized tile without caching it', async (t) => {
|
||||
isolate(t, { TOMTOM_API_KEY: 'fixture-key', TOMTOM_DAILY_TILE_BUDGET: '' });
|
||||
const bodies = [];
|
||||
t.mock.method(globalThis, 'fetch', async () => {
|
||||
const oversized = oversizedResponse('\x01');
|
||||
bodies.push(oversized.state);
|
||||
return oversized.response;
|
||||
});
|
||||
const request = install(tomtomProxy());
|
||||
const first = await request('/flow/8/1/1.pbf');
|
||||
assert.equal(first.status, 502);
|
||||
assert.equal(bodies[0].cancelled, true);
|
||||
assert.ok(bodies[0].pulled <= 9, 'reading stops at the 8 MiB cap');
|
||||
assert.ok(
|
||||
!fsp.writeFile.mock.calls.some(({ arguments: [file] }) =>
|
||||
String(file).endsWith('.pbf'),
|
||||
),
|
||||
'no tile reached the disk cache',
|
||||
);
|
||||
assert.equal((await request('/flow/8/1/1.pbf')).status, 502);
|
||||
assert.equal(bodies.length, 2, 'nothing was cached in memory');
|
||||
});
|
||||
|
||||
test('FIRMS retains a large successful source during partial failure and filters stale data at serve time', async (t) => {
|
||||
isolate(t, { FIRMS_MAP_KEY: '' });
|
||||
let now = Date.UTC(2026, 8, 12, 12);
|
||||
|
||||
@@ -37,6 +37,30 @@ function install(plugin, preview = false) {
|
||||
};
|
||||
}
|
||||
|
||||
/** A response that streams 1 MiB chunks of `text` well past every cap, then ends. */
|
||||
function oversizedResponse(text, init = {}) {
|
||||
const state = { pulled: 0, cancelled: false };
|
||||
const chunk = new TextEncoder().encode(
|
||||
text.repeat(Math.ceil((1 << 20) / text.length)),
|
||||
);
|
||||
const response = new Response(
|
||||
new ReadableStream(
|
||||
{
|
||||
pull(controller) {
|
||||
if (++state.pulled > 40) controller.close();
|
||||
else controller.enqueue(chunk);
|
||||
},
|
||||
cancel() {
|
||||
state.cancelled = true;
|
||||
},
|
||||
},
|
||||
{ highWaterMark: 0 },
|
||||
),
|
||||
init,
|
||||
);
|
||||
return { response, state };
|
||||
}
|
||||
|
||||
function environment(t, values) {
|
||||
for (const [key, value] of Object.entries(values)) {
|
||||
const original = process.env[key];
|
||||
@@ -193,6 +217,130 @@ test('OpenSky global snapshot attempts are time-limited, retried once, then fall
|
||||
assert.equal(stalled.response.headers['x-flight-source'], 'adsb.lol');
|
||||
});
|
||||
|
||||
const openSkyAnonEnv = {
|
||||
OPENSKY_CLIENT_ID: undefined,
|
||||
OPENSKY_CLIENT_SECRET: undefined,
|
||||
OPENSKY_AUTH_MODE: 'anon',
|
||||
OPENSKY_USERNAME: undefined,
|
||||
OPENSKY_PASSWORD: undefined,
|
||||
};
|
||||
const regionalFallback = () =>
|
||||
Response.json({
|
||||
now: Date.now() / 1000,
|
||||
ac: [{ hex: 'abc123', lat: 30, lon: -97, alt_baro: 10000 }],
|
||||
});
|
||||
|
||||
test('OpenSky 429 starts the cooldown from headers and cancels the body unread', async (t) => {
|
||||
environment(t, openSkyAnonEnv);
|
||||
t.mock.method(console, 'warn', () => {});
|
||||
t.mock.method(console, 'error', () => {});
|
||||
let states = 0;
|
||||
let limited;
|
||||
t.mock.method(globalThis, 'fetch', async (url) => {
|
||||
if (url.includes('/states/')) {
|
||||
states++;
|
||||
limited = oversizedResponse('x', {
|
||||
status: 429,
|
||||
headers: { 'x-rate-limit-retry-after-seconds': '600' },
|
||||
});
|
||||
return limited.response;
|
||||
}
|
||||
if (url.includes('/lat/')) return regionalFallback();
|
||||
throw Error(`Unexpected URL: ${url}`);
|
||||
});
|
||||
const fresh = await import(
|
||||
`../../server/providers/aircraft/opensky.js?ratelimit=${Math.random()}`
|
||||
);
|
||||
const request = install(fresh.openSkyProxy());
|
||||
const first = await request('/api/flights', '?lat=30&lon=-97');
|
||||
assert.equal(first.statusCode, 200);
|
||||
assert.equal(first.headers['x-flight-source'], 'adsb.lol');
|
||||
assert.equal(states, 1, 'a rate-limited snapshot is not retried');
|
||||
assert.equal(limited.state.cancelled, true);
|
||||
assert.ok(limited.state.pulled <= 1, 'the 429 body is not read');
|
||||
const cooling = await request('/api/flights', '?lat=30&lon=-97');
|
||||
assert.equal(cooling.statusCode, 200);
|
||||
assert.equal(
|
||||
cooling.headers['x-opensky-auth-reason'],
|
||||
'opensky_cooldown_regional_fallback',
|
||||
);
|
||||
assert.equal(states, 1, 'the cooldown holds off the next upstream attempt');
|
||||
});
|
||||
|
||||
test('OpenSky 429 without retry-after cools down for the 2 minute default', async (t) => {
|
||||
environment(t, openSkyAnonEnv);
|
||||
t.mock.method(console, 'warn', () => {});
|
||||
t.mock.method(console, 'error', () => {});
|
||||
const start = Date.now();
|
||||
let now = start;
|
||||
t.mock.method(Date, 'now', () => now);
|
||||
let states = 0;
|
||||
t.mock.method(globalThis, 'fetch', async (url) => {
|
||||
if (url.includes('/states/')) {
|
||||
states++;
|
||||
return new Response('{}', { status: 429 });
|
||||
}
|
||||
if (url.includes('/lat/')) return regionalFallback();
|
||||
throw Error(`Unexpected URL: ${url}`);
|
||||
});
|
||||
const fresh = await import(
|
||||
`../../server/providers/aircraft/opensky.js?ratelimit-default=${Math.random()}`
|
||||
);
|
||||
const request = install(fresh.openSkyProxy());
|
||||
await request('/api/flights', '?lat=30&lon=-97');
|
||||
assert.equal(states, 1);
|
||||
// Past the 30 s floor but inside the 2 minute default.
|
||||
now = start + 90_000;
|
||||
await request('/api/flights', '?lat=30&lon=-97');
|
||||
assert.equal(states, 1, 'a missing header is not read as a zero wait');
|
||||
});
|
||||
|
||||
test('OpenSky does not retry a snapshot body over the cap', async (t) => {
|
||||
environment(t, openSkyAnonEnv);
|
||||
t.mock.method(console, 'warn', () => {});
|
||||
t.mock.method(console, 'error', () => {});
|
||||
const bodies = [];
|
||||
t.mock.method(globalThis, 'fetch', async (url) => {
|
||||
if (url.includes('/states/')) {
|
||||
const oversized = oversizedResponse(' ');
|
||||
bodies.push(oversized.state);
|
||||
return oversized.response;
|
||||
}
|
||||
if (url.includes('/lat/')) return regionalFallback();
|
||||
throw Error(`Unexpected URL: ${url}`);
|
||||
});
|
||||
const fresh = await import(
|
||||
`../../server/providers/aircraft/opensky.js?oversize=${Math.random()}`
|
||||
);
|
||||
const response = await install(fresh.openSkyProxy())(
|
||||
'/api/flights',
|
||||
'?lat=30&lon=-97',
|
||||
);
|
||||
assert.equal(response.statusCode, 200);
|
||||
assert.equal(response.headers['x-flight-source'], 'adsb.lol');
|
||||
assert.equal(bodies.length, 1, 'an oversized body is not fetched twice');
|
||||
assert.equal(bodies[0].cancelled, true);
|
||||
assert.ok(bodies[0].pulled <= 9, 'reading stops at the 8 MiB cap');
|
||||
});
|
||||
|
||||
test('military aircraft route answers 502 to an oversized body without caching it', async (t) => {
|
||||
t.mock.method(console, 'error', () => {});
|
||||
const bodies = [];
|
||||
t.mock.method(globalThis, 'fetch', async () => {
|
||||
const oversized = oversizedResponse('{"ac":[]}');
|
||||
bodies.push(oversized.state);
|
||||
return oversized.response;
|
||||
});
|
||||
const request = install(providers.adsbLolProxy());
|
||||
const first = await request('/api/military');
|
||||
assert.equal(first.statusCode, 502);
|
||||
assert.equal(bodies[0].cancelled, true);
|
||||
assert.ok(bodies[0].pulled <= 9, 'reading stops at the 8 MiB cap');
|
||||
const again = await request('/api/military');
|
||||
assert.equal(again.statusCode, 502, 'nothing was cached');
|
||||
assert.equal(bodies.length, 2);
|
||||
});
|
||||
|
||||
test('military aircraft route preserves fresh cache and stale response after upstream failure', async (t) => {
|
||||
let now = Date.now();
|
||||
let calls = 0;
|
||||
|
||||
@@ -47,6 +47,29 @@ function isolateDisk(t) {
|
||||
t.mock.method(fsp, 'writeFile', async () => {});
|
||||
}
|
||||
|
||||
/** A response that streams 1 MiB chunks of `text` well past every cap, then ends. */
|
||||
function oversizedResponse(text) {
|
||||
const state = { pulled: 0, cancelled: false };
|
||||
const chunk = new TextEncoder().encode(
|
||||
text.repeat(Math.ceil((1 << 20) / text.length)),
|
||||
);
|
||||
const response = new Response(
|
||||
new ReadableStream(
|
||||
{
|
||||
pull(controller) {
|
||||
if (++state.pulled > 40) controller.close();
|
||||
else controller.enqueue(chunk);
|
||||
},
|
||||
cancel() {
|
||||
state.cancelled = true;
|
||||
},
|
||||
},
|
||||
{ highWaterMark: 0 },
|
||||
),
|
||||
);
|
||||
return { response, state };
|
||||
}
|
||||
|
||||
test('portable requests keep fixed origins, encode group data and preserve the 30-day UTC window', () => {
|
||||
const tle = celestrakTleUrl('stations&FORMAT=json');
|
||||
assert.equal(tle.origin, 'https://celestrak.org');
|
||||
@@ -120,6 +143,27 @@ test('exported CelesTrak plugin coalesces refreshes, retains stale TLEs, and rea
|
||||
assert.equal(disk.body, tle);
|
||||
});
|
||||
|
||||
test('CelesTrak answers 502 to an oversized TLE body without caching it', async (t) => {
|
||||
isolateDisk(t);
|
||||
t.mock.method(console, 'warn', () => {});
|
||||
const bodies = [];
|
||||
t.mock.method(globalThis, 'fetch', async () => {
|
||||
const oversized = oversizedResponse(
|
||||
'ISS\n1 25544U fixture\n2 25544 fixture\n',
|
||||
);
|
||||
bodies.push(oversized.state);
|
||||
return oversized.response;
|
||||
});
|
||||
const request = install(celestrakProxy());
|
||||
const first = await request('/api/celestrak', '/stations');
|
||||
assert.equal(first.status, 502);
|
||||
assert.equal(bodies[0].cancelled, true);
|
||||
assert.ok(bodies[0].pulled <= 17, 'reading stops at the 16 MiB cap');
|
||||
assert.equal(fsp.writeFile.mock.callCount(), 0);
|
||||
assert.equal((await request('/api/celestrak', '/stations')).status, 502);
|
||||
assert.equal(bodies.length, 2, 'nothing was cached in memory');
|
||||
});
|
||||
|
||||
for (const preview of [false, true])
|
||||
test(`exported launch plugin preserves optional server auth and cache in ${preview ? 'preview' : 'development'}`, async (t) => {
|
||||
isolateDisk(t);
|
||||
|
||||
Reference in new issue
Block a user