Add the Data Library: live regional datasets, tiles and packs

DATA LAYERS keeps its curated core layers; regional and niche sources
live in the Data Library, opened from Browse data library at the foot
of the panel. Each entry is a JSON manifest in src/library/entries/
pointing at a live upstream (query API, provider-hosted file or tile
endpoint), so nothing is bundled and a self-hosted instance hosts no
data. Sources without CORS go through one allowlisted /api/library
route. Kinds: dataset (GeoJSON, CSV, JSON records, ArcGIS, partitioned
GeoParquet), tiles (XYZ, WMS/ArcGIS bbox export, terrarium relief) and
pack. Selections persist and travel in share links as lib=, separate
from the core layer tokens. Library rows get Zoom and Remove icon
chips that work while the layer is off, rendered through RailChips.
Strings are translated in every locale.
This commit is contained in:
Said Turksever committed 2026-11-12 12:26:45 +00:00
1 parent 10a788db78
commit db80d82b8d
182 files changed
+15805 -44

No files matched your search

+1
View File
@@ -73,6 +73,7 @@ The data proxies under `server/providers/` are written so the browser cannot tur
- **Radio is not an audio relay.** `/api/radio/stations` contacts only allowlisted Radio Browser HTTPS hosts and paths, rejects redirects, rejects any hostname with a loopback/private/link-local/metadata/non-public A or AAAA result, and pins each TLS connection to a validated address. It returns normalized public HTTPS stream URLs; `/api/radio/click/:uuid` applies the same destination policy and accepts only station IDs from the current bounded catalog. The browser then connects directly to the broadcaster after an explicit playback action, so the broadcaster sees the listener's IP address. GEV never proxies, caches, records, or redistributes audio.
- **Québec 511 video plays in the browser, opt-in and after Play.** With `CCTV_QUEBEC_VIDEO_ENABLED=1`, `/api/cctv/sources` lists, for each MTQ camera, one URL built server-side from its numeric id on `https://www.quebec511.info/Carte/Fenetres/camera.ashx`; the overlay accepts no other host, path or format. The browser loads that clip only after the viewer presses Play for the selected camera, so Québec 511 sees the viewer's IP address, and the panel says so before Play. Selecting a camera by click, auto-hop, a scene or voice never loads it, and switching cameras stops playback. GEV never proxies, caches or reads the video's pixels. With the flag unset (the default) the browser makes no request to Québec 511.
- **No verbatim client headers upstream.** The CCTV media route is the one route that relays a request header (`Range`, for video seeking). It is parsed and canonicalized before it is forwarded: one `bytes=` range only, with every accepted form — explicit span, open-ended and suffix — bounded to the same 64 MiB ceiling the relay applies to a declared response body. Multi-range, malformed, inverted and non-`bytes` values are dropped and the request proceeds without a `Range`, as RFC 7233 §3.1 prescribes. Bounding the request bounds what is asked for: a response that declares no length — live streamed media, or a chunked body from an upstream that ignores the `Range` — has no ceiling. An upstream request the browser has stopped waiting for is cancelled rather than left running, whether the viewer leaves before the headers arrive or during the body.
- **The Data Library proxy fetches manifest upstreams only.** `/api/library/data/<id>`, `/api/library/tile/<id>/<z>/<x>/<y>` and `/api/library/index/<id>` (a partitioned GeoParquet entry's file index, built from an S3 ListObjectsV2 listing and one ranged footer read per file, cached in memory for the entry's `cacheSeconds`) resolve the id against the validated manifests in `src/library/entries/`; the browser never supplies a URL, and every derived request must stay on the manifest's https host. Redirects are refused, dataset bodies are capped while streaming at the manifest's `maxBytes` (at most 32 MiB) and tiles at 2 MiB of `image/png`, `image/jpeg` or `image/webp`, responses are cached in memory only (at most 512 tiles), requests are rate-limited per client, and upstream error text is never returned. A keyed entry must be proxied; its key is read from this server's environment through the key registry and its routes refuse cross-site and forwarded requests like the other cost-bearing routes. Browser-direct entries are fetched by the browser with no cookies or referrer.
- **Transit fetches registered feeds only.** `/api/transit/vehicles/<id>` resolves the id against `src/data/transitFeeds.js`; the browser never supplies a URL, and a feed that is registered but disabled does not resolve at all. Redirects are followed manually and each hop is validated against the feed's own https origin before it is requested, so an off-origin or downgraded hop is refused rather than contacted. Bodies are capped at 8 MB, the protobuf is decoded server-side under entity-count and string-length ceilings, a differential feed is refused, and snapshots live 15 s in memory with no disk cache. A per-feed admission limiter and a failure cooldown ladder bound what this process can ask of any operator.
- **Rail fetches registered feeds only, and none needs a credential.** `/api/rail/feeds` lists the offered feeds; `/api/rail/trains/<id>`, `/api/rail/updates/<id>`, `/api/rail/paths/<id>` and `/api/rail/schedule/<id>` resolve the id against `src/data/railFeeds.js` (`RAIL_FEED_ID_PATTERN`: lowercase letters, digits and hyphens). The browser never supplies a URL, and an id that is not registered, or a trip-updates route for a feed that registers none, does not resolve at all; any method but GET answers 405 (`Allow: GET`, never cached). Every registered feed and every static timetable is open: no rail request carries a credential of the deployer's; one feed's published URL includes a fixed public value its publisher prints for everyone (Bordeaux's `apiKey`), and the only rail setting the server reads is `RAIL_PATHS_DIR`. Upstream requests follow the Transit proxy's policy: https only, each redirect hop validated against the feed's own https origin before it is requested (three hops at most), a request timeout, bodies capped at 8 MiB unless the registry sets a feed's own cap (never above 64 MiB), and a per-feed admission limiter and the failure cooldown ladder bound what the process asks of any operator. Responses are decoded server-side: GTFS-Realtime under entity-count and string-length ceilings, a differential feed is refused, and an empty or header-less GTFS-Realtime reply (an error page served as 200, say) is refused as unreadable, so the last good snapshot is served stale or the request fails with 502. The Irish Rail and SIRI VM XML is read by small scanners, not an XML library: a document that declares a DTD or an entity is refused, and the whole text, the number of trains and every field are capped. Only vehicles the feed's classifier recognises as trains leave the server, and no train position inside Ukraine is ever sent (`src/data/railNoPositions.js`). Snapshots and trip updates live in memory for the feed's own freshness window, with serve-stale after a failure; there is no disk cache and no history. Path and schedule packs are optional build output, read from disk (`.cache/rail-paths/` or `RAIL_PATHS_DIR`) by a reader that re-checks the id against the same pattern, so a name never leaves that directory. A pack over 24 MiB is refused before it is read and answers 503, never a partial file, and so does a pack that cannot be read: only a missing pack is 404; the rest pass through as bytes, through a 64 MiB in-memory read cache keyed by each file's modification time and size, and are served as JSON holding timetables and track shapes only, `no-cache` with an ETag from that time and size (304 when unchanged). A schedule pack of another format version or another feed, or whose window has ended, is not used, and a feed that cannot be read without it answers 503 rather than an empty list. The pack build (`scripts/build-rail-paths.mjs`) runs only when a person runs it, never on app start: it downloads over https only (a redirect to plain http is refused), caps each download in size and time, sends no credential, and refuses an `--out` folder (or a `RAIL_PATHS_DIR` it writes to) inside `src/` or `public/`.
- **Local receiver feeds are operator-configured and local-only.** `/api/local-receivers/aircraft` reads only the `aircraft.json` URLs in the server's `LOCAL_RECEIVER_FEEDS`; the browser never supplies an address. Each host must pass the shared tap address rule (`src/data/tapAddress.js`: loopback, RFC1918, `localhost`, `*.local`; no link-local, IPv6 or other names), the scheme must be http(s), the path must end in `aircraft.json`, and credentials, queries and fragments are refused. Invalid entries are logged at startup and never fetched. Reads use a 2 s timeout, refuse redirects, cap bodies at 2 MB and share one read per second; responses name each feed by band only and never carry upstream error text.