Add Data Library: live-fetched dataset, tile and pack layers

This commit is contained in:
Said Turksever committed 2026-11-06 12:12:15 +00:00
1 parent 5a1598fd9e
commit 976fa6d6ce
169 files changed
+14637 -31

No files matched your search

+1
View File
@@ -67,6 +67,7 @@ The data proxies under `server/providers/` are written so the browser cannot tur
- **Live HLS is bounded and origin-pinned.** Registered HTTP(S) HLS sources use a Node puller that rejects redirects and off-origin playlist references, bounds playlists and segment bodies while streaming, and aborts timed-out or released sessions. At most two sessions retain 12 segments/24 MiB each, plus bounded download buffers. Stores are memory-only and idle sessions expire after 15 seconds. RTMP/ffmpeg execution is not enabled. Locally configured source URLs remain an operator trust boundary.
- **Radio is not an audio relay.** `/api/radio/stations` contacts only allowlisted Radio Browser HTTPS hosts and paths, rejects redirects, rejects any hostname with a loopback/private/link-local/metadata/non-public A or AAAA result, and pins each TLS connection to a validated address. It returns normalized public HTTPS stream URLs; `/api/radio/click/:uuid` applies the same destination policy and accepts only station IDs from the current bounded catalog. The browser then connects directly to the broadcaster after an explicit playback action, so the broadcaster sees the listener's IP address. GEV never proxies, caches, records, or redistributes audio.
- **No verbatim client headers upstream.** The CCTV media route is the one route that relays a request header (`Range`, for video seeking). It is parsed and canonicalized before it is forwarded: one `bytes=` range only, with every accepted form — explicit span, open-ended and suffix — bounded to the same 64 MiB ceiling the relay applies to a declared response body. Multi-range, malformed, inverted and non-`bytes` values are dropped and the request proceeds without a `Range`, as RFC 7233 §3.1 prescribes. Bounding the request bounds what is asked for: a response that declares no length — live streamed media, or a chunked body from an upstream that ignores the `Range` — has no ceiling. An upstream request the browser has stopped waiting for is cancelled rather than left running, whether the viewer leaves before the headers arrive or during the body.
- **The Data Library proxy fetches manifest upstreams only.** `/api/library/data/<id>`, `/api/library/tile/<id>/<z>/<x>/<y>` and `/api/library/index/<id>` (a partitioned GeoParquet entry's file index, built from an S3 ListObjectsV2 listing and one ranged footer read per file, cached in memory for the entry's `cacheSeconds`) resolve the id against the validated manifests in `src/library/entries/`; the browser never supplies a URL, and every derived request must stay on the manifest's https host. Redirects are refused, dataset bodies are capped while streaming at the manifest's `maxBytes` (at most 32 MiB) and tiles at 2 MiB of `image/png`, `image/jpeg` or `image/webp`, responses are cached in memory only (at most 512 tiles), requests are rate-limited per client, and upstream error text is never returned. A keyed entry must be proxied; its key is read from this server's environment through the key registry and its routes refuse cross-site and forwarded requests like the other cost-bearing routes. Browser-direct entries are fetched by the browser with no cookies or referrer.
- **Transit fetches registered feeds only.** `/api/transit/vehicles/<id>` resolves the id against `src/data/transitFeeds.js`; the browser never supplies a URL, and a feed that is registered but disabled does not resolve at all. Redirects are followed manually and each hop is validated against the feed's own https origin before it is requested, so an off-origin or downgraded hop is refused rather than contacted. Bodies are capped at 8 MB, the protobuf is decoded server-side under entity-count and string-length ceilings, a differential feed is refused, and snapshots live 15 s in memory with no disk cache. A per-feed admission limiter and a failure cooldown ladder bound what this process can ask of any operator.
- **Local receiver feeds are operator-configured and local-only.** `/api/local-receivers/aircraft` reads only the `aircraft.json` URLs in the server's `LOCAL_RECEIVER_FEEDS`; the browser never supplies an address. Each host must pass the shared tap address rule (`src/data/tapAddress.js`: loopback, RFC1918, `localhost`, `*.local`; no link-local, IPv6 or other names), the scheme must be http(s), the path must end in `aircraft.json`, and credentials, queries and fragments are refused. Invalid entries are logged at startup and never fetched. Reads use a 2 s timeout, refuse redirects, cap bodies at 2 MB and share one read per second; responses name each feed by band only and never carry upstream error text.
- **Response-size caps and timeouts** on proxied responses.