docs(server): tighten compression comments and docs

changelog.d/795-opensky-compress-cached-body.md

@@ -1,7 +1,5 @@
-- The live flight snapshot is compressed on the wire. `/api/flights` gzips its
+- Compress the live flight snapshot on the wire. `/api/flights` gzips its
-  cached global snapshot and its regional adsb.lol fallback once per upstream
+  cached global snapshot and its regional adsb.lol fallback once per refresh
-  refresh and serves that frame to clients offering gzip — roughly 860 KB
+  and sends that frame to clients that offer gzip: about 860 KB becomes about
-  becomes roughly 258 KB — while a client that offers nothing receives the same
+  250 KB. Other clients get the same bytes uncompressed, and both answers carry
-  bytes uncompressed. Both answers carry `Vary: Accept-Encoding`, and a response
+  `Vary: Accept-Encoding` (daikaginza, #795).
-  the proxy does not cache, such as an auth failure, is never sent under a
-  cached snapshot's encoding (daikaginza, #795).

docs/CURRENT-STATE.md

@@ -6186,7 +6186,7 @@ apply to `public/` assets or built-preview serving.
 - `/api/overpass` is bounded by body/response caps, per-client/global rate limits, concurrency limits, operator-configured upstreams, in-flight dedupe, cache bounds, and static validation that every selector is spatially bounded.
 - `/api/military-installations` uses an independent limiter with the same 90-per-client/300-global one-minute bounds, so viewport installation refreshes never consume `/api/overpass` annotation capacity.
 - `/api/route` proxies bounded OSRM route requests for annotation routes, with profile allowlisting, distance caps, response caps, caching, and sanitized "no route found" errors.
-- `/api/flights` negotiates response encoding. The cached global snapshot and the regional adsb.lol fallback are each gzipped once per refresh, so a client that offers gzip receives roughly 860 KB as 250 KB and one that does not receives the same bytes uncompressed; both answers carry `Vary: Accept-Encoding`. Bodies the proxy does not cache — auth failures, cooldown notices — are never sent under a cached response's encoding.
+- `/api/flights` negotiates gzip. The cached global snapshot and the regional adsb.lol fallback are each gzipped once per refresh (about 860 KB becomes about 250 KB). Clients that offer gzip get that frame; other clients, and bodies under 1 KiB or that gzip cannot shrink, get identity bytes. Both answers carry `Vary: Accept-Encoding`. Uncached bodies, such as auth failures, are never sent under a cached frame's encoding.
 - Track endpoints: `/api/vessels/track?mmsi=` (server-accumulated ring buffers; sub-route handled before the rows snapshot), `/api/flights/track?icao24=` (OAuth, 60s cache, sanitized errors, independent OpenSky credit bucket), `/api/military/track?hex=` (400 for an invalid hex, otherwise `503 history_unavailable` with `Retry-After: 1800` and `Cache-Control: no-store`, no upstream request; military tracking uses its locally accumulated trail).
 - Realtime debug logs redact API keys, bearer tokens, client secrets, and image data URLs before writing to disk; request bodies are size-capped.
 - `/api/realtime/debug-log` enforces an always-on 120-per-client/400-global one-minute limiter (not the opt-in `GEV_RATELIMIT_OPENAI_PER_MIN` bucket the cost-bearing OpenAI routes share), appends asynchronously through a serialized queue, and rotates `realtime-conversations.jsonl` at 32 MB keeping one prior generation, so the sink is bounded at twice that regardless of session length. A malformed record answers 400 and a failed write 500, both with a fixed message.

server/providers/aircraft/opensky.js

@@ -20,9 +20,8 @@ let _openskyTokenPromise = null;
 /** @type {string|null} Cached upstream response body (JSON text). */
 let _openskyCacheBody = null;
 /**
- * @type {Buffer|null} Gzip of `_openskyCacheBody`, written in the same step as
+ * @type {Buffer|null} Gzip of `_openskyCacheBody`, written together with it.
- * the body it encodes so the two can never describe different snapshots. Null
+ * Null when the body is not worth compressing.
- * whenever the cached body is too small to be worth a frame.
  */
 let _openskyCacheGzip = null;
 /** @type {number} HTTP status of the cached response. */
@@ -409,9 +408,8 @@ async function fetchAdsbLolPointFallback(req) {
         const body = JSON.stringify(normalized);
         const record = {
           body,
-          // Compressed with the body it belongs to, so a cache entry served
+          // Compressed once per entry; ADSBLOL_POINT_CACHE_MAX bounds the
-          // many times pays for the frame once. Entries are bounded by
+          // extra memory.
-          // ADSBLOL_POINT_CACHE_MAX, so the extra residency is bounded too.
           gzip: await gzipBody(body),
           cachedAt: Date.now(),
           count: normalized.states.length,
@@ -647,9 +645,8 @@ export function openSkyProxy() {
             // Only cache successful responses — error responses (401/403/429/5xx)
             // should not be served from cache on subsequent requests
             if (upstream.ok) {
-              // The frame is built before either slot is written so the two
+              // Compress before writing either slot, so a request arriving
-              // are always published together: a request arriving mid-refresh
+              // mid-refresh still reads a body with its own frame.
-              // reads the previous body with the previous body's frame.
               const frame = await gzipBody(body);
               _openskyCacheBody = body;
               _openskyCacheGzip = frame;
@@ -763,9 +760,8 @@ export function openSkyProxy() {
           }
         }
 
-        // Only the coalesced refresh caches, so the gzip slot belongs to this
+        // The frame belongs to this body only if this body was cached. An
-        // body only when this body is the one it stored. An error response is
+        // uncached error must not borrow the previous snapshot's frame.
-        // not cached and must not borrow the previous snapshot's frame.
         const encoded = encodedBody(
           req,
           body,

server/providers/common/compress.js

@@ -4,39 +4,25 @@ import { promisify } from 'node:util';
 const gzipAsync = promisify(gzip);
 
 /**
- * Response compression for proxies that already hold a whole body in memory.
+ * Opt-in gzip for text bodies a proxy already holds in memory.
  *
- * Deliberately not a middleware. Two response paths in this tree must never be
+ * Not a middleware: `cctv/media.js` streams bodies and answers range requests,
- * wrapped: `cctv/media.js` streams a piped body and answers byte-range requests
+ * and several providers forward binary upstream types that gzip only grows.
- * with 206, and `wind/gfs.js` also serves ranges — a wrapper that buffered or
+ * `encodedBody` returns the body with its headers so the two cannot disagree.
- * re-encoded those would break playback and range semantics. Seven providers
- * additionally forward an upstream `content-type` verbatim, including already
- * compressed binary formats that gzip would only make larger.
- *
- * So compression is opt-in at the call site, for bodies the proxy holds and
- * knows to be text. The failure mode of forgetting a site is a response that is
- * merely uncompressed, never a corrupt one — but only as long as the encoded
- * body and its `Content-Encoding` are chosen together, which is why
- * `encodedBody` returns both rather than exposing them separately.
  */
 
-/** Below this, a gzip frame costs more than the bytes it saves. */
+/** Skip compression for bodies smaller than 1 KiB. */
 const MIN_COMPRESSED_BYTES = 1024;
 
 /**
- * The `qvalue` grammar of RFC 9110 section 12.4.2: zero to three decimals
+ * RFC 9110 section 12.4.2 `qvalue`. Matching the whole token keeps `q=1oops`
- * below one, or one with only zeros after it. Matching the whole token rather
+ * and `q=2` unreadable instead of rounding them into acceptance.
- * than parsing a numeric prefix is what makes `q=1oops` and the out-of-range
- * `q=2` unreadable instead of quietly rounding to something acceptable.
  */
 const QVALUE = /^(?:0(?:\.\d{0,3})?|1(?:\.0{0,3})?)$/;
 
 /**
- * Whether the client offered gzip in `Accept-Encoding`.
+ * Whether `Accept-Encoding` offers gzip. Anything unreadable (no header, a bad
- *
+ * q value, `x-gzip`) reads as no: every client can decode identity.
- * Anything this cannot read confidently — a missing header, a malformed or
- * out-of-range quality value, the `x-gzip` alias — reads as "no", because the
- * fallback is an identity response every client can decode.
  *
  * @param {import('node:http').IncomingMessage} request
  * @returns {boolean}
@@ -51,23 +37,18 @@ export function acceptsGzip(request) {
       .map((parameter) => parameter.trim().toLowerCase())
       .find((parameter) => parameter.startsWith('q='));
     if (quality === undefined) return true;
-    // `q=0` is the one way a client can name an encoding and still refuse it.
+    // `q=0` names gzip and refuses it.
     const value = quality.slice(2);
     return QVALUE.test(value) && Number(value) > 0;
   });
 }
 
 /**
- * Gzip a body once, for callers that cache it and serve it many times.
+ * Gzip a body once for a caller that caches it. Async, so the ~860 KB OpenSky
+ * snapshot does not block other requests while it compresses.
  *
- * OpenSky's snapshot is ~860 KB and is held for the cache TTL, so this is paid
+ * Returns null, meaning "serve identity", when the body is under 1 KiB or gzip
- * per refresh rather than per request. It runs off the event loop on purpose:
+ * does not shrink it.
- * on that snapshot the synchronous call stalls every other in-flight request by
- * 11 ms, while this stalls them by under 1 ms and costs the caller — which is
- * already waiting on an upstream fetch — about 5 ms more of its own latency.
- *
- * Returns null when the body is too small to be worth a frame, so callers can
- * store the result directly and treat null as "serve identity".
  *
  * @param {string|Buffer} body - the exact bytes the identity response sends.
  * @returns {Promise<Buffer|null>}
@@ -81,11 +62,9 @@ export async function gzipBody(body) {
 }
 
 /**
- * Pick the body to send and the headers that describe it, together.
+ * Pick the body and the headers that describe it. `Vary` is on both branches
- *
+ * so a shared cache never serves one encoding to a client that asked for the
- * `Vary: Accept-Encoding` is set on both branches, not only the compressed one:
+ * other.
- * a cache that stored an identity response without it would later hand those
- * bytes to a client that asked for gzip, and vice versa.
  *
  * @param {import('node:http').IncomingMessage} request
  * @param {string|Buffer} identity - the uncompressed body.

src/tooling/liveProviders.test.mjs

@@ -615,8 +615,7 @@ test('the OpenSky snapshot is compressed for clients that ask, and an error body
     if (upstreamStatus !== 200)
       return new Response('{"error":"upstream"}', { status: upstreamStatus });
     const sourceSeconds = Math.floor((now - snapshotAgeMs) / 1000);
-    // A healthy credit budget, so the proxy keeps its 9 s base TTL and the
+    // Healthy credits keep the 9 s base TTL, so the clock jumps below clear it.
-    // clock advances below are unambiguously past it.
     return Response.json(
       {
         time: sourceSeconds,
@@ -634,8 +633,7 @@ test('the OpenSky snapshot is compressed for clients that ask, and an error body
     );
   });
 
-  // A module instance of its own: the snapshot cache, its frame and the
+  // A fresh module: the snapshot cache, its frame and the TTL are module state.
-  // adaptive TTL are module state, and this test reasons about all three.
   const isolated = await import(
     `../../server/providers/aircraft/opensky.js?compression=${now}`
   );
@@ -648,25 +646,25 @@ test('the OpenSky snapshot is compressed for clients that ask, and an error body
   assert.equal(compressed.headers.vary, 'Accept-Encoding');
   assert.ok(Buffer.isBuffer(compressed.body));
 
-  // Same snapshot from cache, this time to a client that offered nothing: the
+  // Same cached snapshot, no Accept-Encoding: identity bytes, which the frame
-  // identity bytes, and the compressed frame above must decode to exactly them.
+  // above must decode to.
   const identity = await states('/api/flights');
   assert.equal(identity.headers['x-opensky-cache'], 'HIT');
   assert.equal(identity.headers['content-encoding'], undefined);
   assert.equal(identity.headers.vary, 'Accept-Encoding');
   assert.equal(gunzipSync(compressed.body).toString(), identity.body);
 
-  // A client that names gzip and refuses it with q=0 is a non-gzip client.
+  // `gzip;q=0` refuses gzip.
   const refused = await states('/api/flights', '/', 'GET', {
     'accept-encoding': 'gzip;q=0',
   });
   assert.equal(refused.headers['content-encoding'], undefined);
   assert.equal(refused.body, identity.body);
 
-  // An upstream rejection is not cached, so it has no frame of its own. It must
+  // An upstream rejection is not cached, so it must not carry the previous
-  // not be sent under the previous snapshot's Content-Encoding.
+  // snapshot's Content-Encoding.
   upstreamStatus = 403;
-  now += 60000; // past the 9 s base TTL this fresh instance starts with
+  now += 60000; // past the 9 s base TTL
   const rejected = await states('/api/flights', '/', 'GET', {
     'accept-encoding': 'gzip',
   });
@@ -674,9 +672,8 @@ test('the OpenSky snapshot is compressed for clients that ask, and an error body
   assert.equal(rejected.headers['content-encoding'], undefined);
   assert.ok(JSON.parse(rejected.body).error, 'a readable error body');
 
-  // The regional fallback is a fifth body this route can answer with. It is
+  // A snapshot too old to serve falls back to adsb.lol, which negotiates the
-  // cached the same way and so it negotiates the same way: a snapshot too old
+  // same way.
-  // to serve hands the request to adsb.lol, compressed.
   upstreamStatus = 200;
   snapshotAgeMs = 3600000;
   now += 60000;

src/tooling/responseCompression.test.mjs

@@ -17,7 +17,7 @@ function request(acceptEncoding) {
   };
 }
 
-/** Repetitive JSON, the shape of an actual snapshot body. */
+/** Repetitive JSON shaped like a snapshot body. */
 function snapshotBody(states = 200) {
   return JSON.stringify({
     time: 1700000000,
@@ -33,10 +33,7 @@ function snapshotBody(states = 200) {
   });
 }
 
-/**
+/** Seeded high-entropy bytes, so the "gzip made it bigger" check is stable. */
- * Deterministic high-entropy bytes. Random data would do, but a seeded sequence
- * keeps the "gzip made it bigger" assertion from depending on luck.
- */
 function incompressibleBytes(length) {
   const bytes = Buffer.alloc(length);
   let state = 123456789;
@@ -61,16 +58,15 @@ test('accept-encoding is read the way clients actually write it', () => {
   assert.equal(acceptsGzip({}), false);
   assert.equal(acceptsGzip(undefined), false);
 
-  // A quality of zero names gzip and refuses it (RFC 9110 section 12.4.2).
+  // `q=0` names gzip and refuses it (RFC 9110 section 12.4.2).
   assert.equal(acceptsGzip(request('gzip;q=0')), false);
   assert.equal(acceptsGzip(request('gzip;q=0.000')), false);
   assert.equal(acceptsGzip(request('gzip;q=0.001')), true);
   assert.equal(acceptsGzip(request('gzip;q=1')), true);
   assert.equal(acceptsGzip(request('gzip;q=1.000')), true);
 
-  // A quality that is not a whole `qvalue` is unreadable, not a number with
+  // A q that is not a whole `qvalue` is unreadable and must not round into
-  // trailing noise: the grammar allows three decimals and a ceiling of 1, so
+  // acceptance.
-  // these must not round into acceptance.
   assert.equal(acceptsGzip(request('gzip;q=bogus')), false);
   assert.equal(acceptsGzip(request('gzip;q=1oops')), false);
   assert.equal(acceptsGzip(request('gzip;q=0.5abc')), false);
@@ -102,8 +98,7 @@ test('a body is only compressed when compression pays for itself', async () => {
   assert.ok(compressed.byteLength < Buffer.byteLength(identity));
   assert.equal(gunzipSync(compressed).toString(), identity);
 
-  // Buffers are accepted as-is, and data that gzip cannot shrink is refused
+  // Buffers are accepted as-is; data gzip cannot shrink stays uncompressed.
-  // rather than served as a larger response.
   assert.ok((await gzipBody(Buffer.from(identity))) !== null);
   assert.equal(await gzipBody(incompressibleBytes(4096)), null);
 });
@@ -118,8 +113,7 @@ test('the encoded body and the header that names it are chosen together', async
   assert.equal(negotiated.headers.Vary, 'Accept-Encoding');
   assert.equal(gunzipSync(negotiated.body).toString(), identity);
 
-  // Offered but not wanted: identity bytes, and no Content-Encoding to
+  // Offered but refused: identity bytes and no Content-Encoding.
-  // contradict them.
   const refused = encodedBody(request('br'), identity, compressed);
   assert.equal(refused.body, identity);
   assert.equal(refused.headers['Content-Encoding'], undefined);
@@ -130,8 +124,7 @@ test('the encoded body and the header that names it are chosen together', async
   assert.equal(unavailable.body, identity);
   assert.equal(unavailable.headers['Content-Encoding'], undefined);
 
-  // Vary is on both branches: without it a shared cache could hand a stored
+  // Vary is on both branches, so a shared cache cannot serve the wrong encoding.
-  // response to a client whose Accept-Encoding it was not selected for.
   assert.equal(unavailable.headers.Vary, 'Accept-Encoding');
   assert.equal(
     encodedBody(request(undefined), identity, compressed).headers.Vary,