docs(server): tighten compression comments and docs

changelog.d/795-opensky-compress-cached-body.md

@@ -1,7 +1,5 @@
-- The live flight snapshot is compressed on the wire. `/api/flights` gzips its
-  cached global snapshot and its regional adsb.lol fallback once per upstream
-  refresh and serves that frame to clients offering gzip — roughly 860 KB
-  becomes roughly 258 KB — while a client that offers nothing receives the same
-  bytes uncompressed. Both answers carry `Vary: Accept-Encoding`, and a response
-  the proxy does not cache, such as an auth failure, is never sent under a
-  cached snapshot's encoding (daikaginza, #795).
+- Compress the live flight snapshot on the wire. `/api/flights` gzips its
+  cached global snapshot and its regional adsb.lol fallback once per refresh
+  and sends that frame to clients that offer gzip: about 860 KB becomes about
+  250 KB. Other clients get the same bytes uncompressed, and both answers carry
+  `Vary: Accept-Encoding` (daikaginza, #795).

docs/CURRENT-STATE.md

@@ -6186,7 +6186,7 @@ apply to `public/` assets or built-preview serving.
 - `/api/overpass` is bounded by body/response caps, per-client/global rate limits, concurrency limits, operator-configured upstreams, in-flight dedupe, cache bounds, and static validation that every selector is spatially bounded.
 - `/api/military-installations` uses an independent limiter with the same 90-per-client/300-global one-minute bounds, so viewport installation refreshes never consume `/api/overpass` annotation capacity.
 - `/api/route` proxies bounded OSRM route requests for annotation routes, with profile allowlisting, distance caps, response caps, caching, and sanitized "no route found" errors.
-- `/api/flights` negotiates response encoding. The cached global snapshot and the regional adsb.lol fallback are each gzipped once per refresh, so a client that offers gzip receives roughly 860 KB as 250 KB and one that does not receives the same bytes uncompressed; both answers carry `Vary: Accept-Encoding`. Bodies the proxy does not cache — auth failures, cooldown notices — are never sent under a cached response's encoding.
+- `/api/flights` negotiates gzip. The cached global snapshot and the regional adsb.lol fallback are each gzipped once per refresh (about 860 KB becomes about 250 KB). Clients that offer gzip get that frame; other clients, and bodies under 1 KiB or that gzip cannot shrink, get identity bytes. Both answers carry `Vary: Accept-Encoding`. Uncached bodies, such as auth failures, are never sent under a cached frame's encoding.
 - Track endpoints: `/api/vessels/track?mmsi=` (server-accumulated ring buffers; sub-route handled before the rows snapshot), `/api/flights/track?icao24=` (OAuth, 60s cache, sanitized errors, independent OpenSky credit bucket), `/api/military/track?hex=` (400 for an invalid hex, otherwise `503 history_unavailable` with `Retry-After: 1800` and `Cache-Control: no-store`, no upstream request; military tracking uses its locally accumulated trail).
 - Realtime debug logs redact API keys, bearer tokens, client secrets, and image data URLs before writing to disk; request bodies are size-capped.
 - `/api/realtime/debug-log` enforces an always-on 120-per-client/400-global one-minute limiter (not the opt-in `GEV_RATELIMIT_OPENAI_PER_MIN` bucket the cost-bearing OpenAI routes share), appends asynchronously through a serialized queue, and rotates `realtime-conversations.jsonl` at 32 MB keeping one prior generation, so the sink is bounded at twice that regardless of session length. A malformed record answers 400 and a failed write 500, both with a fixed message.

server/providers/aircraft/opensky.js

@@ -20,9 +20,8 @@ let _openskyTokenPromise = null;
 /** @type {string|null} Cached upstream response body (JSON text). */
 let _openskyCacheBody = null;
 /**
- * @type {Buffer|null} Gzip of `_openskyCacheBody`, written in the same step as
- * the body it encodes so the two can never describe different snapshots. Null
- * whenever the cached body is too small to be worth a frame.
+ * @type {Buffer|null} Gzip of `_openskyCacheBody`, written together with it.
+ * Null when the body is not worth compressing.
  */
 let _openskyCacheGzip = null;
 /** @type {number} HTTP status of the cached response. */
@@ -409,9 +408,8 @@ async function fetchAdsbLolPointFallback(req) {
         const body = JSON.stringify(normalized);
         const record = {
           body,
-          // Compressed with the body it belongs to, so a cache entry served
-          // many times pays for the frame once. Entries are bounded by
-          // ADSBLOL_POINT_CACHE_MAX, so the extra residency is bounded too.
+          // Compressed once per entry; ADSBLOL_POINT_CACHE_MAX bounds the
+          // extra memory.
           gzip: await gzipBody(body),
           cachedAt: Date.now(),
           count: normalized.states.length,
@@ -647,9 +645,8 @@ export function openSkyProxy() {
             // Only cache successful responses — error responses (401/403/429/5xx)
             // should not be served from cache on subsequent requests
             if (upstream.ok) {
-              // The frame is built before either slot is written so the two
-              // are always published together: a request arriving mid-refresh
-              // reads the previous body with the previous body's frame.
+              // Compress before writing either slot, so a request arriving
+              // mid-refresh still reads a body with its own frame.
               const frame = await gzipBody(body);
               _openskyCacheBody = body;
               _openskyCacheGzip = frame;
@@ -763,9 +760,8 @@ export function openSkyProxy() {
           }
         }
 
-        // Only the coalesced refresh caches, so the gzip slot belongs to this
-        // body only when this body is the one it stored. An error response is
-        // not cached and must not borrow the previous snapshot's frame.
+        // The frame belongs to this body only if this body was cached. An
+        // uncached error must not borrow the previous snapshot's frame.
         const encoded = encodedBody(
           req,
           body,

server/providers/common/compress.js

@@ -4,39 +4,25 @@ import { promisify } from 'node:util';
 const gzipAsync = promisify(gzip);
 
 /**
- * Response compression for proxies that already hold a whole body in memory.
+ * Opt-in gzip for text bodies a proxy already holds in memory.
  *
- * Deliberately not a middleware. Two response paths in this tree must never be
- * wrapped: `cctv/media.js` streams a piped body and answers byte-range requests
- * with 206, and `wind/gfs.js` also serves ranges — a wrapper that buffered or
- * re-encoded those would break playback and range semantics. Seven providers
- * additionally forward an upstream `content-type` verbatim, including already
- * compressed binary formats that gzip would only make larger.
- *
- * So compression is opt-in at the call site, for bodies the proxy holds and
- * knows to be text. The failure mode of forgetting a site is a response that is
- * merely uncompressed, never a corrupt one — but only as long as the encoded
- * body and its `Content-Encoding` are chosen together, which is why
- * `encodedBody` returns both rather than exposing them separately.
+ * Not a middleware: `cctv/media.js` streams bodies and answers range requests,
+ * and several providers forward binary upstream types that gzip only grows.
+ * `encodedBody` returns the body with its headers so the two cannot disagree.
  */
 
-/** Below this, a gzip frame costs more than the bytes it saves. */
+/** Skip compression for bodies smaller than 1 KiB. */
 const MIN_COMPRESSED_BYTES = 1024;
 
 /**
- * The `qvalue` grammar of RFC 9110 section 12.4.2: zero to three decimals
- * below one, or one with only zeros after it. Matching the whole token rather
- * than parsing a numeric prefix is what makes `q=1oops` and the out-of-range
- * `q=2` unreadable instead of quietly rounding to something acceptable.
+ * RFC 9110 section 12.4.2 `qvalue`. Matching the whole token keeps `q=1oops`
+ * and `q=2` unreadable instead of rounding them into acceptance.
  */
 const QVALUE = /^(?:0(?:\.\d{0,3})?|1(?:\.0{0,3})?)$/;
 
 /**
- * Whether the client offered gzip in `Accept-Encoding`.
- *
- * Anything this cannot read confidently — a missing header, a malformed or
- * out-of-range quality value, the `x-gzip` alias — reads as "no", because the
- * fallback is an identity response every client can decode.
+ * Whether `Accept-Encoding` offers gzip. Anything unreadable (no header, a bad
+ * q value, `x-gzip`) reads as no: every client can decode identity.
  *
  * @param {import('node:http').IncomingMessage} request
  * @returns {boolean}
@@ -51,23 +37,18 @@ export function acceptsGzip(request) {
       .map((parameter) => parameter.trim().toLowerCase())
       .find((parameter) => parameter.startsWith('q='));
     if (quality === undefined) return true;
-    // `q=0` is the one way a client can name an encoding and still refuse it.
+    // `q=0` names gzip and refuses it.
     const value = quality.slice(2);
     return QVALUE.test(value) && Number(value) > 0;
   });
 }
 
 /**
- * Gzip a body once, for callers that cache it and serve it many times.
+ * Gzip a body once for a caller that caches it. Async, so the ~860 KB OpenSky
+ * snapshot does not block other requests while it compresses.
  *
- * OpenSky's snapshot is ~860 KB and is held for the cache TTL, so this is paid
- * per refresh rather than per request. It runs off the event loop on purpose:
- * on that snapshot the synchronous call stalls every other in-flight request by
- * 11 ms, while this stalls them by under 1 ms and costs the caller — which is
- * already waiting on an upstream fetch — about 5 ms more of its own latency.
- *
- * Returns null when the body is too small to be worth a frame, so callers can
- * store the result directly and treat null as "serve identity".
+ * Returns null, meaning "serve identity", when the body is under 1 KiB or gzip
+ * does not shrink it.
  *
  * @param {string|Buffer} body - the exact bytes the identity response sends.
  * @returns {Promise<Buffer|null>}
@@ -81,11 +62,9 @@ export async function gzipBody(body) {
 }
 
 /**
- * Pick the body to send and the headers that describe it, together.
- *
- * `Vary: Accept-Encoding` is set on both branches, not only the compressed one:
- * a cache that stored an identity response without it would later hand those
- * bytes to a client that asked for gzip, and vice versa.
+ * Pick the body and the headers that describe it. `Vary` is on both branches
+ * so a shared cache never serves one encoding to a client that asked for the
+ * other.
  *
  * @param {import('node:http').IncomingMessage} request
  * @param {string|Buffer} identity - the uncompressed body.

src/tooling/liveProviders.test.mjs

@@ -615,8 +615,7 @@ test('the OpenSky snapshot is compressed for clients that ask, and an error body
     if (upstreamStatus !== 200)
       return new Response('{"error":"upstream"}', { status: upstreamStatus });
     const sourceSeconds = Math.floor((now - snapshotAgeMs) / 1000);
-    // A healthy credit budget, so the proxy keeps its 9 s base TTL and the
-    // clock advances below are unambiguously past it.
+    // Healthy credits keep the 9 s base TTL, so the clock jumps below clear it.
     return Response.json(
       {
         time: sourceSeconds,
@@ -634,8 +633,7 @@ test('the OpenSky snapshot is compressed for clients that ask, and an error body
     );
   });
 
-  // A module instance of its own: the snapshot cache, its frame and the
-  // adaptive TTL are module state, and this test reasons about all three.
+  // A fresh module: the snapshot cache, its frame and the TTL are module state.
   const isolated = await import(
     `../../server/providers/aircraft/opensky.js?compression=${now}`
   );
@@ -648,25 +646,25 @@ test('the OpenSky snapshot is compressed for clients that ask, and an error body
   assert.equal(compressed.headers.vary, 'Accept-Encoding');
   assert.ok(Buffer.isBuffer(compressed.body));
 
-  // Same snapshot from cache, this time to a client that offered nothing: the
-  // identity bytes, and the compressed frame above must decode to exactly them.
+  // Same cached snapshot, no Accept-Encoding: identity bytes, which the frame
+  // above must decode to.
   const identity = await states('/api/flights');
   assert.equal(identity.headers['x-opensky-cache'], 'HIT');
   assert.equal(identity.headers['content-encoding'], undefined);
   assert.equal(identity.headers.vary, 'Accept-Encoding');
   assert.equal(gunzipSync(compressed.body).toString(), identity.body);
 
-  // A client that names gzip and refuses it with q=0 is a non-gzip client.
+  // `gzip;q=0` refuses gzip.
   const refused = await states('/api/flights', '/', 'GET', {
     'accept-encoding': 'gzip;q=0',
   });
   assert.equal(refused.headers['content-encoding'], undefined);
   assert.equal(refused.body, identity.body);
 
-  // An upstream rejection is not cached, so it has no frame of its own. It must
-  // not be sent under the previous snapshot's Content-Encoding.
+  // An upstream rejection is not cached, so it must not carry the previous
+  // snapshot's Content-Encoding.
   upstreamStatus = 403;
-  now += 60000; // past the 9 s base TTL this fresh instance starts with
+  now += 60000; // past the 9 s base TTL
   const rejected = await states('/api/flights', '/', 'GET', {
     'accept-encoding': 'gzip',
   });
@@ -674,9 +672,8 @@ test('the OpenSky snapshot is compressed for clients that ask, and an error body
   assert.equal(rejected.headers['content-encoding'], undefined);
   assert.ok(JSON.parse(rejected.body).error, 'a readable error body');
 
-  // The regional fallback is a fifth body this route can answer with. It is
-  // cached the same way and so it negotiates the same way: a snapshot too old
-  // to serve hands the request to adsb.lol, compressed.
+  // A snapshot too old to serve falls back to adsb.lol, which negotiates the
+  // same way.
   upstreamStatus = 200;
   snapshotAgeMs = 3600000;
   now += 60000;

src/tooling/responseCompression.test.mjs

@@ -17,7 +17,7 @@ function request(acceptEncoding) {
   };
 }
 
-/** Repetitive JSON, the shape of an actual snapshot body. */
+/** Repetitive JSON shaped like a snapshot body. */
 function snapshotBody(states = 200) {
   return JSON.stringify({
     time: 1700000000,
@@ -33,10 +33,7 @@ function snapshotBody(states = 200) {
   });
 }
 
-/**
- * Deterministic high-entropy bytes. Random data would do, but a seeded sequence
- * keeps the "gzip made it bigger" assertion from depending on luck.
- */
+/** Seeded high-entropy bytes, so the "gzip made it bigger" check is stable. */
 function incompressibleBytes(length) {
   const bytes = Buffer.alloc(length);
   let state = 123456789;
@@ -61,16 +58,15 @@ test('accept-encoding is read the way clients actually write it', () => {
   assert.equal(acceptsGzip({}), false);
   assert.equal(acceptsGzip(undefined), false);
 
-  // A quality of zero names gzip and refuses it (RFC 9110 section 12.4.2).
+  // `q=0` names gzip and refuses it (RFC 9110 section 12.4.2).
   assert.equal(acceptsGzip(request('gzip;q=0')), false);
   assert.equal(acceptsGzip(request('gzip;q=0.000')), false);
   assert.equal(acceptsGzip(request('gzip;q=0.001')), true);
   assert.equal(acceptsGzip(request('gzip;q=1')), true);
   assert.equal(acceptsGzip(request('gzip;q=1.000')), true);
 
-  // A quality that is not a whole `qvalue` is unreadable, not a number with
-  // trailing noise: the grammar allows three decimals and a ceiling of 1, so
-  // these must not round into acceptance.
+  // A q that is not a whole `qvalue` is unreadable and must not round into
+  // acceptance.
   assert.equal(acceptsGzip(request('gzip;q=bogus')), false);
   assert.equal(acceptsGzip(request('gzip;q=1oops')), false);
   assert.equal(acceptsGzip(request('gzip;q=0.5abc')), false);
@@ -102,8 +98,7 @@ test('a body is only compressed when compression pays for itself', async () => {
   assert.ok(compressed.byteLength < Buffer.byteLength(identity));
   assert.equal(gunzipSync(compressed).toString(), identity);
 
-  // Buffers are accepted as-is, and data that gzip cannot shrink is refused
-  // rather than served as a larger response.
+  // Buffers are accepted as-is; data gzip cannot shrink stays uncompressed.
   assert.ok((await gzipBody(Buffer.from(identity))) !== null);
   assert.equal(await gzipBody(incompressibleBytes(4096)), null);
 });
@@ -118,8 +113,7 @@ test('the encoded body and the header that names it are chosen together', async
   assert.equal(negotiated.headers.Vary, 'Accept-Encoding');
   assert.equal(gunzipSync(negotiated.body).toString(), identity);
 
-  // Offered but not wanted: identity bytes, and no Content-Encoding to
-  // contradict them.
+  // Offered but refused: identity bytes and no Content-Encoding.
   const refused = encodedBody(request('br'), identity, compressed);
   assert.equal(refused.body, identity);
   assert.equal(refused.headers['Content-Encoding'], undefined);
@@ -130,8 +124,7 @@ test('the encoded body and the header that names it are chosen together', async
   assert.equal(unavailable.body, identity);
   assert.equal(unavailable.headers['Content-Encoding'], undefined);
 
-  // Vary is on both branches: without it a shared cache could hand a stored
-  // response to a client whose Accept-Encoding it was not selected for.
+  // Vary is on both branches, so a shared cache cannot serve the wrong encoding.
   assert.equal(unavailable.headers.Vary, 'Accept-Encoding');
   assert.equal(
     encodedBody(request(undefined), identity, compressed).headers.Vary,