Bound fallback retry attempts with a deadline

An Esri or keyless terrain attempt that never settled kept the single-flight
retry open forever, so neither provider was retried again. Each attempt now
counts as failed after 60 seconds and the retry backs off as usual; its
signal is aborted, and a provider that lands after the deadline is released
rather than installed or cached.

CURRENT-STATE now says only the construction fallback to OSM is retried, not
the switch made after repeated Esri tile failures.
This commit is contained in:
Sameh Khamis committed 2026-10-16 16:47:11 -07:00
1 parent f0d43a62e8
commit 759c7964c7
3 files changed
+110 -9

No files matched your search

+5 -5
View File
@@ -385,10 +385,10 @@ plus up to 1.5 s of jitter, and `Retry-After` extending the cooldown when the
upstream exposes it. One console warning is logged per cooldown window. Other upstream exposes it. One console warning is logged per cooldown window. Other
failures keep Cesium's handling and the flat `EllipsoidTerrainProvider` failures keep Cesium's handling and the flat `EllipsoidTerrainProvider`
fallback; while that flat terrain is on a shown globe the map controller fallback; while that flat terrain is on a shown globe the map controller
retries Re:Earth in the background (30 s, doubling to a 10-minute cap) and retries Re:Earth in the background (30 s, doubling to a 10-minute cap, each
installs the real terrain without waiting for a map switch or an imagery attempt abandoned after 60 s) and installs the real terrain without waiting
retry. Behind a 3D tileset it neither retries nor installs; the next globe for a map switch or an imagery retry. Behind a 3D tileset it neither retries
activation does. nor installs; the next globe activation does.
`scripts/qa-terrain-429.mjs` reproduces the burst against a keyless `scripts/qa-terrain-429.mjs` reproduces the burst against a keyless
dev server, counts real throttles with their headers as evidence, synthesises dev server, counts real throttles with their headers as evidence, synthesises
throttles with `--inject N` when the upstream is not throttling, and fails if throttles with `--inject N` when the upstream is not throttling, and fails if
@@ -4776,7 +4776,7 @@ are omitted rather than framing the wrong part of the globe.
- The bottom Visual Presets tray presents a **five-tile MAP SOURCE row** (`#map-stack-chips`, `src/mapStackChips.js`): Google 3D, Esri Satellite, Bing Aerial, Bing Labels, and OSM. The duplicate left `#stack-panel` is retired. The five tiles share one row on desktop and two rows on narrow screens, carry `aria-pressed` on the active source, and remain keyboard-reachable with a visible focus outline. - The bottom Visual Presets tray presents a **five-tile MAP SOURCE row** (`#map-stack-chips`, `src/mapStackChips.js`): Google 3D, Esri Satellite, Bing Aerial, Bing Labels, and OSM. The duplicate left `#stack-panel` is retired. The five tiles share one row on desktop and two rows on narrow screens, carry `aria-pressed` on the active source, and remain keyboard-reachable with a visible focus outline.
- The lit tile follows controller state, not the click: a rejected switch (no ion token) or a superseded one (rapid A→B) leaves the genuinely active source lit, and the tray heading keeps its short-label status readout (`...` while switching, amber on `lastError`). - The lit tile follows controller state, not the click: a rejected switch (no ion token) or a superseded one (rapid A→B) leaves the genuinely active source lit, and the tray heading keeps its short-label status readout (`...` while switching, amber on `lastError`).
- Ion stacks remain visible and keyboard-focusable when no ion token is configured, but expose `aria-disabled="true"` and do not switch. Their accessible label and tooltip quote `getStacks().unavailableReason` — the same string `setStack()` puts in the toast. OSM works keyless. The `ION` badge is gated on the stack's own `requiresIon`, so a `photoreal` chip unavailable because the Google tileset failed says so instead of falsely demanding an ion token. - Ion stacks remain visible and keyboard-focusable when no ion token is configured, but expose `aria-disabled="true"` and do not switch. Their accessible label and tooltip quote `getStacks().unavailableReason` — the same string `setStack()` puts in the toast. OSM works keyless. The `ION` badge is gated on the stack's own `requiresIon`, so a `photoreal` chip unavailable because the Google tileset failed says so instead of falsely demanding an ion token.
- Stack choice participates in share links (`src/sharelink.js`) and falls back to the best available stack when the requested one is unavailable (keyless boots land on Esri; OSM takes over automatically if Esri is unreachable; while that fallback is on screen Esri is retried in the background, 30 s doubling to a 10-minute cap, and replaces OSM once it loads). Share-link restore, the `set_map_stack` voice tool, and the chip row all land on the same `_setMapStack()` path. - Stack choice participates in share links (`src/sharelink.js`) and falls back to the best available stack when the requested one is unavailable (keyless boots land on Esri; OSM takes over automatically if Esri is unreachable; while that construction fallback is on screen Esri is retried in the background, 30 s doubling to a 10-minute cap with each attempt abandoned after 60 s, and replaces OSM once it loads. The switch to OSM after repeated Esri tile failures is not retried; selecting Esri again tries it). Share-link restore, the `set_map_stack` voice tool, and the chip row all land on the same `_setMapStack()` path.
### Voice Map Whiteboard / Annotations (June 2026) ### Voice Map Whiteboard / Annotations (June 2026)
+29 -4
View File
@@ -12,6 +12,8 @@ const AUTOMATIC = Symbol('automatic switch');
/** Background retries of a fallback in use back off from 30 s to 10 min. */ /** Background retries of a fallback in use back off from 30 s to 10 min. */
const FALLBACK_RETRY_MS = 30_000; const FALLBACK_RETRY_MS = 30_000;
const FALLBACK_RETRY_MAX_MS = 600_000; const FALLBACK_RETRY_MAX_MS = 600_000;
/** A retry attempt still unsettled after a minute counts as failed. */
const FALLBACK_ATTEMPT_MS = 60_000;
const resolved = (provider, effectiveStackId) => ({ const resolved = (provider, effectiveStackId) => ({
provider, provider,
@@ -354,17 +356,38 @@ export class MapSourceController {
if (!id && !terrain) return; if (!id && !terrain) return;
this._retrying = true; this._retrying = true;
this._retryRequested = false; this._retryRequested = false;
const attempt = async (create) => { // An attempt that outlives its deadline counts as a failure. Its signal is
// aborted, but the Esri and keyless factories ignore it and Cesium's
// requests have no timeout, so a late result is released, never used.
const attempt = async (create, release) => {
const expiry = new AbortController();
let timer;
const pending = Promise.resolve().then(() =>
create?.({
signal: AbortSignal.any([this._abort.signal, expiry.signal]),
}),
);
const deadline = new Promise((resolve) => {
timer = setTimeout(resolve, FALLBACK_ATTEMPT_MS, expiry);
});
try { try {
return await create?.({ signal: this._abort.signal }); const result = await Promise.race([pending, deadline]);
if (result !== expiry) return result;
expiry.abort();
pending.then(release, () => {});
return null;
} catch { } catch {
return null; // Still unavailable: keep the fallback. return null; // Still unavailable: keep the fallback.
} finally {
clearTimeout(timer);
} }
}; };
// Each recovers as soon as it loads, never waiting on the other. // Each recovers as soon as it loads, never waiting on the other.
const retryImagery = async () => { const retryImagery = async () => {
const cached = this._imageryProviders.get(id); const cached = this._imageryProviders.get(id);
const provider = await attempt(this._sources.get(id).imagery); const provider = await attempt(this._sources.get(id).imagery, (late) =>
this._dispose(late),
);
// Obsolete: destroyed, or the cache moved on while this was in flight. // Obsolete: destroyed, or the cache moved on while this was in flight.
if (this._destroyed || this._imageryProviders.get(id) !== cached) { if (this._destroyed || this._imageryProviders.get(id) !== cached) {
this._dispose(provider); this._dispose(provider);
@@ -388,7 +411,9 @@ export class MapSourceController {
}; };
const retryTerrain = async () => { const retryTerrain = async () => {
const cached = this._terrainProviders.get(terrain.id); const cached = this._terrainProviders.get(terrain.id);
const real = await attempt(terrain.create); const real = await attempt(terrain.create, (late) =>
this._dispose(late?.provider),
);
if ( if (
this._destroyed || this._destroyed ||
this._terrainProviders.get(terrain.id) !== cached || this._terrainProviders.get(terrain.id) !== cached ||
+76
View File
@@ -1180,3 +1180,79 @@ test('a stalled imagery retry does not hold back recovered terrain', async (t) =
await settle(); await settle();
env.controller.destroy(); env.controller.destroy();
}); });
test('a retry attempt that never settles is abandoned at its deadline and released if it lands late', async (t) => {
const hung = deferred();
const late = { id: 'late', destroy: () => (late.destroyed = true) };
const { env, installed, terrain } = terrainFixture(t, async (n) => {
if (n === 2) await hung.promise;
return n === 2 ? { provider: late } : flatTerrain(n);
});
const esri = failingEsri(env);
await env.controller.setStack('esri-imagery');
t.mock.timers.tick(30_000);
await settle();
assert.equal(esri.attempts, 2, 'imagery failed');
assert.equal(terrain.calls, 2, 'terrain hangs');
t.mock.timers.tick(60_000); // The hung attempt's deadline.
await settle();
t.mock.timers.tick(60_000); // The doubled backoff.
await settle();
assert.equal(esri.attempts, 3, 'imagery is retried again');
assert.equal(terrain.calls, 3, 'terrain is retried again');
hung.resolve();
await settle();
assert.equal(late.destroyed, true, 'the late provider is released');
assert.ok(!installed.includes(late), 'and never installed');
env.controller.destroy();
});
test('a failed retry after a mid-retry re-show re-arms once, after the doubled delay', async (t) => {
t.mock.timers.enable({ apis: ['setTimeout'] });
const env = publicFixture();
const esri = failingEsri(env);
await env.controller.setStack('esri-imagery');
const gate = deferred();
esri.next = gate.promise;
t.mock.timers.tick(30_000);
await settle();
await env.controller.setStack('osm');
await env.controller.setStack('esri-imagery');
gate.resolve();
await settle();
assert.equal(esri.attempts, 2);
t.mock.timers.tick(59_999);
await settle();
assert.equal(esri.attempts, 2, 'not before the doubled delay');
t.mock.timers.tick(1);
await settle();
assert.equal(esri.attempts, 3, 'one retry after it');
t.mock.timers.tick(119_999);
await settle();
assert.equal(esri.attempts, 3, 'no second timer was armed');
env.controller.destroy();
});
test('destroy releases recovered imagery and terrain that lands afterwards', async (t) => {
const stalled = deferred();
const real = { id: 'real', destroy: () => (real.destroyed = true) };
const { env, installed } = terrainFixture(t, async (n) => {
if (n === 1) return flatTerrain(n);
await stalled.promise;
return { provider: real };
});
const esri = failingEsri(env);
const imagery = env.providers.get('esri-imagery');
imagery.destroy = () => (imagery.destroyed = true);
await env.controller.setStack('esri-imagery');
esri.failing = false;
t.mock.timers.tick(30_000);
await settle();
assert.equal(env.imagery.at(-1).provider, imagery, 'imagery has landed');
env.controller.destroy();
stalled.resolve();
await settle();
assert.equal(imagery.destroyed, true);
assert.equal(real.destroyed, true);
assert.ok(!installed.includes(real));
});