fix(news): answer within the browser's wait on a first read

A country's first World News API read, or Liveuamap's, could take up to
20 s while the browser waits 15, so the panel showed the browser's own
"signal timed out". A read with nothing to serve yet is now waited for
at most 12 s, as GDELT's already was: past that the section answers
unavailable while the read goes on, and the next refresh is served from
it. A client timeout reads "News feed unavailable".
This commit is contained in:
Denis Groshev committed 2026-11-06 11:41:37 +01:00
1 parent 525543d08a
commit c6ae4c44d7
6 files changed
+87 -25

No files matched your search

+1 -1
View File
@@ -5853,7 +5853,7 @@ request viewbox and continues searching without that geographic preference.
- **Panel.** The right-rail NEWS panel (`src/ui/newsPanel.js`, `src/ui/news/`) has a time window at the top (the server's 24 h, or 12, 6, 3 or 1 h; the hotspot counts are read for the chosen window) and two sections. **Conflicts**: hotspot chips with counts, a severity floor (significant and above by default), a category filter, the chosen event, and a continuous feed, newest first. **Local**, while the camera is below 3,000 km: every report in the area at the view centre, newest first, named at the top ("News from …") and outlined on the globe, which then shows those reports instead of the hotspots' news; at sea, reports within 500 km. A disputed area of the bundled list (11, see `src/data/local_data/natural_earth/README.md`), or a territory Natural Earth gives no country code (Somaliland, Northern Cyprus and others), is named as itself and read from no country's outlets; elsewhere Natural Earth's default country lines apply. Each card shows the publisher's picture when the event has one (hotlinked, no referrer, publisher credited), the headline (a model's English headline marked as such), a mono meta line (severity marker, category, place, sources, age), who reported it, attestation and media chips, and the service it came through ("via GDELT Project"). Open, a card plays the post's clip on request. A click flies the camera straight down from 150 km and selects the report.
- **Notices.** A section whose key the server lacks says "{provider} key required" and names the variable; a refused key says "{provider} refused its key"; a spent daily allowance says "{provider} limit reached for today". The other section keeps working.
- **Source.** `sources.news` (`createNewsSource`, export `./layers/news/source`) reads same-origin `/api/news/*` with a 15 s timeout, a 4 MiB cap and record-by-record validation; `newsQueryAllowed` lets a host admit exactly the client's requests. Core's local server (`server/providers/news.js`) forwards only those GET requests to `GEV_NEWS_ORIGIN` when it is set.
- **Built-in backend.** With `GEV_NEWS_ORIGIN` unset, Core answers the same routes itself (`server/providers/news/keyed.js`). **Conflicts** needs no key: the last 24 h of GDELT 2.0 event exports (15-minute files on `data.gdeltproject.org`, about 100 KB each), physical violence only (CAMEO roots 18–20), one report per story with copies of one agency story merged, placed at GDELT's location for it (a town the article's words name beats a bare country). Titles are the article's page title from GDELT's GKG export of the same quarter hour (about 6 MB each) for the newest 3 h, else the words of the article's address; GDELT's actor coding is never shown. With `LIVEUAMAP_API_KEY`, Liveuamap's mapped items stand beside GDELT's. **Local** reads World News API (`WORLDNEWSAPI_KEY`) by the source country at the view centre, once per country every 30 minutes (about 1.5 points a read, at most `WORLDNEWSAPI_DAILY_READS` reads a day, default 30), plus the Conflicts reports placed inside the area. A country's own stories keep World News API's category (politics, business, else other) and no severity: the rules read fighting and crises, and everyday headlines mislead them. Areas come from the bundled Natural Earth countries and disputed areas (`server/providers/news/areas.js`). Conflicts' categories, relevance and severity are bundled rules (`categorize.js`); a news server may replace them.
- **Built-in backend.** With `GEV_NEWS_ORIGIN` unset, Core answers the same routes itself (`server/providers/news/keyed.js`). **Conflicts** needs no key: the last 24 h of GDELT 2.0 event exports (15-minute files on `data.gdeltproject.org`, about 100 KB each), physical violence only (CAMEO roots 18–20), one report per story with copies of one agency story merged, placed at GDELT's location for it (a town the article's words name beats a bare country). Titles are the article's page title from GDELT's GKG export of the same quarter hour (about 6 MB each) for the newest 3 h, else the words of the article's address; GDELT's actor coding is never shown. With `LIVEUAMAP_API_KEY`, Liveuamap's mapped items stand beside GDELT's. **Local** reads World News API (`WORLDNEWSAPI_KEY`) by the source country at the view centre, once per country every 30 minutes (about 1.5 points a read, at most `WORLDNEWSAPI_DAILY_READS` reads a day, default 30), plus the Conflicts reports placed inside the area. A country's own stories keep World News API's category (politics, business, else other) and no severity: the rules read fighting and crises, and everyday headlines mislead them. Areas come from the bundled Natural Earth countries and disputed areas (`server/providers/news/areas.js`). Conflicts' categories, relevance and severity are bundled rules (`categorize.js`); a news server may replace them. A read with nothing to serve yet (the first of GDELT's day, of Liveuamap or of a country) is waited for at most 12 s, under the client's 15 s timeout: past that the section answers unavailable (502) while the read goes on, and the next refresh is served from it.
- **State.** Share/restore token `9`, enabled-only. Voice is off for this layer (`VOICE_OFF_LAYERS`).
### S2 GhostMaps (October 2026)
+3
View File
@@ -145,6 +145,9 @@ Status: Open (limits)
- A country's own stories in Local keep World News API's category (politics as
political, business as economy, else other) and carry no severity; fighting
there shows through the Conflicts reports placed in the area.
- When World News API takes over 12 s to answer a country's first read, Local
reads unavailable until the layer's next refresh (up to 2 minutes), which is
served from that read.
- Local needs a World News API key. Its free plan allows 50 points a day; Core
stops at `WORLDNEWSAPI_DAILY_READS` reads (default 30) and Local then says the
daily limit is reached.
+26 -16
View File
@@ -59,14 +59,13 @@ const DAY_MS = 24 * HOUR_MS;
// published; one that fails GDELT_EXPORT_TRIES times (a try a round), or
// cannot be read at all, is given up on, so one bad file never holds the
// rest back. Only while GDELT itself answers is a file given up on. The first
// answer waits at most GDELT_FIRST_MS for the first exports.
// answer waits at most FIRST_ANSWER_MS for the first exports.
const GDELT_FRESH_MS = 5 * 60_000;
const GDELT_EXPORTS_PER_READ = 12;
const GDELT_PARALLEL = 4;
const GDELT_NEXT_MS = 2_000;
const GDELT_GONE_MS = 90 * 60_000;
const GDELT_EXPORT_TRIES = 3;
const GDELT_FIRST_MS = 12_000;
const MAX_GDELT_REPORTS = 5000;
// Events kept of one story, to place it by.
const MAX_STORY_EVENTS = 200;
@@ -80,6 +79,9 @@ const COUNTRY_FRESH_MS = 30 * 60_000;
// A failed read is not tried again for a minute; what was read before is
// served meanwhile, up to these ages (GDELT's: since an export last came in).
const RETRY_MS = 60_000;
// A read with nothing to serve yet is waited for at most this long, under the
// browser's own 15 s: it goes on, and the next ask is served from it.
const FIRST_ANSWER_MS = 12_000;
const GDELT_STALE_MS = 2 * HOUR_MS;
const LIVE_STALE_MS = HOUR_MS;
const COUNTRY_STALE_MS = 6 * HOUR_MS;
@@ -276,6 +278,7 @@ export function createKeyedNews({
now = () => Date.now(),
log = (message) => console.warn(message),
later = (run, ms) => setTimeout(run, ms).unref(),
firstAnswerMs = FIRST_ANSWER_MS,
} = {}) {
const keyOf = (id) => String(env()[NEWS_KEYS[id]] ?? '').trim();
const inFlight = new Map();
@@ -373,7 +376,7 @@ export function createKeyedNews({
.finally(() => inFlight.delete(flight)),
);
if (prior.value && at - prior.readAt < staleMs) return prior;
return inFlight.get(flight);
return firstAnswer(inFlight.get(flight), { ...prior, error: null });
}
/** An entry stored as the most recently used (the countries are capped). */
@@ -384,6 +387,22 @@ export function createKeyedNews({
}
/** An entry's reports while they may be served, else why there are none. */
/** `read`, waited for at most firstAnswerMs; then `fallback`, as it goes on. */
async function firstAnswer(read, fallback) {
let timer;
try {
return await Promise.race([
read,
new Promise((resolve) => {
timer = setTimeout(() => resolve(fallback), firstAnswerMs);
timer.unref?.();
}),
]);
} finally {
clearTimeout(timer);
}
}
function served(entry, staleMs, id) {
if (entry.value && now() - entry.readAt < staleMs)
return { reports: entry.value, readAt: entry.readAt };
@@ -602,19 +621,10 @@ export function createKeyedNews({
// newest exports' events, not their titles.
const current = () =>
gdelt.readAt !== null && now() - gdelt.readAt < GDELT_STALE_MS;
if (!current() && inFlight.has('gdelt')) {
// At most GDELT_FIRST_MS: a slow GDELT must not outlast the browser's
// own timeout; what has come in by then is served.
let timer;
await Promise.race([
inFlight.get('gdelt'),
new Promise((resolve) => {
timer = setTimeout(resolve, GDELT_FIRST_MS);
timer.unref?.();
}),
]);
clearTimeout(timer);
}
// A slow GDELT must not outlast the browser's own timeout: what has come
// in by then is served.
if (!current() && inFlight.has('gdelt'))
await firstAnswer(inFlight.get('gdelt'));
if (!current()) return { failed: true };
const dayAgo = new Date(now() - DAY_MS).toISOString();
const reports = [...gdelt.reports.values()]
+32
View File
@@ -1082,3 +1082,35 @@ test("a country's own news keeps World News API's category and no severity, what
assert.equal(read('King visits Amman', 'politics').category, 'political');
assert.equal(read('Parliament session opens', null).category, 'other');
});
test('a first read slower than the browser waits answers unavailable, and the next ask is served from it', async () => {
const fake = upstream();
let release;
const held = new Promise((resolve) => {
release = resolve;
});
const news = createKeyedNews({
env: () => ({ WORLDNEWSAPI_KEY: 'wna-key' }),
fetchImpl: async (url, init) => {
if (new URL(url).host === 'api.worldnewsapi.com') await held;
return fake.fetchImpl(url, init);
},
now: () => NOW,
log: () => {},
later: () => {},
firstAnswerMs: 20,
});
const local = () =>
news.answer('/feed', new URLSearchParams('hotspot=all&at=31.95,35.93'));
assert.deepEqual(await local(), UPSTREAM_ERROR);
// The read went on, and is not asked again.
release();
await new Promise((resolve) => setTimeout(resolve, 5));
const served = await local();
assert.equal(served.status, 200);
assert.deepEqual(
served.body.reports.map((r) => r.id),
['wna-9001', 'wna-9002'],
);
assert.equal(count(fake.calls, 'api.worldnewsapi.com'), 1);
});
+19 -7
View File
@@ -161,13 +161,25 @@ export function createNewsSource({
signal?.throwIfAborted();
const timeout = AbortSignal.timeout(timeoutMs);
const request = signal ? AbortSignal.any([signal, timeout]) : timeout;
const response = await fetchImpl(`${BASE_PATH}${path}`, {
signal: request,
cache: 'no-store',
redirect: 'error',
});
if (!response.ok) throw await responseError(response);
return readResponseJsonCapped(response, MAX_BYTES, request);
try {
const response = await fetchImpl(`${BASE_PATH}${path}`, {
signal: request,
cache: 'no-store',
redirect: 'error',
});
if (!response.ok) throw await responseError(response);
return await readResponseJsonCapped(response, MAX_BYTES, request);
} catch (cause) {
// Out of time the feed reads unavailable, not the browser's own words
// ("signal timed out"); the caller's own abort stays an abort.
if (timeout.aborted && !signal?.aborted)
throw Object.assign(new Error(msg('News feed unavailable')), {
code: 'timeout',
unavailable: false,
retryable: true,
});
throw cause;
}
}
return {
// `hours` asks for counts over a shorter window than the server's own.
+6 -1
View File
@@ -176,7 +176,12 @@ test('reads time out and follow the caller signal', async () => {
);
});
const slow = createNewsSource({ fetchImpl: hang, timeoutMs: 20 });
await assert.rejects(slow.getHotspots(), { name: 'TimeoutError' });
// Out of time it reads unavailable, in the app's words, and is retried.
await assert.rejects(slow.getHotspots(), {
message: 'News feed unavailable',
code: 'timeout',
retryable: true,
});
const caller = new AbortController();
const pending = createNewsSource({ fetchImpl: hang }).getFeed(
{},