diff --git a/.env.example b/.env.example index 2b62f8871..342ed6720 100644 --- a/.env.example +++ b/.env.example @@ -317,6 +317,7 @@ GEMINI_CHAT_MODEL=gemini-3.5-flash-lite # Each reserves up to 40 stations within the existing 750-station directory. # Restart after changing. Leave blank for the existing global selection. # RADIO_PRIORITY_COUNTRIES=IN,PL + # Optional: a news server for the News layer (its /api/news/hotspots, /feed # and /outline routes), such as https://news.example. The local server # forwards only the layer's own GET requests there, with a 15 s timeout. Set, diff --git a/README.md b/README.md index d955ee57c..3a19b82a6 100644 --- a/README.md +++ b/README.md @@ -438,7 +438,7 @@ Twenty-two layers and map sources. **Twenty have a keyless path.** Some offer ad | 🚢 **Live Vessels** | Thousands of ships worldwide | AISStream | 🟡 | | 🛰️ **Satellites** | 838-object catalog, color-coded by class with a live legend — the **DENSE** chip drops in the whole Starlink shell | CelesTrak | 🟢 | | 🌍 **Earthquakes** | Global seismic activity, last 24h | USGS | 🟢 | -| 📰 **News** | The last 24 hours of news in the right-rail **NEWS** panel, narrowed to 12, 6, 3 or 1 hours, with picture cards and clips. **Conflicts**: physical violence worldwide from GDELT's event files, one report per story at GDELT's place for it, plus Liveuamap's mapped items with a key. **Local**: the news of the country at the view centre, its outline and name on the globe; a disputed area of the bundled list is named as itself. Places and categories are machine-coded and can be wrong | GDELT · Liveuamap · World News API, or your own news server | 🟢 Conflicts · 🟡 Local (World News API) · 🔴 Liveuamap items | +| 📰 **News** | The last 24 hours of news in the right-rail **NEWS** panel, narrowed to 12, 6, 3 or 1 hours, with picture cards and clips. **Conflicts**: physical violence worldwide from GDELT's event files, one report per story at GDELT's place for it, plus Liveuamap's mapped items with a key. **Local**: the news of the country at the view centre, its outline and name on the globe; a disputed area of the bundled list, or a territory without a country code, is named as itself. Places and categories are machine-coded and can be wrong | GDELT · Liveuamap · World News API, or your own news server | 🟢 Conflicts · 🟡 Local (World News API) · 🔴 Liveuamap items | | ◍ **Air quality** | Live AQHI on the published 1–10+ health-risk scale, with a Weather-panel card showing the band legend and the highest readings in view. **Canada only** — ECCC is the first registered provider | ECCC MSC GeoMet | 🟢 | | 🚗 **Traffic** | Simulated vehicles on OSM roads. With TomTom, live flow speeds drive the simulation and congestion colors below ~8 km; individual vehicle positions are not live observations | TomTom + OSM | 🟢 simulation · 🟡 live flow speeds | | 📹 **CCTV Mesh** | ~4,000 public cameras (shared cap) projected _into_ the 3D space — Austin · Texas (TxDOT) · California (Caltrans) · London (TfL) · Ontario (511, free developer key) · Finland (Fintraffic) · British Columbia (DriveBC) · Estonia (Tallinn, Tarktee) · Delaware (DelDOT live video) · New South Wales (Live Traffic NSW) · Calgary · Norway (Statens vegvesen) · Toronto · King County (WA) · Washington (WSDOT) · Catalonia (Servei Català de Trànsit) · Hong Kong (Transport Department) · Taiwan (Freeway Bureau, MOTC) · Italy (A22 Brennero motorway), plus opt-in New York City (NYC DOT TMC, needs your own NYC DOT agreement), opt-in Lithuania (Via Lietuva / Eismoinfo, photo licence unconfirmed), opt-in Florida (FL511, individual personal use only; other use needs FDOT's written consent) and opt-in Houston (TranStar, enable only with TxDOT's written permission per TranStar's Content Usage terms). Positions are published; poses are estimated priors **you calibrate by dragging a gizmo on the camera itself** | City APIs | 🟢 | diff --git a/changelog.d/0-news-layer.md b/changelog.d/0-news-layer.md index 6acd7e9d4..30d84d78f 100644 --- a/changelog.d/0-news-layer.md +++ b/changelog.d/0-news-layer.md @@ -18,7 +18,7 @@ (default 30). A section says when its key is required, refused or out of its daily allowance. The Data attribution popover credits the GDELT Project, World News API and, once its items show, Liveuamap - (AUTHOR, #1). + (AUTHOR, #0). - Cockpit Regional News holds its list to the last 24 hours, newest first, on the server and in the client, and labels its four states apart: fresh results with no banner and a `LAST 24 H` footer, earlier results kept after diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index ea0210440..a2976f83c 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -5850,11 +5850,11 @@ request viewbox and continues searching without that geographic preference. ### News (November 2026) - **What it shows.** Events ▸ **News**: the last 24 hours only, or the fewer hours a server names. Zoomed out, one beacon per hotspot with its count for the window and its trend, and a breathing beacon per developing situation (still under reduced motion). Below 2,500 km (7,500 km with a hotspot chosen), the hotspot outlines with disputed areas dotted and named, and every placed report as a ground-clamped marker: fill is severity (0 grey no physical event, 1 yellow minor, 2 orange significant, 3 red major), the icon is the category, the rim is attestation (solid confirmed, dashed claim, thin single), size grows with sources and the marker fades over the day. Colour never marks a side. Up to ten picture cards (physical events with two or more sources, major or pressing) stand over their markers without overlapping. A selected report's card reads "Source: host ↗" and opens the article. -- **Panel.** The right-rail NEWS panel (`src/ui/newsPanel.js`, `src/ui/news/`) has a time window at the top (the server's 24 h, or 12, 6, 3 or 1 h; the hotspot counts are read for the chosen window) and two sections. **Conflicts**: hotspot chips with counts, a severity floor (significant and above by default), a category filter, the chosen event, and a continuous feed, newest first. **Local**, while the camera is below 3,000 km: every report in the area at the view centre, named at the top ("News from …") and outlined on the globe, which then shows those reports instead of the hotspots' news; at sea, reports within 500 km. A disputed area of the bundled list (11, see `src/data/local_data/natural_earth/README.md`) is named as itself and read from no country's outlets; elsewhere Natural Earth's default country lines apply. Each card shows the publisher's picture when the event has one (hotlinked, no referrer, publisher credited), the headline (a model's English headline marked as such), a mono meta line (severity marker, category, place, sources, age), who reported it, attestation and media chips, and the service it came through ("via GDELT Project"). Open, a card plays the post's clip on request. A click flies the camera straight down from 150 km and selects the report. +- **Panel.** The right-rail NEWS panel (`src/ui/newsPanel.js`, `src/ui/news/`) has a time window at the top (the server's 24 h, or 12, 6, 3 or 1 h; the hotspot counts are read for the chosen window) and two sections. **Conflicts**: hotspot chips with counts, a severity floor (significant and above by default), a category filter, the chosen event, and a continuous feed, newest first. **Local**, while the camera is below 3,000 km: every report in the area at the view centre, named at the top ("News from …") and outlined on the globe, which then shows those reports instead of the hotspots' news; at sea, reports within 500 km. A disputed area of the bundled list (11, see `src/data/local_data/natural_earth/README.md`), or a territory Natural Earth gives no country code (Somaliland, Northern Cyprus and others), is named as itself and read from no country's outlets; elsewhere Natural Earth's default country lines apply. Each card shows the publisher's picture when the event has one (hotlinked, no referrer, publisher credited), the headline (a model's English headline marked as such), a mono meta line (severity marker, category, place, sources, age), who reported it, attestation and media chips, and the service it came through ("via GDELT Project"). Open, a card plays the post's clip on request. A click flies the camera straight down from 150 km and selects the report. - **Notices.** A section whose key the server lacks says "{provider} key required" and names the variable; a refused key says "{provider} refused its key"; a spent daily allowance says "{provider} limit reached for today". The other section keeps working. - **Source.** `sources.news` (`createNewsSource`, export `./layers/news/source`) reads same-origin `/api/news/*` with a 15 s timeout, a 4 MiB cap and record-by-record validation; `newsQueryAllowed` lets a host admit exactly the client's requests. Core's local server (`server/providers/news.js`) forwards only those GET requests to `GEV_NEWS_ORIGIN` when it is set. - **Built-in backend.** With `GEV_NEWS_ORIGIN` unset, Core answers the same routes itself (`server/providers/news/keyed.js`). **Conflicts** needs no key: the last 24 h of GDELT 2.0 event exports (15-minute files on `data.gdeltproject.org`, about 100 KB each), physical violence only (CAMEO roots 18–20), one report per story with copies of one agency story merged, placed at GDELT's location for it (a town the article's words name beats a bare country). Titles are the article's page title from GDELT's GKG export of the same quarter hour (about 6 MB each) for the newest 3 h, else the words of the article's address; GDELT's actor coding is never shown. With `LIVEUAMAP_API_KEY`, Liveuamap's mapped items stand beside GDELT's. **Local** reads World News API (`WORLDNEWSAPI_KEY`) by the source country at the view centre, once per country every 30 minutes (about 1.5 points a read, at most `WORLDNEWSAPI_DAILY_READS` reads a day, default 30), plus the Conflicts reports placed inside the area. Areas come from the bundled Natural Earth countries and disputed areas (`server/providers/news/areas.js`). Categories, relevance and severity are bundled rules (`categorize.js`); a news server may replace them. -- **State.** Share/restore token `6`, enabled-only. Voice is off for this layer (`VOICE_OFF_LAYERS`). +- **State.** Share/restore token `9`, enabled-only. Voice is off for this layer (`VOICE_OFF_LAYERS`). ### S2 GhostMaps (October 2026) diff --git a/docs/KNOWN-ISSUES.md b/docs/KNOWN-ISSUES.md index 3a87eb1b5..78d14c6e4 100644 --- a/docs/KNOWN-ISSUES.md +++ b/docs/KNOWN-ISSUES.md @@ -136,10 +136,11 @@ Status: Open (limits) - Copies of one agency story are merged, but the same event under different wording stays as separate reports. - Only the 11 disputed areas in the bundled list - (`src/data/local_data/natural_earth/README.md`) are named as themselves; - everywhere else Natural Earth's default country lines apply, so a place in - Western Sahara or Kashmir, for example, is named with the country those - lines put it in. GDELT places a whole city at one point: Jerusalem's falls + (`src/data/local_data/natural_earth/README.md`) and the territories Natural + Earth gives no country code (Somaliland, Northern Cyprus, Siachen Glacier, + the Spratly Islands and a few more) are named as themselves; everywhere else + Natural Earth's default country lines apply, so a place in Western Sahara, + for example, is named with the country those lines put it in. GDELT places a whole city at one point: Jerusalem's falls inside East Jerusalem, so it is named "Jerusalem" alone. - Local needs a World News API key. Its free plan allows 50 points a day; Core stops at `WORLDNEWSAPI_DAILY_READS` reads (default 30) and Local then says the diff --git a/server/providers/news/categorize.js b/server/providers/news/categorize.js index 4574246de..30ccafc22 100644 --- a/server/providers/news/categorize.js +++ b/server/providers/news/categorize.js @@ -8,7 +8,7 @@ // as the fallback. /** Text as the rules match it: folded case, one apostrophe, no vowel marks. */ -export function normaliseForMatch(text) { +function normaliseForMatch(text) { return String(text || '') .normalize('NFKC') .toLowerCase() @@ -553,11 +553,6 @@ const COMPILED = Object.entries(RULES).map(([category, stems]) => ({ }), })); -/** - * Categorise a report from its title and summary. - * @returns {{category: string, tags: string[]}} - */ -export const RELEVANCE_LEVELS = Object.freeze(['core', 'related', 'minor']); const EVENT_CATEGORIES = new Set([ 'strike', 'air-defence', @@ -771,7 +766,7 @@ const ELECTION = const PHYSICAL = /(? { @@ -511,13 +514,11 @@ export function createKeyedNews({ events = await fetchExport(url, { fetchImpl }); } catch (error) { failed = error; - // An unreadable file stays so. - if (error.unreadable) gdelt.gone.add(url); - else failing.push(url); + (error.unreadable ? unreadable : failing).push(url); return; } if (!events) { - if (at - exportAt(url) >= GDELT_GONE_MS) gdelt.gone.add(url); + if (at - exportAt(url) >= GDELT_GONE_MS) unpublished.push(url); return; } // The events count now; the titles follow (readTitles). @@ -531,12 +532,19 @@ export function createKeyedNews({ }), ); if (failed) log(`[news] gdelt read failed: ${failed.message}`); - // A file that fails while others read is that file's fault, and gets a - // few tries; when none read, GDELT itself is down and no file is blamed. + // A file is blamed only while GDELT itself answers (its list read, or + // another export of the round read): then a 404 long after its time is + // never published, an unreadable file stays so, and a failing one gets a + // few tries, one a GDELT_FRESH_MS. With GDELT down, no file is blamed. + const answering = added > 0 || latest !== null; + if (answering) + for (const url of [...unpublished, ...unreadable]) gdelt.gone.add(url); if (added) for (const url of failing) { - const tries = (gdelt.failures.get(url) || 0) + 1; - gdelt.failures.set(url, tries); + const prior = gdelt.failures.get(url); + if (prior && now() - prior.at < GDELT_FRESH_MS) continue; + const tries = (prior?.tries || 0) + 1; + gdelt.failures.set(url, { tries, at: now() }); if (tries >= GDELT_EXPORT_TRIES) gdelt.gone.add(url); } // Nothing read and a failure: tried again after RETRY_MS, not after diff --git a/server/providers/news/keyed.test.mjs b/server/providers/news/keyed.test.mjs index c7b6706e9..dfcb0bb7f 100644 --- a/server/providers/news/keyed.test.mjs +++ b/server/providers/news/keyed.test.mjs @@ -200,6 +200,8 @@ function upstream(answers = {}) { ); if (answers.corrupt?.includes(href)) return new Response('not a zip at all', { status: 200 }); + if (answers.failing?.includes(href)) + return new Response('busy', { status: 503 }); const gkg = href.endsWith('.gkg.csv.zip'); const listed = answers.exports ?? EXPORTS; const answer = gkg @@ -669,7 +671,12 @@ test('exports never published are given up on, not looked for back to back', asy const quarter = exportName('20261105114500'); const { ask, calls, clock, settle } = keyed({ env: {}, - answers: { exports: { [quarter]: EXPORTS[quarter] }, unlisted: 404 }, + // GDELT's list names the newest; every other export it answers 404. + answers: { + exports: { [quarter]: EXPORTS[quarter] }, + unlisted: 404, + lastupdate: '20261105120000', + }, }); assert.equal((await ask('/hotspots')).body.hotspots[0].counts.last24h, 3); await settle(); @@ -943,7 +950,7 @@ test('no export later than the newest GDELT lists is asked for (its CDN keeps a ); }); -test('a town a disputed area is named after is named alone, on both sides of its line', () => { +test('a town a disputed area is named after is named as the town alone', () => { assert.equal( gdeltPlace({ lat: 31.7667, @@ -986,3 +993,62 @@ test('GDELT down for a few minutes gives up on no export: they are all read once ), ); }); + +test('a GDELT outage answering 404, its list down too, gives up on no export', async () => { + const answers = { exports: {}, unlisted: 404 }; + const { ask, clock, settle } = keyed({ env: {}, answers }); + // Long enough for the stories' exports to be 90 minutes old. + for (let i = 0; i < 20; i++) { + await ask('/hotspots'); + await settle(); + clock.now += 5 * MINUTE; + } + delete answers.exports; + delete answers.unlisted; + for (let i = 0; i < 3; i++) { + clock.now += 5 * MINUTE; + await ask('/hotspots'); + await settle(); + } + const back = await ask('/hotspots'); + assert.equal(back.status, 200); + assert.equal(back.body.hotspots[0].counts.last24h, 4); +}); + +test('a host answering junk for every file gives up on none of them', async () => { + const answers = { + corrupt: [exportName('20261105114500'), exportName('20261105113000')], + exports: {}, + unlisted: 404, + }; + const { ask, clock, settle } = keyed({ env: {}, answers }); + await ask('/hotspots'); + await settle(); + answers.corrupt = []; + delete answers.exports; + delete answers.unlisted; + clock.now += 6 * MINUTE; + await ask('/hotspots'); + await settle(); + const back = await ask('/hotspots'); + assert.equal(back.body.hotspots[0].counts.last24h, 4); +}); + +test('a failing file gets one strike a round, not three within seconds', async () => { + const flaky = exportName('20261105114500'); + const answers = { failing: [flaky] }; + const { ask, calls, clock, settle } = keyed({ env: {}, answers }); + // The startup fill runs several rounds a few seconds apart. + await ask('/hotspots'); + await settle(); + // GDELT answers again for it before three rounds five minutes apart. + answers.failing = []; + clock.now += 6 * MINUTE; + await ask('/hotspots'); + await settle(); + const back = await ask('/hotspots'); + assert.equal(back.body.hotspots[0].counts.last24h, 4); + assert.ok( + gdeltCalls(calls).filter((c) => String(c.url) === flaky).length >= 2, + ); +}); diff --git a/src/data/layerStateTokenLedger.test.mjs b/src/data/layerStateTokenLedger.test.mjs index 04d25dce6..a1fc8c8e0 100644 --- a/src/data/layerStateTokenLedger.test.mjs +++ b/src/data/layerStateTokenLedger.test.mjs @@ -290,11 +290,7 @@ test('PR B manually replaces provisional 00 with 01 after PR A publishes 00', () test('allocation batches cross the last digit and base-36 pair boundaries in order', () => { // News holds the last digit, 9: the ledger has no free digit left. - const digitRows = []; - const allDigits = parseLayerStateTokenReservations([ - ...reservationRows, - ...digitRows, - ]); + const allDigits = parseLayerStateTokenReservations(reservationRows); const beforeNine = { ...allDigits }; delete beforeNine.news; assert.equal( @@ -342,24 +338,20 @@ test('allocation batches cross the last digit and base-36 pair boundaries in ord test('an isolated valid two-character fixture round-trips an l field beyond the old 64-character cap', async () => { const cwd = mkdtempSync(path.join(tmpdir(), 'gev-layer-token-width-')); try { - // News holds the last digit: no prior digit to fill. - const priorDigits = []; + // News holds the last digit: the fixture's layers start at pairs. const fixtureLayers = ['00', '01', '02', '03'].map((token) => ({ id: `qa-layer-${token}`, token, })); const fixtureRows = [ ...reservationRows, - ...priorDigits, ...fixtureLayers.map(({ id, token }) => [id, token]), ]; const fixtureModule = writeCodecFixture(cwd, fixtureRows, fixtureLayers); const codec = await import(pathToFileURL(fixtureModule)); assert.equal(codec.validateLayerStateRegistry(), true); assert.equal( - nextLayerStateToken( - parseLayerStateTokenReservations([...reservationRows, ...priorDigits]), - ), + nextLayerStateToken(parseLayerStateTokenReservations(reservationRows)), '00', ); assert.equal(