From 81f49406b80f2f4c8076ed739360f79ab2e2c34f Mon Sep 17 00:00:00 2001 From: Wilmer Alzate Date: Wed, 30 Sep 2026 12:31:59 -0500 Subject: [PATCH 01/14] perf(flights): skip dead-reckoning contacts that stay behind the horizon The fleet tick dead-reckoned, floored and distance-tested every contact on every ~80 ms tick, then hid the ones behind the horizon. The worldwide feed carries ~13k contacts; zoomed in, well under 1% are on screen, so nearly all of that work produced a hidden billboard. A contact that is already hidden and still behind the horizon at its last position now re-reckons only on its turn in a staggered one-in-12-ticks rotation (about once a second, HIDDEN_CONTACT_DR_TICKS). Anything that position puts in view, and any contact with a fleet model, takes the full pass immediately, so on-screen behaviour is unchanged. Hidden-inclusive proximity queries (getNearby includeHidden, getAllPositions) read positions at most about a second old, well inside the 30 s poll's uncertainty. Measured in Chrome on a Radeon 780M, ~12.9k contacts, 56 on screen over Colombia at 4 km, same view before and after: fleet tick 7.66 -> 2.03 ms/frame, longest tick 50.9 -> ~16 ms frame 20.6 -> 14.0 ms, 42.9 -> 57.8 FPS Teleporting the camera to Madrid showed 975 contacts within 100 ms. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 7 ++++ docs/CURRENT-STATE.md | 2 +- scripts/format-scope.json | 1 + .../flights/hiddenContactRotation.test.mjs | 42 +++++++++++++++++++ src/layers/flights/policy.js | 31 ++++++++++++++ src/layers/flights/rendering.js | 24 +++++++++++ src/layers/flights/state.js | 4 ++ 7 files changed, 110 insertions(+), 1 deletion(-) create mode 100644 src/layers/flights/hiddenContactRotation.test.mjs diff --git a/CHANGELOG.md b/CHANGELOG.md index 58103d04d..4cc44d40c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,12 @@ # Changelog +- Stop dead-reckoning every flight hidden behind the horizon on every fleet + tick. With the worldwide feed (~13k contacts, ~56 on screen over Colombia at + 4 km), the fleet pass drops from 7.7 ms to 2.0 ms per frame and its longest + tick from 51 ms to about 16 ms; the scene goes from 43 to 58 FPS. Hidden + contacts re-reckon about once a second, staggered, and appear as soon as + their position is in view. + - Public Overpass instances are no longer used by default. Street Traffic roads come from TomTom flow tiles, OpenFreeMap vector tiles, or both, chosen on the layer row (TomTom / OSM / Hybrid) or with `?trafficRoads=`. With a diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 22ce1eb3b..ee4269a63 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -3469,7 +3469,7 @@ silently demoting every later lookup for the session. ### Motion & Symbology Correctness (June 10, 2026) -- **Flights (commercial + military)** render one poll interval behind real time (30s/15s) and interpolate between two known feed-stamped fixes (OpenSky `time_position`; adsb.lol `receipt − seen_pos`). The display latency is an intentional product decision — do not "fix" it away. The whole fleet dead-reckons at ~12Hz (1m² write gating); aircraft get a 3-poll grace period (faded icon) before removal. When a position epoch pauses, both layers coast for at least 60 seconds of contact grace with an absolute five-minute ceiling. Source backoff marks each contact and the cockpit `STALE`; the cockpit then holds the exact layer position instead of continuing inertial flight. Repeated-position kinematic changes create a forward-only synthetic fix rather than mutating history, and grounded history is lifted only when no owned 3D model already controls its datum. A nominally successful worldwide OpenSky response whose own snapshot epoch is more than two minutes old prefers the existing 250-nm viewport-scoped adsb.lol fallback and labels the source/coverage accordingly when that upstream is available. If the fallback is also unavailable, the layer's freshness/error fields use the source epoch—never the cache receipt time—so the UI reports an old snapshot rather than “just now.” +- **Flights (commercial + military)** render one poll interval behind real time (30s/15s) and interpolate between two known feed-stamped fixes (OpenSky `time_position`; adsb.lol `receipt − seen_pos`). The display latency is an intentional product decision — do not "fix" it away. The fleet dead-reckons at ~12Hz (1m² write gating); a contact already hidden behind the horizon whose last position still is re-reckons on a staggered one-in-12-ticks rotation (about once a second) instead, so the worldwide feed's ~13k contacts no longer cost a full pass when well under 1% are on screen. Anything that position puts in view, and any contact with a fleet model, takes the full pass immediately, and hidden-inclusive proximity queries read positions at most about a second old; aircraft get a 3-poll grace period (faded icon) before removal. When a position epoch pauses, both layers coast for at least 60 seconds of contact grace with an absolute five-minute ceiling. Source backoff marks each contact and the cockpit `STALE`; the cockpit then holds the exact layer position instead of continuing inertial flight. Repeated-position kinematic changes create a forward-only synthetic fix rather than mutating history, and grounded history is lifted only when no owned 3D model already controls its datum. A nominally successful worldwide OpenSky response whose own snapshot epoch is more than two minutes old prefers the existing 250-nm viewport-scoped adsb.lol fallback and labels the source/coverage accordingly when that upstream is available. If the fallback is also unavailable, the layer's freshness/error fields use the source epoch—never the cache receipt time—so the UI reports an old snapshot rather than “just now.” - **World-space headings at every angle** (`src/data/iconOrientation.js`): aircraft/vessel icon rotation uses the camera right/up basis per tick (alignedAxis always ZERO), which is exact at screen center and for orthographic/nadir views and remains stable through >180° tracked orbits. Perspective rays vary across the viewport, so off-center contacts at oblique pitch can diverge from an exact finite-difference window projection; a regression test pins that known regime, and field evidence decides whether to adopt exact projection with the basis method as fallback. Fleet rotations refresh on camera-pose change; tracked entities per frame. Billboards are horizon-culled via a shared EllipsoidalOccluder. - **Military/OpenSky reconciliation** (`src/data/militaryRegistry.js`): known-military ICAOs render amber in the flights layer (60s self-poll of the cached mil endpoint when the military layer is off) and are suppressed there while the military layer renders them. - **Satellites**: 838-sat core catalog (stations/visual/GPS/GLONASS/Galileo/GEO), tracking lands ~726km out via `viewFrom` (tracked entity owns a point graphic so the tracking camera engages), rings realign via primitive modelMatrix (no per-second rebuild flicker), optional `setParams({catalog:'dense'})` Starlink mode. diff --git a/scripts/format-scope.json b/scripts/format-scope.json index cd836a5cb..a12d5124f 100644 --- a/scripts/format-scope.json +++ b/scripts/format-scope.json @@ -308,6 +308,7 @@ "src/layers/flights/tracking.js", "src/layers/flights/rendering.js", "src/layers/flights/testing.js", + "src/layers/flights/hiddenContactRotation.test.mjs", "src/layers/flights/evidence.js", "src/layers/flights/ingestion.js", "src/layers/flights/queries.js", diff --git a/src/layers/flights/hiddenContactRotation.test.mjs b/src/layers/flights/hiddenContactRotation.test.mjs new file mode 100644 index 000000000..23e81c45b --- /dev/null +++ b/src/layers/flights/hiddenContactRotation.test.mjs @@ -0,0 +1,42 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { + FLEET_DR_INTERVAL_MS, + HIDDEN_CONTACT_DR_INTERVAL_MS, + HIDDEN_CONTACT_DR_TICKS, + hiddenContactDrPhase, +} from './policy.js'; + +// Contacts hidden behind the horizon re-reckon on one fleet tick per rotation +// instead of every tick. Their positions still feed hidden-inclusive proximity +// queries, so the rotation must reach every contact about once a second, and +// it must spread them across ticks so no single tick carries the whole fleet. + +const icao = (n) => n.toString(16).padStart(6, '0'); + +test('the rotation revisits each hidden contact about once a second', () => { + const periodMs = HIDDEN_CONTACT_DR_TICKS * FLEET_DR_INTERVAL_MS; + assert.ok(HIDDEN_CONTACT_DR_TICKS > 1); + assert.ok( + Math.abs(periodMs - HIDDEN_CONTACT_DR_INTERVAL_MS) <= FLEET_DR_INTERVAL_MS, + ); +}); + +test('a contact keeps the same slot, inside the rotation', () => { + for (let n = 0; n < 500; n++) { + const phase = hiddenContactDrPhase(icao(n * 7919)); + assert.ok(Number.isInteger(phase)); + assert.ok(phase >= 0 && phase < HIDDEN_CONTACT_DR_TICKS); + assert.equal(hiddenContactDrPhase(icao(n * 7919)), phase); + } +}); + +test('a worldwide fleet is spread evenly enough that no tick carries a burst', () => { + const fleet = 13000; + const perSlot = new Array(HIDDEN_CONTACT_DR_TICKS).fill(0); + // Real ICAO24 addresses cluster by national block; sample several blocks. + for (let n = 0; n < fleet; n++) + perSlot[hiddenContactDrPhase(icao(0x0a0000 + n * 53))]++; + const fair = fleet / HIDDEN_CONTACT_DR_TICKS; + for (const count of perSlot) assert.ok(count < fair * 1.5, `${count} > 1.5×`); +}); diff --git a/src/layers/flights/policy.js b/src/layers/flights/policy.js index f8358c98f..04313c367 100644 --- a/src/layers/flights/policy.js +++ b/src/layers/flights/policy.js @@ -200,6 +200,37 @@ export const FLEET_DR_INTERVAL_MS = 80; export const ROTATION_REFRESH_MS = 1000; +/** + * @constant {number} Dead-reckoning cadence (ms) for contacts hidden behind the + * horizon. The worldwide feed carries ~13k contacts and a zoomed-in view shows + * well under 1% of them; the rest still need positions for hidden-inclusive + * proximity queries, but ~1 s old (≈250 m for an airliner) is well inside the + * 30 s poll's own uncertainty. Each hidden contact is re-reckoned on one tick in + * every HIDDEN_CONTACT_DR_TICKS, staggered by id so no tick carries them all. + */ + +export const HIDDEN_CONTACT_DR_INTERVAL_MS = 1000; + +export const HIDDEN_CONTACT_DR_TICKS = Math.max( + 1, + Math.round(HIDDEN_CONTACT_DR_INTERVAL_MS / FLEET_DR_INTERVAL_MS), +); + +/** + * Stable slot in [0, HIDDEN_CONTACT_DR_TICKS) for a contact id, so each hidden + * contact re-reckons on the same tick of every rotation. + * @param {string} id Contact id (ICAO24). + * @returns {number} + */ + +export function hiddenContactDrPhase(id) { + const text = String(id); + let hash = 0; + for (let i = 0; i < text.length; i++) + hash = (hash * 31 + text.charCodeAt(i)) >>> 0; + return hash % HIDDEN_CONTACT_DR_TICKS; +} + /** Max course slew (deg/s) — well above real turns (≤4°/s), hides fix-boundary * steps. Scaled down toward COURSE_MIN_DPS at low speed (courseSlewCapDps). */ diff --git a/src/layers/flights/rendering.js b/src/layers/flights/rendering.js index 44e1b02c3..2048128dd 100644 --- a/src/layers/flights/rendering.js +++ b/src/layers/flights/rendering.js @@ -33,6 +33,8 @@ import { COURSE_SLEW_DT_MAX_SEC, ROTATION_REFRESH_MS, COURSE_MAX_DPS, + HIDDEN_CONTACT_DR_TICKS, + hiddenContactDrPhase, } from './policy.js'; export function createRendering({ @@ -811,6 +813,12 @@ export function createRendering({ } } + /** The contact's hidden-rotation slot, cached on its billboard. */ + function _hiddenDrPhase(bb, icao24) { + bb._gevHiddenDrPhase ??= hiddenContactDrPhase(icao24); + return bb._gevHiddenDrPhase; + } + function _fleetTick() { if ( !flightState._viewer || @@ -931,9 +939,25 @@ export function createRendering({ for (const icao of toRelease) _releaseModel(icao); } + const hiddenDrPhase = flightState._fleetTickSeq++ % HIDDEN_CONTACT_DR_TICKS; + for (const [icao24, bb] of flightState._billboards) { if (icao24 === flightState._trackedIcao) continue; // tracked entity owns its own motion + // A contact still behind the horizon at its last position would only be + // re-reckoned to be hidden again, so off its turn it keeps that position + // (see HIDDEN_CONTACT_DR_TICKS). Anything the stale position puts in view, + // and any contact with a fleet model, takes the full pass below at once. + if ( + !bb.show && + _hiddenDrPhase(bb, icao24) !== hiddenDrPhase && + !flightState._models.has(icao24) && + !occluder.isPointVisible( + flightState._cullPositions.get(icao24) || bb.position, + ) + ) + continue; + const info = flightState.records.data.get(icao24); const dr = parts.motion._deadReckon(icao24, flightState._scratchFleetPos); diff --git a/src/layers/flights/state.js b/src/layers/flights/state.js index 2978ceba4..d1adec064 100644 --- a/src/layers/flights/state.js +++ b/src/layers/flights/state.js @@ -204,6 +204,10 @@ export function createFlightState({ source, services }) { flightState._lastFleetTickMs = 0; + /** @type {number} Fleet ticks run; selects which hidden contacts re-reckon */ + + flightState._fleetTickSeq = 0; + /** @type {string} Camera pose signature at the last rotation pass */ flightState._lastCamPoseSig = ''; From aa745bcee5c4f76353c9426fb9055cb5737f0fc0 Mon Sep 17 00:00:00 2001 From: Trygve Hemminghytt <274629477+trygvehhh@users.noreply.github.com> Date: Wed, 7 Oct 2026 04:02:05 +0200 Subject: [PATCH 02/14] data: refresh Norway datacenters from Geofabrik extract Add a maintainer script that merges a pinned, MD5-verified Geofabrik Norway extract (osmium tags-filter telecom/building=data_center) into the bundled datacenter layer. Only features inside the Geofabrik Norway boundary are touched; unchanged features keep their original line. norway-261005 (OSM 2026-10-05) adds Troll Housing (w1551029155) and updates tags on four existing features. Provenance is recorded in the dataset README and the bundled count moves from 4,351 to 4,352. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 6 + DATA_SOURCES.md | 2 +- README.md | 2 +- docs/INFRASTRUCTURE-LAYERS.md | 2 +- scripts/build-datacenters-norway.py | 196 ++++++++++++++++++ src/data/infrastructure.test.mjs | 2 +- src/data/local_data/datacenters/README.md | 16 +- .../datacenters/datacenters.geojsonl | 9 +- 8 files changed, 226 insertions(+), 9 deletions(-) create mode 100644 scripts/build-datacenters-norway.py diff --git a/CHANGELOG.md b/CHANGELOG.md index b433f8c06..031e36751 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,12 @@ ## [Unreleased] +- Refresh Norway's datacenters from a Geofabrik OpenStreetMap extract + (2026-10-05). Troll Housing joins the layer (4,352 features) and four + Norwegian sites pick up current OSM tags; no site moved or was removed. + `scripts/build-datacenters-norway.py` reproduces or `--verify`s the merge, + touching only features inside Geofabrik's Norway boundary, and the dataset + README records the extract, checksum and filter. - Add Norway's road cameras from Statens vegvesen as a CCTV pack. The camera list is one keyless GeoJSON request to the agency's OGC API view of its DATEX 3.1 CCTV table, so no DATEX account is needed. Every working camera diff --git a/DATA_SOURCES.md b/DATA_SOURCES.md index 734e35a69..b4f0dc0cc 100644 --- a/DATA_SOURCES.md +++ b/DATA_SOURCES.md @@ -112,7 +112,7 @@ The [Bhote Koshi event pack](public/events/bhote-koshi-2026/README.md), under `p | Dataset | Folder | License | Commercial use? | Attribution | | --------------------------------------------------------------------------- | --------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | --------------------------------------------------------------------------- | -| **Datacenters** (4,351) | `datacenters/` | **ODbL 1.0** (OpenStreetMap extract) | ✅ (attribution + share-alike on data) | © OpenStreetMap contributors (shared credit) | +| **Datacenters** (4,352) | `datacenters/` | **ODbL 1.0** (OpenStreetMap extract) | ✅ (attribution + share-alike on data) | © OpenStreetMap contributors (shared credit) | | **Dams** (704) | `dams/` | **ODbL 1.0** (OpenInfraMap / OSM extract) | ✅ (attribution + share-alike on data) | © OpenStreetMap contributors (shared credit) + Open Infrastructure Map | | **Military area names** (36,466) | `osm_military_names/` | **ODbL 1.0** (OpenStreetMap, via Overture Maps) | ✅ (attribution + share-alike on data) | © OpenStreetMap contributors (shared credit) + Overture Maps Foundation | | **TeleGeography Submarine Cable Map** (712 cables + 1,917 landing points) | `telegeography_submarine_cables/` | **CC BY-NC-SA 3.0** | ❌ **NonCommercial — remove for commercial use** | "© TeleGeography — submarinecablemap.com" | diff --git a/README.md b/README.md index 724ea18cb..0500645af 100644 --- a/README.md +++ b/README.md @@ -323,7 +323,7 @@ In the app, **Data Layers** groups them as Movement, Cameras, Infrastructure, Ev _The Space Missions layer replaying a Falcon 9 ascent — labeled `RECONSTRUCTED ESTIMATE`, scrubbable 0.25×–4×._ -**Also on the globe:** neighborhood overlays · an optional cockpit WX cloud effect. **Bundled static infrastructure:** Datacenters (4,351), Dams (704), and Submarine Cables (712). +**Also on the globe:** neighborhood overlays · an optional cockpit WX cloud effect. **Bundled static infrastructure:** Datacenters (4,352), Dams (704), and Submarine Cables (712). ![Diving into the Bahamas and revealing labeled submarine cable routes beneath the globe](docs/media/09-undersea-cables.gif) diff --git a/docs/INFRASTRUCTURE-LAYERS.md b/docs/INFRASTRUCTURE-LAYERS.md index 76567e605..1166a9e69 100644 --- a/docs/INFRASTRUCTURE-LAYERS.md +++ b/docs/INFRASTRUCTURE-LAYERS.md @@ -58,6 +58,6 @@ factory and supplies its existing functions. Other consumers should use the pack exports, which do not import standalone application globals. Dataset files and source/license notices remain under `src/data/local_data/`. -Their input counts are 4,351 datacenter features and 704 dam features. Cesium may +Their input counts are 4,352 datacenter features and 704 dam features. Cesium may expand multipart geometries into multiple entities; entity and stem counts are different measurements. This change does not refresh or relicense the datasets. diff --git a/scripts/build-datacenters-norway.py b/scripts/build-datacenters-norway.py new file mode 100644 index 000000000..01a891a8b --- /dev/null +++ b/scripts/build-datacenters-norway.py @@ -0,0 +1,196 @@ +#!/usr/bin/env python3 +"""Maintainer-only Norway refresh of the bundled datacenter extract (osmium-tool). + +Downloads a pinned Geofabrik Norway extract, verifies its MD5, filters +telecom=data_center / building=data_center with osmium, and merges the result +into datacenters.geojsonl: features inside the Geofabrik Norway boundary are +added, updated or removed; every other line is kept byte-for-byte. +""" +import argparse +import hashlib +import json +import math +from pathlib import Path +import shutil +import subprocess +import urllib.request + +ROOT = Path(__file__).resolve().parent.parent +EXTRACT = 'norway-261005' +GEOFABRIK = 'https://download.geofabrik.de/europe/' +DATASET = ROOT / 'src/data/local_data/datacenters/datacenters.geojsonl' +CONTACT_TAGS = {'email', 'phone', 'fax', 'mobile', 'whatsapp', 'contact:email', 'contact:phone', + 'contact:fax', 'contact:mobile', 'contact:whatsapp'} +SAME_GEOMETRY_METERS = 0.05 + + +def fetch(url, target): + if not target.exists(): + with urllib.request.urlopen(url) as response, open(str(target) + '.part', 'wb') as out: + shutil.copyfileobj(response, out) + Path(str(target) + '.part').rename(target) + return target + + +def md5(path): + digest = hashlib.md5() + with open(path, 'rb') as handle: + for block in iter(lambda: handle.read(1 << 20), b''): + digest.update(block) + return digest.hexdigest() + + +def read_poly(path): + """Geofabrik .poly -> list of (ring, is_hole).""" + rings, ring, hole = [], None, False + for raw in path.read_text().splitlines()[1:]: + line = raw.strip() + if ring is None: + if line and line != 'END': + ring, hole = [], line.startswith('!') + elif line == 'END': + rings.append((ring, hole)) + ring = None + elif line: + lon, lat = (float(v) for v in line.split()) + ring.append((lon, lat)) + return rings + + +def inside(rings, lon, lat): + def in_ring(ring): + hit = False + for (x1, y1), (x2, y2) in zip(ring, ring[1:] + ring[:1]): + if (y1 > lat) != (y2 > lat) and lon < x1 + (lat - y1) * (x2 - x1) / (y2 - y1): + hit = not hit + return hit + return any(in_ring(r) for r, h in rings if not h) and not any(in_ring(r) for r, h in rings if h) + + +def points(coordinates): + if isinstance(coordinates[0], (int, float)): + return [coordinates] + return [p for part in coordinates for p in points(part)] + + +def osm_key(feature): + osm_id = feature['properties']['osm_id'] + if osm_id < 0: + return 'r%d' % -osm_id + return ('n' if feature['geometry']['type'] == 'Point' else 'w') + str(osm_id) + + +def export(osmium, pbf, cache): + filtered = cache / (EXTRACT + '-datacenters.osm.pbf') + exported = cache / (EXTRACT + '-datacenters.geojsonseq') + subprocess.run([osmium, 'tags-filter', '-O', '-o', str(filtered), str(pbf), + 'nwr/telecom=data_center', 'nwr/building=data_center'], check=True) + subprocess.run([osmium, 'export', '-O', '-f', 'geojsonseq', '--add-unique-id=type_id', + '-o', str(exported), str(filtered)], check=True) + fresh = {} + for line in exported.read_text().splitlines(): + line = line.lstrip('\x1e').strip() + if not line: + continue + feature = json.loads(line) + tags = {k: v for k, v in feature['properties'].items() if k not in CONTACT_TAGS} + if tags.get('telecom') != 'data_center' and tags.get('building') != 'data_center': + continue # untagged member ways exported only for geometry + key = feature['id'] + geometry = feature['geometry'] + if key[0] == 'a': # osmium area id: way * 2 or relation * 2 + 1 + number = int(key[1:]) + key = ('w%d' % (number // 2)) if number % 2 == 0 else ('r%d' % (number // 2)) + if key[0] == 'r': + tags = {**tags, 'type': 'multipolygon'} + if geometry['type'] == 'MultiPolygon' and len(geometry['coordinates']) == 1: + geometry = {'type': 'Polygon', 'coordinates': geometry['coordinates'][0]} + elif key in fresh or geometry['type'] == 'LineString': + continue # closed ways are taken from their area export + fresh[key] = {'geometry': geometry, 'tags': tags} + return fresh + + +def same_geometry(old, new): + """Same type and vertex set within tolerance; ring start and winding may differ.""" + if old['type'] != new['type']: + return False + a, b = points(old['coordinates']), points(new['coordinates']) + if len(a) != len(b): + return False + + def meters(p, q): + dx = (p[0] - q[0]) * 111320 * math.cos(math.radians(p[1])) + return math.hypot(dx, (p[1] - q[1]) * 110540) + return all(min(meters(p, q) for q in b) <= SAME_GEOMETRY_METERS for p in a) and \ + all(min(meters(q, p) for p in a) <= SAME_GEOMETRY_METERS for q in b) + + +def feature_line(key, sequence, fresh): + osm_id = int(key[1:]) * (-1 if key[0] == 'r' else 1) + feature = {'id': osm_id, 'type': 'Feature', 'geometry': fresh['geometry'], + 'properties': {'id': sequence, 'tags': fresh['tags'], 'type': 'data_center', + 'osm_id': osm_id}} + return json.dumps(feature, separators=(',', ':'), ensure_ascii=False) + + +def merge(lines, fresh, rings): + out, seen, sequence = [], set(), 0 + report = {'kept': 0, 'updated': [], 'removed': [], 'added': []} + for line in lines: + feature = json.loads(line) + sequence = max(sequence, feature['properties']['id']) + key = osm_key(feature) + if key in fresh: + seen.add(key) + old_tags = feature['properties']['tags'] + if old_tags == fresh[key]['tags'] and same_geometry(feature['geometry'], fresh[key]['geometry']): + out.append(line) + report['kept'] += 1 + else: + out.append(feature_line(key, feature['properties']['id'], fresh[key])) + report['updated'].append(key) + continue + lon, lat = points(feature['geometry']['coordinates'])[0][:2] + if inside(rings, lon, lat): + report['removed'].append(key) + continue + out.append(line) + for key in sorted(set(fresh) - seen, key=lambda k: (k[0], int(k[1:]))): + sequence += 1 + out.append(feature_line(key, sequence, fresh[key])) + report['added'].append(key) + if len({osm_key(json.loads(line)) for line in out}) != len(out): + raise ValueError('Duplicate OSM identity') + return out, report + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--cache', type=Path, required=True) + parser.add_argument('--osmium', default=shutil.which('osmium') or 'osmium') + parser.add_argument('--verify', action='store_true') + args = parser.parse_args() + args.cache.mkdir(parents=True, exist_ok=True) + pbf = fetch(GEOFABRIK + EXTRACT + '.osm.pbf', args.cache / (EXTRACT + '.osm.pbf')) + expected = fetch(GEOFABRIK + EXTRACT + '.osm.pbf.md5', + args.cache / (EXTRACT + '.osm.pbf.md5')).read_text().split()[0] + actual = md5(pbf) + if actual != expected: + raise RuntimeError('MD5 mismatch for %s: %s != %s' % (pbf.name, actual, expected)) + rings = read_poly(fetch(GEOFABRIK + 'norway.poly', args.cache / 'norway.poly')) + fresh = export(args.osmium, pbf, args.cache) + current = [line for line in DATASET.read_text().splitlines() if line.strip()] + merged, report = merge(current, fresh, rings) + payload = '\n'.join(merged) + '\n' + if args.verify: + if payload != DATASET.read_text(): + raise SystemExit('datacenters.geojsonl is not the %s merge' % EXTRACT) + else: + DATASET.write_text(payload) + print(json.dumps({'extract': EXTRACT, 'md5': actual, 'norway': len(fresh), + 'features': len(merged), **report})) + + +if __name__ == '__main__': + main() diff --git a/src/data/infrastructure.test.mjs b/src/data/infrastructure.test.mjs index 45fb3d317..c7f1b5a23 100644 --- a/src/data/infrastructure.test.mjs +++ b/src/data/infrastructure.test.mjs @@ -82,7 +82,7 @@ test('infrastructure factory preserves identity and creates independent state wi test('dataset URLs still name the complete bundled sources', () => { for (const [file, count] of [ - ['datacenters', 4351], + ['datacenters', 4352], ['dams', 704], ]) { const lines = readFileSync( diff --git a/src/data/local_data/datacenters/README.md b/src/data/local_data/datacenters/README.md index 454a15007..ef0720ae4 100644 --- a/src/data/local_data/datacenters/README.md +++ b/src/data/local_data/datacenters/README.md @@ -5,7 +5,7 @@ layer. - Source: OpenStreetMap contributors - License: Open Database License (ODbL) 1.0 -- Feature count: 4,351 +- Feature count: 4,352 - Runtime file: `datacenters.geojsonl` The public-release snapshot removes contact-oriented tags such as email, phone, @@ -16,3 +16,17 @@ ODbL 1.0 with the required OpenStreetMap attribution. The original extraction date and query were not recorded alongside this snapshot. Future refreshes should record both before replacing the file. + +## Norway refresh + +Features inside the Geofabrik Norway boundary (`norway.poly`, which includes +Svalbard) were refreshed separately, so this file mixes two vintages: Norway +reflects OSM as of the extract below, everything else the original snapshot. + +- Extract: Geofabrik `norway-261005.osm.pbf`, OSM data + 2026-10-05T20:21:35Z, MD5 `1d2cfdeb98980fce11b8891d0f9146c0` +- Filter: `osmium tags-filter nwr/telecom=data_center nwr/building=data_center` +- Contact tags are dropped as above; unchanged features keep their original + line, so geometry is only rewritten when tags or vertices actually changed. +- Rebuild or check: `python3 scripts/build-datacenters-norway.py --cache ` + (add `--verify` to check without writing; requires `osmium-tool`) diff --git a/src/data/local_data/datacenters/datacenters.geojsonl b/src/data/local_data/datacenters/datacenters.geojsonl index f7f5fc89f..b02bd57b3 100644 --- a/src/data/local_data/datacenters/datacenters.geojsonl +++ b/src/data/local_data/datacenters/datacenters.geojsonl @@ -3770,15 +3770,15 @@ {"id":939340882,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[21.021364183,52.315373019],[21.02232433,52.315435883],[21.022363641,52.315209656],[21.021403578,52.315146875],[21.021364183,52.315373019]]]},"properties":{"id":11407,"tags":{"name":"T-Mobile Warszawa-Szlachecka","telecom":"data_center","building":"office","building:levels":"3"},"type":"data_center","osm_id":939340882}} {"id":1005973518,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[21.021325375,52.315595978],[21.022285857,52.315658423],[21.02232433,52.315435883],[21.021364183,52.315373019],[21.021325375,52.315595978]]]},"properties":{"id":11976,"tags":{"telecom":"data_center","building":"office","building:levels":"3"},"type":"data_center","osm_id":1005973518}} {"id":1156823776,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[21.115130857,52.235385942],[21.115189866,52.236349777],[21.115232027,52.236667954],[21.115818174,52.237174556],[21.116175578,52.237468677],[21.117262124,52.236985628],[21.11728157,52.236976576],[21.117262795,52.236960985],[21.117340495,52.236926033],[21.117360276,52.2369433],[21.117447532,52.236904491],[21.11804097,52.236654459],[21.118030996,52.236645742],[21.118041054,52.236641719],[21.116603055,52.235624072],[21.116518984,52.235676375],[21.115986231,52.235696659],[21.115954966,52.235361551],[21.115130857,52.235385942]]]},"properties":{"id":15342,"tags":{"name":"Atman Data Center Warsaw-1","telecom":"data_center","website":"https://www.atman.pl"},"type":"data_center","osm_id":1156823776}} -{"id":7405586142,"type":"Feature","geometry":{"type":"Point","coordinates":[5.598350034,58.860111293]},"properties":{"id":24849,"tags":{"name":"Maan Group","telecom":"data_center","operator":"Privat"},"type":"data_center","osm_id":7405586142}} +{"id":7405586142,"type":"Feature","geometry":{"type":"Point","coordinates":[5.5983501,58.8601113]},"properties":{"id":24849,"tags":{"name":"Maan Group","opening_hours":"24/7","operator":"Privat","telecom":"data_center"},"type":"data_center","osm_id":7405586142}} {"id":481615983,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[9.572510149,56.481777951],[9.574069938,56.484921751],[9.57465357,56.48483349],[9.574639572,56.484805243],[9.574700676,56.484795939],[9.574714925,56.484824857],[9.575298557,56.484736595],[9.57454888,56.483225422],[9.574609229,56.483216369],[9.574510155,56.483016629],[9.574449722,56.483025849],[9.573738769,56.481592795],[9.573155137,56.481680972],[9.573170643,56.481712321],[9.573109539,56.481721625],[9.573093781,56.48168969],[9.572510149,56.481777951]]]},"properties":{"id":7881,"tags":{"telecom":"data_center","building":"data_center","operator":"Apple","wikidata":"Q19721053","wikipedia":"en:Foulum Data Center","description":"Apple Datacenter"},"type":"data_center","osm_id":481615983}} {"id":618756435,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[10.500360092,57.318668738],[10.500399236,57.318671672],[10.500382891,57.318734871],[10.500552038,57.318744175],[10.500545249,57.318787677],[10.500501495,57.318785666],[10.500458999,57.318957746],[10.500970295,57.31899647],[10.501009187,57.318847189],[10.501084959,57.318852888],[10.50111966,57.318719197],[10.500843896,57.318698159],[10.500865521,57.318615178],[10.500859235,57.318614759],[10.500869461,57.318575699],[10.500954537,57.318582153],[10.501022347,57.31832223],[10.500736691,57.318300437],[10.500669133,57.318559689],[10.500674162,57.318560025],[10.500664272,57.318598498],[10.500383981,57.318577124],[10.500360092,57.318668738]]]},"properties":{"id":8842,"tags":{"ref":"AAL01-01","name":"Prime Data Centers - Saeby Campus AAL01","telecom":"data_center","website":"https://primedatacenters.com/locations/saeby/","building":"industrial","operator":"Prime Data Centers"},"type":"data_center","osm_id":618756435}} -{"id":-14367602,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[11.002243225,59.633471864],[11.003903261,59.633446551],[11.007046726,59.633399696],[11.006983443,59.632420941],[11.006982772,59.63227057],[11.006911023,59.632265289],[11.006852936,59.632253973],[11.006765848,59.632228576],[11.00671782,59.632205275],[11.006119687,59.632213573],[11.006119352,59.632202592],[11.006118933,59.63219245],[11.005930424,59.632194378],[11.005923635,59.632120869],[11.005980799,59.632120031],[11.005979542,59.632104524],[11.005977027,59.63206865],[11.00597225,59.631998242],[11.005412171,59.632035541],[11.005286945,59.632043839],[11.00527848,59.632044342],[11.003707962,59.632147272],[11.003107483,59.63219178],[11.002741864,59.632235533],[11.002729375,59.63223788],[11.002345987,59.632309294],[11.002308352,59.632316251],[11.002302066,59.632477351],[11.002277088,59.632913881],[11.002243225,59.633471864]]]},"properties":{"id":107,"tags":{"name":"STACK Infrastructure OSL04","type":"multipolygon","telecom":"data_center","website":"https://www.stackinfra.com","operator":"Stack Infrastructure","operator:wikidata":"Q115477173"},"type":"data_center","osm_id":-14367602}} +{"id":-14367602,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[11.0022433,59.6334719],[11.0022771,59.6329139],[11.0023021,59.6324774],[11.0023084,59.6323163],[11.002346,59.6323093],[11.0027294,59.6322379],[11.0027419,59.6322356],[11.0031075,59.6321918],[11.003708,59.6321473],[11.0052785,59.6320444],[11.005287,59.6320439],[11.0054122,59.6320356],[11.0059723,59.6319983],[11.0059771,59.6320687],[11.0059796,59.6321046],[11.0059808,59.6321201],[11.0059237,59.6321209],[11.0059305,59.6321944],[11.006119,59.6321925],[11.0061194,59.6322026],[11.0061197,59.6322136],[11.0067179,59.6322053],[11.0067659,59.6322286],[11.006853,59.632254],[11.0069111,59.6322653],[11.0069828,59.6322706],[11.0069835,59.632421],[11.0070468,59.6333997],[11.0039033,59.6334466],[11.0022433,59.6334719]]]},"properties":{"id":107,"tags":{"landuse":"industrial","name":"STACK Infrastructure OSL04","operator":"Stack Infrastructure","operator:wikidata":"Q115477173","telecom":"data_center","website":"https://www.stackinfra.com","type":"multipolygon"},"type":"data_center","osm_id":-14367602}} {"id":1028687251,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[11.004111886,59.632578437],[11.004142983,59.633037262],[11.004275166,59.633034999],[11.004274998,59.63303173],[11.00505133,59.633019157],[11.005056443,59.633099372],[11.005224416,59.63309669],[11.005190637,59.632562344],[11.005060885,59.632564439],[11.005060969,59.632567457],[11.004587978,59.632575419],[11.00458538,59.632536527],[11.004459819,59.632538623],[11.004462418,59.632576844],[11.004242476,59.6325807],[11.004242141,59.632576174],[11.004111886,59.632578437]]]},"properties":{"id":12226,"tags":{"building":"data_center","ref:bygningsnr":"300783262"},"type":"data_center","osm_id":1028687251}} {"id":1164840308,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[11.002600126,59.632599392],[11.002631223,59.633058217],[11.00276349,59.633055954],[11.002763238,59.633052685],[11.00353957,59.633040196],[11.003544683,59.633120327],[11.00371274,59.633117645],[11.003678961,59.632583382],[11.003549042,59.632585478],[11.003549293,59.632588495],[11.003076135,59.632596458],[11.00307362,59.632557566],[11.002948143,59.632559661],[11.002950657,59.632597883],[11.002730716,59.632601571],[11.002730465,59.632597129],[11.002600126,59.632599392]]]},"properties":{"id":15459,"tags":{"building":"data_center","ref:bygningsnr":"300783262"},"type":"data_center","osm_id":1164840308}} {"id":1164840309,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[11.005613672,59.632723444],[11.005644769,59.633182269],[11.005776951,59.633179922],[11.005776784,59.633176737],[11.006553199,59.633164164],[11.006558229,59.633244463],[11.006726286,59.633241781],[11.006692423,59.632707351],[11.006562671,59.632709446],[11.006562839,59.632712464],[11.006089764,59.632720426],[11.006087166,59.632681534],[11.005961772,59.632683797],[11.005964287,59.632721851],[11.005744262,59.632725623],[11.005743927,59.632721181],[11.005613672,59.632723444]]]},"properties":{"id":15460,"tags":{"building":"data_center","ref:bygningsnr":"300783262"},"type":"data_center","osm_id":1164840309}} -{"id":935092287,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[10.991467366,59.759024056],[10.992780978,59.759406942],[10.992773937,59.759423957],[10.992948197,59.759474248],[10.993073842,59.759363859],[10.993022293,59.759348939],[10.993291688,59.759112653],[10.993128827,59.759065547],[10.993181885,59.759019027],[10.993083062,59.758990445],[10.992976947,59.759083484],[10.992845519,59.759045346],[10.992813584,59.759073342],[10.9927876,59.759065798],[10.993409789,59.758524243],[10.99263002,59.758297094],[10.991938848,59.758898663],[10.991693594,59.758827165],[10.991467366,59.759024056]]]},"properties":{"id":11377,"tags":{"name":"Green Mountain Ytre Enebakk","telecom":"data_center","website":"https://greenmountain.no/","building":"warehouse","ref:bygningsnr":"300752944"},"type":"data_center","osm_id":935092287}} -{"id":1161764012,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[10.992826324,59.757448761],[10.994530281,59.757601396],[10.994634888,59.757303168],[10.992928332,59.757152545],[10.992826324,59.757448761]]]},"properties":{"id":15426,"tags":{"name":"Green Mountain Ytre Enebakk","telecom":"data_center","website":"https://greenmountain.no/","building":"industrial"},"type":"data_center","osm_id":1161764012}} +{"id":935092287,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[10.9914674,59.7590241],[10.9916936,59.7588272],[10.9919389,59.7588987],[10.9926301,59.7582971],[10.9934098,59.7585243],[10.9927876,59.7590658],[10.9928136,59.7590734],[10.9928456,59.7590454],[10.992977,59.7590835],[10.9930831,59.7589905],[10.9931819,59.7590191],[10.9931289,59.7590656],[10.9932917,59.7591127],[10.9930223,59.759349],[10.9930739,59.7593639],[10.9929482,59.7594743],[10.992774,59.759424],[10.992781,59.759407],[10.9914674,59.7590241]]]},"properties":{"id":11377,"tags":{"building":"warehouse","name":"Green Mountain Ytre Enebakk","ref:bygningsnr":"300752944","source":"Norway Orthophoto 2024;https://greenmountain.no/data-center/osl1-enebakk/","telecom":"data_center","website":"https://greenmountain.no/"},"type":"data_center","osm_id":935092287}} +{"id":1161764012,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[10.9928264,59.7574488],[10.9929284,59.7571526],[10.9946349,59.7573032],[10.9945303,59.7576014],[10.9928264,59.7574488]]]},"properties":{"id":15426,"tags":{"building":"industrial","name":"Green Mountain Ytre Enebakk","source":"Norway Orthophoto 2024;https://greenmountain.no/data-center/osl1-enebakk/","telecom":"data_center","website":"https://greenmountain.no/"},"type":"data_center","osm_id":1161764012}} {"id":30724580,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[10.808790995,59.924984985],[10.808893925,59.925031923],[10.80893793,59.92500594],[10.808982941,59.925025972],[10.809042955,59.92501298],[10.809083943,59.925031923],[10.809012948,59.925077856],[10.80926583,59.925189755],[10.80955115,59.925025972],[10.809676543,59.925081293],[10.81032899,59.924701928],[10.809890365,59.924508138],[10.80982155,59.924515347],[10.809691631,59.924457847],[10.808820667,59.924557592],[10.809099365,59.924699581],[10.808806418,59.924865291],[10.808918149,59.924914828],[10.808875988,59.924937962],[10.808790995,59.924984985]]]},"properties":{"id":261,"tags":{"name":"Stack Infrastructure OSLO1A","telecom":"data_center","building":"office","operator":"Stack Infrastructure","ref:bygningsnr":"80877293","building:levels":"4","operator:wikidata":"Q115477173"},"type":"data_center","osm_id":30724580}} {"id":80871917,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[10.83408892,59.938588981],[10.834305927,59.938720996],[10.834575992,59.938720996],[10.834574987,59.938736],[10.834581943,59.938736922],[10.834579932,59.938771958],[10.834736925,59.938771958],[10.834734997,59.93893792],[10.836717988,59.938944961],[10.836717988,59.938983937],[10.836817984,59.938983937],[10.83681899,59.938944961],[10.836844974,59.938944961],[10.836846985,59.938774976],[10.836841956,59.938774976],[10.836842962,59.938744968],[10.836849919,59.938744968],[10.836851931,59.938653941],[10.836683957,59.938652935],[10.836684125,59.938640278],[10.836686975,59.938414973],[10.83490297,59.938406926],[10.834899953,59.938646984],[10.834812949,59.938645978],[10.83481496,59.938551933],[10.834579932,59.938550927],[10.834580938,59.938564925],[10.834567946,59.938564925],[10.834567946,59.938551933],[10.834089926,59.938549922],[10.83408892,59.938588981]]]},"properties":{"id":1464,"tags":{"name":"Bulk Data Centers Oslo Internet Exchange","telecom":"data_center","website":"https://bulkinfrastructure.com/data-centers/locations/os-ix","building":"industrial","ref:bygningsnr":"80709943"},"type":"data_center","osm_id":80871917}} {"id":12158409495,"type":"Feature","geometry":{"type":"Point","coordinates":[9.689363435,62.59439936]},"properties":{"id":28723,"tags":{"name":"VisBook AS","telecom":"data_center","website":"https://visbook.com/"},"type":"data_center","osm_id":12158409495}} @@ -4349,3 +4349,4 @@ {"id":540895898,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[101.71747909,56.303855963],[101.71782015,56.304420065],[101.721113819,56.303806929],[101.720772759,56.303242994],[101.71747909,56.303855963]]]},"properties":{"id":8271,"tags":{"name":"Битривер Рус","name:en":"Bitriver Rus","name:ru":"Битривер Рус","telecom":"data_center","website":"https://bitriver.farm/","building":"data_center"},"type":"data_center","osm_id":540895898}} {"id":1082803064,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[123.948021193,47.340264299],[123.948354625,47.340268239],[123.948490663,47.340269831],[123.948488568,47.340349292],[123.948809762,47.340353064],[123.948811942,47.340267485],[123.948918643,47.340268658],[123.948922918,47.340137817],[123.948023959,47.340126082],[123.948021193,47.340264299]]]},"properties":{"id":13377,"tags":{"name":"联通云数据中心·齐齐哈尔","name:en":"Unicom Data Center of Qiqihar","name:zh":"联通云数据中心·齐齐哈尔","telecom":"data_center","building":"yes","name:zh-Hans":"联通云数据中心·齐齐哈尔","name:zh-Hant":"聯通雲數據中心·齊齊哈爾"},"type":"data_center","osm_id":1082803064}} {"id":442479059,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[142.76019226,46.910096898],[142.760192427,46.910325724],[142.760447656,46.910328155],[142.760460397,46.910102263],[142.76019226,46.910096898]]]},"properties":{"id":7471,"tags":{"telecom":"data_center","building":"yes"},"type":"data_center","osm_id":442479059}} +{"id":1551029155,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[7.1069147,62.8627625],[7.1071207,62.8621617],[7.1075917,62.8616616],[7.1088647,62.8620476],[7.1106308,62.862645],[7.1098655,62.8632827],[7.1089604,62.8630042],[7.1086734,62.8632156],[7.1069147,62.8627625]]]},"properties":{"id":29471,"tags":{"layer":"-1","loc_name":"Troll Housing Klempertåsen","name":"Troll Housing","operator":"Troll Housing AS","telecom":"data_center","website":"https://www.trollhousing.no/"},"type":"data_center","osm_id":1551029155}} From 5b5d942d95dddd56b3bacb465c69098913688d1f Mon Sep 17 00:00:00 2001 From: Trygve Hemminghytt <274629477+trygvehhh@users.noreply.github.com> Date: Fri, 16 Oct 2026 22:23:02 +0200 Subject: [PATCH 03/14] fix(data): make the Norway datacenter script portable and re-runnable Read and write files as UTF-8 with LF line endings so the script and --verify work on Windows, where the default codec is cp1252 (thanks to quanticsoul4772 for the report). Geofabrik keeps only its newest dated extract, so --extract now defaults to the current one and accepts a dated name for --verify while it is listed; the dataset README says so. Downloads time out instead of hanging. --- CHANGELOG.md | 1 + scripts/build-datacenters-norway.py | 52 +++++++++++++++-------- src/data/local_data/datacenters/README.md | 8 +++- 3 files changed, 42 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 20a9da4e6..067ca16b6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -184,6 +184,7 @@ `scripts/build-datacenters-norway.py` reproduces or `--verify`s the merge, touching only features inside Geofabrik's Norway boundary, and the dataset README records the extract, checksum and filter. + - Add a keyless Air quality layer to the Weather group, with Environment and Climate Change Canada's AQHI as the first registered provider. Readings carry `{ provider, scale, value, band }` so another network can be added without diff --git a/scripts/build-datacenters-norway.py b/scripts/build-datacenters-norway.py index 01a891a8b..687395f7b 100644 --- a/scripts/build-datacenters-norway.py +++ b/scripts/build-datacenters-norway.py @@ -1,10 +1,13 @@ #!/usr/bin/env python3 """Maintainer-only Norway refresh of the bundled datacenter extract (osmium-tool). -Downloads a pinned Geofabrik Norway extract, verifies its MD5, filters +Downloads a dated Geofabrik Norway extract, verifies its MD5, filters telecom=data_center / building=data_center with osmium, and merges the result into datacenters.geojsonl: features inside the Geofabrik Norway boundary are added, updated or removed; every other line is kept byte-for-byte. + +Geofabrik keeps only its newest dated extract, so --extract defaults to the +current one and a recorded extract can be re-verified only while it is listed. """ import argparse import hashlib @@ -16,17 +19,28 @@ import subprocess import urllib.request ROOT = Path(__file__).resolve().parent.parent -EXTRACT = 'norway-261005' GEOFABRIK = 'https://download.geofabrik.de/europe/' DATASET = ROOT / 'src/data/local_data/datacenters/datacenters.geojsonl' CONTACT_TAGS = {'email', 'phone', 'fax', 'mobile', 'whatsapp', 'contact:email', 'contact:phone', 'contact:fax', 'contact:mobile', 'contact:whatsapp'} SAME_GEOMETRY_METERS = 0.05 +TIMEOUT_SECONDS = 60 + + +def latest_extract(): + """Resolve norway-latest to the dated name Geofabrik redirects it to.""" + request = urllib.request.Request(GEOFABRIK + 'norway-latest.osm.pbf', method='HEAD') + with urllib.request.urlopen(request, timeout=TIMEOUT_SECONDS) as response: + name = response.geturl().rsplit('/', 1)[-1] + if not name.startswith('norway-') or not name.endswith('.osm.pbf') or name == 'norway-latest.osm.pbf': + raise RuntimeError('Unexpected Geofabrik latest target: %s' % name) + return name[:-len('.osm.pbf')] def fetch(url, target): if not target.exists(): - with urllib.request.urlopen(url) as response, open(str(target) + '.part', 'wb') as out: + with urllib.request.urlopen(url, timeout=TIMEOUT_SECONDS) as response, \ + open(str(target) + '.part', 'wb') as out: shutil.copyfileobj(response, out) Path(str(target) + '.part').rename(target) return target @@ -43,7 +57,7 @@ def md5(path): def read_poly(path): """Geofabrik .poly -> list of (ring, is_hole).""" rings, ring, hole = [], None, False - for raw in path.read_text().splitlines()[1:]: + for raw in path.read_text(encoding='utf-8').splitlines()[1:]: line = raw.strip() if ring is None: if line and line != 'END': @@ -80,15 +94,15 @@ def osm_key(feature): return ('n' if feature['geometry']['type'] == 'Point' else 'w') + str(osm_id) -def export(osmium, pbf, cache): - filtered = cache / (EXTRACT + '-datacenters.osm.pbf') - exported = cache / (EXTRACT + '-datacenters.geojsonseq') +def export(osmium, pbf, cache, extract): + filtered = cache / (extract + '-datacenters.osm.pbf') + exported = cache / (extract + '-datacenters.geojsonseq') subprocess.run([osmium, 'tags-filter', '-O', '-o', str(filtered), str(pbf), 'nwr/telecom=data_center', 'nwr/building=data_center'], check=True) subprocess.run([osmium, 'export', '-O', '-f', 'geojsonseq', '--add-unique-id=type_id', '-o', str(exported), str(filtered)], check=True) fresh = {} - for line in exported.read_text().splitlines(): + for line in exported.read_text(encoding='utf-8').splitlines(): line = line.lstrip('\x1e').strip() if not line: continue @@ -169,26 +183,30 @@ def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument('--cache', type=Path, required=True) parser.add_argument('--osmium', default=shutil.which('osmium') or 'osmium') + parser.add_argument('--extract', default='latest', + help="dated Geofabrik name such as norway-261005, or 'latest'") parser.add_argument('--verify', action='store_true') args = parser.parse_args() args.cache.mkdir(parents=True, exist_ok=True) - pbf = fetch(GEOFABRIK + EXTRACT + '.osm.pbf', args.cache / (EXTRACT + '.osm.pbf')) - expected = fetch(GEOFABRIK + EXTRACT + '.osm.pbf.md5', - args.cache / (EXTRACT + '.osm.pbf.md5')).read_text().split()[0] + extract = latest_extract() if args.extract == 'latest' else args.extract + pbf = fetch(GEOFABRIK + extract + '.osm.pbf', args.cache / (extract + '.osm.pbf')) + expected = fetch(GEOFABRIK + extract + '.osm.pbf.md5', + args.cache / (extract + '.osm.pbf.md5')).read_text(encoding='utf-8').split()[0] actual = md5(pbf) if actual != expected: raise RuntimeError('MD5 mismatch for %s: %s != %s' % (pbf.name, actual, expected)) rings = read_poly(fetch(GEOFABRIK + 'norway.poly', args.cache / 'norway.poly')) - fresh = export(args.osmium, pbf, args.cache) - current = [line for line in DATASET.read_text().splitlines() if line.strip()] + fresh = export(args.osmium, pbf, args.cache, extract) + current = [line for line in DATASET.read_text(encoding='utf-8').splitlines() if line.strip()] merged, report = merge(current, fresh, rings) payload = '\n'.join(merged) + '\n' if args.verify: - if payload != DATASET.read_text(): - raise SystemExit('datacenters.geojsonl is not the %s merge' % EXTRACT) + if payload != DATASET.read_text(encoding='utf-8'): + raise SystemExit('datacenters.geojsonl is not the %s merge' % extract) else: - DATASET.write_text(payload) - print(json.dumps({'extract': EXTRACT, 'md5': actual, 'norway': len(fresh), + with open(DATASET, 'w', encoding='utf-8', newline='\n') as handle: + handle.write(payload) + print(json.dumps({'extract': extract, 'md5': actual, 'norway': len(fresh), 'features': len(merged), **report})) diff --git a/src/data/local_data/datacenters/README.md b/src/data/local_data/datacenters/README.md index ef0720ae4..8b803dcc7 100644 --- a/src/data/local_data/datacenters/README.md +++ b/src/data/local_data/datacenters/README.md @@ -28,5 +28,9 @@ reflects OSM as of the extract below, everything else the original snapshot. - Filter: `osmium tags-filter nwr/telecom=data_center nwr/building=data_center` - Contact tags are dropped as above; unchanged features keep their original line, so geometry is only rewritten when tags or vertices actually changed. -- Rebuild or check: `python3 scripts/build-datacenters-norway.py --cache ` - (add `--verify` to check without writing; requires `osmium-tool`) +- Refresh: `python3 scripts/build-datacenters-norway.py --cache ` uses + Geofabrik's current Norway extract and prints the dated name it used; record + that name, its OSM timestamp and MD5 here. Requires `osmium-tool`. +- Check: add `--extract --verify` to compare without writing. + Geofabrik keeps only its newest dated extract, so a recorded extract can be + re-verified only while it is still listed; the MD5 above identifies it. From 4318f51de4114bfb68137ff48bedd070b976f771 Mon Sep 17 00:00:00 2001 From: Trygve Hemminghytt <274629477+trygvehhh@users.noreply.github.com> Date: Sun, 18 Oct 2026 20:03:23 +0200 Subject: [PATCH 04/14] docs: move the changelog entry to changelog.d --- CHANGELOG.md | 7 ------- changelog.d/953-datacenters-norway-refresh.md | 6 ++++++ 2 files changed, 6 insertions(+), 7 deletions(-) create mode 100644 changelog.d/953-datacenters-norway-refresh.md diff --git a/CHANGELOG.md b/CHANGELOG.md index a285bc555..98d32365f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -284,13 +284,6 @@ ## [Unreleased] -- Refresh Norway's datacenters from a Geofabrik OpenStreetMap extract - (2026-10-05). Troll Housing joins the layer (4,352 features) and four - Norwegian sites pick up current OSM tags; no site moved or was removed. - `scripts/build-datacenters-norway.py` reproduces or `--verify`s the merge, - touching only features inside Geofabrik's Norway boundary, and the dataset - README records the extract, checksum and filter. - - Add a keyless Air quality layer to the Weather group, with Environment and Climate Change Canada's AQHI as the first registered provider. Readings carry `{ provider, scale, value, band }` so another network can be added without diff --git a/changelog.d/953-datacenters-norway-refresh.md b/changelog.d/953-datacenters-norway-refresh.md new file mode 100644 index 000000000..48a019605 --- /dev/null +++ b/changelog.d/953-datacenters-norway-refresh.md @@ -0,0 +1,6 @@ +- Refresh Norway's datacenters from a Geofabrik OpenStreetMap extract + (2026-10-05). Troll Housing joins the layer (4,352 features) and four + Norwegian sites pick up current OSM tags; no site moved or was removed. + `scripts/build-datacenters-norway.py` reproduces or `--verify`s the merge, + touching only features inside Geofabrik's Norway boundary, and the dataset + README records the extract, checksum and filter (trygvehhh, #953). From 04d626a4b190149ffcb017290c5d9aa17d32748d Mon Sep 17 00:00:00 2001 From: German Khokhlov Date: Tue, 3 Nov 2026 15:02:42 +0400 Subject: [PATCH 05/14] fix(datacenters): preserve missing geometry and compare ring topology --- scripts/build-datacenters-norway.py | 64 ++++++++++++--- src/data/datacentersNorwayRefresh.test.mjs | 93 ++++++++++++++++++++++ src/data/local_data/datacenters/README.md | 4 +- 3 files changed, 148 insertions(+), 13 deletions(-) create mode 100644 src/data/datacentersNorwayRefresh.test.mjs diff --git a/scripts/build-datacenters-norway.py b/scripts/build-datacenters-norway.py index 687395f7b..03286b864 100644 --- a/scripts/build-datacenters-norway.py +++ b/scripts/build-datacenters-norway.py @@ -4,7 +4,8 @@ Downloads a dated Geofabrik Norway extract, verifies its MD5, filters telecom=data_center / building=data_center with osmium, and merges the result into datacenters.geojsonl: features inside the Geofabrik Norway boundary are -added, updated or removed; every other line is kept byte-for-byte. +added, updated or removed once no longer tagged; every other line is kept +byte-for-byte. Geofabrik keeps only its newest dated extract, so --extract defaults to the current one and a recorded extract can be re-verified only while it is listed. @@ -16,6 +17,7 @@ import math from pathlib import Path import shutil import subprocess +import sys import urllib.request ROOT = Path(__file__).resolve().parent.parent @@ -95,10 +97,15 @@ def osm_key(feature): def export(osmium, pbf, cache, extract): + """Exported features by key, and the keys of every tagged object in the extract.""" filtered = cache / (extract + '-datacenters.osm.pbf') + listed = cache / (extract + '-datacenters.opl') exported = cache / (extract + '-datacenters.geojsonseq') subprocess.run([osmium, 'tags-filter', '-O', '-o', str(filtered), str(pbf), 'nwr/telecom=data_center', 'nwr/building=data_center'], check=True) + subprocess.run([osmium, 'tags-filter', '-O', '-R', '-f', 'opl', '-o', str(listed), str(filtered), + 'nwr/telecom=data_center', 'nwr/building=data_center'], check=True) + tagged = {line.split(' ', 1)[0] for line in listed.read_text(encoding='utf-8').splitlines() if line} subprocess.run([osmium, 'export', '-O', '-f', 'geojsonseq', '--add-unique-id=type_id', '-o', str(exported), str(filtered)], check=True) fresh = {} @@ -122,22 +129,48 @@ def export(osmium, pbf, cache, extract): elif key in fresh or geometry['type'] == 'LineString': continue # closed ways are taken from their area export fresh[key] = {'geometry': geometry, 'tags': tags} - return fresh + return fresh, tagged def same_geometry(old, new): - """Same type and vertex set within tolerance; ring start and winding may differ.""" + """Same type and rings within tolerance; ring start, winding and part or hole order may differ.""" if old['type'] != new['type']: return False - a, b = points(old['coordinates']), points(new['coordinates']) - if len(a) != len(b): - return False def meters(p, q): dx = (p[0] - q[0]) * 111320 * math.cos(math.radians(p[1])) return math.hypot(dx, (p[1] - q[1]) * 110540) - return all(min(meters(p, q) for q in b) <= SAME_GEOMETRY_METERS for p in a) and \ - all(min(meters(q, p) for p in a) <= SAME_GEOMETRY_METERS for q in b) + + def same_line(a, b): + return len(a) == len(b) and all(meters(p, q) <= SAME_GEOMETRY_METERS for p, q in zip(a, b)) + + def same_ring(a, b): + a, b = a[:-1], b[:-1] # closed rings repeat their first vertex + return len(a) == len(b) and \ + any(same_line(a[i:] + a[:i], c) for c in (b, b[::-1]) for i in range(len(a))) + + def same_parts(a, b, same): + rest = list(b) + for part in a: + match = next((i for i, other in enumerate(rest) if same(part, other)), None) + if match is None: + return False + rest.pop(match) + return not rest + + def same_polygon(a, b): + return same_ring(a[0], b[0]) and same_parts(a[1:], b[1:], same_ring) + + a, b = old['coordinates'], new['coordinates'] + if old['type'] == 'Point': + return same_line([a], [b]) + if old['type'] == 'LineString': + return same_line(a, b) + if old['type'] == 'Polygon': + return same_polygon(a, b) + if old['type'] == 'MultiPolygon': + return same_parts(a, b, same_polygon) + return a == b def feature_line(key, sequence, fresh): @@ -148,9 +181,9 @@ def feature_line(key, sequence, fresh): return json.dumps(feature, separators=(',', ':'), ensure_ascii=False) -def merge(lines, fresh, rings): +def merge(lines, fresh, rings, tagged): out, seen, sequence = [], set(), 0 - report = {'kept': 0, 'updated': [], 'removed': [], 'added': []} + report = {'kept': 0, 'updated': [], 'removed': [], 'added': [], 'unexported': []} for line in lines: feature = json.loads(line) sequence = max(sequence, feature['properties']['id']) @@ -165,6 +198,10 @@ def merge(lines, fresh, rings): out.append(feature_line(key, feature['properties']['id'], fresh[key])) report['updated'].append(key) continue + if key in tagged: # still tagged, but osmium built no usable geometry: keep the old line + report['unexported'].append(key) + out.append(line) + continue lon, lat = points(feature['geometry']['coordinates'])[0][:2] if inside(rings, lon, lat): report['removed'].append(key) @@ -196,9 +233,12 @@ def main(): if actual != expected: raise RuntimeError('MD5 mismatch for %s: %s != %s' % (pbf.name, actual, expected)) rings = read_poly(fetch(GEOFABRIK + 'norway.poly', args.cache / 'norway.poly')) - fresh = export(args.osmium, pbf, args.cache, extract) + fresh, tagged = export(args.osmium, pbf, args.cache, extract) current = [line for line in DATASET.read_text(encoding='utf-8').splitlines() if line.strip()] - merged, report = merge(current, fresh, rings) + merged, report = merge(current, fresh, rings, tagged) + if report['unexported']: + print('warning: kept the old line of tagged features osmium did not export: %s' + % ', '.join(report['unexported']), file=sys.stderr) payload = '\n'.join(merged) + '\n' if args.verify: if payload != DATASET.read_text(encoding='utf-8'): diff --git a/src/data/datacentersNorwayRefresh.test.mjs b/src/data/datacentersNorwayRefresh.test.mjs new file mode 100644 index 000000000..2b7b7d5bf --- /dev/null +++ b/src/data/datacentersNorwayRefresh.test.mjs @@ -0,0 +1,93 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { fileURLToPath } from 'node:url'; + +const SCRIPT = fileURLToPath( + new URL('../../scripts/build-datacenters-norway.py', import.meta.url), +); + +function python(t, source) { + const result = spawnSync('python3', ['-c', source, SCRIPT], { + encoding: 'utf8', + }); + if (result.error?.code === 'ENOENT') return t.skip('python3 is not installed'); + assert.equal(result.status, 0, result.stderr); +} + +test('Norway refresh compares rings, not vertex sets', (t) => { + python( + t, + String.raw` +import runpy, sys +same = runpy.run_path(sys.argv[1])['same_geometry'] +poly = lambda *rings: {'type': 'Polygon', 'coordinates': list(rings)} +multi = lambda *polys: {'type': 'MultiPolygon', 'coordinates': [p['coordinates'] for p in polys]} +a, b, c, d, e = [0, 0], [4, 0], [4, 4], [0, 4], [2, 1] +assert not same(poly([a, b, c, d, e, a]), poly([a, b, e, c, d, a])), 'reordered edges' +assert same(poly([a, b, c, d, e, a]), poly([c, b, a, e, d, c])), 'ring start and winding' +assert same(poly([a, b, c, d, a]), poly([[0, 1e-7], b, c, d, [0, 1e-7]])), 'tolerance' +outer = [[0, 0], [9, 0], [9, 9], [0, 9], [0, 0]] +one, two = [[1, 1], [2, 1], [2, 2], [1, 1]], [[5, 5], [6, 5], [6, 6], [5, 5]] +assert same(poly(outer, one, two), poly(outer, two, one)), 'hole order' +assert not same(poly(outer, one), poly(one, outer)), 'outer and inner roles swapped' +assert same(multi(poly(one), poly(two)), multi(poly(two), poly(one))), 'part order' +p, q, r, s, u, v = [0, 0], [1, 0], [0, 1], [5, 5], [6, 5], [5, 6] +assert not same(multi(poly([p, q, r, p]), poly([s, u, v, s])), + multi(poly([p, q, s, p]), poly([r, u, v, r]))), 'vertices moved between parts' +`, + ); +}); + +test('Norway refresh keeps a still-tagged feature osmium did not export', (t) => { + python( + t, + String.raw` +import json, runpy, sys +merge = runpy.run_path(sys.argv[1])['merge'] +square = [[[10, 60], [10.01, 60], [10.01, 60.01], [10, 60.01], [10, 60]]] +def line(osm_id, sequence): + return json.dumps({'id': osm_id, 'type': 'Feature', 'geometry': {'type': 'Polygon', 'coordinates': square}, + 'properties': {'id': sequence, 'tags': {'building': 'data_center'}, + 'type': 'data_center', 'osm_id': osm_id}}) +norway = [([(0, 0), (20, 0), (20, 80), (0, 80)], False)] +lines = [line(7, 1), line(8, 2)] +out, report = merge(lines, {}, norway, {'w7'}) +assert out == [lines[0]], out +assert report['unexported'] == ['w7'] and report['removed'] == ['w8'], report +`, + ); +}); + +test('Norway refresh lists tagged objects apart from the exported geometry', (t) => { + python( + t, + String.raw` +import json, runpy, subprocess, sys, tempfile +from pathlib import Path +export = runpy.run_path(sys.argv[1])['export'] +tags = {'building': 'data_center'} +ring = [[10, 60], [10.01, 60], [10.01, 60.01], [10, 60]] +features = [ + {'id': 'n1', 'properties': {'telecom': 'data_center'}, 'geometry': {'type': 'Point', 'coordinates': [10, 60]}}, + {'id': 'w5', 'properties': tags, 'geometry': {'type': 'LineString', 'coordinates': ring}}, + {'id': 'a10', 'properties': tags, 'geometry': {'type': 'MultiPolygon', 'coordinates': [[ring]]}}, + {'id': 'w7', 'properties': tags, 'geometry': {'type': 'LineString', 'coordinates': ring[:3]}}, +] +outputs = { + '.opl': 'n1 v1 Ttelecom=data_center x10 y60\nw5 v1 Tbuilding=data_center Nn2,n3,n4,n2\n' + 'w7 v1 Tbuilding=data_center Nn2,n3,n4\nr9 v1 Ttelecom=data_center Mw8@outer\n', + '.geojsonseq': ''.join(chr(30) + json.dumps({'type': 'Feature', **f}) + '\n' for f in features), +} +def run(args, check=False): + target = Path(args[args.index('-o') + 1]) + target.write_text(outputs.get(target.suffix, ''), encoding='utf-8') +subprocess.run = run +with tempfile.TemporaryDirectory() as cache: + fresh, tagged = export('osmium', Path(cache) / 'norway-test.osm.pbf', Path(cache), 'norway-test') +assert sorted(fresh) == ['n1', 'w5'], fresh +assert fresh['w5']['geometry'] == {'type': 'Polygon', 'coordinates': [ring]}, fresh['w5'] +assert tagged == {'n1', 'w5', 'w7', 'r9'}, tagged +`, + ); +}); diff --git a/src/data/local_data/datacenters/README.md b/src/data/local_data/datacenters/README.md index 8b803dcc7..5a80df267 100644 --- a/src/data/local_data/datacenters/README.md +++ b/src/data/local_data/datacenters/README.md @@ -27,7 +27,9 @@ reflects OSM as of the extract below, everything else the original snapshot. 2026-10-05T20:21:35Z, MD5 `1d2cfdeb98980fce11b8891d0f9146c0` - Filter: `osmium tags-filter nwr/telecom=data_center nwr/building=data_center` - Contact tags are dropped as above; unchanged features keep their original - line, so geometry is only rewritten when tags or vertices actually changed. + line, so geometry is only rewritten when tags or rings actually changed. + A feature is removed only once it is no longer tagged; one still tagged but + without an exportable geometry keeps its line and is reported as unexported. - Refresh: `python3 scripts/build-datacenters-norway.py --cache ` uses Geofabrik's current Norway extract and prints the dated name it used; record that name, its OSM timestamp and MD5 here. Requires `osmium-tool`. From 2c936ae131ed8b24abfc45cc861c8de75df50061 Mon Sep 17 00:00:00 2001 From: Said Turksever Date: Tue, 3 Nov 2026 16:26:45 +0000 Subject: [PATCH 06/14] docs: correct hidden-contact rotation interval --- docs/CURRENT-STATE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 5be325222..2a6dc5fdb 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -4989,7 +4989,7 @@ silently demoting every later lookup for the session. ### Motion & Symbology Correctness (June 10, 2026) -- **Flights (commercial + military)** render one poll interval behind real time (30s/15s) and interpolate between two known feed-stamped fixes (OpenSky `time_position`; adsb.lol `receipt − seen_pos`). The display latency is an intentional product decision — do not "fix" it away. The fleet dead-reckons at ~12Hz (1m² write gating); a contact already hidden behind the horizon whose last position still is re-reckons on a staggered one-in-12-ticks rotation (about once a second) instead, so the worldwide feed's ~13k contacts no longer cost a full pass when well under 1% are on screen. Anything that position puts in view, and any contact with a fleet model, takes the full pass immediately, and hidden-inclusive proximity queries read positions at most about a second old; aircraft get a 3-poll grace period (faded icon) before removal. When a position epoch pauses, both layers coast for at least 60 seconds of contact grace with an absolute five-minute ceiling. Source backoff marks each contact and the cockpit `STALE`; the cockpit then holds the exact layer position instead of continuing inertial flight. Repeated-position kinematic changes create a forward-only synthetic fix rather than mutating history, and grounded history is lifted only when no owned 3D model already controls its datum. A nominally successful worldwide OpenSky response whose own snapshot epoch is more than two minutes old prefers the existing 250-nm viewport-scoped adsb.lol fallback and labels the source/coverage accordingly when that upstream is available. If the fallback is also unavailable, the layer's freshness/error fields use the source epoch—never the cache receipt time—so the UI reports an old snapshot rather than “just now.” +- **Flights (commercial + military)** render one poll interval behind real time (30s/15s) and interpolate between two known feed-stamped fixes (OpenSky `time_position`; adsb.lol `receipt − seen_pos`). The display latency is an intentional product decision — do not "fix" it away. The fleet dead-reckons at ~12Hz (1m² write gating); a contact already hidden behind the horizon whose last position still is re-reckons on a staggered one-in-13-ticks rotation (about once a second) instead, so the worldwide feed's ~13k contacts no longer cost a full pass when well under 1% are on screen. Anything that position puts in view, and any contact with a fleet model, takes the full pass immediately, and hidden-inclusive proximity queries read positions at most about a second old; aircraft get a 3-poll grace period (faded icon) before removal. When a position epoch pauses, both layers coast for at least 60 seconds of contact grace with an absolute five-minute ceiling. Source backoff marks each contact and the cockpit `STALE`; the cockpit then holds the exact layer position instead of continuing inertial flight. Repeated-position kinematic changes create a forward-only synthetic fix rather than mutating history, and grounded history is lifted only when no owned 3D model already controls its datum. A nominally successful worldwide OpenSky response whose own snapshot epoch is more than two minutes old prefers the existing 250-nm viewport-scoped adsb.lol fallback and labels the source/coverage accordingly when that upstream is available. If the fallback is also unavailable, the layer's freshness/error fields use the source epoch—never the cache receipt time—so the UI reports an old snapshot rather than “just now.” - **World-space headings at every angle** (`src/data/iconOrientation.js`): aircraft/vessel icon rotation uses the camera right/up basis per tick (alignedAxis always ZERO), which is exact at screen center and for orthographic/nadir views and remains stable through >180° tracked orbits. Perspective rays vary across the viewport, so off-center contacts at oblique pitch can diverge from an exact finite-difference window projection; a regression test pins that known regime, and field evidence decides whether to adopt exact projection with the basis method as fallback. Fleet rotations refresh on camera-pose change; tracked entities per frame. Billboards are horizon-culled via a shared EllipsoidalOccluder. - **Military/OpenSky reconciliation** (`src/data/militaryRegistry.js`): known-military ICAOs render amber in the flights layer (60s self-poll of the cached mil endpoint when the military layer is off) and are suppressed there while the military layer renders them. - **Satellites**: 838-sat core catalog (stations/visual/GPS/GLONASS/Galileo/GEO), tracking lands ~726km out via `viewFrom` (tracked entity owns a point graphic so the tracking camera engages), rings realign via primitive modelMatrix (no per-second rebuild flicker), optional `setParams({catalog:'dense'})` Starlink mode. From 1f0a8b5f1640921d0742cd94cdd8520310a90e2f Mon Sep 17 00:00:00 2001 From: Trygve Hemminghytt <274629477+trygvehhh@users.noreply.github.com> Date: Tue, 3 Nov 2026 22:58:09 +0100 Subject: [PATCH 07/14] fix(datacenters): report newly tagged features osmium cannot export A still-tagged feature without exportable geometry already keeps its old line. A newly tagged one had nowhere to go and was dropped silently; it is now listed as skipped with a warning, so a refresh never loses a site or hides a broken one. --- scripts/build-datacenters-norway.py | 10 ++++++++-- src/data/datacentersNorwayRefresh.test.mjs | 3 ++- src/data/local_data/datacenters/README.md | 3 ++- 3 files changed, 12 insertions(+), 4 deletions(-) diff --git a/scripts/build-datacenters-norway.py b/scripts/build-datacenters-norway.py index 03286b864..d1203f38d 100644 --- a/scripts/build-datacenters-norway.py +++ b/scripts/build-datacenters-norway.py @@ -182,12 +182,13 @@ def feature_line(key, sequence, fresh): def merge(lines, fresh, rings, tagged): - out, seen, sequence = [], set(), 0 - report = {'kept': 0, 'updated': [], 'removed': [], 'added': [], 'unexported': []} + out, seen, known, sequence = [], set(), set(), 0 + report = {'kept': 0, 'updated': [], 'removed': [], 'added': [], 'unexported': [], 'skipped': []} for line in lines: feature = json.loads(line) sequence = max(sequence, feature['properties']['id']) key = osm_key(feature) + known.add(key) if key in fresh: seen.add(key) old_tags = feature['properties']['tags'] @@ -211,6 +212,8 @@ def merge(lines, fresh, rings, tagged): sequence += 1 out.append(feature_line(key, sequence, fresh[key])) report['added'].append(key) + # newly tagged, but osmium built no usable geometry: nothing to add yet + report['skipped'] = sorted(set(tagged) - set(fresh) - known, key=lambda k: (k[0], int(k[1:]))) if len({osm_key(json.loads(line)) for line in out}) != len(out): raise ValueError('Duplicate OSM identity') return out, report @@ -239,6 +242,9 @@ def main(): if report['unexported']: print('warning: kept the old line of tagged features osmium did not export: %s' % ', '.join(report['unexported']), file=sys.stderr) + if report['skipped']: + print('warning: did not add newly tagged features osmium did not export: %s' + % ', '.join(report['skipped']), file=sys.stderr) payload = '\n'.join(merged) + '\n' if args.verify: if payload != DATASET.read_text(encoding='utf-8'): diff --git a/src/data/datacentersNorwayRefresh.test.mjs b/src/data/datacentersNorwayRefresh.test.mjs index 2b7b7d5bf..88527295b 100644 --- a/src/data/datacentersNorwayRefresh.test.mjs +++ b/src/data/datacentersNorwayRefresh.test.mjs @@ -52,9 +52,10 @@ def line(osm_id, sequence): 'type': 'data_center', 'osm_id': osm_id}}) norway = [([(0, 0), (20, 0), (20, 80), (0, 80)], False)] lines = [line(7, 1), line(8, 2)] -out, report = merge(lines, {}, norway, {'w7'}) +out, report = merge(lines, {}, norway, {'w7', 'w12'}) assert out == [lines[0]], out assert report['unexported'] == ['w7'] and report['removed'] == ['w8'], report +assert report['skipped'] == ['w12'] and report['added'] == [], report `, ); }); diff --git a/src/data/local_data/datacenters/README.md b/src/data/local_data/datacenters/README.md index 5a80df267..86f573c61 100644 --- a/src/data/local_data/datacenters/README.md +++ b/src/data/local_data/datacenters/README.md @@ -29,7 +29,8 @@ reflects OSM as of the extract below, everything else the original snapshot. - Contact tags are dropped as above; unchanged features keep their original line, so geometry is only rewritten when tags or rings actually changed. A feature is removed only once it is no longer tagged; one still tagged but - without an exportable geometry keeps its line and is reported as unexported. + without an exportable geometry keeps its line and is reported as unexported, + and a newly tagged one osmium cannot export is reported as skipped. - Refresh: `python3 scripts/build-datacenters-norway.py --cache ` uses Geofabrik's current Norway extract and prints the dated name it used; record that name, its OSM timestamp and MD5 here. Requires `osmium-tool`. From e4fda9d4e6d7f0d5c8cc8bfba82dd33d959dbc64 Mon Sep 17 00:00:00 2001 From: Mert Kaya <47145541+mertkayacs@users.noreply.github.com> Date: Thu, 5 Nov 2026 17:39:34 +0300 Subject: [PATCH 08/14] fix(geocode): preserve non-Latin names in keyless search --- changelog.d/1585-keyless-unicode-search.md | 3 + docs/CURRENT-STATE.md | 6 +- scripts/package-boundaries.json | 1 + src/data/fixtures/README.md | 5 + src/data/fixtures/photon-moscow-search.json | 110 ++++++++++++++++++++ src/keylessGeocoder.js | 16 ++- src/keylessGeocoder.test.mjs | 37 ++++++- 7 files changed, 166 insertions(+), 12 deletions(-) create mode 100644 changelog.d/1585-keyless-unicode-search.md create mode 100644 src/data/fixtures/photon-moscow-search.json diff --git a/changelog.d/1585-keyless-unicode-search.md b/changelog.d/1585-keyless-unicode-search.md new file mode 100644 index 000000000..276ade93f --- /dev/null +++ b/changelog.d/1585-keyless-unicode-search.md @@ -0,0 +1,3 @@ +- Keep keyless place search on a nearby exact match for names written in + non-Latin scripts instead of discarding the match and retrying without its + proximity ranking (Mert Kaya, @mertkayacs, #PRNUM). diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index f1244947d..45fe2999e 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -5720,8 +5720,10 @@ HTTP refusals and outages remain retryable. Searches share a 12-second total deadline, with Photon requests capped at six seconds each. Caller/application cancellation stops retries and late cache writes. Replacing a location search cancels the previous lookup; disposing its controls cancels the active lookup. Photon uses a soft proximity bias, up to five -candidates, and an unbiased retry for name mismatches. A "place, region" query -prefers a candidate named exactly for the place whose city, county, state, +candidates, and an unbiased retry for name mismatches. Name comparison folds +Latin accents while preserving letters and marks in other scripts, so an exact +non-Latin name keeps its proximity ranking. A "place, region" query prefers a +candidate named exactly for the place whose city, county, state, country or country code (or USA/UK) contains every word of the region, and a settlement or administrative area over a POI when both fit. Nearby results that fail that check are retried unbiased. After the region checks of both passes, a diff --git a/scripts/package-boundaries.json b/scripts/package-boundaries.json index d008ceaaa..72652d8da 100644 --- a/scripts/package-boundaries.json +++ b/scripts/package-boundaries.json @@ -552,6 +552,7 @@ "src/search/placeSearch.js", "src/search/google.js", "src/keylessGeocoder.js", + "src/data/placeNames.js", "src/search/geospatial.js", "src/search/http.js", "src/search/defaults.js", diff --git a/src/data/fixtures/README.md b/src/data/fixtures/README.md index e9dd55f50..257eaf8a5 100644 --- a/src/data/fixtures/README.md +++ b/src/data/fixtures/README.md @@ -1,5 +1,10 @@ # Test fixtures +- `photon-moscow-search.json` - Photon responses for `Москва`, captured + 2026-11-05 with and without a proximity bias near the Tajikistan namesake. + The fixture is trimmed to the fields the keyless geocoder consumes and is + used only by offline search tests. Data from OpenStreetMap contributors, + ODbL 1.0 (https://www.openstreetmap.org/copyright). - `tomtom-flow-austin-12-935-1686.pbf` — one real TomTom traffic-flow vector tile (Mapbox Vector Tile protobuf, layer `"Traffic flow"`), downtown Austin z12 x935 y1686, captured 2026-07-16 from diff --git a/src/data/fixtures/photon-moscow-search.json b/src/data/fixtures/photon-moscow-search.json new file mode 100644 index 000000000..7b1a4d25a --- /dev/null +++ b/src/data/fixtures/photon-moscow-search.json @@ -0,0 +1,110 @@ +{ + "biased": { + "features": [ + { + "geometry": { + "type": "Point", + "coordinates": [71.8339931, 38.948412] + }, + "properties": { + "name": "Москва", + "country": "Тоҷикистон" + } + }, + { + "geometry": { + "type": "Point", + "coordinates": [37.6063916, 55.625578] + }, + "properties": { + "name": "Москва", + "country": "Россия" + } + }, + { + "geometry": { + "type": "Point", + "coordinates": [37.6174782, 55.7505412] + }, + "properties": { + "name": "Москва", + "country": "Россия" + } + }, + { + "geometry": { + "type": "Point", + "coordinates": [37.0536908, 55.7098009] + }, + "properties": { + "name": "Москва", + "country": "Россия" + } + }, + { + "geometry": { + "type": "Point", + "coordinates": [32.163334, 56.917778] + }, + "properties": { + "name": "Москва", + "country": "Россия" + } + } + ] + }, + "unbiased": { + "features": [ + { + "geometry": { + "type": "Point", + "coordinates": [37.6063916, 55.625578] + }, + "properties": { + "name": "Москва", + "country": "Россия" + } + }, + { + "geometry": { + "type": "Point", + "coordinates": [37.6174782, 55.7505412] + }, + "properties": { + "name": "Москва", + "country": "Россия" + } + }, + { + "geometry": { + "type": "Point", + "coordinates": [37.0536908, 55.7098009] + }, + "properties": { + "name": "Москва", + "country": "Россия" + } + }, + { + "geometry": { + "type": "Point", + "coordinates": [32.163334, 56.917778] + }, + "properties": { + "name": "Москва", + "country": "Россия" + } + }, + { + "geometry": { + "type": "Point", + "coordinates": [71.8339931, 38.948412] + }, + "properties": { + "name": "Москва", + "country": "Тоҷикистон" + } + } + ] + } +} diff --git a/src/keylessGeocoder.js b/src/keylessGeocoder.js index 4e01f1c74..eb881f970 100644 --- a/src/keylessGeocoder.js +++ b/src/keylessGeocoder.js @@ -1,3 +1,5 @@ +import { foldPlaceName } from './data/placeNames.js'; + /** Photon/OpenStreetMap place-search adapter. */ const PHOTON_ENDPOINT = 'https://photon.komoot.io/api/'; @@ -233,19 +235,15 @@ export function photonSearchUrl( } /** - * Place name reduced to comparable form: unaccented, lowercased, punctuation - * collapsed to single spaces. "Huế" and "Hue" have to compare equal, because a - * user typing either means the same city. + * Place name reduced to comparable form: lowercased, Latin accents removed, + * other scripts preserved, and punctuation collapsed to single spaces. "Huế" + * and "Hue" have to compare equal, because a user typing either means the same + * city. * @param {string} text * @returns {string} */ export function normalizeToponym(text) { - return String(text ?? '') - .normalize('NFD') - .replace(/[\u0300-\u036f]/g, '') - .toLowerCase() - .replace(/[^a-z0-9]+/g, ' ') - .trim(); + return foldPlaceName(text); } /** diff --git a/src/keylessGeocoder.test.mjs b/src/keylessGeocoder.test.mjs index 95098a10c..e4c762f62 100644 --- a/src/keylessGeocoder.test.mjs +++ b/src/keylessGeocoder.test.mjs @@ -35,6 +35,12 @@ import { import { geocodeNavigationMode } from './locations.js'; const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); +const MOSCOW_RESPONSES = JSON.parse( + fs.readFileSync( + path.join(ROOT, 'src/data/fixtures/photon-moscow-search.json'), + 'utf8', + ), +); /** Photon's real answer for "Hanoi", trimmed to the fields we consume. */ const HANOI = { @@ -223,7 +229,6 @@ const HUE = { extent: [107.3763, 16.7739, 108.0518, 16.1276], }, }; - /** Answers biased requests with `nearby`, unbiased ones with `anywhere`. */ function photonPair(nearby, anywhere) { const urls = []; @@ -249,6 +254,36 @@ test('a biased near-miss is refused, and the unbiased answer is taken instead', assert.equal(fetchImpl.urls.length, 2, 'the refused bias costs exactly one extra request'); }); +test('a non-Latin exact match keeps the nearby namesake', async () => { + const fetchImpl = photonPair( + MOSCOW_RESPONSES.biased.features, + MOSCOW_RESPONSES.unbiased.features, + ); + + const result = await geocodeKeyless('Москва', { + bias: '38.90,71.80|39.00,71.90', + fetchImpl, + }); + + assert.equal(result.label, 'Москва, Тоҷикистон'); + assert.equal(fetchImpl.urls.length, 1); +}); + +test('toponym folding preserves non-Latin scripts', () => { + const cases = [ + ['Huế', 'hue'], + ['Москва', 'москва'], + ['Αθήνα', 'αθήνα'], + ['القاهرة', 'القاهرة'], + ['東京', '東京'], + ['São 東京', 'sao 東京'], + ]; + + for (const [input, expected] of cases) { + assert.equal(normalizeToponym(input), expected, input); + } +}); + test('a biased match that leads with the name is taken without a second request', async () => { // The other half of the rule, and the reason bias exists at all: "Sixth // Street" over Austin must stay Austin's. From bc65c274da4418edc80df8d3b4fa3b1ed4bb42aa Mon Sep 17 00:00:00 2001 From: Said Turksever Date: Thu, 5 Nov 2026 17:13:00 +0000 Subject: [PATCH 09/14] fix(geocode): keep a leading "the" in keyless name matching MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit foldPlaceName drops a leading "the", so "The Hague" over upstate New York accepted Hague, NY as an exact nearby match. Split the script-aware fold into foldPlaceText and use it for keyless toponyms; foldPlaceName keeps its behaviour for the packs and street matchers. Also: changelog fragment credits #1585, folding test pins the Latin letters NFD does not decompose (ß, ø, ł), and CURRENT-STATE notes that non-Latin names match only when spelled with their marks. --- changelog.d/1585-keyless-unicode-search.md | 2 +- docs/CURRENT-STATE.md | 5 ++-- src/data/placeNames.js | 28 +++++++++++++++------- src/keylessGeocoder.js | 4 ++-- src/keylessGeocoder.test.mjs | 23 ++++++++++++++++++ 5 files changed, 48 insertions(+), 14 deletions(-) diff --git a/changelog.d/1585-keyless-unicode-search.md b/changelog.d/1585-keyless-unicode-search.md index 276ade93f..1f8b3af41 100644 --- a/changelog.d/1585-keyless-unicode-search.md +++ b/changelog.d/1585-keyless-unicode-search.md @@ -1,3 +1,3 @@ - Keep keyless place search on a nearby exact match for names written in non-Latin scripts instead of discarding the match and retrying without its - proximity ranking (Mert Kaya, @mertkayacs, #PRNUM). + proximity ranking (mertkayacs, #1585). diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 45fe2999e..d60b351cc 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -5721,8 +5721,9 @@ deadline, with Photon requests capped at six seconds each. Caller/application cancellation stops retries and late cache writes. Replacing a location search cancels the previous lookup; disposing its controls cancels the active lookup. Photon uses a soft proximity bias, up to five candidates, and an unbiased retry for name mismatches. Name comparison folds -Latin accents while preserving letters and marks in other scripts, so an exact -non-Latin name keeps its proximity ranking. A "place, region" query prefers a +Latin accents while preserving letters and marks in other scripts, so a +non-Latin name spelled exactly, marks included, keeps its proximity ranking. A +leading "the" counts: "The Hague" does not take a nearby "Hague". A "place, region" query prefers a candidate named exactly for the place whose city, county, state, country or country code (or USA/UK) contains every word of the region, and a settlement or administrative area over a POI when both fit. Nearby results that diff --git a/src/data/placeNames.js b/src/data/placeNames.js index 300d09209..90b458dd3 100644 --- a/src/data/placeNames.js +++ b/src/data/placeNames.js @@ -11,17 +11,16 @@ const FOLD = { }; /** - * Comparable form of a place name, shared by the bundled packs, the - * OpenFreeMap street and grounds matchers and the Nominatim server: case - * folded, accents removed from Latin letters only, punctuation and symbols - * collapsed to single spaces, and a leading "the" dropped. Other scripts keep - * their marks, so "कुल मार्ग" and "काली मार्ग" stay distinct while "São Paulo", - * "sao paulo" and "SÃO PAULO!" meet. + * Script-aware fold of a place name: case folded, accents removed from Latin + * letters only, punctuation and symbols collapsed to single spaces. Other + * scripts keep their marks, so "कुल मार्ग" and "काली मार्ग" stay distinct while + * "São Paulo", "sao paulo" and "SÃO PAULO!" meet. Every word is kept: "The + * Hague" is not "Hague". * * @param {unknown} value * @returns {string} */ -export function foldPlaceName(value) { +export function foldPlaceText(value) { return String(value ?? '') .toLowerCase() .replace(/[ßøæłđıœþ]/g, (c) => FOLD[c]) @@ -29,8 +28,19 @@ export function foldPlaceName(value) { .replace(/(\p{Script=Latin})\p{M}+/gu, '$1') .normalize('NFKC') .replace(/[^\p{L}\p{M}\p{N}]+/gu, ' ') - .trim() - .replace(/^the /, ''); + .trim(); +} + +/** + * Comparable form of a place name, shared by the bundled packs, the + * OpenFreeMap street and grounds matchers and the Nominatim server: + * `foldPlaceText` with a leading "the" dropped. + * + * @param {unknown} value + * @returns {string} + */ +export function foldPlaceName(value) { + return foldPlaceText(value).replace(/^the /, ''); } /** diff --git a/src/keylessGeocoder.js b/src/keylessGeocoder.js index eb881f970..3a5cd0035 100644 --- a/src/keylessGeocoder.js +++ b/src/keylessGeocoder.js @@ -1,4 +1,4 @@ -import { foldPlaceName } from './data/placeNames.js'; +import { foldPlaceText } from './data/placeNames.js'; /** Photon/OpenStreetMap place-search adapter. */ const PHOTON_ENDPOINT = 'https://photon.komoot.io/api/'; @@ -243,7 +243,7 @@ export function photonSearchUrl( * @returns {string} */ export function normalizeToponym(text) { - return foldPlaceName(text); + return foldPlaceText(text); } /** diff --git a/src/keylessGeocoder.test.mjs b/src/keylessGeocoder.test.mjs index e4c762f62..7f50ebc2e 100644 --- a/src/keylessGeocoder.test.mjs +++ b/src/keylessGeocoder.test.mjs @@ -229,6 +229,7 @@ const HUE = { extent: [107.3763, 16.7739, 108.0518, 16.1276], }, }; + /** Answers biased requests with `nearby`, unbiased ones with `anywhere`. */ function photonPair(nearby, anywhere) { const urls = []; @@ -277,6 +278,10 @@ test('toponym folding preserves non-Latin scripts', () => { ['القاهرة', 'القاهرة'], ['東京', '東京'], ['São 東京', 'sao 東京'], + ['Straße', 'strasse'], + ['Tromsø', 'tromso'], + ['Łódź', 'lodz'], + ['The Hague', 'the hague'], ]; for (const [input, expected] of cases) { @@ -284,6 +289,24 @@ test('toponym folding preserves non-Latin scripts', () => { } }); +test('a leading "the" is part of the name a nearby match must equal', async () => { + // "The Hague" over upstate New York means the Dutch city, not Hague, NY. + const hagueNy = { + geometry: { type: 'Point', coordinates: [-73.5004, 43.6967] }, + properties: { osm_key: 'place', osm_value: 'town', type: 'city', name: 'Hague', country: 'United States' }, + }; + const theHague = { + geometry: { type: 'Point', coordinates: [4.3113, 52.0799] }, + properties: { osm_key: 'place', osm_value: 'city', type: 'city', name: 'The Hague', country: 'Nederland' }, + }; + const fetchImpl = photonPair([hagueNy], [theHague]); + + const result = await geocodeKeyless('The Hague', { bias: '43.65,-73.55|43.75,-73.45', fetchImpl }); + + assert.equal(Math.round(result.lat), 52, `got ${result.label}`); + assert.equal(fetchImpl.urls.length, 2); +}); + test('a biased match that leads with the name is taken without a second request', async () => { // The other half of the rule, and the reason bias exists at all: "Sixth // Street" over Austin must stay Austin's. From 067383fda80b7bbc66bf792275433df771910426 Mon Sep 17 00:00:00 2001 From: German Khokhlov Date: Thu, 5 Nov 2026 21:42:22 +0400 Subject: [PATCH 10/14] fix(datacenters): drop the redundant seen set and test skipped against exports The merge test now passes an exported newly tagged feature, so removing '- set(fresh)' from the skipped report fails it. --- scripts/build-datacenters-norway.py | 5 ++--- src/data/datacentersNorwayRefresh.test.mjs | 7 ++++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/scripts/build-datacenters-norway.py b/scripts/build-datacenters-norway.py index d1203f38d..3c5eb92dd 100644 --- a/scripts/build-datacenters-norway.py +++ b/scripts/build-datacenters-norway.py @@ -182,7 +182,7 @@ def feature_line(key, sequence, fresh): def merge(lines, fresh, rings, tagged): - out, seen, known, sequence = [], set(), set(), 0 + out, known, sequence = [], set(), 0 report = {'kept': 0, 'updated': [], 'removed': [], 'added': [], 'unexported': [], 'skipped': []} for line in lines: feature = json.loads(line) @@ -190,7 +190,6 @@ def merge(lines, fresh, rings, tagged): key = osm_key(feature) known.add(key) if key in fresh: - seen.add(key) old_tags = feature['properties']['tags'] if old_tags == fresh[key]['tags'] and same_geometry(feature['geometry'], fresh[key]['geometry']): out.append(line) @@ -208,7 +207,7 @@ def merge(lines, fresh, rings, tagged): report['removed'].append(key) continue out.append(line) - for key in sorted(set(fresh) - seen, key=lambda k: (k[0], int(k[1:]))): + for key in sorted(set(fresh) - known, key=lambda k: (k[0], int(k[1:]))): sequence += 1 out.append(feature_line(key, sequence, fresh[key])) report['added'].append(key) diff --git a/src/data/datacentersNorwayRefresh.test.mjs b/src/data/datacentersNorwayRefresh.test.mjs index 88527295b..99278769e 100644 --- a/src/data/datacentersNorwayRefresh.test.mjs +++ b/src/data/datacentersNorwayRefresh.test.mjs @@ -52,10 +52,11 @@ def line(osm_id, sequence): 'type': 'data_center', 'osm_id': osm_id}}) norway = [([(0, 0), (20, 0), (20, 80), (0, 80)], False)] lines = [line(7, 1), line(8, 2)] -out, report = merge(lines, {}, norway, {'w7', 'w12'}) -assert out == [lines[0]], out +fresh = {'w13': {'geometry': {'type': 'Polygon', 'coordinates': square}, 'tags': {'building': 'data_center'}}} +out, report = merge(lines, fresh, norway, {'w7', 'w12', 'w13'}) +assert out[0] == lines[0] and len(out) == 2, out assert report['unexported'] == ['w7'] and report['removed'] == ['w8'], report -assert report['skipped'] == ['w12'] and report['added'] == [], report +assert report['skipped'] == ['w12'] and report['added'] == ['w13'], report `, ); }); From 5d8f2b878f5ef204b5efb4519a7fbbbc3699b59f Mon Sep 17 00:00:00 2001 From: German Khokhlov Date: Fri, 6 Nov 2026 15:47:36 +0400 Subject: [PATCH 11/14] chore(datacenters): drop the one-off Norway refresh script The data refresh is committed and its provenance (extract, timestamp, MD5, filter) stays in the dataset README. The script needed osmium-tool and could only re-verify while Geofabrik still listed that extract, so it goes with its test. --- changelog.d/953-datacenters-norway-refresh.md | 6 +- scripts/build-datacenters-norway.py | 259 ------------------ src/data/datacentersNorwayRefresh.test.mjs | 95 ------- src/data/local_data/datacenters/README.md | 12 +- 4 files changed, 3 insertions(+), 369 deletions(-) delete mode 100644 scripts/build-datacenters-norway.py delete mode 100644 src/data/datacentersNorwayRefresh.test.mjs diff --git a/changelog.d/953-datacenters-norway-refresh.md b/changelog.d/953-datacenters-norway-refresh.md index 48a019605..5db897234 100644 --- a/changelog.d/953-datacenters-norway-refresh.md +++ b/changelog.d/953-datacenters-norway-refresh.md @@ -1,6 +1,4 @@ - Refresh Norway's datacenters from a Geofabrik OpenStreetMap extract (2026-10-05). Troll Housing joins the layer (4,352 features) and four - Norwegian sites pick up current OSM tags; no site moved or was removed. - `scripts/build-datacenters-norway.py` reproduces or `--verify`s the merge, - touching only features inside Geofabrik's Norway boundary, and the dataset - README records the extract, checksum and filter (trygvehhh, #953). + Norwegian sites pick up current OSM tags; no site moved or was removed + (trygvehhh, #953). diff --git a/scripts/build-datacenters-norway.py b/scripts/build-datacenters-norway.py deleted file mode 100644 index 3c5eb92dd..000000000 --- a/scripts/build-datacenters-norway.py +++ /dev/null @@ -1,259 +0,0 @@ -#!/usr/bin/env python3 -"""Maintainer-only Norway refresh of the bundled datacenter extract (osmium-tool). - -Downloads a dated Geofabrik Norway extract, verifies its MD5, filters -telecom=data_center / building=data_center with osmium, and merges the result -into datacenters.geojsonl: features inside the Geofabrik Norway boundary are -added, updated or removed once no longer tagged; every other line is kept -byte-for-byte. - -Geofabrik keeps only its newest dated extract, so --extract defaults to the -current one and a recorded extract can be re-verified only while it is listed. -""" -import argparse -import hashlib -import json -import math -from pathlib import Path -import shutil -import subprocess -import sys -import urllib.request - -ROOT = Path(__file__).resolve().parent.parent -GEOFABRIK = 'https://download.geofabrik.de/europe/' -DATASET = ROOT / 'src/data/local_data/datacenters/datacenters.geojsonl' -CONTACT_TAGS = {'email', 'phone', 'fax', 'mobile', 'whatsapp', 'contact:email', 'contact:phone', - 'contact:fax', 'contact:mobile', 'contact:whatsapp'} -SAME_GEOMETRY_METERS = 0.05 -TIMEOUT_SECONDS = 60 - - -def latest_extract(): - """Resolve norway-latest to the dated name Geofabrik redirects it to.""" - request = urllib.request.Request(GEOFABRIK + 'norway-latest.osm.pbf', method='HEAD') - with urllib.request.urlopen(request, timeout=TIMEOUT_SECONDS) as response: - name = response.geturl().rsplit('/', 1)[-1] - if not name.startswith('norway-') or not name.endswith('.osm.pbf') or name == 'norway-latest.osm.pbf': - raise RuntimeError('Unexpected Geofabrik latest target: %s' % name) - return name[:-len('.osm.pbf')] - - -def fetch(url, target): - if not target.exists(): - with urllib.request.urlopen(url, timeout=TIMEOUT_SECONDS) as response, \ - open(str(target) + '.part', 'wb') as out: - shutil.copyfileobj(response, out) - Path(str(target) + '.part').rename(target) - return target - - -def md5(path): - digest = hashlib.md5() - with open(path, 'rb') as handle: - for block in iter(lambda: handle.read(1 << 20), b''): - digest.update(block) - return digest.hexdigest() - - -def read_poly(path): - """Geofabrik .poly -> list of (ring, is_hole).""" - rings, ring, hole = [], None, False - for raw in path.read_text(encoding='utf-8').splitlines()[1:]: - line = raw.strip() - if ring is None: - if line and line != 'END': - ring, hole = [], line.startswith('!') - elif line == 'END': - rings.append((ring, hole)) - ring = None - elif line: - lon, lat = (float(v) for v in line.split()) - ring.append((lon, lat)) - return rings - - -def inside(rings, lon, lat): - def in_ring(ring): - hit = False - for (x1, y1), (x2, y2) in zip(ring, ring[1:] + ring[:1]): - if (y1 > lat) != (y2 > lat) and lon < x1 + (lat - y1) * (x2 - x1) / (y2 - y1): - hit = not hit - return hit - return any(in_ring(r) for r, h in rings if not h) and not any(in_ring(r) for r, h in rings if h) - - -def points(coordinates): - if isinstance(coordinates[0], (int, float)): - return [coordinates] - return [p for part in coordinates for p in points(part)] - - -def osm_key(feature): - osm_id = feature['properties']['osm_id'] - if osm_id < 0: - return 'r%d' % -osm_id - return ('n' if feature['geometry']['type'] == 'Point' else 'w') + str(osm_id) - - -def export(osmium, pbf, cache, extract): - """Exported features by key, and the keys of every tagged object in the extract.""" - filtered = cache / (extract + '-datacenters.osm.pbf') - listed = cache / (extract + '-datacenters.opl') - exported = cache / (extract + '-datacenters.geojsonseq') - subprocess.run([osmium, 'tags-filter', '-O', '-o', str(filtered), str(pbf), - 'nwr/telecom=data_center', 'nwr/building=data_center'], check=True) - subprocess.run([osmium, 'tags-filter', '-O', '-R', '-f', 'opl', '-o', str(listed), str(filtered), - 'nwr/telecom=data_center', 'nwr/building=data_center'], check=True) - tagged = {line.split(' ', 1)[0] for line in listed.read_text(encoding='utf-8').splitlines() if line} - subprocess.run([osmium, 'export', '-O', '-f', 'geojsonseq', '--add-unique-id=type_id', - '-o', str(exported), str(filtered)], check=True) - fresh = {} - for line in exported.read_text(encoding='utf-8').splitlines(): - line = line.lstrip('\x1e').strip() - if not line: - continue - feature = json.loads(line) - tags = {k: v for k, v in feature['properties'].items() if k not in CONTACT_TAGS} - if tags.get('telecom') != 'data_center' and tags.get('building') != 'data_center': - continue # untagged member ways exported only for geometry - key = feature['id'] - geometry = feature['geometry'] - if key[0] == 'a': # osmium area id: way * 2 or relation * 2 + 1 - number = int(key[1:]) - key = ('w%d' % (number // 2)) if number % 2 == 0 else ('r%d' % (number // 2)) - if key[0] == 'r': - tags = {**tags, 'type': 'multipolygon'} - if geometry['type'] == 'MultiPolygon' and len(geometry['coordinates']) == 1: - geometry = {'type': 'Polygon', 'coordinates': geometry['coordinates'][0]} - elif key in fresh or geometry['type'] == 'LineString': - continue # closed ways are taken from their area export - fresh[key] = {'geometry': geometry, 'tags': tags} - return fresh, tagged - - -def same_geometry(old, new): - """Same type and rings within tolerance; ring start, winding and part or hole order may differ.""" - if old['type'] != new['type']: - return False - - def meters(p, q): - dx = (p[0] - q[0]) * 111320 * math.cos(math.radians(p[1])) - return math.hypot(dx, (p[1] - q[1]) * 110540) - - def same_line(a, b): - return len(a) == len(b) and all(meters(p, q) <= SAME_GEOMETRY_METERS for p, q in zip(a, b)) - - def same_ring(a, b): - a, b = a[:-1], b[:-1] # closed rings repeat their first vertex - return len(a) == len(b) and \ - any(same_line(a[i:] + a[:i], c) for c in (b, b[::-1]) for i in range(len(a))) - - def same_parts(a, b, same): - rest = list(b) - for part in a: - match = next((i for i, other in enumerate(rest) if same(part, other)), None) - if match is None: - return False - rest.pop(match) - return not rest - - def same_polygon(a, b): - return same_ring(a[0], b[0]) and same_parts(a[1:], b[1:], same_ring) - - a, b = old['coordinates'], new['coordinates'] - if old['type'] == 'Point': - return same_line([a], [b]) - if old['type'] == 'LineString': - return same_line(a, b) - if old['type'] == 'Polygon': - return same_polygon(a, b) - if old['type'] == 'MultiPolygon': - return same_parts(a, b, same_polygon) - return a == b - - -def feature_line(key, sequence, fresh): - osm_id = int(key[1:]) * (-1 if key[0] == 'r' else 1) - feature = {'id': osm_id, 'type': 'Feature', 'geometry': fresh['geometry'], - 'properties': {'id': sequence, 'tags': fresh['tags'], 'type': 'data_center', - 'osm_id': osm_id}} - return json.dumps(feature, separators=(',', ':'), ensure_ascii=False) - - -def merge(lines, fresh, rings, tagged): - out, known, sequence = [], set(), 0 - report = {'kept': 0, 'updated': [], 'removed': [], 'added': [], 'unexported': [], 'skipped': []} - for line in lines: - feature = json.loads(line) - sequence = max(sequence, feature['properties']['id']) - key = osm_key(feature) - known.add(key) - if key in fresh: - old_tags = feature['properties']['tags'] - if old_tags == fresh[key]['tags'] and same_geometry(feature['geometry'], fresh[key]['geometry']): - out.append(line) - report['kept'] += 1 - else: - out.append(feature_line(key, feature['properties']['id'], fresh[key])) - report['updated'].append(key) - continue - if key in tagged: # still tagged, but osmium built no usable geometry: keep the old line - report['unexported'].append(key) - out.append(line) - continue - lon, lat = points(feature['geometry']['coordinates'])[0][:2] - if inside(rings, lon, lat): - report['removed'].append(key) - continue - out.append(line) - for key in sorted(set(fresh) - known, key=lambda k: (k[0], int(k[1:]))): - sequence += 1 - out.append(feature_line(key, sequence, fresh[key])) - report['added'].append(key) - # newly tagged, but osmium built no usable geometry: nothing to add yet - report['skipped'] = sorted(set(tagged) - set(fresh) - known, key=lambda k: (k[0], int(k[1:]))) - if len({osm_key(json.loads(line)) for line in out}) != len(out): - raise ValueError('Duplicate OSM identity') - return out, report - - -def main(): - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument('--cache', type=Path, required=True) - parser.add_argument('--osmium', default=shutil.which('osmium') or 'osmium') - parser.add_argument('--extract', default='latest', - help="dated Geofabrik name such as norway-261005, or 'latest'") - parser.add_argument('--verify', action='store_true') - args = parser.parse_args() - args.cache.mkdir(parents=True, exist_ok=True) - extract = latest_extract() if args.extract == 'latest' else args.extract - pbf = fetch(GEOFABRIK + extract + '.osm.pbf', args.cache / (extract + '.osm.pbf')) - expected = fetch(GEOFABRIK + extract + '.osm.pbf.md5', - args.cache / (extract + '.osm.pbf.md5')).read_text(encoding='utf-8').split()[0] - actual = md5(pbf) - if actual != expected: - raise RuntimeError('MD5 mismatch for %s: %s != %s' % (pbf.name, actual, expected)) - rings = read_poly(fetch(GEOFABRIK + 'norway.poly', args.cache / 'norway.poly')) - fresh, tagged = export(args.osmium, pbf, args.cache, extract) - current = [line for line in DATASET.read_text(encoding='utf-8').splitlines() if line.strip()] - merged, report = merge(current, fresh, rings, tagged) - if report['unexported']: - print('warning: kept the old line of tagged features osmium did not export: %s' - % ', '.join(report['unexported']), file=sys.stderr) - if report['skipped']: - print('warning: did not add newly tagged features osmium did not export: %s' - % ', '.join(report['skipped']), file=sys.stderr) - payload = '\n'.join(merged) + '\n' - if args.verify: - if payload != DATASET.read_text(encoding='utf-8'): - raise SystemExit('datacenters.geojsonl is not the %s merge' % extract) - else: - with open(DATASET, 'w', encoding='utf-8', newline='\n') as handle: - handle.write(payload) - print(json.dumps({'extract': extract, 'md5': actual, 'norway': len(fresh), - 'features': len(merged), **report})) - - -if __name__ == '__main__': - main() diff --git a/src/data/datacentersNorwayRefresh.test.mjs b/src/data/datacentersNorwayRefresh.test.mjs deleted file mode 100644 index 99278769e..000000000 --- a/src/data/datacentersNorwayRefresh.test.mjs +++ /dev/null @@ -1,95 +0,0 @@ -import test from 'node:test'; -import assert from 'node:assert/strict'; -import { spawnSync } from 'node:child_process'; -import { fileURLToPath } from 'node:url'; - -const SCRIPT = fileURLToPath( - new URL('../../scripts/build-datacenters-norway.py', import.meta.url), -); - -function python(t, source) { - const result = spawnSync('python3', ['-c', source, SCRIPT], { - encoding: 'utf8', - }); - if (result.error?.code === 'ENOENT') return t.skip('python3 is not installed'); - assert.equal(result.status, 0, result.stderr); -} - -test('Norway refresh compares rings, not vertex sets', (t) => { - python( - t, - String.raw` -import runpy, sys -same = runpy.run_path(sys.argv[1])['same_geometry'] -poly = lambda *rings: {'type': 'Polygon', 'coordinates': list(rings)} -multi = lambda *polys: {'type': 'MultiPolygon', 'coordinates': [p['coordinates'] for p in polys]} -a, b, c, d, e = [0, 0], [4, 0], [4, 4], [0, 4], [2, 1] -assert not same(poly([a, b, c, d, e, a]), poly([a, b, e, c, d, a])), 'reordered edges' -assert same(poly([a, b, c, d, e, a]), poly([c, b, a, e, d, c])), 'ring start and winding' -assert same(poly([a, b, c, d, a]), poly([[0, 1e-7], b, c, d, [0, 1e-7]])), 'tolerance' -outer = [[0, 0], [9, 0], [9, 9], [0, 9], [0, 0]] -one, two = [[1, 1], [2, 1], [2, 2], [1, 1]], [[5, 5], [6, 5], [6, 6], [5, 5]] -assert same(poly(outer, one, two), poly(outer, two, one)), 'hole order' -assert not same(poly(outer, one), poly(one, outer)), 'outer and inner roles swapped' -assert same(multi(poly(one), poly(two)), multi(poly(two), poly(one))), 'part order' -p, q, r, s, u, v = [0, 0], [1, 0], [0, 1], [5, 5], [6, 5], [5, 6] -assert not same(multi(poly([p, q, r, p]), poly([s, u, v, s])), - multi(poly([p, q, s, p]), poly([r, u, v, r]))), 'vertices moved between parts' -`, - ); -}); - -test('Norway refresh keeps a still-tagged feature osmium did not export', (t) => { - python( - t, - String.raw` -import json, runpy, sys -merge = runpy.run_path(sys.argv[1])['merge'] -square = [[[10, 60], [10.01, 60], [10.01, 60.01], [10, 60.01], [10, 60]]] -def line(osm_id, sequence): - return json.dumps({'id': osm_id, 'type': 'Feature', 'geometry': {'type': 'Polygon', 'coordinates': square}, - 'properties': {'id': sequence, 'tags': {'building': 'data_center'}, - 'type': 'data_center', 'osm_id': osm_id}}) -norway = [([(0, 0), (20, 0), (20, 80), (0, 80)], False)] -lines = [line(7, 1), line(8, 2)] -fresh = {'w13': {'geometry': {'type': 'Polygon', 'coordinates': square}, 'tags': {'building': 'data_center'}}} -out, report = merge(lines, fresh, norway, {'w7', 'w12', 'w13'}) -assert out[0] == lines[0] and len(out) == 2, out -assert report['unexported'] == ['w7'] and report['removed'] == ['w8'], report -assert report['skipped'] == ['w12'] and report['added'] == ['w13'], report -`, - ); -}); - -test('Norway refresh lists tagged objects apart from the exported geometry', (t) => { - python( - t, - String.raw` -import json, runpy, subprocess, sys, tempfile -from pathlib import Path -export = runpy.run_path(sys.argv[1])['export'] -tags = {'building': 'data_center'} -ring = [[10, 60], [10.01, 60], [10.01, 60.01], [10, 60]] -features = [ - {'id': 'n1', 'properties': {'telecom': 'data_center'}, 'geometry': {'type': 'Point', 'coordinates': [10, 60]}}, - {'id': 'w5', 'properties': tags, 'geometry': {'type': 'LineString', 'coordinates': ring}}, - {'id': 'a10', 'properties': tags, 'geometry': {'type': 'MultiPolygon', 'coordinates': [[ring]]}}, - {'id': 'w7', 'properties': tags, 'geometry': {'type': 'LineString', 'coordinates': ring[:3]}}, -] -outputs = { - '.opl': 'n1 v1 Ttelecom=data_center x10 y60\nw5 v1 Tbuilding=data_center Nn2,n3,n4,n2\n' - 'w7 v1 Tbuilding=data_center Nn2,n3,n4\nr9 v1 Ttelecom=data_center Mw8@outer\n', - '.geojsonseq': ''.join(chr(30) + json.dumps({'type': 'Feature', **f}) + '\n' for f in features), -} -def run(args, check=False): - target = Path(args[args.index('-o') + 1]) - target.write_text(outputs.get(target.suffix, ''), encoding='utf-8') -subprocess.run = run -with tempfile.TemporaryDirectory() as cache: - fresh, tagged = export('osmium', Path(cache) / 'norway-test.osm.pbf', Path(cache), 'norway-test') -assert sorted(fresh) == ['n1', 'w5'], fresh -assert fresh['w5']['geometry'] == {'type': 'Polygon', 'coordinates': [ring]}, fresh['w5'] -assert tagged == {'n1', 'w5', 'w7', 'r9'}, tagged -`, - ); -}); diff --git a/src/data/local_data/datacenters/README.md b/src/data/local_data/datacenters/README.md index 86f573c61..c360ee527 100644 --- a/src/data/local_data/datacenters/README.md +++ b/src/data/local_data/datacenters/README.md @@ -26,14 +26,4 @@ reflects OSM as of the extract below, everything else the original snapshot. - Extract: Geofabrik `norway-261005.osm.pbf`, OSM data 2026-10-05T20:21:35Z, MD5 `1d2cfdeb98980fce11b8891d0f9146c0` - Filter: `osmium tags-filter nwr/telecom=data_center nwr/building=data_center` -- Contact tags are dropped as above; unchanged features keep their original - line, so geometry is only rewritten when tags or rings actually changed. - A feature is removed only once it is no longer tagged; one still tagged but - without an exportable geometry keeps its line and is reported as unexported, - and a newly tagged one osmium cannot export is reported as skipped. -- Refresh: `python3 scripts/build-datacenters-norway.py --cache ` uses - Geofabrik's current Norway extract and prints the dated name it used; record - that name, its OSM timestamp and MD5 here. Requires `osmium-tool`. -- Check: add `--extract --verify` to compare without writing. - Geofabrik keeps only its newest dated extract, so a recorded extract can be - re-verified only while it is still listed; the MD5 above identifies it. +- Contact tags are dropped as above. From a69d0492d0cda16a1eef15d022629f4949f293b5 Mon Sep 17 00:00:00 2001 From: German Khokhlov Date: Fri, 6 Nov 2026 15:54:53 +0400 Subject: [PATCH 12/14] docs(datacenters): tighten the Norway refresh notes --- changelog.d/953-datacenters-norway-refresh.md | 6 ++---- docs/INFRASTRUCTURE-LAYERS.md | 2 +- src/data/local_data/datacenters/README.md | 6 ++---- 3 files changed, 5 insertions(+), 9 deletions(-) diff --git a/changelog.d/953-datacenters-norway-refresh.md b/changelog.d/953-datacenters-norway-refresh.md index 5db897234..d60d4ac00 100644 --- a/changelog.d/953-datacenters-norway-refresh.md +++ b/changelog.d/953-datacenters-norway-refresh.md @@ -1,4 +1,2 @@ -- Refresh Norway's datacenters from a Geofabrik OpenStreetMap extract - (2026-10-05). Troll Housing joins the layer (4,352 features) and four - Norwegian sites pick up current OSM tags; no site moved or was removed - (trygvehhh, #953). +- Add Troll Housing to the Datacenters layer (4,352 features) by refreshing + Norway from OpenStreetMap data of 2026-10-05 (trygvehhh, #953). diff --git a/docs/INFRASTRUCTURE-LAYERS.md b/docs/INFRASTRUCTURE-LAYERS.md index aa8cd0558..798ff09af 100644 --- a/docs/INFRASTRUCTURE-LAYERS.md +++ b/docs/INFRASTRUCTURE-LAYERS.md @@ -59,4 +59,4 @@ globals. Dataset files and source/license notices remain under `src/data/local_data/`. Their input counts are 4,352 datacenter features and 704 dam features. Cesium may expand multipart geometries into multiple entities; entity and stem counts are -different measurements. This change does not refresh or relicense the datasets. +different measurements. diff --git a/src/data/local_data/datacenters/README.md b/src/data/local_data/datacenters/README.md index c360ee527..a89894fdb 100644 --- a/src/data/local_data/datacenters/README.md +++ b/src/data/local_data/datacenters/README.md @@ -19,11 +19,9 @@ snapshot. Future refreshes should record both before replacing the file. ## Norway refresh -Features inside the Geofabrik Norway boundary (`norway.poly`, which includes -Svalbard) were refreshed separately, so this file mixes two vintages: Norway -reflects OSM as of the extract below, everything else the original snapshot. +Features inside Geofabrik's Norway boundary (`norway.poly`, with Svalbard) come +from a newer extract than the rest of the file. Contact tags are dropped as above. - Extract: Geofabrik `norway-261005.osm.pbf`, OSM data 2026-10-05T20:21:35Z, MD5 `1d2cfdeb98980fce11b8891d0f9146c0` - Filter: `osmium tags-filter nwr/telecom=data_center nwr/building=data_center` -- Contact tags are dropped as above. From c058ecf21b98f3097adea6cd512b4931cb864723 Mon Sep 17 00:00:00 2001 From: German Khokhlov Date: Fri, 6 Nov 2026 15:57:27 +0400 Subject: [PATCH 13/14] docs(infrastructure): keep the dataset scope note --- docs/INFRASTRUCTURE-LAYERS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/INFRASTRUCTURE-LAYERS.md b/docs/INFRASTRUCTURE-LAYERS.md index 798ff09af..aa8cd0558 100644 --- a/docs/INFRASTRUCTURE-LAYERS.md +++ b/docs/INFRASTRUCTURE-LAYERS.md @@ -59,4 +59,4 @@ globals. Dataset files and source/license notices remain under `src/data/local_data/`. Their input counts are 4,352 datacenter features and 704 dam features. Cesium may expand multipart geometries into multiple entities; entity and stem counts are -different measurements. +different measurements. This change does not refresh or relicense the datasets. From 141181f83a0cc2fa6379a4c77af04469aee2df92 Mon Sep 17 00:00:00 2001 From: Said Turksever Date: Fri, 6 Nov 2026 12:02:30 +0000 Subject: [PATCH 14/14] docs(flights): changelog names the one-in-13-ticks rotation, about once a second --- changelog.d/870-hidden-contact-dr.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/changelog.d/870-hidden-contact-dr.md b/changelog.d/870-hidden-contact-dr.md index f88084d7a..aab55229e 100644 --- a/changelog.d/870-hidden-contact-dr.md +++ b/changelog.d/870-hidden-contact-dr.md @@ -1,4 +1,4 @@ - Reduce dead-reckoning work for flights already hidden behind the horizon. - Hidden contacts update on a staggered one-in-12-ticks rotation while visible - contacts and contacts with fleet models retain the full update path - (Wilmer Alzate, #870). + Hidden contacts update on a staggered one-in-13-ticks rotation (about once a + second) while visible contacts and contacts with fleet models retain the full + update path (Wilmer Alzate, #870).