From aa745bcee5c4f76353c9426fb9055cb5737f0fc0 Mon Sep 17 00:00:00 2001 From: Trygve Hemminghytt <274629477+trygvehhh@users.noreply.github.com> Date: Wed, 7 Oct 2026 04:02:05 +0200 Subject: [PATCH 1/9] data: refresh Norway datacenters from Geofabrik extract Add a maintainer script that merges a pinned, MD5-verified Geofabrik Norway extract (osmium tags-filter telecom/building=data_center) into the bundled datacenter layer. Only features inside the Geofabrik Norway boundary are touched; unchanged features keep their original line. norway-261005 (OSM 2026-10-05) adds Troll Housing (w1551029155) and updates tags on four existing features. Provenance is recorded in the dataset README and the bundled count moves from 4,351 to 4,352. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 6 + DATA_SOURCES.md | 2 +- README.md | 2 +- docs/INFRASTRUCTURE-LAYERS.md | 2 +- scripts/build-datacenters-norway.py | 196 ++++++++++++++++++ src/data/infrastructure.test.mjs | 2 +- src/data/local_data/datacenters/README.md | 16 +- .../datacenters/datacenters.geojsonl | 9 +- 8 files changed, 226 insertions(+), 9 deletions(-) create mode 100644 scripts/build-datacenters-norway.py diff --git a/CHANGELOG.md b/CHANGELOG.md index b433f8c06..031e36751 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,12 @@ ## [Unreleased] +- Refresh Norway's datacenters from a Geofabrik OpenStreetMap extract + (2026-10-05). Troll Housing joins the layer (4,352 features) and four + Norwegian sites pick up current OSM tags; no site moved or was removed. + `scripts/build-datacenters-norway.py` reproduces or `--verify`s the merge, + touching only features inside Geofabrik's Norway boundary, and the dataset + README records the extract, checksum and filter. - Add Norway's road cameras from Statens vegvesen as a CCTV pack. The camera list is one keyless GeoJSON request to the agency's OGC API view of its DATEX 3.1 CCTV table, so no DATEX account is needed. Every working camera diff --git a/DATA_SOURCES.md b/DATA_SOURCES.md index 734e35a69..b4f0dc0cc 100644 --- a/DATA_SOURCES.md +++ b/DATA_SOURCES.md @@ -112,7 +112,7 @@ The [Bhote Koshi event pack](public/events/bhote-koshi-2026/README.md), under `p | Dataset | Folder | License | Commercial use? | Attribution | | --------------------------------------------------------------------------- | --------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | --------------------------------------------------------------------------- | -| **Datacenters** (4,351) | `datacenters/` | **ODbL 1.0** (OpenStreetMap extract) | ✅ (attribution + share-alike on data) | © OpenStreetMap contributors (shared credit) | +| **Datacenters** (4,352) | `datacenters/` | **ODbL 1.0** (OpenStreetMap extract) | ✅ (attribution + share-alike on data) | © OpenStreetMap contributors (shared credit) | | **Dams** (704) | `dams/` | **ODbL 1.0** (OpenInfraMap / OSM extract) | ✅ (attribution + share-alike on data) | © OpenStreetMap contributors (shared credit) + Open Infrastructure Map | | **Military area names** (36,466) | `osm_military_names/` | **ODbL 1.0** (OpenStreetMap, via Overture Maps) | ✅ (attribution + share-alike on data) | © OpenStreetMap contributors (shared credit) + Overture Maps Foundation | | **TeleGeography Submarine Cable Map** (712 cables + 1,917 landing points) | `telegeography_submarine_cables/` | **CC BY-NC-SA 3.0** | ❌ **NonCommercial — remove for commercial use** | "© TeleGeography — submarinecablemap.com" | diff --git a/README.md b/README.md index 724ea18cb..0500645af 100644 --- a/README.md +++ b/README.md @@ -323,7 +323,7 @@ In the app, **Data Layers** groups them as Movement, Cameras, Infrastructure, Ev _The Space Missions layer replaying a Falcon 9 ascent — labeled `RECONSTRUCTED ESTIMATE`, scrubbable 0.25×–4×._ -**Also on the globe:** neighborhood overlays · an optional cockpit WX cloud effect. **Bundled static infrastructure:** Datacenters (4,351), Dams (704), and Submarine Cables (712). +**Also on the globe:** neighborhood overlays · an optional cockpit WX cloud effect. **Bundled static infrastructure:** Datacenters (4,352), Dams (704), and Submarine Cables (712). ![Diving into the Bahamas and revealing labeled submarine cable routes beneath the globe](docs/media/09-undersea-cables.gif) diff --git a/docs/INFRASTRUCTURE-LAYERS.md b/docs/INFRASTRUCTURE-LAYERS.md index 76567e605..1166a9e69 100644 --- a/docs/INFRASTRUCTURE-LAYERS.md +++ b/docs/INFRASTRUCTURE-LAYERS.md @@ -58,6 +58,6 @@ factory and supplies its existing functions. Other consumers should use the pack exports, which do not import standalone application globals. Dataset files and source/license notices remain under `src/data/local_data/`. -Their input counts are 4,351 datacenter features and 704 dam features. Cesium may +Their input counts are 4,352 datacenter features and 704 dam features. Cesium may expand multipart geometries into multiple entities; entity and stem counts are different measurements. This change does not refresh or relicense the datasets. diff --git a/scripts/build-datacenters-norway.py b/scripts/build-datacenters-norway.py new file mode 100644 index 000000000..01a891a8b --- /dev/null +++ b/scripts/build-datacenters-norway.py @@ -0,0 +1,196 @@ +#!/usr/bin/env python3 +"""Maintainer-only Norway refresh of the bundled datacenter extract (osmium-tool). + +Downloads a pinned Geofabrik Norway extract, verifies its MD5, filters +telecom=data_center / building=data_center with osmium, and merges the result +into datacenters.geojsonl: features inside the Geofabrik Norway boundary are +added, updated or removed; every other line is kept byte-for-byte. +""" +import argparse +import hashlib +import json +import math +from pathlib import Path +import shutil +import subprocess +import urllib.request + +ROOT = Path(__file__).resolve().parent.parent +EXTRACT = 'norway-261005' +GEOFABRIK = 'https://download.geofabrik.de/europe/' +DATASET = ROOT / 'src/data/local_data/datacenters/datacenters.geojsonl' +CONTACT_TAGS = {'email', 'phone', 'fax', 'mobile', 'whatsapp', 'contact:email', 'contact:phone', + 'contact:fax', 'contact:mobile', 'contact:whatsapp'} +SAME_GEOMETRY_METERS = 0.05 + + +def fetch(url, target): + if not target.exists(): + with urllib.request.urlopen(url) as response, open(str(target) + '.part', 'wb') as out: + shutil.copyfileobj(response, out) + Path(str(target) + '.part').rename(target) + return target + + +def md5(path): + digest = hashlib.md5() + with open(path, 'rb') as handle: + for block in iter(lambda: handle.read(1 << 20), b''): + digest.update(block) + return digest.hexdigest() + + +def read_poly(path): + """Geofabrik .poly -> list of (ring, is_hole).""" + rings, ring, hole = [], None, False + for raw in path.read_text().splitlines()[1:]: + line = raw.strip() + if ring is None: + if line and line != 'END': + ring, hole = [], line.startswith('!') + elif line == 'END': + rings.append((ring, hole)) + ring = None + elif line: + lon, lat = (float(v) for v in line.split()) + ring.append((lon, lat)) + return rings + + +def inside(rings, lon, lat): + def in_ring(ring): + hit = False + for (x1, y1), (x2, y2) in zip(ring, ring[1:] + ring[:1]): + if (y1 > lat) != (y2 > lat) and lon < x1 + (lat - y1) * (x2 - x1) / (y2 - y1): + hit = not hit + return hit + return any(in_ring(r) for r, h in rings if not h) and not any(in_ring(r) for r, h in rings if h) + + +def points(coordinates): + if isinstance(coordinates[0], (int, float)): + return [coordinates] + return [p for part in coordinates for p in points(part)] + + +def osm_key(feature): + osm_id = feature['properties']['osm_id'] + if osm_id < 0: + return 'r%d' % -osm_id + return ('n' if feature['geometry']['type'] == 'Point' else 'w') + str(osm_id) + + +def export(osmium, pbf, cache): + filtered = cache / (EXTRACT + '-datacenters.osm.pbf') + exported = cache / (EXTRACT + '-datacenters.geojsonseq') + subprocess.run([osmium, 'tags-filter', '-O', '-o', str(filtered), str(pbf), + 'nwr/telecom=data_center', 'nwr/building=data_center'], check=True) + subprocess.run([osmium, 'export', '-O', '-f', 'geojsonseq', '--add-unique-id=type_id', + '-o', str(exported), str(filtered)], check=True) + fresh = {} + for line in exported.read_text().splitlines(): + line = line.lstrip('\x1e').strip() + if not line: + continue + feature = json.loads(line) + tags = {k: v for k, v in feature['properties'].items() if k not in CONTACT_TAGS} + if tags.get('telecom') != 'data_center' and tags.get('building') != 'data_center': + continue # untagged member ways exported only for geometry + key = feature['id'] + geometry = feature['geometry'] + if key[0] == 'a': # osmium area id: way * 2 or relation * 2 + 1 + number = int(key[1:]) + key = ('w%d' % (number // 2)) if number % 2 == 0 else ('r%d' % (number // 2)) + if key[0] == 'r': + tags = {**tags, 'type': 'multipolygon'} + if geometry['type'] == 'MultiPolygon' and len(geometry['coordinates']) == 1: + geometry = {'type': 'Polygon', 'coordinates': geometry['coordinates'][0]} + elif key in fresh or geometry['type'] == 'LineString': + continue # closed ways are taken from their area export + fresh[key] = {'geometry': geometry, 'tags': tags} + return fresh + + +def same_geometry(old, new): + """Same type and vertex set within tolerance; ring start and winding may differ.""" + if old['type'] != new['type']: + return False + a, b = points(old['coordinates']), points(new['coordinates']) + if len(a) != len(b): + return False + + def meters(p, q): + dx = (p[0] - q[0]) * 111320 * math.cos(math.radians(p[1])) + return math.hypot(dx, (p[1] - q[1]) * 110540) + return all(min(meters(p, q) for q in b) <= SAME_GEOMETRY_METERS for p in a) and \ + all(min(meters(q, p) for p in a) <= SAME_GEOMETRY_METERS for q in b) + + +def feature_line(key, sequence, fresh): + osm_id = int(key[1:]) * (-1 if key[0] == 'r' else 1) + feature = {'id': osm_id, 'type': 'Feature', 'geometry': fresh['geometry'], + 'properties': {'id': sequence, 'tags': fresh['tags'], 'type': 'data_center', + 'osm_id': osm_id}} + return json.dumps(feature, separators=(',', ':'), ensure_ascii=False) + + +def merge(lines, fresh, rings): + out, seen, sequence = [], set(), 0 + report = {'kept': 0, 'updated': [], 'removed': [], 'added': []} + for line in lines: + feature = json.loads(line) + sequence = max(sequence, feature['properties']['id']) + key = osm_key(feature) + if key in fresh: + seen.add(key) + old_tags = feature['properties']['tags'] + if old_tags == fresh[key]['tags'] and same_geometry(feature['geometry'], fresh[key]['geometry']): + out.append(line) + report['kept'] += 1 + else: + out.append(feature_line(key, feature['properties']['id'], fresh[key])) + report['updated'].append(key) + continue + lon, lat = points(feature['geometry']['coordinates'])[0][:2] + if inside(rings, lon, lat): + report['removed'].append(key) + continue + out.append(line) + for key in sorted(set(fresh) - seen, key=lambda k: (k[0], int(k[1:]))): + sequence += 1 + out.append(feature_line(key, sequence, fresh[key])) + report['added'].append(key) + if len({osm_key(json.loads(line)) for line in out}) != len(out): + raise ValueError('Duplicate OSM identity') + return out, report + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--cache', type=Path, required=True) + parser.add_argument('--osmium', default=shutil.which('osmium') or 'osmium') + parser.add_argument('--verify', action='store_true') + args = parser.parse_args() + args.cache.mkdir(parents=True, exist_ok=True) + pbf = fetch(GEOFABRIK + EXTRACT + '.osm.pbf', args.cache / (EXTRACT + '.osm.pbf')) + expected = fetch(GEOFABRIK + EXTRACT + '.osm.pbf.md5', + args.cache / (EXTRACT + '.osm.pbf.md5')).read_text().split()[0] + actual = md5(pbf) + if actual != expected: + raise RuntimeError('MD5 mismatch for %s: %s != %s' % (pbf.name, actual, expected)) + rings = read_poly(fetch(GEOFABRIK + 'norway.poly', args.cache / 'norway.poly')) + fresh = export(args.osmium, pbf, args.cache) + current = [line for line in DATASET.read_text().splitlines() if line.strip()] + merged, report = merge(current, fresh, rings) + payload = '\n'.join(merged) + '\n' + if args.verify: + if payload != DATASET.read_text(): + raise SystemExit('datacenters.geojsonl is not the %s merge' % EXTRACT) + else: + DATASET.write_text(payload) + print(json.dumps({'extract': EXTRACT, 'md5': actual, 'norway': len(fresh), + 'features': len(merged), **report})) + + +if __name__ == '__main__': + main() diff --git a/src/data/infrastructure.test.mjs b/src/data/infrastructure.test.mjs index 45fb3d317..c7f1b5a23 100644 --- a/src/data/infrastructure.test.mjs +++ b/src/data/infrastructure.test.mjs @@ -82,7 +82,7 @@ test('infrastructure factory preserves identity and creates independent state wi test('dataset URLs still name the complete bundled sources', () => { for (const [file, count] of [ - ['datacenters', 4351], + ['datacenters', 4352], ['dams', 704], ]) { const lines = readFileSync( diff --git a/src/data/local_data/datacenters/README.md b/src/data/local_data/datacenters/README.md index 454a15007..ef0720ae4 100644 --- a/src/data/local_data/datacenters/README.md +++ b/src/data/local_data/datacenters/README.md @@ -5,7 +5,7 @@ layer. - Source: OpenStreetMap contributors - License: Open Database License (ODbL) 1.0 -- Feature count: 4,351 +- Feature count: 4,352 - Runtime file: `datacenters.geojsonl` The public-release snapshot removes contact-oriented tags such as email, phone, @@ -16,3 +16,17 @@ ODbL 1.0 with the required OpenStreetMap attribution. The original extraction date and query were not recorded alongside this snapshot. Future refreshes should record both before replacing the file. + +## Norway refresh + +Features inside the Geofabrik Norway boundary (`norway.poly`, which includes +Svalbard) were refreshed separately, so this file mixes two vintages: Norway +reflects OSM as of the extract below, everything else the original snapshot. + +- Extract: Geofabrik `norway-261005.osm.pbf`, OSM data + 2026-10-05T20:21:35Z, MD5 `1d2cfdeb98980fce11b8891d0f9146c0` +- Filter: `osmium tags-filter nwr/telecom=data_center nwr/building=data_center` +- Contact tags are dropped as above; unchanged features keep their original + line, so geometry is only rewritten when tags or vertices actually changed. +- Rebuild or check: `python3 scripts/build-datacenters-norway.py --cache ` + (add `--verify` to check without writing; requires `osmium-tool`) diff --git a/src/data/local_data/datacenters/datacenters.geojsonl b/src/data/local_data/datacenters/datacenters.geojsonl index f7f5fc89f..b02bd57b3 100644 --- a/src/data/local_data/datacenters/datacenters.geojsonl +++ b/src/data/local_data/datacenters/datacenters.geojsonl @@ -3770,15 +3770,15 @@ {"id":939340882,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[21.021364183,52.315373019],[21.02232433,52.315435883],[21.022363641,52.315209656],[21.021403578,52.315146875],[21.021364183,52.315373019]]]},"properties":{"id":11407,"tags":{"name":"T-Mobile Warszawa-Szlachecka","telecom":"data_center","building":"office","building:levels":"3"},"type":"data_center","osm_id":939340882}} {"id":1005973518,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[21.021325375,52.315595978],[21.022285857,52.315658423],[21.02232433,52.315435883],[21.021364183,52.315373019],[21.021325375,52.315595978]]]},"properties":{"id":11976,"tags":{"telecom":"data_center","building":"office","building:levels":"3"},"type":"data_center","osm_id":1005973518}} {"id":1156823776,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[21.115130857,52.235385942],[21.115189866,52.236349777],[21.115232027,52.236667954],[21.115818174,52.237174556],[21.116175578,52.237468677],[21.117262124,52.236985628],[21.11728157,52.236976576],[21.117262795,52.236960985],[21.117340495,52.236926033],[21.117360276,52.2369433],[21.117447532,52.236904491],[21.11804097,52.236654459],[21.118030996,52.236645742],[21.118041054,52.236641719],[21.116603055,52.235624072],[21.116518984,52.235676375],[21.115986231,52.235696659],[21.115954966,52.235361551],[21.115130857,52.235385942]]]},"properties":{"id":15342,"tags":{"name":"Atman Data Center Warsaw-1","telecom":"data_center","website":"https://www.atman.pl"},"type":"data_center","osm_id":1156823776}} -{"id":7405586142,"type":"Feature","geometry":{"type":"Point","coordinates":[5.598350034,58.860111293]},"properties":{"id":24849,"tags":{"name":"Maan Group","telecom":"data_center","operator":"Privat"},"type":"data_center","osm_id":7405586142}} +{"id":7405586142,"type":"Feature","geometry":{"type":"Point","coordinates":[5.5983501,58.8601113]},"properties":{"id":24849,"tags":{"name":"Maan Group","opening_hours":"24/7","operator":"Privat","telecom":"data_center"},"type":"data_center","osm_id":7405586142}} {"id":481615983,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[9.572510149,56.481777951],[9.574069938,56.484921751],[9.57465357,56.48483349],[9.574639572,56.484805243],[9.574700676,56.484795939],[9.574714925,56.484824857],[9.575298557,56.484736595],[9.57454888,56.483225422],[9.574609229,56.483216369],[9.574510155,56.483016629],[9.574449722,56.483025849],[9.573738769,56.481592795],[9.573155137,56.481680972],[9.573170643,56.481712321],[9.573109539,56.481721625],[9.573093781,56.48168969],[9.572510149,56.481777951]]]},"properties":{"id":7881,"tags":{"telecom":"data_center","building":"data_center","operator":"Apple","wikidata":"Q19721053","wikipedia":"en:Foulum Data Center","description":"Apple Datacenter"},"type":"data_center","osm_id":481615983}} {"id":618756435,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[10.500360092,57.318668738],[10.500399236,57.318671672],[10.500382891,57.318734871],[10.500552038,57.318744175],[10.500545249,57.318787677],[10.500501495,57.318785666],[10.500458999,57.318957746],[10.500970295,57.31899647],[10.501009187,57.318847189],[10.501084959,57.318852888],[10.50111966,57.318719197],[10.500843896,57.318698159],[10.500865521,57.318615178],[10.500859235,57.318614759],[10.500869461,57.318575699],[10.500954537,57.318582153],[10.501022347,57.31832223],[10.500736691,57.318300437],[10.500669133,57.318559689],[10.500674162,57.318560025],[10.500664272,57.318598498],[10.500383981,57.318577124],[10.500360092,57.318668738]]]},"properties":{"id":8842,"tags":{"ref":"AAL01-01","name":"Prime Data Centers - Saeby Campus AAL01","telecom":"data_center","website":"https://primedatacenters.com/locations/saeby/","building":"industrial","operator":"Prime Data Centers"},"type":"data_center","osm_id":618756435}} -{"id":-14367602,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[11.002243225,59.633471864],[11.003903261,59.633446551],[11.007046726,59.633399696],[11.006983443,59.632420941],[11.006982772,59.63227057],[11.006911023,59.632265289],[11.006852936,59.632253973],[11.006765848,59.632228576],[11.00671782,59.632205275],[11.006119687,59.632213573],[11.006119352,59.632202592],[11.006118933,59.63219245],[11.005930424,59.632194378],[11.005923635,59.632120869],[11.005980799,59.632120031],[11.005979542,59.632104524],[11.005977027,59.63206865],[11.00597225,59.631998242],[11.005412171,59.632035541],[11.005286945,59.632043839],[11.00527848,59.632044342],[11.003707962,59.632147272],[11.003107483,59.63219178],[11.002741864,59.632235533],[11.002729375,59.63223788],[11.002345987,59.632309294],[11.002308352,59.632316251],[11.002302066,59.632477351],[11.002277088,59.632913881],[11.002243225,59.633471864]]]},"properties":{"id":107,"tags":{"name":"STACK Infrastructure OSL04","type":"multipolygon","telecom":"data_center","website":"https://www.stackinfra.com","operator":"Stack Infrastructure","operator:wikidata":"Q115477173"},"type":"data_center","osm_id":-14367602}} +{"id":-14367602,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[11.0022433,59.6334719],[11.0022771,59.6329139],[11.0023021,59.6324774],[11.0023084,59.6323163],[11.002346,59.6323093],[11.0027294,59.6322379],[11.0027419,59.6322356],[11.0031075,59.6321918],[11.003708,59.6321473],[11.0052785,59.6320444],[11.005287,59.6320439],[11.0054122,59.6320356],[11.0059723,59.6319983],[11.0059771,59.6320687],[11.0059796,59.6321046],[11.0059808,59.6321201],[11.0059237,59.6321209],[11.0059305,59.6321944],[11.006119,59.6321925],[11.0061194,59.6322026],[11.0061197,59.6322136],[11.0067179,59.6322053],[11.0067659,59.6322286],[11.006853,59.632254],[11.0069111,59.6322653],[11.0069828,59.6322706],[11.0069835,59.632421],[11.0070468,59.6333997],[11.0039033,59.6334466],[11.0022433,59.6334719]]]},"properties":{"id":107,"tags":{"landuse":"industrial","name":"STACK Infrastructure OSL04","operator":"Stack Infrastructure","operator:wikidata":"Q115477173","telecom":"data_center","website":"https://www.stackinfra.com","type":"multipolygon"},"type":"data_center","osm_id":-14367602}} {"id":1028687251,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[11.004111886,59.632578437],[11.004142983,59.633037262],[11.004275166,59.633034999],[11.004274998,59.63303173],[11.00505133,59.633019157],[11.005056443,59.633099372],[11.005224416,59.63309669],[11.005190637,59.632562344],[11.005060885,59.632564439],[11.005060969,59.632567457],[11.004587978,59.632575419],[11.00458538,59.632536527],[11.004459819,59.632538623],[11.004462418,59.632576844],[11.004242476,59.6325807],[11.004242141,59.632576174],[11.004111886,59.632578437]]]},"properties":{"id":12226,"tags":{"building":"data_center","ref:bygningsnr":"300783262"},"type":"data_center","osm_id":1028687251}} {"id":1164840308,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[11.002600126,59.632599392],[11.002631223,59.633058217],[11.00276349,59.633055954],[11.002763238,59.633052685],[11.00353957,59.633040196],[11.003544683,59.633120327],[11.00371274,59.633117645],[11.003678961,59.632583382],[11.003549042,59.632585478],[11.003549293,59.632588495],[11.003076135,59.632596458],[11.00307362,59.632557566],[11.002948143,59.632559661],[11.002950657,59.632597883],[11.002730716,59.632601571],[11.002730465,59.632597129],[11.002600126,59.632599392]]]},"properties":{"id":15459,"tags":{"building":"data_center","ref:bygningsnr":"300783262"},"type":"data_center","osm_id":1164840308}} {"id":1164840309,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[11.005613672,59.632723444],[11.005644769,59.633182269],[11.005776951,59.633179922],[11.005776784,59.633176737],[11.006553199,59.633164164],[11.006558229,59.633244463],[11.006726286,59.633241781],[11.006692423,59.632707351],[11.006562671,59.632709446],[11.006562839,59.632712464],[11.006089764,59.632720426],[11.006087166,59.632681534],[11.005961772,59.632683797],[11.005964287,59.632721851],[11.005744262,59.632725623],[11.005743927,59.632721181],[11.005613672,59.632723444]]]},"properties":{"id":15460,"tags":{"building":"data_center","ref:bygningsnr":"300783262"},"type":"data_center","osm_id":1164840309}} -{"id":935092287,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[10.991467366,59.759024056],[10.992780978,59.759406942],[10.992773937,59.759423957],[10.992948197,59.759474248],[10.993073842,59.759363859],[10.993022293,59.759348939],[10.993291688,59.759112653],[10.993128827,59.759065547],[10.993181885,59.759019027],[10.993083062,59.758990445],[10.992976947,59.759083484],[10.992845519,59.759045346],[10.992813584,59.759073342],[10.9927876,59.759065798],[10.993409789,59.758524243],[10.99263002,59.758297094],[10.991938848,59.758898663],[10.991693594,59.758827165],[10.991467366,59.759024056]]]},"properties":{"id":11377,"tags":{"name":"Green Mountain Ytre Enebakk","telecom":"data_center","website":"https://greenmountain.no/","building":"warehouse","ref:bygningsnr":"300752944"},"type":"data_center","osm_id":935092287}} -{"id":1161764012,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[10.992826324,59.757448761],[10.994530281,59.757601396],[10.994634888,59.757303168],[10.992928332,59.757152545],[10.992826324,59.757448761]]]},"properties":{"id":15426,"tags":{"name":"Green Mountain Ytre Enebakk","telecom":"data_center","website":"https://greenmountain.no/","building":"industrial"},"type":"data_center","osm_id":1161764012}} +{"id":935092287,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[10.9914674,59.7590241],[10.9916936,59.7588272],[10.9919389,59.7588987],[10.9926301,59.7582971],[10.9934098,59.7585243],[10.9927876,59.7590658],[10.9928136,59.7590734],[10.9928456,59.7590454],[10.992977,59.7590835],[10.9930831,59.7589905],[10.9931819,59.7590191],[10.9931289,59.7590656],[10.9932917,59.7591127],[10.9930223,59.759349],[10.9930739,59.7593639],[10.9929482,59.7594743],[10.992774,59.759424],[10.992781,59.759407],[10.9914674,59.7590241]]]},"properties":{"id":11377,"tags":{"building":"warehouse","name":"Green Mountain Ytre Enebakk","ref:bygningsnr":"300752944","source":"Norway Orthophoto 2024;https://greenmountain.no/data-center/osl1-enebakk/","telecom":"data_center","website":"https://greenmountain.no/"},"type":"data_center","osm_id":935092287}} +{"id":1161764012,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[10.9928264,59.7574488],[10.9929284,59.7571526],[10.9946349,59.7573032],[10.9945303,59.7576014],[10.9928264,59.7574488]]]},"properties":{"id":15426,"tags":{"building":"industrial","name":"Green Mountain Ytre Enebakk","source":"Norway Orthophoto 2024;https://greenmountain.no/data-center/osl1-enebakk/","telecom":"data_center","website":"https://greenmountain.no/"},"type":"data_center","osm_id":1161764012}} {"id":30724580,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[10.808790995,59.924984985],[10.808893925,59.925031923],[10.80893793,59.92500594],[10.808982941,59.925025972],[10.809042955,59.92501298],[10.809083943,59.925031923],[10.809012948,59.925077856],[10.80926583,59.925189755],[10.80955115,59.925025972],[10.809676543,59.925081293],[10.81032899,59.924701928],[10.809890365,59.924508138],[10.80982155,59.924515347],[10.809691631,59.924457847],[10.808820667,59.924557592],[10.809099365,59.924699581],[10.808806418,59.924865291],[10.808918149,59.924914828],[10.808875988,59.924937962],[10.808790995,59.924984985]]]},"properties":{"id":261,"tags":{"name":"Stack Infrastructure OSLO1A","telecom":"data_center","building":"office","operator":"Stack Infrastructure","ref:bygningsnr":"80877293","building:levels":"4","operator:wikidata":"Q115477173"},"type":"data_center","osm_id":30724580}} {"id":80871917,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[10.83408892,59.938588981],[10.834305927,59.938720996],[10.834575992,59.938720996],[10.834574987,59.938736],[10.834581943,59.938736922],[10.834579932,59.938771958],[10.834736925,59.938771958],[10.834734997,59.93893792],[10.836717988,59.938944961],[10.836717988,59.938983937],[10.836817984,59.938983937],[10.83681899,59.938944961],[10.836844974,59.938944961],[10.836846985,59.938774976],[10.836841956,59.938774976],[10.836842962,59.938744968],[10.836849919,59.938744968],[10.836851931,59.938653941],[10.836683957,59.938652935],[10.836684125,59.938640278],[10.836686975,59.938414973],[10.83490297,59.938406926],[10.834899953,59.938646984],[10.834812949,59.938645978],[10.83481496,59.938551933],[10.834579932,59.938550927],[10.834580938,59.938564925],[10.834567946,59.938564925],[10.834567946,59.938551933],[10.834089926,59.938549922],[10.83408892,59.938588981]]]},"properties":{"id":1464,"tags":{"name":"Bulk Data Centers Oslo Internet Exchange","telecom":"data_center","website":"https://bulkinfrastructure.com/data-centers/locations/os-ix","building":"industrial","ref:bygningsnr":"80709943"},"type":"data_center","osm_id":80871917}} {"id":12158409495,"type":"Feature","geometry":{"type":"Point","coordinates":[9.689363435,62.59439936]},"properties":{"id":28723,"tags":{"name":"VisBook AS","telecom":"data_center","website":"https://visbook.com/"},"type":"data_center","osm_id":12158409495}} @@ -4349,3 +4349,4 @@ {"id":540895898,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[101.71747909,56.303855963],[101.71782015,56.304420065],[101.721113819,56.303806929],[101.720772759,56.303242994],[101.71747909,56.303855963]]]},"properties":{"id":8271,"tags":{"name":"Битривер Рус","name:en":"Bitriver Rus","name:ru":"Битривер Рус","telecom":"data_center","website":"https://bitriver.farm/","building":"data_center"},"type":"data_center","osm_id":540895898}} {"id":1082803064,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[123.948021193,47.340264299],[123.948354625,47.340268239],[123.948490663,47.340269831],[123.948488568,47.340349292],[123.948809762,47.340353064],[123.948811942,47.340267485],[123.948918643,47.340268658],[123.948922918,47.340137817],[123.948023959,47.340126082],[123.948021193,47.340264299]]]},"properties":{"id":13377,"tags":{"name":"联通云数据中心·齐齐哈尔","name:en":"Unicom Data Center of Qiqihar","name:zh":"联通云数据中心·齐齐哈尔","telecom":"data_center","building":"yes","name:zh-Hans":"联通云数据中心·齐齐哈尔","name:zh-Hant":"聯通雲數據中心·齊齊哈爾"},"type":"data_center","osm_id":1082803064}} {"id":442479059,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[142.76019226,46.910096898],[142.760192427,46.910325724],[142.760447656,46.910328155],[142.760460397,46.910102263],[142.76019226,46.910096898]]]},"properties":{"id":7471,"tags":{"telecom":"data_center","building":"yes"},"type":"data_center","osm_id":442479059}} +{"id":1551029155,"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[7.1069147,62.8627625],[7.1071207,62.8621617],[7.1075917,62.8616616],[7.1088647,62.8620476],[7.1106308,62.862645],[7.1098655,62.8632827],[7.1089604,62.8630042],[7.1086734,62.8632156],[7.1069147,62.8627625]]]},"properties":{"id":29471,"tags":{"layer":"-1","loc_name":"Troll Housing Klempertåsen","name":"Troll Housing","operator":"Troll Housing AS","telecom":"data_center","website":"https://www.trollhousing.no/"},"type":"data_center","osm_id":1551029155}} From 5b5d942d95dddd56b3bacb465c69098913688d1f Mon Sep 17 00:00:00 2001 From: Trygve Hemminghytt <274629477+trygvehhh@users.noreply.github.com> Date: Fri, 16 Oct 2026 22:23:02 +0200 Subject: [PATCH 2/9] fix(data): make the Norway datacenter script portable and re-runnable Read and write files as UTF-8 with LF line endings so the script and --verify work on Windows, where the default codec is cp1252 (thanks to quanticsoul4772 for the report). Geofabrik keeps only its newest dated extract, so --extract now defaults to the current one and accepts a dated name for --verify while it is listed; the dataset README says so. Downloads time out instead of hanging. --- CHANGELOG.md | 1 + scripts/build-datacenters-norway.py | 52 +++++++++++++++-------- src/data/local_data/datacenters/README.md | 8 +++- 3 files changed, 42 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 20a9da4e6..067ca16b6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -184,6 +184,7 @@ `scripts/build-datacenters-norway.py` reproduces or `--verify`s the merge, touching only features inside Geofabrik's Norway boundary, and the dataset README records the extract, checksum and filter. + - Add a keyless Air quality layer to the Weather group, with Environment and Climate Change Canada's AQHI as the first registered provider. Readings carry `{ provider, scale, value, band }` so another network can be added without diff --git a/scripts/build-datacenters-norway.py b/scripts/build-datacenters-norway.py index 01a891a8b..687395f7b 100644 --- a/scripts/build-datacenters-norway.py +++ b/scripts/build-datacenters-norway.py @@ -1,10 +1,13 @@ #!/usr/bin/env python3 """Maintainer-only Norway refresh of the bundled datacenter extract (osmium-tool). -Downloads a pinned Geofabrik Norway extract, verifies its MD5, filters +Downloads a dated Geofabrik Norway extract, verifies its MD5, filters telecom=data_center / building=data_center with osmium, and merges the result into datacenters.geojsonl: features inside the Geofabrik Norway boundary are added, updated or removed; every other line is kept byte-for-byte. + +Geofabrik keeps only its newest dated extract, so --extract defaults to the +current one and a recorded extract can be re-verified only while it is listed. """ import argparse import hashlib @@ -16,17 +19,28 @@ import subprocess import urllib.request ROOT = Path(__file__).resolve().parent.parent -EXTRACT = 'norway-261005' GEOFABRIK = 'https://download.geofabrik.de/europe/' DATASET = ROOT / 'src/data/local_data/datacenters/datacenters.geojsonl' CONTACT_TAGS = {'email', 'phone', 'fax', 'mobile', 'whatsapp', 'contact:email', 'contact:phone', 'contact:fax', 'contact:mobile', 'contact:whatsapp'} SAME_GEOMETRY_METERS = 0.05 +TIMEOUT_SECONDS = 60 + + +def latest_extract(): + """Resolve norway-latest to the dated name Geofabrik redirects it to.""" + request = urllib.request.Request(GEOFABRIK + 'norway-latest.osm.pbf', method='HEAD') + with urllib.request.urlopen(request, timeout=TIMEOUT_SECONDS) as response: + name = response.geturl().rsplit('/', 1)[-1] + if not name.startswith('norway-') or not name.endswith('.osm.pbf') or name == 'norway-latest.osm.pbf': + raise RuntimeError('Unexpected Geofabrik latest target: %s' % name) + return name[:-len('.osm.pbf')] def fetch(url, target): if not target.exists(): - with urllib.request.urlopen(url) as response, open(str(target) + '.part', 'wb') as out: + with urllib.request.urlopen(url, timeout=TIMEOUT_SECONDS) as response, \ + open(str(target) + '.part', 'wb') as out: shutil.copyfileobj(response, out) Path(str(target) + '.part').rename(target) return target @@ -43,7 +57,7 @@ def md5(path): def read_poly(path): """Geofabrik .poly -> list of (ring, is_hole).""" rings, ring, hole = [], None, False - for raw in path.read_text().splitlines()[1:]: + for raw in path.read_text(encoding='utf-8').splitlines()[1:]: line = raw.strip() if ring is None: if line and line != 'END': @@ -80,15 +94,15 @@ def osm_key(feature): return ('n' if feature['geometry']['type'] == 'Point' else 'w') + str(osm_id) -def export(osmium, pbf, cache): - filtered = cache / (EXTRACT + '-datacenters.osm.pbf') - exported = cache / (EXTRACT + '-datacenters.geojsonseq') +def export(osmium, pbf, cache, extract): + filtered = cache / (extract + '-datacenters.osm.pbf') + exported = cache / (extract + '-datacenters.geojsonseq') subprocess.run([osmium, 'tags-filter', '-O', '-o', str(filtered), str(pbf), 'nwr/telecom=data_center', 'nwr/building=data_center'], check=True) subprocess.run([osmium, 'export', '-O', '-f', 'geojsonseq', '--add-unique-id=type_id', '-o', str(exported), str(filtered)], check=True) fresh = {} - for line in exported.read_text().splitlines(): + for line in exported.read_text(encoding='utf-8').splitlines(): line = line.lstrip('\x1e').strip() if not line: continue @@ -169,26 +183,30 @@ def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument('--cache', type=Path, required=True) parser.add_argument('--osmium', default=shutil.which('osmium') or 'osmium') + parser.add_argument('--extract', default='latest', + help="dated Geofabrik name such as norway-261005, or 'latest'") parser.add_argument('--verify', action='store_true') args = parser.parse_args() args.cache.mkdir(parents=True, exist_ok=True) - pbf = fetch(GEOFABRIK + EXTRACT + '.osm.pbf', args.cache / (EXTRACT + '.osm.pbf')) - expected = fetch(GEOFABRIK + EXTRACT + '.osm.pbf.md5', - args.cache / (EXTRACT + '.osm.pbf.md5')).read_text().split()[0] + extract = latest_extract() if args.extract == 'latest' else args.extract + pbf = fetch(GEOFABRIK + extract + '.osm.pbf', args.cache / (extract + '.osm.pbf')) + expected = fetch(GEOFABRIK + extract + '.osm.pbf.md5', + args.cache / (extract + '.osm.pbf.md5')).read_text(encoding='utf-8').split()[0] actual = md5(pbf) if actual != expected: raise RuntimeError('MD5 mismatch for %s: %s != %s' % (pbf.name, actual, expected)) rings = read_poly(fetch(GEOFABRIK + 'norway.poly', args.cache / 'norway.poly')) - fresh = export(args.osmium, pbf, args.cache) - current = [line for line in DATASET.read_text().splitlines() if line.strip()] + fresh = export(args.osmium, pbf, args.cache, extract) + current = [line for line in DATASET.read_text(encoding='utf-8').splitlines() if line.strip()] merged, report = merge(current, fresh, rings) payload = '\n'.join(merged) + '\n' if args.verify: - if payload != DATASET.read_text(): - raise SystemExit('datacenters.geojsonl is not the %s merge' % EXTRACT) + if payload != DATASET.read_text(encoding='utf-8'): + raise SystemExit('datacenters.geojsonl is not the %s merge' % extract) else: - DATASET.write_text(payload) - print(json.dumps({'extract': EXTRACT, 'md5': actual, 'norway': len(fresh), + with open(DATASET, 'w', encoding='utf-8', newline='\n') as handle: + handle.write(payload) + print(json.dumps({'extract': extract, 'md5': actual, 'norway': len(fresh), 'features': len(merged), **report})) diff --git a/src/data/local_data/datacenters/README.md b/src/data/local_data/datacenters/README.md index ef0720ae4..8b803dcc7 100644 --- a/src/data/local_data/datacenters/README.md +++ b/src/data/local_data/datacenters/README.md @@ -28,5 +28,9 @@ reflects OSM as of the extract below, everything else the original snapshot. - Filter: `osmium tags-filter nwr/telecom=data_center nwr/building=data_center` - Contact tags are dropped as above; unchanged features keep their original line, so geometry is only rewritten when tags or vertices actually changed. -- Rebuild or check: `python3 scripts/build-datacenters-norway.py --cache ` - (add `--verify` to check without writing; requires `osmium-tool`) +- Refresh: `python3 scripts/build-datacenters-norway.py --cache ` uses + Geofabrik's current Norway extract and prints the dated name it used; record + that name, its OSM timestamp and MD5 here. Requires `osmium-tool`. +- Check: add `--extract --verify` to compare without writing. + Geofabrik keeps only its newest dated extract, so a recorded extract can be + re-verified only while it is still listed; the MD5 above identifies it. From 4318f51de4114bfb68137ff48bedd070b976f771 Mon Sep 17 00:00:00 2001 From: Trygve Hemminghytt <274629477+trygvehhh@users.noreply.github.com> Date: Sun, 18 Oct 2026 20:03:23 +0200 Subject: [PATCH 3/9] docs: move the changelog entry to changelog.d --- CHANGELOG.md | 7 ------- changelog.d/953-datacenters-norway-refresh.md | 6 ++++++ 2 files changed, 6 insertions(+), 7 deletions(-) create mode 100644 changelog.d/953-datacenters-norway-refresh.md diff --git a/CHANGELOG.md b/CHANGELOG.md index a285bc555..98d32365f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -284,13 +284,6 @@ ## [Unreleased] -- Refresh Norway's datacenters from a Geofabrik OpenStreetMap extract - (2026-10-05). Troll Housing joins the layer (4,352 features) and four - Norwegian sites pick up current OSM tags; no site moved or was removed. - `scripts/build-datacenters-norway.py` reproduces or `--verify`s the merge, - touching only features inside Geofabrik's Norway boundary, and the dataset - README records the extract, checksum and filter. - - Add a keyless Air quality layer to the Weather group, with Environment and Climate Change Canada's AQHI as the first registered provider. Readings carry `{ provider, scale, value, band }` so another network can be added without diff --git a/changelog.d/953-datacenters-norway-refresh.md b/changelog.d/953-datacenters-norway-refresh.md new file mode 100644 index 000000000..48a019605 --- /dev/null +++ b/changelog.d/953-datacenters-norway-refresh.md @@ -0,0 +1,6 @@ +- Refresh Norway's datacenters from a Geofabrik OpenStreetMap extract + (2026-10-05). Troll Housing joins the layer (4,352 features) and four + Norwegian sites pick up current OSM tags; no site moved or was removed. + `scripts/build-datacenters-norway.py` reproduces or `--verify`s the merge, + touching only features inside Geofabrik's Norway boundary, and the dataset + README records the extract, checksum and filter (trygvehhh, #953). From 04d626a4b190149ffcb017290c5d9aa17d32748d Mon Sep 17 00:00:00 2001 From: German Khokhlov Date: Tue, 3 Nov 2026 15:02:42 +0400 Subject: [PATCH 4/9] fix(datacenters): preserve missing geometry and compare ring topology --- scripts/build-datacenters-norway.py | 64 ++++++++++++--- src/data/datacentersNorwayRefresh.test.mjs | 93 ++++++++++++++++++++++ src/data/local_data/datacenters/README.md | 4 +- 3 files changed, 148 insertions(+), 13 deletions(-) create mode 100644 src/data/datacentersNorwayRefresh.test.mjs diff --git a/scripts/build-datacenters-norway.py b/scripts/build-datacenters-norway.py index 687395f7b..03286b864 100644 --- a/scripts/build-datacenters-norway.py +++ b/scripts/build-datacenters-norway.py @@ -4,7 +4,8 @@ Downloads a dated Geofabrik Norway extract, verifies its MD5, filters telecom=data_center / building=data_center with osmium, and merges the result into datacenters.geojsonl: features inside the Geofabrik Norway boundary are -added, updated or removed; every other line is kept byte-for-byte. +added, updated or removed once no longer tagged; every other line is kept +byte-for-byte. Geofabrik keeps only its newest dated extract, so --extract defaults to the current one and a recorded extract can be re-verified only while it is listed. @@ -16,6 +17,7 @@ import math from pathlib import Path import shutil import subprocess +import sys import urllib.request ROOT = Path(__file__).resolve().parent.parent @@ -95,10 +97,15 @@ def osm_key(feature): def export(osmium, pbf, cache, extract): + """Exported features by key, and the keys of every tagged object in the extract.""" filtered = cache / (extract + '-datacenters.osm.pbf') + listed = cache / (extract + '-datacenters.opl') exported = cache / (extract + '-datacenters.geojsonseq') subprocess.run([osmium, 'tags-filter', '-O', '-o', str(filtered), str(pbf), 'nwr/telecom=data_center', 'nwr/building=data_center'], check=True) + subprocess.run([osmium, 'tags-filter', '-O', '-R', '-f', 'opl', '-o', str(listed), str(filtered), + 'nwr/telecom=data_center', 'nwr/building=data_center'], check=True) + tagged = {line.split(' ', 1)[0] for line in listed.read_text(encoding='utf-8').splitlines() if line} subprocess.run([osmium, 'export', '-O', '-f', 'geojsonseq', '--add-unique-id=type_id', '-o', str(exported), str(filtered)], check=True) fresh = {} @@ -122,22 +129,48 @@ def export(osmium, pbf, cache, extract): elif key in fresh or geometry['type'] == 'LineString': continue # closed ways are taken from their area export fresh[key] = {'geometry': geometry, 'tags': tags} - return fresh + return fresh, tagged def same_geometry(old, new): - """Same type and vertex set within tolerance; ring start and winding may differ.""" + """Same type and rings within tolerance; ring start, winding and part or hole order may differ.""" if old['type'] != new['type']: return False - a, b = points(old['coordinates']), points(new['coordinates']) - if len(a) != len(b): - return False def meters(p, q): dx = (p[0] - q[0]) * 111320 * math.cos(math.radians(p[1])) return math.hypot(dx, (p[1] - q[1]) * 110540) - return all(min(meters(p, q) for q in b) <= SAME_GEOMETRY_METERS for p in a) and \ - all(min(meters(q, p) for p in a) <= SAME_GEOMETRY_METERS for q in b) + + def same_line(a, b): + return len(a) == len(b) and all(meters(p, q) <= SAME_GEOMETRY_METERS for p, q in zip(a, b)) + + def same_ring(a, b): + a, b = a[:-1], b[:-1] # closed rings repeat their first vertex + return len(a) == len(b) and \ + any(same_line(a[i:] + a[:i], c) for c in (b, b[::-1]) for i in range(len(a))) + + def same_parts(a, b, same): + rest = list(b) + for part in a: + match = next((i for i, other in enumerate(rest) if same(part, other)), None) + if match is None: + return False + rest.pop(match) + return not rest + + def same_polygon(a, b): + return same_ring(a[0], b[0]) and same_parts(a[1:], b[1:], same_ring) + + a, b = old['coordinates'], new['coordinates'] + if old['type'] == 'Point': + return same_line([a], [b]) + if old['type'] == 'LineString': + return same_line(a, b) + if old['type'] == 'Polygon': + return same_polygon(a, b) + if old['type'] == 'MultiPolygon': + return same_parts(a, b, same_polygon) + return a == b def feature_line(key, sequence, fresh): @@ -148,9 +181,9 @@ def feature_line(key, sequence, fresh): return json.dumps(feature, separators=(',', ':'), ensure_ascii=False) -def merge(lines, fresh, rings): +def merge(lines, fresh, rings, tagged): out, seen, sequence = [], set(), 0 - report = {'kept': 0, 'updated': [], 'removed': [], 'added': []} + report = {'kept': 0, 'updated': [], 'removed': [], 'added': [], 'unexported': []} for line in lines: feature = json.loads(line) sequence = max(sequence, feature['properties']['id']) @@ -165,6 +198,10 @@ def merge(lines, fresh, rings): out.append(feature_line(key, feature['properties']['id'], fresh[key])) report['updated'].append(key) continue + if key in tagged: # still tagged, but osmium built no usable geometry: keep the old line + report['unexported'].append(key) + out.append(line) + continue lon, lat = points(feature['geometry']['coordinates'])[0][:2] if inside(rings, lon, lat): report['removed'].append(key) @@ -196,9 +233,12 @@ def main(): if actual != expected: raise RuntimeError('MD5 mismatch for %s: %s != %s' % (pbf.name, actual, expected)) rings = read_poly(fetch(GEOFABRIK + 'norway.poly', args.cache / 'norway.poly')) - fresh = export(args.osmium, pbf, args.cache, extract) + fresh, tagged = export(args.osmium, pbf, args.cache, extract) current = [line for line in DATASET.read_text(encoding='utf-8').splitlines() if line.strip()] - merged, report = merge(current, fresh, rings) + merged, report = merge(current, fresh, rings, tagged) + if report['unexported']: + print('warning: kept the old line of tagged features osmium did not export: %s' + % ', '.join(report['unexported']), file=sys.stderr) payload = '\n'.join(merged) + '\n' if args.verify: if payload != DATASET.read_text(encoding='utf-8'): diff --git a/src/data/datacentersNorwayRefresh.test.mjs b/src/data/datacentersNorwayRefresh.test.mjs new file mode 100644 index 000000000..2b7b7d5bf --- /dev/null +++ b/src/data/datacentersNorwayRefresh.test.mjs @@ -0,0 +1,93 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { fileURLToPath } from 'node:url'; + +const SCRIPT = fileURLToPath( + new URL('../../scripts/build-datacenters-norway.py', import.meta.url), +); + +function python(t, source) { + const result = spawnSync('python3', ['-c', source, SCRIPT], { + encoding: 'utf8', + }); + if (result.error?.code === 'ENOENT') return t.skip('python3 is not installed'); + assert.equal(result.status, 0, result.stderr); +} + +test('Norway refresh compares rings, not vertex sets', (t) => { + python( + t, + String.raw` +import runpy, sys +same = runpy.run_path(sys.argv[1])['same_geometry'] +poly = lambda *rings: {'type': 'Polygon', 'coordinates': list(rings)} +multi = lambda *polys: {'type': 'MultiPolygon', 'coordinates': [p['coordinates'] for p in polys]} +a, b, c, d, e = [0, 0], [4, 0], [4, 4], [0, 4], [2, 1] +assert not same(poly([a, b, c, d, e, a]), poly([a, b, e, c, d, a])), 'reordered edges' +assert same(poly([a, b, c, d, e, a]), poly([c, b, a, e, d, c])), 'ring start and winding' +assert same(poly([a, b, c, d, a]), poly([[0, 1e-7], b, c, d, [0, 1e-7]])), 'tolerance' +outer = [[0, 0], [9, 0], [9, 9], [0, 9], [0, 0]] +one, two = [[1, 1], [2, 1], [2, 2], [1, 1]], [[5, 5], [6, 5], [6, 6], [5, 5]] +assert same(poly(outer, one, two), poly(outer, two, one)), 'hole order' +assert not same(poly(outer, one), poly(one, outer)), 'outer and inner roles swapped' +assert same(multi(poly(one), poly(two)), multi(poly(two), poly(one))), 'part order' +p, q, r, s, u, v = [0, 0], [1, 0], [0, 1], [5, 5], [6, 5], [5, 6] +assert not same(multi(poly([p, q, r, p]), poly([s, u, v, s])), + multi(poly([p, q, s, p]), poly([r, u, v, r]))), 'vertices moved between parts' +`, + ); +}); + +test('Norway refresh keeps a still-tagged feature osmium did not export', (t) => { + python( + t, + String.raw` +import json, runpy, sys +merge = runpy.run_path(sys.argv[1])['merge'] +square = [[[10, 60], [10.01, 60], [10.01, 60.01], [10, 60.01], [10, 60]]] +def line(osm_id, sequence): + return json.dumps({'id': osm_id, 'type': 'Feature', 'geometry': {'type': 'Polygon', 'coordinates': square}, + 'properties': {'id': sequence, 'tags': {'building': 'data_center'}, + 'type': 'data_center', 'osm_id': osm_id}}) +norway = [([(0, 0), (20, 0), (20, 80), (0, 80)], False)] +lines = [line(7, 1), line(8, 2)] +out, report = merge(lines, {}, norway, {'w7'}) +assert out == [lines[0]], out +assert report['unexported'] == ['w7'] and report['removed'] == ['w8'], report +`, + ); +}); + +test('Norway refresh lists tagged objects apart from the exported geometry', (t) => { + python( + t, + String.raw` +import json, runpy, subprocess, sys, tempfile +from pathlib import Path +export = runpy.run_path(sys.argv[1])['export'] +tags = {'building': 'data_center'} +ring = [[10, 60], [10.01, 60], [10.01, 60.01], [10, 60]] +features = [ + {'id': 'n1', 'properties': {'telecom': 'data_center'}, 'geometry': {'type': 'Point', 'coordinates': [10, 60]}}, + {'id': 'w5', 'properties': tags, 'geometry': {'type': 'LineString', 'coordinates': ring}}, + {'id': 'a10', 'properties': tags, 'geometry': {'type': 'MultiPolygon', 'coordinates': [[ring]]}}, + {'id': 'w7', 'properties': tags, 'geometry': {'type': 'LineString', 'coordinates': ring[:3]}}, +] +outputs = { + '.opl': 'n1 v1 Ttelecom=data_center x10 y60\nw5 v1 Tbuilding=data_center Nn2,n3,n4,n2\n' + 'w7 v1 Tbuilding=data_center Nn2,n3,n4\nr9 v1 Ttelecom=data_center Mw8@outer\n', + '.geojsonseq': ''.join(chr(30) + json.dumps({'type': 'Feature', **f}) + '\n' for f in features), +} +def run(args, check=False): + target = Path(args[args.index('-o') + 1]) + target.write_text(outputs.get(target.suffix, ''), encoding='utf-8') +subprocess.run = run +with tempfile.TemporaryDirectory() as cache: + fresh, tagged = export('osmium', Path(cache) / 'norway-test.osm.pbf', Path(cache), 'norway-test') +assert sorted(fresh) == ['n1', 'w5'], fresh +assert fresh['w5']['geometry'] == {'type': 'Polygon', 'coordinates': [ring]}, fresh['w5'] +assert tagged == {'n1', 'w5', 'w7', 'r9'}, tagged +`, + ); +}); diff --git a/src/data/local_data/datacenters/README.md b/src/data/local_data/datacenters/README.md index 8b803dcc7..5a80df267 100644 --- a/src/data/local_data/datacenters/README.md +++ b/src/data/local_data/datacenters/README.md @@ -27,7 +27,9 @@ reflects OSM as of the extract below, everything else the original snapshot. 2026-10-05T20:21:35Z, MD5 `1d2cfdeb98980fce11b8891d0f9146c0` - Filter: `osmium tags-filter nwr/telecom=data_center nwr/building=data_center` - Contact tags are dropped as above; unchanged features keep their original - line, so geometry is only rewritten when tags or vertices actually changed. + line, so geometry is only rewritten when tags or rings actually changed. + A feature is removed only once it is no longer tagged; one still tagged but + without an exportable geometry keeps its line and is reported as unexported. - Refresh: `python3 scripts/build-datacenters-norway.py --cache ` uses Geofabrik's current Norway extract and prints the dated name it used; record that name, its OSM timestamp and MD5 here. Requires `osmium-tool`. From 1f0a8b5f1640921d0742cd94cdd8520310a90e2f Mon Sep 17 00:00:00 2001 From: Trygve Hemminghytt <274629477+trygvehhh@users.noreply.github.com> Date: Tue, 3 Nov 2026 22:58:09 +0100 Subject: [PATCH 5/9] fix(datacenters): report newly tagged features osmium cannot export A still-tagged feature without exportable geometry already keeps its old line. A newly tagged one had nowhere to go and was dropped silently; it is now listed as skipped with a warning, so a refresh never loses a site or hides a broken one. --- scripts/build-datacenters-norway.py | 10 ++++++++-- src/data/datacentersNorwayRefresh.test.mjs | 3 ++- src/data/local_data/datacenters/README.md | 3 ++- 3 files changed, 12 insertions(+), 4 deletions(-) diff --git a/scripts/build-datacenters-norway.py b/scripts/build-datacenters-norway.py index 03286b864..d1203f38d 100644 --- a/scripts/build-datacenters-norway.py +++ b/scripts/build-datacenters-norway.py @@ -182,12 +182,13 @@ def feature_line(key, sequence, fresh): def merge(lines, fresh, rings, tagged): - out, seen, sequence = [], set(), 0 - report = {'kept': 0, 'updated': [], 'removed': [], 'added': [], 'unexported': []} + out, seen, known, sequence = [], set(), set(), 0 + report = {'kept': 0, 'updated': [], 'removed': [], 'added': [], 'unexported': [], 'skipped': []} for line in lines: feature = json.loads(line) sequence = max(sequence, feature['properties']['id']) key = osm_key(feature) + known.add(key) if key in fresh: seen.add(key) old_tags = feature['properties']['tags'] @@ -211,6 +212,8 @@ def merge(lines, fresh, rings, tagged): sequence += 1 out.append(feature_line(key, sequence, fresh[key])) report['added'].append(key) + # newly tagged, but osmium built no usable geometry: nothing to add yet + report['skipped'] = sorted(set(tagged) - set(fresh) - known, key=lambda k: (k[0], int(k[1:]))) if len({osm_key(json.loads(line)) for line in out}) != len(out): raise ValueError('Duplicate OSM identity') return out, report @@ -239,6 +242,9 @@ def main(): if report['unexported']: print('warning: kept the old line of tagged features osmium did not export: %s' % ', '.join(report['unexported']), file=sys.stderr) + if report['skipped']: + print('warning: did not add newly tagged features osmium did not export: %s' + % ', '.join(report['skipped']), file=sys.stderr) payload = '\n'.join(merged) + '\n' if args.verify: if payload != DATASET.read_text(encoding='utf-8'): diff --git a/src/data/datacentersNorwayRefresh.test.mjs b/src/data/datacentersNorwayRefresh.test.mjs index 2b7b7d5bf..88527295b 100644 --- a/src/data/datacentersNorwayRefresh.test.mjs +++ b/src/data/datacentersNorwayRefresh.test.mjs @@ -52,9 +52,10 @@ def line(osm_id, sequence): 'type': 'data_center', 'osm_id': osm_id}}) norway = [([(0, 0), (20, 0), (20, 80), (0, 80)], False)] lines = [line(7, 1), line(8, 2)] -out, report = merge(lines, {}, norway, {'w7'}) +out, report = merge(lines, {}, norway, {'w7', 'w12'}) assert out == [lines[0]], out assert report['unexported'] == ['w7'] and report['removed'] == ['w8'], report +assert report['skipped'] == ['w12'] and report['added'] == [], report `, ); }); diff --git a/src/data/local_data/datacenters/README.md b/src/data/local_data/datacenters/README.md index 5a80df267..86f573c61 100644 --- a/src/data/local_data/datacenters/README.md +++ b/src/data/local_data/datacenters/README.md @@ -29,7 +29,8 @@ reflects OSM as of the extract below, everything else the original snapshot. - Contact tags are dropped as above; unchanged features keep their original line, so geometry is only rewritten when tags or rings actually changed. A feature is removed only once it is no longer tagged; one still tagged but - without an exportable geometry keeps its line and is reported as unexported. + without an exportable geometry keeps its line and is reported as unexported, + and a newly tagged one osmium cannot export is reported as skipped. - Refresh: `python3 scripts/build-datacenters-norway.py --cache ` uses Geofabrik's current Norway extract and prints the dated name it used; record that name, its OSM timestamp and MD5 here. Requires `osmium-tool`. From 067383fda80b7bbc66bf792275433df771910426 Mon Sep 17 00:00:00 2001 From: German Khokhlov Date: Thu, 5 Nov 2026 21:42:22 +0400 Subject: [PATCH 6/9] fix(datacenters): drop the redundant seen set and test skipped against exports The merge test now passes an exported newly tagged feature, so removing '- set(fresh)' from the skipped report fails it. --- scripts/build-datacenters-norway.py | 5 ++--- src/data/datacentersNorwayRefresh.test.mjs | 7 ++++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/scripts/build-datacenters-norway.py b/scripts/build-datacenters-norway.py index d1203f38d..3c5eb92dd 100644 --- a/scripts/build-datacenters-norway.py +++ b/scripts/build-datacenters-norway.py @@ -182,7 +182,7 @@ def feature_line(key, sequence, fresh): def merge(lines, fresh, rings, tagged): - out, seen, known, sequence = [], set(), set(), 0 + out, known, sequence = [], set(), 0 report = {'kept': 0, 'updated': [], 'removed': [], 'added': [], 'unexported': [], 'skipped': []} for line in lines: feature = json.loads(line) @@ -190,7 +190,6 @@ def merge(lines, fresh, rings, tagged): key = osm_key(feature) known.add(key) if key in fresh: - seen.add(key) old_tags = feature['properties']['tags'] if old_tags == fresh[key]['tags'] and same_geometry(feature['geometry'], fresh[key]['geometry']): out.append(line) @@ -208,7 +207,7 @@ def merge(lines, fresh, rings, tagged): report['removed'].append(key) continue out.append(line) - for key in sorted(set(fresh) - seen, key=lambda k: (k[0], int(k[1:]))): + for key in sorted(set(fresh) - known, key=lambda k: (k[0], int(k[1:]))): sequence += 1 out.append(feature_line(key, sequence, fresh[key])) report['added'].append(key) diff --git a/src/data/datacentersNorwayRefresh.test.mjs b/src/data/datacentersNorwayRefresh.test.mjs index 88527295b..99278769e 100644 --- a/src/data/datacentersNorwayRefresh.test.mjs +++ b/src/data/datacentersNorwayRefresh.test.mjs @@ -52,10 +52,11 @@ def line(osm_id, sequence): 'type': 'data_center', 'osm_id': osm_id}}) norway = [([(0, 0), (20, 0), (20, 80), (0, 80)], False)] lines = [line(7, 1), line(8, 2)] -out, report = merge(lines, {}, norway, {'w7', 'w12'}) -assert out == [lines[0]], out +fresh = {'w13': {'geometry': {'type': 'Polygon', 'coordinates': square}, 'tags': {'building': 'data_center'}}} +out, report = merge(lines, fresh, norway, {'w7', 'w12', 'w13'}) +assert out[0] == lines[0] and len(out) == 2, out assert report['unexported'] == ['w7'] and report['removed'] == ['w8'], report -assert report['skipped'] == ['w12'] and report['added'] == [], report +assert report['skipped'] == ['w12'] and report['added'] == ['w13'], report `, ); }); From 5d8f2b878f5ef204b5efb4519a7fbbbc3699b59f Mon Sep 17 00:00:00 2001 From: German Khokhlov Date: Fri, 6 Nov 2026 15:47:36 +0400 Subject: [PATCH 7/9] chore(datacenters): drop the one-off Norway refresh script The data refresh is committed and its provenance (extract, timestamp, MD5, filter) stays in the dataset README. The script needed osmium-tool and could only re-verify while Geofabrik still listed that extract, so it goes with its test. --- changelog.d/953-datacenters-norway-refresh.md | 6 +- scripts/build-datacenters-norway.py | 259 ------------------ src/data/datacentersNorwayRefresh.test.mjs | 95 ------- src/data/local_data/datacenters/README.md | 12 +- 4 files changed, 3 insertions(+), 369 deletions(-) delete mode 100644 scripts/build-datacenters-norway.py delete mode 100644 src/data/datacentersNorwayRefresh.test.mjs diff --git a/changelog.d/953-datacenters-norway-refresh.md b/changelog.d/953-datacenters-norway-refresh.md index 48a019605..5db897234 100644 --- a/changelog.d/953-datacenters-norway-refresh.md +++ b/changelog.d/953-datacenters-norway-refresh.md @@ -1,6 +1,4 @@ - Refresh Norway's datacenters from a Geofabrik OpenStreetMap extract (2026-10-05). Troll Housing joins the layer (4,352 features) and four - Norwegian sites pick up current OSM tags; no site moved or was removed. - `scripts/build-datacenters-norway.py` reproduces or `--verify`s the merge, - touching only features inside Geofabrik's Norway boundary, and the dataset - README records the extract, checksum and filter (trygvehhh, #953). + Norwegian sites pick up current OSM tags; no site moved or was removed + (trygvehhh, #953). diff --git a/scripts/build-datacenters-norway.py b/scripts/build-datacenters-norway.py deleted file mode 100644 index 3c5eb92dd..000000000 --- a/scripts/build-datacenters-norway.py +++ /dev/null @@ -1,259 +0,0 @@ -#!/usr/bin/env python3 -"""Maintainer-only Norway refresh of the bundled datacenter extract (osmium-tool). - -Downloads a dated Geofabrik Norway extract, verifies its MD5, filters -telecom=data_center / building=data_center with osmium, and merges the result -into datacenters.geojsonl: features inside the Geofabrik Norway boundary are -added, updated or removed once no longer tagged; every other line is kept -byte-for-byte. - -Geofabrik keeps only its newest dated extract, so --extract defaults to the -current one and a recorded extract can be re-verified only while it is listed. -""" -import argparse -import hashlib -import json -import math -from pathlib import Path -import shutil -import subprocess -import sys -import urllib.request - -ROOT = Path(__file__).resolve().parent.parent -GEOFABRIK = 'https://download.geofabrik.de/europe/' -DATASET = ROOT / 'src/data/local_data/datacenters/datacenters.geojsonl' -CONTACT_TAGS = {'email', 'phone', 'fax', 'mobile', 'whatsapp', 'contact:email', 'contact:phone', - 'contact:fax', 'contact:mobile', 'contact:whatsapp'} -SAME_GEOMETRY_METERS = 0.05 -TIMEOUT_SECONDS = 60 - - -def latest_extract(): - """Resolve norway-latest to the dated name Geofabrik redirects it to.""" - request = urllib.request.Request(GEOFABRIK + 'norway-latest.osm.pbf', method='HEAD') - with urllib.request.urlopen(request, timeout=TIMEOUT_SECONDS) as response: - name = response.geturl().rsplit('/', 1)[-1] - if not name.startswith('norway-') or not name.endswith('.osm.pbf') or name == 'norway-latest.osm.pbf': - raise RuntimeError('Unexpected Geofabrik latest target: %s' % name) - return name[:-len('.osm.pbf')] - - -def fetch(url, target): - if not target.exists(): - with urllib.request.urlopen(url, timeout=TIMEOUT_SECONDS) as response, \ - open(str(target) + '.part', 'wb') as out: - shutil.copyfileobj(response, out) - Path(str(target) + '.part').rename(target) - return target - - -def md5(path): - digest = hashlib.md5() - with open(path, 'rb') as handle: - for block in iter(lambda: handle.read(1 << 20), b''): - digest.update(block) - return digest.hexdigest() - - -def read_poly(path): - """Geofabrik .poly -> list of (ring, is_hole).""" - rings, ring, hole = [], None, False - for raw in path.read_text(encoding='utf-8').splitlines()[1:]: - line = raw.strip() - if ring is None: - if line and line != 'END': - ring, hole = [], line.startswith('!') - elif line == 'END': - rings.append((ring, hole)) - ring = None - elif line: - lon, lat = (float(v) for v in line.split()) - ring.append((lon, lat)) - return rings - - -def inside(rings, lon, lat): - def in_ring(ring): - hit = False - for (x1, y1), (x2, y2) in zip(ring, ring[1:] + ring[:1]): - if (y1 > lat) != (y2 > lat) and lon < x1 + (lat - y1) * (x2 - x1) / (y2 - y1): - hit = not hit - return hit - return any(in_ring(r) for r, h in rings if not h) and not any(in_ring(r) for r, h in rings if h) - - -def points(coordinates): - if isinstance(coordinates[0], (int, float)): - return [coordinates] - return [p for part in coordinates for p in points(part)] - - -def osm_key(feature): - osm_id = feature['properties']['osm_id'] - if osm_id < 0: - return 'r%d' % -osm_id - return ('n' if feature['geometry']['type'] == 'Point' else 'w') + str(osm_id) - - -def export(osmium, pbf, cache, extract): - """Exported features by key, and the keys of every tagged object in the extract.""" - filtered = cache / (extract + '-datacenters.osm.pbf') - listed = cache / (extract + '-datacenters.opl') - exported = cache / (extract + '-datacenters.geojsonseq') - subprocess.run([osmium, 'tags-filter', '-O', '-o', str(filtered), str(pbf), - 'nwr/telecom=data_center', 'nwr/building=data_center'], check=True) - subprocess.run([osmium, 'tags-filter', '-O', '-R', '-f', 'opl', '-o', str(listed), str(filtered), - 'nwr/telecom=data_center', 'nwr/building=data_center'], check=True) - tagged = {line.split(' ', 1)[0] for line in listed.read_text(encoding='utf-8').splitlines() if line} - subprocess.run([osmium, 'export', '-O', '-f', 'geojsonseq', '--add-unique-id=type_id', - '-o', str(exported), str(filtered)], check=True) - fresh = {} - for line in exported.read_text(encoding='utf-8').splitlines(): - line = line.lstrip('\x1e').strip() - if not line: - continue - feature = json.loads(line) - tags = {k: v for k, v in feature['properties'].items() if k not in CONTACT_TAGS} - if tags.get('telecom') != 'data_center' and tags.get('building') != 'data_center': - continue # untagged member ways exported only for geometry - key = feature['id'] - geometry = feature['geometry'] - if key[0] == 'a': # osmium area id: way * 2 or relation * 2 + 1 - number = int(key[1:]) - key = ('w%d' % (number // 2)) if number % 2 == 0 else ('r%d' % (number // 2)) - if key[0] == 'r': - tags = {**tags, 'type': 'multipolygon'} - if geometry['type'] == 'MultiPolygon' and len(geometry['coordinates']) == 1: - geometry = {'type': 'Polygon', 'coordinates': geometry['coordinates'][0]} - elif key in fresh or geometry['type'] == 'LineString': - continue # closed ways are taken from their area export - fresh[key] = {'geometry': geometry, 'tags': tags} - return fresh, tagged - - -def same_geometry(old, new): - """Same type and rings within tolerance; ring start, winding and part or hole order may differ.""" - if old['type'] != new['type']: - return False - - def meters(p, q): - dx = (p[0] - q[0]) * 111320 * math.cos(math.radians(p[1])) - return math.hypot(dx, (p[1] - q[1]) * 110540) - - def same_line(a, b): - return len(a) == len(b) and all(meters(p, q) <= SAME_GEOMETRY_METERS for p, q in zip(a, b)) - - def same_ring(a, b): - a, b = a[:-1], b[:-1] # closed rings repeat their first vertex - return len(a) == len(b) and \ - any(same_line(a[i:] + a[:i], c) for c in (b, b[::-1]) for i in range(len(a))) - - def same_parts(a, b, same): - rest = list(b) - for part in a: - match = next((i for i, other in enumerate(rest) if same(part, other)), None) - if match is None: - return False - rest.pop(match) - return not rest - - def same_polygon(a, b): - return same_ring(a[0], b[0]) and same_parts(a[1:], b[1:], same_ring) - - a, b = old['coordinates'], new['coordinates'] - if old['type'] == 'Point': - return same_line([a], [b]) - if old['type'] == 'LineString': - return same_line(a, b) - if old['type'] == 'Polygon': - return same_polygon(a, b) - if old['type'] == 'MultiPolygon': - return same_parts(a, b, same_polygon) - return a == b - - -def feature_line(key, sequence, fresh): - osm_id = int(key[1:]) * (-1 if key[0] == 'r' else 1) - feature = {'id': osm_id, 'type': 'Feature', 'geometry': fresh['geometry'], - 'properties': {'id': sequence, 'tags': fresh['tags'], 'type': 'data_center', - 'osm_id': osm_id}} - return json.dumps(feature, separators=(',', ':'), ensure_ascii=False) - - -def merge(lines, fresh, rings, tagged): - out, known, sequence = [], set(), 0 - report = {'kept': 0, 'updated': [], 'removed': [], 'added': [], 'unexported': [], 'skipped': []} - for line in lines: - feature = json.loads(line) - sequence = max(sequence, feature['properties']['id']) - key = osm_key(feature) - known.add(key) - if key in fresh: - old_tags = feature['properties']['tags'] - if old_tags == fresh[key]['tags'] and same_geometry(feature['geometry'], fresh[key]['geometry']): - out.append(line) - report['kept'] += 1 - else: - out.append(feature_line(key, feature['properties']['id'], fresh[key])) - report['updated'].append(key) - continue - if key in tagged: # still tagged, but osmium built no usable geometry: keep the old line - report['unexported'].append(key) - out.append(line) - continue - lon, lat = points(feature['geometry']['coordinates'])[0][:2] - if inside(rings, lon, lat): - report['removed'].append(key) - continue - out.append(line) - for key in sorted(set(fresh) - known, key=lambda k: (k[0], int(k[1:]))): - sequence += 1 - out.append(feature_line(key, sequence, fresh[key])) - report['added'].append(key) - # newly tagged, but osmium built no usable geometry: nothing to add yet - report['skipped'] = sorted(set(tagged) - set(fresh) - known, key=lambda k: (k[0], int(k[1:]))) - if len({osm_key(json.loads(line)) for line in out}) != len(out): - raise ValueError('Duplicate OSM identity') - return out, report - - -def main(): - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument('--cache', type=Path, required=True) - parser.add_argument('--osmium', default=shutil.which('osmium') or 'osmium') - parser.add_argument('--extract', default='latest', - help="dated Geofabrik name such as norway-261005, or 'latest'") - parser.add_argument('--verify', action='store_true') - args = parser.parse_args() - args.cache.mkdir(parents=True, exist_ok=True) - extract = latest_extract() if args.extract == 'latest' else args.extract - pbf = fetch(GEOFABRIK + extract + '.osm.pbf', args.cache / (extract + '.osm.pbf')) - expected = fetch(GEOFABRIK + extract + '.osm.pbf.md5', - args.cache / (extract + '.osm.pbf.md5')).read_text(encoding='utf-8').split()[0] - actual = md5(pbf) - if actual != expected: - raise RuntimeError('MD5 mismatch for %s: %s != %s' % (pbf.name, actual, expected)) - rings = read_poly(fetch(GEOFABRIK + 'norway.poly', args.cache / 'norway.poly')) - fresh, tagged = export(args.osmium, pbf, args.cache, extract) - current = [line for line in DATASET.read_text(encoding='utf-8').splitlines() if line.strip()] - merged, report = merge(current, fresh, rings, tagged) - if report['unexported']: - print('warning: kept the old line of tagged features osmium did not export: %s' - % ', '.join(report['unexported']), file=sys.stderr) - if report['skipped']: - print('warning: did not add newly tagged features osmium did not export: %s' - % ', '.join(report['skipped']), file=sys.stderr) - payload = '\n'.join(merged) + '\n' - if args.verify: - if payload != DATASET.read_text(encoding='utf-8'): - raise SystemExit('datacenters.geojsonl is not the %s merge' % extract) - else: - with open(DATASET, 'w', encoding='utf-8', newline='\n') as handle: - handle.write(payload) - print(json.dumps({'extract': extract, 'md5': actual, 'norway': len(fresh), - 'features': len(merged), **report})) - - -if __name__ == '__main__': - main() diff --git a/src/data/datacentersNorwayRefresh.test.mjs b/src/data/datacentersNorwayRefresh.test.mjs deleted file mode 100644 index 99278769e..000000000 --- a/src/data/datacentersNorwayRefresh.test.mjs +++ /dev/null @@ -1,95 +0,0 @@ -import test from 'node:test'; -import assert from 'node:assert/strict'; -import { spawnSync } from 'node:child_process'; -import { fileURLToPath } from 'node:url'; - -const SCRIPT = fileURLToPath( - new URL('../../scripts/build-datacenters-norway.py', import.meta.url), -); - -function python(t, source) { - const result = spawnSync('python3', ['-c', source, SCRIPT], { - encoding: 'utf8', - }); - if (result.error?.code === 'ENOENT') return t.skip('python3 is not installed'); - assert.equal(result.status, 0, result.stderr); -} - -test('Norway refresh compares rings, not vertex sets', (t) => { - python( - t, - String.raw` -import runpy, sys -same = runpy.run_path(sys.argv[1])['same_geometry'] -poly = lambda *rings: {'type': 'Polygon', 'coordinates': list(rings)} -multi = lambda *polys: {'type': 'MultiPolygon', 'coordinates': [p['coordinates'] for p in polys]} -a, b, c, d, e = [0, 0], [4, 0], [4, 4], [0, 4], [2, 1] -assert not same(poly([a, b, c, d, e, a]), poly([a, b, e, c, d, a])), 'reordered edges' -assert same(poly([a, b, c, d, e, a]), poly([c, b, a, e, d, c])), 'ring start and winding' -assert same(poly([a, b, c, d, a]), poly([[0, 1e-7], b, c, d, [0, 1e-7]])), 'tolerance' -outer = [[0, 0], [9, 0], [9, 9], [0, 9], [0, 0]] -one, two = [[1, 1], [2, 1], [2, 2], [1, 1]], [[5, 5], [6, 5], [6, 6], [5, 5]] -assert same(poly(outer, one, two), poly(outer, two, one)), 'hole order' -assert not same(poly(outer, one), poly(one, outer)), 'outer and inner roles swapped' -assert same(multi(poly(one), poly(two)), multi(poly(two), poly(one))), 'part order' -p, q, r, s, u, v = [0, 0], [1, 0], [0, 1], [5, 5], [6, 5], [5, 6] -assert not same(multi(poly([p, q, r, p]), poly([s, u, v, s])), - multi(poly([p, q, s, p]), poly([r, u, v, r]))), 'vertices moved between parts' -`, - ); -}); - -test('Norway refresh keeps a still-tagged feature osmium did not export', (t) => { - python( - t, - String.raw` -import json, runpy, sys -merge = runpy.run_path(sys.argv[1])['merge'] -square = [[[10, 60], [10.01, 60], [10.01, 60.01], [10, 60.01], [10, 60]]] -def line(osm_id, sequence): - return json.dumps({'id': osm_id, 'type': 'Feature', 'geometry': {'type': 'Polygon', 'coordinates': square}, - 'properties': {'id': sequence, 'tags': {'building': 'data_center'}, - 'type': 'data_center', 'osm_id': osm_id}}) -norway = [([(0, 0), (20, 0), (20, 80), (0, 80)], False)] -lines = [line(7, 1), line(8, 2)] -fresh = {'w13': {'geometry': {'type': 'Polygon', 'coordinates': square}, 'tags': {'building': 'data_center'}}} -out, report = merge(lines, fresh, norway, {'w7', 'w12', 'w13'}) -assert out[0] == lines[0] and len(out) == 2, out -assert report['unexported'] == ['w7'] and report['removed'] == ['w8'], report -assert report['skipped'] == ['w12'] and report['added'] == ['w13'], report -`, - ); -}); - -test('Norway refresh lists tagged objects apart from the exported geometry', (t) => { - python( - t, - String.raw` -import json, runpy, subprocess, sys, tempfile -from pathlib import Path -export = runpy.run_path(sys.argv[1])['export'] -tags = {'building': 'data_center'} -ring = [[10, 60], [10.01, 60], [10.01, 60.01], [10, 60]] -features = [ - {'id': 'n1', 'properties': {'telecom': 'data_center'}, 'geometry': {'type': 'Point', 'coordinates': [10, 60]}}, - {'id': 'w5', 'properties': tags, 'geometry': {'type': 'LineString', 'coordinates': ring}}, - {'id': 'a10', 'properties': tags, 'geometry': {'type': 'MultiPolygon', 'coordinates': [[ring]]}}, - {'id': 'w7', 'properties': tags, 'geometry': {'type': 'LineString', 'coordinates': ring[:3]}}, -] -outputs = { - '.opl': 'n1 v1 Ttelecom=data_center x10 y60\nw5 v1 Tbuilding=data_center Nn2,n3,n4,n2\n' - 'w7 v1 Tbuilding=data_center Nn2,n3,n4\nr9 v1 Ttelecom=data_center Mw8@outer\n', - '.geojsonseq': ''.join(chr(30) + json.dumps({'type': 'Feature', **f}) + '\n' for f in features), -} -def run(args, check=False): - target = Path(args[args.index('-o') + 1]) - target.write_text(outputs.get(target.suffix, ''), encoding='utf-8') -subprocess.run = run -with tempfile.TemporaryDirectory() as cache: - fresh, tagged = export('osmium', Path(cache) / 'norway-test.osm.pbf', Path(cache), 'norway-test') -assert sorted(fresh) == ['n1', 'w5'], fresh -assert fresh['w5']['geometry'] == {'type': 'Polygon', 'coordinates': [ring]}, fresh['w5'] -assert tagged == {'n1', 'w5', 'w7', 'r9'}, tagged -`, - ); -}); diff --git a/src/data/local_data/datacenters/README.md b/src/data/local_data/datacenters/README.md index 86f573c61..c360ee527 100644 --- a/src/data/local_data/datacenters/README.md +++ b/src/data/local_data/datacenters/README.md @@ -26,14 +26,4 @@ reflects OSM as of the extract below, everything else the original snapshot. - Extract: Geofabrik `norway-261005.osm.pbf`, OSM data 2026-10-05T20:21:35Z, MD5 `1d2cfdeb98980fce11b8891d0f9146c0` - Filter: `osmium tags-filter nwr/telecom=data_center nwr/building=data_center` -- Contact tags are dropped as above; unchanged features keep their original - line, so geometry is only rewritten when tags or rings actually changed. - A feature is removed only once it is no longer tagged; one still tagged but - without an exportable geometry keeps its line and is reported as unexported, - and a newly tagged one osmium cannot export is reported as skipped. -- Refresh: `python3 scripts/build-datacenters-norway.py --cache ` uses - Geofabrik's current Norway extract and prints the dated name it used; record - that name, its OSM timestamp and MD5 here. Requires `osmium-tool`. -- Check: add `--extract --verify` to compare without writing. - Geofabrik keeps only its newest dated extract, so a recorded extract can be - re-verified only while it is still listed; the MD5 above identifies it. +- Contact tags are dropped as above. From a69d0492d0cda16a1eef15d022629f4949f293b5 Mon Sep 17 00:00:00 2001 From: German Khokhlov Date: Fri, 6 Nov 2026 15:54:53 +0400 Subject: [PATCH 8/9] docs(datacenters): tighten the Norway refresh notes --- changelog.d/953-datacenters-norway-refresh.md | 6 ++---- docs/INFRASTRUCTURE-LAYERS.md | 2 +- src/data/local_data/datacenters/README.md | 6 ++---- 3 files changed, 5 insertions(+), 9 deletions(-) diff --git a/changelog.d/953-datacenters-norway-refresh.md b/changelog.d/953-datacenters-norway-refresh.md index 5db897234..d60d4ac00 100644 --- a/changelog.d/953-datacenters-norway-refresh.md +++ b/changelog.d/953-datacenters-norway-refresh.md @@ -1,4 +1,2 @@ -- Refresh Norway's datacenters from a Geofabrik OpenStreetMap extract - (2026-10-05). Troll Housing joins the layer (4,352 features) and four - Norwegian sites pick up current OSM tags; no site moved or was removed - (trygvehhh, #953). +- Add Troll Housing to the Datacenters layer (4,352 features) by refreshing + Norway from OpenStreetMap data of 2026-10-05 (trygvehhh, #953). diff --git a/docs/INFRASTRUCTURE-LAYERS.md b/docs/INFRASTRUCTURE-LAYERS.md index aa8cd0558..798ff09af 100644 --- a/docs/INFRASTRUCTURE-LAYERS.md +++ b/docs/INFRASTRUCTURE-LAYERS.md @@ -59,4 +59,4 @@ globals. Dataset files and source/license notices remain under `src/data/local_data/`. Their input counts are 4,352 datacenter features and 704 dam features. Cesium may expand multipart geometries into multiple entities; entity and stem counts are -different measurements. This change does not refresh or relicense the datasets. +different measurements. diff --git a/src/data/local_data/datacenters/README.md b/src/data/local_data/datacenters/README.md index c360ee527..a89894fdb 100644 --- a/src/data/local_data/datacenters/README.md +++ b/src/data/local_data/datacenters/README.md @@ -19,11 +19,9 @@ snapshot. Future refreshes should record both before replacing the file. ## Norway refresh -Features inside the Geofabrik Norway boundary (`norway.poly`, which includes -Svalbard) were refreshed separately, so this file mixes two vintages: Norway -reflects OSM as of the extract below, everything else the original snapshot. +Features inside Geofabrik's Norway boundary (`norway.poly`, with Svalbard) come +from a newer extract than the rest of the file. Contact tags are dropped as above. - Extract: Geofabrik `norway-261005.osm.pbf`, OSM data 2026-10-05T20:21:35Z, MD5 `1d2cfdeb98980fce11b8891d0f9146c0` - Filter: `osmium tags-filter nwr/telecom=data_center nwr/building=data_center` -- Contact tags are dropped as above. From c058ecf21b98f3097adea6cd512b4931cb864723 Mon Sep 17 00:00:00 2001 From: German Khokhlov Date: Fri, 6 Nov 2026 15:57:27 +0400 Subject: [PATCH 9/9] docs(infrastructure): keep the dataset scope note --- docs/INFRASTRUCTURE-LAYERS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/INFRASTRUCTURE-LAYERS.md b/docs/INFRASTRUCTURE-LAYERS.md index 798ff09af..aa8cd0558 100644 --- a/docs/INFRASTRUCTURE-LAYERS.md +++ b/docs/INFRASTRUCTURE-LAYERS.md @@ -59,4 +59,4 @@ globals. Dataset files and source/license notices remain under `src/data/local_data/`. Their input counts are 4,352 datacenter features and 704 dam features. Cesium may expand multipart geometries into multiple entities; entity and stem counts are -different measurements. +different measurements. This change does not refresh or relicense the datasets.