Bundle Wikidata cities for offline place lookups

A pack of the world's settlements of 15,000 people or more (31,827 places,
1.6 MB, CC0) built from the Wikidata Query Service by
scripts/build-wikidata-cities.mjs. src/data/wikidataCities.js reads it on the
first lookup and answers the nearest city to a point, the cities around it,
a city by name ("Paris, France", or the nearest namesake in view) and a
country's name in the page language. Names match without accents or case.

The build leaves out groupings of settlements (metropolitan areas,
agglomerations) and places that no longer stand, keeps a city that is also a
narrower grouping kind (Istanbul) or a unit dissolved where nothing else
names the spot (the Vietnamese provincial cities of 2025), fills a country
from the area a place lies in, splits a population band the service cannot
answer in time, and fails if well-known cities are missing or a known
grouping gets in. The app never calls Wikidata itself.

Co-authored-by: Sameh Khamis <[email protected]>

DATA_SOURCES.md

@@ -419,6 +419,7 @@ The [Bhote Koshi event pack](public/events/bhote-koshi-2026/README.md), under `p
 | **Natural Earth countries, states and provinces** (260 admin-0/map units + 4,587 admin-1 polygons)             | `natural_earth/`                  | **Public domain**                                                                                         | ✅ (no restrictions)                             | "Made with Natural Earth" (courtesy credit — not legally required)          |
 | **Natural Earth disputed areas** (11 of the source's 99 features, hand-picked; [list](src/data/local_data/natural_earth/README.md#disputed-areas-disputed_areasjson)) | `natural_earth/` | **Public domain** | ✅ (no restrictions) | "Made with Natural Earth" (courtesy credit — not legally required) |
 | **Natural Earth populated places** (7,342 points: name, position, rank)     | `natural_earth/`                  | **Public domain**                                                                                         | ✅ (no restrictions)                             | "Made with Natural Earth" (courtesy credit — not legally required)          |
+| **Wikidata cities** (31,827 settlements of 15,000 people or more: name, position, population, country) | `wikidata_cities/` | **CC0 1.0** (public domain dedication) | ✅ (no restrictions) | "City, town and village names from Wikidata (CC0)" (courtesy credit — not legally required) |
 | **US Census Bureau counties** (3,235 county polygons)                       | `us_census_counties/`             | **Public domain**                                                                                         | ✅ (no restrictions)                             | "U.S. Census Bureau" (courtesy — not legally required)                      |
 | **US Census Bureau places** (32,629 city, town and CDP polygons)            | `us_census_places/`               | **Public domain**                                                                                         | ✅ (no restrictions)                             | "U.S. Census Bureau" (courtesy — not legally required)                      |
 | **Who's On First neighborhoods** (61,441 polygons)                          | `wof_neighborhoods/`              | Per source: CC BY, ODC-By, Open Government licences, CC0, PDDL, public domain (folder `ATTRIBUTION.md`)   | ✅ (attribution)                                 | "Neighborhood boundaries: Who's On First and contributors"; per-source credits in the folder |
@@ -518,6 +519,20 @@ its width ("Sichuan, China · near Chengdu"), offline, with the states and provi
 the division. Public domain; credited "Made with Natural Earth" with the other packs. Rebuild:
 `node scripts/build-admin-packs.mjs --only places`; provenance in the folder README.
 
+### Wikidata cities (`wikidata_cities/`)
+
+Settlements of 15,000 people or more that Wikidata states a population and coordinates for,
+read once from the Wikidata Query Service by `scripts/build-wikidata-cities.mjs`: one row per
+place with its English label, latitude and longitude to 3 decimals, largest stated population,
+ISO 3166-1 country code and Wikidata id (31,827 places, 1.6 MB). The pack is read the
+first time a place name is looked up and gives the nearest city, the places in view and a city
+found by name (`findPlace`: "Paris, France", or the nearest namesake in view). Groupings of
+settlements (metropolitan areas, agglomerations) and places that no longer stand are left out
+unless they are also a living city. Names match without accents ("Sao Paulo" finds São Paulo).
+CC0; credited "City, town and village names from Wikidata (CC0)" as a courtesy once a name from
+the pack is shown. Rebuild: `node scripts/build-wikidata-cities.mjs`; provenance in the folder
+README. The app never calls Wikidata itself.
+
 ### Sky background (`sky/`)
 
 Sources, licences, the changes CC BY asks to state, hashes and build commands

scripts/build-wikidata-cities.mjs

@@ -0,0 +1,455 @@
+#!/usr/bin/env node
+/**
+ * Build the bundled world cities pack read by `src/data/wikidataCities.js`:
+ *
+ *   src/data/local_data/wikidata_cities/cities.json
+ *
+ * Source: Wikidata (CC0), read once from the public query service at build
+ * time. The app itself never calls Wikidata: it reads this file.
+ *
+ * What goes in: items that are a human settlement (an instance of any subclass
+ * of Q486972), have coordinates (P625) and a population (P1082) at or above
+ * the cut-off. Each row keeps the English label, coordinates to three
+ * decimals, the largest stated population, the ISO 3166-1 alpha-2 country
+ * code and the Wikidata id. Groupings of settlements (metropolitan areas,
+ * agglomerations, conurbations) and places that no longer stand are left out.
+ *
+ * The query service limits each query to 60 s, so the population range is
+ * read in bands without labels, the settlement test runs here against the
+ * class list, and labels and countries are read afterwards in batches. One
+ * request runs at a time, with a pause between requests, and a refusal's
+ * Retry-After is honoured.
+ *
+ * Usage:
+ *   node scripts/build-wikidata-cities.mjs [--min-population 15000]
+ */
+
+import { mkdir, writeFile } from 'node:fs/promises';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const ENDPOINT = 'https://query.wikidata.org/sparql';
+const USER_AGENT =
+  'gods-eye-view-places-builder/0.2 (offline data pack; one-off build; https://github.com/bilawalsidhu/gods-eye-view)';
+const SETTLEMENT = 'Q486972';
+const PAUSE_MS = 1500;
+const LABEL_BATCH = 400;
+
+const args = process.argv.slice(2);
+const flag = (name) =>
+  args.includes(name) ? Number(args[args.indexOf(name) + 1]) : NaN;
+const minPopulation = flag('--min-population') || 15000;
+const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
+const outDir = path.join(root, 'src/data/local_data/wikidata_cities');
+
+const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
+const qid = (uri) => uri.slice(uri.lastIndexOf('/') + 1);
+
+/** The service ran out of time on a query: a smaller one may still answer. */
+class QueryTimeout extends Error {}
+
+async function sparql(query, attempt = 1) {
+  let response;
+  try {
+    response = await fetch(ENDPOINT, {
+      method: 'POST',
+      headers: {
+        'User-Agent': USER_AGENT,
+        Accept: 'application/sparql-results+json',
+        'Content-Type': 'application/x-www-form-urlencoded',
+      },
+      body: `query=${encodeURIComponent(query)}`,
+      // The service stops a query at 60 s; past that the answer is lost.
+      signal: AbortSignal.timeout(90000),
+    });
+  } catch (error) {
+    // Our own time limit: the query is too big, as with a 504.
+    if (error.name === 'TimeoutError' && attempt >= 2)
+      throw new QueryTimeout('Wikidata did not answer in time');
+    if (attempt >= 4) throw error;
+    console.warn(`  ${error.message}; waiting ${20 * attempt} s`);
+    await sleep(20000 * attempt);
+    return sparql(query, attempt + 1);
+  }
+  if (response.status === 504 && attempt >= 2)
+    throw new QueryTimeout('Wikidata answered 504');
+  // A malformed query fails the same way every time.
+  if (
+    response.status >= 400 &&
+    response.status < 500 &&
+    response.status !== 429
+  )
+    throw new Error(
+      `Wikidata answered ${response.status}: ${(await response.text()).slice(0, 300)}`,
+    );
+  if (!response.ok) {
+    if (attempt >= 4) throw new Error(`Wikidata answered ${response.status}`);
+    const wait =
+      Number(response.headers.get('retry-after')) * 1000 || 20000 * attempt;
+    console.warn(`  ${response.status}; waiting ${Math.round(wait / 1000)} s`);
+    await sleep(wait);
+    return sparql(query, attempt + 1);
+  }
+  // The service can stop a long answer partway; that arrives as broken JSON.
+  let rows;
+  try {
+    rows = JSON.parse(await response.text()).results.bindings;
+  } catch (error) {
+    // The service stops a query that runs too long partway through its answer.
+    if (attempt >= 2)
+      throw new QueryTimeout(`Wikidata answer was cut short: ${error.message}`);
+    console.warn(`  answer cut short; waiting ${20 * attempt} s`);
+    await sleep(20000 * attempt);
+    return sparql(query, attempt + 1);
+  }
+  await sleep(PAUSE_MS);
+  return rows;
+}
+
+/**
+ * Population bands, narrow where places are many, so each answer stays well
+ * under the service's 60 s limit (about 35,000 places a band).
+ */
+function bands(min, max) {
+  const stepAt = (pop) =>
+    pop < 2000
+      ? 200
+      : pop < 5000
+        ? 500
+        : pop < 10000
+          ? 1000
+          : pop < 20000
+            ? 2500
+            : pop < 30000
+              ? 10000
+              : pop < 50000
+                ? 20000
+                : pop < 100000
+                  ? 50000
+                  : Infinity;
+  const list = [];
+  for (let low = min; low < max;) {
+    const high = Math.min(max, low + stepAt(low));
+    list.push([low, high]);
+    low = high;
+  }
+  return list;
+}
+
+console.log('Settlement classes…');
+const classes = new Set(
+  (await sparql(`SELECT ?c WHERE { ?c wdt:P279* wd:${SETTLEMENT} }`)).map(
+    (row) => qid(row.c.value),
+  ),
+);
+console.log(`  ${classes.size} classes`);
+
+// Groupings of settlements (metropolitan areas, agglomerations, conurbations,
+// capital regions)
+// are settlements in Wikidata too, and are left out so a view is named after
+// the city rather than the area around it. An item stated as one of these
+// classes goes, whatever else it is ("New York metropolitan area" is also a
+// megacity). An item of a narrower kind only goes when it is nothing else:
+// Istanbul is a "metropolitan municipality in Turkey", a kind of
+// metropolitan area, and a megacity, so it stays.
+const GROUPING_TREES = [
+  'Q1907114', // metropolitan area
+  'Q159313', // urban agglomeration
+  'Q245260', // conurbation
+  'Q1062177', // capital region
+  'Q1768043', // metropolitan statistical area
+  'Q3175539', // functional urban area
+];
+// Left out as stated only: "city in the United States" is a subclass of
+// these, so their subclasses must stay.
+const GROUPING_CLASSES = [
+  'Q702492', // urban area
+  'Q2342385', // urban area (statistical)
+  'Q5951278', // urban area of the United States
+];
+// Places that no longer stand as settlements (former entities, ghost towns,
+// abandoned villages, archaeological sites) are settlements in Wikidata too.
+// An item typed only as one of these, with every subclass, is left out; one
+// that is also a living settlement (a city that is also a former capital)
+// stays.
+const HISTORICAL_TREES = [
+  'Q15893266', // former entity
+  'Q19953632', // former administrative territorial entity
+  'Q74047', // ghost town
+  'Q350895', // abandoned village
+  'Q839954', // archaeological site
+];
+const former = new Set(
+  (
+    await sparql(
+      `SELECT ?c WHERE { VALUES ?top { ${HISTORICAL_TREES.map((id) => `wd:${id}`).join(' ')} } ?c wdt:P279* ?top }`,
+    )
+  ).map((row) => qid(row.c.value)),
+);
+console.log(`  ${former.size} former-place classes`);
+const groupings = new Set([
+  ...(
+    await sparql(
+      `SELECT ?c WHERE { VALUES ?top { ${GROUPING_TREES.map((id) => `wd:${id}`).join(' ')} } ?c wdt:P279* ?top }`,
+    )
+  ).map((row) => qid(row.c.value)),
+  ...GROUPING_CLASSES,
+]);
+console.log(`  ${groupings.size} grouping classes left out`);
+// A band's rows; one the service cannot answer in time is asked again in two
+// halves.
+async function bandRows(low, high) {
+  const range = Number.isFinite(high)
+    ? `?pop >= ${low} && ?pop < ${high}`
+    : `?pop >= ${low}`;
+  try {
+    return await sparql(
+      `SELECT ?item ?pop ?coord ?type WHERE { ?item wdt:P1082 ?pop . FILTER(${range}) ?item wdt:P625 ?coord ; wdt:P31 ?type . }`,
+    );
+  } catch (error) {
+    const mid = Number.isFinite(high) ? Math.round((low + high) / 2) : low * 4;
+    if (!(error instanceof QueryTimeout) || mid <= low || mid === high)
+      throw error;
+    console.warn(`  splitting ${low}–${high} at ${mid}`);
+    return [...(await bandRows(low, mid)), ...(await bandRows(mid, high))];
+  }
+}
+
+const GROUPING_STATED = new Set([...GROUPING_TREES, ...GROUPING_CLASSES]);
+const living = new Set();
+const stated = new Set();
+const grouped = new Set();
+const settled = new Set();
+
+const places = new Map();
+for (const [low, high] of bands(minPopulation, Infinity)) {
+  console.log(`Population ${low}–${Number.isFinite(high) ? high : '∞'}…`);
+  const rows = await bandRows(low, high);
+  let kept = 0;
+  for (const row of rows) {
+    const type = qid(row.type.value);
+    if (GROUPING_STATED.has(type)) stated.add(qid(row.item.value));
+    if (groupings.has(type)) grouped.add(qid(row.item.value));
+    if (!classes.has(type)) continue;
+    if (!groupings.has(type) && !former.has(type))
+      settled.add(qid(row.item.value));
+    if (!former.has(type)) living.add(qid(row.item.value));
+    const match = /^Point\(([-\d.eE]+) ([-\d.eE]+)\)$/.exec(row.coord.value);
+    if (!match) continue;
+    const id = qid(row.item.value);
+    const pop = Math.round(Number(row.pop.value));
+    const known = places.get(id);
+    if (known) {
+      known.pop = Math.max(known.pop, pop);
+      // Of several stated coordinates, the same one on every build.
+      const lon = Number(match[1]);
+      const lat = Number(match[2]);
+      if (lat < known.lat || (lat === known.lat && lon < known.lon)) {
+        known.lat = lat;
+        known.lon = lon;
+      }
+      continue;
+    }
+    places.set(id, { id, pop, lon: Number(match[1]), lat: Number(match[2]) });
+    kept++;
+  }
+  console.log(`  ${rows.length} rows, ${kept} new settlements`);
+}
+
+let groupingsLeftOut = 0;
+for (const id of grouped) {
+  if ((settled.has(id) && !stated.has(id)) || !places.delete(id)) continue;
+  groupingsLeftOut++;
+}
+console.log(`${groupingsLeftOut} groupings left out`);
+// A place typed only as a former kind is held until its end date is known: a
+// unit dissolved in a recent reform still stands (below).
+for (const [id, place] of places) if (!living.has(id)) place.formerOnly = true;
+const ids = [...places.keys()];
+console.log(`Labels and countries for ${ids.length} places…`);
+for (let start = 0; start < ids.length; start += LABEL_BATCH) {
+  const values = ids
+    .slice(start, start + LABEL_BATCH)
+    .map((id) => `wd:${id}`)
+    .join(' ');
+  const rows = await sparql(
+    `SELECT ?item ?label ?iso ?ended WHERE { VALUES ?item { ${values} } OPTIONAL { ?item rdfs:label ?label . FILTER(LANG(?label) = "en") } OPTIONAL { ?item wdt:P17 ?country . ?country wdt:P297 ?iso } OPTIONAL { ?item wdt:P576 ?ended } }`,
+  );
+  for (const row of rows) {
+    const place = places.get(qid(row.item.value));
+    if (!place) continue;
+    if (row.label && !place.name) place.name = row.label.value;
+    // Of several stated countries, the same one on every build.
+    if (row.iso && (!place.cc || row.iso.value < place.cc))
+      place.cc = row.iso.value;
+    // Dissolved, abolished or merged as a unit (P576), and when.
+    if (row.ended) {
+      place.ended = true;
+      const year = Number(/^-?\d+/.exec(row.ended.value)?.[0]);
+      if (Number.isFinite(year))
+        place.endedYear = Math.max(place.endedYear ?? -Infinity, year);
+    }
+  }
+  if ((start / LABEL_BATCH) % 10 === 0)
+    console.log(
+      `  ${Math.min(start + LABEL_BATCH, ids.length)} / ${ids.length}`,
+    );
+}
+
+// A dissolved place goes only where another place still names the spot (a
+// city merged into its neighbour, a former district inside its city). Where
+// nothing else does, the town is still there under its name and stays: many
+// of Vietnam's provincial cities ended as units in 2025, and with them their
+// class, so they are typed only as a former kind. A place typed only as a
+// former kind stays only that way: dissolved within the last ten years with
+// nothing else near it. Ghost towns and ancient cities go.
+const SUCCESSOR_KM = 15;
+const RECENT_YEARS = 10;
+const recent = new Date().getUTCFullYear() - RECENT_YEARS;
+const standing = [...places.values()].filter(
+  (place) => place.name && !place.ended && !place.formerOnly,
+);
+const near = (a, b) => {
+  const rad = Math.PI / 180;
+  const h =
+    Math.sin(((b.lat - a.lat) * rad) / 2) ** 2 +
+    Math.cos(a.lat * rad) *
+      Math.cos(b.lat * rad) *
+      Math.sin(((b.lon - a.lon) * rad) / 2) ** 2;
+  return 2 * 6371 * Math.asin(Math.min(1, Math.sqrt(h))) <= SUCCESSOR_KM;
+};
+const byCell = new Map();
+const cellOf = (lat, lon) => `${Math.floor(lat)},${Math.floor(lon)}`;
+for (const place of standing) {
+  const key = cellOf(place.lat, place.lon);
+  if (!byCell.has(key)) byCell.set(key, []);
+  byCell.get(key).push(place);
+}
+let dissolved = 0;
+let formerOnly = 0;
+for (const [id, place] of places) {
+  if (place.formerOnly && !(place.ended && place.endedYear >= recent)) {
+    places.delete(id);
+    formerOnly++;
+    continue;
+  }
+  if (!place.ended) continue;
+  let succeeded = false;
+  for (let dLat = -1; dLat <= 1 && !succeeded; dLat++)
+    for (let dLon = -1; dLon <= 1 && !succeeded; dLon++)
+      succeeded = (
+        byCell.get(cellOf(place.lat + dLat, place.lon + dLon)) || []
+      ).some((other) => near(place, other));
+  if (succeeded) dissolved++;
+  else place.ended = false;
+}
+console.log(`${formerOnly} former places left out`);
+console.log(`${dissolved} dissolved places left out`);
+
+// A place whose stated country is a former one (no ISO code) takes the
+// country of the area it lies in, one area level up at a time (a path of
+// several levels in one query runs past the service's time limit).
+const COUNTRY_BATCH = 50;
+const countryless = () =>
+  [...places.values()]
+    .filter((place) => place.name && !place.cc && !place.ended)
+    .map((place) => place.id);
+for (const hops of [1, 2, 3]) {
+  const ids = countryless();
+  if (!ids.length) break;
+  console.log(`Countries ${hops} area level(s) up for ${ids.length} places…`);
+  const pathToArea = Array.from({ length: hops }, () => 'wdt:P131').join('/');
+  for (let start = 0; start < ids.length; start += COUNTRY_BATCH) {
+    const values = ids
+      .slice(start, start + COUNTRY_BATCH)
+      .map((id) => `wd:${id}`)
+      .join(' ');
+    const rows = await sparql(
+      `SELECT ?item ?iso WHERE { VALUES ?item { ${values} } ?item ${pathToArea} ?area . ?area wdt:P17 ?country . ?country wdt:P297 ?iso }`,
+    );
+    for (const row of rows) {
+      const place = places.get(qid(row.item.value));
+      if (!place || (place.cc && place.hops !== hops)) continue;
+      if (!place.cc || row.iso.value < place.cc) {
+        place.cc = row.iso.value;
+        place.hops = hops;
+      }
+    }
+  }
+}
+console.log(`  ${countryless().length} places still without a country`);
+
+const rows = [...places.values()]
+  .filter(
+    (place) =>
+      place.name &&
+      !place.ended &&
+      Math.abs(place.lat) <= 90 &&
+      Math.abs(place.lon) <= 180,
+  )
+  .sort((a, b) => b.pop - a.pop || a.id.localeCompare(b.id))
+  .map((place) => [
+    place.name,
+    Number(place.lat.toFixed(3)),
+    Number(place.lon.toFixed(3)),
+    place.pop,
+    place.cc || '',
+    place.id,
+  ]);
+
+// A filter that is too wide drops real cities without any error; these must
+// survive, and no settlement grouping may.
+const MUST_HAVE = {
+  Q1490: 'Tokyo',
+  Q84: 'London',
+  Q16559: 'Austin',
+  Q16555: 'Houston',
+  Q1355: 'Bengaluru',
+  Q3630: 'Jakarta',
+  Q174: 'São Paulo',
+  Q34600: 'Kyoto',
+  Q656: 'Saint Petersburg',
+  Q406: 'Istanbul',
+  Q8678: 'Rio de Janeiro',
+  Q1345: 'Philadelphia',
+  Q586: 'Bonn',
+  Q1449: 'Genoa',
+  Q1563: 'Havana',
+  // Provincial cities dissolved as units in 2025 that still stand.
+  Q19491: 'Nha Trang',
+  Q36081: 'Quy Nhon',
+};
+// Groupings that are also typed as cities; none may get in.
+const MUST_NOT = {
+  Q683705: 'New York metropolitan area',
+  Q665894: 'Greater Mexico City',
+  Q13580: 'Metro Manila',
+  Q3114019: 'Katowice urban area',
+};
+const present = new Set(rows.map((row) => row[5]));
+const missing = Object.entries(MUST_HAVE).filter(([id]) => !present.has(id));
+const intruders = Object.entries(MUST_NOT).filter(([id]) => present.has(id));
+if (intruders.length) {
+  throw new Error(
+    `Groupings got in: ${intruders.map(([id, name]) => `${name} (${id})`).join(', ')}`,
+  );
+}
+if (missing.length) {
+  throw new Error(
+    `Expected cities are missing: ${missing.map(([id, name]) => `${name} (${id})`).join(', ')}`,
+  );
+}
+
+const meta = {
+  source: 'Wikidata (https://www.wikidata.org), via the Wikidata Query Service',
+  license: 'CC0 1.0',
+  built: new Date().toISOString(),
+  minPopulation,
+  count: rows.length,
+  columns: ['name', 'lat', 'lon', 'population', 'country', 'wikidata'],
+};
+
+await mkdir(outDir, { recursive: true });
+const file = path.join(outDir, 'cities.json');
+await writeFile(file, `${JSON.stringify({ meta, rows })}\n`);
+console.log(`Wrote ${rows.length} places to ${path.relative(root, file)}`);

scripts/format-scope.json

@@ -1465,5 +1465,8 @@
   "src/ui/styles/cockpit-panels.css",
   "src/ui/styles/layer-section-index.css",
   "src/ui/styles/layer-finder.css",
-  "src/ui/layerFinder.test.mjs"
+  "src/ui/layerFinder.test.mjs",
+  "src/data/wikidataCities.js",
+  "src/data/wikidataCities.test.mjs",
+  "scripts/build-wikidata-cities.mjs"
 ]

src/data/dataCredits.js

@@ -700,6 +700,15 @@ export const NATURAL_EARTH_CREDIT = {
     '<a href="https://www.naturalearthdata.com" target="_blank" rel="noopener">Natural Earth</a> (public domain)',
 };
 
+/** Registered when a name from the bundled Wikidata places packs is first
+ * shown (CC0 — no attribution required; credited as a courtesy). */
+export const WIKIDATA_PLACES_CREDIT = {
+  key: 'wikidata-places',
+  html:
+    'City, town and village names from ' +
+    '<a href="https://www.wikidata.org" target="_blank" rel="noopener">Wikidata</a> (CC0)',
+};
+
 /** Registered when the first bundled US county or place outline resolves
  * (public domain — credited as a courtesy). */
 export const US_CENSUS_CREDIT = {

src/data/local_data/wikidata_cities/README.md

@@ -0,0 +1,42 @@
+# Wikidata cities pack
+
+World cities and towns for naming a spot (the nearest city), the places
+around it and finding a place by name, read by `src/data/wikidataCities.js`. The app reads this file; it
+does not call Wikidata or any place service for these.
+
+| File | Contents |
+|------|----------|
+| `cities.json` | Human settlements with coordinates and a population at or above the cut-off (see `meta.minPopulation`), largest first |
+
+Each row is `[name, lat, lon, population, country, wikidata]`: the English
+label, coordinates to three decimals, the largest stated population, the
+ISO 3166-1 alpha-2 country code (empty when Wikidata gives none) and the
+Wikidata id.
+
+**Source:** Wikidata, read once from the Wikidata Query Service by
+`scripts/build-wikidata-cities.mjs`. The build time and row count are in the
+file's `meta` header.
+
+**License:** CC0 1.0 (https://www.wikidata.org/wiki/Wikidata:Licensing). No
+attribution is legally required.
+
+**What is left out:** metropolitan areas, urban agglomerations, conurbations
+and similar groupings of settlements, so a view is named after the city
+rather than the area around it (an item of a narrower grouping kind that is
+also a city stays: Istanbul is a metropolitan municipality and a megacity);
+places that no longer stand (former entities, ghost towns, abandoned
+villages, archaeological sites) unless they are also a living settlement;
+and places Wikidata states as dissolved or abolished (P576) where another
+place within 15 km still names the spot. A dissolved unit with nothing
+else near it stays: many of Vietnam's provincial cities ended as units in
+2025 and still stand under their names. Also left out: places below the population cut-off and
+places without an English label.
+
+**Rebuild:** `node scripts/build-wikidata-cities.mjs [--min-population 15000]`.
+The build runs one query at a time with pauses and honours the service's
+Retry-After; it takes about ten minutes. It fails if a short list of
+well-known cities is missing from the result, or a known grouping gets in.
+
+**Limits:** populations are whatever Wikidata states and differ in date and
+definition between places (city proper, municipality, urban district). Some
+districts and suburbs appear as their own rows.

src/data/local_data/wikidata_cities/cities.json

@@ -0,0 +1 @@
+{"meta":{"source":"Wikidata (https://www.wikidata.org), via the Wikidata Query Service","license":"CC0 1.0","built":"2026-11-15T02:59:26.673Z","minPopulation":15000,"count":31827,"columns":["name","lat","lon","population","country","wikidata"]},"rows":[["Chongqing",29.55,106.507,32054159,"CN","Q11725"],["Delhi",28.667,77.217,26495000,"IN","Q1353"],["Shanghai",31.233,121.469,24870895,"CN","Q8686"],["Beijing",39.904,116.408,21893095,"CN","Q956"],["Chengdu",30.66,104.063,20937757,"CN","Q30002"],["Guangzhou",23.13,113.26,18676605,"CN","Q16572"],["Shenzhen",22.547,114.054,17494398,"CN","Q15174"],["Dhaka",23.729,90.394,16800000,"BD","Q1354"],["Istanbul",41.01,28.96,15655924,"TR","Q406"],["Mumbai",19.076,72.877,15414288,"IN","Q1156"],["Karachi",24.86,67.01,14910352,"PK","Q8660"],["Kinshasa",-4.322,15.312,14565700,"CD","Q3838"],["Tokyo",35.689,139.692,14264798,"JP","Q1490"],["Ho Chi Minh City",10.776,106.702,14002598,"VN","Q1854"],["Tianjin",39.147,117.206,13866009,"CN","Q11736"],["Moscow",55.751,37.617,13274285,"RU","Q649"],["Xi'an",34.261,108.942,12952907,"CN","Q5826"],["Suzhou",31.3,120.619,12748262,"CN","Q42622"],["Zhengzhou",34.749,113.661,12600574,"CN","Q30340"],["Bengaluru",12.979,77.591,12327000,"IN","Q1355"],["Wuhan",30.595,114.297,12326518,"CN","Q11746"],["Hangzhou",30.25,120.168,11936010,"CN","Q4970"],["Baoding",38.867,115.485,11544036,"CN","Q58584"],["São Paulo",-23.55,-46.634,11451999,"BR","Q174"],["Jakarta",-6.175,106.827,11135191,"ID","Q3630"],["Lahore",31.55,74.344,11126285,"PK","Q11739"],["Linyi",35.104,118.35,11018365,"CN","Q373346"],["Shijiazhuang",38.042,114.51,10640458,"CN","Q58401"],["Dongguan",23.047,113.749,10466625,"CN","Q59218"],["Qingdao",36.117,120.4,10071722,"CN","Q170322"],["Changsha",28.199,112.971,10047914,"CN","Q174091"],["Harbin",45.75,126.633,10009854,"CN","Q42956"],["Lima",-12.06,-77.037,9943800,"PE","Q2868"],["Cairo",30.044,31.236,9801536,"EG","Q85"],["Nanyang",32.999,112.529,9713112,"CN","Q404763"],["Seoul",37.56,126.99,9668465,"KR","Q8684"],["Tokyo",35.684,139.774,9640742,"JP","Q7473516"],["Wenzhou",27.999,120.656,9572903,"CN","Q42635"],["Foshan",23.029,113.106,9498863,"CN","Q34412"],["Handan",36.612,114.489,9413990,"CN","Q58570"],["Ningbo",29.875,121.549,9404283,"CN","Q42780"],["Weifang",36.717,119.1,9386705,"CN","Q217698"],["Hefei",31.864,117.281,9369881,"CN","Q185684"],["Nanjing",32.061,118.779,9314685,"CN","Q16666"],["Mexico City",19.354,-99.136,9209944,"MX","Q1489"],["Jinan",36.667,116.983,9202432,"CN","Q170247"],["Hyderabad",17.362,78.475,9191000,"IN","Q1361"],["Xuzhou",34.261,117.186,9083790,"CN","Q57719"],["Shenyang",41.803,123.428,9070093,"CN","Q11720"],["Changchun",43.88,125.323,9066906,"CN","Q92161"],["Zhoukou",33.625,114.642,9026015,"CN","Q198061"],["Kuala Lumpur",3.148,101.695,9000280,"MY","Q1865"],["Ganzhou",25.829,114.934,8970014,"CN","Q363166"],["New York City",40.713,-74.006,8804190,"US","Q60"],["London",51.507,-0.128,8799728,"GB","Q84"],["Heze",35.233,115.433,8795939,"CN","Q332753"],["Quanzhou",24.914,118.586,8782285,"CN","Q68695"],["Nanning",22.819,108.315,8741584,"CN","Q179608"],["Hanoi",21.024,105.841,8717600,"VN","Q1858"],["Tehran",35.689,51.39,8693706,"IR","Q3616"],["Kunming",25.043,102.706,8460088,"CN","Q182852"],["Jining",35.4,116.567,8357897,"CN","Q372791"],["Fuzhou",26.077,119.292,8291268,"CN","Q68481"],["Fuyang",32.899,115.804,8200264,"CN","Q360584"],["Baghdad",33.315,44.366,8126755,"IQ","Q1530"],["Veracruz",19.19,-96.153,8062579,"MX","Q173270"],["Bogotá",4.61,-74.082,8034649,"CO","Q2841"],["Shangqiu",34.426,115.647,7816831,"CN","Q404817"],["Nantong",31.983,120.887,7726635,"CN","Q57947"],["Tangshan",39.629,118.174,7717983,"CN","Q58422"],["Ahmedabad",23.022,72.58,7645000,"IN","Q1070"],["Wuxi",31.567,120.283,7462135,"CN","Q42651"],["Dalian",38.9,121.6,7450785,"CN","Q74881"],["Hong Kong",22.278,114.159,7413070,"CN","Q8646"],["Cangzhou",38.304,116.845,7300783,"CN","Q58695"],["Xingtai",37.066,114.475,7111106,"CN","Q58576"],["Yantai",37.464,121.448,7102116,"CN","Q210493"],["Luoyang",34.659,112.424,7056699,"CN","Q187136"],["Jinhua",29.105,119.649,7050683,"CN","Q58210"],["Riyadh",24.638,46.715,7009100,"SA","Q3692"],["Zhumadian",32.977,114.025,7008427,"CN","Q198240"],["Zhanjiang",21.197,110.403,6981236,"CN","Q59144"],["Bijie",27.302,105.286,6899636,"CN","Q486319"],["Yangon",16.795,96.16,6874000,"MM","Q37995"],["Yancheng",33.35,120.157,6709629,"CN","Q57866"],["Hengyang",26.897,112.586,6645243,"CN","Q144663"],["Taizhou",28.658,121.422,6622888,"CN","Q58273"],["Zunyi",27.705,106.934,6606675,"CN","Q28994"],["Chennai",13.082,80.275,6599000,"IN","Q1352"],["Shaoyang",27.242,111.472,6563520,"CN","Q194531"],["Shangrao",28.442,117.963,6491088,"CN","Q363479"],["Huanggang",30.45,114.875,6333000,"CN","Q71355"],["Moro",-6,144,6310129,"PG","Q103927888"],["Santiago",-33.438,-70.65,6257516,"CL","Q2887"],["Nanchang",28.684,115.887,6255007,"CN","Q171943"],["Xinxiang",35.299,113.885,6251929,"CN","Q404362"],["Xinyang",32.126,114.067,6234401,"CN","Q404873"],["Rio de Janeiro",-2Line truncated

src/data/wikidataCities.js

@@ -0,0 +1,244 @@
+/**
+ * The bundled world cities pack (Wikidata, CC0), read from
+ * `local_data/wikidata_cities/cities.json` (built by
+ * scripts/build-wikidata-cities.mjs). The file loads once, on first use, and
+ * every answer comes from it: no place service is called.
+ *
+ * Rows are `[name, lat, lon, population, country, wikidata]`, largest first.
+ */
+
+const CELL_DEG = 1;
+
+// Letters that are not a base letter plus an accent, so NFD leaves them.
+const PLAIN_LETTERS = {
+  ł: 'l',
+  ø: 'o',
+  đ: 'd',
+  ð: 'd',
+  ı: 'i',
+  ß: 'ss',
+  æ: 'ae',
+  œ: 'oe',
+  þ: 'th',
+};
+
+/**
+ * A name as the lookups compare it: lower case, without accents, so
+ * "sao paulo" finds São Paulo, "zurich" Zürich, "lodz" Łódź and "tromso"
+ * Tromsø.
+ */
+export const foldName = (name) =>
+  String(name || '')
+    .normalize('NFD')
+    .replace(/\p{M}/gu, '')
+    .toLocaleLowerCase('en')
+    .replace(/[łøđðıßæœþ]/g, (letter) => PLAIN_LETTERS[letter])
+    .trim();
+const EARTH_KM = 6371;
+const rad = (deg) => (deg * Math.PI) / 180;
+
+function distanceKm(lat1, lon1, lat2, lon2) {
+  const a =
+    Math.sin(rad(lat2 - lat1) / 2) ** 2 +
+    Math.cos(rad(lat1)) *
+      Math.cos(rad(lat2)) *
+      Math.sin(rad(lon2 - lon1) / 2) ** 2;
+  return 2 * EARTH_KM * Math.asin(Math.min(1, Math.sqrt(a)));
+}
+
+const cellKey = (lat, lon) =>
+  `${Math.floor(lat / CELL_DEG)},${Math.floor(((((lon % 360) + 540) % 360) - 180) / CELL_DEG)}`;
+
+/** Group rows by one-degree cell so a lookup reads only the cells near it. */
+export function indexCities(rows) {
+  const cells = new Map();
+  for (const row of rows) {
+    const key = cellKey(row[1], row[2]);
+    const cell = cells.get(key);
+    if (cell) cell.push(row);
+    else cells.set(key, [row]);
+  }
+  return cells;
+}
+
+/** Rows by folded name, each list in the rows' own order (largest first). */
+export function indexNames(rows) {
+  const names = new Map();
+  for (const row of rows) {
+    const key = foldName(row[0]);
+    const list = names.get(key);
+    if (list) list.push(row);
+    else names.set(key, [row]);
+  }
+  return names;
+}
+
+function toCity(row, km) {
+  return {
+    name: row[0],
+    latitude: row[1],
+    longitude: row[2],
+    population: row[3],
+    country: row[4] || null,
+    wikidata: row[5],
+    distanceKm: km,
+  };
+}
+
+/**
+ * Half the longitude a circle spans, in degrees: 180 when it reaches a pole,
+ * where every meridian passes through it.
+ */
+export function lonHalfSpan(latitude, radiusKm) {
+  const angle = radiusKm / EARTH_KM;
+  const fromPole = Math.PI / 2 - Math.abs(rad(latitude));
+  if (angle >= fromPole) return 180;
+  return (Math.asin(Math.sin(angle) / Math.cos(rad(latitude))) * 180) / Math.PI;
+}
+const latHalfSpan = (radiusKm) => ((radiusKm / EARTH_KM) * 180) / Math.PI;
+
+/** Rows within `radiusKm` of a point, each with its distance. */
+export function citiesWithin(cells, latitude, longitude, radiusKm) {
+  const latSpan = Math.ceil(latHalfSpan(radiusKm) / CELL_DEG);
+  const lonSpan = Math.ceil(lonHalfSpan(latitude, radiusKm) / CELL_DEG);
+  const found = [];
+  const seen = new Set();
+  for (let dLat = -latSpan; dLat <= latSpan; dLat++) {
+    const lat = latitude + dLat * CELL_DEG;
+    if (lat < -90 - CELL_DEG || lat > 90 + CELL_DEG) continue;
+    for (
+      let dLon = -Math.min(lonSpan, 180);
+      dLon <= Math.min(lonSpan, 180);
+      dLon++
+    ) {
+      const key = cellKey(lat, longitude + dLon * CELL_DEG);
+      if (seen.has(key)) continue;
+      seen.add(key);
+      for (const row of cells.get(key) || []) {
+        const km = distanceKm(latitude, longitude, row[1], row[2]);
+        if (km <= radiusKm) found.push(toCity(row, km));
+      }
+    }
+  }
+  return found;
+}
+
+// Each country in the pack by what a qualifier may call it: its code, its
+// English name and its name in the page language, folded.
+function countryNames(rows) {
+  const names = new Map();
+  for (const code of new Set(rows.map((row) => row[4]).filter(Boolean))) {
+    for (const name of [code, countryName(code, 'en'), countryName(code)])
+      names.set(foldName(name), code);
+  }
+  return names;
+}
+
+let loading = null;
+/** The indexed pack; a failed load is retried on the next call. */
+export function loadCities(fetchImpl = (...args) => globalThis.fetch(...args)) {
+  loading ??= fetchImpl(
+    new URL('./local_data/wikidata_cities/cities.json', import.meta.url),
+  )
+    .then((response) => {
+      if (!response.ok) throw new Error(`cities pack: HTTP ${response.status}`);
+      return response.json();
+    })
+    .then((pack) => ({
+      meta: pack.meta,
+      rows: pack.rows,
+      cells: indexCities(pack.rows),
+      names: indexNames(pack.rows),
+      countries: countryNames(pack.rows),
+    }))
+    .catch((error) => {
+      loading = null;
+      throw error;
+    });
+  return loading;
+}
+
+/**
+ * The city a point is in or nearest to: the largest within `nearKm`, else the
+ * nearest within `radiusKm`, else null. Preferring the largest nearby keeps a
+ * view over a metropolis from being named after one of its suburbs.
+ */
+export async function nearestCity(
+  latitude,
+  longitude,
+  { radiusKm = 60, nearKm = 12 } = {},
+) {
+  if (!Number.isFinite(latitude) || !Number.isFinite(longitude)) return null;
+  const { cells } = await loadCities();
+  const found = citiesWithin(cells, latitude, longitude, radiusKm);
+  if (!found.length) return null;
+  const near = found.filter((city) => city.distanceKm <= nearKm);
+  if (near.length)
+    return near.reduce((a, b) => (b.population > a.population ? b : a));
+  return found.reduce((a, b) => (b.distanceKm < a.distanceKm ? b : a));
+}
+
+/** Cities around a point, most populous first. */
+export async function citiesAround({
+  latitude,
+  longitude,
+  radiusKm = 50,
+  limit = 8,
+  minPopulation = 0,
+}) {
+  if (!Number.isFinite(latitude) || !Number.isFinite(longitude)) return [];
+  const { cells } = await loadCities();
+  return citiesWithin(cells, latitude, longitude, radiusKm)
+    .filter((city) => city.population >= minPopulation)
+    .sort((a, b) => b.population - a.population || a.distanceKm - b.distanceKm)
+    .slice(0, limit);
+}
+
+/** A country's name in the page language, from its ISO code. */
+export function countryName(
+  code,
+  language = globalThis.document?.documentElement?.lang || 'en',
+) {
+  if (!code) return null;
+  try {
+    return (
+      new Intl.DisplayNames([language], { type: 'region' }).of(code) || code
+    );
+  } catch {
+    return code;
+  }
+}
+
+/**
+ * The city with this name, or null. A qualifier after a comma that names a
+ * country ("Paris, France", "Paris, US") keeps only cities there, and finds
+ * nothing when none is; one that names no country ("Paris, Texas") is
+ * ignored. A city of that name within `radiusKm` of `near` (the view) wins,
+ * the nearest first, else the largest city of that name anywhere.
+ */
+export async function findPlace(
+  name,
+  { near = null, radiusKm = 150 } = {},
+  fetchImpl = (...args) => globalThis.fetch(...args),
+) {
+  const [head, ...rest] = String(name || '').split(',');
+  const wanted = foldName(head);
+  if (!wanted) return null;
+  const qualifier = foldName(rest.join(','));
+  const { names, countries } = await loadCities(fetchImpl);
+  const country = qualifier ? countries.get(qualifier) : undefined;
+  const cities = (names.get(wanted) || []).filter(
+    (row) => !country || row[4] === country,
+  );
+  if (!cities.length) return null;
+  if (Number.isFinite(near?.latitude) && Number.isFinite(near?.longitude)) {
+    let best = null;
+    for (const row of cities) {
+      const km = distanceKm(near.latitude, near.longitude, row[1], row[2]);
+      if (km <= radiusKm && (!best || km < best.distanceKm))
+        best = toCity(row, km);
+    }
+    if (best) return best;
+  }
+  return toCity(cities[0], 0);
+}

src/data/wikidataCities.test.mjs

@@ -0,0 +1,121 @@
+// src/data/wikidataCities.test.mjs — lookups over the bundled Wikidata
+// places packs, with the packs served from memory.
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import {
+  citiesWithin,
+  foldName,
+  indexCities,
+  lonHalfSpan,
+} from './wikidataCities.js';
+
+const ROWS = [
+  ['São Paulo', -23.55, -46.633, 11451999, 'BR', 'Q174'],
+  ['Austin', 30.267, -97.743, 961855, 'US', 'Q16559'],
+  ['Round Rock', 30.508, -97.679, 119468, 'US', 'Q128269'],
+  ['Zürich', 47.374, 8.541, 421878, 'CH', 'Q72'],
+];
+
+test('names compare without case or accents', () => {
+  assert.equal(foldName('São Paulo'), 'sao paulo');
+  assert.equal(foldName('  ZÜRICH '), 'zurich');
+  assert.equal(foldName(null), '');
+});
+
+test('a lookup reads the cells around a point and measures each place', () => {
+  const cells = indexCities(ROWS);
+  const near = citiesWithin(cells, 30.27, -97.74, 40).sort(
+    (a, b) => a.distanceKm - b.distanceKm,
+  );
+  assert.deepEqual(
+    near.map((city) => city.name),
+    ['Austin', 'Round Rock'],
+  );
+  assert.ok(near[0].distanceKm < 1);
+  assert.equal(near[1].country, 'US');
+  assert.deepEqual(citiesWithin(cells, 0, 0, 50), []);
+});
+
+test('findPlace finds a city by its name without accents', async () => {
+  // A fresh module, so its pack cache starts empty.
+  const { findPlace } = await import(`./wikidataCities.js?case=${Date.now()}`);
+  const fetchImpl = async (url) => ({
+    ok: true,
+    json: async () =>
+      String(url).includes('cities.json') ? { meta: {}, rows: ROWS } : null,
+  });
+  const found = await findPlace('sao paulo, Brazil', {}, fetchImpl);
+  assert.equal(found?.wikidata, 'Q174');
+  assert.equal(await findPlace('Atlantis', {}, fetchImpl), null);
+});
+
+test('names fold letters that carry no separate accent', () => {
+  assert.equal(foldName('Łódź'), 'lodz');
+  assert.equal(foldName('Tromsø'), 'tromso');
+  assert.equal(foldName('Đà Nẵng'), 'da nang');
+  assert.equal(foldName('Diyarbakır'), 'diyarbakir');
+  assert.equal(foldName('Straße'), 'strasse');
+});
+
+const PARIS = [
+  ['Paris', 48.857, 2.352, 2145906, 'FR', 'Q90'],
+  ['Paris', 33.661, -95.556, 24476, 'US', 'Q830149'],
+];
+
+async function freshPack(rows) {
+  const module = await import(
+    `./wikidataCities.js?case=${Date.now()}-${Math.random()}`
+  );
+  const fetchImpl = async () => ({
+    ok: true,
+    json: async () => ({ meta: {}, rows }),
+  });
+  return { ...module, fetchImpl };
+}
+
+test('findPlace takes a country qualifier and prefers a namesake in view', async () => {
+  const { findPlace, fetchImpl } = await freshPack(PARIS);
+  const texas = { latitude: 33.2, longitude: -96.0 };
+  assert.equal((await findPlace('Paris', {}, fetchImpl)).wikidata, 'Q90');
+  assert.equal(
+    (await findPlace('Paris', { near: texas }, fetchImpl)).wikidata,
+    'Q830149',
+  );
+  assert.equal(
+    (await findPlace('Paris, United States', {}, fetchImpl)).wikidata,
+    'Q830149',
+  );
+  assert.equal(
+    (await findPlace('paris, us', {}, fetchImpl)).wikidata,
+    'Q830149',
+  );
+  // A qualifier the packs cannot place is ignored.
+  assert.equal(
+    (await findPlace('Paris, Texas', { near: texas }, fetchImpl)).wikidata,
+    'Q830149',
+  );
+});
+
+test('a circle that reaches a pole spans every longitude', () => {
+  assert.equal(lonHalfSpan(89, 150), 180);
+  assert.ok(lonHalfSpan(60, 150) < 3);
+  const cells = indexCities([['Pole camp', 89.9, 179, 1, '', 'Q1']]);
+  assert.equal(citiesWithin(cells, 89, 0, 150).length, 1);
+});
+
+test('a country qualifier that matches no namesake finds nothing', async () => {
+  const { findPlace, fetchImpl } = await freshPack([
+    ...PARIS,
+    ['Berlin', 52.52, 13.405, 3755251, 'DE', 'Q64'],
+  ]);
+  assert.equal(await findPlace('Paris, Germany', {}, fetchImpl), null);
+  assert.equal(
+    (await findPlace('Paris, France', {}, fetchImpl)).wikidata,
+    'Q90',
+  );
+});
+
+test('a lookup wraps at the antimeridian', () => {
+  const cells = indexCities([['Taveuni', -16.8, -179.9, 15000, 'FJ', 'Q2']]);
+  assert.equal(citiesWithin(cells, -16.8, 179.9, 40).length, 1);
+});