diff --git a/CHANGELOG.md b/CHANGELOG.md index 36100f297..a0a564a4a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,11 +30,15 @@ under NLOD 2.0. - Import your own GeoJSON, KML, KMZ and GPX files (DISPLAY ▸ Import, or drop - them onto the globe). Each file is drawn on the ground in its own color with - a row to show, hide, fly to or remove it. Files stay in the browser for the - session, and KML/KMZ network links, remote icons, remote styles and balloon - HTML are left out so an imported file cannot make the page fetch anything. - The import half of #890. + them onto the globe). GeoJSON and GPX files are drawn on the ground in the + color shown on their row; KML and KMZ keep the styles written in them. Each + file gets a row to show, hide, fly to or remove it. Files stay in the + browser for the session. An import cannot make the page fetch anything (KML + network links, remote icons and styles are left out), keeps only plain-text + names from the file (no descriptions, balloons or native labels), never + moves the app's clock (dated tracks draw for all time), and keeps GPX track + segments apart. Size limits are checked before anything is built. The import + half of #890. ## [0.2.1] — 2026-10-02 — God's Eye View through Your Agent (with security fixes and other features) diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index e6e062f64..e1d4fcbbf 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -4108,13 +4108,16 @@ are omitted rather than framing the wrong part of the globe. ### Import Your Own Files (October 2026) -- DISPLAY ▸ **Import** puts GeoJSON (`.geojson`/`.json`), KML, KMZ and GPX files on the globe. Files come in through the Import button (a hidden ``) or by dropping them onto the globe; the format is taken from the extension, then from the first bytes. Each file becomes its own Cesium data source (`GeoJsonDataSource`, `KmlDataSource`, `GpxDataSource`), clamped to the ground, in its own color from the whiteboard palette, with a row to show/hide it, fly to it, or remove it. The first file of a batch that loads is flown to. Fly-to frames the features' longitudes and latitudes at the surface and lets the camera ground guard lift the eye once the terrain has loaded (`src/ui/flyToDataSource.js`). `viewer.flyTo` is not used here: before the destination's terrain streams in, it frames ground-clamped points kilometers below the surface. Imported files last for the page session: a reload starts empty, and nothing is uploaded or stored. -- Runtime entry: `createApplicationTools` (`src/app/tools.js`) calls `initGeoFileImport({ viewer })` (`src/ui/geoFileImport.js`) after the draw tool and defers its `destroy()`, which removes every imported data source, revokes every blob URL, and gives back the drop listeners on the viewer container and the `window.__gevGeoImport` handle (`importFiles`, `list`, `setVisible`, `flyTo`, `remove`, `diagnostics`). Parsing lives in `src/data/geoFileImport.js`. -- **An imported file cannot make the page fetch anything.** Before `KmlDataSource` reads a KML/KMZ document, `sanitizeKmlDocument` removes `NetworkLink`, `NetworkLinkControl`, `ScreenOverlay`, `PhotoOverlay`, `Model` and `Tour`, every `href` that is not an image packed in the same KMZ or a `data:image/` URL (an icon then falls back to Cesium's default pin, and a `GroundOverlay` left without an image is dropped), every `styleUrl` that is not a local `#id`, and placemark `description`/`BalloonStyle`/`Snippet` HTML. KMZ images are served from blob URLs. The hint says when something was left out. `scripts/qa-geo-import.mjs` imports files that name a trap host in each of those places and fails on any request to it. +- DISPLAY ▸ **Import** puts GeoJSON (`.geojson`/`.json`), KML, KMZ and GPX files on the globe. Files come in through the Import button (a hidden ``) or by dropping them onto the globe; the format is taken from the extension, then from the first bytes. Each file becomes its own Cesium data source, clamped to the ground, with a row to show/hide it, fly to it, or remove it. GeoJSON and GPX are drawn in the row's color from the whiteboard palette (simplestyle properties in GeoJSON are dropped); KML and KMZ keep the styles written in them, and their row shows a hatched swatch instead of a color. GPX is read by `gpxToGeoJson` and loaded as GeoJSON: waypoints are points, each route a line, and each track one `MultiLineString` with a line per ``, so a pause in a recording stays a gap. Imported files last for the page session: a reload starts empty, and nothing is uploaded or stored. +- **Navigation.** The row's Fly-to is an immediate navigation through the application shell (`runImmediateNavigation('file', …)`): Cockpit refuses it, and the current follow/track owner is released first. The fly after an import is a deferred navigation: `beginDeferredLocationNavigation()` stamps it when the batch starts and `reassertDeferredLocationNavigation()` re-checks it just before flying, so a camera gesture, search, Cockpit or any newer navigation while files load wins and the import does not fly. Fly-to frames the features' longitudes and latitudes at the surface (thinned to 5,000 points by index, never spread into a call) and lets the camera ground guard lift the eye once the terrain has loaded (`src/ui/flyToDataSource.js`); `viewer.flyTo` is not used, because before terrain streams in it frames ground-clamped points kilometers below the surface. +- **Clock.** An import never changes the app's shared clock. The viewer adopts the clock of every data source added to it, so each imported data source reports no clock (`detachImportedTime`); every entity is available for all time, and a time-dynamic KML position (`gx:Track`) is drawn as the line it travels. GPX times are not read at all. +- Runtime entry: `createApplicationTools` (`src/app/tools.js`) calls `initGeoFileImport({ viewer, navigation: styleManager })` (`src/ui/geoFileImport.js`) after the draw tool and defers its `destroy()`, which removes every imported data source, revokes every blob URL, and gives back the drop listeners on the viewer container and the `window.__gevGeoImport` handle (`importFiles`, `list`, `setVisible`, `flyTo`, `remove`, `diagnostics`). Parsing lives in `src/data/geoFileImport.js`. +- **An imported file cannot make the page fetch anything.** Before `KmlDataSource` reads a KML/KMZ document, `sanitizeKmlDocument` removes `NetworkLink`, `NetworkLinkControl`, `ScreenOverlay`, `PhotoOverlay`, `Model` and `Tour`, every `href` that is not an image packed in the same KMZ or a `data:image/` URL (an icon then falls back to Cesium's default pin, drawn locally, and a `GroundOverlay` left without an image is dropped), every `styleUrl` that is not a local `#id`, and placemark `description`/`BalloonStyle`/`Snippet` HTML. GeoJSON `marker-symbol` (a maki icon asset) is dropped. KMZ images are served from blob URLs. The hint says when something was left out. `scripts/qa-geo-import.mjs` imports files that name a trap host in each of those places and fails on any request that leaves the app's own origin. +- **Imported metadata stays inert.** After loading, `scrubImportedEntities` removes every entity's description, native label, properties and KML extended data, and keeps the name only as plain text without control characters, at most 120 characters. GeoJSON is loaded with a `describe` that builds no HTML. The file name is shown with `textContent` only. - **KMZ** is read by a small zip reader (`readZipEntries`: stored and deflated entries through the platform `DecompressionStream`; no encryption, no ZIP64). The main document is `doc.kml`, else the shallowest `.kml`; relative hrefs resolve only to entries inside the archive. -- **Bounds.** A file may be at most 50 MB (`MAX_IMPORT_BYTES`), a KMZ may inflate to at most 120 MB in total (`MAX_KMZ_INFLATED_BYTES`, the zip-bomb stop), one file may create at most 20,000 entities (`MAX_IMPORT_ENTITIES`), and at most 12 files are on the globe at once (`MAX_IMPORTED_FILES`; files over that are not read). A refused file names its reason in the hint and does not stop the rest of the batch; a file with nothing to draw is refused. -- File names are rendered with `textContent` only. -- Gate: `node scripts/qa-geo-import.mjs` (dev server up). Unit: `src/data/geoFileImport.test.mjs`, `src/ui/geoFileImport.test.mjs`. +- **Bounds.** A file may be at most 50 MB (`MAX_IMPORT_BYTES`), a KMZ may inflate to at most 120 MB in total (`MAX_KMZ_INFLATED_BYTES`, the zip-bomb stop), one file may hold at most 20,000 features (`MAX_IMPORT_ENTITIES`) and 250,000 vertices (`MAX_IMPORT_POSITIONS`), counted before Cesium builds anything (`countGeoJsonGeometry`, `countKmlGeometry`, and while reading GPX), and the entity count is checked again after loading. At most 12 files are on the globe at once (`MAX_IMPORTED_FILES`): imports run one batch at a time, so two picks or drops in quick succession cannot pass the cap; files over it are not read. A refused file names its reason in the hint and does not stop the rest of the batch; a file with nothing to draw is refused. +- **Keyboard.** Hide, show and remove rebuild the list without losing focus: the same control on the same row keeps it, a removed row hands it to the next row (else the Import button). +- Gate: `node scripts/qa-geo-import.mjs` (dev server up). Unit: `src/data/geoFileImport.test.mjs`, `src/ui/geoFileImport.test.mjs`, `src/ui/flyToDataSource.test.mjs`. ### Recent Imagery (September 2026) diff --git a/scripts/qa-geo-import.mjs b/scripts/qa-geo-import.mjs index c1c8f6462..4f6627774 100644 --- a/scripts/qa-geo-import.mjs +++ b/scripts/qa-geo-import.mjs @@ -5,11 +5,16 @@ * * Proves: GeoJSON, KML, KMZ (with a packed icon and ground overlay) and GPX * each become one row and one data source; the imported shape actually paints - * on the globe and stops painting when hidden; a KML/KMZ never makes the page - * request anything named in it (network link, remote icon, remote style); a - * dropped file imports like a picked one; a file that is not geodata is refused - * with a reason and leaves nothing behind; Remove gives every data source back; - * no page errors. + * on the globe in its row's color (a GeoJSON `stroke`/`fill` does not override + * it) and stops painting when hidden; no import makes the page request + * anything named in it (network link, remote icon, remote style, palette + * icon, balloon or extended-data HTML, maki marker) or anything outside the + * page for it; no entity keeps a description, label or properties; dated KML + * and GPX leave the app's clock alone and still draw now; a GPX track keeps + * its two segments apart; keyboard focus survives hide/show; a dropped file + * imports like a picked one; a file that is not geodata is refused with a + * reason and leaves nothing behind; Remove gives every data source back; no + * page errors. * * Usage: dev server up (`npm run dev`), then `node scripts/qa-geo-import.mjs`. * QA_BASE_URL (default http://localhost:4173), QA_SHOT_DIR, QA_HEADFUL=1. @@ -40,24 +45,39 @@ const geojson = JSON.stringify({ features: [ { type: 'Feature', - properties: { name: 'QA park' }, + // A file's own simplestyle and HTML must not survive the import. + properties: { + name: 'QA park', + stroke: '#ff0000', + fill: '#ff0000', + 'fill-opacity': 1, + description: ``, + }, geometry: { type: 'Polygon', coordinates: [ring] }, }, { type: 'Feature', - properties: { name: 'QA pin' }, + properties: { name: 'QA pin', 'marker-symbol': 'rail' }, geometry: { type: 'Point', coordinates: [-122.4862, 37.7694] }, }, ], }); const kml = (iconHref) => ` - + + https://${TRAP_HOST}/link.kml QA KML pin#s ]]> + 2019-01-01T00:00:00Z2019-01-02T00:00:00Z + ]]> -122.47,37.77,0 + QA palette pin#palette + -122.48,37.771,0 + QA moving + 2019-01-01T00:00:00Z2019-01-01T01:00:00Z + -122.49 37.767 0-122.475 37.772 0 QA remote stylehttps://${TRAP_HOST}/styles.kml#x -122.50,37.766 -122.46,37.772 https://${TRAP_HOST}/overlay.png @@ -67,8 +87,10 @@ const kml = (iconHref) => ` const gpx = ` QA waypoint - QA track - + QA track<img src="https://${TRAP_HOST}/gpx.png"> + + + `; // 1×1 PNG. @@ -149,8 +171,12 @@ const page = await browser.newPage(); const errors = []; const trapped = []; page.on('pageerror', (error) => errors.push(`pageerror: ${error.message}`)); +// Anything a file names, and the icon assets Cesium would load on a file's +// behalf (KML palette icons from maps.google.com, GeoJSON maki markers). +const FILE_FETCH = [TRAP_HOST, 'maps.google.com/mapfiles', '/maki/']; page.on('request', (request) => { - if (request.url().includes(TRAP_HOST)) trapped.push(request.url()); + if (FILE_FETCH.some((needle) => request.url().includes(needle))) + trapped.push(request.url()); }); let failures = 0; function check(name, passed, detail = '') { @@ -250,6 +276,23 @@ try { )), ); const cyanBefore = await cyanPixels(); + const clock = () => + page.evaluate(() => { + const c = window.__godsEyeView.viewer.clock; + const iso = (t) => window.__CESIUM__.JulianDate.toIso8601(t, 0); + return { + start: iso(c.startTime), + stop: iso(c.stopTime), + range: c.clockRange, + step: c.clockStep, + multiplier: c.multiplier, + animate: c.shouldAnimate, + // Wall-clock drift aside, the current time must not jump to 2018/2019. + year: Number(iso(c.currentTime).slice(0, 4)), + }; + }); + const clockBefore = await clock(); + const sourcesBefore = before.dataSources; // Picked through the real . const input = await page.$('#geo-import-input'); @@ -286,6 +329,65 @@ try { /Added 4 files\..*left out/.test(after.hint), after.hint, ); + const clockAfter = await clock(); + check( + "dated KML and GPX leave the app's clock alone", + JSON.stringify(clockAfter) === JSON.stringify(clockBefore) && + (await page.evaluate( + () => + window.__godsEyeView.viewer.clockTrackedDataSource?.clock === + undefined, + )), + `${JSON.stringify(clockBefore)} → ${JSON.stringify(clockAfter)}`, + ); + const inside = await page.evaluate((from) => { + const viewer = window.__godsEyeView.viewer; + const now = viewer.clock.currentTime; + const sources = []; + for (let i = from; i < viewer.dataSources.length; i++) + sources.push(viewer.dataSources.get(i)); + const entities = sources.flatMap((ds) => ds.entities.values); + const named = (name) => entities.find((e) => e.name === name); + const gpx = sources[3].entities.values; + return { + sources: sources.length, + clocks: sources.filter((ds) => ds.clock !== undefined).length, + described: entities.filter((e) => e.description !== undefined).length, + labelled: entities.filter((e) => e.label !== undefined).length, + withProperties: entities.filter((e) => e.properties !== undefined).length, + withKml: entities.filter((e) => e.kml !== undefined).length, + hidden: entities.filter((e) => !e.isAvailable(now)).length, + datedPinNow: Boolean(named('QA KML pin')?.isAvailable(now)), + movingIsLine: Boolean( + named('QA moving')?.polyline && !named('QA moving')?.position, + ), + gpxLines: gpx.filter((e) => e.polyline).length, + gpxPoints: gpx.filter((e) => e.billboard || e.point).length, + }; + }, sourcesBefore); + check( + 'no imported entity keeps a description, label, properties or KML data', + inside.sources === 4 && + inside.described + + inside.labelled + + inside.withProperties + + inside.withKml === + 0, + JSON.stringify(inside), + ); + check( + 'no imported source has a clock, and dated features draw now', + inside.clocks === 0 && + inside.hidden === 0 && + inside.datedPinNow && + inside.movingIsLine, + JSON.stringify(inside), + ); + check( + 'the GPX track keeps its two segments apart', + inside.gpxLines === 2 && inside.gpxPoints === 1, + JSON.stringify(inside), + ); await page.evaluate((view) => { const viewer = window.__godsEyeView.viewer; viewer.camera.cancelFlight?.(); @@ -325,6 +427,26 @@ try { (await cyanPixels()) > cyanHidden * 4, ); + // Keyboard: focus stays on the same control when the list is rebuilt. + const toggleSelector = `#geo-import-list [data-import-id="${parkId}"] button[data-action="toggle"]`; + await page.focus(toggleSelector); + await page.keyboard.press('Enter'); + const focusAfterHide = await page.evaluate( + (sel) => document.activeElement === document.querySelector(sel), + toggleSelector, + ); + await page.keyboard.press('Enter'); + const focusAfterShow = await page.evaluate( + (sel) => + document.activeElement === document.querySelector(sel) && + document.activeElement.getAttribute('aria-pressed') === 'true', + toggleSelector, + ); + check( + 'keyboard focus stays on the toggle through hide and show', + focusAfterHide && focusAfterShow, + ); + // Dropped onto the globe, as a person drags from their file manager. await page.evaluate((text) => { const container = window.__godsEyeView.viewer.container; @@ -387,7 +509,7 @@ try { ); await wait(1500); check( - 'nothing named inside the KML/KMZ was ever requested', + 'nothing named inside a file, and no icon asset for one, was ever requested', trapped.length === 0, trapped.join(' '), ); diff --git a/src/app/tools.js b/src/app/tools.js index 76f8e7fe6..7acc4939e 100644 --- a/src/app/tools.js +++ b/src/app/tools.js @@ -60,7 +60,10 @@ export function createApplicationTools({ // DISPLAY ▸ Import: the person's own GeoJSON/KML/KMZ/GPX files, each its // own data source. They live for the page session, so the application // lifetime removes them and the drop listeners on the viewer container. - const geoFileImport = initGeoFileImport({ viewer }); + const geoFileImport = initGeoFileImport({ + viewer, + navigation: styleManager, + }); defer(() => geoFileImport?.destroy()); // DATA ▸ Recent Imagery: the box tool claims the pointer like Draw and the // panel lives on the right rail, so both belong to the application diff --git a/src/data/geoFileImport.js b/src/data/geoFileImport.js index ee38b7fa1..53d954957 100644 --- a/src/data/geoFileImport.js +++ b/src/data/geoFileImport.js @@ -8,6 +8,23 @@ * before Cesium sees the document (`sanitizeKmlDocument`). Images packed * inside a KMZ are served to Cesium from blob URLs the caller revokes. * + * What the file says about itself stays inert: descriptions, balloons, + * extended data and GeoJSON properties are dropped from every entity, names + * are kept only as short plain text, and no native Cesium label is drawn + * (`scrubImportedEntities`). GeoJSON and GPX draw in the row's color: + * simplestyle properties are dropped, including `marker-symbol`, so no icon + * asset is loaded on the file's behalf. KML and KMZ keep their own styles. + * + * An import never moves the app's shared clock: dated KML (TimeSpan, + * TimeStamp, gx:Track) is drawn for all time, and the data source exposes no + * clock for the viewer to adopt (`detachImportedTime`). GPX is read here + * rather than by Cesium (`gpxToGeoJson`), keeping each track segment as its + * own line and leaving times out. + * + * Size is checked before Cesium builds anything (`countGeoJsonGeometry`, + * `countKmlGeometry`): a file with too many features or vertices is refused + * up front instead of after the entities exist. + * * The bounded pieces — format detection, the KMZ (zip) reader, the size and * count limits — are plain functions so they run under `node --test`; the * Cesium loaders need a browser DOM and are exercised by @@ -23,6 +40,10 @@ export const MAX_KMZ_INFLATED_BYTES = 120 * 1024 * 1024; export const MAX_IMPORTED_FILES = 12; /** Most entities one file may create before it is refused. */ export const MAX_IMPORT_ENTITIES = 20_000; +/** Most vertices (points of every line, ring and marker) in one file. */ +export const MAX_IMPORT_POSITIONS = 250_000; +/** Longest plain-text name kept from a file's features. */ +export const MAX_IMPORT_NAME_LENGTH = 120; /** What the file picker offers. */ export const GEO_FILE_ACCEPT = '.geojson,.json,.kml,.kmz,.gpx'; @@ -127,6 +148,112 @@ export function parseGeoJsonText(text) { return value; } +/** + * Count features and vertices in a parsed GeoJSON object without recursion + * limits or array spreading, stopping as soon as a limit is passed. + * @param {object} geojson + * @param {{maxFeatures?: number, maxPositions?: number}} [limits] + * @returns {{features: number, positions: number, over: ''|'features'|'positions'}} + */ +export function countGeoJsonGeometry( + geojson, + { + maxFeatures = MAX_IMPORT_ENTITIES, + maxPositions = MAX_IMPORT_POSITIONS, + } = {}, +) { + let features = 0; + let positions = 0; + const stack = [geojson]; + while (stack.length) { + const node = stack.pop(); + if (!node || typeof node !== 'object') continue; + if (node.type === 'FeatureCollection') { + const list = Array.isArray(node.features) ? node.features : []; + for (let i = 0; i < list.length; i++) stack.push(list[i]); + continue; + } + if (node.type === 'Feature') { + features += 1; + if (features > maxFeatures) + return { features, positions, over: 'features' }; + stack.push(node.geometry); + continue; + } + if (node.type === 'GeometryCollection') { + const list = Array.isArray(node.geometries) ? node.geometries : []; + for (let i = 0; i < list.length; i++) stack.push(list[i]); + continue; + } + if (GEOJSON_TYPES.has(node.type)) { + // A bare geometry is one feature: Cesium makes one entity for it. + if (node === geojson) features += 1; + // Walk the coordinate arrays iteratively: a position is an array whose + // first item is a number. + const coords = [node.coordinates]; + while (coords.length) { + const c = coords.pop(); + if (!Array.isArray(c)) continue; + if (typeof c[0] === 'number') { + positions += 1; + if (positions > maxPositions) + return { features, positions, over: 'positions' }; + } else for (let i = 0; i < c.length; i++) coords.push(c[i]); + } + } + } + return { features, positions, over: '' }; +} + +/** The refusal for a file over a size limit, in the person's terms. */ +function overLimitError({ over, features, positions }) { + if (over === 'features') + return new GeoImportError( + `That file has more than ${MAX_IMPORT_ENTITIES.toLocaleString()} features (the limit).`, + ); + return new GeoImportError( + `That file has more than ${MAX_IMPORT_POSITIONS.toLocaleString()} points (the limit)${features ? ` across ${features.toLocaleString()} features` : ''}.`, + ); +} + +// simplestyle-spec keys Cesium honors. `marker-symbol` would load a maki icon +// asset on the file's behalf; the colors would override the row's color. +const GEOJSON_STYLE_KEYS = Object.freeze([ + 'marker-symbol', + 'marker-color', + 'marker-size', + 'stroke', + 'stroke-opacity', + 'stroke-width', + 'fill', + 'fill-opacity', +]); + +/** + * Drop GeoJSON simplestyle properties, so a GeoJSON file draws in its row's + * color like a GPX file and no icon is fetched for it. Mutates and returns + * the object. + * @param {object} geojson + * @returns {object} + */ +export function stripGeoJsonStyleProperties(geojson) { + const stack = [geojson]; + while (stack.length) { + const node = stack.pop(); + if (!node || typeof node !== 'object') continue; + if (node.type === 'FeatureCollection' && Array.isArray(node.features)) + for (let i = 0; i < node.features.length; i++) + stack.push(node.features[i]); + else if ( + node.type === 'Feature' && + node.properties && + typeof node.properties === 'object' + ) + for (const key of GEOJSON_STYLE_KEYS) delete node.properties[key]; + } + return geojson; +} + /** Inflate raw-deflate bytes with the platform's stream, stopping at `maxBytes`. */ async function inflateRaw(bytes, maxBytes) { const stream = new Blob([bytes]) @@ -326,6 +453,253 @@ export function sanitizeKmlDocument(doc, resolveLocal = () => null) { return { removed }; } +/** Child elements of `el` with this local name (any namespace). */ +const childrenNamed = (el, localName) => { + const out = []; + const kids = el?.childNodes || []; + for (let i = 0; i < kids.length; i++) + if (kids[i].nodeType === 1 && kids[i].localName === localName) + out.push(kids[i]); + return out; +}; +const childText = (el, localName) => + (childrenNamed(el, localName)[0]?.textContent || '').trim(); + +/** + * Count placemarks and vertices in a parsed KML document before Cesium reads + * it: every `` tuple and every `gx:coord`. + * @param {Document} doc + * @returns {{features: number, positions: number, over: ''|'features'|'positions'}} + */ +export function countKmlGeometry( + doc, + { + maxFeatures = MAX_IMPORT_ENTITIES, + maxPositions = MAX_IMPORT_POSITIONS, + } = {}, +) { + let features = 0; + let positions = 0; + for (const el of doc.getElementsByTagName('*')) { + const name = el.localName; + if (name === 'Placemark') { + features += 1; + if (features > maxFeatures) + return { features, positions, over: 'features' }; + } else if (name === 'coordinates') { + const text = el.textContent || ''; + // Tuples are separated by whitespace; count runs without splitting + // a large string into a large array. + let inTuple = false; + for (let i = 0; i < text.length; i++) { + const space = text.charCodeAt(i) <= 32; + if (!space && !inTuple) { + positions += 1; + if (positions > maxPositions) + return { features, positions, over: 'positions' }; + } + inTuple = !space; + } + } else if (name === 'coord') { + positions += 1; + if (positions > maxPositions) + return { features, positions, over: 'positions' }; + } + } + return { features, positions, over: '' }; +} + +/** + * Read a parsed GPX document into GeoJSON: waypoints become points, each + * route a line and each track one MultiLineString with a line per `` + * (segment breaks are gaps in the recording and stay gaps). Only names are + * kept, and no times, so a dated track draws without a clock. Throws + * `GeoImportError` past the size limits. + * @param {Document} doc + * @returns {object} A FeatureCollection. + */ +export function gpxToGeoJson( + doc, + { + maxFeatures = MAX_IMPORT_ENTITIES, + maxPositions = MAX_IMPORT_POSITIONS, + } = {}, +) { + const root = doc.documentElement; + if (root?.localName !== 'gpx') + throw new GeoImportError('That GPX file has no document.'); + const features = []; + let positions = 0; + const point = (el) => { + const lat = Number(el.getAttribute('lat')); + const lon = Number(el.getAttribute('lon')); + if ( + !Number.isFinite(lat) || + !Number.isFinite(lon) || + Math.abs(lat) > 90 || + Math.abs(lon) > 180 + ) + return null; + positions += 1; + if (positions > maxPositions) + throw overLimitError({ + over: 'positions', + features: features.length, + positions, + }); + return [lon, lat]; + }; + const add = (name, geometry) => { + if (features.length >= maxFeatures) + throw overLimitError({ + over: 'features', + features: features.length + 1, + positions, + }); + features.push({ + type: 'Feature', + properties: name ? { name } : {}, + geometry, + }); + }; + for (const wpt of childrenNamed(root, 'wpt')) { + const at = point(wpt); + if (at) add(childText(wpt, 'name'), { type: 'Point', coordinates: at }); + } + for (const rte of childrenNamed(root, 'rte')) { + const line = []; + for (const pt of childrenNamed(rte, 'rtept')) { + const at = point(pt); + if (at) line.push(at); + } + if (line.length >= 2) + add(childText(rte, 'name'), { type: 'LineString', coordinates: line }); + } + for (const trk of childrenNamed(root, 'trk')) { + const lines = []; + for (const seg of childrenNamed(trk, 'trkseg')) { + const line = []; + for (const pt of childrenNamed(seg, 'trkpt')) { + const at = point(pt); + if (at) line.push(at); + } + if (line.length >= 2) lines.push(line); + } + if (lines.length) + add(childText(trk, 'name'), { + type: 'MultiLineString', + coordinates: lines, + }); + } + return { type: 'FeatureCollection', features }; +} + +/** A feature name as short plain text (no control characters), or undefined. */ +export function importFeatureName(value) { + const text = String(value ?? '') + .replace(/[\u0000-\u001f\u007f]/g, ' ') + .replace(/\s+/g, ' ') + .trim(); + if (!text) return undefined; + return text.length > MAX_IMPORT_NAME_LENGTH + ? `${text.slice(0, MAX_IMPORT_NAME_LENGTH - 1)}…` + : text; +} + +/** + * Make every entity of an imported data source inert: no description, + * balloon, extended data or properties, no native label, and a name that is + * short plain text. Returns how many entities were touched. + * @param {{entities: {values: object[]}}} dataSource + * @returns {number} + */ +export function scrubImportedEntities(dataSource) { + const list = dataSource?.entities?.values || []; + for (let i = 0; i < list.length; i++) { + const entity = list[i]; + let name; + try { + name = importFeatureName( + typeof entity.name === 'string' ? entity.name : undefined, + ); + } catch { + name = undefined; + } + entity.name = name; + entity.description = undefined; + entity.label = undefined; + entity.properties = undefined; + if (entity.kml && typeof entity.kml === 'object') entity.kml = undefined; + } + return list.length; +} + +const TRACK_SAMPLES = 2000; + +/** + * Take time out of an imported data source so adding it cannot move the app's + * shared clock: the data source reports no clock (the viewer adopts the clock + * of every added source that has one), every entity is available for all + * time, and a time-dynamic position (a KML gx:Track) is drawn as the line it + * travels instead of a moving marker. + * @param {object} dataSource + * @param {{Cesium?: object, color?: object}} [options] + * @returns {object} The same data source. + */ +export function detachImportedTime( + dataSource, + { Cesium: C = Cesium, color } = {}, +) { + Object.defineProperty(dataSource, 'clock', { + configurable: true, + enumerable: false, + get: () => undefined, + set: () => {}, + }); + const list = dataSource?.entities?.values || []; + for (let i = 0; i < list.length; i++) { + const entity = list[i]; + const span = entity.availability; + const position = entity.position; + if (position && position.isConstant === false) { + const start = span?.start; + const stop = span?.stop; + const line = []; + if (start && stop && C.JulianDate.lessThan(start, stop)) { + const seconds = C.JulianDate.secondsDifference(stop, start); + for (let n = 0; n <= TRACK_SAMPLES; n++) { + const t = C.JulianDate.addSeconds( + start, + (seconds * n) / TRACK_SAMPLES, + new C.JulianDate(), + ); + const at = position.getValue(t); + if (at) line.push(at); + } + } + entity.path = undefined; + entity.model = undefined; + if (line.length >= 2 && !entity.polyline) { + entity.billboard = undefined; + entity.point = undefined; + entity.position = undefined; + entity.polyline = new C.PolylineGraphics({ + positions: line, + clampToGround: true, + width: 3, + material: color || C.Color.WHITE, + }); + } else { + entity.position = line.length + ? new C.ConstantPositionProperty(line[0]) + : undefined; + } + } + entity.availability = undefined; + } + return dataSource; +} + function parseXml(text, what) { const doc = new DOMParser().parseFromString( String(text).replace(/^/, ''), @@ -364,22 +738,27 @@ export async function loadGeoFile(file, { viewer, color = '#39d0ff' }) { let dataSource; let removed = 0; try { - if (format === 'geojson') { - dataSource = await Cesium.GeoJsonDataSource.load( - parseGeoJsonText(new TextDecoder().decode(bytes)), - { - clampToGround: true, - stroke, - fill: stroke.withAlpha(0.25), - strokeWidth: 3, - markerColor: stroke, - }, - ); - } else if (format === 'gpx') { - dataSource = await Cesium.GpxDataSource.load( - parseXml(new TextDecoder().decode(bytes), 'GPX'), - { clampToGround: true, trackColor: stroke, routeColor: stroke }, - ); + if (format === 'geojson' || format === 'gpx') { + let geojson; + if (format === 'gpx') { + geojson = gpxToGeoJson( + parseXml(new TextDecoder().decode(bytes), 'GPX'), + ); + } else { + geojson = parseGeoJsonText(new TextDecoder().decode(bytes)); + const size = countGeoJsonGeometry(geojson); + if (size.over) throw overLimitError(size); + stripGeoJsonStyleProperties(geojson); + } + dataSource = await Cesium.GeoJsonDataSource.load(geojson, { + clampToGround: true, + stroke, + fill: stroke.withAlpha(0.25), + strokeWidth: 3, + markerColor: stroke, + // No HTML description table is built from the file's properties. + describe: () => undefined, + }); } else { let text; let resolveLocal = () => null; @@ -402,6 +781,8 @@ export async function loadGeoFile(file, { viewer, color = '#39d0ff' }) { text = new TextDecoder().decode(bytes); } const doc = parseXml(text, format.toUpperCase()); + const size = countKmlGeometry(doc); + if (size.over) throw overLimitError(size); removed = sanitizeKmlDocument(doc, resolveLocal).removed; dataSource = await Cesium.KmlDataSource.load(doc, { camera: viewer.scene.camera, @@ -427,6 +808,11 @@ export async function loadGeoFile(file, { viewer, color = '#39d0ff' }) { revoke(); throw new GeoImportError('That file has nothing to draw.'); } - // GpxDataSource's `name` is read-only, so the display name travels beside it. + scrubImportedEntities(dataSource); + // Only KML has time-dynamic positions (GPX is read without times); its + // track line is drawn in KML's default white, like its other unstyled lines. + detachImportedTime(dataSource); + // The display name travels beside the data source (the file name, not + // anything the document says about itself). return { dataSource, name, format, entityCount, removed, revoke }; } diff --git a/src/data/geoFileImport.test.mjs b/src/data/geoFileImport.test.mjs index 522af2550..c589438fc 100644 --- a/src/data/geoFileImport.test.mjs +++ b/src/data/geoFileImport.test.mjs @@ -1,8 +1,17 @@ import test from 'node:test'; import assert from 'node:assert/strict'; import { deflateRawSync } from 'node:zlib'; +import * as Cesium from 'cesium'; import { GeoImportError, + MAX_IMPORT_POSITIONS, + countGeoJsonGeometry, + countKmlGeometry, + detachImportedTime, + gpxToGeoJson, + importFeatureName, + scrubImportedEntities, + stripGeoJsonStyleProperties, detectGeoFileFormat, importDisplayName, kmzImageType, @@ -213,3 +222,283 @@ test('KMZ hrefs resolve only to entries inside the archive', () => { assert.equal(kmzImageType('files/a.PNG'), 'image/png'); assert.equal(kmzImageType('files/a.svg'), null); }); + +/* ── a tiny element tree with the DOM surface the readers use ─────────── */ +function xmlEl(localName, attrs = {}, kids = []) { + const el = { + nodeType: 1, + localName, + childNodes: [], + getAttribute: (n) => (n in attrs ? String(attrs[n]) : null), + }; + for (const k of typeof kids === 'string' ? [] : kids) el.childNodes.push(k); + const text = typeof kids === 'string' ? kids : null; + Object.defineProperty(el, 'textContent', { + get: () => text ?? el.childNodes.map((c) => c.textContent).join(''), + }); + return el; +} +const xmlDoc = (root) => ({ + documentElement: root, + getElementsByTagName(name) { + assert.equal(name, '*'); + const out = []; + const walk = (el) => { + out.push(el); + el.childNodes.forEach(walk); + }; + walk(root); + return out; + }, +}); +const trkpt = (lat, lon, time) => + xmlEl('trkpt', { lat, lon }, time ? [xmlEl('time', {}, time)] : []); + +test('GPX keeps each track segment as its own line, names as text, no times', () => { + const doc = xmlDoc( + xmlEl('gpx', {}, [ + xmlEl('wpt', { lat: 37.77, lon: -122.49 }, [xmlEl('name', {}, 'Camp')]), + xmlEl('wpt', { lat: 99, lon: 0 }), // out of range: skipped + xmlEl('rte', {}, [ + xmlEl('name', {}, 'Route'), + xmlEl('rtept', { lat: 1, lon: 2 }), + xmlEl('rtept', { lat: 3, lon: 4 }), + ]), + xmlEl('trk', {}, [ + xmlEl('name', {}, 'Morning walk'), + xmlEl('trkseg', {}, [ + trkpt(10, 20, '2024-05-01T10:00:00Z'), + trkpt(10.1, 20.1, '2024-05-01T10:01:00Z'), + ]), + // A pause in the recording: a new segment, not a straight jump. + xmlEl('trkseg', {}, [ + trkpt(11, 21, '2024-05-01T11:00:00Z'), + trkpt(11.1, 21.1, '2024-05-01T11:01:00Z'), + trkpt(11.2, 21.2, '2024-05-01T11:02:00Z'), + ]), + xmlEl('trkseg', {}, [trkpt(12, 22)]), // one point: not a line + ]), + ]), + ); + const fc = gpxToGeoJson(doc); + assert.deepEqual( + fc.features.map((f) => [f.properties.name, f.geometry.type]), + [ + ['Camp', 'Point'], + ['Route', 'LineString'], + ['Morning walk', 'MultiLineString'], + ], + ); + assert.deepEqual(fc.features[2].geometry.coordinates, [ + [ + [20, 10], + [20.1, 10.1], + ], + [ + [21, 11], + [21.1, 11.1], + [21.2, 11.2], + ], + ]); + assert.doesNotMatch(JSON.stringify(fc), /2024|time/); + assert.throws(() => gpxToGeoJson(xmlDoc(xmlEl('kml'))), GeoImportError); +}); + +test('GPX past the point limit is refused while reading, before Cesium', () => { + const seg = xmlEl( + 'trkseg', + {}, + Array.from({ length: 30 }, (_, i) => trkpt(1 + i / 100, 2)), + ); + const doc = xmlDoc(xmlEl('gpx', {}, [xmlEl('trk', {}, [seg])])); + assert.throws( + () => gpxToGeoJson(doc, { maxPositions: 20 }), + (e) => e instanceof GeoImportError && /more than .* points/.test(e.message), + ); + assert.throws( + () => + gpxToGeoJson( + xmlDoc( + xmlEl('gpx', {}, [ + xmlEl('wpt', { lat: 1, lon: 1 }), + xmlEl('wpt', { lat: 2, lon: 2 }), + ]), + ), + { maxFeatures: 1 }, + ), + /more than .* features/, + ); +}); + +test('GeoJSON features and vertices are counted before loading, without recursion', () => { + const line = Array.from({ length: 50 }, (_, i) => [i / 10, 1]); + const fc = { + type: 'FeatureCollection', + features: [ + { type: 'Feature', geometry: { type: 'LineString', coordinates: line } }, + { + type: 'Feature', + geometry: { + type: 'GeometryCollection', + geometries: [ + { type: 'Point', coordinates: [1, 2] }, + { + type: 'Polygon', + coordinates: [ + [ + [0, 0], + [1, 0], + [1, 1], + [0, 0], + ], + ], + }, + ], + }, + }, + { type: 'Feature', geometry: null }, + ], + }; + assert.deepEqual(countGeoJsonGeometry(fc), { + features: 3, + positions: 55, + over: '', + }); + assert.equal( + countGeoJsonGeometry(fc, { maxPositions: 54 }).over, + 'positions', + ); + assert.equal(countGeoJsonGeometry(fc, { maxFeatures: 2 }).over, 'features'); + assert.deepEqual( + countGeoJsonGeometry({ type: 'Point', coordinates: [1, 2] }), + { + features: 1, + positions: 1, + over: '', + }, + ); + // A very long line is counted iteratively and stops at the limit. + const huge = { + type: 'LineString', + coordinates: Array.from({ length: MAX_IMPORT_POSITIONS + 5 }, () => [0, 0]), + }; + assert.equal(countGeoJsonGeometry(huge).over, 'positions'); +}); + +test('KML placemarks and coordinate tuples are counted before loading', () => { + const coords = (n) => + xmlEl( + 'coordinates', + {}, + Array.from({ length: n }, (_, i) => `${i},1,0`).join('\n '), + ); + const doc = xmlDoc( + xmlEl('kml', {}, [ + xmlEl('Placemark', {}, [xmlEl('LineString', {}, [coords(40)])]), + xmlEl('Placemark', {}, [ + xmlEl('Track', {}, [ + xmlEl('coord', {}, '1 2 0'), + xmlEl('coord', {}, '1 3 0'), + ]), + ]), + ]), + ); + assert.deepEqual(countKmlGeometry(doc), { + features: 2, + positions: 42, + over: '', + }); + assert.equal(countKmlGeometry(doc, { maxPositions: 41 }).over, 'positions'); + assert.equal(countKmlGeometry(doc, { maxFeatures: 1 }).over, 'features'); +}); + +test('simplestyle is dropped: no icon fetched, the row color is what draws', () => { + const fc = { + type: 'FeatureCollection', + features: [ + { + type: 'Feature', + properties: { + 'marker-symbol': 'rail', + 'marker-color': '#f00', + stroke: '#0f0', + 'stroke-width': 9, + fill: '#00f', + 'fill-opacity': 1, + name: 'x', + }, + geometry: null, + }, + { type: 'Feature', properties: null, geometry: null }, + ], + }; + stripGeoJsonStyleProperties(fc); + assert.deepEqual(fc.features[0].properties, { name: 'x' }); +}); + +test('imported entities keep only a short plain-text name: no description, label or properties', () => { + const ds = new Cesium.CustomDataSource('t'); + const e = ds.entities.add({ + name: 'Trail\u0007 head\n' + 'x'.repeat(300), + description: '', + label: { text: 'Trail head' }, + properties: { note: '' }, + position: Cesium.Cartesian3.fromDegrees(1, 2), + }); + e.kml = { extendedData: { note: { value: '