diff --git a/CHANGELOG.md b/CHANGELOG.md index a7db1c6a1..4b0e36c60 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,18 +36,7 @@ - Director timeline seeks cross the dateline along the short longitude arc. -- Map state updates stop delivering to subscribers removed during the same notification. - -- Director assets cancelled during the final stream read are discarded with reader cleanup. - -- Analyst queries preserve useful rows when an upstream timestamp is outside the supported date range. - -- Fire detections with missing or impossible positions are skipped while genuine zero coordinates remain valid. - -- Frequently reused terrain batches retain their cached heights ahead of unused cells. - -- Recent Imagery’s exported `setBox` and `searchHls` reject missing, blank or - boolean programmatic box edges instead of interpreting them as zero coordinates. +- Recent Imagery thumbnails require explicit presence evidence before reporting available pixels. - Treat explicitly current, complete empty vessel coverage as a successful update, clearing obsolete contacts while retaining stale or incomplete snapshots. @@ -259,6 +248,15 @@ largest cities) and `CCTV_VEGVESEN_VIDEO=0` keeps stills only. Attributed under NLOD 2.0. +- Import your own GeoJSON, KML, KMZ and GPX files from DISPLAY or by dropping + them onto the app. Show, hide, fly to or remove each file for the session. + GeoJSON/GPX draw on the ground; KML retains its styles and explicit heights. + Large repeated-style exports use shared local pins, while recorded track + vertices and segment gaps stay intact. Imports respect newer camera gestures, + leave the shared clock alone, and report malformed or oversized files. + File bytes stay on the device; rendered overlays may appear in shared map + screenshots. Thanks to [Chase Mills](https://github.com/kapturner) (#925). + - GNSS Integrity (Events, voice-addressable, share token `3`) draws navigation accuracy scores from a keyless regional adsb.lol sample; aircraft-level analyst queries are unavailable for these aggregate cells. Cells do not intercept diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 28c59191a..735157bd2 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -101,6 +101,10 @@ lists at 25 rows by default. Voice offers the same queries next to its app actions: the session lists them, and the browser runs them through the same catalog, loaded on first use. See [tools and the MCP server](TOOLS.md). +Catalog calls check their cancellation signal immediately before executing a +tool. Already-cancelled queries do not start; cancellation after execution starts +continues to follow the existing tool and caller behavior. + ## Cyber HUD — September 30, 2026 Display > HUD > Layout includes Cyber, also available through the HUD voice @@ -1090,6 +1094,9 @@ collapse can still be undone. ## Surface keyboard lifecycle +Surface Tab cycling excludes controls with a negative tabindex, native disabled +state or inert ancestry; ordinary visible enabled controls retain their order. + `ui/surfaces` owns the capture-phase keyboard listener, Tab cycling and return focus shared by the first-run launcher and Provider Settings. Each caller activates it while open and deactivates it on dismissal; destruction releases @@ -1173,7 +1180,9 @@ stalled downloads are cancelled. Development and preview use the same handler. Overpass query validation, geometry simplification, disk caching and upstream transport now have separate modules; military-installation search reuses that transport. Regional briefing combines separate place, news and weather sources, -while weather effects uses only the weather source. Existing process-scoped +while weather effects uses only the weather source. +GDELT compact dates that do not name a real calendar instant keep the article +with an unknown publication time. Existing process-scoped caches, rate limits, stale fallbacks and route ordering are preserved. Local voice has separate HUD-summary, debug-log and Realtime-token handlers, @@ -1202,6 +1211,11 @@ a browser key. Preview is for local build verification, not a production server. ## CCTV and radio provider modules +CCTV direction text resolves travel tokens first, then intercardinals, then +bare cardinals when enabled. Only conflicting headings within the highest +matching tier remain unknown; street quadrants and lower-tier words do not +cancel an explicit travel heading. + CCTV catalog acquisition, source normalization and frame/media delivery now live in separate modules. Each CCTV factory owns its catalog and health state; its `sourceRoot` option resolves relative source files against the application root. @@ -1222,6 +1236,10 @@ The browser layers and their rendering remain in their existing modules. ## Terrain height cache bound +The terrain proxy parser rejects finite longitudes outside [-180, 180] and +latitudes outside [-90, 90] for every client before upstream resolution. One +out-of-range point rejects the whole batch; inclusive WGS84 bounds remain valid. + The client terrain-height resolver's in-memory cache is bounded at 20 000 entries (`DEFAULT_MAX_CACHE_ENTRIES`, overridable per instance through `createTerrainHeights({ maxCacheEntries })`). Coordinate rounding makes repeated @@ -1459,7 +1477,7 @@ accessible name remains "Don't show this again". Control behavior is unchanged. ## FIRMS source status -FIRMS ingestion and adaptation require explicit finite coordinates within WGS84 ranges; genuine zero coordinates remain valid and admitted detection indices stay sequential. +The FIRMS CSV parser rejects impossible acquisition calendar dates before the proxy filters records to the trailing 24 hours, while retaining valid UTC acquisition times. The FIRMS proxy records source success after appending its rows. If aggregation throws, that source reports failure without a contradictory success entry. @@ -4202,7 +4220,8 @@ when its own signal aborts. Configured endpoints have per-instance cooldowns for 406/429, honoring Retry-After; absent delays back off from 30 seconds to five minutes. Empty `elements` is a valid answer. Existing byte, query, concurrency and cache caps -remain. Consumer capability misses never schedule automatic retries. +remain. Malformed capability-probe responses remain unknown and are retried after the existing 30-second backoff; valid configured decisions remain memoized. +Consumer capability misses never schedule automatic retries. Operator-configured Overpass area and focus-footprint queries use body-level `out geom` (with `center` for focus), preserving relation member outlines. @@ -4583,12 +4602,21 @@ are omitted rather than framing the wrong part of the globe. - Pure half: `src/annotations/drawMode.js` (session, minimum vertices, double-click de-dup at 0.5 m, vertex ceiling, degenerate-geometry refusal, ring closing, dateline-safe length/area/centroid, hint text). Regression surface: `src/annotations/drawMode.test.mjs` and `src/data/inputOwnership.test.mjs` (pure), `src/annotations/drawToolBehaviour.test.mjs` (the tool driven against DOM and viewer doubles — leases, borrowed input actions, teardown, the Enter key, preview heights), `src/annotations/drawTool.test.mjs` (structural: where files live, which ids the markup carries, what the docs may claim), and the rendered harness `scripts/qa-draw-tool.mjs`. - The annotation tree is a declared ownership group: `./annotations` → `src/annotations/index.js` in `package.json`, with its module graph listed in `scripts/package-boundaries.json`, so an annotation module reaching outside that graph is a gate failure rather than a surprise in a layer's bundle. `drawMode.js`, `drawTool.js` and their tests are in `scripts/format-scope.json`. +### Import Your Own Files (October 2026) + +- DISPLAY ▸ **Import** reads GeoJSON (`.geojson`/`.json`), KML, KMZ and GPX from the picker or a file drop anywhere in the app. Each file gets show/hide, Fly-to and Remove controls. GeoJSON and GPX use the row's color; KML/KMZ retain their styles and show a hatched swatch. The file bytes remain on the device for the page session. Rendered overlays can appear in map screenshots sent through voice or other existing screenshot tools. +- **Ground and height.** GeoJSON XYZ coordinates become ground footprints. KML ground lines/rings are tessellated; static recorded tracks preserve their original vertices, segment gaps and explicit altitude modes. KML with absolute/relative altitude keeps that meaning. Ground pins resolve in batches of 256 against loaded terrain and, in the nearby camera footprint on photoreal maps, `clampToHeightMostDetailed`, then draw in one owned native BillboardCollection per file, with static positions and no per-frame point Entity visualizer or clamping callbacks. Point metadata remains in `dataSource.importedEntities` for framing/style inspection; non-point geometry stays in the normal source collection. Pin collections mount only after source admission, follow file show/hide, and are destroyed on removal. Settled camera/map-source changes and completed terrain loads refresh visible files through the same serial queue; distant points retain their latest available height. No detailed worldwide terrain sweep is started. Nearby mesh sampling can add requests/time; configured photoreal providers require separate validation. The depth test is disabled for ground pins, matching Draw's marker visibility policy. +- **Navigation.** Row Fly-to uses `runImmediateNavigation('file', …)`; Cockpit can refuse it and existing follow/track ownership is released first. Automatic Fly-to is stamped when a batch is submitted and reasserted before flying. Pointer/wheel gestures and newer navigation supersede it, including time spent in the queue. Framing uses every finite visible point/path/polygon-hole/wall vertex plus rotated overlay bounds, with viewport-aware range and a nadir view. The target uses available terrain/cached ground height and the ground guard on arrival. Empty footprints never fall back to deferred `viewer.flyTo`. +- **Clock.** Imports expose no source clock and no entity availability. KML times/view metadata are removed before loading; recorded tracks become static lines. GPX segments remain separate and GPX times are ignored. The shared clock stays unchanged. +- **Input handling.** Coordinates and GeoJSON geometry types are checked before Cesium allocation. Only short plain names reach GeoJSON property bags; file IDs, custom CRS and styling properties are removed. KML feature/geometry IDs are removed while Style/StyleMap references remain. Local style references are trimmed, incomplete style pairs removed, hidden-folder flags normalized to the import's show control, and unsupported prototype-named elements removed. UTF-16 BOMs and common KML namespace export defects are handled. Empty drawable content is refused. A loader failure or timeout does not hold the next file indefinitely; teardown aborts pending waits. +- **References and images.** File-authored remote links/icons/styles, network links, models and screen/photo overlays are removed. Packed KMZ and embedded PNG/JPEG/GIF/WebP/BMP images remain local; decoded dimensions are checked before image decoding against a total 16-megapixel and 8,192-pixel-side budget per file. Blocked IconStyle images fall back to local pins; authored empty icons remain empty. Colored overlays without an Icon survive; overlays with a removed external image are dropped. Metadata, native labels and descriptions are removed, and generated pins share one lossless image URL per canvas. +- **Bounds.** File size ≤50 MiB; inflated KMZ ≤120 MiB; ≤20,000 entities (including multipart children and Folder containers); ≤250,000 positions; ≤2,000 normalized style definitions and distinct image links; ≤12 files per session. Exact repeated styles share definitions without changing inheritance or inline overrides. Files of 20–30 MB fit the byte budget, subject to geometry/image limits. Refusals name a reason and batches continue; with multiple failures the hint shows the first reason. +- **Lifecycle and keyboard.** `createApplicationTools` wires `initGeoFileImport` with viewer, navigation, ground services and map-source subscription. Destroy aborts pending work, unsubscribes, removes data sources, revokes blob URLs, removes document/canvas listeners and releases `window.__gevGeoImport`. List rebuilds retain keyboard focus; removing a row hands focus to the next row, else the previous row, else Import. +- **Validation.** `node scripts/qa-geo-import.mjs` covers all four formats and real painted show/hide. Run `node scripts/qa-geo-import-regressions.mjs` against both dev and preview for XYZ/track heights, failure recovery, namespace/BOM support, portrait framing and panel drops. `node scripts/qa-geo-import-large.mjs [--file /absolute/path/sample.kml] [--expect-local-pins]` checks a 27.6 MB repeated-style file, native style/image equivalence, large-file painted output and frame times. It records the WebGL renderer; software-rendered runs do not establish GPU performance. Unit tests live beside the data/controller/framing modules. + ### Recent Imagery (September 2026) -Programmatic boxes passed to the exported `setBox` and `searchHls` operations -reject missing, blank or boolean edges instead of interpreting them as zero -coordinates. Valid finite numeric edges remain accepted; the app’s drag tool, -pins and share links already supply complete boxes. +Thumbnail decoding requires explicit `Data-Present: true` evidence; false remains empty, while absent or malformed evidence reports an error. Injected `getThumbnail` sources must return an OK Response with readable `Data-Present: true` to admit image bytes, or `false` for confirmed empty coverage. Cross-origin sources must expose that header to browser JavaScript. - The layer sits in the Data Layers panel's Cameras group beside Cameras (CCTV); Mapped ALPR Cameras moved to the Infrastructure group to make room. - DATA ▸ **Recent Imagery** (`recent-imagery`, `src/layers/recentImagery/`) lists the last 30 UTC days of imagery for a selected box and drapes the days you pick: HLS S30 (Sentinel-2) and L30 (Landsat 8/9) at 30 m, and the VIIRS NOAA-21 daily overview at 250 m. Keyless and browser-direct: NASA CMR for which days have HLS granules in the box (`catalog.js`, both collections concurrently, paged with `CMR-Search-After` up to 2,000 records each, `truncated` against CMR `hits`), GIBS WMTS for tiles, Worldview Snapshots for thumbnails and exports. Worldview's `Data-Present` header is the only availability signal: GIBS answers an empty tile with HTTP 200, and the JPEG overview would drape black, so a day confirmed empty never drapes. diff --git a/index.html b/index.html index 2e630c4a8..78d6e34ec 100644 --- a/index.html +++ b/index.html @@ -13,7 +13,7 @@ with the sources: src/materialSymbolsSubset.test.mjs fails when a source names a glyph this list is missing, because a missing glyph does not draw a box — the ligature never forms and the element renders the literal word. --> - + diff --git a/scripts/format-scope.json b/scripts/format-scope.json index f8c76fea4..6b67601a5 100644 --- a/scripts/format-scope.json +++ b/scripts/format-scope.json @@ -30,6 +30,8 @@ "src/annotations/drawMode.test.mjs", "src/annotations/drawTool.js", "src/annotations/drawTool.test.mjs", + "src/data/geoFileImport.test.mjs", + "src/ui/geoFileImport.test.mjs", "src/annotations/adminOutline.test.mjs", "src/annotations/regionRing.test.mjs", "src/data/adminBoundaries.test.mjs", @@ -203,6 +205,9 @@ "src/styles/snow.js", "scripts/qa-visual-effects.mjs", "scripts/qa-draw-tool.mjs", + "scripts/qa-geo-import.mjs", + "scripts/qa-geo-import-regressions.mjs", + "scripts/qa-geo-import-large.mjs", "src/ui/mapSource.js", "src/ui/mapSourceControls.js", "src/ui/mapSourceControls.test.mjs", @@ -1034,5 +1039,7 @@ "src/tools/globePanel.test.mjs", "src/costEndpointThrottle.test.mjs", "src/sources/gbfsStations.test.mjs", - "scripts/qa-gbfs-names.mjs" + "scripts/qa-gbfs-names.mjs", + "src/ui/flyToDataSource.js", + "src/ui/flyToDataSource.test.mjs" ] diff --git a/scripts/package-boundaries.json b/scripts/package-boundaries.json index e7b494fba..2fa53ac8e 100644 --- a/scripts/package-boundaries.json +++ b/scripts/package-boundaries.json @@ -2027,6 +2027,9 @@ "src/data/inputOwnership.js", "src/annotations/drawMode.js", "src/annotations/drawTool.js", + "src/data/geoFileImport.js", + "src/ui/flyToDataSource.js", + "src/ui/geoFileImport.js", "src/data/routeSteps.js", "src/layers/directions/index.js", "src/layers/directions/source.js", @@ -2251,7 +2254,10 @@ "src/tools/functions.js", "src/voice/actionSchemas.js", "src/view/index.js", - "src/app/embed.js" + "src/app/embed.js", + "src/data/cameraFraming.js", + "src/data/geoImportPins.js", + "src/data/geoImportImages.js" ], "external": [ "hls.js", @@ -2800,7 +2806,8 @@ "src/data/local_data/osm_military_names/names.json", "src/layers/installations/namedMarkers.js", "src/data/infrastructureOverlayEntry.js", - "src/sources/infrastructureData.js" + "src/sources/infrastructureData.js", + "src/data/cameraFraming.js" ], "external": [ "hls.js", @@ -2865,7 +2872,8 @@ "src/sources/overpassFeatures.js", "src/sources/featureSource.js", "src/sources/capability.js", - "src/data/cameraArrival.js" + "src/data/cameraArrival.js", + "src/data/cameraFraming.js" ], "external": [ "cesium", diff --git a/scripts/qa-geo-import-large.mjs b/scripts/qa-geo-import-large.mjs new file mode 100644 index 000000000..679ea14a1 --- /dev/null +++ b/scripts/qa-geo-import-large.mjs @@ -0,0 +1,446 @@ +#!/usr/bin/env node +/** Large KML acceptance: style equivalence, 27.6 MB import and optional real file. + * QA_BASE_URL defaults to http://localhost:4173. Optional --file /path/file.kml. + * For a USGS fixture with blocked icons, add --expect-local-pins to check paint. + */ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import puppeteer from 'puppeteer'; +import { hookRenderErrors, readRenderErrors } from './qa-browserEvidence.mjs'; + +const base = process.env.QA_BASE_URL || 'http://localhost:4173'; +const supplied = process.argv.includes('--file') + ? path.resolve(process.argv[process.argv.indexOf('--file') + 1]) + : null; +const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'gev-large-kml-')); +const file = path.join(directory, 'metadata-rich-27mb.kml'); +const style = + ''; +const notes = 'A local observation with supporting detail. '.repeat(120); +const placemarks = Array.from({ length: 5000 }, (_, i) => { + const lon = -122.6 + (i % 100) * 0.002; + const lat = 37.65 + Math.floor(i / 100) * 0.003; + return `Point ${i}#base${style}${notes}${lon},${lat},0`; +}); +const wrap = (body) => + `${body}`; +const large = wrap(placemarks.join('')); +fs.writeFileSync(file, large); +const bytes = fs.statSync(file).size; +if (bytes < 20 * 1024 * 1024 || bytes > 30 * 1024 * 1024) + throw new Error(`Fixture must be 20–30 MiB, got ${bytes} bytes`); +const small = wrap( + Array.from({ length: 6 }, (_, i) => { + const kind = i % 3; + const geometry = + kind === 0 + ? '-122.49,37.77,0' + : kind === 1 + ? '-122.50,37.77,0 -122.49,37.77,0' + : '-122.50,37.77,0 -122.49,37.77,0 -122.49,37.78,0 -122.50,37.77,0'; + const override = + kind === 0 + ? style + : kind === 1 + ? '' + : ''; + return `Small ${i}#base${override}${geometry}`; + }).join('') + + '' + + '' + + 'normal' + + ['named-a', 'named-b', 'paired'] + .map( + (id) => + `${id}#${id}-122.49,37.77,0`, + ) + .join(''), +); +const browser = await puppeteer.launch({ + headless: true, + args: + process.platform === 'darwin' + ? ['--use-gl=angle', '--use-angle=metal'] + : ['--no-sandbox', '--use-gl=angle', '--use-angle=swiftshader'], +}); +const page = await browser.newPage(); +await hookRenderErrors(page); +const errors = []; +const fileRequests = []; +page.on('pageerror', (e) => errors.push(e.message)); +await page.setRequestInterception(true); +page.on('request', (request) => { + if (new URL(request.url()).hostname === 'earthquake.usgs.gov') { + fileRequests.push(request.url()); + void request.abort(); + } else void request.continue(); +}); +let failures = 0; +const check = (name, ok, detail) => { + console.log(`[${ok ? 'PASS' : 'FAIL'}] ${name}`, detail ?? ''); + if (!ok) failures++; +}; +try { + await page.setViewport({ width: 1440, height: 900 }); + await page.goto(`${base}/?welcome=0`, { waitUntil: 'domcontentloaded' }); + await page.waitForFunction( + () => + window.__gevGeoImport && + document.getElementById('loading-screen')?.classList.contains('hidden'), + { timeout: 60000 }, + ); + await page.evaluate(async () => { + const { viewer, dataManager } = window.__godsEyeView; + const C = window.__CESIUM__; + for (const id of dataManager.layers.keys()) + await dataManager.setEnabled(id, false); + viewer.clock.shouldAnimate = false; + viewer.imageryLayers.removeAll(); + viewer.terrainProvider = new C.EllipsoidTerrainProvider(); + viewer.camera.cancelFlight(); + viewer.camera.setView({ + destination: C.Cartesian3.fromDegrees(-122.5, 37.72, 50000), + orientation: { heading: 0, pitch: -Math.PI / 2, roll: 0 }, + }); + window.__qaStyleSnapshot = (source) => { + const now = viewer.clock.currentTime; + const images = new Map(); + const imageValue = (p) => { + const value = p?.getValue(now); + if (value?.tagName !== 'CANVAS') return value; + if (!images.has(value)) images.set(value, value.toDataURL()); + return images.get(value); + }; + const color = (p) => p?.getValue(now)?.toCssColorString(); + const positions = (p) => p?.getValue(now)?.map((v) => [v.x, v.y, v.z]); + const renderedPins = new Map(); + const collection = source.importedBillboards; + if (collection) + for (let i = 0; i < collection.length; i++) { + const pin = collection.get(i); + renderedPins.set(pin.id, pin); + } + const actual = (e, key, fallback) => { + const pin = renderedPins.get(e); + return pin ? pin[key] : (e.billboard?.[key]?.getValue(now) ?? fallback); + }; + return (source.importedEntities || source.entities.values) + .map((e) => ({ + name: e.name, + point: renderedPins.get(e)?.position || e.position?.getValue(now), + scale: e.billboard ? actual(e, 'scale', 1) : undefined, + rotation: e.billboard ? actual(e, 'rotation', 0) : undefined, + image: renderedPins.get(e)?.image || imageValue(e.billboard?.image), + color: e.billboard + ? actual(e, 'color', C.Color.WHITE).toCssColorString() + : undefined, + width: e.polyline?.width?.getValue(now), + lineColor: color(e.polyline?.material?.color), + line: positions(e.polyline?.positions), + fill: e.polygon?.fill?.getValue(now), + outline: e.polygon?.outline?.getValue(now), + polygonColor: color(e.polygon?.material?.color), + polygon: e.polygon?.hierarchy + ?.getValue(now) + ?.positions?.map((v) => [v.x, v.y, v.z]), + })) + .sort((a, b) => a.name.localeCompare(b.name)); + }; + window.__qaUnsharedKml = async (text) => { + const doc = new DOMParser().parseFromString(text, 'application/xml'); + for (const el of [...doc.getElementsByTagName('*')]) + if ( + ['description', 'ExtendedData', 'BalloonStyle'].includes(el.localName) + ) + el.remove(); + const source = await C.KmlDataSource.load(doc, { + camera: viewer.scene.camera, + canvas: viewer.scene.canvas, + clampToGround: true, + }); + for (const e of source.importedEntities || source.entities.values) + e.label = undefined; + return source; + }; + }); + const equivalent = await page.evaluate(async (text) => { + const { viewer } = window.__godsEyeView; + const control = await window.__qaUnsharedKml(text); + const outcomes = await window.__gevGeoImport.importFiles([ + new File([text], 'style-equivalence.kml'), + ]); + const candidate = viewer.dataSources.get(viewer.dataSources.length - 1); + const result = { + ok: outcomes[0]?.ok, + same: + JSON.stringify(window.__qaStyleSnapshot(control)) === + JSON.stringify(window.__qaStyleSnapshot(candidate)), + entities: (candidate.importedEntities || candidate.entities.values) + .length, + }; + await window.__gevGeoImport.remove(outcomes[0]?.id); + return result; + }, small); + check( + 'shared inline and named styles preserve point, line and polygon overrides/inheritance', + equivalent.ok && equivalent.same && equivalent.entities === 9, + JSON.stringify(equivalent), + ); + + // The control bypasses the former style-count refusal solely in this test, + // to compare native unshared rendering against normalized rendering. + await page.evaluate(async (text) => { + const source = await window.__qaUnsharedKml(text); + window.__qaUnsharedSnapshot = JSON.stringify( + window.__qaStyleSnapshot(source), + ); + window.__qaControl = source; + await window.__godsEyeView.viewer.dataSources.add(source); + }, large); + const measure = async () => + page.evaluate(async () => { + const C = window.__CESIUM__, + { viewer } = window.__godsEyeView; + const scene = viewer.scene; + viewer.camera.cancelFlight(); + viewer.camera.setView({ + destination: C.Cartesian3.fromDegrees(-122.5, 37.72, 50000), + orientation: { heading: 0, pitch: -Math.PI / 2, roll: 0 }, + }); + const started = performance.now(), + samples = []; + let last, + frames = 0; + const remove = scene.postRender.addEventListener(() => { + const t = performance.now(); + if (t - started < 2000) return; + frames++; + if (last !== undefined) samples.push(t - last); + last = t; + }); + await new Promise((resolve) => { + let previous = performance.now(); + function tick(t) { + viewer.camera.rotate( + C.Cartesian3.UNIT_Z, + Math.min(t - previous, 100) * 0.0000002, + ); + previous = t; + scene.requestRender(); + if (t - started < 7000) requestAnimationFrame(tick); + else resolve(); + } + requestAnimationFrame(tick); + }); + remove(); + samples.sort((a, b) => a - b); + const gl = scene.context._gl, + debug = gl.getExtension('WEBGL_debug_renderer_info'); + return { + fps: frames / 5, + p50Ms: samples[Math.floor(samples.length * 0.5)], + p95Ms: samples[Math.floor(samples.length * 0.95)], + renderer: debug ? gl.getParameter(debug.UNMASKED_RENDERER_WEBGL) : null, + }; + }); + console.log('UNSHARED CONTROL', JSON.stringify(await measure())); + await page.evaluate(() => + window.__godsEyeView.viewer.dataSources.remove(window.__qaControl, true), + ); + + const loadPicked = async (filename, expected) => { + const before = await page.evaluate( + () => window.__godsEyeView.viewer.dataSources.length, + ); + const priorIds = await page.evaluate(() => + window.__gevGeoImport.list().map((f) => f.id), + ); + const started = performance.now(); + await (await page.$('#geo-import-input')).uploadFile(filename); + const name = path.basename(filename); + const loaded = await page + .waitForFunction( + ({ name, priorIds }) => + window.__gevGeoImport.list().some((f) => !priorIds.includes(f.id)) || + document + .getElementById('geo-import-hint') + .textContent.startsWith(`${name}:`), + { timeout: 120000 }, + { name, priorIds }, + ) + .then( + () => true, + () => false, + ); + const state = await page.evaluate(() => ({ + files: window.__gevGeoImport.list(), + hint: document.getElementById('geo-import-hint').textContent, + sources: window.__godsEyeView.viewer.dataSources.length, + })); + const item = state.files.find((f) => !priorIds.includes(f.id)); + check( + `picked ${name} loads`, + loaded && + (expected === null + ? item?.entityCount > 0 + : item?.entityCount === expected) && + state.sources === before + 1, + JSON.stringify({ + bytes: fs.statSync(filename).size, + loadMs: Math.round(performance.now() - started), + ...state, + }), + ); + return item; + }; + const item = await loadPicked(file, 5000); + if (item) { + const same = await page.evaluate(() => { + const viewer = window.__godsEyeView.viewer; + const source = viewer.dataSources.get(viewer.dataSources.length - 1); + return ( + JSON.stringify(window.__qaStyleSnapshot(source)) === + window.__qaUnsharedSnapshot + ); + }); + check('all 5,000 normalized graphics match their unshared control', same); + console.log('NORMALIZED CANDIDATE', JSON.stringify(await measure())); + await page.evaluate((id) => window.__gevGeoImport.remove(id), item.id); + } + if (supplied) { + const clockBefore = await page.evaluate(() => { + const v = window.__godsEyeView.viewer, + C = window.__CESIUM__; + return [ + C.JulianDate.toIso8601(v.clock.startTime), + C.JulianDate.toIso8601(v.clock.stopTime), + C.JulianDate.toIso8601(v.clock.currentTime), + v.clock.multiplier, + ]; + }); + const suppliedItem = await loadPicked(supplied, null); + const clockAfter = await page.evaluate(() => { + const v = window.__godsEyeView.viewer, + C = window.__CESIUM__; + return [ + C.JulianDate.toIso8601(v.clock.startTime), + C.JulianDate.toIso8601(v.clock.stopTime), + C.JulianDate.toIso8601(v.clock.currentTime), + v.clock.multiplier, + ]; + }); + check( + 'real dated KML leaves the shared clock unchanged', + JSON.stringify(clockBefore) === JSON.stringify(clockAfter), + ); + if (suppliedItem && process.argv.includes('--expect-local-pins')) { + const markers = await page.evaluate(() => { + const { viewer } = window.__godsEyeView, + C = window.__CESIUM__; + const source = viewer.dataSources.get(viewer.dataSources.length - 1); + const points = ( + source.importedEntities || source.entities.values + ).filter((e) => e.position && e.billboard); + const images = points.filter((e) => + e.billboard.image?.getValue(viewer.clock.currentTime), + ); + if (images.length) { + const at = C.Cartographic.fromCartesian( + images[0].position.getValue(viewer.clock.currentTime), + ); + viewer.camera.cancelFlight(); + viewer.camera.setView({ + destination: C.Cartesian3.fromRadians( + at.longitude, + at.latitude, + 100000, + ), + orientation: { heading: 0, pitch: -Math.PI / 2, roll: 0 }, + }); + } + return { points: points.length, images: images.length }; + }); + check( + 'supplied point markers have drawable images', + markers.points > 0 && markers.images === markers.points, + JSON.stringify(markers), + ); + if (markers.points) { + const yellowPixels = async () => { + const shot = await page.screenshot({ + clip: { x: 620, y: 350, width: 200, height: 200 }, + type: 'png', + encoding: 'base64', + }); + return page.evaluate(async (base64) => { + const bitmap = await createImageBitmap( + await (await fetch(`data:image/png;base64,${base64}`)).blob(), + ); + const canvas = new OffscreenCanvas(bitmap.width, bitmap.height), + context = canvas.getContext('2d'); + context.drawImage(bitmap, 0, 0); + const data = context.getImageData( + 0, + 0, + bitmap.width, + bitmap.height, + ).data; + let count = 0; + for (let i = 0; i < data.length; i += 4) + if (data[i] > 150 && data[i + 1] > 150 && data[i + 2] < 100) + count++; + return count; + }, shot); + }; + const setVisible = (visible) => + page.evaluate( + ({ id, visible }) => window.__gevGeoImport.setVisible(id, visible), + { id: suppliedItem.id, visible }, + ); + await setVisible(false); + await new Promise((resolve) => setTimeout(resolve, 1000)); + const hidden = await yellowPixels(); + await setVisible(true); + let shown = hidden; + const deadline = performance.now() + 10000; + while (performance.now() < deadline && shown <= hidden + 10) { + await new Promise((resolve) => setTimeout(resolve, 500)); + shown = await yellowPixels(); + } + check( + 'real USGS markers paint when shown and disappear when hidden', + shown > hidden + 10, + JSON.stringify({ hidden, shown }), + ); + await page.screenshot({ + path: path.join(directory, 'real-kml-visible.png'), + }); + } + } + if (suppliedItem) + await page.evaluate( + (id) => window.__gevGeoImport.remove(id), + suppliedItem.id, + ); + } + check( + 'file-directed USGS icon/metadata requests never leave the page', + fileRequests.length === 0, + fileRequests.length, + ); + errors.push(...(await readRenderErrors(page))); + check( + 'no uncaught page or render errors', + errors.length === 0, + JSON.stringify(errors), + ); + check( + 'all imported sources are removed', + await page.evaluate(() => window.__gevGeoImport.list().length === 0), + ); + console.log('Fixture retained at', file); +} finally { + await browser.close(); +} +process.exitCode = failures ? 1 : 0; diff --git a/scripts/qa-geo-import-regressions.mjs b/scripts/qa-geo-import-regressions.mjs new file mode 100644 index 000000000..ba5a040a2 --- /dev/null +++ b/scripts/qa-geo-import-regressions.mjs @@ -0,0 +1,564 @@ +#!/usr/bin/env node +/** Import regressions through the real UI controller and Cesium loaders. + * Run against dev and preview: QA_BASE_URL=http://localhost:4173 node scripts/qa-geo-import-regressions.mjs + */ +import puppeteer from 'puppeteer'; +import * as Cesium from 'cesium'; +import { hookRenderErrors, readRenderErrors } from './qa-browserEvidence.mjs'; + +const browser = await puppeteer.launch({ headless: true }); +const page = await browser.newPage(); +const meshBytes = Buffer.alloc(60); +[ + -2000, 2000, -2000, 2000, 2000, -2000, -2000, 2000, 2000, 2000, 2000, 2000, +].forEach((value, i) => meshBytes.writeFloatLE(value, i * 4)); +[0, 1, 2, 2, 1, 3].forEach((value, i) => + meshBytes.writeUInt16LE(value, 48 + i * 2), +); +const fixture = { + '/__qa_import_surface/tileset.json': { + asset: { version: '1.1' }, + geometricError: 0, + root: { + transform: Cesium.Matrix4.toArray( + Cesium.Transforms.eastNorthUpToFixedFrame( + Cesium.Cartesian3.fromDegrees(-122.5, 37.7), + ), + ), + boundingVolume: { box: [0, 0, 2000, 2000, 0, 0, 0, 2000, 0, 0, 0, 100] }, + geometricError: 0, + content: { uri: 'surface.gltf' }, + }, + }, + '/__qa_import_surface/surface.gltf': { + asset: { version: '2.0' }, + scene: 0, + scenes: [{ nodes: [0] }], + nodes: [{ mesh: 0 }], + meshes: [ + { + primitives: [{ attributes: { POSITION: 0 }, indices: 1, material: 0 }], + }, + ], + materials: [{ doubleSided: true }], + buffers: [ + { + byteLength: 60, + uri: `data:application/octet-stream;base64,${meshBytes.toString('base64')}`, + }, + ], + bufferViews: [ + { buffer: 0, byteOffset: 0, byteLength: 48, target: 34962 }, + { buffer: 0, byteOffset: 48, byteLength: 12, target: 34963 }, + ], + accessors: [ + { + bufferView: 0, + componentType: 5126, + count: 4, + type: 'VEC3', + min: [-2000, 2000, -2000], + max: [2000, 2000, 2000], + }, + { bufferView: 1, componentType: 5123, count: 6, type: 'SCALAR' }, + ], + }, +}; +await page.setRequestInterception(true); +page.on('request', (request) => { + const body = fixture[new URL(request.url()).pathname]; + if (body) + void request.respond({ + status: 200, + contentType: 'application/json', + body: JSON.stringify(body), + }); + else void request.continue(); +}); +const errors = [], + requests = []; +page.on('pageerror', (e) => errors.push(e.message)); +page.on('console', (m) => { + if (/Ignoring clampToGround/.test(m.text())) errors.push(m.text()); +}); +await hookRenderErrors(page); +let failed = 0; +try { + await page.setViewport({ width: 390, height: 844 }); + await page.goto( + `${process.env.QA_BASE_URL || 'http://localhost:4173'}/?welcome=0`, + { waitUntil: 'domcontentloaded' }, + ); + await page.waitForFunction( + () => + window.__gevGeoImport && + document.getElementById('loading-screen')?.classList.contains('hidden'), + { timeout: 90000 }, + ); + await page.evaluate(async () => { + const { viewer, dataManager } = window.__godsEyeView, + C = window.__CESIUM__; + for (const id of dataManager.layers.keys()) + await dataManager.setEnabled(id, false); + viewer.imageryLayers.removeAll(); + viewer.terrainProvider = new C.EllipsoidTerrainProvider(); + viewer.clock.shouldAnimate = false; + viewer.camera.cancelFlight(); + viewer.camera.setView({ + destination: C.Cartesian3.fromDegrees(-122.5, 37.7, 100000), + orientation: { heading: 0, pitch: -Math.PI / 2, roll: 0 }, + }); + }); + page.on('request', (r) => { + if (!/^(data:|blob:)/.test(r.url())) requests.push(r.url()); + }); + const results = await page.evaluate(async () => { + const { viewer } = window.__godsEyeView, + C = window.__CESIUM__, + tool = window.__gevGeoImport; + const results = [], + now = C.JulianDate.clone(viewer.clock.currentTime); + const check = (name, pass, detail = '') => + results.push({ name, pass: Boolean(pass), detail }); + const wrap = (body, attrs = '') => + `${body}`; + const point = '-122.5,37.7,0'; + const load = async (value, name) => { + const content = + typeof value === 'object' && !ArrayBuffer.isView(value) + ? JSON.stringify(value) + : value; + const [outcome] = await tool.importFiles([new File([content], name)]); + return { + outcome, + source: outcome.ok + ? viewer.dataSources.get(viewer.dataSources.length - 1) + : null, + }; + }; + const remove = (r) => { + if (r.outcome.ok) tool.remove(r.outcome.id); + }; + let r = await load( + { + type: 'FeatureCollection', + features: [ + { + type: 'Feature', + id: 'kartaview-owned', + properties: { + name: 'XYZ', + addProperty: 1, + 'marker-symbol': 'rail', + }, + geometry: { type: 'Point', coordinates: [-122.5, 37.7, 100] }, + }, + { + type: 'Feature', + geometry: { + type: 'Polygon', + coordinates: [ + [ + [-122.5, 37.7, 10], + [-122.4, 37.7, 10], + [-122.4, 37.8, 10], + [-122.5, 37.7, 10], + ], + ], + }, + }, + ], + }, + 'xyz.geojson', + ); + const pin = (r.source?.importedEntities || r.source?.entities.values).find( + (e) => e.billboard, + ), + polygon = (r.source?.importedEntities || r.source?.entities.values).find( + (e) => e.polygon, + ); + check( + 'XYZ ground footprint and isolated entity IDs', + r.outcome.ok && + pin.id !== 'kartaview-owned' && + Math.abs( + C.Cartographic.fromCartesian(pin.position.getValue(now)).height, + ) < 0.01 && + polygon.polygon.perPositionHeight?.getValue(now) !== true, + ); + check( + 'ground pins have no persistent clamping callbacks', + pin?.billboard.heightReference?.getValue(now) === + C.HeightReference.NONE && + pin.billboard.disableDepthTestDistance.getValue(now) === Infinity, + ); + remove(r); + r = await load( + wrap( + 'ground-122.5 37.7 0-122.4 37.8 0' + + 'absoluteabsolute-122.5 37.7 10000-122.4 37.8 10000' + + 'line-122.5,37.7 -122.4,37.8', + ), + 'tracks.kml', + ); + const find = (name) => + (r.source?.importedEntities || r.source?.entities.values).find( + (e) => e.name === name, + ); + check( + 'ground tracks and untessellated authored lines clamp', + find('ground')?.polyline.clampToGround?.getValue(now) && + find('line')?.polyline.clampToGround?.getValue(now), + ); + check( + 'explicit track altitude and original vertices survive', + find('absolute')?.polyline.positions.getValue(now).length === 2 && + find('absolute') + .polyline.positions.getValue(now) + .every( + (p) => + Math.abs(C.Cartographic.fromCartesian(p).height - 10000) < 0.01, + ), + ); + remove(r); + r = await load( + wrap( + ` #s ${point}`, + ), + 'local-style.kml', + ); + check('whitespace around a local style remains local', r.outcome.ok); + remove(r); + r = await load( + wrap( + `normalhttps://gev-qa-trap.invalid/style${point}`, + ), + 'incomplete-style.kml', + ); + check('removing external style does not hang the queue', r.outcome.ok); + remove(r); + r = await load( + wrap( + `0${point}`, + ), + 'hidden-folder.kml', + ); + check( + 'import show controls own previously hidden KML content', + r.outcome.ok && + (r.source.importedEntities || r.source.entities.values).every( + (e) => e.isShowing, + ), + ); + remove(r); + r = await load( + wrap( + 'ff00ff0037.837.7-122.4-122.5', + ), + 'colored.kml', + ); + check( + 'a colored overlay without an icon is drawable', + r.outcome.ok && + (r.source.importedEntities || r.source.entities.values).some( + (e) => e.rectangle, + ), + ); + remove(r); + const xml = wrap(`${point}`); + const utf16 = new Uint8Array(2 + xml.length * 2); + utf16.set([255, 254]); + for (let i = 0; i < xml.length; i++) { + utf16[2 + i * 2] = xml.charCodeAt(i) & 255; + utf16[3 + i * 2] = xml.charCodeAt(i) >> 8; + } + r = await load(utf16, 'utf16.kml'); + check('UTF-16 BOM is decoded', r.outcome.ok); + remove(r); + r = await load( + wrap( + `${point}`, + 'xmlns:gx="http://www.google.com/kml/ext/2.2" xsi:schemaLocation="test"', + ), + 'namespaces.kml', + ); + check( + 'ordinary duplicate and missing export namespaces are repaired', + r.outcome.ok, + ); + remove(r); + for (const [name, value] of [ + ['null geometry', { type: 'Feature', geometry: null }], + ['invalid coordinates', { type: 'Point', coordinates: ['bad', 37] }], + [ + 'missing feature type', + { + type: 'FeatureCollection', + features: [{ geometry: { type: 'Point', coordinates: [0, 0] } }], + }, + ], + [ + 'nested unsupported geometry', + { type: 'Feature', geometry: { type: 'Topology', arcs: [] } }, + ], + ]) { + r = await load(value, 'refused.geojson'); + check(`${name} is refused without stopping rendering`, !r.outcome.ok); + remove(r); + } + r = await load( + wrap( + 'NaN,37,0', + ), + 'invalid.kml', + ); + check('non-finite KML coordinates are refused', !r.outcome.ok); + remove(r); + r = await load( + { + type: 'LineString', + coordinates: [ + [-122.6, 37.7], + [-122.4, 37.7], + ], + }, + 'portrait.geojson', + ); + await new Promise((resolve) => setTimeout(resolve, 2300)); + const positions = (r.source?.importedEntities || + r.source?.entities.values)[0].polyline.positions.getValue(now); + const screen = positions?.map((p) => + C.SceneTransforms.worldToWindowCoordinates(viewer.scene, p), + ); + check( + 'portrait flight includes both route endpoints', + screen?.every( + (p) => p && p.x > 0 && p.x < innerWidth && p.y > 0 && p.y < innerHeight, + ), + screen?.map((p) => p && [p.x, p.y]), + ); + remove(r); + r = await load( + { + type: 'MultiPoint', + coordinates: [ + [-74, 40.7], + [139.7, 35.7], + ], + }, + 'intercontinental.geojson', + ); + await new Promise((resolve) => setTimeout(resolve, 2300)); + const intercontinental = ( + r.source?.importedEntities || r.source?.entities.values + ) + .filter((e) => e.position) + .map((e) => e.position.getValue(now)); + const occluder = new C.EllipsoidalOccluder( + C.Ellipsoid.WGS84, + viewer.camera.positionWC, + ); + check( + 'intercontinental points remain in frame and in front of the horizon', + intercontinental?.every((p) => { + const s = C.SceneTransforms.worldToWindowCoordinates(viewer.scene, p); + return ( + occluder.isPointVisible(p) && + s && + s.x > 0 && + s.x < innerWidth && + s.y > 0 && + s.y < innerHeight + ); + }), + ); + remove(r); + r = await load( + wrap( + 'absolute14.9,52.37,3000 5.12,52.09,30005.12,52.09', + ), + 'wall.kml', + ); + await new Promise((resolve) => setTimeout(resolve, 2300)); + check( + 'a wall contributes both endpoints to the flight', + r.outcome.ok && + (r.source.importedEntities || r.source.entities.values) + .find((e) => e.wall) + ?.wall.positions.getValue(now) + .every((p) => { + const s = C.SceneTransforms.worldToWindowCoordinates( + viewer.scene, + p, + ); + return ( + s && s.x > 0 && s.x < innerWidth && s.y > 0 && s.y < innerHeight + ); + }), + ); + remove(r); + r = await load( + { + type: 'MultiPoint', + coordinates: [ + [0, 0], + [180, 0], + ], + }, + 'antipodal.geojson', + ); + await new Promise((resolve) => setTimeout(resolve, 1800)); + check( + 'symmetric global extents keep a finite camera', + r.outcome.ok && + Number.isFinite(viewer.camera.positionCartographic.height), + ); + remove(r); + viewer.camera.cancelFlight(); + viewer.camera.setView({ + destination: C.Cartesian3.fromDegrees(-122.5, 37.7, 100000), + orientation: { heading: 0, pitch: -Math.PI / 2, roll: 0 }, + }); + const mesh = await C.Cesium3DTileset.fromUrl( + `${location.origin}/__qa_import_surface/tileset.json`, + ); + viewer.scene.primitives.add(mesh); + // This fixture installs a mesh outside the app's map-source lifecycle. + // Let the new camera/tiles reach a rendered frame before testing sampling. + await new Promise((resolve) => { + let frames = 0; + const remove = viewer.scene.postRender.addEventListener(() => { + if (++frames >= 2) { + remove(); + resolve(); + } else viewer.scene.requestRender(); + }); + viewer.scene.requestRender(); + }); + const fixtureDeadline = performance.now() + 5000; + while (!mesh.tilesLoaded && performance.now() < fixtureDeadline) { + viewer.scene.requestRender(); + await new Promise((resolve) => setTimeout(resolve, 50)); + } + const globeShown = viewer.scene.globe.show; + viewer.scene.globe.show = false; + + r = await load( + { type: 'Point', coordinates: [-122.5, 37.7, 0] }, + 'mesh-point.geojson', + ); + const meshPoint = (r.source?.importedEntities || r.source?.entities.values) + .find((e) => e.billboard) + ?.position.getValue(now); + const meshHeight = + meshPoint && C.Cartographic.fromCartesian(meshPoint).height; + check( + 'native 3D Tiles ground sampling keeps pins on the rendered mesh', + r.outcome.ok && Math.abs(meshHeight - 2000) < 2, + { + meshHeight, + tilesLoaded: mesh.tilesLoaded, + globeShown: viewer.scene.globe.show, + }, + ); + tool.flyTo(r.outcome.id); + await new Promise((resolve) => setTimeout(resolve, 2300)); + const meshScreen = + meshPoint && + C.SceneTransforms.worldToWindowCoordinates(viewer.scene, meshPoint); + check( + 'a high-surface arrival frames the elevated pin', + meshScreen && + Math.abs(meshScreen.x - innerWidth / 2) < 20 && + Math.abs(meshScreen.y - innerHeight / 2) < 20, + { + screen: meshScreen && [meshScreen.x, meshScreen.y], + camera: { + height: viewer.camera.positionCartographic.height, + pitch: viewer.camera.pitch, + }, + meshHeight, + }, + ); + const groundSource = r.source; + const elevated = await load( + wrap( + 'absolute-122.5,37.7,5000', + ), + 'elevated-pin.kml', + ); + await new Promise((resolve) => setTimeout(resolve, 2600)); + check( + 'a higher imported pin never becomes the sampled ground', + elevated.outcome.ok && + Math.abs( + C.Cartographic.fromCartesian( + groundSource.importedBillboards.get(0).position, + ).height - 2000, + ) < 2, + ); + remove(elevated); + remove(r); + viewer.scene.primitives.remove(mesh); + viewer.scene.globe.show = globeShown; + const before = tool.list().length; + const transfer = new DataTransfer(); + transfer.items.add( + new File( + [JSON.stringify({ type: 'Point', coordinates: [-122.5, 37.7] })], + 'panel-drop.geojson', + ), + ); + const event = new DragEvent('drop', { + bubbles: true, + cancelable: true, + dataTransfer: transfer, + }); + document.getElementById('geo-import-button').dispatchEvent(event); + for (let i = 0; i < 100 && tool.list().length === before; i++) + await new Promise((resolve) => setTimeout(resolve, 20)); + check( + 'a panel drop imports and prevents the browser default', + event.defaultPrevented && tool.list().length === before + 1, + ); + for (const file of tool.list()) tool.remove(file.id); + check( + 'the shared clock stays exactly unchanged', + C.JulianDate.equals(now, viewer.clock.currentTime), + ); + check('no file sources remain', tool.list().length === 0); + return results; + }); + for (const r of results) { + console.log( + `[${r.pass ? 'PASS' : 'FAIL'}] ${r.name}`, + JSON.stringify(r.detail), + ); + if (!r.pass) failed++; + } + const renderErrors = await readRenderErrors(page); + if (errors.length || renderErrors.length) { + failed++; + console.log( + '[FAIL] browser/render errors', + JSON.stringify({ errors, renderErrors }), + ); + } + // Terrain belongs to the base map; imported files may sample that provider. + const unexpected = requests.filter( + (url) => + new URL(url).pathname !== '/logo.svg' && + !new URL(url).pathname.startsWith('/__qa_import_surface/') && + !/\/api\/|\.terrain(?:\?|$)|reearth|cesium|arcgis|openstreetmap|\.pbf(?:\?|$)/i.test( + url, + ), + ); + console.log( + `[${unexpected.length ? 'FAIL' : 'PASS'}] no unexpected file-directed requests`, + JSON.stringify(unexpected), + ); + if (unexpected.length) failed++; +} finally { + await browser.close(); +} +process.exitCode = failed ? 1 : 0; diff --git a/scripts/qa-geo-import.mjs b/scripts/qa-geo-import.mjs new file mode 100644 index 000000000..074eb8fbd --- /dev/null +++ b/scripts/qa-geo-import.mjs @@ -0,0 +1,594 @@ +#!/usr/bin/env node +/** + * Rendered acceptance for DISPLAY ▸ Import: real files through the real file + * input and a real drop, parsed by Cesium in the page. + * + * Proves: GeoJSON, KML, KMZ (with a packed icon and ground overlay) and GPX + * each become one row and one data source; the imported shape actually paints + * on the globe in its row's color (a GeoJSON `stroke`/`fill` does not override + * it) and stops painting when hidden; no import makes the page request + * anything named in it (network link, remote icon, remote style, palette + * icon, balloon or extended-data HTML, maki marker) or anything outside the + * page for it; no entity keeps a description, label or properties; dated KML + * and GPX leave the app's clock alone and still draw now; a GPX track keeps + * its two segments apart; keyboard focus survives hide/show; a dropped file + * imports like a picked one; a file that is not geodata is refused with a + * reason and leaves nothing behind; Remove gives every data source back; no + * page errors. + * + * Usage: dev server up (`npm run dev`), then `node scripts/qa-geo-import.mjs`. + * QA_BASE_URL (default http://localhost:4173), QA_SHOT_DIR, QA_HEADFUL=1. + */ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import puppeteer from 'puppeteer'; +import { hookRenderErrors, readRenderErrors } from './qa-browserEvidence.mjs'; + +const BASE_URL = process.env.QA_BASE_URL || 'http://localhost:4173'; +const SHOT_DIR = process.env.QA_SHOT_DIR || 'qa-shots/geo-import'; +const HEADFUL = process.env.QA_HEADFUL === '1'; +// Golden Gate Park: a big, flat, green rectangle the test polygon can cover. +const VIEW = { lon: -122.4862, lat: 37.7694, height: 4200, pitch: -88 }; +const TRAP_HOST = 'gev-qa-trap.invalid'; + +/* ── fixtures ──────────────────────────────────────────────────────────── */ + +const ring = [ + [-122.511, 37.7645], + [-122.4545, 37.7645], + [-122.4545, 37.7745], + [-122.511, 37.7745], + [-122.511, 37.7645], +]; +const geojson = JSON.stringify({ + type: 'FeatureCollection', + features: [ + { + type: 'Feature', + // A file's own simplestyle and HTML must not survive the import. + properties: { + name: 'QA park', + stroke: '#ff0000', + fill: '#ff0000', + 'fill-opacity': 1, + description: ``, + }, + geometry: { type: 'Polygon', coordinates: [ring.map((p) => [...p, 0])] }, + }, + { + type: 'Feature', + properties: { name: 'QA pin', 'marker-symbol': 'rail' }, + geometry: { type: 'Point', coordinates: [-122.4862, 37.7694, 0] }, + }, + ], +}); +const kml = (iconHref) => ` + + + + + https://${TRAP_HOST}/link.kml + QA KML pin#s + ]]> + 2019-01-01T00:00:00Z2019-01-02T00:00:00Z + ]]> + -122.47,37.77,0 + QA palette pin#palette + -122.48,37.771,0 + QA moving + 2019-01-01T00:00:00Z2019-01-01T01:00:00Z + -122.49 37.767 0-122.475 37.772 0 + QA split tracks + -122.50 37.767 0-122.49 37.768 0 + -122.47 37.771 0-122.46 37.772 0 + + QA remote stylehttps://${TRAP_HOST}/styles.kml#x + -122.50,37.766 -122.46,37.772 + https://${TRAP_HOST}/overlay.png + 37.7737.76-122.46-122.47 + +`; +const gpx = ` + + QA waypoint + QA track<img src="https://${TRAP_HOST}/gpx.png"> + + + + +`; +// 1×1 PNG. +const PNG = Buffer.from( + 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8DwHwAFBQIAX8jx0gAAAABJRU5ErkJggg==', + 'base64', +); +/** Stored (uncompressed) zip — enough for a KMZ fixture. */ +function storedZip(files) { + const parts = []; + const central = []; + let offset = 0; + for (const [name, data] of files) { + const n = Buffer.from(name); + const local = Buffer.alloc(30); + local.writeUInt32LE(0x04034b50, 0); + local.writeUInt32LE(data.length, 18); + local.writeUInt32LE(data.length, 22); + local.writeUInt16LE(n.length, 26); + const c = Buffer.alloc(46); + c.writeUInt32LE(0x02014b50, 0); + c.writeUInt32LE(data.length, 20); + c.writeUInt32LE(data.length, 24); + c.writeUInt16LE(n.length, 28); + c.writeUInt32LE(offset, 42); + parts.push(local, n, data); + central.push(c, n); + offset += 30 + n.length + data.length; + } + const dir = Buffer.concat(central); + const end = Buffer.alloc(22); + end.writeUInt32LE(0x06054b50, 0); + end.writeUInt16LE(files.length, 8); + end.writeUInt16LE(files.length, 10); + end.writeUInt32LE(dir.length, 12); + end.writeUInt32LE(offset, 16); + return Buffer.concat([...parts, dir, end]); +} +const kmzDoc = kml('files/pin.png').replace( + `https://${TRAP_HOST}/overlay.png`, + 'files/overlay.png', +); + +const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gev-qa-geo-import-')); +const fixture = (name, data) => { + const file = path.join(dir, name); + fs.writeFileSync(file, data); + return file; +}; +const FILES = [ + fixture('park.geojson', geojson), + fixture('remote-bits.kml', kml(`https://${TRAP_HOST}/icon.png`)), + fixture( + 'packed.kmz', + storedZip([ + ['doc.kml', Buffer.from(kmzDoc)], + ['files/pin.png', PNG], + ['files/overlay.png', PNG], + ]), + ), + fixture('walk.gpx', gpx), +]; + +/* ── harness ───────────────────────────────────────────────────────────── */ + +const browser = await puppeteer.launch({ + headless: !HEADFUL, + args: [ + '--no-sandbox', + '--disable-background-timer-throttling', + '--disable-renderer-backgrounding', + ...(process.platform === 'darwin' + ? ['--use-angle=metal', '--enable-gpu'] + : ['--use-gl=angle', '--use-angle=swiftshader']), + ], +}); +const page = await browser.newPage(); +await hookRenderErrors(page); +const errors = []; +const trapped = []; +page.on('pageerror', (error) => errors.push(`pageerror: ${error.message}`)); +page.on('console', (message) => { + if (/Ignoring clampToGround/.test(message.text())) + errors.push(message.text()); +}); +// Anything a file names, and the icon assets Cesium would load on a file's +// behalf (KML palette icons from maps.google.com, GeoJSON maki markers). +const FILE_FETCH = [TRAP_HOST, 'maps.google.com/mapfiles', '/maki/']; +page.on('request', (request) => { + if (FILE_FETCH.some((needle) => request.url().includes(needle))) + trapped.push(request.url()); +}); +let failures = 0; +function check(name, passed, detail = '') { + console.log( + `[${passed ? 'PASS' : 'FAIL'}] ${name}${detail ? ` — ${detail}` : ''}`, + ); + if (!passed) failures++; +} +fs.mkdirSync(SHOT_DIR, { recursive: true }); +const shot = (name) => + page.screenshot({ + path: path.join(SHOT_DIR, `${name}.jpg`), + type: 'jpeg', + quality: 82, + }); +const wait = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); +const CLIP = { x: 380, y: 240, width: 480, height: 320 }; + +/** Pixels in the clip close to the first import color (#39d0ff). */ +async function cyanPixels() { + await page.evaluate(() => window.__godsEyeView.viewer.scene.requestRender()); + await wait(600); + const base64 = await page.screenshot({ + clip: CLIP, + type: 'png', + encoding: 'base64', + }); + return page.evaluate(async (b64) => { + const bitmap = await createImageBitmap( + await (await fetch(`data:image/png;base64,${b64}`)).blob(), + ); + const canvas = new OffscreenCanvas(bitmap.width, bitmap.height); + const context = canvas.getContext('2d'); + context.drawImage(bitmap, 0, 0); + const { data } = context.getImageData(0, 0, bitmap.width, bitmap.height); + let n = 0; + for (let i = 0; i < data.length; i += 4) { + const [r, g, b] = [data[i], data[i + 1], data[i + 2]]; + if (b > 45 && g > 30 && r < 100 && b - r > 30) n += 1; + } + return n; + }, base64); +} +const state = () => + page.evaluate(() => ({ + files: window.__gevGeoImport.list(), + dataSources: window.__godsEyeView.viewer.dataSources.length, + rows: document.querySelectorAll('#geo-import-list .geo-import-row').length, + hint: document.getElementById('geo-import-hint')?.textContent || '', + })); + +try { + await page.setViewport({ width: 1280, height: 860 }); + await page.goto(`${BASE_URL}/?welcome=0`, { waitUntil: 'domcontentloaded' }); + await page.waitForFunction( + () => + window.__gevGeoImport && + window.__godsEyeView?.viewer && + document.getElementById('loading-screen')?.classList.contains('hidden'), + { timeout: 90_000 }, + ); + await page.evaluate(async (view) => { + const viewer = window.__godsEyeView.viewer; + for (const id of window.__godsEyeView.dataManager.layers.keys()) + await window.__godsEyeView.dataManager.setEnabled(id, false); + viewer.imageryLayers.removeAll(); + viewer.terrainProvider = new window.__CESIUM__.EllipsoidTerrainProvider(); + viewer.scene.globe.baseColor = window.__CESIUM__.Color.BLACK; + viewer.scene.globe.enableLighting = false; + viewer.camera.cancelFlight?.(); + viewer.camera.setView({ + destination: window.__CESIUM__.Cartesian3.fromDegrees( + view.lon, + view.lat, + view.height, + ), + orientation: { heading: 0, pitch: (view.pitch * Math.PI) / 180, roll: 0 }, + }); + viewer.scene.requestRender(); + }, VIEW); + await wait(4000); + await page.evaluate(() => { + const rail = document.getElementById('pp-toggles'); + if (rail?.classList.contains('collapsed')) + document + .querySelector('.panel-collapse-btn[data-collapse-target="pp-toggles"]') + ?.click(); + }); + await page.waitForFunction( + () => + document.getElementById('geo-import-button')?.getBoundingClientRect() + .width > 0, + { timeout: 10_000 }, + ); + const before = await state(); + check( + 'Import is on the DISPLAY rail with an empty, hidden list', + before.rows === 0 && + !(await page.evaluate(() => + document + .getElementById('geo-import-list') + .classList.contains('visible'), + )), + ); + const cyanBefore = await cyanPixels(); + await page.evaluate(() => { + window.__godsEyeView.viewer.clock.shouldAnimate = false; + }); + const clock = () => + page.evaluate(() => { + const c = window.__godsEyeView.viewer.clock; + const iso = (t) => window.__CESIUM__.JulianDate.toIso8601(t, 0); + return { + start: iso(c.startTime), + stop: iso(c.stopTime), + range: c.clockRange, + step: c.clockStep, + multiplier: c.multiplier, + animate: c.shouldAnimate, + // Wall-clock drift aside, the current time must not jump to 2018/2019. + currentTime: window.__CESIUM__.JulianDate.toIso8601(c.currentTime), + }; + }); + const clockBefore = await clock(); + const sourcesBefore = before.dataSources; + + // Picked through the real . + const input = await page.$('#geo-import-input'); + await input.uploadFile(...FILES); + const imported = await page + .waitForFunction(() => window.__gevGeoImport.list().length === 4, { + timeout: 60_000, + }) + .then( + () => true, + () => false, + ); + const after = await state(); + check( + 'four picked files become four rows and four data sources', + imported && + after.rows === 4 && + after.dataSources === before.dataSources + 4, + JSON.stringify( + after.files.map((f) => `${f.name}:${f.format}:${f.entityCount}`), + ), + ); + check( + 'formats are recognized', + after.files.map((f) => f.format).join() === 'geojson,kml,kmz,gpx', + ); + check( + 'the KML and KMZ had their outside references removed', + after.files[1].removed > 0 && after.files[2].removed > 0, + `removed ${after.files[1].removed}/${after.files[2].removed}`, + ); + check( + 'the hint says so', + /Added 4 files\..*left out/.test(after.hint), + after.hint, + ); + const clockAfter = await clock(); + check( + "dated KML and GPX leave the app's clock alone", + JSON.stringify(clockAfter) === JSON.stringify(clockBefore) && + (await page.evaluate( + () => + window.__godsEyeView.viewer.clockTrackedDataSource?.clock === + undefined, + )), + `${JSON.stringify(clockBefore)} → ${JSON.stringify(clockAfter)}`, + ); + const inside = await page.evaluate((from) => { + const viewer = window.__godsEyeView.viewer; + const now = viewer.clock.currentTime; + const sources = []; + for (let i = from; i < viewer.dataSources.length; i++) + sources.push(viewer.dataSources.get(i)); + const entities = sources.flatMap( + (ds) => ds.importedEntities || ds.entities.values, + ); + const named = (name) => entities.find((e) => e.name === name); + const gpx = sources[3].importedEntities || sources[3].entities.values; + const kmz = sources[2].importedEntities || sources[2].entities.values; + return { + sources: sources.length, + clocks: sources.filter((ds) => ds.clock !== undefined).length, + described: entities.filter((e) => e.description !== undefined).length, + labelled: entities.filter((e) => e.label !== undefined).length, + withProperties: entities.filter((e) => e.properties !== undefined).length, + withKml: entities.filter((e) => e.kml !== undefined).length, + hidden: entities.filter((e) => !e.isAvailable(now)).length, + datedPinNow: Boolean(named('QA KML pin')?.isAvailable(now)), + fallbackPins: ['QA KML pin', 'QA palette pin'].filter((name) => + named(name)?.billboard?.image?.getValue(now), + ).length, + movingIsLine: Boolean( + named('QA moving')?.polyline && !named('QA moving')?.position, + ), + movingVertices: + named('QA moving')?.polyline?.positions?.getValue(now)?.length, + splitTracks: (sources[1].importedEntities || sources[1].entities.values) + .filter((e) => e.parent?.name === 'QA split tracks' && e.polyline) + .map((e) => e.polyline.positions.getValue(now).length), + gpxLines: gpx.filter((e) => e.polyline).length, + gpxPoints: gpx.filter((e) => e.billboard || e.point).length, + kmzLocalImage: kmz.some((e) => + String(e.billboard?.image?.getValue(now) || '').startsWith('blob:'), + ), + kmzOverlay: kmz.some((e) => e.rectangle), + groundedLines: sources + .slice(1, 3) + .every((ds) => + (ds.importedEntities || ds.entities.values) + .filter((e) => e.polyline) + .every((e) => e.polyline.clampToGround?.getValue(now)), + ), + xyzPolygon: + (sources[0].importedEntities || sources[0].entities.values) + .find((e) => e.polygon) + ?.polygon.perPositionHeight?.getValue(now) !== true, + }; + }, sourcesBefore); + check( + 'no imported entity keeps a description, label, properties or KML data', + inside.sources === 4 && + inside.described + + inside.labelled + + inside.withProperties + + inside.withKml === + 0, + JSON.stringify(inside), + ); + check( + 'no imported source has a clock, and dated features draw now', + inside.clocks === 0 && + inside.hidden === 0 && + inside.datedPinNow && + inside.movingIsLine && + inside.movingVertices === 2, + JSON.stringify(inside), + ); + check( + 'blocked KML icon sources use drawable local fallback pins', + inside.fallbackPins === 2, + JSON.stringify({ fallbackPins: inside.fallbackPins }), + ); + check( + 'KML MultiTrack keeps its segments and original vertex count', + inside.splitTracks.join() === '2,2', + JSON.stringify(inside.splitTracks), + ); + check( + 'the GPX track keeps its two segments apart', + inside.gpxLines === 2 && inside.gpxPoints === 1, + JSON.stringify(inside), + ); + check( + 'KMZ retains its own packed image and overlay', + inside.kmzLocalImage && inside.kmzOverlay, + ); + check( + 'XYZ polygons and KML/KMZ lines use ground geometry', + inside.xyzPolygon && inside.groundedLines, + ); + await page.evaluate((from) => { + for (const entity of window.__godsEyeView.viewer.dataSources.get(from) + .entities.values) + if (entity.billboard) entity.show = false; + }, sourcesBefore); + await page.evaluate((view) => { + const viewer = window.__godsEyeView.viewer; + viewer.camera.cancelFlight?.(); + viewer.camera.setView({ + destination: window.__CESIUM__.Cartesian3.fromDegrees( + view.lon, + view.lat, + view.height, + ), + orientation: { heading: 0, pitch: (view.pitch * Math.PI) / 180, roll: 0 }, + }); + }, VIEW); + await wait(2500); + const cyanShown = await cyanPixels(); + await shot('imported'); + check( + 'the imported polygon paints on the globe', + cyanShown > cyanBefore * 4 && cyanShown > cyanBefore + 5000, + `${cyanBefore} → ${cyanShown}`, + ); + + const parkId = after.files[0].id; + await page.click( + `#geo-import-list [data-import-id="${parkId}"] button[data-action="toggle"]`, + ); + const cyanHidden = await cyanPixels(); + check( + 'hiding it stops it painting', + cyanHidden < cyanShown / 3, + `${cyanShown} → ${cyanHidden}`, + ); + await page.click( + `#geo-import-list [data-import-id="${parkId}"] button[data-action="toggle"]`, + ); + check( + 'showing it again paints it again', + (await cyanPixels()) > cyanHidden * 4, + ); + + // Keyboard: focus stays on the same control when the list is rebuilt. + const toggleSelector = `#geo-import-list [data-import-id="${parkId}"] button[data-action="toggle"]`; + await page.focus(toggleSelector); + await page.keyboard.press('Enter'); + const focusAfterHide = await page.evaluate( + (sel) => document.activeElement === document.querySelector(sel), + toggleSelector, + ); + await page.keyboard.press('Enter'); + const focusAfterShow = await page.evaluate( + (sel) => + document.activeElement === document.querySelector(sel) && + document.activeElement.getAttribute('aria-pressed') === 'true', + toggleSelector, + ); + check( + 'keyboard focus stays on the toggle through hide and show', + focusAfterHide && focusAfterShow, + ); + + // Dropped onto the globe, as a person drags from their file manager. + await page.evaluate((text) => { + const container = window.__godsEyeView.viewer.container; + const transfer = new DataTransfer(); + transfer.items.add( + new File([text], 'dropped.geojson', { type: 'application/geo+json' }), + ); + for (const type of ['dragenter', 'dragover', 'drop']) + container.dispatchEvent( + new DragEvent(type, { + dataTransfer: transfer, + bubbles: true, + cancelable: true, + }), + ); + }, geojson); + const dropped = await page + .waitForFunction(() => window.__gevGeoImport.list().length === 5, { + timeout: 20_000, + }) + .then( + () => true, + () => false, + ); + check('a file dropped on the globe imports', dropped); + + // Not geodata: refused with a reason, nothing added. + await page.evaluate(() => { + const transfer = new DataTransfer(); + transfer.items.add( + new File(['just some notes'], 'notes.txt', { type: 'text/plain' }), + ); + window.__godsEyeView.viewer.container.dispatchEvent( + new DragEvent('drop', { + dataTransfer: transfer, + bubbles: true, + cancelable: true, + }), + ); + }); + await wait(800); + const refused = await state(); + check( + 'a file that is not geodata is refused with a reason', + refused.files.length === 5 && + /notes\.txt: Only GeoJSON, KML, KMZ and GPX/.test(refused.hint), + refused.hint, + ); + + // Remove every file through its row button. + for (const { id } of refused.files) + await page.click( + `#geo-import-list [data-import-id="${id}"] button[data-action="remove"]`, + ); + const cleared = await state(); + check( + 'Remove gives every data source back', + cleared.rows === 0 && cleared.dataSources === before.dataSources, + `${cleared.dataSources} vs ${before.dataSources}`, + ); + await wait(1500); + check( + 'nothing named inside a file, and no icon asset for one, was ever requested', + trapped.length === 0, + trapped.join(' '), + ); + errors.push(...(await readRenderErrors(page))); + check('no page or rendering errors', errors.length === 0, errors.join(' | ')); +} catch (error) { + check('harness ran to completion', false, error?.stack || String(error)); +} finally { + await browser.close(); + fs.rmSync(dir, { recursive: true, force: true }); +} +console.log(failures ? `\n${failures} check(s) failed` : '\nall checks passed'); +process.exit(failures ? 1 : 0); diff --git a/src/annotations/annotationEngine.test.mjs b/src/annotations/annotationEngine.test.mjs index 813163d8c..3c4b9165b 100644 --- a/src/annotations/annotationEngine.test.mjs +++ b/src/annotations/annotationEngine.test.mjs @@ -313,7 +313,9 @@ test('retry: a plain HTTP 500 still exhausts the existing 8s/25s transient ladde t.mock.timers.enable({ apis: ['setTimeout', 'Date'], now: 100_000 }); globalThis.window = { setTimeout: globalThis.setTimeout, clearTimeout: globalThis.clearTimeout }; const requestTimes = []; - globalThis.fetch = async () => { + globalThis.fetch = async (url) => { + if (String(url).endsWith('/status')) + return Response.json({ configured: true }); requestTimes.push(Date.now()); return httpFailure(500); }; diff --git a/src/app/tools.js b/src/app/tools.js index 25d763e2d..7b3ea83c9 100644 --- a/src/app/tools.js +++ b/src/app/tools.js @@ -1,6 +1,7 @@ import { SceneDirector } from '../scenes/director.js'; import { initAnnotations } from '../annotations/index.js'; import { initDrawTool } from '../annotations/drawTool.js'; +import { initGeoFileImport } from '../ui/geoFileImport.js'; import { initImageryBoxTool } from '../ui/imageryBoxTool.js'; import { createRecentImageryPanel } from '../ui/recentImagery.js'; import { initGevVoiceCommands } from '../voice/gevRealtime.js'; @@ -56,6 +57,16 @@ export function createApplicationTools({ // lifetime rather than to whoever last pressed the button. const drawTool = initDrawTool({ viewer, annotations }); defer(() => drawTool?.destroy()); + // DISPLAY ▸ Import: the person's own GeoJSON/KML/KMZ/GPX files, each its + // own data source. They live for the page session, so the application + // lifetime removes them and the drop listeners on the viewer container. + const geoFileImport = initGeoFileImport({ + viewer, + navigation: styleManager, + ground: operations.surface?.groundFloor, + subscribeMapSource: (listener) => mapStackController?.subscribe?.(listener), + }); + defer(() => geoFileImport?.destroy()); // DATA ▸ Recent Imagery: the box tool claims the pointer like Draw and the // panel lives on the right rail, so both belong to the application // lifetime. The tileset lets the layer drape while the globe is hidden. diff --git a/src/data/ambiguous-camera-direction.test.mjs b/src/data/ambiguous-camera-direction.test.mjs new file mode 100644 index 000000000..b5996d7a7 --- /dev/null +++ b/src/data/ambiguous-camera-direction.test.mjs @@ -0,0 +1,30 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { directionToHeading } from './directionText.js'; +test('conflicting facing directions cannot manufacture one heading', () => { + for (const bare of [false, true]) { + for (const value of ['NB / SB', 'EASTBOUND WESTBOUND', 'NE SW']) + assert.ok(Number.isNaN(directionToHeading(value, bare)), value); + assert.equal(directionToHeading('NB northbound', bare), 0); + assert.equal(directionToHeading('IH-35 SOUTHBOUND AT 15TH', bare), 180); + } + assert.ok(Number.isNaN(directionToHeading('North / East', true))); + assert.equal(directionToHeading('West / SR-24', true), 270); +}); + +test('travel tokens retain precedence over quadrant streets and bare words', () => { + for (const [text, heading] of [ + ['I-5 NB at NE 45th St', 0], + ['SR 520 EB at NE 148th', 90], + ['US 13 SB @ SW Front St', 180], + ]) { + for (const bare of [false, true]) + assert.equal(directionToHeading(text, bare), heading); + } + assert.equal( + directionToHeading('Hwy 401 east of Keele St - Westbound', true), + 270, + ); + assert.equal(directionToHeading('NE near West St', true), 45); + assert.ok(Number.isNaN(directionToHeading('NB SB near NE St', true))); +}); diff --git a/src/data/cameraFraming.js b/src/data/cameraFraming.js new file mode 100644 index 000000000..a2d38b5ec --- /dev/null +++ b/src/data/cameraFraming.js @@ -0,0 +1,15 @@ +import * as Cesium from 'cesium'; + +/** Camera range with breathing room in both viewport dimensions. */ +export function rangeForBoundingSphere(viewer, radius) { + const frustum = viewer.camera?.frustum; + const verticalFov = Number(frustum?.fov) || Cesium.Math.toRadians(60); + const aspectRatio = Math.max(0.5, Number(frustum?.aspectRatio) || 1); + const horizontalFov = 2 * Math.atan(Math.tan(verticalFov / 2) * aspectRatio); + const limitingFov = Math.min(verticalFov, horizontalFov); + // Target 57% sphere occupancy so perspective still leaves at least 40% + // measured roof-to-base breathing room in ordinary oblique building views. + const occupiedViewportFraction = 0.57; + const desiredAngularRadius = (limitingFov * occupiedViewportFraction) / 2; + return (radius / Math.sin(desiredAngularRadius)) * 1.05; +} diff --git a/src/data/directionText.js b/src/data/directionText.js index 78f1be0df..eb78065d2 100644 --- a/src/data/directionText.js +++ b/src/data/directionText.js @@ -28,22 +28,34 @@ export function directionToHeading(value, allowBare = false) { .trim() .toUpperCase(); if (!text) return NaN; - // Explicit travel/intercardinal forms — safe on free-form text (a street - // name almost never contains "NORTHBOUND" or a lone "NB" token). - if (/\bNORTHBOUND\b|\bNB\b/.test(text)) return 0; - if (/\bSOUTHBOUND\b|\bSB\b/.test(text)) return 180; - if (/\bEASTBOUND\b|\bEB\b/.test(text)) return 90; - if (/\bWESTBOUND\b|\bWB\b/.test(text)) return 270; - if (/\bNORTHEAST\b|\bNE\b/.test(text)) return 45; - if (/\bNORTHWEST\b|\bNW\b/.test(text)) return 315; - if (/\bSOUTHEAST\b|\bSE\b/.test(text)) return 135; - if (/\bSOUTHWEST\b|\bSW\b/.test(text)) return 225; - // Bare cardinals — dedicated direction fields only. - if (allowBare) { - if (/\bNORTH\b/.test(text)) return 0; - if (/\bSOUTH\b/.test(text)) return 180; - if (/\bEAST\b/.test(text)) return 90; - if (/\bWEST\b/.test(text)) return 270; + const tiers = [ + [ + [/\bNORTHBOUND\b|\bNB\b/, 0], + [/\bSOUTHBOUND\b|\bSB\b/, 180], + [/\bEASTBOUND\b|\bEB\b/, 90], + [/\bWESTBOUND\b|\bWB\b/, 270], + ], + [ + [/\bNORTHEAST\b|\bNE\b/, 45], + [/\bNORTHWEST\b|\bNW\b/, 315], + [/\bSOUTHEAST\b|\bSE\b/, 135], + [/\bSOUTHWEST\b|\bSW\b/, 225], + ], + ]; + if (allowBare) + tiers.push([ + [/\bNORTH\b/, 0], + [/\bSOUTH\b/, 180], + [/\bEAST\b/, 90], + [/\bWEST\b/, 270], + ]); + for (const forms of tiers) { + const headings = new Set(); + for (const [pattern, heading] of forms) { + if (pattern.test(text)) headings.add(heading); + } + if (headings.size) + return headings.size === 1 ? headings.values().next().value : NaN; } return NaN; } diff --git a/src/data/firms-calendar-validity.test.mjs b/src/data/firms-calendar-validity.test.mjs new file mode 100644 index 000000000..1ccaf7435 --- /dev/null +++ b/src/data/firms-calendar-validity.test.mjs @@ -0,0 +1,23 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { acquisitionMsUtc } from './firmsCsv.js'; +test('FIRMS acquisition dates cannot roll into a different calendar day', () => { + for (const [date, time] of [ + ['2026-02-30', '1200'], + ['2026-04-31', '1200'], + ['2026-13-01', '1200'], + ['2026-07-16junk', '1200'], + ['2026-07-16', '2400'], + ['2026-07-16', '1260'], + ['2026-07-16', null], + ['2026-07-16', ''], + ]) { + assert.ok(Number.isNaN(acquisitionMsUtc(date, time)), `${date}:${time}`); + } + for (const [date, time, expected] of [ + ['2024-02-29', '45', Date.UTC(2024, 1, 29, 0, 45)], + ['2026-07-16', '0', Date.UTC(2026, 6, 16)], + ]) { + assert.equal(acquisitionMsUtc(date, time), expected); + } +}); diff --git a/src/data/firmsCsv.js b/src/data/firmsCsv.js index aafeac50b..368ef4f73 100644 --- a/src/data/firmsCsv.js +++ b/src/data/firmsCsv.js @@ -163,7 +163,13 @@ export function acquisitionMsUtc(acqDate, acqTime) { minutes > 59 ) return NaN; - return Date.UTC(year, month - 1, day, hours, minutes); + const ms = Date.UTC(year, month - 1, day, hours, minutes); + const parsed = new Date(ms); + return parsed.getUTCFullYear() === year && + parsed.getUTCMonth() === month - 1 && + parsed.getUTCDate() === day + ? ms + : NaN; } /** diff --git a/src/data/geoFileImport.js b/src/data/geoFileImport.js new file mode 100644 index 000000000..0ceaf9ef5 --- /dev/null +++ b/src/data/geoFileImport.js @@ -0,0 +1,1390 @@ +/** Session-local geodata loading with bounded geometry and local-only file references. */ +import * as Cesium from 'cesium'; +import { createImportImageBudget } from './geoImportImages.js'; +import { createImportedPins } from './geoImportPins.js'; + +/** Largest file accepted, in bytes. */ +export const MAX_IMPORT_BYTES = 50 * 1024 * 1024; +/** Largest total a KMZ may inflate to — a zip bomb stops here. */ +export const MAX_KMZ_INFLATED_BYTES = 120 * 1024 * 1024; +/** Most files on the globe at once. */ +export const MAX_IMPORTED_FILES = 12; +/** Most entities one file may create before it is refused. */ +export const MAX_IMPORT_ENTITIES = 20_000; +/** Most vertices (points of every line, ring and marker) in one file. */ +export const MAX_IMPORT_POSITIONS = 250_000; +/** Most normalized style definitions and local image links in one KML/KMZ. */ +export const MAX_KML_LINKS = 2_000; +/** Longest plain-text name kept from a file's features. */ +export const MAX_IMPORT_NAME_LENGTH = 120; +/** What the file picker offers. */ +export const GEO_FILE_ACCEPT = '.geojson,.json,.kml,.kmz,.gpx'; + +const GEOJSON_TYPES = new Set([ + 'FeatureCollection', + 'Feature', + 'Point', + 'MultiPoint', + 'LineString', + 'MultiLineString', + 'Polygon', + 'MultiPolygon', + 'GeometryCollection', +]); +const KMZ_IMAGE_TYPES = Object.freeze({ + png: 'image/png', + jpg: 'image/jpeg', + jpeg: 'image/jpeg', + gif: 'image/gif', + webp: 'image/webp', + bmp: 'image/bmp', +}); +// External content and metadata the session-local import does not use. +const KML_REMOVED_ELEMENTS = new Set([ + 'NetworkLink', + 'NetworkLinkControl', + 'ScreenOverlay', + 'PhotoOverlay', + 'Model', + 'Tour', + 'description', + 'BalloonStyle', + 'Snippet', + 'ExtendedData', + 'TimeSpan', + 'TimeStamp', + 'LookAt', + 'Camera', + 'visibility', + 'viewRefreshMode', + 'viewFormat', + 'httpQuery', +]); + +/** An error whose message is meant for the person who chose the file. */ +export class GeoImportError extends Error { + constructor(message) { + super(message); + this.name = 'GeoImportError'; + } +} + +/** + * The format of a file, from its name first and its first bytes second. + * @param {string} name File name. + * @param {Uint8Array} [head] The first bytes of the file. + * @returns {'geojson'|'kml'|'kmz'|'gpx'|null} + */ +export function detectGeoFileFormat(name, head) { + const ext = /\.([a-z0-9]+)$/i.exec(String(name || ''))?.[1]?.toLowerCase(); + if (ext === 'geojson') return 'geojson'; + if (ext === 'kml' || ext === 'kmz' || ext === 'gpx') return ext; + if (!head?.length) return ext === 'json' ? 'geojson' : null; + if ( + head[0] === 0x50 && + head[1] === 0x4b && + head[2] === 0x03 && + head[3] === 0x04 + ) + return 'kmz'; + const text = new TextDecoder() + .decode(head.subarray(0, 1024)) + .replace(/^/, '') + .trimStart(); + if (text.startsWith('{')) return 'geojson'; + if (text.startsWith('<')) { + if (/]/i.test(text)) return 'kml'; + if (/]/i.test(text)) return 'gpx'; + } + return ext === 'json' ? 'geojson' : null; +} + +/** + * A short, printable name for the list: the file name without control + * characters, cut to 60 characters. + * @param {string} name + * @returns {string} + */ +export function importDisplayName(name) { + const clean = String(name || '') + .replace(/[\u0000-\u001f\u007f]/g, '') + .trim(); + if (!clean) return 'Imported file'; + return clean.length > 60 ? `${clean.slice(0, 57)}…` : clean; +} + +/** + * Parse and check GeoJSON text. Throws `GeoImportError` for anything that is + * not a GeoJSON object. + * @param {string} text + * @returns {object} + */ +export function parseGeoJsonText(text) { + let value; + try { + value = JSON.parse(String(text).replace(/^/, '')); + } catch { + throw new GeoImportError('That file is not valid JSON.'); + } + if (!value || typeof value !== 'object' || !GEOJSON_TYPES.has(value.type)) + throw new GeoImportError('That JSON file is not GeoJSON.'); + if (value.type === 'FeatureCollection' && !Array.isArray(value.features)) + throw new GeoImportError('That FeatureCollection has no features list.'); + return value; +} + +/** + * Count features and vertices in a parsed GeoJSON object without recursion + * limits or array spreading, stopping as soon as a limit is passed. + * @param {object} geojson + * @param {{maxFeatures?: number, maxPositions?: number, maxEntities?: number}} [limits] + * @returns {{features: number, positions: number, entities: number, + * over: ''|'features'|'positions'|'entities'}} + */ +export function countGeoJsonGeometry( + geojson, + { + maxFeatures = MAX_IMPORT_ENTITIES, + maxPositions = MAX_IMPORT_POSITIONS, + maxEntities = MAX_IMPORT_ENTITIES, + } = {}, +) { + let features = ['Feature', 'FeatureCollection'].includes(geojson?.type) + ? 0 + : 1; + let positions = 0; + let entities = 0; + const result = (over = '') => ({ features, positions, entities, over }); + const stack = [geojson]; + while (stack.length) { + const node = stack.pop(); + if (!node || typeof node !== 'object') continue; + if (node.type === 'FeatureCollection') { + const list = Array.isArray(node.features) ? node.features : []; + for (let i = 0; i < list.length; i++) { + if (list[i]?.type !== 'Feature') + throw new GeoImportError( + 'Every GeoJSON feature must have type Feature.', + ); + stack.push(list[i]); + } + continue; + } + if (node.type === 'Feature') { + features += 1; + if (features > maxFeatures) return result('features'); + if (node.geometry == null) { + entities += 1; + if (entities > maxEntities) return result('entities'); + } + stack.push(node.geometry); + continue; + } + if (node.type === 'GeometryCollection') { + const list = node.geometries; + if (!Array.isArray(list)) + throw new GeoImportError('That GeoJSON has invalid geometries.'); + for (let i = 0; i < list.length; i++) { + if (!list[i] || ['Feature', 'FeatureCollection'].includes(list[i].type)) + throw new GeoImportError( + 'That GeoJSON contains an unsupported geometry.', + ); + stack.push(list[i]); + } + continue; + } + const depth = { + Point: 0, + MultiPoint: 1, + LineString: 1, + MultiLineString: 2, + Polygon: 2, + MultiPolygon: 3, + }[node.type]; + if (depth !== undefined) { + // Multipart geometries create one Cesium entity per component, even + // when the document wraps them in just one Feature. + entities += node.type.startsWith('Multi') + ? Array.isArray(node.coordinates) + ? node.coordinates.length + : 0 + : 1; + if (entities > maxEntities) return result('entities'); + // Walk the coordinate arrays iteratively: a position is an array whose + // first item is a number. + const coords = [[node.coordinates, depth]]; + while (coords.length) { + const [c, remaining] = coords.pop(); + if (!Array.isArray(c)) + throw new GeoImportError('That GeoJSON has invalid coordinates.'); + if (remaining === 0) { + if ( + c.length < 2 || + c.length > 3 || + !c.every(Number.isFinite) || + Math.abs(c[0]) > 180 || + Math.abs(c[1]) > 90 || + (c.length === 3 && Math.abs(c[2]) > 1_000_000) + ) + throw new GeoImportError('That GeoJSON has invalid coordinates.'); + positions += 1; + if (positions > maxPositions) return result('positions'); + } else + for (let i = 0; i < c.length; i++) coords.push([c[i], remaining - 1]); + } + } else + throw new GeoImportError( + 'That GeoJSON contains an unsupported geometry.', + ); + } + return result(); +} + +/** The refusal for a file over a size limit, in the person's terms. */ +function overLimitError({ over, features }) { + if (over === 'entities') + return new GeoImportError( + `That file creates more than ${MAX_IMPORT_ENTITIES.toLocaleString()} map objects (the limit).`, + ); + if (over === 'links') + return new GeoImportError( + `That file has more than ${MAX_KML_LINKS.toLocaleString()} style definitions and image links after cleanup (the limit).`, + ); + if (over === 'features') + return new GeoImportError( + `That file has more than ${MAX_IMPORT_ENTITIES.toLocaleString()} features (the limit).`, + ); + return new GeoImportError( + `That file has more than ${MAX_IMPORT_POSITIONS.toLocaleString()} points (the limit)${features ? ` across ${features.toLocaleString()} features` : ''}.`, + ); +} + +// The import uses WGS84 ground footprints and a short name only. Remove +// loader-sensitive metadata before Cesium allocates properties or entity IDs. +function prepareGeoJson(geojson) { + const stack = [geojson]; + while (stack.length) { + const node = stack.pop(); + if (!node) continue; + delete node.crs; + delete node.id; + if (node.type === 'FeatureCollection') { + for (const feature of node.features) stack.push(feature); + } else if (node.type === 'Feature') { + const name = importFeatureName(node.properties?.name); + node.properties = name ? { name } : {}; + stack.push(node.geometry); + } else if (node.type === 'GeometryCollection') { + for (const geometry of node.geometries) stack.push(geometry); + } else { + const coords = [node.coordinates]; + while (coords.length) { + const value = coords.pop(); + if (typeof value[0] === 'number') value.length = 2; + else for (const child of value) coords.push(child); + } + } + } +} + +/** Inflate raw-deflate bytes with the platform's stream, stopping at `maxBytes`. */ +async function inflateRaw(bytes, maxBytes) { + const stream = new Blob([bytes]) + .stream() + .pipeThrough(new DecompressionStream('deflate-raw')); + const reader = stream.getReader(); + const chunks = []; + let total = 0; + try { + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + total += value.length; + if (total > maxBytes) + throw new GeoImportError( + 'That KMZ unpacks to more than the import limit.', + ); + chunks.push(value); + } + } catch (error) { + await reader.cancel().catch(() => {}); + if (error instanceof GeoImportError) throw error; + throw new GeoImportError('That KMZ is damaged and could not be unpacked.'); + } + const out = new Uint8Array(total); + let at = 0; + for (const chunk of chunks) { + out.set(chunk, at); + at += chunk.length; + } + return out; +} + +/** + * Read the entries of a zip archive (stored or deflated; no encryption, no + * ZIP64). Directory entries are skipped. + * @param {Uint8Array} bytes The whole archive. + * @param {{maxInflatedBytes?: number}} [options] + * @returns {Promise>} Entry path → contents. + */ +export async function readZipEntries( + bytes, + { maxInflatedBytes = MAX_KMZ_INFLATED_BYTES } = {}, +) { + const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); + const fail = (message = 'That KMZ is not a readable zip archive.') => { + throw new GeoImportError(message); + }; + // End of central directory: 22 bytes plus a comment of up to 64 KiB. + let eocd = -1; + for ( + let i = bytes.length - 22; + i >= Math.max(0, bytes.length - 22 - 0xffff); + i-- + ) { + if (view.getUint32(i, true) === 0x06054b50) { + eocd = i; + break; + } + } + if (eocd < 0) fail(); + const count = view.getUint16(eocd + 10, true); + const dirSize = view.getUint32(eocd + 12, true); + let at = view.getUint32(eocd + 16, true); + if (count === 0xffff || at === 0xffffffff || at + dirSize > eocd) + fail('That KMZ uses a zip format this import does not read.'); + const entries = new Map(); + const names = new TextDecoder(); + let inflated = 0; + for (let n = 0; n < count; n++) { + if (at + 46 > bytes.length || view.getUint32(at, true) !== 0x02014b50) + fail(); + const flags = view.getUint16(at + 8, true); + const method = view.getUint16(at + 10, true); + const compressed = view.getUint32(at + 20, true); + const size = view.getUint32(at + 24, true); + const nameLength = view.getUint16(at + 28, true); + const extraLength = view.getUint16(at + 30, true); + const commentLength = view.getUint16(at + 32, true); + const local = view.getUint32(at + 42, true); + const name = names.decode(bytes.subarray(at + 46, at + 46 + nameLength)); + at += 46 + nameLength + extraLength + commentLength; + if (name.endsWith('/')) continue; + if (flags & 0x1) fail('That KMZ is encrypted.'); + if (compressed === 0xffffffff || size === 0xffffffff) + fail('That KMZ uses a zip format this import does not read.'); + if (local + 30 > bytes.length || view.getUint32(local, true) !== 0x04034b50) + fail(); + const start = + local + + 30 + + view.getUint16(local + 26, true) + + view.getUint16(local + 28, true); + if (start + compressed > bytes.length) fail(); + const data = bytes.subarray(start, start + compressed); + let contents; + if (method === 0) contents = data; + else if (method === 8) + contents = await inflateRaw(data, maxInflatedBytes - inflated); + else fail('That KMZ uses a compression method this import does not read.'); + inflated += contents.length; + if (inflated > maxInflatedBytes) + fail('That KMZ unpacks to more than the import limit.'); + entries.set(name, contents); + } + return entries; +} + +/** + * The KML document inside a KMZ: `doc.kml` when present, else the first + * `.kml` entry nearest the archive root. + * @param {Map} entries + * @returns {string|null} Entry path. + */ +export function kmzMainDocument(entries) { + if (entries.has('doc.kml')) return 'doc.kml'; + const kml = [...entries.keys()] + .filter((name) => /\.kml$/i.test(name)) + .sort( + (a, b) => a.split('/').length - b.split('/').length || a.localeCompare(b), + ); + return kml[0] ?? null; +} + +/** + * Resolve a KMZ-relative href to an entry path, or null. + * @param {string} href As written in the KML. + * @param {string} base The KML entry's path. + * @param {Map} entries + * @returns {string|null} + */ +export function resolveKmzPath(href, base, entries) { + const raw = String(href || '').trim(); + if (!raw || /^[a-z][a-z0-9+.-]*:/i.test(raw) || raw.startsWith('//')) + return null; + let decoded; + try { + decoded = decodeURIComponent(raw.split(/[?#]/)[0]); + } catch { + return null; + } + const parts = base.split('/').slice(0, -1); + for (const part of decoded.replace(/\\/g, '/').split('/')) { + if (!part || part === '.') continue; + if (part === '..') parts.pop(); + else parts.push(part); + } + const path = parts.join('/'); + return entries.has(path) ? path : null; +} + +/** MIME type for a KMZ image entry, or null when it is not an image we serve. */ +export function kmzImageType(path) { + const ext = /\.([a-z0-9]+)$/i.exec(path)?.[1]?.toLowerCase(); + return KMZ_IMAGE_TYPES[ext] ?? null; +} + +const elementsNamed = (doc, localName) => + [...doc.getElementsByTagName('*')].filter((el) => el.localName === localName); + +/** + * A `resolveLocal` for `sanitizeKmlDocument`: the blob URL of an image packed + * in the KMZ, or null. Each entry gets one blob and one URL however many + * hrefs name it, so repeated icons don't copy (and decode) the image again. + * @param {Map} entries + * @param {string} main Path of the KMZ's KML document. + * @param {(url: string) => void} onUrl Called once per new URL (to revoke later). + * @param {(blob: Blob) => string} [createUrl] + * @returns {(href: string) => string|null} + */ +export function kmzImageResolver( + entries, + main, + onUrl, + createUrl = (blob) => URL.createObjectURL(blob), +) { + const urls = new Map(); + return (href) => { + const path = resolveKmzPath(href, main, entries); + const type = path && kmzImageType(path); + if (!type) return null; + let url = urls.get(path); + if (!url) { + url = createUrl(new Blob([entries.get(path)], { type })); + urls.set(path, url); + onUrl(url); + } + return url; + }; +} + +/** + * Strip everything from a parsed KML document that would make the page reach + * outside the file. `resolveLocal(href)` returns a URL for an image packed in + * the same KMZ (or null); every other href is removed, so icons fall back to + * Cesium's default pin and a ground overlay without a local image is dropped. + * @param {Document} doc + * @param {(href: string) => string|null} [resolveLocal] + * @returns {{removed: number}} How many elements were taken out. + */ +export function sanitizeKmlDocument(doc, resolveLocal = () => null) { + let removed = 0; + const drop = (el) => { + if (el.parentNode) { + el.parentNode.removeChild(el); + removed += 1; + } + }; + // One snapshot and pass, rather than rescanning the whole XML tree for + // every discarded element type. Strip unused metadata before the loader + // constructs descriptions, availability intervals and camera objects. + for (const el of [...doc.getElementsByTagName('*')]) + if ( + KML_REMOVED_ELEMENTS.has(el.localName) || + el.localName in Object.prototype + ) + drop(el); + else if (!['Style', 'StyleMap'].includes(el.localName)) + el.removeAttribute('id'); + for (const href of elementsNamed(doc, 'href')) { + const value = (href.textContent || '').trim(); + if (!value) continue; + const local = resolveLocal(value); + if (local) href.textContent = local; + else if (/^data:image\/(png|jpeg|gif|webp|bmp);/i.test(value)) continue; + else { + const icon = href.parentNode; + const overlay = icon?.parentNode; + if (overlay?.localName === 'GroundOverlay') { + drop(overlay); + continue; + } + // An empty Icon explicitly hides the billboard in Cesium. Remove the + // blocked icon definition so the loader supplies its local default pin. + // Authored empty Icons (with no blocked href) still mean no marker. + drop( + icon?.localName === 'Icon' && icon.parentNode?.localName === 'IconStyle' + ? icon + : href, + ); + } + } + for (const styleUrl of elementsNamed(doc, 'styleUrl')) { + const value = (styleUrl.textContent || '').trim(); + if (!value.startsWith('#')) drop(styleUrl); + else styleUrl.textContent = value; + } + for (const pair of elementsNamed(doc, 'Pair')) + if ( + !childText(pair, 'key') || + (!childrenNamed(pair, 'styleUrl').length && + !childrenNamed(pair, 'Style').length) + ) + drop(pair); + // Draw recorded KML paths as their original static vertices. Letting the + // loader build time-dynamic tracks would expand each into thousands of + // interpolated points later and exceed the checked geometry budget. + for (const track of elementsNamed(doc, 'Track')) { + const line = doc.createElementNS( + 'http://www.opengis.net/kml/2.2', + 'LineString', + ); + const coordinates = doc.createElementNS( + 'http://www.opengis.net/kml/2.2', + 'coordinates', + ); + const recordedCoordinates = childrenNamed(track, 'coord').map((coord) => + coord.textContent.trim().split(/\s+/).join(','), + ); + coordinates.textContent = recordedCoordinates.join(' '); + line.appendChild(coordinates); + for (const mode of childrenNamed(track, 'altitudeMode')) + line.appendChild(mode.cloneNode(true)); + track.parentNode?.replaceChild(line, track); + } + for (const tracks of elementsNamed(doc, 'MultiTrack')) { + const geometry = doc.createElementNS( + 'http://www.opengis.net/kml/2.2', + 'MultiGeometry', + ); + for (const line of childrenNamed(tracks, 'LineString')) + geometry.appendChild(line); + tracks.parentNode?.replaceChild(geometry, tracks); + } + for (const line of [ + ...elementsNamed(doc, 'LineString'), + ...elementsNamed(doc, 'LinearRing'), + ]) { + const mode = childText(line, 'altitudeMode'); + if ( + !mode || + mode === 'clampToGround' || + mode === 'clampToSeaFloor' || + !childrenNamed(line, 'tessellate').length + ) { + let tessellate = childrenNamed(line, 'tessellate')[0]; + if (!tessellate) { + tessellate = doc.createElementNS( + 'http://www.opengis.net/kml/2.2', + 'tessellate', + ); + line.appendChild(tessellate); + } + tessellate.textContent = '1'; + } + } + return { removed }; +} + +/** Child elements of `el` with this local name (any namespace). */ +const childrenNamed = (el, localName) => { + const out = []; + const kids = el?.childNodes || []; + for (let i = 0; i < kids.length; i++) + if (kids[i].nodeType === 1 && kids[i].localName === localName) + out.push(kids[i]); + return out; +}; +const childText = (el, localName) => + (childrenNamed(el, localName)[0]?.textContent || '').trim(); + +/** + * Share identical styles before the loader builds their graphics. Inline + * overrides stay inline StyleMaps, so an existing styleUrl still supplies + * properties the override leaves unset. Named references are redirected. + * Random-color styles remain unchanged; duplicate IDs + * keep the loader's original precedence instead of being rewritten. + * @param {Document} doc Sanitized KML. + * @param {{serializeStyle?: Function, maxStyles?: number}} [options] + * @returns {number} Number of original style blocks replaced. + */ +export function deduplicateKmlStyles( + doc, + { serializeStyle, maxStyles = MAX_KML_LINKS } = {}, +) { + const serializer = serializeStyle ? null : new XMLSerializer(); + const serialize = + serializeStyle || ((el) => serializer.serializeToString(el)); + const groups = new Map(); + const usedIds = new Set(); + const duplicateIds = new Set(); + for (const el of doc.getElementsByTagName('*')) { + const id = el.getAttribute('id'); + if (id !== null) { + if (usedIds.has(id)) duplicateIds.add(id); + usedIds.add(id); + } + } + for (const style of elementsNamed(doc, 'Style')) { + if ( + elementsNamed(style, 'colorMode').some( + (el) => el.textContent.trim() === 'random', + ) + ) + continue; + const template = style.cloneNode(true); + template.removeAttribute('id'); + const key = serialize(template); + let group = groups.get(key); + if (!group) { + group = { template, nodes: [] }; + groups.set(key, group); + if (groups.size > maxStyles) throw overLimitError({ over: 'links' }); + } + group.nodes.push(style); + } + const redirects = new Map(); + let replaced = 0; + let serial = 0; + let container; + for (const { template, nodes } of groups.values()) { + if ( + nodes.length < 2 || + nodes.some((el) => duplicateIds.has(el.getAttribute('id'))) + ) + continue; + const ns = template.namespaceURI || 'http://www.opengis.net/kml/2.2'; + if (!container) { + const root = doc.documentElement; + container = + root.localName === 'Document' + ? root + : childrenNamed(root, 'Document')[0]; + if (!container) { + container = doc.createElementNS(ns, 'Document'); + while (root.firstChild) container.appendChild(root.firstChild); + root.appendChild(container); + } + } + let id; + do { + id = `gev-import-style-${++serial}`; + } while (usedIds.has(id)); + usedIds.add(id); + template.setAttribute('id', id); + container.appendChild(template); + for (const style of nodes) { + const originalId = style.getAttribute('id'); + if (originalId !== null) redirects.set(originalId, id); + let reference; + if (style.parentNode.localName === 'Pair') { + reference = doc.createElementNS(ns, 'styleUrl'); + reference.textContent = `#${id}`; + } else { + reference = doc.createElementNS(ns, 'StyleMap'); + for (const state of ['normal', 'highlight']) { + const pair = doc.createElementNS(ns, 'Pair'); + const key = doc.createElementNS(ns, 'key'); + key.textContent = state; + const url = doc.createElementNS(ns, 'styleUrl'); + url.textContent = `#${id}`; + pair.appendChild(key); + pair.appendChild(url); + reference.appendChild(pair); + } + } + style.parentNode.replaceChild(reference, style); + replaced += 1; + } + } + for (const url of elementsNamed(doc, 'styleUrl')) { + const value = url.textContent.trim(); + const id = value.startsWith('#') && redirects.get(value.slice(1)); + if (id) url.textContent = `#${id}`; + } + return replaced; +} + +/** + * Count placemarks and vertices in a parsed KML document before Cesium reads + * it: every `` tuple and every `gx:coord`, plus normalized + * style definitions and distinct image `` values (`links`). + * Inline StyleMaps are references, not additional shared style definitions. + * @param {Document} doc + * @returns {{features: number, entities: number, positions: number, links: number, + * over: ''|'features'|'entities'|'positions'|'links'}} + */ +export function countKmlGeometry( + doc, + { + maxFeatures = MAX_IMPORT_ENTITIES, + maxEntities = MAX_IMPORT_ENTITIES, + maxPositions = MAX_IMPORT_POSITIONS, + maxLinks = MAX_KML_LINKS, + } = {}, +) { + let features = 0; + let entities = 0; + let positions = 0; + let links = 0; + const imageLinks = new Set(); + const result = (over) => ({ features, entities, positions, links, over }); + const geometryNames = new Set([ + 'Point', + 'LineString', + 'LinearRing', + 'Polygon', + 'Track', + 'MultiTrack', + 'MultiGeometry', + 'Model', + ]); + for (const el of doc.getElementsByTagName('*')) { + const name = el.localName; + if ( + ['north', 'south', 'east', 'west', 'rotation', 'altitude'].includes(name) + ) { + const value = Number(el.textContent); + const limit = ['north', 'south'].includes(name) + ? 90 + : name === 'altitude' + ? 1_000_000 + : 360; + if ( + !el.textContent.trim() || + !Number.isFinite(value) || + Math.abs(value) > limit + ) + throw new GeoImportError('That KML has invalid geometry bounds.'); + } + // Cesium creates entities for folders, a parent for every placemark, and + // a child for each geometry in a MultiGeometry, including nested containers. + if ( + name === 'Placemark' || + name === 'Folder' || + name === 'GroundOverlay' || + (geometryNames.has(name) && + ['MultiGeometry', 'MultiTrack'].includes(el.parentNode?.localName)) + ) { + entities += 1; + if (entities > maxEntities) return result('entities'); + } + if ( + name === 'Style' || + (name === 'StyleMap' && el.getAttribute('id') !== null) + ) { + links += 1; + if (links > maxLinks) return result('links'); + } else if (name === 'href') { + const href = el.textContent.trim(); + if (!imageLinks.has(href)) { + imageLinks.add(href); + links += 1; + if (links > maxLinks) return result('links'); + } + } else if (name === 'Placemark') { + features += 1; + if (features > maxFeatures) return result('features'); + } else if (name === 'coordinates') { + const text = el.textContent || ''; + for (const match of text.matchAll(/\S+/g)) { + checkKmlPosition(match[0].split(',')); + positions += 1; + if (positions > maxPositions) return result('positions'); + } + } else if (name === 'coord') { + checkKmlPosition(el.textContent.trim().split(/\s+/)); + positions += 1; + if (positions > maxPositions) return result('positions'); + } + } + return result(''); +} + +function checkKmlPosition(tuple) { + const numbers = tuple.map(Number); + if ( + tuple.length < 2 || + tuple.length > 3 || + tuple.some((v) => !v.trim()) || + !numbers.every(Number.isFinite) || + Math.abs(numbers[0]) > 180 || + Math.abs(numbers[1]) > 90 || + (numbers.length === 3 && Math.abs(numbers[2]) > 1_000_000) + ) + throw new GeoImportError('That KML has invalid coordinates.'); +} + +/** + * Read a parsed GPX document into GeoJSON: waypoints become points, each + * route a line and each track one MultiLineString with a line per `` + * (segment breaks are gaps in the recording and stay gaps). Only names are + * kept, and no times, so a dated track draws without a clock. Throws + * `GeoImportError` past the size limits. + * @param {Document} doc + * @returns {object} A FeatureCollection. + */ +export function gpxToGeoJson( + doc, + { + maxFeatures = MAX_IMPORT_ENTITIES, + maxPositions = MAX_IMPORT_POSITIONS, + } = {}, +) { + const root = doc.documentElement; + if (root?.localName !== 'gpx') + throw new GeoImportError('That GPX file has no document.'); + const features = []; + let positions = 0; + const point = (el) => { + const rawLat = el.getAttribute('lat'); + const rawLon = el.getAttribute('lon'); + if (rawLat == null || rawLon == null || !rawLat.trim() || !rawLon.trim()) + return null; + const lat = Number(rawLat); + const lon = Number(rawLon); + if ( + !Number.isFinite(lat) || + !Number.isFinite(lon) || + Math.abs(lat) > 90 || + Math.abs(lon) > 180 + ) + return null; + positions += 1; + if (positions > maxPositions) + throw overLimitError({ + over: 'positions', + features: features.length, + positions, + }); + return [lon, lat]; + }; + const add = (name, geometry) => { + if (features.length >= maxFeatures) + throw overLimitError({ + over: 'features', + features: features.length + 1, + positions, + }); + features.push({ + type: 'Feature', + properties: name ? { name } : {}, + geometry, + }); + }; + for (const wpt of childrenNamed(root, 'wpt')) { + const at = point(wpt); + if (at) add(childText(wpt, 'name'), { type: 'Point', coordinates: at }); + } + for (const rte of childrenNamed(root, 'rte')) { + const line = []; + for (const pt of childrenNamed(rte, 'rtept')) { + const at = point(pt); + if (at) line.push(at); + } + if (line.length >= 2) + add(childText(rte, 'name'), { type: 'LineString', coordinates: line }); + } + for (const trk of childrenNamed(root, 'trk')) { + const lines = []; + for (const seg of childrenNamed(trk, 'trkseg')) { + const line = []; + for (const pt of childrenNamed(seg, 'trkpt')) { + const at = point(pt); + if (at) line.push(at); + } + if (line.length >= 2) lines.push(line); + } + if (lines.length) + add(childText(trk, 'name'), { + type: 'MultiLineString', + coordinates: lines, + }); + } + return { type: 'FeatureCollection', features }; +} + +/** A feature name as short plain text (no control characters), or undefined. */ +export function importFeatureName(value) { + const text = String(value ?? '') + .replace(/[\u0000-\u001f\u007f]/g, ' ') + .replace(/\s+/g, ' ') + .trim(); + if (!text) return undefined; + return text.length > MAX_IMPORT_NAME_LENGTH + ? `${text.slice(0, MAX_IMPORT_NAME_LENGTH - 1)}…` + : text; +} + +/** + * Make every entity of an imported data source inert: no description, + * balloon, extended data or properties, no native label, and a name that is + * short plain text. Generated pin canvases share a data URL so Cesium uploads + * each image once, rather than allocating a texture entry for every entity. + * Returns how many entities were touched. + * @param {{entities: {values: object[]}}} dataSource + * @returns {number} + */ +export function scrubImportedEntities(dataSource) { + const collection = dataSource?.entities; + const list = collection?.values || []; + const images = new Map(); + collection?.suspendEvents?.(); + try { + for (let i = 0; i < list.length; i++) { + const entity = list[i]; + let name; + try { + name = importFeatureName( + typeof entity.name === 'string' ? entity.name : undefined, + ); + } catch { + name = undefined; + } + entity.name = name; + entity.description = undefined; + entity.label = undefined; + entity.properties = undefined; + if (entity.kml && typeof entity.kml === 'object') entity.kml = undefined; + if ( + entity.billboard?.heightReference?.getValue?.() === + Cesium.HeightReference.CLAMP_TO_GROUND + ) + entity.billboard.disableDepthTestDistance = Number.POSITIVE_INFINITY; + const property = entity.billboard?.image; + if (property?.isConstant === false) continue; + const image = property?.getValue?.(); + if (image?.tagName === 'CANVAS') { + let url = images.get(image); + if (!url) { + url = image.toDataURL(); + images.set(image, url); + } + entity.billboard.image = url; + } + } + } finally { + collection?.resumeEvents?.(); + } + return list.length; +} + +/** + * Take time out of an imported data source so adding it cannot move the app's + * shared clock: the data source reports no clock (the viewer adopts the clock + * of every added source that has one), every entity is available for all + * time. KML tracks have already become their original static vertices in + * `sanitizeKmlDocument`, so no sampled-position fallback is needed here. + * @param {object} dataSource + * @returns {object} The same data source. + */ +export function detachImportedTime(dataSource) { + Object.defineProperty(dataSource, 'clock', { + configurable: true, + enumerable: false, + get: () => undefined, + set: () => {}, + }); + const list = dataSource?.entities?.values || []; + for (let i = 0; i < list.length; i++) { + const entity = list[i]; + entity.availability = undefined; + } + return dataSource; +} + +function parseXml(text, what) { + // Match the namespace repairs Cesium applies to ordinary exported KML. + if (what !== 'GPX') + text = String(text).replace(/]*>/i, (root) => { + const seen = new Set(); + root = root.replace( + /\s+xmlns:([\w.-]+)\s*=\s*(["'])[^"']*\2/g, + (declaration, prefix) => { + if (seen.has(prefix)) return ''; + seen.add(prefix); + return declaration; + }, + ); + if (!seen.has('xsi') && /[<\s]xsi:/.test(text)) + root = root.replace( + />$/, + ' xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">', + ); + return root; + }); + const doc = new DOMParser().parseFromString( + String(text).replace(/^/, ''), + 'application/xml', + ); + if (doc.getElementsByTagName('parsererror').length) + throw new GeoImportError(`That ${what} file is not valid XML.`); + return doc; +} + +function decodeXml(bytes) { + const encoding = + bytes[0] === 0xff && bytes[1] === 0xfe + ? 'utf-16le' + : bytes[0] === 0xfe && bytes[1] === 0xff + ? 'utf-16be' + : 'utf-8'; + return new TextDecoder(encoding).decode(bytes); +} + +function checkLocalImages(doc, entries = new Map(), main = '') { + const charge = createImportImageBudget(); + const checked = new Set(); + for (const href of elementsNamed(doc, 'href')) { + const value = href.textContent.trim(); + if (checked.has(value)) continue; + checked.add(value); + const path = resolveKmzPath(value, main, entries); + try { + if (path && kmzImageType(path)) charge(path, entries.get(path)); + else if (/^data:image\/(png|jpeg|gif|webp|bmp);/i.test(value)) { + const comma = value.indexOf(','); + const header = value.slice(0, comma); + const body = value.slice(comma + 1); + const raw = /;base64$/i.test(header) + ? atob(body) + : decodeURIComponent(body); + charge( + value, + Uint8Array.from(raw, (c) => c.charCodeAt(0)), + ); + } + } catch { + throw new GeoImportError( + 'That file has an unreadable image or exceeds the 16-megapixel image limit.', + ); + } + } +} + +// Cesium also processes KML in deferred tasks; an exception there does not +// reject its load promise. Keep a failed file from blocking the serial queue. +async function loadKml(doc, viewer, signal) { + let removeError; + try { + return await boundedImport( + Promise.race([ + Cesium.KmlDataSource.load(doc, { + camera: viewer.scene.camera, + canvas: viewer.scene.canvas, + clampToGround: true, + }), + new Promise((_, reject) => { + const failed = () => { + reject(new GeoImportError('That KML could not be read.')); + }; + globalThis.addEventListener?.('error', failed); + removeError = () => globalThis.removeEventListener?.('error', failed); + }), + ]), + signal, + ); + } finally { + removeError?.(); + } +} + +// Resolve a static file's ground pins in batches before they are visualized. +// Per-billboard CLAMP_TO_GROUND subscriptions scale poorly on large imports. +const groundPins = new WeakMap(); +async function settleGroundPins(dataSource, viewer, signal) { + let pins = groundPins.get(dataSource); + if (!pins) { + pins = []; + for (const entity of dataSource.entities.values) { + const reference = entity.billboard?.heightReference?.getValue?.(); + if ( + reference === Cesium.HeightReference.CLAMP_TO_GROUND || + reference === Cesium.HeightReference.RELATIVE_TO_GROUND + ) { + const position = entity.position?.getValue?.(Cesium.JulianDate.now()); + if (!position) continue; + const cartographic = Cesium.Cartographic.fromCartesian(position); + if (!cartographic) + throw new GeoImportError('That file has invalid point coordinates.'); + pins.push({ + entity, + original: position, + cartographic, + offset: + reference === Cesium.HeightReference.RELATIVE_TO_GROUND + ? cartographic.height + : 0, + }); + entity.billboard.heightReference = Cesium.HeightReference.NONE; + entity.billboard.disableDepthTestDistance = Number.POSITIVE_INFINITY; + } + } + groundPins.set(dataSource, pins); + } + for (let at = 0; at < pins.length; at += 256) { + signal?.throwIfAborted(); + const batch = pins.slice(at, at + 256); + const positions = batch.map( + ({ cartographic: c }) => + new Cesium.Cartographic(c.longitude, c.latitude, 0), + ); + const mesh = + viewer.scene?.globe?.show === false && + viewer.scene?.clampToHeightSupported; + // A coarse rendered ellipsoid mesh can report a height kilometres below + // zero. Its provider's exact height is zero even while a 3D map hides it; + // a buried starting point can make the downward mesh ray miss the surface. + const ellipsoid = + viewer.terrainProvider instanceof Cesium.EllipsoidTerrainProvider; + for (let i = 0; i < positions.length; i++) { + const sampled = ellipsoid + ? 0 + : viewer.scene?.globe?.getHeight?.(positions[i]); + positions[i].height = mesh + ? (batch[i].groundHeight ?? + Math.max(0, Number.isFinite(sampled) ? sampled : 0)) + : Number.isFinite(sampled) + ? sampled + : batch[i].groundHeight || 0; + } + const points = positions.map((c) => + Cesium.Cartesian3.fromRadians(c.longitude, c.latitude, c.height), + ); + const height = viewer.camera?.positionCartographic?.height; + if (mesh && Number.isFinite(height) && height < 250_000) { + const rectangle = viewer.camera.computeViewRectangle?.(); + const indices = []; + for (let i = 0; i < positions.length; i++) + if (rectangle && Cesium.Rectangle.contains(rectangle, positions[i])) + indices.push(i); + if (indices.length) { + const resolved = await boundedImport( + viewer.scene.clampToHeightMostDetailed( + indices.map((i) => { + const c = positions[i]; + return Cesium.Cartesian3.fromRadians( + c.longitude, + c.latitude, + Math.max(0, c.height), + ); + }), + ), + signal, + ); + for (let i = 0; i < indices.length; i++) + if (resolved[i]) points[indices[i]] = resolved[i]; + } + } + signal?.throwIfAborted(); + for (let i = 0; i < batch.length; i++) { + const item = batch[i]; + const c = points[i] && Cesium.Cartographic.fromCartesian(points[i]); + if (!c) + throw new GeoImportError( + 'Ground heights are not available for that file. Try again when the map has loaded.', + ); + item.groundHeight = c.height; + c.height += item.offset; + item.entity.position = + Math.abs(c.height - item.cartographic.height) < 1e-6 + ? item.original + : Cesium.Cartesian3.fromRadians(c.longitude, c.latitude, c.height); + } + if (at + batch.length < pins.length) + await new Promise((resolve) => setTimeout(resolve, 0)); + } +} + +async function boundedImport(promise, signal) { + let timer, abort; + try { + signal?.throwIfAborted(); + return await Promise.race([ + promise, + new Promise((_, reject) => { + abort = () => reject(signal.reason); + signal?.addEventListener('abort', abort, { once: true }); + timer = setTimeout( + () => reject(new GeoImportError('That file took too long to read.')), + 60_000, + ); + }), + ]); + } finally { + clearTimeout(timer); + signal?.removeEventListener('abort', abort); + } +} + +/** + * Load a ground footprint, retaining explicit KML altitude modes. Call mount after + * adding the source; revoke owns pin and blob cleanup. + * @param {File|Blob & {name?: string}} file + * @param {{viewer: Cesium.Viewer, color?: string}} options `color` is a CSS color. + * @returns {Promise<{dataSource: Cesium.DataSource, name: string, + * format: string, entityCount: number, removed: number, mount: Function, revoke: Function}>} + */ +export async function loadGeoFile( + file, + { viewer, color = '#39d0ff', signal } = {}, +) { + signal = signal + ? AbortSignal.any([signal, AbortSignal.timeout(60_000)]) + : AbortSignal.timeout(60_000); + signal?.throwIfAborted(); + if (!file || !Number.isFinite(file.size)) + throw new GeoImportError('Nothing to import.'); + if (file.size > MAX_IMPORT_BYTES) + throw new GeoImportError( + `That file is larger than ${MAX_IMPORT_BYTES / 1024 / 1024} MiB.`, + ); + let bytes = new Uint8Array(await boundedImport(file.arrayBuffer(), signal)); + const format = detectGeoFileFormat(file.name, bytes.subarray(0, 1024)); + if (!format) + throw new GeoImportError( + 'Only GeoJSON, KML, KMZ and GPX files can be imported.', + ); + const stroke = Cesium.Color.fromCssColorString(color); + const urls = []; + let pins; + const revoke = () => { + pins?.destroy(); + urls.splice(0).forEach((url) => URL.revokeObjectURL(url)); + }; + const name = importDisplayName(file.name); + let dataSource; + let removed = 0; + try { + if (format === 'geojson' || format === 'gpx') { + let geojson; + if (format === 'gpx') { + geojson = gpxToGeoJson(parseXml(decodeXml(bytes), 'GPX')); + } else { + geojson = parseGeoJsonText(new TextDecoder().decode(bytes)); + } + const size = countGeoJsonGeometry(geojson); + if (size.over) throw overLimitError(size); + if (!size.positions) + throw new GeoImportError('That file has nothing to draw.'); + prepareGeoJson(geojson); + bytes = null; + dataSource = await boundedImport( + Cesium.GeoJsonDataSource.load(geojson, { + clampToGround: true, + stroke, + fill: stroke.withAlpha(0.25), + strokeWidth: 3, + markerColor: stroke, + // No HTML description table is built from the file's properties. + describe: () => undefined, + }), + signal, + ); + } else { + let text; + let entries; + let main; + let resolveLocal = () => null; + if (format === 'kmz') { + entries = await readZipEntries(bytes); + main = kmzMainDocument(entries); + if (!main) throw new GeoImportError('That KMZ holds no KML document.'); + text = decodeXml(entries.get(main)); + resolveLocal = kmzImageResolver(entries, main, (url) => urls.push(url)); + } else { + text = decodeXml(bytes); + } + const doc = parseXml(text, format.toUpperCase()); + text = null; + bytes = null; + // Bound geometry before transformations, then count the styles and + // image links that survive sanitizing and sharing repeated styles. + const size = countKmlGeometry(doc, { maxLinks: Infinity }); + if (size.over) throw overLimitError(size); + checkLocalImages(doc, entries, main); + removed = sanitizeKmlDocument(doc, resolveLocal).removed; + entries?.clear(); + deduplicateKmlStyles(doc); + const normalizedSize = countKmlGeometry(doc); + if (normalizedSize.over) throw overLimitError(normalizedSize); + dataSource = await loadKml(doc, viewer, signal); + } + } catch (error) { + revoke(); + if (error instanceof GeoImportError) throw error; + throw new GeoImportError( + `That ${format.toUpperCase()} file could not be read.`, + ); + } + const entityCount = dataSource.entities.values.length; + if (entityCount > MAX_IMPORT_ENTITIES) { + revoke(); + throw new GeoImportError( + `That file has ${entityCount.toLocaleString()} map objects; the limit is ${MAX_IMPORT_ENTITIES.toLocaleString()}.`, + ); + } + const now = Cesium.JulianDate.now(); + const drawable = dataSource.entities.values.some( + (e) => + e.billboard?.image?.getValue(now) || + e.point || + e.rectangle || + e.polyline?.positions?.getValue(now)?.length >= 2 || + e.wall?.positions?.getValue(now)?.length >= 2 || + e.polygon?.hierarchy?.getValue(now)?.positions?.length >= 3, + ); + if (!drawable) { + revoke(); + throw new GeoImportError('That file has nothing to draw.'); + } + scrubImportedEntities(dataSource); + try { + await settleGroundPins(dataSource, viewer, signal); + } catch (error) { + revoke(); + throw error instanceof GeoImportError + ? error + : new GeoImportError( + 'Ground heights could not be read. Try that file again.', + ); + } + // Keep the clock invariant explicit even though unused KML dates have + // already been stripped and all supported geometries are static. + detachImportedTime(dataSource); + try { + pins = createImportedPins(dataSource, viewer); + } catch { + revoke(); + throw new GeoImportError('That file has invalid point graphics.'); + } + // The display name travels beside the data source (the file name, not + // anything the document says about itself). + return { + dataSource, + name, + format, + entityCount, + removed, + revoke, + mount: () => pins.mount(), + refreshGround: (refreshSignal) => + settleGroundPins( + dataSource, + viewer, + refreshSignal + ? AbortSignal.any([refreshSignal, AbortSignal.timeout(15_000)]) + : AbortSignal.timeout(15_000), + ).then(() => pins.sync()), + }; +} diff --git a/src/data/geoFileImport.test.mjs b/src/data/geoFileImport.test.mjs new file mode 100644 index 000000000..af02ec222 --- /dev/null +++ b/src/data/geoFileImport.test.mjs @@ -0,0 +1,1173 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { deflateRawSync } from 'node:zlib'; +import * as Cesium from 'cesium'; +import { + GeoImportError, + loadGeoFile, + MAX_IMPORT_POSITIONS, + MAX_IMPORT_ENTITIES, + countGeoJsonGeometry, + countKmlGeometry, + deduplicateKmlStyles, + detachImportedTime, + gpxToGeoJson, + importFeatureName, + scrubImportedEntities, + sanitizeKmlDocument, + detectGeoFileFormat, + importDisplayName, + kmzImageResolver, + kmzImageType, + kmzMainDocument, + parseGeoJsonText, + readZipEntries, + resolveKmzPath, +} from './geoFileImport.js'; +import { createImportImageBudget, importImageSize } from './geoImportImages.js'; + +test('real loader grounds XYZ GeoJSON and strips loader-sensitive metadata before allocation', async () => { + const previous = Cesium.PinBuilder.prototype.fromColor; + Cesium.PinBuilder.prototype.fromColor = () => ({ + toDataURL: () => 'data:image/png;base64,local', + tagName: 'CANVAS', + }); + try { + const object = { + type: 'FeatureCollection', + crs: { type: 'name', properties: { name: 'unsupported' } }, + features: [ + { + type: 'Feature', + id: 'kartaview-owned', + properties: { name: 'Test', addProperty: 1, 'marker-symbol': 'rail' }, + geometry: { type: 'Point', coordinates: [1, 2, 999] }, + }, + { + type: 'Feature', + geometry: { + type: 'Polygon', + coordinates: [ + [ + [1, 2, 0], + [2, 2, 0], + [2, 3, 0], + [1, 2, 0], + ], + ], + }, + }, + ], + }; + const loaded = await loadGeoFile( + new File([JSON.stringify(object)], 'xyz.geojson'), + { + viewer: { + scene: { + frameState: { mode: Cesium.SceneMode.SCENE3D }, + primitives: new Cesium.PrimitiveCollection(), + globe: { getHeight: () => 1234 }, + }, + }, + }, + ); + const [point, polygon] = + loaded.dataSource.importedEntities || loaded.dataSource.entities.values; + assert.notEqual(point.id, 'kartaview-owned'); + assert.equal(point.name, 'Test'); + assert.equal(point.properties, undefined); + assert.equal( + point.billboard.heightReference.getValue(), + Cesium.HeightReference.NONE, + ); + assert.equal(point.billboard.disableDepthTestDistance.getValue(), Infinity); + assert.ok( + Math.abs( + Cesium.Cartographic.fromCartesian( + point.position.getValue(Cesium.JulianDate.now()), + ).height - 1234, + ) < 1e-6, + ); + assert.notEqual(polygon.polygon.perPositionHeight?.getValue(), true); + assert.equal(loaded.dataSource.clock, undefined); + assert.equal( + loaded.dataSource.entities.values.length, + 1, + 'only polygon geometry remains in the per-frame entity collection', + ); + assert.equal(loaded.dataSource.importedBillboards.length, 1); + const update = Cesium.BillboardCollection.prototype.update; + let passes = 0; + Cesium.BillboardCollection.prototype.update = () => passes++; + try { + const collection = loaded.dataSource.importedBillboards; + collection.update({ passes: { pick: true, offscreen: true } }); + assert.equal(passes, 0, 'height picks ignore imported markers'); + collection.update({ passes: { pick: true, offscreen: false } }); + collection.update({ passes: { render: true, offscreen: true } }); + assert.equal(passes, 2, 'ordinary picks and screenshots include markers'); + } finally { + Cesium.BillboardCollection.prototype.update = update; + } + + loaded.mount(); + loaded.dataSource.show = false; + assert.equal(loaded.dataSource.importedBillboards.show, false); + loaded.dataSource.show = true; + await loaded.refreshGround(); + assert.equal( + point.billboard.heightReference.getValue(), + Cesium.HeightReference.NONE, + ); + loaded.revoke(); + assert.ok(loaded.dataSource.importedBillboards.isDestroyed()); + } finally { + Cesium.PinBuilder.prototype.fromColor = previous; + } +}); + +test('real loader refuses empty files and malformed nested geometry before allocating entities', async () => { + for (const object of [ + { type: 'Feature', geometry: null }, + { + type: 'FeatureCollection', + features: [{ geometry: { type: 'Point', coordinates: [1, 2] } }], + }, + { type: 'Feature', geometry: { type: 'Topology', arcs: [] } }, + ...[ + [1, '2'], + [Infinity, 0], + [0, 91], + [0, 0, 1e100], + ].map((coordinates) => ({ type: 'Point', coordinates })), + ]) + await assert.rejects( + loadGeoFile(new File([JSON.stringify(object)], 'bad.geojson'), { + viewer: {}, + }), + GeoImportError, + ); +}); + +test('KML tracks retain their explicit altitude and ground lines are tessellated', () => { + const track = xmlEl('Track', {}, [ + xmlEl('altitudeMode', {}, 'absolute'), + xmlEl('coord', {}, '1 2 10000'), + xmlEl('coord', {}, '2 3 10000'), + ]); + const line = xmlEl('LineString', {}, [xmlEl('coordinates', {}, '1,2 2,3')]); + const ground = xmlEl('Track', {}, [ + xmlEl('coord', {}, '1 2 0'), + xmlEl('coord', {}, '2 3 0'), + ]); + const root = xmlEl('kml', {}, [track, line, ground]); + sanitizeKmlDocument(xmlDoc(root)); + assert.equal( + root.childNodes[0].childNodes.find((n) => n.localName === 'altitudeMode') + .textContent, + 'absolute', + ); + assert.equal( + root.childNodes[1].childNodes.find((n) => n.localName === 'tessellate') + .textContent, + '1', + ); + assert.equal( + root.childNodes[2].childNodes.find((n) => n.localName === 'tessellate') + .textContent, + '1', + ); +}); + +test('sanitizer normalizes styles, removes incomplete pairs and keeps colored overlays', () => { + const local = xmlEl('styleUrl', {}, ' #s '); + const pair = xmlEl('Pair', {}, [ + xmlEl('key', {}, 'normal'), + xmlEl('styleUrl', {}, 'https://example.invalid/s'), + ]); + const overlay = xmlEl('GroundOverlay', {}, [xmlEl('color', {}, 'ff00ff00')]); + const hidden = xmlEl('visibility', {}, '0'); + const id = xmlEl('Placemark', { id: 'kartaview-123' }); + const root = xmlEl('kml', {}, [ + local, + xmlEl('StyleMap', { id: 's' }, [pair]), + overlay, + hidden, + id, + xmlEl('toString'), + ]); + sanitizeKmlDocument(xmlDoc(root)); + assert.equal(local.textContent, '#s'); + assert.equal(pair.parentNode, null); + assert.ok(root.childNodes.includes(overlay)); + assert.equal(hidden.parentNode, null); + assert.equal(id.getAttribute('id'), null); + assert.ok(!root.childNodes.some((n) => n.localName === 'toString')); +}); + +test('image budget rejects large decoded rasters before decoding and shares repeated images', () => { + const png = (w, h) => { + const b = new Uint8Array(24); + const v = new DataView(b.buffer); + v.setUint32(0, 0x89504e47); + b.set([73, 72, 68, 82], 12); + v.setUint32(16, w); + v.setUint32(20, h); + return b; + }; + assert.deepEqual(importImageSize(png(4096, 4096)), [4096, 4096]); + const charge = createImportImageBudget(); + charge('shared', png(4096, 4096)); + charge('shared', png(4096, 4096)); + assert.throws(() => charge('other', png(1, 1)), /limit/); + assert.throws( + () => createImportImageBudget()('huge', png(16384, 16384)), + /limit/, + ); +}); + +test('pending file reads can be cancelled without waiting for the loader timeout', async () => { + const controller = new AbortController(); + const file = { + name: 'waiting.geojson', + size: 1, + arrayBuffer: () => new Promise(() => {}), + }; + const promise = loadGeoFile(file, { viewer: {}, signal: controller.signal }); + controller.abort(new Error('cancelled by teardown')); + await assert.rejects(promise, /cancelled by teardown/); +}); + +test('a hidden coarse globe never seeds a mesh-clamp ray below the surface', async () => { + const previous = Cesium.PinBuilder.prototype.fromColor; + Cesium.PinBuilder.prototype.fromColor = () => ({ + tagName: 'CANVAS', + toDataURL: () => 'data:image/png;base64,local', + }); + const starts = []; + try { + for (const terrainProvider of [new Cesium.EllipsoidTerrainProvider(), {}]) { + const viewer = { + terrainProvider, + camera: { + positionCartographic: { height: 100000 }, + computeViewRectangle: () => Cesium.Rectangle.MAX_VALUE, + }, + scene: { + frameState: { mode: Cesium.SceneMode.SCENE3D }, + globe: { show: false, getHeight: () => -12311 }, + clampToHeightSupported: true, + clampToHeightMostDetailed: async (points) => + points.map((p) => { + const c = Cesium.Cartographic.fromCartesian(p); + starts.push(c.height); + return c.height < -9000 + ? undefined + : Cesium.Cartesian3.fromRadians(c.longitude, c.latitude, 2000); + }), + }, + }; + const loaded = await loadGeoFile( + new File( + [JSON.stringify({ type: 'Point', coordinates: [1, 2] })], + 'surface.geojson', + ), + { viewer }, + ); + assert.ok( + Math.abs(starts.at(-1)) < 1e-6, + `ray started at ${starts.at(-1)}`, + ); + assert.ok( + Math.abs( + Cesium.Cartographic.fromCartesian( + loaded.dataSource.importedBillboards.get(0).position, + ).height - 2000, + ) < 1e-6, + ); + const sampled = starts.length; + viewer.camera.computeViewRectangle = () => undefined; + await loaded.refreshGround(); + assert.equal( + starts.length, + sampled, + 'distant points do not start a detailed mesh sweep', + ); + assert.ok( + Math.abs( + Cesium.Cartographic.fromCartesian( + loaded.dataSource.importedBillboards.get(0).position, + ).height - 2000, + ) < 1e-6, + 'the last valid mesh height survives outside the camera footprint', + ); + const point = loaded.dataSource.importedEntities[0]; + const other = new Cesium.Entity({ id: 'unrelated-aircraft' }); + viewer.trackedEntity = starts.length === 1 ? other : point; + viewer.selectedEntity = point; + loaded.revoke(); + assert.equal(viewer.selectedEntity, undefined); + assert.equal( + viewer.trackedEntity, + starts.length === 1 ? other : undefined, + ); + } + } finally { + Cesium.PinBuilder.prototype.fromColor = previous; + } +}); + +const bytes = (text) => new TextEncoder().encode(text); + +/** A minimal zip writer: stored (0) or deflated (8) entries, optional flags. */ +function zip(files) { + const locals = []; + const centrals = []; + let offset = 0; + for (const { name, data, method = 8, flags = 0, size } of files) { + const nameBytes = bytes(name); + const body = method === 8 ? new Uint8Array(deflateRawSync(data)) : data; + const local = new Uint8Array(30 + nameBytes.length); + const lv = new DataView(local.buffer); + lv.setUint32(0, 0x04034b50, true); + lv.setUint16(6, flags, true); + lv.setUint16(8, method, true); + lv.setUint32(18, body.length, true); + lv.setUint32(22, size ?? data.length, true); + lv.setUint16(26, nameBytes.length, true); + local.set(nameBytes, 30); + const central = new Uint8Array(46 + nameBytes.length); + const cv = new DataView(central.buffer); + cv.setUint32(0, 0x02014b50, true); + cv.setUint16(8, flags, true); + cv.setUint16(10, method, true); + cv.setUint32(20, body.length, true); + cv.setUint32(24, size ?? data.length, true); + cv.setUint16(28, nameBytes.length, true); + cv.setUint32(42, offset, true); + central.set(nameBytes, 46); + locals.push(local, body); + centrals.push(central); + offset += local.length + body.length; + } + const dirSize = centrals.reduce((n, c) => n + c.length, 0); + const eocd = new Uint8Array(22); + const ev = new DataView(eocd.buffer); + ev.setUint32(0, 0x06054b50, true); + ev.setUint16(8, files.length, true); + ev.setUint16(10, files.length, true); + ev.setUint32(12, dirSize, true); + ev.setUint32(16, offset, true); + const parts = [...locals, ...centrals, eocd]; + const out = new Uint8Array(parts.reduce((n, p) => n + p.length, 0)); + let at = 0; + for (const p of parts) { + out.set(p, at); + at += p.length; + } + return out; +} + +test('format comes from the extension, then from the first bytes', () => { + assert.equal(detectGeoFileFormat('trail.GPX'), 'gpx'); + assert.equal(detectGeoFileFormat('farm.kml'), 'kml'); + assert.equal(detectGeoFileFormat('farm.kmz'), 'kmz'); + assert.equal(detectGeoFileFormat('parcels.geojson'), 'geojson'); + assert.equal(detectGeoFileFormat('parcels.json'), 'geojson'); + assert.equal( + detectGeoFileFormat('export', bytes(' {"type":"Feature"}')), + 'geojson', + ); + assert.equal( + detectGeoFileFormat( + 'export', + bytes(''), + ), + 'kml', + ); + assert.equal( + detectGeoFileFormat( + 'export', + bytes('\n'), + ), + 'gpx', + ); + assert.equal( + detectGeoFileFormat('export', new Uint8Array([0x50, 0x4b, 0x03, 0x04, 0])), + 'kmz', + ); + assert.equal(detectGeoFileFormat('notes.txt', bytes('hello')), null); + assert.equal( + detectGeoFileFormat('image.png', new Uint8Array([0x89, 0x50, 0x4e, 0x47])), + null, + ); +}); + +test('display names are printable and bounded', () => { + assert.equal(importDisplayName(' hike\u0000.gpx '), 'hike.gpx'); + assert.equal(importDisplayName(''), 'Imported file'); + const long = importDisplayName(`${'x'.repeat(80)}.kml`); + assert.equal(long.length, 58); + assert.ok(long.endsWith('…')); +}); + +test('GeoJSON text must be a GeoJSON object', () => { + assert.equal( + parseGeoJsonText('{"type":"FeatureCollection","features":[]}').type, + 'FeatureCollection', + ); + assert.equal( + parseGeoJsonText('{"type":"Point","coordinates":[0,0]}').type, + 'Point', + ); + assert.throws(() => parseGeoJsonText('{nope'), GeoImportError); + assert.throws(() => parseGeoJsonText('{"name":"not geo"}'), /not GeoJSON/); + assert.throws(() => parseGeoJsonText('[1,2]'), /not GeoJSON/); + assert.throws( + () => parseGeoJsonText('{"type":"FeatureCollection"}'), + /no features list/, + ); +}); + +test('a KMZ is read with stored and deflated entries, directories skipped', async () => { + const kml = bytes(''); + const png = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 1, 2, 3]); + const archive = zip([ + { name: 'files/', data: new Uint8Array(0), method: 0 }, + { name: 'doc.kml', data: kml }, + { name: 'files/icon.png', data: png, method: 0 }, + ]); + const entries = await readZipEntries(archive); + assert.deepEqual([...entries.keys()], ['doc.kml', 'files/icon.png']); + assert.deepEqual(entries.get('doc.kml'), kml); + assert.deepEqual(entries.get('files/icon.png'), png); +}); + +test('a KMZ that inflates past the limit is refused', async () => { + const archive = zip([{ name: 'doc.kml', data: new Uint8Array(200_000) }]); + await assert.rejects( + readZipEntries(archive, { maxInflatedBytes: 100_000 }), + /more than the import limit/, + ); +}); + +test('encrypted, damaged and non-zip archives are refused with a reason', async () => { + await assert.rejects( + readZipEntries( + zip([{ name: 'doc.kml', data: bytes(''), flags: 1, method: 0 }]), + ), + /encrypted/, + ); + await assert.rejects( + readZipEntries(bytes('not a zip at all, just text')), + /not a readable zip/, + ); + const truncated = zip([ + { name: 'doc.kml', data: bytes(''), method: 0 }, + ]); + const view = new DataView(truncated.buffer); + view.setUint32(truncated.length - 6, 9999, true); // directory offset past the end + await assert.rejects(readZipEntries(truncated), GeoImportError); + await assert.rejects( + readZipEntries( + zip([{ name: 'doc.kml', data: bytes(''), method: 12 }]), + ), + /compression method/, + ); +}); + +test('the main KML is doc.kml, else the shallowest .kml', () => { + const e = (...names) => new Map(names.map((n) => [n, new Uint8Array(0)])); + assert.equal(kmzMainDocument(e('a/b.kml', 'doc.kml')), 'doc.kml'); + assert.equal( + kmzMainDocument(e('deep/x/z.kml', 'top.kml', 'images/a.png')), + 'top.kml', + ); + assert.equal(kmzMainDocument(e('images/a.png')), null); +}); + +test('KMZ hrefs resolve only to entries inside the archive', () => { + const entries = new Map([ + ['doc.kml', new Uint8Array(0)], + ['files/icon one.png', new Uint8Array(0)], + ['overlay.jpg', new Uint8Array(0)], + ]); + assert.equal( + resolveKmzPath('files/icon%20one.png', 'doc.kml', entries), + 'files/icon one.png', + ); + assert.equal( + resolveKmzPath('./overlay.jpg', 'doc.kml', entries), + 'overlay.jpg', + ); + assert.equal( + resolveKmzPath('../overlay.jpg', 'files/sub.kml', entries), + 'overlay.jpg', + ); + assert.equal( + resolveKmzPath('https://example.com/overlay.jpg', 'doc.kml', entries), + null, + ); + assert.equal( + resolveKmzPath('//example.com/overlay.jpg', 'doc.kml', entries), + null, + ); + assert.equal(resolveKmzPath('file:///etc/passwd', 'doc.kml', entries), null); + assert.equal(resolveKmzPath('missing.png', 'doc.kml', entries), null); + assert.equal(resolveKmzPath('%E0%A4%A', 'doc.kml', entries), null); + assert.equal(kmzImageType('files/a.PNG'), 'image/png'); + assert.equal(kmzImageType('files/a.svg'), null); +}); + +/* ── a tiny element tree with the DOM surface the readers use ─────────── */ +function xmlEl(localName, attrs = {}, kids = []) { + const el = { + nodeType: 1, + localName, + namespaceURI: 'http://www.opengis.net/kml/2.2', + childNodes: [], + getAttribute: (n) => (n in attrs ? String(attrs[n]) : null), + get attributes() { + return Object.entries(attrs); + }, + setAttribute: (n, value) => { + attrs[n] = value; + }, + removeAttribute: (n) => { + delete attrs[n]; + }, + cloneNode() { + return xmlEl( + localName, + { ...attrs }, + text ?? el.childNodes.map((child) => child.cloneNode(true)), + ); + }, + getElementsByTagName(name) { + return xmlDoc(this).getElementsByTagName(name).slice(1); + }, + appendChild(child) { + child.parentNode?.removeChild(child); + child.parentNode = this; + this.childNodes.push(child); + }, + replaceChild(next, previous) { + const index = this.childNodes.indexOf(previous); + assert.ok(index >= 0); + next.parentNode?.removeChild(next); + this.childNodes[index] = next; + next.parentNode = this; + previous.parentNode = null; + }, + removeChild(child) { + this.childNodes.splice(this.childNodes.indexOf(child), 1); + child.parentNode = null; + }, + }; + for (const k of typeof kids === 'string' ? [] : kids) { + k.parentNode = el; + el.childNodes.push(k); + } + let text = typeof kids === 'string' ? kids : null; + Object.defineProperty(el, 'firstChild', { + get: () => el.childNodes[0] || null, + }); + Object.defineProperty(el, 'textContent', { + get: () => text ?? el.childNodes.map((c) => c.textContent).join(''), + set: (value) => { + text = value; + el.childNodes = []; + }, + }); + return el; +} +const xmlDoc = (root) => ({ + documentElement: root, + createElementNS: (_ns, name) => xmlEl(name), + getElementsByTagName(name) { + assert.equal(name, '*'); + const out = []; + const walk = (el) => { + out.push(el); + el.childNodes.forEach(walk); + }; + walk(root); + return out; + }, +}); +const serializeStyle = (el) => + JSON.stringify({ + name: el.localName, + attributes: el.attributes, + text: el.childNodes.length ? null : el.textContent, + children: el.childNodes.map(serializeStyle), + }); + +test('repeated inline styles are shared without replacing their inherited styleUrl', () => { + const style = () => + xmlEl('Style', {}, [xmlEl('IconStyle', {}, [xmlEl('scale', {}, '0.6')])]); + const placemarks = Array.from({ length: 2100 }, () => + xmlEl('Placemark', {}, [xmlEl('styleUrl', {}, '#base'), style()]), + ); + const doc = xmlDoc( + xmlEl('kml', {}, [ + xmlEl('Document', {}, [ + xmlEl('Style', { id: 'gev-import-style-1' }, [xmlEl('LineStyle')]), + ...placemarks, + ]), + ]), + ); + assert.equal(countKmlGeometry(doc).over, 'links'); + assert.equal(deduplicateKmlStyles(doc, { serializeStyle }), 2100); + assert.equal(countKmlGeometry(doc).over, ''); + const styles = doc + .getElementsByTagName('*') + .filter((el) => el.localName === 'Style'); + assert.equal(styles.length, 2); + assert.equal(styles[1].getAttribute('id'), 'gev-import-style-2'); + for (const placemark of placemarks) { + assert.equal(placemark.childNodes[0].textContent, '#base'); + assert.equal(placemark.childNodes[1].localName, 'StyleMap'); + assert.deepEqual( + placemark.childNodes[1].childNodes.map((pair) => + pair.childNodes.map((el) => el.textContent), + ), + [ + ['normal', '#gev-import-style-2'], + ['highlight', '#gev-import-style-2'], + ], + ); + } +}); + +test('identical named styles redirect existing references and Pair styles remain valid', () => { + const style = (id) => + xmlEl('Style', id ? { id } : {}, [ + xmlEl('LineStyle', {}, [xmlEl('width', {}, '3')]), + ]); + const reference = xmlEl('styleUrl', {}, '#b'); + const pair = xmlEl('Pair', {}, [xmlEl('key', {}, 'normal'), style()]); + const doc = xmlDoc( + xmlEl('kml', {}, [ + style('a'), + style('b'), + xmlEl('StyleMap', { id: 'map' }, [pair]), + xmlEl('Placemark', {}, [reference]), + ]), + ); + assert.equal(deduplicateKmlStyles(doc, { serializeStyle }), 3); + assert.equal(doc.documentElement.firstChild.localName, 'Document'); + assert.equal(reference.textContent, '#gev-import-style-1'); + assert.equal(pair.childNodes[1].localName, 'styleUrl'); + assert.equal(pair.childNodes[1].textContent, '#gev-import-style-1'); + assert.equal( + doc.getElementsByTagName('*').filter((el) => el.localName === 'Style') + .length, + 1, + ); +}); + +test('random colors and ambiguous style IDs retain independent evaluations and precedence', () => { + const random = () => + xmlEl('Style', {}, [ + xmlEl('IconStyle', {}, [xmlEl('colorMode', {}, 'random')]), + ]); + const named = (width) => + xmlEl('Style', { id: 'collision' }, [ + xmlEl('LineStyle', {}, [xmlEl('width', {}, width)]), + ]); + const original = [random(), random(), named('1'), named('1'), named('2')]; + const doc = xmlDoc(xmlEl('kml', {}, original)); + assert.equal(deduplicateKmlStyles(doc, { serializeStyle }), 0); + assert.deepEqual(doc.documentElement.childNodes, original); +}); + +test('distinct style definitions and named StyleMaps still have a pre-loader budget', () => { + const doc = xmlDoc( + xmlEl( + 'kml', + {}, + ['1', '2', '3'].map((width) => + xmlEl('Style', {}, [ + xmlEl('LineStyle', {}, [xmlEl('width', {}, width)]), + ]), + ), + ), + ); + assert.throws( + () => deduplicateKmlStyles(doc, { serializeStyle, maxStyles: 2 }), + GeoImportError, + ); + const maps = xmlDoc( + xmlEl('kml', {}, [ + xmlEl('StyleMap', { id: 'a' }), + xmlEl('StyleMap', { id: 'b' }), + ]), + ); + assert.equal(countKmlGeometry(maps, { maxLinks: 1 }).over, 'links'); +}); + +test('KML extended metadata is removed before the document reaches the loader', () => { + const point = xmlEl('Point', {}, [xmlEl('coordinates', {}, '1,2,0')]); + const metadata = xmlEl('ExtendedData', {}, [ + xmlEl('Data', { name: 'note' }, [xmlEl('value', {}, 'ordinary metadata')]), + ]); + const placemark = xmlEl('Placemark', {}, [metadata, point]); + const doc = xmlDoc(xmlEl('kml', {}, [placemark])); + assert.equal(sanitizeKmlDocument(doc).removed, 1); + assert.deepEqual(placemark.childNodes, [point]); + assert.equal(metadata.parentNode, null); +}); +test('blocked icon sources fall back to local pins without changing intentional empty icons', () => { + const blocked = xmlEl('Icon', {}, [ + xmlEl('href', {}, 'https://example.invalid/icon.png'), + ]); + const blank = xmlEl('Icon'); + const embedded = xmlEl('Icon', {}, [ + xmlEl('href', {}, 'data:image/png;base64,AAAA'), + ]); + const packed = xmlEl('Icon', {}, [xmlEl('href', {}, 'packed.png')]); + const iconStyle = xmlEl('IconStyle', {}, [ + xmlEl('scale', {}, '0.3'), + blocked, + ]); + const doc = xmlDoc( + xmlEl('kml', {}, [ + xmlEl('Style', {}, [iconStyle]), + ...[blank, embedded, packed].map((icon) => + xmlEl('Style', {}, [xmlEl('IconStyle', {}, [icon])]), + ), + ]), + ); + assert.equal( + sanitizeKmlDocument(doc, (href) => + href === 'packed.png' ? 'blob:local-image' : null, + ).removed, + 1, + ); + assert.equal(blocked.parentNode, null); + assert.equal(iconStyle.childNodes[0].textContent, '0.3'); + assert.equal(blank.parentNode.localName, 'IconStyle'); + assert.equal( + embedded.childNodes[0].textContent, + 'data:image/png;base64,AAAA', + ); + assert.equal(packed.childNodes[0].textContent, 'blob:local-image'); +}); + +const trkpt = (lat, lon, time) => + xmlEl('trkpt', { lat, lon }, time ? [xmlEl('time', {}, time)] : []); + +test('GPX keeps each track segment as its own line, names as text, no times', () => { + const doc = xmlDoc( + xmlEl('gpx', {}, [ + xmlEl('wpt', { lat: 37.77, lon: -122.49 }, [xmlEl('name', {}, 'Camp')]), + xmlEl('wpt', { lat: 99, lon: 0 }), // out of range: skipped + xmlEl('rte', {}, [ + xmlEl('name', {}, 'Route'), + xmlEl('rtept', { lat: 1, lon: 2 }), + xmlEl('rtept', { lat: 3, lon: 4 }), + ]), + xmlEl('trk', {}, [ + xmlEl('name', {}, 'Morning walk'), + xmlEl('trkseg', {}, [ + trkpt(10, 20, '2024-05-01T10:00:00Z'), + trkpt(10.1, 20.1, '2024-05-01T10:01:00Z'), + ]), + // A pause in the recording: a new segment, not a straight jump. + xmlEl('trkseg', {}, [ + trkpt(11, 21, '2024-05-01T11:00:00Z'), + trkpt(11.1, 21.1, '2024-05-01T11:01:00Z'), + trkpt(11.2, 21.2, '2024-05-01T11:02:00Z'), + ]), + xmlEl('trkseg', {}, [trkpt(12, 22)]), // one point: not a line + ]), + ]), + ); + const fc = gpxToGeoJson(doc); + assert.deepEqual( + fc.features.map((f) => [f.properties.name, f.geometry.type]), + [ + ['Camp', 'Point'], + ['Route', 'LineString'], + ['Morning walk', 'MultiLineString'], + ], + ); + assert.deepEqual(fc.features[2].geometry.coordinates, [ + [ + [20, 10], + [20.1, 10.1], + ], + [ + [21, 11], + [21.1, 11.1], + [21.2, 11.2], + ], + ]); + assert.doesNotMatch(JSON.stringify(fc), /2024|time/); + assert.throws(() => gpxToGeoJson(xmlDoc(xmlEl('kml'))), GeoImportError); +}); + +test('GPX past the point limit is refused while reading, before Cesium', () => { + const seg = xmlEl( + 'trkseg', + {}, + Array.from({ length: 30 }, (_, i) => trkpt(1 + i / 100, 2)), + ); + const doc = xmlDoc(xmlEl('gpx', {}, [xmlEl('trk', {}, [seg])])); + assert.throws( + () => gpxToGeoJson(doc, { maxPositions: 20 }), + (e) => e instanceof GeoImportError && /more than .* points/.test(e.message), + ); + assert.throws( + () => + gpxToGeoJson( + xmlDoc( + xmlEl('gpx', {}, [ + xmlEl('wpt', { lat: 1, lon: 1 }), + xmlEl('wpt', { lat: 2, lon: 2 }), + ]), + ), + { maxFeatures: 1 }, + ), + /more than .* features/, + ); +}); + +test('GeoJSON features and vertices are counted before loading, without recursion', () => { + const line = Array.from({ length: 50 }, (_, i) => [i / 10, 1]); + const fc = { + type: 'FeatureCollection', + features: [ + { type: 'Feature', geometry: { type: 'LineString', coordinates: line } }, + { + type: 'Feature', + geometry: { + type: 'GeometryCollection', + geometries: [ + { type: 'Point', coordinates: [1, 2] }, + { + type: 'Polygon', + coordinates: [ + [ + [0, 0], + [1, 0], + [1, 1], + [0, 0], + ], + ], + }, + ], + }, + }, + { type: 'Feature', geometry: null }, + ], + }; + assert.deepEqual(countGeoJsonGeometry(fc), { + features: 3, + positions: 55, + entities: 4, + over: '', + }); + assert.equal( + countGeoJsonGeometry(fc, { maxPositions: 54 }).over, + 'positions', + ); + assert.equal(countGeoJsonGeometry(fc, { maxFeatures: 2 }).over, 'features'); + assert.deepEqual( + countGeoJsonGeometry({ type: 'Point', coordinates: [1, 2] }), + { + features: 1, + positions: 1, + entities: 1, + over: '', + }, + ); + // A very long line is counted iteratively and stops at the limit. + const huge = { + type: 'LineString', + coordinates: Array.from({ length: MAX_IMPORT_POSITIONS + 5 }, () => [0, 0]), + }; + assert.equal(countGeoJsonGeometry(huge).over, 'positions'); +}); + +test('multipart GeoJSON is refused before loading when it expands past the entity cap', () => { + const parts = Array.from({ length: MAX_IMPORT_ENTITIES + 1 }, () => [1, 2]); + for (const geometry of [ + { type: 'MultiPoint', coordinates: parts }, + { + type: 'GeometryCollection', + geometries: parts.map((coordinates) => ({ type: 'Point', coordinates })), + }, + { type: 'MultiLineString', coordinates: parts.map((p) => [p, [2, 3]]) }, + ]) { + const size = countGeoJsonGeometry({ type: 'Feature', geometry }); + assert.equal(size.over, 'entities'); + } +}); + +test('GPX points with missing or blank coordinates are skipped rather than placed at zero', () => { + const doc = xmlDoc( + xmlEl('gpx', {}, [ + xmlEl('wpt', { lon: 2 }), + xmlEl('wpt', { lat: 1 }), + xmlEl('wpt', { lat: '', lon: 2 }), + xmlEl('wpt', { lat: 1, lon: ' ' }), + xmlEl('wpt', { lat: 0, lon: 0 }), + ]), + ); + assert.equal(gpxToGeoJson(doc).features.length, 1); + assert.deepEqual(gpxToGeoJson(doc).features[0].geometry.coordinates, [0, 0]); +}); + +test('KML placemarks and coordinate tuples are counted before loading', () => { + const coords = (n) => + xmlEl( + 'coordinates', + {}, + Array.from({ length: n }, (_, i) => `${i},1,0`).join('\n '), + ); + const doc = xmlDoc( + xmlEl('kml', {}, [ + xmlEl('Placemark', {}, [xmlEl('LineString', {}, [coords(40)])]), + xmlEl('Placemark', {}, [ + xmlEl('Track', {}, [ + xmlEl('coord', {}, '1 2 0'), + xmlEl('coord', {}, '1 3 0'), + ]), + ]), + ]), + ); + assert.deepEqual(countKmlGeometry(doc), { + features: 2, + entities: 2, + positions: 42, + links: 0, + over: '', + }); + assert.equal(countKmlGeometry(doc, { maxPositions: 41 }).over, 'positions'); + assert.equal(countKmlGeometry(doc, { maxFeatures: 1 }).over, 'features'); +}); + +test('KML vertex counting recognizes the Unicode whitespace Cesium accepts', () => { + for (const separator of ['\u00a0', '\u2003', '\ufeff']) { + const doc = xmlDoc( + xmlEl('kml', {}, [ + xmlEl('Placemark', {}, [ + xmlEl('LineString', {}, [ + xmlEl('coordinates', {}, ['1,2', '3,4', '5,6'].join(separator)), + ]), + ]), + ]), + ); + assert.equal(countKmlGeometry(doc).positions, 3); + assert.equal(countKmlGeometry(doc, { maxPositions: 2 }).over, 'positions'); + } +}); + +test('KML multipart geometries count every Cesium entity before loading', () => { + const doc = xmlDoc( + xmlEl('kml', {}, [ + xmlEl('Placemark', {}, [ + xmlEl('MultiGeometry', {}, [ + xmlEl('Point'), + xmlEl('MultiGeometry', {}, [xmlEl('Point'), xmlEl('LineString')]), + xmlEl('MultiTrack', {}, [xmlEl('Track'), xmlEl('Track')]), + ]), + ]), + xmlEl('GroundOverlay'), + ]), + ); + assert.equal(countKmlGeometry(doc).entities, 9); + assert.equal(countKmlGeometry(doc, { maxEntities: 8 }).over, 'entities'); + const oversized = xmlDoc( + xmlEl('kml', {}, [ + xmlEl('Placemark', {}, [ + xmlEl( + 'MultiGeometry', + {}, + Array.from({ length: MAX_IMPORT_ENTITIES }, () => xmlEl('Point')), + ), + ]), + ]), + ); + assert.equal(countKmlGeometry(oversized).over, 'entities'); +}); + +test('KML folders contribute to the pre-loader entity budget', () => { + const doc = xmlDoc( + xmlEl('kml', {}, [ + xmlEl('Folder', {}, [xmlEl('Folder', {}, [xmlEl('Placemark')])]), + ]), + ); + assert.equal(countKmlGeometry(doc).entities, 3); + assert.equal(countKmlGeometry(doc, { maxEntities: 2 }).over, 'entities'); +}); + +test('KML style definitions and distinct icon links have their own limit', () => { + const style = () => + xmlEl('Style', {}, [ + xmlEl('IconStyle', {}, [xmlEl('Icon', {}, [xmlEl('href', {}, 'a.png')])]), + ]); + const doc = xmlDoc(xmlEl('kml', {}, [style(), style(), style()])); + assert.equal(countKmlGeometry(doc).links, 4); + assert.equal(countKmlGeometry(doc, { maxLinks: 3 }).over, 'links'); + const distinct = xmlDoc( + xmlEl( + 'kml', + {}, + ['a.png', 'b.png', 'c.png'].map((href) => xmlEl('href', {}, href)), + ), + ); + assert.equal(countKmlGeometry(distinct, { maxLinks: 2 }).over, 'links'); +}); + +test('a KMZ image gets one blob URL however many hrefs name it', () => { + const entries = new Map([ + ['doc.kml', new Uint8Array(0)], + ['a.png', new Uint8Array(8)], + ['b.png', new Uint8Array(8)], + ]); + const made = []; + const kept = []; + const resolve = kmzImageResolver( + entries, + 'doc.kml', + (url) => kept.push(url), + (blob) => { + made.push(blob); + return `blob:test/${made.length}`; + }, + ); + for (let i = 0; i < 1000; i++) assert.equal(resolve('a.png'), 'blob:test/1'); + assert.equal(resolve('./b.png'), 'blob:test/2'); + assert.equal(resolve('a.png'), 'blob:test/1'); + assert.equal(resolve('https://example.com/a.png'), null); + assert.equal(resolve('doc.kml'), null); + assert.equal(made.length, 2); + assert.deepEqual(kept, ['blob:test/1', 'blob:test/2']); +}); + +test('imported entities keep only a short plain-text name: no description, label or properties', () => { + const ds = new Cesium.CustomDataSource('t'); + const e = ds.entities.add({ + name: 'Trail\u0007 head\n' + 'x'.repeat(300), + description: '', + label: { text: 'Trail head' }, + properties: { note: '' }, + position: Cesium.Cartesian3.fromDegrees(1, 2), + }); + e.kml = { extendedData: { note: { value: '