perf(import): reuse generated pin textures and trim unused processing

CHANGELOG.md

@@ -176,7 +176,10 @@
   are shared before the style limit is applied, allowing large files with many
   copies of the same style to load without changing their graphics. Blocked
   remote icons now fall back to local pins instead of leaving point markers
-  invisible, including USGS earthquake exports.
+  invisible, including USGS earthquake exports. Generated pins share one image
+  in the GPU atlas; unused time/view metadata is pruned before loading, and
+  obsolete track resampling code is removed. KML vertex counting follows
+  the loader's whitespace rules.
 
 ## [0.2.1] — 2026-10-02 — God's Eye View through Your Agent (with security fixes and other features)
 

docs/CURRENT-STATE.md

@@ -4326,10 +4326,10 @@ are omitted rather than framing the wrong part of the globe.
 
 - DISPLAY ▸ **Import** puts GeoJSON (`.geojson`/`.json`), KML, KMZ and GPX files on the globe. Files come in through the Import button (a hidden `<input type=file multiple>`) or by dropping them onto the globe; the format is taken from the extension, then from the first bytes. Each file becomes its own Cesium data source, clamped to the ground, with a row to show/hide it, fly to it, or remove it. GeoJSON and GPX are drawn in the row's color from the whiteboard palette (simplestyle properties in GeoJSON are dropped); KML and KMZ keep the styles written in them, and their row shows a hatched swatch instead of a color. GPX is read by `gpxToGeoJson` and loaded as GeoJSON: waypoints are points, each route a line, and each track one `MultiLineString` with a line per `<trkseg>`, so a pause in a recording stays a gap. Imported files last for the page session: a reload starts empty, and nothing is uploaded or stored.
 - **Navigation.** The row's Fly-to is an immediate navigation through the application shell (`runImmediateNavigation('file', …)`): Cockpit refuses it, and the current follow/track owner is released first. The fly after an import is a deferred navigation: `beginDeferredLocationNavigation()` stamps it when the pick or drop is submitted, before queued batches wait and `reassertDeferredLocationNavigation()` re-checks it just before flying, so a camera gesture, search, Cockpit or any newer navigation while files load wins and the import does not fly. Fly-to frames the features' longitudes and latitudes at the surface (thinned to 5,000 points by index, never spread into a call) and lets the camera ground guard lift the eye once the terrain has loaded (`src/ui/flyToDataSource.js`); `viewer.flyTo` is not used, because before terrain streams in it frames ground-clamped points kilometers below the surface.
-- **Clock.** An import never changes the app's shared clock. The viewer adopts the clock of every data source added to it, so each imported data source reports no clock (`detachImportedTime`); every entity is available for all time, and KML `gx:Track` / `gx:MultiTrack` geometry is converted to static lines before Cesium loads it, preserving the recorded vertices and segment gaps without expanding them through time sampling. GPX times are not read at all.
+- **Clock.** An import never changes the app's shared clock. The viewer adopts the clock of every data source added to it, so each imported data source reports no clock (`detachImportedTime`); every entity is available for all time. Unused KML `TimeSpan`/`TimeStamp` and `LookAt`/`Camera` metadata is removed before loader construction, and KML `gx:Track` / `gx:MultiTrack` geometry is converted to static lines before Cesium loads it, preserving the recorded vertices and segment gaps without expanding them through time sampling. GPX times are not read at all.
 - Runtime entry: `createApplicationTools` (`src/app/tools.js`) calls `initGeoFileImport({ viewer, navigation: styleManager })` (`src/ui/geoFileImport.js`) after the draw tool and defers its `destroy()`, which removes every imported data source, revokes every blob URL, and gives back the drop listeners on the viewer container and the `window.__gevGeoImport` handle (`importFiles`, `list`, `setVisible`, `flyTo`, `remove`, `diagnostics`). Parsing lives in `src/data/geoFileImport.js`.
 - **An imported file cannot make the page fetch anything.** Before `KmlDataSource` reads a KML/KMZ document, `sanitizeKmlDocument` removes `NetworkLink`, `NetworkLinkControl`, `ScreenOverlay`, `PhotoOverlay`, `Model` and `Tour`, every `href` that is not an image packed in the same KMZ or a `data:image/` URL (a blocked `IconStyle` icon definition is removed so the marker falls back to Cesium's default pin, drawn locally; an intentionally empty icon remains empty, and a `GroundOverlay` left without an image is dropped), every `styleUrl` that is not a local `#id`, and placemark `description`/`BalloonStyle`/`Snippet` HTML and `ExtendedData` before metadata processing. GeoJSON `marker-symbol` (a maki icon asset) is dropped. KMZ images are served from blob URLs, one per packed image however many hrefs name it (`kmzImageResolver`). The hint says when something was left out. `scripts/qa-geo-import.mjs` imports files that name a trap host in each of those places and fails on any request that leaves the app's own origin.
-- **Imported metadata stays inert.** After loading, `scrubImportedEntities` removes every entity's description, native label, properties and KML extended data, and keeps the name only as plain text without control characters, at most 120 characters. GeoJSON is loaded with a `describe` that builds no HTML. The file name is shown with `textContent` only.
+- **Imported metadata stays inert.** After loading, `scrubImportedEntities` removes every entity's description, native label, properties and KML extended data, and keeps the name only as plain text without control characters, at most 120 characters. GeoJSON is loaded with a `describe` that builds no HTML. The file name is shown with `textContent` only. Entity normalization batches collection notifications and converts constant generated pin canvases to one shared data URL per image, avoiding a duplicate GPU texture entry per marker while preserving authored image URLs and dynamic image properties.
 - **KMZ** is read by a small zip reader (`readZipEntries`: stored and deflated entries through the platform `DecompressionStream`; no encryption, no ZIP64). The main document is `doc.kml`, else the shallowest `.kml`; relative hrefs resolve only to entries inside the archive.
 - **Bounds.** A file may be at most 50 MiB (`MAX_IMPORT_BYTES`), a KMZ may inflate to at most 120 MiB in total (`MAX_KMZ_INFLATED_BYTES`, the zip-bomb stop), one file may create at most 20,000 Cesium entities, including multipart geometries and KML parent containers (`MAX_IMPORT_ENTITIES`) and 250,000 vertices (`MAX_IMPORT_POSITIONS`), and a KML/KMZ at most 2,000 normalized style definitions (including named StyleMaps) and distinct surviving image links (`MAX_KML_LINKS`). Identical KML styles are shared after metadata and remote-reference cleanup, preserving inline overrides and local style references; repeated copies do not exhaust the style budget. Folder containers count toward the entity budget. Files of 20–30 MB fit within the byte limit; drawing complexity is bounded separately. Geometry is counted before Cesium builds anything (`countGeoJsonGeometry`, `countKmlGeometry`, and while reading GPX), and the entity count is checked again after loading. At most 12 files are on the globe at once (`MAX_IMPORTED_FILES`): imports run one batch at a time, so two picks or drops in quick succession cannot pass the cap; files over it are not read. A refused file names its reason in the hint and does not stop the rest of the batch; a file with nothing to draw is refused.
 - **Keyboard.** Hide, show and remove rebuild the list without losing focus: the same control on the same row keeps it, a removed row hands it to the next row (else the Import button).

scripts/qa-geo-import-large.mjs

@@ -102,6 +102,13 @@ try {
     });
     window.__qaStyleSnapshot = (source) => {
       const now = viewer.clock.currentTime;
+      const images = new Map();
+      const imageValue = (p) => {
+        const value = p?.getValue(now);
+        if (value?.tagName !== 'CANVAS') return value;
+        if (!images.has(value)) images.set(value, value.toDataURL());
+        return images.get(value);
+      };
       const color = (p) => p?.getValue(now)?.toCssColorString();
       const positions = (p) => p?.getValue(now)?.map((v) => [v.x, v.y, v.z]);
       return source.entities.values
@@ -110,7 +117,7 @@ try {
           point: e.position?.getValue(now),
           scale: e.billboard?.scale?.getValue(now),
           rotation: e.billboard?.rotation?.getValue(now),
-          image: e.billboard?.image?.getValue(now),
+          image: imageValue(e.billboard?.image),
           color: color(e.billboard?.color),
           width: e.polyline?.width?.getValue(now),
           lineColor: color(e.polyline?.material?.color),

src/data/geoFileImport.js

@@ -68,14 +68,22 @@ const KMZ_IMAGE_TYPES = Object.freeze({
   webp: 'image/webp',
   bmp: 'image/bmp',
 });
-// KML elements that fetch, embed or execute something outside the file.
-const KML_REMOVED_ELEMENTS = Object.freeze([
+// External content and metadata the session-local import does not use.
+const KML_REMOVED_ELEMENTS = new Set([
   'NetworkLink',
   'NetworkLinkControl',
   'ScreenOverlay',
   'PhotoOverlay',
   'Model',
   'Tour',
+  'description',
+  'BalloonStyle',
+  'Snippet',
+  'ExtendedData',
+  'TimeSpan',
+  'TimeStamp',
+  'LookAt',
+  'Camera',
 ]);
 
 /** An error whose message is meant for the person who chose the file. */
@@ -222,7 +230,7 @@ export function countGeoJsonGeometry(
 }
 
 /** The refusal for a file over a size limit, in the person's terms. */
-function overLimitError({ over, features, positions }) {
+function overLimitError({ over, features }) {
   if (over === 'entities')
     return new GeoImportError(
       `That file creates more than ${MAX_IMPORT_ENTITIES.toLocaleString()} map objects (the limit).`,
@@ -486,8 +494,11 @@ export function sanitizeKmlDocument(doc, resolveLocal = () => null) {
       removed += 1;
     }
   };
-  for (const name of KML_REMOVED_ELEMENTS)
-    elementsNamed(doc, name).forEach(drop);
+  // One snapshot and pass, rather than rescanning the whole XML tree for
+  // every discarded element type. Strip unused metadata before the loader
+  // constructs descriptions, availability intervals and camera objects.
+  for (const el of [...doc.getElementsByTagName('*')])
+    if (KML_REMOVED_ELEMENTS.has(el.localName)) drop(el);
   for (const href of elementsNamed(doc, 'href')) {
     const value = (href.textContent || '').trim();
     const local = resolveLocal(value);
@@ -511,10 +522,6 @@ export function sanitizeKmlDocument(doc, resolveLocal = () => null) {
   for (const styleUrl of elementsNamed(doc, 'styleUrl')) {
     if (!(styleUrl.textContent || '').trim().startsWith('#')) drop(styleUrl);
   }
-  // Placemark balloons are HTML; the app shows no info box, and keeping their
-  // <img>/<iframe> sources out means no future surface can load them either.
-  for (const name of ['description', 'BalloonStyle', 'Snippet', 'ExtendedData'])
-    elementsNamed(doc, name).forEach(drop);
   // Draw recorded KML paths as their original static vertices. Letting the
   // loader build time-dynamic tracks would expand each into thousands of
   // interpolated points later and exceed the checked geometry budget.
@@ -735,7 +742,10 @@ export function countKmlGeometry(
       // a large string into a large array.
       let inTuple = false;
       for (let i = 0; i < text.length; i++) {
-        const space = text.charCodeAt(i) <= 32;
+        const code = text.charCodeAt(i);
+        // Match Cesium's whitespace splitting, keeping the common ASCII
+        // path cheap and avoiding a large token array during preflight.
+        const space = code <= 32 || (code > 127 && /\s/.test(text[i]));
         if (!space && !inTuple) {
           positions += 1;
           if (positions > maxPositions) return result('positions');
@@ -854,47 +864,61 @@ export function importFeatureName(value) {
 /**
  * Make every entity of an imported data source inert: no description,
  * balloon, extended data or properties, no native label, and a name that is
- * short plain text. Returns how many entities were touched.
+ * short plain text. Generated pin canvases share a data URL so Cesium uploads
+ * each image once, rather than allocating a texture entry for every entity.
+ * Returns how many entities were touched.
  * @param {{entities: {values: object[]}}} dataSource
  * @returns {number}
  */
 export function scrubImportedEntities(dataSource) {
-  const list = dataSource?.entities?.values || [];
-  for (let i = 0; i < list.length; i++) {
-    const entity = list[i];
-    let name;
-    try {
-      name = importFeatureName(
-        typeof entity.name === 'string' ? entity.name : undefined,
-      );
-    } catch {
-      name = undefined;
+  const collection = dataSource?.entities;
+  const list = collection?.values || [];
+  const images = new Map();
+  collection?.suspendEvents?.();
+  try {
+    for (let i = 0; i < list.length; i++) {
+      const entity = list[i];
+      let name;
+      try {
+        name = importFeatureName(
+          typeof entity.name === 'string' ? entity.name : undefined,
+        );
+      } catch {
+        name = undefined;
+      }
+      entity.name = name;
+      entity.description = undefined;
+      entity.label = undefined;
+      entity.properties = undefined;
+      if (entity.kml && typeof entity.kml === 'object') entity.kml = undefined;
+      const property = entity.billboard?.image;
+      if (property?.isConstant === false) continue;
+      const image = property?.getValue?.();
+      if (image?.tagName === 'CANVAS') {
+        let url = images.get(image);
+        if (!url) {
+          url = image.toDataURL();
+          images.set(image, url);
+        }
+        entity.billboard.image = url;
+      }
     }
-    entity.name = name;
-    entity.description = undefined;
-    entity.label = undefined;
-    entity.properties = undefined;
-    if (entity.kml && typeof entity.kml === 'object') entity.kml = undefined;
+  } finally {
+    collection?.resumeEvents?.();
   }
   return list.length;
 }
 
-const TRACK_SAMPLES = 2000;
-
 /**
  * Take time out of an imported data source so adding it cannot move the app's
  * shared clock: the data source reports no clock (the viewer adopts the clock
  * of every added source that has one), every entity is available for all
- * time, and a time-dynamic position (a KML gx:Track) is drawn as the line it
- * travels instead of a moving marker.
+ * time. KML tracks have already become their original static vertices in
+ * `sanitizeKmlDocument`, so no sampled-position fallback is needed here.
  * @param {object} dataSource
- * @param {{Cesium?: object, color?: object}} [options]
  * @returns {object} The same data source.
  */
-export function detachImportedTime(
-  dataSource,
-  { Cesium: C = Cesium, color } = {},
-) {
+export function detachImportedTime(dataSource) {
   Object.defineProperty(dataSource, 'clock', {
     configurable: true,
     enumerable: false,
@@ -904,42 +928,6 @@ export function detachImportedTime(
   const list = dataSource?.entities?.values || [];
   for (let i = 0; i < list.length; i++) {
     const entity = list[i];
-    const span = entity.availability;
-    const position = entity.position;
-    if (position && position.isConstant === false) {
-      const start = span?.start;
-      const stop = span?.stop;
-      const line = [];
-      if (start && stop && C.JulianDate.lessThan(start, stop)) {
-        const seconds = C.JulianDate.secondsDifference(stop, start);
-        for (let n = 0; n <= TRACK_SAMPLES; n++) {
-          const t = C.JulianDate.addSeconds(
-            start,
-            (seconds * n) / TRACK_SAMPLES,
-            new C.JulianDate(),
-          );
-          const at = position.getValue(t);
-          if (at) line.push(at);
-        }
-      }
-      entity.path = undefined;
-      entity.model = undefined;
-      if (line.length >= 2 && !entity.polyline) {
-        entity.billboard = undefined;
-        entity.point = undefined;
-        entity.position = undefined;
-        entity.polyline = new C.PolylineGraphics({
-          positions: line,
-          clampToGround: true,
-          width: 3,
-          material: color || C.Color.WHITE,
-        });
-      } else {
-        entity.position = line.length
-          ? new C.ConstantPositionProperty(line[0])
-          : undefined;
-      }
-    }
     entity.availability = undefined;
   }
   return dataSource;
@@ -1050,8 +1038,8 @@ export async function loadGeoFile(file, { viewer, color = '#39d0ff' }) {
     throw new GeoImportError('That file has nothing to draw.');
   }
   scrubImportedEntities(dataSource);
-  // Only KML has time-dynamic positions (GPX is read without times); its
-  // track line is drawn in KML's default white, like its other unstyled lines.
+  // Keep the clock invariant explicit even though unused KML dates have
+  // already been stripped and all supported geometries are static.
   detachImportedTime(dataSource);
   // The display name travels beside the data source (the file name, not
   // anything the document says about itself).

src/data/geoFileImport.test.mjs

@@ -657,6 +657,22 @@ test('KML placemarks and coordinate tuples are counted before loading', () => {
   assert.equal(countKmlGeometry(doc, { maxFeatures: 1 }).over, 'features');
 });
 
+test('KML vertex counting recognizes the Unicode whitespace Cesium accepts', () => {
+  for (const separator of ['\u00a0', '\u2003', '\ufeff']) {
+    const doc = xmlDoc(
+      xmlEl('kml', {}, [
+        xmlEl('Placemark', {}, [
+          xmlEl('LineString', {}, [
+            xmlEl('coordinates', {}, ['1,2', '3,4', '5,6'].join(separator)),
+          ]),
+        ]),
+      ]),
+    );
+    assert.equal(countKmlGeometry(doc).positions, 3);
+    assert.equal(countKmlGeometry(doc, { maxPositions: 2 }).over, 'positions');
+  }
+});
+
 test('KML multipart geometries count every Cesium entity before loading', () => {
   const doc = xmlDoc(
     xmlEl('kml', {}, [
@@ -786,6 +802,67 @@ test('imported entities keep only a short plain-text name: no description, label
   assert.equal(importFeatureName('   '), undefined);
 });
 
+test('generated pin canvases are encoded once and shared without changing authored image URLs', () => {
+  const ds = new Cesium.CustomDataSource('pins');
+  let encoded = 0;
+  const canvas = {
+    tagName: 'CANVAS',
+    toDataURL: () => {
+      encoded++;
+      return 'data:image/png;base64,shared-pin';
+    },
+  };
+  for (let i = 0; i < 1000; i++)
+    ds.entities.add({
+      billboard: { image: canvas },
+      label: { text: 'discard' },
+      properties: { discard: true },
+    });
+  const authored = ds.entities.add({
+    billboard: { image: 'blob:packed-image' },
+  });
+  const dynamicImage = new Cesium.CallbackProperty(() => canvas, false);
+  const dynamic = ds.entities.add({ billboard: { image: dynamicImage } });
+  let notifications = 0;
+  ds.entities.collectionChanged.addEventListener(() => notifications++);
+  scrubImportedEntities(ds);
+  assert.equal(encoded, 1);
+  assert.equal(
+    notifications,
+    1,
+    'normalization publishes one complete collection change',
+  );
+  for (const e of ds.entities.values.slice(0, 1000))
+    assert.equal(
+      e.billboard.image.getValue(),
+      'data:image/png;base64,shared-pin',
+    );
+  assert.equal(authored.billboard.image.getValue(), 'blob:packed-image');
+  assert.equal(
+    dynamic.billboard.image,
+    dynamicImage,
+    'time-dependent image properties stay intact',
+  );
+});
+
+test('unused KML time and camera metadata is removed before loading without changing geometry', () => {
+  const coordinates = xmlEl('coordinates', {}, '1,2,0 3,4,0');
+  const placemark = xmlEl('Placemark', {}, [
+    xmlEl('TimeSpan'),
+    xmlEl('TimeStamp'),
+    xmlEl('LookAt'),
+    xmlEl('Camera'),
+    xmlEl('LineString', {}, [coordinates]),
+  ]);
+  const doc = xmlDoc(xmlEl('kml', {}, [placemark]));
+  sanitizeKmlDocument(doc);
+  assert.deepEqual(
+    placemark.childNodes.map((el) => el.localName),
+    ['LineString'],
+  );
+  assert.equal(coordinates.textContent, '1,2,0 3,4,0');
+});
+
 test('a dated import exposes no clock and draws for all time', () => {
   // A data source with its own clock, as KmlDataSource has for a dated file.
   class DatedSource extends Cesium.CustomDataSource {
@@ -800,15 +877,15 @@ test('a dated import exposes no clock and draws for all time', () => {
   const span = new Cesium.TimeIntervalCollection([
     new Cesium.TimeInterval({ start, stop }),
   ]);
-  const track = new Cesium.SampledPositionProperty();
-  track.addSample(start, Cesium.Cartesian3.fromDegrees(10, 10));
-  track.addSample(stop, Cesium.Cartesian3.fromDegrees(10.1, 10.1));
-  const mover = ds.entities.add({
+  // KML tracks become their original static vertices before Cesium loading.
+  const track = ds.entities.add({
     availability: span,
-    position: track,
-    path: {},
-    billboard: {},
+    polyline: {
+      positions: Cesium.Cartesian3.fromDegreesArray([10, 10, 10.1, 10.1]),
+      clampToGround: true,
+    },
   });
+  const recorded = track.polyline.positions;
   const pin = ds.entities.add({
     availability: span,
     position: Cesium.Cartesian3.fromDegrees(1, 1),
@@ -821,12 +898,8 @@ test('a dated import exposes no clock and draws for all time', () => {
   assert.equal(ds.clock, undefined);
   assert.equal(pin.availability, undefined);
   assert.ok(pin.isAvailable(Cesium.JulianDate.now()));
-  // The moving marker became the line it travels, visible now.
-  assert.equal(mover.availability, undefined);
-  assert.equal(mover.position, undefined);
-  assert.equal(mover.path, undefined);
-  assert.equal(mover.billboard, undefined);
-  const line = mover.polyline.positions.getValue(Cesium.JulianDate.now());
-  assert.ok(line.length > 100);
-  assert.ok(mover.polyline.clampToGround.getValue());
+  assert.equal(track.availability, undefined);
+  assert.equal(track.polyline.positions, recorded);
+  assert.equal(recorded.getValue(Cesium.JulianDate.now()).length, 2);
+  assert.ok(track.polyline.clampToGround.getValue());
 });

src/ui/flyToDataSource.js

@@ -1,5 +1,5 @@
 /**
- * Fly to everything in a data source (an imported file, a live feed) without
+ * Fly to everything in an imported file without
  * trusting Cesium's own framing of ground-clamped points: before the
  * destination's terrain has streamed in, a clamped billboard reports a
  * bounding sphere kilometers below the surface, and `viewer.flyTo` follows it
@@ -10,11 +10,13 @@
 import * as Cesium from 'cesium';
 import { guardCameraAboveGround } from '../cameraGroundGuard.js';
 
-const MAX_POINTS = 5000;
+const SAMPLE_TARGET_POINTS = 5000;
 
 /**
  * Longitude/latitude pairs of every entity's position, line and polygon,
- * thinned to at most `MAX_POINTS`. Long tracks are walked by index, never
+ * sampling long paths toward `SAMPLE_TARGET_POINTS` while retaining each
+ * geometry's endpoints. Separate point features stay in the footprint so
+ * distant outliers are not lost. Long tracks are walked by index, never
  * spread into a call, so a file with hundreds of thousands of vertices cannot
  * overflow the stack; only the kept points are converted.
  */
@@ -38,7 +40,7 @@ export function dataSourcePoints(dataSource, time = Cesium.JulianDate.now()) {
       total += ring.length;
     }
   }
-  const stride = Math.max(1, Math.ceil(total / MAX_POINTS));
+  const stride = Math.max(1, Math.ceil(total / SAMPLE_TARGET_POINTS));
   const points = [];
   let index = 0;
   for (const part of parts) {