fix(import): share repeated KML styles before resource checks

This commit is contained in:
Mazeyar Moeini Feizabadi committed 2026-10-14 04:06:19 +02:00
1 parent a5b21b5d0f
commit ae51f9aee9
8 files changed
+645 -20

No files matched your search

+3 -1
View File
@@ -172,7 +172,9 @@
segments apart. Size limits are checked before anything is built. The import
half of #890. Thanks to Chase Mills (@kapturner). Multipart geometry limits
apply before loading, recorded KML tracks retain their original vertices,
and queued imports respect navigation made while waiting.
and queued imports respect navigation made while waiting. Repeated KML styles
are shared before the style limit is applied, allowing large files with many
copies of the same style to load without changing their graphics.
## [0.2.1] — 2026-10-02 — God's Eye View through Your Agent (with security fixes and other features)
+2 -2
View File
@@ -4331,9 +4331,9 @@ are omitted rather than framing the wrong part of the globe.
- **An imported file cannot make the page fetch anything.** Before `KmlDataSource` reads a KML/KMZ document, `sanitizeKmlDocument` removes `NetworkLink`, `NetworkLinkControl`, `ScreenOverlay`, `PhotoOverlay`, `Model` and `Tour`, every `href` that is not an image packed in the same KMZ or a `data:image/` URL (an icon then falls back to Cesium's default pin, drawn locally, and a `GroundOverlay` left without an image is dropped), every `styleUrl` that is not a local `#id`, and placemark `description`/`BalloonStyle`/`Snippet` HTML and `ExtendedData` before metadata processing. GeoJSON `marker-symbol` (a maki icon asset) is dropped. KMZ images are served from blob URLs, one per packed image however many hrefs name it (`kmzImageResolver`). The hint says when something was left out. `scripts/qa-geo-import.mjs` imports files that name a trap host in each of those places and fails on any request that leaves the app's own origin.
- **Imported metadata stays inert.** After loading, `scrubImportedEntities` removes every entity's description, native label, properties and KML extended data, and keeps the name only as plain text without control characters, at most 120 characters. GeoJSON is loaded with a `describe` that builds no HTML. The file name is shown with `textContent` only.
- **KMZ** is read by a small zip reader (`readZipEntries`: stored and deflated entries through the platform `DecompressionStream`; no encryption, no ZIP64). The main document is `doc.kml`, else the shallowest `.kml`; relative hrefs resolve only to entries inside the archive.
- **Bounds.** A file may be at most 50 MB (`MAX_IMPORT_BYTES`), a KMZ may inflate to at most 120 MB in total (`MAX_KMZ_INFLATED_BYTES`, the zip-bomb stop), one file may create at most 20,000 Cesium entities, including multipart geometries and KML parent containers (`MAX_IMPORT_ENTITIES`) and 250,000 vertices (`MAX_IMPORT_POSITIONS`), and a KML/KMZ at most 2,000 `<href>` and `<Style>` elements (`MAX_KML_LINKS`), counted before Cesium builds anything (`countGeoJsonGeometry`, `countKmlGeometry`, and while reading GPX), and the entity count is checked again after loading. At most 12 files are on the globe at once (`MAX_IMPORTED_FILES`): imports run one batch at a time, so two picks or drops in quick succession cannot pass the cap; files over it are not read. A refused file names its reason in the hint and does not stop the rest of the batch; a file with nothing to draw is refused.
- **Bounds.** A file may be at most 50 MiB (`MAX_IMPORT_BYTES`), a KMZ may inflate to at most 120 MiB in total (`MAX_KMZ_INFLATED_BYTES`, the zip-bomb stop), one file may create at most 20,000 Cesium entities, including multipart geometries and KML parent containers (`MAX_IMPORT_ENTITIES`) and 250,000 vertices (`MAX_IMPORT_POSITIONS`), and a KML/KMZ at most 2,000 normalized style definitions (including named StyleMaps) and distinct surviving image links (`MAX_KML_LINKS`). Identical KML styles are shared after metadata and remote-reference cleanup, preserving inline overrides and local style references; repeated copies do not exhaust the style budget. Folder containers count toward the entity budget. Files of 20–30 MB fit within the byte limit; drawing complexity is bounded separately. Geometry is counted before Cesium builds anything (`countGeoJsonGeometry`, `countKmlGeometry`, and while reading GPX), and the entity count is checked again after loading. At most 12 files are on the globe at once (`MAX_IMPORTED_FILES`): imports run one batch at a time, so two picks or drops in quick succession cannot pass the cap; files over it are not read. A refused file names its reason in the hint and does not stop the rest of the batch; a file with nothing to draw is refused.
- **Keyboard.** Hide, show and remove rebuild the list without losing focus: the same control on the same row keeps it, a removed row hands it to the next row (else the Import button).
- Gate: `node scripts/qa-geo-import.mjs` (dev server up). Unit: `src/data/geoFileImport.test.mjs`, `src/ui/geoFileImport.test.mjs`, `src/ui/flyToDataSource.test.mjs`.
- Gate: `node scripts/qa-geo-import.mjs` (dev server up). Large-file gate: `node scripts/qa-geo-import-large.mjs [--file /absolute/path/to/sample.kml]` tests a 27.6 MB repeated-style fixture, compares its graphics with an unshared Cesium control, and records matched frame times on the actual WebGL renderer. Unit: `src/data/geoFileImport.test.mjs`, `src/ui/geoFileImport.test.mjs`, `src/ui/flyToDataSource.test.mjs`.
### Recent Imagery (September 2026)
+1
View File
@@ -202,6 +202,7 @@
"scripts/qa-visual-effects.mjs",
"scripts/qa-draw-tool.mjs",
"scripts/qa-geo-import.mjs",
"scripts/qa-geo-import-large.mjs",
"src/ui/mapSource.js",
"src/ui/mapSourceControls.js",
"src/ui/mapSourceControls.test.mjs",
+326
View File
@@ -0,0 +1,326 @@
#!/usr/bin/env node
/** Large KML acceptance: style equivalence, 25 MB import and optional real file.
* QA_BASE_URL defaults to http://localhost:4173. Optional --file /path/file.kml.
*/
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import puppeteer from 'puppeteer';
const base = process.env.QA_BASE_URL || 'http://localhost:4173';
const supplied = process.argv.includes('--file')
? path.resolve(process.argv[process.argv.indexOf('--file') + 1])
: null;
const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'gev-large-kml-'));
const file = path.join(directory, 'metadata-rich-25mb.kml');
const style =
'<Style><IconStyle><scale>0.6</scale><color>ff00dd66</color></IconStyle></Style>';
const notes = 'A local observation with supporting detail. '.repeat(120);
const placemarks = Array.from({ length: 5000 }, (_, i) => {
const lon = -122.6 + (i % 100) * 0.002;
const lat = 37.65 + Math.floor(i / 100) * 0.003;
return `<Placemark><name>Point ${i}</name><styleUrl>#base</styleUrl>${style}<description>${notes}</description><Point><coordinates>${lon},${lat},0</coordinates></Point></Placemark>`;
});
const wrap = (body) =>
`<kml xmlns="http://www.opengis.net/kml/2.2"><Document><Style id="base"><IconStyle><heading>25</heading><scale>1.2</scale></IconStyle><LineStyle><color>ffdd6600</color><width>7</width></LineStyle><PolyStyle><color>6600ddff</color><fill>1</fill><outline>0</outline></PolyStyle></Style>${body}</Document></kml>`;
const large = wrap(placemarks.join(''));
fs.writeFileSync(file, large);
const bytes = fs.statSync(file).size;
if (bytes < 20 * 1024 * 1024 || bytes > 30 * 1024 * 1024)
throw new Error(`Fixture must be 20–30 MiB, got ${bytes} bytes`);
const small = wrap(
Array.from({ length: 6 }, (_, i) => {
const kind = i % 3;
const geometry =
kind === 0
? '<Point><coordinates>-122.49,37.77,0</coordinates></Point>'
: kind === 1
? '<LineString><coordinates>-122.50,37.77,0 -122.49,37.77,0</coordinates></LineString>'
: '<Polygon><outerBoundaryIs><LinearRing><coordinates>-122.50,37.77,0 -122.49,37.77,0 -122.49,37.78,0 -122.50,37.77,0</coordinates></LinearRing></outerBoundaryIs></Polygon>';
const override =
kind === 0
? style
: kind === 1
? '<Style><LineStyle><width>3</width></LineStyle></Style>'
: '<Style><PolyStyle><outline>1</outline></PolyStyle></Style>';
return `<Placemark><name>Small ${i}</name><styleUrl>#base</styleUrl>${override}${geometry}</Placemark>`;
}).join('') +
'<Style id="named-a"><IconStyle><scale>0.8</scale></IconStyle></Style>' +
'<Style id="named-b"><IconStyle><scale>0.8</scale></IconStyle></Style>' +
'<StyleMap id="paired"><Pair><key>normal</key><Style><IconStyle><scale>0.8</scale></IconStyle></Style></Pair></StyleMap>' +
['named-a', 'named-b', 'paired']
.map(
(id) =>
`<Placemark><name>${id}</name><styleUrl>#${id}</styleUrl><Point><coordinates>-122.49,37.77,0</coordinates></Point></Placemark>`,
)
.join(''),
);
const browser = await puppeteer.launch({
headless: true,
args:
process.platform === 'darwin'
? ['--use-gl=angle', '--use-angle=metal']
: ['--no-sandbox', '--use-gl=angle', '--use-angle=swiftshader'],
});
const page = await browser.newPage();
const errors = [];
const fileRequests = [];
page.on('pageerror', (e) => errors.push(e.message));
await page.setRequestInterception(true);
page.on('request', (request) => {
if (new URL(request.url()).hostname === 'earthquake.usgs.gov') {
fileRequests.push(request.url());
void request.abort();
} else void request.continue();
});
let failures = 0;
const check = (name, ok, detail) => {
console.log(`[${ok ? 'PASS' : 'FAIL'}] ${name}`, detail ?? '');
if (!ok) failures++;
};
try {
await page.setViewport({ width: 1440, height: 900 });
await page.goto(`${base}/?welcome=0`, { waitUntil: 'domcontentloaded' });
await page.waitForFunction(
() =>
window.__gevGeoImport &&
document.getElementById('loading-screen')?.classList.contains('hidden'),
{ timeout: 60000 },
);
await page.evaluate(async () => {
const { viewer, dataManager } = window.__godsEyeView;
const C = window.__CESIUM__;
for (const id of dataManager.layers.keys())
await dataManager.setEnabled(id, false);
viewer.imageryLayers.removeAll();
viewer.terrainProvider = new C.EllipsoidTerrainProvider();
viewer.camera.cancelFlight();
viewer.camera.setView({
destination: C.Cartesian3.fromDegrees(-122.5, 37.72, 50000),
orientation: { heading: 0, pitch: -Math.PI / 2, roll: 0 },
});
window.__qaStyleSnapshot = (source) => {
const now = viewer.clock.currentTime;
const color = (p) => p?.getValue(now)?.toCssColorString();
const positions = (p) => p?.getValue(now)?.map((v) => [v.x, v.y, v.z]);
return source.entities.values
.map((e) => ({
name: e.name,
point: e.position?.getValue(now),
scale: e.billboard?.scale?.getValue(now),
rotation: e.billboard?.rotation?.getValue(now),
image: e.billboard?.image?.getValue(now),
color: color(e.billboard?.color),
width: e.polyline?.width?.getValue(now),
lineColor: color(e.polyline?.material?.color),
line: positions(e.polyline?.positions),
fill: e.polygon?.fill?.getValue(now),
outline: e.polygon?.outline?.getValue(now),
polygonColor: color(e.polygon?.material?.color),
polygon: e.polygon?.hierarchy
?.getValue(now)
?.positions?.map((v) => [v.x, v.y, v.z]),
}))
.sort((a, b) => a.name.localeCompare(b.name));
};
window.__qaUnsharedKml = async (text) => {
const doc = new DOMParser().parseFromString(text, 'application/xml');
for (const el of [...doc.getElementsByTagName('*')])
if (
['description', 'ExtendedData', 'BalloonStyle'].includes(el.localName)
)
el.remove();
const source = await C.KmlDataSource.load(doc, {
camera: viewer.scene.camera,
canvas: viewer.scene.canvas,
clampToGround: true,
});
for (const e of source.entities.values) e.label = undefined;
return source;
};
});
const equivalent = await page.evaluate(async (text) => {
const { viewer } = window.__godsEyeView;
const control = await window.__qaUnsharedKml(text);
const outcomes = await window.__gevGeoImport.importFiles([
new File([text], 'style-equivalence.kml'),
]);
const candidate = viewer.dataSources.get(viewer.dataSources.length - 1);
const result = {
ok: outcomes[0]?.ok,
same:
JSON.stringify(window.__qaStyleSnapshot(control)) ===
JSON.stringify(window.__qaStyleSnapshot(candidate)),
entities: candidate.entities.values.length,
};
await window.__gevGeoImport.remove(outcomes[0]?.id);
return result;
}, small);
check(
'shared inline and named styles preserve point, line and polygon overrides/inheritance',
equivalent.ok && equivalent.same && equivalent.entities === 9,
JSON.stringify(equivalent),
);
// The control bypasses the former style-count refusal solely in this test,
// to compare native unshared rendering against normalized rendering.
await page.evaluate(async (text) => {
const source = await window.__qaUnsharedKml(text);
window.__qaUnsharedSnapshot = JSON.stringify(
window.__qaStyleSnapshot(source),
);
window.__qaControl = source;
await window.__godsEyeView.viewer.dataSources.add(source);
}, large);
const measure = async () =>
page.evaluate(async () => {
const C = window.__CESIUM__,
{ viewer } = window.__godsEyeView;
const scene = viewer.scene;
viewer.camera.cancelFlight();
viewer.camera.setView({
destination: C.Cartesian3.fromDegrees(-122.5, 37.72, 50000),
orientation: { heading: 0, pitch: -Math.PI / 2, roll: 0 },
});
const started = performance.now(),
samples = [];
let last,
frames = 0;
const remove = scene.postRender.addEventListener(() => {
const t = performance.now();
if (t - started < 2000) return;
frames++;
if (last !== undefined) samples.push(t - last);
last = t;
});
await new Promise((resolve) => {
let previous = performance.now();
function tick(t) {
viewer.camera.rotate(
C.Cartesian3.UNIT_Z,
Math.min(t - previous, 100) * 0.0000002,
);
previous = t;
scene.requestRender();
if (t - started < 7000) requestAnimationFrame(tick);
else resolve();
}
requestAnimationFrame(tick);
});
remove();
samples.sort((a, b) => a - b);
const gl = scene.context._gl,
debug = gl.getExtension('WEBGL_debug_renderer_info');
return {
fps: frames / 5,
p50Ms: samples[Math.floor(samples.length * 0.5)],
p95Ms: samples[Math.floor(samples.length * 0.95)],
renderer: debug ? gl.getParameter(debug.UNMASKED_RENDERER_WEBGL) : null,
};
});
console.log('UNSHARED CONTROL', JSON.stringify(await measure()));
await page.evaluate(() =>
window.__godsEyeView.viewer.dataSources.remove(window.__qaControl, true),
);
const loadPicked = async (filename, expected) => {
const before = await page.evaluate(
() => window.__godsEyeView.viewer.dataSources.length,
);
const started = performance.now();
await (await page.$('#geo-import-input')).uploadFile(filename);
const name = path.basename(filename);
const loaded = await page
.waitForFunction(
(name) =>
window.__gevGeoImport.list().some((f) => f.name === name) ||
document
.getElementById('geo-import-hint')
.textContent.startsWith(`${name}:`),
{ timeout: 120000 },
name,
)
.then(
() => true,
() => false,
);
const state = await page.evaluate(() => ({
files: window.__gevGeoImport.list(),
hint: document.getElementById('geo-import-hint').textContent,
sources: window.__godsEyeView.viewer.dataSources.length,
}));
const item = state.files.find((f) => f.name === name);
check(
`picked ${name} loads`,
loaded &&
(expected === null
? item?.entityCount > 0
: item?.entityCount === expected) &&
state.sources === before + 1,
JSON.stringify({
bytes: fs.statSync(filename).size,
loadMs: Math.round(performance.now() - started),
...state,
}),
);
return item;
};
const item = await loadPicked(file, 5000);
if (item) {
const same = await page.evaluate(() => {
const viewer = window.__godsEyeView.viewer;
const source = viewer.dataSources.get(viewer.dataSources.length - 1);
return (
JSON.stringify(window.__qaStyleSnapshot(source)) ===
window.__qaUnsharedSnapshot
);
});
check('all 5,000 normalized graphics match their unshared control', same);
console.log('NORMALIZED CANDIDATE', JSON.stringify(await measure()));
await page.evaluate((id) => window.__gevGeoImport.remove(id), item.id);
}
if (supplied) {
const clockBefore = await page.evaluate(() => {
const v = window.__godsEyeView.viewer,
C = window.__CESIUM__;
return [
C.JulianDate.toIso8601(v.clock.startTime),
C.JulianDate.toIso8601(v.clock.stopTime),
v.clock.multiplier,
];
});
const suppliedItem = await loadPicked(supplied, null);
const clockAfter = await page.evaluate(() => {
const v = window.__godsEyeView.viewer,
C = window.__CESIUM__;
return [
C.JulianDate.toIso8601(v.clock.startTime),
C.JulianDate.toIso8601(v.clock.stopTime),
v.clock.multiplier,
];
});
check(
'real dated KML leaves the shared clock unchanged',
JSON.stringify(clockBefore) === JSON.stringify(clockAfter),
);
if (suppliedItem)
await page.evaluate(
(id) => window.__gevGeoImport.remove(id),
suppliedItem.id,
);
}
check(
'file-directed USGS icon/metadata requests never leave the page',
fileRequests.length === 0,
fileRequests.length,
);
check('no uncaught page errors', errors.length === 0, JSON.stringify(errors));
check(
'all imported sources are removed',
await page.evaluate(() => window.__gevGeoImport.list().length === 0),
);
console.log('Fixture retained at', file);
} finally {
await browser.close();
}
process.exitCode = failures ? 1 : 0;
+135 -11
View File
@@ -42,7 +42,7 @@ export const MAX_IMPORTED_FILES = 12;
export const MAX_IMPORT_ENTITIES = 20_000;
/** Most vertices (points of every line, ring and marker) in one file. */
export const MAX_IMPORT_POSITIONS = 250_000;
/** Most `<href>` and `<Style>` elements in one KML/KMZ document. */
/** Most normalized style definitions and local image links in one KML/KMZ. */
export const MAX_KML_LINKS = 2_000;
/** Longest plain-text name kept from a file's features. */
export const MAX_IMPORT_NAME_LENGTH = 120;
@@ -225,11 +225,11 @@ export function countGeoJsonGeometry(
function overLimitError({ over, features, positions }) {
if (over === 'entities')
return new GeoImportError(
`That file draws more than ${MAX_IMPORT_ENTITIES.toLocaleString()} objects (the limit).`,
`That file creates more than ${MAX_IMPORT_ENTITIES.toLocaleString()} map objects (the limit).`,
);
if (over === 'links')
return new GeoImportError(
`That file has more than ${MAX_KML_LINKS.toLocaleString()} styles and icon links (the limit).`,
`That file has more than ${MAX_KML_LINKS.toLocaleString()} style definitions and image links after cleanup (the limit).`,
);
if (over === 'features')
return new GeoImportError(
@@ -548,11 +548,118 @@ const childrenNamed = (el, localName) => {
const childText = (el, localName) =>
(childrenNamed(el, localName)[0]?.textContent || '').trim();
/**
* Share identical styles before the loader builds their graphics. Inline
* overrides stay inline StyleMaps, so an existing styleUrl still supplies
* properties the override leaves unset. Named references are redirected.
* Random-color styles remain unchanged; duplicate IDs
* keep the loader's original precedence instead of being rewritten.
* @param {Document} doc Sanitized KML.
* @param {{serializeStyle?: Function, maxStyles?: number}} [options]
* @returns {number} Number of original style blocks replaced.
*/
export function deduplicateKmlStyles(
doc,
{ serializeStyle, maxStyles = MAX_KML_LINKS } = {},
) {
const serializer = serializeStyle ? null : new XMLSerializer();
const serialize =
serializeStyle || ((el) => serializer.serializeToString(el));
const groups = new Map();
const usedIds = new Set();
const duplicateIds = new Set();
for (const el of doc.getElementsByTagName('*')) {
const id = el.getAttribute('id');
if (id !== null) {
if (usedIds.has(id)) duplicateIds.add(id);
usedIds.add(id);
}
}
for (const style of elementsNamed(doc, 'Style')) {
if (
elementsNamed(style, 'colorMode').some(
(el) => el.textContent.trim() === 'random',
)
)
continue;
const template = style.cloneNode(true);
template.removeAttribute('id');
const key = serialize(template);
let group = groups.get(key);
if (!group) {
group = { template, nodes: [] };
groups.set(key, group);
if (groups.size > maxStyles) throw overLimitError({ over: 'links' });
}
group.nodes.push(style);
}
const redirects = new Map();
let replaced = 0;
let serial = 0;
let container;
for (const { template, nodes } of groups.values()) {
if (
nodes.length < 2 ||
nodes.some((el) => duplicateIds.has(el.getAttribute('id')))
)
continue;
const ns = template.namespaceURI || 'http://www.opengis.net/kml/2.2';
if (!container) {
const root = doc.documentElement;
container =
root.localName === 'Document'
? root
: childrenNamed(root, 'Document')[0];
if (!container) {
container = doc.createElementNS(ns, 'Document');
while (root.firstChild) container.appendChild(root.firstChild);
root.appendChild(container);
}
}
let id;
do {
id = `gev-import-style-${++serial}`;
} while (usedIds.has(id));
usedIds.add(id);
template.setAttribute('id', id);
container.appendChild(template);
for (const style of nodes) {
const originalId = style.getAttribute('id');
if (originalId !== null) redirects.set(originalId, id);
let reference;
if (style.parentNode.localName === 'Pair') {
reference = doc.createElementNS(ns, 'styleUrl');
reference.textContent = `#${id}`;
} else {
reference = doc.createElementNS(ns, 'StyleMap');
for (const state of ['normal', 'highlight']) {
const pair = doc.createElementNS(ns, 'Pair');
const key = doc.createElementNS(ns, 'key');
key.textContent = state;
const url = doc.createElementNS(ns, 'styleUrl');
url.textContent = `#${id}`;
pair.appendChild(key);
pair.appendChild(url);
reference.appendChild(pair);
}
}
style.parentNode.replaceChild(reference, style);
replaced += 1;
}
}
for (const url of elementsNamed(doc, 'styleUrl')) {
const value = url.textContent.trim();
const id = value.startsWith('#') && redirects.get(value.slice(1));
if (id) url.textContent = `#${id}`;
}
return replaced;
}
/**
* Count placemarks and vertices in a parsed KML document before Cesium reads
* it: every `<coordinates>` tuple and every `gx:coord`, plus the `<href>`
* and `<Style>` elements (`links`), each of which can make Cesium load and
* decode an image.
* it: every `<coordinates>` tuple and every `gx:coord`, plus normalized
* style definitions and distinct image `<href>` values (`links`).
* Inline StyleMaps are references, not additional shared style definitions.
* @param {Document} doc
* @returns {{features: number, entities: number, positions: number, links: number,
* over: ''|'features'|'entities'|'positions'|'links'}}
@@ -570,6 +677,7 @@ export function countKmlGeometry(
let entities = 0;
let positions = 0;
let links = 0;
const imageLinks = new Set();
const result = (over) => ({ features, entities, positions, links, over });
const geometryNames = new Set([
'Point',
@@ -583,10 +691,11 @@ export function countKmlGeometry(
]);
for (const el of doc.getElementsByTagName('*')) {
const name = el.localName;
// Cesium creates a parent for every placemark and a child entity for
// each geometry in a MultiGeometry, including nested containers.
// Cesium creates entities for folders, a parent for every placemark, and
// a child for each geometry in a MultiGeometry, including nested containers.
if (
name === 'Placemark' ||
name === 'Folder' ||
name === 'GroundOverlay' ||
(geometryNames.has(name) &&
['MultiGeometry', 'MultiTrack'].includes(el.parentNode?.localName))
@@ -594,9 +703,19 @@ export function countKmlGeometry(
entities += 1;
if (entities > maxEntities) return result('entities');
}
if (name === 'href' || name === 'Style') {
if (
name === 'Style' ||
(name === 'StyleMap' && el.getAttribute('id') !== null)
) {
links += 1;
if (links > maxLinks) return result('links');
} else if (name === 'href') {
const href = el.textContent.trim();
if (!imageLinks.has(href)) {
imageLinks.add(href);
links += 1;
if (links > maxLinks) return result('links');
}
} else if (name === 'Placemark') {
features += 1;
if (features > maxFeatures) return result('features');
@@ -888,9 +1007,14 @@ export async function loadGeoFile(file, { viewer, color = '#39d0ff' }) {
text = new TextDecoder().decode(bytes);
}
const doc = parseXml(text, format.toUpperCase());
const size = countKmlGeometry(doc);
// Bound geometry before transformations, then count the styles and
// image links that survive sanitizing and sharing repeated styles.
const size = countKmlGeometry(doc, { maxLinks: Infinity });
if (size.over) throw overLimitError(size);
removed = sanitizeKmlDocument(doc, resolveLocal).removed;
deduplicateKmlStyles(doc);
const normalizedSize = countKmlGeometry(doc);
if (normalizedSize.over) throw overLimitError(normalizedSize);
dataSource = await Cesium.KmlDataSource.load(doc, {
camera: viewer.scene.camera,
canvas: viewer.scene.canvas,
@@ -908,7 +1032,7 @@ export async function loadGeoFile(file, { viewer, color = '#39d0ff' }) {
if (entityCount > MAX_IMPORT_ENTITIES) {
revoke();
throw new GeoImportError(
`That file has ${entityCount.toLocaleString()} features; the limit is ${MAX_IMPORT_ENTITIES.toLocaleString()}.`,
`That file has ${entityCount.toLocaleString()} map objects; the limit is ${MAX_IMPORT_ENTITIES.toLocaleString()}.`,
);
}
if (entityCount === 0) {
+176 -4
View File
@@ -8,6 +8,7 @@ import {
MAX_IMPORT_ENTITIES,
countGeoJsonGeometry,
countKmlGeometry,
deduplicateKmlStyles,
detachImportedTime,
gpxToGeoJson,
importFeatureName,
@@ -231,8 +232,41 @@ function xmlEl(localName, attrs = {}, kids = []) {
const el = {
nodeType: 1,
localName,
namespaceURI: 'http://www.opengis.net/kml/2.2',
childNodes: [],
getAttribute: (n) => (n in attrs ? String(attrs[n]) : null),
get attributes() {
return Object.entries(attrs);
},
setAttribute: (n, value) => {
attrs[n] = value;
},
removeAttribute: (n) => {
delete attrs[n];
},
cloneNode() {
return xmlEl(
localName,
{ ...attrs },
text ?? el.childNodes.map((child) => child.cloneNode(true)),
);
},
getElementsByTagName(name) {
return xmlDoc(this).getElementsByTagName(name).slice(1);
},
appendChild(child) {
child.parentNode?.removeChild(child);
child.parentNode = this;
this.childNodes.push(child);
},
replaceChild(next, previous) {
const index = this.childNodes.indexOf(previous);
assert.ok(index >= 0);
next.parentNode?.removeChild(next);
this.childNodes[index] = next;
next.parentNode = this;
previous.parentNode = null;
},
removeChild(child) {
this.childNodes.splice(this.childNodes.indexOf(child), 1);
child.parentNode = null;
@@ -242,14 +276,22 @@ function xmlEl(localName, attrs = {}, kids = []) {
k.parentNode = el;
el.childNodes.push(k);
}
const text = typeof kids === 'string' ? kids : null;
let text = typeof kids === 'string' ? kids : null;
Object.defineProperty(el, 'firstChild', {
get: () => el.childNodes[0] || null,
});
Object.defineProperty(el, 'textContent', {
get: () => text ?? el.childNodes.map((c) => c.textContent).join(''),
set: (value) => {
text = value;
el.childNodes = [];
},
});
return el;
}
const xmlDoc = (root) => ({
documentElement: root,
createElementNS: (_ns, name) => xmlEl(name),
getElementsByTagName(name) {
assert.equal(name, '*');
const out = [];
@@ -261,6 +303,118 @@ const xmlDoc = (root) => ({
return out;
},
});
const serializeStyle = (el) =>
JSON.stringify({
name: el.localName,
attributes: el.attributes,
text: el.childNodes.length ? null : el.textContent,
children: el.childNodes.map(serializeStyle),
});
test('repeated inline styles are shared without replacing their inherited styleUrl', () => {
const style = () =>
xmlEl('Style', {}, [xmlEl('IconStyle', {}, [xmlEl('scale', {}, '0.6')])]);
const placemarks = Array.from({ length: 2100 }, () =>
xmlEl('Placemark', {}, [xmlEl('styleUrl', {}, '#base'), style()]),
);
const doc = xmlDoc(
xmlEl('kml', {}, [
xmlEl('Document', {}, [
xmlEl('Style', { id: 'gev-import-style-1' }, [xmlEl('LineStyle')]),
...placemarks,
]),
]),
);
assert.equal(countKmlGeometry(doc).over, 'links');
assert.equal(deduplicateKmlStyles(doc, { serializeStyle }), 2100);
assert.equal(countKmlGeometry(doc).over, '');
const styles = doc
.getElementsByTagName('*')
.filter((el) => el.localName === 'Style');
assert.equal(styles.length, 2);
assert.equal(styles[1].getAttribute('id'), 'gev-import-style-2');
for (const placemark of placemarks) {
assert.equal(placemark.childNodes[0].textContent, '#base');
assert.equal(placemark.childNodes[1].localName, 'StyleMap');
assert.deepEqual(
placemark.childNodes[1].childNodes.map((pair) =>
pair.childNodes.map((el) => el.textContent),
),
[
['normal', '#gev-import-style-2'],
['highlight', '#gev-import-style-2'],
],
);
}
});
test('identical named styles redirect existing references and Pair styles remain valid', () => {
const style = (id) =>
xmlEl('Style', id ? { id } : {}, [
xmlEl('LineStyle', {}, [xmlEl('width', {}, '3')]),
]);
const reference = xmlEl('styleUrl', {}, '#b');
const pair = xmlEl('Pair', {}, [xmlEl('key', {}, 'normal'), style()]);
const doc = xmlDoc(
xmlEl('kml', {}, [
style('a'),
style('b'),
xmlEl('StyleMap', { id: 'map' }, [pair]),
xmlEl('Placemark', {}, [reference]),
]),
);
assert.equal(deduplicateKmlStyles(doc, { serializeStyle }), 3);
assert.equal(doc.documentElement.firstChild.localName, 'Document');
assert.equal(reference.textContent, '#gev-import-style-1');
assert.equal(pair.childNodes[1].localName, 'styleUrl');
assert.equal(pair.childNodes[1].textContent, '#gev-import-style-1');
assert.equal(
doc.getElementsByTagName('*').filter((el) => el.localName === 'Style')
.length,
1,
);
});
test('random colors and ambiguous style IDs retain independent evaluations and precedence', () => {
const random = () =>
xmlEl('Style', {}, [
xmlEl('IconStyle', {}, [xmlEl('colorMode', {}, 'random')]),
]);
const named = (width) =>
xmlEl('Style', { id: 'collision' }, [
xmlEl('LineStyle', {}, [xmlEl('width', {}, width)]),
]);
const original = [random(), random(), named('1'), named('1'), named('2')];
const doc = xmlDoc(xmlEl('kml', {}, original));
assert.equal(deduplicateKmlStyles(doc, { serializeStyle }), 0);
assert.deepEqual(doc.documentElement.childNodes, original);
});
test('distinct style definitions and named StyleMaps still have a pre-loader budget', () => {
const doc = xmlDoc(
xmlEl(
'kml',
{},
['1', '2', '3'].map((width) =>
xmlEl('Style', {}, [
xmlEl('LineStyle', {}, [xmlEl('width', {}, width)]),
]),
),
),
);
assert.throws(
() => deduplicateKmlStyles(doc, { serializeStyle, maxStyles: 2 }),
GeoImportError,
);
const maps = xmlDoc(
xmlEl('kml', {}, [
xmlEl('StyleMap', { id: 'a' }),
xmlEl('StyleMap', { id: 'b' }),
]),
);
assert.equal(countKmlGeometry(maps, { maxLinks: 1 }).over, 'links');
});
test('KML extended metadata is removed before the document reaches the loader', () => {
const point = xmlEl('Point', {}, [xmlEl('coordinates', {}, '1,2,0')]);
const metadata = xmlEl('ExtendedData', {}, [
@@ -495,14 +649,32 @@ test('KML multipart geometries count every Cesium entity before loading', () =>
assert.equal(countKmlGeometry(oversized).over, 'entities');
});
test('KML styles and icon links are counted against their own limit', () => {
test('KML folders contribute to the pre-loader entity budget', () => {
const doc = xmlDoc(
xmlEl('kml', {}, [
xmlEl('Folder', {}, [xmlEl('Folder', {}, [xmlEl('Placemark')])]),
]),
);
assert.equal(countKmlGeometry(doc).entities, 3);
assert.equal(countKmlGeometry(doc, { maxEntities: 2 }).over, 'entities');
});
test('KML style definitions and distinct icon links have their own limit', () => {
const style = () =>
xmlEl('Style', {}, [
xmlEl('IconStyle', {}, [xmlEl('Icon', {}, [xmlEl('href', {}, 'a.png')])]),
]);
const doc = xmlDoc(xmlEl('kml', {}, [style(), style(), style()]));
assert.equal(countKmlGeometry(doc).links, 6);
assert.equal(countKmlGeometry(doc, { maxLinks: 5 }).over, 'links');
assert.equal(countKmlGeometry(doc).links, 4);
assert.equal(countKmlGeometry(doc, { maxLinks: 3 }).over, 'links');
const distinct = xmlDoc(
xmlEl(
'kml',
{},
['a.png', 'b.png', 'c.png'].map((href) => xmlEl('href', {}, href)),
),
);
assert.equal(countKmlGeometry(distinct, { maxLinks: 2 }).over, 'links');
});
test('a KMZ image gets one blob URL however many hrefs name it', () => {
+1 -1
View File
@@ -409,6 +409,6 @@ export function summarize(outcomes) {
0,
);
if (added && stripped)
parts.push('Network links and remote images in KML were left out.');
parts.push('KML descriptions and external references were left out.');
return parts.join(' ');
}
+1 -1
View File
@@ -347,7 +347,7 @@ test('dropping files on the globe imports them; other drags are left alone', asy
test('stripped KML content is mentioned once, for the batch that had it', () => {
assert.equal(
summarize([{ name: 'a.kml', ok: true, removed: 3 }]),
'Added 1 file. Network links and remote images in KML were left out.',
'Added 1 file. KML descriptions and external references were left out.',
);
assert.equal(
summarize([{ name: 'b.geojson', ok: true, removed: 0 }]),