mirror of
https://github.com/bilawalsidhu/gods-eye-view.git
synced 2026-11-04 02:14:01 +00:00
fix(import): share repeated KML styles before resource checks
This commit is contained in:
1 parent
a5b21b5d0f
commit
ae51f9aee9
8 files changed
+645
-20
No files matched your search
+3
-1
@@ -172,7 +172,9 @@
|
||||
segments apart. Size limits are checked before anything is built. The import
|
||||
half of #890. Thanks to Chase Mills (@kapturner). Multipart geometry limits
|
||||
apply before loading, recorded KML tracks retain their original vertices,
|
||||
and queued imports respect navigation made while waiting.
|
||||
and queued imports respect navigation made while waiting. Repeated KML styles
|
||||
are shared before the style limit is applied, allowing large files with many
|
||||
copies of the same style to load without changing their graphics.
|
||||
|
||||
## [0.2.1] — 2026-10-02 — God's Eye View through Your Agent (with security fixes and other features)
|
||||
|
||||
|
||||
@@ -4331,9 +4331,9 @@ are omitted rather than framing the wrong part of the globe.
|
||||
- **An imported file cannot make the page fetch anything.** Before `KmlDataSource` reads a KML/KMZ document, `sanitizeKmlDocument` removes `NetworkLink`, `NetworkLinkControl`, `ScreenOverlay`, `PhotoOverlay`, `Model` and `Tour`, every `href` that is not an image packed in the same KMZ or a `data:image/` URL (an icon then falls back to Cesium's default pin, drawn locally, and a `GroundOverlay` left without an image is dropped), every `styleUrl` that is not a local `#id`, and placemark `description`/`BalloonStyle`/`Snippet` HTML and `ExtendedData` before metadata processing. GeoJSON `marker-symbol` (a maki icon asset) is dropped. KMZ images are served from blob URLs, one per packed image however many hrefs name it (`kmzImageResolver`). The hint says when something was left out. `scripts/qa-geo-import.mjs` imports files that name a trap host in each of those places and fails on any request that leaves the app's own origin.
|
||||
- **Imported metadata stays inert.** After loading, `scrubImportedEntities` removes every entity's description, native label, properties and KML extended data, and keeps the name only as plain text without control characters, at most 120 characters. GeoJSON is loaded with a `describe` that builds no HTML. The file name is shown with `textContent` only.
|
||||
- **KMZ** is read by a small zip reader (`readZipEntries`: stored and deflated entries through the platform `DecompressionStream`; no encryption, no ZIP64). The main document is `doc.kml`, else the shallowest `.kml`; relative hrefs resolve only to entries inside the archive.
|
||||
- **Bounds.** A file may be at most 50 MB (`MAX_IMPORT_BYTES`), a KMZ may inflate to at most 120 MB in total (`MAX_KMZ_INFLATED_BYTES`, the zip-bomb stop), one file may create at most 20,000 Cesium entities, including multipart geometries and KML parent containers (`MAX_IMPORT_ENTITIES`) and 250,000 vertices (`MAX_IMPORT_POSITIONS`), and a KML/KMZ at most 2,000 `<href>` and `<Style>` elements (`MAX_KML_LINKS`), counted before Cesium builds anything (`countGeoJsonGeometry`, `countKmlGeometry`, and while reading GPX), and the entity count is checked again after loading. At most 12 files are on the globe at once (`MAX_IMPORTED_FILES`): imports run one batch at a time, so two picks or drops in quick succession cannot pass the cap; files over it are not read. A refused file names its reason in the hint and does not stop the rest of the batch; a file with nothing to draw is refused.
|
||||
- **Bounds.** A file may be at most 50 MiB (`MAX_IMPORT_BYTES`), a KMZ may inflate to at most 120 MiB in total (`MAX_KMZ_INFLATED_BYTES`, the zip-bomb stop), one file may create at most 20,000 Cesium entities, including multipart geometries and KML parent containers (`MAX_IMPORT_ENTITIES`) and 250,000 vertices (`MAX_IMPORT_POSITIONS`), and a KML/KMZ at most 2,000 normalized style definitions (including named StyleMaps) and distinct surviving image links (`MAX_KML_LINKS`). Identical KML styles are shared after metadata and remote-reference cleanup, preserving inline overrides and local style references; repeated copies do not exhaust the style budget. Folder containers count toward the entity budget. Files of 20–30 MB fit within the byte limit; drawing complexity is bounded separately. Geometry is counted before Cesium builds anything (`countGeoJsonGeometry`, `countKmlGeometry`, and while reading GPX), and the entity count is checked again after loading. At most 12 files are on the globe at once (`MAX_IMPORTED_FILES`): imports run one batch at a time, so two picks or drops in quick succession cannot pass the cap; files over it are not read. A refused file names its reason in the hint and does not stop the rest of the batch; a file with nothing to draw is refused.
|
||||
- **Keyboard.** Hide, show and remove rebuild the list without losing focus: the same control on the same row keeps it, a removed row hands it to the next row (else the Import button).
|
||||
- Gate: `node scripts/qa-geo-import.mjs` (dev server up). Unit: `src/data/geoFileImport.test.mjs`, `src/ui/geoFileImport.test.mjs`, `src/ui/flyToDataSource.test.mjs`.
|
||||
- Gate: `node scripts/qa-geo-import.mjs` (dev server up). Large-file gate: `node scripts/qa-geo-import-large.mjs [--file /absolute/path/to/sample.kml]` tests a 27.6 MB repeated-style fixture, compares its graphics with an unshared Cesium control, and records matched frame times on the actual WebGL renderer. Unit: `src/data/geoFileImport.test.mjs`, `src/ui/geoFileImport.test.mjs`, `src/ui/flyToDataSource.test.mjs`.
|
||||
|
||||
### Recent Imagery (September 2026)
|
||||
|
||||
|
||||
@@ -202,6 +202,7 @@
|
||||
"scripts/qa-visual-effects.mjs",
|
||||
"scripts/qa-draw-tool.mjs",
|
||||
"scripts/qa-geo-import.mjs",
|
||||
"scripts/qa-geo-import-large.mjs",
|
||||
"src/ui/mapSource.js",
|
||||
"src/ui/mapSourceControls.js",
|
||||
"src/ui/mapSourceControls.test.mjs",
|
||||
|
||||
@@ -0,0 +1,326 @@
|
||||
#!/usr/bin/env node
|
||||
/** Large KML acceptance: style equivalence, 25 MB import and optional real file.
|
||||
* QA_BASE_URL defaults to http://localhost:4173. Optional --file /path/file.kml.
|
||||
*/
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import puppeteer from 'puppeteer';
|
||||
|
||||
const base = process.env.QA_BASE_URL || 'http://localhost:4173';
|
||||
const supplied = process.argv.includes('--file')
|
||||
? path.resolve(process.argv[process.argv.indexOf('--file') + 1])
|
||||
: null;
|
||||
const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'gev-large-kml-'));
|
||||
const file = path.join(directory, 'metadata-rich-25mb.kml');
|
||||
const style =
|
||||
'<Style><IconStyle><scale>0.6</scale><color>ff00dd66</color></IconStyle></Style>';
|
||||
const notes = 'A local observation with supporting detail. '.repeat(120);
|
||||
const placemarks = Array.from({ length: 5000 }, (_, i) => {
|
||||
const lon = -122.6 + (i % 100) * 0.002;
|
||||
const lat = 37.65 + Math.floor(i / 100) * 0.003;
|
||||
return `<Placemark><name>Point ${i}</name><styleUrl>#base</styleUrl>${style}<description>${notes}</description><Point><coordinates>${lon},${lat},0</coordinates></Point></Placemark>`;
|
||||
});
|
||||
const wrap = (body) =>
|
||||
`<kml xmlns="http://www.opengis.net/kml/2.2"><Document><Style id="base"><IconStyle><heading>25</heading><scale>1.2</scale></IconStyle><LineStyle><color>ffdd6600</color><width>7</width></LineStyle><PolyStyle><color>6600ddff</color><fill>1</fill><outline>0</outline></PolyStyle></Style>${body}</Document></kml>`;
|
||||
const large = wrap(placemarks.join(''));
|
||||
fs.writeFileSync(file, large);
|
||||
const bytes = fs.statSync(file).size;
|
||||
if (bytes < 20 * 1024 * 1024 || bytes > 30 * 1024 * 1024)
|
||||
throw new Error(`Fixture must be 20–30 MiB, got ${bytes} bytes`);
|
||||
const small = wrap(
|
||||
Array.from({ length: 6 }, (_, i) => {
|
||||
const kind = i % 3;
|
||||
const geometry =
|
||||
kind === 0
|
||||
? '<Point><coordinates>-122.49,37.77,0</coordinates></Point>'
|
||||
: kind === 1
|
||||
? '<LineString><coordinates>-122.50,37.77,0 -122.49,37.77,0</coordinates></LineString>'
|
||||
: '<Polygon><outerBoundaryIs><LinearRing><coordinates>-122.50,37.77,0 -122.49,37.77,0 -122.49,37.78,0 -122.50,37.77,0</coordinates></LinearRing></outerBoundaryIs></Polygon>';
|
||||
const override =
|
||||
kind === 0
|
||||
? style
|
||||
: kind === 1
|
||||
? '<Style><LineStyle><width>3</width></LineStyle></Style>'
|
||||
: '<Style><PolyStyle><outline>1</outline></PolyStyle></Style>';
|
||||
return `<Placemark><name>Small ${i}</name><styleUrl>#base</styleUrl>${override}${geometry}</Placemark>`;
|
||||
}).join('') +
|
||||
'<Style id="named-a"><IconStyle><scale>0.8</scale></IconStyle></Style>' +
|
||||
'<Style id="named-b"><IconStyle><scale>0.8</scale></IconStyle></Style>' +
|
||||
'<StyleMap id="paired"><Pair><key>normal</key><Style><IconStyle><scale>0.8</scale></IconStyle></Style></Pair></StyleMap>' +
|
||||
['named-a', 'named-b', 'paired']
|
||||
.map(
|
||||
(id) =>
|
||||
`<Placemark><name>${id}</name><styleUrl>#${id}</styleUrl><Point><coordinates>-122.49,37.77,0</coordinates></Point></Placemark>`,
|
||||
)
|
||||
.join(''),
|
||||
);
|
||||
const browser = await puppeteer.launch({
|
||||
headless: true,
|
||||
args:
|
||||
process.platform === 'darwin'
|
||||
? ['--use-gl=angle', '--use-angle=metal']
|
||||
: ['--no-sandbox', '--use-gl=angle', '--use-angle=swiftshader'],
|
||||
});
|
||||
const page = await browser.newPage();
|
||||
const errors = [];
|
||||
const fileRequests = [];
|
||||
page.on('pageerror', (e) => errors.push(e.message));
|
||||
await page.setRequestInterception(true);
|
||||
page.on('request', (request) => {
|
||||
if (new URL(request.url()).hostname === 'earthquake.usgs.gov') {
|
||||
fileRequests.push(request.url());
|
||||
void request.abort();
|
||||
} else void request.continue();
|
||||
});
|
||||
let failures = 0;
|
||||
const check = (name, ok, detail) => {
|
||||
console.log(`[${ok ? 'PASS' : 'FAIL'}] ${name}`, detail ?? '');
|
||||
if (!ok) failures++;
|
||||
};
|
||||
try {
|
||||
await page.setViewport({ width: 1440, height: 900 });
|
||||
await page.goto(`${base}/?welcome=0`, { waitUntil: 'domcontentloaded' });
|
||||
await page.waitForFunction(
|
||||
() =>
|
||||
window.__gevGeoImport &&
|
||||
document.getElementById('loading-screen')?.classList.contains('hidden'),
|
||||
{ timeout: 60000 },
|
||||
);
|
||||
await page.evaluate(async () => {
|
||||
const { viewer, dataManager } = window.__godsEyeView;
|
||||
const C = window.__CESIUM__;
|
||||
for (const id of dataManager.layers.keys())
|
||||
await dataManager.setEnabled(id, false);
|
||||
viewer.imageryLayers.removeAll();
|
||||
viewer.terrainProvider = new C.EllipsoidTerrainProvider();
|
||||
viewer.camera.cancelFlight();
|
||||
viewer.camera.setView({
|
||||
destination: C.Cartesian3.fromDegrees(-122.5, 37.72, 50000),
|
||||
orientation: { heading: 0, pitch: -Math.PI / 2, roll: 0 },
|
||||
});
|
||||
window.__qaStyleSnapshot = (source) => {
|
||||
const now = viewer.clock.currentTime;
|
||||
const color = (p) => p?.getValue(now)?.toCssColorString();
|
||||
const positions = (p) => p?.getValue(now)?.map((v) => [v.x, v.y, v.z]);
|
||||
return source.entities.values
|
||||
.map((e) => ({
|
||||
name: e.name,
|
||||
point: e.position?.getValue(now),
|
||||
scale: e.billboard?.scale?.getValue(now),
|
||||
rotation: e.billboard?.rotation?.getValue(now),
|
||||
image: e.billboard?.image?.getValue(now),
|
||||
color: color(e.billboard?.color),
|
||||
width: e.polyline?.width?.getValue(now),
|
||||
lineColor: color(e.polyline?.material?.color),
|
||||
line: positions(e.polyline?.positions),
|
||||
fill: e.polygon?.fill?.getValue(now),
|
||||
outline: e.polygon?.outline?.getValue(now),
|
||||
polygonColor: color(e.polygon?.material?.color),
|
||||
polygon: e.polygon?.hierarchy
|
||||
?.getValue(now)
|
||||
?.positions?.map((v) => [v.x, v.y, v.z]),
|
||||
}))
|
||||
.sort((a, b) => a.name.localeCompare(b.name));
|
||||
};
|
||||
window.__qaUnsharedKml = async (text) => {
|
||||
const doc = new DOMParser().parseFromString(text, 'application/xml');
|
||||
for (const el of [...doc.getElementsByTagName('*')])
|
||||
if (
|
||||
['description', 'ExtendedData', 'BalloonStyle'].includes(el.localName)
|
||||
)
|
||||
el.remove();
|
||||
const source = await C.KmlDataSource.load(doc, {
|
||||
camera: viewer.scene.camera,
|
||||
canvas: viewer.scene.canvas,
|
||||
clampToGround: true,
|
||||
});
|
||||
for (const e of source.entities.values) e.label = undefined;
|
||||
return source;
|
||||
};
|
||||
});
|
||||
const equivalent = await page.evaluate(async (text) => {
|
||||
const { viewer } = window.__godsEyeView;
|
||||
const control = await window.__qaUnsharedKml(text);
|
||||
const outcomes = await window.__gevGeoImport.importFiles([
|
||||
new File([text], 'style-equivalence.kml'),
|
||||
]);
|
||||
const candidate = viewer.dataSources.get(viewer.dataSources.length - 1);
|
||||
const result = {
|
||||
ok: outcomes[0]?.ok,
|
||||
same:
|
||||
JSON.stringify(window.__qaStyleSnapshot(control)) ===
|
||||
JSON.stringify(window.__qaStyleSnapshot(candidate)),
|
||||
entities: candidate.entities.values.length,
|
||||
};
|
||||
await window.__gevGeoImport.remove(outcomes[0]?.id);
|
||||
return result;
|
||||
}, small);
|
||||
check(
|
||||
'shared inline and named styles preserve point, line and polygon overrides/inheritance',
|
||||
equivalent.ok && equivalent.same && equivalent.entities === 9,
|
||||
JSON.stringify(equivalent),
|
||||
);
|
||||
|
||||
// The control bypasses the former style-count refusal solely in this test,
|
||||
// to compare native unshared rendering against normalized rendering.
|
||||
await page.evaluate(async (text) => {
|
||||
const source = await window.__qaUnsharedKml(text);
|
||||
window.__qaUnsharedSnapshot = JSON.stringify(
|
||||
window.__qaStyleSnapshot(source),
|
||||
);
|
||||
window.__qaControl = source;
|
||||
await window.__godsEyeView.viewer.dataSources.add(source);
|
||||
}, large);
|
||||
const measure = async () =>
|
||||
page.evaluate(async () => {
|
||||
const C = window.__CESIUM__,
|
||||
{ viewer } = window.__godsEyeView;
|
||||
const scene = viewer.scene;
|
||||
viewer.camera.cancelFlight();
|
||||
viewer.camera.setView({
|
||||
destination: C.Cartesian3.fromDegrees(-122.5, 37.72, 50000),
|
||||
orientation: { heading: 0, pitch: -Math.PI / 2, roll: 0 },
|
||||
});
|
||||
const started = performance.now(),
|
||||
samples = [];
|
||||
let last,
|
||||
frames = 0;
|
||||
const remove = scene.postRender.addEventListener(() => {
|
||||
const t = performance.now();
|
||||
if (t - started < 2000) return;
|
||||
frames++;
|
||||
if (last !== undefined) samples.push(t - last);
|
||||
last = t;
|
||||
});
|
||||
await new Promise((resolve) => {
|
||||
let previous = performance.now();
|
||||
function tick(t) {
|
||||
viewer.camera.rotate(
|
||||
C.Cartesian3.UNIT_Z,
|
||||
Math.min(t - previous, 100) * 0.0000002,
|
||||
);
|
||||
previous = t;
|
||||
scene.requestRender();
|
||||
if (t - started < 7000) requestAnimationFrame(tick);
|
||||
else resolve();
|
||||
}
|
||||
requestAnimationFrame(tick);
|
||||
});
|
||||
remove();
|
||||
samples.sort((a, b) => a - b);
|
||||
const gl = scene.context._gl,
|
||||
debug = gl.getExtension('WEBGL_debug_renderer_info');
|
||||
return {
|
||||
fps: frames / 5,
|
||||
p50Ms: samples[Math.floor(samples.length * 0.5)],
|
||||
p95Ms: samples[Math.floor(samples.length * 0.95)],
|
||||
renderer: debug ? gl.getParameter(debug.UNMASKED_RENDERER_WEBGL) : null,
|
||||
};
|
||||
});
|
||||
console.log('UNSHARED CONTROL', JSON.stringify(await measure()));
|
||||
await page.evaluate(() =>
|
||||
window.__godsEyeView.viewer.dataSources.remove(window.__qaControl, true),
|
||||
);
|
||||
|
||||
const loadPicked = async (filename, expected) => {
|
||||
const before = await page.evaluate(
|
||||
() => window.__godsEyeView.viewer.dataSources.length,
|
||||
);
|
||||
const started = performance.now();
|
||||
await (await page.$('#geo-import-input')).uploadFile(filename);
|
||||
const name = path.basename(filename);
|
||||
const loaded = await page
|
||||
.waitForFunction(
|
||||
(name) =>
|
||||
window.__gevGeoImport.list().some((f) => f.name === name) ||
|
||||
document
|
||||
.getElementById('geo-import-hint')
|
||||
.textContent.startsWith(`${name}:`),
|
||||
{ timeout: 120000 },
|
||||
name,
|
||||
)
|
||||
.then(
|
||||
() => true,
|
||||
() => false,
|
||||
);
|
||||
const state = await page.evaluate(() => ({
|
||||
files: window.__gevGeoImport.list(),
|
||||
hint: document.getElementById('geo-import-hint').textContent,
|
||||
sources: window.__godsEyeView.viewer.dataSources.length,
|
||||
}));
|
||||
const item = state.files.find((f) => f.name === name);
|
||||
check(
|
||||
`picked ${name} loads`,
|
||||
loaded &&
|
||||
(expected === null
|
||||
? item?.entityCount > 0
|
||||
: item?.entityCount === expected) &&
|
||||
state.sources === before + 1,
|
||||
JSON.stringify({
|
||||
bytes: fs.statSync(filename).size,
|
||||
loadMs: Math.round(performance.now() - started),
|
||||
...state,
|
||||
}),
|
||||
);
|
||||
return item;
|
||||
};
|
||||
const item = await loadPicked(file, 5000);
|
||||
if (item) {
|
||||
const same = await page.evaluate(() => {
|
||||
const viewer = window.__godsEyeView.viewer;
|
||||
const source = viewer.dataSources.get(viewer.dataSources.length - 1);
|
||||
return (
|
||||
JSON.stringify(window.__qaStyleSnapshot(source)) ===
|
||||
window.__qaUnsharedSnapshot
|
||||
);
|
||||
});
|
||||
check('all 5,000 normalized graphics match their unshared control', same);
|
||||
console.log('NORMALIZED CANDIDATE', JSON.stringify(await measure()));
|
||||
await page.evaluate((id) => window.__gevGeoImport.remove(id), item.id);
|
||||
}
|
||||
if (supplied) {
|
||||
const clockBefore = await page.evaluate(() => {
|
||||
const v = window.__godsEyeView.viewer,
|
||||
C = window.__CESIUM__;
|
||||
return [
|
||||
C.JulianDate.toIso8601(v.clock.startTime),
|
||||
C.JulianDate.toIso8601(v.clock.stopTime),
|
||||
v.clock.multiplier,
|
||||
];
|
||||
});
|
||||
const suppliedItem = await loadPicked(supplied, null);
|
||||
const clockAfter = await page.evaluate(() => {
|
||||
const v = window.__godsEyeView.viewer,
|
||||
C = window.__CESIUM__;
|
||||
return [
|
||||
C.JulianDate.toIso8601(v.clock.startTime),
|
||||
C.JulianDate.toIso8601(v.clock.stopTime),
|
||||
v.clock.multiplier,
|
||||
];
|
||||
});
|
||||
check(
|
||||
'real dated KML leaves the shared clock unchanged',
|
||||
JSON.stringify(clockBefore) === JSON.stringify(clockAfter),
|
||||
);
|
||||
if (suppliedItem)
|
||||
await page.evaluate(
|
||||
(id) => window.__gevGeoImport.remove(id),
|
||||
suppliedItem.id,
|
||||
);
|
||||
}
|
||||
check(
|
||||
'file-directed USGS icon/metadata requests never leave the page',
|
||||
fileRequests.length === 0,
|
||||
fileRequests.length,
|
||||
);
|
||||
check('no uncaught page errors', errors.length === 0, JSON.stringify(errors));
|
||||
check(
|
||||
'all imported sources are removed',
|
||||
await page.evaluate(() => window.__gevGeoImport.list().length === 0),
|
||||
);
|
||||
console.log('Fixture retained at', file);
|
||||
} finally {
|
||||
await browser.close();
|
||||
}
|
||||
process.exitCode = failures ? 1 : 0;
|
||||
+135
-11
@@ -42,7 +42,7 @@ export const MAX_IMPORTED_FILES = 12;
|
||||
export const MAX_IMPORT_ENTITIES = 20_000;
|
||||
/** Most vertices (points of every line, ring and marker) in one file. */
|
||||
export const MAX_IMPORT_POSITIONS = 250_000;
|
||||
/** Most `<href>` and `<Style>` elements in one KML/KMZ document. */
|
||||
/** Most normalized style definitions and local image links in one KML/KMZ. */
|
||||
export const MAX_KML_LINKS = 2_000;
|
||||
/** Longest plain-text name kept from a file's features. */
|
||||
export const MAX_IMPORT_NAME_LENGTH = 120;
|
||||
@@ -225,11 +225,11 @@ export function countGeoJsonGeometry(
|
||||
function overLimitError({ over, features, positions }) {
|
||||
if (over === 'entities')
|
||||
return new GeoImportError(
|
||||
`That file draws more than ${MAX_IMPORT_ENTITIES.toLocaleString()} objects (the limit).`,
|
||||
`That file creates more than ${MAX_IMPORT_ENTITIES.toLocaleString()} map objects (the limit).`,
|
||||
);
|
||||
if (over === 'links')
|
||||
return new GeoImportError(
|
||||
`That file has more than ${MAX_KML_LINKS.toLocaleString()} styles and icon links (the limit).`,
|
||||
`That file has more than ${MAX_KML_LINKS.toLocaleString()} style definitions and image links after cleanup (the limit).`,
|
||||
);
|
||||
if (over === 'features')
|
||||
return new GeoImportError(
|
||||
@@ -548,11 +548,118 @@ const childrenNamed = (el, localName) => {
|
||||
const childText = (el, localName) =>
|
||||
(childrenNamed(el, localName)[0]?.textContent || '').trim();
|
||||
|
||||
/**
|
||||
* Share identical styles before the loader builds their graphics. Inline
|
||||
* overrides stay inline StyleMaps, so an existing styleUrl still supplies
|
||||
* properties the override leaves unset. Named references are redirected.
|
||||
* Random-color styles remain unchanged; duplicate IDs
|
||||
* keep the loader's original precedence instead of being rewritten.
|
||||
* @param {Document} doc Sanitized KML.
|
||||
* @param {{serializeStyle?: Function, maxStyles?: number}} [options]
|
||||
* @returns {number} Number of original style blocks replaced.
|
||||
*/
|
||||
export function deduplicateKmlStyles(
|
||||
doc,
|
||||
{ serializeStyle, maxStyles = MAX_KML_LINKS } = {},
|
||||
) {
|
||||
const serializer = serializeStyle ? null : new XMLSerializer();
|
||||
const serialize =
|
||||
serializeStyle || ((el) => serializer.serializeToString(el));
|
||||
const groups = new Map();
|
||||
const usedIds = new Set();
|
||||
const duplicateIds = new Set();
|
||||
for (const el of doc.getElementsByTagName('*')) {
|
||||
const id = el.getAttribute('id');
|
||||
if (id !== null) {
|
||||
if (usedIds.has(id)) duplicateIds.add(id);
|
||||
usedIds.add(id);
|
||||
}
|
||||
}
|
||||
for (const style of elementsNamed(doc, 'Style')) {
|
||||
if (
|
||||
elementsNamed(style, 'colorMode').some(
|
||||
(el) => el.textContent.trim() === 'random',
|
||||
)
|
||||
)
|
||||
continue;
|
||||
const template = style.cloneNode(true);
|
||||
template.removeAttribute('id');
|
||||
const key = serialize(template);
|
||||
let group = groups.get(key);
|
||||
if (!group) {
|
||||
group = { template, nodes: [] };
|
||||
groups.set(key, group);
|
||||
if (groups.size > maxStyles) throw overLimitError({ over: 'links' });
|
||||
}
|
||||
group.nodes.push(style);
|
||||
}
|
||||
const redirects = new Map();
|
||||
let replaced = 0;
|
||||
let serial = 0;
|
||||
let container;
|
||||
for (const { template, nodes } of groups.values()) {
|
||||
if (
|
||||
nodes.length < 2 ||
|
||||
nodes.some((el) => duplicateIds.has(el.getAttribute('id')))
|
||||
)
|
||||
continue;
|
||||
const ns = template.namespaceURI || 'http://www.opengis.net/kml/2.2';
|
||||
if (!container) {
|
||||
const root = doc.documentElement;
|
||||
container =
|
||||
root.localName === 'Document'
|
||||
? root
|
||||
: childrenNamed(root, 'Document')[0];
|
||||
if (!container) {
|
||||
container = doc.createElementNS(ns, 'Document');
|
||||
while (root.firstChild) container.appendChild(root.firstChild);
|
||||
root.appendChild(container);
|
||||
}
|
||||
}
|
||||
let id;
|
||||
do {
|
||||
id = `gev-import-style-${++serial}`;
|
||||
} while (usedIds.has(id));
|
||||
usedIds.add(id);
|
||||
template.setAttribute('id', id);
|
||||
container.appendChild(template);
|
||||
for (const style of nodes) {
|
||||
const originalId = style.getAttribute('id');
|
||||
if (originalId !== null) redirects.set(originalId, id);
|
||||
let reference;
|
||||
if (style.parentNode.localName === 'Pair') {
|
||||
reference = doc.createElementNS(ns, 'styleUrl');
|
||||
reference.textContent = `#${id}`;
|
||||
} else {
|
||||
reference = doc.createElementNS(ns, 'StyleMap');
|
||||
for (const state of ['normal', 'highlight']) {
|
||||
const pair = doc.createElementNS(ns, 'Pair');
|
||||
const key = doc.createElementNS(ns, 'key');
|
||||
key.textContent = state;
|
||||
const url = doc.createElementNS(ns, 'styleUrl');
|
||||
url.textContent = `#${id}`;
|
||||
pair.appendChild(key);
|
||||
pair.appendChild(url);
|
||||
reference.appendChild(pair);
|
||||
}
|
||||
}
|
||||
style.parentNode.replaceChild(reference, style);
|
||||
replaced += 1;
|
||||
}
|
||||
}
|
||||
for (const url of elementsNamed(doc, 'styleUrl')) {
|
||||
const value = url.textContent.trim();
|
||||
const id = value.startsWith('#') && redirects.get(value.slice(1));
|
||||
if (id) url.textContent = `#${id}`;
|
||||
}
|
||||
return replaced;
|
||||
}
|
||||
|
||||
/**
|
||||
* Count placemarks and vertices in a parsed KML document before Cesium reads
|
||||
* it: every `<coordinates>` tuple and every `gx:coord`, plus the `<href>`
|
||||
* and `<Style>` elements (`links`), each of which can make Cesium load and
|
||||
* decode an image.
|
||||
* it: every `<coordinates>` tuple and every `gx:coord`, plus normalized
|
||||
* style definitions and distinct image `<href>` values (`links`).
|
||||
* Inline StyleMaps are references, not additional shared style definitions.
|
||||
* @param {Document} doc
|
||||
* @returns {{features: number, entities: number, positions: number, links: number,
|
||||
* over: ''|'features'|'entities'|'positions'|'links'}}
|
||||
@@ -570,6 +677,7 @@ export function countKmlGeometry(
|
||||
let entities = 0;
|
||||
let positions = 0;
|
||||
let links = 0;
|
||||
const imageLinks = new Set();
|
||||
const result = (over) => ({ features, entities, positions, links, over });
|
||||
const geometryNames = new Set([
|
||||
'Point',
|
||||
@@ -583,10 +691,11 @@ export function countKmlGeometry(
|
||||
]);
|
||||
for (const el of doc.getElementsByTagName('*')) {
|
||||
const name = el.localName;
|
||||
// Cesium creates a parent for every placemark and a child entity for
|
||||
// each geometry in a MultiGeometry, including nested containers.
|
||||
// Cesium creates entities for folders, a parent for every placemark, and
|
||||
// a child for each geometry in a MultiGeometry, including nested containers.
|
||||
if (
|
||||
name === 'Placemark' ||
|
||||
name === 'Folder' ||
|
||||
name === 'GroundOverlay' ||
|
||||
(geometryNames.has(name) &&
|
||||
['MultiGeometry', 'MultiTrack'].includes(el.parentNode?.localName))
|
||||
@@ -594,9 +703,19 @@ export function countKmlGeometry(
|
||||
entities += 1;
|
||||
if (entities > maxEntities) return result('entities');
|
||||
}
|
||||
if (name === 'href' || name === 'Style') {
|
||||
if (
|
||||
name === 'Style' ||
|
||||
(name === 'StyleMap' && el.getAttribute('id') !== null)
|
||||
) {
|
||||
links += 1;
|
||||
if (links > maxLinks) return result('links');
|
||||
} else if (name === 'href') {
|
||||
const href = el.textContent.trim();
|
||||
if (!imageLinks.has(href)) {
|
||||
imageLinks.add(href);
|
||||
links += 1;
|
||||
if (links > maxLinks) return result('links');
|
||||
}
|
||||
} else if (name === 'Placemark') {
|
||||
features += 1;
|
||||
if (features > maxFeatures) return result('features');
|
||||
@@ -888,9 +1007,14 @@ export async function loadGeoFile(file, { viewer, color = '#39d0ff' }) {
|
||||
text = new TextDecoder().decode(bytes);
|
||||
}
|
||||
const doc = parseXml(text, format.toUpperCase());
|
||||
const size = countKmlGeometry(doc);
|
||||
// Bound geometry before transformations, then count the styles and
|
||||
// image links that survive sanitizing and sharing repeated styles.
|
||||
const size = countKmlGeometry(doc, { maxLinks: Infinity });
|
||||
if (size.over) throw overLimitError(size);
|
||||
removed = sanitizeKmlDocument(doc, resolveLocal).removed;
|
||||
deduplicateKmlStyles(doc);
|
||||
const normalizedSize = countKmlGeometry(doc);
|
||||
if (normalizedSize.over) throw overLimitError(normalizedSize);
|
||||
dataSource = await Cesium.KmlDataSource.load(doc, {
|
||||
camera: viewer.scene.camera,
|
||||
canvas: viewer.scene.canvas,
|
||||
@@ -908,7 +1032,7 @@ export async function loadGeoFile(file, { viewer, color = '#39d0ff' }) {
|
||||
if (entityCount > MAX_IMPORT_ENTITIES) {
|
||||
revoke();
|
||||
throw new GeoImportError(
|
||||
`That file has ${entityCount.toLocaleString()} features; the limit is ${MAX_IMPORT_ENTITIES.toLocaleString()}.`,
|
||||
`That file has ${entityCount.toLocaleString()} map objects; the limit is ${MAX_IMPORT_ENTITIES.toLocaleString()}.`,
|
||||
);
|
||||
}
|
||||
if (entityCount === 0) {
|
||||
|
||||
@@ -8,6 +8,7 @@ import {
|
||||
MAX_IMPORT_ENTITIES,
|
||||
countGeoJsonGeometry,
|
||||
countKmlGeometry,
|
||||
deduplicateKmlStyles,
|
||||
detachImportedTime,
|
||||
gpxToGeoJson,
|
||||
importFeatureName,
|
||||
@@ -231,8 +232,41 @@ function xmlEl(localName, attrs = {}, kids = []) {
|
||||
const el = {
|
||||
nodeType: 1,
|
||||
localName,
|
||||
namespaceURI: 'http://www.opengis.net/kml/2.2',
|
||||
childNodes: [],
|
||||
getAttribute: (n) => (n in attrs ? String(attrs[n]) : null),
|
||||
get attributes() {
|
||||
return Object.entries(attrs);
|
||||
},
|
||||
setAttribute: (n, value) => {
|
||||
attrs[n] = value;
|
||||
},
|
||||
removeAttribute: (n) => {
|
||||
delete attrs[n];
|
||||
},
|
||||
cloneNode() {
|
||||
return xmlEl(
|
||||
localName,
|
||||
{ ...attrs },
|
||||
text ?? el.childNodes.map((child) => child.cloneNode(true)),
|
||||
);
|
||||
},
|
||||
getElementsByTagName(name) {
|
||||
return xmlDoc(this).getElementsByTagName(name).slice(1);
|
||||
},
|
||||
appendChild(child) {
|
||||
child.parentNode?.removeChild(child);
|
||||
child.parentNode = this;
|
||||
this.childNodes.push(child);
|
||||
},
|
||||
replaceChild(next, previous) {
|
||||
const index = this.childNodes.indexOf(previous);
|
||||
assert.ok(index >= 0);
|
||||
next.parentNode?.removeChild(next);
|
||||
this.childNodes[index] = next;
|
||||
next.parentNode = this;
|
||||
previous.parentNode = null;
|
||||
},
|
||||
removeChild(child) {
|
||||
this.childNodes.splice(this.childNodes.indexOf(child), 1);
|
||||
child.parentNode = null;
|
||||
@@ -242,14 +276,22 @@ function xmlEl(localName, attrs = {}, kids = []) {
|
||||
k.parentNode = el;
|
||||
el.childNodes.push(k);
|
||||
}
|
||||
const text = typeof kids === 'string' ? kids : null;
|
||||
let text = typeof kids === 'string' ? kids : null;
|
||||
Object.defineProperty(el, 'firstChild', {
|
||||
get: () => el.childNodes[0] || null,
|
||||
});
|
||||
Object.defineProperty(el, 'textContent', {
|
||||
get: () => text ?? el.childNodes.map((c) => c.textContent).join(''),
|
||||
set: (value) => {
|
||||
text = value;
|
||||
el.childNodes = [];
|
||||
},
|
||||
});
|
||||
return el;
|
||||
}
|
||||
const xmlDoc = (root) => ({
|
||||
documentElement: root,
|
||||
createElementNS: (_ns, name) => xmlEl(name),
|
||||
getElementsByTagName(name) {
|
||||
assert.equal(name, '*');
|
||||
const out = [];
|
||||
@@ -261,6 +303,118 @@ const xmlDoc = (root) => ({
|
||||
return out;
|
||||
},
|
||||
});
|
||||
const serializeStyle = (el) =>
|
||||
JSON.stringify({
|
||||
name: el.localName,
|
||||
attributes: el.attributes,
|
||||
text: el.childNodes.length ? null : el.textContent,
|
||||
children: el.childNodes.map(serializeStyle),
|
||||
});
|
||||
|
||||
test('repeated inline styles are shared without replacing their inherited styleUrl', () => {
|
||||
const style = () =>
|
||||
xmlEl('Style', {}, [xmlEl('IconStyle', {}, [xmlEl('scale', {}, '0.6')])]);
|
||||
const placemarks = Array.from({ length: 2100 }, () =>
|
||||
xmlEl('Placemark', {}, [xmlEl('styleUrl', {}, '#base'), style()]),
|
||||
);
|
||||
const doc = xmlDoc(
|
||||
xmlEl('kml', {}, [
|
||||
xmlEl('Document', {}, [
|
||||
xmlEl('Style', { id: 'gev-import-style-1' }, [xmlEl('LineStyle')]),
|
||||
...placemarks,
|
||||
]),
|
||||
]),
|
||||
);
|
||||
assert.equal(countKmlGeometry(doc).over, 'links');
|
||||
assert.equal(deduplicateKmlStyles(doc, { serializeStyle }), 2100);
|
||||
assert.equal(countKmlGeometry(doc).over, '');
|
||||
const styles = doc
|
||||
.getElementsByTagName('*')
|
||||
.filter((el) => el.localName === 'Style');
|
||||
assert.equal(styles.length, 2);
|
||||
assert.equal(styles[1].getAttribute('id'), 'gev-import-style-2');
|
||||
for (const placemark of placemarks) {
|
||||
assert.equal(placemark.childNodes[0].textContent, '#base');
|
||||
assert.equal(placemark.childNodes[1].localName, 'StyleMap');
|
||||
assert.deepEqual(
|
||||
placemark.childNodes[1].childNodes.map((pair) =>
|
||||
pair.childNodes.map((el) => el.textContent),
|
||||
),
|
||||
[
|
||||
['normal', '#gev-import-style-2'],
|
||||
['highlight', '#gev-import-style-2'],
|
||||
],
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('identical named styles redirect existing references and Pair styles remain valid', () => {
|
||||
const style = (id) =>
|
||||
xmlEl('Style', id ? { id } : {}, [
|
||||
xmlEl('LineStyle', {}, [xmlEl('width', {}, '3')]),
|
||||
]);
|
||||
const reference = xmlEl('styleUrl', {}, '#b');
|
||||
const pair = xmlEl('Pair', {}, [xmlEl('key', {}, 'normal'), style()]);
|
||||
const doc = xmlDoc(
|
||||
xmlEl('kml', {}, [
|
||||
style('a'),
|
||||
style('b'),
|
||||
xmlEl('StyleMap', { id: 'map' }, [pair]),
|
||||
xmlEl('Placemark', {}, [reference]),
|
||||
]),
|
||||
);
|
||||
assert.equal(deduplicateKmlStyles(doc, { serializeStyle }), 3);
|
||||
assert.equal(doc.documentElement.firstChild.localName, 'Document');
|
||||
assert.equal(reference.textContent, '#gev-import-style-1');
|
||||
assert.equal(pair.childNodes[1].localName, 'styleUrl');
|
||||
assert.equal(pair.childNodes[1].textContent, '#gev-import-style-1');
|
||||
assert.equal(
|
||||
doc.getElementsByTagName('*').filter((el) => el.localName === 'Style')
|
||||
.length,
|
||||
1,
|
||||
);
|
||||
});
|
||||
|
||||
test('random colors and ambiguous style IDs retain independent evaluations and precedence', () => {
|
||||
const random = () =>
|
||||
xmlEl('Style', {}, [
|
||||
xmlEl('IconStyle', {}, [xmlEl('colorMode', {}, 'random')]),
|
||||
]);
|
||||
const named = (width) =>
|
||||
xmlEl('Style', { id: 'collision' }, [
|
||||
xmlEl('LineStyle', {}, [xmlEl('width', {}, width)]),
|
||||
]);
|
||||
const original = [random(), random(), named('1'), named('1'), named('2')];
|
||||
const doc = xmlDoc(xmlEl('kml', {}, original));
|
||||
assert.equal(deduplicateKmlStyles(doc, { serializeStyle }), 0);
|
||||
assert.deepEqual(doc.documentElement.childNodes, original);
|
||||
});
|
||||
|
||||
test('distinct style definitions and named StyleMaps still have a pre-loader budget', () => {
|
||||
const doc = xmlDoc(
|
||||
xmlEl(
|
||||
'kml',
|
||||
{},
|
||||
['1', '2', '3'].map((width) =>
|
||||
xmlEl('Style', {}, [
|
||||
xmlEl('LineStyle', {}, [xmlEl('width', {}, width)]),
|
||||
]),
|
||||
),
|
||||
),
|
||||
);
|
||||
assert.throws(
|
||||
() => deduplicateKmlStyles(doc, { serializeStyle, maxStyles: 2 }),
|
||||
GeoImportError,
|
||||
);
|
||||
const maps = xmlDoc(
|
||||
xmlEl('kml', {}, [
|
||||
xmlEl('StyleMap', { id: 'a' }),
|
||||
xmlEl('StyleMap', { id: 'b' }),
|
||||
]),
|
||||
);
|
||||
assert.equal(countKmlGeometry(maps, { maxLinks: 1 }).over, 'links');
|
||||
});
|
||||
|
||||
test('KML extended metadata is removed before the document reaches the loader', () => {
|
||||
const point = xmlEl('Point', {}, [xmlEl('coordinates', {}, '1,2,0')]);
|
||||
const metadata = xmlEl('ExtendedData', {}, [
|
||||
@@ -495,14 +649,32 @@ test('KML multipart geometries count every Cesium entity before loading', () =>
|
||||
assert.equal(countKmlGeometry(oversized).over, 'entities');
|
||||
});
|
||||
|
||||
test('KML styles and icon links are counted against their own limit', () => {
|
||||
test('KML folders contribute to the pre-loader entity budget', () => {
|
||||
const doc = xmlDoc(
|
||||
xmlEl('kml', {}, [
|
||||
xmlEl('Folder', {}, [xmlEl('Folder', {}, [xmlEl('Placemark')])]),
|
||||
]),
|
||||
);
|
||||
assert.equal(countKmlGeometry(doc).entities, 3);
|
||||
assert.equal(countKmlGeometry(doc, { maxEntities: 2 }).over, 'entities');
|
||||
});
|
||||
|
||||
test('KML style definitions and distinct icon links have their own limit', () => {
|
||||
const style = () =>
|
||||
xmlEl('Style', {}, [
|
||||
xmlEl('IconStyle', {}, [xmlEl('Icon', {}, [xmlEl('href', {}, 'a.png')])]),
|
||||
]);
|
||||
const doc = xmlDoc(xmlEl('kml', {}, [style(), style(), style()]));
|
||||
assert.equal(countKmlGeometry(doc).links, 6);
|
||||
assert.equal(countKmlGeometry(doc, { maxLinks: 5 }).over, 'links');
|
||||
assert.equal(countKmlGeometry(doc).links, 4);
|
||||
assert.equal(countKmlGeometry(doc, { maxLinks: 3 }).over, 'links');
|
||||
const distinct = xmlDoc(
|
||||
xmlEl(
|
||||
'kml',
|
||||
{},
|
||||
['a.png', 'b.png', 'c.png'].map((href) => xmlEl('href', {}, href)),
|
||||
),
|
||||
);
|
||||
assert.equal(countKmlGeometry(distinct, { maxLinks: 2 }).over, 'links');
|
||||
});
|
||||
|
||||
test('a KMZ image gets one blob URL however many hrefs name it', () => {
|
||||
|
||||
@@ -409,6 +409,6 @@ export function summarize(outcomes) {
|
||||
0,
|
||||
);
|
||||
if (added && stripped)
|
||||
parts.push('Network links and remote images in KML were left out.');
|
||||
parts.push('KML descriptions and external references were left out.');
|
||||
return parts.join(' ');
|
||||
}
|
||||
@@ -347,7 +347,7 @@ test('dropping files on the globe imports them; other drags are left alone', asy
|
||||
test('stripped KML content is mentioned once, for the batch that had it', () => {
|
||||
assert.equal(
|
||||
summarize([{ name: 'a.kml', ok: true, removed: 3 }]),
|
||||
'Added 1 file. Network links and remote images in KML were left out.',
|
||||
'Added 1 file. KML descriptions and external references were left out.',
|
||||
);
|
||||
assert.equal(
|
||||
summarize([{ name: 'b.geojson', ok: true, removed: 0 }]),
|
||||
|
||||
Reference in new issue
Block a user