test(street-level): run the photo flow in CI against a production build

- The CI gate builds the app and serves it with `vite preview`, with a
  dummy non-secret client token (a fork PR never gets secrets), and
  answers the Graph API, image tiles and thumbnails from fixtures: 41
  photos on a line under the parked camera and one generated 640x320 JPEG
  (scripts/fixtures/street-level). All 7 photo steps now run in CI, plus a
  real click on a coverage line and on a cone, the framing flight stopping
  on close, the inert modal, the viewer returning into the panel, and a
  reload keeping a collapsed panel collapsed (share ui token t).
- Fixture interception is limited to Mapillary hosts via CDP
  Fetch.enable: page-wide request interception stalled Cesium's workers,
  so the old fixture mode never drew anything on the globe.
- Failure evidence (screenshots, UI state, console, failed requests,
  render errors) is written to qa-artifacts/ with keys and tokens redacted
  and uploaded on failure; qa-artifacts/ is gitignored.
- Cesium render-loop errors fail the gate; Google 3D branches that cannot
  run print as skips, and --strict rejects any other skip.
- Panel presses log their diagnostics, warn on a retry, and CI now fails
  on any retry (QA_FAIL_ON_RETRY=1) since the rail settles; a press with
  no pointerdown is a failure, not a retry.
- Assertion-gating sleeps 27 -> 1; the rejected-key "no requests" check
  has a positive control.

Verified locally: strict fixture gate on the production build 28 ok + 2
"no Google 3D" skips (twice, fail-on-retry on); panel gate 5/5 plus no
render errors; live gate with a real token 25 ok + 3 fixture-only skips.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Said TurkseverandClaude Opus 5.5 committed 2026-10-06 16:54:06 +01:00
1 parent 3af0f80cdb
commit fb024d582b
10 files changed
+2468 -800

No files matched your search

+35 -11
View File
@@ -60,15 +60,22 @@ jobs:
run: npm run build
street-level-browser:
# The Street Level gate in its hermetic mode: Mapillary status and coverage
# tiles are answered from generated fixtures, so no token or Mapillary
# network is needed. The steps that open a photo need live imagery and
# are skipped here; run `npm run qa:street-level` with a token for them.
# The Street Level gate in its hermetic mode, against a production build
# served by `vite preview`: coverage tiles, the Graph API and the image
# CDN are answered from fixtures (scripts/fixtures/street-level), so no
# real token and no Mapillary network are needed, and the whole photo
# flow runs. The server gets a dummy, non-secret client token: the build
# bakes it into the bundle (the viewer needs one to ask for photos) and
# the preview server's status route reports it as configured. The page
# intercepts every *.mapillary.com request, so the token never leaves it.
name: Street Level browser gate
runs-on: ubuntu-latest
timeout-minutes: 20
timeout-minutes: 30
env:
GEV_REQUIRE_ALLOCATION_GATE: '0'
# A header press that misses is a real layout shift under the pointer
# (the right rail now aligns to where the left rail settles), not noise.
QA_FAIL_ON_RETRY: '1'
steps:
- name: Check out source
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
@@ -84,25 +91,42 @@ jobs:
- name: Install locked dependencies and headless Chrome
run: npm ci
- name: Start the dev server
- name: Build production bundle with a dummy Mapillary token
env:
MAPILLARY_CLIENT_TOKEN: 'MLY|0|qa-fixture'
run: npm run build
- name: Start the preview server
env:
MAPILLARY_CLIENT_TOKEN: 'MLY|0|qa-fixture'
run: |
npx vite --port 4173 --strictPort > vite.log 2>&1 &
npx vite preview --port 4173 --strictPort > preview.log 2>&1 &
for attempt in $(seq 1 90); do
curl -sf http://localhost:4173/ > /dev/null && exit 0
sleep 2
done
cat vite.log
cat preview.log
exit 1
- name: Street Level gate with Mapillary fixtures
run: npm run qa:street-level:fixtures -- --url http://localhost:4173
run: npm run qa:street-level:fixtures -- --strict --url http://localhost:4173
- name: Portable panel gate
if: success() || failure()
run: npm run qa:panel-resize -- --url http://localhost:4173
- name: Dev server log
- name: Preview server log
if: failure()
run: cat vite.log
run: cat preview.log
- name: Upload gate evidence
if: failure()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: browser-gate-evidence
path: qa-artifacts/
if-no-files-found: ignore
retention-days: 14
windows-onboarding:
name: Windows onboarding
+2
View File
@@ -14,6 +14,8 @@ qa-shots/
output/
.gev-logs/
.gev-cache/
# Browser QA gate evidence (screenshots, console and network dumps).
qa-artifacts/
.DS_Store
.gstack/
3d-models/
@@ -0,0 +1,242 @@
import { readFileSync } from 'node:fs';
/**
* Mapillary Graph API and image CDN, answered from fixtures, so the Street
* Level gate's photo flow runs with no network and no real token (the CI
* mode). It covers every request the app's provider and the embedded
* MapillaryJS 4 viewer make while opening, stepping and closing a photo:
*
* provider (src/layers/streetLevel/providers/mapillary/source.js)
* GET graph /images?lat&lng&radius&limit&fields nearest lookup
* GET graph /images?sequence_ids&fields&limit a sequence's cones
* MapillaryJS GraphDataProvider (node_modules/mapillary-js)
* GET graph /images?image_ids&fields image ents (core + spatial)
* GET graph /images?s2&fields core images of an S2 cell
* GET graph /image_ids?sequence_id a sequence's image ids
* GET graph /{imageId}/tiles?z&fields image tiles (imageTiling)
* GET thumb_1024_url / thumb_2048_url the photo itself
* and the CORS preflight of each Graph call (MapillaryJS sends an
* Authorization header).
*
* MapillaryJS never asks for a mesh or an SfM cluster here: the fixtures have
* no `merge_cc` (an unmerged image gets an empty mesh without a request) and
* no `sfm_cluster` url, and the spatial component that reads clusters is off.
* S2 cells answer empty: the viewer then builds no spatial edges, which the
* gate does not use (it steps along the sequence through the cones).
*
* The photo (street-640x320.jpg) is generated for this repository (a sky and
* ground gradient with posts; scripts/fixtures/street-level), so it carries
* no third-party licence.
*/
const DAY_MS = 86_400_000;
/** The one sequence with photos: a north-south line through the parked view. */
export const PHOTO_SEQUENCE_ID = 'fx-photo';
export const PHOTO_LINE = Object.freeze({
lon: -121.4944,
south: 38.579,
north: 38.591,
});
/** Metres between consecutive photos on the line (≥ the 3 m cone thinning). */
export const PHOTO_SPACING_M = 30;
/** Host the fixture thumbnails are served from (answered, never reached). */
export const THUMB_HOST = 'qa-fixture.mapillary.com';
const METRES_PER_DEG_LAT = 110_540;
/** Approximate ground distance in metres, as the app's metresBetween. */
export function metresApart(a, b) {
const lat = (((a.lat + b.lat) / 2) * Math.PI) / 180;
return Math.hypot(
(b.lon - a.lon) * 111_320 * Math.cos(lat),
(b.lat - a.lat) * METRES_PER_DEG_LAT,
);
}
/**
* The fixture photos along PHOTO_LINE, south to north: every third one a
* 360° panorama, the rest flat, all captured a month before `now`.
* @returns {Array<{id: string, lon: number, lat: number, isPano: boolean, capturedAt: number, compassAngle: number}>}
*/
export function photoImages(now = Date.now()) {
const step = PHOTO_SPACING_M / METRES_PER_DEG_LAT;
const count =
Math.floor((PHOTO_LINE.north - PHOTO_LINE.south) / step + 1e-9) + 1;
return Array.from({ length: count }, (_, i) => ({
id: String(9_100_000_000_000 + i),
lon: PHOTO_LINE.lon,
lat: Number((PHOTO_LINE.south + i * step).toFixed(7)),
isPano: i % 3 === 0,
capturedAt: now - 30 * DAY_MS - i * 1000,
compassAngle: 0,
}));
}
const point = (image) => ({
type: 'Point',
coordinates: [image.lon, image.lat],
});
/** Every field either caller asks for; extra fields are harmless to both. */
function imageRecord(image) {
const thumb = `https://${THUMB_HOST}/thumb/${image.id}.jpg`;
return {
id: image.id,
sequence: PHOTO_SEQUENCE_ID,
geometry: point(image),
computed_geometry: point(image),
altitude: 12,
computed_altitude: 12,
atomic_scale: 1,
camera_parameters: image.isPano ? [] : [0.85, 0, 0],
camera_type: image.isPano ? 'spherical' : 'perspective',
captured_at: image.capturedAt,
compass_angle: image.compassAngle,
computed_compass_angle: image.compassAngle,
// Angle-axis world-to-camera rotation: +90° about east looks north, level.
computed_rotation: [Math.PI / 2, 0, 0],
creator: { id: '1', username: 'qa-fixture' },
exif_orientation: 1,
height: 320,
width: 640,
is_pano: image.isPano,
merge_cc: null,
mesh: null,
organization: null,
quality_score: 0.9,
sfm_cluster: null,
thumb_256_url: thumb,
thumb_1024_url: thumb,
thumb_2048_url: thumb,
};
}
/** Pick the requested fields (plus id), as the Graph API answers. */
function withFields(record, fields) {
if (!fields) return record;
const out = { id: record.id };
for (const field of fields.split(','))
if (field in record) out[field] = record[field];
return out;
}
const CORS = Object.freeze({
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Methods': 'GET, OPTIONS',
'Access-Control-Allow-Headers': 'Accept, Authorization, Content-Type',
'Access-Control-Max-Age': '600',
});
const json = (status, body) => ({
status,
contentType: 'application/json',
headers: CORS,
body: JSON.stringify(body),
});
let photoBytes = null;
/** The checked-in fixture photo, read once. */
export function photoJpeg() {
photoBytes ??= readFileSync(new URL('./street-640x320.jpg', import.meta.url));
return photoBytes;
}
/**
* The query parameters that name a Graph call (not their values), so a run
* can report which calls it answered without echoing the access token.
*/
export function describeCall(method, url) {
const keys = [...url.searchParams.keys()]
.filter((key) => key !== 'access_token')
.sort();
const path = url.pathname
.replace(/^\/\d{6,}(?=\/|$)/, '/{imageId}')
.replace(/^\/thumb\/[\w-]+\.jpg$/, '/thumb/{imageId}.jpg');
return `${method} ${url.hostname}${path}${keys.length ? `?${keys.join('&')}` : ''}`;
}
/**
* Answer one Mapillary request from fixtures.
* @param {{method: string, url: string, images?: ReturnType<typeof photoImages>}} request
* @returns {{status: number, contentType?: string, headers?: object, body: string|Buffer, known: boolean}}
* `known` is false for a request no fixture covers (answered 404); the gate
* fails on those so a new MapillaryJS call cannot pass unnoticed.
*/
export function answerMapillaryRequest({ method, url, images }) {
const address = new URL(url);
const photos = images ?? photoImages();
const byId = new Map(photos.map((image) => [image.id, image]));
if (method === 'OPTIONS')
return { status: 204, headers: CORS, body: '', known: true };
if (address.hostname === THUMB_HOST) {
if (/^\/thumb\/[\w-]+\.jpg$/.test(address.pathname))
return {
status: 200,
contentType: 'image/jpeg',
headers: CORS,
body: photoJpeg(),
known: true,
};
return { ...json(404, { error: 'no such fixture' }), known: false };
}
if (address.hostname !== 'graph.mapillary.com')
return { ...json(404, { error: 'no such fixture host' }), known: false };
const q = address.searchParams;
const fields = q.get('fields');
const path = address.pathname;
if (path === '/images') {
if (q.has('image_ids')) {
const data = q
.get('image_ids')
.split(',')
.map((id) => byId.get(id))
.filter(Boolean)
.map((image) => withFields(imageRecord(image), fields));
return { ...json(200, { data }), known: true };
}
if (q.has('sequence_ids')) {
const ids = q.get('sequence_ids').split(',');
const limit = Number(q.get('limit')) || photos.length;
const data = ids.includes(PHOTO_SEQUENCE_ID)
? photos
.slice(0, limit)
.map((image) => withFields(imageRecord(image), fields))
: [];
return { ...json(200, { data }), known: true };
}
if (q.has('lat') && q.has('lng')) {
const at = { lat: Number(q.get('lat')), lon: Number(q.get('lng')) };
const radius = Math.min(50, Number(q.get('radius')) || 50);
const limit = Number(q.get('limit')) || 10;
// In no particular order, as the real API answers.
const data = photos
.filter((image) => metresApart(at, image) <= radius)
.reverse()
.slice(0, limit)
.map((image) => withFields(imageRecord(image), fields));
return { ...json(200, { data }), known: true };
}
if (q.has('s2')) return { ...json(200, { data: [] }), known: true };
return {
...json(400, { error: 'unsupported images query' }),
known: false,
};
}
if (path === '/image_ids' && q.has('sequence_id')) {
const data =
q.get('sequence_id') === PHOTO_SEQUENCE_ID
? photos.map((image) => ({ id: image.id }))
: [];
return { ...json(200, { data }), known: true };
}
const tiles = path.match(/^\/([\w-]+)\/tiles$/);
if (tiles) return { ...json(200, { data: [] }), known: true };
const entity = path.match(/^\/([\w-]+)$/);
if (entity && byId.has(entity[1]))
return {
...json(200, withFields(imageRecord(byId.get(entity[1])), fields)),
known: true,
};
return { ...json(404, { error: 'no such fixture' }), known: false };
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 3.6 KiB

+2
View File
@@ -901,6 +901,8 @@
"src/layers/streetLevel/tileMath.test.mjs",
"scripts/qa-street-level.mjs",
"src/tooling/streetLevelQa.test.mjs",
"scripts/qa-browserEvidence.mjs",
"scripts/fixtures/street-level/mapillaryGraph.mjs",
"src/layers/streetLevel/filter.test.mjs",
"src/layers/streetLevel/registry.test.mjs",
"src/layers/streetLevel/params.test.mjs",
+189
View File
@@ -0,0 +1,189 @@
/**
* Evidence the browser QA gates (qa-street-level, qa-panel-resize) collect
* while they run and save when they fail: every page's console (last 200
* messages), failed requests and HTTP errors, Cesium render-loop errors (the
* "An error occurred while rendering" console line and `scene.renderError`)
* and, on failure, a full-page screenshot of every open page plus the Street
* Level UI state, under `qa-artifacts/<gate>/` (CI uploads that directory).
*
* Keys never reach the evidence: URLs lose their `access_token`, `key` (and
* similar) values, and `MLY|…` client tokens and Google `AIza…` keys are
* masked wherever they appear.
*/
import fs from 'node:fs/promises';
import path from 'node:path';
/** How many console messages and failed requests each page keeps. */
export const EVIDENCE_LIMIT = 200;
/** Cesium's console line when the render loop throws (CesiumWidget). */
export const RENDER_ERROR_PATTERN = /An error occurred while rendering/i;
/** Mask tokens in a URL or message before it is printed or saved. */
export function redact(text) {
return String(text ?? '')
.replace(
/([?&](?:access_token|token|key|api_key|apikey|client_secret|signature)=)[^&\s"']*/gi,
'$1REDACTED',
)
.replace(/MLY\|[^\s&"',)]+/g, 'MLY|REDACTED')
.replace(/AIza[\w-]{30,}/g, 'AIza…REDACTED');
}
function push(list, entry) {
list.push(entry);
if (list.length > EVIDENCE_LIMIT)
list.splice(0, list.length - EVIDENCE_LIMIT);
}
/**
* Subscribe to `viewer.scene.renderError` in every document the page loads
* (reloads included), into `window.__qaRenderErrors`. Call before `goto`.
*/
export function hookRenderErrors(page) {
return page.evaluateOnNewDocument(() => {
window.__qaRenderErrors = [];
const hook = () => {
const scene = window.__godsEyeView?.viewer?.scene;
if (!scene?.renderError) return false;
scene.renderError.addEventListener((_, error) =>
window.__qaRenderErrors.push(
String(error?.stack || error?.message || error),
),
);
return true;
};
const timer = setInterval(() => {
if (hook()) clearInterval(timer);
}, 250);
});
}
/** Render errors the in-page hook saw in the current document. */
export async function readRenderErrors(page) {
try {
return await page.evaluate(() => window.__qaRenderErrors ?? []);
} catch {
return [];
}
}
/**
* Watch one page: console, page errors, failed requests and HTTP errors.
* `errors` (shared across pages) receives page errors, render-loop console
* errors and the extra `isError(text)` console lines a gate cares about.
* @returns {{name: string, page: object, console: Array<object>, failed: Array<object>, renderErrors: Array<string>}}
*/
export function watchPage(page, { name, errors, isError = () => false }) {
const monitor = { name, page, console: [], failed: [], renderErrors: [] };
page.on('console', (message) => {
const text = redact(message.text());
push(monitor.console, { type: message.type(), text });
if (message.type() === 'error' && RENDER_ERROR_PATTERN.test(text)) {
monitor.renderErrors.push(text);
errors.push(`[${name}] render loop: ${text}`);
} else if (isError(text)) errors.push(`[${name}] ${text}`);
});
page.on('pageerror', (error) => {
const text = redact(error?.stack || error?.message || error);
push(monitor.console, { type: 'pageerror', text });
errors.push(`[${name}] ${text}`);
});
page.on('requestfailed', (request) =>
push(monitor.failed, {
url: redact(request.url()),
method: request.method(),
failure: request.failure()?.errorText ?? 'failed',
}),
);
page.on('response', (response) => {
if (response.status() < 400) return;
push(monitor.failed, {
url: redact(response.url()),
method: response.request().method(),
status: response.status(),
});
});
return monitor;
}
const withTimeout = (promise, ms, label) =>
Promise.race([
promise,
new Promise((_, reject) =>
setTimeout(() => reject(new Error(`${label} timed out`)), ms),
),
]);
/** The Street Level layer's UI state on a page, or why it could not be read. */
function streetLevelState(page) {
return page.evaluate(() => {
const module =
window.__godsEyeView?.dataManager?.layers?.get('street-level')?.module;
return module?.getUIState ? module.getUIState() : null;
});
}
/**
* Save the evidence of a failed run under `dir`: per page a full-page
* screenshot, the Street Level UI state, the console, failed requests and
* render errors; plus the error itself. Never throws: a page that cannot be
* captured records why instead.
* @returns {Promise<string>} the directory written
*/
export async function saveFailureArtifacts({ dir, browser, monitors, error }) {
await fs.mkdir(dir, { recursive: true });
await fs.writeFile(
path.join(dir, 'error.txt'),
redact(error?.stack || error?.message || String(error)),
);
const known = new Map(monitors.map((monitor) => [monitor.page, monitor]));
let pages = [];
try {
pages = await browser.pages();
} catch {
pages = monitors.map((monitor) => monitor.page);
}
let index = 0;
for (const page of pages) {
const monitor = known.get(page);
// The browser's first blank tab holds nothing worth saving.
if (!monitor && page.url() === 'about:blank') continue;
const base = `${String(index++).padStart(2, '0')}-${monitor?.name ?? 'page'}`;
const record = { url: redact(page.url()) };
try {
await withTimeout(
page.screenshot({
path: path.join(dir, `${base}.png`),
fullPage: true,
}),
30_000,
'screenshot',
);
} catch (shotError) {
record.screenshotError = String(shotError?.message || shotError);
}
try {
record.streetLevel = await withTimeout(
streetLevelState(page),
10_000,
'getUIState',
);
} catch (stateError) {
record.streetLevelError = String(stateError?.message || stateError);
}
record.renderErrors = [
...(monitor?.renderErrors ?? []),
...(await withTimeout(readRenderErrors(page), 10_000, 'render errors')
.then((list) => list.map((text) => `scene.renderError: ${text}`))
.catch(() => [])),
].map(redact);
record.console = monitor?.console ?? [];
record.failedRequests = monitor?.failed ?? [];
await fs.writeFile(
path.join(dir, `${base}.json`),
redact(JSON.stringify(record, null, 2)),
);
}
return dir;
}
+45 -6
View File
@@ -3,7 +3,10 @@
* Prove the portable CCTV panel in a real browser: a header drag lifts it out
* of the right rail, every resize handle moves only its own edges, minimum
* sizes hold, the window survives a reload and a header double-click snaps
* it back and forgets the stored position.
* it back and forgets the stored position. Page errors and Cesium render-loop
* errors fail the run; a failed run saves screenshots, console and failed
* requests under qa-artifacts/panel-resize/ (scripts/qa-browserEvidence.mjs).
* `--fail-on-retry` (or QA_FAIL_ON_RETRY=1) fails on any header-press retry.
*/
import assert from 'node:assert/strict';
import { pathToFileURL } from 'node:url';
@@ -13,11 +16,20 @@ import {
dockPanelByDoubleClick,
dragPointer,
liftPanelByHeader,
nextFrames,
} from './qa-panelDrag.mjs';
import {
hookRenderErrors,
readRenderErrors,
saveFailureArtifacts,
watchPage,
} from './qa-browserEvidence.mjs';
export const PANEL_ID = 'cctv-panel';
export const STORAGE_KEY = `godsEyeView.v8.panelPos.${PANEL_ID}`;
export const MIN_SIZE = { width: 300, height: 160 };
/** Where a failed run leaves its screenshots and logs. */
export const ARTIFACT_DIR = 'qa-artifacts/panel-resize';
const VIEWPORT = { width: 1400, height: 900 };
const EDGE_BY_SIDE = { n: 'top', s: 'bottom', e: 'right', w: 'left' };
@@ -82,6 +94,7 @@ async function main() {
const args = process.argv.slice(2);
const urlIndex = args.indexOf('--url');
const url = urlIndex >= 0 ? args[urlIndex + 1] : 'http://localhost:4173';
if (args.includes('--fail-on-retry')) process.env.QA_FAIL_ON_RETRY = '1';
const browser = await puppeteer.launch({
headless: true,
executablePath:
@@ -89,11 +102,13 @@ async function main() {
(await puppeteer.executablePath()),
args: ['--no-sandbox', '--use-gl=angle', '--use-angle=swiftshader'],
});
const monitors = [];
try {
const page = await browser.newPage();
await page.setViewport(VIEWPORT);
const errors = [];
page.on('pageerror', (error) => errors.push(error.stack || error.message));
monitors.push(watchPage(page, { name: 'panel', errors }));
await hookRenderErrors(page);
const boot = async () => {
await page.goto(`${url}/`, { waitUntil: 'domcontentloaded' });
@@ -112,7 +127,7 @@ async function main() {
node.remove();
});
await page.mouse.click(VIEWPORT.width / 2, VIEWPORT.height / 2);
await new Promise((resolve) => setTimeout(resolve, 400));
await nextFrames(page);
};
const readPanel = () =>
page.evaluate((id) => {
@@ -259,7 +274,17 @@ async function main() {
`lifted top (${moved})`,
);
near(lifted.width, pressed.width, 2, 'lifted width');
await new Promise((resolve) => setTimeout(resolve, 700));
// The rail lets go of the lifted panel on its next layout pass.
await page
.waitForFunction(
(id) =>
document
.getElementById(id)
.style.getPropertyValue('--right-panel-allocated-height') === '',
{ timeout: 5_000 },
PANEL_ID,
)
.catch(() => {});
const afterLayout = await readPanel();
assert.equal(afterLayout.allocated, '', 'the rail no longer allocates it');
const railExcludes = await page.evaluate((id) => {
@@ -294,7 +319,6 @@ async function main() {
{
const header = await headerPoint();
await drag(header, 0, 120 - header.y);
await new Promise((resolve) => setTimeout(resolve, 200));
}
// Every handle moves only its own edges.
@@ -366,7 +390,22 @@ async function main() {
assert.equal(snapped.stored, null, 'the storage key is gone');
console.log('PASS: header double-click returns the panel to the rail');
assert.deepEqual(errors, []);
const renderErrors = await readRenderErrors(page);
assert.deepEqual(
[...errors, ...renderErrors.map((text) => `scene.renderError: ${text}`)],
[],
'no page errors and no Cesium render-loop errors',
);
console.log('PASS: no page errors and no render-loop errors');
} catch (error) {
const dir = await saveFailureArtifacts({
dir: ARTIFACT_DIR,
browser,
monitors,
error,
}).catch(() => null);
if (dir) console.error(`failure evidence saved in ${dir}/`);
throw error;
} finally {
await browser.close();
}
+131 -21
View File
@@ -5,8 +5,14 @@
* be under another panel by the time the pointer goes down. Each attempt
* re-resolves the header, presses only where the header really is, records
* what the press landed on, and waits for the panel's actual state. Only a
* provable miss (a press that landed outside the header) is retried; a press
* in the header that did not lift or dock is a real failure.
* provable miss (a recorded press that landed outside the header) is
* retried; a press in the header that did not lift or dock, and a press that
* recorded no pointerdown at all, are real failures.
*
* Every retry is logged with what led to it (the pre-press hit check, the
* stillness wait, the panel box at the hit check and at the press) and as a
* GitHub `::warning::`. With QA_FAIL_ON_RETRY=1 (or `--fail-on-retry`) a
* retry fails the run instead.
*/
const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
@@ -15,13 +21,37 @@ const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
export const PRESS_RETRIES = 2;
const RETRY_PAUSE_MS = 1000;
/** Whether a retry fails the run (QA_FAIL_ON_RETRY=1 or --fail-on-retry). */
export function failOnRetryDefault(
env = process.env,
argv = process.argv.slice(2),
) {
return env.QA_FAIL_ON_RETRY === '1' || argv.includes('--fail-on-retry');
}
/**
* Whether an attempt proves nothing about the panel: it did not reach the
* wanted state and the press demonstrably landed outside the header.
* wanted state and the recorded press demonstrably landed outside the
* header. No recorded press (`pressed` null) is not a miss but a failure:
* the pointer never reached the page, which a retry would only hide.
* @param {{done: boolean, pressed: {inHeader: boolean}|null}} attempt
*/
export function pressMissed(attempt) {
return !attempt.done && attempt.pressed?.inHeader !== true;
return !attempt.done && attempt.pressed?.inHeader === false;
}
/** The panel's box, rounded, as `left,top widthxheight`. */
export function formatBox(box) {
if (!box) return 'unknown';
const r = (value) => Math.round(value ?? NaN);
return `${r(box.left)},${r(box.top)} ${r(box.width)}x${r(box.height)}`;
}
function panelBox(page, panelId) {
return page.evaluate((id) => {
const r = document.getElementById(id).getBoundingClientRect();
return { left: r.left, top: r.top, width: r.width, height: r.height };
}, panelId);
}
/** The centre of the panel's header title, read now. */
@@ -50,7 +80,8 @@ export function hitAt(page, point, panelId) {
/**
* The header point to press, re-resolved until it lands on the header (or the
* bounded wait runs out; the press then records the miss).
* bounded wait runs out; the press then records the miss). `timedOut` says
* the hit check never passed; `box` is the panel's box at the last check.
*/
export async function headerTarget(
page,
@@ -64,13 +95,15 @@ export async function headerTarget(
point = await headerPoint(page, panelId);
hit = await hitAt(page, point, panelId);
}
return { point, hit };
const box = await panelBox(page, panelId);
return { point, hit, timedOut: !hit.inHeader, box };
}
/**
* Best effort: wait until the panel's box (to the whole pixel) has held still
* for `stillMs`, so a rail still animating heights does not move the header
* between the hit check and the press. Never fails the run.
* between the hit check and the press. Never fails the run; resolves to
* whether the box held still in time.
*/
export async function waitForStill(
page,
@@ -93,8 +126,10 @@ export async function waitForStill(
panelId,
stillMs,
);
return true;
} catch (error) {
if (error?.name !== 'TimeoutError') throw error;
return false;
}
}
@@ -114,6 +149,7 @@ export function recordNextPress(page, panelId) {
left: r.left,
top: r.top,
width: r.width,
height: r.height,
// What the press landed on, so a failed press says why.
target: `${target.tagName?.toLowerCase()}${target.id ? `#${target.id}` : ''}.${[...(target.classList || [])].join('.')}`,
inHeader: Boolean(target.closest?.(`#${id} .panel-header`)),
@@ -152,6 +188,19 @@ export async function waitForFloating(
}
}
/**
* Wait for the page to paint twice: pointer handlers apply synchronously, so
* by then every layout change an input caused is on screen.
*/
export function nextFrames(page) {
return page.evaluate(
() =>
new Promise((resolve) =>
requestAnimationFrame(() => requestAnimationFrame(() => resolve())),
),
);
}
/** Press at `from`, move by (dx, dy) in two legs, release. */
export async function dragPointer(page, from, dx, dy, { steps = 6 } = {}) {
await page.mouse.move(from.x, from.y);
@@ -159,7 +208,27 @@ export async function dragPointer(page, from, dx, dy, { steps = 6 } = {}) {
await page.mouse.move(from.x + dx / 2, from.y + dy / 2, { steps });
await page.mouse.move(from.x + dx, from.y + dy, { steps });
await page.mouse.up();
await delay(120);
await nextFrames(page);
}
/**
* One line naming why a press is retried: what it landed on, the pre-press
* hit check, the stillness wait and the panel's box at the check and press.
*/
export function describeRetry(
panelId,
attempt,
{ round, retries, extra = '' },
) {
const { hit, pressed, timedOut, still, box } = attempt;
return [
`retry ${round + 1}/${retries}: the press missed the ${panelId} header (on ${pressed?.target ?? hit.target})`,
`pre-press hit check ${timedOut ? `timed out (on ${hit.target})` : 'passed'}`,
`stillness ${still === false ? 'timed out' : 'held'}`,
`panel box at hit check ${formatBox(box)} -> at press ${formatBox(pressed)}`,
]
.concat(extra ? [extra] : [])
.join('; ');
}
/**
@@ -170,19 +239,41 @@ async function pressUntilDecided(
page,
panelId,
press,
{ retries, log, note, pauseMs = RETRY_PAUSE_MS },
{
retries,
log,
note,
pauseMs = RETRY_PAUSE_MS,
failOnRetry = failOnRetryDefault(),
},
) {
for (let round = 0; ; round++) {
await waitForStill(page, panelId);
const still = await waitForStill(page, panelId);
await recordNextPress(page, panelId);
const { point, hit } = await headerTarget(page, panelId);
const { point, hit, timedOut, box } = await headerTarget(page, panelId);
const done = await press(point);
const attempt = { point, hit, done, pressed: await readPress(page) };
const attempt = {
point,
hit,
timedOut,
still,
box,
done,
pressed: await readPress(page),
retries: round,
};
if (round >= retries || !pressMissed(attempt)) return attempt;
const extra = note ? `; ${await note()}` : '';
log(
`note: the press missed the ${panelId} header (on ${attempt.pressed?.target ?? hit.target}${extra}); retrying`,
);
const why = describeRetry(panelId, attempt, {
round,
retries,
extra: note ? await note() : '',
});
if (failOnRetry)
throw new Error(
`QA_FAIL_ON_RETRY=1 turns this retry into a failure: ${why}`,
);
log(`note: ${why}; retrying`);
log(`::warning title=Panel press retried::${why}`);
await delay(pauseMs);
}
}
@@ -196,7 +287,15 @@ async function pressUntilDecided(
export function liftPanelByHeader(
page,
panelId,
{ dx, dy, retries = PRESS_RETRIES, log = console.log, note, pauseMs } = {},
{
dx,
dy,
retries = PRESS_RETRIES,
log = console.log,
note,
pauseMs,
failOnRetry,
} = {},
) {
return pressUntilDecided(
page,
@@ -205,7 +304,7 @@ export function liftPanelByHeader(
await dragPointer(page, point, dx, dy);
return waitForFloating(page, panelId, true);
},
{ retries, log, note, pauseMs },
{ retries, log, note, pauseMs, failOnRetry },
);
}
@@ -217,7 +316,13 @@ export function liftPanelByHeader(
export function dockPanelByDoubleClick(
page,
panelId,
{ retries = PRESS_RETRIES, log = console.log, note, pauseMs } = {},
{
retries = PRESS_RETRIES,
log = console.log,
note,
pauseMs,
failOnRetry,
} = {},
) {
return pressUntilDecided(
page,
@@ -227,12 +332,17 @@ export function dockPanelByDoubleClick(
await page.mouse.click(point.x, point.y, { clickCount: 2 });
return waitForFloating(page, panelId, false);
},
{ retries, log, note, pauseMs },
{ retries, log, note, pauseMs, failOnRetry },
);
}
/** One line naming where an attempt pressed, for an assertion message. */
export function describePress(attempt) {
const { point, hit, pressed } = attempt;
return `pressed ${Math.round(point.x)},${Math.round(point.y)} on ${pressed?.target ?? `${hit.target} (no press recorded)`}, in header: ${pressed?.inHeader ?? hit.inHeader}`;
const where = `pressed ${Math.round(point.x)},${Math.round(point.y)}`;
if (!pressed)
return `${where}: no pointerdown reached the page (hit check on ${hit.target}, in header: ${hit.inHeader})`;
const n = attempt.retries || 0;
const retried = n ? `, after ${n} ${n === 1 ? 'retry' : 'retries'}` : '';
return `${where} on ${pressed.target}, in header: ${pressed.inHeader}${retried}`;
}
+1484 -756
View File
File diff suppressed because it is too large. Load diff
+338 -6
View File
@@ -6,7 +6,24 @@ import {
fixtureTile,
EXPECTED_PROVIDERS,
isCollapsed,
PARK,
STRICT_ALLOWED_SKIPS,
strictViolations,
} from '../../scripts/qa-street-level.mjs';
import {
answerMapillaryRequest,
describeCall,
metresApart,
PHOTO_LINE,
PHOTO_SEQUENCE_ID,
photoImages,
THUMB_HOST,
} from '../../scripts/fixtures/street-level/mapillaryGraph.mjs';
import { redact } from '../../scripts/qa-browserEvidence.mjs';
// The press helpers wait for paint; Node has no frames, so a tick stands in.
globalThis.requestAnimationFrame ??= (callback) =>
setTimeout(() => callback(Date.now()), 0);
test('the harness expects exactly the providers the app registers, in chip order', () => {
// Any method a provider asks its source for is a no-op: only ids matter.
@@ -45,11 +62,12 @@ test('fixture tiles give the hermetic gate something real to filter', async () =
y: 6286,
z: 14,
});
assert.equal(street.sequences.length, 8);
const pano = street.sequences.filter((s) => s.isPano).length;
const grid = street.sequences.filter((s) => s.id !== PHOTO_SEQUENCE_ID);
assert.equal(grid.length, 8);
const pano = grid.filter((s) => s.isPano).length;
assert.ok(pano > 0 && pano < 8, '360° and flat both present');
const year = 365 * 86_400_000;
const old = street.sequences.filter((s) => now - s.capturedAt > year).length;
const old = grid.filter((s) => now - s.capturedAt > year).length;
assert.ok(old > 0 && old < 8, 'recent and older both present');
const orbit = decodeCoverageTile(fixtureTile(3, 1, 3, now), {
x: 1,
@@ -159,6 +177,8 @@ function fakePanelPage({ presses = [], floating = false, stuck = false } = {}) {
const evaluate = async (fn, ...args) => inPage(fn, ...args);
const press = () => {
state.last = presses[state.downs++] ?? 'header';
// 'none': the pointerdown never reaches the page.
if (state.last === 'none') return;
const target = state.last === 'header' ? header : other;
for (const listener of listeners.splice(0)) inPage(listener, { target });
};
@@ -202,7 +222,8 @@ test('a header press is retried only when it provably missed the header', async
pressMissed({ done: false, pressed: { inHeader: false } }),
true,
);
assert.equal(pressMissed({ done: false, pressed: null }), true);
// No recorded press is a failure, not a miss to retry.
assert.equal(pressMissed({ done: false, pressed: null }), false);
});
test('a lift whose press the rail moved off the header is retried, then lifts', async () => {
@@ -218,10 +239,17 @@ test('a lift whose press the rail moved off the header is retried, then lifts',
});
assert.equal(attempt.done, true);
assert.equal(page.state.downs, 2);
assert.equal(notes.length, 1);
assert.equal(notes.length, 2, 'a note and a GitHub warning');
assert.match(notes[0], /missed the street-level-panel header/);
assert.match(notes[0], /section#weather-panel\.panel/, 'names what it hit');
assert.match(describePress(attempt), /in header: true/);
assert.match(notes[0], /pre-press hit check passed/);
assert.match(notes[0], /stillness (held|timed out)/);
assert.match(
notes[0],
/panel box at hit check 100,200 300x400 -> at press 100,200 300x400/,
);
assert.match(notes[1], /^::warning title=Panel press retried::retry 1\/2/);
assert.match(describePress(attempt), /in header: true, after 1 retry$/);
});
test('a press in the header that does not lift fails at once, naming the target', async () => {
@@ -278,3 +306,307 @@ test('both panel gates press the header through the shared helper', () => {
assert.match(source, /dockPanelByDoubleClick\(page, PANEL_ID/, script);
}
});
/* ── Hermetic photo flow: the photo line and the Graph fixtures ────────── */
test('the photo sequence runs through the parked view, clipped to each tile it crosses', async () => {
const { decodeCoverageTile } =
await import('../layers/streetLevel/providers/mapillary/decode.js');
const { lonToTileX, latToTileY } =
await import('../layers/streetLevel/tileMath.js');
const x = lonToTileX(PHOTO_LINE.lon, 14);
const rows = new Set(
[PHOTO_LINE.south, PHOTO_LINE.north].map((lat) => latToTileY(lat, 14)),
);
assert.ok(rows.size > 1, 'the line crosses a tile edge');
let south = Infinity;
let north = -Infinity;
for (const y of rows) {
const line = decodeCoverageTile(fixtureTile(14, x, y), {
x,
y,
z: 14,
}).sequences.find((s) => s.id === PHOTO_SEQUENCE_ID);
assert.ok(line, `tile 14/${x}/${y} carries the photo line`);
for (const [lon, lat] of line.parts[0]) {
assert.ok(Math.abs(lon - PHOTO_LINE.lon) < 1e-4);
south = Math.min(south, lat);
north = Math.max(north, lat);
}
}
assert.ok(Math.abs(south - PHOTO_LINE.south) < 1e-4);
assert.ok(Math.abs(north - PHOTO_LINE.north) < 1e-4);
// Far away there is no photo line, only the grid.
const elsewhere = decodeCoverageTile(fixtureTile(14, 100, 100), {
x: 100,
y: 100,
z: 14,
});
assert.equal(
elsewhere.sequences.some((s) => s.id === PHOTO_SEQUENCE_ID),
false,
);
// The parked camera looks down onto the line.
assert.equal(PARK.lon, PHOTO_LINE.lon);
assert.ok(PARK.lat > PHOTO_LINE.south && PARK.lat < PHOTO_LINE.north);
});
test('the fixture photos: one sequence, 360° and flat, close enough for any nearest lookup on the line', () => {
const images = photoImages(Date.UTC(2026, 9, 1));
assert.ok(images.length >= 3);
assert.ok(images.some((image) => image.isPano));
assert.ok(images.some((image) => !image.isPano));
for (let i = 1; i < images.length; i++) {
const gap = metresApart(images[i - 1], images[i]);
// Above the cones' 3 m thinning, and at most 50 m apart: any point on
// the line has a photo within the Graph API's 50 m radius.
assert.ok(gap > 3 && gap < 50, `gap ${gap}`);
}
assert.equal(new Set(images.map((image) => image.id)).size, images.length);
});
const graph = (path, params) =>
`https://graph.mapillary.com/${path}?${new URLSearchParams({ access_token: 'MLY|0|qa-fixture', ...params })}`;
const answer = (url, method = 'GET') => {
const out = answerMapillaryRequest({ method, url });
return {
...out,
json: out.contentType === 'application/json' ? JSON.parse(out.body) : null,
};
};
test('the Graph fixtures answer the provider’s nearest and sequence lookups', () => {
const images = photoImages();
const middle = images[Math.floor(images.length / 2)];
const near = answer(
graph('images', {
lat: String(middle.lat + 0.0001),
lng: String(middle.lon),
radius: '50',
limit: '8',
fields: 'id,geometry,is_pano,captured_at,sequence',
}),
);
assert.equal(near.status, 200);
assert.ok(near.json.data.length > 0);
assert.ok(near.json.data.some((record) => record.id === middle.id));
for (const record of near.json.data) {
assert.deepEqual(Object.keys(record).sort(), [
'captured_at',
'geometry',
'id',
'is_pano',
'sequence',
]);
assert.equal(record.sequence, PHOTO_SEQUENCE_ID);
}
const far = answer(
graph('images', { lat: '0', lng: '0', radius: '50', fields: 'id' }),
);
assert.deepEqual(far.json.data, [], 'nothing outside the radius');
const sequence = answer(
graph('images', {
sequence_ids: PHOTO_SEQUENCE_ID,
fields: 'id,geometry,compass_angle,captured_at,is_pano',
limit: '2000',
}),
);
assert.equal(sequence.json.data.length, images.length);
assert.deepEqual(
answer(graph('images', { sequence_ids: 'other', fields: 'id' })).json.data,
[],
);
});
test('the Graph fixtures answer every call MapillaryJS makes to open a photo', () => {
const [first, second] = photoImages();
const spatial = answer(
graph('images', {
image_ids: `${first.id},${second.id}`,
fields:
'id,computed_geometry,geometry,sequence,camera_type,computed_rotation,thumb_1024_url,thumb_2048_url,width,height,merge_cc,sfm_cluster,mesh,creator,captured_at',
}),
);
assert.equal(spatial.status, 200);
assert.equal(spatial.headers['Access-Control-Allow-Origin'], '*');
const [pano, flat] = spatial.json.data;
assert.equal(pano.camera_type, 'spherical');
assert.equal(flat.camera_type, 'perspective');
assert.equal(pano.merge_cc, null, 'unmerged: no mesh request');
assert.equal(pano.sfm_cluster, null, 'no cluster request');
assert.equal(pano.computed_rotation.length, 3);
assert.equal(new URL(pano.thumb_2048_url).hostname, THUMB_HOST);
assert.equal(
answer(graph('images', { s2: '9749618446378729472', fields: 'id' })).json
.data.length,
0,
);
assert.deepEqual(
answer(graph('image_ids', { sequence_id: PHOTO_SEQUENCE_ID })).json.data[0],
{ id: first.id },
);
assert.deepEqual(
answer(graph(`${first.id}/tiles`, { z: '11', fields: 'url,z,x,y' })).json,
{ data: [] },
);
const photo = answer(pano.thumb_2048_url);
assert.equal(photo.status, 200);
assert.equal(photo.contentType, 'image/jpeg');
assert.deepEqual([...photo.body.subarray(0, 3)], [0xff, 0xd8, 0xff]);
const preflight = answer(graph('images', { image_ids: first.id }), 'OPTIONS');
assert.equal(preflight.status, 204);
assert.match(
preflight.headers['Access-Control-Allow-Headers'],
/Authorization/,
);
});
test('a Mapillary call no fixture covers is answered but flagged unknown', () => {
for (const url of [
graph('images', { bbox: '0,0,1,1' }),
graph('map_features', { fields: 'id' }),
'https://tiles.mapillary.com/maps/vtp/mly1_public/2/14/1/1',
]) {
const out = answer(url);
assert.equal(out.known, false, url);
assert.ok(out.status >= 400);
}
});
test('answered calls are named without the token', () => {
const call = describeCall(
'GET',
new URL(graph('9100000000000/tiles', { z: '11', fields: 'url' })),
);
assert.equal(call, 'GET graph.mapillary.com/{imageId}/tiles?fields&z');
assert.doesNotMatch(call, /MLY|access_token/);
assert.equal(
describeCall(
'GET',
new URL('https://qa-fixture.mapillary.com/thumb/9100000000016.jpg'),
),
'GET qa-fixture.mapillary.com/thumb/{imageId}.jpg',
);
});
test('evidence never carries a Mapillary token', () => {
assert.equal(
redact(
'GET https://graph.mapillary.com/images?access_token=MLY|123|abc&fields=id',
),
'GET https://graph.mapillary.com/images?access_token=REDACTED&fields=id',
);
assert.equal(
redact('token MLY|123|abc in a log'),
'token MLY|REDACTED in a log',
);
assert.equal(
redact(
'https://tile.googleapis.com/v1/3dtiles/x.glb?session=s1&key=AIzaSyBexampleexampleexampleexample00',
),
'https://tile.googleapis.com/v1/3dtiles/x.glb?session=s1&key=REDACTED',
);
assert.equal(
redact('loaded with AIzaSyBexampleexampleexampleexample00 inline'),
'loaded with AIza…REDACTED inline',
);
});
test('--strict accepts only the documented skips', () => {
assert.deepEqual(
[...STRICT_ALLOWED_SKIPS],
['no Google 3D', 'fixtures only'],
);
assert.deepEqual(
strictViolations([
{ reason: 'no Google 3D', label: 'terrain' },
{ reason: 'fixtures only', label: 'keyless page' },
{ reason: 'no Mapillary key', label: 'the keyed steps' },
]),
[{ reason: 'no Mapillary key', label: 'the keyed steps' }],
);
});
test('a press that records no pointerdown fails at once, without a retry', async () => {
const { liftPanelByHeader, describePress } =
await import('../../scripts/qa-panelDrag.mjs');
const page = fakePanelPage({ presses: ['none', 'header'] });
const attempt = await liftPanelByHeader(page, 'street-level-panel', {
dx: -400,
dy: 80,
log: () => assert.fail('no retry'),
pauseMs: 0,
});
assert.equal(attempt.done, false);
assert.equal(attempt.pressed, null);
assert.equal(page.state.downs, 1);
assert.match(describePress(attempt), /no pointerdown reached the page/);
});
test('QA_FAIL_ON_RETRY turns a retry into a failure that says why', async () => {
const { liftPanelByHeader, failOnRetryDefault } =
await import('../../scripts/qa-panelDrag.mjs');
assert.equal(failOnRetryDefault({ QA_FAIL_ON_RETRY: '1' }, []), true);
assert.equal(failOnRetryDefault({}, ['--fail-on-retry']), true);
assert.equal(failOnRetryDefault({}, []), false);
const page = fakePanelPage({ presses: ['other', 'header'] });
await assert.rejects(
liftPanelByHeader(page, 'street-level-panel', {
dx: -400,
dy: 80,
log: () => {},
pauseMs: 0,
failOnRetry: true,
}),
/QA_FAIL_ON_RETRY=1 .*missed the street-level-panel header.*pre-press hit check passed/,
);
assert.equal(page.state.downs, 1);
});
test('both gates save failure evidence and fail on render-loop errors', () => {
for (const script of ['qa-street-level.mjs', 'qa-panel-resize.mjs']) {
const source = fs.readFileSync(
new URL(`../../scripts/${script}`, import.meta.url),
'utf8',
);
assert.match(source, /saveFailureArtifacts\(/, script);
assert.match(source, /hookRenderErrors\(/, script);
assert.match(source, /readRenderErrors\(/, script);
assert.match(source, /--fail-on-retry/, script);
}
});
test('CI runs the gates hermetically against a production build, strict, with evidence', () => {
const ci = fs.readFileSync(
new URL('../../.github/workflows/ci.yml', import.meta.url),
'utf8',
);
const job = ci.slice(ci.indexOf('street-level-browser:'));
const gate = job.slice(0, job.indexOf('\n windows-onboarding:'));
// The dummy token is baked into the bundle at build time and the preview
// server's status route reads it at run time: both steps carry it.
assert.match(
gate,
/MAPILLARY_CLIENT_TOKEN: 'MLY\|0\|qa-fixture'\s+run: npm run build/,
);
assert.match(
gate,
/MAPILLARY_CLIENT_TOKEN: 'MLY\|0\|qa-fixture'\s+run: \|\s+npx vite preview --port 4173 --strictPort/,
);
assert.doesNotMatch(gate, /npx vite --port/);
assert.equal(
(gate.match(/MAPILLARY_CLIENT_TOKEN: 'MLY\|0\|qa-fixture'/g) || []).length,
2,
);
assert.match(gate, /qa:street-level:fixtures -- --strict/);
assert.match(
gate,
/if: failure\(\)[\s\S]*actions\/upload-artifact@[0-9a-f]{40}/,
);
assert.match(gate, /path: qa-artifacts\//);
assert.match(
gate,
/QA_FAIL_ON_RETRY: '1'/,
'a missed header press fails CI now that the rail settles',
);
});