fix(cctv): add Taiwan Highway Bureau fallback

This commit is contained in:
InoukT committed 2026-11-02 09:58:57 -05:00
1 parent 151ce3dee7
commit ed2aaa2c50
8 files changed
+704 -39

No files matched your search

+1 -1
View File
@@ -107,7 +107,7 @@ prints anything; never fix that by renaming or deleting PR A's published token.
The highest-leverage places to jump in:
- **🌆 Add a CCTV source pack.** Austin is the reference camera source. Adding another city means a clean public camera catalog with coordinates, attribution, and server-registered frame URLs (the proxy only fetches registered URLs — never client-supplied ones, see [SECURITY.md](SECURITY.md)). City packs are the best first lane.
- **🌆 Add a CCTV source pack.** Austin is the reference camera source. Follow the [CCTV integration workflow](docs/CCTV_INTEGRATION.md) for source review, provider registration, proxy security, tests, and attribution. The proxy only fetches server-registered URLs — never client-supplied ones; see [SECURITY.md](SECURITY.md).
- **🛰️ Add or improve a data layer.** Layer factories live in `src/layers/<family>/`, with source, record, controller and renderer owners implementing the layer interface (`init/enable/disable/update/destroy/getStats`, optional `getDetectableObjects`/`getStats`). Use an existing layer as a template, and see [Adding a new layer](#adding-a-new-layer) for what the PR needs.
- **🎙️ Extend voice control.** Voice arguments are defined in `src/voice/actionSchemas.js`, with server-side descriptions in `server/providers/openai/tools.js` and client-side execution in `src/voice/gevActions.js`. Keep the tool surface tight and the responses honest (confirm only what actually happened).
- **🎨 Add a visual style.** Styles are GLSL post-process shaders in `src/styles/`.
+1 -1
View File
@@ -115,7 +115,7 @@ The optional Local ADS-B layer and the broadcast-FM receiver use radio you recei
**Quebec is thinly represented**, and that is upstream rather than a filter here: the province runs its own Info-Smog program, so only a handful of Quebec stations appear in ECCC's national feed. The reporting station count also varies hour to hour as stations report or fall silent.
- **Via Lietuva / Eismoinfo (Lithuania).** Public road-camera catalogs and periodically updated still images from [Eismoinfo](https://eismoinfo.lt/), credited to Via Lietuva / eismoinfo.lt in the app. **Off by default.** The camera endpoints are the eismoinfo.lt website backend and are not among the services listed in Via Lietuva's [open-data statement](https://eismoinfo.lt/eismoinfo-backend/content?id=44), whose conditions are attribution and reuse on the same terms. Until the statement is confirmed to cover the camera photos, the pack loads only when an operator sets `CCTV_LITHUANIA_ENABLED=1`; with the flag unset, empty or any other value, the server makes no request to Eismoinfo. `eismoinfo-backend/camera-info-table` is joined by ID to `layer-static-features/VKR`. Without `lks=true`, the latter returns WGS84 `[latitude, longitude]`; the table's `x/y` instead use LKS-94 metres. Only the official HTTPS `/eismoinfo-backend/image-provider/camera/last?id=…` image endpoint is registered. Catalogs use the shared 15-minute cache; frames are fetched on demand and never bundled or archived. These are periodic snapshots, not continuous video. Camera headings are low-confidence estimates. No API key is required. `CCTV_LITHUANIA_MAX_SOURCES` defaults to 400.
- **Freeway Bureau, MOTC (Taiwan).** National freeway cameras from the Freeway Bureau's keyless [CCTV static table](https://tisvcloud.freeway.gov.tw/history/motc20/CCTV.xml) in the MOTC 2.0 standard, refreshed daily and fetched at runtime (never bundled). Freeway Bureau traffic data is published under the [Open Government Data License, Taiwan (Version 1.0)](https://data.gov.tw/license), which permits reuse, including commercial reuse, with attribution; the app credits the Freeway Bureau, MOTC, names the dataset and links the licence, as its attribution exhibit asks. The table publishes each camera's MJPEG stream on one of four official camera servers (`cctvn`, `cctvn5`, `cctvc`, `cctvs` under `freeway.gov.tw`). The frame proxy relays single images, so each published stream URL is mapped to the same server's single-JPEG path for the same camera number (`/abs2mjpg/bmjpg` → `/abs2mjpg/jpg`, `/live-view/mjpg/video.cgi` → `/live-view/jpg/image.cgi`); only those exact HTTPS pairs are registered, and the still path is not itself listed in the table. These are on-demand snapshots, not continuous video. Camera headings are low-confidence estimates. No API key is required. The table host does not answer from some networks outside Taiwan (the camera servers do); there the pack is empty, logs one warning and retries hourly. On by default; `CCTV_TAIWAN_ENABLED=0` skips the pack, and `CCTV_TAIWAN_MAX_SOURCES` defaults to 600 (bounded to 1–2000).
- **Freeway Bureau and Highway Bureau, MOTC (Taiwan).** National freeway cameras use the Freeway Bureau’s keyless [MOTC 2.0 CCTV table](https://tisvcloud.freeway.gov.tw/history/motc20/CCTV.xml), refreshed daily and fetched at runtime (never bundled). Freeway Bureau data follows the [Open Government Data License, Taiwan v1.0](https://data.gov.tw/license); the app credits the Bureau, names the dataset and links the licence. If that catalog is unreachable and no last-good source list exists, the app falls back to the Highway Bureau’s free [provincial-road CCTV XML feed](https://thbapp.thb.gov.tw/opendata/), with attribution as requested by the Bureau. Its MJPEG streams are accepted only from cctv-ss01 through cctv-ss08.thb.gov.tw over HTTPS; the frame proxy extracts one bounded JPEG per request. This fallback covers provincial roads, not national freeways. Neither feed requires an API key. CCTV_TAIWAN_ENABLED=0 disables both; CCTV_TAIWAN_MAX_SOURCES defaults to 600 (bounded to 1–2000).
- **Google Maps Platform.** You supply your own API key and are bound by [Google's ToS](https://cloud.google.com/maps-platform/terms). Google Maps Content (tiles, geocodes, places) **may not be cached, stored, rehosted, or committed** — this app only ever uses it live, which is the compliant pattern. The "Google" attribution is displayed on the globe and must stay visible. Restrict your key (see [SECURITY.md](SECURITY.md)).
- **OpenSky Network.** Its license is **non-commercial**, and operational use of the REST API in a live product can require a prior written agreement with OpenSky — even for non-profit/government use. If you deploy this commercially, contact OpenSky for your own terms. The flights layer is a toggle and runs anonymously by default.
+133
View File
@@ -0,0 +1,133 @@
# CCTV integration workflow
Use this workflow when adding a camera provider or repairing an existing feed.
The CCTV layer gets its catalog and media through the local server; keep
provider-specific network access and URL policy on the server side.
## 1. Check the source before coding
- Identify the catalog endpoint and the actual frame or live-media endpoint.
They may be different hosts or use different formats.
- Read the provider's license, terms, attribution, rate limits, and access
requirements. Publicly reachable data is not automatically licensed for
reuse. Record whether the app may display it and whether any use is
restricted.
- Prefer a public, keyless feed. If credentials are required, keep them in a
server-only environment variable; never put a secret in browser code or a
Vite-exposed variable. If access terms require an agreement or remain
unconfirmed, make the pack opt-in and ensure it makes no request by default.
- Confirm the feed can be fetched from the app's Node server environment. A URL
opening in a browser does not prove the server can reach it, and browser CORS
behavior does not diagnose a server-side fetch.
- Treat camera directories and aggregators such as [TW Live](https://tw.live/)
as discovery aids. Trace each listing to the original agency or camera owner;
their data and media terms govern reuse. For example, SkylineWebcams says
only camera hosts may embed its live feed and its terms restrict copying or
retransmitting footage, so do not proxy its stream without written
authorization ([FAQ](https://www.skylinewebcams.com/support/faq.html),
[terms](https://www.skylinewebcams.com/en/terms-of-use.html)).
- Decide whether the provider supplies still snapshots or continuous video.
Prefer stills unless live video is necessary and supported by the existing
proxy path.
## 2. Add and register the provider pack
Use the smallest existing provider as a template. Current owners are:
- `server/providers/cctv/sources.js` for standard open-data loaders.
- `server/providers/cctv/<provider>.js` for a provider with its own parser,
fallback, or more involved URL policy.
- `server/providers/cctv/catalog.js` to import the loader and add it to
`LIVE_PACKS`. Each pack is independently gated and the shared catalog cap
allocates space across packs.
- `server/providers/cctv/constants.js` for provider endpoints, caps, and
timeouts.
Return normalized camera records with stable unique IDs, readable names,
`city`/`cityId`, provider attribution, valid `lat`/`lon`, and the correct
`feedType`. Include camera heading only when the source actually provides a
bearing; descriptive direction text is not a numeric heading. Mark uncertain
heading data with the existing low-confidence conventions. Set `sourceKind` so
health and source diagnostics identify the provider. Apply a bounded per-pack
camera limit and geographic filtering where appropriate; avoid one provider
silently consuming the whole shared catalog cap.
Camera coordinates need their own verification. Prefer the original provider's
camera coordinates, then corroborate them against its location description,
route/mile marker, or official map. An aggregator's map dot, a town/county name,
or a geocoded attraction name is not a precise camera position. Do not pin a
camera to a city center or nearby landmark; if its actual location cannot be
verified, leave it out until it can. For manually curated coordinates, record
the evidence in the provider data or an adjacent comment. For example, TW Live
camera pages identify the originating agency; use that agency's catalog for
the camera point when available.
If the source becomes opt-in, use the catalog's opt-in gate (`envOptIn`) and
document the flag and reason in `DATA_SOURCES.md`. Otherwise use the normal
enabled-by-default flag pattern and document how to disable the pack.
## 3. Keep frame and media requests server-registered
The browser should request the local `/api/cctv/frame/:id` or
`/api/cctv/media/:id` endpoint. Do not pass an arbitrary upstream URL from the
browser, and do not fetch an unvalidated URL supplied by a catalog record.
Every provider must have a server-side rule that pins frame and media URLs to
the expected scheme, host, and path; use HTTPS where the provider supports it.
Use `pinnedCameraUrl()` for URLs that belong under a fixed provider directory,
or a stricter provider-specific validator when hosts or paths vary. Register
any new permitted image/media origin in the existing
`server/providers/cctv` media policy.
Catalog and media loaders must also:
- refuse unsafe schemes, credentials, unexpected ports, redirects outside the
registered origin/path, and malformed identifiers;
- set request deadlines and cap catalog and buffered frame bodies;
- parse only the expected format and validate coordinates before returning a
camera;
- handle offline cameras and partial provider failures without failing the
entire merged catalog; and
- avoid persistent frame storage unless the provider's terms explicitly allow
it and the design has been reviewed.
For live media, use the existing media proxy and supported feed type (`mjpeg`,
`mp4`, `webm`, or `hls`) rather than adding a direct browser stream. Validate
the upstream content type and bound or cancel streams according to the
existing proxy behavior. Read [SECURITY.md](../SECURITY.md) before changing
URL validation, redirects, or proxy behavior.
## 4. Add focused regression coverage
Add provider tests under `src/data/` following the existing CCTV test naming
and fixtures. Cover at least:
- representative valid catalog rows and their normalized output;
- missing or invalid IDs, coordinates, and frame URLs;
- duplicate records and the pack's cap/filter behavior;
- unsafe host/path, credentials, and redirect cases;
- timeout, oversized, malformed, and unavailable upstream responses; and
- the frame or live-media content type the provider actually serves.
Use `src/data/cctvProxy.test.mjs` for shared proxy/media behavior and
`src/data/cctvCatalogCap.test.mjs` for shared catalog allocation. Run the
focused provider and proxy tests, then the standard checks from
[CONTRIBUTING.md](../CONTRIBUTING.md) that apply to the change. For runtime
verification, inspect `/api/cctv/sources`, `/api/cctv/health`, and the camera's
`/api/cctv/frame/:id` or `/api/cctv/stream/:id` response locally. Confirm a
successful frame is from the upstream camera by checking its content type and
`X-CCTV-Source`; a fallback image is not proof that the upstream feed works.
## 5. Update source documentation and submit
- Add the provider, endpoint, terms/license, required attribution, key status,
caching behavior, and any restrictions to `DATA_SOURCES.md`.
- Update the CCTV source section in `docs/CURRENT-STATE.md` when runtime
behavior, flags, caps, or fallback behavior changes.
- Register in-app attribution through the existing source/provider credit
fields; update `src/data/dataCredits.js` when the source needs a layer-level
credit.
- Add the required `changelog.d/<pr-number>-<slug>.md` entry for a runtime
change, and include the source, license, attribution, key/runtime-fetch
details, and verification evidence in the PR description.
- For a new layer, also follow the camera/source requirements in the
[new-layer PR checklist](../CONTRIBUTING.md#adding-a-new-layer).
+2 -18
View File
@@ -1341,24 +1341,8 @@ video; camera poses use the existing low-confidence heading prior.
## Taiwan CCTV source pack
The CCTV catalog includes Taiwan national freeway cameras from the Freeway
Bureau, MOTC. One keyless request reads the daily MOTC 2.0 `CCTV.xml` table
(~1,800 cameras); the request refuses redirects, has a 15-second deadline and a
6 MiB response cap, and uses the shared 15-minute cache and fair catalog
allocation. A failed download serves the last good table (up to a day old),
or an empty pack, and is not retried for an hour: the catalog host does not
answer from some networks outside Taiwan, and without the backoff every refresh
there would wait out the connect timeout. The table
publishes MJPEG stream URLs; each is mapped to the same camera server's
single-JPEG path for the same camera number, and only those exact HTTPS pairs
on the four `freeway.gov.tw` camera servers are registered with the frame
proxy. Camera keys come from the server and camera number, because published
CCTV IDs contain Chinese place names and a few repeat across cameras. Poses use
the low-confidence heading prior. `CCTV_TAIWAN_ENABLED=0` disables the pack;
`CCTV_TAIWAN_MAX_SOURCES` sets its cap (default 600, bounded to 1–2000), and
`CCTV_TAIWAN_URL` overrides the table URL.
Gate: `src/data/cctvTaiwan.test.mjs`.
The CCTV catalog includes Taiwan national freeway cameras from the Freeway Bureau, MOTC. The app first reads the daily MOTC 2.0 CCTV.xml table with a keyless HTTPS request, a 15-second deadline, a 6 MiB response cap, and the shared 15-minute cache. If that catalog is unreachable and no last-good source list exists, the loader uses the Highway Bureau’s public provincial-road CCTV XML feed at https://cctv-maintain.thb.gov.tw/opendataCCTVs.xml. The fallback is capped by the same Taiwan source limit and hourly retry backoff; its MJPEG feeds are restricted to the eight cctv-ss01 through cctv-ss08.thb.gov.tw HTTPS servers. The frame proxy returns one bounded JPEG from each stream, not an unbounded video response. The Highway Bureau open-data page states XML downloads are free and asks users to attribute the source; the app credits Highway Bureau, MOTC. These are provincial-road cameras, not a replacement for the national freeway cameras. CCTV_TAIWAN_ENABLED=0 disables both feeds; CCTV_TAIWAN_MAX_SOURCES sets the shared cap (default 600, bounded to 1–2000), and CCTV_TAIWAN_URL overrides the Freeway Bureau table URL.
Gate: src/data/cctvTaiwan.test.mjs and src/data/cctvProxy.test.mjs.
## DriveBC CCTV source pack
The CCTV catalog adds DriveBC highway cameras for British Columbia alongside the
+163 -6
View File
@@ -621,6 +621,151 @@ export function cctvUpstreamUserAgent(url) {
}
}
/** Read one bounded JPEG part from an open MJPEG response, then stop the stream. */
async function readFirstMjpegFrame(upstream, maxBytes) {
const mediaType = cctvLiveMediaType(
'mjpeg',
upstream.headers.get('content-type'),
);
const boundary = parseMediaType(mediaType)?.params.get('boundary') || '';
if (
!boundary ||
!upstream.body ||
!Number.isSafeInteger(maxBytes) ||
maxBytes < 4
) {
try {
await upstream.body?.cancel();
} catch {
/* no-op */
}
return null;
}
const nextBoundary = Buffer.from('\r\n--' + boundary, 'latin1');
const opening = Buffer.from('--' + boundary + '\r\n', 'latin1');
const headerTerminator = Buffer.from('\r\n\r\n', 'ascii');
const jpegStart = Buffer.from([0xff, 0xd8]);
const jpegEnd = Buffer.from([0xff, 0xd9]);
const maxHeaderBytes = 8 * 1024;
const maxBufferedBytes =
opening.length + headerTerminator.length + maxHeaderBytes + maxBytes;
let reader;
let buffer = Buffer.alloc(0);
let parsedHeaders = false;
let bodyStart = -1;
let declaredLength = null;
try {
reader = upstream.body.getReader();
for (;;) {
const { done, value } = await reader.read();
if (done) return null;
const chunk = Buffer.from(value);
const room = Math.max(0, maxBufferedBytes - buffer.length);
const included = chunk.subarray(0, room);
const overflow = included.length < chunk.length;
buffer = Buffer.concat(
[buffer, included],
buffer.length + included.length,
);
if (buffer.length < opening.length) {
if (overflow) return null;
continue;
}
if (!buffer.subarray(0, opening.length).equals(opening)) return null;
if (!parsedHeaders) {
const headerEnd = buffer.indexOf(headerTerminator, opening.length);
if (headerEnd < 0) {
if (buffer.length - opening.length > maxHeaderBytes || overflow)
return null;
continue;
}
if (headerEnd - opening.length > maxHeaderBytes) return null;
const headerText = buffer
.subarray(opening.length, headerEnd)
.toString('latin1');
const headers = new Map();
for (const line of headerText.split('\r\n')) {
const colon = line.indexOf(':');
if (colon <= 0) return null;
const name = line.slice(0, colon).trim().toLowerCase();
if (!name || headers.has(name)) return null;
headers.set(name, line.slice(colon + 1).trim());
}
if (cctvRasterImageType(headers.get('content-type')) !== 'image/jpeg')
return null;
const length = headers.get('content-length');
if (length !== undefined) {
if (!/^\d+$/.test(length)) return null;
declaredLength = Number(length);
if (
!Number.isSafeInteger(declaredLength) ||
declaredLength < jpegStart.length + jpegEnd.length ||
declaredLength > maxBytes
)
return null;
}
bodyStart = headerEnd + headerTerminator.length;
parsedHeaders = true;
}
const available = buffer.length - bodyStart;
if (declaredLength !== null) {
const nextPart = buffer.indexOf(nextBoundary, bodyStart);
if (nextPart >= bodyStart && nextPart - bodyStart < declaredLength)
return null;
if (available >= declaredLength) {
const frame = buffer.subarray(bodyStart, bodyStart + declaredLength);
if (
frame.subarray(0, jpegStart.length).equals(jpegStart) &&
frame.subarray(-jpegEnd.length).equals(jpegEnd)
)
return Buffer.from(frame);
return null;
}
} else if (available >= jpegStart.length) {
const frameStart = buffer.subarray(
bodyStart,
bodyStart + jpegStart.length,
);
if (!frameStart.equals(jpegStart)) return null;
const nextPart = buffer.indexOf(nextBoundary, bodyStart);
const end = buffer.indexOf(jpegEnd, bodyStart + jpegStart.length);
if (
nextPart >= bodyStart &&
(end < 0 || end + jpegEnd.length > nextPart)
)
return null;
if (end >= 0) {
const frameLength = end + jpegEnd.length - bodyStart;
if (frameLength > maxBytes) return null;
return Buffer.from(buffer.subarray(bodyStart, end + jpegEnd.length));
}
if (available > maxBytes) return null;
}
if (overflow) return null;
}
} finally {
if (reader) {
try {
await reader.cancel();
} catch {
/* already closed or aborted */
}
try {
reader.releaseLock();
} catch {
/* already released */
}
}
}
}
/**
* Fetch one upstream CCTV image within the frame-refresh budget.
*
@@ -660,10 +805,7 @@ export async function fetchCctvImageFromUpstream(
fetchImpl,
);
if (!upstream) return null;
const contentType = cctvRasterImageType(
upstream.headers.get('content-type'),
);
if (!upstream.ok || !contentType) {
if (!upstream.ok) {
try {
await upstream.body?.cancel();
} catch {
@@ -671,9 +813,24 @@ export async function fetchCctvImageFromUpstream(
}
return null;
}
const body = await readCappedResponseBytes(upstream, maxBytes);
const upstreamType = upstream.headers.get('content-type');
const contentType = cctvRasterImageType(upstreamType);
if (contentType) {
const body = await readCappedResponseBytes(upstream, maxBytes);
if (!body) return null;
return { ok: true, body, contentType };
}
if (!cctvLiveMediaType('mjpeg', upstreamType)) {
try {
await upstream.body?.cancel();
} catch {
/* no-op */
}
return null;
}
const body = await readFirstMjpegFrame(upstream, maxBytes);
if (!body) return null;
return { ok: true, body, contentType };
return { ok: true, body, contentType: 'image/jpeg' };
} catch {
return null;
} finally {
+161 -10
View File
@@ -10,6 +10,8 @@ import { readResponseTextCapped } from '../common/http.js';
* standard, published keyless and refreshed daily (~1,800 cameras). */
export const TAIWAN_FREEWAY_CCTV_URL =
'https://tisvcloud.freeway.gov.tw/history/motc20/CCTV.xml';
export const TAIWAN_HIGHWAY_CCTV_URL =
'https://cctv-maintain.thb.gov.tw/opendataCCTVs.xml';
/** The whole network is ~1.2 MB. */
export const TAIWAN_MAX_CATALOG_BYTES = 6 * 1024 * 1024;
const DEFAULT_TAIWAN_MAX_SOURCES = 600;
@@ -35,6 +37,16 @@ const STREAM_TO_FRAME = new Map([
]);
const DIRECTIONS = { N: 'N', S: 'S', E: 'E', W: 'W' };
const TAIWAN_HIGHWAY_CAMERA_HOSTS = new Set([
'cctv-ss01.thb.gov.tw',
'cctv-ss02.thb.gov.tw',
'cctv-ss03.thb.gov.tw',
'cctv-ss04.thb.gov.tw',
'cctv-ss05.thb.gov.tw',
'cctv-ss06.thb.gov.tw',
'cctv-ss07.thb.gov.tw',
'cctv-ss08.thb.gov.tw',
]);
/** Prioritization anchors, north to south, so a cap keeps the whole island
* rather than only greater Taipei. */
@@ -211,6 +223,74 @@ export function taiwanCameraToSource(row) {
};
}
/**
* One Highway Bureau row -> a source whose first MJPEG frame is served as a
* snapshot by the existing camera-frame proxy.
*
* @param {object} row - Output of parseTaiwanCctvXml.
* @returns {?object}
*/
export function taiwanHighwayCameraToSource(row) {
if (!row || typeof row !== 'object') return null;
const { lat, lon } = row;
if (!isLikelyTaiwanCoordinate(lat, lon)) return null;
let url;
try {
url = new URL(String(row.streamUrl ?? ''));
} catch {
return null;
}
if (
url.protocol !== 'https:' ||
url.username ||
url.password ||
url.port ||
url.search ||
url.hash ||
!TAIWAN_HIGHWAY_CAMERA_HOSTS.has(url.hostname) ||
!/^\/[A-Za-z0-9._+~-]{1,100}$/.test(url.pathname) ||
url.pathname.includes('..')
)
return null;
const rawId = String(row.cctvId ?? '').trim();
if (!/^[A-Za-z0-9-]{1,80}$/.test(rawId)) return null;
const cameraId = 'tw-thb-' + rawId.toLowerCase();
const road = String(row.roadName ?? '').trim();
const direction = DIRECTIONS[String(row.direction ?? '').trim()] || '';
const mile = String(row.mile ?? '').trim();
const section = [row.sectionStart, row.sectionEnd]
.map((part) => String(part ?? '').trim())
.filter(Boolean)
.join('→');
const name =
(road || mile || section
? [road, direction, mile, section].filter(Boolean).join(' ')
: '') || rawId;
return {
id: cameraId,
name: name.slice(0, 160),
city: 'Taiwan / 臺灣',
cityId: 'taiwan',
provider: 'Highway Bureau, MOTC',
lat,
lon,
headingDeg: fallbackHeadingFromId(cameraId),
headingConfidence: 'low',
pitchDeg: -18,
fovDeg: 44,
rangeM: 145,
mountHeightM: 8,
feedType: 'mjpeg',
url: url.href,
snapshotUrl: url.href,
sourceKind: 'taiwan-highway-motc',
license: 'Highway Bureau, MOTC open data',
code: cameraDisplayCode((road + ' ' + mile).trim().toUpperCase() || rawId),
};
}
let lastGood = null;
let lastGoodAt = 0;
let retryAfter = 0;
@@ -237,6 +317,49 @@ function taiwanSourceCap() {
: DEFAULT_TAIWAN_MAX_SOURCES;
}
/** Fetch the public provincial-road CCTV table as a no-key fallback. */
async function loadTaiwanHighwaySources() {
const signal = AbortSignal.timeout(CCTV_SOURCE_FETCH_TIMEOUT_MS);
const response = await fetch(TAIWAN_HIGHWAY_CCTV_URL, {
headers: { Accept: 'application/xml,text/xml' },
redirect: 'manual',
signal,
});
if (!response.ok) {
await response.body?.cancel().catch(() => {});
throw new Error(
response.status >= 300 && response.status < 400
? 'Highway Bureau catalog redirected; redirects are not followed'
: 'Highway Bureau catalog HTTP ' + response.status,
);
}
const xml = await readResponseTextCapped(
response,
TAIWAN_MAX_CATALOG_BYTES,
signal,
);
const cameras = new Map();
for (const row of parseTaiwanCctvXml(xml)) {
const camera = taiwanHighwayCameraToSource(row);
if (camera && !cameras.has(camera.id)) cameras.set(camera.id, camera);
}
if (!cameras.size)
throw new Error('no usable Highway Bureau cameras in the table');
const sources = prioritizeSources(
[...cameras.values()],
taiwanSourceCap(),
ANCHORS,
);
console.log(
'[CCTV] Loaded Taiwan Highway Bureau fallback: ' +
cameras.size +
' (using ' +
sources.length +
')',
);
return sources;
}
/**
* Fetch Taiwan Freeway Bureau cameras from the keyless MOTC 2.0 XML table.
* One bounded request per shared catalog refresh. A failed download serves the
@@ -262,7 +385,7 @@ export async function loadTaiwanSources() {
throw new Error(
response.status >= 300 && response.status < 400
? 'catalog redirected; redirects are not followed'
: `HTTP ${response.status}`,
: 'HTTP ' + response.status,
);
}
const xml = await readResponseTextCapped(
@@ -282,7 +405,11 @@ export async function loadTaiwanSources() {
ANCHORS,
);
console.log(
`[CCTV] Loaded Taiwan camera sources: ${cameras.size} (using ${sources.length})`,
'[CCTV] Loaded Taiwan camera sources: ' +
cameras.size +
' (using ' +
sources.length +
')',
);
lastGood = sources;
lastGoodAt = Date.now();
@@ -291,13 +418,37 @@ export async function loadTaiwanSources() {
} catch (error) {
retryAfter = Date.now() + TAIWAN_RETRY_AFTER_FAILURE_MS;
const fallback = fallbackSources();
console.warn(
`[CCTV] Taiwan camera download failed (${error?.message || error}); ` +
(fallback.length
? `serving the last good ${fallback.length} cameras`
: 'no Taiwan cameras') +
', next attempt in 1 h',
);
return fallback;
if (fallback.length) {
console.warn(
'[CCTV] Taiwan Freeway camera download failed (' +
(error?.message || error) +
'); serving the last good ' +
fallback.length +
' cameras, next attempt in 1 h',
);
return fallback;
}
try {
const sources = await loadTaiwanHighwaySources();
lastGood = sources;
lastGoodAt = Date.now();
console.warn(
'[CCTV] Taiwan Freeway camera download failed (' +
(error?.message || error) +
'); using ' +
sources.length +
' Highway Bureau cameras as fallback, next attempt in 1 h',
);
return sources;
} catch (fallbackError) {
console.warn(
'[CCTV] Taiwan Freeway camera download failed (' +
(error?.message || error) +
'); Highway Bureau fallback failed (' +
(fallbackError?.message || fallbackError) +
'); no Taiwan cameras, next attempt in 1 h',
);
return [];
}
}
}
+156
View File
@@ -313,3 +313,159 @@ test('CCTV upstream frame fetch accepts only raster image types and sends their
assert.equal(state.cancelled, true, `${contentType}: body cancelled`);
}
});
function multipartResponse(contentType, chunks, { onCancel = () => {} } = {}) {
let nextChunk = 0;
const body = new ReadableStream(
{
pull(controller) {
if (nextChunk >= chunks.length) {
controller.close();
return;
}
controller.enqueue(chunks[nextChunk++]);
},
cancel() {
onCancel();
},
},
{ highWaterMark: 0 },
);
return new Response(body, {
status: 200,
headers: { 'Content-Type': contentType },
});
}
function jpegPart(boundary, body, contentType = 'image/jpeg', length = body.length) {
return Buffer.concat([
Buffer.from(
'--' +
boundary +
'\r\nContent-Type: ' +
contentType +
'\r\nContent-Length: ' +
length +
'\r\n\r\n',
),
body,
Buffer.from('\r\n'),
]);
}
test('CCTV upstream frame fetch extracts only the first JPEG from MJPEG', async () => {
const boundary = 'DIGIEVER';
const first = Buffer.from([0xff, 0xd8, 0x11, 0x22, 0xff, 0xd9]);
const second = Buffer.from([0xff, 0xd8, 0x33, 0x44, 0xff, 0xd9]);
for (const type of [
'multipart/x-mixed-replace; boundary=DIGIEVER',
'multipart/x-mixed-replace; boundary="DIGIEVER"',
]) {
let cancelled = false;
const result = await fetchCctvImageFromUpstream(
'https://cctv-ss02.thb.gov.tw/T62-9K+020',
{
maxBytes: 1024,
fetchImpl: async () =>
multipartResponse(
type,
[
jpegPart(boundary, first),
jpegPart(boundary, second),
Buffer.from('--' + boundary + '--\r\n'),
],
{ onCancel: () => { cancelled = true; } },
),
},
);
assert.equal(result?.contentType, 'image/jpeg', type);
assert.deepEqual(result?.body, first, type);
assert.equal(cancelled, true, type);
}
});
test('CCTV upstream frame fetch rejects malformed and non-JPEG MJPEG parts', async () => {
const jpeg = Buffer.from([0xff, 0xd8, 0x01, 0x02, 0xff, 0xd9]);
for (const [type, chunks] of [
['multipart/x-mixed-replace', []],
[
'multipart/x-mixed-replace; boundary=DIGIEVER',
[jpegPart('DIGIEVER', jpeg, 'image/png')],
],
]) {
let cancelled = false;
const result = await fetchCctvImageFromUpstream(
'https://cctv-ss02.thb.gov.tw/T62-9K+020',
{
fetchImpl: async () =>
multipartResponse(type, chunks, { onCancel: () => { cancelled = true; } }),
},
);
assert.equal(result, null, type);
assert.equal(cancelled, true, type);
}
});
test('CCTV upstream frame fetch rejects truncated or oversized MJPEG parts', async () => {
const boundary = 'DIGIEVER';
const full = Buffer.from([0xff, 0xd8, 0x01, 0x02, 0xff, 0xd9]);
const truncated = Buffer.from([0xff, 0xd8, 0x01, 0x02]);
for (const [chunks, maxBytes] of [
[jpegPart(boundary, truncated, 'image/jpeg', full.length), 1024],
[jpegPart(boundary, full), 5],
]) {
let cancelled = false;
const result = await fetchCctvImageFromUpstream(
'https://cctv-ss02.thb.gov.tw/T62-9K+020',
{
maxBytes,
fetchImpl: async () =>
multipartResponse(
'multipart/x-mixed-replace; boundary=' + boundary,
[chunks],
{ onCancel: () => { cancelled = true; } },
),
},
);
assert.equal(result, null);
assert.equal(cancelled, true);
}
});
test('CCTV upstream frame fetch does not cross into a later part when the first JPEG is truncated', async () => {
const boundary = 'DIGIEVER';
const truncated = Buffer.from([0xff, 0xd8, 0x01, 0x02]);
const valid = Buffer.from([0xff, 0xd8, 0x03, 0x04, 0xff, 0xd9]);
const wire = Buffer.concat([
Buffer.from(
'--' + boundary + '\r\nContent-Type: image/jpeg\r\n\r\n',
),
truncated,
Buffer.from(
'\r\n--' +
boundary +
'\r\nContent-Type: image/jpeg\r\nContent-Length: ' +
valid.length +
'\r\n\r\n',
),
valid,
Buffer.from('\r\n--' + boundary + '--\r\n'),
]);
let cancelled = false;
const result = await fetchCctvImageFromUpstream(
'https://cctv-ss02.thb.gov.tw/T62-9K+020',
{
maxBytes: 1024,
fetchImpl: async () =>
multipartResponse(
'multipart/x-mixed-replace; boundary=' + boundary,
[wire],
{ onCancel: () => { cancelled = true; } },
),
},
);
assert.equal(result, null);
assert.equal(cancelled, true);
});
+87 -3
View File
@@ -182,6 +182,86 @@ test('loader deduplicates cameras that share a camera number', async (t) => {
]);
});
test('falls back to Highway Bureau MJPEG cameras when the Freeway catalog fails', async (t) => {
const highwayUrl = 'https://cctv-maintain.thb.gov.tw/opendataCCTVs.xml';
const requests = [];
t.mock.method(console, 'log', () => {});
t.mock.method(console, 'warn', () => {});
t.mock.method(globalThis, 'fetch', async (url, options) => {
const href = String(url);
requests.push(href);
assert.equal(options.redirect, 'manual');
if (href === TAIWAN_FREEWAY_CCTV_URL)
return new Response('', { status: 503 });
if (href === highwayUrl) {
return new Response(
cctvXml(
block({
id: 'CCTV-14-0620-009-002',
stream: 'https://cctv-ss02.thb.gov.tw:443/T62-9K+020',
lat: '24.9967',
lon: '121.4521',
}),
),
{ headers: { 'content-type': 'application/xml' } },
);
}
throw new Error('Unexpected Taiwan feed URL: ' + href);
});
const sources = await loadTaiwanSources();
assert.deepEqual(requests, [TAIWAN_FREEWAY_CCTV_URL, highwayUrl]);
assert.equal(sources.length, 1);
assert.equal(sources[0].id, 'tw-thb-cctv-14-0620-009-002');
assert.equal(sources[0].cityId, 'taiwan');
assert.equal(sources[0].provider, 'Highway Bureau, MOTC');
assert.equal(sources[0].feedType, 'mjpeg');
assert.equal(sources[0].sourceKind, 'taiwan-highway-motc');
assert.equal(new URL(sources[0].url).hostname, 'cctv-ss02.thb.gov.tw');
assert.equal(new URL(sources[0].url).pathname, '/T62-9K+020');
});
test('the Highway Bureau fallback registers only approved HTTPS camera servers', async (t) => {
const highwayUrl = 'https://cctv-maintain.thb.gov.tw/opendataCCTVs.xml';
t.mock.method(console, 'log', () => {});
t.mock.method(console, 'warn', () => {});
t.mock.method(globalThis, 'fetch', async (url) => {
if (String(url) === TAIWAN_FREEWAY_CCTV_URL)
return new Response('', { status: 503 });
if (String(url) === highwayUrl) {
return new Response(
cctvXml(
block({
id: 'CCTV-14-0620-009-001',
stream: 'https://cctv-ss09.thb.gov.tw/T62-9K+019',
}),
block({
id: 'CCTV-14-0620-009-003',
stream: 'http://cctv-ss02.thb.gov.tw/T62-9K+021',
}),
block({
id: 'CCTV-14-0620-009-004',
stream: 'https://cctv-ss02.thb.gov.tw:8443/T62-9K+022',
}),
block({
id: 'CCTV-14-0620-009-002',
stream: 'https://cctv-ss02.thb.gov.tw:443/T62-9K+020',
}),
),
{ headers: { 'content-type': 'application/xml' } },
);
}
throw new Error('Unexpected Taiwan feed URL: ' + String(url));
});
const sources = await loadTaiwanSources();
assert.deepEqual(
sources.map((source) => source.id),
['tw-thb-cctv-14-0620-009-002'],
);
});
test('a redirected or failed catalog contributes an empty pack', async (t) => {
t.mock.method(console, 'warn', () => {});
for (const status of [302, 503]) {
@@ -214,13 +294,17 @@ test('the Taiwan pack is in the catalog by default', async (t) => {
);
});
test('CCTV_TAIWAN_ENABLED=0 makes no Freeway Bureau request', async (t) => {
test('CCTV_TAIWAN_ENABLED=0 makes no Taiwan catalog request', async (t) => {
const { requested, sources } = await runCatalogWithTaiwanFlag(t, '0');
assert.ok(requested.length > 0, 'the other packs still refreshed');
assert.equal(
requested.some((href) => href.includes('freeway.gov.tw')),
false,
);
assert.equal(
requested.some((href) => href.includes('cctv-maintain.thb.gov.tw')),
false,
);
assert.ok(!sources.some((c) => c.cityId === 'taiwan'));
});
@@ -250,12 +334,12 @@ test('a failed download is not retried until the backoff ends', async (t) => {
t.mock.method(Date, 'now', () => 1_000_000);
assert.deepEqual(await loadTaiwanSources(), []);
assert.deepEqual(await loadTaiwanSources(), []);
assert.equal(calls, 1, 'the second refresh made no request');
assert.equal(calls, 2, 'the second refresh made no request');
Date.now.mock.mockImplementation(
() => 1_000_000 + TAIWAN_RETRY_AFTER_FAILURE_MS + 1,
);
await loadTaiwanSources();
assert.equal(calls, 2, 'the backoff ends after an hour');
assert.equal(calls, 4, 'the backoff ends after an hour');
});
test('a failed download serves the last good table', async (t) => {