Compare commits

...
Author SHA1 Message Date
Claude 541bb78282 fix(intro): keep restricted volumes to areas that fit the scene
Fort Bragg's reservation polygon is larger than the scene, which turned
every road into a convoy and floated curtain walls past the ground edge.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01KqP857pippaUs6konSDfe8
2026-10-19 02:14:10 +00:00
Claude a0f8fcc2ae chore(intro): final city fetch scripts (Overpass fallback endpoints)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01KqP857pippaUs6konSDfe8
2026-10-19 01:14:23 +00:00
Claude 02e535cb55 feat(intro): v2 — cities, accelerating sweep, detections, thriller sound design
- City engine: Esri ground, OSM buildings that rise with the scan, road
  ribbons carrying modelled traffic coloured by congestion, NYC DOT cameras,
  real ADS-B, MBTA GTFS lines + realtime vehicles, LAX runways with approach
  strobes, Fort Bragg restricted volumes, Everest terrain with contours.
- Indexing keeps accelerating into a 260-capture global sweep while the
  planet whips a full turn.
- The eye opens: aperture blades, sensor array waking, optics calibration,
  radar sweep, orbiting coordinate ring.
- Camera wall with YOLOv8n detections and a city-pulse trace.
- Ends on GOD'S EYE VIEW / OMNISCIENCE.
- audio/sfx.py: sound design only, built to sit under a music track.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01KqP857pippaUs6konSDfe8
2026-10-19 01:10:39 +00:00
Claude e9ddc37788 fix(intro): punch exposure on impacts instead of adding a grey veil
Additive flashes lifted the blacks to visible grey after gamma; scale the
exposure instead so only lit pixels brighten. Also drop the scratch GL test.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01KqP857pippaUs6konSDfe8
2026-10-18 03:37:49 +00:00
Claude c9421a3f33 feat(intro): add the First Light intro sequence
A deterministic WebGL film of God's Eye View coming online: a seed on Null
Island grows into a culture that draws the planet, indexes the surface
tile by tile, ingests the live feeds and outgrows its dish, ending on the
logo. Renders 16:9 and 9:16 frame by frame in headless Chromium; the
procedural score is locked to the same clock. Data and imagery are
fetched by the scripts in tools/ and are not committed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01KqP857pippaUs6konSDfe8
2026-10-18 03:21:30 +00:00
60 changed files with 22987 additions and 0 deletions

No files matched your search

+22
View File
@@ -0,0 +1,22 @@
# Fetched snapshots and renders are regenerated by tools/; imagery and camera
# stills carry third-party terms and are not redistributed with the repo.
node_modules/
/out/
data/*.json
data/*.txt
!data/SOURCES.md
/imagery/
/cctv/
audio/*.wav
tools/imagery/cache/
__pycache__/
# fetch caches
tools/*/cache/
# city snapshots (Esri imagery, live feeds) and review/model artefacts
/city/
tools/flurry/review/
tools/flurry/*.jpg
tools/live/models/
tools/live/*.jpg
tools/*/check*.png
+58
View File
@@ -0,0 +1,58 @@
# God's Eye View — First Light
The intro sequence: God's Eye View coming online for the first time. A seed of
light on Null Island grows into a culture that turns out to be drawing the
planet. It then indexes the surface tile by tile, ingests every public feed,
and outgrows the dish it was grown in.
Every number on screen comes from a real snapshot: 5,550 aircraft from OpenSky,
15,928 satellites from CelesTrak, 246,411 fire detections from FIRMS, 10,120
quakes from USGS, GFS wind, 1,948 subsea cable segments, AIS from Kystverket and
Digitraffic, and 117 real public camera frames. Those cameras come from a
catalog of 16,607 across 18 keyless agencies. The imagery is Esri World Imagery
and NASA GIBS, fetched keyless. The score is synthesised from scratch and
frame-locked to the same clock as the picture.
## Acts
| Time | Act | What happens |
|---|---|---|
| 0–5 s | Void | Boot console types, then implodes into a point |
| 5–16 s | Seed / culture | Heartbeat, petri-dish ring. Mycelium grows out from 0°N 0°E; real night lights and coastlines light up as the front passes; the culture learns city names |
| 16–30 s | Indexing | Quadtree tiles z3→z5 index in with rising resolution; the sun model is applied; 24 synthetic Esri captures ("enhance" z13→z18); full-frame Sobel feature extraction over Midtown |
| 30–52 s | Ingestion | Aircraft, vessels, military, satellites, quakes, fires, wind and cloud, subsea cables, public cameras. Each feed has its own visual and sonic signature. Ends on the camera wall |
| 52–60 s | Saturation | Every layer at once, data arcs, the digital-twin lattice, and the dish ring's return; containment falls, the ring shatters, OMNIPRESENT |
| 61–72 s | Resolve | The planet becomes the iris of the logo. GOD'S EYE VIEW · NO PLACE LEFT BEHIND · ONLINE |
## Rebuild
```sh
npm install # three, playwright-core, satellite.js, fonts
python3 -m http.server 8765 --bind 127.0.0.1 &
# data + imagery snapshots (see data/SOURCES.md)
bash tools/fetch/fetch_basic.sh # plus the other scripts in tools/fetch, tools/imagery
node tools/export-cues.mjs # film clock -> audio/cues.json
tools/render-all.sh 1920 1080 16x9 # deterministic frames -> out/video_16x9.mp4
tools/render-all.sh 1080 1920 9x16
tools/finalize.sh # score, -14 LUFS, mux -> out/final/
```
Single stills for review:
`node tools/render.mjs --w 1920 --h 1080 --frames 12s,30s,58.4s --out out/stills`
## Real-time / loading screen
`index.html?play` runs the same frames live in a GPU browser, driven by the audio
clock (the first click starts the sound). `?w=1080&h=1920&play` gives the
vertical cut.
## Layout
- `src/timeline.js`: the clock. Every visual and audio event keys off it.
- `src/main.js`: asset loading, camera choreography, per-frame state.
- `src/globe.js`: planet shader (substrate → growth → quadtree indexing → lit Earth), clouds, atmosphere.
- `src/lines.js`: coastline and mycelium growth fronts, cables, arcs, twin lattice.
- `src/layers.js`: aircraft, vessels, satellites, quakes, fires, wind particles, stars.
- `src/hud.js`: all typography and 2D: boot console, dish, captures, feed stack, camera wall, title.
- `src/post.js`: HDR composite, dual-filter bloom, lens, grain, glitch.
- `audio/score.py`: procedural score and sound design.
File diff suppressed because it is too large. Load diff
+679
View File
@@ -0,0 +1,679 @@
#!/usr/bin/env python3
"""God's Eye View — First Light: procedural score and sound design.
Everything here is synthesised from scratch (no samples) and placed against
audio/cues.json, which tools/export-cues.mjs exports from src/timeline.js, so
every hit lands on its frame.
python3 audio/score.py -> audio/score.wav (48 kHz, 24-bit stereo)
"""
import json
import os
import numpy as np
from scipy import signal
from scipy.io import wavfile
SR = 48000
HERE = os.path.dirname(os.path.abspath(__file__))
C = json.load(open(os.path.join(HERE, 'cues.json')))
T = C['T']
DUR = C['DURATION']
N = int(SR * DUR)
rng = np.random.default_rng(20261018)
# ---------------------------------------------------------------- buses
def bus():
return np.zeros((2, N), dtype=np.float64)
B = {k: bus() for k in ('drone', 'music', 'perc', 'hits', 'sfx', 'ui', 'verb')}
def place(dst, sig, t, gain=1.0, pan=0.0, verb=0.0):
"""Mix mono/stereo `sig` into bus `dst` at time t (s), equal-power pan."""
if sig.ndim == 1:
p = (pan + 1) * np.pi / 4
sig = np.vstack([sig * np.cos(p), sig * np.sin(p)])
i0 = int(round(t * SR))
if i0 >= N:
return
s0 = max(0, -i0)
i0 = max(0, i0)
n = min(sig.shape[1] - s0, N - i0)
if n <= 0:
return
seg = sig[:, s0:s0 + n] * gain
B[dst][:, i0:i0 + n] += seg
if verb > 0:
B['verb'][:, i0:i0 + n] += seg * verb
def tt(dur):
return np.arange(int(dur * SR)) / SR
def note(name):
names = {'C': -9, 'C#': -8, 'Db': -8, 'D': -7, 'D#': -6, 'Eb': -6, 'E': -5, 'F': -4, 'F#': -3, 'Gb': -3, 'G': -2, 'G#': -1, 'Ab': -1, 'A': 0, 'A#': 1, 'Bb': 1, 'B': 2}
n, o = name[:-1], int(name[-1])
return 440.0 * 2 ** ((names[n] + (o - 4) * 12) / 12)
def sos_lp(fc, order=2):
return signal.butter(order, min(fc, SR * 0.45), 'low', fs=SR, output='sos')
def sos_hp(fc, order=2):
return signal.butter(order, max(fc, 10), 'high', fs=SR, output='sos')
def sos_bp(lo, hi, order=2):
return signal.butter(order, [max(lo, 10), min(hi, SR * 0.45)], 'band', fs=SR, output='sos')
def filt(x, sos):
return signal.sosfilt(sos, x, axis=-1)
def sweep_filter(x, cut_fn, kind='low', block=256, order=2):
"""Time-varying Butterworth: coefficients per block, state carried across."""
y = np.zeros_like(x)
zi = None
for b in range(0, len(x), block):
tc = b / SR
fc = float(cut_fn(tc))
if kind == 'low':
sos = sos_lp(fc, order)
elif kind == 'high':
sos = sos_hp(fc, order)
else:
bw = fc * 0.35
sos = sos_bp(fc - bw, fc + bw, order)
if zi is None:
zi = signal.sosfilt_zi(sos) * 0
y[b:b + block], zi = signal.sosfilt(sos, x[b:b + block], zi=zi)
return y
def saw(f, dur, phase=0.0):
t = tt(dur)
if np.isscalar(f):
ph = f * t + phase
else:
ph = np.cumsum(f) / SR + phase
return 2 * (ph - np.floor(ph + 0.5))
def sine(f, dur, phase=0.0):
t = tt(dur)
if np.isscalar(f):
return np.sin(2 * np.pi * f * t + phase)
return np.sin(2 * np.pi * np.cumsum(f) / SR + phase)
def noise(dur):
return rng.standard_normal(int(dur * SR))
def env_exp(dur, decay, attack=0.002):
t = tt(dur)
a = np.minimum(1, t / max(attack, 1e-4))
return a * np.exp(-t * decay)
def env_ar(dur, attack, release, curve=2.0):
t = tt(dur)
up = np.clip(t / max(attack, 1e-4), 0, 1) ** curve
down = np.clip((dur - t) / max(release, 1e-4), 0, 1) ** curve
return np.minimum(up, down)
# ---------------------------------------------------------------- instruments
def click(freq=4000, dur=0.012, q=0.5):
x = noise(dur) * env_exp(dur, 900, 0.0003)
return filt(x, sos_bp(freq * (1 - q * 0.5), freq * (1 + q)))
def blip(f, dur=0.05, harm=0.25, decay=60):
t = tt(dur)
e = env_exp(dur, decay, 0.001)
return (np.sin(2 * np.pi * f * t) + harm * np.sin(4 * np.pi * f * t)) * e
def thump(f0=70, f1=34, dur=0.9, decay=5.0, punch=24):
t = tt(dur)
f = f1 + (f0 - f1) * np.exp(-t * punch)
body = sine(f, dur) * env_exp(dur, decay, 0.001)
tr = filt(noise(0.02), sos_lp(2500)) * env_exp(0.02, 200)
out = body
out[: len(tr)] += tr * 0.4
return out
def braam(dur=4.0, notes=('D1', 'A1', 'D2', 'A2'), bright=2600, gain=1.0, decay=1.6):
"""Detuned saw stack through a fast-opening, slow-closing lowpass + drive."""
n = int(dur * SR)
acc = np.zeros(n)
for nm in notes:
f = note(nm)
for d in (-0.012, -0.005, 0.0, 0.006, 0.013):
acc += saw(f * (1 + d), dur, rng.random())
acc /= len(notes) * 3.0
cut = lambda tc: 160 + (bright - 160) * min(1.0, tc / 0.07) * np.exp(-max(0.0, tc - 0.07) * decay)
y = sweep_filter(acc, cut, 'low', order=2)
y = np.tanh(y * 2.2) * env_ar(dur, 0.012, dur * 0.85, 1.2)
sub = sine(note(notes[0]) * np.ones(n) * (1 + 0.5 * np.exp(-tt(dur) * 18)), dur) * env_exp(dur, 1.3, 0.004)
imp = filt(noise(0.6), sos_lp(900)) * env_exp(0.6, 9)
out = y * 0.85 + sub * 0.9
out[: len(imp)] += imp * 0.5
return out * gain
def whoosh(dur=1.2, f0=400, f1=4000, peak=0.5):
x = noise(dur)
t0 = tt(dur)
cut = lambda tc: f0 * (f1 / f0) ** min(1.0, tc / dur)
y = sweep_filter(x, cut, 'band', order=2)
shape = np.exp(-((t0 / dur - peak) ** 2) / 0.05)
return y * shape
def riser(dur=3.0, f0=200, f1=6000):
x = noise(dur)
cut = lambda tc: f0 * (f1 / f0) ** ((tc / dur) ** 1.6)
y = sweep_filter(x, cut, 'band', order=2)
t0 = tt(dur)
tone = sine(f0 * (f1 / f0 / 8) ** ((t0 / dur) ** 1.6) * np.ones_like(t0), dur) * 0.2
return (y + tone) * (t0 / dur) ** 2.2
def shepard(dur, rate_oct_per_s, base=40.0, layers=9, accel=0.0):
t0 = tt(dur)
pos = rate_oct_per_s * t0 + 0.5 * accel * t0 ** 2
out = np.zeros_like(t0)
for k in range(layers):
o = (k + pos) % layers
f = base * 2 ** o
a = np.exp(-((o - layers / 2) ** 2) / (2 * (layers / 5) ** 2))
ph = np.cumsum(f) / SR
out += np.sin(2 * np.pi * ph) * a
return out / layers * 2
def fm_bell(f, dur=2.0, ratio=3.5, index=4.0, decay=2.5):
t = tt(dur)
e = env_exp(dur, decay, 0.002)
mod = np.sin(2 * np.pi * f * ratio * t) * index * np.exp(-t * 3)
return np.sin(2 * np.pi * f * t + mod) * e
def pluck(f, dur=0.22, cutoff=1800, decay=14):
x = saw(f, dur, rng.random()) + 0.5 * saw(f * 1.005, dur, rng.random())
e = env_exp(dur, decay, 0.002)
cut = lambda tc: 120 + cutoff * np.exp(-tc * 22)
return sweep_filter(x * e, cut, 'low', block=128, order=2) * 0.7
def pad(freqs, dur, cutoff=900, attack=1.2, release=1.5):
n = int(dur * SR)
acc = np.zeros(n)
for f in freqs:
for d in (-0.004, 0.0, 0.0045):
acc += saw(f * (1 + d), dur, rng.random())
acc /= len(freqs) * 3
y = filt(acc, sos_lp(cutoff, 2))
return y * env_ar(dur, attack, release, 1.5)
def static(dur, lo=1500, hi=9000):
return filt(noise(dur), sos_bp(lo, hi))
def stereo_noise_bed(dur, lo, hi):
return np.vstack([filt(noise(dur), sos_bp(lo, hi)), filt(noise(dur), sos_bp(lo, hi))])
# =================================================================== ACT 0
# room tone that the whole film sits on
bed = stereo_noise_bed(DUR, 30, 140) * 0.05
ramp_bed = np.clip(tt(DUR) / 3.0, 0, 1)
B['drone'] += bed * ramp_bed
# sub drone: D1 + A1, breathing; intensity rises with the film, dies at the cut
t_all = tt(DUR)
intensity = np.interp(t_all, [0, 5, 9, 16, 30, 52, 60.4, 60.5, 61.2, 66, 72], [0.0, 0.25, 0.45, 0.6, 0.7, 0.85, 1.0, 0.0, 0.0, 0.7, 0.0])
dr = (np.sin(2 * np.pi * note('D1') * t_all) * 0.6 + np.sin(2 * np.pi * note('A1') * 1.001 * t_all) * 0.25 + np.sin(2 * np.pi * note('D2') * 0.999 * t_all) * 0.18)
dr *= 0.85 + 0.15 * np.sin(2 * np.pi * 0.11 * t_all)
B['drone'] += np.vstack([dr, dr]) * intensity * 0.35
# electrical hum under the boot console
hum_e = np.interp(t_all, [0.4, 1.2, T['collapse'], T['seed']], [0, 1, 1, 0], left=0, right=0)
hum = (sine(100, DUR) * 0.3 + sine(150, DUR) * 0.12 + filt(noise(DUR), sos_bp(90, 400)) * 0.05) * hum_e
place('sfx', hum, 0, 0.18)
# keystrokes
for k in C['boot']:
if k['ch'] == ' ':
continue
f = 3500 + rng.random() * 2500
place('ui', click(f, 0.01), k['t'], 0.22 + rng.random() * 0.08, pan=(rng.random() - 0.5) * 0.3)
# "OK" confirmation
place('ui', blip(1760, 0.09) * 0.6 + blip(2640, 0.09) * 0.3, T['boot'] + 0.75 + 41 / 55, 0.25, verb=0.2)
# cursor appears
place('ui', blip(1200, 0.03), T['cursor'], 0.12)
# collapse: suck-in riser then silence, then the seed ignites
place('sfx', riser(T['seed'] - T['collapse'] + 0.05, 300, 9000)[::-1][::-1] * 1.0, T['collapse'], 0.35)
rev = whoosh(0.6, 6000, 300, 0.85)
place('sfx', rev, T['collapse'] - 0.15, 0.4)
place('hits', thump(90, 32, 1.4, 3.5), T['seed'], 0.85, verb=0.3)
place('ui', fm_bell(note('D6'), 2.5, 2.0, 1.5, 1.6), T['seed'], 0.16, verb=0.6)
# heartbeat
for bt, g in ((T['beat1'], 0.9), (T['beat2'], 1.0)):
place('hits', thump(62, 30, 0.7, 7.5, 30), bt, g * 1.4, verb=0.25)
place('hits', thump(52, 28, 0.6, 9, 30), bt + 0.19, g * 0.55, verb=0.2)
# dish ticks: 120 ticks swept around the ring, panned with the sweep
for i in range(120):
k = i / 120
# easeInOut timing of the sweep
u = np.interp(k, np.linspace(0, 1, 400), [((4 * x ** 3) if x < 0.5 else 1 - (-2 * x + 2) ** 3 / 2) for x in np.linspace(0, 1, 400)])
tk = T['dishDraw'] + u * (T['dishDone'] - T['dishDraw'])
loud = 0.12 if i % 10 == 0 else 0.05
place('ui', click(6500 if i % 10 else 3800, 0.008), tk, loud, pan=np.sin(k * 2 * np.pi) * 0.8)
# =================================================================== ACT 1 growth
gs, ge = T['growStart'], T['growEnd']
# granular crackle, density following exponential growth
for _ in range(2600):
x = rng.random() ** 0.45 # skewed late
tc = gs + x * (ge - gs + 0.3)
k = (np.exp(3 * (tc - gs) / (ge - gs)) - 1) / (np.exp(3) - 1)
if rng.random() > 0.15 + 0.85 * k:
continue
f = 1800 + rng.random() * 7000
place('sfx', click(f, 0.006 + rng.random() * 0.01, 0.8), tc, 0.05 + 0.1 * rng.random() * (0.4 + k), pan=rng.uniform(-0.9, 0.9))
# organic tone: saw cluster gliding up an octave, filter opening
dur = ge - gs + 0.6
tg = tt(dur)
glide = note('D2') * 2 ** (((tg / dur) ** 1.8))
cl = (saw(glide, dur) + saw(glide * 1.007, dur) + saw(glide * 1.5, dur) * 0.4)
cl = sweep_filter(cl, lambda tc: 150 + 2200 * (tc / dur) ** 2, 'low', order=2)
cl *= env_ar(dur, 2.5, 0.35, 1.4) * 0.18
place('music', cl, gs, 1.0, verb=0.25)
# city pings as the culture learns names
for i in range(30):
tc = 10.2 + (i / 30) ** 0.8 * 5.2 + rng.random() * 0.1
place('ui', blip(2000 + rng.choice([0, 300, 500, 700, 1000]), 0.06, 0.2, 45), tc, 0.08, pan=rng.uniform(-0.8, 0.8), verb=0.25)
# reverse swell into the substrate hit
place('sfx', riser(1.6, 120, 5000), T['substrate'] - 1.6, 0.3)
# =================================================================== hits
place('hits', braam(4.5, ('D1', 'A1', 'D2'), 1800, 1.0, 1.4), T['substrate'], 0.75, verb=0.35)
# =================================================================== ACT 2 indexing
BPM = 120.0
beat = 60 / BPM
s16 = beat / 4
grid0 = 17.0
pattern = ['D3', 'D3', 'F3', 'D3', 'A3', 'D3', 'C4', 'D3', 'D3', 'D3', 'F3', 'D3', 'G3', 'D3', 'A3', 'F3']
prog = [('D', ['D3', 'F3', 'A3']), ('Bb', ['Bb2', 'D3', 'F3']), ('G', ['G2', 'Bb2', 'D3']), ('A', ['A2', 'C#3', 'E3'])]
def ostinato(t0, t1, cutoff_fn, gain_fn, transpose=0, every=1):
i = 0
t = t0
while t < t1 - 1e-6:
if i % every == 0:
nm = pattern[i % 16]
bar = int((t - grid0) / (beat * 8)) % 4
f = note(nm) * 2 ** (transpose / 12)
# follow the chord root on bar changes
if prog[bar][0] == 'Bb' and nm == 'A3':
f = note('Bb3')
if prog[bar][0] == 'A' and nm == 'C4':
f = note('C#4')
acc = 1.0 if i % 4 == 0 else 0.7
place('music', pluck(f, 0.25, cutoff_fn(t)), t, gain_fn(t) * acc, pan=0.25 if i % 2 else -0.25, verb=0.12)
i += 1
t += s16
ostinato(17.0, 29.9, lambda t: 500 + 2200 * ((t - 17) / 13) ** 1.5, lambda t: 0.06 + 0.1 * (t - 17) / 13)
# pads under indexing
for k, t0 in enumerate(np.arange(17.0, 30.0, 4.0)):
nm, ch = prog[k % 4]
place('music', pad([note(x) for x in ch], 4.3, 700 + k * 120, 1.0, 1.2), t0, 0.12, verb=0.3)
# tile flips: a spray of pitched blips, hemisphere in view only
scale = [note(x) for x in ('D6', 'F6', 'G6', 'A6', 'C7', 'D7')]
for tl in C['tiles']:
if not tl['vis'] or tl['t'] > 26.5:
continue
z = tl['z']
if z == 5 and rng.random() > 0.35:
continue
f = scale[rng.integers(0, len(scale))] / (2 if z == 3 else 1) * (1 if z < 5 else 1.5)
g = {3: 0.07, 4: 0.04, 5: 0.025}[z]
place('ui', blip(f, 0.05, 0.1, 70), tl['t'], g, pan=tl['pan'] * 0.8, verb=0.1)
if z == 3:
place('ui', click(3000, 0.012), tl['t'], 0.06, pan=tl['pan'] * 0.8)
def shutter(t, g=1.0, pan=0.3):
place('sfx', click(3800, 0.008, 0.6), t, 0.5 * g, pan)
place('sfx', click(2200, 0.014, 0.6), t + 0.045, 0.4 * g, pan)
place('sfx', thump(160, 90, 0.05, 60, 80), t, 0.15 * g, pan)
place('ui', blip(2637, 0.05, 0.1, 60), t + 0.07, 0.12 * g, pan)
for cp in C['caps']:
if cp['slow']:
t0 = cp['t']
shutter(t0, 1.0, -0.35)
# the enhance: rising sweep with data chatter
d = 1.05
tz = tt(d)
f = 300 * (8 ** ((tz / d) ** 1.5))
sw = sine(f, d) * 0.5 + sine(f * 1.5, d) * 0.15
sw *= env_ar(d, 0.1, 0.15) * (0.7 + 0.3 * np.sin(2 * np.pi * 18 * tz))
place('sfx', sw, t0 + 0.28, 0.06, -0.3, verb=0.2)
for j in range(16):
place('ui', blip(1500 + rng.random() * 3000, 0.02, 0, 120), t0 + 0.3 + j * 0.06, 0.05, rng.uniform(-0.6, 0.2))
place('sfx', whoosh(0.5, 800, 5000, 0.5), t0 + 1.35, 0.18, 0.4)
place('ui', click(5000, 0.01), t0 + 1.75, 0.2, 0.6)
else:
shutter(cp['t'], 0.55, 0.6)
# full-frame feature extraction
cf = T['capFull']
place('sfx', whoosh(0.7, 300, 3000, 0.4), cf, 0.3)
place('hits', thump(80, 40, 0.8, 6), cf + 0.5, 0.35, verb=0.3)
scan_d = 1.6
ts = tt(scan_d)
scan = (sine(220, scan_d) + 0.5 * sine(440 * (1 + 0.002 * np.sin(2 * np.pi * 6 * ts)), scan_d) + 0.3 * sine(1320, scan_d)) * env_ar(scan_d, 0.08, 0.2)
scan *= 0.6 + 0.4 * np.sin(2 * np.pi * 12 * ts)
place('sfx', scan, cf + 0.75, 0.07, 0, verb=0.2)
for j in range(70):
place('ui', click(5000 + rng.random() * 3000, 0.006), cf + 0.75 + j * (1.6 / 70), 0.06, rng.uniform(-0.5, 0.5))
# collapse into the planet
place('sfx', riser(0.9, 300, 8000), T['indexed'] - 0.9, 0.4)
place('sfx', whoosh(0.5, 6000, 200, 0.8), T['indexed'] - 0.45, 0.35)
# SURFACE INDEXED
place('hits', braam(5.0, ('D1', 'A1', 'D2', 'F2'), 2600, 1.0, 1.1), T['indexed'], 0.95, verb=0.4)
# =================================================================== ACT 3 ingestion
ostinato(30.5, 52.4, lambda t: 900 + 2600 * min(1, (t - 30.5) / 18), lambda t: 0.09 + 0.06 * (t - 30.5) / 22)
# bass
for k, t0 in enumerate(np.arange(30.5, 52.4, beat * 2)):
bar = int((t0 - grid0) / (beat * 8)) % 4
root = {'D': 'D1', 'Bb': 'Bb0', 'G': 'G1', 'A': 'A1'}[prog[bar][0]]
b = saw(note(root), 0.9) + saw(note(root) * 1.004, 0.9)
b = filt(b, sos_lp(260)) * env_exp(0.9, 3.5, 0.005)
place('music', b, t0, 0.22)
for k, t0 in enumerate(np.arange(30.5, 52.4, 4.0)):
nm, ch = prog[k % 4]
place('music', pad([note(x) for x in ch] + [note(ch[0][:-1] + str(int(ch[0][-1]) + 1))], 4.3, 1100, 0.8, 1.2), t0, 0.12, verb=0.3)
# percussion: low taiko-ish hits + ticks
for i, t0 in enumerate(np.arange(30.5, 52.4, beat)):
if i % 4 in (0, 2) or (i % 8 == 7):
place('perc', thump(95, 45, 0.5, 9, 40), t0, 0.45 if i % 4 == 0 else 0.3, verb=0.12)
place('perc', click(9000, 0.02, 0.4), t0 + beat / 2, 0.06)
if i % 4 == 2:
sn = filt(noise(0.2), sos_bp(900, 4000)) * env_exp(0.2, 22) + sine(190, 0.2) * env_exp(0.2, 30) * 0.5
place('perc', sn, t0, 0.13, verb=0.25)
def feed_sig(key, t):
"""Each feed gets its own sonic signature as it comes online."""
place('ui', blip(1760, 0.06, 0.2, 40) + blip(2349, 0.06, 0.1, 40) * 0.6, t, 0.15, -0.5, verb=0.15)
place('ui', click(4500, 0.01), t, 0.25, -0.5)
if key == 'air':
for j in range(500):
x = rng.random() ** 0.7 * 1.9
place('ui', click(5000 + rng.random() * 4000, 0.004), t + x, 0.05 + rng.random() * 0.05, rng.uniform(-0.9, 0.9))
place('sfx', whoosh(1.4, 500, 7000, 0.35), t, 0.18, 0.3)
elif key == 'sea':
ping = sine(1150 * np.exp(-tt(2.2) * 0.05), 2.2) * env_exp(2.2, 2.2, 0.003)
place('sfx', ping, t + 0.05, 0.22, 0.2, verb=0.8)
place('sfx', ping, t + 1.05, 0.1, -0.3, verb=0.8)
place('sfx', filt(noise(2.0), sos_lp(400)) * env_ar(2.0, 0.5, 1.2), t, 0.2)
elif key == 'mil':
for j in range(3):
tone = (np.sign(np.sin(2 * np.pi * 740 * tt(0.12))) * 0.3 + sine(740, 0.12)) * env_ar(0.12, 0.005, 0.03)
place('ui', filt(tone, sos_lp(3000)), t + 0.1 + j * 0.22, 0.12, 0.3)
tone2 = (np.sign(np.sin(2 * np.pi * 554 * tt(0.12))) * 0.3 + sine(554, 0.12)) * env_ar(0.12, 0.005, 0.03)
place('ui', filt(tone2, sos_lp(3000)), t + 0.21 + j * 0.22, 0.1, 0.3)
elif key == 'orbit':
for j in range(26):
f = scale[rng.integers(0, len(scale))] * rng.choice([0.5, 1, 1])
place('music', fm_bell(f, 2.5, 3.5, 2.5, 2.2), t + rng.random() * 2.2, 0.035, rng.uniform(-0.9, 0.9), verb=0.6)
place('sfx', whoosh(2.8, 2500, 200, 0.4), t + 0.2, 0.3, 0)
place('hits', thump(60, 30, 1.6, 2.5), t + 0.1, 0.4, verb=0.3)
elif key == 'quake':
for j, dt in enumerate((0.05, 0.42, 0.95)):
place('hits', thump(55, 26, 1.4, 3.2, 18), t + dt, 0.6 - j * 0.12, rng.uniform(-0.5, 0.5), verb=0.25)
place('sfx', filt(noise(2.5), sos_lp(90)) * env_ar(2.5, 0.2, 1.8), t, 0.9)
elif key == 'fire':
d = 2.2
for j in range(900):
x = rng.random() ** 1.3 * d
g = (1 - x / d) * 0.12
place('sfx', click(800 + rng.random() * 5000, 0.004 + rng.random() * 0.01, 0.9), t + x, g * rng.random(), rng.uniform(-0.8, 0.8))
place('sfx', filt(noise(d), sos_bp(200, 1500)) * env_ar(d, 0.05, 1.6), t, 0.08)
elif key == 'wind':
d = 3.0
w = sweep_filter(noise(d), lambda tc: 250 + 900 * np.sin(np.pi * tc / d), 'band', order=2)
w2 = sweep_filter(noise(d), lambda tc: max(180, 300 + 700 * np.sin(np.pi * tc / d + 0.6)), 'band', order=2)
place('sfx', np.vstack([w, w2]) * env_ar(d, 0.6, 1.6), t, 0.4)
elif key == 'cable':
d = 1.8
buzz = filt(saw(60, d) + 0.5 * saw(120.5, d), sos_bp(300, 2500)) * env_ar(d, 0.02, 1.0)
gate = (np.sin(2 * np.pi * 7 * tt(d)) > 0.2).astype(float)
place('sfx', buzz * (0.4 + 0.6 * gate), t, 0.08, 0.2)
place('sfx', sine(3000 * np.exp(-tt(0.25) * 12) + 150, 0.25) * env_exp(0.25, 12), t, 0.1, -0.2)
elif key == 'cctv':
for j in range(9):
tc = t + 0.25 + j * 0.12
place('sfx', click(2500, 0.01), tc, 0.3, rng.uniform(-0.7, 0.7))
place('sfx', static(0.12) * env_exp(0.12, 25), tc, 0.05, rng.uniform(-0.7, 0.7))
for fd in C['feeds']:
feed_sig(fd['key'], fd['t'])
# camera moves
place('sfx', whoosh(2.3, 200, 2500, 0.5), 33.6, 0.22, -0.4)
place('sfx', whoosh(2.8, 2200, 180, 0.5), 37.2, 0.22, 0.4)
place('sfx', whoosh(2.0, 300, 3000, 0.5), 40.4, 0.15, -0.2)
# THE WALL
wall_t = T['cctvWall']
place('hits', braam(3.5, ('D1', 'A1', 'D2'), 1900, 0.9, 1.6), wall_t, 0.85, verb=0.3)
for j in range(100):
tc = wall_t + (rng.random() ** 0.8) * 0.85 + rng.random() * 0.12
place('sfx', click(2000 + rng.random() * 2500, 0.012), tc, 0.18, rng.uniform(-0.95, 0.95))
place('sfx', static(0.08) * env_exp(0.08, 30), tc, 0.04, rng.uniform(-0.95, 0.95))
wd = T['cctvOut'] - wall_t + 0.3
wbed = stereo_noise_bed(wd, 800, 7000) * env_ar(wd, 0.4, 0.25)
place('sfx', wbed, wall_t, 0.05)
murmur = (sine(15625 / 2, wd) * 0.02 + filt(noise(wd), sos_bp(100, 300)) * 0.4) * env_ar(wd, 0.3, 0.2)
place('sfx', murmur, wall_t, 0.12)
# the watcher hopping cell to cell (7 Hz)
for j in range(int((T['cctvOut'] - wall_t - 0.8) * 7)):
tc = wall_t + 0.8 + j / 7
place('ui', blip(3136, 0.025, 0.0, 90), tc, 0.06, rng.uniform(-0.8, 0.8))
# rapid list
for j, tc in enumerate(C['rapid']):
place('ui', blip(note('D6') * 2 ** (j / 12 * 2), 0.06, 0.2, 45), tc, 0.12, -0.5, verb=0.15)
# collapse
place('sfx', whoosh(0.6, 5000, 150, 0.7), T['cctvOut'] - 0.1, 0.4)
place('hits', thump(80, 30, 1.6, 2.8), T['cctvOut'] + 0.4, 0.7, verb=0.35)
# =================================================================== ACT 4 saturation
sat0, cut = T['saturate'], T['cut']
# pulse intensifies: 16ths, then doubled at the octave
ostinato(52.5, cut, lambda t: 2400 + 3000 * (t - 52.5) / 8, lambda t: 0.12 + 0.06 * (t - 52.5) / 8)
ostinato(55.5, cut, lambda t: 3000, lambda t: 0.05 + 0.06 * (t - 55.5) / 5, transpose=12)
for i, t0 in enumerate(np.arange(52.5, cut, beat)):
k = (t0 - 52.5) / 8
place('perc', thump(100, 42, 0.55, 8, 40), t0, 0.5 + 0.25 * k, verb=0.15)
if t0 > 55.0:
place('perc', thump(100, 42, 0.4, 10, 40), t0 + beat / 2, 0.3 + 0.2 * k)
if i % 2 == 1:
sn = filt(noise(0.25), sos_bp(900, 5000)) * env_exp(0.25, 16) + sine(190, 0.25) * env_exp(0.25, 26) * 0.5
place('perc', sn, t0, 0.18 + 0.12 * k, verb=0.3)
# 32nd-note snare roll in the last bar
if t0 > 58.4:
for q in range(4):
sn = filt(noise(0.08), sos_bp(1200, 6000)) * env_exp(0.08, 40)
place('perc', sn, t0 + q * beat / 4, 0.1 + 0.05 * q)
for k, t0 in enumerate(np.arange(52.5, cut, 2.0)):
nm, ch = prog[k % 4]
place('music', pad([note(x) for x in ch] + [note(x[:-1] + str(int(x[-1]) + 1)) for x in ch], 2.2, 1600 + k * 300, 0.3, 0.6), t0, 0.14, verb=0.3)
# Shepard riser across the whole act
sd = cut - sat0
place('music', shepard(sd, 0.45, 40, 9, 0.12) * env_ar(sd, 2.5, 0.01, 1.3), sat0, 0.22, verb=0.2)
place('sfx', riser(sd, 150, 9000), sat0, 0.25)
# status lines type in
for st in C['status']:
for j in range(14):
place('ui', click(4000 + rng.random() * 2000, 0.008), st + j * 0.028, 0.12, 0.4)
place('ui', blip(2093, 0.05, 0.2, 50), st + 0.42, 0.1, 0.4)
# lattice building shimmer
for j in range(400):
tc = T['twin'] + rng.random() ** 0.7 * (T['dishBreak'] - T['twin'])
place('ui', blip(4000 + rng.random() * 5000, 0.015, 0, 150), tc, 0.025, rng.uniform(-1, 1), verb=0.3)
# containment alarm
for j, tc in enumerate(np.arange(56.6, T['dishBreak'], 0.36)):
k = (tc - 56.6) / (T['dishBreak'] - 56.6)
tone = filt((np.sign(np.sin(2 * np.pi * 880 * tt(0.16))) * 0.4 + sine(880, 0.16)), sos_lp(2500)) * env_ar(0.16, 0.01, 0.05)
place('ui', tone, tc, 0.04 + 0.08 * k, 0.0, verb=0.2)
place('ui', filt((np.sign(np.sin(2 * np.pi * 660 * tt(0.16))) * 0.4 + sine(660, 0.16)), sos_lp(2500)) * env_ar(0.16, 0.01, 0.05), tc + 0.18, 0.04 + 0.08 * k, verb=0.2)
# the breach: impact + shattering ring
db = T['dishBreak']
place('hits', braam(3.0, ('D1', 'A1', 'D2', 'A2'), 3200, 1.0, 1.2), db, 1.0, verb=0.4)
place('hits', thump(120, 28, 2.5, 1.8, 14), db, 0.9)
for j in range(220):
tc = db + (rng.random() ** 2.2) * 1.1
f = 2500 + rng.random() * 7000
gr = filt(noise(0.25), sos_bp(f * 0.97, f * 1.03)) * env_exp(0.25, 10 + rng.random() * 25)
place('sfx', gr, tc, 0.18 * (1 - (tc - db) / 1.2), rng.uniform(-1, 1), verb=0.3)
# OMNIPRESENT
om = T['omni']
place('hits', braam(2.5, ('D1', 'D2', 'A2', 'D3'), 3800, 1.0, 0.8), om, 0.9, verb=0.45)
place('ui', fm_bell(note('A6'), 1.5, 2.0, 1.2, 2.0), om, 0.08, verb=0.5)
# =================================================================== ACT 5 title
ti = T['title']
# the hush after the cut, then a warm low bloom
d = DUR - ti
warm = (sine(note('D1'), d) * 0.6 + sine(note('A1') * 1.002, d) * 0.3 + sine(note('D2') * 0.998, d) * 0.2)
warm *= np.interp(tt(d), [0, 0.8, 2.0, d - 3.5, d], [0, 0.4, 0.7, 0.6, 0])
place('drone', warm, ti, 0.35)
air = stereo_noise_bed(d, 2000, 12000) * np.interp(tt(d), [0, 1.5, d - 3, d], [0, 1, 1, 0])
place('drone', air, ti, 0.012)
# eye opens
eo = T['eyeOpen']
place('sfx', riser(0.9, 400, 7000), eo - 0.9, 0.18)
place('hits', thump(70, 34, 2.5, 2.2), eo, 0.55, verb=0.4)
for f, g in ((note('D6'), 0.06), (note('A6'), 0.05), (note('E7'), 0.035), (note('D5'), 0.05)):
place('music', fm_bell(f, 4.5, 2.0, 0.8, 0.8), eo, g, rng.uniform(-0.5, 0.5), verb=0.8)
# wordmark: decode chatter then the warm hit
wm = T['wordmark']
for j in range(40):
place('ui', click(3000 + rng.random() * 4000, 0.007), wm + j * 0.022, 0.08, rng.uniform(-0.4, 0.4))
place('hits', braam(6.0, ('D1', 'A1', 'D2', 'E2'), 1400, 1.0, 0.5), wm + 0.85, 0.7, verb=0.5)
place('music', pad([note(x) for x in ('D3', 'A3', 'E4', 'F#4', 'A4')], 7.5, 1400, 1.5, 3.5), wm + 0.85, 0.16, verb=0.5)
# tagline keystrokes
tg_t = T['tagline']
for j, ch in enumerate('NO PLACE LEFT BEHIND.'):
if ch != ' ':
place('ui', click(3800 + rng.random() * 1500, 0.009), tg_t + j / 22, 0.14)
# ONLINE
on = T['online']
place('ui', blip(note('A5'), 0.12, 0.3, 18), on, 0.16, verb=0.5)
place('ui', blip(note('D6'), 0.35, 0.3, 8), on + 0.12, 0.16, verb=0.6)
place('hits', thump(60, 30, 3.5, 1.4, 10), on, 0.6, verb=0.5)
place('music', pad([note(x) for x in ('D2', 'A2', 'D3', 'E3', 'A3')], DUR - on, 900, 0.8, 3.5), on, 0.12, verb=0.4)
# =================================================================== reverb
def make_ir(rt60=3.4, pre=0.025):
n = int((rt60 + pre) * SR)
t0 = np.arange(n) / SR
ir = np.zeros((2, n))
for c in range(2):
x = rng.standard_normal(n) * np.exp(-6.9 * t0 / rt60)
# darker tail: blend toward a lowpassed copy over time
lp = filt(x, sos_lp(2500))
mix = np.clip(t0 / 1.5, 0, 1)
ir[c] = x * (1 - mix) + lp * mix
ir[c, : int(pre * SR)] = 0
return ir / np.sqrt(np.sum(ir ** 2) / 2)
ir = make_ir()
wet = np.vstack([signal.fftconvolve(B['verb'][c], ir[c])[:N] for c in range(2)])
# =================================================================== mix + master
# pre-hit dips: the bed drops out for a beat so each impact lands in a hole
dip = np.ones(N)
for ht, depth in ((T['substrate'], 0.3), (T['indexed'], 0.25), (T['cctvWall'], 0.4), (T['cctvOut'] + 0.4, 0.4), (T['dishBreak'], 0.2), (T['omni'], 0.45), (T['wordmark'] + 0.85, 0.4)):
a, b, c = int((ht - 0.28) * SR), int((ht - 0.03) * SR), int((ht + 0.5) * SR)
dip[a:b] = np.minimum(dip[a:b], np.linspace(1, depth, b - a))
dip[b:c] = np.minimum(dip[b:c], np.linspace(depth, 1, c - b) ** 0.7)
bed_mix = (B['drone'] * 1.0 + B['music'] * 0.9 + B['perc'] * 0.8 + B['sfx'] * 0.9 + B['ui'] * 0.85) * dip
mix = bed_mix + B['hits'] * 1.25 + wet * 0.32
mix = filt(mix, sos_hp(24, 2))
# glitch stutter into the cut, then hard silence until the title
def stutter(x, a, b, slice_s=0.045):
ia, ib = int(a * SR), int(b * SR)
sl = int(slice_s * SR)
src = x[:, ia:ia + sl].copy()
for i in range(ia, ib, sl):
n = min(sl, ib - i)
x[:, i:i + n] = src[:, :n] * (0.9 if ((i - ia) // sl) % 2 == 0 else 0.6)
return x
mix = stutter(mix, cut - 0.22, cut, 0.04)
ic = int(cut * SR)
it = int(T['title'] * SR)
fade = int(0.004 * SR)
mix[:, ic - fade:ic] *= np.linspace(1, 0, fade)
mix[:, ic:it] = 0
# collapse glitch: tiny stutter as the console implodes
mix = stutter(mix, T['collapse'] + 0.02, T['collapse'] + 0.14, 0.03)
# glue: slow RMS compressor, then soft clip and a lookahead peak limiter
def rms_env(x, win_s=0.05):
w = int(win_s * SR)
p = np.mean(x ** 2, axis=0)
k = np.ones(w) / w
return np.sqrt(np.convolve(p, k, mode='same') + 1e-12)
env = rms_env(mix, 0.08)
thr = 10 ** (-15 / 20)
ratio = 1.8
gain = np.where(env > thr, (thr * (env / thr) ** (1 / ratio)) / env, 1.0)
gain = signal.sosfiltfilt(sos_lp(8), gain)
mix *= gain
mix = np.tanh(mix * 1.6) / 1.6
look = int(0.004 * SR)
peak = np.max(np.abs(mix), axis=0)
peak = np.lib.stride_tricks.sliding_window_view(np.pad(peak, (0, look)), look + 1).max(axis=1)[:N]
ceiling = 10 ** (-1.2 / 20)
lg = np.minimum(1.0, ceiling / np.maximum(peak, 1e-9))
lg = np.minimum.reduce([lg, np.roll(lg, 1), np.roll(lg, -1)])
lg = signal.sosfiltfilt(sos_lp(60), lg)
mix *= np.minimum(lg, 1.0)
mix *= ceiling / max(1e-9, np.max(np.abs(mix)))
out = os.path.join(HERE, 'score.wav')
wavfile.write(out, SR, (mix.T * (2 ** 31 - 1)).astype(np.int32))
print('wrote', out, f'{DUR}s', 'peak', np.max(np.abs(mix)))
+527
View File
@@ -0,0 +1,527 @@
#!/usr/bin/env python3
"""God's Eye View — First Light (v2): sound design, built to sit under a music track.
No melody, no beeps. Sub, air, metal, impacts, swells, data texture — a tech
thriller palette that leaves the mids for the score. Everything is synthesised
and placed against audio/cues.json (exported from src/timeline.js).
python3 audio/sfx.py -> audio/sfx.wav (48 kHz, 24-bit stereo)
"""
import json
import os
import numpy as np
from scipy import signal
from scipy.io import wavfile
SR = 48000
HERE = os.path.dirname(os.path.abspath(__file__))
C = json.load(open(os.path.join(HERE, 'cues.json')))
T = C['T']
DUR = C['DURATION']
N = int(SR * DUR)
rng = np.random.default_rng(20261019)
B = {k: np.zeros((2, N)) for k in ('bed', 'hits', 'fx', 'ui', 'verb')}
def place(dst, sig, t, gain=1.0, pan=0.0, verb=0.0):
if sig.ndim == 1:
p = (pan + 1) * np.pi / 4
sig = np.vstack([sig * np.cos(p), sig * np.sin(p)])
i0 = int(round(t * SR))
if i0 >= N:
return
s0 = max(0, -i0)
i0 = max(0, i0)
n = min(sig.shape[1] - s0, N - i0)
if n <= 0:
return
seg = sig[:, s0:s0 + n] * gain
B[dst][:, i0:i0 + n] += seg
if verb > 0:
B['verb'][:, i0:i0 + n] += seg * verb
def tt(d):
return np.arange(int(d * SR)) / SR
def lp(fc, o=2):
return signal.butter(o, min(fc, SR * 0.45), 'low', fs=SR, output='sos')
def hp(fc, o=2):
return signal.butter(o, max(fc, 10), 'high', fs=SR, output='sos')
def bp(lo, hi, o=2):
lo = max(lo, 10)
hi = min(max(hi, lo * 1.05), SR * 0.45)
return signal.butter(o, [lo, hi], 'band', fs=SR, output='sos')
def F(x, sos):
return signal.sosfilt(sos, x, axis=-1)
def sweep(x, fn, kind='low', block=256):
y = np.zeros_like(x)
zi = None
for b in range(0, len(x), block):
fc = float(fn(b / SR))
sos = lp(fc) if kind == 'low' else hp(fc) if kind == 'high' else bp(fc * 0.7, fc * 1.35)
if zi is None:
zi = signal.sosfilt_zi(sos) * 0
y[b:b + block], zi = signal.sosfilt(sos, x[b:b + block], zi=zi)
return y
def noise(d):
return rng.standard_normal(int(d * SR))
def env_exp(d, k, a=0.002):
t = tt(d)
return np.minimum(1, t / max(a, 1e-4)) * np.exp(-t * k)
def env_ar(d, a, r, c=2.0):
t = tt(d)
return np.minimum(np.clip(t / max(a, 1e-4), 0, 1) ** c, np.clip((d - t) / max(r, 1e-4), 0, 1) ** c)
def sine(f, d, ph=0.0):
t = tt(d)
if np.isscalar(f):
return np.sin(2 * np.pi * f * t + ph)
return np.sin(2 * np.pi * np.cumsum(f) / SR + ph)
def saw(f, d):
t = tt(d)
ph = f * t if np.isscalar(f) else np.cumsum(f) / SR
return 2 * (ph - np.floor(ph + 0.5))
# ---------------------------------------------------------------- designed sounds
def tick(d=0.006, f=5000, q=0.6):
"""A dry, expensive-sounding UI tick: filtered noise, no pitch."""
return F(noise(d) * env_exp(d, 1200, 0.0002), bp(f * (1 - q / 2), f * (1 + q)))
def granule(f=6000):
d = 0.012 + rng.random() * 0.02
return F(noise(d) * env_ar(d, 0.002, d * 0.8), bp(f * 0.8, f * 1.25))
def sub_boom(d=3.0, f0=55, f1=24, k=1.4):
t = tt(d)
f = f1 + (f0 - f1) * np.exp(-t * 7)
return sine(f, d) * env_exp(d, k, 0.003)
def metal(d=3.0, base=180.0, k=1.2):
"""Inharmonic struck-metal resonance — the cinematic 'tail'."""
t = tt(d)
out = np.zeros(len(t))
for r, a in ((1.0, 1.0), (2.76, 0.6), (5.4, 0.45), (8.93, 0.3), (13.3, 0.2), (1.51, 0.5)):
out += a * np.sin(2 * np.pi * base * r * (1 + 0.002 * rng.standard_normal()) * t + rng.random() * 6) * np.exp(-t * k * (0.6 + r * 0.25))
return out / 3.0
def crack(d=0.4):
return F(noise(d) * env_exp(d, 18, 0.0005), bp(400, 7000))
def impact(size=1.0, dark=0.0, tail=3.5, base=150):
"""Trailer hit: sub drop + body thud + noise crack + metallic tail."""
d = tail + 0.5
s = sub_boom(d, 62, 22, 1.1) * 1.0
body = F(noise(0.5) * env_exp(0.5, 9, 0.001), lp(420)) * 0.9
out = s
out[: len(body)] += body
c = crack(0.45) * (0.5 - dark * 0.35)
out[: len(c)] += c
m = metal(d, base * (1 - dark * 0.3), 1.0) * 0.28
out += F(m, lp(4000 - dark * 2500))
return np.tanh(out * 1.4 * size) * 0.8
def braam(d=4.0, root=36.71, bright=1400, gain=1.0):
"""Low, distorted brass-like swell — filtered and dirty, not synthy."""
acc = np.zeros(int(d * SR))
for mult in (1, 1.5, 2, 3):
for det in (-0.01, 0, 0.012):
acc += saw(root * mult * (1 + det), d)
acc /= 10
y = sweep(acc, lambda tc: 120 + bright * min(1, tc / 0.09) * np.exp(-max(0, tc - 0.09) * 1.3))
y = np.tanh(y * 3.0) * env_ar(d, 0.015, d * 0.85, 1.2)
return y * gain
def whoosh(d=1.0, f0=300, f1=4000, peak=0.55, width=0.06):
x = noise(d)
y = sweep(x, lambda tc: f0 * (f1 / f0) ** min(1, tc / d), 'band')
t0 = tt(d)
return y * np.exp(-((t0 / d - peak) ** 2) / width)
def swell(d=1.5, f0=200, f1=6000):
"""Reverse-swell into a hit."""
x = noise(d)
y = sweep(x, lambda tc: f0 * (f1 / f0) ** ((tc / d) ** 1.8), 'band')
t0 = tt(d)
m = metal(d, 220, 0.2)[::-1] * 0.25
return (y + m) * (t0 / d) ** 2.6
def servo(d=0.6, f0=180, f1=420):
t0 = tt(d)
f = f0 + (f1 - f0) * np.sin(np.pi * t0 / d / 2)
x = saw(f, d) + 0.4 * saw(f * 2.01, d)
return F(x, bp(200, 2200)) * env_ar(d, 0.05, 0.12) * 0.5
def glitch(d=0.25):
"""Digital stutter: gated, bit-crushed noise and a buzzing carrier."""
x = noise(d)
gate = (np.sin(2 * np.pi * (35 + rng.random() * 40) * tt(d)) > 0).astype(float)
crushed = np.round(x * 4) / 4
buzz = np.sign(np.sin(2 * np.pi * (90 + rng.random() * 80) * tt(d))) * 0.3
return F((crushed * 0.6 + buzz) * gate, bp(300, 9000)) * env_ar(d, 0.002, 0.03)
def drone_bed(d, notes=(36.71, 55.0, 73.42), lpf=180):
t0 = tt(d)
x = sum(np.sin(2 * np.pi * f * (1 + 0.0008 * k) * t0 + k) * (0.6 / (k + 1)) for k, f in enumerate(notes))
x += F(noise(d), lp(lpf)) * 0.25
return x
def tension(d, base=55.0):
"""Dissonant cluster, heavily low-passed: dread, not tune."""
t0 = tt(d)
x = np.zeros(len(t0))
for r in (1.0, 1.059, 1.5, 2.12, 2.0):
x += saw(base * r * (1 + 0.003 * np.sin(2 * np.pi * 0.07 * t0 * r)), d)
x = F(x / 5, lp(380))
return x * (0.7 + 0.3 * np.sin(2 * np.pi * 0.11 * t0))
def rotor(d, rate=5.6):
t0 = tt(d)
thump = (0.5 + 0.5 * np.sin(2 * np.pi * rate * t0)) ** 6
x = F(noise(d), lp(220)) * thump * 1.6 + F(noise(d), bp(900, 3000)) * thump * 0.08
return x
def jet(d):
t0 = tt(d)
whine = sine(2400 + 120 * np.sin(2 * np.pi * 0.2 * t0), d) * 0.04 + sine(3600, d) * 0.015
roar = F(noise(d), bp(120, 900)) * 0.6
return roar + whine
def wind(d):
a = sweep(noise(d), lambda tc: 260 + 500 * (0.5 + 0.5 * np.sin(tc * 0.9)), 'band')
b = sweep(noise(d), lambda tc: 300 + 600 * (0.5 + 0.5 * np.sin(tc * 0.7 + 1.3)), 'band')
return np.vstack([a, b])
def squelch(d=0.35):
x = F(noise(d), bp(800, 4500)) * env_ar(d, 0.005, 0.06)
return x
def heartbeat(t, g=1.0):
for dt, gg, f0 in ((0.0, 1.0, 64), (0.19, 0.55, 54)):
d = 0.7
t0 = tt(d)
f = 30 + (f0 - 30) * np.exp(-t0 * 28)
place('hits', sine(f, d) * env_exp(d, 7.5, 0.002), t + dt, g * gg, verb=0.2)
# =================================================================== the film
t_all = tt(DUR)
# bed: sub + air, riding an intensity curve, silent through the cut
inten = np.interp(t_all, [0, 4, 9, 16, 30, 43, 44, 72, 76.6, T['cut'] - 0.01, T['cut'], T['title'], T['title'] + 3, DUR - 2.5, DUR],
[0, 0.25, 0.4, 0.55, 0.65, 0.7, 0.55, 0.65, 0.75, 1.0, 0, 0, 0.6, 0.5, 0])
bed = drone_bed(DUR) * inten
B['bed'] += np.vstack([bed, bed]) * 0.32
airbed = np.vstack([F(noise(DUR), bp(2500, 11000)), F(noise(DUR), bp(2500, 11000))]) * 0.012 * inten
B['bed'] += airbed
# tension cluster under the indexing and the cities
tz = tension(DUR) * np.interp(t_all, [16, 20, 30, 31, 43, 44, 70, 76, 84.4, 84.41], [0, 0.5, 0.6, 0.35, 0.45, 0.6, 0.7, 0.8, 1.0, 0], left=0, right=0)
B['bed'] += np.vstack([tz, tz * 0.95]) * 0.22
# ACT 0 — console
hum = (sine(100, DUR) * 0.25 + F(noise(DUR), bp(80, 300)) * 0.05) * np.interp(t_all, [0.4, 1.2, T['collapse'], T['seed']], [0, 1, 1, 0], left=0, right=0)
place('fx', hum, 0, 0.12)
for k in C['boot']:
if k['ch'] != ' ':
place('ui', tick(0.006, 4200 + rng.random() * 2500), k['t'], 0.16, pan=(rng.random() - 0.5) * 0.3)
place('fx', swell(T['seed'] - T['collapse'] + 0.05, 300, 9000), T['collapse'], 0.35)
place('fx', glitch(0.18), T['collapse'], 0.25)
place('hits', impact(0.7, dark=0.6, tail=3.0, base=110), T['seed'], 0.8, verb=0.35)
for bt in (T['beat1'], T['beat2']):
heartbeat(bt, 1.1)
# aperture: servos as the blades retract
place('fx', servo(1.2, 140, 360), T['aperture'], 0.22, -0.3, verb=0.2)
place('fx', servo(1.0, 160, 300), T['aperture'] + 0.25, 0.18, 0.3, verb=0.2)
for i in range(120):
place('ui', tick(0.004, 7000 if i % 10 else 4200), T['dishDraw'] + (i / 120) * (T['dishDone'] - T['dishDraw']), 0.05 if i % 10 else 0.1, pan=np.sin(i / 120 * 6.28) * 0.8)
# ACT 1 — growth: organic granular crackle densifying
gs, ge = T['growStart'], T['growEnd']
for _ in range(2400):
x = rng.random() ** 0.45
tc = gs + x * (ge - gs + 0.3)
k = (np.exp(3 * (tc - gs) / (ge - gs)) - 1) / (np.exp(3) - 1)
if rng.random() > 0.15 + 0.85 * k:
continue
place('fx', granule(2000 + rng.random() * 7000), tc, 0.05 + 0.08 * rng.random() * (0.4 + k), pan=rng.uniform(-0.9, 0.9))
# radar sweep: a slow low whoosh per revolution
sweep_period = 2 * np.pi / 2.4
for i in range(int((T['substrate'] - gs) / sweep_period) + 1):
place('fx', whoosh(sweep_period, 120, 900, 0.5, 0.12), gs + i * sweep_period, 0.08, pan=0.0)
place('fx', swell(1.6, 120, 5000), T['substrate'] - 1.6, 0.3)
place('hits', impact(0.9, 0.2, 4.0, 150), T['substrate'], 0.9, verb=0.35)
place('hits', braam(3.5, 36.71, 1100), T['substrate'], 0.35, verb=0.3)
# ACT 2 — indexing: tiles as data grains, captures as shutter clicks
for tl in C['tiles']:
if not tl['vis'] or tl['t'] > 23.5:
continue
if tl['z'] == 5 and rng.random() > 0.3:
continue
place('ui', granule(3000 + rng.random() * 5000), tl['t'], {3: 0.07, 4: 0.04, 5: 0.025}[tl['z']], pan=tl['pan'] * 0.8)
def shutter(t, g=1.0, pan=0.0):
place('fx', tick(0.008, 3800, 0.6), t, 0.45 * g, pan)
place('fx', tick(0.014, 2100, 0.6), t + 0.045, 0.35 * g, pan)
place('fx', F(noise(0.05) * env_exp(0.05, 60), lp(500)), t, 0.25 * g, pan)
for cp in C['caps']:
if cp['slow']:
shutter(cp['t'], 1.0, -0.3)
place('fx', servo(0.9, 220, 900), cp['t'] + 0.3, 0.12, -0.3, verb=0.2)
place('fx', whoosh(0.45, 900, 5000, 0.5), cp['t'] + 1.15, 0.16, 0.4)
else:
shutter(cp['t'], 0.5, 0.5)
# the flurry: shutters dissolving into a roar of capture
fl0, fl1 = T['flurry'], T['flurryEnd']
for i in range(420):
u = rng.random() ** 0.6
tc = fl0 + u * (fl1 - fl0)
shutter(tc, 0.18 + 0.25 * u, rng.uniform(-0.9, 0.9))
sw_d = fl1 - fl0
spin = whoosh(sw_d, 150, 3500, 0.62, 0.05)
place('fx', np.vstack([spin, np.roll(spin, 900)]), fl0, 0.35)
place('fx', swell(1.2, 200, 8000), T['indexed'] - 1.2, 0.35)
place('hits', impact(1.1, 0.0, 4.5, 170), T['indexed'], 1.0, verb=0.4)
place('hits', braam(4.0, 36.71, 1800), T['indexed'], 0.4, verb=0.3)
# ACT 3 — feeds: each a designed arrival, not a beep
def feed(key, t):
place('ui', tick(0.01, 3000), t, 0.25, -0.5)
place('fx', whoosh(0.5, 2500, 400, 0.3), t - 0.05, 0.12, -0.4)
if key == 'air':
for _ in range(380):
place('ui', tick(0.003, 6000 + rng.random() * 3000), t + rng.random() ** 0.7 * 1.6, 0.04 + rng.random() * 0.04, rng.uniform(-0.9, 0.9))
place('fx', jet(1.6) * env_ar(1.6, 0.3, 1.0), t, 0.25, 0.3)
elif key == 'sea':
p = sine(820 * np.exp(-tt(2.5) * 0.06), 2.5) * env_exp(2.5, 2.0)
place('fx', F(p, lp(2000)), t + 0.05, 0.12, 0.2, verb=0.9)
place('fx', F(noise(2.0), lp(300)) * env_ar(2.0, 0.6, 1.2), t, 0.25)
elif key == 'mil':
place('fx', squelch(0.3), t + 0.05, 0.25, 0.4)
place('fx', F(saw(48, 1.2), bp(60, 600)) * env_ar(1.2, 0.1, 0.8), t, 0.18)
elif key == 'orbit':
place('fx', whoosh(2.6, 2600, 160, 0.4, 0.1), t + 0.2, 0.3)
place('hits', sub_boom(2.0, 48, 24, 2.0), t + 0.1, 0.5, verb=0.3)
for _ in range(40):
place('ui', granule(7000 + rng.random() * 4000), t + rng.random() * 2.0, 0.03, rng.uniform(-1, 1), verb=0.6)
elif key == 'quake':
for j, dt in enumerate((0.05, 0.45, 0.95)):
place('hits', sub_boom(1.4, 50, 22, 3.0), t + dt, 0.6 - j * 0.12, verb=0.25)
place('fx', F(noise(2.4), lp(90)) * env_ar(2.4, 0.2, 1.6), t, 1.0)
elif key == 'fire':
for _ in range(700):
x = rng.random() ** 1.3 * 2.0
place('fx', granule(700 + rng.random() * 4000), t + x, (1 - x / 2) * 0.1 * rng.random(), rng.uniform(-0.8, 0.8))
place('fx', F(noise(2.0), bp(150, 1200)) * env_ar(2.0, 0.05, 1.5), t, 0.08)
elif key == 'wind':
place('fx', wind(2.6) * env_ar(2.6, 0.5, 1.5), t, 0.45)
elif key == 'cable':
buzz = F(saw(60, 1.5) + 0.5 * saw(120.4, 1.5), bp(300, 2500)) * env_ar(1.5, 0.02, 0.9)
place('fx', buzz, t, 0.05, 0.2)
place('fx', glitch(0.2), t, 0.12, -0.2)
for fd in C['feeds']:
feed(fd['key'], fd['t'])
for t0, d in ((31.6, 1.8), (33.6, 2.2), (35.0, 2.2)):
place('fx', whoosh(d, 200, 2400, 0.5), t0, 0.16, rng.uniform(-0.5, 0.5))
# ACT 4 — descent and the cities
place('fx', swell(1.4, 120, 9000), T['terrain'] - 1.4, 0.45)
place('fx', whoosh(1.2, 4000, 120, 0.7, 0.08), T['terrain'] - 1.0, 0.4)
place('hits', impact(1.0, 0.4, 3.5, 120), T['terrain'], 0.85, verb=0.4)
for c in C['cities']:
t0, t1 = c['t0'], c['t1']
d = t1 - t0
if c['slug'] != 'everest':
place('fx', glitch(0.22), t0 - 0.05, 0.35)
place('hits', impact(0.6, 0.5, 2.5, 140), t0, 0.5, verb=0.3)
place('fx', whoosh(0.6, 3000, 300, 0.25), t0 - 0.1, 0.25)
amb = F(noise(d), bp(60, 400)) * env_ar(d, 0.4, 0.4) * 0.5
place('fx', np.vstack([amb, np.roll(amb, 2000)]), t0, 0.25)
if c['slug'] == 'everest':
place('fx', wind(d) * env_ar(d, 0.6, 0.6), t0, 0.7)
# LiDAR sweep: a soft rising tone per pass
for i in range(int(d / (1 / 0.35 * 1.2 / 1.2)) + 1):
place('fx', F(sine(np.linspace(300, 1200, int(1.2 * SR)), 1.2), bp(200, 1600)) * env_ar(1.2, 0.3, 0.6), t0 + i * 2.86, 0.03, verb=0.4)
elif c['slug'] == 'nyc':
traffic = F(noise(d), bp(80, 700)) * env_ar(d, 0.8, 0.5)
place('fx', np.vstack([traffic, np.roll(traffic, 5000)]), t0, 0.22)
place('fx', rotor(d) * env_ar(d, 1.0, 1.0), t0 + 0.2, 0.3, 0.4)
for j in range(8):
place('fx', tick(0.01, 2400), t0 + 2.2 + j * 0.35, 0.18, 0.7)
place('fx', F(noise(0.1), bp(1500, 7000)) * env_exp(0.1, 25), t0 + 2.2 + j * 0.35, 0.05, 0.7)
elif c['slug'] == 'boston':
for j in range(4):
place('fx', F(noise(1.4), bp(100, 900)) * env_ar(1.4, 0.4, 0.8) * (1 + 0.3 * np.sin(2 * np.pi * 9 * tt(1.4))), t0 + 0.6 + j * 1.0, 0.1, rng.uniform(-0.7, 0.7))
elif c['slug'] == 'lax':
place('fx', jet(d) * env_ar(d, 0.8, 0.8), t0, 0.35, -0.3)
place('fx', whoosh(2.4, 200, 1400, 0.5, 0.1), t0 + 1.4, 0.3, 0.5)
elif c['slug'] == 'bragg':
place('fx', rotor(d, 4.6) * env_ar(d, 0.8, 0.8), t0 + 0.4, 0.25, -0.4)
for j in range(3):
place('fx', squelch(0.3), t0 + 0.8 + j * 1.3, 0.16, 0.5)
# the camera wall
wt = T['cctvWall']
place('hits', impact(0.9, 0.3, 3.0, 160), wt, 0.8, verb=0.3)
place('hits', braam(3.0, 36.71, 1400), wt, 0.3, verb=0.3)
for _ in range(100):
tc = wt + rng.random() ** 0.8 * 0.85 + rng.random() * 0.12
place('fx', tick(0.012, 2200 + rng.random() * 2000), tc, 0.14, rng.uniform(-0.95, 0.95))
place('fx', F(noise(0.08), bp(1500, 8000)) * env_exp(0.08, 30), tc, 0.03, rng.uniform(-0.95, 0.95))
wd = T['cctvOut'] - wt + 0.3
place('fx', np.vstack([F(noise(wd), bp(800, 6000)), F(noise(wd), bp(800, 6000))]) * env_ar(wd, 0.4, 0.25), wt, 0.035)
for j in range(int((T['cctvOut'] - wt - 0.8) * 7)):
place('ui', tick(0.006, 5200), wt + 0.8 + j / 7, 0.06, rng.uniform(-0.8, 0.8))
place('fx', whoosh(0.6, 5000, 150, 0.7), T['cctvOut'] - 0.1, 0.4)
place('hits', impact(0.9, 0.5, 3.0, 120), T['cctvOut'] + 0.4, 0.7, verb=0.35)
# ACT 5 — saturation: rising pressure, the breach, the state
sat0, cut = T['saturate'], T['cut']
sd = cut - sat0
t0s = tt(sd)
pos = 0.45 * t0s + 0.06 * t0s ** 2
sh = np.zeros(len(t0s))
for k in range(9):
o = (k + pos) % 9
a = np.exp(-((o - 4.5) ** 2) / (2 * 1.8 ** 2))
sh += np.sin(2 * np.pi * np.cumsum(30 * 2 ** o) / SR) * a
sh = F(sh / 9 * 2, lp(2200)) * env_ar(sd, 2.0, 0.01, 1.3)
place('fx', sh, sat0, 0.3, verb=0.2)
place('fx', swell(sd, 120, 9000), sat0, 0.3)
# pulse of low hits accelerating toward the breach
tk = sat0 + 0.5
gap = 0.9
while tk < T['dishBreak'] - 0.2:
place('hits', sub_boom(0.9, 58, 30, 4.0), tk, 0.35 + 0.35 * (tk - sat0) / sd)
place('fx', F(noise(0.12), bp(500, 4000)) * env_exp(0.12, 30), tk, 0.08)
tk += gap
gap = max(0.22, gap * 0.86)
for st in C['status']:
for j in range(16):
place('ui', tick(0.005, 4500 + rng.random() * 1500), st + j * 0.026, 0.1, 0.4)
for _ in range(300):
tc = T['twin'] + rng.random() ** 0.7 * (T['dishBreak'] - T['twin'])
place('ui', granule(5000 + rng.random() * 5000), tc, 0.025, rng.uniform(-1, 1), verb=0.3)
db = T['dishBreak']
place('hits', impact(1.25, 0.0, 5.0, 190), db, 1.1, verb=0.45)
place('hits', braam(3.0, 36.71, 2600), db, 0.45, verb=0.4)
for _ in range(200):
tc = db + rng.random() ** 2.2 * 1.1
f = 2500 + rng.random() * 7000
g = F(noise(0.25), bp(f * 0.97, f * 1.03)) * env_exp(0.25, 10 + rng.random() * 25)
place('fx', g, tc, 0.16 * (1 - (tc - db) / 1.2), rng.uniform(-1, 1), verb=0.3)
place('hits', impact(1.0, 0.3, 4.0, 110), T['omni'], 0.8, verb=0.45)
# ACT 6 — title: low bloom, the eye, the word
ti = T['title']
d = DUR - ti
warm = drone_bed(d, (36.71, 55.0, 73.42, 110.0), 140) * np.interp(tt(d), [0, 0.8, 2.0, d - 3.0, d], [0, 0.4, 0.7, 0.6, 0])
place('bed', warm, ti, 0.3)
place('fx', swell(0.9, 400, 7000), T['eyeOpen'] - 0.9, 0.2)
place('hits', impact(0.8, 0.6, 5.0, 90), T['eyeOpen'], 0.6, verb=0.5)
place('fx', metal(6.0, 220, 0.5) * 0.4, T['eyeOpen'], 0.12, verb=0.7)
for j in range(36):
place('ui', tick(0.006, 3500 + rng.random() * 4000), T['wordmark'] + j * 0.024, 0.07, rng.uniform(-0.4, 0.4))
place('hits', impact(1.0, 0.4, 6.5, 100), T['wordmark'] + 0.85, 0.8, verb=0.55)
place('hits', braam(6.0, 36.71, 900), T['wordmark'] + 0.85, 0.35, verb=0.5)
for j in range(11):
place('ui', tick(0.008, 4000), T['tagline'] + j * 0.08, 0.1)
place('fx', metal(6.0, 330, 0.45) * 0.35, T['tagline'] + 0.9, 0.1, verb=0.8)
# =================================================================== reverb, mix, master
def ir(rt=3.6, pre=0.03):
n = int((rt + pre) * SR)
t0 = np.arange(n) / SR
out = np.zeros((2, n))
for c in range(2):
x = rng.standard_normal(n) * np.exp(-6.9 * t0 / rt)
l2 = F(x, lp(2200))
m = np.clip(t0 / 1.2, 0, 1)
out[c] = x * (1 - m) + l2 * m
out[c, : int(pre * SR)] = 0
return out / np.sqrt(np.sum(out ** 2) / 2)
R = ir()
wet = np.vstack([signal.fftconvolve(B['verb'][c], R[c])[:N] for c in range(2)])
dip = np.ones(N)
for ht, depth in ((T['substrate'], 0.3), (T['indexed'], 0.25), (T['terrain'], 0.3), (T['cctvWall'], 0.4), (T['dishBreak'], 0.2), (T['wordmark'] + 0.85, 0.4)):
a, b, c = int((ht - 0.28) * SR), int((ht - 0.03) * SR), int((ht + 0.5) * SR)
dip[a:b] = np.minimum(dip[a:b], np.linspace(1, depth, b - a))
dip[b:c] = np.minimum(dip[b:c], np.linspace(depth, 1, c - b) ** 0.7)
mix = (B['bed'] + B['fx'] * 0.9 + B['ui'] * 0.8) * dip + B['hits'] * 1.2 + wet * 0.3
mix = F(mix, hp(22))
def stutter(x, a, b, sl=0.04):
ia, ib, s = int(a * SR), int(b * SR), int(sl * SR)
src = x[:, ia:ia + s].copy()
for i in range(ia, ib, s):
n = min(s, ib - i)
x[:, i:i + n] = src[:, :n] * (0.9 if ((i - ia) // s) % 2 == 0 else 0.55)
return x
mix = stutter(mix, cut - 0.22, cut)
ic, it = int(cut * SR), int(T['title'] * SR)
f = int(0.004 * SR)
mix[:, ic - f:ic] *= np.linspace(1, 0, f)
mix[:, ic:it] = 0
# gentle glue + ceiling
p = np.sqrt(np.convolve(np.mean(mix ** 2, axis=0), np.ones(4000) / 4000, mode='same') + 1e-12)
thr = 10 ** (-16 / 20)
g = np.where(p > thr, (thr * (p / thr) ** (1 / 1.8)) / p, 1.0)
g = signal.sosfiltfilt(lp(8), g)
mix *= g
mix = np.tanh(mix * 1.5) / 1.5
look = int(0.004 * SR)
pk = np.lib.stride_tricks.sliding_window_view(np.pad(np.max(np.abs(mix), axis=0), (0, look)), look + 1).max(axis=1)[:N]
ceil = 10 ** (-1.2 / 20)
lg = signal.sosfiltfilt(lp(60), np.minimum(1.0, ceil / np.maximum(pk, 1e-9)))
mix *= np.minimum(lg, 1.0)
mix *= ceil / max(1e-9, np.max(np.abs(mix)))
out = os.path.join(HERE, 'sfx.wav')
wavfile.write(out, SR, (mix.T * (2 ** 31 - 1)).astype(np.int32))
print('wrote', out, DUR, 's')
+62
View File
@@ -0,0 +1,62 @@
# Intro-sequence data snapshot: sources
All files are real downloads from public keyless endpoints, fetched on **2026-10-18 between 02:24 and 02:42 UTC**. Nothing here is synthesized. The scripts only filter, convert units and round. Helper scripts are in `../tools/fetch/` (`fetch_basic.sh`, `process_basic.py`, `process_tle.py`, `process_vessels.py`, `fetch_wind.py`, `fetch_bonus.py`, `cctv_fetch.py`, `cctv_select.py`).
JSON files are compact. Tabular layers use `{"cols":[...],"rows":[[...],...]}`, and each file's `cols` lists its columns. Coordinates are `[lon, lat]` in WGS84.
| File | Rows | Source URL(s) | Fetched (UTC) | Notes / licence |
|---|---|---|---|---|
| `flights.json` | 5,550 airborne aircraft | https://opensky-network.org/api/states/all | 02:24:18 (`t`=1792290258) | OpenSky Network state vectors, anonymous tier. Airborne only (`on_ground=false`), non-null positions. `alt_m` is geometric altitude, or barometric when geometric is missing. Licence: OpenSky Network, non-commercial/research terms; credit "The OpenSky Network, https://opensky-network.org". |
| `military.json` | 45 airborne aircraft | https://api.adsb.lol/v2/mil, plus https://opendata.adsb.fi/api/v2/mil (merged and de-duplicated by ICAO hex; adsb.lol wins) | 02:24 | Includes the `type` column (ICAO type code, e.g. C17, K35R). alt ft→m, gs kt→m/s. `country` is null because these feeds do not provide it. The count is low because the snapshot was taken at a quiet hour. A re-poll at 02:42 gave fewer (30 and 31 airborne). adsb.lol data is ODbL 1.0; adsb.fi open data. airplanes.live /v2/mil returned 403 and was not used. |
| `tle.txt` | 15,928 satellites (3-line TLE) | https://celestrak.org/NORAD/elements/gp.php?GROUP=active&FORMAT=3le | 02:26 | CelesTrak "active" group. The FORMAT=tle request kept failing with connection resets after about 11 s through the proxy, while FORMAT=3le answered in 4 s. Element epochs range from day 264 to 293 of 2026. A descriptive User-Agent is required (as in server/providers/space/celestrak.js). Credit: CelesTrak / Dr T.S. Kelso. Data originates from 18th SDS / space-track.org. |
| `tle_groups.json` | stations 17, starlink 10,651, gps-ops 32, geo 567, weather 72 | `gp.php?GROUP={stations,starlink,gps-ops,geo,weather}&FORMAT=tle` | 02:24–02:27 | `groups_norad`: NORAD ids per group, to join against `tle.txt` (columns 3–7 of line 1). `notable`: name→group for the first entries of each group (ISS (ZARYA), CSS (TIANHE), GOES, NOAA, GPS, …). |
| `quakes.json` | 10,120 events | https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson | 02:24 (`t`=generated ms) | USGS ANSS ComCat. Covers 2026-09-18 02:26 to 2026-10-18 02:20 UTC, max M7.7. Sorted by time. Public domain (USGS). |
| `fires.json` | 246,411 hotspots | https://firms.modaps.eosdis.nasa.gov/data/active_fire/suomi-npp-viirs-c2/csv/SUOMI_VIIRS_C2_Global_24h.csv (111,551); https://firms.modaps.eosdis.nasa.gov/data/active_fire/noaa-20-viirs-c2/csv/J1_VIIRS_C2_Global_24h.csv (118,991); https://firms.modaps.eosdis.nasa.gov/data/active_fire/modis-c6.1/csv/MODIS_C6_1_Global_24h.csv (15,869) | 02:24 | Rows are `[lon, lat, frp_MW, brightness_K]`. Brightness is VIIRS `bright_ti4` or MODIS `brightness`. Despite the "24h" names, the acquisitions run from 2026-10-16 00:11 to 2026-10-17 23:56 UTC. SNPP and NOAA-20 overlap, so the same fires appear twice. NASA FIRMS (LANCE); free use with credit "NASA FIRMS". |
| `vessels.json` | 4,385 vessels | https://kystdatahuset.kystverket.no/ws/api/ais/realtime/geojson (Norway, 3,272 with a position); https://meri.digitraffic.fi/api/ais/v1/locations?from=<now-48h> + https://meri.digitraffic.fi/api/ais/v1/vessels for names (Finland/Baltic, 1,113 after de-duplication against Norway) | 02:28–02:29 | **Not global.** Coverage is the Norwegian coast and EEZ including Svalbard (Kystverket Kystdatahuset open API; NLOD licence) plus Finnish waters and the northern Baltic (Fintraffic Digitraffic; CC BY 4.0). Kystverket returns short trails, and the script uses the last vertex as the current position, checked against COG. `sog_kn`/`cog_deg` are null when AIS sends "not available". No keyless global AIS source was found. AISStream (the repo's source) needs a key, BarentsWatch needs OAuth (401), the Danish DMA host has an expired TLS certificate, and Kystverket's raw TCP NMEA feed cannot pass the HTTPS-only proxy. |
| `wind.json` | 360×181 grid (65,160 cells) | https://noaa-gfs-bdp-pds.s3.amazonaws.com/gfs.20261017/18/atmos/gfs.t18z.pgrb2.1p00.f006 (byte ranges from its `.idx`, UGRD/VGRD at "10 m above ground") | 02:34 | NOAA GFS 1.0° run 2026-10-17 18Z, +6 h forecast, valid 2026-10-18 00Z. This is the same AWS Open Data bucket and `.idx` range approach as server/providers/wind/gfs.js, decoded with ecCodes (pip `eccodes`). The grid is re-indexed so `lo1=-180, la1=90, dx=dy=1`, row-major from north to south, m/s. The 00Z run of 10-18 was not yet published. Public domain (NOAA). |
| `cables.json` | 1,948 polylines (732 cables) | https://www.submarinecablemap.com/api/v3/cable/cable-geo.json | 02:24 | Each LineString part is a separate polyline, rounded to 2 decimals. © TeleGeography, Submarine Cable Map (CC BY-SA 4.0); attribute TeleGeography. |
| `coastline.json` | 1,429 lines, 60,392 vertices | https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_50m_coastline.geojson | 02:24 | Natural Earth 1:50m coastline, rounded to 2 decimals with consecutive duplicate points removed. Public domain. |
| `borders.json` | 288 rings | https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_110m_admin_0_countries.geojson | 02:24 | Natural Earth 1:110m countries, each polygon ring as a line. Public domain. |
| `cities.json` | 3,000 | https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_populated_places_simple.geojson | 02:24 | Top 3,000 by `pop_max`. Natural Earth, public domain. |
| `airports.json` (bonus) | 5,280 (1,176 large, 4,104 medium) | https://davidmegginson.github.io/ourairports-data/airports.csv | 02:24 | OurAirports large and medium airports. `iata` falls back to the ICAO ident when there is no IATA code. `size` is L or M. Public domain. |
| `radio.json` (bonus) | 12,983 stations | https://de1.api.radio-browser.info/json/stations/search?has_geo_info=true&hidebroken=true&limit=100000 | 02:40 | Radio Browser community directory (the repo's radio provider). Stations with geo coordinates only. Public domain / open. |
| `bikes.json` (bonus) | 31,176 docks | 18 GBFS `station_information.json` feeds (Citi Bike, Divvy, Bay Wheels, Capital Bikeshare via gbfs.lyft.com; Bluebikes; Toronto, Barcelona, Madrid, São Paulo, Rio, Santiago, Bogotá via publicbikesystem.net; BIXI; Ecobici; Vélib'; Indego and LA Metro via bcycle; Oslo via urbansharing) + https://api.nextbike.net/maps/nextbike-live.json (all nextbike cities, 16,926 stations) | 02:41 | Per-system counts are in `systems`. Operator open-data terms apply (mostly GBFS open licences). |
| `launch_sites.json` (bonus) | 249 pads | https://ll.thespacedevs.com/2.3.0/pads/?limit=100&mode=list (paged) | 02:42 | Launch Library 2 (The Space Devs), sorted by total launches. Free tier. |
| `power_plants.json` (bonus) | 34,936 plants | https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv | 02:42 | WRI Global Power Plant Database v1.3 (static, 2021). CC BY 4.0. |
Not produced: **military installations**. The repo's provider queries OSM Overpass by viewport only and rejects global requests. A global Overpass pull would be slow and against that design. **Datacenters**: the repo has no keyless source.
## CCTV (`../cctv/`)
`cctv/cam_001.jpg` … `cam_117.jpg`: 117 real frames, each fetched 2026-10-18 02:38:46–02:39:30 UTC and cover-cropped and resized to 640×360 (JPEG q88). `cctv/index.json` gives `file, name, city, country, lat, lon, source`, plus `fetched_utc`, `orig_size` and the frame `url`.
About 235 frames were downloaded from 20 sources, and every one was reviewed on contact sheets. Placeholders were rejected: Caltrans "Temporarily Unavailable", Vegvesen "Kamera er midlertidig ute av drift", Tallinn "NB! ÜHENDUS KAAMERAGA PUUDUB", Barcelona "Interrupció temporal de la imatge", and one all-black King County frame. Frames whose average hash repeats within the set were also dropped as placeholders, as were frames under 320 px wide or with luminance mean ≤25 or std ≤15.
Local time at capture: daylight in Sydney, NZ, Hong Kong and Singapore; evening or night in the Americas; pre-dawn night in Europe. The Hong Kong TD frames carry the department's own privacy-mask polygons.
| Source | Frames | Country | Catalog endpoint (frames from) |
|---|---|---|---|
| Live Traffic NSW | 12 | Australia | https://data.livetraffic.com/cameras/traffic-cam.json (webcams.transport.nsw.gov.au) |
| NZTA Waka Kotahi | 12 | New Zealand | https://trafficnz.info/service/traffic/rest/4/cameras/all (trafficnz.info/camera/*.jpg) |
| HK Transport Department | 10 | Hong Kong | https://static.data.gov.hk/td/traffic-snapshot-images/code/Traffic_Camera_Locations_En.xml (tdcctv.data.one.gov.hk) |
| LTA Singapore | 6 | Singapore | https://api.data.gov.sg/v1/transport/traffic-images (the feed listed only 8 cameras) |
| NYC DOT | 9 | USA | https://webcams.nyctmc.org/api/cameras (…/{id}/image) |
| Caltrans CCTV | 6 | USA | https://cwwp2.dot.ca.gov/data/d{4,7,11,3}/cctv/cctvStatusD0X.json |
| WSDOT | 6 | USA | https://data.wsdot.wa.gov/travelcenter/Cameras.json (images.wsdot.wa.gov only) |
| King County DOT | 3 | USA | ArcGIS TRAFFICCAMERA_POINT_2029 FeatureServer (info.kingcounty.gov) |
| TxDOT ITS | 4 | USA | https://its.txdot.gov/its/DistrictIts/GetCctvStatusListByDistrict?districtCode=… (base64 JSON snapshot; Dallas and San Antonio worked, Houston and El Paso returned null) |
| City of Austin | 4 | USA | https://data.austintexas.gov/api/views/b4k4-adkb/rows.json (cctv.austinmobility.io) |
| DriveBC | 6 | Canada | https://www.drivebc.ca/api/webcams/ |
| City of Toronto RESCU | 6 | Canada | https://opendata.toronto.ca/transportation/tmc/rescucameraimages/Data/tmcearthcameras.json |
| City of Calgary | 4 | Canada | https://data.calgary.ca/resource/k7p9-kppz.json |
| TfL JamCams | 8 | UK | https://api.tfl.gov.uk/Place/Type/JamCam (jamcams.tfl.gov.uk S3) |
| Fintraffic Digitraffic | 4 | Finland | https://tie.digitraffic.fi/api/weathercam/v1/stations (weathercam.digitraffic.fi) |
| Statens vegvesen | 4 | Norway | ogckart-sn1.atlas.vegvesen.no datex_3_1:CctvSimple (kamera.atlas.vegvesen.no) |
| City of Tallinn | 3 | Estonia | repo config/cctv_sources.tallinn.json (ristmikud.tallinn.ee) |
| Transpordiamet Tarktee | 3 | Estonia | tarktee.transpordiamet.ee DATEX2 roadCameraLocations + roadCameraImages |
| Via Lietuva eismoinfo | 4 | Lithuania | https://eismoinfo.lt/eismoinfo-backend/camera-info-table + layer-static-features/VKR |
| Servei Català de Trànsit catalog | 3 | Spain | https://www.gencat.cat/transit/opendata/cameres.xml (the 3 kept frames are Barcelona IMI cameras on www.bcn.cat; SCT's own mct.gencat.cat host would not answer over HTTPS) |
Skipped: Ontario 511 (now answers "Invalid Key"), DelDOT (video only), Warendorf (http-only host, and the proxy allows HTTPS only).
The frames are public agency imagery, used here under each agency's open-data or terms-of-use conditions. Credit the agency named in `source`. TfL: "Powered by TfL Open Data". NSW: © Transport for NSW. HK: © HKSAR Transport Department (data.gov.hk). Singapore: LTA via data.gov.sg (Singapore Open Data Licence). NZ: © NZ Transport Agency Waka Kotahi (CC BY 4.0).
+26
View File
@@ -0,0 +1,26 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>God's Eye View — First Light</title>
<style>
@font-face { font-family: 'JetBrains Mono'; font-weight: 400; src: url(node_modules/@fontsource/jetbrains-mono/files/jetbrains-mono-latin-400-normal.woff2) format('woff2'); }
@font-face { font-family: 'JetBrains Mono'; font-weight: 500; src: url(node_modules/@fontsource/jetbrains-mono/files/jetbrains-mono-latin-500-normal.woff2) format('woff2'); }
@font-face { font-family: 'Inter'; font-weight: 200; src: url(node_modules/@fontsource/inter/files/inter-latin-200-normal.woff2) format('woff2'); }
@font-face { font-family: 'Inter'; font-weight: 300; src: url(node_modules/@fontsource/inter/files/inter-latin-300-normal.woff2) format('woff2'); }
@font-face { font-family: 'Inter'; font-weight: 500; src: url(node_modules/@fontsource/inter/files/inter-latin-500-normal.woff2) format('woff2'); }
html, body { margin: 0; background: #000; overflow: hidden; }
canvas { display: block; }
</style>
<script type="importmap">
{ "imports": {
"three": "./node_modules/three/build/three.module.js",
"satellite.js": "./node_modules/satellite.js/dist/satellite.es.js"
} }
</script>
</head>
<body>
<canvas id="c"></canvas>
<script type="module" src="src/main.js"></script>
</body>
</html>
+53
View File
@@ -0,0 +1,53 @@
{
"name": "gev-intro-sequence",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "gev-intro-sequence",
"dependencies": {
"@fontsource/inter": "^5.1.1",
"@fontsource/jetbrains-mono": "^5.1.1",
"playwright-core": "^1.56.1",
"satellite.js": "^5.0.0",
"three": "^0.169.0"
}
},
"node_modules/@fontsource/inter": {
"version": "5.1.1",
"resolved": "https://registry.npmjs.org/@fontsource/inter/-/inter-5.1.1.tgz",
"integrity": "sha512-weN3E+rq0Xb3Z93VHJ+Rc7WOQX9ETJPTAJ+gDcaMHtjft67L58sfS65rAjC5tZUXQ2FdZ/V1/sSzCwZ6v05kJw==",
"license": "OFL-1.1"
},
"node_modules/@fontsource/jetbrains-mono": {
"version": "5.1.1",
"resolved": "https://registry.npmjs.org/@fontsource/jetbrains-mono/-/jetbrains-mono-5.1.1.tgz",
"integrity": "sha512-5rwvmdQQpXev4LlBX1P+7h2dguu6iwW/9Npjde4+DEq+HgpVJI/7QY8DI1NVVFdvLtXZNP+vO2L/5BQED6FUhA==",
"license": "OFL-1.1"
},
"node_modules/playwright-core": {
"version": "1.56.1",
"resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.56.1.tgz",
"integrity": "sha512-hutraynyn31F+Bifme+Ps9Vq59hKuUCz7H1kDOcBs+2oGguKkWTU50bBWrtz34OUWmIwpBTWDxaRPXrIXkgvmQ==",
"license": "Apache-2.0",
"bin": {
"playwright-core": "cli.js"
},
"engines": {
"node": ">=18"
}
},
"node_modules/satellite.js": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/satellite.js/-/satellite.js-5.0.0.tgz",
"integrity": "sha512-ie3yiJ2LJAJIhVUKdYhgp7V0btXKAMImDjRnuaNfJGl8rjwP2HwVIh4HLFcpiXYEiYwXc5fqh5+yZqCe6KIwWw==",
"license": "MIT"
},
"node_modules/three": {
"version": "0.169.0",
"resolved": "https://registry.npmjs.org/three/-/three-0.169.0.tgz",
"integrity": "sha512-Ed906MA3dR4TS5riErd4QBsRGPcx+HBDX2O5yYE5GqJeFQTPU+M56Va/f/Oph9X7uZo3W3o4l2ZhBZ6f6qUv0w==",
"license": "MIT"
}
}
}
+14
View File
@@ -0,0 +1,14 @@
{
"name": "gev-intro-sequence",
"private": true,
"type": "module",
"description": "God's Eye View: the intro sequence where the system comes online. Deterministic WebGL renderer + procedural score.",
"scripts": {},
"dependencies": {
"@fontsource/inter": "^5.1.1",
"@fontsource/jetbrains-mono": "^5.1.1",
"playwright-core": "^1.56.1",
"satellite.js": "^5.0.0",
"three": "^0.169.0"
}
}
+916
View File
@@ -0,0 +1,916 @@
import * as THREE from 'three';
import { mulberry32, clamp } from './timeline.js';
// City scenes in a local ENU frame (data in metres, scene units in km;
// three.js x = east, y = up, z = -north). One engine renders every location:
// Esri ground, OSM buildings that rise as the scan passes, road ribbons that
// carry modelled traffic coloured by congestion, cameras, aircraft, transit,
// airports, restricted areas and terrain.
const KM = 0.001;
async function getJSON(p) {
try {
const r = await fetch(p);
return r.ok ? await r.json() : null;
} catch {
return null;
}
}
async function getBin(p) {
try {
const r = await fetch(p);
return r.ok ? await r.arrayBuffer() : null;
} catch {
return null;
}
}
function loadTex(p) {
return new Promise((res) =>
new THREE.TextureLoader().load(
p,
(t) => {
t.colorSpace = THREE.SRGBColorSpace;
t.anisotropy = 8;
res(t);
},
undefined,
() => res(null),
),
);
}
export async function loadCity(slug) {
const base = `city/${slug}/`;
const [meta, buildings, roads, extra, cameras, air, transit, vehicles, aeroway, military, peaks, elevMeta, elev, ground] = await Promise.all([
getJSON(base + 'meta.json'),
getJSON(base + 'buildings.json'),
getJSON(base + 'roads.json'),
getJSON(base + 'extra.json'),
getJSON(base + 'cameras.json'),
getJSON(base + 'air.json'),
getJSON(base + 'transit.json'),
getJSON(base + 'vehicles.json'),
getJSON(base + 'aeroway.json'),
getJSON(base + 'military.json'),
getJSON(base + 'peaks.json'),
getJSON(base + 'elev_meta.json'),
getBin(base + 'elev.f32'),
loadTex(base + 'ground.jpg'),
]);
return { slug, meta, buildings, roads, extra, cameras, air, transit, vehicles, aeroway, military, peaks, elevMeta, elev: elev ? new Float32Array(elev) : null, ground };
}
// ---------------------------------------------------------------- shared shader bits
const HASH = /* glsl */ `
float h11(float n) { return fract(sin(n * 127.1) * 43758.5453); }
float h21(vec2 p) { return fract(sin(dot(p, vec2(127.1, 311.7))) * 43758.5453); }
`;
function congestionColor() {
return /* glsl */ `
vec3 congCol(float c) {
vec3 g = vec3(0.10, 1.00, 0.45);
vec3 y = vec3(1.00, 0.80, 0.10);
vec3 r = vec3(1.00, 0.16, 0.12);
return c < 0.5 ? mix(g, y, c * 2.0) : mix(y, r, (c - 0.5) * 2.0);
}
`;
}
// ---------------------------------------------------------------- ground
function makeGround(d, half, elevGeo) {
const size = 2 * half * KM;
const geo = elevGeo || new THREE.PlaneGeometry(size, size, 1, 1).rotateX(-Math.PI / 2);
const mat = new THREE.ShaderMaterial({
uniforms: {
uTex: { value: d.ground },
uReveal: { value: 0 },
uT: { value: 0 },
uHalf: { value: half * KM },
uNight: { value: 0.55 },
uTerrain: { value: elevGeo ? 1 : 0 },
uScan: { value: -10 },
uFade: { value: 1 },
},
vertexShader: /* glsl */ `
attribute float aElev;
varying vec2 vUv;
varying vec3 vP;
varying float vElev;
varying vec3 vN;
void main() {
vUv = uv;
vP = position;
vElev = aElev;
vN = normal;
gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
}
`,
fragmentShader: /* glsl */ `
uniform sampler2D uTex;
uniform float uReveal, uT, uHalf, uNight, uTerrain, uScan, uFade;
varying vec2 vUv;
varying vec3 vP;
varying float vElev;
varying vec3 vN;
void main() {
vec3 img = texture2D(uTex, vUv).rgb;
float l = dot(img, vec3(0.3, 0.59, 0.11));
// night grade: cool, dim, contrasty — real pixels, but the city's lights do the talking
vec3 night = pow(mix(vec3(l), img, 0.35), vec3(1.7)) * vec3(0.5, 0.7, 0.95) * 0.56;
float r = length(vP.xz) / uHalf;
float rev = smoothstep(uReveal, uReveal - 0.08, r);
// tactical grid: 100 m, brighter every km
vec2 g = vP.xz * 1000.0;
vec2 fw = fwidth(g);
vec2 gl = abs(fract(g / 100.0 - 0.5) - 0.5) * 100.0 / fw;
float grid = 1.0 - smoothstep(0.0, 1.0, min(gl.x, gl.y));
vec2 gk = abs(fract(g / 1000.0 - 0.5) - 0.5) * 1000.0 / fw;
float gridK = 1.0 - smoothstep(0.0, 1.3, min(gk.x, gk.y));
vec3 col = night * rev;
col += vec3(0.0, 0.55, 0.7) * (grid * 0.05 + gridK * 0.14) * (0.4 + 0.6 * rev);
// the scan front
float ring = exp(-pow((r - uReveal) * 60.0, 2.0));
col += vec3(0.2, 0.9, 1.0) * ring * 0.6;
if (uTerrain > 0.5) {
// terrain: sunlit relief, 100 m contours, a LiDAR line sweeping across
vec3 L = normalize(vec3(-0.5, 0.7, 0.4));
float sh = clamp(dot(normalize(vN), L), 0.0, 1.0);
vec3 tg = pow(mix(vec3(l), img, 0.5), vec3(1.35));
col = tg * (0.08 + 0.62 * sh) * vec3(0.62, 0.78, 1.0) * rev;
float ce = vElev / 100.0;
float cw = fwidth(ce);
float cont = 1.0 - smoothstep(0.0, 1.2 * cw, abs(fract(ce - 0.5) - 0.5));
float major = 1.0 - smoothstep(0.0, 1.2 * fwidth(vElev / 500.0), abs(fract(vElev / 500.0 - 0.5) - 0.5));
col += vec3(0.1, 0.8, 1.0) * (cont * 0.16 + major * 0.5) * rev;
float sweep = exp(-pow((vP.x - uScan) * 18.0, 2.0));
col += vec3(0.4, 1.0, 1.0) * sweep * 0.9;
col += vec3(0.0, 0.6, 0.8) * smoothstep(uScan, uScan - 3.0, vP.x) * 0.03 * step(vP.x, uScan);
}
float edge = smoothstep(1.0, 0.82, max(abs(vP.x), abs(vP.z)) / uHalf);
gl_FragColor = vec4(col * edge * uFade, 1.0);
}
`,
});
// the ground never writes depth: it is the base layer, and road ribbons a
// metre above it would otherwise lose the depth test at city distances
if (!elevGeo) mat.depthWrite = false;
const mesh = new THREE.Mesh(geo, mat);
mesh.renderOrder = -1;
return mesh;
}
function makeTerrainGeo(d, half) {
const n = d.elevMeta?.size || 1024;
const seg = 384;
const size = 2 * half * KM;
const geo = new THREE.PlaneGeometry(size, size, seg, seg).rotateX(-Math.PI / 2);
const pos = geo.getAttribute('position');
const uv = geo.getAttribute('uv');
const el = new Float32Array(pos.count);
for (let i = 0; i < pos.count; i++) {
const u = uv.getX(i);
const v = uv.getY(i);
const px = Math.min(n - 1, Math.max(0, Math.round(u * (n - 1))));
const py = Math.min(n - 1, Math.max(0, Math.round((1 - v) * (n - 1))));
const h = d.elev[py * n + px];
el[i] = h;
pos.setY(i, h * KM);
}
geo.setAttribute('aElev', new THREE.BufferAttribute(el, 1));
geo.computeVertexNormals();
return geo;
}
// ---------------------------------------------------------------- buildings
function makeBuildings(list) {
const P = [];
const UV = [];
const A = []; // height m, edge length m, dist km, rand
const idx = [];
let vi = 0;
const rnd = mulberry32(5);
const rings = [];
for (const b of list) {
const h = Math.max(4, Math.min(560, b[0] || 10));
const pts = [];
for (let i = 2; i < b.length - 1; i += 2) pts.push([b[i], b[i + 1]]);
if (pts.length < 3) continue;
let cx = 0;
let cy = 0;
for (const p of pts) {
cx += p[0];
cy += p[1];
}
cx /= pts.length;
cy /= pts.length;
const dist = Math.hypot(cx, cy) * KM;
const r = rnd();
// walls
for (let i = 0; i < pts.length; i++) {
const [x1, y1] = pts[i];
const [x2, y2] = pts[(i + 1) % pts.length];
const L = Math.hypot(x2 - x1, y2 - y1);
if (L < 0.3) continue;
P.push(x1 * KM, 0, -y1 * KM, x2 * KM, 0, -y2 * KM, x2 * KM, h * KM, -y2 * KM, x1 * KM, h * KM, -y1 * KM);
UV.push(0, 0, 1, 0, 1, 1, 0, 1);
for (let k = 0; k < 4; k++) A.push(h, L, dist, r);
idx.push(vi, vi + 1, vi + 2, vi, vi + 2, vi + 3);
vi += 4;
}
// roof
const contour = pts.map((p) => new THREE.Vector2(p[0], p[1]));
let tris = [];
try {
tris = THREE.ShapeUtils.triangulateShape(contour, []);
} catch {
tris = [];
}
const base = vi;
for (const p of pts) {
P.push(p[0] * KM, h * KM, -p[1] * KM);
UV.push(-1, 1); // u < 0 marks roof
A.push(h, 0, dist, r);
vi++;
}
for (const t of tris) idx.push(base + t[0], base + t[2], base + t[1]);
rings.push({ pts, h, dist });
}
const geo = new THREE.BufferGeometry();
geo.setAttribute('position', new THREE.Float32BufferAttribute(P, 3));
geo.setAttribute('uv', new THREE.Float32BufferAttribute(UV, 2));
geo.setAttribute('aB', new THREE.Float32BufferAttribute(A, 4));
geo.setIndex(vi > 65535 ? new THREE.Uint32BufferAttribute(idx, 1) : new THREE.Uint16BufferAttribute(idx, 1));
const mat = new THREE.ShaderMaterial({
uniforms: { uReveal: { value: 0 }, uT: { value: 0 }, uAlpha: { value: 1 }, uTint: { value: new THREE.Color('#38e8ff') } },
vertexShader: /* glsl */ `
attribute vec4 aB;
uniform float uReveal;
varying vec2 vUv;
varying vec4 vB;
varying vec3 vW;
varying float vRise;
void main() {
vUv = uv;
vB = aB;
float k = clamp((uReveal - aB.z) / 0.35, 0.0, 1.0);
float rise = 1.0 - pow(1.0 - k, 3.0);
vRise = k;
vec3 p = position;
p.y *= rise;
vec4 wp = modelMatrix * vec4(p, 1.0);
vW = wp.xyz;
gl_Position = projectionMatrix * viewMatrix * wp;
}
`,
fragmentShader: /* glsl */ `
${HASH}
uniform float uReveal, uT, uAlpha;
uniform vec3 uTint;
varying vec2 vUv;
varying vec4 vB;
varying vec3 vW;
varying float vRise;
void main() {
if (vRise <= 0.0) discard;
float h = vB.x;
float L = vB.y;
bool roof = vUv.x < -0.5;
vec3 N = normalize(cross(dFdx(vW), dFdy(vW)));
float lit = 0.35 + 0.65 * abs(dot(N, normalize(vec3(0.4, 0.8, 0.3))));
float tall = smoothstep(30.0, 250.0, h);
vec3 base = mix(vec3(0.010, 0.016, 0.026), vec3(0.016, 0.032, 0.05), tall) * lit;
vec3 col = base;
float fresh = exp(-max(uReveal - vB.z, 0.0) * 5.0);
if (roof) {
col = base * 1.3 + uTint * 0.02;
} else {
// floors and windows
float fy = vUv.y * h / 3.4;
float fx = vUv.x * L / 3.2;
vec2 cell = floor(vec2(fx, fy));
vec2 f = fract(vec2(fx, fy));
float win = step(0.25, f.x) * step(f.x, 0.8) * step(0.3, f.y) * step(f.y, 0.75);
float on = step(0.9, h21(cell + vB.w * 91.0));
float warm = h21(cell * 1.7 + 3.0);
vec3 wc = mix(vec3(1.0, 0.75, 0.45), vec3(0.6, 0.9, 1.0), step(0.7, warm));
float flick = 0.85 + 0.15 * sin(uT * 2.0 + h21(cell) * 40.0);
col += wc * win * on * 0.32 * flick * smoothstep(0.0, 0.1, vRise);
// edges: crown line, vertical corners, base
float wy = fwidth(vUv.y);
float wx = fwidth(vUv.x);
float crown = 1.0 - smoothstep(0.0, wy * 1.5, 1.0 - vUv.y);
float corner = 1.0 - smoothstep(0.0, wx * 1.5, min(vUv.x, 1.0 - vUv.x));
col += uTint * (crown * (0.08 + tall * 0.9) + corner * (0.03 + tall * 0.12)) * (0.6 + fresh * 2.5);
// the scan passing up the facade as it rises
col += uTint * fresh * 0.25 * smoothstep(0.2, 1.0, vUv.y);
}
gl_FragColor = vec4(col * uAlpha, 1.0);
}
`,
});
const mesh = new THREE.Mesh(geo, mat);
mesh.frustumCulled = false;
return { mesh, count: rings.length, rings };
}
// ---------------------------------------------------------------- roads + traffic
// Congestion is modelled, not measured: road class, distance to the core, and
// a slow per-road noise. Labelled MODELED in the HUD.
function makeRoads(list, half, { tint = null, inside = null, scale = 1 } = {}) {
const W = [34, 24, 20, 16, 11, 7];
const P = [];
const A = []; // arc m, side, cong, oneway
const B = []; // class, rnd, width
const idx = [];
let vi = 0;
const rnd = mulberry32(17);
let segs = 0;
for (const r of list) {
const cls = r[0];
const one = r[1] ? 1 : 0;
const pts = [];
for (let i = 3; i < r.length - 1; i += 2) pts.push([r[i], r[i + 1]]);
if (pts.length < 2) continue;
if (inside && !inside(pts[0][0], pts[0][1])) continue;
const w = W[cls] ?? 6;
const cx = pts[Math.floor(pts.length / 2)];
const core = Math.exp(-Math.hypot(cx[0], cx[1]) / (half * 0.55));
const base = [0.55, 0.5, 0.42, 0.35, 0.22, 0.3][cls] ?? 0.3;
const rr = rnd();
const cong = clamp(base * 0.55 + core * 0.55 + (rr - 0.5) * 0.95, 0, 1);
let arc = 0;
for (let i = 0; i < pts.length - 1; i++) {
const [x1, y1] = pts[i];
const [x2, y2] = pts[i + 1];
const L = Math.hypot(x2 - x1, y2 - y1);
if (L < 0.5) continue;
const nx = -(y2 - y1) / L;
const ny = (x2 - x1) / L;
const hw = w / 2;
const y = 0.6 + cls * 0.02;
const v = [
[x1 - nx * hw, y1 - ny * hw, -1, arc],
[x1 + nx * hw, y1 + ny * hw, 1, arc],
[x2 + nx * hw, y2 + ny * hw, 1, arc + L],
[x2 - nx * hw, y2 - ny * hw, -1, arc + L],
];
for (const [x, yy, s, a] of v) {
P.push(x * KM, y * KM, -yy * KM);
A.push(a, s, cong, one);
B.push(cls, rr, w);
}
idx.push(vi, vi + 1, vi + 2, vi, vi + 2, vi + 3);
vi += 4;
arc += L;
segs++;
}
}
const geo = new THREE.BufferGeometry();
geo.setAttribute('position', new THREE.Float32BufferAttribute(P, 3));
geo.setAttribute('aA', new THREE.Float32BufferAttribute(A, 4));
geo.setAttribute('aC', new THREE.Float32BufferAttribute(B, 3));
geo.setIndex(new THREE.Uint32BufferAttribute(idx, 1));
const mat = new THREE.ShaderMaterial({
transparent: true,
side: THREE.DoubleSide,
depthWrite: false,
blending: THREE.AdditiveBlending,
uniforms: {
uT: { value: 0 },
uReveal: { value: 0 },
uHalf: { value: half * KM },
uAlpha: { value: 1 },
uTint: { value: tint ? new THREE.Color(tint) : new THREE.Color(0, 0, 0) },
uUseTint: { value: tint ? 1 : 0 },
uScale: { value: scale },
},
vertexShader: /* glsl */ `
attribute vec4 aA;
attribute vec3 aC;
varying vec4 vA;
varying vec3 vC;
varying float vR;
void main() {
vA = aA;
vC = aC;
vR = length(position.xz);
gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
}
`,
fragmentShader: /* glsl */ `
${HASH}
${congestionColor()}
uniform float uT, uReveal, uHalf, uAlpha, uUseTint, uScale;
uniform vec3 uTint;
varying vec4 vA;
varying vec3 vC;
varying float vR;
void main() {
float r = vR / uHalf;
float k = uReveal - r;
if (k < 0.0) discard;
float cong = vA.z;
float side = vA.y;
float dir = vA.w > 0.5 ? 1.0 : (side > 0.0 ? 1.0 : -1.0);
vec3 cc = mix(congCol(cong), uTint, uUseTint);
// ribbon cross-profile: soft edges, dark median
float a = abs(side);
float body = smoothstep(1.0, 0.7, a);
// cars: dashes flowing along each lane, slower and denser when jammed
float speed = mix(16.0, 2.5, cong) * 7.0 * uScale;
float spacing = mix(70.0, 14.0, cong);
float lane = vA.w > 0.5 ? 1.0 : smoothstep(0.05, 0.3, a) * smoothstep(0.98, 0.7, a);
float ph = (vA.x - dir * uT * speed) / spacing + vC.y * 13.0 + (side > 0.0 ? 0.37 : 0.0);
float fp = fract(ph);
float car = smoothstep(0.0, 0.04, fp) * (1.0 - smoothstep(0.08, 0.22, fp));
float present = step(0.25 + 0.5 * (1.0 - cong), h11(floor(ph) + vC.y * 77.0));
float head = car * present * lane;
vec3 carCol = dir > 0.0 ? vec3(1.0, 0.95, 0.85) : vec3(1.0, 0.25, 0.2);
vec3 col = cc * body * (0.2 + 0.12 * cong) + mix(cc, carCol, 0.22) * head * 1.25;
float fresh = exp(-k * 25.0);
col += cc * fresh * 0.8 * body;
gl_FragColor = vec4(col * uAlpha, 1.0);
}
`,
});
const mesh = new THREE.Mesh(geo, mat);
mesh.frustumCulled = false;
return { mesh, segs };
}
// ---------------------------------------------------------------- lines (generic, glowing)
function makePolylines(lines, { color = '#38e8ff', y = 1, width = 0, alpha = 1, colors = null } = {}) {
const P = [];
const C = [];
const A = [];
for (let li = 0; li < lines.length; li++) {
const l = lines[li];
const col = new THREE.Color(colors ? colors[li] : color);
let arc = 0;
for (let i = 0; i < l.length - 1; i++) {
const [x1, y1, z1 = y] = l[i];
const [x2, y2, z2 = y] = l[i + 1];
const L = Math.hypot(x2 - x1, y2 - y1);
P.push(x1 * KM, z1 * KM, -y1 * KM, x2 * KM, z2 * KM, -y2 * KM);
C.push(col.r, col.g, col.b, col.r, col.g, col.b);
A.push(arc, arc + L);
arc += L;
}
}
const geo = new THREE.BufferGeometry();
geo.setAttribute('position', new THREE.Float32BufferAttribute(P, 3));
geo.setAttribute('color', new THREE.Float32BufferAttribute(C, 3));
geo.setAttribute('aArc', new THREE.Float32BufferAttribute(A, 1));
const mat = new THREE.ShaderMaterial({
transparent: true,
depthWrite: false,
blending: THREE.AdditiveBlending,
vertexColors: true,
uniforms: { uT: { value: 0 }, uAlpha: { value: alpha }, uDraw: { value: 1e9 }, uPulse: { value: 0 } },
vertexShader: /* glsl */ `
attribute float aArc;
varying float vArc;
varying vec3 vCol;
void main() { vArc = aArc; vCol = color; gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0); }
`,
fragmentShader: /* glsl */ `
uniform float uT, uAlpha, uDraw, uPulse;
varying float vArc;
varying vec3 vCol;
void main() {
if (vArc > uDraw) discard;
float p = uPulse > 0.0 ? pow(fract(vArc / 900.0 - uT * 0.8), 8.0) * 2.0 : 0.0;
gl_FragColor = vec4(vCol * (0.7 + p) * uAlpha, 1.0);
}
`,
});
return new THREE.LineSegments(geo, mat);
}
// glowing point sprites updated from the CPU
function makePoints(n, { size = 6, color = '#ffffff' } = {}) {
const geo = new THREE.BufferGeometry();
const pos = new Float32Array(n * 3);
const col = new Float32Array(n * 3);
const sz = new Float32Array(n).fill(1);
const c = new THREE.Color(color);
for (let i = 0; i < n; i++) col.set([c.r, c.g, c.b], i * 3);
geo.setAttribute('position', new THREE.BufferAttribute(pos, 3));
geo.setAttribute('color', new THREE.BufferAttribute(col, 3));
geo.setAttribute('aS', new THREE.BufferAttribute(sz, 1));
const mat = new THREE.ShaderMaterial({
transparent: true,
depthWrite: false,
depthTest: false,
blending: THREE.AdditiveBlending,
vertexColors: true,
uniforms: { uSize: { value: size }, uAlpha: { value: 1 }, uPx: { value: 1 } },
vertexShader: /* glsl */ `
attribute float aS;
uniform float uSize, uPx;
varying vec3 vCol;
void main() { vCol = color; gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0); gl_PointSize = uSize * aS * uPx; }
`,
fragmentShader: /* glsl */ `
uniform float uAlpha;
varying vec3 vCol;
void main() {
float r = length(gl_PointCoord - 0.5) * 2.0;
float a = smoothstep(1.0, 0.0, r);
float core = smoothstep(0.35, 0.0, r);
gl_FragColor = vec4((vCol * a * 0.8 + vec3(1.0) * core) * uAlpha, 1.0);
}
`,
});
const pts = new THREE.Points(geo, mat);
pts.frustumCulled = false;
return { pts, pos, col, sz, geo };
}
// ---------------------------------------------------------------- the engine
export function buildCity(d, opts = {}) {
const half = d.meta?.half || 3500;
const scene = new THREE.Scene();
const group = new THREE.Group();
scene.add(group);
const out = { scene, d, half, labels: [], stats: {} };
const terrainGeo = d.elev ? makeTerrainGeo(d, half) : null;
const ground = makeGround(d, half, terrainGeo);
group.add(ground);
out.ground = ground;
if (d.elev) {
out.elevAt = (x, y) => {
const n = d.elevMeta?.size || 1024;
const px = Math.round(((x + half) / (2 * half)) * (n - 1));
const py = Math.round(((half - y) / (2 * half)) * (n - 1));
return d.elev[clamp(py, 0, n - 1) * n + clamp(px, 0, n - 1)] || 0;
};
}
if (d.buildings?.b?.length) {
const bl = makeBuildings(d.buildings.b);
group.add(bl.mesh);
out.buildings = bl;
out.stats.buildings = bl.count;
}
// military polygons: roads inside get the amber treatment
let insideMil = null;
if (d.military?.military?.length) {
const polys = d.military.military.map((m) => {
const pts = [];
for (let i = 1; i < m.length - 1; i += 2) pts.push([m[i], m[i + 1]]);
return { name: m[0], pts };
}).filter((p) => p.pts.length > 2 && p.pts.every(([x, y]) => Math.abs(x) < half * 1.35 && Math.abs(y) < half * 1.35));
// the reservation itself is bigger than the scene: keep what fits
out.milPolys = polys;
insideMil = (x, y) => polys.some((p) => pip(x, y, p.pts));
}
if (d.roads?.roads?.length) {
const rd = makeRoads(d.roads.roads, half);
group.add(rd.mesh);
out.roads = rd;
out.stats.roads = rd.segs;
if (insideMil) {
const conv = makeRoads(d.roads.roads, half, { tint: '#ffb547', inside: insideMil, scale: 0.6 });
conv.mesh.material.uniforms.uAlpha.value = 0.9;
group.add(conv.mesh);
out.convoy = conv;
}
}
// restricted areas: translucent curtains with scanning bands
if (out.milPolys) {
const P = [];
const U = [];
for (const p of out.milPolys) {
let arc = 0;
for (let i = 0; i < p.pts.length; i++) {
const [x1, y1] = p.pts[i];
const [x2, y2] = p.pts[(i + 1) % p.pts.length];
const L = Math.hypot(x2 - x1, y2 - y1);
const H = 0.35;
P.push(x1 * KM, 0, -y1 * KM, x2 * KM, 0, -y2 * KM, x2 * KM, H, -y2 * KM, x1 * KM, 0, -y1 * KM, x2 * KM, H, -y2 * KM, x1 * KM, H, -y1 * KM);
U.push(arc, 0, arc + L, 0, arc + L, 1, arc, 0, arc + L, 1, arc, 1);
arc += L;
}
}
const g = new THREE.BufferGeometry();
g.setAttribute('position', new THREE.Float32BufferAttribute(P, 3));
g.setAttribute('uv', new THREE.Float32BufferAttribute(U, 2));
const m = new THREE.ShaderMaterial({
transparent: true,
depthWrite: false,
side: THREE.DoubleSide,
blending: THREE.AdditiveBlending,
uniforms: { uT: { value: 0 }, uAlpha: { value: 0 } },
vertexShader: `varying vec2 vUv; void main(){ vUv = uv; gl_Position = projectionMatrix * modelViewMatrix * vec4(position,1.0); }`,
fragmentShader: /* glsl */ `
uniform float uT, uAlpha;
varying vec2 vUv;
void main() {
float band = pow(fract(vUv.y * 6.0 - uT * 0.6), 6.0);
float top = 1.0 - smoothstep(0.0, 0.04, 1.0 - vUv.y);
float fade = 1.0 - vUv.y;
float hatch = step(0.5, fract((vUv.x / 60.0 + vUv.y * 4.0)));
vec3 c = vec3(1.0, 0.62, 0.18) * (fade * 0.10 + band * 0.25 + top * 0.8 + hatch * fade * 0.03);
gl_FragColor = vec4(c * uAlpha, 1.0);
}
`,
});
out.curtain = new THREE.Mesh(g, m);
out.curtain.frustumCulled = false;
group.add(out.curtain);
}
// airport: runways, approach strobes, taxiways
const aw = d.aeroway || d.military;
if (aw?.runways?.length) {
const P = [];
const U = [];
const strobes = [];
for (const rw of aw.runways) {
const w = Math.max(30, rw[0] || 45);
const pts = [];
for (let i = 2; i < rw.length - 1; i += 2) pts.push([rw[i], rw[i + 1]]);
if (pts.length < 2) continue;
const [x1, y1] = pts[0];
const [x2, y2] = pts[pts.length - 1];
const L = Math.hypot(x2 - x1, y2 - y1);
const ux = (x2 - x1) / L;
const uy = (y2 - y1) / L;
const nx = -uy * w / 2;
const ny = ux * w / 2;
const q = [[x1 - nx, y1 - ny, 0, 0], [x1 + nx, y1 + ny, 1, 0], [x2 + nx, y2 + ny, 1, L], [x1 - nx, y1 - ny, 0, 0], [x2 + nx, y2 + ny, 1, L], [x2 - nx, y2 - ny, 0, L]];
for (const [x, y, s, a] of q) {
P.push(x * KM, 0.0012, -y * KM);
U.push(s, a);
}
// approach lighting: 900 m of strobes off each threshold
for (const [sx, sy, dx, dy] of [[x1, y1, -ux, -uy], [x2, y2, ux, uy]]) {
for (let k = 1; k <= 30; k++) strobes.push([sx + dx * k * 30, sy + dy * k * 30, k]);
}
out.labels.push({ kind: 'runway', x: (x1 + x2) / 2, y: (y1 + y2) / 2, text: `RWY ${rw[1] || ''}`.trim() });
}
const g = new THREE.BufferGeometry();
g.setAttribute('position', new THREE.Float32BufferAttribute(P, 3));
g.setAttribute('uv', new THREE.Float32BufferAttribute(U, 2));
const m = new THREE.ShaderMaterial({
transparent: true,
side: THREE.DoubleSide,
depthWrite: false,
blending: THREE.AdditiveBlending,
uniforms: { uT: { value: 0 }, uAlpha: { value: 1 } },
vertexShader: `varying vec2 vUv; void main(){ vUv = uv; gl_Position = projectionMatrix * modelViewMatrix * vec4(position,1.0); }`,
fragmentShader: /* glsl */ `
uniform float uT, uAlpha;
varying vec2 vUv;
void main() {
float edge = 1.0 - smoothstep(0.0, 0.06, min(vUv.x, 1.0 - vUv.x));
float cl = (1.0 - smoothstep(0.0, 0.03, abs(vUv.x - 0.5))) * step(0.5, fract(vUv.y / 60.0));
vec3 c = vec3(0.9, 0.95, 1.0) * (edge * 0.9 + cl * 0.7) + vec3(0.05, 0.08, 0.1);
gl_FragColor = vec4(c * uAlpha, 1.0);
}
`,
});
out.runways = new THREE.Mesh(g, m);
out.runways.frustumCulled = false;
group.add(out.runways);
out.strobes = makePoints(strobes.length, { size: 5, color: '#ffffff' });
strobes.forEach((s, i) => out.strobes.pos.set([s[0] * KM, 0.004, -s[1] * KM], i * 3));
out.strobeK = strobes.map((s) => s[2]);
out.strobes.geo.getAttribute('position').needsUpdate = true;
group.add(out.strobes.pts);
}
if (aw?.taxiways?.length) {
const lines = aw.taxiways.map((t) => {
const pts = [];
for (let i = 0; i < t.length - 1; i += 2) pts.push([t[i], t[i + 1], 1.2]);
return pts;
});
out.taxi = makePolylines(lines, { color: '#ffc04a', alpha: 0.75 });
group.add(out.taxi);
}
if (aw?.aprons?.length) {
const lines = aw.aprons.map((t) => {
const pts = [];
for (let i = 0; i < t.length - 1; i += 2) pts.push([t[i], t[i + 1], 1]);
if (pts.length) pts.push(pts[0]);
return pts;
});
out.aprons = makePolylines(lines, { color: '#2fd0ff', alpha: 0.4 });
group.add(out.aprons);
}
// transit: route-coloured lines with pulses, vehicles riding them
if (d.transit?.shapes?.length) {
const lines = [];
const colors = [];
out.shapes = [];
for (const s of d.transit.shapes) {
const rid = s[0];
const route = d.transit.routes[rid] || {};
const pts = [];
for (let i = 1; i < s.length - 1; i += 2) pts.push([s[i], s[i + 1], route.type === 3 ? 2 : 6]);
const col = route.color ? (route.color.startsWith('#') ? route.color : '#' + route.color) : '#38e8ff';
// subway and rail glow brighter than buses
lines.push(pts);
colors.push(route.type === 3 ? new THREE.Color(col).multiplyScalar(0.45).getStyle() : col);
out.shapes.push({ rid, pts, type: route.type });
}
out.transitLines = makePolylines(lines, { colors });
out.transitLines.material.uniforms.uPulse.value = 1;
group.add(out.transitLines);
out.stats.routes = Object.keys(d.transit.routes).length;
}
if (d.vehicles?.v?.length) {
const vs = d.vehicles.v;
out.veh = makePoints(vs.length, { size: 9 });
vs.forEach((v, i) => {
const r = d.transit?.routes?.[v[4]];
const c = new THREE.Color(r?.color ? (r.color.startsWith('#') ? r.color : '#' + r.color) : '#ffffff');
out.veh.col.set([c.r, c.g, c.b], i * 3);
out.veh.sz[i] = r?.type === 3 ? 0.65 : 1.2;
});
out.veh.geo.getAttribute('color').needsUpdate = true;
out.veh.geo.getAttribute('aS').needsUpdate = true;
group.add(out.veh.pts);
out.stats.vehicles = vs.length;
}
// street cameras: beams + base rings
if (d.cameras?.length) {
const lines = d.cameras.map((c) => [[c[0], c[1], 0], [c[0], c[1], 90]]);
out.camBeams = makePolylines(lines, { color: '#eafcff', alpha: 0.5 });
group.add(out.camBeams);
out.camPts = makePoints(d.cameras.length, { size: 7, color: '#bff6ff' });
d.cameras.forEach((c, i) => out.camPts.pos.set([c[0] * KM, 0.09, -c[1] * KM], i * 3));
out.camPts.geo.getAttribute('position').needsUpdate = true;
group.add(out.camPts.pts);
out.stats.cameras = d.cameras.length;
}
// aircraft (real ADS-B): glyphs + real track history
if (d.air?.a?.length) {
const a = d.air.a;
out.airPts = makePoints(a.length + 8, { size: 8, color: '#ffffff' });
a.forEach((p, i) => {
const heli = p[8] === 'A7' || /^(H|EC|AS|S7|S9|B06|B407|B429|A109|A119|A139|R44|R66|EC|H1|H6)/.test(p[7] || '');
const mil = /^(RCH|CNV|PAT|EVAC|TOPCT|HKY|SPAR|SAM)/.test(p[6] || '');
const c = new THREE.Color(heli ? '#7dffd6' : mil ? '#ffb547' : '#ffffff');
out.airPts.col.set([c.r, c.g, c.b], i * 3);
p.heli = heli;
});
out.airPts.geo.getAttribute('color').needsUpdate = true;
group.add(out.airPts.pts);
const tr = d.air.tracks || {};
const tl = Object.values(tr).filter((t) => t.length > 1).map((t) => t.map((q) => [q[0], q[1], q[2] * 1.5]));
if (tl.length) {
out.airTracks = makePolylines(tl.map((t) => t.filter((q) => Math.abs(q[0]) < half * 1.2 && Math.abs(q[1]) < half * 1.2)).filter((t) => t.length > 1), { color: '#9fe8ff', alpha: 0.3 });
group.add(out.airTracks);
}
out.stats.aircraft = a.length;
out.stats.helis = a.filter((p) => p.heli).length;
}
// modelled extras (labelled MODELED): helicopter tours, circuits
out.modeled = [];
if (opts.modeled) for (const m of opts.modeled) out.modeled.push(m);
if (out.modeled.length) {
out.modPts = makePoints(out.modeled.length, { size: 9, color: '#7dffd6' });
out.modeled.forEach((m, i) => {
const c = new THREE.Color(m.color || '#7dffd6');
out.modPts.col.set([c.r, c.g, c.b], i * 3);
});
out.modPts.geo.getAttribute('color').needsUpdate = true;
group.add(out.modPts.pts);
out.modTrails = makePolylines(out.modeled.map((m) => m.path.concat([m.path[0]])), { color: '#7dffd6', alpha: 0.25 });
out.modTrails.material.vertexColors = true;
group.add(out.modTrails);
}
// ---------------------------------------------------------------- per-frame
out.update = (lt, k, pxScale = 1) => {
// lt: seconds into this scene; k: 0..1 progress
const reveal = Math.min(1.6, 0.22 + lt * 1.05);
ground.material.uniforms.uReveal.value = reveal * 1.3;
ground.material.uniforms.uT.value = lt;
if (d.elev) ground.material.uniforms.uScan.value = -half * KM + ((lt * 0.35) % 1.2) * 2.4 * half * KM;
if (out.buildings) {
out.buildings.mesh.material.uniforms.uReveal.value = reveal * half * KM * 0.9;
out.buildings.mesh.material.uniforms.uT.value = lt;
}
if (out.roads) {
out.roads.mesh.material.uniforms.uReveal.value = reveal * 1.1;
out.roads.mesh.material.uniforms.uT.value = lt;
}
if (out.convoy) {
out.convoy.mesh.material.uniforms.uReveal.value = reveal * 1.1;
out.convoy.mesh.material.uniforms.uT.value = lt;
}
if (out.curtain) {
out.curtain.material.uniforms.uT.value = lt;
out.curtain.material.uniforms.uAlpha.value = clamp((lt - 0.6) * 1.5);
}
if (out.runways) out.runways.material.uniforms.uAlpha.value = clamp(lt * 1.2);
if (out.strobes) {
// the "rabbit": a pulse running in toward each threshold, twice a second
const ph = (lt * 2) % 1;
out.strobeK.forEach((kk, i) => (out.strobes.sz[i] = Math.abs((1 - kk / 30) - ph) < 0.05 ? 2.2 : 0.35));
out.strobes.geo.getAttribute('aS').needsUpdate = true;
out.strobes.pts.material.uniforms.uPx.value = pxScale;
}
if (out.transitLines) {
out.transitLines.material.uniforms.uT.value = lt;
out.transitLines.material.uniforms.uDraw.value = lt * 9000;
}
if (out.veh) {
const vs = d.vehicles.v;
for (let i = 0; i < vs.length; i++) {
const v = vs[i];
// dead-reckon along the reported bearing at the reported speed, sped up
const sp = Math.max(4, v[3] || 8) * 6;
const b = ((v[2] || 0) * Math.PI) / 180;
const x = v[0] + Math.sin(b) * sp * lt;
const y = v[1] + Math.cos(b) * sp * lt;
out.veh.pos.set([x * KM, 0.012, -y * KM], i * 3);
}
out.veh.geo.getAttribute('position').needsUpdate = true;
out.veh.pts.material.uniforms.uAlpha.value = clamp((lt - 0.8) * 2);
out.veh.pts.material.uniforms.uPx.value = pxScale;
}
if (out.camPts) {
out.camPts.pts.material.uniforms.uAlpha.value = clamp((lt - 0.9) * 2);
out.camBeams.material.uniforms.uAlpha.value = clamp((lt - 0.9) * 2) * 0.45;
out.camPts.pts.material.uniforms.uPx.value = pxScale;
}
if (out.airPts) {
const a = d.air.a;
for (let i = 0; i < a.length; i++) {
const p = a[i];
const b = ((p[3] || 0) * Math.PI) / 180;
const sp = (p[4] || 0) * 3;
const x = p[0] + Math.sin(b) * sp * lt;
const y = p[1] + Math.cos(b) * sp * lt;
const z = Math.max(0, (p[2] || 0) + (p[5] || 0) * 3 * lt) * 1.5;
out.airPts.pos.set([x * KM, z * KM + 0.004, -y * KM], i * 3);
p.now = [x, y, z];
}
out.airPts.geo.getAttribute('position').needsUpdate = true;
out.airPts.pts.material.uniforms.uAlpha.value = clamp((lt - 0.5) * 2);
out.airPts.pts.material.uniforms.uPx.value = pxScale;
if (out.airTracks) out.airTracks.material.uniforms.uAlpha.value = clamp((lt - 0.5) * 2) * 0.4;
}
if (out.modPts) {
out.modeled.forEach((m, i) => {
const p = pathAt(m.path, (lt * m.speed + m.phase) % 1);
out.modPts.pos.set([p[0] * KM, p[2] * KM, -p[1] * KM], i * 3);
m.now = p;
});
out.modPts.geo.getAttribute('position').needsUpdate = true;
out.modPts.pts.material.uniforms.uAlpha.value = clamp((lt - 0.7) * 2);
out.modPts.pts.material.uniforms.uPx.value = pxScale;
out.modTrails.material.uniforms.uAlpha.value = clamp((lt - 0.7) * 2) * 0.22;
}
if (out.taxi) out.taxi.material.uniforms.uAlpha.value = clamp((lt - 0.4) * 2) * 0.7;
if (out.aprons) out.aprons.material.uniforms.uAlpha.value = clamp((lt - 0.4) * 2) * 0.35;
};
return out;
}
function pathAt(path, f) {
const n = path.length;
const x = f * n;
const i = Math.floor(x) % n;
const j = (i + 1) % n;
const u = x - Math.floor(x);
const a = path[i];
const b = path[j];
return [a[0] + (b[0] - a[0]) * u, a[1] + (b[1] - a[1]) * u, (a[2] ?? 0) + ((b[2] ?? 0) - (a[2] ?? 0)) * u];
}
function pip(x, y, pts) {
let c = false;
for (let i = 0, j = pts.length - 1; i < pts.length; j = i++) {
const [xi, yi] = pts[i];
const [xj, yj] = pts[j];
if (yi > y !== yj > y && x < ((xj - xi) * (y - yi)) / (yj - yi) + xi) c = !c;
}
return c;
}
// An ellipse/loop path in metres, for modelled helicopter tours and circuits.
export function loopPath(cx, cy, rx, ry, alt, rot = 0, n = 48) {
const out = [];
for (let i = 0; i < n; i++) {
const a = (i / n) * Math.PI * 2;
const x = Math.cos(a) * rx;
const y = Math.sin(a) * ry;
out.push([cx + x * Math.cos(rot) - y * Math.sin(rot), cy + x * Math.sin(rot) + y * Math.cos(rot), alt]);
}
return out;
}
+72
View File
@@ -0,0 +1,72 @@
// Geodesy for the film. Convention (matches the globe shader):
// x = cos(lat) cos(lon), y = sin(lat), z = -cos(lat) sin(lon) (Y up, radius 1)
export const D2R = Math.PI / 180;
export const R2D = 180 / Math.PI;
export const EARTH_KM = 6371;
export function llToVec(lon, lat, r = 1, out = [0, 0, 0]) {
const la = lat * D2R;
const lo = lon * D2R;
const c = Math.cos(la);
out[0] = r * c * Math.cos(lo);
out[1] = r * Math.sin(la);
out[2] = -r * c * Math.sin(lo);
return out;
}
export function vecToLL(x, y, z) {
const r = Math.hypot(x, y, z);
return [Math.atan2(-z, x) * R2D, Math.asin(y / r) * R2D];
}
// Central angle in radians between two lon/lat points.
export function angDist(lon1, lat1, lon2, lat2) {
const p1 = lat1 * D2R;
const p2 = lat2 * D2R;
const dp = p2 - p1;
const dl = (lon2 - lon1) * D2R;
const a = Math.sin(dp / 2) ** 2 + Math.cos(p1) * Math.cos(p2) * Math.sin(dl / 2) ** 2;
return 2 * Math.asin(Math.min(1, Math.sqrt(a)));
}
// Destination point given start, bearing (deg) and central angle (rad).
export function destination(lon, lat, bearingDeg, ang) {
const p1 = lat * D2R;
const l1 = lon * D2R;
const b = bearingDeg * D2R;
const p2 = Math.asin(Math.sin(p1) * Math.cos(ang) + Math.cos(p1) * Math.sin(ang) * Math.cos(b));
const l2 = l1 + Math.atan2(Math.sin(b) * Math.sin(ang) * Math.cos(p1), Math.cos(ang) - Math.sin(p1) * Math.sin(p2));
return [((l2 * R2D + 540) % 360) - 180, p2 * R2D];
}
export function bearing(lon1, lat1, lon2, lat2) {
const p1 = lat1 * D2R;
const p2 = lat2 * D2R;
const dl = (lon2 - lon1) * D2R;
const y = Math.sin(dl) * Math.cos(p2);
const x = Math.cos(p1) * Math.sin(p2) - Math.sin(p1) * Math.cos(p2) * Math.cos(dl);
return (Math.atan2(y, x) * R2D + 360) % 360;
}
// Subsolar point (lon, lat) for a JS Date — NOAA low-precision solar position.
export function subsolar(date) {
const jd = date.getTime() / 86400000 + 2440587.5;
const n = jd - 2451545.0;
const L = (280.46 + 0.9856474 * n) % 360;
const g = ((357.528 + 0.9856003 * n) % 360) * D2R;
const lambda = (L + 1.915 * Math.sin(g) + 0.02 * Math.sin(2 * g)) * D2R;
const eps = (23.439 - 0.0000004 * n) * D2R;
const decl = Math.asin(Math.sin(eps) * Math.sin(lambda));
const ra = Math.atan2(Math.cos(eps) * Math.sin(lambda), Math.cos(lambda));
const gmst = (280.46061837 + 360.98564736629 * n) % 360;
let lon = ra * R2D - gmst;
lon = ((lon + 540) % 360) - 180;
return [lon, decl * R2D];
}
export function fmtLat(lat, dp = 4) {
return `${Math.abs(lat).toFixed(dp)}°${lat >= 0 ? 'N' : 'S'}`;
}
export function fmtLon(lon, dp = 4) {
return `${Math.abs(lon).toFixed(dp)}°${lon >= 0 ? 'E' : 'W'}`;
}
+296
View File
@@ -0,0 +1,296 @@
import * as THREE from 'three';
// The planet: one shader that walks the surface through every state the film
// needs — dark substrate, the growth front lighting up night cities, quadtree
// tiles indexing in with rising resolution, then the lit, living Earth.
const COMMON = /* glsl */ `
#define PI 3.141592653589793
float hash3(vec3 p) {
p = fract(p * 0.3183099 + vec3(0.71, 0.113, 0.419));
p *= 17.0;
return fract(p.x * p.y * p.z * (p.x + p.y + p.z));
}
vec3 llDir(float lon, float lat) {
float c = cos(lat);
return vec3(c * cos(lon), sin(lat), -c * sin(lon));
}
// u wraps 1 -> 0 at the antimeridian; take derivatives from whichever of u or
// u+0.5 is continuous here so mip selection doesn't spike into a seam line
void seamGrad(vec2 uv, out vec2 gx, out vec2 gy) {
vec2 a = uv;
vec2 b = vec2(fract(uv.x + 0.5), uv.y);
vec2 ax = dFdx(a), ay = dFdy(a), bx = dFdx(b), by = dFdy(b);
gx = vec2(abs(ax.x) < abs(bx.x) ? ax.x : bx.x, ax.y);
gy = vec2(abs(ay.x) < abs(by.x) ? ay.x : by.x, ay.y);
}
`;
export function createGlobe({ dayTex, nightTex, cloudTex }) {
const uniforms = {
uDay: { value: dayTex },
uNight: { value: nightTex },
uTime: { value: 0 },
uSeed: { value: new THREE.Vector3(1, 0, 0) },
uSun: { value: new THREE.Vector3(1, 0, 0) },
uFront: { value: -1 }, // growth front, radians from the seed
uHolo: { value: 0 }, // substrate grid visibility
uLights: { value: 0 }, // night-light "neurons" during growth
uWave: { value: new THREE.Vector3(1e3, 1e3, 1e3) },
uWaveSpread: { value: 2.6 },
uImagery: { value: 0 }, // forces imagery fully on (after indexing)
uLit: { value: 0 }, // 0 = flat imagery, 1 = real sun lighting
uTexel: { value: new THREE.Vector2(1 / 8192, 1 / 4096) },
uGridAmt: { value: 1 },
uTint: { value: 0 }, // cyan analysis tint on the whole surface
uDim: { value: 1 },
};
const mat = new THREE.ShaderMaterial({
uniforms,
vertexShader: /* glsl */ `
varying vec3 vObj;
varying vec3 vWorldN;
varying vec3 vView;
void main() {
vObj = position;
vec4 wp = modelMatrix * vec4(position, 1.0);
vWorldN = normalize(mat3(modelMatrix) * position);
vView = normalize(cameraPosition - wp.xyz);
gl_Position = projectionMatrix * viewMatrix * wp;
}
`,
fragmentShader: /* glsl */ `
${COMMON}
uniform sampler2D uDay, uNight;
uniform float uTime, uFront, uHolo, uLights, uWaveSpread, uImagery, uLit, uGridAmt, uTint, uDim;
uniform vec3 uSeed, uSun, uWave;
uniform vec2 uTexel;
varying vec3 vObj;
varying vec3 vWorldN;
varying vec3 vView;
float mercY(float lat) {
lat = clamp(lat, -1.4844, 1.4844);
return 0.5 - log(tan(PI * 0.25 + lat * 0.5)) / (2.0 * PI);
}
float invMercY(float my) {
return atan(sinh(PI * (1.0 - 2.0 * my)));
}
// Per-level quadtree state at this fragment.
// returns x = phase since this tile revealed, y = edge line (0..1)
vec2 tileState(float z, float waveT, vec2 uv, float my, float jitter) {
float n = exp2(z);
vec2 tc = vec2(uv.x * n, my * n);
vec2 ti = floor(tc);
vec2 f = fract(tc);
float lonc = ((ti.x + 0.5) / n) * 2.0 * PI - PI;
float latc = invMercY((ti.y + 0.5) / n);
float d = acos(clamp(dot(uSeed, llDir(lonc, latc)), -1.0, 1.0));
float tRev = waveT + d / PI * uWaveSpread + hash3(vec3(ti, z)) * jitter;
vec2 tcb = vec2(fract(uv.x + 0.5) * n, tc.y);
vec2 w = vec2(min(fwidth(tc.x), fwidth(tcb.x)), fwidth(tc.y));
vec2 e = min(f, 1.0 - f) / max(w, vec2(1e-5));
float line = 1.0 - smoothstep(0.0, 1.25, min(e.x, e.y));
return vec2(uTime - tRev, line);
}
void main() {
vec3 n = normalize(vObj);
float lat = asin(clamp(n.y, -1.0, 1.0));
float lon = atan(-n.z, n.x);
vec2 uv = vec2(lon / (2.0 * PI) + 0.5, lat / PI + 0.5);
float my = mercY(lat);
vec3 N = normalize(vWorldN);
vec3 V = normalize(vView);
float dSeed = acos(clamp(dot(n, uSeed), -1.0, 1.0));
// ---------- substrate (pre-imagery)
float gLat = abs(fract(lat * 180.0 / PI / 10.0 + 0.5) - 0.5) * 10.0;
float gLon = abs(fract(lon * 180.0 / PI / 10.0 + 0.5) - 0.5) * 10.0 * cos(lat);
float gw = fwidth(lat * 180.0 / PI) * 1.2;
float gwl = min(fwidth(lon), fwidth(mod(lon + 2.0 * PI, 2.0 * PI))) * 180.0 / PI * 1.2 * cos(lat);
float grid = max(1.0 - smoothstep(0.0, gw, gLat), 1.0 - smoothstep(0.0, max(gwl, 1e-4), gLon));
float behind = smoothstep(uFront + 0.02, uFront - 0.10, dSeed);
vec3 base = vec3(0.0018, 0.0035, 0.006) * behind;
vec3 holo = base + vec3(0.0, 0.35, 0.42) * grid * 0.10 * uHolo * behind;
// growth front: a thin bright ring with a soft wake
float ring = exp(-pow((dSeed - uFront) * 55.0, 2.0)) * step(0.0, uFront);
float wake = exp(-max(uFront - dSeed, 0.0) * 6.0) * behind;
holo += vec3(0.1, 0.85, 1.0) * (ring * 0.55 + wake * 0.035) * uHolo;
// real night lights, colonised by the front: the "neurons"
vec2 gx, gy;
seamGrad(uv, gx, gy);
float nl = textureGrad(uNight, uv, gx, gy).r;
float nlc = pow(max(nl - 0.06, 0.0) * 1.07, 1.7);
float fresh = exp(-max(uFront - dSeed, 0.0) * 3.0);
holo += mix(vec3(0.55, 0.95, 1.0), vec3(1.0), fresh) * nlc * (1.2 + fresh * 2.5) * behind * uLights;
// ---------- quadtree indexing
vec2 s3 = tileState(3.0, uWave.x, uv, my, 0.35);
vec2 s4 = tileState(4.0, uWave.y, uv, my, 0.30);
vec2 s5 = tileState(5.0, uWave.z, uv, my, 0.25);
float reveal = max(smoothstep(0.05, 0.42, s3.x), uImagery);
// resolution climbs as each finer wave passes
float bias = mix(4.5, 0.0, max(smoothstep(0.0, 0.5, s5.x), uImagery));
bias = min(bias, mix(4.5, 2.2, smoothstep(0.0, 0.5, s4.x)));
float bs = exp2(bias);
vec2 bgx = gx * bs, bgy = gy * bs;
vec3 day = textureGrad(uDay, uv, bgx, bgy).rgb;
// analysis edges (luminance gradient) — only while a tile is being indexed
float ph0 = s3.x;
float edgeAmt0 = step(0.0, ph0) * exp(-max(ph0 - 0.1, 0.0) * 2.2) * (1.0 - uImagery);
float edge = 0.0;
if (edgeAmt0 > 0.002) {
float l0 = dot(textureGrad(uDay, uv + vec2(uTexel.x, 0.0), bgx, bgy).rgb, vec3(0.3, 0.59, 0.11));
float l1 = dot(textureGrad(uDay, uv - vec2(uTexel.x, 0.0), bgx, bgy).rgb, vec3(0.3, 0.59, 0.11));
float l2 = dot(textureGrad(uDay, uv + vec2(0.0, uTexel.y), bgx, bgy).rgb, vec3(0.3, 0.59, 0.11));
float l3 = dot(textureGrad(uDay, uv - vec2(0.0, uTexel.y), bgx, bgy).rgb, vec3(0.3, 0.59, 0.11));
edge = clamp(length(vec2(l0 - l1, l2 - l3)) * 9.0, 0.0, 1.0);
}
// ---------- lighting
float sd = dot(n, uSun);
float dayF = smoothstep(-0.10, 0.20, sd);
vec3 lit = day * (0.035 + 1.25 * pow(max(sd, 0.0), 0.75)) * dayF + day * 0.012;
float ocean = smoothstep(0.015, 0.06, day.b - day.r) * (1.0 - smoothstep(0.08, 0.25, dot(day, vec3(0.33))));
vec3 Hh = normalize(normalize(mat3(1.0) * uSun) + V);
float graze = smoothstep(0.15, 0.5, dot(N, V));
float spec = (pow(max(dot(N, Hh), 0.0), 240.0) * 0.4 + pow(max(dot(N, Hh), 0.0), 30.0) * 0.03) * ocean * dayF * graze;
vec3 cityGlow = vec3(1.0, 0.72, 0.42) * pow(nl, 1.9) * 1.6 * (1.0 - dayF);
float fres = pow(1.0 - max(dot(N, V), 0.0), 3.0);
vec3 haze = vec3(0.25, 0.5, 1.0) * fres * (0.15 + 0.85 * dayF) * 0.7;
vec3 earthLit = lit + vec3(1.0, 0.92, 0.8) * spec + cityGlow + haze;
vec3 earthFlat = day * 0.6;
vec3 earth = mix(earthFlat, earthLit, uLit);
// reveal treatment: flash, edges, then settle
float ph = s3.x;
float flash = step(0.0, ph) * exp(-max(ph, 0.0) * 9.0);
float edgeAmt = step(0.0, ph) * exp(-max(ph - 0.1, 0.0) * 2.2) * (1.0 - uImagery);
vec3 surf = mix(holo, earth, reveal);
surf += vec3(0.0, 0.8, 1.0) * edge * edgeAmt * 0.9;
surf += vec3(0.6, 0.95, 1.0) * flash * 0.10 * (1.0 - uImagery);
surf = mix(surf, vec3(dot(surf, vec3(0.3, 0.59, 0.11))) * vec3(0.4, 1.0, 1.15), uTint);
// tile edges: faint before, bright at their own wave, then gone
float g3 = s3.y * (step(ph, 0.0) * 0.18 * uHolo * behind + step(0.0, ph) * exp(-max(ph, 0.0) * 1.6) * 1.3);
float g4 = s4.y * step(0.0, s4.x) * exp(-max(s4.x, 0.0) * 2.0) * 0.9;
float g5 = s5.y * step(0.0, s5.x) * exp(-max(s5.x, 0.0) * 2.6) * 0.7;
surf += vec3(0.25, 0.9, 1.0) * max(g3, max(g4, g5)) * uGridAmt;
gl_FragColor = vec4(surf * uDim, 1.0);
}
`,
});
const geo = new THREE.SphereGeometry(1, 384, 192);
const mesh = new THREE.Mesh(geo, mat);
// Clouds — real GIBS-derived opacity, lit by the same sun.
const cloudU = {
uClouds: { value: cloudTex },
uSun: uniforms.uSun,
uAmt: { value: 0 },
uTime: uniforms.uTime,
};
const cloudMat = new THREE.ShaderMaterial({
uniforms: cloudU,
transparent: true,
depthWrite: false,
vertexShader: /* glsl */ `
varying vec3 vObj;
varying vec3 vN;
varying vec3 vView;
void main() {
vObj = position;
vec4 wp = modelMatrix * vec4(position, 1.0);
vN = normalize(mat3(modelMatrix) * position);
vView = normalize(cameraPosition - wp.xyz);
gl_Position = projectionMatrix * viewMatrix * wp;
}
`,
fragmentShader: /* glsl */ `
#define PI 3.141592653589793
uniform sampler2D uClouds;
uniform vec3 uSun;
uniform float uAmt, uTime;
varying vec3 vObj;
varying vec3 vN;
varying vec3 vView;
void main() {
vec3 n = normalize(vObj);
float lat = asin(clamp(n.y, -1.0, 1.0));
float lon = atan(-n.z, n.x) - uTime * 0.0015;
vec2 uv = vec2(fract(lon / (2.0 * PI) + 0.5), lat / PI + 0.5);
vec2 ua = uv, ub = vec2(fract(uv.x + 0.5), uv.y);
vec2 ax = dFdx(ua), ay = dFdy(ua), bx = dFdx(ub), by = dFdy(ub);
vec2 gx = vec2(abs(ax.x) < abs(bx.x) ? ax.x : bx.x, ax.y);
vec2 gy = vec2(abs(ay.x) < abs(by.x) ? ay.x : by.x, ay.y);
float c = textureGrad(uClouds, uv, gx, gy).r;
c = smoothstep(0.08, 0.85, c);
float sd = dot(n, uSun);
float dayF = smoothstep(-0.12, 0.25, sd);
vec3 col = vec3(1.0, 0.98, 0.95) * (0.02 + 1.1 * dayF * (0.35 + 0.65 * max(sd, 0.0)));
float edge = pow(1.0 - max(dot(normalize(vN), normalize(vView)), 0.0), 2.0);
gl_FragColor = vec4(col, c * uAmt * (0.92 - edge * 0.4));
}
`,
});
const clouds = new THREE.Mesh(new THREE.SphereGeometry(1.0065, 256, 128), cloudMat);
// Atmosphere shell — limb glow, warm-to-blue at the terminator.
const atmoU = { uSun: uniforms.uSun, uAmt: { value: 0 }, uCyan: { value: 0 } };
const atmoMat = new THREE.ShaderMaterial({
uniforms: atmoU,
side: THREE.BackSide,
transparent: true,
depthWrite: false,
blending: THREE.AdditiveBlending,
vertexShader: /* glsl */ `
varying vec3 vN;
varying vec3 vWP;
void main() {
vec4 wp = modelMatrix * vec4(position, 1.0);
vWP = wp.xyz;
vN = normalize(mat3(modelMatrix) * position);
gl_Position = projectionMatrix * viewMatrix * wp;
}
`,
fragmentShader: /* glsl */ `
uniform vec3 uSun;
uniform float uAmt, uCyan;
varying vec3 vN;
varying vec3 vWP;
void main() {
vec3 V = normalize(cameraPosition - vWP);
vec3 N = normalize(vN);
// distance of this view ray from the planet centre, in radii
vec3 ro = cameraPosition;
vec3 rd = -V;
float b = dot(ro, rd);
float h = length(ro - b * rd);
float shell = 1.065;
float t = clamp((shell - h) / (shell - 1.0), 0.0, 1.0);
float glow = pow(t, 2.6) * (1.0 - smoothstep(0.985, 1.0, t) * 0.0);
if (h < 1.0) glow *= 0.0;
// light side vs dark side of the limb
vec3 limbP = normalize(ro + rd * b);
float sd = dot(limbP, uSun);
float lit = smoothstep(-0.35, 0.4, sd);
vec3 col = mix(vec3(0.02, 0.08, 0.14), vec3(0.35, 0.62, 1.0), lit);
col += vec3(1.0, 0.45, 0.2) * exp(-pow(sd * 6.0, 2.0)) * 0.35;
col = mix(col, vec3(0.1, 0.9, 1.0) * 0.8, uCyan);
gl_FragColor = vec4(col * glow * uAmt * 1.05, 1.0);
}
`,
});
const atmo = new THREE.Mesh(new THREE.SphereGeometry(1.065, 128, 64), atmoMat);
const group = new THREE.Group();
group.add(mesh, clouds, atmo);
return { group, mesh, uniforms, cloudU, atmoU };
}
File diff suppressed because it is too large. Load diff
+512
View File
@@ -0,0 +1,512 @@
import * as THREE from 'three';
import { llToVec, D2R, EARTH_KM } from './geo.js';
import { mulberry32 } from './timeline.js';
// Point and particle layers for the live feeds. Every layer exposes
// `update(t, ctx)`; reveal timing and motion are pure functions of film time.
const GC = /* glsl */ `
vec3 llDir(float lon, float lat) {
float c = cos(lat);
return vec3(c * cos(lon), sin(lat), -c * sin(lon));
}
// great-circle destination: start (lon,lat) rad, bearing rad, distance rad
vec2 gcDest(vec2 ll, float b, float d) {
float lat2 = asin(sin(ll.y) * cos(d) + cos(ll.y) * sin(d) * cos(b));
float lon2 = ll.x + atan(sin(b) * sin(d) * cos(ll.y), cos(d) - sin(ll.y) * sin(lat2));
return vec2(lon2, lat2);
}
`;
// Moving contacts (aircraft, vessels): heads as points, trails as segments.
// rows: [lon, lat, alt_m, track_deg, vel_ms]
export function createMovers(rows, { color, trailColor, size = 2.2, altScale = 8, trail = 0.010, revealSpread = 1.4, seed = 1, gain = 1 }) {
const rnd = mulberry32(seed);
const n = rows.length;
const ll = new Float32Array(n * 2);
const hv = new Float32Array(n * 2);
const alt = new Float32Array(n);
const rr = new Float32Array(n);
rows.forEach((r, i) => {
ll[i * 2] = r[0] * D2R;
ll[i * 2 + 1] = r[1] * D2R;
hv[i * 2] = (r[3] || 0) * D2R;
hv[i * 2 + 1] = (r[4] || 0) / (EARTH_KM * 1000); // rad per second
alt[i] = 1 + ((r[2] || 0) / (EARTH_KM * 1000)) * altScale + 0.0015;
rr[i] = rnd();
});
const uniforms = {
uSim: { value: 0 }, // simulated seconds since the feed went live
uReveal: { value: 0 }, // 0..1 + spread
uAlpha: { value: 0 },
uGain: { value: gain },
uPx: { value: 1 },
uSize: { value: size },
uTrail: { value: trail },
uColor: { value: new THREE.Color(color) },
uTrailColor: { value: new THREE.Color(trailColor || color) },
uSpread: { value: revealSpread },
};
const vs = (isTrail) => /* glsl */ `
${GC}
attribute vec2 aLL;
attribute vec2 aHV;
attribute float aAlt;
attribute float aR;
${isTrail ? 'attribute float aEnd;' : ''}
uniform float uSim, uReveal, uPx, uSize, uTrail, uSpread;
varying float vA;
varying float vFacing;
void main() {
float d = aHV.y * uSim;
${isTrail ? 'd -= aEnd * max(uTrail, aHV.y * 40.0);' : ''}
vec2 p = gcDest(aLL, aHV.x, d);
vec3 pos = llDir(p.x, p.y) * aAlt;
vec4 wp = modelMatrix * vec4(pos, 1.0);
vFacing = dot(normalize(wp.xyz), normalize(cameraPosition - wp.xyz));
float rv = clamp((uReveal - aR * uSpread) * 6.0, 0.0, 1.0);
vA = rv ${isTrail ? '* (1.0 - aEnd)' : ''};
gl_Position = projectionMatrix * viewMatrix * wp;
${isTrail ? '' : 'gl_PointSize = uSize * uPx * (1.0 + 2.5 * exp(-(uReveal - aR * uSpread) * 10.0) * step(0.0, uReveal - aR * uSpread));'}
}
`;
const fs = (isTrail) => /* glsl */ `
uniform vec3 ${isTrail ? 'uTrailColor' : 'uColor'};
uniform float uAlpha, uGain;
varying float vA;
varying float vFacing;
void main() {
float back = smoothstep(-0.02, 0.12, vFacing);
${
isTrail
? 'gl_FragColor = vec4(uTrailColor * vA * uAlpha * uGain * back * 0.9, 1.0);'
: `vec2 c = gl_PointCoord - 0.5;
float r = length(c);
float disc = smoothstep(0.5, 0.15, r);
if (vA * back <= 0.001) discard;
gl_FragColor = vec4(uColor * disc * vA * uAlpha * uGain * back, 1.0);`
}
}
`;
const mk = (isTrail) =>
new THREE.ShaderMaterial({
uniforms,
transparent: true,
depthWrite: false,
blending: THREE.AdditiveBlending,
vertexShader: vs(isTrail),
fragmentShader: fs(isTrail),
});
const gp = new THREE.BufferGeometry();
gp.setAttribute('position', new THREE.BufferAttribute(new Float32Array(n * 3), 3));
gp.setAttribute('aLL', new THREE.BufferAttribute(ll, 2));
gp.setAttribute('aHV', new THREE.BufferAttribute(hv, 2));
gp.setAttribute('aAlt', new THREE.BufferAttribute(alt, 1));
gp.setAttribute('aR', new THREE.BufferAttribute(rr, 1));
gp.boundingSphere = new THREE.Sphere(new THREE.Vector3(), 2);
const heads = new THREE.Points(gp, mk(false));
// trails: two vertices per contact, aEnd 0 (head) / 1 (tail)
const dup = (src, k) => {
const out = new Float32Array(src.length * 2);
for (let i = 0; i < src.length / k; i++)
for (let j = 0; j < k; j++) {
out[i * 2 * k + j] = src[i * k + j];
out[i * 2 * k + k + j] = src[i * k + j];
}
return out;
};
const gt = new THREE.BufferGeometry();
gt.setAttribute('position', new THREE.BufferAttribute(new Float32Array(n * 6), 3));
gt.setAttribute('aLL', new THREE.BufferAttribute(dup(ll, 2), 2));
gt.setAttribute('aHV', new THREE.BufferAttribute(dup(hv, 2), 2));
gt.setAttribute('aAlt', new THREE.BufferAttribute(dup(alt, 1), 1));
gt.setAttribute('aR', new THREE.BufferAttribute(dup(rr, 1), 1));
const ends = new Float32Array(n * 2);
for (let i = 0; i < n; i++) ends[i * 2 + 1] = 1;
gt.setAttribute('aEnd', new THREE.BufferAttribute(ends, 1));
gt.boundingSphere = new THREE.Sphere(new THREE.Vector3(), 2);
const trails = new THREE.LineSegments(gt, mk(true));
const group = new THREE.Group();
group.add(trails, heads);
return { group, uniforms, count: n };
}
// Static or pulsing point events (fires, quakes, cities, cameras).
// rows: [lon, lat, weight]
export function createEvents(rows, { color, size = 3, mode = 'flicker', r = 1.002, spread = 1.2, seed = 3, gain = 1, weightFn = (w) => w }) {
const rnd = mulberry32(seed);
const n = rows.length;
const pos = new Float32Array(n * 3);
const w = new Float32Array(n);
const rr = new Float32Array(n);
const v = [0, 0, 0];
rows.forEach((row, i) => {
llToVec(row[0], row[1], r, v);
pos.set(v, i * 3);
w[i] = weightFn(row[2] ?? 1);
rr[i] = rnd();
});
const uniforms = {
uT: { value: 0 },
uReveal: { value: 0 },
uAlpha: { value: 0 },
uGain: { value: gain },
uPx: { value: 1 },
uSize: { value: size },
uColor: { value: new THREE.Color(color) },
uSpread: { value: spread },
};
const g = new THREE.BufferGeometry();
g.setAttribute('position', new THREE.BufferAttribute(pos, 3));
g.setAttribute('aW', new THREE.BufferAttribute(w, 1));
g.setAttribute('aR', new THREE.BufferAttribute(rr, 1));
const ringMode = mode === 'ring';
const m = new THREE.ShaderMaterial({
uniforms,
transparent: true,
depthWrite: false,
blending: THREE.AdditiveBlending,
vertexShader: /* glsl */ `
attribute float aW;
attribute float aR;
uniform float uT, uReveal, uPx, uSize, uSpread;
varying float vA;
varying float vPh;
varying float vW;
varying float vFacing;
void main() {
vec4 wp = modelMatrix * vec4(position, 1.0);
vFacing = dot(normalize(wp.xyz), normalize(cameraPosition - wp.xyz));
float k = uReveal - aR * uSpread;
vA = clamp(k * 5.0, 0.0, 1.0);
vPh = k;
vW = aW;
gl_Position = projectionMatrix * viewMatrix * wp;
${
ringMode
? 'gl_PointSize = uSize * uPx * (0.6 + aW) * 9.0;'
: 'gl_PointSize = uSize * uPx * (0.7 + aW) * (1.0 + 3.0 * exp(-max(k, 0.0) * 8.0) * step(0.0, k));'
}
}
`,
fragmentShader: /* glsl */ `
uniform vec3 uColor;
uniform float uT, uAlpha, uGain;
varying float vA;
varying float vPh;
varying float vW;
varying float vFacing;
void main() {
float back = smoothstep(-0.02, 0.12, vFacing);
if (vA * back <= 0.001) discard;
vec2 c = gl_PointCoord - 0.5;
float r = length(c) * 2.0;
${
ringMode
? `float ph = fract(vPh * 0.55 + vW * 0.37);
float ringR = ph;
float ring = exp(-pow((r - ringR) * 14.0, 2.0)) * (1.0 - ph);
float core = smoothstep(0.16, 0.0, r);
float first = exp(-max(vPh, 0.0) * 3.0);
float a = ring * (0.8 + first) + core * 1.4;`
: `float flick = 0.65 + 0.35 * sin(uT * (9.0 + vW * 7.0) + vW * 91.0);
float a = smoothstep(1.0, 0.0, r) * smoothstep(1.0, 0.25, r) * 1.5 * flick;`
}
gl_FragColor = vec4(uColor * a * vA * uAlpha * uGain * back, 1.0);
}
`,
});
const pts = new THREE.Points(g, m);
return { group: pts, uniforms, count: n };
}
// Satellites: positions arrive from satellite.js each frame.
export function createSatellites(count, { color = '#d6d0ff', size = 1.7 } = {}) {
const pos = new Float32Array(count * 3);
const rr = new Float32Array(count);
const kind = new Float32Array(count);
const rnd = mulberry32(11);
for (let i = 0; i < count; i++) rr[i] = rnd();
const g = new THREE.BufferGeometry();
const pa = new THREE.BufferAttribute(pos, 3);
pa.setUsage(THREE.DynamicDrawUsage);
g.setAttribute('position', pa);
g.setAttribute('aR', new THREE.BufferAttribute(rr, 1));
g.setAttribute('aK', new THREE.BufferAttribute(kind, 1));
g.boundingSphere = new THREE.Sphere(new THREE.Vector3(), 20);
const uniforms = {
uReveal: { value: 0 },
uAlpha: { value: 0 },
uPx: { value: 1 },
uSize: { value: size },
uColor: { value: new THREE.Color(color) },
};
const m = new THREE.ShaderMaterial({
uniforms,
transparent: true,
depthWrite: false,
depthTest: true,
blending: THREE.AdditiveBlending,
vertexShader: /* glsl */ `
attribute float aR;
attribute float aK;
uniform float uReveal, uPx, uSize;
varying float vA;
varying float vK;
void main() {
float k = uReveal - aR * 1.3;
vA = clamp(k * 5.0, 0.0, 1.0);
vK = aK;
gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
float big = aK > 0.5 ? 1.6 : 1.0;
gl_PointSize = uSize * uPx * big * (1.0 + 3.0 * exp(-max(k, 0.0) * 9.0) * step(0.0, k));
}
`,
fragmentShader: /* glsl */ `
uniform vec3 uColor;
uniform float uAlpha;
varying float vA;
varying float vK;
void main() {
if (vA <= 0.001) discard;
float r = length(gl_PointCoord - 0.5) * 2.0;
float a = smoothstep(1.0, 0.1, r);
vec3 c = mix(uColor * 0.75, vec3(0.75, 1.0, 1.0) * 1.4, step(0.5, vK));
gl_FragColor = vec4(c * a * vA * uAlpha, 1.0);
}
`,
});
const pts = new THREE.Points(g, m);
return { group: pts, uniforms, positions: pos, attr: pa, kind, count };
}
// Orbit rings for a handful of notable objects.
export function createOrbitRings(paths, color = '#bfb6ff') {
const pos = [];
const u = [];
for (const path of paths) {
for (let i = 0; i < path.length - 1; i++) {
pos.push(...path[i], ...path[i + 1]);
u.push(i / (path.length - 1), (i + 1) / (path.length - 1));
}
}
const g = new THREE.BufferGeometry();
g.setAttribute('position', new THREE.Float32BufferAttribute(pos, 3));
g.setAttribute('aU', new THREE.Float32BufferAttribute(u, 1));
const m = new THREE.ShaderMaterial({
transparent: true,
depthWrite: false,
blending: THREE.AdditiveBlending,
uniforms: { uDraw: { value: 0 }, uAlpha: { value: 0 }, uColor: { value: new THREE.Color(color) } },
vertexShader: /* glsl */ `
attribute float aU;
varying float vU;
varying float vNear;
void main() {
vU = aU;
vec4 mv = modelViewMatrix * vec4(position, 1.0);
vNear = smoothstep(1.2, 4.5, -mv.z);
gl_Position = projectionMatrix * mv;
}
`,
fragmentShader: /* glsl */ `
uniform float uDraw, uAlpha;
uniform vec3 uColor;
varying float vU;
varying float vNear;
void main() {
float k = uDraw - vU;
if (k < 0.0) discard;
float hot = exp(-k * 30.0) * 2.0;
gl_FragColor = vec4((uColor * 0.4 + vec3(1.0) * hot) * uAlpha * vNear, 1.0);
}
`,
});
return new THREE.LineSegments(g, m);
}
// Wind: particles advected through a real GFS u/v field, drawn as fading trails.
export function createWind(field, { count = 7000, trailLen = 14, seed = 21 } = {}) {
const { nx, ny, lo1, la1, dx, dy, u, v } = field;
const sample = (lon, lat) => {
let x = (((lon - lo1) % 360) + 360) % 360 / dx;
let y = (la1 - lat) / dy;
const x0 = Math.floor(x);
const y0 = Math.max(0, Math.min(ny - 2, Math.floor(y)));
const fx = x - x0;
const fy = Math.max(0, Math.min(1, y - y0));
const i00 = y0 * nx + (x0 % nx);
const i10 = y0 * nx + ((x0 + 1) % nx);
const i01 = (y0 + 1) * nx + (x0 % nx);
const i11 = (y0 + 1) * nx + ((x0 + 1) % nx);
const bl = (arr) => (arr[i00] * (1 - fx) + arr[i10] * fx) * (1 - fy) + (arr[i01] * (1 - fx) + arr[i11] * fx) * fy;
return [bl(u), bl(v)];
};
const SEGS = trailLen - 1;
const pos = new Float32Array(count * SEGS * 2 * 3);
const alpha = new Float32Array(count * SEGS * 2);
const g = new THREE.BufferGeometry();
const pa = new THREE.BufferAttribute(pos, 3);
pa.setUsage(THREE.DynamicDrawUsage);
const aa = new THREE.BufferAttribute(alpha, 1);
aa.setUsage(THREE.DynamicDrawUsage);
g.setAttribute('position', pa);
g.setAttribute('aA', aa);
g.boundingSphere = new THREE.Sphere(new THREE.Vector3(), 2);
const uniforms = { uAlpha: { value: 0 }, uColor: { value: new THREE.Color('#a6dcff') } };
const m = new THREE.ShaderMaterial({
uniforms,
transparent: true,
depthWrite: false,
blending: THREE.AdditiveBlending,
vertexShader: /* glsl */ `
attribute float aA;
varying float vA;
varying float vFacing;
void main() {
vA = aA;
vec4 wp = modelMatrix * vec4(position, 1.0);
vFacing = dot(normalize(wp.xyz), normalize(cameraPosition - wp.xyz));
gl_Position = projectionMatrix * viewMatrix * wp;
}
`,
fragmentShader: /* glsl */ `
uniform vec3 uColor;
uniform float uAlpha;
varying float vA;
varying float vFacing;
void main() {
float back = smoothstep(0.0, 0.15, vFacing);
gl_FragColor = vec4(uColor * vA * uAlpha * back, 1.0);
}
`,
});
const lines = new THREE.LineSegments(g, m);
// particle state
const P = { lon: new Float32Array(count), lat: new Float32Array(count), age: new Float32Array(count), life: new Float32Array(count), spd: new Float32Array(count) };
const hist = new Float32Array(count * trailLen * 2);
let rnd;
let simT = -1;
const spawn = (i) => {
const lon = rnd() * 360 - 180;
const lat = Math.asin(rnd() * 2 - 1) * (180 / Math.PI) * 0.92;
P.lon[i] = lon;
P.lat[i] = lat;
P.age[i] = 0;
P.life[i] = 1.2 + rnd() * 2.2;
for (let k = 0; k < trailLen; k++) {
hist[(i * trailLen + k) * 2] = lon;
hist[(i * trailLen + k) * 2 + 1] = lat;
}
};
const reset = () => {
rnd = mulberry32(seed);
for (let i = 0; i < count; i++) {
spawn(i);
P.age[i] = rnd() * P.life[i];
}
simT = 0;
};
const DT = 1 / 30;
const K = 0.85; // deg per (m/s) per second of film — visual speed
const step = () => {
for (let i = 0; i < count; i++) {
const [uu, vv] = sample(P.lon[i], P.lat[i]);
P.spd[i] = Math.hypot(uu, vv);
const c = Math.max(0.2, Math.cos(P.lat[i] * D2R));
P.lon[i] += (uu * K * DT) / c;
P.lat[i] += vv * K * DT;
if (P.lon[i] > 180) P.lon[i] -= 360;
if (P.lon[i] < -180) P.lon[i] += 360;
P.age[i] += DT;
const base = i * trailLen * 2;
hist.copyWithin(base + 2, base, base + (trailLen - 1) * 2);
hist[base] = P.lon[i];
hist[base + 1] = P.lat[i];
if (P.age[i] > P.life[i] || Math.abs(P.lat[i]) > 84) spawn(i);
}
simT += DT;
};
const a = [0, 0, 0];
const b = [0, 0, 0];
function update(tSinceStart) {
if (simT < 0 || tSinceStart < simT - 1e-6) reset();
while (simT < tSinceStart) step();
let o = 0;
let q = 0;
for (let i = 0; i < count; i++) {
const base = i * trailLen * 2;
const life = P.age[i] / P.life[i];
const fade = Math.min(1, life * 5) * Math.min(1, (1 - life) * 4);
const sp = Math.min(1, P.spd[i] / 18);
for (let k = 0; k < SEGS; k++) {
const lo1 = hist[base + k * 2];
const la1 = hist[base + k * 2 + 1];
const lo2 = hist[base + (k + 1) * 2];
const la2 = hist[base + (k + 1) * 2 + 1];
const jump = Math.abs(lo2 - lo1) > 90;
llToVec(lo1, la1, 1.004, a);
llToVec(jump ? lo1 : lo2, jump ? la1 : la2, 1.004, b);
pos[o++] = a[0]; pos[o++] = a[1]; pos[o++] = a[2];
pos[o++] = b[0]; pos[o++] = b[1]; pos[o++] = b[2];
const f1 = (1 - k / SEGS) * fade * (0.25 + sp);
const f2 = (1 - (k + 1) / SEGS) * fade * (0.25 + sp);
alpha[q++] = f1;
alpha[q++] = f2;
}
}
pa.needsUpdate = true;
aa.needsUpdate = true;
}
return { group: lines, uniforms, update, count };
}
export function createStars(n = 2600, seed = 77) {
const rnd = mulberry32(seed);
const pos = new Float32Array(n * 3);
const mag = new Float32Array(n);
for (let i = 0; i < n; i++) {
const z = rnd() * 2 - 1;
const t = rnd() * Math.PI * 2;
const s = Math.sqrt(1 - z * z);
pos[i * 3] = 400 * s * Math.cos(t);
pos[i * 3 + 1] = 400 * z;
pos[i * 3 + 2] = 400 * s * Math.sin(t);
mag[i] = Math.pow(rnd(), 6);
}
const g = new THREE.BufferGeometry();
g.setAttribute('position', new THREE.BufferAttribute(pos, 3));
g.setAttribute('aM', new THREE.BufferAttribute(mag, 1));
const uniforms = { uAlpha: { value: 0 }, uPx: { value: 1 } };
const m = new THREE.ShaderMaterial({
uniforms,
transparent: true,
depthWrite: false,
blending: THREE.AdditiveBlending,
vertexShader: /* glsl */ `
attribute float aM;
uniform float uPx;
varying float vM;
void main() {
vM = aM;
gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
gl_PointSize = (1.0 + aM * 2.2) * uPx;
}
`,
fragmentShader: /* glsl */ `
uniform float uAlpha;
varying float vM;
void main() {
float r = length(gl_PointCoord - 0.5) * 2.0;
float a = smoothstep(1.0, 0.0, r);
gl_FragColor = vec4(vec3(0.8, 0.9, 1.0) * a * (0.12 + vM * 0.9) * uAlpha, 1.0);
}
`,
});
return { group: new THREE.Points(g, m), uniforms };
}
+323
View File
@@ -0,0 +1,323 @@
import * as THREE from 'three';
import { llToVec, angDist, destination, bearing, D2R } from './geo.js';
import { mulberry32 } from './timeline.js';
// Line layers that reveal along a scalar "arrival" attribute (radians from the
// seed, or arc length), with a hot tip at the front and a configurable wake.
export function frontLineMaterial({ color, alpha = 1, after = 0.25, tip = 3.0, decay = 4.0, additive = true }) {
return new THREE.ShaderMaterial({
transparent: true,
depthWrite: false,
depthTest: true,
blending: additive ? THREE.AdditiveBlending : THREE.NormalBlending,
uniforms: {
uColor: { value: new THREE.Color(color) },
uFront: { value: -1 },
uAlpha: { value: alpha },
uAfter: { value: after },
uTip: { value: tip },
uDecay: { value: decay },
uFadeBack: { value: 1 },
},
vertexShader: /* glsl */ `
attribute float aDist;
varying float vDist;
varying float vFacing;
void main() {
vDist = aDist;
vec4 wp = modelMatrix * vec4(position, 1.0);
vec3 n = normalize(wp.xyz);
vFacing = dot(n, normalize(cameraPosition - wp.xyz));
gl_Position = projectionMatrix * viewMatrix * wp;
}
`,
fragmentShader: /* glsl */ `
uniform vec3 uColor;
uniform float uFront, uAlpha, uAfter, uTip, uDecay, uFadeBack;
varying float vDist;
varying float vFacing;
void main() {
float behind = uFront - vDist;
if (behind < 0.0) discard;
float a = uAfter + (1.0 - uAfter) * exp(-behind * uDecay);
float hot = exp(-behind * 60.0) * uTip;
float back = mix(1.0, smoothstep(-0.15, 0.25, vFacing), uFadeBack);
vec3 c = uColor * (a + hot) + vec3(1.0) * hot * 0.6;
gl_FragColor = vec4(c * uAlpha * back, 1.0);
}
`,
});
}
function segmentsFromPolylines(lines, r, distFn) {
const pos = [];
const dist = [];
const a = [0, 0, 0];
const b = [0, 0, 0];
for (const line of lines) {
for (let i = 0; i < line.length - 1; i++) {
const [lo1, la1] = line[i];
const [lo2, la2] = line[i + 1];
if (Math.abs(lo2 - lo1) > 180) continue; // antimeridian jump
llToVec(lo1, la1, r, a);
llToVec(lo2, la2, r, b);
pos.push(...a, ...b);
dist.push(distFn(lo1, la1, i, line), distFn(lo2, la2, i + 1, line));
}
}
const g = new THREE.BufferGeometry();
g.setAttribute('position', new THREE.Float32BufferAttribute(pos, 3));
g.setAttribute('aDist', new THREE.Float32BufferAttribute(dist, 1));
return g;
}
export function createCoastline(lines, seedLL, color = '#3ff0ff') {
const g = segmentsFromPolylines(lines, 1.0008, (lo, la) => angDist(seedLL[0], seedLL[1], lo, la));
const m = frontLineMaterial({ color, after: 0.32, tip: 5.0, decay: 3.0 });
return new THREE.LineSegments(g, m);
}
export function createBorders(lines, seedLL, color = '#1aa6c4') {
const g = segmentsFromPolylines(lines, 1.0006, (lo, la) => angDist(seedLL[0], seedLL[1], lo, la));
const m = frontLineMaterial({ color, after: 0.18, tip: 1.5, decay: 3.0 });
return new THREE.LineSegments(g, m);
}
// Mycelium: a branching random walk out from the seed. Each vertex carries its
// path length so the network visibly *grows* rather than appearing.
export function createMycelium(seedLL, { primaries = 22, maxSegs = 60000, seed = 7 } = {}) {
const rnd = mulberry32(seed);
const pos = [];
const dist = [];
const a = [0, 0, 0];
const b = [0, 0, 0];
const stack = [];
for (let i = 0; i < primaries; i++) {
stack.push({ lon: seedLL[0], lat: seedLL[1], brg: (360 * i) / primaries + rnd() * 12, len: 0, depth: 0, life: 150 + rnd() * 120 });
}
let segs = 0;
while (stack.length && segs < maxSegs) {
const br = stack.pop();
let { lon, lat, brg, len } = br;
const step = (0.5 + rnd() * 0.5) * D2R * (br.depth === 0 ? 1.2 : 0.9);
for (let s = 0; s < br.life && segs < maxSegs; s++) {
// outward bias keeps the culture expanding rather than knotting
const away = (bearing(lon, lat, seedLL[0], seedLL[1]) + 180) % 360;
let diff = ((away - brg + 540) % 360) - 180;
brg += diff * 0.04 + (rnd() - 0.5) * 26;
const [nl, nt] = destination(lon, lat, brg, step);
if (Math.abs(nl - lon) < 180) {
llToVec(lon, lat, 1.0012, a);
llToVec(nl, nt, 1.0012, b);
pos.push(...a, ...b);
dist.push(len * 0.82, (len + step) * 0.82);
segs++;
}
lon = nl;
lat = nt;
len += step;
if (len > Math.PI * 1.05) break;
const pBranch = br.depth < 5 ? 0.07 : 0.02;
if (rnd() < pBranch) {
stack.push({ lon, lat, brg: brg + (rnd() < 0.5 ? -1 : 1) * (22 + rnd() * 45), len, depth: br.depth + 1, life: (br.life - s) * (0.35 + rnd() * 0.5) });
}
}
}
const g = new THREE.BufferGeometry();
g.setAttribute('position', new THREE.Float32BufferAttribute(pos, 3));
g.setAttribute('aDist', new THREE.Float32BufferAttribute(dist, 1));
const m = frontLineMaterial({ color: '#2fe8ff', after: 0.07, tip: 4.0, decay: 2.2 });
const obj = new THREE.LineSegments(g, m);
obj.userData.segments = segs;
return obj;
}
// Subsea cables: drawn on, then light pulses ride them.
export function createCables(cables) {
const pos = [];
const arc = [];
const rnd = [];
const a = [0, 0, 0];
const b = [0, 0, 0];
const r = mulberry32(99);
for (const line of cables) {
let L = 0;
const off = r();
for (let i = 0; i < line.length - 1; i++) {
const [lo1, la1] = line[i];
const [lo2, la2] = line[i + 1];
if (Math.abs(lo2 - lo1) > 180) continue;
const d = angDist(lo1, la1, lo2, la2);
llToVec(lo1, la1, 1.0015, a);
llToVec(lo2, la2, 1.0015, b);
pos.push(...a, ...b);
arc.push(L, L + d);
rnd.push(off, off);
L += d;
}
}
const g = new THREE.BufferGeometry();
g.setAttribute('position', new THREE.Float32BufferAttribute(pos, 3));
g.setAttribute('aArc', new THREE.Float32BufferAttribute(arc, 1));
g.setAttribute('aRnd', new THREE.Float32BufferAttribute(rnd, 1));
const m = new THREE.ShaderMaterial({
transparent: true,
depthWrite: false,
blending: THREE.AdditiveBlending,
uniforms: { uT: { value: 0 }, uDraw: { value: 0 }, uAlpha: { value: 0 }, uColor: { value: new THREE.Color('#2bd9ff') } },
vertexShader: /* glsl */ `
attribute float aArc;
attribute float aRnd;
varying float vArc;
varying float vRnd;
varying float vFacing;
void main() {
vArc = aArc; vRnd = aRnd;
vec4 wp = modelMatrix * vec4(position, 1.0);
vFacing = dot(normalize(wp.xyz), normalize(cameraPosition - wp.xyz));
gl_Position = projectionMatrix * viewMatrix * wp;
}
`,
fragmentShader: /* glsl */ `
uniform float uT, uDraw, uAlpha;
uniform vec3 uColor;
varying float vArc;
varying float vRnd;
varying float vFacing;
void main() {
float drawn = uDraw * 3.2 - vArc - vRnd * 0.4;
if (drawn < 0.0) discard;
float tip = exp(-drawn * 30.0) * 3.0;
float p = fract(vArc * 3.0 - uT * 0.9 + vRnd);
float pulse = pow(smoothstep(0.85, 1.0, p), 3.0) * 2.5;
float back = smoothstep(-0.1, 0.25, vFacing);
vec3 c = uColor * (0.28 + pulse) + vec3(1.0) * (tip + pulse * 0.4);
gl_FragColor = vec4(c * uAlpha * back, 1.0);
}
`,
});
return new THREE.LineSegments(g, m);
}
// Great-circle arcs that lift off the surface; a bright head travels along.
export function createArcs(pairs, seed = 5) {
const rnd = mulberry32(seed);
const pos = [];
const u = [];
const t0 = [];
const SEG = 48;
const a = [0, 0, 0];
const b = [0, 0, 0];
for (const [p, q] of pairs) {
const d = angDist(p[0], p[1], q[0], q[1]);
if (d < 0.05) continue;
const brg = bearing(p[0], p[1], q[0], q[1]);
const lift = 0.015 + d * 0.07;
const start = rnd();
let prev = null;
for (let i = 0; i <= SEG; i++) {
const f = i / SEG;
const [lo, la] = destination(p[0], p[1], brg, d * f);
const h = 1.002 + Math.sin(Math.PI * f) * lift;
llToVec(lo, la, h, b);
if (prev) {
pos.push(...prev, ...b);
u.push((i - 1) / SEG, f);
t0.push(start, start);
}
prev = [...b];
}
}
const g = new THREE.BufferGeometry();
g.setAttribute('position', new THREE.Float32BufferAttribute(pos, 3));
g.setAttribute('aU', new THREE.Float32BufferAttribute(u, 1));
g.setAttribute('aT0', new THREE.Float32BufferAttribute(t0, 1));
const m = new THREE.ShaderMaterial({
transparent: true,
depthWrite: false,
blending: THREE.AdditiveBlending,
uniforms: { uT: { value: 0 }, uAlpha: { value: 0 }, uColor: { value: new THREE.Color('#5ff6ff') } },
vertexShader: /* glsl */ `
attribute float aU;
attribute float aT0;
varying float vU;
varying float vT0;
void main() {
vU = aU; vT0 = aT0;
gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
}
`,
fragmentShader: /* glsl */ `
uniform float uT, uAlpha;
uniform vec3 uColor;
varying float vU;
varying float vT0;
void main() {
float head = uT * 0.55 - vT0 * 1.6; // each arc launches at its own time
float k = head - vU;
if (k < 0.0) discard;
float trail = exp(-k * 3.5);
float hot = exp(-k * 40.0) * 2.0;
vec3 c = uColor * (0.15 + trail) + vec3(1.0) * hot;
gl_FragColor = vec4(c * uAlpha, 1.0);
}
`,
});
return new THREE.LineSegments(g, m);
}
// The digital twin's lattice: a geodesic shell that grows outward.
export function createLattice() {
const ico = new THREE.IcosahedronGeometry(1, 4);
const edges = new THREE.WireframeGeometry(ico);
const p = edges.getAttribute('position');
const rnd = [];
// build order: a sweep out from the seed (+X, Null Island) with a little jitter,
// so the shell visibly constructs itself the way the culture grew
for (let i = 0; i < p.count; i += 2) {
const mx = (p.getX(i) + p.getX(i + 1)) / 2;
const my = (p.getY(i) + p.getY(i + 1)) / 2;
const mz = (p.getZ(i) + p.getZ(i + 1)) / 2;
const ang = Math.acos(Math.max(-1, Math.min(1, mx / Math.hypot(mx, my, mz)))) / Math.PI;
const h = Math.sin(mx * 91.7 + my * 47.3 + mz * 13.1) * 43758.5;
const v = Math.min(0.999, ang * 0.9 + (h - Math.floor(h)) * 0.08);
rnd.push(v, v);
}
edges.setAttribute('aRnd', new THREE.Float32BufferAttribute(rnd, 1));
const m = new THREE.ShaderMaterial({
transparent: true,
depthWrite: false,
blending: THREE.AdditiveBlending,
uniforms: { uAlpha: { value: 0 }, uBuild: { value: 0 }, uT: { value: 0 }, uColor: { value: new THREE.Color('#38e8ff') } },
vertexShader: /* glsl */ `
attribute float aRnd;
varying float vRnd;
varying vec3 vP;
varying float vFacing;
void main() {
vRnd = aRnd; vP = normalize(position);
vec4 wp = modelMatrix * vec4(position, 1.0);
vFacing = dot(normalize(wp.xyz), normalize(cameraPosition - wp.xyz));
gl_Position = projectionMatrix * viewMatrix * wp;
}
`,
fragmentShader: /* glsl */ `
uniform float uAlpha, uBuild, uT;
uniform vec3 uColor;
varying float vRnd;
varying vec3 vP;
varying float vFacing;
void main() {
if (vRnd > uBuild) discard;
float fresh = exp(-(uBuild - vRnd) * 18.0);
float shimmer = 0.6 + 0.4 * sin(uT * 3.0 + vRnd * 40.0);
// silhouette-weighted: the shell reads as a sphere, not a net over the planet
float rim = 1.0 - abs(vFacing);
float back = mix(0.18, 1.0, pow(rim, 1.5));
vec3 c = uColor * (0.3 * shimmer + fresh * 1.6) + vec3(1.0) * fresh * 0.7;
gl_FragColor = vec4(c * uAlpha * back, 1.0);
}
`,
});
return new THREE.LineSegments(edges, m);
}
+832
View File
@@ -0,0 +1,832 @@
import * as THREE from 'three';
import * as satellite from 'satellite.js';
import { T, FPS, FEEDS, CITIES, DURATION, captureTimes, clamp, lerp, ramp, smooth, win, easeInOut, expoOut, mulberry32 } from './timeline.js';
import { loadCity, buildCity, loopPath } from './city.js';
import { llToVec, subsolar, angDist, D2R, R2D, EARTH_KM } from './geo.js';
import { createGlobe } from './globe.js';
import { createCoastline, createBorders, createMycelium, createCables, createArcs, createLattice } from './lines.js';
import { createMovers, createEvents, createSatellites, createOrbitRings, createWind, createStars } from './layers.js';
import { createPost } from './post.js';
import { createHUD } from './hud.js';
const qs = new URLSearchParams(location.search);
const W = +qs.get('w') || 1920;
const H = +qs.get('h') || 1080;
const PORTRAIT = H > W;
const SEED_LL = [0, 0]; // `seed --origin null` — Null Island
const status = (m) => {
window.__status = m;
console.log('[gev]', m);
};
async function json(path) {
try {
const r = await fetch(path);
if (!r.ok) throw new Error(r.status);
return await r.json();
} catch (e) {
console.warn('missing', path, e.message);
return null;
}
}
async function text(path) {
try {
const r = await fetch(path);
return r.ok ? await r.text() : null;
} catch {
return null;
}
}
function tex(path, { srgb = true, mips = true } = {}) {
return new Promise((res) => {
new THREE.TextureLoader().load(
path,
(t) => {
if (srgb) t.colorSpace = THREE.SRGBColorSpace;
t.anisotropy = 8;
t.generateMipmaps = mips;
t.minFilter = mips ? THREE.LinearMipmapLinearFilter : THREE.LinearFilter;
t.wrapS = THREE.RepeatWrapping;
res(t);
},
undefined,
() => {
console.warn('missing texture', path);
const d = new THREE.DataTexture(new Uint8Array([0, 0, 0, 255]), 1, 1);
d.needsUpdate = true;
res(d);
},
);
});
}
function img(path) {
return new Promise((res) => {
const im = new Image();
im.onload = () => res(im);
im.onerror = () => res(null);
im.src = path;
});
}
// ------------------------------------------------------------------ boot
async function init() {
status('loading');
const canvas = document.getElementById('c');
canvas.width = W;
canvas.height = H;
canvas.style.width = W + 'px';
canvas.style.height = H + 'px';
await document.fonts.ready;
await Promise.all([
document.fonts.load('400 20px "JetBrains Mono"'),
document.fonts.load('500 20px "JetBrains Mono"'),
document.fonts.load('200 20px Inter'),
document.fonts.load('300 20px Inter'),
document.fonts.load('500 20px Inter'),
]);
const [flights, mil, vessels, quakes, fires, wind, cables, coast, borders, cities, airports, radio, bikes, tleText, tleGroups, heroIdx, camIdx, extra, plants, pads] =
await Promise.all([
json('data/flights.json'),
json('data/military.json'),
json('data/vessels.json'),
json('data/quakes.json'),
json('data/fires.json'),
json('data/wind.json'),
json('data/cables.json'),
json('data/coastline.json'),
json('data/borders.json'),
json('data/cities.json'),
json('data/airports.json'),
json('data/radio.json'),
json('data/bikes.json'),
text('data/tle.txt'),
json('data/tle_groups.json'),
json('imagery/hero/index.json'),
json('cctv/index.json'),
json('data/extra_counts.json'),
json('data/power_plants.json'),
json('data/launch_sites.json'),
]);
const [dayTex, nightTex, cloudTex] = await Promise.all([
tex('imagery/earth_day.jpg'),
tex('imagery/earth_night.jpg'),
tex('imagery/clouds.jpg', { srgb: false }),
]);
// Hero captures (Esri synthetic screenshots) and CCTV stills
const heroes = [];
for (const h of heroIdx || []) {
const [wide, close] = await Promise.all([img(`imagery/hero/${h.slug}_wide.jpg`), img(`imagery/hero/${h.slug}_close.jpg`)]);
if (wide && close) heroes.push({ ...h, wide, close });
}
const cams = [];
for (const c of camIdx || []) {
const im = await img(`cctv/${c.file}`);
if (im) cams.push({ ...c, img: im });
}
status(`assets: ${heroes.length} heroes, ${cams.length} cams`);
// ---------------------------------------------------------------- scene
const renderer = new THREE.WebGLRenderer({ canvas, antialias: false, preserveDrawingBuffer: true, powerPreference: 'high-performance' });
renderer.setPixelRatio(1);
renderer.setSize(W, H, false);
renderer.autoClear = false;
const scene = new THREE.Scene();
const FOV = 26;
const camera = new THREE.PerspectiveCamera(FOV, W / H, 0.005, 3000);
const snapT = new Date((flights?.t || Date.now() / 1000) * 1000);
const sunLL = subsolar(snapT);
const sunV = new THREE.Vector3(...llToVec(sunLL[0], sunLL[1]));
const globe = createGlobe({ dayTex, nightTex, cloudTex });
globe.uniforms.uSeed.value.set(...llToVec(SEED_LL[0], SEED_LL[1]));
globe.uniforms.uSun.value.copy(sunV);
globe.uniforms.uWave.value.set(...T.zWave);
scene.add(globe.group);
const stars = createStars();
scene.add(stars.group);
const coastline = coast ? createCoastline(coast.lines, SEED_LL) : null;
const borderL = borders ? createBorders(borders.lines, SEED_LL) : null;
const myc = createMycelium(SEED_LL);
[coastline, borderL, myc].forEach((o) => o && scene.add(o));
const cableL = cables ? createCables(cables.cables) : null;
if (cableL) scene.add(cableL);
// live feeds
const L = {};
if (flights) L.air = createMovers(flights.rows, { color: '#e8fbff', trailColor: '#4fd8ff', size: 2.0, altScale: 10, seed: 2 });
if (mil) L.mil = createMovers(mil.rows, { color: '#ffbf5a', trailColor: '#ff9a2a', size: 4.0, altScale: 10, revealSpread: 0.6, seed: 4 });
if (vessels) L.sea = createMovers(vessels.rows.map((r) => [r[0], r[1], 0, r[3], (r[2] || 0) * 0.5144]), { color: '#52ffc8', trailColor: '#1fd6a0', size: 1.5, gain: 0.45, altScale: 0, seed: 6 });
const quakeRowsX = (quakes?.rows || []).filter((r) => r[3] >= 2.5).map((r) => [r[0], r[1], r[3]]);
const quakeRows = quakeRowsX;
L.quake = createEvents(quakeRows, { color: '#ff3d3d', size: 1.6, mode: 'ring', weightFn: (m) => clamp((m - 2.4) / 4.5, 0.05, 1.3), spread: 1.6 });
const fireRows = (fires?.rows || []).map((r) => [r[0], r[1], r[2]]);
L.fire = createEvents(fireRows, { color: '#ff6a14', size: 1.15, gain: 0.32, mode: 'flicker', weightFn: (f) => clamp(Math.log10(1 + f) / 2.5, 0, 1.2), spread: 1.4 });
if (wind) L.wind = createWind(wind, { count: PORTRAIT ? 6000 : 7000 });
Object.values(L).forEach((l) => scene.add(l.group));
// satellites
const sats = [];
if (tleText) {
const lines = tleText.split(/\r?\n/).map((s) => s.trimEnd()).filter(Boolean);
for (let i = 0; i < lines.length - 1; i++) {
if (lines[i].startsWith('1 ') && lines[i + 1].startsWith('2 ')) {
const name = i > 0 && !lines[i - 1].startsWith('2 ') && !lines[i - 1].startsWith('1 ') ? lines[i - 1].replace(/^0 /, '').trim() : '';
try {
const rec = satellite.twoline2satrec(lines[i], lines[i + 1]);
sats.push({ rec, name, norad: +lines[i].slice(2, 7) || 0 });
} catch {}
i++;
}
}
}
const stationSet = new Set(tleGroups?.groups_norad?.stations || [25544]);
const satL = createSatellites(Math.max(1, sats.length));
sats.forEach((s, i) => (satL.kind[i] = stationSet.has(s.norad) ? 1 : 0));
satL.group.geometry.getAttribute('aK').needsUpdate = true;
scene.add(satL.group);
const satByNorad = new Map(sats.map((s, i) => [s.norad, i]));
// orbit rings in an ECI-aligned group, rotated by -GMST each frame
const ringGroup = new THREE.Group();
scene.add(ringGroup);
// LEO rings for the orbital beat; GPS/GEO rings only once the camera is far out
const ringIds = [25544, 20580, 48274, 41866, 51850, 40105, 28474, 43226];
const ringPaths = [];
const highPaths = [];
for (const id of ringIds) {
const i = satByNorad.get(id);
if (i === undefined) continue;
const rec = sats[i].rec;
const period = (2 * Math.PI) / rec.no; // minutes
const path = [];
for (let k = 0; k <= 240; k++) {
const d = new Date(snapT.getTime() + (k / 240) * period * 60000);
const pv = satellite.propagate(rec, d);
if (!pv.position) continue;
const p = pv.position;
path.push([p.x / EARTH_KM, p.z / EARTH_KM, -p.y / EARTH_KM]);
}
if (path.length > 10) (Math.hypot(...path[0]) < 1.3 ? ringPaths : highPaths).push(path);
}
const rings = createOrbitRings(ringPaths);
const ringsHigh = createOrbitRings(highPaths, '#a9a2ff');
ringGroup.add(rings, ringsHigh);
// data arcs for saturation: between big cities, cameras and cable ends
const arcPairs = [];
{
const r = mulberry32(42);
const pool = (cities?.rows || []).slice(0, 400).map((c) => [c[0], c[1]]);
for (let i = 0; i < 150 && pool.length > 2; i++) {
const a = pool[Math.floor(Math.pow(r(), 1.6) * pool.length)];
const b = pool[Math.floor(r() * pool.length)];
if (angDist(a[0], a[1], b[0], b[1]) > 0.15) arcPairs.push([a, b]);
}
}
const arcs = createArcs(arcPairs);
scene.add(arcs);
const lattice = createLattice();
scene.add(lattice);
const pxScale = Math.min(W, H) / 1080;
for (const l of [...Object.values(L), satL, stars]) if (l.uniforms.uPx) l.uniforms.uPx.value = pxScale;
// ---------------------------------------------------------------- HUD + post
const hudCanvas = document.createElement('canvas');
hudCanvas.width = W;
hudCanvas.height = H;
const post = createPost(renderer, W, H, hudCanvas);
const hudCtx = hudCanvas.getContext('2d');
const counts = {
air: flights?.rows.length || 0,
mil: mil?.rows.length || 0,
sea: vessels?.rows.length || 0,
orbit: sats.length,
quake: quakes?.rows.length || 0,
fire: fires?.rows.length || 0,
wind: wind ? wind.nx * wind.ny : 0,
cable: cables?.cables.length || 0,
cctv: extra?.cctv_catalog || cams.length,
cities: cities?.rows.length || 0,
cctvCountries: extra?.cctv_countries || new Set(cams.map((c) => c.country)).size,
cctvSources: extra?.cctv_sources ? Object.keys(extra.cctv_sources).length : new Set(cams.map((c) => c.source)).size,
};
const rapidList = [];
if (airports) rapidList.push(['AIRPORTS', airports.rows.length]);
if (radio) rapidList.push(['RADIO STATIONS', radio.rows.length]);
if (bikes) rapidList.push(['BIKESHARE STATIONS', bikes.rows.length]);
if (plants) rapidList.push(['POWER PLANTS', plants.rows.length]);
if (pads) rapidList.push(['LAUNCH PADS', pads.rows.length]);
rapidList.push(['POPULATED PLACES', counts.cities]);
rapidList.push(['LAND BORDERS', borders?.lines.length || 0]);
// notable callouts — real records picked from the feeds
const pick = {
flights: pickFlights(flights),
mil: (mil?.rows || []).filter((r) => r[5] && r[7] && r[2] > 1500 && r[4] > 80).slice(0, 8),
ships: (vessels?.rows || []).filter((r) => r[5] && r[2] > 6).slice(0, 40),
quakeTop: (quakes?.rows || []).slice().sort((a, b) => b[3] - a[3]).slice(0, 4),
fireTop: (fires?.rows || []).slice().sort((a, b) => b[2] - a[2]).slice(0, 3),
cities: (cities?.rows || []).slice(0, 220),
};
const capT = captureTimes(heroes.length);
// v2 assets: the capture flurry, street cameras, detections
const flurryIdx = (await json('imagery/flurry/index.json')) || [];
const flurry = [];
for (const f of flurryIdx) {
const im = await img(`imagery/flurry/${f.file}`);
if (im) flurry.push({ ...f, img: im });
}
const detections = (await json('cctv/detections.json')) || {};
const nycIdx = (await json('cctv/nyc/index.json')) || [];
const nycCams = [];
for (const c of nycIdx) {
const im = await img(`cctv/nyc/${c.file}`);
if (im) nycCams.push({ ...c, img: im, det: detections[`nyc/${c.file}`] || detections[`cctv/nyc/${c.file}`] || [] });
}
for (const c of cams) c.det = detections[c.file] || detections[`cctv/${c.file}`] || [];
status(`v2 assets: ${flurry.length} flurry, ${nycCams.length} nyc cams, ${Object.keys(detections).length} detection sets`);
// ---------------------------------------------------------------- cities
const cityCam = new THREE.PerspectiveCamera(PORTRAIT ? 60 : 36, W / H, 0.05, 600);
const cityScenes = [];
for (const c of CITIES) {
const d = await loadCity(c.slug);
if (!d.meta) {
console.warn('city missing', c.slug);
continue;
}
const built = buildCity(d, { modeled: modeledFor(c.slug, d) });
cityScenes.push({ ...c, d, built });
status(`city ${c.slug}: ${JSON.stringify(built.stats)}`);
}
window.__cities = cityScenes;
const CITY_CAMS = {
everest: [[[17, 26, 22], [0, 4.5, 0]], [[7, 16, 17], [0, 6, -1]], [[-9, 12.5, 13], [0, 6.5, 0]]],
nyc: [[[0.2, 7.5, 2.2], [0, 0, -0.3]], [[1.7, 2.7, 2.5], [0, 0.1, -0.4]], [[2.7, 1.15, 0.5], [0.1, 0.15, -0.6]]],
boston: [[[-0.4, 5.6, 3.4], [0, 0, 0]], [[1.3, 2.5, 2.7], [0, 0, -0.2]], [[2.5, 1.35, 1.0], [0.1, 0.05, -0.3]]],
lax: [[[-4.2, 2.8, 3.6], [0, 0, 0]], [[-1.0, 1.5, 3.0], [0.3, 0, 0]], [[1.9, 0.95, 2.3], [0.6, 0, -0.2]]],
bragg: [[[0, 9, 8], [0, 0, 0]], [[-3.2, 4.8, 5.2], [0.3, 0, 0]], [[-5.2, 2.8, 2.1], [0.5, 0, 0]]],
};
const camCurves = {};
for (const [k, keys] of Object.entries(CITY_CAMS)) {
const s = PORTRAIT ? 1.18 : 1;
camCurves[k] = {
pos: new THREE.CatmullRomCurve3(keys.map(([p, q]) => new THREE.Vector3(q[0] + (p[0] - q[0]) * s, q[1] + (p[1] - q[1]) * s, q[2] + (p[2] - q[2]) * s))),
tgt: new THREE.CatmullRomCurve3(keys.map(([, q]) => new THREE.Vector3(...q))),
};
}
const KMV = 0.001;
const cv = new THREE.Vector3();
function projectCity(x, y, z = 0) {
cv.set(x * KMV, z * KMV, -y * KMV).project(cityCam);
return { x: (cv.x * 0.5 + 0.5) * W, y: (-cv.y * 0.5 + 0.5) * H, vis: cv.z < 1 && cv.z > -1 };
}
const hud = createHUD({ W, H, PORTRAIT, heroes, cams, counts, rapidList, pick, capT, snapT, sunLL, flurry, nycCams });
// ---------------------------------------------------------------- camera director
// Keys: [t, lon, lat, g (globe radius / half the short side), shiftX, shiftY, roll]
const sx = PORTRAIT ? 0 : 0.26;
const sy = PORTRAIT ? 0.2 : 0;
const KEYS = [
[0, 0, 0, 0.6, 0, 0, 0],
[8.7, 0, 0, 0.6, 0, 0, 0],
[16.6, -1.5, 1.5, 0.62, 0, 0, 0],
[19.5, 4, 6, 0.7, 0, 0, -0.01],
[22.6, 14, 12, 0.74, 0, 0, -0.02],
// the flurry: the planet whips a full turn, accelerating then braking
[23.6, 20, 12, 0.74, 0, 0, -0.02],
[25.0, 50, 14, 0.74, 0, 0, -0.03],
[26.2, 120, 16, 0.72, 0, 0, -0.04],
[27.2, 230, 18, 0.72, 0, 0, -0.04],
[28.2, 330, 24, 0.78, 0, 0, -0.03],
[29.2, 375, 30, 0.9, sx * 0.3, sy * 0.3, -0.02],
[30.0, 384, 36, 0.98, sx * 0.5, sy * 0.5, -0.02],
[31.8, 374, 58, 1.85, sx, sy, -0.03],
[32.8, 368, 59, 1.9, sx, sy, -0.03],
[34.4, 305, 43, 1.0, sx, sy, -0.01],
[35.6, 268, 30, 0.5, sx, sy, 0.0],
[37.6, 225, 20, 0.62, sx, sy, 0.02],
[39.4, 175, 2, 0.7, sx, sy, 0.01],
[41.0, 150, 6, 0.78, sx, sy, 0.0],
// descent: swing to the Himalaya and fall in
[42.4, 100, 22, 1.1, sx * 0.4, sy * 0.4, 0.0],
[43.2, 88, 27.5, 3.5, 0, 0, 0.0],
[43.7, 87, 28, 9.0, 0, 0, 0.0],
// behind the camera wall, and back out for saturation
[71.0, 205, 16, 0.9, 0, 0, 0.0],
[76.4, 200, 18, 0.9, 0, 0, 0.0],
[80.0, 178, 24, 0.5, 0, 0, 0.0],
[84.4, 148, 22, 0.16, 0, 0, 0.0],
];
// Title framing: the planet becomes the iris of the logo.
const eye = hud.eyeLayout;
// Each flurry capture fires as its real location rotates into view on the
// leading (east) side of the disc during the spin (18° east of centre).
{
const samples = [];
for (let i = 0; i <= 600; i++) {
const tt = T.flurry + (i / 600) * (T.flurryEnd - T.flurry);
samples.push([tt, camAt(tt).lon]);
}
const lo = samples[0][1];
const hi = samples[samples.length - 1][1];
for (const f of flurry) {
let L = f.lon - 18;
while (L < lo) L += 360;
if (L > hi) continue;
const s = samples.find((q) => q[1] >= L);
if (s) f.t = s[0] + (hash01(f.lat * 13.3) - 0.5) * 0.08;
}
}
function camAt(t) {
if (t >= T.title - 0.05) {
const g = eye.irisR / (Math.min(W, H) / 2);
const k = t - T.title;
return { lon: 108 + k * 1.6, lat: 12, g, shiftX: 0, shiftY: (H / 2 - eye.cy) / (Math.min(W, H) / 2), roll: 0 };
}
let i = 0;
while (i < KEYS.length - 2 && t > KEYS[i + 1][0]) i++;
const k0 = KEYS[Math.max(0, i - 1)];
const k1 = KEYS[i];
const k2 = KEYS[i + 1];
const k3 = KEYS[Math.min(KEYS.length - 1, i + 2)];
const u = clamp((t - k1[0]) / (k2[0] - k1[0]));
const out = [];
for (let c = 1; c < 7; c++) {
const dt = k2[0] - k1[0];
const m1 = ((k2[c] - k0[c]) / Math.max(1e-3, k2[0] - k0[0])) * dt;
const m2 = ((k3[c] - k1[c]) / Math.max(1e-3, k3[0] - k1[0])) * dt;
const u2 = u * u;
const u3 = u2 * u;
out[c] = (2 * u3 - 3 * u2 + 1) * k1[c] + (u3 - 2 * u2 + u) * m1 + (-2 * u3 + 3 * u2) * k2[c] + (u3 - u2) * m2;
}
// breathing drift so the shot never feels locked off
const drift = smooth(t, 8, 12);
return {
lon: out[1] + Math.sin(t * 0.21) * 0.6 * drift,
lat: out[2] + Math.sin(t * 0.17 + 1.3) * 0.4 * drift,
g: out[3],
shiftX: out[4],
shiftY: out[5],
roll: out[6],
};
}
const HITS = [
[T.collapse, 0.5],
[T.substrate, 0.7],
[T.indexed, 0.9],
[T.cctvWall, 0.5],
[T.cctvOut, 0.6],
[T.dishBreak, 1.0],
[T.eyeOpen, 0.4],
];
function impulse(t, decay = 6) {
let v = 0;
for (const [h, a] of HITS) if (t >= h) v += a * Math.exp(-(t - h) * decay);
return v;
}
function placeCamera(t) {
const c = camAt(t);
const minDim = Math.min(W, H);
const tanHalf = Math.tan((FOV * D2R) / 2);
const tanA = (c.g * minDim) / H * tanHalf;
const dist = 1 / Math.sin(Math.atan(tanA));
const shake = impulse(t, 9) * 0.0025;
const sh1 = Math.sin(t * 91.0) * shake;
const sh2 = Math.cos(t * 77.0) * shake;
camera.position.set(...llToVec(c.lon, c.lat, dist));
camera.up.set(0, 1, 0);
camera.lookAt(0, 0, 0);
camera.rotateZ(c.roll + sh1 * 2);
camera.near = Math.max(0.002, dist - 30);
camera.far = dist + 2000;
// shift the globe on screen without changing perspective
const offX = -c.shiftX * (minDim / 2) + sh1 * W;
const offY = c.shiftY * (minDim / 2) + sh2 * H;
camera.setViewOffset(W, H, offX, offY, W, H);
camera.updateProjectionMatrix();
camera.updateMatrixWorld();
return { ...c, dist };
}
// ---------------------------------------------------------------- per-frame state
const tmpV = new THREE.Vector3();
function project(lon, lat, r = 1) {
tmpV.set(...llToVec(lon, lat, r));
const n = tmpV.clone().normalize();
const toCam = camera.position.clone().sub(tmpV).normalize();
const facing = r <= 1.2 ? n.dot(toCam) : 1;
tmpV.project(camera);
return { x: (tmpV.x * 0.5 + 0.5) * W, y: (-tmpV.y * 0.5 + 0.5) * H, vis: facing > 0.05 && tmpV.z < 1, facing };
}
function projectVec(v) {
tmpV.set(v[0], v[1], v[2]);
tmpV.project(camera);
return { x: (tmpV.x * 0.5 + 0.5) * W, y: (-tmpV.y * 0.5 + 0.5) * H, z: tmpV.z };
}
function frontAngle(t) {
if (t < T.growStart) return -1;
const x = ramp(t, T.growStart, T.growEnd);
const k = 3;
const base = Math.PI * (Math.exp(k * x) - 1) / (Math.exp(k) - 1);
return base + Math.max(0, t - T.growEnd) * 0.6;
}
const satScratch = { idx: -1 };
function updateSatellites(t, simDate) {
const gmst = satellite.gstime(simDate);
ringGroup.rotation.y = -gmst;
if (satScratch.t === t) return gmst;
satScratch.t = t;
const p = satL.positions;
for (let i = 0; i < sats.length; i++) {
const pv = satellite.propagate(sats[i].rec, simDate);
const e = pv && pv.position;
if (!e || isNaN(e.x)) {
p[i * 3] = p[i * 3 + 1] = p[i * 3 + 2] = 0;
continue;
}
const ecf = satellite.eciToEcf(e, gmst);
p[i * 3] = ecf.x / EARTH_KM;
p[i * 3 + 1] = ecf.z / EARTH_KM;
p[i * 3 + 2] = -ecf.y / EARTH_KM;
}
satL.attr.needsUpdate = true;
return gmst;
}
function feedAlpha(t, start) {
// everything stays on through saturation, then clears for the title
return smooth(t, start, start + 0.25) * (1 - smooth(t, T.cut - 0.05, T.cut));
}
function renderFrame(frame) {
const t = frame / FPS;
const cam = placeCamera(t);
const cityNow = cityScenes.find((c) => t >= c.t0 && t < c.t1);
const G = globe.uniforms;
const front = frontAngle(t);
const titleOn = t >= T.title - 0.05;
G.uTime.value = t;
G.uFront.value = titleOn ? 10 : front;
G.uHolo.value = smooth(t, T.growStart - 0.4, T.growStart + 1.2) * (1 - smooth(t, T.indexed - 1, T.indexed + 1));
G.uLights.value = smooth(t, T.growStart, T.growStart + 0.8);
G.uImagery.value = titleOn ? 1 : smooth(t, 26.5, 29.5);
G.uLit.value = titleOn ? 1 : smooth(t, 22.6, 26.0);
G.uGridAmt.value = titleOn ? 0 : 1;
G.uDim.value = 1 - 0.35 * win(t, T.cctvWall - 0.2, T.cctvOut + 0.3, 0.3, 0.3);
globe.cloudU.uAmt.value = titleOn ? 0.6 : smooth(t, T.wind, T.wind + 1.6) * 0.62;
globe.atmoU.uAmt.value = titleOn ? 1 : smooth(t, 21.0, 25.0);
globe.group.children[1].visible = globe.cloudU.uAmt.value > 0.0005;
globe.group.children[2].visible = globe.atmoU.uAmt.value > 0.0005;
globe.group.visible = !(t >= T.cut && t < T.title - 0.05) && t >= T.seed - 0.1;
stars.uniforms.uAlpha.value = titleOn ? 0.5 : smooth(t, T.substrate - 0.5, T.substrate + 2.5) * 0.85 * (t < T.cut ? 1 : 0);
const vecFade = 1 - smooth(t, T.indexStart + 1.5, T.indexed - 2);
if (coastline) {
coastline.material.uniforms.uFront.value = front;
coastline.material.uniforms.uAlpha.value = 0.95 * vecFade;
coastline.visible = vecFade > 0.001 && !titleOn;
}
if (borderL) {
borderL.material.uniforms.uFront.value = front - 0.25;
borderL.material.uniforms.uAlpha.value = 0.7 * vecFade;
borderL.visible = vecFade > 0.001 && !titleOn;
}
myc.material.uniforms.uFront.value = front;
myc.material.uniforms.uAlpha.value = 0.85 * (1 - smooth(t, T.substrate, T.indexStart + 3));
myc.visible = t > T.growStart && t < T.indexStart + 3.2;
// feeds
const dens = clamp((cam.g - 0.12) / 0.75, 0.22, 1);
const reveal = (start, dur = 1.6, spread = 1.4) => ((t - start) * (1 + spread)) / dur;
const setMover = (l, start, accel) => {
if (!l) return;
l.uniforms.uSim.value = Math.max(0, t - start) * accel;
l.uniforms.uReveal.value = reveal(start, 1.6, l.uniforms.uSpread.value);
l.uniforms.uAlpha.value = feedAlpha(t, start) * dens;
l.group.visible = t >= start && t < T.cut;
};
setMover(L.air, T.air, 140);
setMover(L.mil, T.mil, 140);
setMover(L.sea, T.sea, 1800);
const setEv = (l, start, dur = 1.8) => {
if (!l) return;
l.uniforms.uT.value = t;
l.uniforms.uReveal.value = reveal(start, dur, l.uniforms.uSpread.value);
l.uniforms.uAlpha.value = feedAlpha(t, start) * dens;
l.group.visible = t >= start && t < T.cut;
};
setEv(L.quake, T.quake, 1.6);
setEv(L.fire, T.fire, 1.5);
if (L.fire) L.fire.uniforms.uAlpha.value *= dens;
if (L.wind) {
const on = t >= T.wind && t < T.cut;
L.wind.group.visible = on;
if (on) {
L.wind.update(t - T.wind);
L.wind.uniforms.uAlpha.value = feedAlpha(t, T.wind) * 2.2 * dens;
}
}
if (cableL) {
const u = cableL.material.uniforms;
u.uT.value = t;
u.uDraw.value = ramp(t, T.cable, T.cable + 1.8);
u.uAlpha.value = feedAlpha(t, T.cable) * 0.9;
cableL.visible = t >= T.cable && t < T.cut;
}
// satellites + rings
const satOn = t >= T.orbit && t < T.cut;
const simDate = new Date(snapT.getTime() + Math.max(0, t - T.orbit) * 45 * 1000);
satL.group.visible = satOn;
rings.visible = satOn;
let gmst = satellite.gstime(simDate);
if (satOn) {
gmst = updateSatellites(t, simDate);
satL.uniforms.uReveal.value = reveal(T.orbit, 1.8, 1.3);
satL.uniforms.uAlpha.value = feedAlpha(t, T.orbit) * Math.max(0.45, dens);
rings.material.uniforms.uDraw.value = easeInOut(ramp(t, T.orbit + 0.3, T.orbit + 2.4));
rings.material.uniforms.uAlpha.value = feedAlpha(t, T.orbit) * 0.8;
}
ringsHigh.visible = t >= T.saturate + 1 && t < T.cut;
ringsHigh.material.uniforms.uDraw.value = easeInOut(ramp(t, T.saturate + 1, T.saturate + 3.5));
ringsHigh.material.uniforms.uAlpha.value = feedAlpha(t, T.saturate + 1) * 0.7;
// saturation: arcs + lattice
arcs.visible = t >= T.arcs && t < T.cut;
arcs.material.uniforms.uT.value = t - T.arcs;
arcs.material.uniforms.uAlpha.value = feedAlpha(t, T.arcs) * 0.9;
const twinOn = t >= T.twin && t < T.cut;
lattice.visible = twinOn;
if (twinOn) {
lattice.scale.setScalar(latticeScale(t, cam.dist));
lattice.rotation.y = t * 0.05;
const lu = lattice.material.uniforms;
lu.uBuild.value = ramp(t, T.twin, T.twin + 2.6);
lu.uT.value = t;
lu.uAlpha.value = smooth(t, T.twin, T.twin + 0.6) * (1 - smooth(t, T.cut - 0.4, T.cut)) * 0.7;
}
// ---- HUD
const S = {
t,
frame,
cam,
front,
gmst,
simDate,
project,
projectVec,
satPos: (norad) => {
const i = satByNorad.get(norad);
if (i === undefined) return null;
const p = satL.positions;
return [p[i * 3], p[i * 3 + 1], p[i * 3 + 2]];
},
limbR: limbRadiusPx(cam.dist),
center: project(...[cam.lon, cam.lat], 0), // screen position of the globe centre
flightPos: (row, accel) => {
const sim = Math.max(0, t - T.air) * accel;
return advance(row, sim);
},
shipPos: (row) => advance([row[0], row[1], 0, row[3], (row[2] || 0) * 0.5144], Math.max(0, t - T.sea) * 1800),
};
S.center = projectVec([0, 0, 0]);
S.ctx = hudCtx;
S.camPos = camera.position;
S.latticeR = latticePx(t, cam.dist);
S.ringR = ringR0;
let renderScene = scene;
let renderCam = camera;
if (cityNow) {
const lt = t - cityNow.t0;
const k = (t - cityNow.t0) / (cityNow.t1 - cityNow.t0);
const u = 1 - Math.pow(1 - k, 2.2);
const cc = camCurves[cityNow.slug];
cityCam.position.copy(cc.pos.getPoint(u));
const tg = cc.tgt.getPoint(u);
// a hair of handheld drift
cityCam.position.x += Math.sin(t * 0.7) * 0.01;
cityCam.position.y += Math.sin(t * 0.53) * 0.008;
cityCam.lookAt(tg);
const sh = impulse(t, 9) * 0.003;
cityCam.rotateZ(Math.sin(t * 91) * sh);
cityCam.updateMatrixWorld();
cityNow.built.update(lt, k, pxScale);
// Boston is the transit showcase: let the GTFS lines lead
if (cityNow.slug === 'boston' && cityNow.built.roads) cityNow.built.roads.mesh.material.uniforms.uAlpha.value = 0.45;
renderScene = cityNow.built.scene;
renderCam = cityCam;
S.city = { ...cityNow, lt, k, project: projectCity, built: cityNow.built };
S.cam = { ...S.cam, lat: cityNow.lat, lon: cityNow.lon, dist: 1 + cityCam.position.y / 6371 };
}
hud.draw(S);
// ---- post
const glitch =
win(t, T.collapse - 0.05, T.collapse + 0.35, 0.02, 0.2) * 0.8 +
win(t, T.substrate - 0.02, T.substrate + 0.25, 0.02, 0.2) * 0.5 +
win(t, T.indexed - 0.02, T.indexed + 0.3, 0.02, 0.25) * 0.7 +
win(t, T.cctvWall - 0.02, T.cctvWall + 0.2, 0.02, 0.15) * 0.5 +
win(t, T.cctvOut - 0.05, T.cctvOut + 0.3, 0.02, 0.25) * 0.8 +
win(t, T.dishBreak - 0.02, T.dishBreak + 0.4, 0.02, 0.3) * 1.0 +
win(t, T.cut - 0.25, T.cut, 0.1, 0.01) * 1.0 +
win(t, T.eyeOpen - 0.02, T.eyeOpen + 0.2, 0.02, 0.15) * 0.4 +
win(t, T.terrain - 0.35, T.terrain + 0.25, 0.25, 0.2) * 1.0 +
CITIES.slice(1).reduce((s, c) => s + win(t, c.t0 - 0.06, c.t0 + 0.22, 0.04, 0.16) * 0.8, 0);
const flash =
Math.exp(-Math.max(0, t - T.substrate) * 12) * 0.035 * (t >= T.substrate ? 1 : 0) +
Math.exp(-Math.max(0, t - T.indexed) * 12) * 0.05 * (t >= T.indexed ? 1 : 0) +
Math.exp(-Math.max(0, t - T.dishBreak) * 10) * 0.06 * (t >= T.dishBreak ? 1 : 0) +
Math.exp(-Math.max(0, t - T.eyeOpen) * 10) * 0.05 * (t >= T.eyeOpen ? 1 : 0);
let fade = smooth(t, 0, 0.6);
if (t >= T.cut && t < T.title) fade = 0;
fade *= 1 - smooth(t, T.fadeOut, DURATION_END);
post.render(renderScene, renderCam, {
time: t,
frame,
bloom: t < T.seed ? 0.7 : lerp(0.85, 1.15, smooth(t, T.saturate, T.cut)),
bloomThresh: 0.78,
exposure: 1.0,
aberr: titleOn ? 0.00025 + glitch * 0.004 : 0.0012 + 0.002 * smooth(t, T.saturate, T.cut) + glitch * 0.004,
grain: 0.05,
vignette: 0.6,
glitch,
scan: t < T.seed ? 0.7 : 0.12,
flash,
fade,
hudGain: 1.0,
});
}
const RING_R = Math.min(W, H) * (PORTRAIT ? 0.43 : 0.4);
function latticePx(t, dist) {
const limb = limbRadiusPx(dist) * 1.12;
if (t < T.dishBreak) return lerp(limb, RING_R, Math.pow(easeInOut(ramp(t, T.twin, T.dishBreak)), 1.2));
return RING_R * (1 + 1.6 * Math.pow(ramp(t, T.dishBreak, T.cut), 0.8));
}
function latticeScale(t, dist) {
const px = latticePx(t, dist);
const tanA = (px / (H / 2)) * Math.tan((FOV * D2R) / 2);
return dist * Math.sin(Math.atan(tanA));
}
function limbRadiusPx(dist) {
const tanA = Math.tan(Math.asin(1 / dist));
return (tanA / Math.tan((FOV * D2R) / 2)) * (H / 2);
}
function advance(row, sim) {
// row: [lon, lat, alt, track, vel(m/s)] -> [lon, lat] after sim seconds
const d = ((row[4] || 0) / (EARTH_KM * 1000)) * sim;
const b = (row[3] || 0) * D2R;
const la = row[1] * D2R;
const lo = row[0] * D2R;
const lat2 = Math.asin(Math.sin(la) * Math.cos(d) + Math.cos(la) * Math.sin(d) * Math.cos(b));
const lon2 = lo + Math.atan2(Math.sin(b) * Math.sin(d) * Math.cos(la), Math.cos(d) - Math.sin(la) * Math.sin(lat2));
return [lon2 * R2D, lat2 * R2D, row[2] || 0];
}
// the dish's second coming is sized so the twin breaches it exactly on cue
const ringR0 = RING_R;
window.GEV = { renderFrame, ready: true, W, H, frames: Math.round(DURATION_END * FPS), counts, heroes: heroes.length, cams: cams.length, capT };
status('ready');
if (qs.has('play')) startPlayback(renderFrame);
}
const DURATION_END = DURATION;
const hash01 = (n) => {
const s = Math.sin(n * 127.1 + 311.7) * 43758.5453;
return s - Math.floor(s);
};
// Modelled actors (always labelled MODELED in the HUD) where real feeds are thin.
function modeledFor(slug, d) {
const out = [];
if (slug === 'nyc') {
const realHelis = (d.air?.a || []).filter((p) => p[8] === 'A7').length;
if (realHelis < 3) {
// Hudson and East River corridors, and a hover orbit over Midtown
out.push({ name: 'HUDSON CORRIDOR', path: [[-1750, -3300, 300], [-1950, -1000, 330], [-1800, 1500, 350], [-1450, 3300, 320], [-1250, 1500, 300], [-1400, -1000, 290]], speed: 0.09, phase: 0.1, color: '#7dffd6' });
out.push({ name: 'EAST RIVER', path: [[1650, -3200, 260], [1900, -800, 280], [2150, 1800, 300], [1750, 2600, 300], [1450, 800, 270], [1350, -1500, 250]], speed: 0.08, phase: 0.55, color: '#7dffd6' });
out.push({ name: 'MIDTOWN ORBIT', path: loopPath(-150, 900, 420, 380, 450, 0.3, 40), speed: 0.16, phase: 0.2, color: '#7dffd6' });
}
}
if (slug === 'bragg') {
const rw = d.military?.runways?.[0];
let cx = 0, cy = 0, rot = 0;
if (rw) {
const x1 = rw[2], y1 = rw[3], x2 = rw[rw.length - 2], y2 = rw[rw.length - 1];
cx = (x1 + x2) / 2;
cy = (y1 + y2) / 2;
rot = Math.atan2(y2 - y1, x2 - x1);
}
out.push({ name: 'PATTERN', path: loopPath(cx, cy, 2600, 1100, 330, rot, 48), speed: 0.07, phase: 0.0, color: '#ffb547' });
out.push({ name: 'PATTERN', path: loopPath(cx, cy, 2600, 1100, 330, rot, 48), speed: 0.07, phase: 0.5, color: '#ffb547' });
out.push({ name: 'ROTARY', path: loopPath(cx - 3000, cy + 2500, 900, 700, 180, 0.4, 32), speed: 0.11, phase: 0.3, color: '#ffb547' });
}
return out;
}
// Real-time mode (?play): the same deterministic frames, driven by the audio
// clock, for use as a live loading sequence on a GPU. Browsers block audio
// until a gesture, so the first click starts it.
function startPlayback(renderFrame) {
const audio = new Audio('audio/score.wav');
const cv = document.getElementById('c');
cv.style.cssText = 'width:100vw;height:100vh;object-fit:contain;display:block';
const gate = document.createElement('div');
gate.textContent = 'CLICK TO INITIALISE';
gate.style.cssText = 'position:fixed;inset:0;display:flex;align-items:center;justify-content:center;color:#00f6ff;font:500 14px "JetBrains Mono";letter-spacing:6px;cursor:pointer;background:#000';
document.body.appendChild(gate);
gate.addEventListener('click', () => {
gate.remove();
audio.play().catch(() => {});
const t0 = performance.now();
const tick = () => {
const t = audio.currentTime > 0 ? audio.currentTime : (performance.now() - t0) / 1000;
if (t < DURATION_END) {
renderFrame(Math.min(DURATION_END * FPS - 1, Math.floor(t * FPS)));
requestAnimationFrame(tick);
}
};
requestAnimationFrame(tick);
});
}
function pickFlights(f) {
if (!f) return [];
// long-haul, high, fast, with recognisable callsigns
return f.rows
.filter((r) => r[5] && r[2] > 9000 && r[4] > 220 && /^[A-Z]{3}\d/.test(r[5]))
.sort((a, b) => b[4] - a[4])
.slice(0, 120);
}
init().catch((e) => {
status('error: ' + e.message);
console.error(e);
});
+218
View File
@@ -0,0 +1,218 @@
import * as THREE from 'three';
// HDR scene -> HUD composited in -> dual-filter bloom -> lens + film finish.
const QUAD_VS = /* glsl */ `
varying vec2 vUv;
void main() { vUv = uv; gl_Position = vec4(position.xy, 0.0, 1.0); }
`;
export function createPost(renderer, W, H, hudCanvas) {
const rtOpts = { type: THREE.HalfFloatType, format: THREE.RGBAFormat, depthBuffer: false };
const scene = new THREE.WebGLRenderTarget(W, H, { type: THREE.HalfFloatType, samples: 4 });
const merged = new THREE.WebGLRenderTarget(W, H, rtOpts);
const levels = [];
let w = W;
let h = H;
for (let i = 0; i < 6; i++) {
w = Math.max(2, Math.floor(w / 2));
h = Math.max(2, Math.floor(h / 2));
levels.push({ down: new THREE.WebGLRenderTarget(w, h, rtOpts), up: new THREE.WebGLRenderTarget(w, h, rtOpts) });
}
const hudTex = new THREE.CanvasTexture(hudCanvas);
hudTex.colorSpace = THREE.SRGBColorSpace;
hudTex.minFilter = THREE.LinearFilter;
hudTex.generateMipmaps = false;
const quad = new THREE.Mesh(new THREE.PlaneGeometry(2, 2));
const qScene = new THREE.Scene();
qScene.add(quad);
const qCam = new THREE.OrthographicCamera(-1, 1, 1, -1, 0, 1);
const mergeMat = new THREE.ShaderMaterial({
uniforms: { tScene: { value: null }, tHud: { value: hudTex }, uHudGain: { value: 1.25 } },
vertexShader: QUAD_VS,
fragmentShader: /* glsl */ `
uniform sampler2D tScene, tHud;
uniform float uHudGain;
varying vec2 vUv;
void main() {
vec3 s = texture2D(tScene, vUv).rgb;
vec4 hud = texture2D(tHud, vUv);
// canvas uploads un-premultiplied: straight-alpha over-composite
vec3 c = s * (1.0 - hud.a) + hud.rgb * hud.a * uHudGain;
gl_FragColor = vec4(c, 1.0);
}
`,
depthTest: false,
});
const downMat = new THREE.ShaderMaterial({
uniforms: { tSrc: { value: null }, uTexel: { value: new THREE.Vector2() }, uThresh: { value: 0 }, uFirst: { value: 0 } },
vertexShader: QUAD_VS,
fragmentShader: /* glsl */ `
uniform sampler2D tSrc;
uniform vec2 uTexel;
uniform float uThresh, uFirst;
varying vec2 vUv;
vec3 pre(vec3 c) {
if (uFirst < 0.5) return c;
float l = max(c.r, max(c.g, c.b));
float k = max(l - uThresh, 0.0) / max(l, 1e-4);
return min(c * k, vec3(40.0));
}
void main() {
vec2 o = uTexel;
vec3 c = pre(texture2D(tSrc, vUv).rgb) * 4.0;
c += pre(texture2D(tSrc, vUv + vec2(-o.x, -o.y)).rgb);
c += pre(texture2D(tSrc, vUv + vec2( o.x, -o.y)).rgb);
c += pre(texture2D(tSrc, vUv + vec2(-o.x, o.y)).rgb);
c += pre(texture2D(tSrc, vUv + vec2( o.x, o.y)).rgb);
gl_FragColor = vec4(c / 8.0, 1.0);
}
`,
depthTest: false,
});
const upMat = new THREE.ShaderMaterial({
uniforms: { tSrc: { value: null }, tBase: { value: null }, uTexel: { value: new THREE.Vector2() } },
vertexShader: QUAD_VS,
fragmentShader: /* glsl */ `
uniform sampler2D tSrc, tBase;
uniform vec2 uTexel;
varying vec2 vUv;
void main() {
vec2 o = uTexel;
vec3 c = texture2D(tSrc, vUv + vec2(-o.x * 2.0, 0.0)).rgb;
c += texture2D(tSrc, vUv + vec2(-o.x, o.y)).rgb * 2.0;
c += texture2D(tSrc, vUv + vec2(0.0, o.y * 2.0)).rgb;
c += texture2D(tSrc, vUv + vec2(o.x, o.y)).rgb * 2.0;
c += texture2D(tSrc, vUv + vec2(o.x * 2.0, 0.0)).rgb;
c += texture2D(tSrc, vUv + vec2(o.x, -o.y)).rgb * 2.0;
c += texture2D(tSrc, vUv + vec2(0.0, -o.y * 2.0)).rgb;
c += texture2D(tSrc, vUv + vec2(-o.x, -o.y)).rgb * 2.0;
gl_FragColor = vec4(c / 12.0 + texture2D(tBase, vUv).rgb, 1.0);
}
`,
depthTest: false,
});
const finalMat = new THREE.ShaderMaterial({
uniforms: {
tSrc: { value: null },
tBloom: { value: null },
uBloom: { value: 0.9 },
uExposure: { value: 1 },
uTime: { value: 0 },
uFrame: { value: 0 },
uAberr: { value: 0.0015 },
uGrain: { value: 0.045 },
uVignette: { value: 0.55 },
uGlitch: { value: 0 },
uScan: { value: 0 },
uFlash: { value: 0 },
uFade: { value: 1 },
uRes: { value: new THREE.Vector2(W, H) },
},
vertexShader: QUAD_VS,
fragmentShader: /* glsl */ `
uniform sampler2D tSrc, tBloom;
uniform float uBloom, uExposure, uTime, uFrame, uAberr, uGrain, uVignette, uGlitch, uScan, uFlash, uFade;
uniform vec2 uRes;
varying vec2 vUv;
float h21(vec2 p) { p = fract(p * vec2(123.34, 456.21)); p += dot(p, p + 45.32); return fract(p.x * p.y); }
vec3 aces(vec3 x) {
const float a = 2.51, b = 0.03, c = 2.43, d = 0.59, e = 0.14;
return clamp((x * (a * x + b)) / (x * (c * x + d) + e), 0.0, 1.0);
}
void main() {
vec2 uv = vUv;
// block glitch: horizontal slices displaced on transitions
if (uGlitch > 0.0) {
float row = floor(uv.y * 48.0);
float blk = floor(uv.x * 6.0);
float r = h21(vec2(row, floor(uFrame * 0.5)));
float r2 = h21(vec2(blk, row + floor(uFrame)));
if (r < uGlitch * 0.55) uv.x += (r2 - 0.5) * 0.08 * uGlitch;
if (r2 < uGlitch * 0.12) uv.y += (r - 0.5) * 0.02;
}
vec2 d = uv - 0.5;
float rr = dot(d, d);
vec2 ab = d * rr * uAberr * 7.0 + vec2(uGlitch * 0.006, 0.0);
vec3 col;
col.r = texture2D(tSrc, uv + ab).r;
col.g = texture2D(tSrc, uv).g;
col.b = texture2D(tSrc, uv - ab).b;
vec3 bl;
bl.r = texture2D(tBloom, uv + ab * 1.5).r;
bl.g = texture2D(tBloom, uv).g;
bl.b = texture2D(tBloom, uv - ab * 1.5).b;
col += bl * uBloom;
// flashes punch exposure instead of adding a veil, so blacks stay black
col *= uExposure * (1.0 + uFlash * 6.0);
col = aces(col);
// vignette
float vig = smoothstep(0.95, 0.18, length(d * vec2(uRes.x / uRes.y, 1.0) * 0.82));
col *= mix(1.0, vig, uVignette);
// scanlines (boot / CRT moments)
col *= 1.0 - uScan * 0.22 * (0.5 + 0.5 * sin(gl_FragCoord.y * 3.14159 * 0.5));
col = pow(col, vec3(1.0 / 2.2));
// grain after encoding so it lives in the shadows too
float g = h21(gl_FragCoord.xy + fract(uFrame * 0.6180339) * 913.0) - 0.5;
col += g * uGrain * (1.0 - 0.6 * dot(col, vec3(0.33)));
col *= uFade;
gl_FragColor = vec4(col, 1.0);
}
`,
depthTest: false,
});
function pass(mat, target) {
quad.material = mat;
renderer.setRenderTarget(target);
renderer.render(qScene, qCam);
}
function render(sceneObj, camera, params) {
renderer.setRenderTarget(scene);
renderer.setClearColor(0x000000, 1);
renderer.clear();
renderer.render(sceneObj, camera);
hudTex.needsUpdate = true;
mergeMat.uniforms.tScene.value = scene.texture;
mergeMat.uniforms.uHudGain.value = params.hudGain ?? 1.2;
pass(mergeMat, merged);
let src = merged;
for (let i = 0; i < levels.length; i++) {
downMat.uniforms.tSrc.value = src.texture;
downMat.uniforms.uTexel.value.set(0.5 / src.width, 0.5 / src.height);
downMat.uniforms.uFirst.value = i === 0 ? 1 : 0;
downMat.uniforms.uThresh.value = params.bloomThresh ?? 0.55;
pass(downMat, levels[i].down);
src = levels[i].down;
}
let up = levels[levels.length - 1].down;
for (let i = levels.length - 2; i >= 0; i--) {
upMat.uniforms.tSrc.value = up.texture;
upMat.uniforms.tBase.value = levels[i].down.texture;
upMat.uniforms.uTexel.value.set(0.5 / up.width, 0.5 / up.height);
pass(upMat, levels[i].up);
up = levels[i].up;
}
const u = finalMat.uniforms;
u.tSrc.value = merged.texture;
u.tBloom.value = up.texture;
for (const k of ['bloom', 'exposure', 'aberr', 'grain', 'vignette', 'glitch', 'scan', 'flash', 'fade']) {
const key = 'u' + k[0].toUpperCase() + k.slice(1);
if (params[k] !== undefined) u[key].value = params[k];
}
u.uTime.value = params.time;
u.uFrame.value = params.frame;
pass(finalMat, null);
}
return { render };
}
+154
View File
@@ -0,0 +1,154 @@
// The film's clock. Everything visual and audible keys off these numbers, so the
// renderer and the score (audio/score.py reads cues.json, exported from here)
// stay frame-locked. Pure module: no DOM, safe to import from Node.
export const FPS = 30;
export const DURATION = 95;
export const FRAMES = FPS * DURATION;
export const T = {
// ACT 0 — the void
cursor: 0.9,
boot: 1.4,
collapse: 4.9,
seed: 5.35,
beat1: 6.1,
beat2: 7.35,
dishDraw: 6.3,
dishDone: 8.3,
aperture: 7.35,
// ACT 1 — the eye opens; the culture grows and turns out to be drawing the planet
growStart: 8.7,
coastReached: 13.4,
growEnd: 15.6,
substrate: 16.0,
// ACT 2 — indexing the surface, accelerating until it is whipping round the world
indexStart: 17.0,
zWave: [17.0, 18.5, 19.8], // z3, z4, z5 quadtree waves
capStart: 18.0,
capRapid: 22.6,
flurry: 23.6,
flurryEnd: 29.4,
capFull: 999, // v1's full-frame extraction now lives in the city act
indexed: 30.0,
// ACT 3 — global ingestion
air: 30.6,
sea: 32.0,
mil: 33.4,
orbit: 34.6,
quake: 36.6,
fire: 38.0,
wind: 39.4,
cable: 41.0,
descend: 42.6,
// ACT 4 — down into the world
terrain: 43.6,
nyc: 49.0,
boston: 57.0,
lax: 61.8,
bragg: 66.4,
cctv: 71.4,
cctvWall: 71.4,
rapid: 72.8,
cctvOut: 76.4,
// ACT 5 — saturation; the twin outgrows the dish
saturate: 76.6,
arcs: 77.3,
twin: 79.1,
dishBreak: 82.3,
omni: 82.9,
cut: 84.4,
// ACT 6 — resolve
title: 85.1,
eyeOpen: 85.9,
wordmark: 87.2,
tagline: 88.9,
online: 999,
fadeOut: 93.0,
};
// Feeds as they register in the stack (ACT 3). `key` matches a layer.
export const FEEDS = [
{ key: 'air', t: T.air, code: 'ADS-B', label: 'AIRCRAFT', src: 'OPENSKY NETWORK' },
{ key: 'sea', t: T.sea, code: 'AIS', label: 'VESSELS', src: 'KYSTVERKET · DIGITRAFFIC' },
{ key: 'mil', t: T.mil, code: 'ADS-B', label: 'MILITARY', src: 'ADSB.LOL · ADSB.FI' },
{ key: 'orbit', t: T.orbit, code: 'TLE', label: 'SATELLITES', src: 'CELESTRAK' },
{ key: 'quake', t: T.quake, code: 'SEIS', label: 'EARTHQUAKES', src: 'USGS' },
{ key: 'fire', t: T.fire, code: 'IR', label: 'ACTIVE FIRES', src: 'NASA FIRMS' },
{ key: 'wind', t: T.wind, code: 'NWP', label: 'WIND · CLOUD', src: 'NOAA GFS · GIBS' },
{ key: 'cable', t: T.cable, code: 'GEO', label: 'SUBSEA CABLES', src: 'TELEGEOGRAPHY' },
];
// ACT 4 — the cities, in order
export const CITIES = [
{ slug: 'everest', t0: T.terrain, t1: T.nyc, name: 'MOUNT EVEREST', sub: 'KHUMBU · NEPAL / TIBET', lat: 27.9881, lon: 86.925 },
{ slug: 'nyc', t0: T.nyc, t1: T.boston, name: 'NEW YORK', sub: 'MIDTOWN · MANHATTAN', lat: 40.74, lon: -73.99 },
{ slug: 'boston', t0: T.boston, t1: T.lax, name: 'BOSTON', sub: 'MBTA · GTFS REALTIME', lat: 42.3555, lon: -71.0605 },
{ slug: 'lax', t0: T.lax, t1: T.bragg, name: 'LOS ANGELES INTL', sub: 'KLAX · AIRSIDE', lat: 33.9425, lon: -118.4081 },
{ slug: 'bragg', t0: T.bragg, t1: T.cctv, name: 'FORT BRAGG', sub: 'NORTH CAROLINA · POPE FIELD', lat: 35.158, lon: -79.009 },
];
// Synthetic captures in ACT 2: the first few take the slow "enhance" path, the
// rest fire rapidly into the contact sheet, then one blows up full frame.
export const CAPTURE_SLOW = 3;
export function captureTimes(count) {
const out = [];
const slot = [T.capStart, T.capStart + 1.6, T.capStart + 3.1];
for (let i = 0; i < Math.min(CAPTURE_SLOW, count); i++) out.push(slot[i]);
const rapidN = count - CAPTURE_SLOW;
// Accelerating rapid-fire: intervals shrink from ~0.32 s to ~0.11 s.
let t = T.capRapid;
for (let i = 0; i < rapidN; i++) {
out.push(t);
const k = i / Math.max(1, rapidN - 1);
t += 0.12 - 0.08 * k;
}
return out;
}
// ------------------------------------------------------------------ helpers
export const clamp = (x, a = 0, b = 1) => Math.min(b, Math.max(a, x));
export const lerp = (a, b, t) => a + (b - a) * t;
export const ramp = (t, a, b) => clamp((t - a) / (b - a));
export const smooth = (t, a, b) => {
const x = ramp(t, a, b);
return x * x * (3 - 2 * x);
};
export const easeInOut = (x) => (x < 0.5 ? 4 * x * x * x : 1 - Math.pow(-2 * x + 2, 3) / 2);
export const easeOut = (x) => 1 - Math.pow(1 - x, 3);
export const easeIn = (x) => x * x * x;
export const expoOut = (x) => (x >= 1 ? 1 : 1 - Math.pow(2, -10 * x));
export const expoInOut = (x) =>
x <= 0 ? 0 : x >= 1 ? 1 : x < 0.5 ? Math.pow(2, 20 * x - 10) / 2 : (2 - Math.pow(2, -20 * x + 10)) / 2;
// Window: 0 before a, ramps up over `inDur`, holds, ramps down to 0 ending at b.
export function win(t, a, b, inDur = 0.3, outDur = 0.3) {
return Math.min(smooth(t, a, a + inDur), 1 - smooth(t, b - outDur, b));
}
export function mulberry32(seed) {
let a = seed >>> 0;
return function () {
a = (a + 0x6d2b79f5) >>> 0;
let t = a;
t = Math.imul(t ^ (t >>> 15), t | 1);
t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}
export function hash1(n) {
const s = Math.sin(n * 127.1 + 311.7) * 43758.5453;
return s - Math.floor(s);
}
// Counter that eases toward `to` and keeps visibly ticking while it does.
export function count(t, a, b, from, to, curve = expoOut) {
return Math.round(lerp(from, to, curve(ramp(t, a, b))));
}
+79
View File
@@ -0,0 +1,79 @@
"""Render city/<slug>/check.png: ground imagery (left) and imagery + OSM overlay (right) for alignment QA.
usage: python3 -I tools/city/check.py nyc boston [--crop x y size]
"""
import sys, os, json
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from PIL import Image, ImageDraw
from geo import CITIES, OUT, CACHE
ROADC = [(255, 60, 60), (255, 150, 40), (255, 220, 40), (200, 255, 80), (255, 255, 255), (160, 160, 160)]
def render(slug, S=1400, crop=None):
d = os.path.join(OUT, slug)
h = CITIES[slug]['half']
cx, cy, cs = crop if crop else (0, 0, h) # center and half-size of view in meters
g = Image.open(os.path.join(d, 'ground.jpg')).convert('RGB')
G = g.size[0]
# crop region of ground in pixels
l = (cx - cs + h) / (2 * h) * G
t = (h - (cy + cs)) / (2 * h) * G
r = (cx + cs + h) / (2 * h) * G
b = (h - (cy - cs)) / (2 * h) * G
base = g.crop((int(l), int(t), int(r), int(b))).resize((S, S), Image.LANCZOS)
def P(x, y):
return ((x - (cx - cs)) / (2 * cs) * S, ((cy + cs) - y) / (2 * cs) * S)
ov = base.copy()
dr = ImageDraw.Draw(ov, 'RGBA')
wp = os.path.join(d, 'water.json')
if os.path.exists(wp):
for poly in json.load(open(wp))['polys']:
for k, ring in enumerate(poly):
pts = [P(ring[i], ring[i + 1]) for i in range(0, len(ring), 2)]
if len(pts) >= 3:
dr.polygon(pts, outline=(60, 160, 255, 255), fill=(30, 90, 255, 70) if k == 0 else None)
for bld in json.load(open(os.path.join(d, 'buildings.json')))['b']:
hgt = bld[0]
pts = [P(bld[i], bld[i + 1]) for i in range(2, len(bld), 2)]
a = min(255, 60 + int(hgt))
col = (0, 255, 200, 255) if hgt >= 100 else (255, 0, 220, 255)
dr.polygon(pts, outline=col, fill=(col[0], col[1], col[2], 40 if hgt < 100 else 90))
for rd in json.load(open(os.path.join(d, 'roads.json')))['roads']:
pts = [P(rd[i], rd[i + 1]) for i in range(3, len(rd), 2)]
dr.line(pts, fill=ROADC[rd[0]] + (230,), width=2 if rd[0] < 3 else 1)
rp = os.path.join(d, 'rail.json')
if os.path.exists(rp):
for ln in json.load(open(rp))['lines']:
c = ln[0]
col = tuple(int(c[i:i + 2], 16) for i in (1, 3, 5)) + (255,)
pts = [P(ln[i], ln[i + 1]) for i in range(1, len(ln), 2)]
dr.line(pts, fill=col, width=3)
ep = os.path.join(d, 'extra.json')
if os.path.exists(ep):
for x, y, nm in json.load(open(ep))['helipads']:
px, py = P(x, y)
dr.ellipse((px - 7, py - 7, px + 7, py + 7), outline=(255, 255, 0, 255), width=3)
out = Image.new('RGB', (S * 2 + 10, S), (0, 0, 0))
out.paste(base, (0, 0))
out.paste(ov, (S + 10, 0))
return out
if __name__ == '__main__':
args = [a for a in sys.argv[1:] if not a.startswith('--')]
crop = None
if '--crop' in sys.argv:
i = sys.argv.index('--crop')
crop = tuple(float(v) for v in sys.argv[i + 1:i + 4])
args = [a for a in args if a not in sys.argv[i + 1:i + 4]]
for slug in args or list(CITIES):
im = render(slug, crop=crop)
if crop:
p = os.path.join(CACHE, f'{slug}_check_crop_{int(crop[0])}_{int(crop[1])}_{int(crop[2])}.png')
else:
p = os.path.join(OUT, slug, 'check.png')
im.save(p, optimize=True)
print('wrote', p)
+107
View File
@@ -0,0 +1,107 @@
"""Esri World Imagery mosaic -> local ENU square, 4096^2 + 2048^2 JPEG.
usage: python3 -I tools/city/fetch_imagery.py nyc boston [--zoom 17]
"""
import sys, os, math, io, time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from concurrent.futures import ThreadPoolExecutor
import numpy as np
import requests
from PIL import Image
from scipy.ndimage import map_coordinates
from geo import CITIES, OUT, CACHE, Frame
URL = 'https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{z}/{y}/{x}'
TS = 256
N = 4096
SS = 2 # supersample factor before Lanczos downsample
def merc_px(lat, lon, z):
s = TS * 2 ** z
x = (lon + 180.0) / 360.0 * s
lr = np.radians(lat)
y = (1.0 - np.log(np.tan(lr) + 1.0 / np.cos(lr)) / math.pi) / 2.0 * s
return x, y
sess = requests.Session()
sess.headers['User-Agent'] = 'gods-eye-view-intro/1.0 (film asset build)'
def get_tile(z, x, y):
d = os.path.join(CACHE, 'tiles', str(z), str(x))
os.makedirs(d, exist_ok=True)
p = os.path.join(d, f'{y}.jpg')
if os.path.exists(p) and os.path.getsize(p) > 0:
return p, 'cache'
last = None
for attempt in range(5):
try:
r = sess.get(URL.format(z=z, x=x, y=y), timeout=60)
if r.status_code == 200 and r.content[:2] in (b'\xff\xd8', b'\x89P'):
with open(p, 'wb') as f:
f.write(r.content)
return p, 'net'
last = f'HTTP {r.status_code} {r.headers.get("content-type")}'
except Exception as e: # noqa
last = str(e)[:120]
time.sleep(1.5 * (attempt + 1))
raise RuntimeError(f'tile {z}/{x}/{y} failed: {last}')
def build(slug, z):
c = CITIES[slug]
F = Frame(**c)
h = F.half
s, w, n, e = F.bbox(h * 1.02)
x0, y0 = merc_px(n, w, z)
x1, y1 = merc_px(s, e, z)
tx0, ty0 = int(x0 // TS), int(y0 // TS)
tx1, ty1 = int(x1 // TS), int(y1 // TS)
tiles = [(tx, ty) for ty in range(ty0, ty1 + 1) for tx in range(tx0, tx1 + 1)]
print(f'== {slug} z{z}: {tx1-tx0+1}x{ty1-ty0+1} = {len(tiles)} tiles')
with ThreadPoolExecutor(8) as ex:
res = list(ex.map(lambda t: get_tile(z, t[0], t[1]), tiles))
print(f' net {sum(1 for r in res if r[1]=="net")} cache {sum(1 for r in res if r[1]=="cache")}')
W, H = (tx1 - tx0 + 1) * TS, (ty1 - ty0 + 1) * TS
mos = np.zeros((H, W, 3), np.uint8)
for (tx, ty), (p, _) in zip(tiles, res):
im = np.asarray(Image.open(p).convert('RGB'))
mos[(ty - ty0) * TS:(ty - ty0 + 1) * TS, (tx - tx0) * TS:(tx - tx0 + 1) * TS] = im
M = N * SS
t = (np.arange(M) + 0.5) / M
xs = -h + 2 * h * t # east
ys = h - 2 * h * t # north, row 0 at top
lon = F.lon0 + xs / F.kx
lat = F.lat0 + ys / F.ky
px, _ = merc_px(np.full_like(lon, F.lat0), lon, z)
_, py = merc_px(lat, np.full_like(lat, F.lon0), z)
px = px - tx0 * TS - 0.5 # pixel centers
py = py - ty0 * TS - 0.5
PX, PY = np.meshgrid(px, py)
out = np.empty((M, M, 3), np.uint8)
for ch in range(3):
out[..., ch] = np.clip(map_coordinates(mos[..., ch].astype(np.float32), [PY, PX], order=1,
mode='nearest'), 0, 255).astype(np.uint8)
del PX, PY
img = Image.fromarray(out)
if SS != 1:
img = img.resize((N, N), Image.LANCZOS)
d = os.path.join(OUT, slug)
os.makedirs(d, exist_ok=True)
img.save(os.path.join(d, 'ground.jpg'), quality=90, optimize=True)
img.resize((2048, 2048), Image.LANCZOS).save(os.path.join(d, 'ground_2k.jpg'), quality=90, optimize=True)
img.resize((768, 768), Image.LANCZOS).save(os.path.join(CACHE, f'{slug}_ground_preview.jpg'), quality=85)
mpp_src = 156543.03392 * math.cos(math.radians(F.lat0)) / 2 ** z
print(f' wrote ground.jpg {os.path.getsize(os.path.join(d, "ground.jpg"))/1e6:.1f}MB; '
f'source {mpp_src:.2f} m/px, output {2*h/N:.2f} m/px')
if __name__ == '__main__':
args = [a for a in sys.argv[1:] if not a.startswith('--')]
z = 17
if '--zoom' in sys.argv:
z = int(sys.argv[sys.argv.index('--zoom') + 1])
for slug in args or list(CITIES):
build(slug, z)
+495
View File
@@ -0,0 +1,495 @@
"""Fetch OSM buildings / roads / water / extras for the city scenes and write city/<slug>/*.json.
usage: python3 -I tools/city/fetch_osm.py nyc boston
"""
import sys, os, math, re, json
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from geo import (CITIES, OUT, Frame, overpass, rdp, rdp_ring, area, centroid, open_ring,
assemble_rings, clip_polyline, clip_polygon, flat, r1, write_json, dedupe)
FT = 0.3048
def parse_len(v):
"""Parse an OSM length tag to meters. Returns None if unparseable."""
if v is None:
return None
s = str(v).strip().lower().replace(',', '.')
s = s.split(';')[0].strip()
m = re.match(r"^(\d+(?:\.\d+)?)\s*'\s*(?:(\d+(?:\.\d+)?)\s*\"?)?$", s) # 12'6"
if m:
return float(m.group(1)) * FT + (float(m.group(2)) * 0.0254 if m.group(2) else 0)
m = re.match(r'^(-?\d+(?:\.\d+)?)\s*(m|meters?|metres?|ft|feet|foot)?$', s)
if not m:
return None
val = float(m.group(1))
u = m.group(2) or 'm'
if u in ('ft', 'feet', 'foot'):
val *= FT
return val
def parse_num(v):
if v is None:
return None
m = re.match(r'^\s*(\d+(?:\.\d+)?)', str(v).replace(',', '.'))
return float(m.group(1)) if m else None
HOUSE = {'house', 'detached', 'semidetached_house', 'terrace', 'bungalow', 'cabin', 'shed', 'garage',
'garages', 'hut', 'carport', 'static_caravan'}
def building_height(t):
"""Returns (height, min_height, source) where source in h/lv/def."""
h = parse_len(t.get('height'))
src = 'h'
if h is None or h <= 0:
lv = parse_num(t.get('building:levels'))
if lv is not None and lv > 0:
h = lv * 3.2 + (parse_num(t.get('roof:levels')) or 0) * 3.2
src = 'lv'
else:
b = t.get('building', 'yes')
h = 7.0 if b in HOUSE else 12.0
if b in ('shed', 'garage', 'garages', 'carport', 'hut'):
h = 3.5
src = 'def'
mh = parse_len(t.get('min_height'))
if mh is None:
ml = parse_num(t.get('building:min_level'))
mh = ml * 3.2 if ml else 0.0
mh = max(0.0, mh)
if mh >= h:
mh = 0.0
return h, mh, src
def ll_list(geom):
return [(g['lat'], g['lon']) for g in geom if g is not None]
# ------------------------------------------------------------------ buildings
def buildings(slug, F):
h = F.half
s, w, n, e = F.bbox(h + 60)
# tile the bbox to keep each response manageable
T = 2 if h > 3000 else 1
elems = {}
for i in range(T):
for j in range(T):
ts = s + (n - s) * i / T
tn = s + (n - s) * (i + 1) / T
tw = w + (e - w) * j / T
te = w + (e - w) * (j + 1) / T
bb = f'{ts:.6f},{tw:.6f},{tn:.6f},{te:.6f}'
q = (f'[out:json][timeout:180];(way["building"]({bb});'
f'relation["building"]["type"="multipolygon"]({bb}););out geom;')
print(f' buildings tile {i},{j}')
d = overpass(q, f'{slug}_bld_{i}{j}')
for el in d['elements']:
elems[(el['type'], el['id'])] = el
print(f' {len(elems)} building elements')
out = []
stats = dict(h=0, lv=0, df=0, dropped_small=0, dropped_out=0, rel_rings=0)
for (typ, _id), el in elems.items():
t = el.get('tags', {})
if t.get('building') in ('no', 'construction_site'):
continue
rings = []
if typ == 'way':
if 'geometry' not in el:
continue
g = ll_list(el['geometry'])
if len(g) >= 4 and g[0] == g[-1]:
rings.append(g)
else:
outers = [ll_list(m['geometry']) for m in el.get('members', [])
if m['type'] == 'way' and m.get('role') == 'outer' and 'geometry' in m]
rings = assemble_rings(outers)
stats['rel_rings'] += len(rings)
hgt, mh, src = building_height(t)
for g in rings:
pts = open_ring([F.xy(la, lo) for la, lo in g])
pts = rdp_ring(pts, 0.5)
if len(pts) < 3 or abs(area(pts)) < 8:
stats['dropped_small'] += 1
continue
cx, cy = centroid(pts)
if abs(cx) > h or abs(cy) > h:
stats['dropped_out'] += 1
continue
stats[{'h': 'h', 'lv': 'lv', 'def': 'df'}[src]] += 1
out.append((cx * cx + cy * cy, [r1(hgt), r1(mh)] + flat(pts)))
out.sort(key=lambda a: a[0])
b = [o[1] for o in out]
write_json(os.path.join(OUT, slug, 'buildings.json'), {'count': len(b), 'b': b})
hs = sorted(x[0] for x in b)
print(f' buildings {len(b)} height-src tag={stats["h"]} levels={stats["lv"]} default={stats["df"]} '
f'dropped small={stats["dropped_small"]} outside={stats["dropped_out"]} tallest={hs[-5:]}')
return dict(count=len(b), height_from_tag=stats['h'], height_from_levels=stats['lv'],
height_default=stats['df'], tallest_m=hs[-1] if hs else None)
# ------------------------------------------------------------------ roads
CLS = {'motorway': 0, 'trunk': 0, 'primary': 1, 'secondary': 2, 'tertiary': 3,
'residential': 4, 'unclassified': 4, 'service': 5}
DEF_LANES = {0: 3, 1: 2, 2: 2, 3: 2, 4: 1, 5: 1}
def plen(pts):
return sum(math.hypot(pts[i + 1][0] - pts[i][0], pts[i + 1][1] - pts[i][1]) for i in range(len(pts) - 1))
def roads(slug, F):
h = F.half
s, w, n, e = F.bbox(h + 100)
bb = f'{s:.6f},{w:.6f},{n:.6f},{e:.6f}'
q = (f'[out:json][timeout:180];way["highway"~"^(motorway|trunk|primary|secondary|tertiary|residential|'
f'unclassified|service|motorway_link|trunk_link|primary_link|secondary_link|tertiary_link)$"]({bb});out geom;')
print(' roads')
d = overpass(q, f'{slug}_roads')
out = []
cnt = [0] * 6
tunnels = 0
short_service = 0
for el in d['elements']:
t = el.get('tags', {})
hw = t['highway']
if t.get('area') == 'yes':
continue
if t.get('tunnel') in ('yes', 'covered') or t.get('layer', '0').startswith('-') and t.get('tunnel'):
tunnels += 1
continue
pts = [F.xy(la, lo) for la, lo in ll_list(el.get('geometry', []))]
pts = dedupe(pts)
if len(pts) < 2:
continue
if hw.endswith('_link'):
c = 5
else:
c = CLS[hw]
if hw == 'service' and plen(pts) <= 80:
short_service += 1
continue
ow = t.get('oneway', '')
oneway = 0
if ow in ('yes', '1', 'true'):
oneway = 1
elif ow == '-1':
oneway = 1
pts = pts[::-1]
elif ow == 'no':
oneway = 0
elif hw in ('motorway', 'motorway_link') or t.get('junction') in ('roundabout', 'circular'):
oneway = 1
ln = parse_num(t.get('lanes'))
lanes = int(ln) if ln else DEF_LANES[c]
for piece in clip_polyline(pts, -h, -h, h, h):
piece = rdp(piece, 1.0)
if len(piece) < 2 or plen(piece) < 1:
continue
out.append([c, oneway, lanes] + flat(piece))
cnt[c] += 1
write_json(os.path.join(OUT, slug, 'roads.json'), {'roads': out})
print(f' roads {len(out)} by class {cnt} (dropped tunnels {tunnels}, short service {short_service})')
return dict(count=len(out), by_class=cnt, dropped_tunnel_ways=tunnels, dropped_short_service=short_service)
# ------------------------------------------------------------------ water
def coast_water(slug, F, h):
"""Build water polygons inside the square from natural=coastline (land left, water right)."""
s, w, n, e = F.bbox(h * 1.6)
bb = f'{s:.6f},{w:.6f},{n:.6f},{e:.6f}'
q = f'[out:json][timeout:180];way["natural"="coastline"]({bb});out geom;'
print(' coastline')
d = overpass(q, f'{slug}_coast')
ways = [[F.xy(la, lo) for la, lo in ll_list(el['geometry'])] for el in d['elements'] if 'geometry' in el]
# directional merge into chains
chains = []
pool = [wy for wy in ways if len(wy) >= 2]
while pool:
cur = pool.pop()
grown = True
while grown and cur[0] != cur[-1]:
grown = False
for i, wy in enumerate(pool):
if wy[0] == cur[-1]:
cur = cur + wy[1:]
elif wy[-1] == cur[0]:
cur = wy + cur[1:]
else:
continue
pool.pop(i)
grown = True
break
chains.append(cur)
eps = 1e-6
def on_edge(p):
return abs(abs(p[0]) - h) < 1e-3 or abs(abs(p[1]) - h) < 1e-3
def spos(p): # clockwise perimeter coordinate from top-left
x, y = p
if abs(y - h) < 1e-3:
return x + h
if abs(x - h) < 1e-3:
return 2 * h + (h - y)
if abs(y + h) < 1e-3:
return 4 * h + (h - x)
return 6 * h + (y + h)
corners = [(0.0, (-h, h)), (2 * h, (h, h)), (4 * h, (h, -h)), (6 * h, (-h, -h))]
pieces, islands, bad = [], [], 0
for ch in chains:
closed = ch[0] == ch[-1]
inside_all = all(abs(x) <= h and abs(y) <= h for x, y in ch)
if closed and inside_all:
islands.append(open_ring(ch))
continue
for pc in clip_polyline(ch, -h, -h, h, h):
if on_edge(pc[0]) and on_edge(pc[-1]):
pieces.append(pc)
else:
bad += 1
polys = []
if pieces:
used = [False] * len(pieces)
starts = [spos(p[0]) for p in pieces]
for k in range(len(pieces)):
if used[k]:
continue
ring = []
cur = k
guard = 0
while True:
used[cur] = True
ring += pieces[cur]
se = spos(pieces[cur][-1])
# next entry clockwise from se
best, bd = None, None
for m in range(len(pieces)):
dd = (starts[m] - se) % (8 * h)
if bd is None or dd < bd:
best, bd = m, dd
# add corners passed between se and se+bd
for cs, cp in sorted(corners, key=lambda c: (c[0] - se) % (8 * h)):
if 0 < (cs - se) % (8 * h) < bd:
ring.append(cp)
cur = best
guard += 1
if cur == k or guard > len(pieces) + 2:
break
if used[cur]:
break
polys.append([dedupe(open_ring(ring))])
# attach islands (land holes) to the containing water polygon
def pip(pt, ring):
x, y = pt
c = False
for i in range(len(ring)):
x1, y1 = ring[i]
x2, y2 = ring[i - 1]
if (y1 > y) != (y2 > y) and x < (x2 - x1) * (y - y1) / (y2 - y1) + x1:
c = not c
return c
for isl in islands:
if abs(area(isl)) < 50:
continue
for p in polys:
if pip(isl[0], p[0]):
p.append(isl)
break
print(f' coastline chains {len(chains)} pieces {len(pieces)} islands {len(islands)} '
f'water polys {len(polys)} (unclosed pieces skipped {bad})')
return polys, bad
def water(slug, F):
h = F.half
s, w, n, e = F.bbox(h + 50)
bb = f'{s:.6f},{w:.6f},{n:.6f},{e:.6f}'
q = (f'[out:json][timeout:180];(way["natural"="water"]({bb});relation["natural"="water"]({bb});'
f'way["waterway"="riverbank"]({bb});relation["waterway"="riverbank"]({bb}););out geom;')
print(' water areas')
d = overpass(q, f'{slug}_water')
polys = []
for el in d['elements']:
t = el.get('tags', {})
if t.get('water') in ('pond', 'reflecting_pool', 'fountain', 'basin', 'reservoir') and el['type'] == 'way':
pass # keep small ponds too; they are real water
if el['type'] == 'way':
if 'geometry' not in el:
continue
g = ll_list(el['geometry'])
if g[0] != g[-1]:
continue
outers, inners = [g], []
else:
mem = [m for m in el.get('members', []) if m['type'] == 'way' and 'geometry' in m]
outers = assemble_rings([ll_list(m['geometry']) for m in mem if m.get('role') == 'outer'])
inners = assemble_rings([ll_list(m['geometry']) for m in mem if m.get('role') == 'inner'])
inn_xy = [open_ring([F.xy(*p) for p in r]) for r in inners]
for o in outers:
ring = open_ring([F.xy(*p) for p in o])
ring = clip_polygon(ring, -h, -h, h, h)
ring = rdp_ring(open_ring(ring), 1.0)
if len(ring) < 3 or abs(area(ring)) < 20:
continue
p = [ring]
for ir in inn_xy:
ir2 = rdp_ring(open_ring(clip_polygon(ir, -h, -h, h, h)), 1.0)
if len(ir2) >= 3 and abs(area(ir2)) >= 20:
p.append(ir2)
polys.append(('water', p))
n_area = len(polys)
cpolys, bad = coast_water(slug, F, h)
for p in cpolys:
p2 = [rdp_ring(r, 1.0) for r in p]
p2 = [r for r in p2 if len(r) >= 3]
if p2:
polys.append(('coast', p2))
out = [[flat(r) for r in p] for _, p in polys]
src = [s_ for s_, _ in polys]
write_json(os.path.join(OUT, slug, 'water.json'),
{'count': len(out), 'format': 'polys[i] = [outerRing, holeRing...], rings flat x,y', 'src': src,
'polys': out})
print(f' water polys {len(out)} (osm water areas {n_area}, coastline-derived {len(cpolys)})')
return dict(count=len(out), water_areas=n_area, coastline_polys=len(cpolys), coastline_unclosed_pieces=bad)
# ------------------------------------------------------------------ extras
KNOWN_HELI = ['Downtown Manhattan Heliport', 'West 30th Street Heliport', 'East 34th Street Heliport']
def nyc_extra(slug, F):
s, w, n, e = F.bbox(F.half + 2000)
bb = f'{s:.6f},{w:.6f},{n:.6f},{e:.6f}'
q = (f'[out:json][timeout:180];(nwr["aeroway"~"^(helipad|heliport)$"]({bb});'
f'nwr["name"~"heliport|helistop|helipad",i]({bb}););out center tags;')
print(' helipads')
d = overpass(q, f'{slug}_heli2')
pads = []
for el in d['elements']:
t = el.get('tags', {})
if el['type'] == 'node':
la, lo = el['lat'], el['lon']
else:
c = el.get('center')
if not c:
continue
la, lo = c['lat'], c['lon']
x, y = F.xy(la, lo)
name = t.get('name') or t.get('operator') or ''
kind = t.get('aeroway') or ('heliport' if 'heliport' in name.lower() else 'named')
pads.append(dict(x=r1(x), y=r1(y), name=name, kind=kind, ref=t.get('ref', ''), osm=f"{el['type']}/{el['id']}",
tags={k: v for k, v in t.items() if k in ('aeroway', 'faa', 'operator', 'man_made', 'amenity')},
inside=abs(x) <= F.half and abs(y) <= F.half))
pads.sort(key=lambda p: (p['kind'] != 'heliport', p['x'] ** 2 + p['y'] ** 2))
for p in pads:
print(f" {p['kind']:9s} {p['x']:8.1f} {p['y']:8.1f} {'in ' if p['inside'] else 'OUT'} {p['name']}")
found = {k: [p for p in pads if p['kind'] == 'heliport' and k.split()[0].lower() in p['name'].lower()
and 'heliport' in p['name'].lower()] for k in KNOWN_HELI}
helipads = [[p['x'], p['y'], p['name'] or (f"{p['kind']} {p['ref']}".strip())] for p in pads]
write_json(os.path.join(OUT, slug, 'extra.json'),
{'helipads': helipads,
'detail': pads,
'note': 'aeroway=helipad|heliport from OSM in a bbox 2 km wider than the scene; '
'detail[].inside says whether the point is inside the scene square'})
return dict(helipad_points=len(pads), named_heliports={k: (v[0]['x'], v[0]['y'], v[0]['inside']) if v else None
for k, v in found.items()})
def boston_rail(slug, F):
h = F.half
s, w, n, e = F.bbox(h + 100)
bb = f'{s:.6f},{w:.6f},{n:.6f},{e:.6f}'
q = (f'[out:json][timeout:180];(relation["route"~"^(subway|light_rail|train|tram)$"]({bb});'
f'way["railway"~"^(subway|light_rail|rail|tram)$"]({bb}););out geom;')
print(' rail')
d = overpass(q, f'{slug}_rail')
ways = {el['id']: el for el in d['elements'] if el['type'] == 'way'}
rels = [el for el in d['elements'] if el['type'] == 'relation']
lines, seen = [], set()
rel_names = []
for r in rels:
t = r.get('tags', {})
col = (t.get('colour') or t.get('color') or '').strip()
if not re.match(r'^#?[0-9a-fA-F]{6}$', col):
col = {'subway': '#888888', 'light_rail': '#00843D', 'tram': '#00843D', 'train': '#80276C'}[t['route']]
if not col.startswith('#'):
col = '#' + col
col = col.upper()
rel_names.append(f"{t.get('route')}:{t.get('name', t.get('ref', ''))}:{col}")
for m in r.get('members', []):
if m['type'] != 'way' or 'geometry' not in m or m.get('role') in ('platform', 'stop'):
continue
wt = ways.get(m['ref'], {}).get('tags', {})
if wt.get('public_transport') or wt.get('railway') == 'platform':
continue
key = (m['ref'], col)
if key in seen:
continue
seen.add(key)
pts = [F.xy(la, lo) for la, lo in ll_list(m['geometry'])]
for pc in clip_polyline(dedupe(pts), -h, -h, h, h):
pc = rdp(pc, 1.0)
if len(pc) >= 2:
lines.append([col] + flat(pc))
covered = {k[0] for k in seen}
extra = 0
for wid, el in ways.items():
t = el.get('tags', {})
if wid in covered or t.get('service') or 'geometry' not in el:
continue
pts = [F.xy(la, lo) for la, lo in ll_list(el['geometry'])]
for pc in clip_polyline(dedupe(pts), -h, -h, h, h):
pc = rdp(pc, 1.0)
if len(pc) >= 2:
lines.append(['#7F7F7F'] + flat(pc))
extra += 1
write_json(os.path.join(OUT, slug, 'rail.json'),
{'lines': lines, 'routes': sorted(set(rel_names)),
'note': 'OSM route relations (subway/light_rail/train/tram) colour tags; #7F7F7F = railway way not in any route relation'})
print(f' rail lines {len(lines)} from {len(rels)} route relations (+{extra} unassigned way pieces)')
for nm in sorted(set(rel_names)):
print(' ', nm)
return dict(polylines=len(lines), route_relations=len(rels), unassigned_pieces=extra)
def main():
slugs = sys.argv[1:] or list(CITIES)
report = {}
for slug in slugs:
c = CITIES[slug]
F = Frame(**c)
print(f'== {slug}')
write_json(os.path.join(OUT, slug, 'meta.json'),
{'slug': slug, 'name': c['name'], 'lat0': c['lat0'], 'lon0': c['lon0'], 'half': c['half'],
'frame': 'ENU m',
'projection': 'x=(lon-lon0)*cos(lat0)*111320, y=(lat-lat0)*110540',
'source': 'OpenStreetMap contributors (ODbL) via Overpass; imagery Esri World Imagery'})
r = {}
r['buildings'] = buildings(slug, F)
r['roads'] = roads(slug, F)
try:
r['water'] = water(slug, F)
except Exception as ex: # noqa
print(' WATER FAILED', ex)
r['water'] = f'failed: {ex}'
if slug == 'nyc':
r['extra'] = nyc_extra(slug, F)
if slug == 'boston':
r['rail'] = boston_rail(slug, F)
report[slug] = r
print(json.dumps(report, indent=1))
with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'cache', 'osm_report.json'), 'w') as f:
json.dump(report, f, indent=1)
if __name__ == '__main__':
main()
+288
View File
@@ -0,0 +1,288 @@
"""Shared helpers for the city scene geodata fetchers (ENU frame, Overpass, geometry)."""
import json, math, os, time, hashlib
import requests
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.abspath(os.path.join(HERE, '..', '..'))
OUT = os.path.join(ROOT, 'city')
CACHE = os.path.join(HERE, 'cache')
os.makedirs(CACHE, exist_ok=True)
CITIES = {
'nyc': dict(name='New York — Midtown / Chelsea / Lower Manhattan', lat0=40.7400, lon0=-73.9900, half=3500),
'boston': dict(name='Boston — Downtown / Back Bay / Cambridge / Seaport', lat0=42.3555, lon0=-71.0605, half=3200),
}
OVERPASS = [
'https://overpass-api.de/api/interpreter',
'https://overpass.kumi.systems/api/interpreter',
'https://overpass.private.coffee/api/interpreter',
'https://maps.mail.ru/osm/tools/overpass/api/interpreter',
]
class Frame:
def __init__(self, lat0, lon0, half, **_):
self.lat0, self.lon0, self.half = lat0, lon0, half
self.kx = math.cos(math.radians(lat0)) * 111320.0
self.ky = 110540.0
def xy(self, lat, lon):
return ((lon - self.lon0) * self.kx, (lat - self.lat0) * self.ky)
def ll(self, x, y):
return (self.lat0 + y / self.ky, self.lon0 + x / self.kx)
def bbox(self, half=None):
h = self.half if half is None else half
s, w = self.ll(-h, -h)
n, e = self.ll(h, h)
return s, w, n, e
def overpass(query, tag):
"""Run an Overpass query, caching the raw JSON response under cache/osm_<tag>.json."""
path = os.path.join(CACHE, f'osm_{tag}.json')
if os.path.exists(path):
with open(path) as f:
return json.load(f)
last = None
for attempt in range(8):
for url in OVERPASS:
try:
t = time.time()
r = requests.post(url, data={'data': query}, timeout=400)
if r.status_code != 200:
last = f'{url} HTTP {r.status_code}'
print(' overpass fail', last)
continue
d = r.json()
if 'remark' in d and 'error' in d.get('remark', '').lower():
last = f'{url} remark {d["remark"][:200]}'
print(' overpass remark', last)
continue
print(f' overpass ok {url} {len(r.content)/1e6:.1f}MB {time.time()-t:.0f}s {len(d["elements"])} el')
with open(path, 'w') as f:
json.dump(d, f)
return d
except Exception as e: # noqa
last = f'{url} {type(e).__name__}: {str(e)[:200]}'
print(' overpass fail', last)
time.sleep(15 + 15 * attempt)
raise RuntimeError('all overpass endpoints failed: ' + str(last))
# ---------------- geometry ----------------
def rdp(pts, eps):
"""Douglas-Peucker on an open polyline (list of (x,y))."""
n = len(pts)
if n < 3:
return list(pts)
keep = [False] * n
keep[0] = keep[-1] = True
stack = [(0, n - 1)]
while stack:
a, b = stack.pop()
ax, ay = pts[a]
bx, by = pts[b]
dx, dy = bx - ax, by - ay
L = math.hypot(dx, dy)
dmax, idx = -1.0, -1
for i in range(a + 1, b):
px, py = pts[i]
if L < 1e-9:
d = math.hypot(px - ax, py - ay)
else:
d = abs(dx * (ay - py) - dy * (ax - px)) / L
if d > dmax:
dmax, idx = d, i
if dmax > eps and idx > 0:
keep[idx] = True
stack.append((a, idx))
stack.append((idx, b))
return [p for p, k in zip(pts, keep) if k]
def rdp_ring(ring, eps):
"""Douglas-Peucker for a closed ring given without the repeated closing vertex."""
n = len(ring)
if n < 4:
return list(ring)
# split at vertex 0 and the vertex farthest from it
x0, y0 = ring[0]
far = max(range(n), key=lambda i: (ring[i][0] - x0) ** 2 + (ring[i][1] - y0) ** 2)
a = rdp(ring[0:far + 1], eps)
b = rdp(ring[far:] + [ring[0]], eps)
return a[:-1] + b[:-1]
def area(ring):
s = 0.0
n = len(ring)
for i in range(n):
x1, y1 = ring[i]
x2, y2 = ring[(i + 1) % n]
s += x1 * y2 - x2 * y1
return s / 2.0
def centroid(ring):
a = area(ring)
if abs(a) < 1e-9:
xs = [p[0] for p in ring]
ys = [p[1] for p in ring]
return sum(xs) / len(xs), sum(ys) / len(ys)
cx = cy = 0.0
n = len(ring)
for i in range(n):
x1, y1 = ring[i]
x2, y2 = ring[(i + 1) % n]
c = x1 * y2 - x2 * y1
cx += (x1 + x2) * c
cy += (y1 + y2) * c
return cx / (6 * a), cy / (6 * a)
def dedupe(pts):
out = []
for p in pts:
if not out or p != out[-1]:
out.append(p)
return out
def open_ring(pts):
pts = dedupe(pts)
if len(pts) > 1 and pts[0] == pts[-1]:
pts = pts[:-1]
return pts
def assemble_rings(ways):
"""Join way coordinate lists (lists of (lat,lon) tuples) into closed rings by matching endpoints."""
rings = []
pool = [list(w) for w in ways if len(w) >= 2]
while pool:
cur = pool.pop()
changed = True
while cur[0] != cur[-1] and changed:
changed = False
for i, w in enumerate(pool):
if w[0] == cur[-1]:
cur = cur + w[1:]
elif w[-1] == cur[-1]:
cur = cur + w[::-1][1:]
elif w[-1] == cur[0]:
cur = w + cur[1:]
elif w[0] == cur[0]:
cur = w[::-1] + cur[1:]
else:
continue
pool.pop(i)
changed = True
break
if cur[0] == cur[-1] and len(cur) >= 4:
rings.append(cur)
return rings
def clip_polyline(pts, xmin, ymin, xmax, ymax):
"""Clip a polyline to a rectangle (Liang-Barsky per segment); returns list of pieces."""
pieces, cur = [], []
def inside(p):
return xmin <= p[0] <= xmax and ymin <= p[1] <= ymax
for i in range(len(pts) - 1):
(x1, y1), (x2, y2) = pts[i], pts[i + 1]
dx, dy = x2 - x1, y2 - y1
t0, t1 = 0.0, 1.0
ok = True
for p, q in ((-dx, x1 - xmin), (dx, xmax - x1), (-dy, y1 - ymin), (dy, ymax - y1)):
if p == 0:
if q < 0:
ok = False
break
else:
t = q / p
if p < 0:
if t > t1:
ok = False
break
t0 = max(t0, t)
else:
if t < t0:
ok = False
break
t1 = min(t1, t)
if not ok:
if cur:
pieces.append(cur)
cur = []
continue
a = (x1, y1) if t0 == 0.0 else (x1 + t0 * dx, y1 + t0 * dy)
b = (x2, y2) if t1 == 1.0 else (x1 + t1 * dx, y1 + t1 * dy)
if not cur:
cur = [a, b]
else:
if cur[-1] != a:
pieces.append(cur)
cur = [a, b]
else:
cur.append(b)
if t1 < 1.0:
pieces.append(cur)
cur = []
if cur:
pieces.append(cur)
return [dedupe(p) for p in pieces if len(dedupe(p)) >= 2]
def clip_polygon(ring, xmin, ymin, xmax, ymax):
"""Sutherland-Hodgman clip of a ring to a rectangle."""
def clip(poly, inside, inter):
out = []
n = len(poly)
for i in range(n):
cur, prev = poly[i], poly[i - 1]
if inside(cur):
if not inside(prev):
out.append(inter(prev, cur))
out.append(cur)
elif inside(prev):
out.append(inter(prev, cur))
return out
def ix(xc):
return lambda p, q: (xc, p[1] + (q[1] - p[1]) * (xc - p[0]) / (q[0] - p[0]))
def iy(yc):
return lambda p, q: (p[0] + (q[0] - p[0]) * (yc - p[1]) / (q[1] - p[1]), yc)
poly = list(ring)
for ins, inter in ((lambda p: p[0] >= xmin, ix(xmin)), (lambda p: p[0] <= xmax, ix(xmax)),
(lambda p: p[1] >= ymin, iy(ymin)), (lambda p: p[1] <= ymax, iy(ymax))):
if not poly:
break
poly = clip(poly, ins, inter)
return dedupe(poly)
def r1(v):
return round(v, 1)
def flat(pts):
out = []
for x, y in pts:
out.append(r1(x))
out.append(r1(y))
return out
def write_json(path, obj):
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, 'w') as f:
json.dump(obj, f, separators=(',', ':'))
print(f' wrote {os.path.relpath(path, ROOT)} {os.path.getsize(path)/1e6:.2f}MB')
+188
View File
@@ -0,0 +1,188 @@
"""Build real-geodata scene packs. usage: python3 build_scene.py lax|bragg|everest [steps...]
steps: osm ground elev check (default all)"""
import json
import os
import sys
import numpy as np
from PIL import Image, ImageDraw
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import common as C # noqa
SCENES = {
"lax": dict(name="Los Angeles International Airport", lat0=33.9425, lon0=-118.4081, half=3500, z=17),
"bragg": dict(name="Fort Bragg, North Carolina", lat0=35.1580, lon0=-79.0090, half=6000, z=16),
"everest": dict(name="Mount Everest / Khumbu", lat0=27.9881, lon0=86.9250, half=15000, z=15),
}
NT = {"lax": 4, "bragg": 6}
AERO_Q = '[out:json][timeout:600];nwr["aeroway"~"^(runway|taxiway|apron|terminal|gate|parking_position)$"]({bb});out geom;'
BLDG_Q = '[out:json][timeout:300];(way["building"]({bb});relation["building"]({bb}););out geom;'
ROAD_Q = ('[out:json][timeout:300];way["highway"~"^(motorway|trunk|primary|secondary|tertiary|'
'residential|unclassified|living_street|motorway_link|trunk_link|primary_link|secondary_link|tertiary_link|service)$"]({bb});out geom;')
MIL_Q = '[out:json][timeout:600];(way["landuse"="military"]({bb});relation["landuse"="military"]({bb});way["military"]({bb});relation["military"]({bb}););out geom;'
PEAK_Q = '[out:json][timeout:300];node["natural"="peak"]({bb});out;'
def step_osm(f, slug):
bb = f.bbox_str()
if slug in ("lax", "bragg"):
aero = C.build_aeroway(f, C.overpass(AERO_Q.format(bb=bb), f"{slug}_aero"))
if slug == "lax":
C.wjson(os.path.join(f.out, "aeroway.json"), aero)
else:
mil = C.overpass(MIL_Q.format(bb=bb), f"{slug}_mil")
polys = []
for el in mil["elements"]:
tags = el.get("tags", {})
for ring in C.outer_rings(el, f):
name = tags.get("name") or (f"[{tags['military']}]" if "military" in tags else "")
polys.append([name] + C.flat(C.dp_ring(ring, 2.0)))
names = sorted({p[0] for p in polys if p[0]})
print(f" military polygons: {len(polys)}; names: {names[:40]}")
out = {"military": polys, "runways": aero["runways"], "taxiways": aero["taxiways"],
"aprons": aero["aprons"], "apronNames": aero["apronNames"], "terminals": aero["terminals"],
"gates": aero["gates"], "parking": aero["parking"]}
C.wjson(os.path.join(f.out, "military.json"), out)
C.wjson(os.path.join(f.out, "buildings.json"), C.build_buildings(f, C.overpass_tiled(f, BLDG_Q, f"{slug}_bldg", NT[slug])))
C.wjson(os.path.join(f.out, "roads.json"), C.build_roads(f, C.overpass_tiled(f, ROAD_Q, f"{slug}_roads", NT[slug] // 2)))
else:
d = C.overpass(PEAK_Q.format(bb=bb), f"{slug}_peaks")
peaks = []
skipped = 0
for el in d["elements"]:
t = el.get("tags", {})
name = t.get("name:en") or t.get("name")
ele = t.get("ele")
if not name or not ele:
skipped += 1
continue
try:
s = ele.lower().replace(",", ".").strip()
v = float(s.replace("ft", "").replace("m", "").strip().split()[0].split(";")[0])
if "ft" in s:
v *= 0.3048
except ValueError:
skipped += 1
continue
x, y = f.xy(el["lon"], el["lat"])
peaks.append([C.r1(x), C.r1(y), C.r1(v), name])
peaks.sort(key=lambda p: -p[2])
print(f" peaks with name+ele: {len(peaks)} (skipped {skipped} lacking name/ele); top: {peaks[:6]}")
C.wjson(os.path.join(f.out, "peaks.json"), peaks)
def step_elev(f):
z = 12
url = "https://s3.amazonaws.com/elevation-tiles-prod/terrarium/{z}/{x}/{y}.png"
mos, ox, oy, nf = C.mosaic(f, z, url, "terrarium", "png", "RGB", pad_m=1500)
if nf:
print(f" WARNING: {nf} terrarium tiles missing")
m = mos.astype(np.float32)
h = m[..., 0] * 256 + m[..., 1] + m[..., 2] / 256 - 32768
N = 1024
e = C.reproject(f, h[..., None], ox, oy, z, N)[..., 0].astype("<f4")
e.tofile(os.path.join(f.out, "elev.f32"))
meta = {"size": N, "min": C.r1(e.min()), "max": C.r1(e.max()), "half": f.half,
"source": "AWS Terrain Tiles terrarium z12", "rowOrder": "row0=north", "dtype": "float32le"}
C.wjson(os.path.join(f.out, "elev_meta.json"), meta)
print(f" elev min {e.min():.1f} max {e.max():.1f}")
# hillshade
cell = 2 * f.half / N
gy, gx = np.gradient(e.astype(np.float64), cell) # gy: d/drow (southward)
dzdx, dzdy = gx, -gy
az, alt = np.radians(315), np.radians(45)
slope = np.arctan(np.hypot(dzdx, dzdy))
aspect = np.arctan2(-dzdx, -dzdy) # direction of steepest descent... simple form
lx, ly, lz = np.sin(az) * np.cos(alt), np.cos(az) * np.cos(alt), np.sin(alt)
nx, ny, nz = -dzdx, -dzdy, np.ones_like(dzdx)
nn = np.sqrt(nx * nx + ny * ny + 1)
shade = np.clip((nx * lx + ny * ly + nz * lz) / nn, 0, 1)
t = (e - e.min()) / max(1, e.max() - e.min())
rgb = np.stack([0.35 + 0.65 * t, 0.45 + 0.55 * t, 0.55 + 0.45 * t], -1) * shade[..., None]
Image.fromarray((np.clip(rgb, 0, 1) * 255).astype(np.uint8)).save(os.path.join(f.out, "elev_preview.png"))
print(" wrote elev_preview.png")
def step_check(f, slug):
N = 2048
im = C.check_canvas(f)
dr = ImageDraw.Draw(im, "RGBA")
P = lambda a: C.to_px(f, a, N) # noqa
if slug in ("lax", "bragg"):
roads = json.load(open(os.path.join(f.out, "roads.json")))["roads"]
col = {0: (255, 60, 60, 230), 1: (255, 160, 40, 220), 2: (255, 230, 60, 200), 3: (200, 255, 120, 180),
4: (160, 200, 255, 150), 5: (180, 180, 180, 120)}
for r in roads:
dr.line(P(r[3:]), fill=col[r[0]], width=2 if r[0] < 3 else 1)
bl = json.load(open(os.path.join(f.out, "buildings.json")))["b"]
for b in bl:
pts = P(b[2:])
if len(pts) >= 3:
dr.polygon(pts, outline=(0, 255, 255, 200))
aero = json.load(open(os.path.join(f.out, "aeroway.json" if slug == "lax" else "military.json")))
if slug == "bragg":
for m in aero["military"]:
dr.polygon(P(m[1:]), outline=(255, 0, 255, 255))
for a in aero["aprons"]:
dr.polygon(P(a), outline=(255, 255, 255, 200))
for t in aero["taxiways"]:
dr.line(P(t), fill=(255, 255, 0, 230), width=1)
for rw in aero["runways"]:
w = max(2, int(rw[0] / (2 * f.half) * N))
dr.line(P(rw[2:6]), fill=(255, 40, 40, 110), width=w)
dr.line(P(rw[2:6]), fill=(255, 255, 255, 255), width=1)
for t in aero["terminals"]:
dr.polygon(P(t[1:]), outline=(255, 120, 0, 255))
for g in aero["gates"]:
(x, y), = P(g[:2])
dr.ellipse([x - 2, y - 2, x + 2, y + 2], fill=(0, 255, 0, 255))
else:
peaks = json.load(open(os.path.join(f.out, "peaks.json")))
for x0, y0, ele, name in peaks:
(x, y), = P([x0, y0])
dr.ellipse([x - 4, y - 4, x + 4, y + 4], outline=(255, 60, 60, 255), width=2)
if ele > 6000:
dr.text((x + 6, y - 6), f"{name} {ele:.0f}", fill=(255, 255, 255, 255))
# contour-ish overlay from DEM to check imagery/DEM alignment
e = np.fromfile(os.path.join(f.out, "elev.f32"), "<f4").reshape(1024, 1024)
e2 = np.asarray(Image.fromarray(e).resize((N, N), Image.BILINEAR))
band = (np.floor(e2 / 500) % 2).astype(np.int8)
edge = (np.abs(np.diff(band, axis=0, prepend=band[:1])) + np.abs(np.diff(band, axis=1, prepend=band[:, :1]))) > 0
arr = np.asarray(im).copy()
arr[edge] = (0.4 * arr[edge] + 0.6 * np.array([0, 255, 255])).astype(np.uint8)
im = Image.fromarray(arr)
dr = ImageDraw.Draw(im, "RGBA")
for x0, y0, ele, name in peaks:
(x, y), = P([x0, y0])
dr.ellipse([x - 4, y - 4, x + 4, y + 4], outline=(255, 60, 60, 255), width=2)
if ele > 6500:
dr.text((x + 6, y - 6), f"{name} {ele:.0f}", fill=(255, 255, 255, 255))
p = os.path.join(f.out, "check.png")
im.save(p)
print(f" wrote {p}")
def main():
slug = sys.argv[1]
steps = sys.argv[2:] or ["osm", "ground", "elev", "check"]
s = SCENES[slug]
f = C.Frame(slug, s["name"], s["lat0"], s["lon0"], s["half"])
f.write_meta()
print(f"== {slug} bbox {f.bbox_str()}")
if "osm" in steps:
step_osm(f, slug)
if "ground" in steps:
C.build_ground(f, s["z"])
if "ground_s2" in steps:
# alternative uniform real imagery (EOX Sentinel-2 cloudless 2020) — Esri is a patchwork here
C.build_ground(f, 14, C.EOX_S2, "eox_s2", "ground_s2")
if "elev" in steps and slug == "everest":
step_elev(f)
if "check" in steps:
step_check(f, slug)
if __name__ == "__main__":
main()
+571
View File
@@ -0,0 +1,571 @@
"""Shared helpers for real-geodata scene builds (city2).
Local frame: ENU metres relative to (lat0, lon0):
x = (lon - lon0) * cos(lat0) * 111320
y = (lat - lat0) * 110540
"""
import hashlib
import io
import json
import math
import os
import sys
import time
from concurrent.futures import ThreadPoolExecutor
import numpy as np
import requests
from PIL import Image
HERE = os.path.dirname(os.path.abspath(__file__))
CACHE = os.path.join(HERE, "cache")
CITY = os.path.abspath(os.path.join(HERE, "..", "..", "city"))
os.makedirs(CACHE, exist_ok=True)
OVERPASS = [
"https://overpass.kumi.systems/api/interpreter",
"https://overpass.private.coffee/api/interpreter",
"https://maps.mail.ru/osm/tools/overpass/api/interpreter",
"https://overpass-api.de/api/interpreter",
]
UA = {"User-Agent": "gods-eye-view-intro/1.0 (cinematic geodata build)"}
Image.MAX_IMAGE_PIXELS = None
class Frame:
def __init__(self, slug, name, lat0, lon0, half):
self.slug, self.name, self.lat0, self.lon0, self.half = slug, name, lat0, lon0, half
self.kx = math.cos(math.radians(lat0)) * 111320.0
self.ky = 110540.0
self.out = os.path.join(CITY, slug)
os.makedirs(self.out, exist_ok=True)
def xy(self, lon, lat):
return (lon - self.lon0) * self.kx, (lat - self.lat0) * self.ky
def ll(self, x, y):
return self.lon0 + x / self.kx, self.lat0 + y / self.ky
def bbox(self, pad=0.0):
"""(south, west, north, east)"""
h = self.half + pad
return (self.lat0 - h / self.ky, self.lon0 - h / self.kx,
self.lat0 + h / self.ky, self.lon0 + h / self.kx)
def bbox_str(self, pad=0.0):
s, w, n, e = self.bbox(pad)
return f"{s:.6f},{w:.6f},{n:.6f},{e:.6f}"
def write_meta(self):
meta = {"slug": self.slug, "name": self.name, "lat0": self.lat0, "lon0": self.lon0,
"half": self.half, "frame": "ENU m"}
wjson(os.path.join(self.out, "meta.json"), meta)
def geom_xy(self, geom):
"""list of {lat,lon} -> Nx2 float array in local metres"""
a = np.array([[g["lon"], g["lat"]] for g in geom if g is not None], dtype=np.float64)
if len(a) == 0:
return np.zeros((0, 2))
return np.stack([(a[:, 0] - self.lon0) * self.kx, (a[:, 1] - self.lat0) * self.ky], 1)
def wjson(path, obj):
with open(path, "w") as f:
json.dump(obj, f, separators=(",", ":"))
print(f" wrote {path} ({os.path.getsize(path)/1e6:.2f} MB)")
def r1(v):
return round(float(v), 1)
def flat(arr):
return [r1(v) for v in np.asarray(arr).reshape(-1)]
# ---------------------------------------------------------------- Overpass
def overpass(query, tag):
key = hashlib.sha1(query.encode()).hexdigest()[:12]
path = os.path.join(CACHE, f"overpass_{tag}_{key}.json")
if os.path.exists(path):
with open(path) as f:
return json.load(f)
last = None
for attempt in range(10):
for url in OVERPASS:
try:
print(f" overpass {tag} -> {url}", flush=True)
t = time.time()
r = requests.get(url, params={"data": query}, headers=UA, timeout=900)
if r.status_code != 200:
last = f"{url}: HTTP {r.status_code} {r.text[:200]}"
print(" ", last)
continue
d = r.json()
rem = d.get("remark", "")
if rem and ("runtime error" in rem or "timed out" in rem or "out of memory" in rem):
last = f"{url}: remark {rem[:200]}"
print(" ", last)
continue
print(f" {len(d.get('elements', []))} elements in {time.time()-t:.0f}s")
with open(path, "w") as f:
json.dump(d, f)
return d
except Exception as e: # noqa
last = f"{url}: {e}"
print(" ", last)
time.sleep(30)
raise RuntimeError(f"overpass failed for {tag}: {last}")
def overpass_tiled(frame, qfmt, tag, n):
"""Run a query template ({bb} placeholder) over an n x n split of the frame bbox; dedupe elements."""
s, w, nn, e = frame.bbox()
seen, els = set(), []
for i in range(n):
for j in range(n):
bb = (f"{s + (nn - s) * i / n:.6f},{w + (e - w) * j / n:.6f},"
f"{s + (nn - s) * (i + 1) / n:.6f},{w + (e - w) * (j + 1) / n:.6f}")
d = overpass(qfmt.format(bb=bb), f"{tag}_{i}{j}")
for el in d["elements"]:
k = (el["type"], el["id"])
if k not in seen:
seen.add(k)
els.append(el)
print(f" {tag}: {len(els)} unique elements over {n}x{n} tiles")
return {"elements": els}
# ---------------------------------------------------------------- geometry
def assemble_rings(ways):
"""ways: list of Nx2 arrays (open or closed). Returns list of closed rings."""
rings = []
pending = [w for w in ways if len(w) >= 2]
while pending:
cur = pending.pop(0)
changed = True
while not np.allclose(cur[0], cur[-1], atol=1e-3) and changed:
changed = False
for i, w in enumerate(pending):
if np.allclose(cur[-1], w[0], atol=1e-3):
cur = np.vstack([cur, w[1:]])
elif np.allclose(cur[-1], w[-1], atol=1e-3):
cur = np.vstack([cur, w[::-1][1:]])
elif np.allclose(cur[0], w[-1], atol=1e-3):
cur = np.vstack([w, cur[1:]])
elif np.allclose(cur[0], w[0], atol=1e-3):
cur = np.vstack([w[::-1], cur[1:]])
else:
continue
pending.pop(i)
changed = True
break
if np.allclose(cur[0], cur[-1], atol=1e-3) and len(cur) >= 4:
rings.append(cur)
return rings
def outer_rings(el, frame):
"""Outer rings (local metres, closed) for a way or multipolygon relation with out geom."""
if el["type"] == "way":
g = el.get("geometry")
if not g:
return []
a = frame.geom_xy(g)
if len(a) >= 4 and np.allclose(a[0], a[-1], atol=1e-3):
return [a]
return []
if el["type"] == "relation":
ways = [frame.geom_xy(m["geometry"]) for m in el.get("members", [])
if m.get("type") == "way" and m.get("role", "outer") in ("outer", "") and m.get("geometry")]
return assemble_rings(ways)
return []
def dp(pts, eps):
"""Douglas-Peucker on Nx2 array (iterative)."""
n = len(pts)
if n < 3:
return pts
keep = np.zeros(n, bool)
keep[0] = keep[-1] = True
stack = [(0, n - 1)]
while stack:
a, b = stack.pop()
if b <= a + 1:
continue
p, q = pts[a], pts[b]
seg = pts[a + 1:b]
d = q - p
L = math.hypot(d[0], d[1])
if L < 1e-9:
dist = np.hypot(seg[:, 0] - p[0], seg[:, 1] - p[1])
else:
dist = np.abs(d[0] * (seg[:, 1] - p[1]) - d[1] * (seg[:, 0] - p[0])) / L
i = int(np.argmax(dist))
if dist[i] > eps:
m = a + 1 + i
keep[m] = True
stack.append((a, m))
stack.append((m, b))
return pts[keep]
def dp_ring(ring, eps):
"""DP for a closed ring: split at the vertex farthest from ring[0]."""
r = ring[:-1] if np.allclose(ring[0], ring[-1]) else ring
if len(r) < 4:
return r
far = int(np.argmax(np.hypot(*(r - r[0]).T)))
a = dp(np.vstack([r[:far + 1]]), eps)
b = dp(np.vstack([r[far:], r[:1]]), eps)
return np.vstack([a[:-1], b[:-1]]) # open ring (no repeated closing vertex)
def area(ring):
x, y = ring[:, 0], ring[:, 1]
return 0.5 * abs(np.dot(x, np.roll(y, -1)) - np.dot(y, np.roll(x, -1)))
def poly_len(a):
return float(np.sum(np.hypot(*np.diff(a, axis=0).T))) if len(a) > 1 else 0.0
def clip_polyline(a, lim):
"""Split a polyline into runs inside the square |x|,|y|<=lim, cutting segments at the edge."""
out, cur = [], []
inside = (np.abs(a[:, 0]) <= lim) & (np.abs(a[:, 1]) <= lim)
def cut(p, q):
# point on segment p->q at the square boundary (p inside, q outside)
t = 1.0
d = q - p
for k in range(2):
if d[k] > 0 and q[k] > lim:
t = min(t, (lim - p[k]) / d[k])
elif d[k] < 0 and q[k] < -lim:
t = min(t, (-lim - p[k]) / d[k])
return p + d * t
for i in range(len(a)):
if inside[i]:
if not cur and i > 0:
cur.append(cut(a[i], a[i - 1]))
cur.append(a[i])
else:
if cur:
cur.append(cut(a[i - 1], a[i]))
out.append(np.array(cur))
cur = []
if cur:
out.append(np.array(cur))
return [c for c in out if len(c) >= 2]
def tag_height(tags, default=10.0):
def num(s):
try:
s = str(s).strip().lower().replace(",", ".")
ft = s.endswith("ft") or s.endswith("'")
s = s.replace("m", "").replace("ft", "").replace("'", "").strip().split(";")[0].split()[0]
v = float(s)
return v * 0.3048 if ft else v
except Exception:
return None
h = num(tags["height"]) if "height" in tags else None
if h is None and "building:levels" in tags:
lv = num(tags["building:levels"])
if lv is not None:
h = lv * 3.2
if h is None or h <= 0:
h = default
mh = num(tags["min_height"]) if "min_height" in tags else None
if mh is None and "building:min_level" in tags:
lv = num(tags["building:min_level"])
mh = lv * 3.2 if lv is not None else None
mh = mh if (mh is not None and 0 <= mh < h) else 0.0
return h, mh
# ---------------------------------------------------------------- buildings / roads
def build_buildings(frame, data, min_area=6.0):
b = []
dropped = 0
lim = frame.half
for el in data["elements"]:
if el["type"] not in ("way", "relation"):
continue
tags = el.get("tags", {})
if "building" not in tags:
continue
h, mh = tag_height(tags)
for ring in outer_rings(el, frame):
c = ring[:-1].mean(0)
if abs(c[0]) > lim or abs(c[1]) > lim:
continue
s = dp_ring(ring, 0.5)
if len(s) < 3 or area(s) < min_area:
dropped += 1
continue
b.append([r1(h), r1(mh)] + flat(s))
print(f" buildings: {len(b)} kept, {dropped} tiny dropped")
return {"count": len(b), "b": b}
ROAD_CLASS = {
"motorway": 0, "trunk": 0,
"primary": 1, "secondary": 2, "tertiary": 3,
"residential": 4, "unclassified": 4, "living_street": 4,
"motorway_link": 5, "trunk_link": 5, "primary_link": 5, "secondary_link": 5,
"tertiary_link": 5, "service": 5,
}
DEFAULT_LANES = {0: 3, 1: 2, 2: 2, 3: 2, 4: 1, 5: 1}
def build_roads(frame, data):
roads = []
counts = {}
lim = frame.half + 50
for el in data["elements"]:
if el["type"] != "way" or not el.get("geometry"):
continue
tags = el.get("tags", {})
hw = tags.get("highway")
if hw not in ROAD_CLASS:
continue
cls = ROAD_CLASS[hw]
a = frame.geom_xy(el["geometry"])
if hw == "service" and poly_len(a) <= 80:
continue
ow = tags.get("oneway", "")
oneway = 0
if ow in ("yes", "1", "true") or (hw in ("motorway",) and ow != "no") or tags.get("junction") in ("roundabout", "circular"):
oneway = 1
elif ow == "-1":
oneway = 1
a = a[::-1]
try:
lanes = int(str(tags.get("lanes", "")).split(";")[0])
except ValueError:
lanes = DEFAULT_LANES[cls] * (1 if oneway else 2) if cls < 4 else (1 if oneway else 2)
for part in clip_polyline(a, lim):
if poly_len(part) < 2:
continue
roads.append([cls, oneway, lanes] + flat(part))
counts[cls] = counts.get(cls, 0) + 1
print(f" roads: {len(roads)} polylines, by class {dict(sorted(counts.items()))}")
return {"roads": roads}
# ---------------------------------------------------------------- aeroway
def build_aeroway(frame, data):
lim = frame.half + 200
runways, taxiways, aprons, terminals, gates, parking = [], [], [], [], [], []
apron_names = []
for el in data["elements"]:
tags = el.get("tags", {})
aw = tags.get("aeroway")
if el["type"] == "node":
x, y = frame.xy(el["lon"], el["lat"])
if abs(x) > lim or abs(y) > lim:
continue
if aw == "gate":
gates.append([r1(x), r1(y), tags.get("ref", tags.get("name", ""))])
elif aw == "parking_position":
parking.append([r1(x), r1(y), tags.get("ref", tags.get("name", ""))])
continue
if aw == "runway" and el["type"] == "way" and el.get("geometry"):
a = frame.geom_xy(el["geometry"])
closed = len(a) > 3 and np.allclose(a[0], a[-1], atol=1e-3)
try:
w = float(str(tags.get("width", "")).replace("m", "").strip())
except ValueError:
w = None
if closed or tags.get("area") == "yes":
# area runway: use its long axis (PCA) as centreline, width = short extent
c = a.mean(0)
u, s, vt = np.linalg.svd(a - c, full_matrices=False)
ax = vt[0]
t = (a - c) @ ax
n = (a - c) @ vt[1]
p1, p2 = c + ax * t.min(), c + ax * t.max()
w = w or float(n.max() - n.min())
else:
p1, p2 = a[0], a[-1]
runways.append([r1(w or 45.0), tags.get("ref", ""), r1(p1[0]), r1(p1[1]), r1(p2[0]), r1(p2[1])])
elif aw == "taxiway" and el["type"] == "way" and el.get("geometry"):
a = frame.geom_xy(el["geometry"])
if tags.get("area") == "yes":
continue
for part in clip_polyline(a, lim):
taxiways.append(flat(part))
elif aw == "apron":
for ring in outer_rings(el, frame):
c = ring.mean(0)
if abs(c[0]) > lim or abs(c[1]) > lim:
continue
aprons.append(flat(dp_ring(ring, 0.5)))
apron_names.append(tags.get("name", ""))
elif aw == "terminal":
h, _ = tag_height(tags)
for ring in outer_rings(el, frame):
c = ring.mean(0)
if abs(c[0]) > lim or abs(c[1]) > lim:
continue
terminals.append([r1(h)] + flat(dp_ring(ring, 0.5)))
# OSM often splits one runway into several ways (displaced thresholds, blast pads):
# merge segments sharing a ref into one centreline spanning their extreme endpoints.
merged, groups = [], {}
for rw in runways:
if rw[1]:
groups.setdefault(rw[1], []).append(rw)
else:
merged.append(rw)
for ref, segs in groups.items():
pts = np.array([[s[2], s[3]] for s in segs] + [[s[4], s[5]] for s in segs])
main = max(segs, key=lambda s: math.hypot(s[4] - s[2], s[5] - s[3]))
ax = np.array([main[4] - main[2], main[5] - main[3]])
ax /= np.linalg.norm(ax)
t = (pts - pts[0]) @ ax
p1, p2 = pts[int(np.argmin(t))], pts[int(np.argmax(t))]
merged.append([main[0], ref, r1(p1[0]), r1(p1[1]), r1(p2[0]), r1(p2[1])])
if len(merged) != len(runways):
print(f" runways: merged {len(runways)} OSM segments into {len(merged)}")
runways = merged
out = {"runways": runways, "taxiways": taxiways, "aprons": aprons, "apronNames": apron_names,
"terminals": terminals, "gates": gates, "parking": parking}
print(" aeroway: " + ", ".join(f"{k}={len(v)}" for k, v in out.items()))
return out
# ---------------------------------------------------------------- tiles / imagery
def merc_px(lon, lat, z, ts=256):
n = 2 ** z
x = (lon + 180.0) / 360.0 * n * ts
lat_r = np.radians(lat)
y = (1.0 - np.log(np.tan(lat_r) + 1.0 / np.cos(lat_r)) / math.pi) / 2.0 * n * ts
return x, y
def fetch_tiles(url_fmt, tiles, subdir, ext, workers=8):
d = os.path.join(CACHE, subdir)
os.makedirs(d, exist_ok=True)
sess = requests.Session()
sess.headers.update(UA)
fails = []
def get(t):
z, x, y = t
p = os.path.join(d, f"{z}_{x}_{y}.{ext}")
if os.path.exists(p) and os.path.getsize(p) > 0:
return p
for i in range(4):
try:
r = sess.get(url_fmt.format(z=z, x=x, y=y), timeout=60)
if r.status_code == 200 and len(r.content) > 100:
with open(p + ".tmp", "wb") as f:
f.write(r.content)
os.replace(p + ".tmp", p)
return p
if r.status_code == 404:
break
except Exception:
pass
time.sleep(1.5 * (i + 1))
fails.append(t)
return None
with ThreadPoolExecutor(workers) as ex:
res = list(ex.map(get, tiles))
return dict(zip(tiles, res)), fails
def mosaic(frame, z, url_fmt, subdir, ext, mode, pad_m=300.0):
s, w, n, e = frame.bbox(pad_m)
x0, y1 = merc_px(w, s, z)
x1, y0 = merc_px(e, n, z)
tx0, tx1 = int(x0 // 256), int(x1 // 256)
ty0, ty1 = int(y0 // 256), int(y1 // 256)
tiles = [(z, x, y) for y in range(ty0, ty1 + 1) for x in range(tx0, tx1 + 1)]
print(f" {subdir} z{z}: {tx1-tx0+1}x{ty1-ty0+1} = {len(tiles)} tiles", flush=True)
paths, fails = fetch_tiles(url_fmt, tiles, subdir, ext)
if fails:
print(f" WARNING: {len(fails)} tiles failed: {fails[:10]}")
W, H = (tx1 - tx0 + 1) * 256, (ty1 - ty0 + 1) * 256
nb = 3 if mode == "RGB" else 3
mos = np.zeros((H, W, nb), np.uint8)
for (zz, x, y), p in paths.items():
if p is None:
continue
im = Image.open(p).convert("RGB")
if im.size != (256, 256):
im = im.resize((256, 256))
mos[(y - ty0) * 256:(y - ty0 + 1) * 256, (x - tx0) * 256:(x - tx0 + 1) * 256] = np.asarray(im)
return mos, tx0 * 256, ty0 * 256, len(fails)
def bilinear(img, px, py):
"""img HxWxC (float or uint8), px/py float arrays in pixel coords (pixel centres at i+0.5)."""
H, W = img.shape[:2]
fx = np.clip(px - 0.5, 0, W - 1.000001)
fy = np.clip(py - 0.5, 0, H - 1.000001)
x0 = np.floor(fx).astype(np.int64)
y0 = np.floor(fy).astype(np.int64)
ax = (fx - x0)[..., None]
ay = (fy - y0)[..., None]
im = img if img.ndim == 3 else img[..., None]
a = im[y0, x0].astype(np.float32)
b = im[y0, x0 + 1].astype(np.float32)
c = im[y0 + 1, x0].astype(np.float32)
d = im[y0 + 1, x0 + 1].astype(np.float32)
out = (a * (1 - ax) + b * ax) * (1 - ay) + (c * (1 - ax) + d * ax) * ay
return out if img.ndim == 3 else out[..., 0]
def reproject(frame, img, ox, oy, z, N, rows_per=256):
"""Sample a Mercator mosaic (origin global px ox,oy at zoom z) onto the local NxN square."""
h = frame.half
out = np.zeros((N, N) + img.shape[2:], np.float32)
xs = -h + (np.arange(N) + 0.5) * (2 * h / N)
for r0 in range(0, N, rows_per):
r = np.arange(r0, min(N, r0 + rows_per))
ys = h - (r + 0.5) * (2 * h / N)
X, Y = np.meshgrid(xs, ys)
lon, lat = frame.ll(X, Y)
mx, my = merc_px(lon, lat, z)
out[r0:r0 + len(r)] = bilinear(img, mx - ox, my - oy)
return out
ESRI = "https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{z}/{y}/{x}"
EOX_S2 = "https://tiles.maps.eox.at/wmts/1.0.0/s2cloudless-2020_3857/default/g/{z}/{y}/{x}.jpg"
def build_ground(frame, z, url=ESRI, sub="esri", name="ground"):
mos, ox, oy, nf = mosaic(frame, z, url, sub, "jpg", "RGB")
out = reproject(frame, mos, ox, oy, z, 4096)
im = Image.fromarray(np.clip(out + 0.5, 0, 255).astype(np.uint8))
p = os.path.join(frame.out, f"{name}.jpg")
im.save(p, quality=90, optimize=True)
print(f" wrote {p} ({os.path.getsize(p)/1e6:.2f} MB)")
im2 = im.resize((2048, 2048), Image.LANCZOS)
p2 = os.path.join(frame.out, f"{name}_2k.jpg")
im2.save(p2, quality=88, optimize=True)
print(f" wrote {p2} ({os.path.getsize(p2)/1e6:.2f} MB)")
return nf
# ---------------------------------------------------------------- check overlay
def to_px(frame, a, N):
a = np.asarray(a, dtype=np.float64).reshape(-1, 2)
return [((x + frame.half) / (2 * frame.half) * N, (frame.half - y) / (2 * frame.half) * N) for x, y in a]
def check_canvas(frame, dark=0.45):
from PIL import ImageEnhance
im = Image.open(os.path.join(frame.out, "ground_2k.jpg")).convert("RGB")
im = ImageEnhance.Brightness(im).enhance(dark)
return im
+40
View File
@@ -0,0 +1,40 @@
// Export the film clock for the score: every visual event the audio locks to.
import fs from 'node:fs';
import { T, FEEDS, CITIES, DURATION, captureTimes, CAPTURE_SLOW } from '../src/timeline.js';
import { angDist } from '../src/geo.js';
const heroes = JSON.parse(fs.readFileSync('imagery/hero/index.json', 'utf8'));
const caps = captureTimes(heroes.length).map((t, i) => ({ t, slow: i < CAPTURE_SLOW, lon: heroes[i].lon }));
// tile reveal times — same formula as the globe shader and HUD
const f = (v) => v - Math.floor(v);
const hash3 = (x, y, z) => {
const px = f(x * 0.3183099 + 0.71) * 17, py = f(y * 0.3183099 + 0.113) * 17, pz = f(z * 0.3183099 + 0.419) * 17;
return f(px * py * pz * (px + py + pz));
};
const tiles = [];
[[3, T.zWave[0], 0.35], [4, T.zWave[1], 0.3], [5, T.zWave[2], 0.25]].forEach(([z, wave, jit]) => {
const n = 2 ** z;
for (let x = 0; x < n; x++)
for (let y = 0; y < n; y++) {
const lon = ((x + 0.5) / n) * 360 - 180;
const lat = (Math.atan(Math.sinh(Math.PI * (1 - 2 * ((y + 0.5) / n)))) * 180) / Math.PI;
// only the hemisphere facing the camera is heard
const d = angDist(0, 0, lon, lat);
tiles.push({ t: wave + (d / Math.PI) * 2.6 + hash3(x, y, z) * jit, z, pan: Math.max(-1, Math.min(1, lon / 90)), vis: angDist(10, 8, lon, lat) < 1.4 });
}
});
// boot console keystrokes
const boot = [];
[['GEV://OBSERVER-1 rev 0.2.1', T.boot, 55], ['allocating substrate ........ 1.0000 OK', T.boot + 0.75, 55], ['loading priors .............. NONE', T.boot + 1.35, 55], ['> seed --origin 0.0000°N 0.0000°E', T.boot + 2.1, 26]].forEach(([s, st, cps], line) => {
for (let i = 0; i < s.length; i++) {
const t = st + i / cps;
if (t < T.collapse) boot.push({ t, ch: s[i], line });
}
});
const status = [77.4, 78.5, 79.6, 80.7, 81.7];
const rapid = Array.from({ length: 7 }, (_, i) => T.rapid + i * 0.22);
fs.writeFileSync('audio/cues.json', JSON.stringify({ T, DURATION, feeds: FEEDS.map((x) => ({ key: x.key, t: x.t })), cities: CITIES.map((c) => ({ slug: c.slug, t0: c.t0, t1: c.t1 })), caps, tiles, boot, status, rapid }, null, 1));
console.log('cues:', caps.length, 'captures,', tiles.length, 'tiles,', boot.length, 'keystrokes');
+511
View File
@@ -0,0 +1,511 @@
#!/usr/bin/env python3
"""Count distinct cameras in every keyless CCTV catalog the app supports.
Fetches each source's FULL public catalog (no anchor selection, no
DEFAULT_*_MAX_SOURCES caps) and counts distinct cameras two ways:
raw - distinct camera ids listed in the catalog at all
valid - distinct ids passing the app's own validity filters (in service /
online / official still or stream URL / plausible coordinates),
i.e. what server/providers/cctv/* would accept before capping.
The headline number written to data/extra_counts.json is the conservative
`valid` count. Sources that fail are skipped, never estimated.
Usage: python3 -I cctv_count.py <out_json> [<detail_json>]
"""
import json, math, re, sys, time
from concurrent.futures import ThreadPoolExecutor
from urllib.parse import urlparse, parse_qs
import requests
UA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'
REPO = '/home/user/gods-eye-view'
def sess():
s = requests.Session()
s.headers.update({'User-Agent': UA, 'Digitraffic-User': 'gods-eye-view'})
return s
def get(url, timeout=60, **kw):
r = sess().get(url, timeout=timeout, **kw)
r.raise_for_status()
return r
def fnum(v):
try:
f = float(v)
return f if math.isfinite(f) else None
except (TypeError, ValueError):
return None
def inbox(lat, lon, la0, la1, lo0, lo1):
return lat is not None and lon is not None and la0 <= lat <= la1 and lo0 <= lon <= lo1
SRC = {}
def source(label, country):
def deco(fn):
SRC[label] = (fn, country)
return fn
return deco
@source('TfL JamCams', 'United Kingdom')
def tfl():
d = get('https://api.tfl.gov.uk/Place/Type/JamCam').json()
raw, ok = set(), set()
for p in d:
raw.add(p['id'])
props = {a['key']: a['value'] for a in p.get('additionalProperties', [])}
if str(props.get('available')).lower() != 'true':
continue
if not (props.get('imageUrl') or '').startswith('https://s3-eu-west-1.amazonaws.com/jamcams.tfl.gov.uk/'):
continue
if fnum(p.get('lat')) is None or fnum(p.get('lon')) is None:
continue
ok.add(p['id'])
return raw, ok, 'one Place per camera; available=true + official S3 still'
@source('NYC DOT', 'United States')
def nyc():
d = get('https://webcams.nyctmc.org/api/cameras').json()
raw = {c['id'] for c in d}
ok = {c['id'] for c in d if str(c.get('isOnline')).strip().lower() == 'true'
and inbox(fnum(c.get('latitude')), fnum(c.get('longitude')), 40.4, 41.0, -74.3, -73.6)}
return raw, ok, 'one row per camera; isOnline=true'
@source('Caltrans', 'United States')
def caltrans():
raw, ok = set(), set()
notes = []
def one(dist):
err = None
for attempt in range(5): # cwwp2 resets connections under parallel load
try:
return dist, get(f'https://cwwp2.dot.ca.gov/data/d{dist}/cctv/cctvStatusD{dist:02d}.json').json()
except Exception as e:
err = e
time.sleep(1 + attempt)
return dist, err
with ThreadPoolExecutor(3) as ex:
res = list(ex.map(one, range(1, 13)))
for dist, d in res:
if isinstance(d, Exception):
notes.append(f'D{dist} failed: {str(d)[:60]}')
continue
for i, row in enumerate(d.get('data', [])):
c = row.get('cctv') or {}
loc = c.get('location') or {}
name = (loc.get('locationName') or '').strip()
m = re.match(r'^([A-Za-z0-9_-]+)\s*--', name)
key = f"d{dist}-{(m.group(1) if m else c.get('index') or name or i)}".lower()
raw.add(key)
if str(c.get('inService')).lower() != 'true':
continue
if fnum(loc.get('latitude')) is None or fnum(loc.get('longitude')) is None:
continue
url = ((c.get('imageData') or {}).get('static') or {}).get('currentImageURL') or ''
if not url.startswith('https://cwwp2.dot.ca.gov/'):
continue
ok.add(key)
return raw, ok, 'all 12 districts; inService=true + cwwp2 still. ' + '; '.join(notes)
@source('WSDOT', 'United States')
def wsdot():
d = json.loads(get('https://data.wsdot.wa.gov/travelcenter/Cameras.json').content.decode('cp1252'))
raw, ok = set(), set()
for f in d['features']:
a = f.get('attributes') or {}
cid = a.get('CameraID')
if not isinstance(cid, int) or cid <= 0:
continue
raw.add(cid)
if (a.get('ImageURL') or '').strip().startswith('https://images.wsdot.wa.gov/') and fnum((f.get('geometry') or {}).get('x')) is not None:
ok.add(cid)
return raw, ok, 'one feature per CameraID; official images.wsdot.wa.gov still'
@source('King County DOT', 'United States')
def kingco():
d = get('https://services.arcgis.com/Ej0PsM5Aw677QF1W/arcgis/rest/services/TRAFFICCAMERA_POINT_2029/FeatureServer/0/query?where=1%3D1&outFields=AssetID,Location,ImageURL,Live,Model,Manufacturer,Owner,CurrentStatus&f=geojson').json()
raw, ok = set(), set()
for f in d['features']:
p = f.get('properties') or {}
aid = p.get('AssetID')
if not isinstance(aid, int) or aid <= 0:
continue
raw.add(aid)
if p.get('Live') != 1 or str(p.get('CurrentStatus') or '').strip() != 'Active':
continue
if str(p.get('Owner') or '').strip() == 'WSDOT':
continue # WSDOT-owned rows are WSDOT cameras (counted under WSDOT)
url = (p.get('ImageURL') or '').strip().replace('http://info.kingcounty.gov/', 'https://info.kingcounty.gov/')
if not url.startswith('https://info.kingcounty.gov/'):
continue
ok.add(aid)
note = 'Live=1 + Active, WSDOT-owned rows excluded (avoid double count)'
if d.get('exceededTransferLimit') or (d.get('properties') or {}).get('exceededTransferLimit'):
note += ' WARNING: transfer limit exceeded'
return raw, ok, note
@source('DriveBC', 'Canada')
def drivebc():
d = get('https://www.drivebc.ca/api/webcams/').json()
raw, ok = set(), set()
for c in d:
cid = c.get('id')
if not isinstance(cid, int) or cid <= 0:
continue
raw.add(cid)
if c.get('is_on') is not True or c.get('should_appear') is not True:
continue
coords = (c.get('location') or {}).get('coordinates') or [None, None]
if not inbox(fnum(coords[1]), fnum(coords[0]), 48.0, 60.5, -139.5, -113.5):
continue
ok.add(cid)
return raw, ok, 'one webcam per id; is_on + should_appear'
@source('Fintraffic Digitraffic', 'Finland')
def fintraffic():
d = get('https://tie.digitraffic.fi/api/weathercam/v1/stations', headers={'Accept-Encoding': 'gzip'}).json()
raw, ok, presets = set(), set(), 0
for f in d['features']:
p = f.get('properties') or {}
sid = str(p.get('id') or '')
if not sid:
continue
raw.add(sid)
if str(p.get('collectionStatus') or '').upper() != 'GATHERING':
continue
live = [pr for pr in p.get('presets') or [] if pr.get('inCollection') is True and re.fullmatch(r'C\d{7}', str(pr.get('id'))) and str(pr['id']).startswith(sid)]
if live:
ok.add(sid)
presets += len(live)
return raw, ok, f'counted per STATION (camera site), not per preset view; {presets} live presets (the app treats each preset as a camera)'
@source('Statens vegvesen', 'Norway')
def vegvesen():
d = get('https://ogckart-sn1.atlas.vegvesen.no/ogc/features/v1/collections/datex_3_1:CctvSimple/items?f=application/geo%2Bjson&limit=5000',
headers={'Accept': 'application/geo+json,application/json'}).json()
raw, ok = set(), set()
for f in d['features']:
p = f.get('properties') or {}
rid = str(p.get('cameraId') or '').strip()
if not re.fullmatch(r'[A-Za-z0-9_-]{1,40}', rid):
continue
raw.add(rid)
av = str(p.get('status.stillImageAvailability') or '').strip()
if av and av != 'videoOrImagesAvailable':
continue
if (p.get('stillImageUrl') or '') != f'https://kamera.atlas.vegvesen.no/api/images/{rid}':
continue
c = (f.get('geometry') or {}).get('coordinates') or [None, None]
if not inbox(fnum(c[1]), fnum(c[0]), 57.5, 71.5, 4.0, 31.5):
continue
ok.add(rid)
note = f"numberMatched={d.get('numberMatched')} numberReturned={d.get('numberReturned')}"
return raw, ok, 'stillImageAvailability ok + official still; ' + note
@source('City of Tallinn', 'Estonia')
def tallinn():
d = json.load(open(f'{REPO}/config/cctv_sources.tallinn.json'))
raw, ok = set(), set()
for c in d:
cid = str(c.get('id') or '').strip()
if not cid:
continue
raw.add(cid)
url = c.get('snapshotUrl') or c.get('url') or ''
if inbox(fnum(c.get('lat')), fnum(c.get('lon')), 59.2, 59.7, 24.3, 25.4) and url.startswith('https://ristmikud.tallinn.ee/'):
ok.add(url)
return raw, ok, 'curated repo catalog (config/cctv_sources.tallinn.json) of ristmikud.tallinn.ee stills; distinct frame URLs'
@source('Transpordiamet Tarktee', 'Estonia')
def tarktee():
lx = get('https://tarktee.transpordiamet.ee/api/v1/datex/roadCameraLocations', headers={'Accept': 'application/xml'}).text
ix = get('https://tarktee.transpordiamet.ee/api/v1/datex/roadCameraImages', headers={'Accept': 'application/xml'}).text
locs = {}
for m in re.finditer(r'<predefinedLocation\s+id="([^"]+)"[^>]*>([\s\S]*?)</predefinedLocation>', lx):
la = re.search(r'<latitude>\s*(-?\d+(?:\.\d+)?)\s*</latitude>', m.group(2), re.I)
lo = re.search(r'<longitude>\s*(-?\d+(?:\.\d+)?)\s*</longitude>', m.group(2), re.I)
if la and lo:
locs[m.group(1)] = (float(la.group(1)), float(lo.group(1)))
imgs = {}
for m in re.finditer(r'<trafficView\b[^>]*>([\s\S]*?)</trafficView>', ix):
r = re.search(r'<linearPredefinedLocationReference\b[^>]*\bid="([^"]+)"', m.group(1), re.I)
u = re.search(r'<urlLinkAddress>\s*([^<\s]+)\s*</urlLinkAddress>', m.group(1), re.I)
if r and u:
imgs[r.group(1)] = u.group(1)
raw = set(locs) | set(imgs)
ok = set()
for lid, (lat, lon) in locs.items():
u = imgs.get(lid, '')
if not u.startswith('https://tarktee.transpordiamet.ee/images/'):
continue
if not inbox(lat, lon, 57.4, 59.9, 21.5, 28.4):
continue
n = re.search(r'/images/(\d+)/', u)
ok.add(n.group(1) if n else lid)
return raw, ok, 'DATEX location joined to image; distinct camera numbers'
@source('Via Lietuva eismoinfo', 'Lithuania')
def lithuania():
rows = get('https://eismoinfo.lt/eismoinfo-backend/camera-info-table', headers={'Accept': 'application/json'}).json()
layers = get('https://eismoinfo.lt/eismoinfo-backend/layer-static-features/VKR', headers={'Accept': 'application/json'}).json()
feats = {}
for L in layers:
if L.get('layer') == 'VKR':
feats = {str(f.get('id')): f for f in L.get('features') or [] if f}
raw, ok = set(), set()
for r in rows:
rid = str(r.get('id') or '')
if not re.fullmatch(r'[1-9]\d{0,8}', rid):
continue
raw.add(rid)
f = feats.get(rid)
try:
lat, lon = f['points'][0]['point']
except Exception:
continue
if not inbox(fnum(lat), fnum(lon), 53.8, 56.5, 20.8, 26.9):
continue
u = urlparse(str(r.get('image') or ''))
if f'{u.scheme}://{u.netloc}' != 'https://eismoinfo.lt' or u.path != '/eismoinfo-backend/image-provider/camera/last' or parse_qs(u.query).get('id', [''])[0] != rid:
continue
ok.add(rid)
return raw, ok, 'camera table joined to VKR layer; official still endpoint'
@source('Live Traffic NSW', 'Australia')
def nsw():
d = get('https://data.livetraffic.com/cameras/traffic-cam.json').json()
raw, ok = set(), set()
for f in d['features']:
p = f.get('properties') or {}
fid = str(f.get('id') or p.get('id') or p.get('title') or '').strip()
if not fid:
continue
raw.add(fid)
href = (p.get('href') or '').replace('http://', 'https://')
c = (f.get('geometry') or {}).get('coordinates') or [None, None]
if href.startswith('https://webcams.transport.nsw.gov.au/') and inbox(fnum(c[1]), fnum(c[0]), -38.0, -28.0, 140.0, 154.5):
ok.add(fid)
return raw, ok, 'one feature per camera; official webcams.transport.nsw.gov.au still'
@source('City of Calgary', 'Canada')
def calgary():
d = get('https://data.calgary.ca/resource/k7p9-kppz.json?$limit=5000').json()
raw, ok = set(), set()
for c in d:
url = ((c.get('camera_url') or {}).get('url') or '').strip()
if not url:
continue
u = re.sub(r'^http://', 'https://', url)
m = re.search(r'loc(\d+)\.jpg$', urlparse(u).path, re.I)
key = m.group(1) if m else urlparse(u).path
raw.add(key)
coords = (c.get('point') or {}).get('coordinates') or [None, None]
if u.startswith('https://trafficcam.calgary.ca/') and inbox(fnum(coords[1]), fnum(coords[0]), 50.7, 51.4, -114.5, -113.7):
ok.add(key)
return raw, ok, f'{len(d)} rows ($limit=5000); distinct trafficcam.calgary.ca image ids'
@source('City of Toronto RESCU', 'Canada')
def toronto():
t = get('https://opendata.toronto.ca/transportation/tmc/rescucameraimages/Data/tmcearthcameras.json').text
d = json.loads(t[t.index('(') + 1: t.rindex(')')])
raw, ok = set(), set()
for c in d['Data']:
n = str(c.get('Number') or '').strip()
if not n:
continue
raw.add(n)
if re.fullmatch(r'\d{1,6}', n) and inbox(fnum(c.get('Latitude')), fnum(c.get('Longitude')), 43.4, 44.0, -79.8, -79.0):
ok.add(n)
return raw, ok, 'one row per numbered RESCU camera'
@source('Servei Català de Trànsit', 'Spain')
def catalonia():
x = get('https://www.gencat.cat/transit/opendata/cameres.xml').text
hosts = {'mct.gencat.cat': '/mct2bo/', 'www.bcn.cat': '/transit/imatges/', 'emap.terrassa.cat': '/it_terrassa/'}
raw, ok = set(), set()
for m in re.finditer(r'<gml:featureMember>([\s\S]*?)</gml:featureMember>', x):
b = m.group(1)
co = re.search(r'<gml:coordinates[^>]*>([^<]+)</gml:coordinates>', b)
lk = re.search(r'<cite:link>([^<]*)</cite:link>', b)
if not co or not lk:
continue
link = lk.group(1).replace('&amp;', '&').strip()
u = urlparse(link)
cam = parse_qs(u.query).get('sctidcam', [''])[0]
key = f'sct:{cam}' if u.hostname == 'mct.gencat.cat' and cam else f'{u.hostname}{u.path}'
raw.add(key)
lon, lat = (fnum(v) for v in co.group(1).split(',')[:2])
if not inbox(lat, lon, 40.4, 42.95, 0.1, 3.4):
continue
d = hosts.get(u.hostname or '')
if not d or not u.path.startswith(d) or u.username or u.password or u.port:
continue
f = u.path[len(d):]
if not f or '/' in f:
continue
if u.hostname == 'mct.gencat.cat' and (f != 'RenderService' or not re.fullmatch(r'[a-z0-9]+\.(gif|jpe?g)', cam, re.I)):
continue
ok.add(key)
return raw, ok, 'SCT + Barcelona IMI + Terrassa; keyed by frame file (feed fid is unstable)'
@source('City of Austin', 'United States')
def austin():
d = get('https://data.austintexas.gov/api/views/b4k4-adkb/rows.json?accessType=DOWNLOAD').json()
cols = [c['fieldName'] for c in d['meta']['view']['columns']]
raw, ok = set(), set()
for row in d['data']:
r = dict(zip(cols, row))
cid = str(r.get('camera_id') or '').strip()
if not cid:
continue
raw.add(cid)
st = (r.get('camera_status') or '').strip().upper()
if st and st != 'TURNED_ON':
continue
m = re.match(r'POINT \(([-\d.]+) ([-\d.]+)\)', r.get('location') or '')
if not m or not inbox(fnum(m.group(2)), fnum(m.group(1)), 29.9, 30.8, -98.2, -97.3):
continue
ok.add(cid)
return raw, ok, 'camera_status TURNED_ON'
TXDOT = 'ABL AMA ATL AUS BMT BWD BRY CHS CRP DAL ELP FTW HOU LRD LBB LFK ODA PAR PHR SJT SAT TYL WAC WFS YKM'.split()
@source('TxDOT ITS', 'United States')
def txdot():
def one(code):
for attempt in range(2):
try:
return code, get(f'https://its.txdot.gov/its/DistrictIts/GetCctvStatusListByDistrict?districtCode={code}',
headers={'Accept': 'application/json'}, timeout=60).json()
except Exception as e:
err = e
return code, err
with ThreadPoolExecutor(8) as ex:
res = list(ex.map(one, TXDOT))
raw, ok, notes, per = set(), set(), [], {}
for code, d in res:
if isinstance(d, Exception):
notes.append(f'{code} failed: {str(d)[:50]}')
continue
n = 0
for rows in (d.get('roadwayCctvStatuses') or {}).values():
for r in rows or []:
icd = str(r.get('icd_Id') or '').strip()
if not icd:
continue
raw.add(icd)
if r.get('statusDescription') != 'Device Online' or r.get('hasSnapshot') is False:
continue
lat, lon = r.get('latitude'), r.get('longitude')
if not (isinstance(lat, (int, float)) and isinstance(lon, (int, float)) and inbox(lat, lon, 25.5, 36.7, -106.8, -93.3)):
continue
if icd not in ok:
n += 1
ok.add(icd)
per[code] = n
return raw, ok, f'all 25 districts; Device Online + snapshot; per-district {per}. ' + '; '.join(notes)
@source('Ontario 511', 'Canada')
def ontario():
d = get('https://511on.ca/api/v2/get/cameras?format=json&lang=en', headers={'Accept': 'application/json'}).json()
raw, ok, views = set(), set(), 0
for r in d:
rid = str(r.get('Id') or r.get('id') or '').strip()
if not rid:
continue
raw.add(rid)
if not inbox(fnum(r.get('Latitude')), fnum(r.get('Longitude')), 41.0, 57.5, -95.6, -74.0):
continue
good = []
for v in r.get('Views') or r.get('views') or []:
if str(v.get('Status') or '').strip().lower() != 'enabled':
continue
u = urlparse(str(v.get('Url') or '').strip())
if u.scheme == 'https' and (u.hostname == '511on.ca' or (u.hostname or '').endswith('.traveliq.co')) and re.fullmatch(r'/map/Cctv/[A-Za-z0-9_.-]+', u.path):
good.append(v)
if good:
ok.add(rid)
views += len(good)
return raw, ok, f'counted per camera row, not per view; {views} enabled views'
@source('DelDOT', 'United States')
def deldot():
d = get('https://tmc.deldot.gov/json/videocamera.json', headers={'Accept': 'application/json'}).json()
raw, ok = set(), set()
for r in d.get('videoCameras') or []:
cid = str(r.get('id') or '').strip()
if not cid:
continue
raw.add(cid)
if str(r.get('status')).lower() != 'active':
continue
if not inbox(fnum(r.get('lat')), fnum(r.get('lon')), 38.4, 39.9, -75.85, -75.0):
continue
u = urlparse(str((r.get('urls') or {}).get('m3u8s') or ''))
if u.scheme != 'https' or u.hostname != 'video.deldot.gov' or u.port not in (None, 443) or u.username or not re.fullmatch(r'/live/[A-Za-z0-9_.-]+/playlist\.m3u8', u.path):
continue
if not re.fullmatch(r'[A-Za-z0-9_-]{1,80}', cid):
continue
ok.add(cid)
return raw, ok, 'status active + official HLS stream'
def run(label):
fn, country = SRC[label]
t0 = time.time()
try:
raw, ok, note = fn()
if not ok:
raise RuntimeError('catalog parsed to zero cameras')
return label, dict(country=country, raw=len(raw), valid=len(ok), note=note, secs=round(time.time() - t0, 1))
except Exception as e:
return label, dict(country=country, error=f'{type(e).__name__}: {str(e)[:200]}', secs=round(time.time() - t0, 1))
if __name__ == '__main__':
out = sys.argv[1]
detail = sys.argv[2] if len(sys.argv) > 2 else None
with ThreadPoolExecutor(8) as ex:
results = dict(ex.map(run, list(SRC)))
counted = {k: v for k, v in results.items() if 'valid' in v}
for k, v in results.items():
print(f"{k:28s} {v.get('country', ''):15s} raw={v.get('raw', '-'):>6} valid={v.get('valid', '-'):>6} {v.get('error') or v.get('note')}")
total = sum(v['valid'] for v in counted.values())
res = {
'cctv_catalog': total,
'cctv_sources': {k: v['valid'] for k, v in counted.items()},
'cctv_countries': len({v['country'] for v in counted.values()}),
'fetched_utc': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
}
print('TOTAL valid', total, 'raw', sum(v['raw'] for v in counted.values()), 'countries', res['cctv_countries'])
json.dump(res, open(out, 'w'), ensure_ascii=False, indent=2)
if detail:
json.dump(results, open(detail, 'w'), ensure_ascii=False, indent=2)
+406
View File
@@ -0,0 +1,406 @@
#!/usr/bin/env python3
"""Download real current still frames from keyless public traffic-camera feeds.
Stage 1 (this script): build candidate lists from each source's public catalog
(endpoints mirror server/providers/cctv/*), sample a spread of cameras, download
frames, compute quality metrics, write <work>/frames/*.jpg + <work>/candidates.json.
Stage 2 (cctv_select.py) filters, crops to 640x360 and writes cctv/index.json.
Usage: python3 -I cctv_fetch.py <work_dir>
"""
import base64, hashlib, io, json, math, random, re, sys, time
import xml.etree.ElementTree as ET
from concurrent.futures import ThreadPoolExecutor
import requests
from PIL import Image, ImageStat, ImageFilter
WORK = sys.argv[1]
import os
os.makedirs(f'{WORK}/frames', exist_ok=True)
UA_BROWSER = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'
S = requests.Session()
S.headers.update({'User-Agent': UA_BROWSER, 'Digitraffic-User': 'gods-eye-intro'})
rng = random.Random(42)
def get(url, **kw):
kw.setdefault('timeout', 40)
r = S.get(url, **kw)
r.raise_for_status()
return r
def C(source, name, city, country, lat, lon, url, kind='image'):
return dict(source=source, name=name.strip(), city=city, country=country, lat=round(float(lat), 5), lon=round(float(lon), 5), url=url, kind=kind)
def nearest(cands, anchors, n):
"""Spread a pick across anchors: round-robin, nearest cameras to each anchor first, randomized a bit."""
buckets = []
for (alat, alon, label) in anchors:
b = sorted(cands, key=lambda c: (c['lat'] - alat) ** 2 + ((c['lon'] - alon) * math.cos(math.radians(alat))) ** 2)[:max(n * 3, 12)]
rng.shuffle(b)
for c in b:
c['city'] = label if c.get('city') in (None, '', '?') else c['city']
buckets.append(b)
out, seen = [], set()
while len(out) < n and any(buckets):
for b in buckets:
while b:
c = b.pop()
if c['url'] not in seen:
seen.add(c['url']); out.append(c); break
return out[:n]
def sample(cands, n):
cands = list(cands)
rng.shuffle(cands)
return cands[:n]
SOURCES = {}
def source(fn):
SOURCES[fn.__name__] = fn
return fn
@source
def tfl(n):
d = get('https://api.tfl.gov.uk/Place/Type/JamCam').json()
out = []
for p in d:
props = {a['key']: a['value'] for a in p.get('additionalProperties', [])}
if props.get('available') != 'true' or not props.get('imageUrl'):
continue
out.append(C('TfL JamCams', p['commonName'], 'London', 'United Kingdom', p['lat'], p['lon'], props['imageUrl']))
return nearest(out, [(51.5074, -0.1278, 'London')], n)
@source
def nyc(n):
d = get('https://webcams.nyctmc.org/api/cameras').json()
out = [C('NYC DOT', c['name'], f"New York ({c.get('area')})", 'United States', c['latitude'], c['longitude'],
f"https://webcams.nyctmc.org/api/cameras/{c['id']}/image") for c in d if str(c.get('isOnline')).lower() == 'true']
return nearest(out, [(40.758, -73.9855, 'New York'), (40.7128, -74.006, 'New York'), (40.6782, -73.9442, 'New York'), (40.7282, -73.7949, 'New York')], n)
@source
def caltrans(n):
out = []
for dist, city in [(4, 'San Francisco Bay Area'), (7, 'Los Angeles'), (11, 'San Diego'), (3, 'Sacramento')]:
try:
d = get(f'https://cwwp2.dot.ca.gov/data/d{dist}/cctv/cctvStatusD{dist:02d}.json').json()
except Exception as e:
print('caltrans', dist, e); continue
for row in d['data']:
c = row['cctv']
url = (c.get('imageData', {}).get('static') or {}).get('currentImageURL')
if c.get('inService') != 'true' or not url:
continue
loc = c['location']
out.append(C('Caltrans CCTV', loc['locationName'], loc.get('nearbyPlace') or city, 'United States', loc['latitude'], loc['longitude'], url))
return nearest(out, [(37.7793, -122.4193, 'San Francisco'), (34.0537, -118.2428, 'Los Angeles'), (32.7157, -117.1611, 'San Diego'), (38.5816, -121.4944, 'Sacramento')], n)
@source
def wsdot(n):
d = json.loads(get('https://data.wsdot.wa.gov/travelcenter/Cameras.json').content.decode('cp1252'))
out = []
for f in d['features']:
a, g = f['attributes'], f['geometry']
url = a.get('ImageURL') or ''
if not url.startswith('https://images.wsdot.wa.gov/'):
continue
lon = g['x'] / 6378137 * 180 / math.pi
lat = (2 * math.atan(math.exp(g['y'] / 6378137)) - math.pi / 2) * 180 / math.pi
out.append(C('WSDOT', a['CameraTitle'], '', 'United States', lat, lon, url))
return nearest(out, [(47.6062, -122.3321, 'Seattle'), (47.6588, -117.426, 'Spokane')], n)
@source
def kingco(n):
d = get('https://services.arcgis.com/Ej0PsM5Aw677QF1W/arcgis/rest/services/TRAFFICCAMERA_POINT_2029/FeatureServer/0/query?where=1%3D1&outFields=AssetID,Location,ImageURL,Live,Model,Manufacturer,Owner,CurrentStatus&f=geojson').json()
out = []
for f in d['features']:
p = f['properties']
url = (p.get('ImageURL') or '').replace('http://info.kingcounty.gov/', 'https://info.kingcounty.gov/')
if p.get('CurrentStatus') != 'Active' or not url.startswith('https://info.kingcounty.gov/'):
continue
lon, lat = f['geometry']['coordinates'][:2]
out.append(C('King County DOT', p['Location'], 'King County, WA', 'United States', lat, lon, url))
return sample(out, n)
@source
def drivebc(n):
d = get('https://www.drivebc.ca/api/webcams/').json()
out = []
for c in d:
if not c.get('is_on') or c.get('marked_stale') or not c.get('should_appear', True):
continue
lon, lat = c['location']['coordinates'][:2]
out.append(C('DriveBC', c['name'], c.get('region_name') or 'British Columbia', 'Canada', lat, lon, f"https://www.drivebc.ca/images/{c['id']}.jpg"))
return nearest(out, [(49.2827, -123.1207, 'Vancouver'), (48.4284, -123.3656, 'Victoria'), (50.1163, -122.9574, 'Whistler')], n)
@source
def calgary(n):
d = get('https://data.calgary.ca/resource/k7p9-kppz.json?$limit=500').json()
out = []
for c in d:
url = (c.get('camera_url') or {}).get('url', '').replace('http://', 'https://')
if not url.startswith('https://trafficcam.calgary.ca/'):
continue
lon, lat = c['point']['coordinates'][:2]
out.append(C('City of Calgary', c['camera_location'], 'Calgary', 'Canada', lat, lon, url))
return nearest(out, [(51.0461, -114.0626, 'Calgary')], n)
@source
def toronto(n):
t = get('https://opendata.toronto.ca/transportation/tmc/rescucameraimages/Data/tmcearthcameras.json').text
d = json.loads(t[t.index('(') + 1: t.rindex(')')])
out = [C('City of Toronto RESCU', c['Name'].title(), 'Toronto', 'Canada', c['Latitude'], c['Longitude'],
f"https://opendata.toronto.ca/transportation/tmc/rescucameraimages/CameraImages/loc{c['Number']}.jpg")
for c in d['Data'] if re.fullmatch(r'\d+', str(c['Number']))]
return nearest(out, [(43.6532, -79.3832, 'Toronto')], n)
@source
def txdot(n):
out = []
for code, city in [('HOU', 'Houston'), ('DAL', 'Dallas'), ('AUS', 'Austin'), ('SAT', 'San Antonio'), ('ELP', 'El Paso')]:
try:
d = get(f'https://its.txdot.gov/its/DistrictIts/GetCctvStatusListByDistrict?districtCode={code}').json()
except Exception as e:
print('txdot', code, e); continue
seen = set()
for rows in d['roadwayCctvStatuses'].values():
for r in rows:
if r.get('statusDescription') != 'Device Online' or r.get('hasSnapshot') is False or r['icd_Id'] in seen:
continue
if not isinstance(r.get('latitude'), (int, float)) or not r['latitude']:
continue
seen.add(r['icd_Id'])
url = 'https://its.txdot.gov/its/DistrictIts/GetCctvSnapshotByIcdId?' + requests.compat.urlencode({'icdId': r['icd_Id'], 'districtCode': code})
out.append(C('TxDOT ITS', r['name'], city, 'United States', r['latitude'], r['longitude'], url, kind='txdot'))
return nearest(out, [(29.7604, -95.3698, 'Houston'), (32.7767, -96.797, 'Dallas'), (29.4241, -98.4936, 'San Antonio'), (31.7619, -106.485, 'El Paso')], n)
@source
def austin(n):
d = get('https://data.austintexas.gov/api/views/b4k4-adkb/rows.json?accessType=DOWNLOAD').json()
cols = [c['fieldName'] for c in d['meta']['view']['columns']]
out = []
for row in d['data']:
r = dict(zip(cols, row))
if (r.get('camera_status') or '').strip().upper() != 'TURNED_ON':
continue
m = re.match(r'POINT \(([-\d.]+) ([-\d.]+)\)', r.get('location') or '')
if not m or not r.get('camera_id'):
continue
out.append(C('City of Austin', (r.get('location_name') or '').strip(), 'Austin', 'United States', m.group(2), m.group(1),
f"https://cctv.austinmobility.io/image/{r['camera_id']}.jpg"))
return nearest(out, [(30.2672, -97.7431, 'Austin')], n)
@source
def fintraffic(n):
d = get('https://tie.digitraffic.fi/api/weathercam/v1/stations').json()
out = []
for f in d['features']:
p = f['properties']
if p.get('collectionStatus') != 'GATHERING':
continue
lon, lat = f['geometry']['coordinates'][:2]
for pr in p.get('presets', []):
if pr.get('inCollection'):
out.append(C('Fintraffic Digitraffic', p['name'].replace('_', ' '), '', 'Finland', lat, lon, f"https://weathercam.digitraffic.fi/{pr['id']}.jpg"))
break
return nearest(out, [(60.1699, 24.9384, 'Helsinki'), (61.4978, 23.761, 'Tampere'), (65.0121, 25.4651, 'Oulu'), (66.5039, 25.7294, 'Rovaniemi'), (60.4518, 22.2666, 'Turku')], n)
@source
def vegvesen(n):
d = get('https://ogckart-sn1.atlas.vegvesen.no/ogc/features/v1/collections/datex_3_1:CctvSimple/items?f=application/geo%2Bjson&limit=5000').json()
out = []
for f in d['features']:
p = f['properties']
url = p.get('stillImageUrl') or ''
if not url.startswith('https://kamera.atlas.vegvesen.no/api/images/'):
continue
lon, lat = f['geometry']['coordinates'][:2]
name = ' - '.join(x for x in [p.get('description'), p.get('roadNumber')] if x)
out.append(C('Statens vegvesen', name, '', 'Norway', lat, lon, url))
return nearest(out, [(59.9139, 10.7522, 'Oslo'), (60.3913, 5.3221, 'Bergen'), (63.4305, 10.3951, 'Trondheim'), (69.6492, 18.9553, 'Tromsø'), (58.97, 5.7331, 'Stavanger')], n)
@source
def tallinn(n):
d = json.load(open('/home/user/gods-eye-view/config/cctv_sources.tallinn.json'))
out = [C('City of Tallinn', c['name'], 'Tallinn', 'Estonia', c['lat'], c['lon'], c['snapshotUrl']) for c in d if c.get('snapshotUrl', '').startswith('https://ristmikud.tallinn.ee/')]
return sample(out, n)
@source
def tarktee(n):
ns = {'d': 'http://datex2.eu/schema/2/2_0'}
loc = ET.fromstring(get('https://tarktee.transpordiamet.ee/api/v1/datex/roadCameraLocations').content)
locs = {}
for pl in loc.iter('{http://datex2.eu/schema/2/2_0}predefinedLocation'):
lat = pl.find('.//d:latitude', ns); lon = pl.find('.//d:longitude', ns); nm = pl.find('.//d:value', ns)
if lat is not None and lon is not None:
locs[pl.get('id')] = (float(lat.text), float(lon.text), nm.text if nm is not None else '')
img = ET.fromstring(get('https://tarktee.transpordiamet.ee/api/v1/datex/roadCameraImages').content)
out = []
for tv in img.iter('{http://datex2.eu/schema/2/2_0}linearTrafficView'):
ref = tv.find('d:linearPredefinedLocationReference', ns)
u = tv.find('.//d:urlLinkAddress', ns)
if ref is None or u is None or ref.get('id') not in locs:
continue
lat, lon, nm = locs[ref.get('id')]
out.append(C('Transpordiamet Tarktee', nm, '', 'Estonia', lat, lon, u.text))
return nearest(out, [(59.437, 24.753, 'Tallinn'), (58.378, 26.729, 'Tartu'), (58.3859, 24.4971, 'Pärnu')], n)
@source
def lithuania(n):
rows = get('https://eismoinfo.lt/eismoinfo-backend/camera-info-table').json()
layer = get('https://eismoinfo.lt/eismoinfo-backend/layer-static-features/VKR').json()
feats = {str(f.get('id')): f for L in layer for f in L.get('features', [])}
out = []
for r in rows:
f = feats.get(str(r.get('id')))
try:
lat, lon = f['points'][0]['point']
except Exception:
continue
out.append(C('Via Lietuva eismoinfo', r['name'], '', 'Lithuania', lat, lon, f"https://eismoinfo.lt/eismoinfo-backend/image-provider/camera/last?id={r['id']}"))
return nearest(out, [(54.6872, 25.2797, 'Vilnius'), (54.8985, 23.9036, 'Kaunas'), (55.7033, 21.1443, 'Klaipėda')], n)
@source
def catalonia(n):
x = get('https://www.gencat.cat/transit/opendata/cameres.xml').text
out = []
for m in re.finditer(r'<gml:coordinates[^>]*>([-\d.]+),([-\d.]+)</gml:coordinates>.*?<cite:carretera>(.*?)</cite:carretera>.*?<cite:municipi>(.*?)</cite:municipi>.*?<cite:link>(.*?)</cite:link>.*?<cite:font>(.*?)</cite:font>', x, re.S):
lon, lat, road, muni, link, font = m.groups()
link = link.replace('&amp;', '&').replace('http://', 'https://')
if font not in ('SCT', 'IMI', 'Terrassa'):
continue
city = 'Barcelona' if font == 'IMI' else ('Terrassa' if font == 'Terrassa' else muni)
out.append(C('Servei Català de Trànsit', f'{road} {muni}', city, 'Spain', lat, lon, link))
return nearest(out, [(41.3874, 2.1686, 'Barcelona'), (41.5636, 2.0111, 'Terrassa'), (41.45, 2.2, 'Barcelona area')], n)
@source
def nsw(n):
d = get('https://data.livetraffic.com/cameras/traffic-cam.json').json()
out = []
for f in d['features']:
p = f['properties']
href = (p.get('href') or '').replace('http://', 'https://')
if not href.startswith('https://webcams.transport.nsw.gov.au/'):
continue
lon, lat = f['geometry']['coordinates'][:2]
out.append(C('Live Traffic NSW', p['title'], '', 'Australia', lat, lon, href))
return nearest(out, [(-33.8688, 151.2093, 'Sydney'), (-32.9283, 151.7817, 'Newcastle'), (-34.4278, 150.8931, 'Wollongong'), (-33.8, 151.0, 'Western Sydney')], n)
@source
def singapore(n):
d = get('https://api.data.gov.sg/v1/transport/traffic-images').json()
out = [C('LTA Singapore (data.gov.sg)', f"Camera {c['camera_id']}", 'Singapore', 'Singapore', c['location']['latitude'], c['location']['longitude'], c['image'])
for c in d['items'][0]['cameras']]
return sample(out, n)
@source
def hongkong(n):
x = ET.fromstring(get('https://static.data.gov.hk/td/traffic-snapshot-images/code/Traffic_Camera_Locations_En.xml').content)
out = []
for im in x.iter('image'):
g = lambda k: (im.findtext(k) or '').strip()
if not g('url').startswith('https://tdcctv.data.one.gov.hk/'):
continue
out.append(C('HK Transport Department', re.sub(r'\s*\[.*?\]$', '', g('description')), f"Hong Kong ({g('district')})", 'Hong Kong', g('latitude'), g('longitude'), g('url')))
return sample(out, n)
@source
def nz(n):
d = get('https://trafficnz.info/service/traffic/rest/4/cameras/all', headers={'Accept': 'application/json'}).json()
out = []
for c in d['response']['camera']:
if c.get('offline') or c.get('underMaintenance'):
continue
out.append(C('NZTA Waka Kotahi', (c.get('name') or '').strip() + (' - ' + c['description'] if c.get('description') else ''),
(c.get('region') or {}).get('name', ''), 'New Zealand', c['latitude'], c['longitude'], 'https://trafficnz.info' + c['imageUrl']))
return nearest(out, [(-36.8485, 174.7633, 'Auckland'), (-41.2865, 174.7762, 'Wellington'), (-43.5321, 172.6362, 'Christchurch'), (-37.787, 175.279, 'Hamilton'), (-45.0312, 168.6626, 'Queenstown')], n)
TARGETS = dict(nsw=26, hongkong=24, singapore=18, nz=20, nyc=20, caltrans=18, wsdot=12, kingco=6, drivebc=14, calgary=8,
toronto=10, txdot=10, austin=8, tfl=14, fintraffic=10, vegvesen=10, tallinn=6, tarktee=6, lithuania=8, catalonia=10)
def fetch_frame(c):
try:
hdr = {}
if c['kind'] == 'txdot':
r = S.get(c['url'], timeout=30, headers={'Accept': 'application/json'})
r.raise_for_status()
js = r.json()
b64 = js.get('snippet') or js.get('Snippet') or ''
data = base64.b64decode(b64.split(',')[-1])
else:
r = S.get(c['url'], timeout=30)
r.raise_for_status()
data = r.content
im = Image.open(io.BytesIO(data))
im.load()
im = im.convert('RGB')
g = im.convert('L')
st = ImageStat.Stat(g)
edges = ImageStat.Stat(g.resize((320, 180)).filter(ImageFilter.FIND_EDGES)).mean[0]
sat = ImageStat.Stat(im.convert('HSV').split()[1]).mean[0]
small = g.resize((16, 9))
px = list(small.tobytes())
ahash = ''.join('1' if p > sum(px) / len(px) else '0' for p in px)
h = hashlib.md5(c['url'].encode()).hexdigest()[:12]
path = f'{WORK}/frames/{h}.jpg'
im.save(path, quality=95)
c.update(ok=True, file=path, w=im.width, h=im.height, mean=round(st.mean[0], 1), std=round(st.stddev[0], 1),
edges=round(edges, 2), sat=round(sat, 1), ahash=ahash, bytes=len(data), fetched=time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()))
except Exception as e:
c.update(ok=False, err=str(e)[:160])
return c
if __name__ == '__main__':
only = sys.argv[2].split(',') if len(sys.argv) > 2 else list(SOURCES)
cands = []
for name in only:
try:
got = SOURCES[name](TARGETS.get(name, 8))
print(f'{name}: {len(got)} candidates')
for g in got:
g['src_key'] = name
cands += got
except Exception as e:
print(f'{name}: CATALOG FAILED {e}')
with ThreadPoolExecutor(24) as ex:
res = list(ex.map(fetch_frame, cands))
prev = []
pth = f'{WORK}/candidates.json'
if os.path.exists(pth):
prev = [p for p in json.load(open(pth)) if p['src_key'] not in only]
json.dump(prev + res, open(pth, 'w'), ensure_ascii=False, indent=0)
from collections import Counter
print('ok per source', Counter(c['src_key'] for c in res if c['ok']))
print('fail per source', Counter(c['src_key'] for c in res if not c['ok']))
for c in res:
if not c['ok']:
print(' FAIL', c['src_key'], c['url'][:90], c['err'][:100])
+42
View File
@@ -0,0 +1,42 @@
#!/usr/bin/env python3
"""Stage 2: pick a spread of good frames from cctv_fetch.py output, cover-crop to 640x360, write cctv/index.json.
Usage: python3 -I cctv_select.py <work_dir> <out_cctv_dir>
EXCLUDE lists frames rejected on visual review of the contact sheets (placeholders such as
'Temporarily Unavailable', 'Kamera er midlertidig ute av drift', 'NB! ÜHENDUS KAAMERAGA PUUDUB',
'Interrupció temporal de la imatge', and an all-black King County frame)."""
import json, os, sys
from collections import Counter
from PIL import Image
W, OUT = sys.argv[1], sys.argv[2]
ok = json.load(open(f'{W}/ok.json'))
EXCLUDE = {40, 43, 45, 46, 49, 66, 126, 139, 148, 150}
PREFER = {'hongkong': [187, 195, 197, 198, 201, 205, 206, 196, 200, 202]} # fewest privacy-mask polygons
QUOTA = dict(nsw=12, hongkong=10, singapore=6, nz=12, nyc=9, caltrans=6, wsdot=6, kingco=3, drivebc=6, calgary=4, toronto=6,
txdot=4, austin=4, tfl=8, fintraffic=4, vegvesen=4, tallinn=3, tarktee=3, lithuania=4, catalonia=3)
hash_count = Counter(x['ahash'] for x in ok)
good = [x for x in ok if x['i'] not in EXCLUDE and hash_count[x['ahash']] == 1 and x['w'] >= 320 and x['mean'] > 25 and x['std'] > 15]
pick = []
for src, q in QUOTA.items():
pool = [x for x in good if x['src_key'] == src]
pref = [x for i in PREFER.get(src, []) for x in pool if x['i'] == i]
rest = sorted([x for x in pool if x not in pref], key=lambda x: -(x['std'] * x['edges']))
pick += (pref + rest)[:q]
os.makedirs(OUT, exist_ok=True)
for f in os.listdir(OUT):
if f.startswith('cam_') and f.endswith('.jpg'):
os.remove(os.path.join(OUT, f))
index = []
for n, x in enumerate(pick, 1):
im = Image.open(x['file']).convert('RGB')
w, h = im.size
s = max(640 / w, 360 / h)
im = im.resize((max(640, round(w * s)), max(360, round(h * s))), Image.LANCZOS)
w, h = im.size
l, t = (w - 640) // 2, (h - 360) // 2
im = im.crop((l, t, l + 640, t + 360))
fn = f'cam_{n:03d}.jpg'
im.save(os.path.join(OUT, fn), quality=88)
index.append(dict(file=fn, name=x['name'], city=x['city'], country=x['country'], lat=x['lat'], lon=x['lon'], source=x['source'],
fetched_utc=x['fetched'], orig_size=[x['w'], x['h']], url=x['url']))
json.dump(index, open(os.path.join(OUT, 'index.json'), 'w'), ensure_ascii=False, indent=1)
print(len(index), Counter(i['country'] for i in index))
+33
View File
@@ -0,0 +1,33 @@
#!/usr/bin/env bash
# Re-fetch the raw keyless feeds used by process_basic.py / process_tle.py / process_vessels.py.
# Usage: fetch_basic.sh <raw_dir> then:
# python3 -I process_basic.py <raw_dir> ../../data
# python3 -I process_tle.py <raw_dir> ../../data
# python3 -I process_vessels.py <raw_dir> ../../data
# <venv with eccodes,numpy,requests>/python -I fetch_wind.py https://noaa-gfs-bdp-pds.s3.amazonaws.com/gfs.YYYYMMDD/HH/atmos/gfs.tHHz.pgrb2.1p00.f006 ../../data/wind.json
# python -I fetch_bonus.py ../../data <raw_dir>
# python -I cctv_fetch.py <work_dir> && python -I cctv_select.py <work_dir> ../../cctv (after rebuilding ok.json/contact sheets and re-reviewing EXCLUDE)
set -u
R="$1"; mkdir -p "$R"; cd "$R"
date -u +%Y-%m-%dT%H:%M:%SZ > fetch_start.txt
CT='gods-eye-view-celestrak-proxy/1.0 (+https://github.com/bilawalsidhu/gods-eye-view)' # CelesTrak 403s bulk groups without a descriptive UA
f() { curl -sS -m 300 --compressed "${@:3}" -o "$1" -w "$1 %{http_code} %{size_download}\n" "$2"; }
f opensky.json https://opensky-network.org/api/states/all &
f mil.json https://api.adsb.lol/v2/mil &
f mil_adsbfi.json https://opendata.adsb.fi/api/v2/mil &
f quakes.geojson https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson &
f viirs_snpp.csv https://firms.modaps.eosdis.nasa.gov/data/active_fire/suomi-npp-viirs-c2/csv/SUOMI_VIIRS_C2_Global_24h.csv &
f viirs_n20.csv https://firms.modaps.eosdis.nasa.gov/data/active_fire/noaa-20-viirs-c2/csv/J1_VIIRS_C2_Global_24h.csv &
f modis.csv https://firms.modaps.eosdis.nasa.gov/data/active_fire/modis-c6.1/csv/MODIS_C6_1_Global_24h.csv &
f cables.json https://www.submarinecablemap.com/api/v3/cable/cable-geo.json &
f coast50.geojson https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_50m_coastline.geojson &
f countries110.geojson https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_110m_admin_0_countries.geojson &
f places.geojson https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_populated_places_simple.geojson &
f airports.csv https://davidmegginson.github.io/ourairports-data/airports.csv &
f kystverket_ais.json https://kystdatahuset.kystverket.no/ws/api/ais/realtime/geojson &
f dt_ais_from.json "https://meri.digitraffic.fi/api/ais/v1/locations?from=$(( ($(date +%s)-172800)*1000 ))" -H "Digitraffic-User: gods-eye-intro" &
f digitraffic_vessels.json https://meri.digitraffic.fi/api/ais/v1/vessels -H "Digitraffic-User: gods-eye-intro" &
wait
# CelesTrak: sequential (parallel requests get connection resets); FORMAT=3le endpoint answered when FORMAT=tle timed out
f tle_active3.txt "https://celestrak.org/NORAD/elements/gp.php?GROUP=active&FORMAT=3le" -A "$CT"
for g in stations starlink gps-ops geo weather; do sleep 3; f tle_$g.txt "https://celestrak.org/NORAD/elements/gp.php?GROUP=$g&FORMAT=tle" -A "$CT"; done
+133
View File
@@ -0,0 +1,133 @@
#!/usr/bin/env python3
"""Bonus layers: radio stations (Radio Browser), GBFS bike stations, launch pads (Launch Library 2),
power plants (WRI Global Power Plant Database). Usage: python3 -I fetch_bonus.py <out_data_dir> <raw_dir>"""
import csv, io, json, os, sys, time, zipfile
import requests
OUT, RAW = sys.argv[1], sys.argv[2]
only = sys.argv[3].split(',') if len(sys.argv) > 3 else ['radio', 'gbfs', 'pads', 'power']
S = requests.Session()
S.headers['User-Agent'] = 'GodsEyeView/1.0 intro-sequence data snapshot'
log = {}
def dump(name, obj):
with open(os.path.join(OUT, name), 'w') as f:
json.dump(obj, f, separators=(',', ':'), ensure_ascii=False)
if 'radio' in only:
rows, used = [], None
for host in ['https://de1.api.radio-browser.info', 'https://de2.api.radio-browser.info', 'https://nl1.api.radio-browser.info']:
try:
r = S.get(f'{host}/json/stations/search', params={'has_geo_info': 'true', 'hidebroken': 'true', 'limit': 100000}, timeout=180,
headers={'User-Agent': 'GodsEyeView/1.0 (Radio Browser directory client)'})
r.raise_for_status()
data = r.json(); used = host
break
except Exception as e:
print('radio', host, e)
seen = set()
for s in data:
lat, lon = s.get('geo_lat'), s.get('geo_long')
if lat is None or lon is None or (abs(lat) < 0.01 and abs(lon) < 0.01):
continue
key = (round(lon, 3), round(lat, 3), s['name'].strip())
if key in seen:
continue
seen.add(key)
rows.append([round(lon, 4), round(lat, 4), s['name'].strip(), s.get('countrycode') or ''])
dump('radio.json', {'cols': ['lon', 'lat', 'name', 'cc'], 'rows': rows})
log['radio'] = dict(url=f'{used}/json/stations/search?has_geo_info=true&hidebroken=true&limit=100000', rows=len(rows))
print('radio', len(rows))
if 'gbfs' in only:
SYSTEMS = {
'Citi Bike (New York)': 'https://gbfs.lyft.com/gbfs/2.3/bkn/en/station_information.json',
'Divvy (Chicago)': 'https://gbfs.lyft.com/gbfs/2.3/chi/en/station_information.json',
'Bay Wheels (SF Bay)': 'https://gbfs.lyft.com/gbfs/2.3/bay/en/station_information.json',
'Capital Bikeshare (DC)': 'https://gbfs.lyft.com/gbfs/2.3/dca-cabi/en/station_information.json',
'Bluebikes (Boston)': 'https://gbfs.bluebikes.com/gbfs/en/station_information.json',
'Bike Share Toronto': 'https://toronto.publicbikesystem.net/customer/gbfs/v2/en/station_information.json',
'BIXI (Montreal)': 'https://gbfs.velobixi.com/gbfs/en/station_information.json',
'Ecobici (Mexico City)': 'https://gbfs.mex.lyftbikes.com/gbfs/en/station_information.json',
'Velib Metropole (Paris)': 'https://velib-metropole-opendata.smovengo.cloud/opendata/Velib_Metropole/station_information.json',
'Bicing (Barcelona)': 'https://barcelona.publicbikesystem.net/customer/gbfs/v2/en/station_information.json',
'BiciMAD (Madrid)': 'https://madrid.publicbikesystem.net/customer/gbfs/v2/en/station_information.json',
'Bike Sampa (Sao Paulo)': 'https://saopaulo.publicbikesystem.net/customer/gbfs/v2/en/station_information.json',
'Bike Rio (Rio de Janeiro)': 'https://riodejaneiro.publicbikesystem.net/customer/gbfs/v2/en/station_information.json',
'Bike Santiago': 'https://santiago.publicbikesystem.net/customer/gbfs/v2/en/station_information.json',
'Tembici Bogota': 'https://bogota.publicbikesystem.net/customer/gbfs/v2/en/station_information.json',
'Indego (Philadelphia)': 'https://gbfs.bcycle.com/bcycle_indego/station_information.json',
'Metro Bike Share (LA)': 'https://gbfs.bcycle.com/bcycle_lametro/station_information.json',
'Oslo Bysykkel': 'https://gbfs.urbansharing.com/oslobysykkel.no/station_information.json',
}
rows, per = [], {}
for name, url in SYSTEMS.items():
try:
d = S.get(url, timeout=60, headers={'Client-Identifier': 'gods-eye-view'}).json()
st = d['data']['stations']
for s in st:
rows.append([round(float(s['lon']), 5), round(float(s['lat']), 5)])
per[name] = len(st)
except Exception as e:
per[name] = f'FAILED {str(e)[:80]}'
# nextbike: one keyless global live feed covering every nextbike city
try:
d = S.get('https://api.nextbike.net/maps/nextbike-live.json', timeout=120).json()
n = 0
for c in d['countries']:
for city in c.get('cities', []):
for p in city.get('places', []):
if p.get('spot'): # stations only, not free-floating bikes
rows.append([round(p['lng'], 5), round(p['lat'], 5)]); n += 1
per['nextbike (all cities, global)'] = n
except Exception as e:
per['nextbike'] = f'FAILED {e}'
dump('bikes.json', {'cols': ['lon', 'lat'], 'systems': per, 'rows': rows})
log['gbfs'] = dict(systems=per, urls=list(SYSTEMS.values()) + ['https://api.nextbike.net/maps/nextbike-live.json'], rows=len(rows))
print('bikes', len(rows), per)
if 'pads' in only:
url = 'https://ll.thespacedevs.com/2.3.0/pads/?limit=100&mode=list'
rows = []
while url:
r = S.get(url, timeout=60)
r.raise_for_status()
d = r.json()
for p in d['results']:
try:
lat, lon = float(p['latitude']), float(p['longitude'])
except (TypeError, ValueError, KeyError):
continue
loc = p.get('location') or {}
rows.append([round(lon, 4), round(lat, 4), p['name'], loc.get('name') if isinstance(loc, dict) else None,
p.get('total_launch_count') or 0, bool(p.get('active'))])
url = d.get('next')
time.sleep(1)
rows.sort(key=lambda x: -x[4])
dump('launch_sites.json', {'cols': ['lon', 'lat', 'pad', 'location', 'total_launches', 'active'], 'rows': rows})
log['pads'] = dict(url='https://ll.thespacedevs.com/2.3.0/pads/?limit=100&mode=list (paged)', rows=len(rows))
print('pads', len(rows))
if 'power' in only:
url = 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'
r = S.get(url, timeout=180)
r.raise_for_status()
rows = []
for row in csv.DictReader(io.StringIO(r.content.decode('utf-8'))):
try:
rows.append([round(float(row['longitude']), 4), round(float(row['latitude']), 4), round(float(row['capacity_mw']), 1), row['primary_fuel'], row['name'], row['country']])
except ValueError:
pass
rows.sort(key=lambda x: -x[2])
dump('power_plants.json', {'cols': ['lon', 'lat', 'capacity_mw', 'fuel', 'name', 'iso3'], 'rows': rows})
log['power'] = dict(url=url, rows=len(rows))
print('power', len(rows))
prev = {}
p = os.path.join(RAW, 'bonus_log.json')
if os.path.exists(p):
prev = json.load(open(p))
prev.update(log)
json.dump(prev, open(p, 'w'), indent=1)
+28
View File
@@ -0,0 +1,28 @@
#!/usr/bin/env python3
"""GFS 1.0 deg 10 m wind U/V from NOAA's AWS Open Data bucket (noaa-gfs-bdp-pds) via .idx byte
ranges, same approach as server/providers/wind/gfs.js; decoded with python `eccodes` (pip).
Usage: python fetch_wind.py <gfs file base url (no .idx)> <out wind.json>"""
import json, sys, requests, numpy as np, eccodes
base, out = sys.argv[1], sys.argv[2]
lines = [l.split(':') for l in requests.get(base + '.idx', timeout=60).text.strip().split('\n')]
def field(var):
i = next(k for k, p in enumerate(lines) if p[3] == var and p[4] == '10 m above ground')
s, e = int(lines[i][1]), int(lines[i + 1][1]) - 1
b = requests.get(base, headers={'Range': f'bytes={s}-{e}'}, timeout=120).content
h = eccodes.codes_new_from_message(b)
g = {k: eccodes.codes_get(h, k) for k in ['Ni', 'Nj', 'dataDate', 'dataTime', 'step']}
g.update({k: eccodes.codes_get_double(h, k) for k in ['longitudeOfFirstGridPointInDegrees', 'latitudeOfFirstGridPointInDegrees', 'iDirectionIncrementInDegrees', 'jDirectionIncrementInDegrees']})
g['values'] = eccodes.codes_get_values(h).reshape(g['Nj'], g['Ni'])
eccodes.codes_release(h)
return g
U, V = field('UGRD'), field('VGRD')
assert U['longitudeOfFirstGridPointInDegrees'] == 0 and U['latitudeOfFirstGridPointInDegrees'] == 90
dx = U['iDirectionIncrementInDegrees']; shift = int(round(180 / dx))
u = np.roll(U['values'], -shift, axis=1); v = np.roll(V['values'], -shift, axis=1) # lon 0..359 -> -180..179
ny, nx = u.shape
json.dump({'source': 'NOAA GFS 1.0deg ' + base.split('/')[-1], 'refTime': f"{U['dataDate']} {U['dataTime']:04d}Z", 'forecastHour': U['step'],
'level': '10 m above ground', 'units': 'm/s', 'nx': nx, 'ny': ny, 'lo1': -180, 'la1': 90, 'dx': dx, 'dy': U['jDirectionIncrementInDegrees'],
'u': [round(float(x), 2) for x in u.ravel()], 'v': [round(float(x), 2) for x in v.ravel()]},
open(out, 'w'), separators=(',', ':'))
sp = np.hypot(u, v)
print(nx, ny, 'max', sp.max().round(1), 'mean', sp.mean().round(2), U['dataDate'], U['dataTime'], 'f', U['step'])
+172
View File
@@ -0,0 +1,172 @@
#!/usr/bin/env python3
"""Convert raw fetched feeds (see fetch_basic.sh) into compact intro-sequence JSON.
Usage: python3 -I process_basic.py <raw_dir> <out_data_dir>
All inputs are real downloads; this only filters, reprojects units and rounds.
"""
import csv, json, os, sys, time
RAW, OUT = sys.argv[1], sys.argv[2]
meta = {}
def r(x, n):
return None if x is None else round(float(x), n)
def dump(name, obj):
p = os.path.join(OUT, name)
with open(p, 'w') as f:
json.dump(obj, f, separators=(',', ':'), ensure_ascii=False)
print(f'{name}: {os.path.getsize(p)/1e6:.2f} MB')
def raw(name):
return os.path.join(RAW, name)
# 1. flights (OpenSky)
o = json.load(open(raw('opensky.json')))
rows = []
for s in o['states']:
lon, lat, on_ground = s[5], s[6], s[8]
if lon is None or lat is None or on_ground:
continue
alt = s[13] if s[13] is not None else s[7]
rows.append([r(lon, 4), r(lat, 4), r(alt, 0), r(s[10], 1), r(s[9], 1),
(s[1] or '').strip(), s[2]])
dump('flights.json', {'t': o['time'], 'cols': ['lon', 'lat', 'alt_m', 'track_deg', 'vel_ms', 'callsign', 'country'], 'rows': rows})
meta['flights'] = len(rows)
# 2. military (adsb.lol /v2/mil, unioned with adsb.fi /v2/mil, dedup by hex; adsb.lol wins)
FT, KT = 0.3048, 0.514444
seen, rows, tmax = set(), [], 0
for fname in ['mil.json', 'mil_adsbfi.json']:
if not os.path.exists(raw(fname)):
continue
d = json.load(open(raw(fname)))
tmax = max(tmax, int((d.get('now') or 0) / (1000 if (d.get('now') or 0) > 1e11 else 1)))
for a in d.get('ac') or d.get('aircraft') or []:
lat, lon = a.get('lat'), a.get('lon')
if lat is None or lon is None or a['hex'] in seen:
continue
alt = a.get('alt_geom', a.get('alt_baro'))
if a.get('alt_baro') == 'ground' or alt == 'ground':
continue # airborne only
seen.add(a['hex'])
track = a.get('track', a.get('true_heading'))
gs = a.get('gs')
rows.append([r(lon, 4), r(lat, 4), r(alt * FT, 0) if isinstance(alt, (int, float)) else None,
r(track, 1), r(gs * KT, 1) if gs is not None else None,
(a.get('flight') or '').strip(), None, a.get('t')])
dump('military.json', {'t': tmax, 'cols': ['lon', 'lat', 'alt_m', 'track_deg', 'vel_ms', 'callsign', 'country', 'type'], 'rows': rows,
'note': 'country not provided by adsb.lol/adsb.fi feeds (null)'})
meta['military'] = len(rows)
# 4. quakes
q = json.load(open(raw('quakes.geojson')))
rows = []
for f in q['features']:
c, p = f['geometry']['coordinates'], f['properties']
if p.get('mag') is None:
continue
rows.append([r(c[0], 3), r(c[1], 3), r(c[2], 1), r(p['mag'], 2), p['time'], p.get('place') or ''])
rows.sort(key=lambda x: x[4])
dump('quakes.json', {'t': q['metadata']['generated'], 'cols': ['lon', 'lat', 'depth_km', 'mag', 'time_ms', 'place'], 'rows': rows})
meta['quakes'] = len(rows)
# 5. fires (FIRMS 24h global: SNPP VIIRS, NOAA-20 VIIRS, MODIS)
rows, per = [], {}
for fname, bcol, tag in [('viirs_snpp.csv', 'bright_ti4', 'VIIRS_SNPP'), ('viirs_n20.csv', 'bright_ti4', 'VIIRS_NOAA20'), ('modis.csv', 'brightness', 'MODIS')]:
if not os.path.exists(raw(fname)):
continue
n = 0
with open(raw(fname), newline='') as fh:
for row in csv.DictReader(fh):
try:
rows.append([r(row['longitude'], 3), r(row['latitude'], 3), r(row['frp'], 1), r(row[bcol], 1)])
n += 1
except (KeyError, ValueError):
pass
per[tag] = n
dump('fires.json', {'cols': ['lon', 'lat', 'frp', 'brightness'], 'sources': per, 'rows': rows})
meta['fires'] = len(rows)
meta['fires_per'] = per
# 8. cables
c = json.load(open(raw('cables.json')))
lines = []
for f in c['features']:
g = f['geometry']
parts = g['coordinates'] if g['type'] == 'MultiLineString' else [g['coordinates']]
for part in parts:
pl = [[r(x, 2), r(y, 2)] for x, y in (pt[:2] for pt in part)]
if len(pl) >= 2:
lines.append(pl)
dump('cables.json', {'cables': lines})
meta['cables'] = (len(c['features']), len(lines))
# 9. coastline + borders
def lines_of(geo):
out = []
for f in geo['features']:
g = f['geometry']
if g is None:
continue
t, co = g['type'], g['coordinates']
if t == 'LineString':
parts = [co]
elif t == 'MultiLineString':
parts = co
elif t == 'Polygon':
parts = co
elif t == 'MultiPolygon':
parts = [ring for poly in co for ring in poly]
else:
parts = []
for part in parts:
pl, last = [], None
for x, y in (pt[:2] for pt in part):
pt = [r(x, 2), r(y, 2)]
if pt != last:
pl.append(pt)
last = pt
if len(pl) >= 2:
out.append(pl)
return out
coast = lines_of(json.load(open(raw('coast50.geojson'))))
dump('coastline.json', {'lines': coast})
borders = lines_of(json.load(open(raw('countries110.geojson'))))
dump('borders.json', {'lines': borders})
meta['coast'] = len(coast)
meta['borders'] = len(borders)
# 10. cities
pp = json.load(open(raw('places.geojson')))
rows = []
for f in pp['features']:
p = f['properties']
lon, lat = f['geometry']['coordinates'][:2]
rows.append([r(lon, 3), r(lat, 3), int(p.get('pop_max') or 0), p.get('name') or p.get('nameascii'), p.get('adm0name') or p.get('sov0name')])
rows.sort(key=lambda x: -x[2])
rows = rows[:3000]
dump('cities.json', {'cols': ['lon', 'lat', 'pop_max', 'name', 'country'], 'rows': rows})
meta['cities'] = len(rows)
# 11. airports (OurAirports large + medium)
rows = []
with open(raw('airports.csv'), newline='', encoding='utf-8') as fh:
for row in csv.DictReader(fh):
if row['type'] not in ('large_airport', 'medium_airport'):
continue
rows.append([r(row['longitude_deg'], 4), r(row['latitude_deg'], 4), row['iata_code'] or row['ident'], row['name'],
'L' if row['type'] == 'large_airport' else 'M'])
rows.sort(key=lambda x: x[4])
dump('airports.json', {'cols': ['lon', 'lat', 'iata', 'name', 'size'], 'rows': rows})
meta['airports'] = (len(rows), sum(1 for x in rows if x[4] == 'L'))
json.dump(meta, open(os.path.join(RAW, 'meta_basic.json'), 'w'))
print(meta)
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env python3
"""tle.txt (3-line TLEs, CelesTrak GROUP=active) + tle_groups.json (NORAD ids per group).
Usage: python3 -I process_tle.py <raw_dir> <out_dir>"""
import json, os, sys
RAW, OUT = sys.argv[1], sys.argv[2]
def parse(path):
L = [l.rstrip() for l in open(path) if l.strip()]
out = []
i = 0
while i + 2 < len(L) + 0 and i + 2 <= len(L) - 1:
if L[i+1].startswith('1 ') and L[i+2].startswith('2 '):
name = L[i][2:] if L[i].startswith('0 ') else L[i]
out.append((name.strip(), L[i+1], L[i+2])); i += 3
else:
i += 1
return out
act = parse(os.path.join(RAW, 'tle_active3.txt'))
with open(os.path.join(OUT, 'tle.txt'), 'w') as f:
for n, a, b in act:
f.write(f'{n}\n{a}\n{b}\n')
groups, notable = {}, {}
for g in ['stations', 'starlink', 'gps-ops', 'geo', 'weather']:
p = os.path.join(RAW, f'tle_{g}.txt')
if os.path.exists(p) and os.path.getsize(p) > 0:
sats = parse(p)
groups[g] = sorted(int(a[2:7]) for _, a, _ in sats)
for n, _, _ in sats[: (40 if g != 'starlink' else 3)]:
notable[n] = g
epochs = sorted(a[18:32] for _, a, _ in act)
json.dump({'groups_norad': groups, 'notable': notable}, open(os.path.join(OUT, 'tle_groups.json'), 'w'), separators=(',', ':'))
print(len(act), {k: len(v) for k, v in groups.items()}, 'epoch range', epochs[0], epochs[-1])
@@ -0,0 +1,38 @@
#!/usr/bin/env python3
"""vessels.json from keyless AIS: Fintraffic Digitraffic (Finland/Baltic) + Kystverket Kystdatahuset realtime (Norway).
Usage: python3 -I process_vessels.py <raw_dir> <out_dir>"""
import json, os, sys, time
RAW, OUT = sys.argv[1], sys.argv[2]
rows, src = {}, {}
# Norway (Kystverket) - LineString trails, last coordinate = latest position
k = json.load(open(os.path.join(RAW, 'kystverket_ais.json')))
for f in k['features']:
c, p = f['geometry']['coordinates'], f['properties']
if not c:
continue
lon, lat = c[-1]
cog = p.get('cog')
cog = None if cog is None or cog >= 360 else round(cog, 1)
sog = p.get('speed')
sog = None if sog is None or sog >= 102.3 else round(sog, 1)
rows[p['mmsi']] = [round(lon, 4), round(lat, 4), sog, cog, p['mmsi'], (p.get('ship_name') or '').strip() or None]
src[p['mmsi']] = 'NO'
# Finland / Baltic (Digitraffic)
names = {v['mmsi']: (v.get('name') or '').strip() or None for v in json.load(open(os.path.join(RAW, 'digitraffic_vessels.json')))}
d = json.load(open(os.path.join(RAW, 'dt_ais_from.json')))
for f in d['features']:
p = f['properties']
lon, lat = f['geometry']['coordinates'][:2]
if p['mmsi'] in rows:
continue
cog = p.get('cog'); cog = None if cog is None or cog >= 360 else round(cog, 1)
sog = p.get('sog'); sog = None if sog is None or sog >= 102.3 else round(sog, 1)
rows[p['mmsi']] = [round(lon, 4), round(lat, 4), sog, cog, p['mmsi'], names.get(p['mmsi'])]
src[p['mmsi']] = 'FI'
out = list(rows.values())
json.dump({'t': int(time.time()), 'cols': ['lon', 'lat', 'sog_kn', 'cog_deg', 'mmsi', 'name'],
'coverage': 'Norwegian coast/EEZ (Kystverket) + Finnish waters & Baltic Sea (Fintraffic Digitraffic); not global',
'counts': {'kystverket_norway': sum(1 for v in src.values() if v == 'NO'), 'digitraffic_finland_baltic': sum(1 for v in src.values() if v == 'FI')},
'rows': out}, open(os.path.join(OUT, 'vessels.json'), 'w'), separators=(',', ':'), ensure_ascii=False)
lons = [r[0] for r in out]; lats = [r[1] for r in out]
print(len(out), 'bbox', min(lons), max(lons), min(lats), max(lats))
+23
View File
@@ -0,0 +1,23 @@
#!/usr/bin/env bash
# v2: sound design only (made to sit under a music track). Normalise the SFX
# mix to -16 LUFS, keep it as a stem, and mux share + master cuts.
set -euo pipefail
cd "$(dirname "$0")/.."
node tools/export-cues.mjs
python3 audio/sfx.py
stats=$(ffmpeg -hide_banner -i audio/sfx.wav -af loudnorm=I=-16:TP=-1.5:LRA=14:print_format=json -f null - 2>&1 | sed -n '/^{/,/^}/p')
g() { echo "$stats" | python3 -c "import json,sys;print(json.load(sys.stdin)['$1'])"; }
ffmpeg -y -loglevel error -i audio/sfx.wav -af "loudnorm=I=-16:TP=-1.5:LRA=14:measured_I=$(g input_i):measured_TP=$(g input_tp):measured_LRA=$(g input_lra):measured_thresh=$(g input_thresh):linear=true,aresample=48000" -c:a pcm_s24le audio/sfx_norm.wav
mkdir -p out/final_v2
cp audio/sfx_norm.wav out/final_v2/GodsEyeView_FirstLight_v2_SFX_stem.wav
for tag in 16x9 9x16; do
v=out/v2_video_$tag.mp4
[ -s "$v" ] || { echo "missing $v"; continue; }
ffmpeg -y -loglevel error -i $v -i audio/sfx_norm.wav -map 0:v -map 1:a -c:v copy -c:a aac -b:a 320k -shortest -movflags +faststart out/final_v2/GodsEyeView_FirstLight_v2_${tag}_master.mp4
ffmpeg -y -loglevel error -i $v -c:v libx264 -preset slow -b:v 16M -maxrate 24M -bufsize 32M -pass 1 -passlogfile out/final_v2/p_$tag -an -f null /dev/null
ffmpeg -y -loglevel error -i $v -i audio/sfx_norm.wav -map 0:v -map 1:a -c:v libx264 -preset slow -b:v 16M -maxrate 24M -bufsize 32M -pass 2 -passlogfile out/final_v2/p_$tag \
-pix_fmt yuv420p -colorspace bt709 -color_primaries bt709 -color_trc bt709 -c:a aac -b:a 256k -shortest -movflags +faststart out/final_v2/GodsEyeView_FirstLight_v2_${tag}.mp4
rm -f out/final_v2/p_$tag*
echo "finalized $tag"
done
ls -la out/final_v2
+26
View File
@@ -0,0 +1,26 @@
#!/usr/bin/env bash
# Loudness-normalise the score (-14 LUFS, -1 dBTP) and mux it into both cuts.
# tools/finalize.sh (after tools/render-all.sh has produced out/video_*.mp4)
set -euo pipefail
cd "$(dirname "$0")/.."
python3 audio/score.py
# two-pass loudnorm
stats=$(ffmpeg -hide_banner -i audio/score.wav -af loudnorm=I=-14:TP=-1:LRA=11:print_format=json -f null - 2>&1 | sed -n '/^{/,/^}/p')
mi=$(echo "$stats" | python3 -c "import json,sys;d=json.load(sys.stdin);print(d['input_i'])")
mtp=$(echo "$stats" | python3 -c "import json,sys;d=json.load(sys.stdin);print(d['input_tp'])")
mlra=$(echo "$stats" | python3 -c "import json,sys;d=json.load(sys.stdin);print(d['input_lra'])")
mth=$(echo "$stats" | python3 -c "import json,sys;d=json.load(sys.stdin);print(d['input_thresh'])")
ffmpeg -y -loglevel error -i audio/score.wav -af "loudnorm=I=-14:TP=-1:LRA=11:measured_I=$mi:measured_TP=$mtp:measured_LRA=$mlra:measured_thresh=$mth:linear=true,aresample=48000" -c:a pcm_s24le audio/score_norm.wav
mkdir -p out/final
for tag in 16x9 9x16; do
[ -s out/video_$tag.mp4 ] || { echo "missing out/video_$tag.mp4"; continue; }
# master: the CRF 14 picture untouched + 320k AAC
ffmpeg -y -loglevel error -i out/video_$tag.mp4 -i audio/score_norm.wav -map 0:v -map 1:a -c:v copy -c:a aac -b:a 320k -ar 48000 -shortest -movflags +faststart \
-metadata title="God's Eye View — First Light ($tag)" out/final/GodsEyeView_FirstLight_${tag}_master.mp4
# delivery: lighter encode for posting/sharing
ffmpeg -y -loglevel error -i out/video_$tag.mp4 -i audio/score_norm.wav -map 0:v -map 1:a -c:v libx264 -preset slow -crf 19 -tune grain -pix_fmt yuv420p \
-colorspace bt709 -color_primaries bt709 -color_trc bt709 -c:a aac -b:a 256k -ar 48000 -shortest -movflags +faststart \
-metadata title="God's Eye View — First Light ($tag)" out/final/GodsEyeView_FirstLight_${tag}.mp4
echo "finalized $tag"
done
ls -la out/final
+175
View File
@@ -0,0 +1,175 @@
"""Build the flurry montage captures.
python3 select_cities.py # -> city_candidates.json (primary + backup per region)
python3 build.py # -> imagery/flurry/fx_###.jpg, index.json, tools/flurry/sheet.jpg
Each capture: Esri World Imagery, 3x3 tiles around the point, centred 512x512
crop, JPEG q86. Placeholder / low-variance captures are rejected automatically
(tiles.quality_ok); captures that passed the metrics but looked wrong on the
review sheets (forest, farmland, haze) are listed in VISUAL_REJECT. Rejected
items are replaced by the next backup of the same region (cities) or by the
next backup landmark.
"""
import glob
import json
import os
import sys
from concurrent.futures import ThreadPoolExecutor
from PIL import Image
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from landmarks import LANDMARKS # noqa: E402
from tiles import HERE, MAX_WORKERS, OUT, capture, quality_ok # noqa: E402
N_CITIES = 170
N_LANDMARKS = 90
# Natural Earth points that landed on a park/water/cemetery: moved to the dense centre.
CITY_OVERRIDES = {
"Lagos": (6.4541, 3.3947), # Lagos Island (was open lagoon)
"Cairns": (-16.9230, 145.7740), # CBD (was mangrove mudflat)
"Calgary": (51.0460, -114.0650), # downtown (was cemetery)
"Dubai": (25.0805, 55.1403), # Dubai Marina (was construction lot)
"Perth": (-31.9530, 115.8575), # CBD (was Kings Park bush)
"Dhaka": (23.7330, 90.4172), # Motijheel (was hazy open ground)
"Delhi": (28.6315, 77.2167), # Connaught Place
"Montréal": (45.5017, -73.5673), # downtown (was Mount Royal park)
"Busan": (35.1000, 129.0350), # Nampo / harbour (was forested hill)
"Jeddah": (21.4858, 39.1925), # Al-Balad old town (was empty lot)
"Tokyo": (35.6717, 139.7650), # Ginza (was Imperial Palace woods)
}
VISUAL_REJECT = {
"Venice": "blurry low-res imagery",
"Conakry": "heavy haze",
"Maracaibo": "farmland / scrub",
"Barquisimeto": "dry riverbed, no city",
"Natal": "solid forest canopy",
"João Pessoa": "forest canopy",
"Mbuji-Mayi": "forest/scrub",
"Huambo": "scrub, sparse",
"Port Harcourt": "haze",
"Qaraghandy": "dark featureless rail yard",
"Herat": "low-res blur",
"Chengdu": "smog haze",
"Xian": "washed-out industrial lot",
"Kaohsiung": "forested hill",
"Sapporo": "university farm fields",
"Isfahan": "low contrast blur",
"Stuttgart": "hazy woodland",
"Townsville": "airport apron, not city",
}
LANDMARK_BACKUPS = [
"Glen Canyon Dam", "Grand Coulee Dam", "Mount St. Helens", "Groom Lake", "Diavik Diamond Mine",
"Syncrude Mildred Lake oil sands", "Centre-pivot fields, Kansas", "Indianapolis Motor Speedway",
"Brooklyn Bridge", "Panama Canal, Miraflores Locks", "Escondida Mine", "ALMA Observatory",
"Meeting of the Waters, Manaus", "Brasilia, National Congress", "PS10 / PS20 Solar Towers", "Tower Bridge",
"Port of Tema", "Akosombo Dam", "Okavango Delta", "FNB Stadium, Soweto", "East Oweinat pivots",
"Kilimanjaro, Kibo crater", "Ras Laffan Industrial City", "Dubai International Airport", "Bagan temples",
"Jiuquan Launch Centre", "Mount Bromo", "Vostochny Cosmodrome", "Pine Gap", "Tanegashima Space Center",
"Teide", "Faa'a International Airport", "Hartsfield-Jackson Atlanta Airport", "Niagara Horseshoe Falls",
"Iguazu Falls, Devil's Throat", "Vandenberg SLC-4E", "Paranal Observatory", "Cerro Dominador Solar",
"Gibraltar Airport runway", "Erg Chebbi dunes", "Millau Viaduct", "Garzweiler Lignite Mine", "Inga Dams",
"Grand Ethiopian Renaissance Dam", "Ras Tanura refinery", "Udachnaya Pipe", "Busan New Port",
"Mount Taranaki", "Beijing National Stadium", "Port of Hamburg, Burchardkai", "Port of Santos",
"Jebel Ali Port", "Mont-Saint-Michel", "Angkor Wat", "Mount Vesuvius", "Horseshoe Bend",
"Benban Solar Park", "Borobudur", "Bhadla Solar Park", "Kariba Dam", "Wake Island Airfield",
"Diamond Head", "Port Hedland salt ponds",
]
def evaluate(cands):
def job(c):
try:
img, info = capture(c["lat"], c["lon"], c["zoom"])
ok, why = quality_ok(info)
except Exception as e: # network failure counts as a reject
img, info, ok, why = None, {}, False, f"fetch error: {e}"
if ok and c["name"] in VISUAL_REJECT and c.get("kind") == "city":
ok, why = False, "visual: " + VISUAL_REJECT[c["name"]]
return img, ok, why
with ThreadPoolExecutor(MAX_WORKERS) as ex:
return list(ex.map(job, cands))
def pick(cands, results, quota_by_group):
"""Primaries first in order; failures replaced by the group's backups in order."""
chosen, rejects = [], []
for g, quota in quota_by_group.items():
items = [(c, r) for c, r in zip(cands, results) if c["group"] == g]
prim = [x for x in items if x[0]["rank"] == "primary"]
back = [x for x in items if x[0]["rank"] == "backup"]
got = []
for c, r in prim + back:
if len(got) >= quota:
break
if r[1]:
got.append((c, r[0]))
elif c["rank"] == "primary":
rejects.append((c, r[2]))
if len(got) < quota:
print(f"!! group {g}: only {len(got)}/{quota}")
chosen += got
return chosen, rejects
def main():
cities = json.load(open(os.path.join(HERE, "city_candidates.json")))
for c in cities:
c["kind"], c["group"] = "city", "c:" + c["region"]
if c["name"] in CITY_OVERRIDES:
c["lat"], c["lon"] = CITY_OVERRIDES[c["name"]]
cquota = {}
for c in cities:
if c["rank"] == "primary":
cquota[c["group"]] = cquota.get(c["group"], 0) + 1
assert sum(cquota.values()) == N_CITIES, cquota
lms = [dict(name=n, country=co, lat=la, lon=lo, zoom=z, kind="landmark", group="landmark",
rank="backup" if n in LANDMARK_BACKUPS else "primary") for n, co, la, lo, z in LANDMARKS]
lms.sort(key=lambda c: c["rank"] != "primary") # stable: primaries first
nprim = sum(c["rank"] == "primary" for c in lms)
assert nprim == N_LANDMARKS, nprim
allc = cities + lms
print(f"evaluating {len(allc)} candidates ...", flush=True)
res = evaluate(allc)
quotas = dict(cquota)
quotas["landmark"] = N_LANDMARKS
chosen, rejects = pick(allc, res, quotas)
chosen.sort(key=lambda x: x[0]["lon"])
os.makedirs(OUT, exist_ok=True)
for f in glob.glob(os.path.join(OUT, "fx_*.jpg")):
os.remove(f)
index = []
for i, (c, img) in enumerate(chosen):
fn = f"fx_{i:03d}.jpg"
img.save(os.path.join(OUT, fn), quality=86, optimize=True)
index.append({"file": fn, "name": c["name"], "country": c["country"], "lat": round(c["lat"], 5),
"lon": round(c["lon"], 5), "zoom": c["zoom"], "kind": c["kind"]})
with open(os.path.join(OUT, "index.json"), "w") as f:
json.dump(index, f, indent=1, ensure_ascii=False)
# contact sheet: 16 x 17 grid of 96px thumbs, west->east reading order
cols, cell = 16, 96
rows = (len(chosen) + cols - 1) // cols
sheet = Image.new("RGB", (cols * cell, rows * cell), (10, 10, 10))
for i, (c, img) in enumerate(chosen):
sheet.paste(img.resize((cell, cell), Image.LANCZOS), ((i % cols) * cell, (i // cols) * cell))
sheet.save(os.path.join(HERE, "sheet.jpg"), quality=88)
from collections import Counter
print(f"wrote {len(index)} captures: {Counter(x['kind'] for x in index)}")
print("rejected primaries (replaced by backups):")
for c, why in rejects:
print(f" - {c['name']} ({c['country']}): {why}")
bands = Counter(int((x["lon"] + 180) // 30) for x in index)
print("30-degree longitude bands W->E:", [bands.get(i, 0) for i in range(12)])
json.dump([{"name": c["name"], "country": c["country"], "reason": why} for c, why in rejects],
open(os.path.join(HERE, "rejects.json"), "w"), indent=1, ensure_ascii=False)
if __name__ == "__main__":
main()
File diff suppressed because it is too large. Load diff
+205
View File
@@ -0,0 +1,205 @@
"""Hand-picked non-city landmarks for the flurry montage.
(name, country, lat, lon, zoom). zoom 16 for compact features, 15 for very large
ones (deltas, glaciers, big pits, pivot fields). Coordinates are verified
visually via tools/flurry/review.py sheets; entries that came back as featureless
terrain were corrected or dropped (see REJECTED at the bottom).
"""
LANDMARKS = [
# --- ports / container terminals ---
("Port of Rotterdam, Maasvlakte", "Netherlands", 51.9575, 4.0500, 16),
("Port of Singapore, Pasir Panjang", "Singapore", 1.2735, 103.7750, 16),
("Yangshan Deep-Water Port", "China", 30.6280, 122.0650, 16),
("Port of Los Angeles, Pier 400", "United States of America", 33.7350, -118.2550, 16),
("Jebel Ali Port", "United Arab Emirates", 25.0100, 55.0600, 16),
("Port of Santos", "Brazil", -23.9700, -46.3000, 16),
("Port of Hamburg, Burchardkai", "Germany", 53.5350, 9.9200, 16),
("Port Hedland salt ponds", "Australia", -20.3450, 118.6550, 15),
("Busan New Port", "South Korea", 35.0800, 128.8000, 16),
("Tanger Med Port", "Morocco", 35.8850, -5.5000, 16),
("Port of Tema", "Ghana", 5.6330, 0.0100, 16),
# --- airports ---
("Hartsfield-Jackson Atlanta Airport", "United States of America", 33.6407, -84.4277, 16),
("Dubai International Airport", "United Arab Emirates", 25.2528, 55.3644, 16),
("Hong Kong International Airport", "Hong Kong S.A.R.", 22.3080, 113.9185, 16),
("Denver International Airport", "United States of America", 39.8494, -104.6738, 16),
("Kansai International Airport", "Japan", 34.4320, 135.2304, 16),
("Beijing Daxing Airport", "China", 39.5098, 116.4105, 16),
("Davis-Monthan Aircraft Boneyard", "United States of America", 32.1500, -110.8300, 16),
("Princess Juliana Airport", "Sint Maarten", 18.0410, -63.1089, 16),
("Gibraltar Airport runway", "Gibraltar", 36.1512, -5.3497, 16),
("Faa'a International Airport", "French Polynesia", -17.5537, -149.6067, 16),
("Funafuti Airfield", "Tuvalu", -8.5250, 179.1960, 16),
("Midway Atoll, Sand Island", "United States of America", 28.2100, -177.3800, 16),
("Wake Island Airfield", "United States of America", 19.2820, 166.6470, 16),
# --- dams ---
("Hoover Dam", "United States of America", 36.0160, -114.7377, 16),
("Three Gorges Dam", "China", 30.8233, 111.0036, 16),
("Itaipu Dam", "Brazil", -25.4085, -54.5888, 16),
("Grand Coulee Dam", "United States of America", 47.9560, -118.9817, 16),
("Kariba Dam", "Zimbabwe", -16.5222, 28.7614, 16),
("Aswan High Dam", "Egypt", 23.9707, 32.8780, 16),
("Grand Ethiopian Renaissance Dam", "Ethiopia", 11.2150, 35.0900, 16),
("Glen Canyon Dam", "United States of America", 36.9375, -111.4844, 16),
("Akosombo Dam", "Ghana", 6.2999, 0.0592, 16),
("Inga Dams", "Congo (Kinshasa)", -5.5240, 13.6200, 16),
# --- open-pit mines ---
("Bingham Canyon Mine", "United States of America", 40.5230, -112.1510, 15),
("Chuquicamata Mine", "Chile", -22.2900, -68.9000, 15),
("Mir Diamond Mine", "Russia", 62.5275, 113.9936, 15),
("Escondida Mine", "Chile", -24.2700, -69.0700, 15),
("Kalgoorlie Super Pit", "Australia", -30.7775, 121.5040, 15),
("Udachnaya Pipe", "Russia", 66.4300, 112.3200, 15),
("Garzweiler Lignite Mine", "Germany", 51.0600, 6.4800, 15),
("Kimberley Big Hole", "South Africa", -28.7383, 24.7585, 16),
("Diavik Diamond Mine", "Canada", 64.4960, -110.2700, 15),
("Syncrude Mildred Lake oil sands", "Canada", 57.0400, -111.6000, 15),
# --- solar ---
("Noor Ouarzazate Solar", "Morocco", 31.0498, -6.8686, 16),
("Ivanpah Solar", "United States of America", 35.5525, -115.4680, 16),
("Crescent Dunes Solar", "United States of America", 38.2386, -117.3636, 15),
("Bhadla Solar Park", "India", 27.5400, 71.9100, 16),
("Gemasolar", "Spain", 37.5606, -5.3317, 16),
("PS10 / PS20 Solar Towers", "Spain", 37.4425, -6.2550, 16),
("Benban Solar Park", "Egypt", 24.4500, 32.7300, 16),
("Cerro Dominador Solar", "Chile", -22.7700, -69.4800, 16),
# --- centre-pivot irrigation ---
("Centre-pivot fields, Kansas", "United States of America", 37.9000, -100.9000, 15),
("Centre-pivot fields, Wadi As-Sirhan", "Saudi Arabia", 30.2000, 38.0800, 15),
("Kufra agricultural project", "Libya", 24.1400, 23.3500, 15),
("East Oweinat pivots", "Egypt", 22.6000, 28.6000, 15),
("Centre-pivot fields, Nebraska", "United States of America", 41.1000, -101.7000, 15),
# --- river deltas / water ---
("Lena Delta", "Russia", 72.8000, 126.5000, 15),
("Okavango Delta", "Botswana", -19.3000, 22.9000, 15),
("Betsiboka Estuary", "Madagascar", -15.7500, 46.3500, 15),
("Meeting of the Waters, Manaus", "Brazil", -3.1370, -59.9100, 15),
# --- salt / evaporation ---
("Salar de Atacama lithium ponds", "Chile", -23.5280, -68.3680, 16),
("Dead Sea evaporation ponds", "Israel", 31.1000, 35.4000, 15),
("San Francisco Bay salt ponds", "United States of America", 37.5000, -122.0500, 15),
("Lake Retba (Pink Lake)", "Senegal", 14.8400, -17.2300, 15),
("Hutt Lagoon", "Australia", -28.1700, 114.2500, 15),
# --- glaciers / ice ---
("Perito Moreno Glacier", "Argentina", -50.4800, -73.0500, 15),
("Aletsch Glacier, Konkordiaplatz", "Switzerland", 46.5000, 8.0300, 15),
("Hubbard Glacier", "United States of America", 60.0200, -139.5000, 15),
# Antarctica: Esri has only Landsat-class imagery here (z14+ is placeholder),
# so these use z11-12 where the source pixels are native rather than upsampled.
("Taylor Valley, Canada Glacier", "Antarctica", -77.6300, 162.9500, 12),
("McMurdo Dry Valleys, Wright Valley", "Antarctica", -77.5700, 161.8500, 11),
("Deception Island caldera", "Antarctica", -62.9600, -60.6300, 12),
("Mount Everest", "Nepal", 27.9881, 86.9250, 15),
# --- volcanoes / craters ---
("Mount Bromo", "Indonesia", -7.9420, 112.9530, 15),
("Erta Ale", "Ethiopia", 13.6065, 40.6655, 16),
("Kilauea, Halemaumau", "United States of America", 19.4060, -155.2830, 15),
("Mount Fuji summit crater", "Japan", 35.3628, 138.7307, 16),
("Nyiragongo", "Congo (Kinshasa)", -1.5200, 29.2500, 16),
("Meteor Crater", "United States of America", 35.0275, -111.0225, 15),
("Mount St. Helens", "United States of America", 46.2000, -122.1900, 15),
("Kawah Ijen", "Indonesia", -8.0580, 114.2420, 16),
("Teide", "Spain", 28.2725, -16.6425, 16),
("Mount Vesuvius", "Italy", 40.8216, 14.4263, 16),
("Rano Kau, Easter Island", "Chile", -27.1860, -109.4350, 15),
("Diamond Head", "United States of America", 21.2620, -157.8050, 15),
("Darvaza Gas Crater", "Turkmenistan", 40.2525, 58.4394, 16),
("Kilimanjaro, Kibo crater", "Tanzania", -3.0758, 37.3533, 15),
("Mount Taranaki", "New Zealand", -39.2968, 174.0634, 15),
("Whakaari / White Island", "New Zealand", -37.5200, 177.1800, 15),
("Dallol", "Ethiopia", 14.2400, 40.3000, 16),
# --- stadiums / venues ---
("Maracana Stadium", "Brazil", -22.9122, -43.2302, 16),
("Rungrado May Day Stadium", "North Korea", 39.0497, 125.7756, 16),
("Camp Nou", "Spain", 41.3809, 2.1228, 16),
("Beijing National Stadium", "China", 39.9925, 116.3910, 16),
("Melbourne Cricket Ground", "Australia", -37.8200, 144.9834, 16),
("Indianapolis Motor Speedway", "United States of America", 39.7950, -86.2350, 15),
("Estadio Azteca", "Mexico", 19.3029, -99.1505, 16),
("FNB Stadium, Soweto", "South Africa", -26.2348, 27.9825, 16),
# --- bridges / canals ---
("Golden Gate Bridge", "United States of America", 37.8100, -122.4770, 16),
("Sydney Harbour Bridge & Opera House", "Australia", -33.8550, 151.2120, 16),
("Tower Bridge", "United Kingdom", 51.5055, -0.0754, 16),
("Oresund Bridge, Peberholm", "Denmark", 55.5900, 12.7740, 16),
("Millau Viaduct", "France", 44.0794, 3.0225, 16),
("Brooklyn Bridge", "United States of America", 40.7061, -73.9969, 16),
("Panama Canal, Miraflores Locks", "Panama", 8.9970, -79.5915, 16),
("Panama Canal, Agua Clara Locks", "Panama", 9.2720, -79.9180, 16),
# --- launch sites ---
("Kennedy Space Center LC-39A", "United States of America", 28.6083, -80.6041, 16),
("Baikonur, Gagarin's Start", "Kazakhstan", 45.9203, 63.3422, 16),
("Vandenberg SLC-4E", "United States of America", 34.6320, -120.6110, 16),
("Starbase, Boca Chica", "United States of America", 25.9972, -97.1566, 16),
("Wenchang Launch Site", "China", 19.6145, 110.9510, 16),
("Jiuquan Launch Centre", "China", 40.9580, 100.2910, 16),
("Tanegashima Space Center", "Japan", 30.4009, 130.9749, 16),
("Vostochny Cosmodrome", "Russia", 51.8840, 128.3330, 16),
("Rocket Lab LC-1, Mahia", "New Zealand", -39.2615, 177.8649, 16),
# --- energy / industry ---
("Permian Basin well pads", "United States of America", 32.2100, -103.8300, 15),
("Ras Tanura refinery", "Saudi Arabia", 26.6400, 50.1600, 16),
("Jamnagar Refinery", "India", 22.3500, 69.8500, 16),
("Ras Laffan Industrial City", "Qatar", 25.9100, 51.5500, 16),
("Yamal LNG, Sabetta", "Russia", 71.2700, 72.0700, 16),
("Chernobyl Nuclear Power Plant", "Ukraine", 51.3890, 30.0990, 16),
# --- monuments / sites ---
("Jiayuguan Pass, Great Wall", "China", 39.8016, 98.2133, 16),
("Pyramids of Giza", "Egypt", 29.9792, 31.1342, 16),
("Palm Jumeirah", "United Arab Emirates", 25.1124, 55.1390, 15),
("Angkor Wat", "Cambodia", 13.4125, 103.8670, 16),
("The Pentagon", "United States of America", 38.8719, -77.0563, 16),
("Groom Lake", "United States of America", 37.2350, -115.8111, 15),
("FAST Radio Telescope", "China", 25.6529, 106.8566, 16),
("Very Large Array", "United States of America", 34.0784, -107.6184, 16),
("Uluru", "Australia", -25.3444, 131.0369, 15),
("Mont-Saint-Michel", "France", 48.6361, -1.5115, 16),
("Grand Prismatic Spring", "United States of America", 44.5251, -110.8382, 16),
("Masjid al-Haram, Mecca", "Saudi Arabia", 21.4225, 39.8262, 16),
("Forbidden City", "China", 39.9155, 116.3910, 16),
("Taj Mahal", "India", 27.1751, 78.0421, 16),
("Greenhouses of Almeria", "Spain", 36.7500, -2.7200, 16),
("Pine Gap", "Australia", -23.7990, 133.7370, 16),
("ALMA Observatory", "Chile", -23.0229, -67.7552, 16),
("Paranal Observatory", "Chile", -24.6275, -70.4044, 16),
("Brasilia, National Congress", "Brazil", -15.7998, -47.8645, 16),
("Iguazu Falls, Devil's Throat", "Argentina", -25.6950, -54.4367, 16),
("Niagara Horseshoe Falls", "Canada", 43.0773, -79.0757, 16),
("Victoria Falls", "Zambia", -17.9243, 25.8572, 16),
("Horseshoe Bend", "United States of America", 36.8791, -111.5103, 16),
("Sossusvlei dunes", "Namibia", -24.7300, 15.2900, 15),
("Erg Chebbi dunes", "Morocco", 31.1500, -3.9800, 15),
("Bora Bora", "French Polynesia", -16.4950, -151.7050, 15),
("Borobudur", "Indonesia", -7.6079, 110.2038, 16),
("Bagan temples", "Myanmar", 21.1700, 94.8600, 16),
("Svalbard, Longyearbyen", "Norway", 78.2200, 15.6500, 16),
("Edinburgh of the Seven Seas, Tristan da Cunha", "United Kingdom", -37.0670, -12.3100, 16),
("Diego Garcia", "United Kingdom", -7.3100, 72.4100, 15),
("Kwajalein Atoll", "Marshall Islands", 8.7200, 167.7300, 16),
]
REJECTED = {
'Pituffik Space Base': 'placeholder above z13; z13 too blurry',
'Ilulissat Icefjord': 'placeholder above z13; open water at z13',
'McMurdo Station': 'placeholder above z13; blurry ice at z13',
'Rothera Research Station': 'placeholder above z13; mostly sea',
'Mount Erebus crater': 'placeholder above z13; cloud-like blur',
'Port of Durban': "featureless/cloud/low-res/off-target at tested coords",
'Port of Mombasa': "featureless/cloud/low-res/off-target at tested coords",
'Sundarbans, Ganges Delta': "featureless/cloud/low-res/off-target at tested coords",
"Mississippi Bird's-Foot Delta": "featureless/cloud/low-res/off-target at tested coords",
'Volga Delta': "featureless/cloud/low-res/off-target at tested coords",
'Great Salt Lake causeway': "featureless/cloud/low-res/off-target at tested coords",
'Jokulsarlon Glacier Lagoon': "featureless/cloud/low-res/off-target at tested coords",
'Baltoro Glacier, Concordia': "featureless/cloud/low-res/off-target at tested coords",
'Cotopaxi': "featureless/cloud/low-res/off-target at tested coords",
'Guiana Space Centre ELA-3': "featureless/cloud/low-res/off-target at tested coords",
'Satish Dhawan Space Centre': "featureless/cloud/low-res/off-target at tested coords",
'Abqaiq processing plant': "featureless/cloud/low-res/off-target at tested coords",
'Prudhoe Bay oil field': "featureless/cloud/low-res/off-target at tested coords",
'Great Wall at Mutianyu': "featureless/cloud/low-res/off-target at tested coords",
'Machu Picchu': "featureless/cloud/low-res/off-target at tested coords",
'Baltra Island airport, Galapagos': "featureless/cloud/low-res/off-target at tested coords",
'Moynaq ship graveyard, Aral Sea': "featureless/cloud/low-res/off-target at tested coords",
}
+78
View File
@@ -0,0 +1,78 @@
"""Locator sheets: a wider mosaic around a guess with a lat/lon grid, to find exact coords.
usage: python3 locate.py out.jpg "name|lat|lon|z|ntiles" ...
Each panel is ntiles x ntiles tiles at zoom z (resized to 640px) with grid
lines every `step` degrees, labelled. Panels are stacked in a 2-column grid.
"""
import io
import math
import os
import sys
from concurrent.futures import ThreadPoolExecutor
from PIL import Image, ImageDraw
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from tiles import fetch_tile, pixel_xy # noqa: E402
P = 640
def panel(name, lat, lon, z, nt):
px, py = pixel_xy(lat, lon, z)
tx, ty = int(px // 256), int(py // 256)
h = nt // 2
coords = [(tx - h + i, ty - h + j) for j in range(nt) for i in range(nt)]
with ThreadPoolExecutor(8) as ex:
datas = list(ex.map(lambda c: fetch_tile(z, c[0], c[1]), coords))
m = Image.new("RGB", (nt * 256, nt * 256))
for (x, y), d in zip(coords, datas):
m.paste(Image.open(io.BytesIO(d)).convert("RGB"), ((x - tx + h) * 256, (y - ty + h) * 256))
ox, oy = (tx - h) * 256, (ty - h) * 256
s = P / (nt * 256)
m = m.resize((P, P), Image.LANCZOS)
d = ImageDraw.Draw(m)
n = 2 ** z * 256
def lon2x(lo):
return ((lo + 180) / 360 * n - ox) * s
def lat2y(la):
si = math.sin(math.radians(la))
return ((0.5 - math.log((1 + si) / (1 - si)) / (4 * math.pi)) * n - oy) * s
span = nt * 256 / n * 360
step = [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5][
min(range(9), key=lambda k: abs(span / [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5][k] - 6))]
lo0 = math.floor((lon - span) / step) * step
for k in range(40):
lo = lo0 + k * step
x = lon2x(lo)
if 0 <= x <= P:
d.line((x, 0, x, P), fill=(255, 255, 0), width=1)
d.text((x + 2, P - 12), f"{lo:.3f}", fill=(255, 255, 0))
la0 = math.floor((lat - span) / step) * step
for k in range(40):
la = la0 + k * step
y = lat2y(la)
if 0 <= y <= P:
d.line((0, y, P, y), fill=(0, 255, 255), width=1)
d.text((2, y + 1), f"{la:.3f}", fill=(0, 255, 255))
cx, cy = lon2x(lon), lat2y(lat)
d.ellipse((cx - 5, cy - 5, cx + 5, cy + 5), outline=(255, 0, 0), width=2)
d.rectangle((0, 0, P, 14), fill=(0, 0, 0))
d.text((3, 1), f"{name} z{z}", fill=(255, 255, 255))
return m
if __name__ == "__main__":
out = sys.argv[1]
specs = [a.split("|") for a in sys.argv[2:]]
ps = [panel(s[0], float(s[1]), float(s[2]), int(s[3]), int(s[4])) for s in specs]
cols = 2 if len(ps) > 1 else 1
rows = (len(ps) + cols - 1) // cols
sheet = Image.new("RGB", (cols * P, rows * P))
for i, p in enumerate(ps):
sheet.paste(p, ((i % cols) * P, (i // cols) * P))
sheet.save(out, quality=85)
print(out)
+32
View File
@@ -0,0 +1,32 @@
[
{
"name": "Venice",
"country": "Italy",
"reason": "visual: blurry low-res imagery"
},
{
"name": "Chengdu",
"country": "China",
"reason": "visual: smog haze"
},
{
"name": "Xian",
"country": "China",
"reason": "visual: washed-out industrial lot"
},
{
"name": "Maracaibo",
"country": "Venezuela",
"reason": "visual: farmland / scrub"
},
{
"name": "Barquisimeto",
"country": "Venezuela",
"reason": "visual: dry riverbed, no city"
},
{
"name": "Conakry",
"country": "Guinea",
"reason": "visual: heavy haze"
}
]
+69
View File
@@ -0,0 +1,69 @@
"""Fetch candidates and render labelled review sheets for visual verification.
usage: python3 review.py landmarks|cities [start] [count]
Writes tools/flurry/review/<kind>_NN.jpg (5x4 grid, 256px cells, labelled).
"""
import json
import os
import sys
from concurrent.futures import ThreadPoolExecutor
from PIL import Image, ImageDraw
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from tiles import HERE, MAX_WORKERS, capture, quality_ok # noqa: E402
REV = os.path.join(HERE, "review")
def load(kind):
if kind == "landmarks":
from landmarks import LANDMARKS
return [dict(name=n, country=c, lat=la, lon=lo, zoom=z) for n, c, la, lo, z in LANDMARKS]
return json.load(open(os.path.join(HERE, "city_candidates.json")))
def run(cands, prefix, cell=256, cols=5, rows=4):
os.makedirs(REV, exist_ok=True)
def job(c):
try:
img, info = capture(c["lat"], c["lon"], c["zoom"])
return img, info
except Exception as e:
return None, {"error": str(e)}
with ThreadPoolExecutor(MAX_WORKERS) as ex:
res = list(ex.map(job, cands))
per = cols * rows
report = []
for s in range(0, len(cands), per):
sheet = Image.new("RGB", (cols * cell, rows * cell), (20, 20, 20))
d = ImageDraw.Draw(sheet)
for k, (c, (img, info)) in enumerate(zip(cands[s:s + per], res[s:s + per])):
x, y = (k % cols) * cell, (k // cols) * cell
idx = s + k
if img is not None:
sheet.paste(img.resize((cell, cell), Image.LANCZOS), (x, y))
ok, why = quality_ok(info)
lab2 = f"s{info['std']:.0f} g{info['grad']:.1f}" + ("" if ok else " X " + why[:18])
else:
ok, lab2 = False, "ERR"
d.rectangle((x, y, x + cell, y + 24), fill=(0, 0, 0))
d.text((x + 3, y + 1), f"{idx} {c['name'][:36]}", fill=(255, 255, 0))
d.text((x + 3, y + 12), lab2, fill=(255, 120, 120) if not ok else (160, 255, 160))
report.append((idx, c["name"], ok, lab2))
out = os.path.join(REV, f"{prefix}_{s // per:02d}.jpg")
sheet.save(out, quality=85)
print(out)
for r in report:
if not r[2]:
print("REJECT", r)
if __name__ == "__main__":
kind = sys.argv[1]
cands = load(kind)
st = int(sys.argv[2]) if len(sys.argv) > 2 else 0
n = int(sys.argv[3]) if len(sys.argv) > 3 else len(cands)
run(cands[st:st + n], kind[:2] + (f"_{st}" if st else ""))
@@ -0,0 +1,109 @@
"""Pick ~170 city centres from data/cities.json with geographic spread.
Writes tools/flurry/city_candidates.json: ordered list of dicts with
rank ("primary"/"backup") and region. Backups replace rejected primaries of the
same region in build.py.
"""
import json
import math
import os
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
AFRICA = {
"Nigeria", "Egypt", "Congo (Kinshasa)", "South Africa", "Algeria", "Sudan", "Morocco", "Tanzania",
"Cameroon", "Mozambique", "Libya", "Ethiopia", "Ivory Coast", "Tunisia", "Zambia", "Angola", "Ghana",
"Uganda", "Kenya", "Guinea", "Madagascar", "Senegal", "Zimbabwe", "Mali", "Somalia", "Chad", "Niger",
"Benin", "Eritrea", "Somaliland", "South Sudan", "Sierra Leone", "Mauritania", "Malawi", "Togo",
"Congo (Brazzaville)", "Burkina Faso", "Rwanda", "Mauritius", "Gabon", "Equatorial Guinea", "Liberia",
"Djibouti", "Central African Republic", "Guinea Bissau", "Lesotho", "Burundi", "Namibia", "Botswana",
"The Gambia", "Comoros", "Cape Verde", "eSwatini",
}
SOUTH_AMERICA = {"Brazil", "Argentina", "Colombia", "Venezuela", "Peru", "Chile", "Ecuador", "Bolivia",
"Paraguay", "Uruguay", "Guyana", "Suriname"}
OCEANIA = {"Australia", "New Zealand", "Papua New Guinea", "Fiji", "French Polynesia", "Guam"}
CENTRAL_ASIA = {"Kazakhstan", "Uzbekistan", "Turkmenistan", "Kyrgyzstan", "Tajikistan", "Afghanistan", "Mongolia"}
# (region, primary quota, backups, per-country cap, min spacing km)
PASSES = [
("africa", AFRICA, 31, 16, 3, 300),
("south_america", SOUTH_AMERICA, 22, 10, 5, 250),
("oceania", OCEANIA, 12, 4, 5, 120),
("central_asia", CENTRAL_ASIA, 9, 5, 3, 150),
("rest", None, 96, 40, 11, 200),
]
# Picked first (in "rest" unless their country is in a quota region): fill the
# Pacific / high-latitude longitude bands and add visually distinctive cities.
FORCED = [
"Honolulu", "Anchorage", "Papeete", "Agana", "Petropavlovsk-Kamchatsky", "Vancouver", "San Francisco",
"Las Vegas", "Seattle", "Phoenix", "Denver", "Calgary", "Guatemala City", "Havana", "Reykjavík",
"Dubai", "Doha", "Abu Dhabi", "Hong Kong", "Macau", "Venice", "Amsterdam", "København", "Oslo",
"Stockholm", "Helsinki", "Prague", "Kathmandu", "Colombo", "Vladivostok", "Novosibirsk", "Irkutsk",
"Yakutsk", "Norilsk", "Murmansk", "Manaus", "Mombasa", "Windhoek", "Nouakchott", "Marrakesh",
"Samarkand", "Cairns", "Muscat", "Québec",
]
# Natural Earth rows whose point lands in farmland/hills rather than a dense centre.
EXCLUDE = {"Amaravati", "Hechi", "Saidu", "Cranbourne", "Manukau", "Waitakere", "North Shore", "Wahiawa"}
COUNTRY_CAP_TOTAL = 12
def km(a, b):
la1, lo1, la2, lo2 = map(math.radians, (a[1], a[0], b[1], b[0]))
h = math.sin((la2 - la1) / 2) ** 2 + math.cos(la1) * math.cos(la2) * math.sin((lo2 - lo1) / 2) ** 2
return 6371 * 2 * math.asin(math.sqrt(h))
def main():
rows = json.load(open(os.path.join(ROOT, "data", "cities.json")))["rows"]
special = AFRICA | SOUTH_AMERICA | OCEANIA | CENTRAL_ASIA
chosen, out = [], []
byname = {}
for r in rows:
byname.setdefault(r[3], r) # first = most populous
per = {}
def region_of(country):
for reg, mem, *_ in PASSES:
if mem and country in mem:
return reg
return "rest"
for nm in FORCED:
lon, lat, pop, name, country = byname[nm]
chosen.append((lon, lat))
per[country] = per.get(country, 0) + 1
out.append(dict(name=name, country=country, lat=lat, lon=lon, zoom=16, pop=pop,
region=region_of(country), rank="primary", forced=True))
for region, members, quota, nback, cap, spacing in PASSES:
got = sum(1 for c in out if c["region"] == region)
for lon, lat, pop, name, country in rows:
if name in EXCLUDE or name in byname and any(c["name"] == name and c["country"] == country for c in out):
continue
inreg = (country in members) if members else (country not in special)
if not inreg:
continue
if per.get(country, 0) >= cap and got < quota or per.get(country, 0) >= COUNTRY_CAP_TOTAL:
continue
if any(km((lon, lat), (c[0], c[1])) < spacing for c in chosen):
continue
rank = "primary" if got < quota else "backup"
chosen.append((lon, lat))
per[country] = per.get(country, 0) + 1
out.append(dict(name=name, country=country, lat=lat, lon=lon, zoom=16, pop=pop,
region=region, rank=rank))
got += 1
if got >= quota + nback:
break
json.dump(out, open(os.path.join(HERE, "city_candidates.json"), "w"), indent=0, ensure_ascii=False)
prim = [c for c in out if c["rank"] == "primary"]
print(len(prim), "primary,", len(out) - len(prim), "backup")
from collections import Counter
print(Counter(c["region"] for c in prim))
print(Counter(c["country"] for c in prim).most_common(12))
bands = Counter(int((c["lon"] + 180) // 30) for c in prim)
print("30deg bands:", [bands.get(i, 0) for i in range(12)])
if __name__ == "__main__":
main()
+132
View File
@@ -0,0 +1,132 @@
"""Polite cached Esri World Imagery fetch + 512px centred capture + quality metrics."""
import hashlib
import io
import math
import os
import threading
import time
import numpy as np
import requests
from PIL import Image
HERE = os.path.dirname(os.path.abspath(__file__))
CACHE = os.path.join(HERE, "cache")
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
OUT = os.path.join(ROOT, "imagery", "flurry")
ESRI = "https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{z}/{y}/{x}"
UA = "gods-eye-view-intro/1.0 (flurry montage prep)"
MAX_WORKERS = 8
PLACEHOLDER_MD5 = {"f27d9de7f80c13501f470595e327aa6d"} # "Map data not yet available"
_local = threading.local()
def _session():
s = getattr(_local, "s", None)
if s is None:
s = requests.Session()
s.headers["User-Agent"] = UA
_local.s = s
return s
def fetch_tile(z, x, y, retries=6):
n = 2 ** z
x %= n
path = os.path.join(CACHE, str(z), str(x), f"{y}.jpg")
if os.path.exists(path) and os.path.getsize(path) > 0:
with open(path, "rb") as f:
return f.read()
os.makedirs(os.path.dirname(path), exist_ok=True)
url = ESRI.format(z=z, x=x, y=y)
last = None
for attempt in range(retries):
try:
r = _session().get(url, timeout=45)
if r.status_code == 200 and r.content:
tmp = path + f".{threading.get_ident()}.part"
with open(tmp, "wb") as f:
f.write(r.content)
os.replace(tmp, path)
return r.content
last = f"HTTP {r.status_code}"
if r.status_code in (400, 404):
break
except Exception as e:
last = repr(e)
time.sleep(min(20, 1.0 * 2 ** attempt))
raise RuntimeError(f"failed {url}: {last}")
def is_placeholder(data):
if hashlib.md5(data).hexdigest() in PLACEHOLDER_MD5:
return True
a = np.asarray(Image.open(io.BytesIO(data)).convert("RGB")).astype(np.float32)
m = a.mean(axis=(0, 1))
return bool(abs(m - 204.7).max() < 4 and a.std() < 9)
def pixel_xy(lat, lon, z):
n = 2 ** z * 256
px = (lon + 180.0) / 360.0 * n
s = math.sin(math.radians(lat))
py = (0.5 - math.log((1 + s) / (1 - s)) / (4 * math.pi)) * n
return px, py
def capture(lat, lon, z, size=512):
"""Return (image, info). info has placeholder flag and quality metrics."""
px, py = pixel_xy(lat, lon, z)
tx, ty = int(px // 256), int(py // 256)
mosaic = Image.new("RGB", (768, 768))
ox, oy = (tx - 1) * 256, (ty - 1) * 256
left, top = int(round(px - ox - size / 2)), int(round(py - oy - size / 2))
ph = False
for j in range(3):
for i in range(3):
data = fetch_tile(z, tx - 1 + i, ty - 1 + j)
# does this tile overlap the crop window?
tl, tt = i * 256, j * 256
overlaps = tl < left + size and tl + 256 > left and tt < top + size and tt + 256 > top
if overlaps and is_placeholder(data):
ph = True
mosaic.paste(Image.open(io.BytesIO(data)).convert("RGB"), (tl, tt))
img = mosaic.crop((left, top, left + size, top + size))
info = metrics(img)
info["placeholder"] = ph
return img, info
def metrics(img):
a = np.asarray(img).astype(np.float32)
lum = a @ np.array([0.299, 0.587, 0.114], np.float32)
small = np.asarray(img.convert("L").resize((128, 128), Image.BILINEAR)).astype(np.float32)
gx = np.abs(np.diff(lum, axis=1)).mean()
gy = np.abs(np.diff(lum, axis=0)).mean()
# coarse-scale structure: std of 32px block means (catches flat ocean/cloud/ice)
blocks = lum.reshape(16, 32, 16, 32).mean(axis=(1, 3))
sat = (a.max(axis=2) - a.min(axis=2)).mean()
return {
"std": float(lum.std()),
"grad": float((gx + gy) / 2),
"block_std": float(blocks.std()),
"small_std": float(small.std()),
"sat": float(sat),
"mean": float(lum.mean()),
}
def quality_ok(info):
if info["placeholder"]:
return False, "placeholder"
if info["std"] < 12:
return False, f"low variance (std {info['std']:.1f})"
if info["grad"] < 1.3:
return False, f"featureless (grad {info['grad']:.1f})"
if info["mean"] > 225 and info["std"] < 25:
return False, "cloud/snow white-out"
if info["mean"] < 22:
return False, "too dark"
return True, ""
+1
View File
@@ -0,0 +1 @@
cache/
@@ -0,0 +1,76 @@
"""Deliverable 3: clouds.jpg (4096x2048) grayscale cloud opacity map for one recent day.
Source: NASA GIBS daily VIIRS Corrected Reflectance True Colour for the same UTC
date from three satellites (NOAA-20, NOAA-21, Suomi NPP), EPSG:4326 level 3
(10x5 tiles of 512px = 5120x2560), downsampled to 4096x2048.
Cloud metric per satellite:
whiteness = min(R,G,B) (clouds are bright in every channel)
greyness = 1 - saturation (clouds are colourless)
c = ramp(whiteness) * smoothstep(greyness)
The same metric computed on the Esri base map (earth_day_4k.jpg) is subtracted
to suppress bright deserts, salt flats, snow and ice that would otherwise read
as cloud. The three per-satellite results are combined with a per-pixel median
(nan-aware), which removes swath-edge haze / glint streaks that only one
satellite shows, and fills swath gaps. Remaining no-data (polar night) -> 0.
Usage: python3 build_clouds.py [YYYY-MM-DD]
"""
import os
import sys
import numpy as np
from PIL import Image
from common import OUT, gibs_mosaic
Image.MAX_IMAGE_PIXELS = None
DATE = sys.argv[1] if len(sys.argv) > 1 else "2026-10-15"
LAYERS = [
"VIIRS_NOAA20_CorrectedReflectance_TrueColor",
"VIIRS_NOAA21_CorrectedReflectance_TrueColor",
"VIIRS_SNPP_CorrectedReflectance_TrueColor",
]
W, H = 4096, 2048
def smooth(x, a, b):
t = np.clip((x - a) / (b - a), 0, 1)
return t * t * (3 - 2 * t)
def metric(rgb):
rgb = rgb.astype(np.float32) / 255.0
mn = rgb.min(2)
mx = rgb.max(2)
sat = (mx - mn) / (mx + 1e-4)
white = np.clip((mn - 0.14) / 0.86, 0, 1) ** 1.15 # linear-ish: keeps texture inside thick cloud
return white * smooth(1.0 - sat, 0.65, 0.95), mx
def main():
base = np.asarray(Image.open(os.path.join(OUT, "earth_day_4k.jpg")).convert("RGB"))
base_c, _ = metric(base)
# dilate the base metric a few px so small Esri/VIIRS coastline misregistration
# does not leave bright outlines along desert and ice coasts
from PIL import ImageFilter
base_c = np.asarray(Image.fromarray((base_c * 255).astype(np.uint8)).filter(ImageFilter.MaxFilter(3)).filter(ImageFilter.GaussianBlur(2))).astype(np.float32) / 255
stack = []
for L in LAYERS:
m = gibs_mosaic(L, DATE, "250m", 3, 5, 10, "jpg").resize((W, H), Image.LANCZOS)
c, mx = metric(np.asarray(m))
c = np.clip(c - base_c * 1.1, 0, 1)
c[mx < 0.03] = np.nan # no-data (black)
stack.append(c)
stack = np.stack(stack)
with np.errstate(all="ignore"):
import warnings
warnings.simplefilter("ignore")
cloud = np.nanmedian(stack, axis=0)
cloud = np.nan_to_num(cloud, nan=0.0)
cloud = np.clip((cloud - 0.06) / 0.94, 0, 1) # black point: kill faint surface/haze residue
img = Image.fromarray(np.clip(cloud * 255 + 0.5, 0, 255).astype(np.uint8), "L")
img.save(os.path.join(OUT, "clouds.jpg"), quality=92, optimize=True, progressive=True)
print("done", DATE, "mean opacity %.3f" % cloud.mean())
if __name__ == "__main__":
main()
+91
View File
@@ -0,0 +1,91 @@
"""Deliverable 1: earth_day.jpg (8192x4096) + earth_day_4k.jpg (4096x2048).
Esri World Imagery z5 (32x32 tiles, 8192x8192 Web Mercator) reprojected to
equirectangular with bilinear sampling. Because both grids are 8192 px wide and
linear in longitude, the horizontal axis maps 1:1 and only latitude needs
resampling (linear interpolation between Mercator rows = bilinear).
Polar caps beyond the Mercator limit (+-85.0511 deg) come from NASA GIBS
BlueMarble_ShadedRelief_Bathymetry (EPSG:4326, level 4 = 10240x5120; its shaded
bathymetry matches Esri's small-scale ocean rendering), colour-matched to Esri
with a least-squares affine colour transform fitted on co-registered overlap pixels and cross-faded between 80 and 84.5 deg.
"""
import os
import numpy as np
from PIL import Image
from common import OUT, esri_jobs, fetch_many, decode, gibs_mosaic
Image.MAX_IMAGE_PIXELS = None
W, H = 8192, 4096
Z = 5
N = 2 ** Z
TS = 256
MERC = N * TS # 8192
MERC_LIMIT = np.degrees(np.arctan(np.sinh(np.pi))) # 85.0511
FADE_A, FADE_B = 80.0, 84.5 # pure Esri below A, pure Blue Marble above B
CAP_LAYER = os.environ.get("CAP_LAYER", "BlueMarble_ShadedRelief_Bathymetry")
def esri_mercator():
coords = [(x, y) for y in range(N) for x in range(N)]
res = fetch_many(esri_jobs(Z, coords), desc="esri z5")
merc = np.zeros((MERC, MERC, 3), np.uint8)
for (x, y), b in res.items():
if b is None:
raise SystemExit(f"missing esri tile {x},{y}")
merc[y * TS:(y + 1) * TS, x * TS:(x + 1) * TS] = np.asarray(decode(b))
return merc
def main():
merc = esri_mercator()
print("reprojecting")
lat = 90.0 - (np.arange(H) + 0.5) * 180.0 / H
latc = np.clip(lat, -MERC_LIMIT + 1e-6, MERC_LIMIT - 1e-6)
yn = (1.0 - np.log(np.tan(np.pi / 4 + np.radians(latc) / 2)) / np.pi) / 2.0
v = yn * MERC - 0.5
v0 = np.clip(np.floor(v).astype(int), 0, MERC - 1)
v1 = np.clip(v0 + 1, 0, MERC - 1)
f = np.clip(v - np.floor(v), 0, 1).astype(np.float32)[:, None, None]
eq = np.empty((H, W, 3), np.float32)
for s in range(0, H, 256):
e = s + 256
eq[s:e] = merc[v0[s:e]].astype(np.float32) * (1 - f[s:e]) + merc[v1[s:e]].astype(np.float32) * f[s:e]
del merc
print("fetching polar caps:", CAP_LAYER)
bm = gibs_mosaic(CAP_LAYER, None, "500m", 4, 10, 20, "jpg")
bm = np.asarray(bm.resize((W, H), Image.LANCZOS)).astype(np.float32)
# colour-match Blue Marble to Esri per hemisphere: fit an affine 3x3+offset colour
# transform (least squares) on co-registered pixels in the 76-84.5 deg overlap band
alat = np.abs(lat)
band = (alat > 76) & (alat < 84.5)
t = np.clip((alat - FADE_A) / (FADE_B - FADE_A), 0, 1)
t = (t * t * (3 - 2 * t)).astype(np.float32)[:, None, None] # smoothstep
out = eq.copy()
for hemi in (lat > 0, lat < 0):
rows = band & hemi
a = eq[rows][:, ::4].reshape(-1, 3)
b = bm[rows][:, ::4].reshape(-1, 3)
X = np.hstack([b, np.ones((len(b), 1), np.float32)])
# ridge-regularised toward identity: near-grey ice makes the plain fit ill-conditioned
lam = len(b) * 50.0
D = np.diag([1.0, 1.0, 1.0, 0.0])
I0 = np.vstack([np.eye(3), np.zeros((1, 3))])
X64, a64 = X.astype(np.float64), a.astype(np.float64)
M = np.linalg.solve(X64.T @ X64 + lam * D, X64.T @ a64 + lam * D @ I0)
print(" colour transform\n", M.round(3))
h = bm[hemi]
bm[hemi] = (h.reshape(-1, 3) @ M[:3] + M[3]).reshape(h.shape)
out[hemi] = eq[hemi] * (1 - t[hemi]) + bm[hemi] * t[hemi]
img = Image.fromarray(np.clip(out + 0.5, 0, 255).astype(np.uint8))
os.makedirs(OUT, exist_ok=True)
img.save(os.path.join(OUT, "earth_day.jpg"), quality=92, optimize=True, progressive=True)
img.resize((4096, 2048), Image.LANCZOS).save(os.path.join(OUT, "earth_day_4k.jpg"), quality=92, optimize=True, progressive=True)
print("done")
if __name__ == "__main__":
main()
+133
View File
@@ -0,0 +1,133 @@
"""Deliverable 4: Esri World Imagery 'synthetic screenshots' of iconic places.
For each location: <slug>_wide.jpg (zoom 13) and <slug>_close.jpg (zoom 18,
falling back to 17/16 when Esri serves "Map data not yet available"
placeholder tiles), both 1536x1024 centred on the point. Writes index.json.
Usage: python3 build_hero.py [--contact DIR] (contact sheets for visual QA)
Per-location overrides: CLOSE_ZOOM_OVERRIDES below.
"""
import io
import json
import math
import os
import sys
import numpy as np
from PIL import Image
from common import OUT, esri_jobs, fetch_many
HERO = os.path.join(OUT, "hero")
CW, CH = 1536, 1024
TS = 256
LOCATIONS = [
("manhattan", "Midtown Manhattan", 40.7580, -73.9855),
("palm_jumeirah", "Palm Jumeirah", 25.1124, 55.1390),
("singapore_port", "Port of Singapore (Pasir Panjang Terminal)", 1.2730, 103.7690), # was 1.2650,103.8180 (Keppel Bay residential, no port),
("shibuya", "Shibuya Crossing", 35.6595, 139.7005),
("arc_de_triomphe", "Arc de Triomphe", 48.8738, 2.2950),
("giza", "Giza Pyramids", 29.9792, 31.1342),
("rotterdam_port", "Port of Rotterdam", 51.9500, 4.0500),
("lax", "Los Angeles International Airport", 33.9416, -118.4085),
("hong_kong", "Victoria Harbour, Hong Kong", 22.2930, 114.1700),
("panama_canal", "Panama Canal, Miraflores Locks", 8.9966, -79.5915), # was 9.0800,-79.6800 (blurry jungle/lake shore),
("sydney_opera", "Sydney Opera House", -33.8568, 151.2153),
("kennedy_lc39a", "Kennedy Space Center LC-39A", 28.6083, -80.6041),
("heathrow", "London Heathrow Airport", 51.4700, -0.4543),
("rio_maracana", "Maracana Stadium, Rio de Janeiro", -22.9122, -43.2302),
("venice", "Venice", 45.4340, 12.3388),
("las_vegas_strip", "Las Vegas Strip", 36.1147, -115.1728),
("vatican", "Vatican City", 41.9022, 12.4539),
("burj_khalifa", "Burj Khalifa, Dubai", 25.1972, 55.2744),
("golden_gate", "Golden Gate Bridge", 37.8199, -122.4783),
("kremlin", "Moscow Kremlin", 55.7520, 37.6175),
("tiananmen", "Tiananmen Square, Beijing", 39.9037, 116.3917), # was 39.9055,116.3976 (east of the square),
("mumbai_port", "Mumbai Port", 18.9400, 72.8400),
("suez_canal", "Suez Canal, Ismailia", 30.6141, 32.3228), # was 30.5850,32.2650 (Ismailia city blocks, canal off-frame),
("pentagon", "The Pentagon", 38.8719, -77.0563),
]
WIDE_ZOOM = 13
CLOSE_ZOOM = 18
CLOSE_ZOOM_OVERRIDES = {} # slug -> zoom, set after visual QA if needed
DROP = set() # slugs rejected after visual QA
def world_px(lat, lon, z):
n = TS * 2 ** z
x = (lon + 180.0) / 360.0 * n
s = math.sin(math.radians(lat))
y = (0.5 - math.log((1 + s) / (1 - s)) / (4 * math.pi)) * n
return x, y
def is_placeholder(tile):
a = np.asarray(tile).astype(np.float32)
m = a.reshape(-1, 3).mean(0)
return a.std() < 8 and abs(m[0] - m[1]) < 4 and abs(m[1] - m[2]) < 4
def render(lat, lon, z):
"""Return (image, placeholder_fraction)."""
cx, cy = world_px(lat, lon, z)
x0, y0 = int(round(cx - CW / 2)), int(round(cy - CH / 2))
tx0, ty0 = x0 // TS, y0 // TS
tx1, ty1 = (x0 + CW - 1) // TS, (y0 + CH - 1) // TS
n = 2 ** z
coords = [(tx % n, ty) for ty in range(ty0, ty1 + 1) for tx in range(tx0, tx1 + 1)]
data = fetch_many(esri_jobs(z, coords), desc=f"z{z}")
canvas = Image.new("RGB", ((tx1 - tx0 + 1) * TS, (ty1 - ty0 + 1) * TS))
ph = 0
for ty in range(ty0, ty1 + 1):
for tx in range(tx0, tx1 + 1):
b = data.get((tx % n, ty))
if b is None:
ph += 1
continue
t = Image.open(io.BytesIO(b)).convert("RGB")
if is_placeholder(t):
ph += 1
canvas.paste(t, ((tx - tx0) * TS, (ty - ty0) * TS))
crop = canvas.crop((x0 - tx0 * TS, y0 - ty0 * TS, x0 - tx0 * TS + CW, y0 - ty0 * TS + CH))
return crop, ph / len(coords)
def main():
os.makedirs(HERO, exist_ok=True)
contact = None
if "--contact" in sys.argv:
contact = sys.argv[sys.argv.index("--contact") + 1]
os.makedirs(contact, exist_ok=True)
index = []
for slug, name, lat, lon in LOCATIONS:
if slug in DROP:
for k in ("wide", "close"):
p = os.path.join(HERO, f"{slug}_{k}.jpg")
if os.path.exists(p):
os.remove(p)
print(f"{slug}: dropped")
continue
wide, _ = render(lat, lon, WIDE_ZOOM)
z = CLOSE_ZOOM_OVERRIDES.get(slug, CLOSE_ZOOM)
while True:
close, ph = render(lat, lon, z)
if ph == 0 or z <= 16:
break
print(f" {slug}: {ph:.0%} placeholder tiles at z{z}, falling back")
z -= 1
wide.save(os.path.join(HERO, f"{slug}_wide.jpg"), quality=92, optimize=True)
close.save(os.path.join(HERO, f"{slug}_close.jpg"), quality=92, optimize=True)
index.append({"slug": slug, "name": name, "lat": lat, "lon": lon, "close_zoom": z})
print(f"{slug}: close z{z}")
if contact:
sheet = Image.new("RGB", (1536, 512))
sheet.paste(wide.resize((768, 512), Image.LANCZOS), (0, 0))
sheet.paste(close.resize((768, 512), Image.LANCZOS), (768, 0))
sheet.save(os.path.join(contact, f"{slug}.jpg"), quality=85)
with open(os.path.join(HERO, "index.json"), "w") as f:
json.dump(index, f, indent=2)
print(f"{len(index)} locations written")
if __name__ == "__main__":
main()
@@ -0,0 +1,20 @@
"""Deliverable 2: earth_night.jpg (8192x4096) + earth_night_4k.jpg (4096x2048).
NASA Black Marble (layer VIIRS_Black_Marble, 2016 annual composite) from GIBS
EPSG:4326 level 4 (20x10 tiles of 512px = 10240x5120), Lanczos-downsampled.
Colour kept as served.
"""
import os
from PIL import Image
from common import OUT, gibs_mosaic
Image.MAX_IMAGE_PIXELS = None
LAYER = os.environ.get("NIGHT_LAYER", "VIIRS_Black_Marble")
TIME = os.environ.get("NIGHT_TIME", "2016-01-01")
EXT = "png" if LAYER == "VIIRS_Black_Marble" else "jpg"
m = gibs_mosaic(LAYER, TIME if LAYER == "VIIRS_Black_Marble" else None, "500m", 4, 10, 20, EXT)
big = m.resize((8192, 4096), Image.LANCZOS)
big.save(os.path.join(OUT, "earth_night.jpg"), quality=92, optimize=True, progressive=True)
big.resize((4096, 2048), Image.LANCZOS).save(os.path.join(OUT, "earth_night_4k.jpg"), quality=92, optimize=True, progressive=True)
print("done", LAYER, TIME)
@@ -0,0 +1,15 @@
"""Deliverable 5: Esri World Imagery z3 (8x8) and z4 (16x16) tiles for the quadtree indexing effect."""
import os, shutil
from common import OUT, esri_jobs, fetch_many, esri_tile_path
for z in (3, 4):
n = 2 ** z
d = os.path.join(OUT, f"tiles_z{z}")
os.makedirs(d, exist_ok=True)
coords = [(x, y) for y in range(n) for x in range(n)]
res = fetch_many(esri_jobs(z, coords), desc=f"esri z{z}")
bad = [k for k, v in res.items() if v is None]
for x, y in coords:
if res[(x, y)] is not None:
shutil.copyfile(esri_tile_path(z, x, y), os.path.join(d, f"z{z}_{x}_{y}.jpg"))
print(f"z{z}: {len(coords)-len(bad)} tiles written, {len(bad)} failed")
+116
View File
@@ -0,0 +1,116 @@
"""Shared helpers: polite cached tile fetching for Esri World Imagery and NASA GIBS."""
import io
import os
import sys
import time
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
import requests
from PIL import Image
HERE = os.path.dirname(os.path.abspath(__file__))
CACHE = os.path.join(HERE, "cache")
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
OUT = os.path.join(ROOT, "imagery")
MAX_WORKERS = 8
UA = "gods-eye-view-intro/1.0 (texture prep; contact: project maintainer)"
ESRI = "https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{z}/{y}/{x}"
GIBS4326 = "https://gibs.earthdata.nasa.gov/wmts/epsg4326/best/{layer}/default/{time}/{tms}/{z}/{row}/{col}.{ext}"
_local = threading.local()
def _session():
s = getattr(_local, "s", None)
if s is None:
s = requests.Session()
s.headers["User-Agent"] = UA
_local.s = s
return s
def fetch(url, cache_path, retries=6):
"""Return raw bytes for url, caching to cache_path. Retries with backoff."""
if os.path.exists(cache_path) and os.path.getsize(cache_path) > 0:
with open(cache_path, "rb") as f:
return f.read()
os.makedirs(os.path.dirname(cache_path), exist_ok=True)
last = None
for attempt in range(retries):
try:
r = _session().get(url, timeout=60)
if r.status_code == 200 and r.content:
tmp = cache_path + ".part"
with open(tmp, "wb") as f:
f.write(r.content)
os.replace(tmp, cache_path)
return r.content
if r.status_code in (400, 404):
last = f"HTTP {r.status_code}"
break
last = f"HTTP {r.status_code}"
except Exception as e: # network hiccup
last = repr(e)
time.sleep(min(30, 1.5 * 2 ** attempt))
raise RuntimeError(f"failed {url}: {last}")
def fetch_many(jobs, desc=""):
"""jobs: list of (key, url, cache_path). Returns {key: bytes or None}."""
out = {}
done = 0
with ThreadPoolExecutor(MAX_WORKERS) as ex:
futs = {ex.submit(fetch, u, p): k for k, u, p in jobs}
for f in as_completed(futs):
k = futs[f]
try:
out[k] = f.result()
except Exception as e:
print(f" ! {e}", file=sys.stderr)
out[k] = None
done += 1
if done % 100 == 0 or done == len(jobs):
print(f" {desc} {done}/{len(jobs)}", flush=True)
return out
def esri_tile_path(z, x, y):
return os.path.join(CACHE, "esri", str(z), str(y), f"{x}.jpg")
def esri_jobs(z, coords):
return [((x, y), ESRI.format(z=z, x=x, y=y), esri_tile_path(z, x, y)) for x, y in coords]
def gibs_path(layer, time_, tms, z, row, col, ext):
return os.path.join(CACHE, "gibs", layer, time_ or "default", tms, str(z), str(row), f"{col}.{ext}")
def gibs_jobs(layer, time_, tms, z, rows, cols, ext):
t = time_ or "default"
return [((r, c), GIBS4326.format(layer=layer, time=t, tms=tms, z=z, row=r, col=c, ext=ext),
gibs_path(layer, time_, tms, z, r, c, ext)) for r in range(rows) for c in range(cols)]
def decode(b, mode="RGB"):
return Image.open(io.BytesIO(b)).convert(mode)
def gibs_mosaic(layer, time_, tms, z, rows, cols, ext, mode="RGB"):
"""Fetch and stitch a full-globe GIBS EPSG:4326 level (512px tiles)."""
jobs = gibs_jobs(layer, time_, tms, z, rows, cols, ext)
data = fetch_many(jobs, desc=f"{layer} z{z}")
W, H = cols * 512, rows * 512
canvas = Image.new(mode, (W, H))
missing = 0
for (r, c), b in data.items():
if b is None:
missing += 1
continue
canvas.paste(decode(b, mode), (c * 512, r * 512))
if missing:
print(f" {missing} tiles missing for {layer}", file=sys.stderr)
return canvas
@@ -0,0 +1,82 @@
"""MBTA static GTFS -> city/boston/transit.json (subway, Mattapan, commuter rail, bus).
usage: python3 build_transit.py <dir with routes.txt shapes.txt trips.txt>"""
import csv, json, math, os, sys
from collections import defaultdict
sys.path.insert(0, os.path.dirname(__file__))
from common import SCENES, enu
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
G = sys.argv[1]
H = SCENES['boston'][2]; M = H + 1000
def rdp(pts, eps):
if len(pts) < 3: return pts
keep = [False] * len(pts); keep[0] = keep[-1] = True
stack = [(0, len(pts) - 1)]
while stack:
a, b = stack.pop()
ax, ay = pts[a]; bx, by = pts[b]; dx, dy = bx - ax, by - ay; L = math.hypot(dx, dy)
best, bi = -1, -1
for i in range(a + 1, b):
px, py = pts[i]
d = abs(dy * (px - ax) - dx * (py - ay)) / L if L else math.hypot(px - ax, py - ay)
if d > best: best, bi = d, i
if best > eps:
keep[bi] = True; stack += [(a, bi), (bi, b)]
return [p for p, k in zip(pts, keep) if k]
def length(p): return sum(math.hypot(p[i+1][0]-p[i][0], p[i+1][1]-p[i][1]) for i in range(len(p)-1))
routes = {}
for r in csv.DictReader(open(os.path.join(G, 'routes.txt'), encoding='utf-8-sig')):
t = int(r['route_type'])
if t not in (0, 1, 2, 3): continue
if 'Replacement' in r['route_desc'] or r['route_id'].startswith('Shuttle'): continue
name = r['route_short_name'] or r['route_long_name']
if t == 3 and r['route_short_name'] and r['route_long_name']:
name = f"{r['route_short_name']} {r['route_long_name']}"
routes[r['route_id']] = {'name': name, 'color': '#' + (r['route_color'] or '888888').upper(), 'type': t}
shape_rd = {}
for t in csv.DictReader(open(os.path.join(G, 'trips.txt'), encoding='utf-8-sig')):
if t['route_id'] in routes and t['shape_id']:
shape_rd.setdefault(t['shape_id'], (t['route_id'], t['direction_id']))
shapes = defaultdict(list)
for s in csv.DictReader(open(os.path.join(G, 'shapes.txt'), encoding='utf-8-sig')):
if s['shape_id'] in shape_rd:
shapes[s['shape_id']].append((int(s['shape_pt_sequence']), float(s['shape_pt_lat']), float(s['shape_pt_lon'])))
best = {} # (route, dir) -> (len_in_square, segments)
for sid, pts in shapes.items():
pts.sort()
xy = [enu('boston', la, lo) for _, la, lo in pts]
if not any(abs(x) <= H and abs(y) <= H for x, y in xy): continue # must pass within the square
segs, cur = [], []
for i, (x, y) in enumerate(xy):
if abs(x) <= M and abs(y) <= M:
if not cur and i > 0: cur.append(xy[i-1]) # carry edge point
cur.append((x, y))
elif cur:
cur.append((x, y)); segs.append(cur); cur = []
if cur: segs.append(cur)
segs = [rdp(s, 2.0) for s in segs if len(s) > 1]
L = sum(length(s) for s in segs)
key = shape_rd[sid]
if key not in best or L > best[key][0]: best[key] = (L, segs)
out_shapes, seen = [], set()
for (rid, d), (L, segs) in sorted(best.items()):
for s in segs:
flat = tuple(round(v, 1) for p in s for v in p)
if (rid, flat) in seen: continue
seen.add((rid, flat))
out_shapes.append([rid] + list(flat))
used = {s[0] for s in out_shapes}
out = {'routes': {k: v for k, v in routes.items() if k in used}, 'shapes': out_shapes,
'source': 'MBTA GTFS ' + open(os.path.join(G, 'feed_info.txt')).read().splitlines()[1].split(',"')[1].split('"')[0]}
json.dump(out, open(os.path.join(ROOT, 'city', 'boston', 'transit.json'), 'w'), separators=(',', ':'))
bytype = defaultdict(int)
for k in used: bytype[routes[k]['type']] += 1
print('routes', len(used), dict(bytype), 'shapes', len(out_shapes), 'pts', sum((len(s)-1)//2 for s in out_shapes))
print('rail routes:', sorted(k for k in used if routes[k]['type'] != 3))
print('missing requested:', [k for k in ['Red','Orange','Blue','Green-B','Green-C','Green-D','Green-E','Mattapan'] if k not in used])
+15
View File
@@ -0,0 +1,15 @@
import math
SCENES = {
'nyc': (40.7400, -73.9900, 3500),
'boston': (42.3555, -71.0605, 3200),
'lax': (33.9425, -118.4081, 3500),
'bragg': (35.1580, -79.0090, 6000),
}
def enu(slug, lat, lon):
lat0, lon0, _ = SCENES[slug]
x = (lon - lon0) * math.cos(math.radians(lat0)) * 111320
y = (lat - lat0) * 110540
return round(x, 1), round(y, 1)
def inside(slug, x, y, margin=0.0):
h = SCENES[slug][2] + margin
return abs(x) <= h and abs(y) <= h
+76
View File
@@ -0,0 +1,76 @@
"""YOLOv8n (ONNX, CPU) over cctv/cam_*.jpg and cctv/nyc/*.jpg -> cctv/detections.json.
Model auto-downloads to tools/live/models/yolov8n.onnx (git-ignored)."""
import glob, json, os, sys, time, requests
import numpy as np, onnxruntime as ort
from PIL import Image, ImageDraw
from collections import Counter
HERE = os.path.dirname(os.path.abspath(__file__)); ROOT = os.path.abspath(os.path.join(HERE, '..', '..'))
MODEL = os.path.join(HERE, 'models', 'yolov8n.onnx')
MODEL_URL = 'https://huggingface.co/Kalray/yolov8/resolve/main/yolov8n.onnx'
KEEP = {0: 'person', 1: 'bicycle', 2: 'car', 3: 'motorcycle', 5: 'bus', 7: 'truck', 9: 'traffic light', 11: 'stop sign'}
CONF, IOU, S = 0.16, 0.45, 640
if not os.path.exists(MODEL):
os.makedirs(os.path.dirname(MODEL), exist_ok=True)
open(MODEL, 'wb').write(requests.get(MODEL_URL, timeout=120).content)
sess = ort.InferenceSession(MODEL, providers=['CPUExecutionProvider'])
inp = sess.get_inputs()[0].name
def nms(b, s, thr):
order = s.argsort()[::-1]; keep = []
while order.size:
i = order[0]; keep.append(i)
xx0 = np.maximum(b[i, 0], b[order[1:], 0]); yy0 = np.maximum(b[i, 1], b[order[1:], 1])
xx1 = np.minimum(b[i, 2], b[order[1:], 2]); yy1 = np.minimum(b[i, 3], b[order[1:], 3])
inter = np.clip(xx1 - xx0, 0, None) * np.clip(yy1 - yy0, 0, None)
a = (b[:, 2] - b[:, 0]) * (b[:, 3] - b[:, 1])
iou = inter / (a[i] + a[order[1:]] - inter + 1e-9)
order = order[1:][iou <= thr]
return keep
def detect(path):
im = Image.open(path).convert('RGB'); W, H = im.size
r = min(S / W, S / H); nw, nh = round(W * r), round(H * r); px, py = (S - nw) // 2, (S - nh) // 2
canvas = Image.new('RGB', (S, S), (114, 114, 114)); canvas.paste(im.resize((nw, nh), Image.BILINEAR), (px, py))
x = np.asarray(canvas, dtype=np.float32).transpose(2, 0, 1)[None] / 255.0
p = sess.run(None, {inp: x})[0][0].T # 8400 x 84
sc = p[:, 4:]; cls = sc.argmax(1); conf = sc.max(1)
m = (conf >= CONF) & np.isin(cls, list(KEEP))
p, cls, conf = p[m], cls[m], conf[m]
if not len(p): return []
b = np.stack([p[:, 0] - p[:, 2] / 2, p[:, 1] - p[:, 3] / 2, p[:, 0] + p[:, 2] / 2, p[:, 1] + p[:, 3] / 2], 1)
out = []
for c in np.unique(cls): # class-wise NMS
idx = np.where(cls == c)[0]
for k in nms(b[idx], conf[idx], IOU):
j = idx[k]; x0, y0, x1, y1 = b[j]
x0 = min(max((x0 - px) / r / W, 0), 1); x1 = min(max((x1 - px) / r / W, 0), 1)
y0 = min(max((y0 - py) / r / H, 0), 1); y1 = min(max((y1 - py) / r / H, 0), 1)
if x1 - x0 > 0 and y1 - y0 > 0:
out.append([round(float(x0), 4), round(float(y0), 4), round(float(x1), 4), round(float(y1), 4), round(float(conf[j]), 3), KEEP[int(c)]])
return sorted(out, key=lambda d: -d[4])
COL = {'person': (255, 60, 60), 'car': (60, 200, 255), 'truck': (255, 180, 0), 'bus': (255, 0, 255), 'bicycle': (0, 255, 120),
'motorcycle': (180, 255, 0), 'traffic light': (255, 255, 255), 'stop sign': (255, 0, 0)}
def render(path, dets, dst):
im = Image.open(path).convert('RGB'); W, H = im.size; d = ImageDraw.Draw(im)
for x0, y0, x1, y1, c, n in dets:
d.rectangle([x0 * W, y0 * H, x1 * W, y1 * H], outline=COL[n], width=2)
d.text((x0 * W + 2, max(0, y0 * H - 11)), f'{n} {c:.2f}', fill=COL[n])
im.save(dst, quality=90)
if __name__ == '__main__':
files = sorted(glob.glob(os.path.join(ROOT, 'cctv', 'cam_*.jpg'))) + sorted(glob.glob(os.path.join(ROOT, 'cctv', 'nyc', '*.jpg')))
res, tot, t0 = {}, Counter(), time.time()
for f in files:
rel = os.path.relpath(f, os.path.join(ROOT, 'cctv'))
res[rel] = detect(f); tot.update(d[5] for d in res[rel])
dt = time.time() - t0
json.dump(res, open(os.path.join(ROOT, 'cctv', 'detections.json'), 'w'), separators=(',', ':'))
print('images', len(files), 'secs', round(dt, 1), 'ms/img', round(1000 * dt / max(1, len(files))))
print('totals', dict(tot.most_common()), 'all', sum(tot.values()))
print('nyc totals', dict(Counter(d[5] for k, v in res.items() if k.startswith('nyc/') for d in v)))
picks = sys.argv[1:] or [max((k for k in res if k.startswith('nyc/')), key=lambda k: len(res[k])),
max((k for k in res if not k.startswith('nyc/')), key=lambda k: len(res[k])),
max(res, key=lambda k: sum(d[5] == 'person' for d in res[k]))]
for i, k in enumerate(picks, 1):
render(os.path.join(ROOT, 'cctv', k), res[k], os.path.join(HERE, f'det_check_{i}.jpg')); print('render', i, k, len(res[k]))
+72
View File
@@ -0,0 +1,72 @@
"""Poll live ADS-B (adsb.lol, fallback adsb.fi) around each scene 3x, 20 s apart.
Writes city/<slug>/air.json with latest sample per aircraft + short track history."""
import json, os, sys, time, requests
sys.path.insert(0, os.path.dirname(__file__))
from common import SCENES, enu
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
RAD = {'nyc': 25, 'lax': 25, 'bragg': 25, 'boston': 20}
HELI_TYPES = {'EC35','EC30','EC45','EC55','EC20','EC25','EC75','EC13','EC40','AS50','AS55','AS65','AS32','AS3B','H125','H130','H135','H145','H160','H175','B06','B06T','B407','B412','B427','B429','B430','B47G','B505','B212','B222','B230','B105','BK17','R22','R44','R66','S76','S92','S61','S64','S70','H60','UH60','MH60','HH60','H47','CH47','H53','A109','A119','A139','A169','A189','AW09','MI8','MI17','UH1','UH1Y','AH1','AH64','H64','V22','MV22','CABR','G2CA','EN28','EN48','H269','H500','MD52','MD60','NH90','LYNX','PUMA','GAZL','EH10','H1','B505'}
H = {'User-Agent': 'gods-eye-view-intro/1.0 (film research)'}
def fetch(slug):
lat, lon, _ = SCENES[slug]; nm = RAD[slug]
for url, key in ((f'https://api.adsb.lol/v2/point/{lat}/{lon}/{nm}', 'ac'),
(f'https://opendata.adsb.fi/api/v2/lat/{lat}/lon/{lon}/dist/{nm}', 'aircraft')):
try:
r = requests.get(url, headers=H, timeout=20); r.raise_for_status(); j = r.json()
ac = j.get('ac') or j.get('aircraft') or []
now = j.get('now') or time.time() * 1000
return url, now / 1000 if now > 1e11 else now, ac
except Exception as e:
print(slug, 'fail', url, e, file=sys.stderr)
return None, time.time(), []
def num(v):
try: return float(v)
except Exception: return None
def conv(slug, a):
lat, lon = num(a.get('lat')), num(a.get('lon'))
if lat is None or lon is None:
lp = a.get('lastPosition') or {}
lat, lon = num(lp.get('lat')), num(lp.get('lon'))
if lat is None: return None
x, y = enu(slug, lat, lon)
ab = a.get('alt_baro')
if ab == 'ground': alt = 0.0
else:
v = num(ab)
if v is None: v = num(a.get('alt_geom'))
alt = round(v * 0.3048, 1) if v is not None else None
trk = num(a.get('track')); trk = num(a.get('true_heading')) if trk is None else trk
gs = num(a.get('gs')); gs = round(gs * 0.514444, 1) if gs is not None else None
vr = num(a.get('baro_rate')); vr = num(a.get('geom_rate')) if vr is None else vr
vr = round(vr * 0.00508, 2) if vr is not None else None
return [x, y, alt, trk, gs, vr, (a.get('flight') or '').strip(), a.get('t') or '', a.get('category') or '']
def main():
polls = {s: [] for s in SCENES}
for i in range(3):
t0 = time.time()
for s in SCENES:
url, t, ac = fetch(s); polls[s].append((url, t, ac)); print(i, s, len(ac), url, flush=True)
if i < 2: time.sleep(max(0, 20 - (time.time() - t0)))
summary = {}
for s in SCENES:
latest, tracks, src = {}, {}, set()
tlast = 0
for url, t, ac in polls[s]:
if url: src.add(url.split('/')[2])
tlast = max(tlast, t)
for a in ac:
hx = a.get('hex'); row = conv(s, a)
if not hx or row is None: continue
if (num(a.get('seen_pos')) or 0) > 60: continue
latest[hx] = row
tr = tracks.setdefault(hx, [])
p = [row[0], row[1], row[2]]
if not tr or tr[-1][:2] != p[:2]: tr.append(p)
heli = [h for h, r in latest.items() if r[8] == 'A7' or r[7] in HELI_TYPES]
out = {'t': int(tlast), 'source': sorted(src), 'radius_nm': RAD[s], 'hex': list(latest.keys()),
'a': list(latest.values()), 'tracks': {h: v for h, v in tracks.items() if len(v) > 1}}
json.dump(out, open(os.path.join(ROOT, 'city', s, 'air.json'), 'w'), separators=(',', ':'))
summary[s] = {'aircraft': len(latest), 'with_track': len(out['tracks']), 'heli': len(heli),
'heli_list': [[latest[h][6], latest[h][7], latest[h][8], latest[h][2]] for h in heli]}
print(json.dumps(summary, indent=1))
if __name__ == '__main__': main()
@@ -0,0 +1,21 @@
"""Live MBTA vehicle positions -> city/boston/vehicles.json (square + 1 km)."""
import json, os, sys, requests
from collections import Counter
sys.path.insert(0, os.path.dirname(__file__))
from common import enu, inside
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
j = requests.get('https://cdn.mbta.com/realtime/VehiclePositions_enhanced.json', timeout=30).json()
rows, total = [], 0
for e in j['entity']:
v = e.get('vehicle') or {}; p = v.get('position') or {}
if 'latitude' not in p: continue
total += 1
x, y = enu('boston', p['latitude'], p['longitude'])
if not inside('boston', x, y, 1000): continue
rid = (v.get('trip') or {}).get('route_id', '')
lab = (v.get('vehicle') or {}).get('label') or (v.get('vehicle') or {}).get('id', '')
sp = p.get('speed'); rows.append([x, y, p.get('bearing'), round(sp, 2) if sp is not None else None, rid, lab])
out = {'t': int(j['header']['timestamp']), 'source': 'cdn.mbta.com VehiclePositions_enhanced.json', 'v': rows}
json.dump(out, open(os.path.join(ROOT, 'city', 'boston', 'vehicles.json'), 'w'), separators=(',', ':'))
c = Counter(r[4].split('-')[0] if r[4].startswith(('Green', 'CR')) else ('bus' if r[4][:1].isdigit() or r[4][:2] in ('SL','CT') else r[4]) for r in rows)
print('t', out['t'], 'feed total', total, 'in square+1km', len(rows), dict(c))
@@ -0,0 +1,53 @@
"""NYC DOT TMC cameras -> city/nyc/cameras.json + current stills for ~24 nearest cams."""
import hashlib, io, json, os, sys, time, requests
import numpy as np
from PIL import Image, ImageOps
from collections import Counter
sys.path.insert(0, os.path.dirname(__file__))
from common import enu, inside
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
OUT = os.path.join(ROOT, 'cctv', 'nyc'); DBG = os.path.join(os.path.dirname(__file__), 'nyc_rejects')
os.makedirs(OUT, exist_ok=True); os.makedirs(DBG, exist_ok=True)
H = {'User-Agent': 'gods-eye-view-intro/1.0'}
cat = requests.get('https://webcams.nyctmc.org/api/cameras', headers=H, timeout=30).json()
cams = []
for c in cat:
if str(c.get('isOnline')).lower() != 'true': continue
try: lat, lon = float(c['latitude']), float(c['longitude'])
except Exception: continue
x, y = enu('nyc', lat, lon)
if inside('nyc', x, y): cams.append((x, y, c['id'], c['name'].strip(), lat, lon))
json.dump([[x, y, i, n] for x, y, i, n, _, _ in cams], open(os.path.join(ROOT, 'city', 'nyc', 'cameras.json'), 'w'), separators=(',', ':'))
print('catalog', len(cat), 'online in square', len(cams))
cams.sort(key=lambda c: c[0]**2 + c[1]**2)
for f in os.listdir(OUT):
if f.startswith('cam_') and f.endswith('.jpg'): os.remove(os.path.join(OUT, f))
idx, rejects, hashes, raw_imgs = [], Counter(), Counter(), []
fetched_utc = time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())
for x, y, cid, name, lat, lon in cams[:80]:
if len(raw_imgs) >= 40: break
try:
r = requests.get(f'https://webcams.nyctmc.org/api/cameras/{cid}/image', headers=H, timeout=20)
r.raise_for_status(); im = Image.open(io.BytesIO(r.content)).convert('RGB')
except Exception as e:
rejects['error'] += 1; continue
h = hashlib.md5(r.content).hexdigest(); hashes[h] += 1
g = np.asarray(im.convert('L'), dtype=np.float32)
raw_imgs.append(dict(x=x, y=y, id=cid, name=name, lat=lat, lon=lon, im=im, h=h, mean=float(g.mean()), std=float(g.std()), size=im.size, n=len(r.content)))
n = 0
for d in raw_imgs:
reason = None
if hashes[d['h']] > 1: reason = 'duplicate/placeholder'
elif d['mean'] < 14: reason = 'black'
elif d['std'] < 10: reason = 'flat'
elif d['size'][0] < 200: reason = 'tiny'
if reason:
rejects[reason] += 1; d['im'].save(os.path.join(DBG, f"{reason.split('/')[0]}_{d['id'][:8]}.jpg")); continue
if n >= 24: break
n += 1; fn = f'cam_{n:03d}.jpg'
ImageOps.fit(d['im'], (640, 360), Image.LANCZOS).save(os.path.join(OUT, fn), quality=88)
idx.append({'file': fn, 'id': d['id'], 'name': d['name'], 'x': d['x'], 'y': d['y'], 'lat': d['lat'], 'lon': d['lon'],
'orig_size': list(d['size']), 'fetched_utc': fetched_utc, 'source': 'NYC DOT TMC webcams.nyctmc.org'})
print(fn, d['name'], d['size'], round(d['mean']), round(d['std']), round((d['x']**2+d['y']**2)**.5))
json.dump(idx, open(os.path.join(OUT, 'index.json'), 'w'), indent=1)
print('saved', n, 'rejects', dict(rejects), 'tried', len(raw_imgs))
+15
View File
@@ -0,0 +1,15 @@
#!/usr/bin/env bash
# Render one format in 8 s chunks (skips chunks already on disk), then concat.
# tools/render-all.sh 1920 1080 16x9
set -u
W=$1; H=$2; TAG=$3
CH=240; TOTAL=${4:-2850}
mkdir -p out/${PREFIX:-}chunks_$TAG
for ((s=0; s<TOTAL; s+=CH)); do
e=$((s+CH)); f=out/${PREFIX:-}chunks_$TAG/c_$(printf %05d $s).mp4
[ -s "$f" ] && continue
e=$(( e > TOTAL ? TOTAL : e ))
node tools/render.mjs --port ${PORT:-8766} --w $W --h $H --from $s --to $e --video "$f.tmp.mp4" --crf 14 --preset slow && mv "$f.tmp.mp4" "$f"
done
ls out/${PREFIX:-}chunks_$TAG/c_*.mp4 | sed "s|^|file '$PWD/|; s|$|'|" > out/${PREFIX:-}chunks_$TAG/list.txt
ffmpeg -y -loglevel error -f concat -safe 0 -i out/${PREFIX:-}chunks_$TAG/list.txt -c copy out/${PREFIX:-}video_$TAG.mp4 && echo "CONCAT OK $TAG"
+91
View File
@@ -0,0 +1,91 @@
// Deterministic frame renderer: drives src/main.js frame by frame in headless
// Chromium and either writes stills or pipes straight into ffmpeg.
//
// node tools/render.mjs --w 1920 --h 1080 --frames 480,900,1500 --out out/stills
// node tools/render.mjs --w 1920 --h 1080 --from 0 --to 2160 --video out/video_16x9.mp4
import { chromium } from 'playwright-core';
import { spawn } from 'node:child_process';
import fs from 'node:fs';
import path from 'node:path';
const args = Object.fromEntries(
process.argv
.slice(2)
.join(' ')
.split('--')
.filter(Boolean)
.map((s) => {
const [k, ...v] = s.trim().split(' ');
return [k, v.join(' ') || true];
}),
);
const W = +args.w || 1920;
const H = +args.h || 1080;
const URL = `http://127.0.0.1:${args.port || 8765}/index.html?w=${W}&h=${H}`;
const CHROME = '/opt/pw-browsers/chromium-1194/chrome-linux/chrome';
const browser = await chromium.launch({
executablePath: CHROME,
args: ['--use-gl=angle', '--use-angle=swiftshader', '--enable-unsafe-swiftshader', '--ignore-gpu-blocklist', '--disable-gpu-vsync', '--js-flags=--max-old-space-size=8192'],
});
const page = await browser.newPage({ viewport: { width: W, height: H }, deviceScaleFactor: 1 });
page.on('console', (m) => {
const t = m.text();
if (/error|warn|missing|\[gev\]/i.test(t)) console.log(' page:', t.slice(0, 300));
});
page.on('pageerror', (e) => console.log(' PAGE ERROR:', e.message));
await page.goto(URL);
await page.waitForFunction(() => window.__status === 'ready' || String(window.__status).startsWith('error'), null, { timeout: 600000, polling: 500 });
const st = await page.evaluate(() => window.__status);
if (st !== 'ready') {
console.error('init failed:', st);
process.exit(1);
}
const info = await page.evaluate(() => ({ counts: GEV.counts, heroes: GEV.heroes, cams: GEV.cams, frames: GEV.frames }));
console.log('ready', JSON.stringify(info));
const cdp = await page.context().newCDPSession(page);
async function grab(format = 'png') {
const r = await cdp.send('Page.captureScreenshot', { format, quality: format === 'jpeg' ? 96 : undefined, fromSurface: true, captureBeyondViewport: false });
return Buffer.from(r.data, 'base64');
}
let frames = [];
if (args.frames) frames = String(args.frames).split(',').map((x) => (x.includes('s') ? Math.round(parseFloat(x) * 30) : +x));
else {
const from = +(args.from || 0);
const to = +(args.to || info.frames);
const step = +(args.step || 1);
for (let f = from; f < to; f += step) frames.push(f);
}
let ff = null;
if (args.video) {
fs.mkdirSync(path.dirname(args.video), { recursive: true });
ff = spawn('ffmpeg', ['-y', '-loglevel', 'error', '-f', 'image2pipe', '-framerate', '30', '-c:v', 'png', '-i', '-', '-c:v', 'libx264', '-preset', args.preset || 'slow', '-crf', String(args.crf || 14), '-pix_fmt', 'yuv420p', '-colorspace', 'bt709', '-color_primaries', 'bt709', '-color_trc', 'bt709', '-movflags', '+faststart', args.video], { stdio: ['pipe', 'inherit', 'inherit'] });
} else {
fs.mkdirSync(args.out || 'out/stills', { recursive: true });
}
const t0 = Date.now();
for (let i = 0; i < frames.length; i++) {
const f = frames[i];
await page.evaluate((fr) => window.GEV.renderFrame(fr), f);
const buf = await grab('png');
if (ff) {
if (!ff.stdin.write(buf)) await new Promise((r) => ff.stdin.once('drain', r));
} else {
fs.writeFileSync(path.join(args.out || 'out/stills', `${args.prefix || 'f'}_${String(f).padStart(5, '0')}.png`), buf);
}
if (i % 30 === 0 || i === frames.length - 1) {
const el = (Date.now() - t0) / 1000;
const per = el / (i + 1);
console.log(`frame ${f} (${i + 1}/${frames.length}) ${per.toFixed(2)} s/frame, eta ${((frames.length - i - 1) * per / 60).toFixed(1)} min`);
}
}
if (ff) {
ff.stdin.end();
await new Promise((r) => ff.on('close', r));
}
await browser.close();
console.log('done in', ((Date.now() - t0) / 60000).toFixed(1), 'min');
+19
View File
@@ -0,0 +1,19 @@
# usage: python3 tools/sheet.py <dir> <out_prefix> [per_sheet] [cell_w]
import sys, glob
from PIL import Image
d, outp = sys.argv[1], sys.argv[2]
per = int(sys.argv[3]) if len(sys.argv) > 3 else 4
cw = int(sys.argv[4]) if len(sys.argv) > 4 else 960
fs = sorted(glob.glob(d + '/*.png'))
for s in range(0, len(fs), per):
group = fs[s:s + per]
ims = [Image.open(f) for f in group]
w0, h0 = ims[0].size
ch = int(cw * h0 / w0)
cols = 2 if w0 > h0 else 4
rows = (len(ims) + cols - 1) // cols
sheet = Image.new('RGB', (cw * cols, ch * rows))
for i, im in enumerate(ims):
sheet.paste(im.resize((cw, ch), Image.LANCZOS), ((i % cols) * cw, (i // cols) * ch))
sheet.save(f'{outp}_{s // per:02d}.jpg', quality=88)
print(f'{outp}_{s // per:02d}.jpg', [f.split('/')[-1] for f in group])