How RaceCut renders a 60fps music video in your browser
When you press Race it, RaceCut simulates a physics race, draws 1,800 frames at 1080×1920, encodes them to H.264, muxes AAC audio, and hands you an MP4 - and none of it touches a server. Your phone is the render farm. Here’s the pipeline, and the scars.
Why client-side at all
Three reasons, in honest order. Cost: a render farm for 60fps vertical video would be our biggest bill, and we’d be paying it mostly for free users. Privacy falls out for free: your audio never uploads, which we get to say truthfully because there is nowhere for it to go. Latency: an M-series laptop renders a 15-second race in about 9 seconds - faster than we could upload the song.
The price is that our “server” is whatever device walks in the door, and telemetry says what walks in the door is 95% phones, many with 4GB of RAM and GPUs we’d never heard of before their crash reports introduced us.
The pipeline
Everything below runs inside a Web Worker, so the page stays responsive while the tab quietly does the work of a small studio:
- Bake. The physics engine is Rapier - a Rust rigid-body engine - compiled to WebAssembly. We simulate the entire race up front, deterministically from a seed, and record the timeline: every position, every collision, and crucially who leads when. Baking first is what makes rigged races possible (re-roll seeds until your pick wins, honestly) and what gives the voice pipeline exact hand-off times.
- Compose. For the music formats, the course itself is generated from the song before any physics runs: spectral-flux onset detection finds the beats and the bright sections (rustfft, in the same wasm), and corridor generation places geometry so the action lands on the music. This step has a hard time budget - if a course can’t be composed in 20 seconds on your device, we take the best candidate found so far rather than let a phone spin forever.
- Render. WebGPU draws the baked timeline at 60fps into an offscreen canvas - Bevy’s renderer, also Rust-in-wasm. A post-pass burns in the watermark and the creator’s logo. We ask WebGPU for its guaranteed defaults rather than everything the adapter advertises, because mobile drivers overstate what they support and then die at pipeline creation. We learned that from the crash telemetry, repeatedly.
- Encode. WebCodecs hands us the device’s hardware H.264 encoder. Phones that can’t do High profile get offered the next rung down instead of failing. Audio is encoded to AAC the same way, and an mp4 muxer stitches it all into the file you download.
The Safari audio bug
Our favourite scar. Safari would produce videos that played perfectly - silently. Same file played fine in Chrome, fine in VLC. The cause: Safari’s AAC decoder demands an AudioSpecificConfig - a few bytes describing sample rate and channel layout - that other players will happily infer when it’s missing. The encoder wasn’t attaching it, so Safari shrugged and muted the track. The fix is almost insulting: we synthesise those bytes ourselves and staple them into the MP4. Two bytes of config, a week of confusion.
What 4GB Android phones taught us
- Load the engine early. The wasm bundle is ~8MB. We start downloading it the moment a format is chosen, because a phone that spends its 45-second compose budget downloading is a phone that reports “composing took too long” through no fault of the composer.
- A failed load must be forgotten. One aborted wasm download used to poison every race until the page reloaded, because the broken module got cached. Now a load that never finished evicts itself.
- Screens sleep mid-render. A render is a minute of full-tilt GPU work; phones dim and suspend. We hold a wake lock for exactly the duration of the render.
- When the GPU device is lost, say so. “unreachable” panics from wasm are useless; a crash that knows the GPU device died mid-render is actionable. Every failure we see now carries where Rust died and what the device was - 4GB, Mali Bifrost, Adreno 7xx - which is how most of the fixes above got found.
The part that still amazes us
A kid in Colombia with a mid-range Android opens a browser tab, and that tab detects beats with a Fourier transform, simulates rigid-body physics, renders with the same GPU API games use, and encodes broadcast-format video - then he posts it and his classmates argue about which marble deserved to win. No install, no upload, no queue. The browser grew up while nobody was watching, and we’re happily exploiting it.