Skip to content
all posts

~/blog/f1-live-deep-dive_

Building F1 Live: real-time fan-out to 10,000 viewers

7 min readwebsocketsredistypescriptsystem design

Live scores, trading screens and multiplayer games all solve the same problem: push the same changing state to many people at once, quickly, without one slow connection dragging everyone down. I wanted to understand how that actually works, and what breaks when you do it.

Live F1 timing data is paid, but every past race is free from OpenF1. So I built F1 Live: it replays a real race as if it were happening now, as one shared broadcast. Cars move on the track, the leaderboard updates, race control calls out yellow flags and penalties, team radio plays, and the start lights go out at the same moment for every viewer.

This post walks through how it works, the decisions that mattered, and the bugs that taught me the most.

The shape of the system

OpenF1 ─(download once)─► data/<race>/ ─► replay ─► Redis ─► gateway(s) ─► browsers

Three small Node.js processes, written in TypeScript:

  • replay reads a recorded race and publishes it as if it were live.
  • Redis sits in the middle: a pub/sub channel for the live messages, and a key holding the latest full state.
  • gateway holds the WebSocket connections and fans each message out to every browser.

The replay doesn't know how many gateways exist, and the gateways don't know about each other. That's what makes the system scale: need more viewers, run more gateways.

Record once, replay forever

A race comes from several OpenF1 endpoints: car locations, positions, intervals, laps and race-control messages. Instead of calling the API during playback, I download everything once and normalise it into one time-sorted timeline of small tuples:

['L', 60050, 1, 4660, -5202]   // location: t (ms), driver, x, y
['P', 63117, 4, 3]             // position: driver 4 is now P3
['R', 416, 'GREEN', 'GREEN LIGHT - PIT EXIT OPEN']  // race control

A full race is about 650,000 events and 17 MB. Sorting happens once, at download time, so playback is just walking an array. Recording also separates the system from the data source: tests are repeatable, free, and never hit a rate limit.

One message every 250 ms

Playing events one by one would mean about 80 messages a second at 1x, and 50 times that at 50x. Instead, the replay batches: every 250 ms it sends one message containing whatever changed in that window. Position, gap and lap keep only their latest value; locations keep every sample, so animation stays smooth. The message rate is constant at any speed; only the size grows.

Each message is a delta. For anyone arriving late, the replay also keeps a full snapshot in a Redis key. A new viewer gets the snapshot, then the deltas.

Every message carries a sequence number, so a consumer can spot a missed one. That turned out not to be enough (see "what went wrong").

Subscribe first, then read the snapshot

Redis pub/sub is fire-and-forget: a message goes to whoever is subscribed at that instant, and nothing is stored. So when a gateway starts, the order of two steps matters:

  • Read the snapshot, then subscribe: a message published between the two steps is lost forever.
  • Subscribe, then read the snapshot: the worst case is a duplicate, which the sequence number filters out.

A duplicate is recoverable; a lost message isn't. That one line of ordering is easy to get wrong and hard to debug later.

Fan-out: serialize once

The first version of the gateway encoded the message to JSON separately for each client. At 10,000 clients, re-encoding a 2 KB snapshot took about 50 ms per tick, about a fifth of a CPU core, on a runtime with one thread for your JavaScript. Encoding once and sending the same bytes to every socket takes about 0.1 ms.

When a client can't keep up

One viewer on a bad mobile connection must not slow down the other 9,999. Each WebSocket reports how many bytes are queued for it (bufferedAmount), and the gateway uses that to decide, per client, per tick:

if (c.ws.bufferedAmount > SOFT_LIMIT) {
  c.lagSince ??= now;
  if (now - c.lagSince > MAX_LAG_MS) {
    // Behind for too long to be worth serving: evict it.
    // It can reconnect and start again from a snapshot.
    c.ws.terminate();
    continue;
  }
  // Lossy is fine for a live map: a newer frame supersedes this one.
  c.needsSnapshot = true;
  continue;
}

Over 64 KB queued, the client's ticks are skipped and it's flagged for a snapshot. When it catches up, one snapshot covers everything it missed. If it stays behind for 10 seconds, it's disconnected. Losing frames is fine for live data, because the next frame replaces it anyway.

A heartbeat handles the other failure: connections that die without saying goodbye (a laptop lid closing, Wi-Fi dropping). The gateway pings every 15 seconds and terminates anything that didn't answer the last one.

Reconnecting without a stampede

When a gateway restarts, every browser connected to it drops at the same moment. If they all retry on the same schedule, they all hit the next server in the same millisecond. The client uses exponential backoff with full jitter:

const delay = Math.random() * Math.min(10_000, 500 * 2 ** attempt++);

The randomness spreads the reconnects out. The gateway also closes connections with code 1012 ("service restart") on shutdown, so clients know to come back.

Drawing the past to make it smooth

Car positions arrive about four times a second. Drawing them as they arrive makes cars jump. The browser instead draws the race 600 ms in the past, where it always has a sample before and after the current moment, and moves each car at constant speed between them. Multiplayer games call this an interpolation buffer. The trade-off is 600 ms of extra delay, which nobody watching a replay notices.

Who controls the broadcast

Everyone watches the same race at the same speed, so someone has to be in charge of switching races and changing speed. The rules are a small pure function: a lone viewer controls; if an admin is present, only admins control; otherwise the longest-connected viewer does.

Every gateway has to agree on who that is. Rather than storing every viewer in Redis, each gateway writes a tiny summary every 2 seconds (its viewer count, how many admins, and its earliest join time) with a 6-second expiry. That's about 100 bytes no matter how many viewers there are, and a crashed gateway simply expires. If presence can't be read, nobody can control: it fails closed.

Load test: 10,000 viewers

"It works in one tab" says nothing about ten thousand. I used k6 to open thousands of WebSocket connections, measure latency from publish to receive, and count sequence gaps (which must be zero):

Viewersp50 / p99 latencyGapsGateway CPUGateway memory
1,00029 / 58 ms0~12% of 1 core104 MB
10,00058 / 298 ms0~50% of 1 core240 MB

One gateway handled 10,000 viewers with headroom: no skipped ticks, no evictions. The load generator used three to five CPU cores, more than the gateway, so some of that latency is k6's own. The next step for honest numbers is running k6 from a different machine.

Running it for $0

Most free hosting tiers don't fit an always-on real-time app: they put idle apps to sleep, are time-limited trials, or cap Redis far below the roughly 700,000 commands a day this uses. What works is a real always-on machine, and Oracle Cloud's Always Free tier includes an ARM VM with 12 GB of memory.

The deployment is one Docker Compose file: Redis, the replay, the gateway, and Caddy in front. Caddy gets and renews a Let's Encrypt certificate on its own and proxies WebSockets with no extra configuration. Only Caddy publishes ports; Redis and the gateway are reachable only inside the private Docker network.

What went wrong

  • Redis pub/sub ignores the database number. I ran "isolated" tests against a separate Redis database, and they leaked into the live broadcast, because channels are shared by the whole server. Fix: prefix every key and channel with a namespace.
  • A restarted replay started counting from 1 again. The gateway, which had reached message 2,242, dropped everything as a duplicate and froze. Fix: every message carries an epoch, an ID for the replay run, and sequence numbers are only compared within one epoch.
  • The canvas grew in a feedback loop on Retina screens. Its pixel size (twice its CSS size) fed back into the grid layout, so the track slowly got cut off. Fix: absolute positioning, and resizing only when the size actually changes.
  • The admin link returned 404, because static files were matched including the query string. And browser extensions logged the secret in that URL before the page could remove it. Fix: match on the path only, rotate the secret, and treat URLs as public.
  • Oracle's firewall rule landed after its own REJECT rule. The usual advice inserts the rule at a fixed position, which on this image came after the catch-all REJECT, so ports 80 and 443 stayed closed. Fix: insert at the REJECT's position.

What's next

A shared rewind of 10–20 seconds for everyone (a "seek" command), and possibly limiting race downloads to the admin on the public site.

Race data from OpenF1 (CC BY-NC-SA). Team radio © Formula 1. A non-commercial learning project.