The problem
The platform needed both the virtual environment and the live video inside it to stay synchronised between users. The game state was under our control. The video was not.
Different users watching the same Twitch or YouTube live stream could be roughly 0–8 seconds apart. That is acceptable for normal individual viewing, but it breaks a shared virtual event: one user can react to something several seconds before another user sees it.
The first solution: perfect, but expensive
Our first implementation used RTSP through VLC. The server received the original stream and redistributed audio and video to every user. Technically, it worked very well because the platform controlled the media path.
The problem was cost. At the transfer price used in our original estimate, one hour of 720p video was about 1.75 GB per user. At $0.09/GB that was approximately $0.157 per user per hour. The cost then scaled with users, event duration and the number of simultaneous stadiums.
Higher resolutions and frame rates made the problem worse. The architecture was technically clean, but the commercial model was wrong: we were paying to distribute video that Twitch or YouTube were already distributing.
Trying to control the live stream
I first investigated whether we could control which HLS segments VLC played. Live streams expose an M3U8 playlist containing short media segments, so in theory manipulating those segments looked promising.
I worked through VLC source code, experimented with its demuxing behaviour, tried runtime playlist modification and also tested an FFmpeg-based local cache. None of those approaches gave us a stable production solution. Modifying only part of VLC’s data flow caused mismatches and instability, while the FFmpeg approach introduced read/write locking issues when passing cached media back into Unreal.
The useful observation
The breakthrough came from looking at a much simpler boundary: VLC was already decoding the stream and handing audio and video frames to Unreal.
Instead of trying to move the live source backwards, I tried buffering the decoded frames inside the client. New frames continued entering the buffer while playback consumed older frames. That effectively moved the user behind live time without changing the upstream stream itself.
The key change: stop trying to synchronise the providers. Let each client receive the provider stream normally, then synchronise playback of locally buffered frames.
A shared master timecode
Once clients could deliberately play behind live time, the remaining problem was deciding where they should be.
The server became the source of truth and sent a lightweight master timecode to every user. Each client measured the difference between its local stream position and the target, buffered the required amount, and played from the matching point.
There was one important edge case: a client cannot seek into future live content. The solution was to deliberately move the server’s target behind real live time. With enough artificial delay, the shared target stayed inside content that every client could already have buffered.
The infrastructure trade-off
The server no longer had to redistribute the video. It only had to send synchronisation information, while users downloaded the stream directly from the original provider.
That changed the scaling behaviour of the system: adding viewers no longer meant that our infrastructure also had to send another copy of the video stream.
The new bottleneck: memory
Moving work to the client introduced a different cost. Raw decoded video frames are large. The original calculations showed that even a few seconds of buffered 720p frames could require several gigabytes of memory.
We compressed the VLC frame data with ZSTD. After R&D around compression level versus CPU cost, we settled on a practical middle ground. In one representative configuration, around eight seconds of buffered media used roughly 1.5 GB of RAM.
This was not free - no architecture is - but it moved the cost from recurring server bandwidth to a controlled client-side memory/CPU trade-off.
What I like about this solution
The interesting part was not the buffer itself. It was recognising that the technically “perfect” architecture was the wrong system once operational cost was included.
The final design came from changing the boundary of the problem: rather than owning video distribution, the platform owned only a shared notion of time. The video providers continued doing what they were already good at, and Unreal handled the part we actually needed to control.
Technologies
Unreal Engine · C++ · VLC · HLS / M3U8 · RTSP · ZSTD · multiplayer time synchronisation · media buffering