How VLC plays it?

You drag movie.mp4 onto the little orange traffic cone in your dock. A window opens. A frame appears. Sound starts. Everything is running in sync less than a second later.

That single second is doing an enormous amount of work. The file on disk is not a movie in any sense your eyes or ears can use — it is a compressed, packaged, interleaved stream of bytes that has to be unpacked, decompressed, and pushed to two completely different pieces of hardware (your screen and your speakers) at exactly the right instants so that a character's lips move when they speak. If VLC gets the timing wrong by even a few dozen milliseconds, you will notice.

This is the walk — from the moment you release the mouse to the moment you see the first frame — explained for someone who has never opened a video codec spec in their life.

1. What is even in an MP4?

Before anything else, it helps to know what VLC is actually looking at. An MP4 file is not a video. It is a container — a shipping crate with several kinds of cargo inside. Somewhere in that crate there is a compressed video track, a compressed audio track, maybe a subtitle track, and a bunch of labels that explain how it all fits together: how many frames per second, what language the audio is in, when the movie is supposed to start and end.

The reason for this crate-and-cargo design is that video and audio are compressed differently, by completely different algorithms, but they need to travel together and stay in sync. So the container's job is to package them side by side, timestamp everything, and let the player pull the pieces apart when it needs to. MKV, MOV, AVI, and WebM all work the same way — different crates, same idea.

The compressed video track is not made of pictures. It is made of instructions like "keep the previous frame but shift this block of pixels three to the right." That is what H.264 (and its cousins HEVC, VP9, AV1) actually store: a set of tiny instructions that, when replayed in order, reconstruct one frame at a time from the one before it. A two-hour movie that would be about a terabyte of raw pixels ends up being a few gigabytes because most frames are only small deltas on top of the one before.

Same story for audio, using a different algorithm. AAC or MP3 or Opus compresses raw sound waves down to about 10% of their original size by throwing away frequencies your ear cannot hear anyway.

2. The probe

The first thing VLC does when you open movie.mp4 is refuse to trust the filename. Extensions lie. Someone can rename a WebM file to .mp4 and it is still a WebM file. So VLC opens the file, reads the first few hundred bytes, and inspects them against a set of known magic numbers — short byte sequences that different container formats stamp near the start of every file to identify themselves.

Once the probe is confident about the format, VLC loads the piece of code that knows how to open that specific crate. This piece of code is called a demuxer — short for demultiplexer, which is engineering jargon for "the thing that pulls the mixed-together streams apart." There is an MP4 demuxer, an MKV demuxer, a WebM demuxer, and a few dozen others. They all follow the same contract: give me raw bytes, and I will give you back separate video, audio, and subtitle streams, each one with timestamps attached.

This design is the reason VLC can play almost any file you throw at it. Adding support for a new container is a matter of writing one new demuxer plugin. The rest of the machine does not need to change.

3. Opening the crate

The MP4 demuxer's job is to walk through the file and pull out the tracks. In practice this means reading the box headers scattered through the file (MP4 is internally a tree of "boxes" called atoms), finding the video track, finding the audio track, and figuring out where each chunk of each track lives on disk.

Once the demuxer knows the layout, it can start streaming. It reads a chunk of compressed video, hands it off to whoever wants to decode video. It reads a chunk of compressed audio, hands it off to whoever wants to decode audio. Each chunk comes with a presentation timestamp attached — a number that says "this chunk of audio belongs at exactly 4.312 seconds into the movie." That timestamp is going to matter later.

Notice what the demuxer is not doing. It is not turning anything into pictures or sound. Those compressed packets it hands out are still gibberish to anything except a decoder. The demuxer is a mailroom — it sorts the envelopes and stamps the delivery time on each one. The actual opening of the envelope happens next.

4. The unzip

A compressed video packet coming out of an MP4 is closer to a recipe than a picture. It might say "here is a new full frame" (called a keyframe), or, more often, "here are some small changes to apply to the previous frame." A decoder is the piece of code that follows those recipes and produces a raw grid of pixels.

H.264 is the most common video codec inside MP4 files, so let us follow it. VLC hands each compressed H.264 packet to its H.264 decoder. The decoder maintains a small memory of recent frames, applies the packet's instructions to the most relevant one, and out pops a full frame of raw pixel data — typically about 6 megabytes of pixels for a 1080p frame, at 24 or 30 or 60 of those every second.

This is the most CPU-intensive part of playing a video by a comfortable margin, which is why almost every modern computer has dedicated hardware for it. VLC will happily use that hardware if it can. On a Mac it talks to VideoToolbox, on Windows to DXVA, on Linux to VA-API. When hardware decoding is available, the CPU barely does any work — the whole thing is offloaded to a chip in your GPU designed specifically for unzipping H.264. That is why your laptop does not melt when you watch a 4K movie.

Audio decoding is the same shape, less dramatic. The AAC decoder takes each compressed audio packet and reconstructs the waveform — a stream of numbers that describe how the speaker cone should move, thousands of times per second. Every stereo second of AAC audio at CD quality is about 350 kilobytes of raw samples; on disk it was maybe 32.

At this point VLC is holding two completely different kinds of decoded data in memory: a queue of pixel frames waiting to be shown, and a queue of audio samples waiting to be heard. Neither has hit your screen or speakers yet. The next problem is getting them there at the right moment.

5. Two pipelines, one movie

From this point onward, video and audio go on completely separate journeys through completely different hardware, and they will never touch each other again until you sense them at the same instant.

The video frames travel through what VLC calls the video output module. Its job is to take each raw frame and push it onto your screen — usually through OpenGL, Metal, or Direct3D, depending on your operating system. Modern GPUs are extremely good at this: you hand them a frame, they upload it to video memory, and they draw it during the next screen refresh. A 60-hertz display refreshes every 16.7 milliseconds, so the video output module's window to hit each refresh is small but predictable.

The audio samples travel through the audio output module. Its job is to push samples into whatever the operating system's audio system is — CoreAudio on macOS, WASAPI on Windows, PulseAudio or PipeWire on Linux. That audio system in turn feeds a small hardware buffer on your sound card, which the sound card drains at exactly the sample rate (typically 44.1 or 48 kilohertz). If that buffer ever runs dry, you hear a pop.

Each of these pipelines runs on its own thread. The video output thread wakes up whenever the screen is about to refresh and asks for the next frame. The audio output thread wakes up whenever the sound card's buffer is getting low and asks for more samples. If those two threads ran independently, they would drift out of sync within seconds — and that is where the last, most delicate part of the machine comes in.

6. The conductor

Keeping video and audio in step across two separate pipelines is called A/V sync, and it is the single hardest problem in playback. Humans are shockingly sensitive to it. Audio arriving 40 milliseconds ahead of the matching frame is enough for a viewer to feel that something is off, even if they cannot say what.

VLC's solution is to appoint a master clock and make everything else follow it. In almost every case, the master is the audio clock — because the ear is more forgiving of a dropped or duplicated video frame than the ear is of a stuttering audio sample. Audio, in other words, gets to play whenever it wants; video has to catch up.

Every frame comes out of the decoder with a presentation timestamp — the same timestamp the demuxer stamped on the original packet, all the way back at step 3. When the video output is about to draw a frame, it compares that timestamp to what the audio clock says the current time is. If the frame is slightly ahead of the clock, video output waits a few milliseconds. If it is slightly behind, video output shows it immediately. If it is way behind — maybe the decoder is falling behind on a heavy scene — video output drops the frame and grabs the next one to catch up.

This is why, on an underpowered machine, a struggling video plays with visible skips but the audio still sounds fine. The audio never yields. The video is the one making sacrifices.

7. Everything is a plugin

Something worth pausing on: every single one of the pieces we just walked through — the demuxer, the decoder, the video output, the audio output — is a plugin. VLC does not have a big built-in list of "here is how to play MP4." It has an empty plugin socket for demuxers, and at startup it scans a folder full of shared libraries and asks each one, "what can you do?"

The MP4 demuxer plugin says, "I can open MP4 files." The MKV plugin says, "I can open MKV files." The H.264 decoder plugin says, "I can decode H.264 video." The VideoToolbox plugin says, "I can decode H.264 video, and I can do it on the GPU." The CoreAudio plugin says, "I can push audio to Mac speakers." VLC keeps a registry of who does what, and when it needs a capability, it goes shopping.

This is why VLC plays every strange video file your uncle sends you from a 2007 camcorder. Someone, somewhere, has written a plugin for that format. Drop the plugin in the folder, and VLC picks it up on next start. The core engine never changes.

It also explains why VLC is used as the playback engine inside hundreds of other apps. The whole thing is exposed as a library called libVLC, and any application can embed it, tell it "play this URL," and let VLC handle the entire mess we just walked through.

8. The whole journey, one diagram

Here is the full trip from double-click to first frame, everything talking at once:

Four processes and a chain of plugins doing the work. The file gets probed, the crate gets opened, the streams get pulled apart, each one gets decoded on its own thread, and two independent output modules push pixels and samples to two independent pieces of hardware — while a shared clock keeps them from drifting apart. When the video output writes the first frame to your screen, the audio output is already a few milliseconds into the first sound, and both of them agree, to within a millisecond, that the movie is currently at time zero.

9. Why this matters now

You can play videos without VLC. Your operating system ships a player. Every browser has one built in. YouTube exists. What VLC gives you that none of those do is a design that plays every video, because every piece of the machine is a plugin and the community has written a plugin for every format anyone has ever cared about — going back three decades.

It is also worth knowing, the next time a multimodal AI model asks you to upload a video for it to analyze, or your image-generation pipeline preprocesses a training clip with ffmpeg, that the machinery under the hood is exactly what we just walked through. A container gets probed, a demuxer pulls out the tracks, a decoder turns compressed packets into raw frames, and a clock keeps everything in step. ffmpeg is a close cousin of libVLC; the difference is that ffmpeg saves the decoded output to a file instead of drawing it on your screen. The core pipeline is the same idea.

If you want to see the rest of what VLC does — the adaptive streaming for HLS and DASH, the way it handles subtitles across a dozen formats, the Rust rewrite of its core engine, the memory-safety work in vlcrs-core, the security sandbox around network streams — the full interactive breakdown of the VLC codebase is on Revibe.