The WebCodecs Frame Limit Nobody Documents

Video playback in FlashFX froze a few seconds in. Not a stutter, not a dropped frame. It played, and then it stopped, and nothing in the console had anything to say about it.

I want to write this one down because I lost real time to it, and because almost nothing on the open web describes the actual failure mode.

What it looked like

Decode a video, hold the frames, draw them as the playhead moves. Standard. The decoder was configured correctly, the chunks were going in, and for a few seconds the frames came out.

Then the output callback simply stopped firing. No error. No dequeue event. No state change on the decoder. It sat there in configured, accepting chunks, emitting nothing.

What I assumed

Memory. Obviously memory. I had a byte budget of 512MB across decoded frames, and my first instinct was that the budget was wrong, or that something was retaining frames past their eviction.

So I went looking for a leak. There wasn’t one. The budget was being respected. At the moment playback froze, I was comfortably under it.

That is the part worth sitting with: every measurement I had said the system was healthy. The thing that was actually saturated was not being measured at all.

What it actually is

A hardware VideoDecoder has a pool of output frames. It is not documented as a number you can query, and in practice it is small — somewhere around 16 to 24 open frames, depending on the platform and the codec.

Once you are holding that many, the decoder stops emitting. It does not error. It waits for you to give something back.

And here is the part that makes it hard to see: a VideoFrame that has been transferred still holds its slot in the decoder pool until you call .close(). Transferring it to a worker does not release it. Drawing it does not release it. Dropping your last reference does not release it, because VideoFrame is not garbage-collected in a way you can rely on for this. Only .close() gives the slot back.

So my byte accounting was measuring the wrong resource entirely. Frames are small in bytes and expensive in slots, and I had built a budget in the unit that did not matter.

The fix

Two parts.

First, a cap on the count of open decoded frames per asset, not on their size. I set it at 12 — deliberately below the platform floor, because the real limit is not knowable and being wrong here means a silent freeze rather than a warning.

Second, an eviction order that matters. When you are over the cap you have to close something, and closing the wrong frame means decoding it again a moment later. The frames worth keeping are the ones nearest each layer’s playhead, so eviction works outward from there rather than by age. Age is the obvious heuristic and it is wrong on a timeline, where the user can be anywhere.

What I would tell myself

If a decoder stops emitting and nothing errors, stop looking for a leak. You are not out of memory, you are out of slots, and those are different resources with different units.

More generally: I had instrumentation, and the instrumentation said everything was fine. That is not evidence that everything was fine. It is evidence that I was measuring the thing I had thought to measure. The bug lived in the gap between what I had chosen to count and what the system actually rationed.

Byte accounting alone never touched it.