Historical evidence, superseded guidance. This is the earlier article, preserved with its original workload, measurements and revision links. Its defaults, branch/deployment labels and recommendations are not current. Read the current recipe.

AVPlumber · Mixer optimization cookbook

Browser frames go back one per new frame

Implementation links checked 2026-10-12 against develop at b708e59f. Measured on 2026-10-12 on the 16 vCPU L4 host that runs the demo. Watch the current 192-input L4 demo.

A browser window's frames reach the mixer as DMA-BUFs, and the mixer tells the browser when it has finished with each one. It used to say so the moment a frame was let go, several at a time. Chromium's buffer pool then destroyed and allocated buffers all the time, every new buffer was a new CUDA import and a released one, and each release held a lock inside the CUDA library that every other GPU call waits for. Now one frame goes back for each frame that arrives. The pool settles on a fixed set of buffers, and the 192-source show that ran at 1080p25 runs at 1080p30.

• 28 / 21 → 0 / 0Program deadlines missed on bus A / B in ten minutes at 1080p30, 192 sources
• 97 → 0AUX deadlines missed over 22 buses in the same ten minutes
• 462 → 0draws that took longer than the 33 ms slot; the longest was 198 ms
• ≈10 a second → 0new browser buffers imported into CUDA
• 192 at 25 and at 30 fpssources on one L4; the 30 fps limit was 170

The symptom

The show: 192 sources at 1080p, SDR, 256 scenes, 24 encoded outputs. At 30 fps that is 105 NVDEC decodes, 40 browser windows and 47 raw NV12 uploads. It ran clean at 25 fps and missed deadlines at 30, with room left everywhere a gauge looks: CUDA at 31 to 68 %, no thread above 45 % of a core, VRAM at 19.5 of 23 GB.

The show did not degrade smoothly. It swung between a clean state and a collapsed one on its own, and in the collapsed state each Program bus missed about 140 deadlines a minute. One short clean run therefore proved nothing; the measurements below hold in either state.

The cause

1. The buffers are Chromium's, and it keeps two idle. A window's frames come out of Chromium's capture pool. The pool keeps at most two idle buffers and destroys the rest. The browser service's ringSize is not a ring of buffers: it only limits how many frames may be out of the browser at once.
2. The mixer sent frames back in bunches. A frame went back when its last user let it go: the playout buffer, a compositor, a monitor. That released one to six frames of a window together. Chromium found more than two idle, destroyed the surplus, and allocated new buffers for the frames after.
3. Every new buffer is a new import. A new buffer is a new DMA-BUF. The window's drm_prime_to_cuda node registers it with CUDA (cuGraphicsEGLRegisterImage) and later drops the import of a destroyed one (cuGraphicsUnregisterResource) on its release thread. At 30 fps the 40 windows together imported about ten new buffers a second.
4. The release holds a lock the whole context needs. cuGraphicsUnregisterResource takes a reader-writer lock inside the user-space CUDA library for writing and can sleep while it holds it. Every other call into the context queues behind it: the uploads, the compositors' draws, the encoders' inputs. At 30 fps this tipped into a state where that one thread held the lock most of the time.

Raw frames read straight into the memory the GPU reads has an earlier show whose traces showed upload, import and compositor threads waiting on one another inside the driver; uploads wait on this lock too, so it is very likely the same lock, though that show was not measured again. Here the holder was read from the lock itself; see How the holder was found.

The fix

The mixer still lets a frame go as soon as nothing uses it. What changed is when the browser hears about it. A finished frame waits in a return queue, and each frame that arrives sends the oldest waiting one back. The pool sees one buffer come back for each one it hands out, however unevenly the mixer finishes with frames, so it never has more than two idle and never runs out.

• In the engine. ipc_dmabuf_source has a parameter return_queue, 0 to 64 frames, default 0. With 0 every release is acknowledged at once, as before. With N, a finished frame waits; one goes back per arriving frame, and only beyond N waiting does the oldest go at once. frame_stats reports the waiting ones as returns_queued.
• In the mixer. A show's browser_frame_queue (1 to 9, default 9) is how many frames a window may have out. Three of the nine are the return queue; a shorter queue keeps three for the mixer first, so a queue of three or fewer has none. The earlier key browser_ring_size is still read, and a value above 9 means 9. Setup has no control for it and uses the default.
• At every rate. 25, 30, 50 and 60 fps all use six in flight and three in the return queue.
• On reconnect or stop. When a connection is retired, everything still waiting is sent back before the drain marker, so the browser gets every buffer back.
• In the browser service. Nothing changed. The window is opened with ringSize 9, and frames in the return queue count against it like any frame not yet returned.

Before and after at 1080p30

The same show on the same host, ten minutes each, start-up excluded.

The show then ran 42 minutes with cuts being made: no deadline missed on Program A, on Program B or on any of the 22 AUX buses (1.65 million AUX frames), and no frame dropped. A draw takes about 3 ms of its 33 ms. The before column was not counted for dropped paints.

With that, the nvidia_l4_cuarray profile admits 192 sources at 30 fps as it does at 25; the 30 fps limit was 170. See Source limits by mode and frame rate.

At 1080p60

50 and 60 fps used to keep all nine frames for the mixer and had no return queue. The 60 fps show: 99 sources (40 NVDEC, 40 browser, 19 raw NV12) and 16 encoded outputs, eight minutes each.

At 60 fps the return queue removes the slow draws and about two thirds of the new buffers, not all of them as at 30 fps: the time the mixer holds a frame varies by more frames than three waiting returns even out. Neither run missed a deadline, so here it buys margin. The importer logs a window's first two new buffers and then every 64th, so the line counts compare the two runs and are not buffer counts.

What it was not

Each of these was a candidate and was measured away on the same show.

• NVENC or the number of outputs. With 4 encodes instead of 24, Program missed more deadlines, not fewer.
• GPU compute. CUDA stayed between 31 and 68 %.
• NVDEC. 105 decodes missed like 120 did, and scenes with no decoded source still missed.
• One saturated thread. None ran above 45 % of a core.
• The device's CUDA stream. FFmpeg leaves a CUDA device on the legacy default stream, and work queued there takes the same lock for writing. A trial build gave the device its own stream: the uploaders' queue for the lock went away as predicted, and the collapse did not. That build was dropped; the stream layout is unchanged.
• Thread priority. A higher priority (nice −10) for the threads that must meet the deadline made the AUX buses worse.
• A longer deadline. Not tried. It would have added latency to hide a stall, not removed the stall.

How the holder was found

Everything below reads /proc or runs in a throwaway container; nothing was installed on the host.

1. Waiting, not working. Per-thread schedstat gives the time each thread ran and the time it was runnable without a core. Per-thread state and system call (/proc/<tid>/stat and syscall) showed the Program compositors asleep in a futex 65 % of the time at 30 fps and 0 % at 25.
2. Which lock. The futex address is the second argument in /proc/<tid>/syscall. Threads waited on two addresses four bytes apart, which is one glibc pthread_rwlock_t: its writer-phase and writer futex words. The owner word of the same structure names the thread that holds it for writing. That is a thread id in the container's PID namespace; NSpid in /proc/<tid>/status maps it to the host's.
3. Who holds it. The holder was the same thread in every sample. One gdb attach from a container sharing the mixer's PID namespace gave its stack: cuGraphicsUnregisterResource under the importer's release. An attach stops the mixer for about 17 seconds, so never on a show that matters.
4. Where a tick's time goes. mixer.status now carries playout.timing per bus: histograms of the whole draw, of the part that queues it into CUDA, of the wait for the GPU and of the hand-off downstream, in quarters of the frame period. Every slow draw was slow in the part that queues the work; none was slow waiting for the GPU.

Do not measure with anything CPU-bound beside the show. One single-core test container on the same host cost the 30 fps show 87 deadlines in a minute.

Two changes that came with it

• Imports stay ten idle seconds. The mixer told each importer to drop a buffer it had not seen for four frame-queue lengths, under a second at 30 fps. A mostly still page paints rarely, so its buffers expired and were imported again. import_ttl_ms is now at least 10 s. It did not stop the stall by itself. Since 2026-10-13 the mixer also asks each importer, before it goes on air, to release the buffers the start-up went through and left idle for 500 ms (purge_idle): the start-up peak of GPU memory fell from 22.4 to 19.6 GB, and those releases no longer stall draws once the outputs are on air.
• Per-tick timing. playout.timing, as above. Read it after any deployment: it shows how many draws fitted their slot without waiting for a deadline to be missed.

At a glance

• WhatThe mixer acknowledges a browser frame when the next one arrives, one for one, from a return queue of three. A window may have nine frames out: six in use, three waiting to go back.
• WhyChromium's pool keeps two idle buffers. Frames that come back in bunches make it destroy and allocate buffers; each new one is a CUDA import and a release, and the release blocks every GPU call in the process. At 30 fps and 192 sources that stalled both Program buses.
• Latency0 frames added. The return queue delays only the message that hands a finished buffer back; the frame's way to the compositor is unchanged.
• Side effectsA window keeps up to three finished buffers out of the browser. That is why the frame queue is nine and not six: with six, the return queue would take half of what the mixer's own pipeline needs. Not measured as memory by itself; the show read 19.5 of 23 GB with it.
• CostOne small queue per connection, under the mutex the acknowledgements already had. No thread, no timer, no change in the browser service or its image.
• RiskLow at 25 and 30 fps, where the 192-source show ran 42 minutes clean. At 60 fps some buffers are still replaced. A page that stops painting leaves up to three returns waiting until its next frame or until the connection is retired, which the queue of nine has room for.
• Switch it offNo runtime switch. On a hand-built graph leave return_queue at 0. In a show, a browser_frame_queue of 3 or fewer has no return queue; do not do that on a large show.

Related pages

• Browser frame hand-off: the socket, the acknowledgements and what happens on reconnect.
• The compositor's wait costs the frames that arrived: the busiest thread once the stall was gone; its wait now uses epoll.
• An earlier proposal: shorter decoded frame queues: a different queue, the decoder's, and a proposal that was not selected.
• Unused GPU contexts: the importer that registers each browser buffer.
• Chromium paints into the buffer the mixer reads: a patched browser whose own buffer pool, reused last in first out, brought a stall of the same kind back, and the pool that never frees.

Where in the code

• src/nodes/hwaccel/DmabufReleaseQueue.hpp: enqueue() puts a finished frame in the return queue, frameArrived() sends the oldest back, retire() sends what still waits.
• src/nodes/hwaccel/ipc_dmabuf_source.cpp: the receiver calls frameArrived() for each frame; 437-439: the return_queue parameter; returns_queued in frame_stats.
• pyplumber/mixer/dmabuf_inputs.py: RETURN_QUEUE, MIN_FRAMES_IN_FLIGHT, return_queue() and IMPORT_IDLE_MS; the same file opens each window with ringSize and sets import_ttl_ms.
• pyplumber/mixer/config.py: BROWSER_FRAME_QUEUE_MAX, the default, and the earlier key.
• src/nodes/hwaccel/drm_prime_to_cuda.cpp: the release that holds the lock; the node's import_ttl_ms.
• src/mixer/primitives/TickTimes.hpp: the histogram behind playout.timing.
• demos/mixer/instance_profiles.py: 192 sources at 30 fps on nvidia_l4_cuarray.
• tests/cpp/test_dmabuf_release.cpp: one acknowledgement per arrival, the oldest at once beyond the queue, everything on retire. tests/mixer/test_graph.py: the graph gives each browser source its return queue.

Markdown references: NODES.md · ipc_dmabuf_source documents return_queue; the browser service's README says what ringSize limits; capacity.md has the limits.

Source links are pinned to the reviewed revision.