Historical evidence, superseded guidance. This is the earlier article, preserved with its original workload, measurements and revision links. Its defaults, branch/deployment labels and recommendations are not current. Read the current recipe.

AVPlumber · Mixer optimization cookbook

The compositor's wait costs the frames that arrived

Measured on 2026-10-13 on the 16 vCPU L4 host that runs the demo. Both commits are on develop since 2026-10-13: cc716b6f, the wait with epoll, and 13ea5848, the sleep on the input events. Source links are pinned to develop at 8fa15c92, which has both. Watch the current 192-input L4 demo.

The busiest thread of the 192-source show is the Program compositor. It sleeps until any source has a new frame, and it did that with poll() over one descriptor per source: 193 descriptors handed to the kernel, hooked, unhooked and scanned on each of about 2,200 wakes a second, to learn that two or three had fired. With epoll the list of sources is handed to the kernel once, when the compositor is built. After that each sleep is one call that carries no list, and the answer is only the sources that have a new frame. The thread fell from 52 to 35 % of a core; nothing about when it wakes or what it draws changed.

• 52 → 35 %of one core, Program compositor A, 192 sources at 1080p30
• 17.4 → 3.0points of that which are kernel time
• 81 → 45 %the 20 AUX compositors together, which wait the same way
• 43 → 32 %the same thread on one tree with and without the commit, on a second L4
• 84 → 51 %projected for that thread at 1080p50 with both commits, not measured
• 0missed deadlines in 47 minutes on the new build, after its start-up

Why this thread matters

More vCPUs help a show whose threads together need more than the machine has. They do not help one thread that needs more than a core. On the 192-source show at 1080p30 the Program compositor read 52 % of a core, and its cost is not linear:

• Each source frame wakes the thread, or shares a wake with one or two others: about 2,200 wakes a second for 5,760 source frames.
• Each wake does work for every source, not only for the ones that fired. So the per-wake part of the cost goes with source frames per second times the number of sources: with the square of the source count.
• Three shows fit that: 40 % at 192 sources and 25 fps, 50 % at 192 and 30 fps, 34 % at 99 sources and 60 fps. The last two carry almost the same number of source frames per second.

Carried to 1080p50 with 192 sources, the fit gave 84 % of a core for this one thread, and a full core at about 214 sources. Both Program buses run the loop, and so does every AUX bus.

What one wake did

1. One poll() over 193 descriptors. The kernel walks the list, hooks the thread onto each descriptor's wait queue, sleeps, and unhooks from each on the way out.
2. A pass over the results to read the descriptors that fired, and a pass over all 192 source edges to find one with data.
3. A second pass over all 192 edges to take the new frames into the playout buffer.
4. A third pass when the tick cannot be drawn yet, before sleeping again.

The Program compositor keeps every scene source subscribed, on air or not, so that a cut starts on the next frame. That is why its list is the whole show. An AUX compositor waits on the same 193 descriptors but subscribes only the 4 to 12 sources it shows, so it wakes about twenty times less often.

Event and wait, in plain terms

• An event is the signal: every source edge has a small counter in the kernel (an eventfd), and the source's thread writes to it each time it puts a frame on the edge.
• The wait is the call the compositor makes to sleep until any of those events fires. It is the waiting, not the thing waited for.
• With poll() the program keeps the list of 193 events and hands all of it to the kernel on every sleep. The kernel attaches the thread to each one, sleeps, detaches from each one, and forgets the list.
• With epoll the kernel keeps the list. A sleep is epoll_wait() with no list, and it returns with the events that fired.

It is the difference between reading the whole guest list to the doorman every time you ask whether anyone has arrived, and giving him the list once and asking who is new.

The change

MultiEventWait, the engine's wait for any of a set of events, now registers its events with epoll when it is built and calls epoll_wait() to sleep. Everything else is as it was:

• The same event descriptors, signalled by the same producers on every frame.
• Level-triggered, as poll() was: an event that is signalled and not yet read ends the next wait at once.
• One wait still reads every event that fired.
• An event listed twice, or one that has no descriptor, is accepted as before.

The class is shared: the Program and AUX compositors, the muxers and the clip cache all wait through it.

Before and after

The running 192-source show at 1080p30 (105 NVDEC, 40 browser, 47 raw NV12; 24 encoded outputs), one 20-second reading of every mixer thread before and one after the build was swapped in.

• Sampled every 50 ms, compositor A was asleep in poll() in 33 % of samples before and asleep in epoll_wait() in 43 % after.
• The pacing loops are the control: they do not use this class and did not move.
• The compositors together gave back about 0.7 of a core (195 to 124 points); the mixer as a whole read 52 points lower.
• The show then ran 47 minutes with no missed deadline and no dropped frame, apart from its start-up.

Isolated on one tree

The table above compares the trial image the demo host was running with a build of develop plus this commit, so it is not the commit alone. A second L4 host of the same shape ran the same 192-source show from three images built from one tree: develop, develop with the epoll commit, and with the next commit as well, which drops two of the compositor's three passes (see below). Two 20-second readings each; the two agreed within half a point, and the table has their mean.

The epoll commit alone takes 11 points off each Program compositor and 29 off the AUX compositors, all of it kernel time. The absolute figures differ from the demo host's, which was on another build and has viewers attached; the kernel time goes the same way on both.

The second step, and what it showed

After taking in every input the compositor called findSourceWithData() to wait. That looked at all 192 edges before sleeping and again after waking, for an index the compositor never used. The next commit makes it sleep on its events directly. A frame that arrives after the pass leaves its event signalled and ends the wait at once; an input that still holds frames beyond the pass's bound of eight skips the wait.

• AUX compositors: 44 to 32 % for all 20. They wake often and take in few frames per wake, so the passes were a large share of their work.
• Program compositors: 32.1 to 30.5 %. Much less than a model fitted to the first reading had predicted (it put about 19 points on the passes). A pass over 192 edges costs about 10 µs, not 30.
• Threads waiting for a core fell by a quarter (456 to 326 points), more than the CPU saved: shorter work per wake means the compositors are off the cores sooner at the moment every other per-frame thread wants one.

So what is left of the Program compositor, about 28 points of user time, is not the passes. It works out at about 48 µs for each of the 5,760 source frames a second, and what that time is spent on was not measured here. A later profile found 26 % of the thread in looking at all 192 input queues on every wake; see below. Until then the projection for 1080p50 with 192 sources is the measured figure scaled by the frame rate: about 51 % of a core, down from 84.

Two things that were open have since been addressed:

• Every prewarmed source's frame woke the Program compositor. On a branch since 14 October, canvas.input_poll_ms lets it sleep to its deadline and look for frames at most every few milliseconds, and at once for a cut: at 8 ms each Program compositor fell from 12.1 to 7.3 % and its wakes from 3,400 to 230 a second (Input, decode and wake-ups).
• The pacing loops' 26 % each at 30 fps was a separate fault and is fixed. realtime counted in 1/120000 s against a clock that reads whole milliseconds, and asked the millisecond timer again and again for a release time already inside the current millisecond. Such a time now counts as due, and each loop reads about 8 % (Event-loop sharding). It did not occur at 25 or 50 fps and was not part of this change.

Since this measurement

• The start is clean. The 47 minutes above leave out the start-up. The mixer now holds its outputs until the buses have settled and zeroes the compositors' counters when it goes on air: a start shows 0 missed deadlines and 0 dropped frames from the first frame on air.
• The totals in both tables are those of the builds they name. Three later changes on develop moved them, on the same show and host on 2026-10-13: realtime stopped spinning on a time base finer than the clock (each pacing loop 26–30 % to about 8 %, the whole mixer 791 to 715 %; Event-loop sharding), colour tags moved to the fan-out (1,472 to 1,327 threads; Colour tags at the fan-out), and raw files are read straight into page-locked memory (679 to 510 % of a core, 1,327 to 1,186 threads; Raw frames read straight into the memory the GPU reads). On the build with all three the mixer reads 510 % of a core in 1,186 threads, and each Program compositor about 30 %.
• On branches since 14 October the wait also says which events fired, so a pass takes frames only from those sources, and the compositors look for frames at an interval. With the input changes of the same night the two Program compositors together went from 59.5 to 14.6 % and the mixer from 502 to about 234 % (Input, decode and wake-ups).

How it was measured

From /proc only, with nothing installed on the host and nothing CPU-bound beside the show:

• Per thread, over 20 seconds: run time and time runnable without a core from schedstat, user and kernel time from stat, voluntary and involuntary switches from status.
• For the named threads, every 50 ms: the state from stat and the system call from syscall (7 is poll, 232 is epoll_wait, 202 is futex).
• The show's own counters before and after the reading, to see that the reading cost nothing.

The user and kernel split is what made the first step obvious: a third of the thread's time was in the kernel, and the only thing it asks the kernel for at that rate is the wait.

At a glance

• WhatThe engine's wait for any of a set of events registers them once with epoll, so a wake costs the events that fired, not the size of the set. A second commit lets the compositor sleep on those events without two extra passes over its inputs.
• WhyA compositor with 192 sources paid for 193 descriptors about 2,200 times a second. That was a third of the busiest thread's CPU, and the part of it that grows with the square of the source count.
• Latency0 frames added. The thread wakes for the same events at the same moments.
• Side effectsNone seen. Every node that waits on several inputs now uses epoll; a producer's signal does a little more work in the kernel when the consumer is awake, because the registration is permanent.
• CostOne epoll descriptor per waiting node. About thirty lines changed in one header and a unit test; the second commit is a dozen lines in the compositor.
• RiskLow. The class's behaviour is pinned by a test: timeouts, every fired event read by one wait, a wake from another thread, an event listed twice. Linux only, as the eventfd-based events already were.
• Switch it offNo switch. Revert the commits; they are independent of each other.

Related pages

• Browser frames go back one per new frame: the stall that had to go first, and the per-tick timing in the mixer status.
• Browser frame ticks staggered at 60 fps: threads that wake together, and how run-queue pressure is read.
• Event-loop sharding: the pacing loops.
• Prewarmed DSK keys: why sources stay subscribed while off air.

Where in the code

• src/MultiEventWait.hpp: the wait, with epoll_create1, one EPOLL_CTL_ADD per event and epoll_wait.
• src/Event.hpp: an event is an eventfd; a producer signals it on every frame it puts on an edge.
• src/graph_base.hpp: a node with several inputs builds its wait from one event per source edge; findSourceWithData() and waitForInput() wait on it.
• src/nodes/mixer_compositor.cpp: the compositor's pass over its inputs and its wait until the next deadline.
• tests/cpp/test_multi_event_wait.cpp: what the wait must keep doing.

Source links are pinned to develop at 8fa15c92, which has both commits.