Historical evidence, superseded guidance. This is the earlier article, preserved with its original workload, measurements and revision links. Its defaults, branch/deployment labels and recommendations are not current. Read the current recipe.

AVPlumber · Mixer optimization cookbook

Input, decode and wake-ups: 502 to 234 % of a core

Measured on 13 and 14 October 2026 on the 16 vCPU NVIDIA L4 host that runs the demo, one build at a time. The evening builds ran the full show on a second L4 host first; the later ones passed the unit tests and smoke tests, with light GPU smokes on the demo host, before they went into the show. One change, the HEVC table clears, is on develop since 2026-10-13. The others are on avplumber branches that are not merged to develop yet and run in the demo show; each section names its branch, and source links are pinned to the branch commits. Watch the current 192-input L4 demo.

On the 192-source show at 1080p30 the mixer process read 502 % of one vCPU in 1,186 threads after the raw-upload, colour-tag and compositor changes of 13 October. Ten more changes, aimed at what each source's frames cost on their way from the file to the compositors, took it to about 234 % in 889 threads, with no missed deadline and no dropped output at any step. The whole stack, browser included, then ran fifteen minutes confined to 8 vCPUs without missing a deadline, about half of them busy.

• 502 → 234 %of one vCPU, the whole mixer process, 192 sources at 1080p30 with 24 encodes
• 1,186 → 889mixer threads: no demux thread per file and no fan-out thread per source
• 139.7 → 27.3 %the 152 reading threads, once raw files stay in page-locked memory
• 59.5 → 14.6 %the two Program compositors together (7.3 each), once the busiest threads of the show
• 108 → 28 %time the mixer's threads spent runnable without a core, over the evening's builds
• 0missed deadlines and output drops in every reading, and in fifteen minutes on 8 vCPUs

Every step

The running show: 192 sources at 1080p30 (105 HEVC files through NVDEC, 40 browser windows, 47 raw NV12 files), 256 scenes and 24 encoded outputs. Every figure is a 20-second reading of the mixer process from /proc, in % of one vCPU, three to four minutes after that build started. Every row: 0 missed deadlines and 0 output drops over its reading.

“–”: not read for that build. Run-queue time was read for the evening's builds only. The zero-copy browser trial ran between the last two rows; the mixer read 238–243 with it.

Where it came from, by thread

The evening's builds, before the first and after the input hops (% of one vCPU, all threads of a group together):

With the polling builds that followed, each Program compositor fell again, from 12.1 to 7.3 %: the two together read 14.6 %, against 59.5 that evening and about 98 earlier on 13 October, before the wait used epoll. At the end of the evening the whole host read about 3.0 vCPUs for the mixer, 1.7 for the browser container and 0.2 for the WebRTC gateway.

Each change, in the order it went into the show:

HEVC frames skip the software decode tables · on develop

A profile of the decoder threads found a quarter of their time in five memset calls in FFmpeg's hevc_frame_start: about 650 KB of per-picture tables cleared for every frame, about 66 of the 260 µs a frame cost. Only FFmpeg's software decode and its loop filters read those tables, and with NVDEC doing the decode neither runs. FFmpeg patch 0020 skips the clears when a hardware decoder is in use; software decoding is unchanged. The decoder threads fell from 86 to 62 %, the mixer from 502 to 471.

Raw files resident in page-locked memory · mixer/raw-preload

With canvas.raw_upload: "direct" (raw frames read straight into page-locked memory) each of the 1,410 raw pictures a second was still copied once by the kernel, out of the page cache. The resident mode keeps a looped file's pictures in one block of page-locked memory the size of the file: each picture is read into its place the first time it is shown, and on every later pass only the transfer to the GPU remains. Where a pass starts the node compares the file's modification time, and reads the file again if it has changed. The reading threads fell from 139.7 to 27.3 %, the mixer from 471 to 335, and the time threads spent runnable without a core from 85 to 34.

• Why not the page cache itself. On this driver cuMemHostRegister refuses a read-only or shared file mapping, so the page cache cannot be what the GPU reads from. A private writable mapping registers, at the price of a copy of the file; the node does the same thing openly.
• The cost. Host memory equal to the files, and it cannot be swapped out: 8.8 GB for the show's 47 two-second clips. With resident sources the mixer process holds about 10 GB of host memory.
• Is it still a real input? Every picture of every source is still transferred to the GPU for every showing, on the source's own clock, so the GPU, the bus and the compositors carry the full load of 47 uncompressed 1080p30 inputs. What is skipped is the copy out of a file, which a capture card that writes into page-locked memory does not make either: for an uncompressed live input this is the honest shape. Keeping frames on the GPU and replaying them stays rejected.

The compositor takes frames only from inputs that signalled · mixer/compositor-signalled-inputs

A Program compositor woke about 2,700 times a second and on every wake looked at all 192 input queues for new frames: 26 % of its time was in the queue's peek. The engine's wait for several events, which already used epoll, now says which events fired, and a pass takes frames only from those sources. The two Program compositors together fell from 53 to 39 %.

Input hops · mixer/input-hops

Three changes in one build, measured together (322 → 265 %, 1,186 → 1,081 threads):

• No metadata strings. input_rec stamped eight timestamp keys as formatted strings on every decoded frame, and every hand-off after it copied that dictionary. Nothing in the mixer's path reads them; frame_metadata now selects the keys, and the mixer asks for none. Decoder threads 57 → 49 %.
• No demux hop for one-stream files. With routing, input_rec sends the video stream to the decoder's edge itself, and the per-file demux node and its thread are gone: 105 threads. canvas.input_routing is input by default; demux restores the earlier wiring. The node-group ordering rule knew only demux as a routing node, which the AUX smoke test found.
• One signal per hand-off. Putting a frame on an edge signalled the other side twice; the second write woke nobody.

Compositors look for frames at an interval · mixer/compositor-poll, mixer/aux-input-poll

A compositor's playout only ever draws at a tick's deadline, so waking for every arriving frame only moves frames from one queue to another earlier than needed. With canvas.input_poll_ms a Program compositor sleeps to its next deadline and looks for frames at most that often in between, and at once for a cut. At 8 ms each Program compositor fell from 12.1 to 7.3 %, and its wakes from 3,400 to 230 a second. canvas.aux_input_poll_ms does the same for the AUX buses: at 8 ms the 20 AUX compositors fell from 20.6 to 17.6 % (264 to 184 wakes a second each) and the event loops from 31.4 to 28.6 %, about 4 points in all. A tick is drawn at its deadline either way; a source's queue of three frames must hold what arrives within one interval, and 8 ms suits rates up to 60 fps.

Fan-out on the pacing loops · mixer/fan-out-on-loop

Every source had a one_to_many node and thread that woke once per frame to hand the frame to its compositors and AUX buses, 47 % of it in the kernel. With canvas.source_fan_out: "event_loop" the fan-out (one_to_many_nb) runs on the event loop that already paces the source: 192 threads fewer (1,081 → 889). It saved no CPU: the 20 points moved onto the event loops (24 → 44 %), which then cost about 25 µs per handler because the loop rebuilt and polled its whole descriptor list on every pass. The first try failed to start because the setup's readiness check knew only the node type one_to_many; fixed.

The event loop · mixer/event-loop-self-wake, mixer/coalesced-signals

• No self wake-ups. A handler that registered its next wait from inside the loop woke the loop again; now a wait, work or a timer added from the loop's own thread does not wake it.
• The kernel holds the waits. The loop now keeps its waits in epoll with one-shot registrations instead of rebuilding a poll() list every pass. In a load test of 200 sources a handler cost 16 µs against 28, and the test is clean under the thread sanitizer. Its first two starts on the show failed, because every start of the show then ran the GPU out of memory and this time a raw reader took the failed allocation (GPU memory has the fix). It runs in the builds the show has used since; its share was not read on its own.
• Coalesced signals. Every push and pop on an edge wrote its eventfd whether or not anyone waited. Now a write happens only if the last one was read; each reader (a single wait, the multi-event wait, the event loop) clears the flag after its read and checks its condition again. The Program compositors' writes fell from 5,700 to 61 a second each, the event loops' from 33,274 to 15,051, and with the AUX interval to 10,433. That is about 2 points of CPU, within the noise of the total. A unit test with a mutation check covers lost wake-ups, re-arming by each reader and reused descriptor numbers, clean under the thread sanitizer.

What did not help, and what is left

• NVENC delay: 1 instead of 0. No change at 1080p30: 1.6 to 2.05 % per encoder thread either way. The encoder threads' 600 wakes a second are NVIDIA's own wait inside nvEncLockBitstream. Left at 0.
• The decoder threads. A profile of all 106: C library 23 % (two copies of the bitstream, about 8 %, and allocation), FFmpeg 19 %, NVIDIA's two libraries 28 % (including string formatting inside the CUDA driver library), kernel 14 % (context switches). Nothing large that is ours to change.
• The network senders, about 20 points: one sendto per RTP packet. Batching or UDP segmentation would be an FFmpeg patch.
• The CUDA event handler, about 10 points: woken about 3,600 times a second, 1,410 of them by the raw readers' stream synchronisation.

On 8 vCPUs

The stack's containers confined with a cpuset to fewer vCPUs (whole cores, both hyper-threads of each), after the evening's changes; the confinement reverts at the first missed deadline or drop.

• The last row: mixer, browser and Janus together used about 4.2 vCPUs of the 8 (mixer about 2.3, browser about 1.5), so 4 vCPUs cannot hold this show.
• The same 8 vCPU confinement on 13 October, before these changes, gave 2 output drops and 1,152 % of run-queue time.
• What the confinement does not show: a real 8 vCPU machine also confines the kernel's and the driver's threads, and has 32 GB of memory (the mixer holds about 10 GB with resident sources, the browser 1.9).

The settings the show runs

"canvas": {
  "raw_upload": "resident",
  "input_poll_ms": 8,
  "aux_input_poll_ms": 8,
  "source_fan_out": "event_loop"
}

input_routing is input by default on the branch, and the metadata and signal changes have no setting. Setup builds shows with raw_upload: "direct" and leaves the others unset, which is the earlier behaviour.

At a glance

• WhatRaw pictures kept in page-locked memory, one decoder patch, fewer hops and strings per frame, compositors that take only what signalled and look at an interval, fan-outs on the pacing loops, an epoll event loop, and queue signals written only when someone will read them.
• WhyAt 192 sources the mixer's CPU went on per-frame plumbing: copies out of the page cache, cleared tables, formatted strings, thread hops and wake-ups. None of it touched pixels the viewer sees.
• Latency0 frames added. Ticks are drawn at the same deadlines; with an interval a frame is taken in up to 8 ms later, still before its tick, and a cut looks at once.
• Side effectsResident raw sources hold their files in unswappable host memory (8.8 GB for 47). The fan-out on the loops moves CPU onto them rather than saving it. An interval needs a source queue that covers it.
• CostNo new nodes or queues; two fewer kinds of thread (demux and fan-out). Four new canvas settings and a raw_upload mode, all off by default except input_routing.
• RiskEight branches not yet merged. The evening builds ran the full show on a second L4 host before the demo host; the later ones went in after the Python suite, the C++ unit tests, CPU smokes and light GPU smokes. Measured on one show at 1080p30; 25, 50 and 60 fps not re-measured.
• Switch it offraw_upload: "direct", input_routing: "demux", input_poll_ms and aux_input_poll_ms unset or 0, source_fan_out: "thread". The metadata, signal, event-loop and decoder changes have no switch; revert the commits.

Related pages

• Raw frames read straight into the memory the GPU reads: the direct mode this builds on.
• The compositor's wait costs the frames that arrived and Event-loop sharding: the earlier steps on the same threads, and how the per-thread readings are taken.
• GPU memory: frames no tick can show any more: the memory changes made on the same nights.
• Chromium paints into the buffer the mixer reads: the browser's side of the same show.

Where in the code

• deps/ffmpeg/9/0020-avcodec-hevc-hwaccel-frame-start.patch (develop): the table clears skipped under a hardware decoder.
• src/nodes/hwaccel/raw_video_file_cuda.cpp (mixer/raw-preload): resident, the block the size of the file and the check of the file's modification time.
• src/MultiEventWait.hpp and src/nodes/mixer_compositor.cpp (mixer/compositor-signalled-inputs): which events fired, and a pass over those inputs only.
• src/nodes/input_rec.cpp and src/graph_core.hpp (mixer/input-hops): frame_metadata, routing, and one signal per hand-off.
• src/nodes/mixer_compositor.cpp (mixer/compositor-poll): input_poll_ms; pyplumber/mixer/aux.py (mixer/aux-input-poll): the same for the AUX buses.
• src/nodes/one_to_many_nb.cpp (mixer/fan-out-on-loop): the fan-out on an event loop.
• src/EventLoop.hpp (mixer/event-loop-self-wake): the loop on epoll, and no wake from its own thread.
• src/Event.hpp (mixer/coalesced-signals): a queue's event writes its descriptor once per read.
• config.md · canvas (mixer/vram, which carries the earlier branches): raw_upload, input_routing, input_poll_ms, source_fan_out.

Source links are pinned to the branch commits that run in the show, or to develop where the change has landed.