Historical recipe — not current setup guidance. Measurements and implementation links below belong to their original workload and revision. Deployment labels and defaults may be obsolete. Use the reviewed performance recipes; browse historical evidence.

AVPlumber · Mixer optimization cookbook

Browser frame ticks staggered at 60 fps

Implementation links are pinned to develop at c20464a5 (2026-10-09); read against develop at 8fa15c92 on 2026-10-13 and still current. Measurements retain their original workload and scope. Watch the current 192-input L4 demo.

Current scope. Staggering remains opt-in. The runnable-thread measurements below belong to the stated 60 fps workload; zero deadlines before and after does not establish extra admission capacity.

At 60 fps every browser window of every worker started its frame at the same instant. Once per 16,666 us the count of runnable threads spiked to about 57 on 16 vCPUs for about 2 ms, against about 6 for the rest of the period; the one-minute load average floated between about 9 and 23 while CPU use stayed at 53 %. DMA_BROWSER_STAGGER_FRAMES=1 starts each browser worker's frames at its own point of the period: the spike's peak fell to 22.9, the mean runnable count from 14.3 to 9.8 and the threads waiting for a CPU from 4.55 to 1.30, with 0 missed deadlines before and after. The one-minute load average is not a capacity gauge for this workload.

• 57 → 22.9peak of the runnable-thread count folded on the browsers' 16,666 us clock, on 16 vCPUs
• 14.3 → 9.8runnable threads, mean over 20 minutes
• 4.55 → 1.30threads waiting for a CPU, on average
• 16.4 → 7.9 %PSI cpu “some”: the share of time at least one thread waited for a CPU
• 0 and 0missed deadlines and output drops, in both 20-minute runs

The symptom

On the 88-source 60 fps show, the one-minute load average floated between about 9 and 23 on 16 vCPUs while CPU use stayed at 53 %, with no missed deadline and no dropped output. A load average above the CPU count reads as an overloaded host; half the CPUs idle says the opposite.

What was measured

The show: 88 sources at 1080p60, HLG 10-bit 4:2:2: 44 NVDEC file decodes, 4 raw v210 uploads and 40 browser windows (36 test pages and 4 key pages) in 5 browser workers of 8 windows; 256 scenes, 14 aux buses and 17 encoded outputs.

A passive sampler read the kernel's runnable-thread count (nr_running in /proc/loadavg) for 20 minutes at Poisson-timed instants, 100 a second on average: random spacing cannot alias with a frame clock, which a fixed-rate sampler can. It also read each thread's schedstat at the start and the end, which gives the time every thread ran and the time it waited for a CPU. Before the fix:

• Host CPU 52.8 % (sd 0.9). Runnable threads: mean 14.3, p50 9, p95 51, p99 67, at most 103; more than 16 for 27.9 % of the time.
• From schedstat: 8.18 cores running and 4.55 threads waiting for a CPU on average. The browser container ran 2.86 cores and had 2.25 threads waiting; the mixer ran 5.13 and had 2.22 waiting. 106,000 wake-ups a second.
• The runnable count has a component at 60.0024 Hz, a period of 16,666 us (first-harmonic amplitude 12.75), and a smaller one at exactly 60.000 Hz, the mixer's own grid (amplitude 4.5).
• Folded on exactly 16,666,000 ns of CLOCK_MONOTONIC (the first panel above): about 6 runnable for most of the period, 51 within 0.33 ms after the period start, the peak of 57 at 0.67 ms, 44 at 1.3 ms, 30 at 2 ms and 10 at 6 ms; 1.7 ms wide at half height. That spike carries 8.2 of the mean of 14.3.
• The sampler's own wake-up delay, that of a normal-priority thread: p50 0.09 ms, p95 0.21 ms, p99 0.60 ms; in the 3 ms after the period start, p95 0.5–0.58 ms.
• Missed deadlines 0, output drops 0.

Why the load average floats

Two things move the number, and only one of them is load.

1. The artefact: the kernel samples, it does not average. The load average is built from one instantaneous count of runnable threads every LOAD_FREQ = 5 × HZ + 1 ticks, 5.001 s on this kernel (CONFIG_HZ = 1000). The measured update period is 5.001 s; the update instants rule out 5.000, 5.002 and 5.004 s. Against a 16,666 us period, each sampling instant falls 1.2 ms later in the period than the one before, so the sampling walks through the spike once every 69.4 s (the periodogram of the kernel's samples peaks at 68.5 s). The kernel's point samples ranged from 0 to 78 (sd 15.9), while the real mean of each 5 s ranged from 11.7 to 19.0 (sd 1.6). Their correlation is 0.13; the point samples follow the fold curve at their own phase instead (correlation 0.81). load1 read 8.8 to 23.6; a one-minute average of the same data, taken over time, stays between 13.1 and 16.4.
2. The real part: two clocks that slide past each other. The browsers' period is 16,666 us and the mixer's frame is 16,666.67 us, so the browsers' spike slides across the mixer's frame once per 417 s. While it lands on the mixer's own per-frame work, more threads really do wait: the real one-minute average rises from about 13.3 to about 16.2 for roughly 1.5 minutes.

What to read instead. The one-minute load average is not a capacity gauge for a frame-locked workload: it is a point sample that beats against the frame clock. Read load15, CPU pressure (PSI, /proc/pressure/cpu), the compositors' missed deadlines, or a runnable count averaged over time. How to see it on a host has the commands.

The cause in Chromium

Read from the source of Electron 44.5.1 and Chromium 152; not traced on the host.

• Electron's offscreen view sets a window's frame timing in OffScreenRenderWidgetHostView::SetupFrameRate (shell/browser/osr/osr_render_widget_host_view.cc): it calls ui::Compositor::SetDisplayVSyncParameters with the timebase base::TimeTicks::Now() and an interval of 1,000,000 / fps in whole microseconds, 16,666 us at 60 fps.
• ui::Compositor::SetDisplayVSyncParameters (ui/compositor/compositor.cc) stores both and sends them to the window's display if it has one. It sets has_vsync_params_ only when the interval differs from the one it holds, which starts as viz::BeginFrameArgs::DefaultInterval() (components/viz/common/frame_sinks/begin_frame_args.h): 1/60 s, 16,666 us in whole microseconds.
• ui::Compositor::SetLayerTreeFrameSink hands the stored timebase and interval to a display created later only if has_vsync_params_ is set.

A 60 fps window sets its rate before its display exists, and its interval equals the default. The flag stays unset, the display never gets the window's timebase, and Chromium keeps timebase zero for it: every 60 fps window of every process ticks on multiples of 16,666 us of CLOCK_MONOTONIC. At 25, 30 and 50 fps the interval differs from the default, so each window ticks from the moment its rate was set and the windows are already apart.

The fix and its switch

DMA_BROWSER_STAGGER_FRAMES=1 in the browser service, off by default; docker-compose/mixer/compose.yaml passes the variable through. With it on:

• After every navigation of a 60 fps window, the worker sets the window's rate again, to 59 and then to 60, at the instant the monotonic clock is at the worker's phase of the 16,666 us period. The window's ticks start there. The other rate first is what makes Chromium keep the timebase for a display it creates later, such as after a GPU-process restart.
• The phase is the worker's index read in reversed binary: 0, 1/2, 1/4, 3/4, 1/8 and so on. The supervisor runs only as many workers as a show needs, and any first n of these phases stay at least half an even spacing apart, with no worker count to configure.
• Windows of one worker share a phase, so that worker's compositor and GPU threads still serve them in one wake-up.
• Windows at other rates are left alone. Frame rate and frame count do not change; a frame's timestamp is still the monotonic time of its paint, which now falls at the worker's point of the period.
• A worker reaches the instant with timer sleeps and a busy wait for the last 2–3 ms. After 20 sleeps that still missed it, the worker leaves the window on Chromium's timing and logs frame phase not set: timers woke too late.

Before and after

Two runs of 20 minutes on the same show: one before the fix, and 20 quiet minutes on the final build with the switch on. An earlier build that spaced the five workers evenly (index / 5) measured mean 10.05, peak 22.9 and 1.38 waiting, the same within noise. That run is not a clean one: the show was being cut during it (the program changed slot, which is why its counters cover 58,920 frames of the 72,000), and it shows 3 missed program deadlines that the two quiet runs do not have. Whether cutting or the build caused them was not separated; the final build's quiet run has none.

Who stopped waiting, per thread group (cores running / threads waiting, from schedstat; thread names as the kernel shows them, with a number written as #):

The work is the same and its cost barely moved: 8.18 cores running before, 7.87 after. What fell is the waiting. The compositors also counted fewer repeated input frames with the stagger on: 0.552 per program frame against 0.816 on the program, and 291.9 against 580.7 a minute on the multiviewer, with 444.8 against 981.8 discards a minute there. These are counted, not explained: no run isolated the cause.

How to see it on a host

• Fold the runnable count: sample nr_running from /proc/loadavg at random intervals, stamp each sample with CLOCK_MONOTONIC, and bin the stamps modulo 16,666,000 ns. One spike at one phase means the windows share a tick; one bump per worker means they are staggered.
• Fold on the browsers' period, not on 1/60 s: the spike drifts 40 us a second against 1/60 s and smears. The samples of the run before the fix, folded on 16,666.667 us, peak at 21.9 over a floor of 9.6 and are 5.7 ms wide at half height, against 57.0, 6.2 and 1.7 ms on 16,666 us. An earlier reading of a 68-input 60 fps show on the T4 test host, folded on 1/60 s, showed the same spike smeared into a burst about 8 ms wide, with the load average near 25 at 69 % CPU and PSI cpu “some” at 27 %.
• Per worker: GET /status of the browser service returns each window's lastPaintTsMs; fold those on 16.666 ms. With the switch on, the five workers painted at 4.3, 11.3, 7.7, 16.1 and 5.9 ms into the period (phases 0, 1/2, 1/4, 3/4 and 1/8).
• Each worker logs its framePhase when it starts (null with the switch off), and frame phase not set or frame phase failed for a window it could not align.
• CPU pressure: /proc/pressure/cpu, line some: avg10 is the share of the last 10 s during which at least one thread waited for a CPU; total is that time in microseconds since boot.
• Run-queue wait per process: the second field of /proc/<pid>/task/*/schedstat is nanoseconds spent waiting on a run queue. Take it twice, sum the differences per process and divide by the interval: the average number of that process's threads waiting for a CPU.
• Worst waits per thread: perf sched record -- sleep 2, then perf sched latency --sort max. It traces every context switch while it runs; keep it short.
• Missed frames: node.object.get <compositor> status returns playout.missed_deadlines, a running total since the show went on air.

Still open

• Why a resting window ticks at all. The stagger spreads the ticks; it does not remove them. 1,536 of the 2,400 window-ticks a second paint nothing (Browser CPU is per painted frame).
• The eight windows of a worker still tick together: the five bumps that remain. Whether spreading windows within a worker is worth the extra wake-ups was not measured.
• 25, 30 and 50 fps. The source says those windows already tick apart; no host run folded them.
• It is off by default. Each run was 20 minutes on one show. Hours of uptime and a GPU-process restart during a run were not measured, and the final run's status held no cut-latency sample.

At a glance

• What

  An opt-in switch in the browser service, DMA_BROWSER_STAGGER_FRAMES=1. Each browser worker sets its 60 fps windows' frame rate again at its own phase of the 16,666 us frame period, after every navigation. No avplumber or mixer change.

• Why

  All 40 windows of the 88-source show ticked at the same instant: about 57 runnable threads on 16 vCPUs at the peak of a 2 ms spike in every 16.7 ms, 4.55 threads waiting for a CPU on average at 53 % CPU use, and a one-minute load average that floated between 9 and 23.

• Latency

  0 frames added: frame rate, frame count and the paint timestamps' meaning are unchanged. A worker's paints move within the period by its phase; they had no fixed phase to the mixer's tick before either, since the two clocks slide past each other every 417 s, and the mixer uses smooth_timestamps to smooth browser arrival timestamps (smooth_timestamps node reference). A normal thread's wake-up delay fell from p95 0.21 to 0.14 ms and p99 0.60 to 0.31 ms. Cut latency was not measured in the final run.

• Side effects

  For each 60 fps window, after each navigation: a few timer sleeps and a busy wait of 2–3 ms on the worker's main thread, then two setFrameRate calls, 59 and 60, back to back. The windows of one worker still tick together.

• Cost

  0 nodes, 0 queues, 0 threads, 0 VRAM. Code: one 99-line module, FramePhase.ts, about 20 lines in ManagedWindow.ts and one environment variable.

• Risk

  It leans on Chromium behaviour read from the source of one version. A Chromium that hands the timebase on differently makes the second setFrameRate pointless or ineffective; the fold above shows which. Measured on one host and one show for 20 minutes, at 60 fps only.

• Switch it off

  Leave DMA_BROWSER_STAGGER_FRAMES unset or set it to 0, the default, and restart the browser service. Every 60 fps window then ticks at phase 0 again.

Where in the code

• deps/dma-browser/src/main/capture/FramePhase.ts: the cause, in a comment; source Chromium's default interval; source alignFramePhase: the wait for the instant, then 59 and 60 (source); source workerFramePhase: the switch and the reversed-binary phase.
• deps/dma-browser/src/main/ManagedWindow.ts: the window's first setFrameRate and the realignment on every did-navigate; Application.ts the worker's phase from its environment.
• deps/dma-browser/tests/unit/capture/FramePhase.test.ts: the instant, the other rate first, twenty windows aligning at once, late timers, and the phases per worker index.
• docker-compose/mixer/compose.yaml: the variable passed through, default 0.
• Electron 44.5.1: OffScreenRenderWidgetHostView::SetFrameRate and SetupFrameRate in shell/browser/osr/osr_render_widget_host_view.cc. Chromium 152: ui::Compositor::SetDisplayVSyncParameters and SetLayerTreeFrameSink in ui/compositor/compositor.cc, has_vsync_params_ and vsync_interval_ in ui/compositor/compositor.h, viz::BeginFrameArgs::DefaultInterval in components/viz/common/frame_sinks/begin_frame_args.h.
• Docs: dma-browser README (the switch); How to see it on a host (the counters and commands); Browser CPU is per painted frame (what a tick and a paint cost).

Measured on the L4 host (one NVIDIA L4, 16 vCPUs; kernel 7.0, CONFIG_HZ = 1000) with the 88-source 60 fps show: 44 NVDEC decodes, 4 raw v210 uploads and 40 browser windows in 5 workers. Two runs of 20 minutes, before and with the switch on.