Historical recipe — not current setup guidance. Measurements and implementation links below belong to their original workload and revision. Deployment labels and defaults may be obsolete. Use the reviewed performance recipes; browse historical evidence.

GPU memory: frames no tick can show any more

Measured on 14 October 2026 on the 16 vCPU NVIDIA L4 host that runs the demo; GPU memory is the whole card as the driver reports it, unless a line says per process. Not on develop: the changes are on the avplumber branch mixer/vram at 7310870e and run in the demo show. Watch the current 192-input L4 demo.

Until every input of a compositor has a picture, at the start of a show, the compositor cannot draw a tick, and it queued up to eight frames of every other input meanwhile. Each of those frames is a GPU buffer from its source's pool, the pools grew to hold them, and such pools never shrink: the start set the memory the show then kept. The playout now lets a frame go as soon as a newer one is at or before the tick due now, which no tick can undo. What is drawn is unchanged. The show's steady GPU memory fell from 20.1 to 18.8 GB, and the start no longer runs the card out of memory.

Status: on a branch, running in the demo show

20.1 → 18.8 GBwhole-GPU memory of the 192-source show at 1080p30, steady
22.4 → 19.2 GBthe peak during a start, on the starts before and with the change
~50 → 0Cannot allocate memory lines in the log of a start
33 → 18 MiBextra memory a decoder kept when it fed a compositor that could not tick, against a null sink, in a probe
3,000 ticksof mixed rates, offsets, stalls and late wakes drawn identically before and after, in the equivalence test

Illustration, not a measurement. The rule is per input and runs before every attempt to draw: a frame whose successor is stamped at or before the earliest tick still to be drawn can never be shown. An input with a frame-rate cadence keeps one more, because a cadence can move queued frames one tick later.

Why a start decided the show's memory

Two kinds of GPU frame pool in the mixer behave differently, and both were hit at the start (from the code, checked against the measurements below):

Pools that grow and keep. A raw source's pool and every compositor's canvas pool allocate a new frame whenever all of theirs are still in use, and return nothing to the driver while they live. Their size is the most frames that were ever out at once, and the start, with its stalls while encoders open and every scene is prewarmed, is where that happens.
The decoders' fixed bank. On the CUDA array path each HEVC decoder allocates 9 + extra_hw_frames arrays when it opens (12 extras in the recipe: 21 arrays) and never more. A frame anywhere in the graph is a reference to one of them. When none is free the decoder fails that picture (No decoder surfaces left, logged as Cannot allocate memory); the lost picture was a reference, so the source shows corruption until its next keyframe.

A probe made it concrete: a decoder feeding a compositor that could not tick kept 33 MiB more GPU memory than one feeding a null sink, for the life of the process; with the change, 18 MiB. On the show, a start logged about 50 Cannot allocate memory lines, and two starts in a row failed outright when a raw reader's allocation found the card full.

The change

Before each attempt to draw, the playout drops, per input, every frame whose successor is at or before the tick due now. A tick shows the newest frame stamped at or before it, and the earliest tick still to be drawn is the one due now, so such a frame can never be drawn. Nothing is dropped while a decision awaits its commit. An equivalence test drew 3,000 ticks with mixed rates, offsets, stalls and late wakes with and without the rule: the same frames, tick for tick.

The start that followed logged no Cannot allocate memory line and peaked at 19.2 GB, against 22.4 GB on the start before it. Peaks vary from start to start: purging idle browser imports before the outputs go on air (on develop since 13 October) had brought an earlier start's peak from 22.4 to 19.6 GB.

Step by step

The running 192-source show at 1080p30 (105 HEVC files through NVDEC into CUDA arrays, 40 browser windows, 47 raw NV12 files, 24 encodes). Each reading was taken after a restart on that build; between two starts of the same show the whole-GPU figure has differed by up to 1.9 GB.

Build or setting	Whole GPU	Mixer CPU	What it is
After the input hops	20.2 GB	265 %	the reference for this page
Fan-out on the pacing loops	19.3–20.1 GB	247 %	a CPU change (its page)
`decoded_frames_ahead: 1`	20.1 GB	247 %	one decoded frame waits in front of pacing instead of three; the raw reader waits for GPU memory instead of failing
Playout drops superseded frames	18.76 GB	246 %	the change above
Sources at `extra_hw_frames: 8`	17.78 GB	248 %	8 spare decoder arrays instead of 12
Sources at `extra_hw_frames: 6`	16.21 GB	241 %	the first start attempt failed twice with `No decoder surfaces left`; the show went back to 12

Every step: 0 missed deadlines and 0 output drops over the readings, no repeats of decoded or raw sources, no frame dropped at any of the 5,398 queues, and no decoder error at start or after, except where the table says otherwise.

Three settings, and what each is worth

canvas.decoded_frames_ahead: how many decoded frames of a file or stream source may wait in front of its pacing node (default 3). On the CUDA array path a waiting frame is a reference to one of the decoder's fixed arrays, not a buffer of its own, so at 1 the whole-GPU reading did not fall (20.1 GB, within the previous build's 19.3 to 20.1). It matters only as room for fewer spare arrays. This is the shorter decoded queue that an earlier proposal asked for, now as an option; its effect on repeated frames over an hour has still not been measured.
Spare decoder arrays (extra_hw_frames per source). With superseded frames released, a worst-case probe (decoders into a compositor that cannot tick) never ran out of arrays even at 4 spares. On the show, 8 read 17.78 GB and 6 read 16.21 GB, but at 6 two start attempts in a row failed before Setup's retry succeeded. A start that loses decoder pictures is not acceptable, so the show runs at 12. Reading the code, the playout queue (eight frames at every rate) would have to be bounded by the latency before the spares can come down safely; that is not built.
canvas.raw_pool_frames: at most this many GPU frames per direct or resident raw source; the source waits for a frame to come back instead of allocating. Unset, the pool keeps its high-water mark. Not measured on the show yet. From the code, a raw source needs about 7 frames when the Program compositors tick steadily and reaches 18 to 20 after a stall, so for 47 sources the spread is about 1.8 GB (an estimate), close to the 1.9 GB measured between two starts of the same show.

What things cost, measured one at a time

Probe processes on the L4, per-process GPU memory, 14 October 2026:

Unit	GPU memory	Note
CUDA context, no nodes	186 MiB	once per process
HEVC decoder, 1080p, NVDEC	71–74 MiB	on the linear output path; the show uses CUDA arrays, not measured this way
Raw NV12 1080p source	12 MiB	a one-slot chain into a null sink; the show's fan-out holds more
NVENC H.264 1080p p3 with the AUX settings	48 MiB	34 MiB with `dpb_size` 1, which the AUX encoders get (one reference frame)

Before these changes the mixer process alone held 15,450 MiB of the show's 19.2 to 20.2 GB. The browser's GPU processes hold about 4.7 GB at a 16.2 GB whole-GPU reading, and the zero-copy export adds about 0.5 GB.

At a glance

WhatThe playout releases frames no tick can show any more, before every draw, not only when a tick is drawn. Two settings bound what waits before pacing and what a raw source may allocate; the raw reader waits for GPU memory instead of failing.
WhyA compositor that could not tick yet held eight frames of every input, the pools grew to match and kept that size, and the start ran the card out of memory: about 50 failed allocations in a start, and two failed starts.
Latency0 frames added. The same frame is drawn on every tick; the equivalence test checks it.
Side effectsA compositor's discarded-frame counter rises during a start, where frames used to wait. None in steady state.
CostNo node, queue or thread. A pass over the inputs' queue heads before each draw attempt.
RiskOn one show at 1080p30. Fewer spare decoder arrays saves memory but made starts fail; the recipe keeps 12.
Switch it offNo switch for the playout rule; revert the commit. Leave decoded_frames_ahead and raw_pool_frames unset for the earlier behaviour.

NVDEC arrays without a staging copy: the decoders' fixed array bank and the 4.88 GiB it saved.
Browser frames go back one per new frame: the import purge before the outputs go on air.
Input, decode and wake-ups: the CPU changes of the same nights.

Where in the code

src/mixer/Playout.hpp: discardSuperseded, the rule and why a cadence keeps one more.
src/nodes/mixer_compositor.cpp: the call before each attempt to draw.
src/nodes/hwaccel/raw_video_file_cuda.cpp and src/nodes/hwaccel/CudaPinnedUploader.hpp: pool_frames, and the wait for GPU memory.
config.md · canvas: decoded_frames_ahead and raw_pool_frames.

Source links are pinned to the branch commit that runs in the show.