Historical recipe — not current setup guidance. Measurements and implementation links below belong to their original workload and revision. Deployment labels and defaults may be obsolete. Use the reviewed performance recipes; browse historical evidence.

AVPlumber · Mixer optimization cookbook

A wipe clip is cached whole or the start fails

Implementation links are pinned to develop at c20464a5 (2026-10-09). The mixer has changed since; this page was checked against develop at 8fa15c92 on 2026-10-13. Measurements retain their original workload and scope. Watch the current 192-input L4 demo.

Current scope. EOF-based preload and count checks are implemented, with loader deadline/cleanup and one retry. Native builds and Python tests do not establish dedicated wipe GPU qualification. That validation remains separate from the cadence/transform smoke tests.

The wipe preload used to decide that a clip had ended when no frame arrived for 200 ms. A stalled GPU upload looks the same, so a start on a fresh container cached a wipe of 2 frames. A load now ends only on the decode chain's end-of-stream marker, the stored frame count is checked, and the stall itself is avoided on every start but a host's first.

What went wrong

A freshly created mixer container compiles FFmpeg's CUDA filter kernels on first use. On the L4 host that stalled GPU work for about 15 s during startup, while the wipe clips were being decoded into the GPU cache. Two things followed:

• A stall was taken for the end. clip_cache finished a load after 200 ms without a frame. The upload of the diagonal wipe stalled after its first frames, and the clip was stored with 2 frames instead of 120. The operator saw a broken wipe.
• Late frames went to the next clip. Stopping the loader group is only a request. The next load was accepted while the stalled upload node was still alive, and the frames it delivered afterwards were stored in the next clip, which held 129 frames instead of 120.

After a restart with the kernels already compiled, both clips held 120 frames.

The rule now

1. The end is a fact, not a silence. input_rec sends an end-of-stream marker at the end of the file, and demux, the decoder and the upload filter each pass it on behind their last frame. realtime used to consume it; with forward_eof it passes it on too, in order. clip_cache ends a load on that marker and on nothing else, so a stall of any length only delays it.
2. What was stored is counted. When a load has ended, clipcache.preload() compares the cached frame count with the number of frames the decoder delivered for that load. A clip with a different count, fewer or more, is dropped (forget) and loaded once more, and so is a clip that ended without a picture or did not fit the cache budget. A second difference fails the start with the clip's name and both counts; a clip of which nothing was cached twice is named without counts. A load that has not ended within --preheat-timeout fails the start at once.
3. Every clip is still there at the end. The cache makes room for a clip by evicting the least recently used ones, so a later wipe can push out an earlier one that passed its own check. After the last load clipcache.require_cached() reads the status once and fails the start if any wipe is missing, naming the evicted clips, the bytes cached and the budget (--wipe-cache-mb). A clip the cache already holds is not loaded again.
4. Nothing is left over. After each load the loader group is stopped, the preload waits until none of its nodes is running, clears the edges between them and waits until the cache has discarded what still reached it. Only then is the next load sent. Inside clip_cache a frame leaves the input edge under the same lock that accepts a load.
5. The clip keeps its own timestamps. The loader's realtime still paces the clip but no longer restamps it (set_pts off when caching). A replay places frames by the timestamps they were stored with, and a restamped load that waited through a stall would have stored the stall as a gap in the wipe.
6. Compiled kernels outlive the container. CUDA_CACHE_PATH=/media/.nv-cache puts the driver's kernel cache on the media volume, so a new container finds the kernels compiled and does not stall.

At a glance

• WhatA wipe load ends on the end-of-stream marker; the cached frame count must equal the decoder's; one reload, then the start fails. The loader is fully stopped and emptied between clips. The driver's kernel cache is kept in media/.nv-cache.
• WhyWith the cache on, every wipe must be in it, whole. A fresh container stored one wipe with 2 frames instead of 120 and the next with 129 instead of 120, and nothing reported it.
• LatencyProgram and command-to-screen latency: unchanged. A take replays the cache as before.
• Side effectsA start that used to come up with a broken wipe now waits for the clip, reloads it, or fails with a message naming it. Cached wipe frames carry the clip's own timestamps instead of load-time ones. clip_cache status gains loading; the node gains forget; realtime gains forward_eof (default off).
• Cost0 nodes, 0 queues, 0 threads, no GPU memory. Startup polls a few more states per clip. The kernel cache takes disk space on the media volume. The first start of a new host, or one with an empty media/.nv-cache, still compiles once and is slower by that stall.
• RiskA clip the decode chain cannot deliver whole twice now stops the mixer from starting, where it used to start with a damaged wipe. The check trusts the decoder's count: frames the decoder itself never produces are not missed. The C++ changes are built; no dedicated wipe run on a GPU has qualified them.
• Switch it offNo switch for the rule. --wipe-cache-mb 0 turns the cache and its preload off and decodes on each take. Remove CUDA_CACHE_PATH from compose.yaml to return the kernel cache to the container.

Where in the code

• src/nodes/clip_cache/clip_cache.cpp: process() ends a load on the marker; forget; loading in the status.
• src/nodes/realtime.cpp: forward_eof.
• pyplumber/mixer/clipcache.py: preload(), the count check, the reload and the stop-and-discard between loads; require_cached() after the last one.
• pyplumber/mixer/graph.py: the loader's realtime with forward_eof and without set_pts when caching.
• docker-compose/mixer/compose.yaml: CUDA_CACHE_PATH, also in effect for deploy/l4, which layers its file over this one.
• tests/mixer/test_wipe_preload.py: short, long, never-ending, uncached and already cached loads, and the order between two loads, against a simulated loader; test_prewarm.py: a wipe evicted by a later one.

To see the stall again on a GPU host, start a new container with an empty media/.nv-cache.