Playout bench
For: contributors changing playout, or checking a new host’s encoder. You’ll get: how to run the bench, and whether this host still meets the playout numbers.
The playout bench answers one question on any host: does Loomarr’s real playout pipeline still meet
the beta.8 numbers here? It runs a generated, redistributable corpus through playout.Build, the
same pipeline builder the live chain uses, measures it, judges the result against the thresholds and
diffs it against the last accepted report for that hardware family.
make playout-bench # detect the host as the app does at bootPLAYOUT_BENCH_FAMILY=software make playout-bench # force a family (a GPU host, GPU ignored)It needs ffmpeg and ffprobe on PATH (or PLAYOUT_BENCH_FFMPEG), with libx264, libx265 and,
for the VMAF number, libvmaf. Run it on a quiet machine: it measures speed and CPU, so another heavy
job on the host makes the numbers meaningless.
What it measures
| Metric | How | Threshold (hardware families) |
|---|---|---|
start_p95_ms/<class> | encoder start (warm): fresh process per run, seeking to a different second, using production FragmentArgs and the real packager until a complete one-second fMP4 segment is stamped and written. Encoder surplus and exit cleanup are reaped after taking the clock. Source files are page-cached, so cold network-share reads are excluded; those are host-specific and belong to supervised household runs, not CI. | H.264 1080p ≤ 400 ms, HEVC 1080p ≤ 500 ms, 4K HDR ≤ 1800 ms; NVENC 1080p ≤ 700 ms, because the driver spends ~460 ms initialising every new process |
speed_x/<class> | one whole clip, unpaced, less the start-up of a one-frame run of the same pipeline (start-up is start_p95_ms’s job; NVENC’s ~460 ms would otherwise cap a 10 s clip near 10x) | ≥ 15x (1080p), ≥ 2x (4K HDR) |
cores_per_stream/<class> | child CPU time (rusage) per second of media, so the cores one stream costs at 1x | ≤ 0.1 core per stream |
concurrency/max_streams | largest number of simultaneous 1080p H.264 streams that each hold ≥ 1.2x | ≥ 1 (every family) |
concurrency/total_cores | CPU of those streams at 1x | reported; a GPU that runs more channels spends more in total, so CPU is judged per stream |
break/pts_gaps, break/audio_off_grid | video PTS off the frame grid, audio PTS off the AAC frame grid, across every corpus clip | 0 |
break/sps_variants | distinct H.264 SPS across every clip’s output, so items of any geometry, cadence or codec splice into one decoder | 1 |
break/loudness_dev_lu | worst integrated-loudness deviation from −23 LUFS on the generated clips, no loudnorm | ≤ 1 LU |
break/first_packet_ms | worst time to first output byte across the commercial spots | ≤ 250 ms; NVENC ≤ 400 ms, because each spot is a new process that pays the driver’s start-up |
vmaf_mean/<class> | VMAF of the output against the source scaled to the rung | tracked against the baseline only |
Classes are h264-1080p, hevc-1080p and hdr-4k. A source the builder refuses on this host (a
software host that cannot tone-map 4K HDR in real time) is reported as refused, its metrics are
marked skipped and it is not a failure: the slate covers that slot.
Report schema 2 corrects startup’s measurement boundary: schema 1 timed an MPEG-TS encoder’s process exit, including surplus encoding and device teardown. Those reports remain historical evidence and cannot be compared or accepted as schema-2 baselines. Thresholds, quality, seek distribution, whole-clip speed and CPU methodology are unchanged; hardware must be remeasured.
Known failures. A check that fails until referenced work lands is listed in expectedFailures
(internal/playout/bench/compare.go) and reported as expected fail with that reference, never skipped.
Once the fix lands the check passes, which the bench reports as an unexpected pass and fails on, so the
entry is deleted in the same change. The list is empty: break/sps_variants (an 854×480 source scaled to
SAR 1281:1280 in the SPS) passes since the packager’s items encode with setsar=1 (#1526).
The software family has no GPU thresholds; it is judged on sustaining a stream at 1.2x and on
correctness; the channel capacity it measures (concurrency/max_streams) is reported with every run.
PLAYOUT_BENCH_THRESHOLDS=correctness judges only the exact checks (gaps, SPS,
loudness). CI uses it on the virtualised macOS runner. off skips judging.
The corpus
Everything is synthetic, made with ffmpeg lavfi, and generated on first use into
$LOOMARR_ARTIFACT_DIR/playout-bench-corpus: H.264 (23.976, 25 and 29.97 fps, interlaced, 10-bit),
HEVC 10-bit, a 4K HDR10-tagged HEVC clip, and AAC, AC-3, E-AC-3 and TrueHD 5.1 audio. Four
commercial spots (720p, 1080p, 480p, HEVC) with fades, all built at −23 LUFS, form the break sequence.
Changing any clip changes corpus, which invalidates every baseline.
For realistic content, fetch Blender’s Tears of Steel (CC BY 3.0) with a pinned hash and pass the directory in. Household media must never enter the bench.
scripts/playout-bench-open-films.sh "$LOOMARR_ARTIFACT_DIR/films" 720p # add 1080pPLAYOUT_BENCH_FILMS="$LOOMARR_ARTIFACT_DIR/films" make playout-benchEach film contributes a film-<name> class over its first 30 seconds.
Reading a report
make playout-bench prints markdown and writes playout-bench-<family>.json and .md to
$LOOMARR_ARTIFACT_DIR. It has three parts:
- Thresholds: each beta.8 limit with the measured value. A threshold whose metric is absent
fails; only a metric the host declares skipped (no
libvmaf, refused HDR) is excused. - Against the accepted baseline: every metric’s change. A change is a regression only if it is
worse than the baseline by more than the tolerance (15% by default,
PLAYOUT_BENCH_TOLERANCE) and by more than a small absolute slack per unit (25 ms, 0.3x, 0.01 cores), so a timer tick on a fast start is not a failure. Correctness counts must not change at all. A metric that vanishes without a skip reason is a regression. - Metrics and Clips: everything measured, and each clip’s outcome and any stage that left the
GPU (
decode: …,tonemap: …).
The command exits non-zero on a failed threshold, a regression or a clip that failed to encode.
Accepting a new baseline
Only from a supervised run on the real hardware, on a commit you have read the numbers for:
PLAYOUT_BENCH_ACCEPT=1 make playout-benchgit add docs/engineering/playout-bench/<family>.jsonThe bench refuses to accept a report that fails a threshold or regresses. Fix the cause, or change
the threshold in internal/playout/bench/compare.go in the same PR that explains why. A baseline is
tied to the schema and corpus version; bumping either forces a re-accept.
CI
| Job | When | What |
|---|---|---|
Playout bench — software-only pipeline (ci-playout-bench.yml) | every PR that touches playout, the bench or its scripts (classifier gate playout_bench, part of CI) | 720p software family, pinned production ffmpeg, against docs/engineering/playout-bench/ci/ |
macos-15 — VideoToolbox (playout-bench.yml) | workflow_dispatch and nightly | VideoToolbox family; correctness thresholds only, since the runner is virtualised |
GitHub T4 GPU — NVENC | workflow_dispatch and nightly, only when the repository variable PLAYOUT_BENCH_T4_RUNNER is set | needs a paid GPU larger runner (about $0.052/min); set the variable to its label to enable |
Self-hosted — Intel Arc, Self-hosted — NVIDIA GeForce | workflow_dispatch only | the maintainer’s machines |
Hardware coverage (playout-bench.yml) | every run, after the legs | lists which legs ran in the run summary; red only when a leg whose gate is set did not pass or wrote no report |
A skipped leg is not a pass. The Hardware coverage job writes one line per leg, such as NVENC leg (GitHub T4) skipped: PLAYOUT_BENCH_T4_RUNNER unset, and repeats it as a run annotation. A leg whose
gate is unset is a notice, so a homelab without a T4 subscription or a self-hosted GPU runner stays
green on purpose. A leg whose gate is set but that was skipped, failed, or passed without writing its
playout-bench-<family>.json report turns the run red (#1566). The self-hosted legs’ gate is the
run_arc / run_geforce dispatch input: the nightly run never uses the household GPUs, and a token
that could check whether a runner is online would need admin rights the workflow does not have.
macos-15 arm64 runners expose VideoToolbox hardware encode; macos-14 does not.
The macOS job installs a SHA-256-pinned static arm64 ffmpeg (VideoToolbox included) with the same bounded
retry as scripts/ci-ffmpeg.sh, never an unpinned Homebrew bottle. It is ffmpeg 9.0.2 because no
older macOS arm64 build is published; that is not the production 8.1 pin. ffmpeg 9 breaks the concat
advance over a chunked HTTP body, which the bench never exercises: it drives playout.Build pipelines
and reads the output directly. The bench films are Tears of Steel 720p and 1080p only; the 4K class
runs on generated HDR10 clips.
Self-hosted runners
Register each machine as a repository runner with the custom label loomarr-playout-arc or
loomarr-playout-geforce, with ffmpeg, ffprobe, go and make on its PATH. The workflow
playout-bench.yml must never gain a pull_request, pull_request_target or merge_group trigger:
the repository is public, and a pull request must not be able to schedule work on these machines. A
test (TestPlayoutBenchWorkflowNeverRunsOnPullRequest) enforces that. Dispatch the workflow from the
Actions tab, tick run_arc or run_geforce, and tick accept to have the run write the family
baseline into the uploaded artifact for you to commit.