Skip to content

Publishing a release

For: maintainers cutting a release. You’ll get: what a v* tag publishes, and which workflow owns each part.

Server image publication and GitHub Release publication intentionally run in separate workflows from the same v* tag. The image workflow owns package write and keyless-signing permissions. The release notes workflow owns only contents: write; it cannot build, sign, or promote an image.

One-time repository setup

Add a dedicated OpenRouter key as the masked Actions secret used only by release notes:

Terminal window
gh secret set OPENROUTER_RELEASE_API_KEY --repo loomarr/loomarr

The default model is openai/gpt-5-mini. To change it without editing the workflow, set the optional repository variable RELEASE_NOTES_MODEL to an OpenRouter model that supports strict structured outputs. The release process is designed for a small classification call, not model-authored prose.

Add the TMDB key Loomarr ships with (#2151 decision 51). Register it as Loomarr’s own application at TMDB, not a personal key. The release build stamps it into the binary through a BuildKit secret, so it never appears in the image history or the provenance. A household’s own key, from Settings or TMDB_API_KEY, always wins over it. Without the secret, a release ships no key, and TMDB features wait until the household adds one.

Terminal window
gh secret set TMDB_BUNDLED_KEY --repo loomarr/loomarr

Preview before tagging

Authenticate gh, export the release-specific key locally, and generate a preview for the proposed tag. The tag may already exist or be a commit-ish accepted by GitHub’s generated-notes endpoint.

Terminal window
export OPENROUTER_RELEASE_API_KEY='...'
make release-notes-preview TAG=v0.2.0 PREVIOUS_TAG=v0.1.0-beta.1

The default output is .artifacts/release-notes-v0.2.0.md. Set OUTPUT=/path/to/notes.md to choose another destination. Do not commit or paste the key. A tag-specific project/release/<tag>.md file, when present, is prepended for human-authored framing and known limitations.

Check a candidate on real data and real GPUs

Do both before a final tag.

Rehearse the migrations. A final tag (one without -rc.) is refused until this has passed on its exact commit. Run it where the household backup lives, so the data stays there:

Terminal window
make migration-rehearsal BACKUP=<backup directory> IMAGE=ghcr.io/loomarr/loomarr-staging:<full commit id> RECORD=1

It boots the candidate on a copy of the backup and reports whether it came up. Then it stops the candidate and checks the migrated copy. SQLite’s integrity check and foreign-key check must both come back clean. It also lists every table whose row count changed. Compare that list with what this release’s migrations meant to delete before you tag. It needs sqlite3. The backup is only read, and the container has no network, because the database names the household’s real services. RECORD=1 then sets the loomarr/migration-rehearsal commit status, which release.yml checks. The status says how long the migration took and nothing about the backup. Dispatching staging.yml from a commit builds that commit’s image.

Check the hardware. Software, Intel and NVIDIA block a release; AMD is best-effort. Software is covered by required CI. For the others, dispatch playout-bench.yml with run_arc and run_geforce, then run:

Terminal window
make hardware-gate COMMIT=<full commit id>

A skipped leg counts as missing. release.yml runs the same check and refuses a final tag until both legs have passed on its commit; a -rc. tag doesn’t need them.

Docs pass

The docs ship with every release (#1682, maintainer decision D3; step 5 of #1572). Two things happen without any extra action:

  • The docs site (Pages workflow) redeploys on every v* tag push, not only on pushes to main. It renders docs/** in place, so the published site always matches the tagged commit’s documentation, even for a release that changed no docs.
  • The in-app Help set is //go:embed-ded into the server binary (docs/embed.go), and the release image (release.yml) builds from the tagged source. Help is current the moment the image is current — there is no separate build step.

The gate runs automatically on the tag. The Release notes workflow runs scripts/check-release-docs-gate.sh right after it renders the notes and before it creates the GitHub Release. It fails that workflow — publishing no Release — when the rendered notes contain a New Features, Improvements, Bug Fixes, or Security Fixes entry and the tag range touched nothing under docs/, docs-site/, or README.md. It reuses the same seven-category classification the release notes already show (cmd/release-notes) rather than a second taxonomy, and it is unconditional like every other step in that workflow: fixing the docs, or the PR classification that misled it, is the only way past it. A release that only ships Documentation, Dependencies, or Maintenance changes is not held to a docs change it has no user-facing reason to need. release.yml, the separately hardened image-publication workflow, is unchanged — it still owns only build, sign, and promote, with no added step or job.

What still needs a human: catch it before tagging, not after a failed Release publication. Generate the preview notes and run the same gate locally over the same range:

Terminal window
make release-notes-preview TAG=v0.2.0 PREVIOUS_TAG=v0.1.0-beta.1
make docs-release-gate TAG=v0.2.0 PREVIOUS_TAG=v0.1.0-beta.1

See scripts/check-release-docs-gate.sh for the gate itself, shared by both the local preview and the workflow.

What the model can and cannot do

The helper first asks GitHub to generate the exact merged-PR list, contributor list, and compare link. It sends only each PR number and title to OpenRouter. The structured response is one closed object: every exact PR number is a required property, and its value must be one of these fixed sections. That shape encodes one assignment per PR even for unusually large releases instead of asking the model to repeat PR numbers across seven independent arrays:

  • New Features
  • Improvements
  • Bug Fixes
  • Security Fixes
  • Documentation
  • Dependencies
  • Maintenance

Repository code renders GitHub’s original bullet for each assignment. It independently rejects unknown JSON fields, invented PRs, duplicate keys, omitted PRs, invalid categories, malformed output, and unrecognized GitHub change lines. It retries inference three times and then fails without creating a GitHub Release. It never silently publishes uncategorized or model-authored notes.

Tag and verify

Before requesting remote release certification, test the exact current main commit on the maintainer’s local machine. Record the commit, commands, and results with the release evidence. At minimum, run every release-relevant gate this host supports: Go/Rust contracts and tests, Postgres, web unit/build, visual/e2e/tuner browser evidence, shared clients, both Expo Android app builds, legacy Android TV including its release bundle contract, image-worker certification, release policy verification, and the docs pass (make docs-release-gate). Host-incompatible evidence such as iOS/tvOS and native arm64 image builds must be named explicitly and remain required in protected CI; a local Linux pass cannot stand in for them. Agent sessions still never run the maintainer’s live-stack make smoke* targets.

Push the protected version tag only after that local evidence is green, the required commit gates are green, and the exact current main commit has passed the proportional release-candidate scope:

Terminal window
gh workflow run ci.yml --repo loomarr/loomarr --ref main -f scope=release-candidate

That scope runs repository contracts, real-codec image-worker certification, and the native amd64 and arm64 image builds. Each image build also compares the packaged LICENSE and THIRD_PARTY_NOTICES.md with the tagged source, then runs the unchanged make test-ffmpeg gate against the ffmpeg and ffprobe copied out of that image (#661). That ties the gate to the exact bytes the published GPL source releases correspond to. It deliberately leaves unrelated platform and UI matrices to their normal change-based CI. A normal push run or -f scope=full run is never accepted as Docker-release evidence, even when green, because neither proves that publication is isolated from those unrelated jobs. Use -f scope=full only when a complete manual rerun is independently required.

Both release workflows start from the tag. If OpenRouter is temporarily unavailable, rerun the failed Release notes workflow after service recovers; the separately hardened image publication is unaffected.

Before signing, the publish helper (scripts/publish-release-image.sh) refuses an image index unless both platform images carry a non-empty SPDX SBOM and a build provenance whose every recorded source revision is the tagged commit of this repository. After both workflows finish, verify the GitHub Release body (including its links to the third-party source releases named in THIRD_PARTY_NOTICES.md), the GHCR manifest, signature, SBOM, and provenance against the tagged commit. The tag-specific header remains the place to state limitations that cannot be derived from pull requests.

Apple TestFlight

The Apple TestFlight betas workflow (.github/workflows/apple-testflight.yml) builds, verifies, and uploads the two permanent-identity Apple apps. Each is its own App Store Connect record:

AppappRecordBundle IDPlatformVariable
iPhone (ADR 0042)iphoneLoomarrmedia.loomarr.mobileiphoneosASC_APPLE_APP_ID
Apple TVapple-tvLoomarr TVmedia.loomarr.tvappletvosASC_APPLE_TV_APP_ID

It runs only on main and uses the ios-testflight environment, whose deployment branch policy releases the secrets only to main. It never runs for a pull request or a merge group.

Every push to main that changes an Apple app uploads a build of that app with no manual step. A secretless plan job decides, using the classifier CI uses for the Apple mobile and Apple TV gates (scripts/ci-impact.sh, apple_mobile and apple_tv), so a push that changes neither app uploads nothing, and a push that changes both uploads both. As in CI, a push whose diff cannot be established counts as a change to both. The pushed build’s version is the newest v* tag reachable from the commit (0.2.0-beta.8 until v0.2.0-beta.9 is tagged); TestFlight shows only its x.y.z.

The release job runs once per planned app, from a matrix over the plan’s list. One app’s failure does not cancel the other’s upload. Release runs of one app queue one at a time. A newer pending one replaces an older one, and it builds a later main that already contains the older change.

One script, web/scripts/build-apple-testflight.sh <app>, archives, exports, verifies, validates, and uploads either app; its table of what differs between them (Expo package, bundle ID, Xcode scheme, destination, platform, and record variable) is the only per-app code. Both app configs take the release identity from web/scripts/apple-testflight-config.cjs.

The environment has no required reviewer (a maintainer decision): the reviewed, merge-queue-tested merge is the review. release-verify pins the triggers, inputs, both jobs, the matrix, the environment, and every script step’s command, environment, and condition (internal/platform/releaseverify/apple_testflight_workflow.go).

One-time Apple setup

  1. Enrol in the Apple Developer Program and note the 10-character Team ID.
  2. Register each app’s App ID (media.loomarr.mobile, media.loomarr.tv). Automatic signing can register it, but doing it first removes one failure mode.
  3. Create each App Store Connect app record by hand (name, language, bundle ID, SKU). Note its numeric Apple ID; App Store Connect cannot create the record from the workflow.
  4. Create a Team API key (Users and Access, Integrations; Admin only). An individual key cannot use provisioning endpoints. Download the .p8 once. Apple’s cloud signing then holds the distribution certificate, so the workflow needs no certificate, profile, or keychain. If a role below Admin cannot cloud-sign, use Admin. One key serves both records.
  5. Add internal testers in TestFlight, in a group on each record.

Repository setup

Create the ios-testflight environment and limit its deployment branches to main; it has no required reviewer. Then set:

NameKindValue
ASC_API_KEY_P8_BASE64secretbase64 -w0 AuthKey_<id>.p8
ASC_API_KEY_IDsecretthe key’s 10-character ID
ASC_API_ISSUER_IDsecretthe issuer UUID
APPLE_TEAM_IDvariablethe Team ID
ASC_APPLE_APP_IDvariablethe iPhone record’s numeric Apple ID
ASC_APPLE_TV_APP_IDvariablethe Apple TV record’s numeric Apple ID

The credentials step, which runs after dependency installation and before the archive, fails with a message naming every value its app is missing. An app needs only its own record’s variable.

Dispatch

Terminal window
gh workflow run apple-testflight.yml --repo loomarr/loomarr --ref main -f app=apple-tv -f version=0.2.0-beta.9
gh workflow run apple-testflight.yml --repo loomarr/loomarr --ref main -f app=iphone -f version=0.2.0-beta.9 -f upload=true

Dispatch for a dry run or a re-upload of one app; it rebuilds the current main with the version you name. With upload unset (the default) it signs, exports, verifies, and runs altool --validate-app without uploading, and it logs xcodebuild -version and sw_vers. App Store Connect picks the build number: one past the highest build the record already holds (web/scripts/asc-next-build-number.mjs), with github.run_number only as a floor, so a re-run or an upload made elsewhere never collides.

Each hosted Mac starts with an empty keychain, so the archive’s automatic signing creates a new Apple Development certificate on every run. Once the export is done (or has failed), the build revokes it (web/scripts/asc-revoke-runner-certificates.mjs). It revokes only a Development certificate that is in that runner’s keychain and was issued in the last week, never a Distribution certificate or a Mac’s own. If an archive fails with “Your account has reached the maximum number of certificates”, the Account Holder revokes the old Development certificates at developer.apple.com (Certificates, IDs & Profiles), keeping the ones for Macs still in use.

The version is x.y.z with an optional suffix. Every prerelease of 0.2.0 shares the 0.2.0 TestFlight version, and the build number orders them.

What verification checks

web/scripts/build-apple-testflight.sh <app> build opens the exported IPA and fails unless:

  • the bundle ID is the app’s, and CFBundleShortVersionString and CFBundleVersion match the requested version and build number;
  • ITSAppUsesNonExemptEncryption is false, and for the iPhone NSLocalNetworkUsageDescription is present (the Apple TV app does not browse the local network: its LAN discovery pod is iPhone-only), and for the Apple TV the compiled home-screen icon and both Top Shelf images are declared;
  • PrivacyInfo.xcprivacy is at the app bundle root, has NSPrivacyTracking false or absent and no tracking domains, declares UserDefaults with reason CA92.1, and every accessed-API entry has a category, at least one reason, and appears once (the entry count is not pinned; see Privacy manifest). Every declared category and its reasons are printed in the log and written to the evidence JSON;
  • the platform is the app’s (iphoneos or appletvos), every Mach-O in the app bundle (executable, frameworks, dylibs, and plug-ins) is arm64 only and not built for a simulator, and codesign verifies.

The Apple TV home-screen icon and Top Shelf images come from make brand-assets, which draws them from the brand contract at the exact sizes tvOS requires; @react-native-tvos/config-tv builds the asset catalog from them (appleTVImages in web/apps/tv/app.json).

It writes the IPA’s SHA-256 and the Xcode version next to the IPA, and the workflow keeps both as an artifact for 30 days. validate and upload re-check the IPA against that SHA-256 and refuse any other bytes.

An ITMS-90111 result means Apple’s ingestion rejected the runner’s Xcode or macOS as pre-release. The script names it and does not retry; wait for the xcode-27 image to ship a GM Xcode and macOS.

Privacy manifest

The audit of the autolinked iPhone pods found UserDefaults in expo-system-ui, which persists the root view colour there. The Apple TV app’s pods are a subset of the iPhone app’s, so the same audit and the same app-level declaration cover both apps. expo-system-ui ships its own ios/PrivacyInfo.xcprivacy declaring it, and the app declares NSPrivacyAccessedAPICategoryUserDefaults with reason CA92.1 at app level as well. react-native-tvos, expo-constants, and expo-file-system also carry their own manifests, expo-modules-core reads only a file size, and the other pods use no required-reason API. Repeat the audit when either app’s Apple pod set changes.

The shipped app manifest has more entries than the one the app declares. React Native’s pod install (add_aggregated_privacy_manifest in react-native/scripts/cocoapods/privacy_manifest_utils.rb) merges the required-reason APIs of every pod’s manifest, plus React Native core’s own (FileTimestamp with C617.1, UserDefaults with CA92.1, SystemBootTime with 35F9.1), into the app’s PrivacyInfo.xcprivacy. The IPA check therefore validates the manifest’s shape and the app’s own declaration, not its entry count.

Image-worker certification

Moved verbatim from design.md §22 (#1572). The image service’s design is in design/images.md.

The required worker is release infrastructure, so its gate is broader than unit codec coverage. make image-cert drives the installed loomarr-image executable through the same bounded protocol and manifest validation used by the application, then writes a machine-readable report under this worktree’s .artifacts/<instance>/ directory. It has two corpus modes:

  • With no arguments, it uses the repository’s deterministic certification corpus. That corpus covers opaque JPEG, transparent PNG, static WebP, animated GIF, APNG and WebP, finite and infinite loops, fractional and zero frame delays, a one-frame animated container, corrupt input, and every resource ceiling whose refusal is observable without allocating the forbidden resource.
  • make image-cert IMAGE_CERT_CORPUS=/absolute/read-only/path scans an operator’s existing raster corpus. It never modifies source files, follows no symlinks, performs no network I/O, and treats a supported-looking file that cannot complete inspection plus the requested ladder as a failure. Unsupported files are reported as skipped; stable budget refusals are reported separately from crashes, malformed manifests, and I/O failures.

Every accepted case produces an inspection plus a 320-pixel JPEG/WebP/AVIF ladder. Motion-preserving WebP is required for an animated source; JPEG and AVIF must be the first composited presentation frame. The certifier independently verifies the source hash, output signatures, dimensions, hashes, motion flag, and that staging is empty after each case. A run fails on a worker crash, protocol or manifest violation, unexpected refusal, leaked staging file, incorrect visible timeline, or a resource ceiling breach.

The deterministic gate is intentionally generous enough to survive shared CI hardware while still catching runaway work: each static case must complete within 10 seconds, each animated case within 30 seconds, and a worker process must remain below 768 MiB peak resident memory. The report records per-case wall time, source and output bytes, peak RSS where the host exposes it, plus p50/p95/max summaries. These are certification ceilings, not product SLOs; lowering them requires corpus evidence and raising them is a design change.

Production exposes the same boundary at /metrics: worker operations and stable outcomes, wall time, input/output bytes, peak RSS, queue wait, and in-flight count. Labels are bounded vocabulary (inspect/render and stable result classes), never an image hash, path, URL, MIME supplied by a caller, or free-form error text.

Those measurements are also the admission gate for another Rust capability. A proposal must name the production operation that dominates a captured worker-duration, queue-wait, peak-RSS, byte, or failure distribution, then reproduce it through the certification or benchmark seam. An operation does not move merely because it handles media or because a Rust implementation is possible. The Image service itself imports no Go image package; its deterministic corpus generator lives only in cmd/image-cert, and a source architecture test keeps that boundary closed. Filler-era frame hints remain in internal/media/tools: they sample at most 1,024 pixels per keyframe in background work and no captured Image-worker evidence identifies them as a bottleneck. Therefore V59b selects no next capability. A future measured case extends this worker protocol and resource boundary rather than creating a second Rust service.

make image-bench is the performance companion to certification. It drives the installed release worker through internal/media/images/rustgen with deterministic poster, backdrop, and icon sources and the complete AVIF width ladder for each role. Source creation and the capabilities self-test happen outside the timed region. After one warm-up per role, three runs record the recipe, host architecture and logical CPU count, process/Rendition counts, output bytes, complete-ladder throughput, p50/p95/max worker time, median ladder time, and maximum child peak RSS in a machine-readable report under the worktree artifact directory. The current baseline deliberately performs one worker request per missing AVIF Rendition, matching the background job shape that later batching will replace.

The benchmark is opt-in locally and manually dispatched on native amd64 and arm64 CI runners. It is not part of comprehensive verification and has no wall-clock pass/fail threshold: shared-runner timing is comparative evidence, not a product SLO or correctness gate. Comparisons are valid only for the same corpus, recipe, release profile, architecture, and CPU profile. make image-cert remains the authority for protocol, output, and resource-ceiling correctness.

The consumer half of the gate is observable through public seams. Guide programme art, Watch timeline art, and Filler still/hover art must carry a real Image record through their HTTP DTO and render through the shared frontend ServedImage primitive. An animated filler hover must offer an animated WebP rendition while its still fallback remains non-animated. Tests exercise those HTTP responses and rendered elements; querying image tables or asserting private renderer calls is not certification evidence.

Bundled ffmpeg pin

The image ships BtbN’s newest release-series GPL build of ffmpeg (nX.Y builds, never master’s N- builds), pinned by build id and per-architecture SHA256. BtbN keeps only its newest 14 daily releases and deletes each build’s dependency download cache after 14 days. Each pin is therefore published, together with its corresponding source, as a Loomarr third-party-sources-<date> release. The image and CI download the binaries from that release, never from BtbN.

scripts/ffmpeg-pin-refresh.sh does the whole update:

Terminal window
./scripts/ffmpeg-pin-refresh.sh check # newest BtbN build vs the pin; changes nothing
./scripts/ffmpeg-pin-refresh.sh dry-run .artifacts/ffmpeg-pin # download, verify, stage; no publish, no write
./scripts/ffmpeg-pin-refresh.sh publish .artifacts/ffmpeg-pin # publish the release, rewrite the pin

publish finds the BtbN run that published the newest autobuild and downloads both GPL archives, checking them against BtbN’s checksums. It also downloads the ffmpeg source archive at the build’s commit, the BtbN recipe archive at the commit the run built, and the run’s download-cache artifact. It fails if that artifact has expired. It then publishes the release, which is split into parts over GitHub’s 2 GiB asset limit, and checks every published asset’s digest. Last, it rewrites:

  • the five FFMPEG_* ARGs in the Dockerfile;
  • the ffmpeg entry in project/evidence/redistribution-manifest-v1.json;
  • project/evidence/source-distribution-<date>.json and project/evidence/ffmpeg-pin-<date>.md;
  • the ffmpeg-pin block in THIRD_PARTY_NOTICES.md.

internal/platform/releaseverify fails unless the pinned build’s source release is recorded with both binaries, the source, the recipe and the cache.

The ffmpeg-pin-refresh.yml workflow runs it every Monday and on dispatch (dry_run stages only). When BtbN has a newer build, the workflow publishes the release and pushes deps/ffmpeg-<build>. It then opens a PR and dispatches ci.yml with scope=full, because a bot push triggers no CI. That CI run builds the image on amd64 and arm64 and runs make test-ffmpeg against the image’s own ffmpeg, which includes the software-encode picture checks (TestLive_SoftwareRungsProduceAPictureInOneFormat fails on a black picture). A new release series opens as a draft labelled ready-for-human; it merges only after explicit approval.

CI has no GPU. Before merging a pin PR, run the GPU checks on a GPU host, one at a time, with each GPU command under flock /tmp/loomarr-gpu.lock:

  1. Put the pinned ffmpeg first on PATH. Then run make test-ffmpeg against it, and once against the previous pin.
  2. Encode the packager’s arguments (packagerItemArgs) for the software and hardware paths, SDR and HDR, from a non-black 1080p source. Check the output with signalstats: YAVG and YMAX must be well above 16 and match the previous pin. Look at a saved frame from each path.