Publishing a release
For: maintainers cutting a release.
You’ll get: what a v* tag publishes, and which workflow owns each part.
Server image publication and GitHub Release publication intentionally run in separate workflows from
the same v* tag. The image workflow owns package write and keyless-signing permissions. The release
notes workflow owns only contents: write; it cannot build, sign, or promote an image.
One-time repository setup
Add a dedicated OpenRouter key as the masked Actions secret used only by release notes:
gh secret set OPENROUTER_RELEASE_API_KEY --repo loomarr/loomarrThe default model is openai/gpt-5-mini. To change it without editing the workflow, set the optional
repository variable RELEASE_NOTES_MODEL to an OpenRouter model that supports strict structured
outputs. The release process is designed for a small classification call, not model-authored prose.
Add the TMDB key Loomarr ships with (#2151 decision 51). Register it as Loomarr’s own application at
TMDB, not a personal key. The release build stamps it into the binary through a BuildKit secret, so it
never appears in the image history or the provenance. A household’s own key, from Settings or
TMDB_API_KEY, always wins over it. Without the secret, a release ships no key, and TMDB features wait
until the household adds one.
gh secret set TMDB_BUNDLED_KEY --repo loomarr/loomarrPreview before tagging
Authenticate gh, export the release-specific key locally, and generate a preview for the proposed
tag. The tag may already exist or be a commit-ish accepted by GitHub’s generated-notes endpoint.
export OPENROUTER_RELEASE_API_KEY='...'make release-notes-preview TAG=v0.2.0 PREVIOUS_TAG=v0.1.0-beta.1The default output is .artifacts/release-notes-v0.2.0.md. Set OUTPUT=/path/to/notes.md to choose
another destination. Do not commit or paste the key. A tag-specific
project/release/<tag>.md file, when present, is prepended for human-authored framing and known
limitations.
Check a candidate on real data and real GPUs
Do both before a final tag.
Rehearse the migrations. A final tag (one without -rc.) is refused until this has passed on
its exact commit. Run it where the household backup lives, so the data stays there:
make migration-rehearsal BACKUP=<backup directory> IMAGE=ghcr.io/loomarr/loomarr-staging:<full commit id> RECORD=1It boots the candidate on a copy of the backup and reports whether it came up. Then it stops the
candidate and checks the migrated copy. SQLite’s integrity check and foreign-key check must both
come back clean. It also lists every table whose row count changed. Compare that list with what this
release’s migrations meant to delete before you tag. It needs sqlite3. The backup is only read,
and the container has no network, because the database names the household’s real services.
RECORD=1 then sets the loomarr/migration-rehearsal commit status, which release.yml checks.
The status says how long the migration took and nothing about the backup. Dispatching
staging.yml from a commit builds that commit’s image.
Check the hardware. Software, Intel and NVIDIA block a release; AMD is best-effort. Software is
covered by required CI. For the others, dispatch playout-bench.yml with run_arc and
run_geforce, then run:
make hardware-gate COMMIT=<full commit id>A skipped leg counts as missing. release.yml runs the same check and refuses a final tag until
both legs have passed on its commit; a -rc. tag doesn’t need them.
Docs pass
The docs ship with every release (#1682, maintainer decision D3; step 5 of #1572). Two things happen without any extra action:
- The docs site (Pages workflow) redeploys on every
v*tag push, not only on pushes tomain. It rendersdocs/**in place, so the published site always matches the tagged commit’s documentation, even for a release that changed no docs. - The in-app Help set is
//go:embed-ded into the server binary (docs/embed.go), and the release image (release.yml) builds from the tagged source. Help is current the moment the image is current — there is no separate build step.
The gate runs automatically on the tag. The Release notes workflow
runs scripts/check-release-docs-gate.sh right after it renders the notes and before it creates
the GitHub Release. It fails that workflow — publishing no Release — when the rendered notes
contain a New Features, Improvements, Bug Fixes, or Security Fixes entry and the tag range
touched nothing under docs/, docs-site/, or README.md. It reuses the same seven-category
classification the release notes already show (cmd/release-notes) rather than a second
taxonomy, and it is unconditional like every other step in that workflow: fixing the docs, or
the PR classification that misled it, is the only way past it. A release that only ships
Documentation, Dependencies, or Maintenance changes is not held to a docs change it has no
user-facing reason to need. release.yml, the separately hardened image-publication workflow, is
unchanged — it still owns only build, sign, and promote, with no added step or job.
What still needs a human: catch it before tagging, not after a failed Release publication. Generate the preview notes and run the same gate locally over the same range:
make release-notes-preview TAG=v0.2.0 PREVIOUS_TAG=v0.1.0-beta.1make docs-release-gate TAG=v0.2.0 PREVIOUS_TAG=v0.1.0-beta.1See scripts/check-release-docs-gate.sh for the gate
itself, shared by both the local preview and the workflow.
What the model can and cannot do
The helper first asks GitHub to generate the exact merged-PR list, contributor list, and compare link. It sends only each PR number and title to OpenRouter. The structured response is one closed object: every exact PR number is a required property, and its value must be one of these fixed sections. That shape encodes one assignment per PR even for unusually large releases instead of asking the model to repeat PR numbers across seven independent arrays:
- New Features
- Improvements
- Bug Fixes
- Security Fixes
- Documentation
- Dependencies
- Maintenance
Repository code renders GitHub’s original bullet for each assignment. It independently rejects unknown JSON fields, invented PRs, duplicate keys, omitted PRs, invalid categories, malformed output, and unrecognized GitHub change lines. It retries inference three times and then fails without creating a GitHub Release. It never silently publishes uncategorized or model-authored notes.
Tag and verify
Before requesting remote release certification, test the exact current main commit on the
maintainer’s local machine. Record the commit, commands, and results with the release evidence. At
minimum, run every release-relevant gate this host supports: Go/Rust contracts and tests, Postgres,
web unit/build, visual/e2e/tuner browser evidence, shared clients, both Expo Android app builds,
legacy Android TV including its release bundle contract, image-worker certification, release
policy verification, and the docs pass (make docs-release-gate). Host-incompatible
evidence such as iOS/tvOS and native arm64 image builds must be named explicitly and remain
required in protected CI; a local Linux pass cannot stand in for them. Agent sessions still never
run the maintainer’s live-stack make smoke* targets.
Push the protected version tag only after that local evidence is green, the required commit gates
are green, and the exact current main commit has passed the proportional release-candidate scope:
gh workflow run ci.yml --repo loomarr/loomarr --ref main -f scope=release-candidateThat scope runs repository contracts, real-codec image-worker certification, and the native amd64
and arm64 image builds. Each image build also compares the packaged LICENSE and
THIRD_PARTY_NOTICES.md with the tagged source, then runs the unchanged make test-ffmpeg gate
against the ffmpeg and ffprobe copied out of that image (#661). That ties the gate to the exact
bytes the published GPL source releases correspond to. It deliberately leaves unrelated platform and UI matrices to their normal
change-based CI. A normal push run or -f scope=full run is never accepted as Docker-release
evidence, even when green, because neither proves that publication is isolated from those unrelated
jobs. Use -f scope=full only when a complete manual rerun is independently required.
Both release workflows start from the tag. If OpenRouter is temporarily unavailable, rerun the failed Release notes workflow after service recovers; the separately hardened image publication is unaffected.
Before signing, the publish helper (scripts/publish-release-image.sh) refuses an image index unless
both platform images carry a non-empty SPDX SBOM and a build provenance whose every recorded source
revision is the tagged commit of this repository. After both workflows finish, verify the GitHub
Release body (including its links to the third-party source releases named in
THIRD_PARTY_NOTICES.md), the GHCR manifest, signature, SBOM, and provenance against the tagged
commit. The tag-specific header remains the place to state limitations
that cannot be derived from pull requests.
Apple TestFlight
The Apple TestFlight betas workflow (.github/workflows/apple-testflight.yml) builds, verifies,
and uploads the two permanent-identity Apple apps. Each is its own App Store Connect record:
| App | app | Record | Bundle ID | Platform | Variable |
|---|---|---|---|---|---|
| iPhone (ADR 0042) | iphone | Loomarr | media.loomarr.mobile | iphoneos | ASC_APPLE_APP_ID |
| Apple TV | apple-tv | Loomarr TV | media.loomarr.tv | appletvos | ASC_APPLE_TV_APP_ID |
It runs only on main and uses the ios-testflight environment, whose deployment branch policy
releases the secrets only to main. It never runs for a pull request or a merge group.
Every push to main that changes an Apple app uploads a build of that app with no manual step. A
secretless plan job decides, using the classifier CI uses for the Apple mobile and Apple TV gates
(scripts/ci-impact.sh, apple_mobile and apple_tv), so a push that changes neither app uploads
nothing, and a push that changes both uploads both. As in CI, a push whose diff cannot be established
counts as a change to both. The pushed build’s version is the newest v* tag reachable from the
commit (0.2.0-beta.8 until v0.2.0-beta.9 is tagged); TestFlight shows only its x.y.z.
The release job runs once per planned app, from a matrix over the plan’s list. One app’s failure
does not cancel the other’s upload. Release runs of one app queue one at a time. A newer pending one
replaces an older one, and it builds a later main that already contains the older change.
One script, web/scripts/build-apple-testflight.sh <app>, archives, exports, verifies, validates,
and uploads either app; its table of what differs between them (Expo package, bundle ID, Xcode
scheme, destination, platform, and record variable) is the only per-app code. Both app configs take
the release identity from web/scripts/apple-testflight-config.cjs.
The environment has no required reviewer (a maintainer decision): the reviewed, merge-queue-tested
merge is the review. release-verify pins the triggers, inputs, both jobs, the matrix, the
environment, and every script step’s command, environment, and condition
(internal/platform/releaseverify/apple_testflight_workflow.go).
One-time Apple setup
- Enrol in the Apple Developer Program and note the 10-character Team ID.
- Register each app’s App ID (
media.loomarr.mobile,media.loomarr.tv). Automatic signing can register it, but doing it first removes one failure mode. - Create each App Store Connect app record by hand (name, language, bundle ID, SKU). Note its numeric Apple ID; App Store Connect cannot create the record from the workflow.
- Create a Team API key (Users and Access, Integrations; Admin only). An individual key cannot use
provisioning endpoints. Download the
.p8once. Apple’s cloud signing then holds the distribution certificate, so the workflow needs no certificate, profile, or keychain. If a role below Admin cannot cloud-sign, use Admin. One key serves both records. - Add internal testers in TestFlight, in a group on each record.
Repository setup
Create the ios-testflight environment and limit its deployment branches to main; it has no required
reviewer. Then set:
| Name | Kind | Value |
|---|---|---|
ASC_API_KEY_P8_BASE64 | secret | base64 -w0 AuthKey_<id>.p8 |
ASC_API_KEY_ID | secret | the key’s 10-character ID |
ASC_API_ISSUER_ID | secret | the issuer UUID |
APPLE_TEAM_ID | variable | the Team ID |
ASC_APPLE_APP_ID | variable | the iPhone record’s numeric Apple ID |
ASC_APPLE_TV_APP_ID | variable | the Apple TV record’s numeric Apple ID |
The credentials step, which runs after dependency installation and before the archive, fails with a message naming every value its app is missing. An app needs only its own record’s variable.
Dispatch
gh workflow run apple-testflight.yml --repo loomarr/loomarr --ref main -f app=apple-tv -f version=0.2.0-beta.9gh workflow run apple-testflight.yml --repo loomarr/loomarr --ref main -f app=iphone -f version=0.2.0-beta.9 -f upload=trueDispatch for a dry run or a re-upload of one app; it rebuilds the current main with the version you
name. With upload unset (the default) it signs, exports, verifies, and runs altool --validate-app
without uploading, and it logs xcodebuild -version and sw_vers. App Store Connect picks the build
number: one past the highest build the record already holds (web/scripts/asc-next-build-number.mjs),
with github.run_number only as a floor, so a re-run or an upload made elsewhere never collides.
Each hosted Mac starts with an empty keychain, so the archive’s automatic signing creates a new Apple
Development certificate on every run. Once the export is done (or has failed), the build revokes it
(web/scripts/asc-revoke-runner-certificates.mjs). It revokes only a Development certificate that is in
that runner’s keychain and was issued in the last week, never a Distribution certificate or a Mac’s own.
If an archive fails with “Your account has reached the maximum number of certificates”, the Account
Holder revokes the old Development certificates at developer.apple.com (Certificates, IDs & Profiles),
keeping the ones for Macs still in use.
The version is x.y.z with an optional suffix. Every prerelease of 0.2.0 shares the 0.2.0 TestFlight
version, and the build number orders them.
What verification checks
web/scripts/build-apple-testflight.sh <app> build opens the exported IPA and fails unless:
- the bundle ID is the app’s, and
CFBundleShortVersionStringandCFBundleVersionmatch the requested version and build number; ITSAppUsesNonExemptEncryptionisfalse, and for the iPhoneNSLocalNetworkUsageDescriptionis present (the Apple TV app does not browse the local network: its LAN discovery pod is iPhone-only), and for the Apple TV the compiled home-screen icon and both Top Shelf images are declared;PrivacyInfo.xcprivacyis at the app bundle root, hasNSPrivacyTrackingfalse or absent and no tracking domains, declaresUserDefaultswith reasonCA92.1, and every accessed-API entry has a category, at least one reason, and appears once (the entry count is not pinned; see Privacy manifest). Every declared category and its reasons are printed in the log and written to the evidence JSON;- the platform is the app’s (
iphoneosorappletvos), every Mach-O in the app bundle (executable, frameworks, dylibs, and plug-ins) isarm64only and not built for a simulator, andcodesignverifies.
The Apple TV home-screen icon and Top Shelf images come from make brand-assets, which draws them
from the brand contract at the exact sizes tvOS requires; @react-native-tvos/config-tv builds the
asset catalog from them (appleTVImages in web/apps/tv/app.json).
It writes the IPA’s SHA-256 and the Xcode version next to the IPA, and the workflow keeps both as an
artifact for 30 days. validate and upload re-check the IPA against that SHA-256 and refuse any
other bytes.
An ITMS-90111 result means Apple’s ingestion rejected the runner’s Xcode or macOS as pre-release. The
script names it and does not retry; wait for the xcode-27 image to ship a GM Xcode and macOS.
Privacy manifest
The audit of the autolinked iPhone pods found UserDefaults in expo-system-ui, which persists the
root view colour there. The Apple TV app’s pods are a subset of the iPhone app’s, so the same audit
and the same app-level declaration cover both apps. expo-system-ui ships its own ios/PrivacyInfo.xcprivacy declaring it, and the app
declares NSPrivacyAccessedAPICategoryUserDefaults with reason CA92.1 at app level as well.
react-native-tvos, expo-constants, and expo-file-system also carry their own manifests,
expo-modules-core reads only a file size, and the other pods use no required-reason API. Repeat the
audit when either app’s Apple pod set changes.
The shipped app manifest has more entries than the one the app declares. React Native’s pod install
(add_aggregated_privacy_manifest in react-native/scripts/cocoapods/privacy_manifest_utils.rb) merges
the required-reason APIs of every pod’s manifest, plus React Native core’s own (FileTimestamp with
C617.1, UserDefaults with CA92.1, SystemBootTime with 35F9.1), into the app’s
PrivacyInfo.xcprivacy. The IPA check therefore validates the manifest’s shape and the app’s own
declaration, not its entry count.
Image-worker certification
Moved verbatim from design.md §22 (#1572). The image service’s design is in
design/images.md.
The required worker is release infrastructure, so its gate is broader than unit codec coverage.
make image-cert drives the installed loomarr-image executable through the same bounded protocol
and manifest validation used by the application, then writes a machine-readable report under this
worktree’s .artifacts/<instance>/ directory. It has two corpus modes:
- With no arguments, it uses the repository’s deterministic certification corpus. That corpus covers opaque JPEG, transparent PNG, static WebP, animated GIF, APNG and WebP, finite and infinite loops, fractional and zero frame delays, a one-frame animated container, corrupt input, and every resource ceiling whose refusal is observable without allocating the forbidden resource.
make image-cert IMAGE_CERT_CORPUS=/absolute/read-only/pathscans an operator’s existing raster corpus. It never modifies source files, follows no symlinks, performs no network I/O, and treats a supported-looking file that cannot complete inspection plus the requested ladder as a failure. Unsupported files are reported as skipped; stable budget refusals are reported separately from crashes, malformed manifests, and I/O failures.
Every accepted case produces an inspection plus a 320-pixel JPEG/WebP/AVIF ladder. Motion-preserving WebP is required for an animated source; JPEG and AVIF must be the first composited presentation frame. The certifier independently verifies the source hash, output signatures, dimensions, hashes, motion flag, and that staging is empty after each case. A run fails on a worker crash, protocol or manifest violation, unexpected refusal, leaked staging file, incorrect visible timeline, or a resource ceiling breach.
The deterministic gate is intentionally generous enough to survive shared CI hardware while still catching runaway work: each static case must complete within 10 seconds, each animated case within 30 seconds, and a worker process must remain below 768 MiB peak resident memory. The report records per-case wall time, source and output bytes, peak RSS where the host exposes it, plus p50/p95/max summaries. These are certification ceilings, not product SLOs; lowering them requires corpus evidence and raising them is a design change.
Production exposes the same boundary at /metrics: worker operations and stable outcomes, wall
time, input/output bytes, peak RSS, queue wait, and in-flight count. Labels are bounded vocabulary
(inspect/render and stable result classes), never an image hash, path, URL, MIME supplied by a
caller, or free-form error text.
Those measurements are also the admission gate for another Rust capability. A proposal must name
the production operation that dominates a captured worker-duration, queue-wait, peak-RSS, byte, or
failure distribution, then reproduce it through the certification or benchmark seam. An operation
does not move merely because it handles media or because a Rust implementation is possible. The
Image service itself imports no Go image package; its deterministic corpus generator lives only in
cmd/image-cert, and a source architecture test keeps that boundary closed. Filler-era frame hints
remain in internal/media/tools: they sample at most 1,024 pixels per keyframe in background work and
no captured Image-worker evidence identifies them as a bottleneck. Therefore V59b selects no next
capability. A future measured case extends this worker protocol and resource boundary rather than
creating a second Rust service.
make image-bench is the performance companion to certification. It drives the installed release
worker through internal/media/images/rustgen with deterministic poster, backdrop, and icon sources and
the complete AVIF width ladder for each role. Source creation and the capabilities self-test happen
outside the timed region. After one warm-up per role, three runs record the recipe, host architecture
and logical CPU count, process/Rendition counts, output bytes, complete-ladder throughput,
p50/p95/max worker time, median ladder time, and maximum child peak RSS in a machine-readable report
under the worktree artifact directory. The current baseline deliberately performs one worker request
per missing AVIF Rendition, matching the background job shape that later batching will replace.
The benchmark is opt-in locally and manually dispatched on native amd64 and arm64 CI runners. It is
not part of comprehensive verification and has no wall-clock pass/fail threshold: shared-runner timing is comparative
evidence, not a product SLO or correctness gate. Comparisons are valid only for the same corpus,
recipe, release profile, architecture, and CPU profile. make image-cert remains the authority for
protocol, output, and resource-ceiling correctness.
The consumer half of the gate is observable through public seams. Guide programme art, Watch
timeline art, and Filler still/hover art must carry a real Image record through their HTTP DTO and
render through the shared frontend ServedImage primitive. An animated filler hover must offer an
animated WebP rendition while its still fallback remains non-animated. Tests exercise those HTTP
responses and rendered elements; querying image tables or asserting private renderer calls is not
certification evidence.
Bundled ffmpeg pin
The image ships BtbN’s newest release-series GPL build of ffmpeg (nX.Y builds, never master’s
N- builds), pinned by build id and per-architecture SHA256. BtbN keeps only its newest 14 daily
releases and deletes each build’s dependency download cache after 14 days. Each pin is therefore
published, together with its corresponding source, as a Loomarr third-party-sources-<date> release.
The image and CI download the binaries from that release, never from BtbN.
scripts/ffmpeg-pin-refresh.sh does the whole update:
./scripts/ffmpeg-pin-refresh.sh check # newest BtbN build vs the pin; changes nothing./scripts/ffmpeg-pin-refresh.sh dry-run .artifacts/ffmpeg-pin # download, verify, stage; no publish, no write./scripts/ffmpeg-pin-refresh.sh publish .artifacts/ffmpeg-pin # publish the release, rewrite the pinpublish finds the BtbN run that published the newest autobuild and downloads both GPL archives,
checking them against BtbN’s checksums. It also downloads the ffmpeg source archive at the build’s
commit, the BtbN recipe archive at the commit the run built, and the run’s download-cache
artifact. It fails if that artifact has expired. It then publishes the release, which is split into
parts over GitHub’s 2 GiB asset limit, and checks every published asset’s digest. Last, it rewrites:
- the five
FFMPEG_*ARGs in theDockerfile; - the ffmpeg entry in
project/evidence/redistribution-manifest-v1.json; project/evidence/source-distribution-<date>.jsonandproject/evidence/ffmpeg-pin-<date>.md;- the
ffmpeg-pinblock inTHIRD_PARTY_NOTICES.md.
internal/platform/releaseverify fails unless the pinned build’s source release is recorded with both
binaries, the source, the recipe and the cache.
The ffmpeg-pin-refresh.yml workflow runs it every Monday and on dispatch (dry_run stages only).
When BtbN has a newer build, the workflow publishes the release and pushes deps/ffmpeg-<build>. It
then opens a PR and dispatches ci.yml with scope=full, because a bot push triggers no CI. That
CI run builds the image on amd64 and arm64 and runs make test-ffmpeg against the image’s own
ffmpeg, which includes the software-encode picture checks
(TestLive_SoftwareRungsProduceAPictureInOneFormat fails on a black picture). A new release series
opens as a draft labelled ready-for-human; it merges only after explicit approval.
CI has no GPU. Before merging a pin PR, run the GPU checks on a GPU host, one at a time, with each
GPU command under flock /tmp/loomarr-gpu.lock:
- Put the pinned ffmpeg first on
PATH. Then runmake test-ffmpegagainst it, and once against the previous pin. - Encode the packager’s arguments (
packagerItemArgs) for the software and hardware paths, SDR and HDR, from a non-black 1080p source. Check the output withsignalstats: YAVG and YMAX must be well above 16 and match the previous pin. Look at a saved frame from each path.