CI
For: contributors whose pull request is waiting on checks. You’ll get: how CI picks what to run, and the one check that gates main.
Each family contains independently filtered jobs. The required aggregate always runs and checks every top-level result, including jobs that correctly skipped.
Fast PR feedback, authoritative queued integration
Ordinary pull-request pushes run the always-on policy bootstrap plus affected repository-contract, static-analysis,
compile/type, unit, documentation, and shared-client feedback and return the required CI
result quickly. They do not run race-policy shards, Postgres conformance, Playwright, release-image
builds, runtime image certification, Apple mobile, Apple TV, Android TV, Expo Android mobile, the
tuner matrix, or the macOS harness.
The required main merge queue then builds at most eight cumulative candidates concurrently against
the current base. A queued PR waits only for merge-gate.yml, which reports the required CI check after proving
the merged tree compiles, so it merges in minutes. merge-gate-cache.yml refreshes its Go cache from
main at most hourly and keeps one entry. Beside it, the merge_group run of ci.yml is the
complete integration lane, reported as CI assurance: the same fail-closed classifier selects the
complete affected gate set, and every selected result remains a dependency of that job. It finishes
after the merge, so its evidence no longer blocks one. When it fails on a commit that reached main it
files an issue. Revert first: anyone may revert the commit to get main green, and the fix lands as a
new PR. Rerun instead when the failure is a flake or an outage. Every queue entry keeps its own run and
none is cancelled, because a run selects gates from its own PR’s diff and a later run can’t replace it.
This is a maintainer decision (2026-11-07): merge latency over pre-merge evidence. Explicit
manual runs retain release-candidate and full recovery scopes plus an isolated Apple compilation
cache portability scope. A normal queue-produced push to
main runs publication workflows only rather than validating the admitted commit again.
CodeQL is a separate security workflow because its unit of selection is a language database, not a
product gate. Pull requests and pushes to main diff from a trustworthy base and scan only changed
CodeQL languages. A workflow/classifier edit, an unknown path, or an unusable base fails closed to all
six repository languages. The Wednesday schedule and every manual dispatch always scan all six, so
selective change feedback cannot erode periodic repository coverage. The repository uses CodeQL
advanced setup for this path-aware policy; enabling GitHub’s default setup would disable the checked-in
workflow and restore an all-language scan on every change.
The root workflow owns triggering, classification, admission, manual scopes, and the required aggregate. The lightweight CI policy job always runs independently of classifier output, so a broken classifier cannot skip its own verification. Product job implementations live in family-named reusable workflows, so an edit to one family selects only its owning product gate. Classifier, admission, and release-evidence control-plane code selects policy rather than a product build that does not execute it. Agent harness and design-contract prose likewise select only their real consumers. They do not select unrelated Rust, Android, Apple, image, browser, frontend, Postgres, or application-Go builds. The root Make interface and unknown paths still select everything.
The Go repository-contract job keeps the stable make privacy-verify interface, but its captured
fixture guard is one in-process tracked-tree scan in releaseverify. The previous shell loop
started hashing processes for every unique candidate and measured 54 seconds locally by itself;
the same case-folded SHA-256 policy now measures about two seconds, reports only labels and digests,
and keeps household-name/profile-PIN candidates restricted to the four audited response fixtures.
Make uses the same ownership boundary. The root Makefile is the stable interface for shared
variables, help, and ordered mk/*.mk includes; each module owns one command family. The generated
command reference follows those includes and rejects missing, cyclic, escaping, or duplicate target
definitions, so splitting the physical files does not split the public command catalog.
This is admission control, not weaker assurance. A pull request cannot merge directly after its fast
result: it must enter the queue and pass the generated current-base commit. The queue uses squash
merges, ALLGREEN, eight concurrent builds, one PR per merge, and a three-hour check-response
timeout. The concurrency cap is eight by maintainer decision on 2026-11-07, so a burst of queued PRs
clears together instead of waiting in pairs. That accepts more CI minutes, and a larger invalidation
cost when an early entry fails, because every cumulative build behind it restarts. Apple runners are
the known pressure point: a burst of builds that select the Apple gates competes for the same macOS
capacity. The queue still gives accepted work a stable place in line instead of repeatedly
invalidating successful strict-mode runs. scripts/ci-merge-queue-policy.sh checks the live ruleset;
APPLY=1 updates only its build-concurrency field and then re-checks every admission parameter.
The policy has three coupled halves:
- GitHub ruleset Main merge queue admission targets
refs/heads/main. - Branch protection applies the strict, Actions-owned
CIrequirement to administrators as well as ordinary contributors, so removing post-merge validation does not create a bypass path. .github/workflows/ci.ymltriggers onpull_request,merge_group, and explicit manual dispatch; it does not trigger product validation for a normalmainpush.
releaseverify.VerifyCINativeAdmission rejects loss of the queue trigger, renewed PR admission, a
queue-only condition that drops manual evidence, a restored post-merge product trigger, or a missing
scarce-capacity job. The live ruleset is verified through scripts/ci-merge-queue-policy.sh and
GitHub’s branch-protection API when changing repository protection.
Jobs run only when their inputs changed
A changes job diffs against the merge base and each job gates on its output. It fails safe: no
usable merge base — first push, force-push, new branch — runs everything.
scripts/ci-diff-base.sh resolves that base from the event payload. A PR diffs against its base
commit; a merge-queue batch diffs against merge_group.base_sha, the main commit it is stacked on,
so the queue selects the same gates as the PR it carries and a docs-only or Go-only batch never
waits on the native client builds. scripts/ci-diff-base-test.sh proves both, end to end through
the classifier.
Adding a new build input means adding it to the filter in the same PR.
Three non-obvious entries: the help pages (docs/get-started.md, docs/guides/,
docs/explanation/) are in the Go filter because they are embedded and the doc-claims test reads them, internal/filler/location/ contains generated indexes embedded
in the backend, and scripts/ is there because the job executes them.
Specialized gate classifier activates one job at a time
The changes job also runs scripts/ci-impact.sh and publishes impact_* outputs for the
specialized contract, Go, full-Go, Rust, Postgres, web, shared-client, iOS, tvOS, Expo
Android mobile, Expo Android TV, visual, e2e, tuner, image, docs, agent, and legacy Android TV
gates. Its run summary places those proposed decisions beside the current broad families.
Classification follows execution ownership. A product gate is selected when a changed path is an input to the product or to the gate implementation that executes it. A path that only decides which gates run, admits a release, or interprets existing evidence is control-plane policy and does not borrow an unrelated product build as proof. Unknown paths still select every gate. Local edit-loop test planning is a separate interface: it shares changed-path collection but deliberately chooses fast direct or related unit tests rather than authoritative CI gates.
Browser-only test-card fixtures under web/packages/fixtures/src/testcard/ select browser unit,
visual, and image evidence, not native client builds. Other shared fixture paths remain native inputs.
Postgres was the first active specialized output. store-postgres consumes impact_postgres
directly while remaining in the required CI aggregate. The explicit release-candidate scope
continues to exclude database conformance. This first activation intentionally treats every .go
file plus go.mod, go.sum, store migrations, and unknown paths as Postgres-sensitive. That
conservative boundary skips proven
non-Go over-selection without guessing which transitive Go dependency can change a real-Postgres
assertion. Dependency-aware narrowing is a later shadow change.
Container acquisition is bounded before product tests
Container acquisition has one retry boundary: scripts/ensure-container-image.sh. A locally
inspected image returns without a pull. Otherwise the helper makes at most five pull attempts with
exactly two seconds between attempts, fails immediately if a wait cannot run, and propagates the
fifth pull failure status. The retry ends before docker run or testcontainers starts product tests;
product execution is never retried.
The Playwright runner scripts/run-playwright-container.sh owns its fixed image identity. Each Make
target first invokes its bounded ensure mode, then one product mode. Its reviewed source builds
Docker arguments as a shell array, places the image at a fixed boundary, and accepts only a mode plus
the validated LOOMARR_PLAYWRIGHT_SHARD=--shard=N/M visual-suite input. The Postgres pin has one
Make-readable and Go-embedded authority at
internal/testkit/postgresimage/image.txt; direct Postgres-module Run calls must use
postgresimage.Name(). Release verification structurally pins that owner to one private
//go:embed image.txt string and the single strings.TrimSpace(image) return; missing, transformed,
environment-derived, or alternate implementations fail closed. Both owner and data-file changes
select the contracts, Go, full-Go, Postgres, image, and policy gates. The package must contain
exactly one untagged non-test production Go file, image.go; build constraints, another production
file, or a second embed/Name owner fail closed, while _test.go files do not participate in
production ownership. The verifier scans non-generated Go throughout the repository, including
cmd/, tooling, and future top-level packages while excluding only vendor and named generated/cache
trees. Deprecated postgres.RunContainer is forbidden because its image cannot use the repository
authority. postgres.Run admits only the repository’s direct database, username, password, and wait
customizers; opaque, variadic, nested, image-replacing, pull-policy, or shadowed-package customizers
are rejected. Generic testcontainers Run admits no customizers. GenericContainer, request
aliases, pointer aliases, request builders, and construction-time or later ContainerRequest.Image
assignments accept an explicit unrelated image literal, but Postgres-bearing or opaque values must
use that authority. AlwaysPullImage and ImageSubstitutors are forbidden at construction or later
mutation so request objects cannot restore unbounded acquisition.
The direct AST audit classifies ordinary, multi-result, compound, and range-key/value writes to
whole requests or Image, including nested selectors, dereferences, and parentheses. A pointer to
either target may not cross an opaque call boundary, directly or through a tracked alias; this also
rejects direct reflection and unsafe mutation paths. Request or Image pointers may remain direct,
tracked local aliases, but cannot enter slices, arrays, maps, structs, interfaces, channels, returns,
or opaque aggregate calls; aggregate indexing or selection therefore cannot become an unaudited
mutation path. No pointer-taking authority seam is approved:
the Postgres authority is the value-returning postgresimage.Name(). This is a direct structural
contract, not a claim to infer arbitrary reflection, unsafe, return-value, or interprocedural
dataflow. Dereferencing a tracked pointer retains whether it targets a whole request or only Image,
so ordinary, compound, multi-result, and range writes cannot erase that authority distinction.
Taking acquisition functions as values is rejected because their arguments can no longer be audited.
The alternate exported acquisition surfaces are closed too: ParallelContainers and the
DockerProvider.CreateContainer, ReuseOrCreateContainer, and RunContainer methods are rejected
rather than accepting opaque requests or pull policies outside the audited Run and
GenericContainer seams.
Postgres image literals are also rejected outside count-bounded exact verifier-fixture values and the
one exact unrelated Authentik direct-Docker fixture value; no whole Go file is exempt.
Testcontainers’ cleanup image is separately pinned in
internal/testkit/ryukimage/image.txt, acquired through the same bounded helper, and exported to
testcontainers only by the exact test-pg recipe while Ryuk remains enabled. The Ryuk pin is a
fully-qualified docker.io name; the Postgres pin is its implicit library/... form because the
product recipe sets the exact non-empty
TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX=docker.io, which takes precedence over an operator’s
~/.testcontainers.properties. Testcontainers-go v0.44 prepends that runtime authority exactly
once to the implicit Postgres pin, while Docker resolves the helper’s same implicit form to Docker
Hub; an inherited prefix therefore cannot transform either product image into a different pull.
TESTCONTAINERS_RYUK_DISABLED=false likewise overrides a user property that would disable cleanup.
The Postgres and Ryuk recipes read their repository-owned pin files directly; no environment or
command-line Make variable can replace either authority. A
focused contract binds the qualified Ryuk file to the installed testcontainers module’s exported
default plus the exact Hub prefix, so a dependency update cannot silently introduce a second image.
Each product Make target depends directly on one shared ensure target.
Release verification requires the ensure and protected product targets to remain phony, checks each
protected target’s own exact recipe, and closes the full prerequisite graph over an exact
target/recipe allowlist: the five Playwright targets may
reach only their shared ensure target and their existing Storybook or frontend build prerequisite,
while test-pg may reach only its shared ensure target and the Rust worker build. A wrapper or
changed prerequisite therefore cannot hide another runtime, helper, service, pull, or alternate
container acquisition. The five Playwright recipes and their ensure target invoke only fixed runner
modes; removed PW_DOCKER_USER, PW_CI, PW_REAL_CI, PW_IMAGE, and PW_SHARD variables are
unexported and forbidden as Make assignments. Command-line, environment, or recursively expanded
values therefore have no pre-image argument seam. Make’s dependency graph executes the shared phony
ensure target once even when the Playwright targets are selected together with -j.
Every Make recipe that invokes the bounded helper, reviewed runner, a fixed
Docker/Podman/nerdctl/Buildah/docker-compose executable, or a variable engine with container grammar seeds
the workflow-protected target set, regardless of global options, Compose options, or the eventual
verb. GNU Make’s target: prerequisites ; recipe form is parsed as an inline recipe rather than as
prerequisite text. Backslash-newline recipe continuations are lexically spliced into one logical
command before that classification, including commands reached through reverse prerequisite closure.
Exact active literal Make assignments are resolved only for command classification. Outside recipes,
GNU Make’s first unescaped # begins a comment even when it is adjacent to a value or prerequisite;
an escaped \# remains data. Recipe and inline-recipe text uses the shell’s distinct comment rule,
where an adjacent hash remains part of its token. Audited non-recipe Make source rejects logical
continuations, define, and parse-time eval rather than partially interpreting those contexts. An
unresolved Make command variable followed by container grammar fails closed.
An unresolved variable in recipe command position also seeds the protected set, so a helper or
runner supplied through a Make command-line or environment override cannot hide behind
$(RUNNER). The root’s exact GO and CARGO assignments are resolved before that check, and
recursive $(MAKE) recipes are followed through the protected-target graph, so known benign tools
do not become container routes. Literal shell command substitutions inside recipes are traversed
through the same recursive-Make closure. Recursive Make admits only a positive numeric -jN or
--jobs=N parallelism option and no variable assignments; even a non-graph assignment can retarget
a command-position variable in a reachable recipe. Leading command-prefix assignments and
assignments consumed by nested env wrappers remain attached to the effective recursive Make
invocation, so wrappers cannot erase that fact. Alternate Makefiles, directories, includes,
evaluation, execution controls, dynamic option values, and every unknown option fail closed because
they can select a different graph or executable. The shared
command walker consumes the closed option grammar for env, command, exec, timeout, nice,
and xargs before classifying the effective executable; a wrapper operand cannot be mistaken for
the command. Shell eval bodies are recursively classified, with only the exact checked-in
dev-env.sh export import admitted for the image tooling recipes. Release verification never launches a host Make executable. A checked-in minimal
Make-policy fixture and golden normalization/closure snapshot preserve the security-relevant
adjacent/escaped-hash boundary, prerequisite graph edges, inline and continued recipes, shell-recipe
hash tokens, recursive protected-target reachability, and untrusted alternate-Makefile rejection.
The policy parser canonicalizes harmless surrounding whitespace, including the trailing space GNU
Make retains before a whitespace-delimited comment; the fixture is not a claim of byte-for-byte GNU
Make evaluation. Independent public mutations own continued assignment and define/eval rejection,
parse-time side effects, and untrusted recursive graphs. This keeps the required policy hermetic
across Linux, Homebrew, and the macOS system Make handoff.
This includes the existing local dev and dev-gpu Compose targets. Reverse prerequisites and
recursive Make recipes close that set transitively. Classification grants no new authority: a
workflow route to an arbitrary engine-bearing target is rejected; only the exact audited Postgres
and Playwright commands below may reach their separately verified image authorities.
Repository shell scripts invoked by a reachable Make or workflow route are indexed and traversed
recursively. Literal paths, transparent wrappers, prefix assignments, shell interpreters, and unique
repository-script suffixes share one bounded 32-edge, cycle-safe traversal. Docker, Podman, nerdctl,
Buildah, Compose, variable-engine acquisition grammar, if/then bodies, and input or output
process substitutions are classified without executing the script. A cycle is not acquisition by
itself, but any acquisition-bearing member poisons the complete reachable cycle.
The reusable Postgres workflow retains one exact make test-pg TEST_PG_SUITE=${{ matrix.lane }}
step, run once per suite: its run job’s matrix must list exactly store, backendtransition, and
app with fail-fast: false, so each suite gets its own runner (still -p=1 under the race
detector) and no suite can drop out of the gate. Unset, TEST_PG_SUITE runs all three. Across every
job and step, release verification admits only the current workflow, run job, and exact step
schemas. It rejects
job containers, services, unaudited reusable-workflow delegation, container-acquiring actions, unknown keys,
metadata-only or malformed steps, multiple execution mechanisms, and unapproved environment,
defaults, condition, error-tolerance, shell, working-directory, or MAKEFLAGS controls. The root
tool-version environment and the cache-save condition are admitted only at their exact current
values. Its only permitted shell steps are the exact cache-epoch command and the suite’s make test-pg; this
closes dry-run, variable, wrapper, and quoting indirection without parsing arbitrary shell. Every
repository workflow is also parsed structurally. Any service or job container or docker:// action
is rejected. Workflow shell is not interpreted: route-shaped steps are classified from the raw
scalar and statically declared workflow, job, and step environment, then compared with exact
workflow-name-and-whole-scalar authorities. The classifier recognizes Make by executable basename,
including absolute paths and wrappers, the bounded helper and Playwright runner wherever they occur,
and fixed Docker, Podman, nerdctl, Buildah, and Compose acquisition grammar across quoting and
command-substitution text. It applies the shell’s backslash-newline splice before classification, so
an engine, helper, runner, or Make executable cannot be split across YAML scalar lines. Command
and process substitutions are inspected recursively without evaluation; shell control words cannot
hide an engine command, and ambiguous acquisition-bearing syntax fails closed. Ambiguous nesting and a variable
executable inside any nested env, command, or interpreter command fail closed. Any variable in
executable position is rejected generically, including adjacent or braced construction and
invocation through exec, source, ., quotes, or wrappers. Legacy backtick substitution is not
evaluated or partially parsed: balanced, unbalanced, escaped, nested, wrapped, and CRLF forms all
fail closed unless the entire checked-in step matched an earlier source-bound authority. A
candidate step containing assignments, variable executables or targets, command substitution,
control operators, or any other added shell context is not evaluated and cannot match an authority;
it fails closed.
The only bounded test-image download routes are
the exact per-suite make test-pg step
in ci-postgres.yml, the exact visual shard invocation, and the exact make e2e step in
ci-playwright.yml; ci-image.yml retains one source-exact local packaged-image metadata inspection
that cannot pull an image. Release publication’s two existing manifest/signing scripts are also
source-bound exact authorities because their audited bodies use Docker buildx manifest commands;
they do not become test-image acquisition authorities and cannot be replaced by an arbitrary script.
Each bounded test-image authority is required exactly once at its declared
workflow, job, absolute steps[] index, step kind and name, whole command, target set, and inherited
execution context. Pinned action steps count in that absolute index, so inserting, removing, or
moving either a shell or action step cannot preserve an authority accidentally; removal,
duplication, movement, relabelling, swapping, or an extra Playwright mode fails repository verification.
Every registered run authority, including container-free harness and documentation commands, is
also required exactly once. Its job name, runner, dependencies, timeout, and typed strategy/matrix
shape are source-bound alongside the existing workflow/job/step environment, permissions,
condition, shell, and working-directory controls; absence, addition, or replacement changes the
authority rather than silently changing where it executes. Every pinned action occurrence in every
checked-in workflow is a complete exact authority at its workflow, job, and absolute step index.
Its step kind, name and id, full-SHA uses scalar, complete with inputs, condition, and environment
must match the checked-in authority; the repository inventory proves both that no action occurrence
is unregistered and that no registered workflow or occurrence disappeared. Action-only workflows
receive the same exact workflow/job environment, defaults, permissions, runner, dependency,
timeout, deployment-environment, and typed strategy/matrix audit. Integer authorities must remain
YAML integers rather than numerically equivalent quoted strings. An action-to-action or action-to-run
same-slot replacement is therefore rejected without resolving the action over the network.
Overrides or additional shell syntax make those routes non-exact and are rejected. Existing
unrelated Make steps are separately source-bound by workflow name, exact whole
scalar, and their explicit target list. They remain admitted only while the independent Make graph
proves every named target is outside the acquisition-bearing reverse closure; changing an Android,
cache-cleanup, Go, or other admitted target into a container route therefore invalidates the workflow
authority. This is a closed checked-in command grammar, not arbitrary shell data-flow interpretation.
Every protected and currently unrelated Make authority binds the workflow name, job name, exact run
scalar, absolute steps[] index, run-step kind and exact name, explicit target list, and complete
inherited workflow/job/step execution context together.
Workflow and job conditions, step conditions, legitimate step environments, and the packaged-image
inspection’s bash shell are admitted only at their exact current values. Any additional or changed
environment, defaults, condition, error tolerance, shell, or working directory rejects the authority.
Thus MAKEFLAGS, GNUMAKEFLAGS, MFLAGS, MAKEFILES, MAKEOVERRIDES, MAKELEVEL, MAKE, SHELL,
PATH, BASH_ENV, ENV, GOENV, GOFLAGS, and equivalent controls cannot redirect or suppress an
exact-looking command. An exact command is authorized only at the scalar node inside a real
jobs.<job>.steps[] entry whose complete inherited context passed this audit; matching text in a
malformed or metadata-only YAML shape receives no exemption.
Job-level uses: is rejected except for the root CI workflow’s exact local family calls. Those
calls admit exactly name, needs, if, and uses at their source-bound values and are closed by
VerifyCIFamilyWorkflows: each job must name its registered local workflow and the target must
expose only workflow_call. Product families contain exactly one implementation job. The isolated
Apple cache portability workflow is the sole registered exception: its exact producer and dependent
consumer jobs prove transfer across distinct runners. Every implementation job is scanned
independently by this container policy. Remote,
renamed, additional, or otherwise unresolved reusable workflows fail closed.
The workflow topology is itself an authority: the checked-in workflow filenames, job names, step
counts, and run-versus-action kind at each slot must match exactly. An extra workflow, job, or step
therefore fails before command classification; pinned actions and registered run authorities retain
their deeper exact content and context checks.
Topology, run commands, pinned actions, job contexts, reusable callers, and family-workflow paths
are constructed together through one typed catalog. Each verification receives fresh maps rather
than package-mutable authority state, and a drift contract rejects missing workflows, jobs, contexts,
actions, commands, or family registrations and colliding absolute step identities.
The cache-cleanup workflow is one such run authority: its complete deletion command, step name and
environment, exact Cache cleanup workflow name, closed pull_request trigger with only the
closed type, workflow permissions, job name and runner, and absence of alternate execution context
are source-bound rather than inferred from its one-step topology.
Make execution is closed over the checked-in root Makefile and every mk/*.mk: each module must be
included exactly once, and non-recipe syntax is limited to the current plain assignments, targets,
and exact root includes. Conditional directives are rejected rather than interpreted; define/endef,
dynamic evaluation, modifiers, alternate includes, and non-recipe logical continuations are also
forbidden. Recipe continuations are instead normalized as shell lexical splices and audited as one
logical command.
Every parenthesized or braced Make reference in assignments, targets, and every logical recipe is
structurally balanced. Escaped $$ recipe expansions remain shell-owned, while Make-owned references
are inspected before shell command classification. Ordinary variable references and the current pure
if, or, sort, subst, and wildcard functions are allowed;
shell, eval, file, Guile, != shell assignments, call, dynamically constructed function
names, value, foreach, and any other function are rejected even when nested or continued inside
an otherwise allowed assignment, recipe, or pure function. Playwright host-user discovery now occurs inside the exact reviewed runtime script,
so Make has no parse-time shell exception. The Postgres image pin remains one Go-embedded and
Make-readable file, and the Ryuk pin is a second repository-owned testkit file. Make reads both by
literal path with recipe-time cat immediately before the bounded helper rather than exposing an
overridable variable or exempting a parse-time file function.
GO ?= go and CARGO ?= cargo are fixed, and global shell, executable, flag, recipe-prefix, and
special-target controls that could suppress or redirect a protected recipe are forbidden.
The always-on ci-policy job has a source-bound bootstrap chain: exact pinned checkout and setup-go actions,
then go run ./cmd/releaseverify -root . as the first shell step, followed by the three exact Make/Go
policy commands. Its always() condition is source-bound and independent of impact-classifier
output. The workflow root admits only exact Go and Node version environment values and no
defaults; the job and all six steps admit no inherited environment, alternate shell, working
directory, tolerance, or unknown execution keys; the job’s existing needs, condition, and runner are
source-bound, and no step may add a condition. Setup-go’s exact inputs are also bound, so a preceding
action cannot redefine the Go executable or path. BASH_ENV, ENV, PATH,
GOENV, GOFLAGS, constructed values, or an alternate Make/runtime context therefore fail before
the checked-in policy can claim protection. The exported direct-bootstrap verifier enforces this
independently and VerifyCIContainerDownloads composes it into repository policy. A Make-only
mutation cannot suppress this audit. This bootstrap still has an unavoidable
self-verification boundary: the required-check configuration, GitHub Actions runner, and review of a
change that removes or poisons the direct step must cause the verifier to run before the repository
can inspect itself. Release verification therefore proves the checked-in structure once invoked; it
does not claim authority over the hosting platform or an execution that never starts.
The public verifier admits only the exact reviewed helper and Playwright runner sources,
so no test-only package links into cmd/releaseverify. Tests execute that source through the shared
testkit process boundary with fake local Docker and sleep executables to prove exact image arguments,
cached zero-pull behavior, the five-attempt/four-wait bound, wait failure, and final-status
propagation without a registry or network.
The helper’s classifier consumers are repository contracts, Postgres, visual, e2e, tuner, and CI
policy. The Playwright runner selects repository contracts, visual, e2e, tuner, and CI policy. The
broad Web unit gate is not selected: its make fe-unit and make fe-checks paths never invoke
either script, while each container-backed Playwright target and test-pg owns an explicit
prerequisite.
Any production or test change under internal/platform/releaseverify or cmd/releaseverify selects both the
policy job and the sharded Go job, because only the latter executes the package’s RED-capable tests.
Playwright is the second active job. Its four shards consume the union of impact_visual and
impact_e2e; the combined job still runs both suites exactly as before. Shipping Web runtime
sources are conservatively visual-sensitive because Storybook alias imports make filename-only
transitive narrowing unsafe. Shared API/core/fixture inputs and OpenAPI select both suites, while
visual/e2e tests and committed baselines select their owner. Only proven unit-test-only Web sources
skip Playwright in this first slice.
Tuner is the third active job. Its macOS browser matrix consumes impact_tuner directly. Every
shipping Web runtime source remains tuner-sensitive because the matrix loads the real SPA and HLS
controller; unit, spec, and story-only modules may skip it. Tuner e2e inputs, browser build
configuration, shared API/core/fixture/player packages, runtime tokens, and OpenAPI select it
explicitly.
Apple mobile is the fourth active decision and Apple TV is the fifth. iOS and tvOS are separate top-level jobs with hard-coded app commands, dedicated impact selectors, and independent required results. Existing cache-key strings are preserved so splitting job identity does not discard compatible pnpm, CocoaPods, or ExpoModulesJSI entries.
Every job consumes its dedicated impact_* output. A missing base, classifier failure, or unknown
path selects every specialized gate. The manual release-candidate scope remains unchanged and
excludes Postgres, Playwright, tuner, and application-client builds.
scripts/testdata/ci-impact.tsv records the exact ordered gate set for representative paths and
multi-path changes across every specialized gate. The classifier contract test compares complete
sets, so both a missed gate and an unexplained extra gate require an explicit fixture decision.
Non-code files consumed by Go tests are Go inputs too. In particular, design/configuration/command docs, install docs and README, the committed OpenAPI document, production Compose, and embedded help select the complete Go test set. Dockerfile and packaged license/notices select repository contracts as well as the image build. These mappings are fixtures because file extensions cannot reveal those dependencies.
Client decisions follow the actual consumer graph. An app-local mobile change selects shared-client,
iOS, and Expo Android mobile evidence; a TV change selects shared-client, tvOS, and Expo Android TV.
Changes to api, core, fixtures, player, design-system, or ui select both apps on both
native platforms because those packages are transitive inputs to both. The shared player package
also selects Web, Tuner, Image, and Android TV because its browser and native adapters share one
transport contract. Browser-only client-proof and Turborepo contract changes select the shared
JavaScript gate without spending a native runner.
The committed OpenAPI document is a schema input rather than a native binary input. An OpenAPI-only
change regenerates the client, then the shared-client gate typechecks, tests, and bundles the browser,
iOS, Android, and TV JavaScript consumers. A mobile or TV app change, a shared native/runtime package,
native build tooling, or a platform workflow still selects the corresponding native compilation.
This keeps additive unused endpoints from compiling product binaries while a real consumer change
continues to fail closed through its own path.
Apple mobile, Apple TV, and Expo Android mobile are active. The mobile Android job consumes only
impact_expo_android_mobile, builds CLIENT_APP=mobile, and retains its standalone debug APK for
seven days. Expo Android TV remains a classifier decision until its independently required job and
current-main evidence exist.
make client-android-debug keeps workstation-safe defaults: one Gradle worker, one native job, one
Metro worker, a 1280 MB Gradle heap, and a 4 GB memory scope. The CI mobile job raises them through
the environment, the same way the TV producer does: two Gradle workers (the wrapper’s ceiling), four
native jobs for the single-ABI app CMake build, four Metro workers, a 3 GB heap, and a 12 GB scope on
the 16 GB runner. It reuses the TV producer’s pinned ccache, the same wrapper validation
(web/scripts/android-ccache-env.sh), and the same generated-Ninja launcher proof. It restores no
cache: it builds a debug variant, so TV’s release objects never hit for it.
The permanent-identity Android TV producer restores a 2 GiB ccache directory across source
identities and keeps no Gradle cache. The workflow installs the exact §14 pin,
while the build wrapper rejects a relative/wrong-version launcher, a foreign base directory, or any
sloppiness setting. Ccache uses compiler-content identity and checkout-relative paths; the key may
restore prior compiler objects, but it never includes generated android/ or .cxx trees, bundles,
signing material, or evidence. Version, configuration, pre/post JSON statistics, and generated Ninja
launcher coverage are retained with the existing Android profile artifact. CI also permits two
Gradle projects to execute in parallel while keeping each CMake task’s compile/link pool at one;
local builds retain the one-worker default, and the wrapper accepts no value above two. A developer build
automatically acquires the same pinned macOS or Linux tool into .artifacts/, reuses its local
compiler-result directory across later builds in that worktree, and falls back to the unchanged cold
path when acquisition is unavailable. Three Gradle workers are intentionally rejected: the hosted
comparison regressed from 16m15s to 19m07s without adding cache hits. Set
LOOMARR_ANDROID_CCACHE=off to request the cold path.
GitHub scopes pull-request caches to their merge refs and merge-queue caches to disposable queue
refs, so neither can warm the next queue candidate. After a successful Android queue producer lands,
it instead hands its bounded ccache tree to android-ccache-promotion.yml through a one-day immutable
artifact. The promotion starts when that merge-group CI run completes successfully (workflow_run),
because the run can finish after the merge (#2288). It publishes only a commit main already contains,
and proves the source CI run, Android job, artifact name, commit/run/workflow manifest, and cache-tree digest,
then saves that generation in the default-branch cache scope without running another product build.
It deletes the transfer and all superseded Android generations. Android jobs do not run on pull
requests, and queue jobs restore but never spend time saving an unreachable cache.
Per-run measurements
The required CI aggregate appends a timing table after it has evaluated every required result.
scripts/ci-run-metrics.sh reads GitHub’s run and job records and reports queue delay, execution
time, end-to-end time, the longest job, and total occupied runner time. The report is observational:
an API or checkout failure emits a warning but cannot turn verified code red or hide a failed gate.
Its formatter is tested against a pinned API fixture without touching the network.
The distinction between queue and execution is load-bearing for native work. A macOS job can be the critical path because it waited for capacity, because Xcode compiled slowly, or both; changing cache policy cannot solve the first case, while adding shards can make it worse.
The image job is the exception
The image filter follows every source family copied by the Dockerfile: Docker metadata, packaged
LICENSE/notices, Cargo and Rust sources, Go sources/modules/embedded migrations, embedded help, the frontend,
OpenAPI, and the bundle guard. Makefile and workflow-only changes do not change image bytes and
therefore do not trigger it.
It builds each release platform on a native runner, loads the resulting image without pushing it, and inspects the packaged LICENSE/notices and OCI labels. The Dockerfile’s build-time commands prove the bundled tools; the post-build inspection proves the final runtime filesystem rather than a comment or an intermediate stage.
The image job has a timeout-minutes because its release builds are independently bounded; GitHub’s
default is six hours.
The reusable Apple mobile and Apple TV jobs have explicit job limits of 75 and 60 minutes. These bounds are evidence-derived from recent successful merge-group runs (mobile: 29:14–50:46, TV: 16:38–34:49) and cover prebuild, dependency installation, Xcode build, simulator readiness, installation, and launch. Review them after at least 20 successful native runs or at the first native timeout. A hard timeout can prevent the screenshot step from running; the screenshot remains useful for ordinary failures but is not guaranteed after timeout.
It exists because a Dockerfile that could never build for arm64 sat undetected. Build both platforms or it can’t catch that.
Playout bench
ci-playout-bench.yml runs the software-only playout bench on every pull request that touches playout,
the bench or its scripts. It is selected by the playout_bench classifier gate and is part of CI, so
it is covered by ci-ok. The hardware runs live in playout-bench.yml, which is workflow_dispatch
and nightly only and must never gain a pull_request, pull_request_target or merge_group trigger:
the repository is public and a pull request must not schedule work on the maintainer’s GPU machines.
Playout bench has the metrics, thresholds and baselines.
Manual scopes are explicit
Manual CI defaults to release-candidate. That scope is for certifying an exact main commit before
tagging: it runs repository contracts, the real-codec image-worker certification, and both native
release-image builds. It does not rerun Android, PostgreSQL, Go race, frontend, Playwright,
tuner, docs, or the macOS harness; their normal push and pull-request impact coverage is unchanged.
Select full explicitly when an investigation genuinely needs every matrix. Both modes publish a
scope-marker job, but scripts/validate-release-source.sh accepts only the release-candidate marker
for Docker publication. It rejects normal push CI and full manual runs even when green, so mobile,
client, and unrelated platform matrices cannot become release prerequisites. The release-candidate
marker also makes the contract and certification jobs mandatory evidence rather than relying only on
the workflow’s overall conclusion.
Select apple-cache-validation only while proving the Apple compilation-cache protocol. It runs no
ordinary product family: one xcode-27 job performs the complete mobile and TV Release
install-launch-liveness gates while populating an LLVM CAS, validates and packs that store, and
transfers it as a one-day workflow artifact. A dependent xcode-27 job restores and validates the
archive before proving mobile and TV hits, a real Swift source-change hit-and-miss, corrupt-archive
rejection, fingerprint invalidation, and cold mobile/TV fallback. The runner boundary is deliberate;
a cache that works only within one machine is not eligible for later CI consumption. This manual
scope is included in ci-ok when selected and is skipped for pull requests, merge groups, release
certification, and full dispatches.
Select tuner to measure a browser-engine-specific tuner flake without paying for the rest of the
matrix (#1443). It runs only the tuner job, on the manual-tuner lane, and takes two optional inputs:
project (all, chromium, firefox, or webkit) and repeat_each (1 to 100), which
scripts/tuner-args.sh turns into Playwright --project and --repeat-each flags. Invalid values
fail closed rather than running less than asked, and retries stay owned by
playwright.tuner.config.ts (0), so a repeated run can never hide a failure behind a retry. At the
defaults (all, 1) the script prints nothing and make tuner-e2e-host is byte-identical to the
merge-queue run. Other scopes ignore both inputs, and pull requests and merge groups never see them.
Flake quarantine (#1570). A tuner test with an open flaky issue can leave the required matrix
without being skipped. It enters quarantine only by a line in scripts/flake-quarantine.tsv that
names the suite, project, spec file, exact test title and tracking issue. In pull requests and merge
groups, make tuner-quarantine keeps the lines whose issue is still OPEN. The required step runs
every other (project, test) pair, and a later non-blocking step (make tuner-e2e-quarantine) runs
the quarantined pairs, with zero retries as always. Its failure shows on that step and uploads
tuner-quarantine-failures, but it cannot fail CI.
When the issue closes, the test is back in the required step on the next run; the line then only
prints a notice until someone removes it. Every doubt resolves toward required: an issue whose state
can’t be read, a malformed line, or an unknown mode fails closed. The manual tuner scope turns the
quarantine off, so it measures every test. That non-blocking step is the only source-bound step the
workflow policy lets set continue-on-error (nonBlocking in internal/platform/releaseverify).
After that portability proof is green, Apple compilation cache is the only workflow authorized to
publish compiler results. It is manual-only, refuses every ref except refs/heads/main, and builds
the complete mobile and TV Release install-launch-liveness gates before saving. A restored seed is
hash-validated before use; the candidate CAS and compressed archive are validated again, and a save
is refused if its size plus a 512 MiB reserve would exceed the repository’s 10 GiB cache budget.
Before installing dependencies or starting either native build, the writer also requires enough
live headroom for the maximum permitted archive plus that reserve; this conservative preflight
avoids spending the full cold-build window on a candidate that cannot possibly be published. The
post-build admission still uses the archive’s exact compressed size.
After a successful save, cleanup is limited to the exact fingerprint prefix on refs/heads/main and
retains one generation. Only this workflow has actions: write for the compiler cache.
The ordinary Apple mobile and Apple TV jobs are compiler-cache consumers only. Each computes the toolchain/SDK/native-input fingerprint, uses the restore-only cache action, and exposes a store only after zstd and LLVM CAS hash validation. Missing or rejected archives select the unchanged cold gate. A valid store that fails during the warm build is quarantined and receives one clean cold retry; every architecture, install, launch, screenshot, and liveness assertion still runs. These jobs never save the compiler archive, and the obsolete pnpm/CocoaPods and ExpoModulesJSI lookup steps have no writer and are omitted. Pull-request and merge-queue refs therefore cannot create sibling-scoped Apple generations that later groups cannot read.
The Apple verifier starts the selected simulator before Expo’s required clean prebuild, then runs
pod install explicitly while the simulator boots. simctl bootstatus remains a hard barrier
before Expo builds, installs, or launches, and expo run:ios --no-install skips only the duplicate
CocoaPods invocation. The uploaded phase-timings.tsv records clean prebuild, CocoaPods, simulator
readiness, native build/install, and artifact/runtime assertions. Simulator readiness contains the
overlapped prebuild and CocoaPods intervals, so those rows are deliberately not additive.
Do not add a generated ios/Pods cache without re-proving both integrity and repository capacity.
Measured 2026-08-31, one clean mobile native workspace was 981 MiB and compressed to 339 MiB while
the repository already held 9.11 GiB of its 10 GiB Actions-cache budget. Restoring Pods alone still
pays CocoaPods project generation; restoring the integrated workspace overwrites clean-prebuild
authority. Neither seam justified consuming the remaining cache headroom.
ci-ok is the only required check
It always runs and inspects needs.*.result explicitly. That has to be explicit: a skipped job
doesn’t fail an aggregate by default, and neither does a failed one under if: always().
A cancelled job passes only on a pull request, where a newer push superseded it. In the merge queue
and on manual runs it fails the check: a job that crosses its timeout-minutes reports cancelled
with its tests unfinished, so it has proved nothing.
The main branch requires the GitHub Actions-owned CI check in strict mode: a pull request must
be tested against the current base before it can merge. Preserve both the check name and its app
binding when editing branch protection.
make release-verify parses the workflow and requires every top-level job to appear in
ci-ok.needs. This prevents a newly added or accidentally removed dependency from producing a
green required check while its real job is red.
Never add a workflow-level paths:. A run that doesn’t trigger reports no checks, so a required
check sits “expected” forever and the PR can’t merge. Filter per job.
The workflow handles merge_group, and the organization-owned repository requires the merge queue
through its main ruleset. The strict, GitHub-Actions-owned CI check remains the protected
current-base boundary inside the queue. Never remove the queue rule merely to bypass a delayed or
failing native result.
Sharding
Go tests, frontend and Playwright split across runners for wall-clock only. Repository-wide Go
contracts run once: the workflow, release, harness, shell and generated-docs checks in CI policy on
every PR, and the rest in go-contracts, whose two shards share them without overlap. Rust
contracts once in rust-contracts, in parallel with
two ordinary Go test lanes, two serial lanes for packages that must not share a runner, and the independent release-worker
certification. Their union is the same
assurance as make verify SCOPE=all plus the existing CI-only certification. The ci-ok aggregate
requires every job, so moving a contract out of the test lanes cannot make it optional.
make go-shard-verify runs in go-contracts and asserts the Go shards are a true partition of
go list ./... — a split that drops a package would otherwise pass by not running it.
The Go partition uses longest-processing-time assignment over a reviewed table containing only
material package timings. Packages below five measured seconds, including new packages awaiting a
hosted measurement, receive a one-second planning floor and remain fail-safe members of the exact
partition. Merge-group run 35472062915 exposed 1430/657/583 package-seconds in the former three-way
alphabetical split and a 31m10s critical path. Two ordinary lanes use bounded -p=4 package
parallelism. The reviewed latency-sensitive playout/capacity set runs in two concurrent -p=1
lanes, so each group remains serial while separate runners safely overlap the groups. Merge-group
run 35497917692 measured the former combined certification lane at 9m53s, including internal/app
at 159.215 seconds and the complementary playout packages at 157.874 reported package-seconds.
That natural seam produces modeled certification groups of 173 and 172 seconds after the latest
hosted-weight refresh. The same run
measured the refactored internal/proposals/suggest at 34.908 seconds; refreshing its obsolete 368-second
weight removed that package bottleneck. Merge-group run 35500779353 then put every Go job between
6m59s and 9m01s, but the ordinary job spread was still about 29%. Its package summary exposed
internal/people/auth at 300.797 seconds and refreshed every material upper-envelope weight. Both auth
test-store helpers had independently replayed the complete SQLite migration history for every
private test database. Routing them through the existing isolated migrated template reduced the
local SSO race subset from 30.59 to 5.64 seconds and the complete auth race package from 118.89 to
19.64 seconds, with the real RS256 verifier, allowlist refusals, disabled-person checks, session
revocation, password hashing, and every existing assertion retained. A conservative provisional
auth ceiling first produced a six-lane plan of 372/371/371/371/371/371 seconds, but the repaired
tree no longer has enough ordinary work to keep six hosted runners efficiently occupied. The first
authoritative current-tree four-ordinary-lane run, 35504999984,
validated the repaired auth package at 23.614 seconds and completed all six Go jobs successfully.
Its ordinary job/test times were 9m36s/9m02s, 8m31s/7m52s, 7m02s/6m28s and 9m30s/8m52s.
The 9m53s queue-to-finish critical path was materially lower than 14+ minutes, but the roughly 40%
ordinary test-time spread still failed acceptance. The run therefore did not merge. Refreshing every
material weight from that exact tree with uniform 10% headroom produced checked-in aggregate loads
of 533/533/532/532 seconds and bounded-worker makespans of 267/267/267/266 seconds. The next
authoritative run, 35507559795, proved why more shard tuning was the wrong fix: lane 2 reported only
about four minutes of longest-worker package execution (internal/platform/store at 239.922 seconds), yet
spent 9m37s in the test step. Its cache log showed a complete miss followed by repeated module
downloads, a 40.94-second Rust build, and a 55-second eval-contract invocation before the lane even
started its ordinary package phase. Those prerequisites ran identically in all six jobs.
The resulting architecture preserves the same work with fewer runners. Two ordinary lanes use
bounded -p=4, and the two serial certification lanes remain unchanged. Lane-scoped make test
omits the eager Rust and eval prerequisites: go-contracts executes eval-contract once, while
composition packages that need the real image worker acquire it through internal/testkit and
pass its executable at the explicit application override seam. Unsharded local make test retains
both explicit prerequisites and remains self-contained.
The first authoritative two-ordinary-lane run, 35510845578, proved that balancing aggregate work
was still insufficient. Both certification jobs passed in 5m57s and 7m15s from queue creation, but
ordinary test steps took 10m23s and 10m41s and completed 11m18s and 11m35s after queue creation.
Their 18-second spread ruled out shard imbalance. The executable package order was the real defect:
the sharder modeled longest-processing-time workers, then restored go list order, and the race
policy sorted that order lexically again. internal/platform/store and internal/channels/recurate consequently
started about 6m21s and 6m34s into their test steps. Four large, fixture-isolated packages also ran
all top-level tests serially. The runner now emits and preserves descending measured-cost order,
and research/internal/fillerreview, internal/playout/backendtransition’s non-integration tests,
internal/channels/recurate, and internal/channels/approval use bounded t.Parallel execution. The PostgreSQL-tagged
backend tests remain serial because they mutate process environment. Same-machine four-CPU race
controls measured fillerreview at 115.237s serial versus 41.154s parallel; the other three
packages completed together in 58.83s versus 82.61s with their internal parallelism disabled.
Scaling the hosted weights by those ratios and repartitioning yields 921/920 aggregate seconds and
244/231-second bounded-worker makespans. Those scaled weights remain provisional until consecutive
successful merge-group runs replace them with complete hosted measurements.
Merge-group run 35515087033 measured ordinary lane 1/2 at 8m31s job time and 8m02s test time,
while ordinary lane 2/2 reached its test failure in 8m10s. Certification lane 2/2 passed in 5m09s;
lane 1/2 passed in 7m00s after waiting 2m39s for the repository’s 20-runner concurrency ceiling.
The run was not accepted: lane 2/2 exposed that the integration composition harness had relied on
the old eager Rust build instead of acquiring its required worker. The harness now obtains the
worker through internal/testkit and passes it through app.Overrides.ImageWorkerExecutable, so a
focused clean go test ./internal/integration is self-contained without rebuilding Rust in lanes
that do not consume it. Fresh consecutive merge-group passes remain required.
The weights are now regenerated, never hand-kept: make go-race-weights RUNS="..."
(scripts/go-race-weights-refresh.sh) takes the median ok time of every package across the
race-lane jobs of the given merge-group runs, rounded up. A lane that timed out still counts for the
packages that finished in it: ok is printed only for a whole run, and the lane over its timeout is
the one that needs re-measuring. Regenerated from runs
36330002206, 36330402570, 36330935829, 36331908558 and 36333318208 (#1570), the real costs were
about 2.4× the earlier weights (internal/platform/store 516s against 242), and the old unitless budgets
had been passing on those stale inputs.
Race lanes build modernc’s SQLite with -gcflags=modernc.org/...=-d=checkptr=0
(scripts/go-test-packages.sh). -race turns on checkptr everywhere, and modernc is C translated to
Go, with unsafe pointer arithmetic on nearly every line: checkptr there doubled the cost of every
migration (#2716). The race detector still instruments modernc, and checkptr stays on for every other
package. Tests copy a migrated SQLite template rather than migrating a fresh file
(testkit.MigratedSQLiteDSN, enforced by internal/migratedstore_test.go).
The budgets derive from #1570’s target, a Go-only merge-queue run in at most ten minutes
(./scripts/go-shard.sh --budgets). The measured queue time outside a lane’s test step is 131s:
116s median from queue start to the test step, 4s of teardown, and 11s for the CI aggregator.
That leaves a 469s lane test step. An ordinary lane’s summed package-seconds must fit four -p=4
workers in that step (1,876s). Its bounded-worker makespan, and each serial certification lane,
must fit the step itself. No single package may exceed that step either: a package is the
sharder’s indivisible unit, so an oversized one must run its tests in parallel, move to a serial
lane of its own, or be split, never be given a larger budget. (Until #1650 a temporary cap of
567s, internal/platform/store’s measured time plus 10%, stood in for that rule.)
Pull request #1631 made the 21 heaviest internal/platform/store tests (88% of its time) run in parallel. Pinned to four
cores locally, that cut the package from 146.9s to 110.1s, but merge-group run 36359955148 measured
it at 542s in ordinary lane 1/2: beside three other -p=4 packages on a four-vCPU runner, its
parallel tests only competed for the same cores. So the store package (now internal/platform/store) runs alone in the second
serial lane, where one package at a time owns the runner, and the playout/capacity set shares the
first. There, workflow-dispatch run 36366596157 measured it at 171s, which left it too light to
balance the first lane’s 405s, so internal/integration, whose tests are serial, shares its lane.
The serial lanes hold two kinds of package: media certification that must not compete for a
worker while asserting latency, and CPU-heavy packages whose own parallelism needs a runner to
themselves.
Since #2716 there are three serial lanes, certification-1/3 to certification-3/3. The highest
lane number in scripts/go-certification-lanes.tsv is the count, and the workflow matrix, cache
keys and release-verifier pins name each lane. Two lanes passed the 15-minute job limit once
internal/channels/schedule and internal/demolibrary joined them. Four lanes could not balance:
merge-group run 43037503041 measured store alone at 291s, and no four-way split of the rest stayed
inside the 125% balance budget. The three lanes model 322s, 336s and 378s.
make go-shard-verify SHARDS=2 rejects missing or duplicated packages, a package above the
per-package cap, an ordinary aggregate or a bounded-worker or serial-certification makespan above
those derived budgets, or more than 25% imbalance within any group. Aggregate and worker limits are
deliberately separate: package overlap
cannot hide unbounded total work, and a balanced aggregate cannot hide one saturated worker. Release
verification additionally rejects an unreviewed serial package, grouping or workflow lane. The
workflow also caps each whole job at 15 minutes, leaving six minutes for setup and compilation while
making renewed latency drift a hard failure. TestGoShardUsesMeasuredLongestProcessingTime pins
deterministic assignment and executable work order, TestGoRacePolicyPreservesSharderWorkOrder
prevents the policy split from sorting it away, and TestGoShardBalancesMeasuredRaceWork pins the
latency and balance budgets against the current tree.
Within the SQLite store package, conformance assertions clone one closed, fully migrated template database rather than replaying the complete migration history for every assertion. The 2026-09-01 profile measured 169.3s for the package and 134.3s for SQLite conformance: 131 fresh stores each replayed 86 migrations, more than 11,000 migration applications in one run. Every clone remains a private file opened through the production SQLite adapter. Migration ordering, boot seeding, downgrade behavior, historical data, and restart behavior retain dedicated fresh-migration tests; Postgres conformance remains unchanged. Re-running the exact race profile with the cloned fixture reduced the package to 40.3s and SQLite conformance to 6.2s (−76.2% and −95.4%, respectively). Five consecutive race-enabled conformance/factory runs remained green, followed by the unchanged real Postgres integration gate over store, backend transition, and app. In the authoritative queue, the race steps moved from 7m42s / 6m38s / 11m30s to 9m13s / 6m35s / 5m33s: the critical path fell 2m17s (19.9%) and aggregate race-step time fell 4m29s (17.4%), despite normal per-runner variation moving shard 1 upward.
Domain tests that need current persistence state, rather than migration behavior, use
testkit.MigratedSQLiteStore for the same reason. A race-enabled 2026-09-01 profile found
internal/channels spending 74.46s while its helpers replayed every migration for each private test
database. Routing those helpers through the existing isolated migrated fixture reduced the exact
package profile to 4.19s (94.4%). Tests that exercise startup, migration, downgrade, historical data,
or restart behavior must continue to open and migrate fresh databases.
The Proposal Job worker tests use that same isolated migrated-store fixture and start a due-work drain
before entering their two-second recurring poll. Profiling also found theme scoring recalculating the
same series-title scope for every item and qualifier; preparing it once per score preserves every
2,000-iteration property assertion. On 2026-09-20, the one-time preparation reduced the complete
local internal/proposals/suggest race package from 112.28 to 19.35 seconds after the worker/fixture changes;
the deterministic property alone fell from 66.16 to 6.16 seconds. Hosted lane evidence remains the
authority for the merge-queue budget.
PostgreSQL conformance also uses an isolated template, but through PostgreSQL’s own database-clone
interface rather than file copying. Seven successful merge-queue samples put the real-Postgres step
between 8m20s and 14m18s; internal/platform/store dominated every sample at 366–722 package-seconds while
backend transition and app ran concurrently. The local race profile measured 124.43s for the full
integration-tagged store package and 46.23s for Postgres conformance. Its factory had dropped and
recreated public, then replayed all 86 migrations for every assertion. Migrating and boot-seeding
one closed template, creating a private database from it per assertion, and opening each clone
through the production Postgres adapter reduced the exact profile to 85.46s / 6.85s (−31.3% and
−85.2%). Clone cleanup closes the Store before a maintenance connection force-drops only the named
disposable database. Migration, preflight, and cross-backend data-migration tests still create fresh
databases; queue evidence remains the proportional gate.
Sharding is free on a public repo. Check the bill before copying it into a private one.
Caching
actions/cachenever overwrites an existing key. A cache whose contents track something its key doesn’t gets written once and frozen. Use a rolling key withrestore-keys.- The 10GB cap evicts LRU across all refs, so closed PRs’ caches push out live ones.
cache-cleanup.ymldeletes bothrefs/pull/N/mergeand matching merge-queue caches on close. - Default-branch cache writes require a trusted trigger. GitHub permits a default-branch writer
from
pushorworkflow_dispatch, butmerge_groupandworkflow_runcannot promote their outputs into that scope. The Go cache-save conditions therefore remain useful only for a deliberate manual run onmain; ordinary PR and queue runs are restore-only. Android is the bounded exception justified by the measured 25m54s cold queue build: the admitted queue run emits an immutable compiler-cache transfer, and the queue-producedpushvalidates and saves it without repeating any compilation. Aworkflow_runwriter or apushwarmer that rebuilds the product remains forbidden.
Apple compilation caching is a validated artifact protocol, not an unchecked DerivedData restore.
The fingerprint binds the runner OS and architecture, exact Xcode and Swift identities, both
simulator SDKs, Release build settings, and the pnpm lockfile. Restores are decompressed into a
temporary path and accepted only after the installed llvm-cas verifies hashes; failed warm builds
quarantine that exact store and retry the complete gate once in cold mode. Archive size, repository
cache usage, unrelated-cache headroom, and main-ref retention are checked before any trusted writer
may save. PR and merge-queue consumers remain restore-only, and a warm-required validation cannot
turn its cold fallback into a passing cache claim.
Go runtime workers use the image’s ffmpeg
Go runtime workers install the production Dockerfile’s retained ffmpeg build, pinned by
architecture-specific SHA-256 for both ffmpeg and ffprobe, never the runner’s distribution
package. The archive is cached by its exact digest and verified on every use before extraction or
execution, and the installed pair must report the declared build. A change to the installer, the
Go workflow or the Dockerfile’s pin source selects the complete Go runtime gate.
Hand-maintained lists, and what guards them
Three lists in this repo are written by hand and would rot silently. Each has an executable guard that fails when it drifts, and all three run in CI:
| List | Guard | Runs via |
|---|---|---|
TAGS in the Makefile | scripts/check-tags.sh | make tags-verify, part of comprehensive verification |
| Retired identifiers | scripts/check-retired.sh | make retired-verify, its own CI step |
| Release-image source-family probes | releaseverify.VerifyCIImageInputs | make release-verify, part of comprehensive verification |
tags-verify compares the tags in //go:build lines against TAGS and fails both ways:
- In the tree, not in
TAGS— those files are invisible tovet-tagsandlint. Nothing compiles them, which is how a live ffmpeg test sat uncompiled for months. - In
TAGS, not in the tree — the list claims coverage it doesn’t have. Drop the tag in the PR that removed its last file.
Downgrading the second direction to a warning is the obvious-looking fix when it’s inconvenient. Don’t — a warning printed by a job that exits 0 is one nobody reads.
The CI path filter includes scripts/, so a PR editing only a guard still runs the job that
executes it.
Screenshot baselines render on every frontend push
visual-baselines.yml is outside the required PR gate. It runs on every pull_request push
(opened, synchronize, reopened) from a branch of this repository that touches web/, and by manual
dispatch. Its resolve job feeds the pull request’s changed files to scripts/ci-impact.sh and renders
only when the visual or e2e gate is selected, so it follows the same classifier as the Playwright
job. The render (visual-baselines-render.yml, dispatched at the branch, read-only) regenerates the
visual and e2e snapshots in the pinned Playwright image. The push job commits only PNGs under a
tests/visual/ or tests/e2e/ snapshot directory. It has a workflow-level paths: filter, which a
required check must never have; this workflow is not one.
The bot’s push starts pull_request runs whose actor is github-actions[bot]. The repository’s
fork-run approval policy is “first-time contributors”, and the bot counts as one, so GitHub parks
those runs at action_required and the parked CI keeps the pull request out of the merge queue. The
workflow’s release job approves exactly the parked runs on the commit it pushed, which needs only
actions: write. Forks are untouched: their runs keep the approval rule, and the resolve job refuses
a fork before any job holds a write token. A newer push cancels the older head’s run, and the bot’s
own commit never starts a render. Testing has the full flow.
Scheduled Go dead code report
deadcode.yml is also outside the required PR gate. On the first of each month, and on manual
dispatch, it counts Go functions no ./cmd binary reaches and opens or updates one tracking issue
when the total exceeds the committed baseline. See Code hygiene.
Scheduled Go vulnerability scan
go-maintenance.yml is outside the required PR gate too. Every Monday, and on manual dispatch, it
runs make vuln: pinned govulncheck over the untagged build that ships, failing on any known
vulnerability a Loomarr call path reaches, standard library included. It is weekly rather than a PR
gate because a newly published advisory would otherwise fail every Go PR regardless of its content.
A red run is the prompt to bump the module, or go.mod’s toolchain line together with the
Dockerfile’s golang image.
The same workflow runs make scan-scale: the library scanner’s scale guard over a 33k-file library,
the household’s size. It checks that step 2 counts each file once, and that passes made while the
header stage waits for playback leave nothing held. The PR suite runs the same guard at 2k files.
At 33k it takes about a minute, which is too long for every PR.
Scheduled ffmpeg pin refresh
ffmpeg-pin-refresh.yml runs every Monday, and on manual dispatch with an optional dry_run. It
runs scripts/ffmpeg-pin-refresh.sh automate. When BtbN has a newer release-series build, the
script does four things:
- publishes that build’s
third-party-sources-<date>release (binaries plus corresponding source); - pushes
deps/ffmpeg-<build>; - opens the pin PR, as a draft labelled
ready-for-humanfor a new series; - dispatches
ci.ymlwithscope=fullat that branch, because a push made withGITHUB_TOKENstarts no workflow.
Its single job holds contents, pull-requests and actions write, one for each of those side
effects. The procedure is in releasing.md.
Scheduled Rust maintenance
rust-maintenance.yml is intentionally outside the required PR gate. Every Monday, and on manual
dispatch, it installs pinned cargo-deny and cargo-fuzz versions, checks both Cargo lockfiles for
RustSec advisories, approved SPDX licenses, and untrusted sources, then fuzzes the worker’s bounded
JSON-to-decoder boundary under nightly libFuzzer. A crash retains its reproducer for 30 days.
This job is allowed to be expensive and network-sensitive. The fast deterministic protections stay
in comprehensive verification: Cargo lock enforcement, clippy/tests, and #![forbid(unsafe_code)] on Loomarr-owned
shipping crates.