Skip to content

CI

For: contributors whose pull request is waiting on checks. You’ll get: how CI picks what to run, and the one check that gates main.

A pull request enters ci.yml: the path classifier picks the affected families, the PR run or merge queue admits them, the family workflows run only what changed, and every result goes to one required check that gates main

D2 source

Each family contains independently filtered jobs. The required aggregate always runs and checks every top-level result, including jobs that correctly skipped.

Fast PR feedback, authoritative queued integration

Ordinary pull-request pushes run the always-on policy bootstrap plus affected repository-contract, static-analysis, compile/type, unit, documentation, and shared-client feedback and return the required CI result quickly. They do not run race-policy shards, Postgres conformance, Playwright, release-image builds, runtime image certification, Apple mobile, Apple TV, Android TV, Expo Android mobile, the tuner matrix, or the macOS harness. The required main merge queue then builds at most eight cumulative candidates concurrently against the current base. A queued PR waits only for merge-gate.yml, which reports the required CI check after proving the merged tree compiles, so it merges in minutes. merge-gate-cache.yml refreshes its Go cache from main at most hourly and keeps one entry. Beside it, the merge_group run of ci.yml is the complete integration lane, reported as CI assurance: the same fail-closed classifier selects the complete affected gate set, and every selected result remains a dependency of that job. It finishes after the merge, so its evidence no longer blocks one. When it fails on a commit that reached main it files an issue. Revert first: anyone may revert the commit to get main green, and the fix lands as a new PR. Rerun instead when the failure is a flake or an outage. Every queue entry keeps its own run and none is cancelled, because a run selects gates from its own PR’s diff and a later run can’t replace it. This is a maintainer decision (2026-11-07): merge latency over pre-merge evidence. Explicit manual runs retain release-candidate and full recovery scopes plus an isolated Apple compilation cache portability scope. A normal queue-produced push to main runs publication workflows only rather than validating the admitted commit again.

CodeQL is a separate security workflow because its unit of selection is a language database, not a product gate. Pull requests and pushes to main diff from a trustworthy base and scan only changed CodeQL languages. A workflow/classifier edit, an unknown path, or an unusable base fails closed to all six repository languages. The Wednesday schedule and every manual dispatch always scan all six, so selective change feedback cannot erode periodic repository coverage. The repository uses CodeQL advanced setup for this path-aware policy; enabling GitHub’s default setup would disable the checked-in workflow and restore an all-language scan on every change.

The root workflow owns triggering, classification, admission, manual scopes, and the required aggregate. The lightweight CI policy job always runs independently of classifier output, so a broken classifier cannot skip its own verification. Product job implementations live in family-named reusable workflows, so an edit to one family selects only its owning product gate. Classifier, admission, and release-evidence control-plane code selects policy rather than a product build that does not execute it. Agent harness and design-contract prose likewise select only their real consumers. They do not select unrelated Rust, Android, Apple, image, browser, frontend, Postgres, or application-Go builds. The root Make interface and unknown paths still select everything.

The Go repository-contract job keeps the stable make privacy-verify interface, but its captured fixture guard is one in-process tracked-tree scan in releaseverify. The previous shell loop started hashing processes for every unique candidate and measured 54 seconds locally by itself; the same case-folded SHA-256 policy now measures about two seconds, reports only labels and digests, and keeps household-name/profile-PIN candidates restricted to the four audited response fixtures.

Make uses the same ownership boundary. The root Makefile is the stable interface for shared variables, help, and ordered mk/*.mk includes; each module owns one command family. The generated command reference follows those includes and rejects missing, cyclic, escaping, or duplicate target definitions, so splitting the physical files does not split the public command catalog.

This is admission control, not weaker assurance. A pull request cannot merge directly after its fast result: it must enter the queue and pass the generated current-base commit. The queue uses squash merges, ALLGREEN, eight concurrent builds, one PR per merge, and a three-hour check-response timeout. The concurrency cap is eight by maintainer decision on 2026-11-07, so a burst of queued PRs clears together instead of waiting in pairs. That accepts more CI minutes, and a larger invalidation cost when an early entry fails, because every cumulative build behind it restarts. Apple runners are the known pressure point: a burst of builds that select the Apple gates competes for the same macOS capacity. The queue still gives accepted work a stable place in line instead of repeatedly invalidating successful strict-mode runs. scripts/ci-merge-queue-policy.sh checks the live ruleset; APPLY=1 updates only its build-concurrency field and then re-checks every admission parameter.

The policy has three coupled halves:

  • GitHub ruleset Main merge queue admission targets refs/heads/main.
  • Branch protection applies the strict, Actions-owned CI requirement to administrators as well as ordinary contributors, so removing post-merge validation does not create a bypass path.
  • .github/workflows/ci.yml triggers on pull_request, merge_group, and explicit manual dispatch; it does not trigger product validation for a normal main push.

releaseverify.VerifyCINativeAdmission rejects loss of the queue trigger, renewed PR admission, a queue-only condition that drops manual evidence, a restored post-merge product trigger, or a missing scarce-capacity job. The live ruleset is verified through scripts/ci-merge-queue-policy.sh and GitHub’s branch-protection API when changing repository protection.

Jobs run only when their inputs changed

A changes job diffs against the merge base and each job gates on its output. It fails safe: no usable merge base — first push, force-push, new branch — runs everything.

scripts/ci-diff-base.sh resolves that base from the event payload. A PR diffs against its base commit; a merge-queue batch diffs against merge_group.base_sha, the main commit it is stacked on, so the queue selects the same gates as the PR it carries and a docs-only or Go-only batch never waits on the native client builds. scripts/ci-diff-base-test.sh proves both, end to end through the classifier.

Adding a new build input means adding it to the filter in the same PR.

Three non-obvious entries: the help pages (docs/get-started.md, docs/guides/, docs/explanation/) are in the Go filter because they are embedded and the doc-claims test reads them, internal/filler/location/ contains generated indexes embedded in the backend, and scripts/ is there because the job executes them.

Specialized gate classifier activates one job at a time

The changes job also runs scripts/ci-impact.sh and publishes impact_* outputs for the specialized contract, Go, full-Go, Rust, Postgres, web, shared-client, iOS, tvOS, Expo Android mobile, Expo Android TV, visual, e2e, tuner, image, docs, agent, and legacy Android TV gates. Its run summary places those proposed decisions beside the current broad families.

Classification follows execution ownership. A product gate is selected when a changed path is an input to the product or to the gate implementation that executes it. A path that only decides which gates run, admits a release, or interprets existing evidence is control-plane policy and does not borrow an unrelated product build as proof. Unknown paths still select every gate. Local edit-loop test planning is a separate interface: it shares changed-path collection but deliberately chooses fast direct or related unit tests rather than authoritative CI gates.

Browser-only test-card fixtures under web/packages/fixtures/src/testcard/ select browser unit, visual, and image evidence, not native client builds. Other shared fixture paths remain native inputs.

Postgres was the first active specialized output. store-postgres consumes impact_postgres directly while remaining in the required CI aggregate. The explicit release-candidate scope continues to exclude database conformance. This first activation intentionally treats every .go file plus go.mod, go.sum, store migrations, and unknown paths as Postgres-sensitive. That conservative boundary skips proven non-Go over-selection without guessing which transitive Go dependency can change a real-Postgres assertion. Dependency-aware narrowing is a later shadow change.

Container acquisition is bounded before product tests

Container acquisition has one retry boundary: scripts/ensure-container-image.sh. A locally inspected image returns without a pull. Otherwise the helper makes at most five pull attempts with exactly two seconds between attempts, fails immediately if a wait cannot run, and propagates the fifth pull failure status. The retry ends before docker run or testcontainers starts product tests; product execution is never retried.

The Playwright runner scripts/run-playwright-container.sh owns its fixed image identity. Each Make target first invokes its bounded ensure mode, then one product mode. Its reviewed source builds Docker arguments as a shell array, places the image at a fixed boundary, and accepts only a mode plus the validated LOOMARR_PLAYWRIGHT_SHARD=--shard=N/M visual-suite input. The Postgres pin has one Make-readable and Go-embedded authority at internal/testkit/postgresimage/image.txt; direct Postgres-module Run calls must use postgresimage.Name(). Release verification structurally pins that owner to one private //go:embed image.txt string and the single strings.TrimSpace(image) return; missing, transformed, environment-derived, or alternate implementations fail closed. Both owner and data-file changes select the contracts, Go, full-Go, Postgres, image, and policy gates. The package must contain exactly one untagged non-test production Go file, image.go; build constraints, another production file, or a second embed/Name owner fail closed, while _test.go files do not participate in production ownership. The verifier scans non-generated Go throughout the repository, including cmd/, tooling, and future top-level packages while excluding only vendor and named generated/cache trees. Deprecated postgres.RunContainer is forbidden because its image cannot use the repository authority. postgres.Run admits only the repository’s direct database, username, password, and wait customizers; opaque, variadic, nested, image-replacing, pull-policy, or shadowed-package customizers are rejected. Generic testcontainers Run admits no customizers. GenericContainer, request aliases, pointer aliases, request builders, and construction-time or later ContainerRequest.Image assignments accept an explicit unrelated image literal, but Postgres-bearing or opaque values must use that authority. AlwaysPullImage and ImageSubstitutors are forbidden at construction or later mutation so request objects cannot restore unbounded acquisition. The direct AST audit classifies ordinary, multi-result, compound, and range-key/value writes to whole requests or Image, including nested selectors, dereferences, and parentheses. A pointer to either target may not cross an opaque call boundary, directly or through a tracked alias; this also rejects direct reflection and unsafe mutation paths. Request or Image pointers may remain direct, tracked local aliases, but cannot enter slices, arrays, maps, structs, interfaces, channels, returns, or opaque aggregate calls; aggregate indexing or selection therefore cannot become an unaudited mutation path. No pointer-taking authority seam is approved: the Postgres authority is the value-returning postgresimage.Name(). This is a direct structural contract, not a claim to infer arbitrary reflection, unsafe, return-value, or interprocedural dataflow. Dereferencing a tracked pointer retains whether it targets a whole request or only Image, so ordinary, compound, multi-result, and range writes cannot erase that authority distinction. Taking acquisition functions as values is rejected because their arguments can no longer be audited. The alternate exported acquisition surfaces are closed too: ParallelContainers and the DockerProvider.CreateContainer, ReuseOrCreateContainer, and RunContainer methods are rejected rather than accepting opaque requests or pull policies outside the audited Run and GenericContainer seams. Postgres image literals are also rejected outside count-bounded exact verifier-fixture values and the one exact unrelated Authentik direct-Docker fixture value; no whole Go file is exempt.

Testcontainers’ cleanup image is separately pinned in internal/testkit/ryukimage/image.txt, acquired through the same bounded helper, and exported to testcontainers only by the exact test-pg recipe while Ryuk remains enabled. The Ryuk pin is a fully-qualified docker.io name; the Postgres pin is its implicit library/... form because the product recipe sets the exact non-empty TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX=docker.io, which takes precedence over an operator’s ~/.testcontainers.properties. Testcontainers-go v0.44 prepends that runtime authority exactly once to the implicit Postgres pin, while Docker resolves the helper’s same implicit form to Docker Hub; an inherited prefix therefore cannot transform either product image into a different pull. TESTCONTAINERS_RYUK_DISABLED=false likewise overrides a user property that would disable cleanup. The Postgres and Ryuk recipes read their repository-owned pin files directly; no environment or command-line Make variable can replace either authority. A focused contract binds the qualified Ryuk file to the installed testcontainers module’s exported default plus the exact Hub prefix, so a dependency update cannot silently introduce a second image. Each product Make target depends directly on one shared ensure target. Release verification requires the ensure and protected product targets to remain phony, checks each protected target’s own exact recipe, and closes the full prerequisite graph over an exact target/recipe allowlist: the five Playwright targets may reach only their shared ensure target and their existing Storybook or frontend build prerequisite, while test-pg may reach only its shared ensure target and the Rust worker build. A wrapper or changed prerequisite therefore cannot hide another runtime, helper, service, pull, or alternate container acquisition. The five Playwright recipes and their ensure target invoke only fixed runner modes; removed PW_DOCKER_USER, PW_CI, PW_REAL_CI, PW_IMAGE, and PW_SHARD variables are unexported and forbidden as Make assignments. Command-line, environment, or recursively expanded values therefore have no pre-image argument seam. Make’s dependency graph executes the shared phony ensure target once even when the Playwright targets are selected together with -j. Every Make recipe that invokes the bounded helper, reviewed runner, a fixed Docker/Podman/nerdctl/Buildah/docker-compose executable, or a variable engine with container grammar seeds the workflow-protected target set, regardless of global options, Compose options, or the eventual verb. GNU Make’s target: prerequisites ; recipe form is parsed as an inline recipe rather than as prerequisite text. Backslash-newline recipe continuations are lexically spliced into one logical command before that classification, including commands reached through reverse prerequisite closure. Exact active literal Make assignments are resolved only for command classification. Outside recipes, GNU Make’s first unescaped # begins a comment even when it is adjacent to a value or prerequisite; an escaped \# remains data. Recipe and inline-recipe text uses the shell’s distinct comment rule, where an adjacent hash remains part of its token. Audited non-recipe Make source rejects logical continuations, define, and parse-time eval rather than partially interpreting those contexts. An unresolved Make command variable followed by container grammar fails closed. An unresolved variable in recipe command position also seeds the protected set, so a helper or runner supplied through a Make command-line or environment override cannot hide behind $(RUNNER). The root’s exact GO and CARGO assignments are resolved before that check, and recursive $(MAKE) recipes are followed through the protected-target graph, so known benign tools do not become container routes. Literal shell command substitutions inside recipes are traversed through the same recursive-Make closure. Recursive Make admits only a positive numeric -jN or --jobs=N parallelism option and no variable assignments; even a non-graph assignment can retarget a command-position variable in a reachable recipe. Leading command-prefix assignments and assignments consumed by nested env wrappers remain attached to the effective recursive Make invocation, so wrappers cannot erase that fact. Alternate Makefiles, directories, includes, evaluation, execution controls, dynamic option values, and every unknown option fail closed because they can select a different graph or executable. The shared command walker consumes the closed option grammar for env, command, exec, timeout, nice, and xargs before classifying the effective executable; a wrapper operand cannot be mistaken for the command. Shell eval bodies are recursively classified, with only the exact checked-in dev-env.sh export import admitted for the image tooling recipes. Release verification never launches a host Make executable. A checked-in minimal Make-policy fixture and golden normalization/closure snapshot preserve the security-relevant adjacent/escaped-hash boundary, prerequisite graph edges, inline and continued recipes, shell-recipe hash tokens, recursive protected-target reachability, and untrusted alternate-Makefile rejection. The policy parser canonicalizes harmless surrounding whitespace, including the trailing space GNU Make retains before a whitespace-delimited comment; the fixture is not a claim of byte-for-byte GNU Make evaluation. Independent public mutations own continued assignment and define/eval rejection, parse-time side effects, and untrusted recursive graphs. This keeps the required policy hermetic across Linux, Homebrew, and the macOS system Make handoff. This includes the existing local dev and dev-gpu Compose targets. Reverse prerequisites and recursive Make recipes close that set transitively. Classification grants no new authority: a workflow route to an arbitrary engine-bearing target is rejected; only the exact audited Postgres and Playwright commands below may reach their separately verified image authorities. Repository shell scripts invoked by a reachable Make or workflow route are indexed and traversed recursively. Literal paths, transparent wrappers, prefix assignments, shell interpreters, and unique repository-script suffixes share one bounded 32-edge, cycle-safe traversal. Docker, Podman, nerdctl, Buildah, Compose, variable-engine acquisition grammar, if/then bodies, and input or output process substitutions are classified without executing the script. A cycle is not acquisition by itself, but any acquisition-bearing member poisons the complete reachable cycle.

The reusable Postgres workflow retains one exact make test-pg TEST_PG_SUITE=${{ matrix.lane }} step, run once per suite: its run job’s matrix must list exactly store, backendtransition, and app with fail-fast: false, so each suite gets its own runner (still -p=1 under the race detector) and no suite can drop out of the gate. Unset, TEST_PG_SUITE runs all three. Across every job and step, release verification admits only the current workflow, run job, and exact step schemas. It rejects job containers, services, unaudited reusable-workflow delegation, container-acquiring actions, unknown keys, metadata-only or malformed steps, multiple execution mechanisms, and unapproved environment, defaults, condition, error-tolerance, shell, working-directory, or MAKEFLAGS controls. The root tool-version environment and the cache-save condition are admitted only at their exact current values. Its only permitted shell steps are the exact cache-epoch command and the suite’s make test-pg; this closes dry-run, variable, wrapper, and quoting indirection without parsing arbitrary shell. Every repository workflow is also parsed structurally. Any service or job container or docker:// action is rejected. Workflow shell is not interpreted: route-shaped steps are classified from the raw scalar and statically declared workflow, job, and step environment, then compared with exact workflow-name-and-whole-scalar authorities. The classifier recognizes Make by executable basename, including absolute paths and wrappers, the bounded helper and Playwright runner wherever they occur, and fixed Docker, Podman, nerdctl, Buildah, and Compose acquisition grammar across quoting and command-substitution text. It applies the shell’s backslash-newline splice before classification, so an engine, helper, runner, or Make executable cannot be split across YAML scalar lines. Command and process substitutions are inspected recursively without evaluation; shell control words cannot hide an engine command, and ambiguous acquisition-bearing syntax fails closed. Ambiguous nesting and a variable executable inside any nested env, command, or interpreter command fail closed. Any variable in executable position is rejected generically, including adjacent or braced construction and invocation through exec, source, ., quotes, or wrappers. Legacy backtick substitution is not evaluated or partially parsed: balanced, unbalanced, escaped, nested, wrapped, and CRLF forms all fail closed unless the entire checked-in step matched an earlier source-bound authority. A candidate step containing assignments, variable executables or targets, command substitution, control operators, or any other added shell context is not evaluated and cannot match an authority; it fails closed.

The only bounded test-image download routes are the exact per-suite make test-pg step in ci-postgres.yml, the exact visual shard invocation, and the exact make e2e step in ci-playwright.yml; ci-image.yml retains one source-exact local packaged-image metadata inspection that cannot pull an image. Release publication’s two existing manifest/signing scripts are also source-bound exact authorities because their audited bodies use Docker buildx manifest commands; they do not become test-image acquisition authorities and cannot be replaced by an arbitrary script. Each bounded test-image authority is required exactly once at its declared workflow, job, absolute steps[] index, step kind and name, whole command, target set, and inherited execution context. Pinned action steps count in that absolute index, so inserting, removing, or moving either a shell or action step cannot preserve an authority accidentally; removal, duplication, movement, relabelling, swapping, or an extra Playwright mode fails repository verification. Every registered run authority, including container-free harness and documentation commands, is also required exactly once. Its job name, runner, dependencies, timeout, and typed strategy/matrix shape are source-bound alongside the existing workflow/job/step environment, permissions, condition, shell, and working-directory controls; absence, addition, or replacement changes the authority rather than silently changing where it executes. Every pinned action occurrence in every checked-in workflow is a complete exact authority at its workflow, job, and absolute step index. Its step kind, name and id, full-SHA uses scalar, complete with inputs, condition, and environment must match the checked-in authority; the repository inventory proves both that no action occurrence is unregistered and that no registered workflow or occurrence disappeared. Action-only workflows receive the same exact workflow/job environment, defaults, permissions, runner, dependency, timeout, deployment-environment, and typed strategy/matrix audit. Integer authorities must remain YAML integers rather than numerically equivalent quoted strings. An action-to-action or action-to-run same-slot replacement is therefore rejected without resolving the action over the network. Overrides or additional shell syntax make those routes non-exact and are rejected. Existing unrelated Make steps are separately source-bound by workflow name, exact whole scalar, and their explicit target list. They remain admitted only while the independent Make graph proves every named target is outside the acquisition-bearing reverse closure; changing an Android, cache-cleanup, Go, or other admitted target into a container route therefore invalidates the workflow authority. This is a closed checked-in command grammar, not arbitrary shell data-flow interpretation. Every protected and currently unrelated Make authority binds the workflow name, job name, exact run scalar, absolute steps[] index, run-step kind and exact name, explicit target list, and complete inherited workflow/job/step execution context together. Workflow and job conditions, step conditions, legitimate step environments, and the packaged-image inspection’s bash shell are admitted only at their exact current values. Any additional or changed environment, defaults, condition, error tolerance, shell, or working directory rejects the authority. Thus MAKEFLAGS, GNUMAKEFLAGS, MFLAGS, MAKEFILES, MAKEOVERRIDES, MAKELEVEL, MAKE, SHELL, PATH, BASH_ENV, ENV, GOENV, GOFLAGS, and equivalent controls cannot redirect or suppress an exact-looking command. An exact command is authorized only at the scalar node inside a real jobs.<job>.steps[] entry whose complete inherited context passed this audit; matching text in a malformed or metadata-only YAML shape receives no exemption. Job-level uses: is rejected except for the root CI workflow’s exact local family calls. Those calls admit exactly name, needs, if, and uses at their source-bound values and are closed by VerifyCIFamilyWorkflows: each job must name its registered local workflow and the target must expose only workflow_call. Product families contain exactly one implementation job. The isolated Apple cache portability workflow is the sole registered exception: its exact producer and dependent consumer jobs prove transfer across distinct runners. Every implementation job is scanned independently by this container policy. Remote, renamed, additional, or otherwise unresolved reusable workflows fail closed.

The workflow topology is itself an authority: the checked-in workflow filenames, job names, step counts, and run-versus-action kind at each slot must match exactly. An extra workflow, job, or step therefore fails before command classification; pinned actions and registered run authorities retain their deeper exact content and context checks. Topology, run commands, pinned actions, job contexts, reusable callers, and family-workflow paths are constructed together through one typed catalog. Each verification receives fresh maps rather than package-mutable authority state, and a drift contract rejects missing workflows, jobs, contexts, actions, commands, or family registrations and colliding absolute step identities. The cache-cleanup workflow is one such run authority: its complete deletion command, step name and environment, exact Cache cleanup workflow name, closed pull_request trigger with only the closed type, workflow permissions, job name and runner, and absence of alternate execution context are source-bound rather than inferred from its one-step topology.

Make execution is closed over the checked-in root Makefile and every mk/*.mk: each module must be included exactly once, and non-recipe syntax is limited to the current plain assignments, targets, and exact root includes. Conditional directives are rejected rather than interpreted; define/endef, dynamic evaluation, modifiers, alternate includes, and non-recipe logical continuations are also forbidden. Recipe continuations are instead normalized as shell lexical splices and audited as one logical command. Every parenthesized or braced Make reference in assignments, targets, and every logical recipe is structurally balanced. Escaped $$ recipe expansions remain shell-owned, while Make-owned references are inspected before shell command classification. Ordinary variable references and the current pure if, or, sort, subst, and wildcard functions are allowed; shell, eval, file, Guile, != shell assignments, call, dynamically constructed function names, value, foreach, and any other function are rejected even when nested or continued inside an otherwise allowed assignment, recipe, or pure function. Playwright host-user discovery now occurs inside the exact reviewed runtime script, so Make has no parse-time shell exception. The Postgres image pin remains one Go-embedded and Make-readable file, and the Ryuk pin is a second repository-owned testkit file. Make reads both by literal path with recipe-time cat immediately before the bounded helper rather than exposing an overridable variable or exempting a parse-time file function. GO ?= go and CARGO ?= cargo are fixed, and global shell, executable, flag, recipe-prefix, and special-target controls that could suppress or redirect a protected recipe are forbidden. The always-on ci-policy job has a source-bound bootstrap chain: exact pinned checkout and setup-go actions, then go run ./cmd/releaseverify -root . as the first shell step, followed by the three exact Make/Go policy commands. Its always() condition is source-bound and independent of impact-classifier output. The workflow root admits only exact Go and Node version environment values and no defaults; the job and all six steps admit no inherited environment, alternate shell, working directory, tolerance, or unknown execution keys; the job’s existing needs, condition, and runner are source-bound, and no step may add a condition. Setup-go’s exact inputs are also bound, so a preceding action cannot redefine the Go executable or path. BASH_ENV, ENV, PATH, GOENV, GOFLAGS, constructed values, or an alternate Make/runtime context therefore fail before the checked-in policy can claim protection. The exported direct-bootstrap verifier enforces this independently and VerifyCIContainerDownloads composes it into repository policy. A Make-only mutation cannot suppress this audit. This bootstrap still has an unavoidable self-verification boundary: the required-check configuration, GitHub Actions runner, and review of a change that removes or poisons the direct step must cause the verifier to run before the repository can inspect itself. Release verification therefore proves the checked-in structure once invoked; it does not claim authority over the hosting platform or an execution that never starts.

The public verifier admits only the exact reviewed helper and Playwright runner sources, so no test-only package links into cmd/releaseverify. Tests execute that source through the shared testkit process boundary with fake local Docker and sleep executables to prove exact image arguments, cached zero-pull behavior, the five-attempt/four-wait bound, wait failure, and final-status propagation without a registry or network.

The helper’s classifier consumers are repository contracts, Postgres, visual, e2e, tuner, and CI policy. The Playwright runner selects repository contracts, visual, e2e, tuner, and CI policy. The broad Web unit gate is not selected: its make fe-unit and make fe-checks paths never invoke either script, while each container-backed Playwright target and test-pg owns an explicit prerequisite. Any production or test change under internal/platform/releaseverify or cmd/releaseverify selects both the policy job and the sharded Go job, because only the latter executes the package’s RED-capable tests.

Playwright is the second active job. Its four shards consume the union of impact_visual and impact_e2e; the combined job still runs both suites exactly as before. Shipping Web runtime sources are conservatively visual-sensitive because Storybook alias imports make filename-only transitive narrowing unsafe. Shared API/core/fixture inputs and OpenAPI select both suites, while visual/e2e tests and committed baselines select their owner. Only proven unit-test-only Web sources skip Playwright in this first slice.

Tuner is the third active job. Its macOS browser matrix consumes impact_tuner directly. Every shipping Web runtime source remains tuner-sensitive because the matrix loads the real SPA and HLS controller; unit, spec, and story-only modules may skip it. Tuner e2e inputs, browser build configuration, shared API/core/fixture/player packages, runtime tokens, and OpenAPI select it explicitly.

Apple mobile is the fourth active decision and Apple TV is the fifth. iOS and tvOS are separate top-level jobs with hard-coded app commands, dedicated impact selectors, and independent required results. Existing cache-key strings are preserved so splitting job identity does not discard compatible pnpm, CocoaPods, or ExpoModulesJSI entries.

Every job consumes its dedicated impact_* output. A missing base, classifier failure, or unknown path selects every specialized gate. The manual release-candidate scope remains unchanged and excludes Postgres, Playwright, tuner, and application-client builds.

scripts/testdata/ci-impact.tsv records the exact ordered gate set for representative paths and multi-path changes across every specialized gate. The classifier contract test compares complete sets, so both a missed gate and an unexplained extra gate require an explicit fixture decision.

Non-code files consumed by Go tests are Go inputs too. In particular, design/configuration/command docs, install docs and README, the committed OpenAPI document, production Compose, and embedded help select the complete Go test set. Dockerfile and packaged license/notices select repository contracts as well as the image build. These mappings are fixtures because file extensions cannot reveal those dependencies.

Client decisions follow the actual consumer graph. An app-local mobile change selects shared-client, iOS, and Expo Android mobile evidence; a TV change selects shared-client, tvOS, and Expo Android TV. Changes to api, core, fixtures, player, design-system, or ui select both apps on both native platforms because those packages are transitive inputs to both. The shared player package also selects Web, Tuner, Image, and Android TV because its browser and native adapters share one transport contract. Browser-only client-proof and Turborepo contract changes select the shared JavaScript gate without spending a native runner. The committed OpenAPI document is a schema input rather than a native binary input. An OpenAPI-only change regenerates the client, then the shared-client gate typechecks, tests, and bundles the browser, iOS, Android, and TV JavaScript consumers. A mobile or TV app change, a shared native/runtime package, native build tooling, or a platform workflow still selects the corresponding native compilation. This keeps additive unused endpoints from compiling product binaries while a real consumer change continues to fail closed through its own path. Apple mobile, Apple TV, and Expo Android mobile are active. The mobile Android job consumes only impact_expo_android_mobile, builds CLIENT_APP=mobile, and retains its standalone debug APK for seven days. Expo Android TV remains a classifier decision until its independently required job and current-main evidence exist.

make client-android-debug keeps workstation-safe defaults: one Gradle worker, one native job, one Metro worker, a 1280 MB Gradle heap, and a 4 GB memory scope. The CI mobile job raises them through the environment, the same way the TV producer does: two Gradle workers (the wrapper’s ceiling), four native jobs for the single-ABI app CMake build, four Metro workers, a 3 GB heap, and a 12 GB scope on the 16 GB runner. It reuses the TV producer’s pinned ccache, the same wrapper validation (web/scripts/android-ccache-env.sh), and the same generated-Ninja launcher proof. It restores no cache: it builds a debug variant, so TV’s release objects never hit for it.

The permanent-identity Android TV producer restores a 2 GiB ccache directory across source identities and keeps no Gradle cache. The workflow installs the exact §14 pin, while the build wrapper rejects a relative/wrong-version launcher, a foreign base directory, or any sloppiness setting. Ccache uses compiler-content identity and checkout-relative paths; the key may restore prior compiler objects, but it never includes generated android/ or .cxx trees, bundles, signing material, or evidence. Version, configuration, pre/post JSON statistics, and generated Ninja launcher coverage are retained with the existing Android profile artifact. CI also permits two Gradle projects to execute in parallel while keeping each CMake task’s compile/link pool at one; local builds retain the one-worker default, and the wrapper accepts no value above two. A developer build automatically acquires the same pinned macOS or Linux tool into .artifacts/, reuses its local compiler-result directory across later builds in that worktree, and falls back to the unchanged cold path when acquisition is unavailable. Three Gradle workers are intentionally rejected: the hosted comparison regressed from 16m15s to 19m07s without adding cache hits. Set LOOMARR_ANDROID_CCACHE=off to request the cold path.

GitHub scopes pull-request caches to their merge refs and merge-queue caches to disposable queue refs, so neither can warm the next queue candidate. After a successful Android queue producer lands, it instead hands its bounded ccache tree to android-ccache-promotion.yml through a one-day immutable artifact. The promotion starts when that merge-group CI run completes successfully (workflow_run), because the run can finish after the merge (#2288). It publishes only a commit main already contains, and proves the source CI run, Android job, artifact name, commit/run/workflow manifest, and cache-tree digest, then saves that generation in the default-branch cache scope without running another product build. It deletes the transfer and all superseded Android generations. Android jobs do not run on pull requests, and queue jobs restore but never spend time saving an unreachable cache.

Per-run measurements

The required CI aggregate appends a timing table after it has evaluated every required result. scripts/ci-run-metrics.sh reads GitHub’s run and job records and reports queue delay, execution time, end-to-end time, the longest job, and total occupied runner time. The report is observational: an API or checkout failure emits a warning but cannot turn verified code red or hide a failed gate. Its formatter is tested against a pinned API fixture without touching the network.

The distinction between queue and execution is load-bearing for native work. A macOS job can be the critical path because it waited for capacity, because Xcode compiled slowly, or both; changing cache policy cannot solve the first case, while adding shards can make it worse.

The image job is the exception

The image filter follows every source family copied by the Dockerfile: Docker metadata, packaged LICENSE/notices, Cargo and Rust sources, Go sources/modules/embedded migrations, embedded help, the frontend, OpenAPI, and the bundle guard. Makefile and workflow-only changes do not change image bytes and therefore do not trigger it.

It builds each release platform on a native runner, loads the resulting image without pushing it, and inspects the packaged LICENSE/notices and OCI labels. The Dockerfile’s build-time commands prove the bundled tools; the post-build inspection proves the final runtime filesystem rather than a comment or an intermediate stage.

The image job has a timeout-minutes because its release builds are independently bounded; GitHub’s default is six hours.

The reusable Apple mobile and Apple TV jobs have explicit job limits of 75 and 60 minutes. These bounds are evidence-derived from recent successful merge-group runs (mobile: 29:14–50:46, TV: 16:38–34:49) and cover prebuild, dependency installation, Xcode build, simulator readiness, installation, and launch. Review them after at least 20 successful native runs or at the first native timeout. A hard timeout can prevent the screenshot step from running; the screenshot remains useful for ordinary failures but is not guaranteed after timeout.

It exists because a Dockerfile that could never build for arm64 sat undetected. Build both platforms or it can’t catch that.

Playout bench

ci-playout-bench.yml runs the software-only playout bench on every pull request that touches playout, the bench or its scripts. It is selected by the playout_bench classifier gate and is part of CI, so it is covered by ci-ok. The hardware runs live in playout-bench.yml, which is workflow_dispatch and nightly only and must never gain a pull_request, pull_request_target or merge_group trigger: the repository is public and a pull request must not schedule work on the maintainer’s GPU machines. Playout bench has the metrics, thresholds and baselines.

Manual scopes are explicit

Manual CI defaults to release-candidate. That scope is for certifying an exact main commit before tagging: it runs repository contracts, the real-codec image-worker certification, and both native release-image builds. It does not rerun Android, PostgreSQL, Go race, frontend, Playwright, tuner, docs, or the macOS harness; their normal push and pull-request impact coverage is unchanged.

Select full explicitly when an investigation genuinely needs every matrix. Both modes publish a scope-marker job, but scripts/validate-release-source.sh accepts only the release-candidate marker for Docker publication. It rejects normal push CI and full manual runs even when green, so mobile, client, and unrelated platform matrices cannot become release prerequisites. The release-candidate marker also makes the contract and certification jobs mandatory evidence rather than relying only on the workflow’s overall conclusion.

Select apple-cache-validation only while proving the Apple compilation-cache protocol. It runs no ordinary product family: one xcode-27 job performs the complete mobile and TV Release install-launch-liveness gates while populating an LLVM CAS, validates and packs that store, and transfers it as a one-day workflow artifact. A dependent xcode-27 job restores and validates the archive before proving mobile and TV hits, a real Swift source-change hit-and-miss, corrupt-archive rejection, fingerprint invalidation, and cold mobile/TV fallback. The runner boundary is deliberate; a cache that works only within one machine is not eligible for later CI consumption. This manual scope is included in ci-ok when selected and is skipped for pull requests, merge groups, release certification, and full dispatches.

Select tuner to measure a browser-engine-specific tuner flake without paying for the rest of the matrix (#1443). It runs only the tuner job, on the manual-tuner lane, and takes two optional inputs: project (all, chromium, firefox, or webkit) and repeat_each (1 to 100), which scripts/tuner-args.sh turns into Playwright --project and --repeat-each flags. Invalid values fail closed rather than running less than asked, and retries stay owned by playwright.tuner.config.ts (0), so a repeated run can never hide a failure behind a retry. At the defaults (all, 1) the script prints nothing and make tuner-e2e-host is byte-identical to the merge-queue run. Other scopes ignore both inputs, and pull requests and merge groups never see them.

Flake quarantine (#1570). A tuner test with an open flaky issue can leave the required matrix without being skipped. It enters quarantine only by a line in scripts/flake-quarantine.tsv that names the suite, project, spec file, exact test title and tracking issue. In pull requests and merge groups, make tuner-quarantine keeps the lines whose issue is still OPEN. The required step runs every other (project, test) pair, and a later non-blocking step (make tuner-e2e-quarantine) runs the quarantined pairs, with zero retries as always. Its failure shows on that step and uploads tuner-quarantine-failures, but it cannot fail CI.

When the issue closes, the test is back in the required step on the next run; the line then only prints a notice until someone removes it. Every doubt resolves toward required: an issue whose state can’t be read, a malformed line, or an unknown mode fails closed. The manual tuner scope turns the quarantine off, so it measures every test. That non-blocking step is the only source-bound step the workflow policy lets set continue-on-error (nonBlocking in internal/platform/releaseverify).

After that portability proof is green, Apple compilation cache is the only workflow authorized to publish compiler results. It is manual-only, refuses every ref except refs/heads/main, and builds the complete mobile and TV Release install-launch-liveness gates before saving. A restored seed is hash-validated before use; the candidate CAS and compressed archive are validated again, and a save is refused if its size plus a 512 MiB reserve would exceed the repository’s 10 GiB cache budget. Before installing dependencies or starting either native build, the writer also requires enough live headroom for the maximum permitted archive plus that reserve; this conservative preflight avoids spending the full cold-build window on a candidate that cannot possibly be published. The post-build admission still uses the archive’s exact compressed size. After a successful save, cleanup is limited to the exact fingerprint prefix on refs/heads/main and retains one generation. Only this workflow has actions: write for the compiler cache.

The ordinary Apple mobile and Apple TV jobs are compiler-cache consumers only. Each computes the toolchain/SDK/native-input fingerprint, uses the restore-only cache action, and exposes a store only after zstd and LLVM CAS hash validation. Missing or rejected archives select the unchanged cold gate. A valid store that fails during the warm build is quarantined and receives one clean cold retry; every architecture, install, launch, screenshot, and liveness assertion still runs. These jobs never save the compiler archive, and the obsolete pnpm/CocoaPods and ExpoModulesJSI lookup steps have no writer and are omitted. Pull-request and merge-queue refs therefore cannot create sibling-scoped Apple generations that later groups cannot read.

The Apple verifier starts the selected simulator before Expo’s required clean prebuild, then runs pod install explicitly while the simulator boots. simctl bootstatus remains a hard barrier before Expo builds, installs, or launches, and expo run:ios --no-install skips only the duplicate CocoaPods invocation. The uploaded phase-timings.tsv records clean prebuild, CocoaPods, simulator readiness, native build/install, and artifact/runtime assertions. Simulator readiness contains the overlapped prebuild and CocoaPods intervals, so those rows are deliberately not additive.

Do not add a generated ios/Pods cache without re-proving both integrity and repository capacity. Measured 2026-08-31, one clean mobile native workspace was 981 MiB and compressed to 339 MiB while the repository already held 9.11 GiB of its 10 GiB Actions-cache budget. Restoring Pods alone still pays CocoaPods project generation; restoring the integrated workspace overwrites clean-prebuild authority. Neither seam justified consuming the remaining cache headroom.

ci-ok is the only required check

It always runs and inspects needs.*.result explicitly. That has to be explicit: a skipped job doesn’t fail an aggregate by default, and neither does a failed one under if: always().

A cancelled job passes only on a pull request, where a newer push superseded it. In the merge queue and on manual runs it fails the check: a job that crosses its timeout-minutes reports cancelled with its tests unfinished, so it has proved nothing.

The main branch requires the GitHub Actions-owned CI check in strict mode: a pull request must be tested against the current base before it can merge. Preserve both the check name and its app binding when editing branch protection.

make release-verify parses the workflow and requires every top-level job to appear in ci-ok.needs. This prevents a newly added or accidentally removed dependency from producing a green required check while its real job is red.

Never add a workflow-level paths:. A run that doesn’t trigger reports no checks, so a required check sits “expected” forever and the PR can’t merge. Filter per job.

The workflow handles merge_group, and the organization-owned repository requires the merge queue through its main ruleset. The strict, GitHub-Actions-owned CI check remains the protected current-base boundary inside the queue. Never remove the queue rule merely to bypass a delayed or failing native result.

Sharding

Go tests, frontend and Playwright split across runners for wall-clock only. Repository-wide Go contracts run once: the workflow, release, harness, shell and generated-docs checks in CI policy on every PR, and the rest in go-contracts, whose two shards share them without overlap. Rust contracts once in rust-contracts, in parallel with two ordinary Go test lanes, two serial lanes for packages that must not share a runner, and the independent release-worker certification. Their union is the same assurance as make verify SCOPE=all plus the existing CI-only certification. The ci-ok aggregate requires every job, so moving a contract out of the test lanes cannot make it optional.

make go-shard-verify runs in go-contracts and asserts the Go shards are a true partition of go list ./... — a split that drops a package would otherwise pass by not running it.

The Go partition uses longest-processing-time assignment over a reviewed table containing only material package timings. Packages below five measured seconds, including new packages awaiting a hosted measurement, receive a one-second planning floor and remain fail-safe members of the exact partition. Merge-group run 35472062915 exposed 1430/657/583 package-seconds in the former three-way alphabetical split and a 31m10s critical path. Two ordinary lanes use bounded -p=4 package parallelism. The reviewed latency-sensitive playout/capacity set runs in two concurrent -p=1 lanes, so each group remains serial while separate runners safely overlap the groups. Merge-group run 35497917692 measured the former combined certification lane at 9m53s, including internal/app at 159.215 seconds and the complementary playout packages at 157.874 reported package-seconds. That natural seam produces modeled certification groups of 173 and 172 seconds after the latest hosted-weight refresh. The same run measured the refactored internal/proposals/suggest at 34.908 seconds; refreshing its obsolete 368-second weight removed that package bottleneck. Merge-group run 35500779353 then put every Go job between 6m59s and 9m01s, but the ordinary job spread was still about 29%. Its package summary exposed internal/people/auth at 300.797 seconds and refreshed every material upper-envelope weight. Both auth test-store helpers had independently replayed the complete SQLite migration history for every private test database. Routing them through the existing isolated migrated template reduced the local SSO race subset from 30.59 to 5.64 seconds and the complete auth race package from 118.89 to 19.64 seconds, with the real RS256 verifier, allowlist refusals, disabled-person checks, session revocation, password hashing, and every existing assertion retained. A conservative provisional auth ceiling first produced a six-lane plan of 372/371/371/371/371/371 seconds, but the repaired tree no longer has enough ordinary work to keep six hosted runners efficiently occupied. The first authoritative current-tree four-ordinary-lane run, 35504999984, validated the repaired auth package at 23.614 seconds and completed all six Go jobs successfully. Its ordinary job/test times were 9m36s/9m02s, 8m31s/7m52s, 7m02s/6m28s and 9m30s/8m52s. The 9m53s queue-to-finish critical path was materially lower than 14+ minutes, but the roughly 40% ordinary test-time spread still failed acceptance. The run therefore did not merge. Refreshing every material weight from that exact tree with uniform 10% headroom produced checked-in aggregate loads of 533/533/532/532 seconds and bounded-worker makespans of 267/267/267/266 seconds. The next authoritative run, 35507559795, proved why more shard tuning was the wrong fix: lane 2 reported only about four minutes of longest-worker package execution (internal/platform/store at 239.922 seconds), yet spent 9m37s in the test step. Its cache log showed a complete miss followed by repeated module downloads, a 40.94-second Rust build, and a 55-second eval-contract invocation before the lane even started its ordinary package phase. Those prerequisites ran identically in all six jobs.

The resulting architecture preserves the same work with fewer runners. Two ordinary lanes use bounded -p=4, and the two serial certification lanes remain unchanged. Lane-scoped make test omits the eager Rust and eval prerequisites: go-contracts executes eval-contract once, while composition packages that need the real image worker acquire it through internal/testkit and pass its executable at the explicit application override seam. Unsharded local make test retains both explicit prerequisites and remains self-contained.

The first authoritative two-ordinary-lane run, 35510845578, proved that balancing aggregate work was still insufficient. Both certification jobs passed in 5m57s and 7m15s from queue creation, but ordinary test steps took 10m23s and 10m41s and completed 11m18s and 11m35s after queue creation. Their 18-second spread ruled out shard imbalance. The executable package order was the real defect: the sharder modeled longest-processing-time workers, then restored go list order, and the race policy sorted that order lexically again. internal/platform/store and internal/channels/recurate consequently started about 6m21s and 6m34s into their test steps. Four large, fixture-isolated packages also ran all top-level tests serially. The runner now emits and preserves descending measured-cost order, and research/internal/fillerreview, internal/playout/backendtransition’s non-integration tests, internal/channels/recurate, and internal/channels/approval use bounded t.Parallel execution. The PostgreSQL-tagged backend tests remain serial because they mutate process environment. Same-machine four-CPU race controls measured fillerreview at 115.237s serial versus 41.154s parallel; the other three packages completed together in 58.83s versus 82.61s with their internal parallelism disabled. Scaling the hosted weights by those ratios and repartitioning yields 921/920 aggregate seconds and 244/231-second bounded-worker makespans. Those scaled weights remain provisional until consecutive successful merge-group runs replace them with complete hosted measurements.

Merge-group run 35515087033 measured ordinary lane 1/2 at 8m31s job time and 8m02s test time, while ordinary lane 2/2 reached its test failure in 8m10s. Certification lane 2/2 passed in 5m09s; lane 1/2 passed in 7m00s after waiting 2m39s for the repository’s 20-runner concurrency ceiling. The run was not accepted: lane 2/2 exposed that the integration composition harness had relied on the old eager Rust build instead of acquiring its required worker. The harness now obtains the worker through internal/testkit and passes it through app.Overrides.ImageWorkerExecutable, so a focused clean go test ./internal/integration is self-contained without rebuilding Rust in lanes that do not consume it. Fresh consecutive merge-group passes remain required.

The weights are now regenerated, never hand-kept: make go-race-weights RUNS="..." (scripts/go-race-weights-refresh.sh) takes the median ok time of every package across the race-lane jobs of the given merge-group runs, rounded up. A lane that timed out still counts for the packages that finished in it: ok is printed only for a whole run, and the lane over its timeout is the one that needs re-measuring. Regenerated from runs 36330002206, 36330402570, 36330935829, 36331908558 and 36333318208 (#1570), the real costs were about 2.4× the earlier weights (internal/platform/store 516s against 242), and the old unitless budgets had been passing on those stale inputs.

Race lanes build modernc’s SQLite with -gcflags=modernc.org/...=-d=checkptr=0 (scripts/go-test-packages.sh). -race turns on checkptr everywhere, and modernc is C translated to Go, with unsafe pointer arithmetic on nearly every line: checkptr there doubled the cost of every migration (#2716). The race detector still instruments modernc, and checkptr stays on for every other package. Tests copy a migrated SQLite template rather than migrating a fresh file (testkit.MigratedSQLiteDSN, enforced by internal/migratedstore_test.go).

The budgets derive from #1570’s target, a Go-only merge-queue run in at most ten minutes (./scripts/go-shard.sh --budgets). The measured queue time outside a lane’s test step is 131s: 116s median from queue start to the test step, 4s of teardown, and 11s for the CI aggregator. That leaves a 469s lane test step. An ordinary lane’s summed package-seconds must fit four -p=4 workers in that step (1,876s). Its bounded-worker makespan, and each serial certification lane, must fit the step itself. No single package may exceed that step either: a package is the sharder’s indivisible unit, so an oversized one must run its tests in parallel, move to a serial lane of its own, or be split, never be given a larger budget. (Until #1650 a temporary cap of 567s, internal/platform/store’s measured time plus 10%, stood in for that rule.)

Pull request #1631 made the 21 heaviest internal/platform/store tests (88% of its time) run in parallel. Pinned to four cores locally, that cut the package from 146.9s to 110.1s, but merge-group run 36359955148 measured it at 542s in ordinary lane 1/2: beside three other -p=4 packages on a four-vCPU runner, its parallel tests only competed for the same cores. So the store package (now internal/platform/store) runs alone in the second serial lane, where one package at a time owns the runner, and the playout/capacity set shares the first. There, workflow-dispatch run 36366596157 measured it at 171s, which left it too light to balance the first lane’s 405s, so internal/integration, whose tests are serial, shares its lane. The serial lanes hold two kinds of package: media certification that must not compete for a worker while asserting latency, and CPU-heavy packages whose own parallelism needs a runner to themselves.

Since #2716 there are three serial lanes, certification-1/3 to certification-3/3. The highest lane number in scripts/go-certification-lanes.tsv is the count, and the workflow matrix, cache keys and release-verifier pins name each lane. Two lanes passed the 15-minute job limit once internal/channels/schedule and internal/demolibrary joined them. Four lanes could not balance: merge-group run 43037503041 measured store alone at 291s, and no four-way split of the rest stayed inside the 125% balance budget. The three lanes model 322s, 336s and 378s.

make go-shard-verify SHARDS=2 rejects missing or duplicated packages, a package above the per-package cap, an ordinary aggregate or a bounded-worker or serial-certification makespan above those derived budgets, or more than 25% imbalance within any group. Aggregate and worker limits are deliberately separate: package overlap cannot hide unbounded total work, and a balanced aggregate cannot hide one saturated worker. Release verification additionally rejects an unreviewed serial package, grouping or workflow lane. The workflow also caps each whole job at 15 minutes, leaving six minutes for setup and compilation while making renewed latency drift a hard failure. TestGoShardUsesMeasuredLongestProcessingTime pins deterministic assignment and executable work order, TestGoRacePolicyPreservesSharderWorkOrder prevents the policy split from sorting it away, and TestGoShardBalancesMeasuredRaceWork pins the latency and balance budgets against the current tree.

Within the SQLite store package, conformance assertions clone one closed, fully migrated template database rather than replaying the complete migration history for every assertion. The 2026-09-01 profile measured 169.3s for the package and 134.3s for SQLite conformance: 131 fresh stores each replayed 86 migrations, more than 11,000 migration applications in one run. Every clone remains a private file opened through the production SQLite adapter. Migration ordering, boot seeding, downgrade behavior, historical data, and restart behavior retain dedicated fresh-migration tests; Postgres conformance remains unchanged. Re-running the exact race profile with the cloned fixture reduced the package to 40.3s and SQLite conformance to 6.2s (−76.2% and −95.4%, respectively). Five consecutive race-enabled conformance/factory runs remained green, followed by the unchanged real Postgres integration gate over store, backend transition, and app. In the authoritative queue, the race steps moved from 7m42s / 6m38s / 11m30s to 9m13s / 6m35s / 5m33s: the critical path fell 2m17s (19.9%) and aggregate race-step time fell 4m29s (17.4%), despite normal per-runner variation moving shard 1 upward.

Domain tests that need current persistence state, rather than migration behavior, use testkit.MigratedSQLiteStore for the same reason. A race-enabled 2026-09-01 profile found internal/channels spending 74.46s while its helpers replayed every migration for each private test database. Routing those helpers through the existing isolated migrated fixture reduced the exact package profile to 4.19s (94.4%). Tests that exercise startup, migration, downgrade, historical data, or restart behavior must continue to open and migrate fresh databases.

The Proposal Job worker tests use that same isolated migrated-store fixture and start a due-work drain before entering their two-second recurring poll. Profiling also found theme scoring recalculating the same series-title scope for every item and qualifier; preparing it once per score preserves every 2,000-iteration property assertion. On 2026-09-20, the one-time preparation reduced the complete local internal/proposals/suggest race package from 112.28 to 19.35 seconds after the worker/fixture changes; the deterministic property alone fell from 66.16 to 6.16 seconds. Hosted lane evidence remains the authority for the merge-queue budget.

PostgreSQL conformance also uses an isolated template, but through PostgreSQL’s own database-clone interface rather than file copying. Seven successful merge-queue samples put the real-Postgres step between 8m20s and 14m18s; internal/platform/store dominated every sample at 366–722 package-seconds while backend transition and app ran concurrently. The local race profile measured 124.43s for the full integration-tagged store package and 46.23s for Postgres conformance. Its factory had dropped and recreated public, then replayed all 86 migrations for every assertion. Migrating and boot-seeding one closed template, creating a private database from it per assertion, and opening each clone through the production Postgres adapter reduced the exact profile to 85.46s / 6.85s (−31.3% and −85.2%). Clone cleanup closes the Store before a maintenance connection force-drops only the named disposable database. Migration, preflight, and cross-backend data-migration tests still create fresh databases; queue evidence remains the proportional gate.

Sharding is free on a public repo. Check the bill before copying it into a private one.

Caching

  • actions/cache never overwrites an existing key. A cache whose contents track something its key doesn’t gets written once and frozen. Use a rolling key with restore-keys.
  • The 10GB cap evicts LRU across all refs, so closed PRs’ caches push out live ones. cache-cleanup.yml deletes both refs/pull/N/merge and matching merge-queue caches on close.
  • Default-branch cache writes require a trusted trigger. GitHub permits a default-branch writer from push or workflow_dispatch, but merge_group and workflow_run cannot promote their outputs into that scope. The Go cache-save conditions therefore remain useful only for a deliberate manual run on main; ordinary PR and queue runs are restore-only. Android is the bounded exception justified by the measured 25m54s cold queue build: the admitted queue run emits an immutable compiler-cache transfer, and the queue-produced push validates and saves it without repeating any compilation. A workflow_run writer or a push warmer that rebuilds the product remains forbidden.

Apple compilation caching is a validated artifact protocol, not an unchecked DerivedData restore. The fingerprint binds the runner OS and architecture, exact Xcode and Swift identities, both simulator SDKs, Release build settings, and the pnpm lockfile. Restores are decompressed into a temporary path and accepted only after the installed llvm-cas verifies hashes; failed warm builds quarantine that exact store and retry the complete gate once in cold mode. Archive size, repository cache usage, unrelated-cache headroom, and main-ref retention are checked before any trusted writer may save. PR and merge-queue consumers remain restore-only, and a warm-required validation cannot turn its cold fallback into a passing cache claim.

Go runtime workers use the image’s ffmpeg

Go runtime workers install the production Dockerfile’s retained ffmpeg build, pinned by architecture-specific SHA-256 for both ffmpeg and ffprobe, never the runner’s distribution package. The archive is cached by its exact digest and verified on every use before extraction or execution, and the installed pair must report the declared build. A change to the installer, the Go workflow or the Dockerfile’s pin source selects the complete Go runtime gate.

Hand-maintained lists, and what guards them

Three lists in this repo are written by hand and would rot silently. Each has an executable guard that fails when it drifts, and all three run in CI:

ListGuardRuns via
TAGS in the Makefilescripts/check-tags.shmake tags-verify, part of comprehensive verification
Retired identifiersscripts/check-retired.shmake retired-verify, its own CI step
Release-image source-family probesreleaseverify.VerifyCIImageInputsmake release-verify, part of comprehensive verification

tags-verify compares the tags in //go:build lines against TAGS and fails both ways:

  • In the tree, not in TAGS — those files are invisible to vet-tags and lint. Nothing compiles them, which is how a live ffmpeg test sat uncompiled for months.
  • In TAGS, not in the tree — the list claims coverage it doesn’t have. Drop the tag in the PR that removed its last file.

Downgrading the second direction to a warning is the obvious-looking fix when it’s inconvenient. Don’t — a warning printed by a job that exits 0 is one nobody reads.

The CI path filter includes scripts/, so a PR editing only a guard still runs the job that executes it.

Screenshot baselines render on every frontend push

visual-baselines.yml is outside the required PR gate. It runs on every pull_request push (opened, synchronize, reopened) from a branch of this repository that touches web/, and by manual dispatch. Its resolve job feeds the pull request’s changed files to scripts/ci-impact.sh and renders only when the visual or e2e gate is selected, so it follows the same classifier as the Playwright job. The render (visual-baselines-render.yml, dispatched at the branch, read-only) regenerates the visual and e2e snapshots in the pinned Playwright image. The push job commits only PNGs under a tests/visual/ or tests/e2e/ snapshot directory. It has a workflow-level paths: filter, which a required check must never have; this workflow is not one.

The bot’s push starts pull_request runs whose actor is github-actions[bot]. The repository’s fork-run approval policy is “first-time contributors”, and the bot counts as one, so GitHub parks those runs at action_required and the parked CI keeps the pull request out of the merge queue. The workflow’s release job approves exactly the parked runs on the commit it pushed, which needs only actions: write. Forks are untouched: their runs keep the approval rule, and the resolve job refuses a fork before any job holds a write token. A newer push cancels the older head’s run, and the bot’s own commit never starts a render. Testing has the full flow.

Scheduled Go dead code report

deadcode.yml is also outside the required PR gate. On the first of each month, and on manual dispatch, it counts Go functions no ./cmd binary reaches and opens or updates one tracking issue when the total exceeds the committed baseline. See Code hygiene.

Scheduled Go vulnerability scan

go-maintenance.yml is outside the required PR gate too. Every Monday, and on manual dispatch, it runs make vuln: pinned govulncheck over the untagged build that ships, failing on any known vulnerability a Loomarr call path reaches, standard library included. It is weekly rather than a PR gate because a newly published advisory would otherwise fail every Go PR regardless of its content. A red run is the prompt to bump the module, or go.mod’s toolchain line together with the Dockerfile’s golang image.

The same workflow runs make scan-scale: the library scanner’s scale guard over a 33k-file library, the household’s size. It checks that step 2 counts each file once, and that passes made while the header stage waits for playback leave nothing held. The PR suite runs the same guard at 2k files. At 33k it takes about a minute, which is too long for every PR.

Scheduled ffmpeg pin refresh

ffmpeg-pin-refresh.yml runs every Monday, and on manual dispatch with an optional dry_run. It runs scripts/ffmpeg-pin-refresh.sh automate. When BtbN has a newer release-series build, the script does four things:

  1. publishes that build’s third-party-sources-<date> release (binaries plus corresponding source);
  2. pushes deps/ffmpeg-<build>;
  3. opens the pin PR, as a draft labelled ready-for-human for a new series;
  4. dispatches ci.yml with scope=full at that branch, because a push made with GITHUB_TOKEN starts no workflow.

Its single job holds contents, pull-requests and actions write, one for each of those side effects. The procedure is in releasing.md.

Scheduled Rust maintenance

rust-maintenance.yml is intentionally outside the required PR gate. Every Monday, and on manual dispatch, it installs pinned cargo-deny and cargo-fuzz versions, checks both Cargo lockfiles for RustSec advisories, approved SPDX licenses, and untrusted sources, then fuzzes the worker’s bounded JSON-to-decoder boundary under nightly libFuzzer. A crash retains its reproducer for 30 days.

This job is allowed to be expensive and network-sensitive. The fast deterministic protections stay in comprehensive verification: Cargo lock enforcement, clippy/tests, and #![forbid(unsafe_code)] on Loomarr-owned shipping crates.