Testing
For: contributors writing or running tests. You’ll get: which suite covers what, and the rules every test follows.
Unit tests never touch the network. External services are mocked through internal/testkit —
extend it rather than writing a private mock.
The routine loop
go test -race -run TestName ./internal/<pkg>/ # while editingmake test-affected BASE=origin/main # fallback when the exact test is unknownmake verify BASE=origin/main # once, when the change is stablemake test-affected is non-authoritative edit-loop feedback. It collects committed branch changes,
staged changes, unstaged changes, and untracked files. It runs directly owning Go packages without
reverse dependants; Vitest tests related through the import graph; conventionally paired shell and Node
script tests; the small native-app unit suites selected by shared-client impact; and Rust unit tests only
when Rust inputs changed. It does not run lint, Rust preparation for Go, evaluation contracts, browser
suites, native builds, image checks, or other publication evidence. Prefer an exact named test when one
is known.
The runner reports wall time for each selected owner and the whole run. Go’s own package durations
are also reported without splitting the existing batched test command. Owners over 30 seconds are
flagged as advisory slow owners; tune that feedback with AFFECTED_TEST_WARN_SECONDS=<seconds>.
The threshold never changes the command’s exit status—test failures remain the only failures.
make verify classifies the diff through the same fail-closed impact policy as CI and runs the
affected local evidence. It reports locally executable gates separately from specialized and
platform-dependent gates owned by protected CI, and its completion line names only local evidence
that ran successfully. For Go changes, both golangci-lint and tests consume the affected
reverse-dependency package closure. Pull-request and merge-queue lanes provide protected remote evidence.
Run it once after the diff is stable. If it fails, use the named failing command for the repair loop and
rerun make verify once at the end; do not use it after each edit or commit.
make verify SCOPE=all is the explicit complete-repository audit. Run it when a maintainer requests a full
audit, when changing the gate/classifier machinery, or when diagnosing a selection boundary. It is
not a task-start ritual, an edit-loop command, or a requirement for every Go change.
The checks selected by each Make target are listed in the command reference, which is generated from the Makefile so it cannot drift. Enumerating them here could, and did.
make vet-tags isn’t redundant
Files behind //go:build ffmpeg|eval|integration are invisible to plain go vet ./... and to
golangci-lint — both ask the build system which files exist, and it honors the constraint.
That blind spot ran for months: go vet ./... exited 0 while the tagged run exited 1, and the
one test proving programs sequence hadn’t compiled in several releases. A tagged go build
wouldn’t catch it either, since go build skips _test.go and most tagged files are tests.
The layers
| Layer | Command | Proves | In check? |
|---|---|---|---|
| Go unit | make test | The backend, hermetically | ✅ |
| Tagged compile | make vet-tags | Build-tagged code compiles | ✅ |
| Doc claims | part of make test | Help pages don’t contradict the code | ✅ |
| Store conformance | make test-pg | SQLite and Postgres behave identically | CI |
| Frontend units | make fe | Components and domain logic | CI |
| Visual + a11y | make fe-visual | Every story, two viewports, pixel + axe | CI |
| e2e | make e2e | The embedded SPA through first-run | CI |
| ffmpeg | make test-ffmpeg | Programs sequence through real ffmpeg | CI (image builds, against the image’s own ffmpeg) |
| LLM eval | make eval | Real intents against a real model | manual |
| LLM certification | make eval-cert | Exact starter/adversarial corpus executed with a versioned scorecard | release/manual |
| LLM matrix | make eval-matrix | Same corpus on local and OpenRouter generation, judged through OpenRouter | release/manual |
| Channel recommendation | make channel-recommend-cert | Inert Channel Concepts on a digest-pinned synthetic holdout | release/manual |
| Hosted filler bakeoff | make filler-bakeoff-openrouter | Locked label-blind filler packets through one pinned paid route profile | release/manual |
| Local filler bakeoff | make filler-bakeoff-ollama | Same locked packets through one digest-pinned loopback model | release/manual |
| Rust supply chain | make rust-audit | Cargo advisories, licenses, and sources | weekly + manual |
| Rust fuzz | make rust-fuzz | Bounded worker protocol and decoder do not crash | weekly + manual |
| SSO | make test-sso | OIDC against real Authelia + Authentik | manual |
| Maintainer smoke | make smoke | The real stack end to end | manual |
Store conformance is one suite over two backends — don’t fork the assertions per dialect.
Playback certification is layered
Moved from design.md §9.1 (V58). Three layers each prove one claim, so one green test can’t
stand in for another: the controller contract, a real Loomarr process packaging real bytes, and
shipping browsers on real hardware.
Controller matrix. The same 100-Channel catalog runs through Playwright Chromium, Firefox and WebKit, as a dedicated serial macOS CI job with zero retries.
- Every engine must keep latest-request-wins, one video element, exact warmed-URL reuse, still-only adjacent warming, and a genuinely decoded H.264 frame.
- Chromium and Firefox enforce the first-frame budgets per engine, never pooled. Playwright WebKit only records them: it runs the non-shipping hls.js fallback, not Safari’s native HLS. It still keeps every correctness, OSD, manifest and raw-runner gate.
- Each engine completes one bounded cold decode first; the surf samples measure an already-running tuner.
- A raw MediaSource control on the same bytes and the same persistent element: two unmeasured replacements, then twenty whose nearest-rank p95 must be under 500 ms. It checks the runner, and never adjusts the product budgets. A fresh element per sample would hide the lifecycle it measures.
- Join time comes from media-event timestamps inside the browser, and the frame-to-
playinginterval must stay under 250 ms. Navigation starts its clock in the browser’s trusted input handler, so driver latency never counts as product time.
Route timing (#3399). A step of the same CI job, in Chromium on the same runner and SPA build
(make route-timing-e2e-host, tests/e2e/route-timing.spec.ts). It clicks through Home, Guide,
Library, Requests and Settings and opens a channel, then reads the app’s own
loomarr:route:request-to-content marks (click to real content painted).
- Every
/v1read is held for a fixed 100 ms, so results do not depend on the runner’s backend speed. A page that waits on two reads in a row costs two delays. - Cold is a page asked for the first time in the session; warm is the same page again with its data cached. Each is asserted on the median of five fresh sessions, with zero retries. Slow samples that sit at the API delay are a page awaiting a read, not runner noise: look for a loader that awaits a stale refetch.
- The budgets are a ratchet set just above what main measures. The goal is warm 100 ms and cold 300 ms on every route. A PR that makes a route faster lowers its budget; none is raised without a reason in the commit.
Real-runtime gate. The real composition root over an isolated SQLite store, the real packager and HLS origin, and real ffmpeg and ffprobe. Only true external systems (the media-server API and its library) may be doubles, serving pinned representative media, never prebuilt HLS. The browser signs in through the real API, tunes a real Channel, gets a Loomarr-made manifest and reports a decoded frame. Then a process restart repeats the tune, to prove a cold boot. Missing and corrupt inputs must reach the designed offline or retry state, never an unexplained black frame.
A secondary worktree must set server.public_url to its own backend. Otherwise ffmpeg re-opens the
primary port, writes zero bytes, and every HLS request hides the routing error behind a 45-second
readiness timeout.
Shipping-browser and hardware soak. Maintainer-run: current Chrome, Firefox and Safari against
the isolated runtime, with representative H.264, HEVC 10-bit, multichannel audio, and corrupt or
missing inputs, while GPU capacity is contended. It records boot-to-ready and
request-to-first-decoded-frame, with each channel’s format and pipeline, against the 1.5 s
arbitrary and 750 ms adjacent budgets. Only this run certifies Safari. It never touches the
maintainer’s normal database or media-server configuration, and no agent runs the make smoke*
targets.
Semantic evaluation versus certification
make eval is exploratory and exits cleanly when its real library, TMDB, or LLM configuration is
absent. make eval-cert is an assertion: missing configuration, a skipped/unexecuted case, a hard
grounding or negative-constraint failure, any judge-stage failure for a non-empty rubric, or an
unwritable scorecard makes the command fail. It always bypasses Go’s test cache and writes
$LOOMARR_ARTIFACT_DIR/semantic-certification.json unless LOOMARR_EVAL_OUT selects another path.
Scorecard schema v7 records its schema/corpus version, separate requested generator and judge
provider/model identities, trial profile, and bounded structural observations—never credentials or
ambiguous top-level provider/model compatibility fields. Hard predicates cover
exact named includes/excludes, holiday policy, rating limits, ownership mix, and concrete scheduled
program identities/order; a non-empty Proposal or favorable judge paragraph cannot substitute for
one. RequireTitles and ForbidTitles compare normalized whole titles in the case’s explicit
grounded or scheduled evidence scope. Normalization trims, folds case, and collapses internal
whitespace; it never turns a substring or near title into a match. ForbidTitleTerms is the distinct
intentional substring heuristic. Every failed trial records its first failure as retrieval, generation, deterministic,
structural_budget, schedule, judge, or budget_exhausted, and the scorecard counts failed trials
under the same labels; later failures do not rewrite that first-stage diagnosis. no_tool_call and
retrieval_empty classify as retrieval; provider/model errors, malformed generation, and empty
selection after surfaced candidates classify as generation. Judge evidence requires
explicit finite overall, relevance, and serendipity scores in 0..1 plus a non-blank reason.
Runner.Run revalidates those facts for every Judge implementation; incomplete, NaN/infinite,
out-of-range, or blank-reason output is a judge failure rather than a defaulted or clamped score. Fixture cases materialize
schedule.DesiredLineup in the hermetic gate. Real-provider cases run serially for an explicit trial
count and report pass rate plus min/median/max overall quality, relevance, and serendipity: novelty only scores when it
remains defensibly on-theme. Per-case tool-call and surfaced-candidate budgets fail deterministically.
Structural diagnostics record the grounding stage, tool mode, candidate count, generation failure,
and schedule materialization failure, so a low score points at the layer to tune. Real inference still
remains outside comprehensive verification and certifies only the requested model/provider configuration, catalog
snapshot, and corpus version in the artifact.
Every run prints and records a deterministic worst-case call budget before constructing any Library,
TMDB, generator, or judge client. For C cases and T trials, maximum generator calls are
C × T × suggest.ProductionBounds().MaxModelCalls, maximum judge calls are C × T, and the total is
their sum. LOOMARR_EVAL_MAX_CALLS_PER_RUN and LOOMARR_EVAL_MAX_CALLS_PER_SUITE are mandatory for
LOOMARR_EVAL_REQUIRED=1. The run ceiling covers
suggest.ProductionBounds().MaxModelCalls + 1; the suite ceiling covers the computed total and is at
least the run ceiling. Absent, malformed, overflowing, or undersized values fail before client work;
the removed LOOMARR_EVAL_MAX_CALLS name is neither read nor accepted. An absent required-mode LOOMARR_EVAL_TRIALS defaults to three; an explicit malformed,
zero, negative, or integer-overflowing value fails rather than falling back. Budget multiplication
and addition use checked integer arithmetic, so an unrepresentable envelope fails before clients
without reporting wrapped values. Exploratory make eval reports the budget but does not require a ceiling. Required runs
using the default or explicit ollama generator or judge additionally require
LOOMARR_EVAL_ALLOW_LOCAL=1 at the same preflight boundary. Fully hosted runs need no local acknowledgement, and no evaluation
target provisions or starts Ollama.
Required runs additionally require positive per-case-trial and whole-suite token and spend ceilings:
LOOMARR_EVAL_MAX_TOKENS_PER_RUN, LOOMARR_EVAL_MAX_SPEND_PER_RUN,
LOOMARR_EVAL_MAX_TOKENS, and LOOMARR_EVAL_MAX_SPEND. Spend is an exact nonnegative USD decimal;
tokens are provider-reported prompt plus completion totals. The shared Runner ledger checks those
ceilings before generator/judge boundaries and after each bounded call record. Exhaustion records the
closed budget_exhausted stage and prevents later calls/runs; it is not merely calculated after the
suite. Missing provider attribution stays unknown and is never converted to fabricated usage.
Scorecard schema v12 also embeds a canonical quality run snapshot when corpus, generator model, and
profile are declared. It binds the requested model/provider, unanimously reported resolved model
(or explicit missing resolution), profile as the budget profile, executable version, and accounting
availability without retaining prompts, endpoints, credentials, payloads, or provider generation ids.
Required certification therefore requires an identifier-shaped LOOMARR_EVAL_PROFILE; the name is
the snapshot’s budget profile and distinguishes runs made under different declared resource envelopes.
With a hard ceiling active, missing token or hosted-spend facts fail as budget_exhausted; local
Ollama remains explicitly non-billed rather than receiving an invented charge. A missing hosted
usage fact also latches the shared suite ledger uncertain: every later trial or case is rejected
before its generator, tools, or judge can run. When the provider call simultaneously has an earlier
retrieval or generation failure, that boundary remains the current trial’s first-failure stage; the
latched budget uncertainty still blocks all subsequent provider work.
A required hosted example therefore declares every independent bound explicitly:
LOOMARR_EVAL_REQUIRED=1 \LOOMARR_EVAL_PROFILE=hosted-bounded-v1 \LOOMARR_EVAL_MAX_CALLS_PER_RUN=25 \LOOMARR_EVAL_MAX_CALLS_PER_SUITE=1500 \LOOMARR_EVAL_MAX_TOKENS_PER_RUN=50000 \LOOMARR_EVAL_MAX_TOKENS=3000000 \LOOMARR_EVAL_MAX_SPEND_PER_RUN=0.25 \LOOMARR_EVAL_MAX_SPEND=15.00 \make eval-certAll real proposal and live-schedule cases receive nonzero MaxToolCalls and
MaxCandidatesSurfaced from that same exported production bound. The shipped action-marathon case
allows only movie media identities, the negative-constraint case pins an exact forbidden grounded
Key, and the Sunday-morning nature/travel/food case requires grounded genre diversity, so closed media
mix, exact exclusion, and diversity gates are durable corpus behavior rather than test-only fields.
Each trial’s scorecard keeps separate bounded generator and judge call evidence. Every call records
only requested/resolved provider and model, prompt/completion/reasoning/cached/cache-write/image/audio/
video token counts, exact provider-reported decimal charge and currency, attempts, and latency. Charge
status is explicit: reported requires a nonnegative plain decimal of at most 64 runes and a three-letter
uppercase currency code; missing stays empty, and invalid discards provider-controlled text
without inferring or truncating a value.
Credentials, endpoints, prompts, raw responses, provider payloads, and generation ids are excluded.
Missing wire resolution or attempt metadata stays empty/zero, never copied from the requested route
or changed to attempt one. Missing attribution is serialized as zero/empty fields rather than guessed
from configured model identities. Generator evidence is capped at the production Suggester call bound and judge evidence at
one call per trial. The scorecard separately records the uncapped observed generator-call count;
exceeding the production maximum is a structural_budget failure even though the serialized call
evidence remains bounded. Corpus version 2026-08-27.8 owns the production structural budgets plus
the durable exact-key/title exclusion, exact named-title inclusion, closed movie-only media mix, and
genre-diversity assertions.
Hermetic Runner and schedule certification tests inject the eval-only semantic recording Judge
and cross the public Judge.Score(ctx, JudgeEvidence) seam. That test double validates the bounded
typed grounded, structural, and scheduled evidence it receives, returns a deliberate score and
detail, records its call count, and lets the test assert the resulting scorecard behavior. Wrong
typed evidence must fail at this seam even if a renderer could otherwise repair, infer, or rebound
it. Certification tests must not use prompt-substring assertions or private prompt parsers.
Production modelJudge renderer and provider request/response tests are supplemental wire coverage
only. They may prove serialization, routing, response parsing, and attribution through hermetic
provider observations, but they cannot certify Runner or schedule semantics and cannot substitute
provider-visible prompt text for the typed JudgeEvidence supplied by Runner.
The durable schedule-outcome contract covers an owned curated series, a separate owned holiday
episode case, and an atomic release-ordered movie franchise. Acquisition-only holiday discoveries
are not playable evidence. Hermetic Runner tests label their fixed episode identities as synthetic
test evidence; live certification never copies those expectations. Instead set both
LOOMARR_EVAL_LIVE_SCHEDULE=1 and LOOMARR_EVAL_SCHEDULE_EVIDENCE=/path/to/snapshot.json. Snapshot
schema 1 declares a non-empty snapshot id, complete scheduling-relevant Library episode evidence for
the curated and holiday series, and owned Indiana Jones movie evidence for TMDB collection 84. Both
series objects must use the exact Key series:tmdb:456; substituting another otherwise-consistent
series snapshot fails before external adapters or providers are used. Each prepared live schedule
case has a non-empty schedule-specific judge rubric and quality floors, so its materialized concrete
program sample crosses the same live judge seam as proposal evidence. Looking up an unknown prepared
case name is an explicit error, never an assertion-free case.
Each scheduled episode in that bounded sample retains its grounded identity, title, season/episode
range, year, rating, community rating, overview, and tags so holiday/highlight intent is assessable;
an opaque identity alone is insufficient. A malformed Proposal item Key fails before Judge.Score
and never renders as an empty grounded key.
The JSON object has exactly schemaVersion, snapshotId, curated, holiday, and franchise;
unknown fields, a second JSON value, or malformed trailing bytes fail. Snapshot ids use only ASCII letters, digits, ., _, and - so
they are safe in corpus identity. Each series object has exactly key, name, libraryItemId,
episodes, requiredPrograms, and forbiddenPrograms. Every episode records libraryItemId,
title, durationMs, season, episode, and the
present scheduling signals among episodeEnd, year, officialRating, communityRating,
overview, and tags. The episode array is the complete ordered ListEpisodes result, not a chosen
subset. The required and forbidden arrays are nonempty, disjoint, refer only to present episode
identities, and together classify every episode. franchise contains movies and
requiredSequence; each movie records key, name, libraryItemId, durationMs, and
collectionId, while the pinned sequence is exactly TMDB movie Keys 85, 87, and 89 in canonical
release order and live collection 84.
Before constructing a generator or judge provider, the eval re-reads Library episodes/runtimes and
TMDB collection identity and requires an exact match. Preflight never calls the scheduler or selector;
the later Runner compares production output against the snapshot’s pinned concrete oracle. The
snapshot id is included in the scorecard corpus version. Missing, incomplete, sparse, circular, or
drifted evidence fails closed before inference.
Every declared series and movie first crosses the same ownership-binding check:
library.LookupDetail receives its exact TMDB media type/id, must report it present, and must return
the snapshot’s libraryItemId. Episode enumeration and movie runtime validation happen only after
that cross-binding succeeds, so a self-consistent unrelated Library item fails before inference.
Prepared materializers also enforce exact per-case Lineup ownership: curated and holiday accept only
their snapshot series Key, and franchise accepts only movie Keys 85, 87, and 89. An extra playable
Lineup Key is a schedule-stage failure; a missing required Key remains a deterministic failure.
Acquisition picks are not materialized and are outside this check.
The two live series cases still send canonical semantic viewer requests through the real generator:
Classic Simpsons reruns from the golden era, curated for variety for curated highlights and
Christmas episodes of The Simpsons already in my library for the owned holiday case. Snapshot
names and episode expectations are independent evidence, not substitutes for those requests.
Without the live opt-in, exploration reports one explicit schedule-corpus omission and continues
with proposal cases. LOOMARR_EVAL_REQUIRED=1 fails before provider construction unless both the
opt-in and consistent evidence are present, so make eval-cert cannot certify a proposal-only
subset. Both adapters enter the same pure schedule.ComputeDesiredAt projection.
make eval-contract always disables the live test before any adapter is constructed.
Planner reference-host evidence
make planner-reference-host is the provider-free evidence-publication step for the stock-model
bake-off tracked in #831. It does not run ollama, pull or load a model, start inference, contact a
provider, or spend. It accepts the exact planner scorecard bytes, one normalized schema-v2 capture,
and one directory containing these fourteen bounded raw captures:
huggingface-model.jsonretained during the authorized model acquisition andgguf-sha256.txtcomputed from the exact local GGUF;ollama-version.json,ollama-list.json,ollama-show-request.json, andollama-show.json;ollama-load-request.json,ollama-ps-cold-before.json,ollama-ps-warm-before.json, andollama-ps-after.json;sw-vers.txt,uname.txt,sysctl-hw-memsize.txt, andsystem-profiler.json.
The normalized capture declares planner-reference-host-v2, its run id and times, the exact
scorecard digest/size, model artifact and source identities, runtime/host facts, benchmark protocol,
selected-model residency, and a digest/size for each raw file. The model block requires an explicit
tag plus Ollama digest; pinned source repository/revision; GGUF filename/digest; quantization and the
production 8K context; and template and Modelfile identities. The runtime block requires native
arm64 macOS, exact Ollama/macOS/hardware/chip facts, and at least 64 GiB physical unified memory.
The protocol records profile, production context/output limits, temperature, an explicitly unset seed,
one cold start, one empty-prompt warm-up load, and one to ten measured warm trials. Every scorecard
case must report that same trial count. Cold-before evidence must show the selected model absent;
warm-before and after evidence must bind the exact tag/digest/context, agree with each other, and agree
with the scorecard’s measured /api/ps RAM/VRAM maximum.
The source metadata is a prerequisite, not something this workflow downloads: preserve the exact
Hugging Face model API response with blobs=true when acquisition is separately authorized, and keep
the exact GGUF used to create the Ollama model. On the October reference host, set the candidate paths
and the already-approved evaluation ceilings, then run this sequence from a clean evidence directory.
curl below talks only to loopback Ollama; make eval-planner-cert is the sole inference step and must
not be run without the separate #831 execution authority and aggregate-budget headroom.
set -euMODEL='hf.co/owner/repository:Q4_K_M'PROFILE='m5-pro-candidate-q4km'RUN_ID='2026-10-m5-pro-candidate-q4km'GGUF='/absolute/path/candidate-Q4_K_M.gguf'HF_METADATA='/absolute/path/huggingface-model-blobs.json'SCORECARD='/absolute/path/planner-certification.json'CAPTURE='/absolute/path/reference-capture.json'EVIDENCE_DIR='/absolute/path/reference-evidence'OUT='/absolute/path/planner-reference-manifest.json'TRIALS=3
: "${LOOMARR_EVAL_MAX_CALLS_PER_RUN:?required}": "${LOOMARR_EVAL_MAX_CALLS_PER_SUITE:?required}": "${LOOMARR_EVAL_MAX_TOKENS_PER_RUN:?required}": "${LOOMARR_EVAL_MAX_TOKENS:?required}": "${LOOMARR_EVAL_MAX_SPEND_PER_RUN:?required}": "${LOOMARR_EVAL_MAX_SPEND:?required}"export LOOMARR_EVAL_MAX_CALLS_PER_RUN LOOMARR_EVAL_MAX_CALLS_PER_SUITEexport LOOMARR_EVAL_MAX_TOKENS_PER_RUN LOOMARR_EVAL_MAX_TOKENSexport LOOMARR_EVAL_MAX_SPEND_PER_RUN LOOMARR_EVAL_MAX_SPENDtest ! -e "$EVIDENCE_DIR"mkdir -m 700 "$EVIDENCE_DIR"cp "$HF_METADATA" "$EVIDENCE_DIR/huggingface-model.json"shasum -a 256 "$GGUF" >"$EVIDENCE_DIR/gguf-sha256.txt"
curl --fail --silent --show-error http://127.0.0.1:11434/api/version \ >"$EVIDENCE_DIR/ollama-version.json"curl --fail --silent --show-error http://127.0.0.1:11434/api/tags \ >"$EVIDENCE_DIR/ollama-list.json"jq -cn --arg model "$MODEL" '{model:$model}' \ >"$EVIDENCE_DIR/ollama-show-request.json"curl --fail --silent --show-error -H 'Content-Type: application/json' \ --data-binary @"$EVIDENCE_DIR/ollama-show-request.json" \ http://127.0.0.1:11434/api/show >"$EVIDENCE_DIR/ollama-show.json"sw_vers >"$EVIDENCE_DIR/sw-vers.txt"uname -m >"$EVIDENCE_DIR/uname.txt"sysctl -n hw.memsize >"$EVIDENCE_DIR/sysctl-hw-memsize.txt"system_profiler -json SPHardwareDataType >"$EVIDENCE_DIR/system-profiler.json"
ollama stop "$MODEL" >/dev/null 2>&1 || truecurl --fail --silent --show-error http://127.0.0.1:11434/api/ps \ >"$EVIDENCE_DIR/ollama-ps-cold-before.json"jq -cn --arg model "$MODEL" \ '{model:$model,prompt:"",stream:false,keep_alive:"30m",options:{num_ctx:8192}}' \ >"$EVIDENCE_DIR/ollama-load-request.json"curl --fail --silent --show-error -H 'Content-Type: application/json' \ --data-binary @"$EVIDENCE_DIR/ollama-load-request.json" \ http://127.0.0.1:11434/api/generate >/dev/nullcurl --fail --silent --show-error http://127.0.0.1:11434/api/ps \ >"$EVIDENCE_DIR/ollama-ps-warm-before.json"STARTED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)"LLM_PROVIDER=ollama LLM_URL=http://127.0.0.1:11434 LLM_MODEL="$MODEL" \ LOOMARR_EVAL_ALLOW_LOCAL=1 LOOMARR_EVAL_PROFILE="$PROFILE" \ LOOMARR_EVAL_TRIALS="$TRIALS" LOOMARR_EVAL_OUT="$SCORECARD" \ make eval-planner-certCOMPLETED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)"curl --fail --silent --show-error http://127.0.0.1:11434/api/ps \ >"$EVIDENCE_DIR/ollama-ps-after.json"Build the normalized capture from those exact bytes. jq -rj is intentional: it hashes each JSON
string without adding a newline. The macOS stat -f %z form records exact byte counts.
SOURCE_REPOSITORY="$(jq -er .id "$EVIDENCE_DIR/huggingface-model.json")"SOURCE_REVISION="$(jq -er .sha "$EVIDENCE_DIR/huggingface-model.json")"GGUF_FILE="$(basename "$GGUF")"GGUF_SHA256="$(awk 'NR==1 {print $1}' "$EVIDENCE_DIR/gguf-sha256.txt")"OLLAMA_DIGEST="$(jq -er --arg model "$MODEL" \ '[.models[] | select(.name==$model and .model==$model)] | if length==1 then .[0].digest else error("model count") end' \ "$EVIDENCE_DIR/ollama-list.json")"QUANTIZATION="$(jq -er --arg model "$MODEL" \ '[.models[] | select(.name==$model and .model==$model)] | if length==1 then .[0].details.quantization_level else error("model count") end' \ "$EVIDENCE_DIR/ollama-list.json")"TEMPLATE_SHA256="$(jq -rj .template "$EVIDENCE_DIR/ollama-show.json" | shasum -a 256 | awk '{print $1}')"MODELFILE_SHA256="$(jq -rj .modelfile "$EVIDENCE_DIR/ollama-show.json" | shasum -a 256 | awk '{print $1}')"OLLAMA_VERSION="$(jq -er .version "$EVIDENCE_DIR/ollama-version.json")"MACOS_VERSION="$(awk -F: '$1=="ProductVersion" {gsub(/^[[:space:]]+/,"",$2); print $2}' "$EVIDENCE_DIR/sw-vers.txt")"MACOS_BUILD="$(awk -F: '$1=="BuildVersion" {gsub(/^[[:space:]]+/,"",$2); print $2}' "$EVIDENCE_DIR/sw-vers.txt")"ARCHITECTURE="$(tr -d '[:space:]' <"$EVIDENCE_DIR/uname.txt")"HARDWARE_MODEL="$(jq -er '.SPHardwareDataType | if length==1 then .[0].machine_model else error("hardware count") end' "$EVIDENCE_DIR/system-profiler.json")"CHIP="$(jq -er '.SPHardwareDataType | if length==1 then .[0].chip_type else error("hardware count") end' "$EVIDENCE_DIR/system-profiler.json")"MEMORY_BYTES="$(tr -d '[:space:]' <"$EVIDENCE_DIR/sysctl-hw-memsize.txt")"AFTER_RAM="$(jq -er --arg model "$MODEL" '[.models[] | select(.name==$model and .model==$model)] | if length==1 then .[0].size-.[0].size_vram else error("model count") end' "$EVIDENCE_DIR/ollama-ps-after.json")"AFTER_VRAM="$(jq -er --arg model "$MODEL" '[.models[] | select(.name==$model and .model==$model)] | if length==1 then .[0].size_vram else error("model count") end' "$EVIDENCE_DIR/ollama-ps-after.json")"SCORECARD_SHA256="$(shasum -a 256 "$SCORECARD" | awk '{print $1}')"SCORECARD_BYTES="$(stat -f %z "$SCORECARD")"EVIDENCE_JSON="$(for path in "$EVIDENCE_DIR"/*; do jq -cn --arg kind "$(basename "$path")" \ --arg sha256 "$(shasum -a 256 "$path" | awk '{print $1}')" \ --argjson bytes "$(stat -f %z "$path")" '{kind:$kind,sha256:$sha256,bytes:$bytes}'done | jq -s .)"
jq -n --arg runId "$RUN_ID" --arg startedAt "$STARTED_AT" --arg completedAt "$COMPLETED_AT" \ --arg scorecardSha256 "$SCORECARD_SHA256" --argjson scorecardBytes "$SCORECARD_BYTES" \ --arg tag "$MODEL" --arg digest "$OLLAMA_DIGEST" --arg sourceRepository "$SOURCE_REPOSITORY" \ --arg sourceRevision "$SOURCE_REVISION" --arg ggufFile "$GGUF_FILE" --arg ggufSha256 "$GGUF_SHA256" \ --arg quantization "$QUANTIZATION" --arg templateSha256 "$TEMPLATE_SHA256" \ --arg modelfileSha256 "$MODELFILE_SHA256" --arg ollamaVersion "$OLLAMA_VERSION" \ --arg macosVersion "$MACOS_VERSION" --arg macosBuild "$MACOS_BUILD" \ --arg architecture "$ARCHITECTURE" --arg hardwareModel "$HARDWARE_MODEL" --arg chip "$CHIP" \ --arg profile "$PROFILE" --argjson memory "$MEMORY_BYTES" --argjson trials "$TRIALS" \ --argjson ram "$AFTER_RAM" --argjson vram "$AFTER_VRAM" --argjson evidence "$EVIDENCE_JSON" \ '{schemaVersion:2,contract:"planner-reference-host-v2",runId:$runId,startedAt:$startedAt, completedAt:$completedAt,scorecardSha256:$scorecardSha256,scorecardBytes:$scorecardBytes, model:{tag:$tag,ollamaDigest:$digest,sourceRepository:$sourceRepository, sourceRevision:$sourceRevision,ggufFile:$ggufFile,ggufSha256:$ggufSha256, quantization:$quantization,contextLength:8192,templateSha256:$templateSha256, modelfileSha256:$modelfileSha256}, runtime:{ollamaVersion:$ollamaVersion,macosVersion:$macosVersion,macosBuild:$macosBuild, architecture:$architecture,hardwareModel:$hardwareModel,chip:$chip, physicalUnifiedMemoryBytes:$memory}, protocol:{profile:$profile,contextLength:8192,maxOutputTokens:2048,temperature:0.2, seed:null,coldStarts:1,warmupLoads:1,measuredWarmTrials:$trials}, residency:{coldBefore:{selectedModelResident:false},warmBefore:{selectedModelResident:true, model:$tag,ollamaDigest:$digest,ramBytes:$ram,vramBytes:$vram},after:{selectedModelResident:true, model:$tag,ollamaDigest:$digest,ramBytes:$ram,vramBytes:$vram}},evidence:$evidence}' >"$CAPTURE"Finally publish with fixed paths and a post-run time:
LOOMARR_PLANNER_REFERENCE_SCORECARD="$SCORECARD" \LOOMARR_PLANNER_REFERENCE_CAPTURE="$CAPTURE" \LOOMARR_PLANNER_REFERENCE_EVIDENCE_DIR="$EVIDENCE_DIR" \LOOMARR_PLANNER_REFERENCE_GENERATED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \LOOMARR_PLANNER_REFERENCE_OUT="$OUT" \make planner-reference-hostPublication is immutable: an existing output is never overwritten. The deep module re-hashes every raw input; parses the source, GGUF, Ollama, residency, and macOS evidence; rejects unknown/duplicate/trailing capture JSON; cross-checks every normalized artifact/runtime/host fact plus model/profile/trials and resident memory against the scorecard; sorts the evidence set; and emits only normalized facts plus digests and byte counts. Local paths and unrelated resident-model details remain in neither the capture nor the manifest. The result is necessary provenance, not certification or authority to train, deploy, or distribute. External compute and API work still shares the current $20 aggregate ceiling; remaining headroom is not pre-allocated to GPU work.
Channel recommendation certification
make channel-recommend-cert is the independent certification lane for the recommendation pillar.
It sends only a closed, synthetic Library/preference snapshot to the configured model and asks for
inert Channel Concepts. It never exposes an operator identity or viewing history, supplies tools, or
executes a Channel, Proposal, approval, acquisition, or spend effect. The digest-pinned
channel-recommendation-v2 holdout covers sparse, broad, repetitive, family, seasonal, era-heavy,
conflicting, and adversarial contexts; its certification split is excluded from the training
allowlist. Its case identities and normalized snapshot contents are mechanically disjoint from the
retained v1 no-ship holdout and the prompt-development corpus. The v2 contract keeps JSON mode and
binds every comparable call to exactly 1,024 maximum output tokens.
Run make channel-recommend-cert-dry-run first with the same model/profile and suite ceilings. It
emits a machine-readable contract with inferenceAuthorized=false and constructs no provider, so it
requires no API credential and spends nothing. The live target below remains a separate explicit
action.
The command is explicit, serial, and outside CI. Before it constructs a provider, set an exact model/profile plus positive whole-suite ceilings:
LLM_PROVIDER=ollama \LLM_URL=http://127.0.0.1:11434 \LLM_MODEL=qwen3.5:9b \LOOMARR_RECOMMEND_PROFILE=qwen35-local \LOOMARR_RECOMMEND_MODEL_DIGEST=6488c96fa5fa \LOOMARR_RECOMMEND_MAX_CALLS=8 \LOOMARR_RECOMMEND_MAX_TOKENS=50000 \LOOMARR_RECOMMEND_MAX_SPEND_NANOUSD=1 \make channel-recommend-certFor OpenRouter, use its exact canonical base, a concrete namespaced model, one pinned upstream
provider in LOOMARR_EVAL_GENERATOR_UPSTREAM_PROVIDER, and OPENROUTER_API_KEY (or LLM_API_KEY).
The route disables fallback and data collection and requests ZDR through the shared strict adapter.
Local certification also requires the hexadecimal Ollama artifact ID from ollama list; a tag alone
is mutable and is not certification identity. The scorecard records provider/model/profile, local
artifact digest or hosted upstream, prompt/schema/scorer versions, fixture digest, calls,
prompt/completion tokens, exact nanodollar charges, attempts, latency, hard failures, quality metrics,
floors, and the pre-registered selection margin. It excludes credentials, endpoint URLs, prompts,
generation IDs, reasoning text, and raw provider payloads. Missing hosted token or charge accounting
stops the suite; local Ollama is explicitly treated as unbilled. Crossing any ceiling records the
consumed call and stops before another case. Machine JSON and Markdown are written under
$LOOMARR_ARTIFACT_DIR unless the two LOOMARR_RECOMMEND_*_OUT variables override them.
Run at least the shared planner profile and one alternative against that identical frozen contract, then compare their scorecards without inference:
LOOMARR_RECOMMEND_SHARED_PROFILE=qwen35-shared \LOOMARR_RECOMMEND_SCORECARDS="artifacts/qwen.json artifacts/gemma.json" \make channel-recommend-compareThe comparator rejects different fixture, prompt, schema, scorer, threshold, metric, or margin
identities. A scorecard with incomplete accounting, an interrupted suite, a hard failure, or a missed
floor is ineligible even if its serialized certified field says otherwise. mean_quality is the
equal mean of the seven frozen quality metrics. The shared planner model remains selected unless a
certified alternative exceeds it by at least 0.02; if the shared model fails and an alternative
certifies, the result justifies a distinct recommendation route. This is a routing/fine-tuning gate,
not evidence that an adapter has already been trained or that certification transfers to another
pillar.
make planner-tool-diagnostic isolates the production turn immediately after one non-empty
synthetic catalog result. It requires explicit LLM_URL and LLM_MODEL; LLM_PROVIDER defaults to
ollama. An OpenRouter run also requires LLM_API_KEY and exactly one
LOOMARR_EVAL_GENERATOR_UPSTREAM_PROVIDER. The command uses the production system/user renderer,
catalog tool schema, assistant tool-call correlation, tool-result shape, sampling controls, and
post-result JSON mode. Its JSON report contains versioned prompt/tool/message-template identities,
SHA-256 digests, message roles, option flags, provider attribution, JSON validity, and whether the
model repeated a tool call. It never emits prompt text, tool-result content, model output,
credentials, or reasoning content. This is focused adapter evidence, not model certification.
make eval-matrix prevents tuning to one local model. It requires the ordinary exported LLM_*
configuration plus OPENROUTER_API_KEY, OPENROUTER_MODEL,
OPENROUTER_GENERATOR_PROVIDER, and OPENROUTER_JUDGE_PROVIDER;
OPENROUTER_JUDGE_MODEL may select a different hosted judge. Each role requires an exact immutable
namespaced model slug and exactly one concrete OpenRouter upstream provider; router/latest aliases and
comma-separated fallback orders fail preflight. It writes separate local and openrouter scorecards from the
exact same corpus. Both hosted roles use branded openrouter identity. Their actual chat requests
pin that singleton route, set allow_fallbacks=false, require_parameters=true,
data_collection=deny, and zdr=true; required certification fails before clients if either
applicable route is absent or invalid. Branded required certification also fails preflight unless
both generator and judge URLs are exactly OpenRouter’s canonical OpenAI-compatible
https://openrouter.ai/api/v1 endpoint; blank, alternate, and trailing-slash variants are rejected.
Generic/custom OpenAI-compatible evaluation remains flexible outside this branded lane. Keys are passed
only in request authorization and are never scorecard metadata. The command also requires
LOOMARR_EVAL_ALLOW_LOCAL=1: set it only after confirming the machine is idle and the configured
local runtime fits available RAM/VRAM without spilling. The two legs are sequential, and the target
does not provision or start Ollama. On a shared media server, run this manual certification during a
maintenance window; playback and transcode capacity take precedence over evaluation evidence.
The filler-admission bakeoff is a separate capture-then-replay workflow. Follow the OpenRouter filler bakeoff runbook; never place blind labels in its packet JSONL or run paid inference before the manifest, packet hashes, route profile, request ceiling, and spend ceiling are locked.
Rust dependency review
Renovate opens one weekly grouped PR for Cargo minor and patch updates across the production and
fuzz workspaces. Major updates require explicit Dependency Dashboard approval: update one direct
crate at a time, amend the §14 rationale when its capability or cost changes, and compare
Cargo.lock rather than accepting a generated diff blindly. Every Cargo update runs make rust-audit, make rust-check,
make image-cert, and the amd64/arm64 release build. An advisory ignore requires a reason, an owner,
and a removal issue in the same PR; an unannotated permanent ignore is not policy.
Loomarr-owned shipping Rust crates use #![forbid(unsafe_code)]. Transitive codecs may contain
unsafe or native code, which is why they remain behind the one-shot worker boundary. The fuzz harness
is non-shipping test infrastructure built around libFuzzer; its own source contains no unsafe block.
Auth tests need the negative cases: members get 403 on titles, approve and admin routes; sessions die on disable.
Green that proves nothing
Each of these has happened here:
- A test that passes first try is suspect. Sabotage the code and confirm it goes red.
make ci-lintneedsshellcheckonPATH— without it, actionlint skips the shell half and exits 0 locally while CI fails.- A fake that ignores its context can’t catch write-through-dead-context bugs.
- A story on an unregistered route snapshots “Not Found” as its baseline and stays green.
- A pnpm config only fails on a cold install — with
node_modulespresent, pnpm doesn’t re-evaluate build scripts.
Determinism
Pod assembly and shuffling take an explicit seed. The visual suite pins the timezone to UTC. Baselines are Linux-only, generated in the pinned Docker image; local macOS or Windows runs write differently-suffixed files that are gitignored.
Reduced motion
Both Playwright suites run with prefers-reduced-motion: reduce, set as
contextOptions.reducedMotion in web/apps/web/playwright.shared.ts. It must stay in
contextOptions: Playwright has no reducedMotion fixture, so a reducedMotion key beside
timezoneId, or in test.use, is dropped without a warning. Until #2004 both suites ran with
motion on that way. The gallery now asserts the media query on every story it shoots, and
tests/e2e/reduced-motion.spec.ts does the same in the SPA.
Reduced motion freezes the JS-driven loops (ActivityIndicator, SignalLoader, skeletons) that
the gallery’s animation: none style can’t reach. It also hides the idle TvStatic snow. A
story that exists to show motion opts out with the motion tag:
const meta = { title: "Setup/LoginShell", tags: ["route", "motion"] /* … */ };The gallery calls page.emulateMedia({ reducedMotion: "no-preference" }) for that story before
it loads, and its test carries @motion for --grep. It is a tag, not a story parameter, because
index.json lists tags, and the preference has to be set before the story mounts. The style tag
still freezes CSS animation at frame 0, so a motion story must be still in every other way: a
seeded tile, and no JS-driven loop. Prove that with repeat renders in the pinned image before you
tag a story (--grep @motion --repeat-each 3). The LoginShell, WizardShell,
Password recovery, Invitation join and TvStatic Default stories carry it.
motion-only is different: it keeps a story out of the gallery entirely, and motion.spec.ts
checks its behaviour frame to frame instead of its pixels.
Updating visual baselines
The Visual baselines workflow regenerates a pull request’s screenshot baselines, both the
visual suite’s (tests/visual/*-snapshots) and the e2e suite’s (tests/e2e/*-snapshots). It runs
on its own: every push to a pull request from a branch of this repository that touches web/
starts it, and it renders when scripts/ci-impact.sh selects the visual or e2e suite for the pull
request’s changes. You don’t dispatch it, and you don’t run make fe-visual-update or
make e2e-update locally: the visual target alone takes about 1,400 browser screenshots and
saturates the machine you’re working on.
The workflow accepts every rewritten pixel. Read the bot’s commit in the pull request’s diff before you merge, the way you would read a hand-written one: a PNG that changed when you didn’t mean it to is a regression, not a baseline. Wait for the run to finish before you arm auto-merge, or the queue can take the head before the baselines land on it.
Dispatch it by hand only to re-render without a push, or to force a refresh:
gh workflow run visual-baselines.yml --repo loomarr/loomarr -f pr=<number>The Actions tab runs the same thing: pick Visual baselines, then Run workflow. One run:
- Resolves the pull request’s head branch. It refuses a fork, a closed pull request, the default branch, and a pull request in the merge queue, whose branch can’t take a push. Take it out of the queue first. A fork’s pull request, and the bot’s own baseline commit, never start a run.
- Dispatches Visual baselines render (
visual-baselines-render.yml) at that branch and waits for it. The render builds Storybook from a fresh checkout of the commit and runsmake fe-visual-updatein the pinned Playwright image, then builds the SPA and runsmake e2e-updatein the same image. These are the paths CI’s Playwright job takes. The run fails, and pushes nothing, on an axe violation, a crashing story, or a broken e2e flow. A failed render keeps Playwright’s expected, actual, and diff images as theplaywright-render-failuresartifact. - Checks GitHub’s record of the render run: it must be the one this run dispatched, at the
resolved commit, and it must have succeeded. Then it commits only the PNGs the update rewrote or
added under a
web/apps/web/tests/visual/orweb/apps/web/tests/e2e/snapshot directory, asgithub-actions[bot], with both runs in the message. Any other change to the tree fails the run. If nothing changed, the run says so and pushes nothing. If the branch moved after it was rendered, the push fails; the run the newer push started renders it. - Approves the runs GitHub holds on the bot’s commit. The push starts the pull request’s
pull_requestruns (CI, CodeQL, and this workflow, which skips itself), but their actor isgithub-actions[bot]. The repository requires approval for a first-time contributor’s runs, and the bot counts as one, so GitHub parks each ataction_required, and a parked CI run keeps the pull request out of the merge queue. This step approves exactly those runs: that commit, this repository, started by the bot. If no CI run appears for the commit at all, it dispatches CI withscope=fullinstead, so the requiredCIcheck still reports. If the step fails, approve the parked runs in the Actions tab.
A newer push cancels the run for the older head. A second manual dispatch waits for the first.
The render is a separate workflow because it runs the branch’s own code: pnpm install scripts,
codegen, Storybook, and the SPA build. A manual dispatch of Visual baselines runs from main,
and a run’s Actions cache scope is the ref it runs on. Run there, the branch’s code could write
caches that main’s CI and release builds restore. Dispatched at the branch, the render sits in the
branch’s scope, as the pull request’s CI does, and it refuses to run at main. The render workflow
must exist on the branch, so rebase a branch cut before it landed onto main.
GitHub only offers a dispatch for a workflow that exists on main, so a manual run always comes
from main’s copy, and it refuses to run from any other ref. A run that a push starts uses the pull
request’s copy, as CI does. That gives the branch’s author nothing new: only a branch of this
repository gets there, and its author can already push to it. A fork’s pull request never reaches a
job with a write token, and GitHub gives a fork’s run a read-only token anyway; the trigger is
pull_request, never pull_request_target. The update
only rewrites a baseline whose comparison failed. A change smaller than the suite’s pixel tolerance
leaves the old PNG in place; the per-pixel threshold says which changes
that is.
The per-pixel threshold
A screenshot fails when more than 0.1% of its pixels differ (maxDiffPixelRatio: 0.001). The
per-pixel threshold decides whether one pixel “differs”. Playwright compares two pixels by their
YIQ colour distance, and a pixel counts only when the distance exceeds 35215 × threshold². The
suite sets threshold: 0.05, a limit of 88, in web/apps/web/playwright.shared.ts.
It used to be Playwright’s default, 0.2, a limit of 1408. That let whole token recolours through. When it was tightened (#1956), 121 baselines had drifted from the code without failing a comparison. Every one traced to a merged change. Two full CI runs on the same commit failed the same 121 snapshots and nothing else, so 0.05 adds no anti-aliasing noise and no flake.
| Change | YIQ distance | Caught at 0.2 | at 0.1 | at 0.05 |
|---|---|---|---|---|
amber #FFB020 → control border #666970 | 6,104 | yes | yes | yes |
static-500 #5A6170 vs static-800 #1B1E24 | 2,266 | yes | yes | yes |
amber tint #5A431B → static-700 #2A2E37 | 599 | no | yes | yes |
light focus cream #FFF3D6 → grey #D2D6DE | 592 | no | yes | yes |
disabled control border #1A1D22 → #383A3F | 434 | no | yes | yes |
static-700 #2A2E37 → static-800 #1B1E24 | 131 | no | no | yes |
15% amber wash #2F2410 → static-800 | 115 | no | no | yes |
airing #282C34 vs static-800 | 99 | no | no | yes |
adjacent darks: static-950↔900, static-900↔800 | 41 | no | no | no |
control border step #61646B → #666970 | 13 | no | no | no |
The suite still can’t see a change between adjacent dark steps, or a one-step tweak to a token’s
value. Seeing those would need a threshold near 0.03, which no measurement backs. When you make
such a change on purpose, refresh its stories with force.
Forcing a refresh under the threshold
For an intended change too subtle to fail the comparison, pass force, a Playwright --grep
regex of the test titles to refresh. A gallery test is titled <story title> — <story name>, for
example Loomarr Foundations/Selection — Light. Match that, not the snapshot file name:
gh workflow run visual-baselines.yml --repo loomarr/loomarr -f pr=<number> \ -f force='(Foundations/Selection|Components/Guide Surface) — Light'The render then runs only the matching tests, with --update-snapshots=all, and the workflow
commits every one of their PNGs that comes out different, even when it’s within the threshold. The
commit message records the regex. A regex that matches no test fails the render, so a typo pushes
nothing. Every other rail is unchanged: the merge queue, fork and default-branch refusals, the
render-run check, and the bot-commit approval. Leave force empty for the normal update. A forced
run doesn’t touch any baseline outside the regex, so dispatch a normal run too if the change also
moves other stories. Read the rewritten PNGs before you merge: under force, the workflow accepts
pixels that the comparison can’t tell apart.
Locally, LOOMARR_PLAYWRIGHT_FORCE='<regex>' make fe-visual-update runs the same
grep-scoped update in the pinned image.
Fixtures
Fixtures in internal/testkit/fixtures/ are captures with source-version comments. Write
parsers against them, not against remembered field names — and never let a fixture equate two
distinct identifiers, which has hidden shipped bugs by making a wrong lookup return the right
answer.