Observability
Formerly design.md §17 and §5’s activity and diagnostic evidence tables. Loomarr keeps three
different kinds of record, each answering one question:
| Record | Answers | Written by |
|---|---|---|
activity | What did Loomarr do? (the Dashboard’s Recent activity) | the subsystem making the change, at the domain transition |
| Diagnostic events and Process runs | What did this part of Loomarr observe? | internal/platform/diagnostics |
| Metrics | Is Loomarr operating? | the generation’s metrics.Recorder |
A fourth, the discovery-quality ledger, answers how far requested Proposals progressed.
Activity
activity holds one row per notable event ({id, at, kind, level, text, subject_id}).
- Rows are written at each domain transition, never from the event bus: the bus is in-memory and lossy, and only the subsystem making the change knows what happened.
- Best-effort: a failed insert is logged and the operation continues.
textis composed at the write point and stored, so a historical row never changes wording when a channel is renamed.levelisinfo | warn | error.ListActivity(limit)reads newest first and nothing else.activity.retentionis enforced by the housekeeping job.
Diagnostics
internal/platform/diagnostics owns retained technical evidence behind a small interface: record one
structured Diagnostic event, or begin one Process run and report its progress, output and result.
Callers never choose columns, paths, batching, redaction, retention or export formats. It has SQLite
and PostgreSQL adapters through the shared store and an in-memory test adapter, and receives a narrow
sink role rather than the whole store.Store.
- Logging fans out. Every accepted
slogrecord goes to JSON stdout (available even when the store is down) and to a bounded recorder that redacts a copy and enqueues it without blocking. Saturation drops lower-severity evidence and counts it. The durable threshold isinfo; an admin may capturedebugfor 1–15 minutes, optionally scoped, and it expires on its own. - Events are structured at ingestion: time fields in Unix milliseconds (
occurred_atfrom the producer,received_atfrom the server, so client skew stays visible), level, source, subsystem, a stableeventname, a humanmessage, the correlation ids, bounded JSON attributes and a normalisedsize_bytes. - Correlation. Every request has one
request_id(echoed inX-Request-Idand RFC 7807instance). Producers add only the ids they already know:playback_session_id,channel_id,schedule_block_id,job_id,process_run_id,actor_id,instance_id. Empty means unknown; no lookups are made to fill a log.schedule_block_idis derived by the server from Channel, start, kind and content, so client transitions, guide rows and ffmpeg runs join without exposing ids or paths. - Best-effort by contract. An event is never part of the operation it describes and can never fail a request, job, reconcile or playout process. Shutdown offers a bounded flush.
- Redacted before persistence. Credentials, authorization, cookies, sessions, secrets, signed query parameters, form or DOM content and unnecessary full paths are removed before any row or output byte is written; read-time masking is only defence in depth.
Process runs. One row per external media process: purpose, parent, instance, related channel, block and job, executable and version, a redacted command summary, times, status, exit and termination reason, first and last error, and an output reference. ffmpeg has three streams with three owners: stdout is media and never logged, structured progress is downsampled into the run, and stderr drains continuously into a bounded file (the first 256 KiB plus a rolling 768 KiB tail) with per-line timestamps. Writes never apply backpressure; dropped lines are counted and marked. Process exit and output draining have separate lifetimes, so reaping a child never loses buffered media. Active runs are never aged out.
Live pages (#3352). Diagnostics never polls. Once a batch of events is stored, or verbose
capture starts or stops, the recorder announces it: an admin-only, empty diagnostic_events frame,
at most once every 2 s with a trailing frame. Each stored run update sends diagnostic_processes,
at most once a second while output is written and at once when a run ends. Each page reads again on
its frame. Reads of the log and of the event stream are not themselves logged, so during verbose
capture a page reading again stores nothing that would announce itself. A verbose capture that runs out, and a health check whose result goes stale, change on
time alone, with nothing stored and no frame, so the page reads again at the report’s endsAt or
nextRefreshAt.
Retention. diagnostic_retained_bytes keeps one running total per evidence table, adjusted in
the same transaction as each row change, so the budget check reads two rows instead of scanning
(#1398). diagnostics.max_storage_mb bounds logical payload, not the file: indexes and pages add to
it, and SQLite’s file only shrinks on VACUUM. Time is the workhorse index; the six correlation-id
indexes are partial (WHERE col <> ''), and low-selectivity columns are not indexed.
Client diagnostics
POST /v1/diagnostics/client-events takes 1–20 observations from web, android_tv, apple_tv, ios or android, from a closed
vocabulary: client.error_boundary, client.unhandled_error, client.api_failed,
player.attached, player.detached, player.ready, player.source_replaced,
player.buffering_started, player.buffering_ended, player.seeking, player.seeked,
player.media_error, player.schedule_block_changed, player.playhead_drift. Each carries only
its allowed fields; messages, stacks, URLs, bodies and console output are never accepted. The server
derives actor_id and owns severity. Any invalid field rejects the whole batch. Accepted batches
return 202 after enqueue. The limit is 120 observations a minute per actor with a burst of 30.
Clients keep a 100-observation memory queue, send at most 20 every two seconds, drop routine events
before errors, and never block playback or keep a second log.
Vocabulary versions. Because one unknown event or field refuses the whole batch, and a paired
app is often newer than its server, the vocabulary is versioned (#4134). Each accepted batch’s
202 answer carries vocabulary, the version this server accepts (diagnostics.ClientVocabulary);
a server that sends none is version 0. Version 0 is the fourteen events above; version 1 adds only
the vocabulary answer. A client sends only what its server’s version defines: until the first
answer it sends version-0 observations only, and newer ones wait in the bounded queue, then are
dropped if the server turns out too old to take them (EVENT_SINCE and FIELD_SINCE in
@loomarr/core/client-diagnostics). Every event or field added later names the version that
introduced it here, in the same change that raises ClientVocabulary.
The batch’s platform must belong to its source: web takes the browsers, android_tv takes
android_tv, shield_tv and unknown_android_tv, and apple_tv, ios and android each take their
own name. Both native apps derive their pair from the platform they run on (createNativeApp), and
the observation’s transport names the player that played: native_hls (AVPlayer) on Apple
platforms, media3 on Android.
Client crash reports
Paired apps report their own crashes to their own server (#1816). Reports stay inside Loomarr: no third-party crash service or SDK receives them. This is the one client surface that accepts a message and a stack, so it is a separate table and contract, not a client-diagnostics event.
- Write.
POST /v1/diagnostics/client-crashesaccepts any signed-in caller; a paired device’s token is a session. The server records the user, the paired device’s id (its token hash) and its name from the credential; the body cannot name them, and unknown fields are refused. The body is akind(js-fatal,js-error,native), aplatform(ios,android,tvos,android-tv), app version and build, OS version, device model,occurredAt, amessage(≤2 KiB), an optionalstack(≤32 KiB), up to 50breadcrumbs(navigation,http,lifecycle,log; ≤256 bytes each), and fornativeonly the MetricKitMXDiagnosticPayloadJSON object (≤64 KiB). It returns202 {id}. A body over 128 KiB is413; a contract violation, bad enums included, is400; malformed JSON, a wrong JSON type or an unknown field is the framework’s422. Each device may send a burst of 10, then 2 a minute, and one person’s devices together a burst of 30, then 30 an hour, so no one person can fill the retained cap (429). A browser session is charged as its person and an API token as one caller. - Scrubbed on the server too. Clients never send a token, cookie or
Authorizationheader, and URLs keep their path only. The server enforces the same independently on every text field, labels included: credential header lines, authorization values with their scheme, quoted or baretoken/session/cookie/sig/key/password values, URL user-info, and the query and fragment of any URL, path or bare relative URI (seg00012.ts?sig=…, percent-encoded or not) are removed before storage, as are NUL bytes. Signed HLS play URLs and their segments carry the token in the query. The native payload is scrubbed value by value and key by key, so it stays valid JSON. - Read.
GET /v1/diagnostics/client-crashes(newest first,limit≤100 with an opaquecursor, without stacks, breadcrumbs or native payloads) andGET /v1/diagnostics/client-crashes/{id}(the whole report) are admin only. Each accepted report also writes oneclient.crash_reportedwarning to the Diagnostics timeline, carrying its id. - Retention is fixed product policy, applied by the daily housekeeping pass: 30 days or the newest 1000 reports, whichever is fewer. The diagnostics byte budget does not count these rows; the row cap bounds them instead.
- Capture on the phone. React Native’s global error handler writes each JS error to an on-disk
queue synchronously, before a fatal one ends the app, then hands it on. iOS native crashes come
from Apple’s MetricKit on the next launch, one report per crash, carrying the crashed build’s
identity. The queue keeps the newest 20 (
@loomarr/core/crash-reports). Once paired, the app sends pending reports at launch, on returning to the foreground, after a non-fatal error and when MetricKit delivers. A report is deleted only after a2xx, or after a400/413/422that would never succeed;401,429,5xxor no network keeps it for the next attempt. Breadcrumbs are the last 50 navigation, request (method, path, status) and lifecycle events. Android captures JS errors only; its native analogue,ApplicationExitInfo, is not built.
Current Health
Each generation owns one Current Health value. Checks have a key, label, required flag and
startup | continuous mode; observations are pending | passed | warning | failed | skipped | stale
with bounded detail, remediation and a freshness deadline. Unconfigured optional integrations are
skipped. A continuous pass goes stale after its deadline.
- The overall state is
starting,healthy,degraded(only optional checks need attention) orunhealthy(a required check failed or went stale). A previously passing required check needs two consecutive periodic failures to fail; one success recovers it. /readyzis the required-check projection of the same value;/healthzis liveness only.- Startup is the first pass. Configured integrations reuse the setup probes concurrently in one
bounded window; the
server.public_urlcheck runs only after the listener exists. Thesystem-healthjob refreshes continuous checks (default every 30 s), and settings changes prompt a run. No request or playout path waits on a probe. system-healthruns on a job worker of its own (Job.Watchdog) and never waits behind another job. An ordinary job may hold the shared worker for River’s full minute, longer than the 65 s a continuous check stays fresh. On 2026-11-18, maintenance waiting on a busy Emby did exactly that twice: readiness went stale and the reverse proxy took Loomarr off the network.- The value survives a database failure. Only transitions into or out of an incident are checkpointed as events. Each generation freezes one Startup report; the last 20 are kept.
Settings → System → Diagnostics has three views: Logs (the default), Current Health and Media processes. Logs refreshes automatically, newest first by default, with an explicit oldest first that starts at the earliest retained event. Severity filters are All, Errors, Warnings and Info; exact correlation filters sit behind a disclosure. Pages are 50 records, virtualised.
Reading and exporting
The read surfaces are admin-only, typed and cursor-paged, for the UI and Bearer-token agents alike.
The event read defaults to the last hour, newest first, 100 records (maximum 200); the cursor is the
(occurred_at, id) position bound to its order. Filters are capped at 128 bytes and the text search
at 256. JSON and NDJSON are projections of the same page, and one Process run downloads as text.
A troubleshooting report is a redacted ZIP assembled on demand from one selection: a window of at
most 24 hours; events, process metadata and process output; the same correlation filters. Caps: 2,000
events, 50 runs, eight outputs, 16 MiB, 30 s; hitting one is a declared truncation. Preview and
download share the selection. Entry names are fixed (manifest.json, system.json, events.ndjson,
processes/…) with fixed timestamps; caller identifiers never become paths. manifest.json declares
windows, counts, drops, truncation and redaction; system.json contains no host names, addresses,
targets, paths or user data. Assembly redacts again. Nothing is ever sent anywhere automatically.
Metrics
Prometheus metrics are an operator contract, not a mirror of domain events. One generation-scoped
internal/platform/metrics.Recorder owns a private registry, the collectors, HTTP middleware, store collection
and every label classifier; callers use semantic callbacks and never pass label values.
/v1/metrics(alias/metrics) is unauthenticated on the trusted-LAN listener; deployments keep it on a private scrape network.- No identifying labels: no usernames, emails, titles, channel or media ids, request ids, URLs,
paths, prompts, raw errors or secrets.
routeis the matched template.jobis the registry’s code-defined name; every other label is a closed enum, and unexpected values collapse toother. - Families cover build info, inbound HTTP (including outbound fan-out per route), outbound dependencies and retries, the database pool, retained-object gauges, scheduler jobs, channel reconciles, playout sessions and failures, auth, LLM tokens, filler pods and rotation, the image worker, and scrape errors. The exposition endpoint is the list.
- Name, type, unit, label names and value domains are compatibility commitments. A rename dual-emits for one release line before removal.
- One provisioned Grafana Overview dashboard (stable UID, read-only) and example rules ship as
source; Prometheus and Grafana are never default services.
make observability-verifychecks the manifest, hostile-label exclusion, dashboard queries and rules.make observability-devruns a local stack againstmake dev-be.
Discovery quality
A separate local ledger, never Prometheus labels, records how far requested Proposals progressed.
Stages are retrieval, generation, grounding, approval, acquisition and scheduling, each
with a closed outcome set.
- Only the authoritative transition writes, after it commits: the Proposal Job for the first three,
the approval transaction for
approved/declined, the provisioning transition forplayable/failed(timed fromrequested_at), and the first successful live reconcile forscheduled. A stage never reached is absent. Writes are best-effort and idempotent by an opaque key. - A declined Proposal is not dislike. Only explicit
keep,less,neverandsurpriseactions are taste signals; denial, detach, stopping playback, inactivity and non-selection are never recorded as preference. - Receipts hold stage, outcome, time and bounded quantities only: no titles, intents, errors, URLs, credentials or ids, hashed or not. Housekeeping folds them into daily aggregates and purges them after 30 days; aggregates last 24 months, with at most 256 unreferenced evaluation-run snapshots.
- One admin-only JSON export returns aggregates and referenced snapshots, never receipts or keys. Nothing is uploaded. SQLite and PostgreSQL share one conformance suite.
Tests that pin this
Formerly design.md §19.
- Metrics scrape-token negatives: with a token configured, no credential, a wrong token, a
prefix or superset of it, the admin
API_TOKEN, a member token or a session cookie each get401on/v1/metricsand/metrics; the correct bearer gets the exposition. With no token configured every request gets403namingLOOMARR_METRICS_TOKENand no series. The token never appears in logs, config errors or refusals, and the composition-root test proves the same throughapp.Build.