Skip to content

0018. Ingest is one ordered, watchable pipeline

  • Date: 2026-08-09
  • Status: accepted
  • Supersedes: one cron sweep per capability

Context

Filler grew one cron job per capability (sync, fetch, language, split, transcribe, vision, taxonomy), each sweeping the whole catalog on its own schedule. A clip’s progress was invisible for up to an hour, nothing owned the order (tagging could run before transcription and never re-run), cheap and expensive work competed with no budget, and failures retried at full cost forever.

Decision

Replace the sweeps with one ordered per-clip pipeline and one driver job (V51b). Stages run in a fixed order, one clip at a time, within per-run budgets, with backoff and a defined resolution for each failure. Stage state is persisted beside the clip cache and served by the API; SSE only speeds it up.

Consequences

  • An operator can see which stage each clip is in, and the Incoming view is one conveyor.
  • Media work never starves live channels, because the pipeline is sequential.
  • Later work (enrichment, preparation progress, recovery rewinds) extends the same driver instead of adding new sweeps.