Skip to content

Dev watchers

For: anyone running long sessions or supervising agent lanes on a dev machine (Linux or macOS). You’ll get: how to arm the watchers, what each one reports, how to tune it, and how to retire a worktree without leaving processes behind.

Parallel lanes slow a machine down gradually. Orphaned air watchers keep polling deleted worktrees, scratch fills a RAM-backed /tmp, a PR drops out of the merge queue, or a lane waits on a question nobody sees. The watchers catch these. Each one prints a line when a condition starts and another when it clears. When nothing is wrong, they print nothing.

The watchers only observe. None of them stops a process, types into a terminal, or changes a PR. You decide what to do about each alert.

Arm them

Terminal window
make dev-watch

This runs every watcher that applies to your machine, in the foreground, until you press Ctrl-C. Each line starts with the time and the watcher’s name. Every Claude Code session in this repo prints a one-line snapshot of the machine when it starts, as a reminder. The snapshot comes from the shared .claude/settings.json hook.

Supervisors and long-running agents arm the watchers for the whole session (see AGENTS.md).

WatcherRuns whenReports
scripts/dev/watch-resources.shalwayshigh load, low memory, a full RAM-backed /tmp, processes in deleted worktrees, a CPU hog, a busy GPU, an idle sleep holding a lock
scripts/dev/watch-prs.shgh is signed inyour open PRs against main that are conflicted, failing checks, BEHIND with auto-merge on, dropped from the merge queue, or unarmed and not queued
scripts/dev/watch-lanes.shthe orca CLI is installedagent lanes crossing a token budget level, finishing a turn, waiting on a question menu, or showing API errors

Each script also runs on its own, and --once makes a single check and exits.

Resource thresholds

The defaults scale to the machine: load is measured against the core count and memory against the total. You can override every threshold with an environment variable.

ConditionDefaultVariables
1-minute load above 70% of cores for 3 minutes0.7, 3WATCH_LOAD_RATIO, WATCH_LOAD_MINUTES
Available memory below 10% of total10WATCH_MEM_MIN_PCT
/tmp above 60% full, only when it’s tmpfs60WATCH_TMP_PCT, WATCH_TMP_DIR, WATCH_TMP_ANY_FS=1
A process whose working directory is in a deleted or trashed worktree—WATCH_WORKTREE_ROOTS (colon-separated)
One process above 1.5 cores for 5 minutes1.5, 5WATCH_HOG_CORES, WATCH_HOG_MINUTES, WATCH_HOG_IGNORE
NVIDIA GPU above 90% for 5 minutes90, 5WATCH_GPU_PCT, WATCH_GPU_MINUTES
The GPU or heavy-gate lock held by a sleep for 2 pollsthe two /tmp/loomarr-*.lock filesWATCH_LOCKS, WATCH_LOCK_POLLS

The watcher checks every WATCH_INTERVAL seconds (default 60). WATCH_HOG_IGNORE is a space-separated list of process-name globs for desktop apps that are allowed to be busy. It covers common browsers and desktop shells by default. Worktree roots are found automatically: the primary checkout, its <primary>-<topic> siblings, and the directory of every other worktree, which is how Orca’s workspace directory is found.

CPU use is measured from each process’s cumulative CPU time between polls. ps -o pcpu isn’t used because on Linux it’s an average over the process’s whole life, so a long-running process that has only just started spinning wouldn’t show up.

PR and lane alerts

A PR alert fires once for each combination of PR, condition and head commit. The watcher stores this in ~/.local/state/loomarr-watch (or under $XDG_STATE_HOME), so restarting the watcher doesn’t repeat old alerts. If the same problem comes back after a new push, it alerts again. Drafts and stacked PRs belong to their owners and are skipped. A PR with auto-merge off is only reported after WATCH_PRS_AUTO_GRACE seconds (default 1200), because lanes arm auto-merge once CI is green.

The lane watcher finds agent terminals through Orca in this repo’s secondary worktrees. It honors ORCA_CLI_COMMAND, uses orca-dev in an Orca dev session, and otherwise uses orca-ide on Linux to avoid invoking the GNOME screen reader. It observes output tokens across a worktree’s newest Claude session and that session’s subagents, counting each API message once. It reports at WATCH_LANE_WARN, WATCH_LANE_CUTOFF and WATCH_LANE_LIMIT (90k, 105k and 150k by default). If a supervisor’s checkpoint spans several sessions, write its start time (epoch seconds) to $WATCH_LANES_CHECKPOINTS/<lane>. The watcher then counts every transcript created since that time.

These are advisory output thresholds, not the full checkpoint meter or an enforced token cap. Input tokens and tool results are not counted, and worktree-based discovery cannot distinguish multiple Claude sessions sharing a checkout. Codex and other agents, and Claude lanes without a transcript, report usage as UNKNOWN. For supervised edits, verify an authoritative worker-scoped meter and enforcement as required by the supervisor workflow. Silence from this watcher does not establish that a worker is within its checkpoint budget.

Retire a worktree cleanly

Removing a worktree doesn’t stop the processes that were started in it. Air, vite, storybook and headless browsers keep running against the deleted tree and keep using memory. Before and after you retire a lane, check for them:

Terminal window
make agent-reap WORKTREE=../loomarr-my-topic # list processes working in it
make agent-reap WORKTREE=../loomarr-my-topic APPLY=1 # stop them: TERM, then KILL after 5 s
make agent-reap ORPHANS=1 # anything left in deleted worktrees

make agent-reap matches processes by their working directory, subdirectories included. It stops them by PID and never matches itself or the shell that called it. make agent-gc keeps any worktree that still has such a process, and lists the processes so you can stop them first. Leave the primary checkout’s dev backend alone unless you started it.

Check the watchers on a machine

Terminal window
scripts/dev/watch-verify.sh

This takes about two minutes. It runs the unit tests (scripts/dev/watch-test.sh, also part of make agent-harness-test), then creates each condition for real and confirms the watcher reports it once and then reports it clear. The conditions are an orphan in a deleted sibling directory, a yes > /dev/null hog, a sleep holding a private lock file, and on Linux an 8 MiB tmpfs. It cleans up everything it starts. On macOS the tmpfs check is skipped because /tmp is on disk.

Portability

The scripts run on macOS’s stock bash 3.2. They keep state in small files instead of associative arrays, use cksum for hashing, and handle both GNU and BSD stat, ps and find. Linux data comes from /proc. On macOS it comes from sysctl, vm_stat and lsof. The GPU check needs nvidia-smi and is skipped on macOS.