1.15.6 — A deploy stops waiting for a cron tick, and stops lying about the job that did it

2026-08-20

A single-node release spent most of its tail doing nothing: after the app was already serving, the deploy killed the job engine that was running it and then waited up to a full minute for a once-a-minute cron entry to start a replacement. That wait is the entire difference between the fastest and slowest releases we measured (49.7s and 95.7s). The update script now restarts the engine itself. It also stopped marking its own successful deploys as failed jobs — the engine was killed with the deploy's job row still open, so the replacement engine's reaper found an abandoned job and failed it. Every deploy now records where its seconds went, so the next slow release names its own slow step. Note the first deploy after upgrading still pays the old wait; the improvement shows on the one after it.

The two waits

A deploy ends by stopping the job engine — necessarily, because that engine is running the very job doing the deploy. Two things then went wrong.

The engine was never restarted by anything but cron. /etc/cron.d/3_mojo_jobs runs once a minute, so a node sat engine-less for anywhere between 0 and 60 seconds before finalization could run and the deployment could reach converged. That jitter accounted for the whole spread between our fastest and slowest observed release.

And the stop always burned its full grace window. Stopping waits for active jobs to drain, but the active job is the deploy, which is blocked on the script, which is blocked on the stop. The drain could never succeed, so the ten-second window ran to exhaustion every time before the process was killed.

update.sh now restarts the engine itself and cron reverts to being the crash backstop. Two safeguards worth knowing about, because both are load-bearing:

• It refuses to run the engine as root. The script runs under sudo, but the engine must run as the account the cron entry names. A root-started engine leaves root-owned pidfiles and logs, after which every subsequent start fails and the node cannot be deployed to again without a shell on the box. The owning account is now taken from the cron entry, then $SUDO_USER, then the ownership of var/pids — and if none of those yields a real, non-root user, the script starts nothing and lets cron do it.
• jobman stop now waits for the process to actually be gone. It previously sent the kill and returned immediately, and a following start would see the not-yet-reaped process and report "already running" while starting nothing.

Successful deploys are no longer recorded as failures

The engine was killed with the deploy's own job row still marked running and its lease still held, so the replacement engine's reaper found what looked like an abandoned job and marked it failed — on every self-updating node, while the deployment history alongside it said converged. The job now reaches a terminal state before the engine dies, and job history agrees with deployment history.

The job reaper also stops rewriting the recorded error on jobs that had already finished, and no longer claims a job "exceeded max retries" when it was never retried.

Phase timing

Every deploy writes a per-phase timing table that lands on the deployment record, so a slow release identifies its own slow step instead of inviting a guess. This is also the measurement needed before deciding which deploy steps are safe to skip — that work is deliberately not in this release.

Two things to expect

• The first deploy after upgrading will look unchanged. It runs the previous update.sh, which is already executing when the new one is installed, so it still waits for cron. The improvement appears on the next deploy.
• Non-deploy jobs sharing that engine get a shorter shutdown window — two seconds instead of ten. They are killed and requeued rather than finishing, as they were before, just sooner. If a node runs long jobs on the deploy engine, the new phase timings will show what that costs.

Also in this release

The admin portal now says when an action has finished, and its menus close the way every other menu does.