1.40.1 — The scheduler ticks cron

2026-10-28

The per-minute cron.d entry that booted a full Django process on every node every minute is gone. Each node's jobs scheduler now ticks the @schedule functions from a daemon thread (JOBS_SCHEDULER_CRON_TICK, default on). The next deploy sweeps the old entry and restarts the scheduler after the API probe passes; cron functions then run as the application account.

Changed

• Every jobs.py scheduler process now runs the @schedule cron functions every minute from a daemon thread: once at start, then at every wall-clock minute boundary, on every node and not only the leadership-lock holder. Fleet-once claims and per_node keep the meaning they had under crond. JOBS_SCHEDULER_CRON_TICK controls it, default on.
• The framework no longer ships the per-minute cron.d/2_mojo_cron entry.
• The deploy's API activation gains two steps after the API probe passes: a sweep of stale project cron files from /etc/cron.d, then a stop of the running scheduler so the jobs cron starts the ticking replacement. A scheduler that does not stop is a logged warning, not a failed release; aws-check reports it as cron.heartbeat_missing.
• check_node grades stale cron files by the exact project path followed by /, whitespace or end of line, so a sibling path such as /opt/api2 no longer keeps a node in a FAIL no deploy can clear.
• aws-check's cron.heartbeat_missing remediation names the scheduler instead of an external per-minute shell loop.
• A Scheduler started off the main thread no longer crashes installing signal handlers; it runs without them and stops through its stop event.

Fixed

• The deploy's file snapshot matched destinations as substrings, so a recorded /etc/cron.d/5_reports_v2 hid /etc/cron.d/5_reports from being snapshotted. Whole-field match now.
• Run as root, post_deploy ignores the NGINX_ETC, SYSTEMD_ETC and CRON_ETC environment overrides and uses the fixed /etc paths.

Upgrade notes

• The next deploy's API activation removes any regular file in /etc/cron.d that names the project path and is not in the rendered aws/cron.d/ set, snapshotting it first so rollback puts it back, then restarts the scheduler. On the first adopting deploy the old crond tick covers the minutes the activation takes. A deployment that keeps a project cron file outside aws/cron.d/ loses it: move it into aws/cron.d/ first.
• Cron functions now run as the application account, the job engine's user, instead of the web user, and inherit that account's sudoers grants.
• JOBS_SCHEDULER_CRON_TICK = False with no external tick means no @schedule function runs anywhere. Set it only for a deployment that runs its own tick.
• A deploy whose API probe fails sweeps nothing and leaves the old scheduler running, so the old cron entry keeps ticking until a deploy succeeds.