1.8.0 — Fleet-wide jobs actually reach the fleet, and web app releases can be published

2026-08-09

Three fixes, all found publishing a real web app end to end for the first time. Two of them made the release plane unusable: no release could ever pass verification, and presigned uploads rejected any client that sent a Content-Type. The third is worse and older — work published to every runner on a channel was reaching exactly one of them, so edge convergence and certificate renewal landed on one node per publish while the rest of the fleet drifted. Breaking: an immediate broadcast now returns a list of job ids rather than one.

Breaking

• jobs.publish(..., broadcast=True) returns a list of job ids — one per live runner — where it previously returned a single id. Callers that used the returned id of a broadcast were relying on it having gone to one runner.

Fixed

• broadcast=True delivered to exactly one runner instead of all of them. Every runner pops the same channel queue, so fleet-wide work — edge convergence after a promote, dnsman's certificate-updated sweep — reached one node per publish and the rest drifted until a later sweep happened to pick them. Measured on two nodes: six broadcasts landed three and three, never six and six.
• Web app releases could never complete. HeadObject was not asked for stored checksums, so every file in every manifest verified as having none and release/complete rejected the whole release.
• Presigned S3 upload URLs were SigV2, which signs Content-Type. Any uploader whose HTTP client adds one by default — curl, Python's urllib — got 403 SignatureDoesNotMatch, an error that reads like a credentials problem.

Upgrade notes

• A fleet upgrading from 1.7.1 or earlier may have nodes holding an older certificate or an older vhost generation, because the broadcasts that should have reached them never did. One convergence sweep after upgrading settles it.
• Delayed broadcasts (delay/run_at) are still not fanned out — the roster at publish time is not the roster at fire time. They now log a warning instead of silently reaching a single runner.