Ghost builds

Last winter our build bill was about $43 a month. On Saturday it was projected to close the month at $714.42 — 16.6 times higher. Nothing on the site looked different. Nothing was broken. The money was going somewhere invisible.

We host on Vercel, which charges for the computer time it takes to build each new version of a site. That model is fair: ship more, pay more. Our bill said we were shipping constantly. We weren't. $651.81 of the month's charges was build time, and most of it bought nothing anyone ever saw.

We measured before we touched anything. Four checks ran at once on Saturday afternoon: what the bill actually charges for, every deployment from the last 26 days, how each project is wired, and how often our automated helpers push code.

The census came back with one loud answer. One project produced 4,392 builds in 26 days — 16,046 minutes of build time — and 3,469 of those builds were of temporary branches that GitHub's merge queue creates while it lines up changes for release. The queue makes a short-lived branch for every merge attempt, tests it, and throws it away. Our build platform was dutifully building a full preview of every one, on its most expensive machine size. Nobody opens those previews. Nobody can — the branch is gone minutes later. They were 76% of every build minute the team paid for.

We call them ghost builds.

Turning the ghost builds off looked like one line of configuration. It nearly was. But our release checklist — the automatic gate every change passes before it reaches the live site — used those very builds as its proof that a change compiles. Turn them off carelessly and every release stops.

The fix taught the gate a cheaper habit: accept the preview that already exists. Every change gets one preview built when a person opens it for review. The gate now uses that build as its proof instead of demanding a second, identical one on a throwaway branch. Same evidence, built once. A second set of AI reviewers attacked the change before it shipped and made it stronger in several places. Landing it also meant repairing four unrelated broken checks that had slipped onto our main branch through narrow test runs. The gate is stricter now than before we started.

In the first 21 hours after the fix merged: 51 deployments, zero ghost builds, 30 real releases — one production build per merged change, exactly — and zero failed builds. The pipeline did not slow down. It was a heavy day, clearing a backlog at more than four times our normal merge rate.

Build time fell from 608 minutes a day to 158 — a 74% drop, on a day busier than most. The same change went onto this website's own pipeline, which is how this post shipped through the very system it describes.

The measured part: at the platform's posted rates, the ghost builds we removed were costing between $423 and $497 for a month like August. We show a range because two honest calculation methods give two answers, and we would rather show both than pick the flattering one.

The forward part is a projection, and we label it as one: about $420–500 a month, roughly $5,000–6,000 a year, from one afternoon of measuring and one day of careful fixing. The formula is just the minutes the ghosts consumed, times the machine's price per minute. Our first clean billing month starts today. We will publish the real invoice against this projection either way.

Three things, and none of them are about our stack in particular.

Cloud bills scale with activity unless you make them scale with outcomes. Our robots got busier; the bill tracked their motion, not our results. Now cost follows merged work — the thing that matters — and a much busier fleet would barely move it.

Measure before touching. The bill named a symptom; only a deployment-by-deployment census named the cause. Two of our early suspects — an hourly publisher, a pile of dormant projects — turned out to cost almost nothing.

Safety gates are teachable. The wrong move was bypassing the gate that made ghost builds necessary. The right move was giving it equivalent proof from work already done. The checks are all still there. They just stopped charging us rent.