Every Monday morning, the same Slack message lands in your team channel: “Did something run over the weekend? The bill looks high.” In three years of managing cloud infrastructure for other companies, we’ve seen this pattern dozens of times. And in roughly 80% of cases, the culprit is one of three things.
The weekend is the worst possible window for any of them. Between Friday evening and Monday morning there are around 62 hours in which nobody is looking at a dashboard, nobody is deploying, and the meter never stops. Whatever state your infrastructure was in at 6pm on Friday is the state you pay for until someone notices.
1. Auto-scaling groups that forgot to scale down
Your Friday deployment triggered a scale-up event — perfectly normal. But the scale-in policy was either misconfigured or the cooldown period was set too conservatively. You entered the weekend at 4× capacity and stayed there until your first engineer logged in Monday morning.
Scale-out and scale-in are not symmetrical, and that asymmetry is deliberate. Cloud providers bias towards keeping instances alive because a missed scale-in costs money, while a missed scale-out costs an outage. So a target-tracking policy will add capacity after a couple of breached data points, but it wants a longer run of quiet before it removes any. Add a generous cooldown, an instance warm-up period, and a minimum capacity that somebody bumped during an incident three months ago and never reverted, and you have a group that is very good at growing and very bad at shrinking.
We have also seen scale-in protection enabled on instances “temporarily” for a debugging session and left on. The group cannot terminate them, so it doesn’t, and nothing in the console shouts about it.
The fix is almost never “set a more aggressive cooldown.” It’s scheduling-based scale-in for non-production environments, and alerts that fire within 2 hours of an unexpected capacity floor breach.
Concretely: a scheduled action that sets minimum and desired capacity to zero (or one, if you need a warm box) at 7pm on Friday, and another that restores them at 7am on Monday. For production, where you can’t do that, a CloudWatch alarm on in-service instance count that sits above your expected floor for two consecutive hours, routed to whoever is on call. Two hours is the right number because it catches the problem on Friday night rather than Monday morning, but doesn’t page anyone for a normal traffic bump.
2. Orphaned development environments
Someone spun up an environment for a Friday demo. The demo happened. The environment did not get torn down. By Monday you’ve paid 60+ hours of compute, RDS, and data transfer for infrastructure serving zero traffic.
The compute is the visible part. What actually makes an orphaned environment expensive is everything attached to it: the RDS instance that bills whether or not a query ever arrives, the NAT gateway with its hourly charge, the load balancer, the EBS volumes, the Elastic IP that now costs money simply for existing. An idle environment is not cheap because it is idle. Most of what it runs is priced by the hour, not by the request.
“Every EC2 instance without a TTL tag is a liability waiting to mature.”
A TTL tag is exactly what it sounds like: a tag on every non-production resource carrying an expiry timestamp. A small scheduled function runs nightly, reads the tag, and terminates anything past its date. Anything in a non-production account with no tag at all gets the same treatment, after a warning to the owner. It feels harsh the first week. By the third week nobody creates a resource without one, because they know what happens.
The better version is to stop creating environments by hand at all. If a demo environment is spun up by a pipeline from a branch, the pipeline can also destroy it when the branch is merged or after a fixed number of days. The lifecycle lives in code rather than in someone’s memory.
3. Batch jobs with missing cost ceilings
A weekly batch job kicked off Saturday night. It hit an unexpected data volume (maybe a client imported 10× more records than usual), scaled out to handle it, and ran until Sunday afternoon. No alerts were configured on spend, just on error rate.
This is the one that gets past well-run teams, because from an operational standpoint nothing went wrong. The job scaled exactly as designed. Every record was processed. The error rate stayed at zero. The dashboards were green all weekend. The only signal that anything was unusual was the bill, and nobody was watching that.
The solution here is always the same: cost-based circuit breakers on long-running jobs, not just operational metrics. Your batch job can complete successfully and still be costing you 5× the expected amount.
A circuit breaker can be crude and still work. Cap the maximum number of workers the job can scale to. Give it a hard wall-clock limit after which it stops and pages someone, rather than running until it finishes. Check the input volume before scaling out, and refuse to start if it is more than, say, three times the usual size — a human can approve the exception on Monday. And at the account level, use the provider’s budget tooling to alert on daily spend, not just monthly, so a Saturday-night surge is visible on Sunday morning rather than at month end.
The other 20%
When it isn’t one of the three above, it is usually something smaller that compounds quietly: snapshots on a schedule that nobody prunes, log groups with no retention set, cross-region data transfer from a replication job that was meant to be a one-off, or a load test that somebody ran against a real environment at 5pm on Friday. The common thread is the same — nothing was configured to stop, and nothing was configured to tell you.
What to do this week
Audit your auto-scaling groups for missing schedule-based policies. Tag every non-production resource with a TTL. Configure budget alerts at 80% and 100% of your weekly expected spend. None of these take more than an afternoon — and each one pays for itself the first time it fires.
- Monday: list every auto-scaling group and note which ones have scheduled actions. For any non-production group without one, add the Friday-evening and Monday-morning schedule.
- Tuesday: agree the TTL tag key and format with the team, then write the nightly reaper. Run it in report-only mode for a week before it starts terminating anything.
- Wednesday: set weekly budgets with alerts at 80% and 100%, and make sure they land somewhere people actually read — a channel, not an inbox.
- Thursday: pick your longest-running batch job and give it a worker cap and a wall-clock limit.
- Friday: before you log off, look at the capacity of every environment. That is the state you are paying for until Monday.
The Monday Slack message doesn’t go away because you got better at reading the bill. It goes away because the infrastructure was told, in advance, when to stop.
