Stray service

I got a surprise bill from AWS. More than $300 for May and almost $100 for June. Until then, I thought that surprise bills were something that happened to others.

About a month ago, I completed the Amazon MQ workshop. I was sure that I destroyed all the services when I completed it. It turns out that was not the case. The CloudFormation template failed to delete the Amazon MQ broker, and I overlooked that error.

Billing alerts

Naturally, I can only blame myself for overlooking the error and not setting up budget alerts in the first place. One way or another, it’s time to set up my billing alerts – notifications that AWS can send you when you start spending too much money.

As with everything on AWS, there’s more than one way to set them up.

• The classic way: billing alarms in CloudWatch
• A better way: AWS Budgets. With AWS Budgets, you can have more fine-grained alerts and base your billing expectations on a subset of services, slicing them by regions, service types, etc. It also works well for simple alerts, as an enhanced version of CloudWatch alarms that’s also easy to set up.

Eventually, it took me less than a minute to set up my AWS budget.

Refund

As soon as I discovered a stray service, I deleted it and created a support request, explaining the situation and asking if they could reverse the charge as a goodwill gesture.

The next day, a support specialist got back to me, asking me to double-check if I deleted all the resources and then reversed both the May and the June charges. 🥲

Aside from being happy about not being poor, I learned that AWS support could reverse charges due to the accidental creation of resources. As in any large organization, they follow protocols and use templated responses. There is no way the support specialist made a refund out of benevolence. It’s reassuring that they have a protocol for reversing charges.

AWS Nuke for your AWS playground

A tool called aws-nuke can destroy all AWS account resources. In a follow-up discussion about the incident with my colleagues, I was advised to create a separate AWS account for my playground and periodically run aws-nuke on it.

The authors of the tool provide more use cases in their README.

• We are testing our Terraform code with Jenkins. Sometimes a Terraform run fails during development and messes up the account. With aws-nuke we can simply clean up the failed account so it can be reused for the next build.

• Our platform developers have their own AWS Accounts where they can create their own Kubernetes clusters for testing purposes. With aws-nuke it is very easy to clean up these accounts at the end of the day and keep the costs low.

Lessons learned

A few lessons were learned and documented.

• Sometimes, CloudFormation can fail to remove resources.
• Make sure your billing alerts are the first thing you set up when you create your account.
• AWS support has a protocol to reverse charges made due to accidental creation of resources.
• Consider creating a separate playground AWS account and use aws-nuke to keep resources under tight control.