Stray service

I got a surprise bill from AWS. More than $300 for May and almost $100 for June. Until that, I thought that surprise bills were something that happened to others.

About a month ago, I passed the Amazon MQ workshop. I was sure that I destroyed all the services as I completed it. It turns out that it was not the case. The CloudFormation template failed to delete the Amazon MQ broker, and I overlooked that error.

Billing alerts

Naturally, I can only blame myself for overlooking the error and not setting up the budget alerts in the first place. One way or another, it’s time to set up my billing alerts – notifications that AWS can send you when you start wasting too much money.

As with everything on AWS, there’s more than one way of setting them up.

• The classic way: billing alarms in CloudWatch
• A better way: AWS Budgets. With AWS budgets, you can have more fine-grained alerts, and base your billing expectations on a subset of services, slicing them by regions, service types, etc. It also works well for simple alerts, as an enhanced version of the CloudWatch alarms that’s also easy to set up.

Eventually, it took me less than a minute to set up my AWS budget.

Refund

As soon as I discovered a stray service, I deleted it and created a support request, explaining the situation and asking if they could reverse the charge as a goodwill gesture.

The next day, a support specialist got back to me, asking me to double-check if I deleted all the resources and then reversed both the May and the June charges. 🥲

Aside from being happy about not being poor, I learned AWS support could reverse charges due to the accidental creation of resources. As in any large organization, they follow protocols and use templated responses. There is no way the support specialist made a refund by their benevolence. It’s reassuring that they have a protocol for reversing the charges.

AWS Nuke for your AWS playground

A tool called aws-nuke can destroy all AWS account resources. In a follow-up discussion about the incident with my colleagues, I got advised to create a separate AWS account for my playground and periodically run aws-nuke on this.

The authors of the tool provide more use cases in their README.

• We are testing our Terraform code with Jenkins. Sometimes a Terraform run fails during development and messes up the account. With aws-nuke we can simply clean up the failed account so it can be reused for the next build.

• Our platform developers have their own AWS Accounts where they can create their own Kubernetes clusters for testing purposes. With aws-nuke it is very easy to clean up these account at the end of the day and keep the costs low.

Lessons learned

A few lessons got learned and documented.

• Sometimes, CloudFormation can fail to remove resources.
• Make sure your billing alerts are the first you set up to create your account.
• AWS support has a protocol to reverse charges made due to accidental creation of resources.
• Consider creating a separate playground AWS account and use aws-nuke to keep resources under tight control.