Tuesday, December 27, 2011

Monitor your estimated AWS charges with Amazon CloudWatch

At BitNami we are focused (some will say obsessed, really :) on making it easier to deploy and manage web applications in general and in the Amazon cloud in particular. In our experience, one of the main issues mainstream developers and admins have when moving to the Amazon Cloud is that it is sometimes difficult to estimate what the total cost of a deployment. Even the Simple Monthly Calculator is well, a bit scary in comparison with the more or less straightforward pricing of traditional dedicated or shared hosting. Its 'pay per usage' model, combined with reserved and spot instances, can help reduce your bill for many kinds of deployments. It also makes it highly desirable to be able to control whether it grows more than expected, either because a surge in traffic/usage or because somebody accidentally left running hundreds of machines from a traffic load test (it happens!)

Up to now, AWS did not offer any programmatic access to billing data. Although part of it could be derived from CloudWatch usage data, to get accurate data you had to manually log in directly to the AWS console. Because of these shortcomings, third-party tools for monitoring AWS spending had to ask end users for their username and password (as opposed to being able to provide IAM credentials). That gives us the creepies and it is the reason why we have preferred instead to just present our users with rough estimates... until now: in the past few days Amazon quietly rolled out support for monitoring estimated charges using Cloudwatch. We received the following the following email:

Dear AWS customer,
We are pleased to inform you that Premium Support customers can now monitor AWS estimated charges using Amazon CloudWatch. You can view your estimated charges as Amazon CloudWatch metrics and create alarms that automatically send you email notifications when these charges exceed levels that you set. This feature is currently available as a limited beta and you can share feedback on how we can make it better by submitting a case at: https://aws.amazon.com/support/createCase and selecting ‘Amazon CloudWatch’ in the case description.
To enable monitoring for your estimated charges, visit the AWS account activity page at: http://aws.amazon.com/account/ and click ‘Enable Now’. Once your account is enabled, you can view metric data and set alarms using the Amazon CloudWatch APIs or the AWS Management Console at: http://aws.amazon.com/console/cloudwatch/
You can use this feature to track and manage your AWS spending more easily. For example, you can graph estimated charges for Amazon EC2, Amazon S3, and AWS Data Transfer over the last two weeks and then set alarms that send you an email as soon as one exceeds $100, or you can set an alarm that emails you if total AWS estimated charges exceed $300.
These metrics are provided at no additional charge, and any alarms that you create are priced at standard Amazon CloudWatch rates. More information about Amazon CloudWatch pricing is available at:http://aws.amazon.com/pricing/cloudwatch/.
If you have any questions regarding this new feature, the AWS Support Team is available at: http://aws.amazon.com/support/.
Amazon Web Services
As mentioned in the email, this functionality is available only to Premium support customers, while Amazon works on further refining it. Since we have not seen a review or mention of it anywhere, we gave it a spin and documented the experience. The first step was to enable it from the AWS account activity page:
AWS Account activity

Estimated charges functionality enabled

After enabling the functionality and waiting a few minutes, it is possible to start querying estimated monthly charges, either in aggregate or separately by each service, including EC2, RDS, Route53, SNS and SQS are supported. The system is also able to track the cost of running Premium support and charges related to data transfer.

Estimated monthly charges for EC2

In order to keep an eye on the costs, it is possible to create alarms that get triggered when the projected costs exceed a certain target. For example, the following screen captures show the process to setup an email alert in case the projected bandwidth costs go over $500

Selecting data transfer metric

Naming the alarm and setting a threshold
Send an email if the alarm is triggered

Review and create the alarm

This functionality is just a taste of many new features Amazon is going to be adding in the next few months regarding billing automation and API access. As mentioned, this is particularly interesting for us, as we are building our cloud application hosting offering on top of Amazon's infrastructure. As an aside it is simply amazing the rate of innovation that keeps coming out of AWS, where other cloud vendors offering is still pretty much limited to starting/stopping VMs on demand...