EBS and RDS snapshot management script for Amazon AWS
NB: When I became IT Director at work this year my team of brainy software engineers, who are much cleverer than me, forced me to retire from coding after 30 years experience (20 of them commercial). It doesn’t mean I can’t code any more, obviously, but it isn’t my day job any more and so I can’t promise that I’m fully up-to-date with the best methods and techniques any more.
We run a number of workloads in Amazon AWS, which many do these days, and one of the common things you have to think about it snapshots for your Elastic Block Store (EBS) volumes and Relational Database Service (RDS) instances. You can create these manually through the console, which is obviously a painful ball-ache if you want to introduce any sort of schedule whatsoever, or you can automate it, since every action on AWS is operable through their awesome API.
For a couple of years we used a commercial EC2 appliance to do this for us. It cost $1,800 per year. It wasn’t particularly pretty but it was easy enough to use and it did the job. But all it did was consume the AWS API on a schedule; it didn’t do anything particularly clever and certainly nothing that I wasn’t capable of doing myself by consuming the API directly. So I’ve replaced with a script which I’ve grown myself and now I’ve been testing it for a month or so I’m happy that it works.
I thought perhaps that others might find this useful too, especially if they want to save $1,800 per year, so here it is:
It has Composer dependencies, so satisfy them and then create an AWS key/secret pair which has access to EC2 and RDS. Enter these credentials into credentials.json. Using the standard policies AWSEC2FullAccess and AWSRDSFullAccess achieve this although you might want to create a custom policy which only allows listing of volumes and instances and the creation and deletion of snapshots, up to you and how anal you are about the scope of your AWS keys.
Then you’ll need to edit the schedule, which is in the execute() method. If people want me to I’ll refactor the schedule definition out into a JSON file, but in the meantime you’ll need to get familiar with the PHP date() function. The default schedule is as follows:
// EBS snapshots
// every day at 8pm, 7 day retention
if (date('G') == 20) $this->snapshotEBS(7);
// every sunday at midnight, 4 week retention
if (date('D') == 'Sun' && date('G') == 0) $this->snapshotEBS(28);
// every month, 12 month retention
if (date('j') == 1 && date('G') == 0) $this->snapshotEBS(360);
// RDS snapshots
// every two hours, 1 day retention
if (in_array(date('G'), [1,3,5,7,9,11,13,15,17,19,21,23])) $this->snapshotRDS(1);
// every day at 8pm, 7 day retention
if (date('G') == 20) $this->snapshotRDS(7);
// every sunday at midnight, 4 week retention
if (date('D') == 'Sun' && date('G') == 0) $this->snapshotRDS(28);
// every month, 12 month retention
if (date('j') == 1 && date('G') == 0) $this->snapshotRDS(360);
Run it every hour from the cron. Suggested crontab entry:
0 * * * * php /path/to/aws-snapshot-manager.php
It works properly and is reliable so long as cron doesn’t fall over and your AWS credentials don’t get rusty for some reason. However, its biggest weakness is that you won’t get told if it fails for any reason, which I guess would be the major improvement that it requires. It would need to be able to send alerts, by e-mail, SNS topic or SQS queue, in the event of a problem. So use at your own risk and periodically jump onto the AWS console to make sure your snapshots are up to date.
Don’t forget that AWS will charge you for snapshot storage space (EBS pricing, RDS pricing). Bear this in mind when defining your schedule and don’t wave your AWS bill in my face, I don’t want to see it.
PS: I realise that this is very boring to non-AWS people and I also realise that the AWS snapshot icon looks like a toilet.