Uncategorized

EBS and RDS snapshot management script for Amazon AWS

NB: When I became IT Director at work this year my team of brainy software engineers, who are much cleverer than me, forced me to retire from coding after 30 years experience (20 of them commercial). It doesn’t mean I can’t code any more, obviously, but it isn’t my day job any more and so I can’t promise that I’m fully up-to-date with the best methods and techniques any more.

ebs-snapshot-repositoryWe run a number of workloads in Amazon AWS, which many do these days, and one of the common things you have to think about it snapshots for your Elastic Block Store (EBS) volumes and Relational Database Service (RDS) instances. You can create these manually through the console, which is obviously a painful ball-ache if you want to introduce any sort of schedule whatsoever, or you can automate it, since every action on AWS is operable through their awesome API.

For a couple of years we used a commercial EC2 appliance to do this for us. It cost $1,800 per year. It wasn’t particularly pretty but it was easy enough to use and it did the job. But all it did was consume the AWS API on a schedule; it didn’t do anything particularly clever and certainly nothing that I wasn’t capable of doing myself by consuming the API directly. So I’ve replaced with a script which I’ve grown myself and now I’ve been testing it for a month or so I’m happy that it works.

I thought perhaps that others might find this useful too, especially if they want to save $1,800 per year, so here it is:

It has Composer dependencies, so satisfy them and then create an AWS key/secret pair which has access to EC2 and RDS. Enter these credentials into credentials.json. Using the standard policies AWSEC2FullAccess and AWSRDSFullAccess achieve this although you might want to create a custom policy which only allows listing of volumes and instances and the creation and deletion of snapshots, up to you and how anal you are about the scope of your AWS keys.

Then you’ll need to edit the schedule, which is in the execute() method. If people want me to I’ll refactor the schedule definition out into a JSON file, but in the meantime you’ll need to get familiar with the PHP date() function. The default schedule is as follows:

// EBS snapshots

// every day at 8pm, 7 day retention
if (date('G') == 20) $this->snapshotEBS(7); 

// every sunday at midnight, 4 week retention
if (date('D') == 'Sun' && date('G') == 0) $this->snapshotEBS(28); 

// every month, 12 month retention
if (date('j') == 1 && date('G') == 0) $this->snapshotEBS(360); 

// RDS snapshots

// every two hours, 1 day retention
if (in_array(date('G'), [1,3,5,7,9,11,13,15,17,19,21,23])) $this->snapshotRDS(1);

// every day at 8pm, 7 day retention
if (date('G') == 20) $this->snapshotRDS(7); 

// every sunday at midnight, 4 week retention
if (date('D') == 'Sun' && date('G') == 0) $this->snapshotRDS(28); 

// every month, 12 month retention
if (date('j') == 1 && date('G') == 0) $this->snapshotRDS(360); 

Run it every hour from the cron. Suggested crontab entry:

0 * * * * php /path/to/aws-snapshot-manager.php

It works properly and is reliable so long as cron doesn’t fall over and your AWS credentials don’t get rusty for some reason. However, its biggest weakness is that you won’t get told if it fails for any reason, which I guess would be the major improvement that it requires. It would need to be able to send alerts, by e-mail, SNS topic or SQS queue, in the event of a problem. So use at your own risk and periodically jump onto the AWS console to make sure your snapshots are up to date.

Don’t forget that AWS will charge you for snapshot storage space (EBS pricing, RDS pricing). Bear this in mind when defining your schedule and don’t wave your AWS bill in my face, I don’t want to see it.

PS: I realise that this is very boring to non-AWS people and I also realise that the AWS snapshot icon looks like a toilet.

Uncategorized

Home energy monitor reveals consumption horrors

I’ve moved flat recently (all planned, fancied an upgrade and the right opportunity came along), and my new flat is situated close to to the cupboard on my floor where the electricity meters are kept. This differs from previous apartment buildings in which I have lived because in those buildings the meters were all in the basement, far far away from the apartment itself. This prevented me from using an energy monitor, because for them to work you need to install a transmitter on the meter and that transmitter needs to be within a certain distance of the receiver inside the apartment.

Keen to work out why I kept getting over usage bills from my own employer, I bought an Owl Intuition-E and Micro+ bundle, which gave me the transmitter, the receiver and a network receiver, which allows me to upload usage data over the Internet to their online portal for analysis. Being within 30 metres of the meter, it works a treat.

I was a little surprised, however, with the results it gave me. I’ve found out the following:

  1. My television, surround sound and Bluray setup in the living room uses 60W on standby. 525 kWh per year (£78.75). I put a remote controlled socket on that lot straight away. I have no idea why 60W is necessary to keep four items AV equipment on standby.
  2. The TV / stereo setup in the bedroom uses 16W on standby. Another 140 kWh per year (£21.00) saved with another remote socket.
  3. I use a minimum of 241W. It never goes below that. This is during the day with no lights on (not that they bother me much, they’re all LED), with the fridge/freezer on (60W), my server and networking gear on (90W – this is lean, trust me) and my desktop computer on (60W), but with my monitors and everything else switched off. This means that there’s still 31W of background usage, 24 hours per day (271 kWh per year, £40.73).
  4. Consumption has been as high as > 9kW. This, presumably, was when I had the water heater and the cooker and the microwave and the telly on at the same time. Fortunately periods like this are always short-lived.

It’s a really good product, works very well, but I absolutely hate the web portal you have to log in to view your statistics. It’s awful. Fortunately, the network device can be configured to also sends its readings to a specified IP address on a specified UDP port, which is exactly what I’ve done and I store all my readings in a local database with a collector listening on that port. I then wrote my own software to analyse it and now I get a report like this every day. To this I have also added a web browser dashboard.

daily-electricity-report

Update 19/05/2014: I’ve now created a web dashboard (which also works well on mobile devices).

Screen Shot 2014-05-19 at 20.03.27

Code’s here if you’re interested (NodeMon isn’t really a thing, just a project framework for various bits of tinkering). Needs Phalcon.

Uncategorized

MRTG Dashboard

I’m one of those die-hards whose been using MRTG for almost as long as I’ve had a computer with a network connection. It’s an old tool for monitoring network traffic and its not pretty by modern standards but it does still do that job very well. However, its blocky output does rather leave much to be desired in this day and age of interactivity and so I’ve knocked together an MRTG Dashboard.

It’s a single PHP script which you just pop in your MRTG output directory (workdir) on your PHP-enabled web server. That’s all you need, all the required libraries are loaded from CDNs. It’s not perfect, but it is an improvement.

MRTG Dashboard screenshot

MRTG Dashboard screenshot

You will find that the timescales on the interactive graphs can be a little hit-and-miss. This is because while Highcharts demands data at consistent intervals when creating time-based graphs MRTG’s data is anything but consistently intervalled. I will try to improve this at some point in the future.

You can get MRTG Dashboard from Github.

Uncategorized

Mac, Apache, MySQL and PHP (MAMP)

Mac OS X serves as an excellent development environment, even if you are not actually developing Mac OS or iOS applications. It is the darling of many a LAMP (Linux, Apache, MySQL and PHP) developer, who enjoys a slick desktop operating system with good UNIX-like underpinnings but who don’t necessarily want to put up with all the various limitations and complications that running a Linux desktop brings, consistent improvements in this regard over recent years notwithstanding.

The only trouble with this is that if you want to develop LAMP applications and work on a Mac then traditionally you’ve needed a two-box setup; a Mac on your desk and Linux on a development server. For many this isn’t an issue, and indeed when you’ve got a team of developers, optimal, but what if you wanted a self-contained development environment that was restricted to just one box? What if you wanted that box to be your laptop so you could take it anywhere?

Solutions

“Virtual machine!”, I hear you cry. Yes, this is a possible solution, and for many works well. Good virtualisation software is free these days, but using a local VM is cumbersome. Not only does it consume a large slice of your RAM but it also puts a lot of strain on the CPU, meaning that if you are running off your battery your battery life will be decreased. It’s also cumbersome; you have to start up the VM when you need it and there can be complications with the networking, for example, if you have connected to a public wireless network it’s possible that your VM might not be extended the same resource.

There is a software package for Mac OS called MAMP (the M for Mac OS replacing the L for Linux). This is a point-and-click installer which bundles Apache, Linux and PHP for installation on Mac OS. I don’t like this solution, for a number of reasons, including:

  1. Limited functionality unless you “go pro” (at quite considerable cost). Any self-respecting developer will require multiple virtual hosts as a minimum and won’t need or want a clicky-button interface to get what they want.
  2. You are entirely at the mercy of the distributors of MAMP with regards to component software versions that are made available to you and when.

Alternative solution

There’s an alternative to this. You don’t have to fork out £39 for a package of what it otherwise freely and widely available software. With the help of my friend and colleague Ben Nimmo I present the following assembled and tested instructions for turning your Mac into a native MAMP server without using the packages download.

MySQL

  1. Download and install the latest .dmg and install both the *.pkgs within it (don’t use the TAR/GZ archives). You may wish to install the Workbench too, it’s really good these days.
  2. Find where the mysql.sock file is expected to be in /etc/php.ini (should be /var/mysql/mysql.sock)
  3. Create the folder and link the socket file to the expected location.
sudo mkdir /var/mysql
sudo ln -s /private/tmp/mysql.sock /var/mysql/mysql.sock
  1. Add MySQL to command line by editing /Users/username/.bash_profile and adding this line and then either restarting terminal or source-ing the file:
export PATH=$PATH:/usr/local/mysql/bin

PHP

PHP comes with Mac OS, so it’s not necessary to download and install it, however, there are a couple of necessary steps to configure it:

  1. Copy the default php.ini file:
sudo cp /etc/php.ini.default to /etc/php.ini
  1. Edit /etc/php.ini and uncomment this line to enable xdebug (not essential, but recommended):
zend_extension="/usr/lib/php/extensions/no-debug-non-zts-20090626/xdebug.so"

Apache

Apache too comes with Mac OS, so again, no need to download and install it. Its configuration, however, is a little more complex, but nothing scary. The described configuration will provide a special Apache “sandbox” environment for your projects. It uses the existing “Sites” directory in your Mac OS home directory.

  1. Create a subdirectory in this directory for each of your projects, ensuring that the directory name does not contain any characters that would be illegal in a URL. Within each of these subdirectories create another subdirectory called “web”; this will be become the web root of each project. The extra subdirectory is in case you wish to use a framework in your projects which may keep some of its files outside of the web server root (Symfony is a good example of this).
  2. Create a subdirectory called “logs” in your “Sites” directory; Apache will maintain two log files, access and error, for all the sandbox sites.
  3. Enable PHP5 with Apache by editing /etc/apache2/httpd.conf and uncomment the following line:
LoadModule php5_module libexec/apache2/libphp5.so
  1. Change the user and group to your username and “staff” respectively, also in /etc/apache2/httpd.conf:
User sbf
Group staff
  1. While still in /etc/apache2/httpd.conf, find the following configuration and change “Deny from all” to “Allow from all”:
<Directory />
    Options FollowSymLinks
    AllowOverride None
    Order deny,allow
    Deny from all
</Directory>
  1. Create and edit /etc/apache/users/user.conf with the following, changing “sbf” to the username:
<VirtualHost *:80>

    ServerName dev.local
    DocumentRoot /Users/sbf/Sites/

    RewriteEngine on
    RewriteLogLevel 1
    RewriteLog /var/log/apache2/rewrite.log

    # sites in the format http://[site].dev.local
    RewriteCond %{HTTP_HOST} ^[^.]+\.dev\.local
    RewriteCond %{REQUEST_URI} !^/error/.*
    RewriteCond %{REQUEST_URI} !^/icons/.*
    RewriteRule ^(.+) %{HTTP_HOST}$1 [C]
    RewriteRule ^([^.]+)\.dev\.local/(.*) /Users/sbf/Sites/$1/web/$2

    # Logging
    CustomLog /Users/sbf/Sites/logs/sandbox.access.log combined
    ErrorLog /Users/sbf/Sites/logs/sandbox.error.log

</VirtualHost>
  1. Restart Apache:
sudo apachectl restart

Then, for each of your sites, add an entry in /etc/hosts for with the format “name.dev.local” pointing to 127.0.0.1, where name corresponds to a subdirectory in your “Sites” directory. Don’t forget that the public subdirectory of each site is assumed to be “web”, so make a symlink to this if the framework you use has a different convention.

You should then be able to access each of your sites from URLs using the convention http://name.dev.local/ – where “name” again is a subdirectory within your “Sites” directory.

I’ve tested this setup procedure and It Works For Me [tm]. If, however, it doesn’t quite work for you as described, please let me know where you’re going wrong and how, if you were able, to resolve it, and I will update these instructions accordingly.

Uncategorized

Python SimpleHTTPServer

I don’t use Python on a daily basis, barely at all in fact (knowingly), but I stumbled across a handy purpose for it today which I see myself using. Type this command in a Linux shell or a Mac OS terminal and it will start a local web server on port 8000 using the current working directory as the document root and sending access requests to STDOUT:

python -m SimpleHTTPServer 8000

Potential uses include:

  1. Developing local web applications on the local filesystem which use no server-side processing but do use Javascript and AJAX calls that a web browser would not normally allow without starting it with special flags (anyone who’s seen “XMLHttpRequest cannot load $RESOURCE. Origin null is not allowed by Access-Control-Allow-Origin” in their error console will know what I’m talking about).
  2. Quickly sharing a directory with a colleague on the same network rather than uploading files to a shared resource such as a file server or e-mail system. Obviously you’d need to ensure that whatever port you choose is allowed in your local incoming firewall if you have one.

There’s even better news for PHP users. PHP 5.4, released this week, includes a built-in web server which can be started with an equivalent command. The advantages of this of course is that it’ll also process PHP scripts instead of being limited to static files.

Uncategorized

Flattening LaTeX source files for use with conversion software

I’ve been working with LaTeX quite a bit recently and I came across a very silly shortcoming that some of the programs that convert LaTeX to other formats suffer from. It’s moronic, but certain of them (pandoc, I’m looking at you), don’t follow includes (i.e. \input commands).

I have a PHP script which compiles all my LaTeX source files into various destination formats, including PDF, HTML, MediaWiki markup and text, and so in order to compensate for this ridiculous incapability I added this function which creates a “flattened” version of any LaTeX file by inserting the contents of any included files at the point they are included. It’s recursive, so if those included files also include files, they too are processed.

It’s simple but effective. Pass the path to the file to flatten. A string is returned with the entire flattened file. You may need to pay attention to relative paths in your \input commands if you includes are in sub-directories.

Uncategorized

PHP Vacancy

We’re hiring for a Software Engineer at work to join our technical team based in Birmingham city centre (Jewellery Quarter). If you’re looking for a challenging PHP role and you think you might fit the bill, please do drop me a line with your current CV. Full job specification below. Please don’t be put off by the crappy website, we’re literally just about to launch its replacement :)

Glide is a rapidly growing energy and telecommunications company specialising in looking after tenants, landlords, letting agents and property investors across the UK. They supply gas, electricity, telephony and broadband services to residential and commercial customers.

About Glide

They believe in hard work in an informal atmosphere that encourages people to express their ideas. They look for talented people who want to work with technology and can tackle problems in a smart and creative way. Their employees enjoy working in a challenging environment that brings out the best in them. Customers and customer service are at the heart of everything they do – their business revolves around their customers.

They have their own in-house development, customer support and sales teams and remain focused on keeping their costs low through automation and keeping customers informed through good communication. As technology evolves, they aim to bring new and improved services to their customers as early as possible.

About The Role

They currently have an opportunity for a full-time senior software engineer, starting immediately, as part of their in-house software team.

The role will focus on developing their bespoke systems. Their systems are written in object-orientated PHP with a SQL database. You will need to be a highly competent programmer  in PHP, SQL, Javascript, AJAX, Smarty templates, Model-View-Controller and in documenting your work. Good familiarity with Ubuntu/Debian based systems is required. Experience with invoicing routines, SVN or accountancy systems would be useful.

The role will involve working on projects to develop existing products, launch new products and services, improve internal customer service systems, update and maintain accountancy systems and invoicing scripts, debugging and diagnosing problems with their existing code base and working closely with their customer service teams. The role also includes integrating their system with multiple third party suppliers, redeveloping customer portals and revamping their website.

You will be enthusiastic about new technology, eager to learn, and will hold a degree in Computer Science or equivalent. You will be capable of managing your own time.

Application Details

The company is a place where everyone can be heard. If you like the sound of this opportunity, they’d love to hear from you. Please direct your communication and correspondence through my team or direct to me. Please view the website at www.glide.uk.com for more background information to assist you with your response.

1 2