Uncategorized

EBS and RDS snapshot management script for Amazon AWS

NB: When I became IT Director at work this year my team of brainy software engineers, who are much cleverer than me, forced me to retire from coding after 30 years experience (20 of them commercial). It doesn’t mean I can’t code any more, obviously, but it isn’t my day job any more and so I can’t promise that I’m fully up-to-date with the best methods and techniques any more.

ebs-snapshot-repositoryWe run a number of workloads in Amazon AWS, which many do these days, and one of the common things you have to think about it snapshots for your Elastic Block Store (EBS) volumes and Relational Database Service (RDS) instances. You can create these manually through the console, which is obviously a painful ball-ache if you want to introduce any sort of schedule whatsoever, or you can automate it, since every action on AWS is operable through their awesome API.

For a couple of years we used a commercial EC2 appliance to do this for us. It cost $1,800 per year. It wasn’t particularly pretty but it was easy enough to use and it did the job. But all it did was consume the AWS API on a schedule; it didn’t do anything particularly clever and certainly nothing that I wasn’t capable of doing myself by consuming the API directly. So I’ve replaced with a script which I’ve grown myself and now I’ve been testing it for a month or so I’m happy that it works.

I thought perhaps that others might find this useful too, especially if they want to save $1,800 per year, so here it is:

It has Composer dependencies, so satisfy them and then create an AWS key/secret pair which has access to EC2 and RDS. Enter these credentials into credentials.json. Using the standard policies AWSEC2FullAccess and AWSRDSFullAccess achieve this although you might want to create a custom policy which only allows listing of volumes and instances and the creation and deletion of snapshots, up to you and how anal you are about the scope of your AWS keys.

Then you’ll need to edit the schedule, which is in the execute() method. If people want me to I’ll refactor the schedule definition out into a JSON file, but in the meantime you’ll need to get familiar with the PHP date() function. The default schedule is as follows:

// EBS snapshots

// every day at 8pm, 7 day retention
if (date('G') == 20) $this->snapshotEBS(7); 

// every sunday at midnight, 4 week retention
if (date('D') == 'Sun' && date('G') == 0) $this->snapshotEBS(28); 

// every month, 12 month retention
if (date('j') == 1 && date('G') == 0) $this->snapshotEBS(360); 

// RDS snapshots

// every two hours, 1 day retention
if (in_array(date('G'), [1,3,5,7,9,11,13,15,17,19,21,23])) $this->snapshotRDS(1);

// every day at 8pm, 7 day retention
if (date('G') == 20) $this->snapshotRDS(7); 

// every sunday at midnight, 4 week retention
if (date('D') == 'Sun' && date('G') == 0) $this->snapshotRDS(28); 

// every month, 12 month retention
if (date('j') == 1 && date('G') == 0) $this->snapshotRDS(360); 

Run it every hour from the cron. Suggested crontab entry:

0 * * * * php /path/to/aws-snapshot-manager.php

It works properly and is reliable so long as cron doesn’t fall over and your AWS credentials don’t get rusty for some reason. However, its biggest weakness is that you won’t get told if it fails for any reason, which I guess would be the major improvement that it requires. It would need to be able to send alerts, by e-mail, SNS topic or SQS queue, in the event of a problem. So use at your own risk and periodically jump onto the AWS console to make sure your snapshots are up to date.

Don’t forget that AWS will charge you for snapshot storage space (EBS pricing, RDS pricing). Bear this in mind when defining your schedule and don’t wave your AWS bill in my face, I don’t want to see it.

PS: I realise that this is very boring to non-AWS people and I also realise that the AWS snapshot icon looks like a toilet.

Uncategorized

#OrlandoStrong #OrlandoUnited

Context: This blog is an amalgamation and update of two previous Facebook posts. I was in Orlando, Florida on the day of the Orlando Pulse nightclub terrorist attack, but not involved. I did not want the posts to be lost in the firehose of social media.

I’ve only ever once experienced direct homophobia

When I was at university I was walking along the seafront one evening, hand-in-hand with boyfriend-du-jour. We passed a group of three lads, who went stony silent as they passed us. They waited until we were about 30 metres away before one of them yelled “QUEER BOYS!” at us. How original.

I was furious. University was my fresh start with my sexuality and I had always been open about it since arriving, having hidden it at Sixth Form (where I experienced indirect homophobia, almost always through ignorance rather than anything else), and this cunt violated that.

I started to run towards the one who I knew yelled it, the one with the biggest grin on his face. He broke away from his two friends and I chased him into the town. I caught up with him and I threw him to the ground. He put his arms up and said he was sorry and begged me to leave him alone, having very clearly underestimated me (bear in mind I was nowhere near as physically strong then as I am now). Which I did, after telling him what I thought of him at the top of my voice and instructed him in rather frank terms never to do such a thing again.

Why did I do this? Because what he did, even though it did not cause me or boyfriend any physical harm, was NOT FUCKING ACCEPTABLE, and I was not going to let him get away with it. I hope he never did it again and always thought twice about such idiocy and ignorance after that night.

(In hindsight it was a pretty stupid idea to abandon the boyfriend with the other two, but nothing else happened, I just have to chalk that one up to seeing red.)

Since then, some 20 years on, I have never suffered any sort of homophobic abuse. There are a couple of advantageous factors which have contributed to this, the first being that I “don’t look gay” (whatever that means) and the second being that I have lived in major cities for the majority of my years since university, which are generally considered to be safer for gay people.

But I realise that I am very lucky, and I also realise that it could happen to me, someone close to me or just any other gay person, known or unknown to me at any time. It scares me, and it isn’t right, and I still DO NOT FUCKING ACCEPT IT.

The attack in Orlando on 12th June was deliberately targeted at gay people. Again, it did not affect me directly. The reality is because I’m 40 and don’t go to nightclubs any more it would have never had a chance of affecting me personally. But it did affect 103 people with whom I share community, nearly half of which are now dead. And that is not fucking acceptable.

This will happen again if we don’t start to stand up to the plague of hate that we are suffering at the hands of extremists. This has nothing to do with guns, this was a hate attack, plain and simple, and it could have happened anywhere. I am sick of reading in the news about the plight of gay people in Muslim countries, and now not even that is good enough; now gay people in western countries are being targeted for public massacre. That is not fucking acceptable, and we cannot ignore it or make excuses for it.

Nobody should be in the least bit comfortable about what happened on that day and nobody should ever forget it. Homophobia in any form and from any source is never justifiable. This direct attack on our community, solidarity, hard-earned freedoms, rights, culture and way of life is way over the line, and we need to stand up to it and say that we will not fucking accept this bullshit any longer, regardless of who we may upset in the process.

When I came out to my parents, I told them I was frightened about what might happen to me because I’m gay. That fear upset them, especially my Mum. She’s long gone now, but it turns out that her upset wasn’t without merit. I need to show her that I’m not frightened, and I won’t fucking accept this.

I’m tired, it’s been a stressful day and I’ve been through a wide range of negative emotions, from anger and rage to frustration, disbelief, disgust, shock and of course grief and sadness. Thank you to all those who reached out to me today, it’s been very touching and it meant a lot to me, especially those whose care I arguably do not deserve.

Here’s the Orlando Eye, photographed the day after the attack. Many other buildings around the world were lit up in the rainbow colours in its wake.

IMG_4379

This could have happened to any of us

A man from Illinois makes white crosses for every victim after a mass shooting in the United States. Now it’s Orlando’s turn. He’s made 49 white crosses and drove 1,200 miles to Orlando Health Medical Centre, just a couple of blocks from Pulse nightclub. Most of the victims were taken to this hospital.

I went there one week after the atrocity against gay people in Orlando to see these beautiful crosses and to pay my respects to the victims. It’s the least I could do.

People were solemn, people were silent, people were private in their respect and grief. I watched an elderly Japanese man bow to every single one of the 49 crosses. All I could hear was the water from the waterfall and the wind in the trees. There was no traffic to speak of because of the road closures still in place.

I wrote on one of the crosses, that of Alejandro Martinez, who was only just born at the time I was coming out at university. The man from Illinois will, after a while, deliver each of the crosses to their respective families.

Rest in peace, my 49 brothers and sisters.

IMG_4414b

IMG_4404

IMG_4408

IMG_4406

Why I’m bowling for #brexit

12771700_10154012287447578_5263534090863096797_o

Tl;dr: Because economic myths, Turkey and migration.

It’s now been more than two months since the Prime Minister returned from his negotiations in Brussels and announced that there would be a legally binding in/out referendum on the United Kingdom’s membership of the European Union (EU). For most of those two months I’ve been genuinely undecided as to how I will vote on Thursday 23rd June, but I have now made up my mind, and in this post I’m going to tell you why I’ve made this decision.

As the title of the post suggests, I think the United Kingdom should secede from the EU. This decision is not, as many might assume, born out of some knee-jerk right wing reaction to foreigners; those who know me well enough will realise that there’s more to my political views than that. I have made this decision after carefully considering facts and arguments from both “sides” in this debate.

Firstly let’s talk about those “sides”, specifically the “Leave” campaign and the “Remain” campaign. Both have been an utter shambles in equal measure so far, which has played no small part in determining the amount of time I have needed to come to my decision. They’re both run by hysterical loonies who spout sensationalist nonsense instead of educating people on the real facts and the issues that actually matter. I don’t care if there will be more or less potholes/booze cruises/pubs/cheap holidays/mobile phone bills abroad (delete as appropriate). These things do not matter. They don’t matter to me and they shouldn’t matter to anyone else. The following issues, however, do matter:

Economy

The economy is my primary argument. There are other arguments as you will see later on, but the economic argument is twice as important as the rest of them combined.

It’s always about the economy. Any reasonable person knows that without a strong economy nobody can have nice things like hospitals and nurses, schools and teachers, police officers and firefighters, and, yes, state handouts, if that’s your thing. Everyone, working or otherwise, should be primarily concerned with the economy.

Trade

One of the major arguments from those who wish to remain is that the EU is [one of] our biggest trading partner[s]. The words and letter in the square brackets are sometimes included, sometimes they are not; it depends who you talk to. Regardless, the message here seems to be that the UK “needs” the EU for its economy to strive and, if the UK leaves the EU, suddenly all trade between the UK and the EU will cease. This would, obviously, put quite a sizeable hole in our economy, but it’s not going to happen. Why? Because it would put an even bigger hole in the remaining EU economy.

Here’s how it stacks up: The map below shows the top 20 countries to which the UK exported goods and services during 2014, both EU and otherwise. Check my maths if you want, but I make it £131.7bn of exports to EU countries. £131.7bn of stuff we made and sold in 2014 was sold to our EU neighbours.

UK exports 2014

Now let’s look at the other map. This shows the top 20 countries from which the UK imported during the same year, again, EU and otherwise. Check my maths again, but you should get to £201.2bn of imports from EU countries.

UK imports 2014

That means we import £69.5bn more from the EU than we export to the EU. Our net worth in terms of trade to the European Union was nearly seventy billion quid in 2014. Now, would someone please tell exactly who “needs” who here, and why this sort of flow of trade would suddenly cease if we were to leave the EU? It wouldn’t happen, of course it wouldn’t happen.

It’s absolutely absurd to say that we have to be in a political union with a bloc of countries in order to trade with them. We traded with them for centuries before the EU came along and it will be perfectly possible to trade with them again independently of political union. This is scaremongering at its worst.

Our economic world will not end if the UK cedes from the EU, despite what Barrack Obama and George Osborne would like us to believe, the claims from the latter of whom have been debunked so heavily and by practically everyone that I simply can’t ever trust a word that comes out of his mouth ever again. And really, Mr. President? “Back of the queue”? We’ll remind you of those words when you want help with starting another war to prop up your economy!

Contribution

This subject, although one of the more popular subjects bandied around by the Leave campaign, pales into comparison when compared to the trade argument and it’s not one of my high priority issues, but since it is so often a flag waved I need to cover it. The chart below shows the net contributors and net beneficiaries of EU membership.

uk-eu-net-contribution

So we’d save another £4.7bn in (net) contributions if we were to leave the EU. It’s better than a stab in the face, and just one year of this saving would, by way of a topical example, allow Tata Steel’s UK business to keep making a £1m per day loss for over 12 years. If that’s not issue-du-jour when you read this then let’s fall back on that old favourite of the Labour party: £4.7bn would pay for nearly 200,000 extra fully qualified NHS nurses every year.

Single currency

The United Kingdom didn’t join the european single currency (Euro), and it turns out that was one of the best decisions we ever made. We also wouldn’t ever be obliged to do so were we to remain in the EU, so the extent of my comment on this subject will be limited. Nonetheless, I believe the single currency to be a bad idea and it has caused extremely serious problems for the EU, which of course have had (albeit relatively limited) knock-on effects in the UK. The only way to maximise our distance from this basket-case of a currency union is to also leave the political union associated with it.

Turkey

For the past twenty years the European Union has continued to expand by admitting more and more states, with more planned and proposed for the future. Some recent additions have been very successful, most notably Poland, which enjoyed an average of 4.2% economic growth in the ten years since the nation acceded to the union. The country now sits in spectacular and extreme contrast to its years under communism and is without a doubt an EU success story.

But the formula which worked in Poland, much like the single currency, is not a universal, one-size-fits all formula. To assume that it will simply work as well with all new member states is foolish and naive. It worked in Poland because the Poles are a modern, hardworking, tolerant and outward-looking people who were willing to embrace the ideals of the western-European nations which welcomed the country out from behind the Iron Curtain and into the EU.

Turkey, on the other hand, is quite different. With a population of 70 million Muslims the country could barely be more misaligned with the ideals and culture of western Europe. I believe that a political union with Turkey is completely unacceptable. Yet, that is exactly what is proposed. There is an application and a roadmap in place for Turkey to accede to the European Union.

Here’s what bothers me the most:

  1. Islam has an infamous and well-documented problem with homosexuality, an issue which is of course very close to my heart. Turkey aren’t throwing gays off buildings, sure, but Turkey has a very long way to go with regards to its legislation and culture surrounding homosexuality before I will consider entering into any sort of union with them. Homosexuality is not illegal or punishable in Turkey, as it is in most other Islamic nations, however:
    1. No laws exist that protect LGBT people from discrimination in employment, education, housing, health care, public accommodations or credit.
    2. The Turkish military openly discriminates against passive homosexuals by barring them from serving in the military.
    3. LGBT persons in Turkey may face discrimination, harassment and even violence from their relatives, neighbors, co-workers, bosses, employees, teachers, and even members of the Turkish police.
    4. Homosexuality is widely a taboo subject in Turkey and the culture of honour killings can be observed in Turkish society families murdering members who engage in sexual or moral behaviours regarded as inappropriate.
    5. Turkey does not recognise same-sex marriages, civil unions or domestic partnership benefits.
    6. In 2015 police in Istanbul dispersed a Pride parade with tear gas and water cannon, after the parade was suddenly banned by the Governor’s office, citing that it was the month of Ramadan as the reason.
  2. In March 2016 the Turkish government seized the country’s largest newspaper, removed its dditor-in-chief and forced it to start printing pro-government propaganda. Then the government tear-gassed in the streets anybody who objected to this flagrant crackdown on media freedom.  No country in the EU has ever witnessed such brutal suppression and control of the media since the Second World War.
  3. In April 2016 the Turkish government then seized all the Christian churches in one city and declared them state property, citing urban regeneration as the reason. The seizure fits into a pattern in the Middle East, where Christians are systematically displaced and persecuted.
  4. Turkey’s record on human rights remains appalling and, despite them since 2005 being central to negotiations to them one day acceding to the EU have actually become worse. Despite this the EU remains hellbent on moving forward with this process (see more on this below).

Why are we even giving half a passing thought to joining in political union with this country?

The European Union is hellbent on admitting Turkey regardless of costs or disadvantages. At the moment this is being primarily fuelled by the migrant crisis, but it was not always the case and still isn’t the only issue. Only days before the referendum Turkey will be granted visa-free access to the European Union, a key condition in the shady, underhand deal Turkey has struck with the EU regarding the management of migrants from the Middle East. To think that such demands will end with this is naive; Turkey will continue to make demands and threats up to and after accession to further its own self-interests. Turkey has the European Union over a barrel and I am not impressed with their blackmail in the slightest.

Even the pro-Remain Home Secretary Theresa May admits that admitting Turkey and four other countries to the union would be dangerous for the United Kingdom. I firmly believe that this referendum is as much about a decision to be in a political union with Turkey as it is the UK being part of the EU. I’m going to vote against such a union.

Immigration and security

It’s not been a good twelve months for the EU. With two major terrorist attacks, one of which was on its own de-facto capital city, plus an overwhelming wave of migrants and numerous reports of problems with their integration, the most notable of which being the organised mass sexual assaults of women in Cologne, immigration is a public relations disaster for the EU at the moment and it’s only getting worse. Immigration has always been a hot-potato subject with regards to the EU and the current migrant crisis could not have come at a worse time for those who advocate EU expansion and open borders.

Although I do feel strongly about this issue I am not going to go into so much detail as I have with the other issues. This is chiefly because I am mindful of the hysteria which is so often prompted when discussing this subject, with most arguments very quickly deteriorating in accusations of racism. My main points are therefore as follows:

  1. I believe that Labour’s immigration policies implemented while they were in power were largely cynical and inappropriate.
  2. I believe that “multiculturalism” is a failed project, not just in the UK but across the EU, and there are many very serious and large-scale examples of how integration has failed.
  3. I am uncomfortable with the recent rise of extremism and terrorism in European countries and I believe that both will become worse if migration continues the way it is.
  4. I completely disagree with the European Union dictating to the United Kingdom what we do and do not do with regards to migration.
  5. I am aghast at our apparent inability to tackle these issues and stand up to those who would abuse this country in the name of political correctness.
  6. I am uncomfortable at the apparent blanket tolerance being afforded to migrants who follow one particular religion known for its extreme intolerance and I believe this will have an effect on my way of life in the future.

Don’t get me wrong, I am not completely against immigration in any form. There have been some notable success stories (I refer back to my reverence for Polish people). But these examples are dwarfed by large, serious and glaring failures. Positive immigration is what the United Kingdom needs, not mass, indiscriminate migration.

Dave’s Dodgy Deal

During the week before announcing the referendum David Cameron was in Brussels furiously negotiating a deal for the United Kingdom within a “reformed EU”. The key points of this are:

  1. An “emergency brake” on in-word benefits for immigrants from a particular state if there was immigration pressure from it. This would have to be approved on a state-by-state basis by the EU council. This falls short of what Cameron asked for, which is a bar on in-work benefits and social housing for all immigrants for four years, regardless.
  2. An agreement that British taxpayers’ money can never be liable to support the eurozone, a win for Cameron in broad terms.
  3. Less regulation and stronger commitment to the free flow of capital, goods and services, key for our trading relationships with EU states. Again, in-line with what Cameron asked for.
  4. A “red card” for national parliaments, enabling them to block unwanted EU legislation. A win for Cameron, the deal says that 55% of national EU parliaments can bloke EU legislation if they object to it.
  5. An opt-out of the commitment to an “ever closer union” enshrined in the treaty to which every country has to sign up. Cameron secured this, but only for the United Kingdom. The EU super-state dream will continue to apply to other member states.

So, some wins, some with watered-down conditions. But it’s all meaningless. It doesn’t matter what Cameron negotiated. He could have negotiated industrial monopolies for the UK and a brand new BMW for every British family for all I care. Why? Because it’s not legally binding.

The Vice President of the European Parliament has stated that Brussels “clearly went too far” during its negotiations with  Cameron and that “their agreement is in no way a document of the European Union, but a text of hybrid character, which is unspecified and not legally binding”.

Cameron’s deal is worthless, and there is no “reformed EU”. It was all just a show to convince the voters that he’s done something about our unfavourable position within the EU. He hasn’t.

Attitude to democracy

The European Union has a terrible attitude towards democracy and will act unilaterally when it thinks it can get away with it. They’re even quite open about the fact; during April a top “Eurocrat” said that “Referenda are becoming a huge problem for the EU” and that “Perhaps it is time for an EU ban on referenda!“. He’s just one man, I realise, but this was really eye-opening for me. He clearly has no problem expressing his opinion, but that doesn’t mean that he’s the only person within the organisation which holds the same opinion. What will they call for a ban on next? Elections? Where does that end?

Don Juncker, who is very upset that we no longer want his family’s friendship and protection, might give us a clue to where it might end. In May he complained that Prime Ministers listen too much to their voters instead of being “full time Europeans”. More of that pesky democracy stuff getting in the way of the master plan! But he’s just one man. In addition to that other man. I wonder how many more men like these there are in the corridors of power in Brussels.

Here are a few examples of when pesky democracy stuff got in the way of the correct result and of the contempt the EU treats democratic processes (referenda specifically):

Country Date Issues ‘No’ vote Outcome
Denmark 1992 Maastricht Treaty 51.7% Made to vote again
Ireland 2001 Nice Treaty 53.9% Made to vote again
France 2005 EU Constitution 54.9% Ignored
Netherlands 2005 EU Constitution 61.5% Ignored
Ireland 2008 Lisbon Treaty 53.2% Made to vote again
Greece 2015 Euro bailout 61.3% Ignored

Which way will it go?

It’s all very well declaring support for one side or the other, but coming out in support of one side doesn’t necessarily mean that it will prevail on polling day. I stand by my decision and opinions stated in this post, of course I do, but do I believe that the UK will leave the EU? No, I do not. I believe that the Remain campaign will be successful, not by a landslide, certainly, but they will win.

The thing is about referenda is that they are, in the United Kingdom and many other countries, entirely voluntary democratic processes enacted by the government in power at the time. Whether they are declared “legally binding” or not (this one is, but not all are) there is no mandatory compulsion to call them in the first place, regardless of whether or not they formed part of a party’s general election manifesto. General elections, by-elections, local elections, yes, they are all mandatory processes; governments must call them regularly whether they like it or not and whether they think they will go in their favour or not. Not so with referenda.

Therefore, governments will only ever risk calling a referendum on an issue if they are reasonably confident that the outcome will be in their favour. This was true of the Scottish Independence referendum in 2014 (which went the government’s way) and also of the Lisbon Treaty referendum in 2007. Don’t remember that one? No, you won’t, because when it became clear to the contemporary Labour government that they would lose it they cancelled it, despite having promised it two years earlier in their general election manifesto.

Referenda give voters the illusion of power, but it is just that, an illusion. You think you’re making a difference by partaking in them, but actually the decision’s already been made, and there’s nothing you can do about it. I’m extremely sceptical of them.

Her Majesty’s Government has come out in support of the United Kingdom remaining in the European Union, and so that is what we shall do.