Design rules are the basis for every decision we have made in the development of the beta of GOV.UK and hosting is no exception. The key design rule is not to be locked in to a single supplier and to make it trivial to swap hosting partners.
The beta of GOV.UK hasn't been built as rapidly as the Alpha that preceded it. But we've still been working at pace building out admin systems, developing an accessible frontend, putting together custom tools, and setting up the variety of servers to support it all. Through all of it we've maintained the agile, iterative approach that's been discussed here before, and that applies to writing content and designing user experiences right through to setting up the servers that power it all. When iterating so quickly we needed a hosting environment that would give us maximum flexibility so we wouldn't be unduly slowed down waiting for new servers when we needed to add another element to our infrastructure.
For short-term expediency, we chose Amazon Web Services (AWS) because for our immediate needs it offers the right balance of stability, high quality tools we could easily integrate into our scripts, a range of complimentary services and excellent technical support. Critically, however, we are not using any unique AWS components or services which we could not easily migrate to a new supplier. Once the G-Cloud framework is up and running we switch as quickly as is prudent to most suitable product in that framework.
Cloud hosting options are continuing to develop rapidly, and so are the requirements for the GOV.UK platform. As parts of our platform stabilise we may well find that we want a hybrid hosting environment, where some aspects live somewhere like AWS and others are hosted on other cloud platforms or even on traditional dedicated hardware. We're primarily focused on working out what will provide value now rather than being too preoccupied with potential future scenarios, but we have been careful to follow our own rules and not tie ourselves too tightly to any one hosting platform.
So, for example, we've chosen to use Amazon's Relational Database Service for handling our relational databases because as far as our applications are concerned it looks just like MySQL. When developing we can use MySQL and when we deploy we can hand over responsibility for scaling our databases to Amazon without making any changes to our code. That means that if we were to later decide to move that code to another hosting environment we would be able to focus on moving the data, and not have to spend time rebuilding our apps too. We'll be talking a bit more about the databases in use for the beta soon.
By contrast we were very cautious about using the AWS email service (SES) as there are some quirks in how it handles standard email protocols and so we had to add some specific code to interface with it. We decided to go ahead, but only after we were convinced that switching away again would be a straightforward matter. Thankfully as time passes more and more of the components of most cloud hosting platforms are becoming accessible using standard and well understood interfaces. Long may that progression continue.
Without access to cloud hosting with the flexibility such as is provided by AWS we'd not have been able to develop the beta with nearly as much freedom. And without that we'd have had to spend far more time focussed on the detail of hosting which would mean less time addressing user needs.
James Stewart is Tech Lead on the beta of GOV.UK. You can follow @jystewart on Twitter
16 comments
Comment by Steve posted on
Hi, "When developing we can use MySQL and when we deploy we can hand over responsibility for scaling our databases to Amazon without making any changes to our code. "
While you can provision bigger RDS instances, it isn't scalable and Amazon will not want responsibility. Also we've seen many problems with RDS so it maybe worth considering some other high availability technology (such as Continuent Tungsten).
Comment by James Stewart posted on
We're actually managing MySQL ourselves now that we've moved from AWS to Skyscape for our primary infrastructure. With our usage patterns RDS served us really well, but if we'd been using it more heavily we would have needed to do more work to explore its fit for the characteristics we cared about and consider alternatives.
Comment by The beta of GOV.UK « a work on process posted on
[...] the release I’ve been writing quite a bit on the GovUK blog: an explanation of our hosting choices (AWS/EC2), there’s a colophon to list our key tools, a high level overview of how we’re using [...]
Comment by Pulling the servers’ strings | Government Digital Service posted on
[...] already talked about why we chose Amazon Web Services for hosting the beta of GOV.UK, and how we approached some of the decisions that came with it. Cloud hosting gives us the [...]
Comment by Value for money: one year on | CatN posted on
[...] I think the attitude towards development and web hosting described in James Stewart's blog on the Government Digital Service site is greatly encouraging, and suggests that the attitude in Central Government towards SIs and relationships with Incumbent Suppliers truly is changing. James writes.. Cloud hosting options are continuing to develop rapidly, and so are the requirements for the GOV.UK platform. As parts of our platform stabilise we may well find that we want a hybrid hosting environment, where some aspects live somewhere like AWS and others are hosted on other cloud platforms or even on traditional dedicated hardware. We’re primarily focused on working out what will provide value now rather than being too preoccupied with potential future scenarios, but we have been careful to follow our own rules and not tie ourselves too tightly to any one hosting platform. http://digital.cabinetoffice.gov.uk/2012/01/24/hosting-the-beta-of-gov-uk/ [...]
Comment by Nick posted on
Doesn't hosting on cloud services where data will be stored outside of the EU go against EU data protection regulations?
Comment by James Stewart posted on
We're not storing any significant amounts of personal data (some user accounts for team members, but that's it) which definitely simplifies these considerations for the time being.But we've also been very careful to ensure that the amazon services we use are in their EU-West-1 region, so all the data is stored (and code is running) within the EU.
Comment by Alain Williams posted on
''we’ve also been very careful to ensure that the amazon services we use are in their EU-West-1 region''
Yes: but amazon is a US company & so the patriot act still gives the US government the power to just grab anything that it wants.
Comment by Alain Williams posted on
I am somewhat concerned that this is hosted on Amazon - a US company. The issue is one of data protection:
* people will be entering personal information into this web site, so:
+ where will the web servers be ? Could be you be (inadvertantly) exporting personal data to servers outside of the UK ?
+ Amazon is subject to the US Patriot Act which allows the US govt to ask for any data that is available to a US company no matter where it is held in the world.
Please assure me that before this goes live that it will be hosted in G-cloud and that that is entirely in the UK.
Comment by Introducing the beta of GOV.UK | Government Digital Service posted on
[...] We’re using open software and tools as much as possible, and developing in the open. The site is hosted in the cloud. Our processes are iterative and agile, we have daily stand-ups and our walls are covered in [...]
Comment by js posted on
Hi James, will this Amazon solution just be for the new test site when it launches or will it take be for the full weight of direct gov?
Comment by James Stewart posted on
We've no plans to move Directgov to EC2, though I've no doubt that it could handle those levels of traffic if we needed it to.
The only firm decision we've made is that EC2 is where the beta is hosted - we'll see how that goes and review our options for the future in due course.
Comment by David Pomphret posted on
Great to see portability being given the consideration it deserves. 🙂
Comment by Paul posted on
Another interesting article, I'm really enjoying reading this blog & seeing things evolve in a transparent manner.
Out of curiosity, what language/framework are you using on the beta?
Comment by James Stewart posted on
Thanks Paul. We're primarily developing with Ruby, using a mixture of Rails and Sinatra. There's also one app written in Scala as it needs to be able to handle a high level of concurrency. For that we did a comparison of Ruby and Scala implementations and were convinced that Scala was better for the job.
Almost all the code's available at https://github.com/alphagov if you want to take a closer look.
Comment by Paul posted on
Oh nice one, thanks for the link.