Tuesday, February 24, 2009

You're not a cloud provider if you don't provide an API

Cloud computing is all the rage these days, with everybody and their brother claiming to be a 'cloud provider'. Just because hosting companies have a farm of virtual servers that they can parcel out to their customers, it doesn't mean that they are operating 'in the cloud'. For that to be the case, they need to offer a solid API that allows their customers to manage resources such as virtual server instances, storage mounts, IP addresses, load balancer pools, firewall rules, etc.

A short discussion on 'XaaS' nomenclature is in order here: 'aaS' stands for 'as a Service', and X can take various values, for example P==Platform, S==Software, I==Infrastructure. You will see these acronyms in pretty much every industry-sponsored article about cloud computing. Pundits seem to love this kind of stuff. When I talk about cloud providers in this post, I mean providers of 'Infrastructure as a Service', things like the ones I mentioned above -- virtual servers, networking and storage resources, in short the low-level plumbing of an infrastructure.

A good example of 'Platform as a Service' is Google AppEngine, which offers both a development environment (right now Python-specific), and an API to interact with the 'Google cloud' when deploying your GAE application.

'Software as a Service' is pretty much what 'ASP' used to be in the dot com days (ASP == Application Service Provider if you don't remember your acronyms). The poster child for SaaS these days seems to be salesforce.com. I do however emphasize that one significant difference between SaaS and ASP is that SaaS providers DO offer an API for your application to interact with the resources they expose.

So...the common thread between the XaaS offerings is the existence of an API which allows you, as a systems and/or application architect, to interact with and manage the resources offered by the particular provider.

I've been using two cloud APIs here at OpenX, one from AppNexus and one from Amazon EC2. The AppNexus API allows you to reserve physical servers, start up, shut down and delete virtual instances on each server, clone a virtual instance, manage load balancer pools and SSL certificates at the LB level, etc. In short, it's a very solid and easy to use API.

The Amazon EC2 API is more fine grained than the one from AppNexus, which can be an advantage, but also makes it hard to coordinate the management of various resources. For example, to launch an EC2 instance you first need to create a keypair, potentially a security group, maybe an EBS volume and an elastic IP, and only then you can tie everything together via yet other EC2 API calls. For this reason, we're building our own tools around the Amazon API, tools which allow us to deploy an instance with all its associated resources via a single command-line script (and yes, we call this collection of tools the MCP). We're also using slack to deploy specific packages and applications to each instance we launch, but that's a topic for another post.

So what does all this mean to you as a systems or application architect? For a system administrator, I think it means that you need to shore up your programming skills so that you will be able to take advantage of these APIs and automate the deployment, testing and scaling of your infrastructure. For an application architect, it means that you need to shore up your sysadmin skills so you can understand the lower-level resources exposed by cloud APIs and use them to your full advantage. I think the future is bright for people who possess both sets of skills.

5 comments:

Brian O'Kelley said...

Grig -

Thanks for the kind words about the AppNexus API. We've been working on some even higher-level tools to deal with things like centralized management and monitoring of servers, which makes AppNexus even more attractive for larger customers that don't want to reinvent that particular wheel.

Brian

Anonymous said...

Hi Grig,
have you got a look azure. What are you think about it and its services ?

Anonymous said...

Grig,
Does AppNexus support something similar to EC2 AMIs? What about "user data"? If it can do those two things, it might be a good option for me to explore for use with our load testing service.

Patrick

Grig Gheorghiu said...

Patrick - you can launch instances in AppNexus based on what they call a 'bundle', which is similar to an AMI. It's actually easier to clone an instance in AppNexus than it is in EC2. You take a snapshot of its file system (you 'bundle' it), then you launch a new instance based on the bundle.

As far as user data, I don't know, but poke around their wiki, which is pretty well documented.

Grig

Anonymous said...

Another very intricate cloud API comes from IaaS cloud provider ReliaCloud. You can view their API documentation at http://www.reliacloud.com/howitworks/api.

Modifying EC2 security groups via AWS Lambda functions

One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...