Thursday, October 14, 2010

Introducing project "Overmind"

Overmind is the brainchild of Miquel Torres. In its current version, released today, Overmind is what is sometimes called a 'controller fabric' for managing cloud instances, based on libcloud. However, Miquel's Roadmap for the project is very ambitious, and includes things like automated configuration management and monitoring for the instances launched and managed via Overmind.

A little bit of history: Miquel contacted me via email in late July because he read my blog post on "Automated deployment systems: push vs. pull" and he was interested in collaborating on a queue-based deployment/config management system. The first step in such a system is to actually deploy the instances you need configured. Hence the need for something like Overmind.

I'm sure you're asking yourself -- why do these guys wanted to roll their own system? Why not use something like OpenStack? Note in late July OpenStack had only just been announced, and to this day (mid-October 2010) they have yet to release their controller fabric code. In the mean time, we have a pretty functional version of a deployment tool in Overmind, supporting Amazon EC2 and Rackspace, with a Django Web interface, and also with a REST API interface.

I am aware there are many other choices out there in terms of managing and deploying cloud instances -- Cloudkick, RightScale, Scalarium ...and the list goes on. The problem is that none of these is Open Source. They do have great ideas though that we can steal ;-)

I am also aware of Ruby-based tools such as Marionette Collective and its close integration with Puppet (which is now even closer since it has been acquired by Puppet Labs). The problem is that it's Ruby and not Python ;-)

In short, what Overmind brings to the table today is a Python-based, Django-based, libcloud-based tool for deploying (and destroying, but be careful out there) cloud instances. For the next release, Miquel and I are planning to add some configuration management capabilities. We're looking at kokki as a very interesting Python-based alternative to chef, although we're planning on supporting chef-solo too.

If you're interested in contributing to the project, please do! Miquel is an amazingly talented, focused and relentless developer, but he can definitely use more help (my contributions have been minimal in terms of actual code; I mostly tested Miquel's code and did some design and documentation work, especially in the REST API area).

Here are some pointers to Overmind-related resources:


tobami said...


I would like to add something here. Actually, nobody but potential contributors should care about whether Overmind is written in Python. It could use Ruby on Rails and it would be the same. You use a UI, or you use a REST API.

The motivation was that no project exists that is a provider independent front-end and integrates configuration management and monitoring. We hope to get there.

What Python devops do care about is that Chef is ruby.Because they actually have to use ruby cookbooks and develop new deployments in ruby.

While it is not guaranteed, I hope to be able to design a plugin architecture with Chef-solo, Puppetd, kokki and fabric plugins so that both Ruby and Python devs can be happy and use Overmind.

Exactly the same way that we have integrated libcloud drivers as plugins, as well as allowed for anyone to include any custom provider as just another plugin.

Have a nice day!

Anonymous said...

Sorry for reporting this via a comment, but both ATOM and RSS feeds tell me this in Firefox (I replaced angle brackets with round brackets): XML Parsing Error: undefined entity
Location: jar:file:///usr/lib/firefox-3.6.11/chrome/browser.jar!/content/browser/feeds/subscribe.xhtml
Line Number 22, Column 12: (title)&feedPage.title;(/title)

Modifying EC2 security groups via AWS Lambda functions

One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...