Monday, March 14, 2011

Deployment and hosting open space at PyCon

One of the most interesting events for me this year at PyCon was an Open Space session organized by Nate Aune on deployment, hosting and configuration management. The session was very well attended, and it included representatives of a large range of companies. Here are some of them, if memory serves well: Disqus, NASA, Opscode, DjangoZoom, Eucalyptus,, Gondor, Whiskey Media ... and many more that I wish I could remember (if you were there and want to add anything, please leave a comment here).

Here are some things my tired brain remembers from the discussions we had:

  • everybody seems to be using virtualenv when deploying their Python applications
  • everybody seems to be using Fabric in one way or another to push changes to remote nodes
  • the participants seemed to be split almost equally between Puppet and Chef for provisioning
  • the more disciplined of the companies ( for example) use Puppet/Chef both for provisioning and application deployment and configuration ( still uses Fabric for stopping/starting services on remote nodes for example)
  • other companies (including us at Evite) use Chef/Puppet for automated provisioning of the OS + pre-requisite packages, then use Fabric to push the deployment of the application because they prefer the synchronous aspect of a push approach
  • upgrading database schemas is hard; many people only do additive changes (NoSQL makes this easier, and as far as relational databases go, PostgreSQL makes it easier than MySQL )
  • many people struggle with how best to bundle their application with other types of files, such as haproxy or nginx configurations
    • at Evite we face the same issue, and we came up with the notion of a bundle, a directory structure that contains the virtualenv of the application, the configuration files for the application, and all the other configuration files for programs that interact with our application -- haproxy, nginx, supervisord for example
    • when we do a deploy, we check out a bundle via a revision tag, then we push the bundle to a given app server
  • some people prefer to take the OS package approach here, and bundle all the above types of files in an rpm or deb package
  • Noah Kantrowitz has released 2 Chef-related Python tools that I was not aware of: PyChef (a Python client that knows how to query a Chef server) and commis (a Python implementation of a Chef server, with the goal of being less complicated to install than its Ruby counterpart)
  • LittleChef was mentioned as a way to run Chef Solo on a remote node via fabric, thus giving you the control of a 'push' method combined with the advantage of using community cookbooks already published for Chef
  • I had to leave towards the end of the meeting, when people started to discuss the hosting aspect, so I don't have a lot to add here -- but it is interesting to me to see quite a few companies that have Platform-as-a-Service (PaaS) offerings for Python hosting: DjangoZoom,, Gondor ( can host any WSGI application, while the DjangoZoom and Gondor are focused on Django)

All in all, there were some very interesting discussions that showed that pretty much everybody is struggling with similar issues. There is no silver bullet, but there are some tools and approaches that can help make your life easier in this area. My impression is that the field of automated deployments and configuration management, even though changing fast, is also maturing fast, with a handful of tools dominating the space. It's an exciting space to play in!


Anonymous said...

Thanks for the writeup. One question I didn't get a chance to ask was whether anyone had any good Chef recipes for virtualenv and pip. The one I wrote isn't great and the ones I've seen on github are equally not-great.

Also, I was wondering if anyone knew anyone with a good solution for schema migrations with continuous deployments. All of the writeups in the area I've read hand-wave that part of the problem.

I got a chance to talk to some Evite guys and they said you had actually switched your application from Java to django/python. I'd really love to read about the challenges you guys faced while doing that.

Anyway, love the blog and wish I had run in to you at pycon. Maybe next year.

Grig Gheorghiu said...

Hi Wes -- thanks for the comments. I personally rely on the Ubuntu package manager for installing virtualenv and pip. It's good enough for what we need. So in Chef I just use the 'package' resource.

I agree with you RE db schema migrations, I haven't really seen a solid 'lessons learned' document about that either.

We are close to start blogging about more of our experiences at Evite on an Evite-maintained blog. I'll link to those articles in my blog as they appear (hopefully as soon as this week).

Hope to meet you next year!


Modifying EC2 security groups via AWS Lambda functions

One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...