Automated deployments with Fabric - tips and tricks

Fabric has become my automated deployment tool of choice. You can find tons of good blog posts on it if you google it. I touched upon it in 2 blog posts, 'Automated deployments with Puppet and Fabric' and 'Automated deployment systems: push vs. pull'. Here are some more tips and tricks extracted from my recent work with Fabric.

If it's not in a Fabric fabfile, it's not deployable

This is a rule I'm trying to stick to. All deployments need to be automated. No ad-hoc, one-off deployments allowed. If you do allow them, they can quickly snowball into an unmaintainable, unreproducible mess.

Update (via njl): Only deploy files that are checked out of source control

So true. I should have included this when I wrote the post. Make sure your fabfiles, as well as all files you deploy remotely are indeed what you expect them to be. It's best to keep them in a source control repository, and to be disciplined about checking them in every time you update them.

Along the same lines: when deploying our Tornado-based Web services here at Evite, we make sure our continuous integration system (we use Hudson) builds the eggs and saves them in a central location, and we install those eggs with Fabric from the central location.

Common Fabric operations

I find myself using a small number of the Fabric functions. Basically I mainly use these three operations:

1) copy files to a remote server (with the 'put' function)
2) run a command as a regular user on a remote server (with the 'run' function)
3) run a command as root on a remote server (with the 'sudo' function)

I also make use sometimes of the 'sed' function, which allows me for example to comment or uncomment lines in an Nginx configuration file, so I can take servers in and out of the load balancer (for commenting lines out, you can also use the 'comment' function).

Complete example

Here is a complete example of installing munin-node on a server:

# cat fab_munin.py

from fabric.api import *

# Globals

env.project='MUNIN'
env.user = 'root'

# Environments
from environments import *

# Tasks

def install():
   install_prereqs()
   copy_munin_files()
   start()

def install_prereqs():
   sudo("apt-get -y install munin-node chkconfig")
   sudo("chkconfig munin-node on")

def copy_munin_files():
   run('mkdir -p munin')
   put('munin/munin-node.conf', 'munin')
   put('munin/munin-node', 'munin')
   sudo('cp munin/munin-node.conf /etc/munin/')
   sudo('cp munin/munin-node /etc/munin/plugin-conf.d')

def start():
   sudo('service munin-node start')

def stop():
   sudo('service munin-node stop')

def restart():
   sudo('service munin-node restart')

The main function in this fabfile is 'install'. Inside it, I call 3 other functions which can be also called on their own: 'install_prereqs' installs the munin-node package via apt-get, 'copy_munin_files' copies various configuration files to the remote server under ~/munin, then sudo copies them to /etc/munin, and finally 'start' starts up the munin-node service. I find that this is a fairly common pattern for my deployments: install base packages, copy over configuration files, start or restart service.

Note that I'm importing a module called 'environments'. That's where I define and name lists of hosts that I want to target during the deployment. For example, your environments.py file could contain the following:

from fabric.api import *

def stgapp():
    """App staging environment."""
    env.hosts = [
        'stgapp01', 
        'stgapp02',
    ]

def stgdb():
    """DB staging environment."""
    env.hosts = [
        'stgdb01',
        'stgdb02',
    ]

def prdapp():
    """App production environment."""
    env.hosts = [
        'prdapp01', 
        'prdapp02',
    ]

def prddb():
    """DB production environment."""
    env.hosts = [
        'prddb01',
        'prddb02',
    ]

To install munin-node on all the staging app servers, you would run:

fab -f fab_munin stgapp install

If you wanted to install munin-node on a server which is not part of any env.hosts lists defined in environments.py, you would just call:

fab -f fab_munin install

...and the fab utility will ask you for a host name or IP on which to run the 'install' function. Simple and powerful.

Dealing with configurations for different environment types

A common issue I've seen is that different environment types (test, staging, production) need different configurations, both for services such as nginx or apache, and for my own applications. One solution I found is to prefix my environment names with their types (tst for testing, stg for staging and prd for production), and to add a sufix to the configuration files, for example nginx.conf.tst for the testing environment, nginx.conf.stg for staging and nginx.conf.prd for production.

Then, in my fabfiles, I automatically send the correct configuration file over, based on the environment I specify on the command line. For example, I have this function in my fab_nginx.py fabfile:

def deploy_config():
    host_prefix = env.host[:3]
    env.config_file = 'nginx.conf.%s' % host_prefix
    put('nginx/%(config_file)s' % env, '~')
    sudo('mv ~/%(config_file)s /usr/local/nginx/conf/nginx.
    with settings(warn_only=True):
        sudo('service nginx restart')

When I run:

fab -f fab_nginx tstapp deploy_config

...this will deploy the configuration on the 2 hosts defined inside the tstapp list -- tstapp01 and tstapp02. The deploy_config function will capture 'tst' as the first 3 characters of the current host name, and it will then operate on the nginx.conf.tst file, sending it to the remote server as /usr/local/nginx/conf/nginx.conf.

I find that it is a good practice to not have local files called nginx.conf, because the risk of overwriting the wrong file in the wrong environment is increased. Instead, I keep 3 files around -- nginx.conf.tst, nginx.conf.stg and nginx.conf.prd -- and I copy them to remote servers accordingly.

Defining short functions inside a fabfile

My fabfiles are composed of short functions that can be called inside a longer function, or on their own. This gives me the flexibility to only deploy configuration files for example, or only install base packages, or only restart a service.

Automated deployments ensure repeatability

I may repeat myself here, but it is worth rehashing this point: in my view, the best feature of an automated deployment system is that it transforms your deployments from an ad-hoc, off-the-cuff procedure into a repeatable and maintainable process that both developer and operation teams can use (#devops anybody?). An added benefit is that you get for free a good documentation for installing your application and its requirements. Just copy and paste your fabfiles (or Puppet manifests) into a wiki page and there you have it.