Working with Chef cookbooks and roles

Welcome to the second installment of my Chef saga (you can read the first one here). This time I will walk you through creating your own cookbook, modifying an existing cookbook, creating a role and adding a client machine to that role. As usual, I got much help from the good people on the #chef IRC channel, especially the omnipresent @kallistec. All these tasks are also documented in one form or another on the Chef wiki, but I found it hard to put them all together, hence this blog post.

Downloading the Opscode Chef cookbooks

Step 0 for this task is to actually clone the Opscode repository. I created a directory called /srv/chef/repos on my Chef server box and ran this command inside it:

# git clone git://github.com/opscode/chef-repo.git

This will create /srv/chef/repos/chef-repo with a bunch of sub-directories, one of them being cookbooks.

I deleted the cookbooks directory and cloned the Opscode cookbooks in its place:

# cd /srv/chef/repos/chef-repo
# git clone git://github.com/opscode/cookbooks


Uploading the cookbooks to the Chef server

Just downloading the cookbooks somewhere on the file system is not enough. The Chef server needs to be made aware of their existence. You do that with the following knife command (which I ran on the Chef server box):

# knife cookbook upload -a -o /srv/chef/repos/chef-repo/cookbooks

BTW, if you specified a non-default configuration file location for knife when you configured it (I specified /etc/chef/knife.rb for example) then you need to make a symlink from that file to ~/.chef/knife.rb, otherwise knife will complain about not finding a config file. At least it complained to me in its strange Swedish accent.

To go back to the knife command above: it says to upload all (-a) cookbooks it finds under the directory specified with -o.

Modifying an existing cookbook

If you look closely under the Opscode cookbooks, there's one called python. A cookbook contains one or more recipes, which reside under COOKBOOK_NAME/recipes. Most cookbooks have only one recipe which is a file called default.rb. In the case of the python cookbook, this recipe ensures that certain python packages such as python-dev, python-imaging, etc. get installed on the node running Chef client. To add more packages, simply edit default.rb (there's a certain weirdness in modifying a Ruby file to make a node install more Python packages....) and add your packages of choice.

Again, modifying a cookbook recipe on the file system is not enough; you need to let the Chef server know about the modification, and you do it by uploading the modified cookbook to the Chef server via knife:

# knife cookbook upload python -o /srv/chef/repos/chef-repo/cookbooks

Note the modified version of the 'knife cookbook upload' command. In this case, we don't specify '-a' for all cookbooks, but instead we specify a cookbook name (python). However, the directory remains the same. Do not make the mistake of specifying /srv/chef/repos/chef-repo/cookbooks/python as the target of your -o parameter, because it will not work. Trust me, I tried it until I was enlightened by @kallistec on the #chef IRC channel.

Creating your own cookbook

It's time to bite the bullet and create your own cookbook. Chef makes it easy to create all the files needed inside a cookbook. Run this command when in the top-level chef repository directory (/srv/chef/repos/chef-repo in my case):

# rake new_cookbook COOKBOOK=octopus

(like many other people, I felt the urge to cook Paul the Octopus when he accurately predicted Germany's loss to Spain)

This will create a directory cscp neo-app01:/opt/evite/etc/ad_urls.json .

alled octopus under chef-repo/cookbooks and it will populate it with many other directories and files. At a minimum, you need to modify only octopus/recipes/default.rb. Let's assume you want to install some packages. You can take some inspiration from other recipes (build-essential for example), but it boils down to something like this:

include_recipe "build-essential"

include_recipe "ntp"

include_recipe "python"

include_recipe "screen"

include_recipe "git"

%w{chkconfig libssl-dev syslog-ng munin-node}.each do |pkg|

  package pkg do

    action :install

  end

end

Note that I'm also including other recipes in my default.rb file. They will be executed on the Chef client node, BUT they also need to be specified in the metadata file cookbooks/octopus/metadata.rb as dependencies, so that the Chef client node knows that it needs to download them before running them. Trust me on this one, I speak again from bitter experience. This is how my metadata.rb file looks like:

maintainer        "My Organization"

maintainer_email  "admin@example.com"

license           "Apache 2.0"

description       "Installs required packages for My Organization applications"

version           "0.1"

recipe            "octopus", "Installs required packages for My Organization applications"

depends           "build-essential"

depends           "ntp"

depends           "python"

depends           "screen"

depends           "git"

%w{ fedora redhat centos ubuntu debian }.each do |os|

  supports os

end

Now it's time to upload our brand new cookbook to the Chef server. As before, we'll use the knife utility:

# knife cookbook upload octopus -o /srv/chef/repos/chef-repo/cookbooks

If you have any syntax errors in the recipe file or the metadata file, knife will let you know about them. To verify that the cookbook was uploaded successfully to the server, run this command, which should list your cookbook along the others that you uploaded previously:

# knife cookbook list

Creating a role and associating a chef client machine with it

Chef supports the notion of roles, which is very important for automated configuration management because you can assign a client node to one or more roles (such as 'web' or 'db' for example) and have it execute recipes associated with those roles.

To add a role, simply create a file under chef-repo/roles. I called mine base.rb, with the contents:

name "base"

description "Base role (installs common packages)"

run_list("recipe[octopus]")

It's pretty self-explanatory. Clients associated with the 'base' role will run the 'octopus' recipe.

As with cookbooks, we need to upload the newly created role to the Chef server. The following knife command will do it (assuming you're in the chef-repo directory):

# knife role from file roles/base.rb

To verify that the role has been uploaded, you can run:

# knife role show base

and it should show a JSON output similar to:

{

    "name": "base",

    "default_attributes": {

    },

    "json_class": "Chef::Role",

    "run_list": [

      "recipe[octopus]"

    ],

    "description": "Base role (installs common packages)",

    "chef_type": "role",

    "override_attributes": {

    }

}

Now it's time to associate a Chef client with the new role. You can see which clients are already registered with your Chef server by running:

# knife client list

Now pick a client and see which roles it is associated with:

# knife node show client01.example.com -r

{

  "run_list": [

  ]

}

The run_list is empty for this client. Let's associate it with the role we created, called 'base':

# knife node run_list add client01.example.com "role[base]"

You should now see the 'base' role in the run_list:

# knife node show client01.example.com -r
{
  "run_list": [
    "role[base]"
  ]
}

The next time client01.example.com will run chef-client, it will figure out it is part of the 'base' role, and it will follow the role' run_list by downloading and running the 'octopus' recipe from the Chef server.

In the next installment, I will talk about how to automatically bootstrap a Chef client in an EC2 environment. The goal there is to launch a new EC2 instance, have it install Chef client, have it assign a role to itself, then have it talk to the Chef server, download and run the recipes associated with the role.