Friday, February 28, 2014

Example of Chef workflow

Here is a quick example of a Chef workflow that has been working for us. It can be easily improved on, especially around testing, but it's a good foundation.

1) Put your chef-repo on Github.
2) When you want to modify a cookbook, do a git pull to get the latest version of the cookbook.
3) Modify the cookbook.
4) Check your environments (I'll assume staging and production for now, to keep it simple) to see what version of the cookbook is used in production vs staging. Let's assume both staging and production environments use the latest version of the cookbook, say 0.1.
5) Modify metadata.rb and bump up the version of the cookbook to 0.2.
6) Modify the staging environment file (for example environments/stg.rb) and pin the cookbook you modified to version 0.2. Make sure the production environment is still pinned to 0.1.
7) Update the staging environment on the Chef server via: 'knife environment from file environments/stg.rb'
8) Upload the new version of the cookbook (0.2) to the Chef server via: 'knife cookbook upload mycookbook' (it should report version 0.2 after the upload)
9) Run chef-client on a staging box that uses the cookbook you modified. Check that everything looks good.
10) Assuming everything looks good in staging, modify the production environment file (for example environments/prod.rb) and pin the cookbook you modified to the new version 0.2.
11) Update the production environment on the Chef server via: 'knife environment from file environments/prod.rb'.
12) Run chef-client on a prod box and check that everything is OK. If it looks good, either let chef-client run by itself on all prod boxes, or run chef-client manually to force the change.
13) Commit your coobook and environment changes into git and push to Github.

Note that there is the possibility of screw-ups if somebody forgets step #13. For this reason, I usually am double careful and check especially my local version of the environment files (stg.rb and prod.rb) against what is actually running on the Chef server. I run 'knife environment show stg' and compare the result to stg.rb. I also run 'knife environment show prod' and compare the result to prod.rb. Only if they both look good do I modify my local copies of stg.rb and prod.rb and then upload them to the Chef server. We've had issues in the past with changes that were made to the Chef server directly (via 'knife environment edit') that got overwritten when somebody uploaded their version of the environment file that contained an older version of the given cookbook. For this reason I don't recommed making changes directly on the Chef server by editing roles, environments, etc, but instead making all changes on your local files, then uploading those files to Chef and also committing those changes to Github.

As I said in the beginning, there is the opportunity to run various testing tools (at a minimum rubocop and Foodcritic) on your cookbook before uploading it to the Chef server. But that is for another post.


Jonathan Q said...

You could automate some of the cookbook version changes/environment pinning by using Knife Spork from Etsy: knife-spork

May remove some of the manual steps from your workflow (editing version and pinning in the environment).

Grig Gheorghiu said...

Thanks, Jonathan. I did see Knife Sport mentioned in some blog posts, it's on my TODO list to look into it. Thanks for the reminder.

ScottInNH said...

Curious, in the past you were using Fabric for systems building and configuration. Now you are using Chef, which arguably is more standard for building systems.

Are you still using Fabric in any capacity, and where do you see it heading?

Fabric still seems to be the choice for executing remote tasks (where the task is not a permanent part of configuration -- for example if you want to remotely execute tests, or gather system information from multiple systems).

Your introduction article on Fabric was a lifesaver for me (previously having been defining SSH based tasks using Paramiko directly, which is far too low level for any productive work).

Grig Gheorghiu said...

Hi Scot -- I'm not using Fabric anymore. For better or worse, we use Ruby as our scripting language of choice at my current job, so we use a combination Jenkins/Capistrano/Chef for deployments. Works pretty well. I would still use Fabric if I were using Python.

Jonathan Q said...


I don't see fabric (or Capistrano) and Chef as mutually exclusive. They solve different problems in my mind.

I use Chef to configure the state of a machine (packages, directory structure, configuration, etc).

I use Fabric to perform operations on those machines. On example, I have some chef recipes that are not run on a server on a regular basis. I can actually trigger chef runs on a node using Fabric and execute those specific recipes.

Or as Grig mentioned, using Fabric/Capistrano to trigger deployments on your servers if you don't have a continuous deployment set up (or want to control deployment timing).

If you use Fabric + Chef - take a look at pychef (

Modifying EC2 security groups via AWS Lambda functions

One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...