DKIM setup with postfix and OpenDKIM

If sending email is a critical part of your online presence, then it pays to look at ways to enhance the probability that messages you send will find their way into your recipients' inboxes, as opposed to their spam folders. This is fairly hard to achieve and there are no silver bullet guarantees, but there are some things you can do to try to enhance the reputation of your mail servers.

One thing you can do is use DKIM, which stands for DomainKeys Identified Mail. DKIM is a result of a merging between Yahoo's DomainKeys and Cisco's Identified Internet Mail. There's a great Wikipedia article on DKIM which I strongly recommend you read.

DKIM is a method for email authentication -- in a nutshell, mail senders use DKIM to digitally sign messages they send with a private key. They also publish the corresponding public key as a DNS record. On the receiving side, mail servers use the public key to verify the digital signature. So by using DKIM, you as a mail sender prove to your mail recipients that you are who you say you are.

Note that DKIM doesn't prevent spam. Spammers can also use DKIM, and sign their messages. However, DKIM achieves an important goal, which is to prevent spammers from spoofing the source of their emails and impersonate users in other mail domains by forging the 'From:' header in their spam emails. If spammers want to use DKIM, they are thus forced to use their real domain name in the 'From' header, and this makes it easier for the receiving mail servers to reject that email. See also 'Three myths about DKIM' by John R. Levine.

As an email sender, if you use DKIM, then your chances of your mail servers being whitelisted and of your mail domain being considered 'reputable' are increased.

Enough theory, let's see how you can set up DKIM with postfix. I'm going to use OpenDKIM and postfix on an Ubuntu 9.04 server.

1) Install prerequisites

# apt-get install libssl-dev libmilter-dev

2) Download and install OpenDKIM

# wget http://downloads.sourceforge.net/project/opendkim/opendkim-2.0.1.tar.gz
# tar xvfz opendkim-2.0.1.tar.gz
# cd opendkim-2.0.1
# ./configure
# make
# make install

You will also need to add /usr/local/lib to the paths inspected by ldconfig, otherwise opendkim will not find its required shared libraries when starting up. I created a file called /etc/ld.so.conf.d/opendkim.conf containing just one line:

/usr/local/lib

Then I ran:

# ldconfig

3) Add a 'dkim' user and group

# useradd dkim

4) Use the opendkim-genkey.sh script to generate a private key (in PEM format) and a corresponding public key inside a TXT record that you will publish in your DNS zone. The opendkim-genkey.sh script takes as arguments your DNS domain name (-d) and a so-called selector, which identifies this particular private key/DNS record combination. You can choose any selector name you want. In my example I chose MAILOUT.

# cd ~/opendkim-2.0.1/opendkim
# ./opendkim-genkey.sh -d mydomain.com -s MAILOUT
# mkdir -p /var/db/dkim
# cp MAILOUT.private /var/db/dkim/MAILOUT.key.pem

The opendkim-genkey.sh script generates a private key called MAILOUT.private, which I'm copying to /var/db/dkim/MAILOUT.key.pem. It also generates a file called MAILOUT.txt which contains the TXT record that you need to add to your DNS zone:

# cat MAILOUT.txt
MAILOUT._domainkey IN TXT "v=DKIM1; g=*; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADGTeQKBgQDATldYm37cGn6W1TS1Ukqy2ata8w2mw+JagOC+GNEi02gvDyDtfTT0ivczPl45V1SqyQhdwF3dPnhkUXgxjjzBvU6+5uTFU4kzrKz0Ew7vywsCMcfKUjYhtVTAbKAy+5Vf3CNmOlFlJnnh/fdQHVsHqqNjQII/13bVtcfYPGOXJwIDAQAB" ; ----- DKIM MAILOUT for mydomain.com

5) Configure DNS

Add the generated TXT record to the DNS zone for mydomain.com.

6) Configure opendkim

There is a sample file called opendkim.conf.sample located in the root directory of the opendkim source distribution. I copied it as /etc/opendkim.conf, then I set the following variables (this article by Eland Systems was very helpful):

Canonicalization relaxed/simple
Domain mydomain.com
InternalHosts /var/db/dkim/internal_hosts
KeyFile /var/db/dkim/MAILOUT.key.pem
Selector MAILOUT
Socket inet:9999@localhost
Syslog Yes
UserID dkim
X-Header yes

Note the InternalHosts setting. It points to a file listing IP addresses that belong to servers which use your mail server as a mail relay. They could be for example your application servers that send email via your mail server. You need to list their IP addresses in that file, otherwise mail sent by them will NOT be signed by your mail server running opendkim.

7) Start up opendkim


# /usr/local/sbin/opendkim -x /etc/opendkim.conf

At this point, opendkim is running and listening on the port you specified in the config file -- in my case 9999.

8) Configure postfix

You need to configure postfix to use opendkim as an SMTP milter.

Note that the postfix syntax in the opendkim documentation is wrong. I googled around and found this blog post which solved the issue.

Edit /etc/postfix/main.cf and add the following line:

smtpd_milters = inet:localhost:9999

(the port needs to be the same as the one opendkim is listening on)

Reload the postfix configuration:

# service postfix reload

9) Troubleshooting

At first, I didn't know about the InternalHosts trick, so I was baffled when email sent from my application servers wasn't being signed by opendkim. To troubleshoot, make sure you set

LogWhy yes

in opendkim.conf and then inspect /var/log/mail.log and google for any warnings you find (remember to always RTFL!!!).

When you're done troubleshooting, set LogWhy back to 'no' so that you don't log excessively.

10) Verifying your DKIM setup

At this point, try to send email to some of your email accounts such as gmail, yahoo, etc. When you get the email there, show all headers and make sure you see the DKIM-Signature and X-DKIM headers. Here's an example from an email I received in my GMail account (you need to click the 'Reply' drop-down and choose 'Show original' to see all the headers):

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mydomain.com;
 s=MAILOUT; t=1269364019;
 bh=5UokauIClLiW/0JovG5US3xUr2LcjGTnutud9Ke1V2E=;
 h=MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID:
  Subject:From:Reply-to:To:Date;
 b=X7cgUltyjJpqN7xavEHPrLSTnEqyj7fsuS/4HDrs3YfZg2d8K9EdvqJ5gwdrmkVEL
  M27ZDugI0yaK5C+cdOZ2Fyj8nG83nLwcnMz7X3EqCkHP0CPlv9FCXtjispxLJ2W3xc
  AUQnp4vVChYb/TEGmQV+Ilzzf9a9WDUMhWlaljjM=
X-DKIM: OpenDKIM Filter v2.0.1 mymailserver 4B4B864925

That's about it. There are quite a few moving parts here, so I hope somebody will find this useful.

Automated deployments with Fabric - tips and tricks

Fabric has become my automated deployment tool of choice. You can find tons of good blog posts on it if you google it. I touched upon it in 2 blog posts, 'Automated deployments with Puppet and Fabric' and 'Automated deployment systems: push vs. pull'. Here are some more tips and tricks extracted from my recent work with Fabric.

If it's not in a Fabric fabfile, it's not deployable

This is a rule I'm trying to stick to. All deployments need to be automated. No ad-hoc, one-off deployments allowed. If you do allow them, they can quickly snowball into an unmaintainable, unreproducible mess.

Update (via njl): Only deploy files that are checked out of source control

So true. I should have included this when I wrote the post. Make sure your fabfiles, as well as all files you deploy remotely are indeed what you expect them to be. It's best to keep them in a source control repository, and to be disciplined about checking them in every time you update them.

Along the same lines: when deploying our Tornado-based Web services here at Evite, we make sure our continuous integration system (we use Hudson) builds the eggs and saves them in a central location, and we install those eggs with Fabric from the central location.

Common Fabric operations

I find myself using a small number of the Fabric functions. Basically I mainly use these three operations:

1) copy files to a remote server (with the 'put' function)
2) run a command as a regular user on a remote server (with the 'run' function)
3) run a command as root on a remote server (with the 'sudo' function)

I also make use sometimes of the 'sed' function, which allows me for example to comment or uncomment lines in an Nginx configuration file, so I can take servers in and out of the load balancer (for commenting lines out, you can also use the 'comment' function).

Complete example

Here is a complete example of installing munin-node on a server:

# cat fab_munin.py

from fabric.api import *

# Globals

env.project='MUNIN'
env.user = 'root'

# Environments
from environments import *

# Tasks

def install():
   install_prereqs()
   copy_munin_files()
   start()

def install_prereqs():
   sudo("apt-get -y install munin-node chkconfig")
   sudo("chkconfig munin-node on")

def copy_munin_files():
   run('mkdir -p munin')
   put('munin/munin-node.conf', 'munin')
   put('munin/munin-node', 'munin')
   sudo('cp munin/munin-node.conf /etc/munin/')
   sudo('cp munin/munin-node /etc/munin/plugin-conf.d')

def start():
   sudo('service munin-node start')

def stop():
   sudo('service munin-node stop')

def restart():
   sudo('service munin-node restart')

The main function in this fabfile is 'install'. Inside it, I call 3 other functions which can be also called on their own: 'install_prereqs' installs the munin-node package via apt-get, 'copy_munin_files' copies various configuration files to the remote server under ~/munin, then sudo copies them to /etc/munin, and finally 'start' starts up the munin-node service. I find that this is a fairly common pattern for my deployments: install base packages, copy over configuration files, start or restart service.

Note that I'm importing a module called 'environments'. That's where I define and name lists of hosts that I want to target during the deployment. For example, your environments.py file could contain the following:

from fabric.api import *

def stgapp():
    """App staging environment."""
    env.hosts = [
        'stgapp01', 
        'stgapp02',
    ]

def stgdb():
    """DB staging environment."""
    env.hosts = [
        'stgdb01',
        'stgdb02',
    ]

def prdapp():
    """App production environment."""
    env.hosts = [
        'prdapp01', 
        'prdapp02',
    ]

def prddb():
    """DB production environment."""
    env.hosts = [
        'prddb01',
        'prddb02',
    ]

To install munin-node on all the staging app servers, you would run:

fab -f fab_munin stgapp install

If you wanted to install munin-node on a server which is not part of any env.hosts lists defined in environments.py, you would just call:

fab -f fab_munin install

...and the fab utility will ask you for a host name or IP on which to run the 'install' function. Simple and powerful.

Dealing with configurations for different environment types

A common issue I've seen is that different environment types (test, staging, production) need different configurations, both for services such as nginx or apache, and for my own applications. One solution I found is to prefix my environment names with their types (tst for testing, stg for staging and prd for production), and to add a sufix to the configuration files, for example nginx.conf.tst for the testing environment, nginx.conf.stg for staging and nginx.conf.prd for production.

Then, in my fabfiles, I automatically send the correct configuration file over, based on the environment I specify on the command line. For example, I have this function in my fab_nginx.py fabfile:

def deploy_config():
    host_prefix = env.host[:3]
    env.config_file = 'nginx.conf.%s' % host_prefix
    put('nginx/%(config_file)s' % env, '~')
    sudo('mv ~/%(config_file)s /usr/local/nginx/conf/nginx.
    with settings(warn_only=True):
        sudo('service nginx restart')

When I run:

fab -f fab_nginx tstapp deploy_config

...this will deploy the configuration on the 2 hosts defined inside the tstapp list -- tstapp01 and tstapp02. The deploy_config function will capture 'tst' as the first 3 characters of the current host name, and it will then operate on the nginx.conf.tst file, sending it to the remote server as /usr/local/nginx/conf/nginx.conf.

I find that it is a good practice to not have local files called nginx.conf, because the risk of overwriting the wrong file in the wrong environment is increased. Instead, I keep 3 files around -- nginx.conf.tst, nginx.conf.stg and nginx.conf.prd -- and I copy them to remote servers accordingly.

Defining short functions inside a fabfile

My fabfiles are composed of short functions that can be called inside a longer function, or on their own. This gives me the flexibility to only deploy configuration files for example, or only install base packages, or only restart a service.

Automated deployments ensure repeatability

I may repeat myself here, but it is worth rehashing this point: in my view, the best feature of an automated deployment system is that it transforms your deployments from an ad-hoc, off-the-cuff procedure into a repeatable and maintainable process that both developer and operation teams can use (#devops anybody?). An added benefit is that you get for free a good documentation for installing your application and its requirements. Just copy and paste your fabfiles (or Puppet manifests) into a wiki page and there you have it.

Automated deployment systems: push vs. pull

I've been immersed in the world of automated deployment systems for quite a while. Because I like Python, I've been using Fabric, but I also dabbled in Puppet. When people are asked about alternatives to Puppet in the Python world, many mention Fabric, but in fact these two systems are very different. Their main difference is the topic of this blog post.

Fabric is what I consider a 'push' automated deployment system: you install Fabric on a server, and from there you push deployments by running remote commands via ssh on a set of servers. In the Ruby world, an example of a push system is Capistrano.

The main advantages of a 'push' system are:

• control: everything is synchronous, and under your control. You can see right away is something went wrong, and you can correct it immediately.
• simplicity: in the case of Fabric, a 'fabfile' is just a collection of Python functions that copy files over to a remote server and execute commands over ssh on that server; it's all very easy to set up and run

The main disadvantages of a 'push' system are:

• lack of full automation: it's not usually possible to boot a server and have it configure itself without some sort of client/server protocol which push systems don't generally support (see 'pull' systems below for that)
• lack of scalability: when you're dealing with hundreds of servers, a push system starts showing its limits, unless it makes heavy use of threading or multi-processing

Puppet is what I consider a 'pull' automated deployment system (actually to be more precise, it is a configuration management system). In such a system, you have a server which acts as a master, and clients which contact the master to find out what they need to do, thus pulling their configuration information from the master. In Puppet, configuration files are called manifests. They are written in a specific language and they are declarative, i.e. they tell each client what to do, not how to do it. The Puppet client software running on each server knows how to interpret the manifest files and how to translate them into actions specific to the operating system of that server. For example, you specify in your manifest file that you want a user created and you don't need to say 'run the adduser command on server X'. Other examples of 'pull' deployment/configuration management systems are bcfg2 (Python),Chef (Ruby) and slack (Perl). A newcomer in the Python world is a port of Chef called kokki (it looks like it's very much in its infancy still, but I hope the author will continue to actively develop it).

The main advantages of a 'pull' system are:

• full automation capabilities: it is possible, and indeed advisable, to fully automate the configuration of a newly booted server using a 'pull' deployment system (for details on how I've done it with Puppet, see this post)
• increased scalability: in a 'pull' system, clients contact the server independently of each other, so the system as a whole is more scalable than a 'push' system

The main disadvantages of a 'pull' system are:

• proprietary configuration management language: with the notable exception of Chef, which uses pure Ruby for its configuration 'recipes', most other pull system use their own proprietary way of specifying the configuration to be deployed (Puppet's language looks like a cross between Perl and Ruby, while bcfg2 uses...gasp...XML); this turns out to be a pretty big drawback, because if you're not using the system on a daily basis, you're guaranteed to forget it (as happened to me with Puppet)
• scalability is still an issue: unless you deploy several master servers and keep them in sync, that one master will start getting swamped as you add more and more clients and thus will become your bottleneck

My particular preference is to use a 'pull' system for the initial configuration of a server, including all the packages necessary to deploy my application (for example tornado). For the actual application code deployment, I prefer to use a 'push' system, because it gives me more control over how exactly I do the deployment. I can take a server out of the load balancer, deploy, test, then put it back, rinse and repeat.

In discussions with Holger Krekel at PyCon, I realized that execnet might be a good replacement for Fabric for my needs. It already provides remote command execution via ssh, and an rsync-like file transfer protocol. All it needs is a small library of functions on top to do common system administration tasks such as running commands as sudo, etc. I also want to look into kokki as a replacement for Puppet in my deployment architecture.

A parting thought: my colleague Dan Mesh suggested using a queuing mechanism for the client-server protocol in a 'pull' system. In fact, I am becoming more and more convinced that as far as scalability is concerned, when in doubt, use a queuing mechanism. In this deployment architecture, the master would post tasks to be done by a specific client to a central queue. The client would check the queue periodically for a task assigned to it, would execute it then would send a report back to the server when done. Of course, you need to worry about authentication in this scenario, but it seems that it would solve a lot of the scalability issues that both push and pull systems exhibit. Who knows, we may build it at Evite and open source it...so stay tuned ;-)

Getting past hung remote processes in Fabric

This is a note for myself, but maybe it will be useful to other people too.

I've been using Fabric version 1.0a lately, and it's been working very well, with an important exception: when launching remote processes that get daemonized, the 'run' Fabric command which launches those processes hangs, and needs to be forcefully killed on the server where I run the 'fab' commands.

I remembered vaguely reading on the Fabric mailing list something about the ssh channel not being closed properly, so I hacked the source code (operations.py) to close the channel before waiting for the stdout/stderr capture threads to exit.

Here's my diff:

git diff fabric/operations.py 
diff --git a/fabric/operations.py b/fabric/operations.py
index 9c567c1..fe12450 100644
--- a/fabric/operations.py
+++ b/fabric/operations.py
@@ -498,12 +498,13 @@ def _run_command(command, shell=True, pty=False, sudo=False, user=None):
     # Close when done
     status = channel.recv_exit_status()
     
+    # Close channel
+    channel.close()
+
     # Wait for threads to exit so we aren't left with stale threads
     out_thread.join()
     err_thread.join()

-    # Close channel
-    channel.close()

     # Assemble output string
     out = _AttributeString("".join(capture_stdout).strip())

I realize this is a hack, but it solved my particular problem...If you've seen this and have found a different solution, please leave a comment.

Use HAProxy 1.4 if you need MySQL health checks

I was using the latest version of HAProxy 1.3 and was load balancing backend MySQL servers while also checking their ports, so if one server went down it would be taken out of the load balancing pool. However, since the port checks in HAProxy happen at the TCP level, the MySQL instance which was being hit by the port checks wasn't happy, because it wasn't a proper MySQL connection. As a result, after a number of some checks, MySQL refused to allow clients to connect, with a message like this:

OperationalError: (1129, "Host 'myhost' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'")

Solution: upgrade HAProxy to the newly released version 1.4 (at the date of this writing, the exact version is 1.4.0). Documentation is here.

For MySQL specific checks, you can specify 'option mysql-check' in a backend or 'listen' section of the configuration file. For example, I have something similar to this in my HAProxy configuration file:


listen mysql 0.0.0.0:3306
mode tcp
option mysql-check
balance roundrobin
server mysql1 10.0.1.1:3306 check port 3306
server mysql2 10.0.1.2:3306 check port 3306 backup

My top 10 PyCon takeaways

PyCon 2010 in Atlanta was a blast as always. While I still have things fresh on my mind, here are my top 10 takeaways from the conference, in no particular order.

1) Alternative Python implementations are getting increased attention

It seemed to me that PyPy, Unladen Swallow, IronPython and Jython got much more buzz this year. Maybe this is also due to the announcement that Unladen Swallow will be merged into Python 3.x. I recommend you watch Holger Krekel's talk on the topic of the diverse and healthy Python ecosystem, 'The Ring of Python'.

I was also glad to see that 2 core Jython developers, Frank Wierzbicki and Jim Baker, were hired by Sauce Labs.

2) Testing has gone mainstream

When Mark Shuttleworth mentions automated testing in his keynote as one of the most important ingredients of a sound software engineering process, you know that automated testing has arrived.

There were also no less than 6 testing-related talks, all very good, given by the usual suspects in the testing world, people like Ned Batchelder, Titus Brown, Holger Krekel and Michael Foord.

3) Packaging and packaging

I liked Antonio Rodriguez's distinction between packaging with small 'p' (distutils, setuptools, distribute) and Packaging with big 'P' (the python.org web site). Both are very important. There was a lot of attention to packaging, and a great show of support for Tarek Ziade's efforts in leading the way to improving the way of distributing Python packages. And I think Antonio was right in pointing out that the python.org site needs some redesign in terms of getting a more modern and streamlined look and feel.

4) I tweet, thus I exist

I came late to the Twitter party, barely a month ago. I was resistant at first because I considered tweeting a waste of time. I still think it has a strong tendency to shorten your attention span and break your focus, so I personally need to discipline myself in how I use it.

But Twitter is a great way to keep your finger on the pulse of topics that interest you -- and at PyCon, if you didn't tweet or at least read other people's tweets, you were out of touch, out of the picture. Alex Gaynor and company did a great job with their PyCon Live Stream site, which was pretty much the dashboard of the conference.

5) The testing goat

Terry Peppers started a new meme during the TiP BoF: the testing goat. Read Terry's post and also Titus's post for more details on it, but suffice it to say it was a huge success. And speaking of the TiP Bof, it ballooned from last year to this one. I estimate around 120 attendees, so more than 10% of the people at the conference. Pizza provided by Disney (thanks to Paul Hildebrandt and Roy Turner), beer provided by Dr. Brown and friends, great lightning talks and unceasing heckling made this into one of the highlights of the conference.

6) Healthy ecosystem of Python web frameworks

Two or three years ago, all the buzz was about Django and maybe TurboGears. This year, a lot of presenters talked about other frameworks -- Pylons in particular, but also tornado, CherryPy, restish. It does feel like Django is the granddaddy of them all, but it also feels to me like Pylons is being preferred by big name/big traffic web sites such as reddit. Tornado of course is a newcomer, and we're using it very successfully at Evite. The presenter from Lolapps said they were also experimenting with it and were going to put it in production for some portions of their site.

7) Inspirational keynotes

I thought the keynotes were of much higher quality than in previous years. Mark Shuttleworth talked about 'Cadence, quality and design' (see the bitsource interview), while Antonio Rodriguez gave a very inspirational presentation on topics such as involving everybody in your company in coding (he knows, it sounds crazy...), about the strategic advantages of using Python, about putting more stable libraries into the stdlib (he mentioned httplib2, and I couldn't agree more -- we need that library in the stdlib!), and other stuff that you can see on his pycon 2010 page. You need to watch the video of his keynote though in order to appreciate the impact that it had (videos from PyCon are being made available as we speak on blip.tv).

One thing though -- I am a big Ubuntu fan, I have it on both my laptop(s) and desktop, and yet I was pained to see that Mark Shuttleworth couldn't use his slide deck because his laptop couldn't properly display a dual screen when using the conference projector. I struggled to make it work myself before delivering my presentation. Ubuntu really needs to get better dual-screen configuration management software.

8) It's all about the hallway discussions

For first comers to PyCon, or for people who intend to go next year, a word of advice: skip some of the presentations and instead join random people in hallway discussions, or for a beer at the bar. Trust me, you'll learn more than in almost any presentation. And you'll potentially make friends that you'll recognize the next time you go to PyCon. I've done this for 6 years now, and it never fails to amaze me how easy it is to get into deep technical discussions over a mind-bending range of topics. Non-technical discussions are usually mind-bending at PyCon too ;-)

9) More talks of the advanced type please

I heard from many people (and Titus has been saying it for years) that they wished the talks were a bit more advanced. I realize PyCon needs to cater to all types of Python users, from beginning to intermediate to expert, but still the conference track could use a larger number of advanced, mind-exploding, challenging presentations (such as Raymond Hettinger's talk). I understand though that next year there will be exactly such a track, dubbed 'Extreme Python', so I'm very much looking forward to it.

10) Top-notch organization

Finally, kudos to Van Lindberg, this year's PyCon chair, and the rest of the organizers, for delivering an almost perfect experience to the more than 1,000 attendees. I though the food was great, the WiFi was better than usual, the sessions went almost always smoothly (minus projector issues), and there was a great fun and camaraderie in the air. That's why PyCon is my (and many other people's) favorite conference. Keep it up guys!

Slides for 'Creating RESTful Web Services with restish'

For those who are interested, the slides for my PyCon 2010 talk 'Creating RESTful Web Services with restish' are online. Leave comments please if you attended the talk and want to start discussions on some of the topics I mentioned.

Update 02/25/10: the video is up too

Web site monitoring techniques and tools

When monitoring a Web site, you need to look at it both from a 'micro' perspective (i.e. are the individual devices and servers in your infrastructure running smoothly?) and from a 'macro' perspective (i.e. do your customers have a pleasant experience when accessing and using your site?; can they use your site's functionality to conduct their business online?). You can also think about these types of monitoring as 'engineering' monitoring vs. 'business' monitoring. They're both important, but most system administrators obviously focus on the engineering/micro-type monitoring, since it's closer to their skills and interests, and tend to not put much emphasis on the business/macro-type monitoring.

In this post I'll talk about various types of monitoring we're doing for the Evite Web site. I'll focus on what I call 'deep application monitoring', which to me means gathering and displaying metrics about your Web application's behavior as seen by the users.

System infrastructure monitoring

This is the 'micro' or 'engineering' type of monitoring that is the bread and butter of a sysops team. I need to eat crow here and say that there is a great open source monitoring tool out there, and its name is Nagios. That's what we use to monitor the health of all our internal servers and network devices, and to alert us via email and SMS if anything goes wrong. It works great, it's easy to set up and deploy automatically, it has tons of plugins to monitor pretty much everything under the sun. My mistake in the past was expecting it to also be a resource graphing tool, but that was simply asking too much from one single tool.

For resource graphing and visualization we use a combination of cacti, ganglia and munin. They're all good, but going forward I'll use munin because it's easy to set up and automate, and also allows you to group devices together and see the same metric (for example system load) across the devices in the group. This makes it easy to spot trends and outliers.

One non-typical aspect of our use of Nagios is that we run passive checks from the servers being monitored to the main Nagios server using NSCA. This avoids the situation where the central Nagios server becomes a bottleneck because it needs to poll all the devices it monitors either via ssh or via NRPE. In our setup, each plugin deployed on a monitored server pushes its own notifications to the central server using send_nsca. It's also fairly easy to write your own Nagios plugins.

What kinds of things do we monitor and graph? The usual suspects -- system load, CPU, memory, disk, processes -- but also things like number of database connections on each Web/app server, number of Apache and Tomcat/Java processes AND threads, size of the mail queue on mail servers, MySQL queries/second, threads, throughput...and many others.

In general, the more metrics you capture and graph, the easier it is to spot trends and correlations. Because that's what you need to do during normal day-to-day traffic on your Web site -- you need to establish baseline numbers which will tell you right away if something goes wrong, for example when you're being hit with more traffic than usual. Having a baseline makes it easy to spot spikes and flare-ups in the metrics you capture. Noticing correlations between these metrics makes it easier for you to troubleshoot things like site slowness or unresponsiveness.

For example, if you see that the Web servers aren't being pounded with more traffic than usual, yet the number of database connections on each Web server keeps going up, you can quickly look at the database metrics and see that indeed the database is slower than usual. It could be that there's a disk I/O problem, or maybe the database is working hard to reindex a large table...but in any case your monitoring and graphing systems should be your friends in diagnosing the problem.

I compare system infrastructure monitoring with unit testing in a development project. Just like when you develop you write a unit test every time you find a bug, with monitoring you add another metric or resource to be monitored and graphed every time you discover a problem with your Web site. Both system monitoring and unit testing work at the micro/engineering level.

By the way, one of the best ways to learn a new system architecture is to delve into the monitoring system, or to roll one out if there's none. This will make you deeply familiar with every server and device in your infrastructure.

External HTTP monitoring

This type of monitoring typically involves setting up your own monitoring server outside of your site's infrastructure, or using a 3rd party service such as Pingdom, Keynote or Gomez. We've been using Pingdom and have been very happy with it. While it doesn't provide the in-depth analysis that Keynote or Gomez do, it costs much less and still gives you a great window into the performance and uptime of your site, as seen by an external user.

In any such system, you choose certain pages within your Web application and you have the external monitor check for a specific character string within the page, to make sure the expected content is there, as opposed to a cryptic error message.

However, with Pingdom you can go deeper than that. They provide 'custom HTTP checks' which allow you to deploy an internal monitor reachable from the outside via HTTP, which checks whatever resource you need within your internal infrastructure, and sends some well-formatted XML back to Pingdom. This is useful when you have a farm of web/app servers behind a load balancer, and you want to check each one but without exposing an external IP address for each server.  Or you can have processes running on various port numbers on your web/app servers, and you don't want to expose those directly to Pingdom checks. Instead, you need to open port 80 to just one server, the one running the custom checks.

We've built a RESTful Web service based on restish for this purpose. We call it from Pingdom with certain parameters in the query string, and based on those we run various checks such as 'is our Web application app1 running on web servers A, B, C and D functioning properly?'. If the application doesn't respond as expected on server C, we send details about the error in the XML payload back to Pingdom and we get alerted on it.

We also use Pingdom for monitoring cloud-based resources that we interact with, for example Amazon S3 or SQS. For SQS, we have another custom HTTP check which goes against our restish-based Web service, which then connects to SQS, does some operations and reports back the elapsed time. We want to know if that time/latency exceeds certain thresholds. While we could have done that from our internal Nagios monitoring system, it would have required writing a Nagios plugin, while our restish application already knew how to send the properly formatted XML payload back to Pingdom.

This type of monitoring doesn't go very deep into checking the business logic rules of your application. It is however a 'macro'-type monitoring technique that should be part of your monitoring strategy because it mimics the user experience of your site's customers. I compare it to doing functional testing for your application. You don't test things end-to-end, but you choose some specific portion of your functionality (a specific Web page that is being hit by many users for example) and you make sure that your application provides indeed that functionality to its users in a timely manner.

Also, if you use a CDN provider, then they probably offer you a way to see statistics about your HTTP traffic using some sort of Web dashboard. Most providers also give you a breakdown of HTTP codes returned over some period of time. Again trending is your friend here. If you notice that the percentage of non-200 HTTP codes suddenly increases, you know something is not going well somewhere in your infrastructure. Many CDN providers offer a Web service API for you to query and get these stats, so you can integrate that functionality into your monitoring and alerting system.

Business intelligence monitoring

We've come to the 'deep application monitoring' section of this post. Business intelligence (BI) deals with metrics pertaining to the business of your Web site. Think of all the metrics you can get with Google Analytics -- and yes, Google Analytics in itself is a powerful monitoring tool.

BI is very much concerned with trends. Everybody wants to see traffic and number of users to the site going up and up. Your site may have very specific metrics that you want to keep track of. Let's say you want users to register and fill in details about themselves, upload a photo, etc. You want to keep track of how many user profiles are created per day, and you want to keep track of the rate of change from day to day, or even from hour to hour. It would be nice if you could spot a drop in that rate right away, and then identify the root cause of the problem. Well, we do this here at Evite with a combination of tools, the main one being a non-open-source non-free tool called SAM (for Simple Application Monitoring) written by Randy Wigginton.

In a nutshell, SAM uses a modified memcached server to capture URLs and SQL queries as they are processed by your application, and aggregates them by ignoring query strings in URLs and values in SQL statements. It saves their count, average/min/max time of completion and failure count in a database, then displays stats and graphs in a simple Web application that queries that database. The beauty of this approach is that you can see metrics related to specific portions of your application aggregated across all users hitting that functionality. So if a user profile is saved via a POST to a URL with a specific query string and HTTP payload, SAM will aggregate all such requests and will allow you to see exactly how many users have saved their profiles in the last hour, how many failures you had, how long it took, etc. What's more, you can query the database directly and take action based on certain thresholds (for example if the failure rate for a specific action, e.g. a POST to a specific URL, has surpassed a given threshold). We're doing exactly this within our restish-based Web service monitoring tool.

Note that this technique based on aggregation can be implemented using other approaches, for example by mining your Web server logs, or database logs. The nice thing about SAM is that it captures all this information in real time. As far as your application is concerned, it only interacts with memcached using standard memcache client calls, and it doesn't even bother to check the result of the memcache push.

Another advantage of having such a macro/business-type of monitoring and graphing in place is that your business stakeholders (including your CEO) can see these metrics and trends almost in real time, and can alert you (the ops team) if something seems out of whack. This is almost guaranteed to correlate with something going amiss in your infrastructure. Once you figure out what that is, see if your internal system monitoring system caught it or not. If it didn't, it's a great opportunity for you to add it in so as to know about it next time before your CEO does. Trust me, these things can happen, I speak from experience.

End-to-end application monitoring

Companies such as BrowserMob and Sauce Labs allow you to run Selenium scripts against your Web application using clients hosted by them 'in the cloud'. With BrowserMob, you can also use these scripts to monitor a given path through your application. This is obviously the monitoring equivalent of an end-to-end application test. If you have an e-commerce site for example, you can test and monitor your entire check out process this way. I am still scratching the surface when it comes to this type of monitoring, but it's high on my TODO list.

So there you have it. Similar to having a sound application testing strategy, your best bet is to implement all these types of monitoring, or as many of them as your time and budget allow. If you do, you can truly say you have a holistic monitoring strategy in place. And remember, the more dashboards you have into the health of your application and your systems, the better off you are. It's an uncontested rule that the one with the most dashboards wins.

For further reading and more dashboard ideas, see these resources:

• Flickr's John Allspaw on 'Context and operational metrics'
• Mike Brittain on 'Two Munin graphs for monitoring our code'
• WebOps Visualization page on Flickr

Bug in tornado-0.2

If you're using the latest packaged tornado distribution, tornado-0.2.tar.gz, then you might hit on the following bug when using AsyncHTTPClient:

ERROR:root:Exception in callback >
Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/tornado/ioloop.py", line 238, in _run_callback
    callback()
  File "/usr/local/lib/python2.6/dist-packages/tornado/httpclient.py", line 214, in _perform
    self.io_loop.remove_handler(fd)
  File "/usr/local/lib/python2.6/dist-packages/tornado/ioloop.py", line 133, in remove_handler
    self._impl.unregister(fd)
IOError: [Errno 2] No such file or directory


For me, this was in the context of using a load testing script written by my colleague Dan Mesh. The script makes use of AsyncHTTPClient and runs several such objects in parallel, hitting a number of URLs. When the number of concurrent HTTP requests reaches around 100, I start seeing this error.

Solution? Install tornado from the latest source code on GitHub.

Twittering

I've been holding back from using Twitter, which I generally still consider a waste of time. However, Bryan Landers convinced me that it's a good idea to tweet occasionally, for example when I post to my blog. Apparently many people (Bryan included) are past using RSS feed readers and other such antiquated tools that used to be popular in the first decade of this millennium. Also, there have been discussions within the SoCal Piggies group about increasing the communication and collaboration within the group, and with the Python community at large.

So...I created 2 twitter accounts, one for the SoCal Piggies (@socalpiggies) and one for myself (@griggheo). I'll generally post links to stuff I come across that might be of interest to people who share my interests.

Ian Bicking does it again with toppcloud

I haven't seen this on Planet Python yet, so I'll give it a shout out in the hope that more people will benefit from it. Via Ben Bangert, I heard yesterday about Ian Bicking's new brainchild, toppcloud -- a tool for automated deployments of cloud instances AND Python web apps. It's based on libcloud, and it currently only supports Rackspace's cloud, but it seems very promising already. The Python deployment seems to be inspired by Google AppEngine's deployment philosophy. I haven't played with it yet, but I'll do so soon and report back here. I've been using Fabric (and Puppet) successfully for automated application deployments, but toppcloud seems to be a very valuable tool to add to my arsenal.

Deploying Tornado in production

We've been using Tornado at Evite very successfully for a while, for a subset of functionality on our web site. While the instructions in the official documentation make it easy to get started with Tornado and get an application up and running, they don't go a long way towards explaining issues that you face when deploying to a production environment -- things such as running the Tornado processes as daemons, logging, automated deployments. Here are some ways we found to solve these issues.

Running Tornado processes as Unix daemons

When we started experimenting with Tornado, we ran the Tornado processes via nohup. It did the job, but it was neither solid nor elegant. I ended up using the grizzled Python library, specifically the os module, which encapsulates best practices in Unix system programming. This module offers a function called daemonize, which converts the calling process into a Unix daemon. The function's docstring says it all:

Convert the calling process into a daemon. To make the current Python
process into a daemon process, you need two lines of code:

from grizzled.os import daemon

daemon.daemonize()

I combined this with standard library's os.execve, which replaces the current (already daemonized) process with another program -- in my case with the Tornado process.

I'll show some code in a second, but first I also want to mention...

Logging

We use the standard Python logging module and send the log output to stdout/stderr. However, we wanted to also rotate the log files using the rotatelogs utility, so we looked for a way to pipe stdout/stderr to the rotatelogs binary, while also daemonizing the Tornado process.

Here's what I came up with (the stdout/stderr redirection was inspired by this note on StackOverflow):

from socket import gethostname
from grizzled.os import daemonize
PYTHON_BINARY = "python2.6"
PATH_TO_PYTHON_BINARY = "/usr/bin/%s" % PYTHON_BINARY
ROTATELOGS_CMD = "/usr/sbin/rotatelogs"
LOGDIR = "/opt/tornado/logs"
LOGDURATION = 86400

logdir = LOGDIR
logger = ROTATELOGS_CMD
hostname = gethostname()
# service is the name of the Python module pointing to your Tornado web server
# for example myapp.web
execve_args = [PYTHON_BINARY, "-m", service]
logfile = "%s_%s_log.%%Y-%%m-%%d" % (service, hostname)
pidfile = "%s/%s.pid" % (logdir, service)
logpipe ="%s %s/%s %d" % (logger, logdir, logfile, LOGDURATION)
execve_path = PATH_TO_PYTHON_BINARY

# open the pipe to ROTATELOGS
so = se = os.popen(logpipe, 'w')

# re-open stdout without buffering
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

# redirect stdout and stderr to the log file opened above
os.dup2(so.fileno(), sys.stdout.fileno())
os.dup2(se.fileno(), sys.stderr.fileno()) 

# daemonize the calling process and replace it with the Tornado process
daemonize(no_close=True, pidfile=pidfile)
os.execv(execve_path, execve_args)

The net result of all this is that our Tornado processes run as daemons, and the logging output is captured in files managed by the rotatelogs utility. Note that it is easy to switch out rotatelogs and use scribe instead (which is something we'll do very soon).

Automated deployments

I already wrote about our use of Fabric for automated deployments. We use Fabric to deploy Python eggs containing the Tornado code to each production server in turn. Each server runs N Tornado processes, and we run nginx to load balance between all M x N tornados (M servers with N processes each).

Here's how easy it is to run our Fabric-based deployment scripts:

fab -f fab_myapp.py nginx disable_tornado_in_lb:web1
fab -f fab_myapp.py web1 deploy
# run automated tests, then:
fab -f fab_myapp.py nginx enable_tornado_in_lb:web1

We first disable web1 in the nginx configuration file (as detailed here), then we deploy the egg to web1, then we run a battery of tests against web1 to make sure things look good, and finally we re-enable web1 in nginx. Rinse and repeat for all the other production web servers.

NetApp SNMP monitoring with Nagios

Here are some tips regarding the monitoring of NetApp filers with Nagios. First off, the Nagios Exchange includes many NetApp-specific monitoring scripts, all based on SNMP. I ended up using check_netapp3.pl, but I hit some roadblocks when it came to checking disk space on NetApp volumes (the SNMP queries were timing out in that case).

The check_netapp3.pl script works fine for things such as CPU load. For example, I created a new command called check_netapp_cpu in /usr/local/nagios/etc/objects/commands.cfg on my Nagios server:

define command {
command_name check_netapp_cpu
command_line $USER1$/check_netapp3.pl -H $HOSTADDRESS$ -C mycommunity -v CPULOAD -w 50 -c 80
}

However, for things such as percent of disk used for a given NetApp volume, I had to use good old SNMP checks directly against the NetApp. Any time you use SNMP, you need to know which OIDs to hit. In this case, the task is a bit easier because you can look inside the check_netapp3.pl script to see some example of NetApp-specific OIDs. But let's assume you have no clue where to start. Here's a step-by-step procedure:

1) Find the NetApp MIB -- I found one online here.

2) Do an snmpwalk against the top-level OID, which in this case is 1.3.6.1.4.1.789. Save the output in a file.
Example: snmpwalk -v 1 -c mycommunity IP_OF_NETAPP_FILER 1.3.6.1.4.1.789 > myfiler.out

3) Search for a volume name that you know about in myfiler.out. I searched for /vol/vol0 and found this line:
SNMPv2-SMI::enterprises.789.1.5.4.1.2.5 = STRING: "/vol/vol0/"
This will give you a clue as to the OID range that corresponds to volume information. If you search for "1.5.4.1.2" in the NetApp MIB, you'll see that it corresponds to dfTable.dfEntry.dfFileSys. So the entries
1.5.4.1.2.1 through 1.5.4.1.2.N will show the N file systems available on that particular filer.

4) I was interested in percentage of disk used on those volumes, so I found the variable dfPerCentKBytesCapacity in the MIB, corresponding to the OID 1.3.6.1.4.1.789.1.5.4.1.6. This means that for /vol/vol0 (which is the 5th entry in my file system table), I need to use 1.3.6.1.4.1.789.1.5.4.1.6.5 to get the percentage of disk used.

So, to put all this detective work together, it's easy to create specific commands that query a particular filer for the percentage disk used for a particular volume. Here's an example that uses the check_snmp Nagios plugin:

define command {
  command_name check_netapp_percent_diskused_myfiler_vol0
  command_line $USER1$/check_snmp -H $HOSTADDRESS$ -C mycommunity -o .1.3.6.1.4.1.789.1.5.4.1.6.3 -w 75 -c 90
}

Then I defined a service corresponding to that filer, similar to this:

define service{
use active-service
host_name myfiler
check_command check_netapp_percent_diskused_myfiler_vol0
service_description PERCENT DISK USED VOL0
is_volatile 0
check_period 24x7
max_check_attempts 3
normal_check_interval 5
retry_check_interval 1
contact_groups admins
notification_interval 1440
notification_period 24x7
notification_options w,c,r
}

Hope this helps somebody out there!

Compiling Python 2.6 with sqlite3 support

Quick note to self, hopefully useful to others too:

If you compile Python 2.6 (or 2.5) from source, and you want to enable sqlite3 support (which is included in the stdlib for 2.5 and above), then you need to pass a special USE flag to the configuration command line, like this:

./configure USE="sqlite"

(note "sqlite" and not "sqlite3")

5 years of blogging

Today marks the 5th anniversary of my blog. It's been a fun and rewarding experience, and I hope to never run out of interesting topics to post about ;-)

As a sort of retrospective, I was curious to see which of my blog posts have been getting the most traffic. Here's the top 10 over the last 9 months, according to Google Analytics:

1. Performance vs. load vs. stress testing (as an aside, I think this has been wildly popular because I inadvertently hit on a lot of keywords in the title)
2. Experiences deploying a large-scale infrastructure in Amazon EC2
3. Ajax testing with Selenium using waitForCondition
4. Useful tools for writing Selenium tests
5. Load balancing in EC2 with HAProxy
6. Python unit testing part 1: the unittest module
7. HTTP performance testing with httperf, autobench and openload
8. Running a Python script as a Windows service
9. Apache virtual hosting with Tomcat and mod_jk
10. Configuring Apache 2 and Tomcat 5.5 with mod_jk

It's interesting that 2 of the top 5 posts are Selenium-related. I think Selenium documentation is not where it needs to be generally speaking, hence people find my old posts on this topic. Adam, you really need to write a Selenium RC book!

Monitoring multiple MySQL instances with Munin

I've been using Munin for its resource graphing capabilities. I especially like the fact that you can group servers together and watch a common metric (let's say system load) across all servers in a group -- something that is hard to achieve with other similar tools such as Cacti and Ganglia.

I did have the need to monitor multiple MySQL instances running on the same server. I am using mysql-sandbox to launch and manage these instances. I haven't found any pointers on how to use Munin to monitor several MySQL instances, so I rolled my own solution.

First of all, here is my scenario:

• server running Ubuntu 9.04 64-bit
  
• N+1 MySQL instances installed as sandboxes rooted in /mysql/m0, /mysql/m1,..., /mysql/mN
• munin-node package version 1.2.6-8ubuntu3 (installed via 'apt-get install munin-node')
  

Step 1

Locate mysql_* plugins already installed by the munin-node package in /usr/share/munin/plugins. I have 5 such plugins: mysql_bytes, mysql_isam_space_, mysql_queries, mysql_slowqueries and mysql_threads. I don't use ISAM, so I am ignoring mysql_isam_space_.


Step 2

Make a copy of each plugin for each MySQL instance you want to monitor. I know this contradicts the DRY principle, but I just wanted something quick that worked. The alternative is to modify the plugins and add extra parameters so they refer to specific MySQL instances.

For example, I made N + 1 copies of mysql_bytes and called them mysql_m0_bytes, mysql_m1_bytes,..., mysql_mN_bytes. In each copy, I modified the line "echo 'graph_title MySQL throughput'" to say "echo 'graph_title MySQL throughput for mN'". I did the same for mysql_threads, mysql_queries and mysql_slowqueries. So at the end of this step I have 4 x (N+1) new plugins in /usr/share/munin/plugins.


As I said, the alternative is to modify for example mysql_bytes and add new parameters, e.g. a parameter for the title of the graph. However, I don't know exactly how the plugin is called from within Munin, and I don't want to fiddle with the number and order of parameters it's called with -- which is why I chose the easy way out.

Step 3

Create symlinks in /etc/munin/plugins to the newly created plugins. Example:

ln -s /usr/share/munin/plugins/mysql_m0_bytes /etc/munin/plugins/mysql_m0_bytes

(and similar for all the other plugins).


Step 4

Specify the path to msyqladmin for the newly defined plugins. You do this by editing the plugin configuration file /etc/munin/plugin-conf.d/munin-node.

Here's what I have in this file related to MySQL:

[mysql_m0*]
user m0
env.mysqladmin /mysql/m0/my sqladmin

[mysql_m1*]
user m1
env.mysqladmin /mysql/m1/my sqladmin

[mysql_m2*]
user m2
env.mysqladmin /mysql/m2/my sqladmin

[mysql_m3*]
user m3
env.mysqladmin /mysql/m3/my sqladmin

What the above lines say is that for each class of plugins starting with mysql_mN, I want to use the mysqladmin utility for that particular MySQL instance. The way mysql-sandbox works, mysqladmin is actually available per instance as "/mysql/mN/my sqladmin".

Note that the naming convention is important. The syntax of the munin-node plugin configuration file says that the plugin name "May include one wildcard ('*') at the start or end of the plugin-name, but not both, and not in the middle." Trust me, I haven't read this fine print initially, and I named my new plugins something like mysql_bytes_mN, then tried to configure the plugins as mysql_*mN. Pulling hair time ensued.

Step 5

Restart munin-node via 'service munin-node restart'. At this point you're supposed to see the new graphs under the Mysql link corresponding to the munin node where you set all this up. You should see N+1 graphs for each type of plugin (mysql_bytes, mysql_threads, mysql_queries and mysql_slowqueries). The graphs can be easily differentiated by their titles, e.g. 'MySQL throughput for m0' or 'MySQL queries for m1', etc.

One other quick tip: if you want to easily group nodes together, come up with some domain name which doesn't need to correspond to a real DNS domain name. For example, I called my MySQL servers something like mysqlN.myproject.mydomain.com in /etc/munin/munin.conf on the Munin server side. This allows me to see the myproject.mydomain.com group at a glance, with all the metrics for the nodes in that group shown side by side.

Here's how I defined each node in munin.conf:

[mysqlN.myproject.mydomain.com]
 address 192.168.0.N
 use_node_name yes

(where N is 1, 2, etc)

Behaviour Driven Infrastructure

I just read a post by Matthew Flanagan on Behaviour Driven Infrastructure or BDI, a concept that apparently originates with Martin Englund's post on this topic. The idea is that you describe what you need your system to do in natural language, using for example a tool such as Cucumber. What's more, you can then use the cucumber-nagios plugin to express the desired behaviour of the new system as a series of Nagios checks. The checks will initiall fail (just like in a TDD or BDD development cycle), but you will make them pass by deploying the appropriate packages and applications to the system.

I also expressed the need for automated testing of production deployments in one of my blog posts. However, BDI goes one step further, by describing a test plan for production deployments in natural language. Pretty cool, and again I can only wish that the Python testing tools kept up with Ruby-based tools such as Cucumber and friends....

Great series of posts on Tokyo Tyrant

Matt Yonkovit has started a series of posts on Tokyo Tyrant at Percona's MySQL Performance Blog. Great in-depth analysis of the reliability and performance of TT.

Part 1: Tokyo Tyrant -- is it durable?
Part 2: Tokyo Tyrant -- the performance wall
Part 3: Tokyo Tyrant -- write bottleneck

(parts 4 and 5, about replication and scaling, are hopefully coming soon)

Google using buildbot for Chromium continuous integration

Via Ben Bangert, this gem of a page showing the continuous integration status for the Chromium project at Google. It's cool to see that they're using buildbot. But just like Ben says -- I wish they open sourced the look and feel of that buildbot status page ;-)

NFS troubleshooting with iostat and lsof

Scenario: you mount a volume exported from a NetApp on several Linux clients via NFS

Problem: you see constant high CPU usage on the NetApp, and some of the Linux clients become sluggish, primarily in terms of I/O

Troubleshooting steps:

1) If iostat is not already on the clients, install the sysstat utilities.

2) On each client mounting from the filer, or on a representative sample of the clients, run iostat with -n so that it shows NFS-related statistics. The following
command will run iostat every 5 seconds and show NFS stats in a nicely tabulated output:

# iostat -nh 5

3) Notice which client exhibits the most NFS operations per second, and correlate it with the NFS volume on that client which shows the most NFS reads and/or writes per second.

At this point you found the most likely culprit in terms of sending NFS traffic to the filer (there could be several client machines in this position, for example if they are part of a cluster).

5) If not already installed, download and install lsof.

6) Run lsof on the client(s) discovered in step 4, and grep for the directory representing the mount point of the NFS volume with the most reads and/or writes. For example:

# lsof | grep /var/log

This will show you, among other things, which processes are accessing which files under that directory. Usually something will jump out at you in terms of things that are going on outside of the ordinary. In my case, it was logrotate kicking off from a daily cron and compressing a huge log file -- since the log file was on a volume NFS-mounted from the filer, this caused the filer to do extra work, hence its increased CPU usage.

That's about it. Of courser these steps can be refined/modified/added to -- but even in this simple form, they can help you pinpoint NFS issues fairly quickly.