Another entertaining blog post from Joel Spolsky, this time on some issues they had with servers and networking equipment hosted at a data center in Manhattan. It all comes down to a network switch which had its ports configured to automatically negotiate their speed. As a result, one port was misbehaving and brough their whole web site down. The conclusion reached by Joel and his sysadmin team: we need documentation, we need checklists. I concur, but as I said in a recent post, this is still not enough. Human beings are notoriously prone to skipping tests on checklists. What Joel and his team really need are AUTOMATED TESTS that run periodically and check every single thing on those checklists. You can easily automate the step which verifies that a port on the switch is set to 100 Mbps or 1 Gbps; you can either use SNMP, or some expect-like script.
In fact, at my own company I'm developing a pretty extensive automated test suite (written in Python of course, and using nose) that verifies all the steps we go through whenever we deploy a server or a network device. It's very satisfying to see those dots and have a total of N passed tests and 0 failed tests, with N increasing daily. Automated tests for sysadmin tasks is an area little explored, so there's lots of potential for cool stuff to happen. If you're doing something similar and have ideas to share, please leave a comment.
Subscribe to:
Post Comments (Atom)
-
Here's a good interview question for a tester: how do you define performance/load/stress testing? Many times people use these terms inte...
-
Update 02/26/07 -------- The link to the old httperf page wasn't working anymore. I updated it and pointed it to the new page at HP. Her...
-
I've been using dnspython lately for transferring some DNS zone files from one name server to another. I found the package extremely us...
9 comments:
I've starting using nose and twill to verify our various websites (some static, some dynamic) are up and running properly after web server configuration changes. This includes checking that passwords are requested where expected, and that important links work.
Checklists can be ignored, but generally the known existence of a checklist is enough to improve quality. If you can automate it, great, do so. Many things cannot be automated (with current technology), but a human can still look at the checklist.
Note that I specified known. Checklists that nobody knows about are worthless. A checklist that people know about will improve quality.
My dad used to do data center support for one of the major banks up here. They had entire binders of procedures on how to install a new banking platform onto the as/400s. I recall coming home one night (well, morning ;)) to have him sitting on the phone to some island trying to recover their database. His comment to the person at the time was something along the lines of "Yes, you were supposed to follow the stuff on page 12. That's why why put page 12 in there. It's not as if we had to keep pages 11 and 13 from each other!" :)
Henry -- that has not been my experience. I was fully aware of the existence of a checklist for deploying a complex Web application, but that didn't stop me from sometimes skipping important steps by mistake.
Grig
Adam -- I sympathize with your dad....Great story, thanks for sharing it!
Grig
Stan -- thanks for sharing your scenario. nose and twill are a great combination!
Grig
"Automated tests for sysadmin tasks", I'd really like to read an article or tutorial on this.
Our sys admin writes tests, and runs them continuously, using Zabbix.
Great stuff--it pushes notifications, makes nice graphs for system dashboards, etc.
Stephen -- having a monitoring system is necessary, but sometimes not sufficient. Let's say you have a checklist for deploying a LAMP server, or a Apache/Tomcat server. Your monitoring system will be hard pressed to verify that all the steps in the checklist have been applied. Examples:
* is httpd set to start at boot time?
* are the tomcat directories owned by the correct user?
* does the sudoers file contain certain user IDs?
All these things can be tested automatically with a real scripting language, using a test framework.
Grig
Post a Comment