Pybots success stories and a call for help

Any report of the death of the Pybots project is an exaggeration. But not by much. First, some history.

Some history

The idea behind the Pybots project is to allow people to run automated tests for their Python projects, while using Python binaries built from the very latest source code from the Python subversion repository.

The idea originated from Glyph, of Twisted fame. He sent out a message to the python-dev mailing list in which he said:

"I would like to propose, although I certainly don't have time to implement, a program by which Python-using projects could contribute buildslaves which would run their projects' tests with the latest Python trunk. This would provide two useful incentives: Python code would gain a reputation as generally well-tested (since there is a direct incentive to write tests for your project: get notified when core python changes might break it), and the core developers would have instant feedback when a "small" change breaks more code than it was expected to."

This was back in July 2006. I volunteered to maintain a buildbot master (running on a server belonging to the PSF) and also to rally a community of people interested in running this type of tests. The hard part was (and still is) to find people willing to donate client machines to act as build slaves for a particular project, and even more so people willing to keep up with the status of their build slaves. The danger here, as in any continuous integratin system, is that once the status turns to red and doesn't go back to green, people start to ignore the failed steps. Even if those steps exhibit new and interesting failures, it's too late at this point (this is related to the broken windows theory).

The project starting fairly strong, gained some momentum, but then slowly ran out of steam. It was a combination of me not having the time to do the rallying, and of people not being interested in participating in the project anymore. At the height of its momentum, in early 2007, the Pybots farm consisted of 11 buildslaves running automated tests for more than 20 Python projects, including Twisted, Django, SQLAlchemy, MySQLdb, Bazaar, nose, twill, Storm, Trac, CherryPy, Genshi, Roundup. Pretty much a who's who of the Python project world.

Early success stories

Here are some examples of bugs discovered by the buildslaves in the Python farm:

• new keywords 'as' and 'with' in Python 2.6 causing problems for projects that had variables with those names
• Python install step failing even though all unit tests were passing (this underscores the importance of functional testing)
• platform-specific issues -- for example Bazaar issues on Windows due to TCP client behavior, Twisted issues on Red Hat 9 due to multicast behavior, Python core issues on OS X due to string formatting errors

(for a more thorough overview of the Pybots project, including lessons learned, see also my PyCon07 presentation)

Recent signs of life and more success stories

In the last month or so there has been a flurry of activity related to the Pybots farm. It all started with an upgrade of the buildbot version on the machine hosting the Pybots buildmaster. This broke the master's configuration file, so the Pybots status page went completely dark.

As a result, Steve Holden posted a plea for help answered by a few people who showed interest in adding build slaves to the project. In parallel, Jean-Paul Calderone jumped in to help on the buildmaster side and he managed to fix the buildbot upgrade issue (thanks, JP!) David Stanek also expressed interest in taking a more active role on the buildmaster side.

Jean-Paul also sent more success stories to the Pybots mailing list. Here they are, verbatim, with his permission:

"The skip story:

The Twisted pybots slave started skipping every Twisted test one day. I noticed and filed http://twistedmatrix.com/trac/ticket/3703 (which goes into a bit of detail about why this happened). This happened to come up during the PyCon language summit, so there was some real-time discussion about it, resulting in a Python bug being filed, http://bugs.python.org/issue5571. Then, as that ticket shows, Benjamin Peterson was nice enough to fix the incompatibility.

The array/buffer story:

The Twisted pybots slave started to fail some Twisted tests one day. ;) The tests in question were actually calling into some PyCrypto code, so this failure wasn't in Twisted directly. PyCrypto loads some bytes into an array.array and then tries to hash them (for some part of its random pool API). I filed http://bugs.python.org/issue6071 on which someone explained that hashlib switched over to the new buffer API, lost support for hashing anything that only provides the old buffer API, and that array.array still only supports the old buffer API. This one hasn't been fixed yet, but it sounds like Gregory Smith plans to fix it before 2.7 is released.

There are other success stories too, incompatible changes that are more like bugs on the Twisted side than on the Python side (assuming one is generous and believes that incompatible changes in Python can actually be Twisted bugs ;). Things like typos that didn't result in syntax errors in an older version of Python but became syntax errors in newer versions (in particular, a variable was defined as 0x+80000000 instead of 0x80000000 - the former actually being valid syntax in 2.5 but became illegal in 2.6)."

My hope is that stories like these will convince more people about the usefulness of running tests for their projects against 'live' changes in the Python trunk (or other Python branches). I am not aware of any other testing project that accomplishes this for other programming languages.

In particular, if there is enough interest, we can also configure the Pybots master to trigger test runs for your project of choice using Py3k binaries! Think how cool you'll appear to your grandchildren!

How you can help

If you want to be involved in the Pybots project, please subscribe to the Pybots mailing list and show your interest by sending a message to the list. Here are some resources to get you started:

• Setting up a Pybots buildslave (scroll to the end for an update)
• Pybots FAQ
• Pybots Google Code page
  
• Examples of buildslave scripts that trigger the automated builds and tests
• Sidnei Da Silva's experiences on setting up a Win32 buildslave
  

Pybots project keeps rolling

Some updates on the Pybots project:

• Mike Taylor aka Bear from OSAF contributed 2 buildslaves: an Ubuntu Dapper box for testing his parsedatetime package, and an Intel Mac OSX box for testing two libraries used by OSAF -- vobject and zanshin
• Manuzhai contributed a Gentoo box for testing pysqlite and Trac
• Elliot Murphy from mysql.com will contribute an AMD-64 Ubuntu Dapper box for testing MySQLdb
• In summary, we're up to 5 (soon to be 6) buildslaves, with more on the way
  
• Seo Sanghyeon created a Google Code project for pybots; he and I are the current admins for this project; you can browse the Subversion repository for various scripts and buildbot config. files
• As a cool side note, Sanghyeon rewrote the home page for pybots.org in reST; the page is kept in subversion and the server hosting pybots.org is doing a svn update every hour, followed by a rest2html call

As I said in a previous post, the Pybots setup already proved its usefulness by uncovering issues with new keywords such as 'with' and 'as'. Some of the projects affected by these new keywords are zope.interface, roundup and zanshin. According to the Zope developers, the issue had been fixed a while ago in the svn repository, but no release has happened since. The roundup developers already fixed the issue -- nice to see this.

Also, the Trac unit tests, when running against the latest Python trunk, are failing with an ugly backtrace. If somebody can shed some light, please do.

Again, if you're interested in running a Pybots buildslave, take a look at the various pieces of documentation at pybots.org and send a message to the Pybots mailing list.

Pay attention to the new 'with' and 'as' keywords

As of Sept. 6th (revision 51767), Python 2.6 has two new keywords: with and as. Python code that uses either one of these words as a variable name will be in trouble. How do I know that? Because the Twisted unit tests have been failing in the Twisted Pybots buildslave ever since. Actually the issue is not with Twisted code, but with zope.interface, which is one of the Twisted pre-requisites. Here's the offending code:

Traceback (most recent call last):
File "/tmp/Twisted/bin/trial", line 23, in 
from twisted.scripts.trial import run
File "/tmp/Twisted/twisted/scripts/trial.py", line 10, in 
from twisted.application import app
File "/tmp/Twisted/twisted/application/app.py", line 10, in 
from twisted.application import service
File "/tmp/Twisted/twisted/application/service.py", line 20, in 
from twisted.python import components
File "/tmp/Twisted/twisted/python/components.py", line 37, in 
from zope.interface.adapter import AdapterRegistry
File "/tmp/python-buildbot/local/lib/python2.6/site-packages/zope/interface/adapter.py", line 201
for with, objects in v.iteritems():
       ^
SyntaxError: invalid syntax

It would be great if the Zope folks fixed their code so that the Twisted tests will start passing again. This issue actually impacts all packages that depend on zope.interface -- for example zanshin, which also fails in the Pybots buildslave for the OSAF libraries (graciously set up by Bear from OSAF).

If you're interested in following such issues as they arise in the Pybots buildbot farm, I encourage you to subscribe to the Pybots mailing list.

Update

I mentioned the 'with' keyword already causing problems. As it turns out, Seo Sanghyeon's buildslave, which is running tests for docutils and roundup, uncovered an issue in roundup, related to the 'as' keyword:

Traceback (most recent call last):
 File "run_tests.py", line 889, in 
   process_args()
 File "run_tests.py", line 879, in process_args
   bad = main(module_filter, test_filter, libdir)
 File "run_tests.py", line 671, in main
   runner(files, test_filter, debug)
 File "run_tests.py", line 585, in runner
   s = get_suite(file)
 File "run_tests.py", line 497, in get_suite
   mod = package_import(modname)
 File "run_tests.py", line 489, in package_import
   mod = __import__(modname)
 File "/home/buildslave/pybots/roundup/./test/test_actions.py", line 6, in 
   from roundup import hyperdb
 File "/home/buildslave/pybots/roundup/roundup/hyperdb.py", line 29, in 
   import date, password
 File "/home/buildslave/pybots/roundup/roundup/date.py", line 735
   as = a[0]
    ^
SyntaxError: invalid syntax

Setting up a Pybots buildslave

If you're interested in setting up a buildbot buildslave for the Pybots project, here are some instructions:

Step 1

Install buildbot on your machine. Instructions can be found here, here, here and here.

Step 2

Create a user that will run the buildbot slave process. Let's call it buildslave, with a home directory of /home/buildslave. Also create a /home/buildslave/pybot directory.

Step 3

Create the file buildbot.tac in /home/buildslave/pybot, with content similar to this:

from twisted.application import service
from buildbot.slave.bot import BuildSlave

# set your basedir appropriately
basedir = r'/home/buildslave/pybot'
host = 'www.python.org'
port = 9070
slavename = 'abc'
passwd = 'xyz'
keepalive = 600
usepty = 1

application = service.Application('buildslave')
s = BuildSlave(host, port, slavename, passwd, basedir, keepalive, usepty)
s.setServiceParent(application)

Step 4

Create a python-tool directory under /home/buildslave/pybots. You must name this directory python-tool, as the buildmaster will use this name in the build steps.

Step 5

Create a file called run_tests.py under the python-tool directory. This is where you will invoke the automated tests for your projects.

How this all works

The buildmaster will have your buildslave execute the following steps, every time a check-in is made into the python subversion trunk (and also every time a check-in is made in the 2.5 branch):

1. Update step: runs "svn update" from the python svn trunk
2. Configure step: runs "./configure --prefix=/tmp/python-buildbot/local"
3. Make step: runs "make all"
4. Test step: runs "make test" (note: this step runs the python unit tests, not your project's unit tests)
5. Make install step: runs "make install"; this will install the newly-built python binary in /tmp/python-buildbot/local/bin/python
6. Project-specific tests step: this is when your run_tests.py file will be run via "/tmp/python-buildbot/local/bin/python ../../python-tool/run_tests.py"
7. Clean step: runs "make distclean"

Important note: since your automated tests will be run via the newly-built python binary installed in /tmp/python-buildbot/local/bin/python, you need to make sure you install all the pre-requisite packages for your package using this custom python binary, otherwise your unit tests will fail because they will not find these pre-requisites. For example, for the Twisted unit tests, I had to install setuptools, ZopeInterface, pycrypto and pyOpenSSL, before I could actually run the Twisted test suite.

So in my run_tests.py file I first call a prepare_packages.sh shell script, before I launch the actual test suite (I copied the pre-requisite packages in /home/buildslave):

$ cat prepare_packages.sh

#!/bin/bash

cd /tmp

rm -rf setuptools*
cp ~/setuptools-0.6c1.zip .
unzip setuptools-0.6c1.zip
cd setuptools-0.6c1
/tmp/python-buildbot/local/bin/python setup.py install
cd ..

rm -rf ZopeInterface*
cp ~/ZopeInterface-3.1.0c1.tgz .
tar xvfz ZopeInterface-3.1.0c1.tgz
cd ZopeInterface-3.1.0c1
/tmp/python-buildbot/local/bin/python setup.py install
cd ..

rm -rf pycrypto-2.0.1*
cp ~/pycrypto-2.0.1.tar.gz .
tar xvfz pycrypto-2.0.1.tar.gz
cd pycrypto-2.0.1
/tmp/python-buildbot/local/bin/python setup.py install
cd ..

rm -rf pyOpenSSL-0.6*
cp ~/pyOpenSSL-0.6.tar.gz .
tar xvfz pyOpenSSL-0.6.tar.gz
cd pyOpenSSL-0.6
/tmp/python-buildbot/local/bin/python setup.py install
cd ..

rm -rf Twisted
svn co svn://svn.twistedmatrix.com/svn/Twisted/trunk Twisted

Then I call the actual Twisted test suite, via:

/tmp/python-buildbot/local/bin/python -Wall /tmp/Twisted/bin/trial --reporter=bwverbose --random=0 twisted

You can see the current Pybots status page here.

If you are interested in setting up your own buildslave to participate in the Pybots project, please send a message to the Pybots mailing list. I will send you a slavename and a password, and then we can test the integration of your buildslave with the buildmaster.

Update 10/16/09

I realized that these instructions for setting up a Pybot buildslave are a bit outdated. Discussions on the Pybots mailing list prompted certain changes to run_tests.py, even though you're still OK if you follow the instructions above to the letter.

Here are some enhancements that you can take advantage of:

1. You can test several projects, each in its own build step, simply by having your run_tests.py script be aware of an extra command-line argument, which will be the name of the project under tests. An example of such a script is here: run_tests.py. The script receives a command-line argument (let's call it proj_name) and invokes a shell script called proj_name.sh. The shell script checks out the latest code for project proj_name (or downloads the latest distribution), then runs its unit tests. Here is an example: Cheesecake.sh.

2. You do not have to hardcode the path to the newly built Python binary in your run_tests.py or your shell scripts. You can simply retrieve the path to the binary via sys.executable. This run_tests.py script sets an environment variable called PYTHON via a call to

os.putenv('PYTHON', sys.executable)

Then the variable is used as $PYTHON in the shell scripts invoked by run_tests.py (thanks to Elliot Murphy and Seo Sanghyeon for coming up with this solution.)

Any projects that need Pybots buildslaves?

Know of any projects that need Pybots buildslaves?

This is a question that's been asked twice already on the Pybots mailing list, or in emails addressed directly to me. Every time I answered that I don't really know any, and I advised the people asking the question to go through the list of their favorite Python projects, and pick some that they'd like to test. Or go through the list of major Web frameworks and pick some.

Anyway -- maybe a better way is to ask all the readers of this blog: do you have a project that you'd like to test in the Pybots buildbot farm, but maybe you don't have the hardware to run the buildslave on? Then let me know, or even better, let the Pybots mailing list know. There are people such as Jeff McNiel and Skip Montanaro who have hardware and resources, and are looking for projects to test.

BTW, the Pybots farm now has 6 buildslaves (the 6th was courtesy of Elliot Murphy, who contributed an AMD-64 Ubuntu Dapper box running the MySQLdb tests), with 3 or 4 buildslaves on the way.

Thanks a lot to all the people who have contributed or will contribute soon. It's very rewarding to see the Python community responding in an enthusiastic manner to a call to do better testing of Python projects.

Pybots update

I'm happy to report that the Pybots project got its first user other than yours truly. Seo Sanghyeon set up a buildslave running on Debian Unstable which is running the docutils unit tests every time a checkin is made into Python trunk or in the 2.5-maint branch.

Marc-Andre Lemburg also offered to run a buildslave for the egenix-mx-base tests, while Manuzhai offered to run a buildslave for the Trac tests. Skip Montanaro also expressed interest in running a buildslave, but he hasn't decided on a project yet.

You can see the current pybots buildslaves here. Expect to see more active buildslaves in the next few days.

Marc-Andre suggested I write a Pybots FAQ and put some info on the Python wiki, so here they are:

• Pybots page on Python wiki
• Pybots FAQ on Python wiki
  

PyCon news

I was very glad to see that the 3 proposals I submitted to PyCon07 were accepted: a "Testing Tools in Python" tutorial presented jointly with Titus, a "Testing Tools Panel" that I will moderate, and a talk on the Pybots project. The complete list of accepted talks and panels is here.

Here are the brief description and the outline for the Testing Tools tutorial that Titus and I will present. We will cover much more than just testing tools actually -- we'll talk about test and development techniques and strategies. It should be as good or better than the one we gave last year, which attracted a lot of people.

The Testing Tools Panel has a Wiki page. If you're interested in attending, please consider adding questions or topics of interest to you. If there is enough interest, I'm thinking about also organizing a BoF session on Testing Tools and Techniques, since the panel's duration will be only 45 minutes.

Finally, my Pybots talk will consist of an overview of the Pybots project: I will talk about the setup of the Pybots buildbot farm, about the issues that the Pybots farm has helped uncover,
and also about lessons learned in building, sustaining and growing an open-source community project.

The program for PyCon07 looks very solid, with a lot of interesting talks and tutorials. I'm very much looking forward to the 4 days I'll spend in beautiful Addison, TX :-)

Pybots news

I'm happy to report that the Pybots project continues to gain momentum. In raw numbers, we have 8 buildslaves running the automated tests for 17 projects, and also testing the installation of 18 other packages. Pretty impressive, if I may say so myself. This table is copied from the main pybots.org page and shows the current setup:

Builder name	Project(s) tested	Pre-requisites installed	Owner
x86 Red Hat 9	Twisted	setuptools, zope.interface, pycrypto, pyOpenSSL	Grig Gheorghiu
x86 Debian Unstable	docutils, roundup	N/A	Seo Sanghyeon
x86 Ubuntu Dapper	parsedatetime	setuptools	Mike Taylor (Bear)
x86 OSX	vobject, zanshin	setuptools, zope.interface, Twisted	Mike Taylor (Bear)
x86 Gentoo	pysqlite, Genshi, Trac, feedvalidator	clearsilver, pysqlite	Manuzhai
amd64 Ubuntu Dapper	MySQLdb, CherryPy	N/A	Elliot Murphy
x86 Windows 2003	lxml (dev, stable) Bazaar (dev, stable)	libxml2, libxslt, zlib, iconv	Sidnei da Silva
x86 Ubuntu Breezy	Cheesecake	setuptools, nose, logilab-astng, pylint	Grig Gheorghiu

Some more projects and buildslaves are in the pipeline, so I hope to be able to announce them soon. I'd like to thank all the contributors so far, in chronological order of their contributions: Seo Sanghyeon, Mike Taylor (Bear), Manuzhai, Elliot Murphy and Sidnei da Silva.

People interested in this project -- whether they'd like their project to be tested on an existing buildslave, or they'd like to contribute a buildslave -- are encouraged to peruse the documentation on the Pybots page, then send a message to the Pybots mailing list.

Pybots updates

After a long hibernation period, the Pybots project shows some signs of life -- I should probably say mixed with signs of death. Elliot Murphy from Canonical added the Storm ORM project to his AMD64 Ubuntu Gutsy buildslave, while Manuzhai and Jeff McNeil had to drop their buildslaves out of the mix, hopefully only temporarily. In Manuzhai's case though, the project he was testing -- Trac -- proved to have maintainers that were not interested in fixing their failing tests. In this case, there is no point in testing that project in Pybots. Hopefully Manuzhai will find a different, more test-infected project to run in Pybots.

Speaking of test-infected projects, it was nice to see unit testing topping the list of topics in the Django tutorial given at OSCON by Jeremy Dunck, Jacob Kaplan-Moss and Simon Willison. In fact, Titus is quoted too on this slide, which seems to be destined to be a classic (and I'm proud he uttered those words during the Testing Tools Panel that I moderated at PyCon07). Way to go, Titus, but I'd really like to see some T-shirts sporting that quote :-)

Pybots updates

Time for the periodical update on the Pybots project. Since my last post in July, John Hampton added a buildslave running Gentoo on x86 and testing Trac and SQLAlchemy. A belated thank you to John.

I also had to disable the tests for bzr dev on my RH 9 buildslave, because for some reason they were leaving a lot of orphaned/zombie processes around.

With help from Jean-Paul Calderone from the Twisted team, we managed to get the Twisted buildslave (running RH 9) past some annoying multicast-related failures. Jean-Paul had me add an explicit iptables rule to allow multicast traffic. The rule is:

iptables -A INPUT -j ACCEPT -d 225.0.0.0/24

This seemed to have done the trick. There are some Twisted unit tests that still fail -- some of them are apparently due to the fact that raising string exceptions is now illegal in the Python trunk (2.6). Jean-Paul will investigate and I'll report on the findings -- after all, this type of issues is exactly why we set up the Pybots farm in the first place.

As usual, I end with a plea to people interested in running Pybots buidlslaves to either send a message to the mailing list, or contact me directly at grig at gheorghiu dot net.

Pybots updates

Since my last Pybots-related blog post, a lot has happened. We added 2 buildslaves, a Sparc Solaris 10 host running the Django unit test suite (courtesy of Matthew Flanagan) and a G5 OSX host running the SQLAlchemy test suite (courtesy of Skip Montanaro). So now we have 10 buildslaves altogether. I also added more test suites to my Ubuntu Breezy buildslave. In addition to the Cheesecake test suite, I'm now running the unit test suites for the py library, nose, twill and Testoob.

Email notifications finally started to work too, after I finally figured out I was passing the wrong builder names to the MailNotification class. And we also have RSS feeds available. If you want to be notified of failures from all builders, subscribe to:

http://www.python.org/dev/buildbot/community/all/rss or
http://www.python.org/dev/buildbot/community/all/atom

To be notified of failures from the trunk builders, subscribe to:

http://www.python.org/dev/buildbot/community/trunk/rss or
http://www.python.org/dev/buildbot/community/trunk/atom

To be notified of failures from the 2.5 branch builders, subscribe to:

http://www.python.org/dev/buildbot/community/2.5/rss or
http://www.python.org/dev/buildbot/community/2.5/atom

Matthew Flanagan also added functionality that allows you to subscribe to a feed for a particular builder. For example, to subscribe to the feed for the "x86 Ubuntu Breezy trunk" builder, use this URL:

http://www.python.org/dev/buildbot/community/all/rss?show=x86%20Ubuntu%20Breezy%20trunk

If you are interested in contributing a buildslave to the Pybots project, please send a message to the Pybots mailing list, or to me (grig at gheorghiu dot net), or leave a comment here.

Pybots -- Python Community Buildbots

The idea behind the Pybots project (short for "Python Community Buildbots") is to allow people to run automated tests for their Python projects, while using Python binaries built from the very latest source code from the Python subversion repository.

The idea originated from Glyph, of Twisted fame. He sent out a message to the python-dev mailing list (thanks to John J. Lee for bringing this message to my attention), in which he said:

"I would like to propose, although I certainly don't have time to implement, a program by which Python-using projects could contribute buildslaves which would run their projects' tests with the latest Python trunk. This would provide two useful incentives: Python code would gain a reputation as generally well-tested (since there is a direct incentive to write tests for your project: get notified when core python changes might break it), and the core developers would have instant feedback when a "small" change breaks more code than it was expected to."

Well, Neal Norwitz made this happen by setting up a buildmaster process on one of the servers maintained by the PSF. He graciously allowed me to maintain this buildmaster, and I already added a buildslave which runs the Twisted unit tests (in honor of Glyph, who was the originator
of this idea) every time a check-in is made in the Python trunk. You can see the buildmaster's status page here.

Note that some of the Twisted unit tests sometimes fail for various reasons. Most of these reasons are environment-related -- for example the user that the buildbot slave runs as used to not have a login shell in /etc/passwd, and thus a specific test which was trying to run the login shell as a child process was failing. Not necessarily a Twisted bug, but still something that's nice to catch.

And this brings me to a point I want to make about running your project's automated tests in buildbot: I can almost guarantee that you will find many environment-specific issues that would otherwise remain dormant, since most likely the way you're usually running your tests is in a controlled environment that you had set up carefully some time ago. There's nothing like running the same tests under a different user's account and environment.

Of course, if you've never run your tests under a continuous integration process such as buildbot, you'll also be pleasantly -- or maybe not so pleasantly -- surprised at the amount of stuff that can get broken by one of your check-ins that you considered foolproof. This is because buildbot tirelessly checks out the very latest source code of your project, then runs your unit tests against that code. When you run your unit tests on your local machine, chances are you might not have synchronized your local copy with the repository.

This does assume that you have unit tests for your project, but since you have been reading this post this far, I assume you either do, or are interested in adding them. I strongly urge you to do so, and also to contribute to the Pybots project by setting up a buildslave for your project.

I'll post another entry with instructions on how to configure a buildslave so that it can be coordinated by the Pybots buildmaster. I also have a mailing list here (thanks, Titus!) for people who are interested in this project. Please send a message there, and I'll respond to you.

The buildmaster is currently running the build steps every time a check-in is made to the Python trunk, and to the 2.4 branch. In the near future, there will be a 2.5 branch, and the trunk will be used for 2.6 check-ins. I'll modify the buildmaster configuration to account for this.

Update: The buildmaster is now aware of changes in both the trunk and the newly created release25-maint branch. You can watch the HTML status page for all builders, or for the trunk builders.

BTW, if you need instructions on setting up buildbot, you can find some here, here, here and here.

On the importance of functional testing

I did not need further proof of the fact that functional tests are a vital piece in a project's overall testing strategy. I got that proof anyway last night, while watching the Pybots buildmaster status page. I noticed that the Twisted unit tests were failing, but not because of errors within the Twisted package, but because pre-requisite packages such as ZopeInterface could not be installed anymore. If you followed my post on setting up a Pybots buildslave, you know that before running the Twisted unit tests, I attempt to instal ZopeInterface and other packages using the newly-built python binary, via "/tmp/python-buildbot/local/bin/python setup.py install".

Well, all of a sudden last night this last command was failing with errors such as:

error: invalid Python installation:
unable to open /tmp/python-buildbot/local/lib/python2.6/config/Makefile
(No such file or directory)

This proved to be was a transient error, due to some recent checkins that modified the release numbers in the python svn trunk from 2.5 to 2.6. This issue was fixed within an hour, but the interesting thing to me was that, while this step was failing in the Pybots Twisted tests, the Python builbots running the Python-specific unit tests against the trunk were merrily chugging along, green and happy (at least on Unix platforms). This was of course to be expected, since nothing major changed as far as the internal Python unit tests were concerned. However, when running a functional test involving the newly-built Python binary -- and in my case that functional test consisted simply in running "python setup.py install" on some packages -- things started to break.

Lesson learned? Always make sure you test your application from the outside in, by exercising it as a user would. Unit tests are necessary (indeed, they are essential), but they are not sufficient by any means. A 'holistic' test strategy takes into consideration both white-box-type unit tests, and black-box-type functional tests. Of course, the recommended way of running all these types of tests is via a continuous integration tool such as buildbot.

Testing tutorial at PyCon07

Titus and I will present a tutorial on "Testing Tools in Python" at PyCon07. It is scheduled for the afternoon session on Thursday Feb.22. It will be an improved version of our "Agile Development and Testing in Python" tutorial from last year.

Here is the tutorial outline (courtesy of Titus). If you have any suggestions, please leave a comment.

Introduction
* Why test?
* What to test?
* Using testing to boost maintainability of code.

Setting up a project
* Source control management with Subversion.
* A brief introduction to using Trac for project
documentation and ticket management.
* Packaging with distutils
* Packaging with setuptools
* Registering your project with the Python Cheeseshop
* What else is out there? (distributed vs svn, roundup, ...)

Unit testing
* How to think about unit testing
* Using nose to run unit tests
* doctest-style unit tests
* What else is out there? (unittest, py.test, testosterone...)

Functional Web testing with twill
* Writing twill scripts
* Running twill scripts
* Using scotch to record actions
* Using wsgi_intercept to avoid network sockets
* What else is out there? (zope.testbrowser, mechanize, mechanoid)

Using code coverage in conjunction with unit/functional testing
* Basic code coverage with figleaf
* Monitoring code coverage in remote servers
* Combining figleaf code coverage analyses
* What else is out there (coverage)

** BREAK **

Acceptance testing with FitNesse/PyFit
* How FitNesse works
* Writing fixtures
* Running Python fixtures

Web application testing with Selenium
* How Selenium works
* Writing and recording Selenium tests
* Scripting Selenium tests remotely with SeleniumRC
* What else is out there? (Sahi, Watir)

Continuous integration with buildbot
* Introduction to buildbot
* Discussion of concepts, demonstration.
* Integrating tests into buildbot.
* GUI testing in buildbot.
* Using pybots to test your open source project

Conclusion
* Why test, revisited
* Maintainability and testing

The 90-9-1 rule and building an open source community

Jakob Nielsen talks about the 90-9-1 rule in his latest Alertbox newsletter: "Participation inequality: encouraging more users to contribute". Simply put, the rule states that in a typical online community, 90% of the users are lurkers, 9% are occasional contributors, and only 1% are active contributors. This should be interesting for people trying to build and grow open source projects. Nielsen has some suggestions to offer on how to overcome this "participation inequality". Read the article for his suggestions.

Here are some of my own observations and lessons learned from various open source efforts I've been part of (many of them are things I've tried to do on the Pybots project):

How to build an open source community

* Blog, blog, blog
* Send messages to mailing lists related to the area of your project
* Write extensive documentation, make it easy for people to join
* Create a project repository (Google Code)
* Get help from early adopters, involve them in the project

How to sustain and grow an open source community

* Blog, blog, blog
* Send messages to individuals who might be interested in contributing
* Acknowledge contributions
* Respond quickly to issues on mailing list
* Demonstrate usefulness of the project, both to contributors, and to any organizations involved (e.g. the PSF)
* Market/promote/evangelize the project tirelessly
* Recommended reading: "Fearless change: Patterns for introducing new ideas" by Mary Lynn Manns and Linda Rising

Comments about your own experience in building an open source community are much appreciated.

GK-H on "Myths, lies and truths about the Linux kernel"

"Myths, lies and truths about the Linux kernel" is the title of Greg Koah-Hartman's closing keynote at OLS 2006. Fascinating read, especially when Greg talks about the apparently chaotic Linux kernel development process, which turns out to be amazingly flexible and evolutionary. I was also impressed by the arguments for open-source drivers that are maintained and modified at the same time with the kernel -- this make a stable internal kernel API unnecessary, and allows the kernel to evolve.

Titus sent me the link to the keynote, and he also underlined this paragraph related to testing:

"Now, this is true, it would be great to have a simple set of tests that everyone could run for every release to ensure that nothing was broken and that everything's just right. But unfortunately, we don't have such a test suite just yet. The only set of real tests we have, is for everyone to run the kernel on their machines, and to let us know if it works for them."

It's pretty amazing to me that the Linux kernel manages to be that stable without a regression test suite. Imagine how much better it would be with such a regression suite. Clearly, a community project waiting to happen.

In the mean time, back to my pybots.... also waiting for that to happen -- but at least it's a bit more realistic from my perspective, and I hope to be able to give you more details as the PSF is working on getting a server to configure the buildmaster on.