Setting up a Pybots buildslave

If you're interested in setting up a buildbot buildslave for the Pybots project, here are some instructions:

Step 1

Install buildbot on your machine. Instructions can be found here, here, here and here.

Step 2

Create a user that will run the buildbot slave process. Let's call it buildslave, with a home directory of /home/buildslave. Also create a /home/buildslave/pybot directory.

Step 3

Create the file buildbot.tac in /home/buildslave/pybot, with content similar to this:

from twisted.application import service
from buildbot.slave.bot import BuildSlave

# set your basedir appropriately
basedir = r'/home/buildslave/pybot'
host = 'www.python.org'
port = 9070
slavename = 'abc'
passwd = 'xyz'
keepalive = 600
usepty = 1

application = service.Application('buildslave')
s = BuildSlave(host, port, slavename, passwd, basedir, keepalive, usepty)
s.setServiceParent(application)

Step 4

Create a python-tool directory under /home/buildslave/pybots. You must name this directory python-tool, as the buildmaster will use this name in the build steps.

Step 5

Create a file called run_tests.py under the python-tool directory. This is where you will invoke the automated tests for your projects.

How this all works

The buildmaster will have your buildslave execute the following steps, every time a check-in is made into the python subversion trunk (and also every time a check-in is made in the 2.5 branch):

1. Update step: runs "svn update" from the python svn trunk
2. Configure step: runs "./configure --prefix=/tmp/python-buildbot/local"
3. Make step: runs "make all"
4. Test step: runs "make test" (note: this step runs the python unit tests, not your project's unit tests)
5. Make install step: runs "make install"; this will install the newly-built python binary in /tmp/python-buildbot/local/bin/python
6. Project-specific tests step: this is when your run_tests.py file will be run via "/tmp/python-buildbot/local/bin/python ../../python-tool/run_tests.py"
7. Clean step: runs "make distclean"

Important note: since your automated tests will be run via the newly-built python binary installed in /tmp/python-buildbot/local/bin/python, you need to make sure you install all the pre-requisite packages for your package using this custom python binary, otherwise your unit tests will fail because they will not find these pre-requisites. For example, for the Twisted unit tests, I had to install setuptools, ZopeInterface, pycrypto and pyOpenSSL, before I could actually run the Twisted test suite.

So in my run_tests.py file I first call a prepare_packages.sh shell script, before I launch the actual test suite (I copied the pre-requisite packages in /home/buildslave):

$ cat prepare_packages.sh

#!/bin/bash

cd /tmp

rm -rf setuptools*
cp ~/setuptools-0.6c1.zip .
unzip setuptools-0.6c1.zip
cd setuptools-0.6c1
/tmp/python-buildbot/local/bin/python setup.py install
cd ..

rm -rf ZopeInterface*
cp ~/ZopeInterface-3.1.0c1.tgz .
tar xvfz ZopeInterface-3.1.0c1.tgz
cd ZopeInterface-3.1.0c1
/tmp/python-buildbot/local/bin/python setup.py install
cd ..

rm -rf pycrypto-2.0.1*
cp ~/pycrypto-2.0.1.tar.gz .
tar xvfz pycrypto-2.0.1.tar.gz
cd pycrypto-2.0.1
/tmp/python-buildbot/local/bin/python setup.py install
cd ..

rm -rf pyOpenSSL-0.6*
cp ~/pyOpenSSL-0.6.tar.gz .
tar xvfz pyOpenSSL-0.6.tar.gz
cd pyOpenSSL-0.6
/tmp/python-buildbot/local/bin/python setup.py install
cd ..

rm -rf Twisted
svn co svn://svn.twistedmatrix.com/svn/Twisted/trunk Twisted

Then I call the actual Twisted test suite, via:

/tmp/python-buildbot/local/bin/python -Wall /tmp/Twisted/bin/trial --reporter=bwverbose --random=0 twisted

You can see the current Pybots status page here.

If you are interested in setting up your own buildslave to participate in the Pybots project, please send a message to the Pybots mailing list. I will send you a slavename and a password, and then we can test the integration of your buildslave with the buildmaster.

Update 10/16/09

I realized that these instructions for setting up a Pybot buildslave are a bit outdated. Discussions on the Pybots mailing list prompted certain changes to run_tests.py, even though you're still OK if you follow the instructions above to the letter.

Here are some enhancements that you can take advantage of:

1. You can test several projects, each in its own build step, simply by having your run_tests.py script be aware of an extra command-line argument, which will be the name of the project under tests. An example of such a script is here: run_tests.py. The script receives a command-line argument (let's call it proj_name) and invokes a shell script called proj_name.sh. The shell script checks out the latest code for project proj_name (or downloads the latest distribution), then runs its unit tests. Here is an example: Cheesecake.sh.

2. You do not have to hardcode the path to the newly built Python binary in your run_tests.py or your shell scripts. You can simply retrieve the path to the binary via sys.executable. This run_tests.py script sets an environment variable called PYTHON via a call to

os.putenv('PYTHON', sys.executable)

Then the variable is used as $PYTHON in the shell scripts invoked by run_tests.py (thanks to Elliot Murphy and Seo Sanghyeon for coming up with this solution.)