Thursday, August 17, 2006

Setting up a Pybots buildslave

If you're interested in setting up a buildbot buildslave for the Pybots project, here are some instructions:

Step 1

Install buildbot on your machine. Instructions can be found here, here, here and here.

Step 2

Create a user that will run the buildbot slave process. Let's call it buildslave, with a home directory of /home/buildslave. Also create a /home/buildslave/pybot directory.

Step 3

Create the file buildbot.tac in /home/buildslave/pybot, with content similar to this:

from twisted.application import service
from import BuildSlave

# set your basedir appropriately
basedir = r'/home/buildslave/pybot'
host = ''
port = 9070
slavename = 'abc'
passwd = 'xyz'
keepalive = 600
usepty = 1

application = service.Application('buildslave')
s = BuildSlave(host, port, slavename, passwd, basedir, keepalive, usepty)

Step 4

Create a python-tool directory under /home/buildslave/pybots. You must name this directory python-tool, as the buildmaster will use this name in the build steps.

Step 5

Create a file called under the python-tool directory. This is where you will invoke the automated tests for your projects.

How this all works

The buildmaster will have your buildslave execute the following steps, every time a check-in is made into the python subversion trunk (and also every time a check-in is made in the 2.5 branch):

1. Update step: runs "svn update" from the python svn trunk
2. Configure step: runs "./configure --prefix=/tmp/python-buildbot/local"
3. Make step: runs "make all"
4. Test step: runs "make test" (note: this step runs the python unit tests, not your project's unit tests)
5. Make install step: runs "make install"; this will install the newly-built python binary in /tmp/python-buildbot/local/bin/python
6. Project-specific tests step: this is when your file will be run via "/tmp/python-buildbot/local/bin/python ../../python-tool/"
7. Clean step: runs "make distclean"

Important note: since your automated tests will be run via the newly-built python binary installed in /tmp/python-buildbot/local/bin/python, you need to make sure you install all the pre-requisite packages for your package using this custom python binary, otherwise your unit tests will fail because they will not find these pre-requisites. For example, for the Twisted unit tests, I had to install setuptools, ZopeInterface, pycrypto and pyOpenSSL, before I could actually run the Twisted test suite.

So in my file I first call a shell script, before I launch the actual test suite (I copied the pre-requisite packages in /home/buildslave):

$ cat


cd /tmp

rm -rf setuptools*
cp ~/ .
cd setuptools-0.6c1
/tmp/python-buildbot/local/bin/python install
cd ..

rm -rf ZopeInterface*
cp ~/ZopeInterface-3.1.0c1.tgz .
tar xvfz ZopeInterface-3.1.0c1.tgz
cd ZopeInterface-3.1.0c1
/tmp/python-buildbot/local/bin/python install
cd ..

rm -rf pycrypto-2.0.1*
cp ~/pycrypto-2.0.1.tar.gz .
tar xvfz pycrypto-2.0.1.tar.gz
cd pycrypto-2.0.1
/tmp/python-buildbot/local/bin/python install
cd ..

rm -rf pyOpenSSL-0.6*
cp ~/pyOpenSSL-0.6.tar.gz .
tar xvfz pyOpenSSL-0.6.tar.gz
cd pyOpenSSL-0.6
/tmp/python-buildbot/local/bin/python install
cd ..

rm -rf Twisted
svn co svn:// Twisted

Then I call the actual Twisted test suite, via:

/tmp/python-buildbot/local/bin/python -Wall /tmp/Twisted/bin/trial --reporter=bwverbose --random=0 twisted

You can see the current Pybots status page here.

If you are interested in setting up your own buildslave to participate in the Pybots project, please send a message to the Pybots mailing list. I will send you a slavename and a password, and then we can test the integration of your buildslave with the buildmaster.

Update 10/16/09

I realized that these instructions for setting up a Pybot buildslave are a bit outdated. Discussions on the Pybots mailing list prompted certain changes to, even though you're still OK if you follow the instructions above to the letter.

Here are some enhancements that you can take advantage of:

1. You can test several projects, each in its own build step, simply by having your script be aware of an extra command-line argument, which will be the name of the project under tests. An example of such a script is here: The script receives a command-line argument (let's call it proj_name) and invokes a shell script called The shell script checks out the latest code for project proj_name (or downloads the latest distribution), then runs its unit tests. Here is an example:

2. You do not have to hardcode the path to the newly built Python binary in your or your shell scripts. You can simply retrieve the path to the binary via sys.executable. This script sets an environment variable called PYTHON via a call to
os.putenv('PYTHON', sys.executable)
Then the variable is used as $PYTHON in the shell scripts invoked by (thanks to Elliot Murphy and Seo Sanghyeon for coming up with this solution.)


Anonymous said...

Do you have a sample script to avoid having to reinvent the wheel?

Grig Gheorghiu said...


Take a look at the files checked in here: will be called with an extra argument; on that specific buildslave, the argument will be either "Cheesecake" or "py.lib", since these are the 2 projects I'm testing there, and I want a separate build step for each project.

In your case, if you want to test SQLAlchemy, expect to be called with an argument of "SQLAlchemy". You can then have a script called, which checks out the SQLAlchemy code, then runs its tests -- just as and do.



Modifying EC2 security groups via AWS Lambda functions

One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...