The idea originated from Glyph, of Twisted fame. He sent out a message to the python-dev mailing list (thanks to John J. Lee for bringing this message to my attention), in which he said:
"I would like to propose, although I certainly don't have time to implement, a program by which Python-using projects could contribute buildslaves which would run their projects' tests with the latest Python trunk. This would provide two useful incentives: Python code would gain a reputation as generally well-tested (since there is a direct incentive to write tests for your project: get notified when core python changes might break it), and the core developers would have instant feedback when a "small" change breaks more code than it was expected to."
Well, Neal Norwitz made this happen by setting up a buildmaster process on one of the servers maintained by the PSF. He graciously allowed me to maintain this buildmaster, and I already added a buildslave which runs the Twisted unit tests (in honor of Glyph, who was the originator
of this idea) every time a check-in is made in the Python trunk. You can see the buildmaster's status page here.
Note that some of the Twisted unit tests sometimes fail for various reasons. Most of these reasons are environment-related -- for example the user that the buildbot slave runs as used to not have a login shell in /etc/passwd, and thus a specific test which was trying to run the login shell as a child process was failing. Not necessarily a Twisted bug, but still something that's nice to catch.
And this brings me to a point I want to make about running your project's automated tests in buildbot: I can almost guarantee that you will find many environment-specific issues that would otherwise remain dormant, since most likely the way you're usually running your tests is in a controlled environment that you had set up carefully some time ago. There's nothing like running the same tests under a different user's account and environment.
Of course, if you've never run your tests under a continuous integration process such as buildbot, you'll also be pleasantly -- or maybe not so pleasantly -- surprised at the amount of stuff that can get broken by one of your check-ins that you considered foolproof. This is because buildbot tirelessly checks out the very latest source code of your project, then runs your unit tests against that code. When you run your unit tests on your local machine, chances are you might not have synchronized your local copy with the repository.
This does assume that you have unit tests for your project, but since you have been reading this post this far, I assume you either do, or are interested in adding them. I strongly urge you to do so, and also to contribute to the Pybots project by setting up a buildslave for your project.
I'll post another entry with instructions on how to configure a buildslave so that it can be coordinated by the Pybots buildmaster. I also have a mailing list here (thanks, Titus!) for people who are interested in this project. Please send a message there, and I'll respond to you.
The buildmaster is currently running the build steps every time a check-in is made to the Python trunk, and to the 2.4 branch. In the near future, there will be a 2.5 branch, and the trunk will be used for 2.6 check-ins. I'll modify the buildmaster configuration to account for this.
Update: The buildmaster is now aware of changes in both the trunk and the newly created release25-maint branch. You can watch the HTML status page for all builders, or for the trunk builders.
BTW, if you need instructions on setting up buildbot, you can find some here, here, here and here.
No comments:
Post a Comment