Xooglers

Fascinating blog I came across a couple of days ago: Xooglers (which stands for "ex-Googlers"). A real page-turner about the anticipated ups, but mostly about the unexpected downs of life as a Googler.

Updates to PTTT

...where PTTT stands of course for Python Testing Tools Taxonomy. I'm glad to see that people are updating the Wiki page and adding more stuff to their tool description or adding other tools to the list. Some examples:

• Ori Peleg updated the description of his TestOOB tool: "unittest enhancements; test filtering via regex/glob patterns; reporting in XML/HTML; colorized output; runs pdb on failing tests; run in parallel in threads/processes; verbose asserts; report failures immediately; and a little more;"
• Ian Bicking added his paste TestFileEnvironment tool ("A simple environment for testing command-line applications, running commands and seeing what files they write to")
• Geoff Bache added his TextTest tool, an acceptance testing tool "written in python but it can be used to test programs written in any language. Comes with extensive self tests which serve as examples of how to use it, ie how to test a non-trivial application with a pyGTK GUI"; TextTest looks really intriguing, I'll check it out and comment on it soon
  
• Zack Cerza added two GUI testing tools:
• dogtail ("Created by Redhat engineers on linux. Uses the X11 accessability framework (AT-SPI) to drive applications so works well with the gnome desktop on Unixes. Has flash movies")
• ldtp ("Also uses the X11 accessability framework (AT-SPI) to drive applications so works well with the gnome desktop on Unixes. Has extensive tests for the evolution groupware client.")

Martin Fowler on In-Memory Databases and Testing

Just came across Martin Fowler's post on In-Memory Databases. I was glad to see his mention of Firebird, which has been a favorite of mine for a number of years. Fowler talks about the primary use of in-memory databases: testing, or to be more precise test-driven development, where speed is of the essence. He also mentions SQLite, which I've only recently started playing with, mainly by going through the Django tutorial. I'd like to explore it further though in the context of testing.

Python Testing Tools Taxonomy

There's been a flurry of blog posts recently on the subject of Python testing, especially on Web application testing tools. I thought it would be a good idea to put up a Wiki page with the tools I know of, so that anybody who's interested in contributing to the "Python Testing Tools Taxonomy" can do so easily. Here is what I have so far. Feel free to modify it.

Update 05/01/06

The PTTT page has moved here. Please update your bookmarks etc.

Cheesecake project update

Micah Elliott graciously offered to host my Cheesecake project at his TracOS site, a Trac-based Wiki that hosts a collection of Open Source projects. Check out the brand new Cheesecake home page and let me know what you think. I haven't had time to properly package my code (which is kind of ironic, considering that checking packages for their 'goodness' is after all the goal of the Cheesecake project), but you can grab the source code via subversion:

svn co http://svn.tracos.org/cheesecake

I'll put the code up on SourceForge too at some point, but it seems to me that the Trac-based Wiki is much more "agile" than the SourceForge interface, so I'll make TracOS the primary home for my project.

Bob Koss on "Refrigerator Code"

Seen via Jeffrey Fredrick's blog: Bob Koss talks about Refrigerator Code, i.e. code that you're so proud of that you're ready to put it on the refrigerator, next to your kids' drawings. Nice metaphor. It reminds me of an expression used by the late Chick Hearn, the famous play-by-play announcer for the Lakers: "This game is in the refrigerator!"

Lightning Talks session at Star West 2005

Yesterday I gave a 5-minute Lightning Talk on Selenium at the Star West 2005 testing conference in Anaheim. There were 9 speakers in all, coordinated by Erik Petersen. It was my second Lightning Talk experience, after the one at PyCon 2005 earlier this year. It wasn't quite as interactive as at PyCon, mainly because it was based on slides rather than live demos (I did do a live demo of Selenium though), but it was still very interesting and intense. Erik made it even more fun by introducing every speaker with a little Vegas-style tune that evoked the Rat Pack's apparition on the stage.

My favorite presentation was Rob Sabourin's, who talked about the Iron Ring given to all Professional Engineers in Canada during a secret "Calling of the Engineers" ceremony. The ring is made of iron and is initially very coarse, but the engineers who receive it are supposed to wear it permanently on their working hand, so in time it becomes round and smooth. It is a symbol of pride to be part of an important profession, but it's also a symbol of humility and social responsibility. Rob linked the ring to the Quebec Bridge Disaster which happened at the beginning of the 20th century, when what was supposed to be the longest cantilever bridge in the world collapsed twice during construction because of a scaling problem (the weight of the bridge turned out to be more than what the engineers were used to from previous constructions of smaller bridges). What I took away from the talk was the need for more social responsibility from software developers and testers in particular.

Another fun presentation was Jon Bach's, who talked about an interesting form of testing: open-book exams. He has an article online on this subject. The main idea is that if you want your application to be tested by technical and non-technical people alike, just give them an open-book exam with 30-40 questions related to various areas of functionality, and let them collaborate in finding the answers by exploring the application. This offers a much more focused testing environment than just telling them to go explore the application on their own. Interesting indeed...I have to think about ways to apply this in my day-to-day work.

Other interesting topics from some of the other Lightning Talks:

• a test team's success with using session-based testing (here's a good introductory article by the same Jon Bach)
• keeping track of testing progress by means of simple Excel spreadsheets (as opposed to cumbersome project management software and Gantt charts that quickly become obsolete) and using burndown charts to show where the test team stands
• using virtual machines to simplify the administration of the testing environment
• how gaining even a millisecond is important when doing performance testing at VeriSign
• how RUP could not be easily adapted to non-software manufacturing environments

On the whole, it was a very focused session, choke-full of ideas and lessons to take away. Kudos to Erik Petersen for putting it together. If you want my advice, the next time you attend a conference, don't miss the Lightning Talks if they are offered!

Agile Dilbert

Saw this on Roy Osherove's blog.

Article on All-pairs testing technique

From the agile-testing mailing list, courtesy of Todd Bradley, here's a link to a PDF version of an article by Bernie Berger on the All-pairs testing technique. If you're a tester, you need to get acquainted with this technique, so that you can mitigate the combinatorial explosion of your test cases.

Exciting times in the Python testing world

If you are a developer or tester using Python, you live in exciting, ebullient times. There are Python-based testing frameworks newly-announced or recently-updated almost every day. Here is a rundown of the latest I'm aware of:

Unit testing

• py.test: no recent new release, but changes are happening almost daily in svn
• TestOOB: version 0.7 was released recently (TestOOB is an enhancement to the standard unittest module, offering many features that py.test offers)
• nose: version 0.7.2 was freshly released yesterday (nose, in its author's words, "provides an alternate test discovery and running process for unittest, one that is intended to mimic the behavior of py.test as much as is reasonably possible without resorting to too much magic"; nose will become, if it's not already, the official test framework for TurboGears)

Web application testing

• twill: version 0.7.4 was released on Nov. 11th, with unit tests that use nose, and with new commands to help developers use twill to unit test their own Web apps; also, Titus Brown announced yesterday that he extended twill to add in-process testing of WSGI applications (I blogged about twill here)
• FunkLoad: version 1.3.1 was released on Nov.10th (FunkLoad offers functional, performance, load and stress testing of Web applications)
• zope.testbrowser: version 0.9.0 was released on Nov.12th (zope.testbrowser is the stand-alone version of the Zope 3 functional testing framework)
• Sancho: version 2.1 was released on Nov. 2nd (Sancho is the unit test framework for the MEMS Exchange applications; for those who are not familiar with MEMS Exchange, they are the guys behind Quixote, Durus and other Python apps)
  
• Django's own doctest-based test framework: read my blog post about it
  
• ibofobi's doctest-based framework for Django: blog post from Nov. 8th announces a new unit test framework for Django-based Web applications
• [Update from Ian Bicking, the creator of Paste]: paste.test.fixture: the unit test framework used in Paste (read "What is Paste, yet again" on Ian's blog)
• [Update from Robert Brewer, the creator of CherryPy]: webtest is a small, but helpful and isolated web-application-test module (it extends unittest) used to test CherryPy

GUI testing

• guitest: version 0.3 was released on Nov. 13th (guitest is a Python helper library for testing Python GUI applications, with pyGTK support being the most mature)

Various

• retest: version 0.5.1 was released on Sept. 23rd (retest enables tests of Python regular expressions in a webbrowser; it uses SimpleHTTPServer and AJAX)

I also want to mention MochiKit as an example of an application that makes it a point to offer top-notch tests and documentation. MochiKit is a JavaScript library that is very "Pythonical" in nature, which is not surprising given that one of its main developers is Bob Ippolito, well-known for his contributions in the Python community. One of the goals of MochiKit is to maintain 100% documentation coverage, and another is to test itself mercilessly. If only all applications followed these tenets, the world would truly be a better place :-)

SoCal Code Camp

From the xpsocal mailing list: the Southern California Code Camp will be held on Jan.21-22 2006 at Cal State Fullerton. I checked out the Sessions page and the organizers say "All technologies are welcome C++, C#, VB.Net, Java, Ruby, COBOL???, SQL, etc... if it is code... we want it." I noticed at least one glaring omission (what? no Python?) so I decided to remedy it by sending a proposal for a Python session, which is basically my PyCon 2005 talk on "Agile testing with Python test frameworks". Let's see if they go for it.

ibofobi's Django doctest framework

In the "This is way too cool" category: a new doctest-based framework for testing your Django apps. Other than doctest, it also uses Beautiful Soup and YAML. I need to check it out at some point.

Drucker on agility and testing

The other day I picked up a copy of "The Daily Drucker" from the local library. It was again one of those fortuitous events, almost as if the book told me to pick it up from the shelf. I had another similar experience last year at the same library when I picked up "XP Explained", so I think it's particularly fitting to find references to agility and testing in Drucker's ideas. I don't think Peter Drucker needs any introduction, but I have to confess I haven't read any of his books yet (although this is about to change!). "The Daily Drucker" is a collection of fragments from his books, put together in "a thought a day" format that's pretty agile in itself :-)

Here's what Drucker has to say on organizational inertia: "All organizations need to know that virtually no program or activity will perform effectively for a long time without modification and redesign. Eventually every activity becomes obsolete."

These are sobering words that can be applied verbatim to software development (note the use of the word 'program' in the above quote!). They say that you must be prepared for change, and indeed welcome it, as XP teaches. In fact, in the same fragment (from "The Age of Discontinuity", Drucker goes on to say: "All organizations must be capable of change. We need concepts and measurements that give to other kinds of organizations what the market test and profitability yardstick give to business. Those tests and yardsticks will be quite different."

Note the use of the word "tests". To apply this idea to software, you can't prove that your application is capable/performing if you don't have tests and yardsticks. Again, an idea that appears over and over in agile software methodologies. Here's Drucker again: "Each institution will be the stronger the more clearly it defines its objectives. It will be more effective the more yardsticks and measurements there are against which its performance can be appraised. It will be more legitimate the more strictly it bases authority on justification by performance." You couldn't give a better justification for having unit/acceptance/performance tests in place for your application even if you tried :-) !!!

I was also struck by some of Drucker's ideas on abandoning what doesn't work. It reminded me about the virtue of courage in XP, which gives you the fortitude to throw away code and start anew, knowing that you'll probably do a much better job. Here's Drucker: "Without systematic and purposeful abandonment, an organization will be overtaken by events. It will squander its best resources on things it should bever have been doing or should no longer do. As a result, it will lack the resources, especially capable people, needed to exploit the opportunities that arise." It's almost like Drucker is talking about the YAGNI principle and about Big Design Up Front. The alternative, of course, is to Do The Simplest Thing That Could Possibly Work. In fact, according to Drucker: "The question to be asked - and asked seriously - 'If we did not do this already, would we, knowing what we know, go into it now?' If the answer is no, the reaction must be 'What do we do now?'" In XP, we know what to do now because we have the current iteration to work on, having obtained feedback from the customers.

Drucker talks of course about feedback too. He associates it with 'continuous learning' (another agile concept) and he advises everybody to write down the results they anticipate before doing anything of significance. Then, after a number of months, they will feed back from the actual results to the anticipations. This accomplishes three things: firstly, it shows you what you did well and what your strengths are; secondly, it shows you what you have to learn and what habits to change; and thirdly, it show you what you are not gifted for and cannot do well. These are the keys to continuous learning, and it seems to me that they're used very effectively in the agile practices of test-driven development, continuous integration, constant feedback and frequent iterations.

Finally, Drucker has something to say about agility itself: "Large organizations cannot be versatile. A large organization is effective through its mass rather than through its agility. An organization, no matter what it would like to do, can only do only a small number of tasks at the same time. This is not something that better organization or 'effective communication' can cure. The law of organization is concentration. Yet modern organization must be capable of change. Indeed it must be capable of initiating change, that is innovation. It must be able to move scarce and expensive resources of knowledge from areas of low productivity and nonresults to opportunities for achievement and contribution. This however, requires the ability to stop doing what wastes resources." The lesson for agile development? Keep your teams small and focused, encourage an environment of collaboration and freely flowing ideas, don't pidgeon-hole people into roles, be prepared to react quickly and to throw away what doesn't work.

I am amazed by the depth and breadth of Drucker's ideas, and about their direct applicability to software development. My TODO list has just been augmented with "Read at least one Druckerian wisdom nugget every day".

"Articles and tutorials" page updated

As blog posts come and go to the archive bucket, finding a specific article in a blog is hard to do. This page wants to be a central repository of links to the various articles and tutorials that I posted so far.

Updated 01/19/2006

Unit testing
Unit testing in Python part 1: unittest
Unit testing in Python part 2: doctest
Unit testing in Python part 3: py.test
Michael Feathers on unit testing rules

Acceptance testing with FitNesse
PyFIT/FitNesse Tutorial Part 1
PyFIT/FitNesse Tutorial Part 2
PyFIT/FitNesse Tutorial Part 3

Web application testing
Web app testing with Jython and HttpUnit
Web app testing with Python part 1: MaxQ
Web app testing with Python part 3: twill
Acceptance tests for Web apps: GUI vs. business logic

Selenium-specific articles
Web app testing with Python part 2: Selenium and Twisted
Quick update on Selenium in TestRunner mode
Quick update on Selenium in Twisted Server mode
Using Selenium to test a Plone site (part 1)
Using Selenium to test a Plone site (part 2)
New features in Selenium 0.3
Article on Selenium in Oct. 2005 issue of "Better Software"
Selenium at OpenQA
Running Selenium in Python Driven Mode
Testing Commentary (and thus Ajax) with Selenium

Performance/load/stress testing
pyUnitPerf Tutorial
Performance vs. load vs. stress testing
HTTP performance testing with httperf, autobench and openload
More on performance vs. load testing

Automated test distribution, execution and reporting
STAF/STAX Tutorial

General testing topics
Quick black-box testing example
White-box vs. black-box testing

Agile Documentation
Agile Documentation with doctest and epydoc
Agile documentation in the Django project

The Cheesecake project
Cheesecake: How tasty is your code?
Cheesecake project on SourceForge
A whiff of Cheesecake
More Cheesecake bits
New home for Cheesecake and Sparkplot projects

Databases
Installing and using cx_Oracle on Unix
Installing Python 2.4.1 and cx_Oracle on AIX
Installing the Firebird database on a 64-bit RHEL Linux server

The py library
Keyword-based logging with the py library
py lib gems: greenlets and py.xml
'py library overview' slides

Python on Windows
Handling the 'Path' Windows registry value correctly
Running a Python script as a Windows service

System Administration HOWTOS
Telecommuting via ssh tunneling
Managing DNS zone files with dnspython
Configuring OpenLDAP as a replacement for NIS
Chroot-ed FTP with wu-ftpd
System monitoring via SNMP
Compiling and installing a custom Linux kernel
Configuring Apache 2 and Tomcat 5.5 with mod_jk

Data visualization
sparkplot: creating sparklines with matplotlib

Usability
Jakob Nielsen on Usability Testing
Jakob Nielsen on Blog Usability

Other articles
Python as an agile language
Oblique Strategies and testing