testing-in-python mailing list

Titus created a mailing list dedicated to topics related to testing in Python. If you're interested, you can subscribe via its mailman interface here. We hope it will become a discussion forum for things such as:

• testing tools that people have successfully used in their projects
• testing techniques that help in certain situations (mocking for example)
• real life scenarios where a specific type of testing (e.g. functional) helped more than another type of testing (e.g. unit)
• etc. etc.
  

I think I will affectionately refer to this list as TIP from now on :-)

Update 02/28/07

What do you know, the name TIP struck a chord, so Titus created an alias for it. You can now send email to the list via tip at lists.idyll.org too.

William McVey's PyCon notes as mindmap

Via Elliot Murphy's blog, a very nice mindmap created by William McVey and showing his notes from PyCon07. It demonstrates the power of mindmaps: a lot of information condensed in one page, with links to more detailed information below.

Also from William, other PyCon07 notes.

Testing Tools Panel at PyCon

I'd like to thank the participants in the Testing Tools Panel at PyCon07 for sharing their insights into testing with the audience. Here they are, in alphabetical order of first name:

• Benji York: zope.testbrowser
• Brian Dorsey: py.test (representing Holger Krekel)
• Chad Whitacre: testosterone (created) switched to nose
• Ian Bicking: paste.test.fixture, minimock, FitLoader
• Jeff Younker: PyMock
• Kumar McMillan: fixture - module for loading and referencing test data
• Martin Taylor: test framework within TI
• Neal Norwitz: PyChecker
• Tim Couper: WATSUP (Windows GUI Testing)
• Titus Brown: twill, scotch, figleaf, pinocchio

Matt Harrison has a very good write-up on the discussions we had during the panel (actually I lifted the list above from his blog post, because he summarized it so well).

One thing that I think all the participants felt, and maybe the audience too, was that 45 minutes was totally not sufficient for this kind of panel. And I know I felt the same thing with the other two panels, for Python-dev and for the Web frameworks. So I'd like to ask people to leave some comments on this post, with ideas about turning these panels into discussions that would last longer, at least 1 hour and even more.

My gut feeling is that there would be a lot of interest in getting framework/library/tool creators together and having a discussion/Q&A with them in front of the audience, with audience participation of course. I'm not sure what the best format would be for this kind of thing -- maybe a round table? But if we get enough ideas, maybe we can fit something like this in next year's PyCon schedule, and allocate it a generous amount of time.

Testability

During the Testing Tools tutorial that Titus and I gave at PyCon, there was a short discussion on testability -- what makes software more testable? I mentioned a list put together by Michael Bolton, and summarized/enhanced by Adam Goucher in this blog post. Recommended reading, both for developers who want to add testing hooks into their software, and for testers who want to know what to ask for from developers so that their life gets easier (and if you're one of the unfortunate souls who have to deal with Java or .NET, this blog post by Roy Osherove talks about testability and pure OOP.)

Although our tutorial was focused on tools and techniques for implementing test automation, we also mentioned that you will never be able to get rid of manual testing. Even though the Google testing team says that 'Life is too short for manual testing' (and I couldn't agree more with them), they hasten to qualify this slogan by adding that automated testing frees you up to do more meaningful exploratory testing.

My experience as a tester shows that the nastiest bugs are often discovered by manual testing. But when you do discover them manually, the best strategy is to write automated tests for them, so that you'll check your application in that particular area from that moment on, via an automated test suite which runs in your continuous integration system.

You do have an automated test suite, right? And it does run periodically (daily or upon on every check-in) in a continuous integration system, right? And you have everything set up so that you're notified by email or RSS feeds when something fails, right? And you fix failures quickly so that everything turns back to green, because you know that too much red, too often, leads to broken windows and bit rot, right?

If you answered No to any of these questions, then you are not testing your application, period (but you already knew this if you took our tutorial -- it was on the last slide :-)

Photos from PyCon panels

Here are photos I took today at the python-dev panel and at the Web frameworks panel. Enjoy!

BTW, here are two very good write-ups on the Web framework panel: one from Matt Harrison, the other from James Bennett.

PyCon day 1

Gave the testing tutorial with Titus yesterday; went pretty well from the feedback we got. We'll publish the slides soon.

Just got out of the first keynote, Ivan Krstic's talk on the "One Laptop Per Child" project. Pretty interesting -- here are some tidbits I remember:

• OLPC wants to change the way teaching and learning is done these days; they want to go back to the time when preschool kids interacted with each other by playing, and learned naturally peer-to-peer (as opposed to institutionalized teaching, which is one-to-many)
  
• contrary to popular opinion, the laptop does not have a hand crank (it would wear down too fast if it had one); however, the laptop can be powered by a pull string that reacts to the puller's strength and powers the device accordingly
  
• the 2 rabbit ears are used for wireless; the laptop can speak 802.11s, a new protocol that can be used for fully meshed networking; as soon as one laptop is connected to the internet, all the other ones in its mesh will be connected too
• the CPU is an AMD Geode at 366 MHz (not 400 or 500, actually 366)
• no hard drive, uses 512 MB of flash storage
• OS is a stripped-down version of Fedora
• runs Python wherever it can (including the init boot daemon); some exceptions are the X.org windowing system, the mDNS daemon, and the bus communication; pretty much all other user-level software, including the file system, is written in Python
• the laptop has a 'show source' button which obviously shows Python source code that can be edited, etc.
• no adult has ever been able to open the laptop in less than 2 minutes
• no child has ever needed more than 30 seconds to open the laptop
• two fortunate souls got an OLPC XO laptop today: Guido (as the creator of Python), and a guy who was able to recognize a very complicated formula that Ivan showed on a slide (the BBP formula for computing the n-th decimal place of pi); the guy needed approx. 1 minute to open the laptop; Guido's was already open, in a sign of respect I guess
• OLPC needs good Python developers; if you're interested, check out dev.laptop.org
  

Wikipatterns

Via Grady Booch's blog, a site which looks very promising: Wikipatterns. It identifies patterns and anti-patterns for wiki adoption. Here's an excerpt:

"Any grassroots, or bottom-up, strategy is the best place to start since the success of a wiki depends on building active, sustainable participation and this only happens when people see that the software is simple enough to immediately be useful, and meets their needs without requiring them to spend lots of extra time.

A good first step is to identify a group or department who would likely benefit the most from using a wiki, and whose people are open to trying new tools. If you're looking to expand wiki use in another group, look for the thought leader in the group - someone who is very forward thinking, respected by peers, and willing to Champion a new idea and get others around them involved."

Seems like a very good complement to a book I read recently: "Fearless Change: Patterns for Introducing New Ideas" by Mary Lynn Manns and Linda Rising.

Anybody doing LDom on Solaris Sparc?

Lazy Web-type question: has anybody tried out the brand new LDom functionality available in Solaris 10? From some googling around I've done, it seems Sun hasn't yet shipped the full-fledged LDom functionality in Solaris 10 version 11/06.

LDom seems like a cool way to partition some big Solaris Sparc boxes, if you have them. I wonder if the logical domains/virtual machines created with LDom can have Ubuntu installed on top of them (because Ubuntu supports Solaris Sparc).

The Buildbot project has a Trac instance as its home

Brian Warner just announced on the buildbot-devel mailing list that his Buildbot project has a brand new home in the form of a Trac instance. Check it out here. Glad to see the move, as Trac is so much nicer to work with than the clunky SourceForge interface.

Ubuntu not to activate proprietary drivers by default

This just in via Jonathan Carter's blog on Planet Ubuntu: the Ubuntu Technical Board has decided that Ubuntu will not activate proprietary drivers by default. Proprietary drivers will be provided for convenience, and users will of course be free to install them if they so choose. This should settle a controversy which has been raging for a while in the Ubuntu community.

Cheesecake Service launched

The Cheesecake Service is a result of Michał Kwiatkowski's hard work during the Google Summer of Code 2006. The Web interface for the service is based on web.py and it has been up and running since last August, but we're making it public now, to coincide with the release of Cheesecake 0.6.1.

The back-end of the Cheesecake Service talks directly to the PyPI repository using the PyPI API in order to find out about packages newly posted to PyPI. Then the service uses Cheesecake and tries to download, install, and score the package. If your package is not there, it might mean you haven't released a new version after August 10, 2006, date from which we started to score packages. Let us know and we can manually score it so that it appears in the list.

Michał just posted a blog entry on Cheesecake and its Service, so please read it and let us know how we can improve on the various things he describes. We are aware that scoring packages is controversial, and we've been called names before because of it, but as Michał also says, the Cheesecake score is meant to be used as a relative number by creators of Python packages who can try to improve it, and not as an absolute ranking among packages. Think of it as an Apgar score for your software.

Kudos to Michał for his continuing hard work on new Cheesecake features and improvements.

Internal blogs as project tracking tools

I'm reading "The Corporate Blogging Book" by Debbie Weil. She talks about two different types of blogs -- internal (Intranet-type) and external (public access) -- and she mentions how some big name companies use internal blogs for all sorts of purposes: knowledge/information sharing, email replacement, and, the thing that caught my attention, project management. Apparently IBM is big on all these things when it comes to internal blogs.

So I had a mini-revelation: an internal blog is a very good tool for tracking time you spend on various projects. Take the example of a hosting company -- it can set up an internal blog and have categories corresponding to various projects/customers; employees can jot down a summary of what they worked on each day, and put it in the appropriate category. After a while, a timeline of work done on particular projects emerges. Because posts are automatically dated, it's easy to see what you were working on 3 weeks or 3 months ago. And each blog post can contain links to more detailed howtos that are kept in a wiki which serves as a knowledge base. Blogs and wikis make entering information a snap, as opposed to more complicated project management/tracking tools. Blogs and wikis are also searchable, so finding information is easy. To me, this is a lean/agile way of keeping track of your work.

Anyway, maybe this is an obvious use of blogs, but to me it's new, and of course I'm going to implement it :-)

New job

Today was my first day at RIS Technology, a Web hosting/managed services company which hosts grammy.com and missuniverse.com, among other sites. My responsibilities include selecting and helping implement various technologies that make up our offerings to our customers. Of course, if you follow my blog, you know that Python and automated testing are big in my book, so you can be sure that our offerings will include both :-) Stay tuned for more details.

Connecting to people on LinkedIn

I wasn't such a big LinkedIn fan until a short time ago, when a post by Guy Kawasaki caught my attention. Then synchronicity kicked in and Tennessee Leeuwenburg sent a message to the python-advocacy mailing list, asking Python developers to connect to each other on LinkedIn; here's what Tennessee had to say:

"One way to help spread Python would be to have a strong presence of Python developers in various online networks. One that springs to mind is LinkedIn, a job related social networking site.

If we could encourage Python developers to start adding eachother to their LinkedIn network, then we shoud be able to create a well-connected developer network with business and industry contacts. This should benefit everyone -- both people looking for Python developers, and also people looking for work."

So in the past week or so I started to send LinkedIn invitations to people I know, either by having worked with them, or through the various forums, mailing lists and Open Source communities I have been part of. It's amazing how many people we all know, if we think about it.

LinkedIn has several nice features that can help when you're looking for people to hire, or when you're looking for a job. Perhaps the easiest way to find people is to click on 'Advanced search' (the small link next to the main search box) and type something in the Keywords field. Try it with 'python' for example -- you'll see that a lot of people whose blogs are aggregated on Planet Python have a LinkedIn profile. Your next step, if you are a Python developer yourself, is to send invitations to people you want to connect with. If enough of us Pythonistas do this, our networks will become more and more interconnected, to everybody's advantage. And you can replace 'Pythonistas' with 'agilistas', 'rubyistas' or whatever your interest is.

It's also interesting to see how LinkedIn displays the number of the degrees of separation between yourself and people you are searching for. Amazingly enough, that number is usually 2 or 3, if not 1. This makes me think of Malcolm Gladwell's theory about Connectors in 'The Tipping Point', namely that there is a small number of people that have a LOT of connections. If you are connected to one of these Connectors, then all of a sudden you have a huge number of people in your network, and you can potentially benefit by introducing yourself to them as someone only 2 or 3 degrees of separation apart from them. This is true in my own network, where I am only 2 degrees of separation away from Guy Kawasaki for example. Why? Because a long long time ago I accepted a LinkedIn invitation from one Paul Davis, who has 500+ LinkedIn connections.

If I made you curious about LinkedIn, I'd advise you one more time to read Guy Kawasaki's blog post on how to improve your LinkedIn profile.

Speaking of jobs and hiring, if you are a hardcore Python programmer looking for work, especially in the D.C. area, the Zope Corporation is hiring.

Pybots updates

Since my last Pybots-related blog post, a lot has happened. We added 2 buildslaves, a Sparc Solaris 10 host running the Django unit test suite (courtesy of Matthew Flanagan) and a G5 OSX host running the SQLAlchemy test suite (courtesy of Skip Montanaro). So now we have 10 buildslaves altogether. I also added more test suites to my Ubuntu Breezy buildslave. In addition to the Cheesecake test suite, I'm now running the unit test suites for the py library, nose, twill and Testoob.

Email notifications finally started to work too, after I finally figured out I was passing the wrong builder names to the MailNotification class. And we also have RSS feeds available. If you want to be notified of failures from all builders, subscribe to:

http://www.python.org/dev/buildbot/community/all/rss or
http://www.python.org/dev/buildbot/community/all/atom

To be notified of failures from the trunk builders, subscribe to:

http://www.python.org/dev/buildbot/community/trunk/rss or
http://www.python.org/dev/buildbot/community/trunk/atom

To be notified of failures from the 2.5 branch builders, subscribe to:

http://www.python.org/dev/buildbot/community/2.5/rss or
http://www.python.org/dev/buildbot/community/2.5/atom

Matthew Flanagan also added functionality that allows you to subscribe to a feed for a particular builder. For example, to subscribe to the feed for the "x86 Ubuntu Breezy trunk" builder, use this URL:

http://www.python.org/dev/buildbot/community/all/rss?show=x86%20Ubuntu%20Breezy%20trunk

If you are interested in contributing a buildslave to the Pybots project, please send a message to the Pybots mailing list, or to me (grig at gheorghiu dot net), or leave a comment here.

Steve Rowe on "Letting test drive the process"

Just came across a blog post by Microsoft' Steve Rowe called "Letting test drive the process". Steve quotes an article by Richard Collins -- "Test, test and test again" -- and adds his own observations on the practices of involving testers early in the development process, and of building testable interfaces into the product instead of heavy UIs.

According to Steve Rowe, Microsoft's development and testing process follows these recommended practices. I quote Steve:

"Also, at Microsoft, testing begins from day one. Every product I've ever been involved with at Microsoft has had daily builds from very early on. Every product has also had what we call BVTs (build verfication tests) that are run every day right after the build completes. If any of their tests fail, the product is held until they can be fixed."

Hmmm...I would expect Microsoft to have less problems with their products in this case. But I think a couple of problems that plague Microsoft in particular are backwards compatibility and the sheer amount of hardware/OS/service pack combinations that they need to test.

Speaking of Microsoft and testing, I found The Braidy Tester's blog very informative.

Testing tutorial at PyCon07

Titus and I will present a tutorial on "Testing Tools in Python" at PyCon07. It is scheduled for the afternoon session on Thursday Feb.22. It will be an improved version of our "Agile Development and Testing in Python" tutorial from last year.

Here is the tutorial outline (courtesy of Titus). If you have any suggestions, please leave a comment.

Introduction
* Why test?
* What to test?
* Using testing to boost maintainability of code.

Setting up a project
* Source control management with Subversion.
* A brief introduction to using Trac for project
documentation and ticket management.
* Packaging with distutils
* Packaging with setuptools
* Registering your project with the Python Cheeseshop
* What else is out there? (distributed vs svn, roundup, ...)

Unit testing
* How to think about unit testing
* Using nose to run unit tests
* doctest-style unit tests
* What else is out there? (unittest, py.test, testosterone...)

Functional Web testing with twill
* Writing twill scripts
* Running twill scripts
* Using scotch to record actions
* Using wsgi_intercept to avoid network sockets
* What else is out there? (zope.testbrowser, mechanize, mechanoid)

Using code coverage in conjunction with unit/functional testing
* Basic code coverage with figleaf
* Monitoring code coverage in remote servers
* Combining figleaf code coverage analyses
* What else is out there (coverage)

** BREAK **

Acceptance testing with FitNesse/PyFit
* How FitNesse works
* Writing fixtures
* Running Python fixtures

Web application testing with Selenium
* How Selenium works
* Writing and recording Selenium tests
* Scripting Selenium tests remotely with SeleniumRC
* What else is out there? (Sahi, Watir)

Continuous integration with buildbot
* Introduction to buildbot
* Discussion of concepts, demonstration.
* Integrating tests into buildbot.
* GUI testing in buildbot.
* Using pybots to test your open source project

Conclusion
* Why test, revisited
* Maintainability and testing

Testing Tools Panel at PyCon07: questions needed

I will be moderating the Testing Tools Panel at PyCon07, currently scheduled from 11:40 AM to 12:25 PM on Sat. Feb. 24th (and immediately followed by Guido's keynote). I put together a Wiki page for the panel, with questions and topics that I thought would be interesting for the audience (and with some input from Ian Bicking.)

I'd be very grateful if people who plan on attending the panel could add more questions or topics of interest either by directly editing the Wiki page, or by leaving a comment here, or by sending me an email at grig at gheorghiu.net. Thanks in advance!

New Year's resolution

Here's one New Year resolution I'm trying to keep: each day, read the corresponding page for that date from "The Daily Drucker". I blogged about this book before, and I continue to be amazed at the insight and wisdom that Drucker manages to pack in almost every sentence he writes. Although Drucker writes about general practices of management and leadership, many of his ideas can be easily applied to software development in general, and testing in particular.

Here are some fragments from January 4th, on "Organizational inertia", which can be applied just as well to any software project ("bitrot" and "goldplating" come to mind):

"All organizations need to know that virtually no program or activity will perform effectively for a long time without modifications and redesign. Eventually every activity becomes obsolete."

"Businessmen are just as sentimental about yesterday as bureaucrats. They are just as likely to respond to the failure of a product or program by doubling the efforts invested in it. But they are, fortunately, unable to indulge freely in their predilections. They stand under an objective discipline, the discipline of the market. They have an objective outside measurement, profitability.And so they are forced to slough off the unsuccessful and unproductive sooner or later."

And how do you measure the efficiency of an organization? By testing, testing, testing:

"All organizations must be capable of change. We need concepts and measurements that give to other kinds of organizations what the market test and profitability yardstick give to business. Those tests and yardsticks will be quite different."

Cheesecake now including PEP8 checks

The inclusion actually happened a couple of weeks ago. I saw Johann Rocholl's message on conp.lang.python.announce where he talked about his pep8.py module -- a tool which checks Python modules against some of the style conventions in PEP8.

Here's a sample output of running pep8.py against one of the modules in the Cheesecake project. By default, pep8 reports only the first occurrence of the error or warning. The numbers after the file name represent the line and column where the error/warning occurred:

$ python pep8.py logger.py
logger.py:1:11: E401 multiple imports on one line
logger.py:7:23: W291 trailing whitespace
logger.py:8:5: E301 expected 1 blank line, found 0
logger.py:40:33: W602 deprecated form of raising exception
logger.py:60:1: E302 expected 2 blank lines, found 1
logger.py:114:80: E501 line too long (85 characters)

If you want to see all occurrences, use the --repeat flag.

If you just want to see how many lines in a given file have PEP8-related errors/warnings, use the --statistics flag, along with -qq, which quiets the default output:

$ python pep8.py logger.py --statistics -qq
3       E301 expected 1 blank line, found 0
4       E302 expected 2 blank lines, found 1
1       E401 multiple imports on one line
1       E501 line too long (85 characters)
40      W291 trailing whitespace
1       W602 deprecated form of raising exception

You can also pass multiple file and directory names to pep8.py, and it will give you an overall line count when you use the --statistics flag.

So now cheesecake_index.py includes a check for PEP8 compatibility as part of the 'code kwalitee' index. To compute the PEP8 score, it only looks at types of errors and warnings, not at the line count for each type. It subtracts 1 from the code kwalitee score for each warning type reported by pep8, and 2 for each error type reported. Johann told me he'll try to come up with a scoring scheme within the pep8 module, so when that's ready I'll just use it instead of my ad-hoc one. Kudos to Johann for creating a very useful module.