Python unit testing part 1: the unittest module

Python developers who are serious about testing their code are fortunate to have a choice between at least three unit test frameworks: unittest, doctest and py.test. I'll discuss these frameworks and I'll focus on features such as availability, ease of use, API complexity, test execution customization, test fixture management, test reuse and organization, assertion syntax, dealing with exceptions. This post is the first in a series of three. It discusses the unittest module.

The SUT (software under test) I'll use in this discussion is a simple Blog management application, based on the Universal Feed Parser Python module written by Mark Pilgrim of Dive Into Python fame. I discussed this application in a previous PyFIT-related post. I implemented the blog management functionality in a module called Blogger (all the source code used in this discussion can be found here.)

unittest

Availability

The unittest module (called PyUnit by Steve Purcell, its author) has been part of the Python standard library since version 2.1.

Ease of use

Since unittest is based on jUnit, people familiar with the xUnit framework will have no difficulty picking up the unittest API. Due to the jUnit heritage, some Python pundits consider that unittest is too much "java-esque" and not enough "pythonic". I think the opinions are split though. I tried to initiate a discussion on this topic at comp.lang.python, but I didn't have much success.

API complexity

The canonical way of writing unittest tests is to derive a test class from unittest.TestCase. The test class exists in its own module, separate from the module containing the SUT. Here is a short example of a test class for the Blogger module. I saved the following in a file called unittest_blogger.py:

import unittest

import Blogger



class testBlogger(unittest.TestCase):

    """

    A test class for the Blogger module.

    """



    def setUp(self):

        """

        set up data used in the tests.

        setUp is called before each test function execution.

        """

        self.blogger = Blogger.get_blog()



    def testGetFeedTitle(self):

        title = "fitnessetesting"

        self.assertEqual(self.blogger.get_title(), title)



    def testGetFeedPostingURL(self):

        posting_url = "http://www.blogger.com/atom/9276918"

        self.assertEqual(self.blogger.get_feed_posting_url(), posting_url)



    def testGetFeedPostingHost(self):

        posting_host = "www.blogger.com"

        self.assertEqual(self.blogger.get_feed_posting_host(), posting_host)



    def testPostNewEntry(self):

        init_num_entries = self.blogger.get_num_entries()

        title = "testPostNewEntry"

        content = "testPostNewEntry"

        self.assertTrue(self.blogger.post_new_entry(title, content))

        self.assertEqual(self.blogger.get_num_entries(), init_num_entries+1)

        # Entries are ordered most-recent first

        # Newest entry should be first

        self.assertEqual(title, self.blogger.get_nth_entry_title(1))

        self.assertEqual(content, self.blogger.get_nth_entry_content_strip_html(1))



    def testDeleteAllEntries(self):

        self.blogger.delete_all_entries()

        self.assertEqual(self.blogger.get_num_entries(), 0)



def suite():

    suite = unittest.TestSuite()

    suite.addTest(unittest.makeSuite(testBlogger))

    return suite



if __name__ == '__main__':

    #unittest.main()



    suiteFew = unittest.TestSuite()

    suiteFew.addTest(testBlogger("testPostNewEntry"))

    suiteFew.addTest(testBlogger("testDeleteAllEntries"))

    #unittest.TextTestRunner(verbosity=2).run(suiteFew)

    unittest.TextTestRunner(verbosity=2).run(suite())

A few API-related things to note in the code above:

• test method names that start with "test" are automatically invoked by the framework
  
• each test method is executed independently from all other methods
  
• unittest.TestCase provides a setUp method for setting up the fixture, and a tearDown method for doing necessary clean-up
• setUp is automatically called by TestCase before any other test method is invoked
• tearDown is automatically called by TestCase after all other test methods have been invoked
• unittest.TestCase also provides custom assertions (for example assertEqual, assertTrue assertNotEqual) that generate more meaningful error messages than the default Python assertions

All in all, not a huge number of APIs to remember, but it's enough to draw some people away from using unittest. For example, in his PyCon 2004 presentation, Jim Fulton complains that unittest has too much support for abstraction, which makes the test code intent's less clear, while making it look too different from the SUT code.

Test execution customization

The canonical way of running tests in unittest is to include this code at the end of the module containing the test class:

if __name__ == '__main__':

    unittest.main()

By default, unittest.main() builds a TestSuite object containing all the tests whose method names start with "test", then it invokes a TextTestRunner which executes each test method and prints the results to stderr.

Let's try it with unittest_blogger:

# python unittest_blogger.py

.....

----------------------------------------------------------------------

Ran 5 tests in 10.245s



OK

The default output is pretty terse. Verbosity can be increased by passing a -v flag at the command line:

# python unittest_blogger.py -v

testDeleteAllEntries (__main__.testBlogger) ... ok

testGetFeedPostingHost (__main__.testBlogger) ... ok

testGetFeedPostingURL (__main__.testBlogger) ... ok

testGetFeedTitle (__main__.testBlogger) ... ok

testPostNewEntry (__main__.testBlogger) ... ok





----------------------------------------------------------------------

Ran 5 tests in 17.958s



OK

One note here: the order in which the tests are run is based on the alphanumerical order of their names, which can be sometimes annoying.

Individual test cases can be run by simply specifying their names (prefixed by the test class name) on the command line:

# python unittest_blogger.py testBlogger.testGetFeedPostingHost testBlogger.testGetFeedPostingURL

..

----------------------------------------------------------------------

Ran 2 tests in 0.053s



OK

In conclusion, it's fair to say that unittest offers a lot of flexibility in test case execution.

Test fixture management

I already mentioned that unittest.TestCase provides the setUp and tearDown methods that can be used in derived test classes in order to create/destroy "test fixtures", i.e. environments were data is set up so that each test method can act on it in isolation from all other test methods. In general, the setUp/tearDown methods are used for creating/destroying database connections, opening/closing files and other operations that need to maintain state during the test run.

In my unittest_blogger example, I'm using the setUp method for creating a Blogger object that can then be referenced by all test methods in my test class. Note that setUp and tearDown are called by the unittest framework before and after each test method is called. This ensures test independence, so that data created by a test does not interfere with data used by another test.

Test organization and reuse

The unittest framework makes it easy to aggregate individual tests into test suites. There are several ways to create test suites. The easiest way is similar to this:

def suite():

    suite = unittest.TestSuite()

    suite.addTest(unittest.makeSuite(testBlogger))

    return suite

Here I created a TestSuite object, then I used the makeSuite helper function to build a test suite out of all tests whose names start with "test". I added the resulting suite to the initial TestSuite object via the addTest method.

A suite can also be created from individual tests by calling addTest with the name of the test method (which in this case does not have to start with test). Here is a fragment from unittest_blogger.py:

suiteFew = unittest.TestSuite()

suiteFew.addTest(testBlogger("testPostNewEntry"))

suiteFew.addTest(testBlogger("testDeleteAllEntries"))

In order to run a given suite, I used a TextTestRunner object:

unittest.TextTestRunner().run(suiteFew)

unittest.TextTestRunner(verbosity=2).run(suite())

The first line runs a TextTestRunner with the default terse output and using the suiteFew suite, which contains only 2 tests.

The second line increases the verbosity of the output, then runs the suite returned by the suite() method, which contains all tests starting with "test" (and all of them do in my example).

The suite mechanism also allows for test reuse across modules. Say for example that I have another test class, which tests some properties of the sort() method. I saved the following in a file called unitest_sort.py:

import unittest



class TestSort(unittest.TestCase):



   def setUp(self):

       self.alist = [5, 2, 3, 1, 4]



   def test_ascending_sort(self):

       self.alist.sort()

       self.assertEqual(self.alist, [1, 2, 3, 4, 5])



   def test_custom_sort(self):

       def int_compare(x, y):

           x = int(x)

           y = int(y)

           return x - y

       self.alist.sort(int_compare)

       self.assertEqual(self.alist, [1, 2, 3, 4, 5])



       b = ["1", "2", "10", "20", "100"]

       b.sort()

       self.assertEqual(b, ['1', '10', '100', '2', '20'])

       b.sort(int_compare)

       self.assertEqual(b, ['1', '2', '10', '20', '100'])



   def test_sort_reverse(self):

       self.alist.sort()

       self.alist.reverse()

       self.assertEqual(self.alist, [5, 4, 3, 2, 1])



   def test_sort_exception(self):

       try:

           self.alist.sort(int_compare)

       except NameError:

           pass

       else:

           fail("Expected a NameError")

       self.assertRaises(ValueError, self.alist.remove, 6)



def suite():

   suite = unittest.TestSuite()

   suite.addTest(unittest.makeSuite(TestSort))

   return suite



if __name__ == "__main__":

   unittest.main()

I can now run the tests in both unittest_blogger and unittest_sort by means of a test suite that aggregates the test suites defined in each of the 2 modules:

# cat unittest_aggregate.py

import unittest

import unittest_sort

import unittest_blogger



suite1 = unittest_sort.suite()

suite2 = unittest_blogger.suite()



suite = unittest.TestSuite()

suite.addTest(suite1)

suite.addTest(suite2)

unittest.TextTestRunner(verbosity=2).run(suite)



# python unittest_aggregate.py

test_ascending_sort (unittest_sort.TestSort) ... ok

test_custom_sort (unittest_sort.TestSort) ... ok

test_sort_exception (unittest_sort.TestSort) ... ok

test_sort_reverse (unittest_sort.TestSort) ... ok

testDeleteAllEntries (unittest_blogger.testBlogger) ... ok

testGetFeedPostingHost (unittest_blogger.testBlogger) ... ok

testGetFeedPostingURL (unittest_blogger.testBlogger) ... ok

testGetFeedTitle (unittest_blogger.testBlogger) ... ok

testPostNewEntry (unittest_blogger.testBlogger) ... ok



----------------------------------------------------------------------

Ran 9 tests in 17.873s



OK

Assertion syntax

As I said previously, unittest provides its own custom assertions. Here are some of the reasons for this choice:

• if tests are run with the optimization option turned on, the standard Python assert statements will be skipped; for this reason, unittest provides the assert_ statement, equivalent with the standard assert, but that will not be optimized away
  
• the output of the standard Python assert statements does not show the expected and actual values that are compared
  

For example, let's make the testDeleteAllEntries test fail by comparing the value of get_num_entries() with 1 instead of 0:

def testDeleteAllEntries(self):

    self.blogger.delete_all_entries()

    self.assertEqual(self.blogger.get_num_entries(), 1)

Running the test will produce a failure:

# python unittest_blogger.py testBlogger.testDeleteAllEntries

F

======================================================================

FAIL: testDeleteAllEntries (__main__.testBlogger)

----------------------------------------------------------------------

Traceback (most recent call last):

File "unittest_blogger.py", line 42, in testDeleteAllEntries

self.assertEqual(self.blogger.get_num_entries(), 1)

AssertionError: 0 != 1



----------------------------------------------------------------------

Ran 1 test in 0.082s



FAILED (failures=1)

The output of the AssertionError is enhanced with the values being compared: 0 != 1. Now instead of assertEqual let's use assert_:

def testDeleteAllEntries(self):

    self.blogger.delete_all_entries()

    self.assert_ self.blogger.get_num_entries() == 1

The output is now:

# python unittest_blogger.py testBlogger.testDeleteAllEntries

F

======================================================================

FAIL: testDeleteAllEntries (__main__.testBlogger)

----------------------------------------------------------------------

Traceback (most recent call last):

File "unittest_blogger.py", line 43, in testDeleteAllEntries

self.assert_(self.blogger.get_num_entries() == 1)

AssertionError



----------------------------------------------------------------------

Ran 1 test in 0.077s



FAILED (failures=1)

There's no indication of what went wrong when the actual and the expected values were compared.

Dealing with exceptions

The unittest_sort module listed above has 2 examples of testing for exceptions. In the test_sort_exception method, I first test that calling sort with an undefined function as a sort function results in a NameError exception. The test will pass only when NameError is raised and will fail otherwise:

try:

    self.alist.sort(int_compare)

except NameError:

    pass

else:

    fail("Expected a NameError")

A more concise way of testing for exceptions is to use the assertRaises statement, passing it the expected exception type and the function/method to be called, followed by its arguments:

self.assertRaises(ValueError, self.alist.remove, 6)

To summarize, here are some Pros and Cons of using the unittest framework.

unittest Pros

• available in the Python standard library
• easy to use by people familiar with the xUnit frameworks
• flexibility in test execution via command-line arguments
• support for test fixture/state management via set-up/tear-down hooks
  
• strong support for test organization and reuse via test suites

unittest Cons

• xUnit flavor may be too strong for "pure" Pythonistas
• API can get in the way and can make the test code intent hard to understand
  
• tests can end up having a different look-and-feel from the code under test
• tests are executed in alphanumerical order
• assertions use custom syntax