Thursday, January 27, 2005

Python unit testing part 1: the unittest module

Python developers who are serious about testing their code are fortunate to have a choice between at least three unit test frameworks: unittest, doctest and py.test. I'll discuss these frameworks and I'll focus on features such as availability, ease of use, API complexity, test execution customization, test fixture management, test reuse and organization, assertion syntax, dealing with exceptions. This post is the first in a series of three. It discusses the unittest module.

The SUT (software under test) I'll use in this discussion is a simple Blog management application, based on the Universal Feed Parser Python module written by Mark Pilgrim of Dive Into Python fame. I discussed this application in a previous PyFIT-related post. I implemented the blog management functionality in a module called Blogger (all the source code used in this discussion can be found here.)

unittest

Availability

The unittest module (called PyUnit by Steve Purcell, its author) has been part of the Python standard library since version 2.1.

Ease of use

Since unittest is based on jUnit, people familiar with the xUnit framework will have no difficulty picking up the unittest API. Due to the jUnit heritage, some Python pundits consider that unittest is too much "java-esque" and not enough "pythonic". I think the opinions are split though. I tried to initiate a discussion on this topic at comp.lang.python, but I didn't have much success.

API complexity

The canonical way of writing unittest tests is to derive a test class from unittest.TestCase. The test class exists in its own module, separate from the module containing the SUT. Here is a short example of a test class for the Blogger module. I saved the following in a file called unittest_blogger.py:

import unittest
import Blogger

class testBlogger(unittest.TestCase):
"""
A test class for the Blogger module.
"""

def setUp(self):
"""
set up data used in the tests.
setUp is called before each test function execution.
"""
self.blogger = Blogger.get_blog()

def testGetFeedTitle(self):
title = "fitnessetesting"
self.assertEqual(self.blogger.get_title(), title)

def testGetFeedPostingURL(self):
posting_url = "http://www.blogger.com/atom/9276918"
self.assertEqual(self.blogger.get_feed_posting_url(), posting_url)

def testGetFeedPostingHost(self):
posting_host = "www.blogger.com"
self.assertEqual(self.blogger.get_feed_posting_host(), posting_host)

def testPostNewEntry(self):
init_num_entries = self.blogger.get_num_entries()
title = "testPostNewEntry"
content = "testPostNewEntry"
self.assertTrue(self.blogger.post_new_entry(title, content))
self.assertEqual(self.blogger.get_num_entries(), init_num_entries+1)
# Entries are ordered most-recent first
# Newest entry should be first
self.assertEqual(title, self.blogger.get_nth_entry_title(1))
self.assertEqual(content, self.blogger.get_nth_entry_content_strip_html(1))

def testDeleteAllEntries(self):
self.blogger.delete_all_entries()
self.assertEqual(self.blogger.get_num_entries(), 0)

def suite():
suite = unittest.TestSuite()
suite.addTest(unittest.makeSuite(testBlogger))
return suite

if __name__ == '__main__':
#unittest.main()

suiteFew = unittest.TestSuite()
suiteFew.addTest(testBlogger("testPostNewEntry"))
suiteFew.addTest(testBlogger("testDeleteAllEntries"))
#unittest.TextTestRunner(verbosity=2).run(suiteFew)
unittest.TextTestRunner(verbosity=2).run(suite())
A few API-related things to note in the code above:
  • test method names that start with "test" are automatically invoked by the framework
  • each test method is executed independently from all other methods
  • unittest.TestCase provides a setUp method for setting up the fixture, and a tearDown method for doing necessary clean-up
    • setUp is automatically called by TestCase before any other test method is invoked
    • tearDown is automatically called by TestCase after all other test methods have been invoked
  • unittest.TestCase also provides custom assertions (for example assertEqual, assertTrue assertNotEqual) that generate more meaningful error messages than the default Python assertions
All in all, not a huge number of APIs to remember, but it's enough to draw some people away from using unittest. For example, in his PyCon 2004 presentation, Jim Fulton complains that unittest has too much support for abstraction, which makes the test code intent's less clear, while making it look too different from the SUT code.

Test execution customization

The canonical way of running tests in unittest is to include this code at the end of the module containing the test class:

if __name__ == '__main__':
unittest.main()

By default, unittest.main() builds a TestSuite object containing all the tests whose method names start with "test", then it invokes a TextTestRunner which executes each test method and prints the results to stderr.

Let's try it with unittest_blogger:

# python unittest_blogger.py
.....
----------------------------------------------------------------------
Ran 5 tests in 10.245s

OK

The default output is pretty terse. Verbosity can be increased by passing a -v flag at the command line:

# python unittest_blogger.py -v
testDeleteAllEntries (__main__.testBlogger) ... ok
testGetFeedPostingHost (__main__.testBlogger) ... ok
testGetFeedPostingURL (__main__.testBlogger) ... ok
testGetFeedTitle (__main__.testBlogger) ... ok
testPostNewEntry (__main__.testBlogger) ... ok


----------------------------------------------------------------------
Ran 5 tests in 17.958s

OK

One note here: the order in which the tests are run is based on the alphanumerical order of their names, which can be sometimes annoying.

Individual test cases can be run by simply specifying their names (prefixed by the test class name) on the command line:

# python unittest_blogger.py testBlogger.testGetFeedPostingHost testBlogger.testGetFeedPostingURL
..
----------------------------------------------------------------------
Ran 2 tests in 0.053s

OK
In conclusion, it's fair to say that unittest offers a lot of flexibility in test case execution.

Test fixture management

I already mentioned that unittest.TestCase provides the setUp and tearDown methods that can be used in derived test classes in order to create/destroy "test fixtures", i.e. environments were data is set up so that each test method can act on it in isolation from all other test methods. In general, the setUp/tearDown methods are used for creating/destroying database connections, opening/closing files and other operations that need to maintain state during the test run.

In my unittest_blogger example, I'm using the setUp method for creating a Blogger object that can then be referenced by all test methods in my test class. Note that setUp and tearDown are called by the unittest framework before and after each test method is called. This ensures test independence, so that data created by a test does not interfere with data used by another test.

Test organization and reuse

The unittest framework makes it easy to aggregate individual tests into test suites. There are several ways to create test suites. The easiest way is similar to this:

def suite():
suite = unittest.TestSuite()
suite.addTest(unittest.makeSuite(testBlogger))
return suite
Here I created a TestSuite object, then I used the makeSuite helper function to build a test suite out of all tests whose names start with "test". I added the resulting suite to the initial TestSuite object via the addTest method.

A suite can also be created from individual tests by calling addTest with the name of the test method (which in this case does not have to start with test). Here is a fragment from unittest_blogger.py:

suiteFew = unittest.TestSuite()
suiteFew.addTest(testBlogger("testPostNewEntry"))
suiteFew.addTest(testBlogger("testDeleteAllEntries"))

In order to run a given suite, I used a TextTestRunner object:

unittest.TextTestRunner().run(suiteFew)
unittest.TextTestRunner(verbosity=2).run(suite())
The first line runs a TextTestRunner with the default terse output and using the suiteFew suite, which contains only 2 tests.

The second line increases the verbosity of the output, then runs the suite returned by the suite() method, which contains all tests starting with "test" (and all of them do in my example).

The suite mechanism also allows for test reuse across modules. Say for example that I have another test class, which tests some properties of the sort() method. I saved the following in a file called unitest_sort.py:
import unittest


class TestSort(unittest.TestCase):

def setUp(self):
self.alist = [5, 2, 3, 1, 4]

def test_ascending_sort(self):
self.alist.sort()
self.assertEqual(self.alist, [1, 2, 3, 4, 5])

def test_custom_sort(self):
def int_compare(x, y):
x = int(x)
y = int(y)
return x - y
self.alist.sort(int_compare)
self.assertEqual(self.alist, [1, 2, 3, 4, 5])

b = ["1", "2", "10", "20", "100"]
b.sort()
self.assertEqual(b, ['1', '10', '100', '2', '20'])
b.sort(int_compare)
self.assertEqual(b, ['1', '2', '10', '20', '100'])

def test_sort_reverse(self):
self.alist.sort()
self.alist.reverse()
self.assertEqual(self.alist, [5, 4, 3, 2, 1])

def test_sort_exception(self):
try:
self.alist.sort(int_compare)
except NameError:
pass
else:
fail("Expected a NameError")
self.assertRaises(ValueError, self.alist.remove, 6)

def suite():
suite = unittest.TestSuite()
suite.addTest(unittest.makeSuite(TestSort))
return suite

if __name__ == "__main__":
unittest.main()
I can now run the tests in both unittest_blogger and unittest_sort by means of a test suite that aggregates the test suites defined in each of the 2 modules:

# cat unittest_aggregate.py
import unittest
import unittest_sort
import unittest_blogger

suite1 = unittest_sort.suite()
suite2 = unittest_blogger.suite()

suite = unittest.TestSuite()
suite.addTest(suite1)
suite.addTest(suite2)
unittest.TextTestRunner(verbosity=2).run(suite)

# python unittest_aggregate.py
test_ascending_sort (unittest_sort.TestSort) ... ok
test_custom_sort (unittest_sort.TestSort) ... ok
test_sort_exception (unittest_sort.TestSort) ... ok
test_sort_reverse (unittest_sort.TestSort) ... ok
testDeleteAllEntries (unittest_blogger.testBlogger) ... ok
testGetFeedPostingHost (unittest_blogger.testBlogger) ... ok
testGetFeedPostingURL (unittest_blogger.testBlogger) ... ok
testGetFeedTitle (unittest_blogger.testBlogger) ... ok
testPostNewEntry (unittest_blogger.testBlogger) ... ok

----------------------------------------------------------------------
Ran 9 tests in 17.873s

OK
Assertion syntax

As I said previously, unittest provides its own custom assertions. Here are some of the reasons for this choice:
  • if tests are run with the optimization option turned on, the standard Python assert statements will be skipped; for this reason, unittest provides the assert_ statement, equivalent with the standard assert, but that will not be optimized away
  • the output of the standard Python assert statements does not show the expected and actual values that are compared
For example, let's make the testDeleteAllEntries test fail by comparing the value of get_num_entries() with 1 instead of 0:

def testDeleteAllEntries(self):
self.blogger.delete_all_entries()
self.assertEqual(self.blogger.get_num_entries(), 1)
Running the test will produce a failure:
# python unittest_blogger.py testBlogger.testDeleteAllEntries

F
======================================================================
FAIL: testDeleteAllEntries (__main__.testBlogger)
----------------------------------------------------------------------
Traceback (most recent call last):
File "unittest_blogger.py", line 42, in testDeleteAllEntries
self.assertEqual(self.blogger.get_num_entries(), 1)
AssertionError: 0 != 1

----------------------------------------------------------------------
Ran 1 test in 0.082s

FAILED (failures=1)
The output of the AssertionError is enhanced with the values being compared: 0 != 1. Now instead of assertEqual let's use assert_:

def testDeleteAllEntries(self):
self.blogger.delete_all_entries()
self.assert_ self.blogger.get_num_entries() == 1
The output is now:
# python unittest_blogger.py testBlogger.testDeleteAllEntries

F
======================================================================
FAIL: testDeleteAllEntries (__main__.testBlogger)
----------------------------------------------------------------------
Traceback (most recent call last):
File "unittest_blogger.py", line 43, in testDeleteAllEntries
self.assert_(self.blogger.get_num_entries() == 1)
AssertionError

----------------------------------------------------------------------
Ran 1 test in 0.077s

FAILED (failures=1)
There's no indication of what went wrong when the actual and the expected values were compared.

Dealing with exceptions

The unittest_sort module listed above has 2 examples of testing for exceptions. In the test_sort_exception method, I first test that calling sort with an undefined function as a sort function results in a NameError exception. The test will pass only when NameError is raised and will fail otherwise:

try:
self.alist.sort(int_compare)
except NameError:
pass
else:
fail("Expected a NameError")
A more concise way of testing for exceptions is to use the assertRaises statement, passing it the expected exception type and the function/method to be called, followed by its arguments:

self.assertRaises(ValueError, self.alist.remove, 6)
To summarize, here are some Pros and Cons of using the unittest framework.

unittest Pros
  • available in the Python standard library
  • easy to use by people familiar with the xUnit frameworks
  • flexibility in test execution via command-line arguments
  • support for test fixture/state management via set-up/tear-down hooks
  • strong support for test organization and reuse via test suites
unittest Cons
  • xUnit flavor may be too strong for "pure" Pythonistas
  • API can get in the way and can make the test code intent hard to understand
  • tests can end up having a different look-and-feel from the code under test
  • tests are executed in alphanumerical order
  • assertions use custom syntax

10 comments:

Anonymous said...

Great! Especially looking forward to the part on "py.test", what I have seen so far about "py.test" looks very good.

Currently I am using unittest and doctest together (verifying my docstrings becomes one of my unittest tests). Each test gets put into the place where it feels more natural, usually doctest, but if a testing framework is required (loops, setup, teardown, etc.), it goes into unittest, instead of fussing with doctest.

Anonymous said...

Consider using TestOOB (http://testoob.sourceforge.net) which extends Python's unittest module capabilities.

Any unittest/doctest suites work without changes, and it's got some really cool features, like color console output, xml/html reports, and firing up pdb on failed tests.

Anonymous said...

Cons: tests are executed in alphanumerical order

So naming them
test_01_<real_name>
test_02_<real_name2> ...

forces unittest executing tests in defined and easy modificable order. I'm used to do so.

Just my 5 cents

Facility to run test in defined order, but break the run after first failed test lacks to me in unittest. I know how to do it (runnig each test namelly and testing its result) but I don't like it and it's hard to extend.

Anonymous said...

Very interesting article, congratulations. Just what I needed

Carl Trachte said...

Grig,
Needed to refer back to this today (2009). Never gets old. Thanks.
Carl T.

Evans said...

I see you used both unittest and doctest together in your example, but when does one choose a particular testing framework over another? Coming from Java land, JUnit seems to be the defacto for Java.

Is there something like that for Python?

Nice article BTW :)

Anonymous said...

Shouldn't tests be independent of one another, hence the order in which they're run should not matter?

Anonymous said...

It is 2013. This is the first example of unittest in Python that I have actually understood. I now actually understand where everything is supposed to be placed when creating test suites. Thank you, a million times over, thank you!

Unknown said...

If you use a nice test runner like Green, you can omit the entire if..."__main__": section of the code. (Full disclosure, I wrote Green) https://github.com/CleanCut/green

Unknown said...

def tearDown(self):
if self.failed:
return
duration = time.time() - self.startTime_
self.cleanup(True)
if self.reportStatus_:
#self.log.info("=== Test %s completed normally (%d sec)", self.name_, duration

I tried to use this code on tearDown() but it shows me exception on invalid syntax in I guess it is due to time function I have import the library even though why this syntax error is showing please suggest and help.

Modifying EC2 security groups via AWS Lambda functions

One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...