Agile Documentation with doctest and epydoc

This post was inspired by an article I read in the Feb. 2005 issue of Better Software: "Double Duty" by Brian Button. The title refers to having unit tests serve the double role of testing and documentation. Brian calls this Agile Documentation. For Python developers, this is old news, since the doctest module already provides what is called "literate testing" or "executable documentation". However, Brian also introduces some concepts that I think are worth exploring: Test Lists and Tests Maps.

Test Lists

A Test List tells a story about the behavior expected from the module/class under test. It is composed of one-liners, each line describing what a specific unit test tries to achieve. For example, in the case of a Blog management application, you could have the following (incomplete) Test List:

• Deleting all entries results in no entries in the blog.
• Posting single entry results in single valid entry.
• Deleting a single entry by index results in no entries in the blog.
• Posting new entry results in valid entry and increases the number of entries by 1.
• Etc.

I find it very valuable to have such a Test List for every Python module that I write, especially if the list is easy to generate from the unit tests that I write. I will show later in this post how the combination of doctest and epydoc makes it trivial to achieve this goal.

Test Maps

A Test Map is a list of unit tests associated with a specific function/method under test. It helps you see how that specific function/method is being exercised via unit tests. A Test Map could look like this:

Testmap for method delete_all_entries:

• test_delete_all_entries
• test_delete_single_entry
• test_post_single_entry
• test_post_two_entries
• test_delete_first_of_two_entries
• test_delete_second_of_two_entries

Generating Test Lists

As an example of a module under test, I will use the Blog management application that I discussed in several previous posts. The source code can be found here. I have a directory called blogmgmt which contains a module called blogger.py. The blogger module contains several classes, the main one being Blogger, and a top-level function called get_blog. I also created an empty __init__.py file, so that blogmgmt can be treated as a package. I wrote a series of doctest-based tests for the blogger module in a file I called testlist_blogger.py. Here is part of that file:

"""
Doctest unit tests for module L{blogger}
"""

def test_get_blog():
  """
  get_blog() mimics a singleton by always returning the same object.

  Function(s) tested:
       - L{blogger.get_blog}


  >>> from blogger import get_blog
  >>> blog1 = get_blog()
  >>> blog2 = get_blog()
  >>> id(blog1) == id(blog2)
  True

  """

def test_get_feed_title():
  """
  Can retrieve the feed title.

  Method(s) tested:
     - L{blogger.Blogger.get_title}

  >>> from blogger import get_blog
  >>> blog = get_blog()
  >>> print blog.get_title()
  fitnessetesting

  """

def test_delete_all_entries():
  """
  Deleting all entries results in no entries in the blog.

  Method(s) tested:
     - L{blogger.Blogger.delete_all_entries}
     - L{blogger.Blogger.get_num_entries}

  >>> from blogger import get_blog
  >>> blog = get_blog()
  >>> blog.delete_all_entries()
  >>> print blog.get_num_entries()
  0

  """

def test_post_new_entry():
  """
  Posting new entry results in valid entry and increases the number of entries by 1.

  Method(s) tested:
     - L{blogger.Blogger.post_new_entry}
     - L{blogger.Blogger.get_nth_entry_title}
     - L{blogger.Blogger.get_nth_entry_content_strip_html}
     - L{blogger.Blogger.get_num_entries}

  >>> from blogger import get_blog
  >>> blog = get_blog()
  >>> init_num_entries = blog.get_num_entries()
  >>> rc = blog.post_new_entry("Test title", "Test content")
  >>> print rc
  True
  >>> print blog.get_nth_entry_title(1)
  Test title
  >>> print blog.get_nth_entry_content_strip_html(1)
  Test content
  >>> num_entries = blog.get_num_entries()
  >>> num_entries == init_num_entries + 1
  True

  """

Each unit test function is composed of a docstring and nothing else. The docstring starts with a one-line description of what the unit test tries to achieve. The docstring continues with a list of methods/functions tested by that unit test. Finally, the interactive shell session output is copied and pasted into the docstring so that it can be processed by doctest.

For the purpose of generating a Test List, only the first line in each docstring is important. If you simply run

epydoc -o blogmgmt testlist_blogger.py

you will get a directory called blogmgmt that contains the epydoc-generated documentation. I usually then move this directory somewhere under the DocumentRoot of one of my Apache Virtual Servers. When viewed in a browser, this is what the epydoc page for the summary of the testlist_blogger module looks like this (also available here):

Module blogmgmt.testlist_blogger

Doctest unit tests for module blogger

---

Function Summary
	test_delete_all_entries() Deleting all entries results in no entries in the blog.
	test_delete_first_of_two_entries() Posting two entries and deleting entry with index 1 leaves oldest entry in place.
	test_delete_second_of_two_entries() Posting two entries and deleting entry with index 2 leaves newest entry in place.
	test_delete_single_entry() Deleting a single entry by index results in no entries in the blog.
	test_get_blog() get_blog() mimics a singleton by always returning the same object.
	test_get_feed_posting_host() Can retrieve the feed posting host.
	test_get_feed_posting_url() Can retrieve the feed posting URL.
	test_get_feed_title() Can retrieve the feed title.
	test_post_new_entry() Posting new entry results in valid entry and increases the number of entries by 1.
	test_post_single_entry() Posting single entry results in single valid entry.
	test_post_two_entries() Posting two entries results in 2 valid entries ordered most recent first.

This is exactly the Test List we wanted. Note that epydoc dutifully generated it for us, since in the Function Summary section it shows the name of every function it finds, plus the first line of that function's docstring. The main value of this Test List for me is that anybody can see at a glance what the methods of the Blogger class are expected to do. It's a nice summary of expected class behavior that enhances the documentation.

So all you need to do to get a nicely formatted Test List is to make sure that you have the test description as the first line of the unit test's docstring; epydoc will then do the grungy work for you.

If you click on the link with the function name on it, you will go to the Function Detail section and witness the power of doctest/epydoc. Since all the tests are copied and pasted from an interactive session and included in the docstring, epydoc will format the docstring very nicely and it will even color-code the blocks of code. Here is an example of the detail for test_delete_all_entries.

Generating Test Maps

Each docstring in the testlist_blogger module contains lines such as these:

Method(s) tested:
  - L{blogger.Blogger.post_new_entry}
  - L{blogger.Blogger.get_nth_entry_title}
  - L{blogger.Blogger.get_nth_entry_content_strip_html}
  - L{blogger.Blogger.get_num_entries}

(the L{...} notation is epydoc-specific and represents a link to another object in the epydoc-generated documentation)

The way I wrote the unit tests, each of them actually exercises several functions/methods from the blogger module. Some unit test purists might think these are not "real" unit tests, but in practice I found it is easier to work this way. For example, the get_blog function is called by each and every unit test in order to retrieve the same "blog" object. However, I am not specifically testing get_blog in every unit test, only calling it as a helper function. The way I see it, a method is tested when there is an assertion made about its behavior. All the other methods are merely called as helpers.

So whenever I write a unit test, I manually specify the list of methods/functions under test. This makes it easy to then parse the testlist file and build a mapping from each function/method under test to a list of unit tests that test it, i.e. what we called the Test Map.

For example, in the testlist_blogger module, the Blogger.delete_all_entries method is listed in the docstrings of 6 unit tests: test_delete_all_entries, test_delete_single_entry, test_post_single_entry, test_post_two_entries, test_delete_first_of_two_entries, test_delete_second_of_two_entries. These 6 unit test represent the Test Map for Blogger.delete_all_entries. It's easy to build the Test Map programatically by parsing the testlist_blogger.py file and creating a Python dictionary having the methods under tests as keys and the unit test lists corresponding to them as values.

An issue I had while putting this together was how to link a method in the Blogger class (for example Blogger.delete_all_entries) to its Test Map. One way would have been to programatically insert the Test Map into the docstring for that method. But this would mean that every time a new unit test is added that tests that method, the Test Map will change and thus the module containing the Blogger class will get changed. This is unfortunate especially when the files are under source control. I think a better solution, and the one I ended up implementing, is to have a third module called for example testmap_blogger that will be automatically generated from testlist_blogger. A method M in the Blogger class will then link to a single function in testmap_blogger. That function will contain in its docstring the Test Map for the Blogger method M.

Again, an example to make all this clearer. Here is the docstring of the Blogger.delete_all_entries method in the blogger module:

"""
Delete all entries in the blog

Test map (set of unit tests that exercise this method):
  - L{testmap_blogger.testmap_Blogger_delete_all_entries}
"""

Here is the epydoc-generated documentation for the Blogger.delete_all_entries method (in the Method Details section):

delete_all_entries(self)

Delete all entries in the blog

Test map (set of unit tests that exercise this method):

• testmap_blogger.testmap_Blogger_delete_all_entries

I manually inserted in the docstring an epydoc link to a function called testmap_Blogger_delete_all_entries in a module called testmap_blogger. Assuming that the testmap_blogger module was already generated and epydoc-documented, clicking on the link will bring up the epydoc detail for that particular function, which contains the 6 unit tests for te delete_all_entries method:

testmap_Blogger_delete_all_entries()

Testmap for blogger.Blogger.delete_all_entries:

• testlist_blogger.test_delete_all_entries
• testlist_blogger.test_delete_single_entry
• testlist_blogger.test_post_single_entry
• testlist_blogger.test_post_two_entries
• testlist_blogger.test_delete_first_of_two_entries
• testlist_blogger.test_delete_second_of_two_entries

Here is the programatically-generated testmap_blogger.py file.

To have all this mechanism work, I use some naming conventions:

• The module containing the Test Maps for module blogger is called testmap_blogger
• In testmap_blogger, the function containing the Test Map for method Blogger.M from the blogger module is called testmap_Blogger_M
• In testmap_blogger, the function containing the Test Map for function F from the blogger module is called testmap_F
• In the docstring of the testmap function itself there is a link which points back to the method Blogger.M; the name of the link needs to be blogger.Blogger.M, otherwise epydoc will not find it

Here's an end-to-end procedure for using the doctest/epydoc combination to write Agile Documentation:

1. We'll unit test a Python module we'll call P which contains a class C.

2. We start by writing a unit test for the method C.M1 from the P module. We write the unit test by copying and pasting a Python shell session output in another Python module called testlist_P. We call the unit test function test_M1. It looks something like this:

def test_M1():
  """
  Short description of the behavior we're testing for M1.

  Method(s) tested:
      - L{P.C.M1}

  >>> from P import C
  >>> c = C()
  >>> rc = c.M1()
  >>> print rc
  True

  """

The testlist_P module has a "main" section of the form:

if __name__ == "__main__":
  import doctest
  doctest.testmod()

This is the typical doctest way of running unit tests. To actually execute the tests, we need to run "python testlist_P.py" at a command line (for more details on doctest, see a previous blog post).

3. At this point, we fleshed out an initial implementation for method M1 in module P. In its docstring, we add a link to the test map:

def M1(self):
  """
  Short description of M1

  Test map (set of unit tests that exercise this method):
      - L{testmap_P.testmap_C_M1}

  """

Note that I followed the naming convention I described earlier.

4. We programatically generate the Test Map for module P by running something like this: build_testmap.py. It will create a file called testmap_P.py with the following content:

def testmap_C_M1():
  """
  Testmap for L{P.C.M1}:

      - L{testlist_P.test_M1}
"""

5. We run epydoc:

epydoc -o P_docs P.py testlist_P.py testmap_P.py

A directory called P_docs will be generated; we can move this directory to a public area of our Web server and thus make the documentation available online. When we click on the testlist_P
module link, we will see the Test List for module P. It will show something like:

Module P_docs.testlist_P

Doctest unit tests for module P

---

Function Summary
	test_M1() Short description of the behavior we're testing for M1.

When we click on the test map link inside the docstring of method C.M1, we see:

testmap_C_M1()

Testmap for P.C.M1:

• testlist_P.test_M1

Now repeat steps 2-5 for method M2:

6. Let's assume we now unit test method M2, but in the process we also test method M1. The function test_M2 will look something like this:

def test_M2():
  """
  Short description of the behavior we're testing for M2.

  Method(s) tested:
      - L{P.C.M1}
      - L{P.C.M2}

  >>> from P import C
  >>> c = C()
  >>> rc = c.M1()
  >>> print rc
  True
  >>> rc = c.M2()
  >>> print rc
  True

  """

We listed both methods in the "Method(s) tested" section.

7. We add a link to the testmap in method M2's docstring (in module P):

def M2(self):
  """
  Short description of M2

  Test map (set of unit tests that exercise this method):
      - L{testmap_P.testmap_C_M2}
  """

8. We recreate the testmap_P file by running build_testmap.py. The testmap for M1 will now contain 2 functions: test_M1 and test_M2, while the testmap for M2 will contain test_M2:

def testmap_C_M1():
  """
  Testmap for L{P.C.M1}:

      - L{testlist_P.test_M1}
      - L{testlist_P.test_M2}
  """

def testmap_C_M2():
  """
  Testmap for L{P.C.M2}:

      - L{testlist_P.test_M2}
  """

9. We run epydoc again:

epydoc -o P_docs P.py testlist_P.py testmap_P.py

Now clicking on testlist_P will show:

Module P_docs.testlist_P

Doctest unit tests for module P

---

Function Summary
	test_M1() Short description of the behavior we're testing for M1.

test_M2() Short description of the behavior we're testing for M2.

Clicking on the test map link inside the docstring of method C.M1 shows:

testmap_C_M1()

Testmap for P.C.M1:

• testlist_P.test_M1
• testlist_P.test_M2

10. Repeat steps 2-5 for each unit test that you add to the testlist_P module.

Conclusion

I find the combination doctest/epydoc very powerful and easy to use in generating Agile Documentation, or "literate testing", or "executable documentation", or however you want to call it. The name is not important, but what you can achieve with it is: a way of documenting your APIs by means of unit tests that live in your code as docstrings. It doesn't get much more "agile" than this. Kudos to the doctest developers and to Edward Loper, the author of epydoc. Also, kudos to Brian Button for his insightful article which inspired my post. Brian's examples used .NET, but hopefully he'll switch to Python soon :-)

If you want to see the full documentation I generated for my blogmgmt package, you can find it here.