Web app testing with Python part 3: twill

In a recent thread on comp.lang.python, somebody was inquiring about ways to test whether a Web site is up or not from within Python code. Some options were proposed, among which I referred the OP to twill, a Web application testing package written in pure Python by Titus Brown (who, I can proudly say, is a fellow SoCal Piggie).

I recently took the latest version of twill for a ride and I'll report here some of my experiences. My application testing scenario was to test a freshly installed instance of Bugzilla. I wanted to see that I can correctly post bugs and retrieve bugs by bug number. Using twill, all this proved to be a snap.

First, a few words about twill: it's a re-implementation of Cory Dodt's PBP package based on the mechanize module written by John J. Lee. Since mechanize implements the HTTP request/response protocol and parses the resulting HTML, we can categorize twill as a "Web protocol driver" tool (for more details on such taxonomies, see a previous post of mine).

Twill can be used as a domain specific language via a command shell (twill-sh), or it can be used as a normal Python module, from within your Python code. I will show both usage models.

After downloading twill and installing it via the usual "python setup.py install" method, you can start its command line interpreter via the twill-sh script installed in /usr/local/bin. At the interpreter prompt, you can then issue commands such as:

• go -- visit the given URL.
• code -- assert that the last page loaded had this HTTP status, e.g. code 200 asserts that the page loaded fine.
• find -- assert that the page contains this regular expression.
• showforms -- show all of the forms on the page.
• formvalue --- set the given field in the given form to the given value. For read-only form widgets/controls, the click may be recorded for use by submit, but the value is not changed.
• submit [] -- click the n'th submit button, if given; otherwise submit via the last submission button clicked; if nothing clicked, use the first submit button on the form.

Let's see a quick example of the twill shell in action. As I mentioned before, I wanted to test a freshly-installed instance of Bugzilla, namely I wanted to verify that I can add new bugs and then retrieve them via their bug number. Here is a shell session fragment that opens the Bugzilla main page via the go command and clicks on the "Enter a new bug report" link via the follow command:

[ggheo@concord twill-latest]$ twill-sh

-= Welcome to twill! =-

current page: *empty page*
>> go http://example.com/bugs/
==> at http://example.com/bugs/
current page: http://example.com/bugs/
>> follow "Enter a new bug report"
==> at http://example.com/bugs/enter_bug.cgi
current page: http://example.com/bugs/enter_bug.cgi

At this point, we can issue the showforms command to see what forms are available on the current page.

>> showforms
Form #1
## __Name______ __Type___ __ID________ __Value__________________
Bugzilla ... text (None)
Bugzilla ... password (None)
product hidden (None) TestProduct
1 GoAheadA ... submit (None) Login
Form #2
## __Name______ __Type___ __ID________ __Value__________________
a hidden (None) reqpw
loginname text (None)
1 submit (None) Submit Request
Form #3
## __Name______ __Type___ __ID________ __Value__________________
id text (None)
1 submit (None) Find
current page: http://example.com/bugs/enter_bug.cgi

It looks like we're on the login page. We can then use the formvalue (or fv for short) command to fill in the required fields (user name and password), then the submit command in order to complete the log in process. The submit command takes an optional argument -- the number of the submit button you want to click. With no arguments, it activates the first submit button it finds.

>> fv 1 Bugzilla_login grig@example.com
current page: http://example.com/bugs/enter_bug.cgi
>> fv 1 Bugzilla_password mypassword
current page: http://example.com/bugs/enter_bug.cgi
>> submit 1
current page: http://example.com/bugs/enter_bug.cgi

At this point, we can verify that we received the expected HTTP status code (200 when everything was OK) via the code command:

>> code 200
current page: http://example.com/bugs/enter_bug.cgi

We run showforms again to see what forms and fields are available on the current page, then we use fv to fill in a bunch of fields for the new bug we want to enter, and finally we submit the form (note how nicely twill displays the available fields, as well as the first few selections available in drop-down combo boxes) :

>> showforms
Form #1
## __Name______ __Type___ __ID________ __Value__________________
product hidden (None) TestProduct
version select (None) ['other'] of ['other']
component select (None) ['TestComponent'] of ['TestComponent']
rep_platform select (None) ['Other'] of ['All', 'DEC', 'HP', 'M ...
op_sys select (None) ['other'] of ['All', 'Windows 3.1', ...
priority select (None) ['P2'] of ['P1', 'P2', 'P3', 'P4', 'P5']
bug_severity select (None) ['normal'] of ['blocker', 'critical' ...
bug_status hidden (None) NEW
assigned_to text (None)
cc text (None)
bug_file_loc text (None) http://
short_desc text (None)
comment textarea (None)
form_name hidden (None) enter_bug
1 submit (None) Commit
2 maketemplate submit (None) Remember values as bookmarkable template
Form #2
## __Name______ __Type___ __ID________ __Value__________________
id text (None)
1 submit (None) Find
current page: http://example.com/bugs/enter_bug.cgi
>> fv 1 op_sys "Linux"
current page: http://example.com/bugs/enter_bug.cgi
>> fv 1 priority P1
current page: http://example.com/bugs/enter_bug.cgi
>> fv 1 assigned_to grig@example.com
current page: http://example.com/bugs/enter_bug.cgi
>> fv 1 short_desc "twill-generated bug"
current page: http://example.com/bugs/enter_bug.cgi
>> fv 1 comment "This is a new bug opened automatically via twill"
current page: http://example.com/bugs/enter_bug.cgi
>> submit
Note: submit is using submit button: name="None", value=" Commit "
current page: http://example.com/bugs/post_bug.cgi

Now we can verify that the bug with the specified description was posted. We use the find command, which takes a regular expression as an argument:

>> find "Bug \d+ Submitted"
current page: http://example.com/bugs/post_bug.cgi
>> find "twill-generated bug"
current page: http://example.com/bugs/post_bug.cgi

No errors were reported, which means the validations succeeded. At this point, we can also inspect the current page via the show_html command in order to see the bug number that Bugzilla automatically assigned. I won't actually show all the HTML, suffice to say that the bug was assigned number 2. We can then go directly to the page for bug #2 and verify that the various bug elements we indicated were indeed posted correctly:

>> go "http://example.com/bugs/show_bug.cgi?id=2"
==> at http://example.com/bugs/show_bug.cgi?id=2
current page: http://example.com/bugs/show_bug.cgi?id=2
>> find "Linux"
current page: http://example.com/bugs/show_bug.cgi?id=2
>> find "P1"
current page: http://example.com/bugs/show_bug.cgi?id=2
>> find "grig@example.com"
current page: http://example.com/bugs/show_bug.cgi?id=2
>> find "twill-generated bug"
current page: http://example.com/bugs/show_bug.cgi?id=2
>> find "This is a new bug opened automatically via twill"
current page: http://example.com/bugs/show_bug.cgi?id=2

I mentioned that all the commands available in the interactive twill-sh command interpreter are also available as top-level functions to be used inside your Python code. All you need to do is import the necessary functions from the twill.commands module.

Here's how a Python script that tests functionality similar to the one I described above would look like:

#!/usr/bin/env python

from twill.commands import go, follow, showforms, fv, submit, find, code, save_html
import os, time, re

def get_bug_number(html_file):
 h = open(html_file)
 bug_number = "-1"
 for line in h:
     s = re.search("Bug (\d+) Submitted", line)
     if s:
         bug_number = s.group(1)
         break
 return bug_number

# MAIN
crt_time = time.strftime("%Y%m%d%H%M%S", time.localtime())
temp_html = "temp.html"

# Open a new bug report
go("http://www.example.com/bugs")
follow("Enter a new bug report")

# Log in
fv("1", "Bugzilla_login", "grig@example.com")
fv("1", "Bugzilla_password", "mypassword")
submit()
code("200")

# Enter bug info
fv("1", "op_sys", "Linux")
fv("1", "priority", "P1")
fv("1", "assigned_to", "grig@example.com")
fv("1", "short_desc",  "twill-generated bug at " + crt_time)
fv("1", "comment", "This is a new bug opened automatically via twill at " + crt_time)
submit()
code("200")

# Verify bug info
find("Bug \d+ Submitted")
find("twill-generated bug at " + crt_time)

# Get bug number
save_html(temp_html)
bug_number = get_bug_number(temp_html)
os.unlink(temp_html)

assert bug_number != "-1"

# Go to bug page and verify more detauled info
go("http://example.com/bugs/show_bug.cgi?id=" + bug_number)
code("200")
find("P1")
find("Linux")
find("grig@example.com")
find("This is a new bug opened automatically via twill at " + crt_time)

I added some extra functionality to the Python script -- such as adding the current time to the bug description, so that whenever the test script will be run, a different bug description will be inserted into the Bugzilla database (the current time doesn't of course guarantee uniqueness, but it will do for now :-) I also used the save_html function in order to save the "Bug posted" page to a temporary file, so that I can retrieve the bug number and query the individual bug page.

Conclusion

Twill is an excellent tool for testing Web applications. It can also be used to automate form handling, especially for Web sites that require a login. I especially like the fact that everything can be run from the command line -- both the twill shell and the Python scripts based on twill. This means that deploying twill is a snap, and there are no cumbersome GUIs to worry about. The assertion commands built into twill (code, find and notfind) should be enough for testing Web sites that use straight HTML and forms. For more complicated, Javascript-intensive Web sites, a tool such as Selenium might be more appropriate.

I haven't looked into twill's cookie-handling capabilities, but they're available, according to the README. Some more aspects of twill that I haven't experimented with yet:

• Script recording: Titus has written a maxq add-on that can be used to automatically record twill-based scripts while browsing the Web site under test; for more details on maxq, see also a previous post of mine
• Extending twill: you can easily add commands to the twill interpreter

Kudos to Titus for writing a powerful, yet easy to use testing tool.