Web app testing with Python part 2: Selenium and Twisted

In a previous post I mentioned Selenium as a Web app testing tool that is like no other in terms of functionality and implementation. I've been experimenting with Selenium for the past few days and I'm very impressed (a reaction which seems to be common to everybody who witnessed the tool in action.)

The main Selenium developer is Jason Huggins, who initially created Selenium as a tool for acceptance/functional testing of Web sites based on Plone. Jason is the author of two Python implementations for Selenium:

• the "Plone Product" version
• written as a Plone product,
  
• it is used specifically for testing Plone-based sites
  
• it needs to be installed on the Plone site under test
• the "Twisted Server" version
• written as a stand-alone Twisted-based server
  
• can be used for testing any Web site

This post is a tutorial in using the "Twisted Server" version of Selenium.

Update 3/31/04: For specifics on running the Twisted Server version of Selenium on Unix-like systems, as well as tips to work around CGI errors, please scroll to the end of this post.

Before I delve into the specifics of installing and running Selenium on Twisted, I'll discuss some of the goals and novel ideas of Selenium. Here are Jason's own words:

"The key concept to know about Selenium is that it uses a real, living and breathing web browser to play back your testing scripts. This means it can test client-side browser-specific things like JavaScript that other testing frameworks like Mechanize or HTTPUnit cannot test. Whereas the best that Mechanize can do is emulate a browser at the HTTP protocol level, Selenium runs in the same environment your users use. Also, Selenium is a cross-browser, cross-platfrom solution supporting Internet Explorer on Windows and Mozilla-based browsers on Windows, Mac OSX and Linux and Unix. Other popular tools, like SAMIE or Watir are IE/Windows only solutions."

To accomplish this goal, Selenium uses several interesting ideas and technologies:

• Selenium runs acceptance tests via a real browser that is driven by a JavaScript engine which is called "the BrowserBot"
• the acceptance tests themselves are usually written as HTML tables, very similar to tests written in FIT and FitNesse; they contain commands such as "click", "select", "type" and assertions such as "verifyValue" and "verifyTextPresent" (this mini-language is called "Selenese" by the way)
  
• the BrowserBot translates the tests written in Selenese into JavaScript commands that it sends to the browser
• because JavaScript has built-in protections against cross-site scripting, it is generally not possible to host the tests on one Web site and run them against a different Web site; this means that the Selenium framework needs to be deployed on the Web server under test
• however, the Twisted-based Selenium server provides a work-around by means of a reverse HTTP proxy

What I described so far is the so-called "TestRunner" mode of operation for Selenium; in it, the framework and the tests themselves are deployed as Web pages inside the Web server under test. You, as a tester, point your browser to a URL such as www.yourwebsite.com/TestRunner.html. At that point, the BrowserBot engine is downloaded in your browser and is ready to process tests written as HTML tables.

The second mode of operation is called "Driven" and provides a way to automatically drive a browser from an application written in Python, Ruby, Java or .NET. The Selenium framework provides "drivers" for these languages, which translate from a given language into "Selenese" commands that get fed to the BrowserBot engine inside the browser. This will become clearer when I'll talk about the way it is done in the Twisted-based server implementation.

I'll continue with a step-by-step tutorial on installing Selenium, configuring it on your local machine, then writing and running acceptance tests against any Web application you want.

Installing and configuring Selenium

1) My setup was: Windows XP with ActivePython 2.3.2. As pre-requisites for Selenium, I installed the latest versions of Twisted and pyCrypto.

2) I created a directory called C:\Selenium and checked out the Selenium code from svn://beaver.codehaus.org/selenium/scm/trunk using the TortoiseSVN subversion client. I'll call the C:\Selenium directory the SELENIUM_ROOT directory.

3) I copied the contents of the SELENIUM_ROOT\code\javascript directory (all its files and sub-directories, not the directory itself) into SELENIUM_ROOT\code\python\twisted\src\selenium\selenium_driver

4) In the current version of the code, there is a "Selenese" command called verifyLocation that checks absolute URLs instead of relative URLs. This makes many of the sample tests shipped with Selenium fail. To fix it, Jason told me to edit SELENIUM_ROOT\code\python\twisted\src\selenium\selenium_driver\selenium-api.js and delete the assertLocation function, then paste these lines in its place:

/*
* Verify the location of the current page.
*/
Selenium.prototype.assertAbsoluteLocation = function(expectedLocation) {
assertEquals(expectedLocation, this.page().location);
};

/*
* Verify the location of the current page ends with the expected location
*/
Selenium.prototype.assertLocation = function(expectedLocation) {
var docLocation = this.page().location.toString();
assertTrue(docLocation.length == docLocation.indexOf(expectedLocation) + expectedLocation.length);
};

Running the Selenium Twisted server

I went to a command prompt and cd-ed into SELENIUM_ROOT\code\python\twisted\src\selenium\, then I ran:

python selenium_server.py

This command launches a Twisted-based server on port 8080 on localhost. According to the docstring in selenium_server.py, this server provides the following functionality:

• A static content Web server for the TestRunner files (HTML test tables, JavaScript files, CSS files)
• A reverse proxy server which provides a work-around for the JavaScript cross-site scripting (XSS) limitation
• the server uses the Perl-based CGIProxy (located in the twisted\src\selenium\cgi-bin directory); however, Jason already compiled the nph-proxy.cgi Perl module into nph-proxy.exe, so you don't need to have Perl installed
  
• in order to comply with the JavaScript XSS limitation, the acceptance tests use URLs such as http://localhost:8080/AUT/00000A/http/www.amazon.com
• the proxy intercepts all requests to these URLs and fetches pages from standard URLs such http://www.amazon.com, then relays the pages to the JavaScript engine, which is blissfully ignorant of this trick
• A driver interface that translates into Selenese
• An XML-RPC server that accepts requests written in standard Python, Ruby, Java or C# and relays them to the driver for translation into Selenese

Perhaps the most important functionality offered by the Twisted Server version of Selenium is the side-stepping of the XSS limitation via the reverse proxy mechanism. One of the main perceived disadvantages of "standard" Selenium is that it needs to be deployed on the server side. The Twisted Server version allows testers to deploy Selenium on their local host and from there test any Web site they want.

Running the sample tests

I opened a browser and went to http://localhost:8080/selenium-driver/TestRunner.html . Note that if you go directly to http://localhost:8080, you get a 404 error, since selenium_server.py explicitly maps /selenium-driver and not / as a resource.

If you followed along, you should see several frames in your browser. The top left frame contains a TestSuite with the sample tests shipped with Selenium. The first test, TestOpen, is already selected in the frame below TestSuite. Ignore a button with a caption that says "True" for now. Its placement is not the most fortunate (I actually edited TestRunner.html and got rid of it). If you click on the green "Selected Test" button, you'll see the TestOpen table turn to green, like this:

Test Open
open	./tests/html/test_open.html
verifyLocation	/tests/html/test_open.html
verifyTextPresent	This is a test of the open command.
open	./tests/html/test_slowloading_page.html
verifyLocation	/tests/html/test_slowloading_page.html
verifyTitle	Slow Loading Page

The large frame on the right of the TestOpen table contains the last page that was opened by the test -- ./tests/html/test_slowloading_page.html -- which contains the text "This is a dummy page".

The TestOpen test is a simple example of what acceptance tests look like in Selenium. If you're familiar with FIT or FitNesse, this is nothing new to you. The first row contains a single cell with the name of the test. All the other rows contain 3 cells: the first one specifies a command such as an action ("open") or a check ("verifyTitle"), the second one specifies the resource to act upon or check, and the third one specifies the expected value for that resource. Many of the check-type commands have the third cell empty, in which case the second cell specifies the value that the command checks for. In the TestOpen example, the last verifyTitle command expects "Slow Loading Page" as the value.

The Selenium test runner will go through each row of the table, run the command, get back the result, compare it with the expected value, and color the row green or red, depending whether the result matched the expectation or not. The "Run mode" radio buttons in TestRunner.html allow the tester to either run the whole table at once, or to walk or step through each row (walk seems to mean go through all the rows, but slowly).

Note also how the summary of the tests ran so far changes:

Elapsed Time:	00:03
Test Results		Command Results
Total run:	1	Passes:	4
Failures:	0	Failures:	0
		Errors:	0

I encourage you at this point to click on other links in the TestSuite frame, then run each one by clicking the "Selected test" button and see what happens. You can also run all the tests in the suite by clicking "All tests" button. Some of them will fail, some expectedly, some not so expectedly. The tests exercise most of the Selenese commands, so this is a very good way to get familiar with their syntax. A command reference for HTML-based tests is also available here.


Adding a test to the default TestSuite

It is instructive at this point to look at the HTML source of TestRunner.html in SELENIUM_ROOT\code\python\twisted\src\selenium\selenium_driver. The file starts by importing JavaScript files, among them the BrowserBot and the selenium-api (I can't show the lines here, since Blogger objects to the "script" tag).

The file then fills the TestSuite frame with the contents of another HTML file, ./tests/TestSuite.html. So the easiest way to add a new test to the TestSuite is to edit TestSuite.html and add a new row to the table defined in it. The tests directory (sub-directory of selenium-driver) contains all the default acceptance tests in HTML format (TestOpen.html, TestType.html etc.) ; among them is GoogleTestSearch.html, which contains this table (Blogger might mangle it and show it with no borders...):

Google Test Search
open	http://www.google.com
verifyTitle	Google
type	q	Selenium ThoughtWorks
verifyValue	q	Selenium ThoughtWorks
click	btnG
verifyTextPresent	selenium.thoughtworks.com
verifyTitle	Google Search: Selenium ThoughtWorks

To add this test to the TestSuite, I added a new row containing ./GoogleTestSearch.html at the top of the table section of TestSuite.html.

I then went back to TestRunner.html in my browser, reloaded the page, clicked on GoogleTestSearch, then ran the test by clicking the "Selected test" button. Since the URL that was opened was http://www.google.com, the JavaScript XSS limitation kicked in and I got "Permission denied" for the verifyTitle command:

Google Test Search
open	http://www.google.com
verifyTitle	Google	Permission denied

To get around this limitation, I needed to use the reverse CGIProxy mechanism. Instead of opening http://www.google.com directly, I specified the following URL: http://localhost:8080/AUT/000000A/http/www.google.com

This is a bit obscure, so let me see if I can explain it. If you look at the source code for selenium-server.py, you'll see these lines:

# The proxy server (aka "The Funnel")
path = os.path.join(os.getcwd(),"cgi-bin","nph-proxy.exe")
proxy = twcgi.CGIScript(path)
root.putChild("AUT",proxy)

The Twisted server specifies AUT as the virtual directory where cgi-bin/nph-proxy.exe (the compiled Perl module) will get executed. AUT signifies "Application Under Test" -- it is just a naming convention used by the Selenium Twisted server.

If you now look inside the nph-proxy.cgi module and search for "sub pack_flags", you'll see that 000000A is a "flag segment", a representation of special flags given as individual characters. This notation is specific to the CGIProxy module. You can by and large ignore its meaning and just remember to insert the 000000A string between AUT and the rest of the URL.

I didn't come up with this notation out of the blue. It was fortunately available in the SELENIUM_ROOT\code\python\twisted\src\examples directory, in the file google-test-xmlrpc.py.

To make the Google search test work, I edited GoogleTestSearch.html and changed the second row so that it contained:

open

http://localhost:8080/AUT/000000A/http/www.google.com

I ran the GoogleTestSearch again and this time it was successful -- all the "verify" lines in the table turned green:

Google Test Search
open	http://localhost:8080/AUT/000000A/http/www.google.com
verifyTitle	Google
type	q	Selenium ThoughtWorks
verifyValue	q	Selenium ThoughtWorks
click	btnG
verifyTextPresent	selenium.thoughtworks.com
verifyTitle	Google Search: Selenium ThoughtWorks

If you followed along, note how the frame next to the result table shows the actual browser rendering of the commands in the test table. The browser first fetches the home page for google.com, then "Selenium ThoughtWorks" is typed in the search box, and finally the first result page for the search is returned.

This exemplifies the power of Selenium: a real browser is driven by the commands in the test table and the results are shown real-time in a frame next to the nicely-colored test result table. This is sure to impress your customers who are supposed to help writing those acceptance tests :-)

Running Selenium in "driven" mode

The "driven" mode of Selenium makes it possible to drive the framework via scripts written in various programming languages. "Selenese" commands are exposed as an API that can then be called from within scripts written in Python, Ruby, Perl, Java or C#. Test "toolsmiths" can then apply their programming expertise to writing real programs, as opposed to writing HTML tables.

Here is an example shipped with Selenium. It tests the same Google search functionality that was tested above via the HTML table.

import xmlrpclib

# Make an object to represent the XML-RPC server.
server_url = "http://localhost:8080/selenium-driver/RPC2"
app = xmlrpclib.ServerProxy(server_url)

# Bump timeout a little higher than the default 5 seconds
app.setTimeout(15)

import os
#os.system('start run_firefox.bat')
os.system('\"C:\\Program Files\\Mozilla Firefox\\firefox.exe\" http://localhost:8080/selenium-driver/SeleneseRunner.html')

print app.open('http://localhost:8080/AUT/000000A/http/www.google.com/')
print app.verifyTitle('Google')
print app.type('q','Selenium ThoughtWorks')
print app.verifyValue('q','Selenium ThoughtWorks')
print app.clickAndWait('btnG')
print app.verifyTextPresent('selenium.thoughtworks.com','')
print app.verifyTitle('Google Search: Selenium ThoughtWorks')
print app.testComplete()

There are similar examples shipped with Selenium, one in Ruby and one in Perl.

All the script needs to do, regardless of the language it's written in, is to open an XML-RPC connection to the server running at http://localhost:8080/selenium-driver/RPC2. The Selenese test commands are then available as methods on the object that represents that connection.

The script then opens a browser and points it to the special URL http://localhost:8080/selenium-driver/SeleneseRunner.html. In the example shipped with Selenium, this is done via the run_firefox.bat or run_ie.bat batch files, but it can also be done directly from the Python code, as shown in the code above.

SeleneseRunner.html is similar to TestRunner.html in that it includes the BrowserBot engine and drives a browser embedded inside one of its frames. The page also shows the commands sent to the BrowserBot by the test script. Successful commands are shown in green and failures are shown in red. If you hover with the mouse over a failed command, you'll see a tooltip showing the reason for the failure.

At the same time, the scripts prints the results of the commands at the command prompt. Here's an example:

C:\Selenium\code\python\twisted\src\examples>python google-test-xmlrpc.py
OK
PASSED
OK
PASSED
OK
PASSED
PASSED
test complete

The mechanism by which the script communicates with the BrowserBot is a queue of commands and command results maintained by the XML-RPC server launched by selenium_server.py. Commands given by the script via the API, such as app.verifyTitle('Google'), get posted to the queue, translated into Selenese by an intrepreter, and retrieved by SeleneseRunner.html via an HTTP GET, using the JavaScript XmlHttp module. SeleneseRunner.html then runs the Selenese commands through the BrowserBot and posts back the results of the commands to the queue. When posting the result for a command, SeleneseRunner.html also inspects the queue to see if there is another command available for execution.

If you're interested to see how all this mechanism works together, look at the source code for SeleneseRunner.html and RPC2.rpy in SELENIUM_ROOT\code\python\twisted\src\selenium\selenium-driver, and for Interpreter.py and Dispatcher.py in SELENIUM_ROOT\code\python\twisted\src\selenium\.
See also the Driven Selenium reference for more details on what they call the "reply/request" mechanism by which the browser retrieves commands to be executed from the queue.

Writing a new test

I'll show another example of a test written for Selenium. It tests the search functionality of Amazon.com. The actions and checks are the following:

• open Amazon.com home page
• verify that "All Products" is selected by default
  
• choose "Books" from the drop-down menu
• verify that the search text box is empty
  
• type "Python Cookbook" as the search text
• verify that the correct search page title is displayed
• verify that the title of the book searched for is displayed in the page
• verify that the book's authors are displayed in the page

Here is the test as an HTML table-- I used Mozilla Composer to write it, a tool that is used by the Selenium developers too. I put it in a file I called AmazonTestSearch.html, in the SELENIUM_ROOT\code\python\twisted\src\selenium\selenium-driver\tests directory:

Amazon Test Search
open	http://localhost:8080/AUT/000000A/http/www.amazon.com/
verifyTitle	Amazon.com: Welcome
verifySelected	url	All Products
select	url	Books
verifySelected	url	Books
verifyValue	field-keywords
type	field-keywords	Python Cookbook
click	Go
verifyTitle	Amazon.com: Books Search Results: Python Cookbook
verifyTextPresent	Python Cookbook
verifyTextPresent	Alex Martellibot, David Ascher

(I misspelled the name of Alex Martelli on purpose, to show an example of a failing test.)

I added a new row to tests/TestSuite.html with a link to the newly created tests\AmazonTestSearch.html, then I pointed a browser to http://localhost:8080/selenium-driver/TestRunner.html, clicked on AmazonTestSearch, then ran the test by clicking on the "Run selected" button. The BrowserBot then sent the commands to the browser embedded in the TestRunner page frame and at the same time colored the rows of the test table according to the outcome of the commands:

Amazon Test Search
open	http://localhost:8080/AUT/000000A/http/www.amazon.com/
verifyTitle	Amazon.com: Welcome
verifySelected	url	All Products
select	url	Books
verifySelected	url	Books
verifyValue	field-keywords
type	field-keywords	Python Cookbook
click	Go
verifyTitle	Amazon.com: Books Search Results: Python Cookbook
verifyTextPresent	Python Cookbook
verifyTextPresent	Alex Martellibot, David Ascher	'Alex Martellibot, David Ascher' not found in page text.

When I tried to run the same commands in a Python script, I ran into a problem with the "select" command, which was not implemented by the Selenium Python driver. All it took to fix it was to send a message to the selenium-users mailing list at lists.public.thoughtworks.org. Jason Huggins promptly sent me 2 code snippets. If you want to replicate the example, you need to edit SELENIUM_ROOT\code\python\twisted\src\selenium\selenium-driver\RPC2.rpy and add the following lines above the selectAndWait function:

    def xmlrpc_select(self, field, value):
   return deferToThread(interpreter.select, field, value)

You also need to edit SELENIUM_ROOT\code\python\twisted\src\selenium\Interpreter.py and add the following lines above selectAndWait:

    def select(self, field, value):
   """ Select the option from the located select element."""

   return self.dispatchCommand("select",field, value)

This actually is a good example of what you need to do if you want to implement a Selenese command in a scripting language.

My test-amazon.py script looks like this:

import xmlrpclib

# Make an object to represent the XML-RPC server.
server_url = "http://localhost:8080/selenium-driver/RPC2"
app = xmlrpclib.ServerProxy(server_url)

# Bump timeout a little higher than the default 5 seconds
app.setTimeout(15)

import os
os.system('start run_firefox.bat')

print app.open('http://localhost:8080/AUT/000000A/http/www.amazon.com/')
print app.verifyTitle('Amazon.com: Welcome')
print app.verifySelected('url', 'All Products')
print app.select('url', 'Books')
print app.verifySelected('url', 'Books')
print app.verifyValue('field-keywords', '')
print app.type('field-keywords', 'Python Cookbook')
print app.clickAndWait('Go')
print app.verifyTitle('Amazon.com: Books Search Results: Python Cookbook')
print app.verifyTextPresent('Python Cookbook', '')
print app.verifyTextPresent('Alex Martellibot, David Ascher', '')
print app.testComplete()

When I ran it in a command prompt (while selenium_server.py was running in another), a browser was automatically opened and directed to the SeleniumRunner page, where the embedded browser then went through all the commands and displayed them colored green or red, depending on their outcome. This is what I got at the command prompt:

C:\Selenium\code\python\twisted\src\examples>python test-amazon.py
OK
PASSED
PASSED
OK
PASSED
PASSED
OK
OK
PASSED
PASSED
'Alex Martellibot, David Ascher' not found in page text.
test complete

Some observations

Overall, I think Selenium is an amazing acceptance test tool for Web applications and I hope it will be adopted on a wide scale. My post focused on the stand-alone Twisted-based server, since it offers a feature not available in other Selenium implementations, namely the ability to test Web sites without instrumenting them on the server side. Another very valuable feature is the ability to run in Driven mode via scripting. Note that the Twisted server, even though Python-based, can still be driven via Ruby or Perl scripts. I tried the Ruby script shipped with the Twisted server and it ran with no problems.

I intend to write another post about Selenium as a Plone product. All concepts related to the TestRunner mode still apply, so I'll focus on some Plone-specific test scenarios.

Here are some issues I ran into, which are mainly due to my inexperience in using the tool:

• I'm still experimenting with some of the Selenese commands, especially the ones related to clicking on various HTML elements
• this is easy to do when elements such as links are identified in the HTML source with IDs, but it gets more complicated when they're not
• Selenium can use "element locators" to locate various HTML elements
• the element locators can be expressed as an elementID, a DOM path or an XPath expression
• I'm currently trying to use Mozilla's DOM Inspector tool to correctly identify the DOM paths to the links I want to click on via Selenium
• I had some problems with Selenium in Driven mode
• sometimes the reply/request mechanism that ties together XML-RPC server and the SeleniumRunner got out of sync and I had to restart the selenium_server, then re-run the tests
• the queue mechanism might need some calibration

Update 3/31/04

1) Some people had problems running the GoogleSearchTest on Windows XP. All that was displayed in the AUT frame was:

CGI Script Error
Premature end of script

In this case, recompiling nph-proxy.cgi into nph-proxy.exe using PAR 0.87 (from http://www.bribes.org/perl/ppmdir.html) solved the problem.

2) If you are on a Unix-like system, you will need to change selenium_server.py to use nph-proxy.cgi instead of nph-proxy.exe. You will also need to do a "dos2unix" on nph-proxy.cgi to convert that file from Windows to Unix line feeds.

(fixes reported by Marc Tremblay)

Update 3/29/05

On Linux/Unix systems, if you keep getting:

CGI Script Error
Premature end of script

check the nph-proxy.cgi file. It needs to be executable, and it needs to have the correct path to perl on the first line. By default, it has a Windows-specific path to perl.exe. You need to change that to the path to perl on your system (/usr/bin/perl for example). To check that nph-proxy.cgi is sane, just run it in a terminal window. You should get back valid HTML.

See this selenium-users forum thread for details on this issue.