Quick update on Selenium in Twisted Server mode

Several people experienced problems when trying to follow my "Selenium and Twisted" tutorial. Here are some issues and some workarounds:

1) If you are on a Windows XP machine and, when trying to run google-test-xmlrpc.py, all you see in the SeleniumRunner page in the browser is:

CGI Script Error
Premature end of script headers

then you need to recompile nph-proxy.cgi into nph-proxy.exe by using the PAR utility. Marc Tremblay reported success by using PAR version 0.87 from here.

2) If you are on a Unix-like machine (Linux, Mac OS X, FreeBSD), then you need to modify selenium_server.py to have it use nph-proxy.cgi instead of nph-proxy.exe.

You also need to run "dos2unix" on nph-proxy.cgi in order to convert from DOS line feeds to Unix line feeds.

3) The Twisted-based server doesn't seem to work properly on Windows 2000 Pro machines. Assuming selenium_server.py is running in one command prompt window, then running the google-test-xmlrpc.py script in another window produces this output:

ERROR: Result queue was empty
ERROR: Result queue was empty
ERROR: Result queue was empty
ERROR: Result queue was empty
ERROR: Result queue was empty
ERROR: Result queue was empty
ERROR: Result queue was empty
test complete

I'm not sure what causes this. I suspect the nph-proxy.cgi script is the culprit, but recompiling it into nph-proxy.exe did not solve the problem for me.

PyCon05

I gave my talk at PyCon05 yesterday. I surprised myself by managing to stay within the 20 minutes allocated for the talk. I even got to show "live" demos of FitNesse and Selenium. The only 2 questions I had at the end were both Selenium-related. It's clear that this tool fills a need that many people have.

Today I gave a "Lightning Talk" presentation with only 5 minutes at my disposal. I showed again a FitNesse demo testing the PyGoogle module, and 2 Selenium demos, one testing a Plone suite with the TestRunner, and the other testing a Google search in "driven" mode, using the Twisted-based implementation. I like to think that I was able to show how testing should really be done at 2 levels:

• the business logic/API level -- represented in my demo by testing the PyGoogle code via FitNesse
• the GUI level -- represented in my demo by testing the Google Web site via Selenium
  

The Lightning Talk session was really fun. People had to stay within the 5 minutes, so they had to get to the essence of what they were trying to show. It looked to me like a martial arts competition, were everybody was trying to show their "kata" moves. Most of the time, people went to a command prompt and started to type away, showing pieces of code and functionality they had been working on.

Good thing I didn't know what a PyCon Lightning Talk is, otherwise I wouldn't have dared to mingle with the masters. It was an amazing and humbling experience for me, and I'm sure an exhilarating experience for the "competitors".

Using Selenium to test a Plone site (part 1)

I will give an example of using Selenium to test a Plone site. I will use a default, out-of-the-box installation of Plone, with no customizations. The functional test I'll present is written as an HTML table. In this post, I'm using Plone only as an example of a Web application. The test table I present uses generally-available "Selenese" commands that are not specific to Plone, so this example can be used as a mini-tutorial for writing HTML table-based tests in Selenium. If you want to see how Selenium can be used in conjunction with a stand-alone Twisted-based server, read this post.

Update: I just found out that Jason Huggins checked in a lot of Plone-specific Selenium code today, so obviously I haven't had the chance to experiment with that yet. I will follow up this post with another one, more targeted to specific features available in the Plone product version of Selenium (setUp, tearDown methods, and postResults for summarizing the test run).

Selenium is under very active development and new and important features are added to the tool on a daily basis. For this post, I checked out via subversion the latest version of the code as of March 8th.

Installing Selenium as a Plone product

One of the implementations of Selenium is a Plone product. I'll give a short procedure for installing Selenium inside an existing Plone site. In my case, I'm running Plone2-2.0.3-2 installed as an RPM on a Red Hat 9 server. My main Plone site is under /var/lib/plone2/main/.

1. Check out Selenium via subversion from svn://selenium.codehaus.org/selenium/scm/trunk
(I'll call SELENIUM_ROOT the directory where you checked out the code. In my case, it is /usr/local/selenium)

2. Copy SELENIUM_ROOT/code/python/Zope/Selenium to the Products directory of your Plone installation. In my case, I ran:

cp -r /usr/local/selenium/code/python/Zope/Selenium /var/lib/plone2/main/Products

3. Copy all the files and directories under SELENIUM_ROOT/code/javascript to the Products/Selenium/skins/selenium_javascript directory. In my case, I ran:

cp -r /usr/local/selenium/code/javascript/* /var/lib/plone2/main/Products/Selenium/skins/selenium_javascript

4. Go to the Products/Selenium/skins/selenium_javascript directory of your Plone site and rename all the .html, .js and .css files to their original name plus .dtml. A small Python script is provided in one of the readme files. Here it is (I called it rename_files.py):

import os
from os.path import join

# We need to append ".dtml" to all html, js, and css files so the "original" filename can be
# called in the browser.
# For example:
# By default, in Zope, TestRunner.html would be available from a URL as "http://localhost/TestRunner" with
# no ".html" attached. To preserve the original file name, we append ".dtml" to the file name.

for root, dirs, files  in os.walk(os.getcwd()):
    if '.svn' in dirs or 'CVS' in dirs:
        dirs.remove('.svn') # don't visit Subversion or CVS directories

    for file in files:
        if file.endswith('.html') or file.endswith('.js') or file.endswith('.css'):
            old_file = join(root, file)
            new_file = old_file + '.dtml'
            os.rename(old_file,new_file)

In my case, I cd-ed into /var/lib/plone2/main/Products/Selenium/skins/selenium_javascript and ran python rename_files.py.

5. Restart your Plone instance (this will need to be done every time you change anything on the file system, unless you start up Zope in debug mode). In my case, I ran:

/etc/rc.d/init.d/plone2 restart

6. Use the Plone QuickInstaller to install the Plone product.

• Login to Plone as the administrator user -- not into the Zope Management Interface (ZMI), but into the Plone interface
• Click on the "My Preferences" link in the navigation bar
• Click on "Add/Remove products" in the "Plone setup" portlet
  
• Select Selenium from the products available for installation and click Install
  

7. Point your browser to http://PLONE_ROOT_URL/TestRunner.html. In my case, PLONE_ROOT_URL is www.example.com:8080/Plone, so I pointed my browser to http://www.example.com:8080/Plone/TestRunner.html (example.com is of course not the real domain name for my Web site)

You should now see the default TestSuite shipped with Selenium. Feel free to click on the tests in the upper-left frame, then click on the green "Selected test" button to run the test. Playing with the tests in the default TestSuite is a good way to learn what kind of actions and checks are available in Selenium.

Creating a custom Test Suite and adding a test table

A non-documented feature of Selenium is that you can run test suites other than the default TestSuite.html by passing an argument to TestRunner.html. In my case, I created an HTML file called CustomTestSuite.html.dtml in the tests subdirectory of selenium_javascript (the full path to this directory is in my case /var/lib/plone2/main/Products/Selenium/skins/selenium_javascript/tests). Note that although CustomTestSuite is an HTML file, you need to end its name with .dtml, otherwise Plone will strip the .html portion from its name and interfere with the internal Selenium logic.

My CustomTestSuite.html.dtml file contains a single table:

Custom Test Suite

TestNewUser

TestNewUser is a link to tests/TestNewUser.html.dtml.

I then created a file called TestNewUser.html.dtml in the same tests directory. This file contains a table with only one row:

Test New User

To have Selenium load my custom test suite, I restarted Plone, then pointed my browser to:

http://www.example.com:8080/Plone/TestRunner.html?test=./tests/CustomTestSuite.html

This loaded the table from CustomTestSuite.html.dtml in the top left frame of the browser.

I then started to fill the table in TestNewUser.html.dtml by adding "Selenese" commands as rows. I'll show here the final version of the table:

Test New User
setVariable	base_url	'http://www.example.com:8080/Plone'
setVariable	logout_url	'${base_url}/logout'
setVariable	join_url	'${base_url}/join_form'
open	${logout_url}
open	${base_url}
verifyTextPresent	Welcome to Plone
click	//a[@href='${join_url}']
verifyTitle	Portal - Please sign in
verifyLocation	join_form
verifyTextPresent	Registration Form
verifyValue	fullname
type	fullname	Test User Full Name
verifyValue	username
setVariable	random_user	'user'+(new Date()).getTime()
type	username	${random_user}
verifyValue	email
type	email	${random_user}@example.com
verifyValue	password
type	password	testUserPassword
verifyValue	confirm
type	confirm	testUserPassword
click	form.button.Register
verifyTextPresent	You have been registered as a member
click	//input[@value='Log in']
verifyTextPresent	You are now logged in
click	link:set up your Preferences
verifyLocation	/plone_memberprefs_panel
verifyTextPresent	My Preferences
click	//img[@src='user.gif']
verifyLocation	/personalize_form
verifyTextPresent	Personal Preferences
select	wysiwyg_editor	Epoz
click	listed
click	form.button.Save
verifyTextPresent	Your personal settings have been saved
verifyValue	wysiwyg_editor	Epoz
verifyValue	listed	off

This is a pretty long test that involves navigating through 6 or 7 pages. In a real-life testing situation this table should probably be split into several smaller tables, each one testing a specific piece of functionality. For the purpose of this tutorial I wanted as many various Selenese commands as I could fit into a still reasonably-sized table.

As it stands, the TestNewUser table tests the following functionality of a default, out-of-the-box Plone installation:

• log out the existing user, if any
• click the "New User?" button
• fill in the registration form and save it
• click the "log in" button
• go to the My Preferences page
• go to the Personal Preferences page
  
• edit some of the preferences and save the form
  
• check that the edited preferences were correctly saved

Instead of discussing the test table row by row, I'll discuss types of commands such as clicking on links, entering text, selecting value, clicking buttons, validating elements, etc. and I'll refer to the rows which use these commands.

Using variables

A brand-new feature of Selenium is the ability to use variables directly in the test tables. In the official documentation, the only way to deal with variables is via separate HTML pages where these variables are set. In the TesNewUser table, I'm using the setVariable command to set 3 variables: base_url, join_url and logout_url. The syntax for setting a variable is:

setVariable

var_name

var_value

(if the value is a string, it needs to be enclosed in quotes)

The syntax for getting the value of a variable is: ${var_name}

Note that join_url and logout_url use the value of base_url via interpolation, by enclosing the expression containing the variable in quotes:

setVariable

logout_url

'${base_url}/logout'

An important use of a variable, especially when testing Web sites that provide log in functionality, is to set a random user ID that will be used in the test. TestNewUser does this via:

setVariable

random_user

'user'+(new Date()).getTime()

Note that in this case the value for the random_user variable is obtained by concatenating a string ('user') with the value returned by a JavaScript function (the getTime() method for a Date object). So you can use JavaScript code in your variable assignments.

Opening Web pages by URL

Web pages can be opened by their URL via the open command. An example is:

open

${logout_url}

This particular "log out" command is the first real action in the TestNewUser table. It is necessary because after creating a user and logging in, that user will still be logged in when running the test table again. In Plone, the home page for a logged-in user is different from the home page of a non-logged in user, and I wanted to test the former situation.

Clicking on links

This is one of the trickiest aspects of writing tests in Selenium. In general, HTML elements on the Web pages you're trying to test can be referred to in Selenium commands via "element locators", which can be one of the following:

• identifiers: the id or name attribute of the element
• DOM traversal syntax: document.forms['myForm'].myDropdown
• XPath syntax: //img[@alt='The image alt text']

Life is easy when HTML elements such as links have id attributes such as id="the_link_id". In this case, the command you need to use for clicking on the link is simply:

click

the_link_id

Life is also pretty easy when links have text that is on the same line with the starting a tag and the closing /a tag. In this case, you can use the following XPath syntax (a good XPath tutorial recommended by Ian Bicking is here):

click

//a[text() = "the link text]

A recently introduced Selenium command for accessing links by their text is the link: command:

click

link:the link text

Note that you need to follow link by a colon, then immediately by the text of the link with no quotes. I used this command in the TestNewUser table like this:

click

link:set up your Preferences

In other cases, especially when the link text is on a line by itself or spans multiple lines, the text method will not work and you will need to identify the link by some other attributes. Here is an example from TestNewUser:

click

//a[@href='${join_url}']

This represents the command for clicking on the "New user?" link at the bottom of the "log in" portlet on the Plone home page. I initially tried to identify the link by the "New user?" text, but that method didn't work, because the starting a tag, the link text and the closing /a tag were on different lines. The only solution I found was to identify the link by its href tag. The XPath syntax I used was //a[@href='url'] where url is identified by the value of the variable ${join_url}.

Here is another example from TestNewUser:

click

//img[@src='user.gif']

This represents the command for clicking on the Personal Preferences link on the "My Preferences" page. Again the text method didn't work, so this time I identified the link via the src tag of its image.

Clicking on submit buttons

The command you need to use for clicking on form submit buttons is click. The target of the command is the element locator for the button. This can be either the button's name attribute or its value attribute. An example of clicking a submit button by its name in TestNewUser is:

click

form.button.Register

This represents the command for clicking on the "register" button at the bottom of the new user registration form.

Here is an example of clicking a submit button by its value, using an XPath expression:

click

//input[@value='Log in']

This represents clicking on the "log in" button on the Welcome page that is shown immediately after a successful registration.

Clicking on check boxes

The click command again accomplishes this. You need to indicate the name of the check-box field as the target of the command. The click command, when applied to a check box, toggles the value of that box. Here is an example from TestNewUser, where the check box for "Listed in searches" on the Personal Preferences page is clicked. The name of that box is "listed":

click

listed

Note that the click command can also be used for clicking on radio buttons.

Entering text in input fields

If you need to fill in input fields in forms, use the type command, which takes 2 arguments: the name of the input field (you'll need to figure what that name is by inspecting the HTML source) and the value you need to type in that field. Here are some examples from TestNewUser, for filling in the user name and email information on the registration form:

type

username

${random_user}

type

email

${random_user}@example.com

Selecting values in drop-down lists

This is accomplished via the select command, which also takes 2 arguments: the name of the drop-down list field and the value you need to select in that list. An example from TestNewUser shows how to select the Epoz editor in the Personal Preferences page:

select

wysiwyg_editor

Epoz

Verifying the state of the application

So far I have shown how Selenium can drive an embedded browser via "Selenese" commands. This is only one aspect of testing a Web application, since it indirectly verifies that the HTML elements it expects to click or open are indeed present. For direct verifications, Selenium provides a variety of "verify" commands that check the values of the different elements under test. Perhaps the simplest check is verifyTextPresent, which makes sure that a given snippet of text is present in a Web page. This example from TestNewUser checks that the default Plone home page contains "Welcome to Plone":

verifyTextPresent

Welcome to Plone

Values for elements of a form (input fields, drop-down lists, check boxes) can be verified via the verifyValue command:

verifyValue

wysiwyg_editor

Epoz

verifyValue

listed

off

(note that for a check box such as "listed", the value that we compare with is either off or on)

To verify that a Web page contains a specific URL, use either verifyAbsouteLocation (which checks that the URL of the page is identical with a given string) or verifyLocation (which checks that the URL of the page ends with a given string).

This example from TestNewUser checks that the URL of the new user registration page ends with /join_form:

verifyLocation

join_form

There are many other types of check commands available on the Selenium TestRunner reference page.

To be continued...

As I mentioned before, new and exciting features are added to Selenium on a daily basis. If you're interested in the tool and want to see what is going on with its development, browse the selenium-devel mailing list archive and consider joining the list and contributing.

I intend to follow up this article with other Selenium-related posts that will cover things such as:

• using wildcards and regular expressions in verification commands
• embedding JavaScript code in Selenese commands
• organizing and reusing tests
  
• using Plone-specific features such as Setup, TearDown and PostResults pages
  

Acceptance tests for Web apps: GUI vs.business logic

I've been looking at Selenium lately (see previous posts) as a tool for acceptance/functional testing of Web applications. In its standard "TestRunner" mode, Selenium allows you to write tests as HTML tables that contain actions related to the HTML elements of the Web pages you want to test. Actions can be commands such as "open" a page, "click" on a link, "type" in a text field, "select" a value from a drop-down box. Actions can also be checks that compare the values of HTML elements against expected values: "verifyText", "verifyValue", "verifyTitle", etc.

A typical Selenium test table looks like this:

Google Test Search
open	http://www.google.com
verifyTitle	Google
type	q	Selenium ThoughtWorks
verifyValue	q	Selenium ThoughtWorks
click	btnG
verifyTextPresent	selenium.thoughtworks.com
verifyTitle	Google Search: Selenium ThoughtWorks

This type of test exercises the AUT (application under test) at the GUI level. It is an important part of your testing strategy, since you want to make sure that your users can actually access all the functionality offered by your Web application. However, testing at the GUI level is notoriously brittle. Any time you make a change in the structure of the HTML pages under test, you run the risk of breaking the acceptance/functional tests that act on and verify elements of those HTML pages. Selenium tries to alleviate this issue by offering different ways of referring to HTML elements in your tests: by elementID, by specifying a DOM node, and by indicating an XPath expression. If you start developing a Web application from scratch, then you can collaborate with the HTML designers on the team and have them use identifiers for those HTML elements that you know will be exercised more by your tests. Or you can come up with a naming scheme for the HTML elements that will change as little as possible as development progresses.

There is another problem with testing at the GUI level, even assuming the HTML pages are stable and your tests will not break. GUI-level tests exercise the business logic of your application only indirectly. For example, say you're in the business of selling widgets. You put together an online store application, with a shopping cart, credit card processing, etc. If you only test the application at the GUI level, how do you know that orders for your widgets really go through, that the inventory is correspondingly modified, that the credit card was really valid? All you can verify via a GUI-level test is that users can succesfully navigate through your site and see the pages that you expect them to see. This is only an indirect validation of your business logic rules. For a direct validation, you need tests that exercise the database backend of your application. These tests can also be written as HTML tables and ran through a framework such as FitNesse. Here's a typical FitNesse test that specifies an acceptance test scenario for a payroll application (I copied it directly from this page):

First we add a few employees.

Employees
id	name	address	salary
1	Jeff Languid	10 Adamant St; Laurel, MD 20707	1005.00
2	Kelp Holland	12B Baker St; Cottonmouth, IL 60066	2000.00

Next we pay them.

Pay day.
pay date	check number
1/31/2001	1000

We make sure their paychecks are correct. The blank cells will be filled in by the PaycheckInspector fixture. The cells with data in them already will be checked.

Paycheck Inspector.
id	amount	number	name	date
1	1005
2	2000

Finally we make sure that the output contained two, and only two paychecks, and that they had the right check numbers.

Paycheck inspector.

number

1000

1001

There is a very different look and feel for these tests, compared to the Selenium tests. Each table has a name such as Employees, which corresponds to a "fixture", a piece of code that takes the data specified in the table rows as arguments and passes them along to the AUT, then retrieves values that can be checked against expected values in the tables. Acceptance tests written in FitNesse often exhibit the "Build, Operate, Check" pattern: build the test data, operate on it, then check it against expected values. It can truly be said that a FitNesse page containing test tables is another thin GUI layer into your application, but a GUI layer that exercises your business rules directly. For this to work, the developers need to provide clean interfaces into the business logic code. The application needs to have a design that separates the GUI logic from the business logic. In an agile environment, customers are supposed to pitch in and start writing acceptance tests in FitNesse as soon as the team gets started on a new iteration. Developers can then see what kind of hooks they need to provide as an interface for the fixture code. These hooks will evolve into a testing interface for the application. The resulting design will clearly separate the GUI logic from the business logic, and in fact it will be much easier to change GUIs altogether, or to provide different "views" into the business logic which do not even have to be GUI-based -- I'm thinking primarily of Web services.

The testing interfaces I mentioned can also evolve from the so-called "admin modules" that many Web applications have. These are alternate interfaces into the application, used by the business people for checking and updating inventory, running reports against the database, etc. All these functions directly exercise business rules by interfacing with the database backend. While writing them, the developers might as well think of them as testing interfaces that can be used in acceptance tests.

In conclusion, I think that acceptance tests for a Web application (or any application that has a GUI for that matter) need to be run at both levels: GUI and business logic. The GUI tests can be used as a "smoke test" strategy, as a sanity check that navigation through the site works and that users are not faced with ugly 404 errors. For this type of testing, a tool such as Selenium, which drives a real browser, is invaluable. But the bulk of the acceptance testing should be done at the business logic level. Being able to run FitNesse-type acceptance tests not only enhances the testability of the application, but most importantly forces a clean design that separates the GUI layer from the business logic layer and allows the application to easily adapt to GUI changes. Another benefit is that it becomes easy for the application to offer several interfaces into its business logic, for example a Web services interface in addition to the standard HTML-based interface.

One more note: one thing that Selenium-style tests and FitNesse-style tests have in common is that tests can be specified via HTML tables. This provides a nice visual feedback during the test run: the rows of these tables get colored green or red, depending on the test outcome. The HTML table format however is not the most friendly one for business customers who are supposed to write these tests side-by-side with the testers. FitNesse does offer ways of importing tables from spreadsheets, and Selenium is being extended at the moment so that it can deal with CSV formats (see Ian Bicking's post on some of the work he's doing on this, as well as on automatically generating Selenium scripts via TCPWatch). Selenium can also be run in "driven mode", where scripts written in Python, Ruby or Perl can drive the browser via an API.

Quick update on Selenium in TestRunner mode

Ian Bicking made a good point today on the selenium-devel mailing list: the introductory Selenium documentation makes things look much more complicated than they really are. And my previous post on Selenium falls in the same trap. It's true that I focused my tutorial on the Twisted server-driven mode of Selenium, but I should have made it clear that it's really easy to get started with Selenium in TestRunner mode if you have a Web application that you need to test.

Basically, all you need to do is to either download Selenium or check it out via svn from svn://selenium.codehaus.org/selenium/scm/trunk, then copy the contents of the javascript directory (all its files and sub-directories) to a directory that can be served via your Web server.

I created a directory called selenium under the DocumentRoot directory of one of my Apache virtual servers, then I pointed my browser to http://www.mywebsite.com/selenium/TestRunner.html and I was good to go -- that is, I could see all the sample tests and start experiment with them.

Note that in the current version, many of the shipped tests will fail, unless you do the following correction: edit selenium-api.js and delete the assertLocation function, then paste these lines in its place:

/*
* Verify the location of the current page.
*/
Selenium.prototype.assertAbsoluteLocation = function(expectedLocation) {
assertEquals(expectedLocation, this.page().location);
};

/*
* Verify the location of the current page ends with the expected location
*/
Selenium.prototype.assertLocation = function(expectedLocation) {
var docLocation = this.page().location.toString();
assertTrue(docLocation.length == docLocation.indexOf(expectedLocation) + expectedLocation.length);
};

To add your own tests, start from one of the sample tests, add your HTML test file to the "tests" subdirectory and edit the tests/TestSuite.html file by adding a new row pointing to your test.

Note: in the TestRunner mode of Selenium you will only be able to test the functionality of your own Web site. You will not be able to run tests against third party sites such as Google or Amazon, due to the cross-scripting security limitation of JavaScript.

By the way, Jason Huggins is asking for suggestions for better naming the 2 main Selenium implementation/execution modes. Currently, the nomenclature is:

1) "TestRunner mode", a.k.a "HTML table-style", "FIT style", "browser driven"
2) "Driven mode", a.k.a "source code-style", "server driven"

If you have a better descriptive name for Selenium in these modes, please leave a comment.

Web app testing with Python part 2: Selenium and Twisted

In a previous post I mentioned Selenium as a Web app testing tool that is like no other in terms of functionality and implementation. I've been experimenting with Selenium for the past few days and I'm very impressed (a reaction which seems to be common to everybody who witnessed the tool in action.)

The main Selenium developer is Jason Huggins, who initially created Selenium as a tool for acceptance/functional testing of Web sites based on Plone. Jason is the author of two Python implementations for Selenium:

• the "Plone Product" version
• written as a Plone product,
  
• it is used specifically for testing Plone-based sites
  
• it needs to be installed on the Plone site under test
• the "Twisted Server" version
• written as a stand-alone Twisted-based server
  
• can be used for testing any Web site

This post is a tutorial in using the "Twisted Server" version of Selenium.

Update 3/31/04: For specifics on running the Twisted Server version of Selenium on Unix-like systems, as well as tips to work around CGI errors, please scroll to the end of this post.

Before I delve into the specifics of installing and running Selenium on Twisted, I'll discuss some of the goals and novel ideas of Selenium. Here are Jason's own words:

"The key concept to know about Selenium is that it uses a real, living and breathing web browser to play back your testing scripts. This means it can test client-side browser-specific things like JavaScript that other testing frameworks like Mechanize or HTTPUnit cannot test. Whereas the best that Mechanize can do is emulate a browser at the HTTP protocol level, Selenium runs in the same environment your users use. Also, Selenium is a cross-browser, cross-platfrom solution supporting Internet Explorer on Windows and Mozilla-based browsers on Windows, Mac OSX and Linux and Unix. Other popular tools, like SAMIE or Watir are IE/Windows only solutions."

To accomplish this goal, Selenium uses several interesting ideas and technologies:

• Selenium runs acceptance tests via a real browser that is driven by a JavaScript engine which is called "the BrowserBot"
• the acceptance tests themselves are usually written as HTML tables, very similar to tests written in FIT and FitNesse; they contain commands such as "click", "select", "type" and assertions such as "verifyValue" and "verifyTextPresent" (this mini-language is called "Selenese" by the way)
  
• the BrowserBot translates the tests written in Selenese into JavaScript commands that it sends to the browser
• because JavaScript has built-in protections against cross-site scripting, it is generally not possible to host the tests on one Web site and run them against a different Web site; this means that the Selenium framework needs to be deployed on the Web server under test
• however, the Twisted-based Selenium server provides a work-around by means of a reverse HTTP proxy

What I described so far is the so-called "TestRunner" mode of operation for Selenium; in it, the framework and the tests themselves are deployed as Web pages inside the Web server under test. You, as a tester, point your browser to a URL such as www.yourwebsite.com/TestRunner.html. At that point, the BrowserBot engine is downloaded in your browser and is ready to process tests written as HTML tables.

The second mode of operation is called "Driven" and provides a way to automatically drive a browser from an application written in Python, Ruby, Java or .NET. The Selenium framework provides "drivers" for these languages, which translate from a given language into "Selenese" commands that get fed to the BrowserBot engine inside the browser. This will become clearer when I'll talk about the way it is done in the Twisted-based server implementation.

I'll continue with a step-by-step tutorial on installing Selenium, configuring it on your local machine, then writing and running acceptance tests against any Web application you want.

Installing and configuring Selenium

1) My setup was: Windows XP with ActivePython 2.3.2. As pre-requisites for Selenium, I installed the latest versions of Twisted and pyCrypto.

2) I created a directory called C:\Selenium and checked out the Selenium code from svn://beaver.codehaus.org/selenium/scm/trunk using the TortoiseSVN subversion client. I'll call the C:\Selenium directory the SELENIUM_ROOT directory.

3) I copied the contents of the SELENIUM_ROOT\code\javascript directory (all its files and sub-directories, not the directory itself) into SELENIUM_ROOT\code\python\twisted\src\selenium\selenium_driver

4) In the current version of the code, there is a "Selenese" command called verifyLocation that checks absolute URLs instead of relative URLs. This makes many of the sample tests shipped with Selenium fail. To fix it, Jason told me to edit SELENIUM_ROOT\code\python\twisted\src\selenium\selenium_driver\selenium-api.js and delete the assertLocation function, then paste these lines in its place:

/*
* Verify the location of the current page.
*/
Selenium.prototype.assertAbsoluteLocation = function(expectedLocation) {
assertEquals(expectedLocation, this.page().location);
};

/*
* Verify the location of the current page ends with the expected location
*/
Selenium.prototype.assertLocation = function(expectedLocation) {
var docLocation = this.page().location.toString();
assertTrue(docLocation.length == docLocation.indexOf(expectedLocation) + expectedLocation.length);
};

Running the Selenium Twisted server

I went to a command prompt and cd-ed into SELENIUM_ROOT\code\python\twisted\src\selenium\, then I ran:

python selenium_server.py

This command launches a Twisted-based server on port 8080 on localhost. According to the docstring in selenium_server.py, this server provides the following functionality:

• A static content Web server for the TestRunner files (HTML test tables, JavaScript files, CSS files)
• A reverse proxy server which provides a work-around for the JavaScript cross-site scripting (XSS) limitation
• the server uses the Perl-based CGIProxy (located in the twisted\src\selenium\cgi-bin directory); however, Jason already compiled the nph-proxy.cgi Perl module into nph-proxy.exe, so you don't need to have Perl installed
  
• in order to comply with the JavaScript XSS limitation, the acceptance tests use URLs such as http://localhost:8080/AUT/00000A/http/www.amazon.com
• the proxy intercepts all requests to these URLs and fetches pages from standard URLs such http://www.amazon.com, then relays the pages to the JavaScript engine, which is blissfully ignorant of this trick
• A driver interface that translates into Selenese
• An XML-RPC server that accepts requests written in standard Python, Ruby, Java or C# and relays them to the driver for translation into Selenese

Perhaps the most important functionality offered by the Twisted Server version of Selenium is the side-stepping of the XSS limitation via the reverse proxy mechanism. One of the main perceived disadvantages of "standard" Selenium is that it needs to be deployed on the server side. The Twisted Server version allows testers to deploy Selenium on their local host and from there test any Web site they want.

Running the sample tests

I opened a browser and went to http://localhost:8080/selenium-driver/TestRunner.html . Note that if you go directly to http://localhost:8080, you get a 404 error, since selenium_server.py explicitly maps /selenium-driver and not / as a resource.

If you followed along, you should see several frames in your browser. The top left frame contains a TestSuite with the sample tests shipped with Selenium. The first test, TestOpen, is already selected in the frame below TestSuite. Ignore a button with a caption that says "True" for now. Its placement is not the most fortunate (I actually edited TestRunner.html and got rid of it). If you click on the green "Selected Test" button, you'll see the TestOpen table turn to green, like this:

Test Open
open	./tests/html/test_open.html
verifyLocation	/tests/html/test_open.html
verifyTextPresent	This is a test of the open command.
open	./tests/html/test_slowloading_page.html
verifyLocation	/tests/html/test_slowloading_page.html
verifyTitle	Slow Loading Page

The large frame on the right of the TestOpen table contains the last page that was opened by the test -- ./tests/html/test_slowloading_page.html -- which contains the text "This is a dummy page".

The TestOpen test is a simple example of what acceptance tests look like in Selenium. If you're familiar with FIT or FitNesse, this is nothing new to you. The first row contains a single cell with the name of the test. All the other rows contain 3 cells: the first one specifies a command such as an action ("open") or a check ("verifyTitle"), the second one specifies the resource to act upon or check, and the third one specifies the expected value for that resource. Many of the check-type commands have the third cell empty, in which case the second cell specifies the value that the command checks for. In the TestOpen example, the last verifyTitle command expects "Slow Loading Page" as the value.

The Selenium test runner will go through each row of the table, run the command, get back the result, compare it with the expected value, and color the row green or red, depending whether the result matched the expectation or not. The "Run mode" radio buttons in TestRunner.html allow the tester to either run the whole table at once, or to walk or step through each row (walk seems to mean go through all the rows, but slowly).

Note also how the summary of the tests ran so far changes:

Elapsed Time:	00:03
Test Results		Command Results
Total run:	1	Passes:	4
Failures:	0	Failures:	0
		Errors:	0

I encourage you at this point to click on other links in the TestSuite frame, then run each one by clicking the "Selected test" button and see what happens. You can also run all the tests in the suite by clicking "All tests" button. Some of them will fail, some expectedly, some not so expectedly. The tests exercise most of the Selenese commands, so this is a very good way to get familiar with their syntax. A command reference for HTML-based tests is also available here.


Adding a test to the default TestSuite

It is instructive at this point to look at the HTML source of TestRunner.html in SELENIUM_ROOT\code\python\twisted\src\selenium\selenium_driver. The file starts by importing JavaScript files, among them the BrowserBot and the selenium-api (I can't show the lines here, since Blogger objects to the "script" tag).

The file then fills the TestSuite frame with the contents of another HTML file, ./tests/TestSuite.html. So the easiest way to add a new test to the TestSuite is to edit TestSuite.html and add a new row to the table defined in it. The tests directory (sub-directory of selenium-driver) contains all the default acceptance tests in HTML format (TestOpen.html, TestType.html etc.) ; among them is GoogleTestSearch.html, which contains this table (Blogger might mangle it and show it with no borders...):

Google Test Search
open	http://www.google.com
verifyTitle	Google
type	q	Selenium ThoughtWorks
verifyValue	q	Selenium ThoughtWorks
click	btnG
verifyTextPresent	selenium.thoughtworks.com
verifyTitle	Google Search: Selenium ThoughtWorks

To add this test to the TestSuite, I added a new row containing ./GoogleTestSearch.html at the top of the table section of TestSuite.html.

I then went back to TestRunner.html in my browser, reloaded the page, clicked on GoogleTestSearch, then ran the test by clicking the "Selected test" button. Since the URL that was opened was http://www.google.com, the JavaScript XSS limitation kicked in and I got "Permission denied" for the verifyTitle command:

Google Test Search
open	http://www.google.com
verifyTitle	Google	Permission denied

To get around this limitation, I needed to use the reverse CGIProxy mechanism. Instead of opening http://www.google.com directly, I specified the following URL: http://localhost:8080/AUT/000000A/http/www.google.com

This is a bit obscure, so let me see if I can explain it. If you look at the source code for selenium-server.py, you'll see these lines:

# The proxy server (aka "The Funnel")
path = os.path.join(os.getcwd(),"cgi-bin","nph-proxy.exe")
proxy = twcgi.CGIScript(path)
root.putChild("AUT",proxy)

The Twisted server specifies AUT as the virtual directory where cgi-bin/nph-proxy.exe (the compiled Perl module) will get executed. AUT signifies "Application Under Test" -- it is just a naming convention used by the Selenium Twisted server.

If you now look inside the nph-proxy.cgi module and search for "sub pack_flags", you'll see that 000000A is a "flag segment", a representation of special flags given as individual characters. This notation is specific to the CGIProxy module. You can by and large ignore its meaning and just remember to insert the 000000A string between AUT and the rest of the URL.

I didn't come up with this notation out of the blue. It was fortunately available in the SELENIUM_ROOT\code\python\twisted\src\examples directory, in the file google-test-xmlrpc.py.

To make the Google search test work, I edited GoogleTestSearch.html and changed the second row so that it contained:

open

http://localhost:8080/AUT/000000A/http/www.google.com

I ran the GoogleTestSearch again and this time it was successful -- all the "verify" lines in the table turned green:

Google Test Search
open	http://localhost:8080/AUT/000000A/http/www.google.com
verifyTitle	Google
type	q	Selenium ThoughtWorks
verifyValue	q	Selenium ThoughtWorks
click	btnG
verifyTextPresent	selenium.thoughtworks.com
verifyTitle	Google Search: Selenium ThoughtWorks

If you followed along, note how the frame next to the result table shows the actual browser rendering of the commands in the test table. The browser first fetches the home page for google.com, then "Selenium ThoughtWorks" is typed in the search box, and finally the first result page for the search is returned.

This exemplifies the power of Selenium: a real browser is driven by the commands in the test table and the results are shown real-time in a frame next to the nicely-colored test result table. This is sure to impress your customers who are supposed to help writing those acceptance tests :-)

Running Selenium in "driven" mode

The "driven" mode of Selenium makes it possible to drive the framework via scripts written in various programming languages. "Selenese" commands are exposed as an API that can then be called from within scripts written in Python, Ruby, Perl, Java or C#. Test "toolsmiths" can then apply their programming expertise to writing real programs, as opposed to writing HTML tables.

Here is an example shipped with Selenium. It tests the same Google search functionality that was tested above via the HTML table.

import xmlrpclib

# Make an object to represent the XML-RPC server.
server_url = "http://localhost:8080/selenium-driver/RPC2"
app = xmlrpclib.ServerProxy(server_url)

# Bump timeout a little higher than the default 5 seconds
app.setTimeout(15)

import os
#os.system('start run_firefox.bat')
os.system('\"C:\\Program Files\\Mozilla Firefox\\firefox.exe\" http://localhost:8080/selenium-driver/SeleneseRunner.html')

print app.open('http://localhost:8080/AUT/000000A/http/www.google.com/')
print app.verifyTitle('Google')
print app.type('q','Selenium ThoughtWorks')
print app.verifyValue('q','Selenium ThoughtWorks')
print app.clickAndWait('btnG')
print app.verifyTextPresent('selenium.thoughtworks.com','')
print app.verifyTitle('Google Search: Selenium ThoughtWorks')
print app.testComplete()

There are similar examples shipped with Selenium, one in Ruby and one in Perl.

All the script needs to do, regardless of the language it's written in, is to open an XML-RPC connection to the server running at http://localhost:8080/selenium-driver/RPC2. The Selenese test commands are then available as methods on the object that represents that connection.

The script then opens a browser and points it to the special URL http://localhost:8080/selenium-driver/SeleneseRunner.html. In the example shipped with Selenium, this is done via the run_firefox.bat or run_ie.bat batch files, but it can also be done directly from the Python code, as shown in the code above.

SeleneseRunner.html is similar to TestRunner.html in that it includes the BrowserBot engine and drives a browser embedded inside one of its frames. The page also shows the commands sent to the BrowserBot by the test script. Successful commands are shown in green and failures are shown in red. If you hover with the mouse over a failed command, you'll see a tooltip showing the reason for the failure.

At the same time, the scripts prints the results of the commands at the command prompt. Here's an example:

C:\Selenium\code\python\twisted\src\examples>python google-test-xmlrpc.py
OK
PASSED
OK
PASSED
OK
PASSED
PASSED
test complete

The mechanism by which the script communicates with the BrowserBot is a queue of commands and command results maintained by the XML-RPC server launched by selenium_server.py. Commands given by the script via the API, such as app.verifyTitle('Google'), get posted to the queue, translated into Selenese by an intrepreter, and retrieved by SeleneseRunner.html via an HTTP GET, using the JavaScript XmlHttp module. SeleneseRunner.html then runs the Selenese commands through the BrowserBot and posts back the results of the commands to the queue. When posting the result for a command, SeleneseRunner.html also inspects the queue to see if there is another command available for execution.

If you're interested to see how all this mechanism works together, look at the source code for SeleneseRunner.html and RPC2.rpy in SELENIUM_ROOT\code\python\twisted\src\selenium\selenium-driver, and for Interpreter.py and Dispatcher.py in SELENIUM_ROOT\code\python\twisted\src\selenium\.
See also the Driven Selenium reference for more details on what they call the "reply/request" mechanism by which the browser retrieves commands to be executed from the queue.

Writing a new test

I'll show another example of a test written for Selenium. It tests the search functionality of Amazon.com. The actions and checks are the following:

• open Amazon.com home page
• verify that "All Products" is selected by default
  
• choose "Books" from the drop-down menu
• verify that the search text box is empty
  
• type "Python Cookbook" as the search text
• verify that the correct search page title is displayed
• verify that the title of the book searched for is displayed in the page
• verify that the book's authors are displayed in the page

Here is the test as an HTML table-- I used Mozilla Composer to write it, a tool that is used by the Selenium developers too. I put it in a file I called AmazonTestSearch.html, in the SELENIUM_ROOT\code\python\twisted\src\selenium\selenium-driver\tests directory:

Amazon Test Search
open	http://localhost:8080/AUT/000000A/http/www.amazon.com/
verifyTitle	Amazon.com: Welcome
verifySelected	url	All Products
select	url	Books
verifySelected	url	Books
verifyValue	field-keywords
type	field-keywords	Python Cookbook
click	Go
verifyTitle	Amazon.com: Books Search Results: Python Cookbook
verifyTextPresent	Python Cookbook
verifyTextPresent	Alex Martellibot, David Ascher

(I misspelled the name of Alex Martelli on purpose, to show an example of a failing test.)

I added a new row to tests/TestSuite.html with a link to the newly created tests\AmazonTestSearch.html, then I pointed a browser to http://localhost:8080/selenium-driver/TestRunner.html, clicked on AmazonTestSearch, then ran the test by clicking on the "Run selected" button. The BrowserBot then sent the commands to the browser embedded in the TestRunner page frame and at the same time colored the rows of the test table according to the outcome of the commands:

Amazon Test Search
open	http://localhost:8080/AUT/000000A/http/www.amazon.com/
verifyTitle	Amazon.com: Welcome
verifySelected	url	All Products
select	url	Books
verifySelected	url	Books
verifyValue	field-keywords
type	field-keywords	Python Cookbook
click	Go
verifyTitle	Amazon.com: Books Search Results: Python Cookbook
verifyTextPresent	Python Cookbook
verifyTextPresent	Alex Martellibot, David Ascher	'Alex Martellibot, David Ascher' not found in page text.

When I tried to run the same commands in a Python script, I ran into a problem with the "select" command, which was not implemented by the Selenium Python driver. All it took to fix it was to send a message to the selenium-users mailing list at lists.public.thoughtworks.org. Jason Huggins promptly sent me 2 code snippets. If you want to replicate the example, you need to edit SELENIUM_ROOT\code\python\twisted\src\selenium\selenium-driver\RPC2.rpy and add the following lines above the selectAndWait function:

    def xmlrpc_select(self, field, value):
   return deferToThread(interpreter.select, field, value)

You also need to edit SELENIUM_ROOT\code\python\twisted\src\selenium\Interpreter.py and add the following lines above selectAndWait:

    def select(self, field, value):
   """ Select the option from the located select element."""

   return self.dispatchCommand("select",field, value)

This actually is a good example of what you need to do if you want to implement a Selenese command in a scripting language.

My test-amazon.py script looks like this:

import xmlrpclib

# Make an object to represent the XML-RPC server.
server_url = "http://localhost:8080/selenium-driver/RPC2"
app = xmlrpclib.ServerProxy(server_url)

# Bump timeout a little higher than the default 5 seconds
app.setTimeout(15)

import os
os.system('start run_firefox.bat')

print app.open('http://localhost:8080/AUT/000000A/http/www.amazon.com/')
print app.verifyTitle('Amazon.com: Welcome')
print app.verifySelected('url', 'All Products')
print app.select('url', 'Books')
print app.verifySelected('url', 'Books')
print app.verifyValue('field-keywords', '')
print app.type('field-keywords', 'Python Cookbook')
print app.clickAndWait('Go')
print app.verifyTitle('Amazon.com: Books Search Results: Python Cookbook')
print app.verifyTextPresent('Python Cookbook', '')
print app.verifyTextPresent('Alex Martellibot, David Ascher', '')
print app.testComplete()

When I ran it in a command prompt (while selenium_server.py was running in another), a browser was automatically opened and directed to the SeleniumRunner page, where the embedded browser then went through all the commands and displayed them colored green or red, depending on their outcome. This is what I got at the command prompt:

C:\Selenium\code\python\twisted\src\examples>python test-amazon.py
OK
PASSED
PASSED
OK
PASSED
PASSED
OK
OK
PASSED
PASSED
'Alex Martellibot, David Ascher' not found in page text.
test complete

Some observations

Overall, I think Selenium is an amazing acceptance test tool for Web applications and I hope it will be adopted on a wide scale. My post focused on the stand-alone Twisted-based server, since it offers a feature not available in other Selenium implementations, namely the ability to test Web sites without instrumenting them on the server side. Another very valuable feature is the ability to run in Driven mode via scripting. Note that the Twisted server, even though Python-based, can still be driven via Ruby or Perl scripts. I tried the Ruby script shipped with the Twisted server and it ran with no problems.

I intend to write another post about Selenium as a Plone product. All concepts related to the TestRunner mode still apply, so I'll focus on some Plone-specific test scenarios.

Here are some issues I ran into, which are mainly due to my inexperience in using the tool:

• I'm still experimenting with some of the Selenese commands, especially the ones related to clicking on various HTML elements
• this is easy to do when elements such as links are identified in the HTML source with IDs, but it gets more complicated when they're not
• Selenium can use "element locators" to locate various HTML elements
• the element locators can be expressed as an elementID, a DOM path or an XPath expression
• I'm currently trying to use Mozilla's DOM Inspector tool to correctly identify the DOM paths to the links I want to click on via Selenium
• I had some problems with Selenium in Driven mode
• sometimes the reply/request mechanism that ties together XML-RPC server and the SeleniumRunner got out of sync and I had to restart the selenium_server, then re-run the tests
• the queue mechanism might need some calibration

Update 3/31/04

1) Some people had problems running the GoogleSearchTest on Windows XP. All that was displayed in the AUT frame was:

CGI Script Error
Premature end of script

In this case, recompiling nph-proxy.cgi into nph-proxy.exe using PAR 0.87 (from http://www.bribes.org/perl/ppmdir.html) solved the problem.

2) If you are on a Unix-like system, you will need to change selenium_server.py to use nph-proxy.cgi instead of nph-proxy.exe. You will also need to do a "dos2unix" on nph-proxy.cgi to convert that file from Windows to Unix line feeds.

(fixes reported by Marc Tremblay)

Update 3/29/05

On Linux/Unix systems, if you keep getting:

CGI Script Error
Premature end of script

check the nph-proxy.cgi file. It needs to be executable, and it needs to have the correct path to perl on the first line. By default, it has a Windows-specific path to perl.exe. You need to change that to the path to perl on your system (/usr/bin/perl for example). To check that nph-proxy.cgi is sane, just run it in a terminal window. You should get back valid HTML.

See this selenium-users forum thread for details on this issue.