Friday, March 11, 2005

Using Selenium to test a Plone site (part 1)

I will give an example of using Selenium to test a Plone site. I will use a default, out-of-the-box installation of Plone, with no customizations. The functional test I'll present is written as an HTML table. In this post, I'm using Plone only as an example of a Web application. The test table I present uses generally-available "Selenese" commands that are not specific to Plone, so this example can be used as a mini-tutorial for writing HTML table-based tests in Selenium. If you want to see how Selenium can be used in conjunction with a stand-alone Twisted-based server, read this post.

Update: I just found out that Jason Huggins checked in a lot of Plone-specific Selenium code today, so obviously I haven't had the chance to experiment with that yet. I will follow up this post with another one, more targeted to specific features available in the Plone product version of Selenium (setUp, tearDown methods, and postResults for summarizing the test run).

Selenium is under very active development and new and important features are added to the tool on a daily basis. For this post, I checked out via subversion the latest version of the code as of March 8th.

Installing Selenium as a Plone product

One of the implementations of Selenium is a Plone product. I'll give a short procedure for installing Selenium inside an existing Plone site. In my case, I'm running Plone2-2.0.3-2 installed as an RPM on a Red Hat 9 server. My main Plone site is under /var/lib/plone2/main/.

1. Check out Selenium via subversion from svn://selenium.codehaus.org/selenium/scm/trunk
(I'll call SELENIUM_ROOT the directory where you checked out the code. In my case, it is /usr/local/selenium)

2. Copy SELENIUM_ROOT/code/python/Zope/Selenium to the Products directory of your Plone installation. In my case, I ran:

cp -r /usr/local/selenium/code/python/Zope/Selenium /var/lib/plone2/main/Products

3. Copy all the files and directories under SELENIUM_ROOT/code/javascript to the Products/Selenium/skins/selenium_javascript directory. In my case, I ran:

cp -r /usr/local/selenium/code/javascript/* /var/lib/plone2/main/Products/Selenium/skins/selenium_javascript

4. Go to the Products/Selenium/skins/selenium_javascript directory of your Plone site and rename all the .html, .js and .css files to their original name plus .dtml. A small Python script is provided in one of the readme files. Here it is (I called it rename_files.py):

import os
from os.path import join

# We need to append ".dtml" to all html, js, and css files so the "original" filename can be
# called in the browser.
# For example:
# By default, in Zope, TestRunner.html would be available from a URL as "http://localhost/TestRunner" with
# no ".html" attached. To preserve the original file name, we append ".dtml" to the file name.

for root, dirs, files in os.walk(os.getcwd()):
if '.svn' in dirs or 'CVS' in dirs:
dirs.remove('.svn') # don't visit Subversion or CVS directories

for file in files:
if file.endswith('.html') or file.endswith('.js') or file.endswith('.css'):
old_file = join(root, file)
new_file = old_file + '.dtml'
os.rename(old_file,new_file)


In my case, I cd-ed into /var/lib/plone2/main/Products/Selenium/skins/selenium_javascript and ran python rename_files.py.

5. Restart your Plone instance (this will need to be done every time you change anything on the file system, unless you start up Zope in debug mode). In my case, I ran:

/etc/rc.d/init.d/plone2 restart

6. Use the Plone QuickInstaller to install the Plone product.
  • Login to Plone as the administrator user -- not into the Zope Management Interface (ZMI), but into the Plone interface
  • Click on the "My Preferences" link in the navigation bar
  • Click on "Add/Remove products" in the "Plone setup" portlet
  • Select Selenium from the products available for installation and click Install
7. Point your browser to http://PLONE_ROOT_URL/TestRunner.html. In my case, PLONE_ROOT_URL is www.example.com:8080/Plone, so I pointed my browser to http://www.example.com:8080/Plone/TestRunner.html (example.com is of course not the real domain name for my Web site)

You should now see the default TestSuite shipped with Selenium. Feel free to click on the tests in the upper-left frame, then click on the green "Selected test" button to run the test. Playing with the tests in the default TestSuite is a good way to learn what kind of actions and checks are available in Selenium.

Creating a custom Test Suite and adding a test table

A non-documented feature of Selenium is that you can run test suites other than the default TestSuite.html by passing an argument to TestRunner.html. In my case, I created an HTML file called CustomTestSuite.html.dtml in the tests subdirectory of selenium_javascript (the full path to this directory is in my case /var/lib/plone2/main/Products/Selenium/skins/selenium_javascript/tests). Note that although CustomTestSuite is an HTML file, you need to end its name with .dtml, otherwise Plone will strip the .html portion from its name and interfere with the internal Selenium logic.

My CustomTestSuite.html.dtml file contains a single table:

Custom Test Suite
TestNewUser

TestNewUser is a link to tests/TestNewUser.html.dtml.

I then created a file called TestNewUser.html.dtml in the same tests directory. This file contains a table with only one row:

Test New User

To have Selenium load my custom test suite, I restarted Plone, then pointed my browser to:

http://www.example.com:8080/Plone/TestRunner.html?test=./tests/CustomTestSuite.html

This loaded the table from CustomTestSuite.html.dtml in the top left frame of the browser.

I then started to fill the table in TestNewUser.html.dtml by adding "Selenese" commands as rows. I'll show here the final version of the table:

Test New User
setVariable base_url 'http://www.example.com:8080/Plone'
setVariable logout_url '${base_url}/logout'
setVariable join_url '${base_url}/join_form'
open ${logout_url}
open ${base_url}
verifyTextPresent Welcome to Plone
click //a[@href='${join_url}']
verifyTitle Portal - Please sign in
verifyLocation join_form
verifyTextPresent Registration Form
verifyValue fullname
type fullname Test User Full Name
verifyValue username
setVariable random_user 'user'+(new Date()).getTime()
type username ${random_user}
verifyValue email
type email ${random_user}@example.com
verifyValue password
type password testUserPassword
verifyValue confirm
type confirm testUserPassword
click form.button.Register
verifyTextPresent You have been registered as a member
click //input[@value='Log in']
verifyTextPresent You are now logged in
click link:set up your Preferences
verifyLocation /plone_memberprefs_panel
verifyTextPresent My Preferences
click //img[@src='user.gif']
verifyLocation /personalize_form
verifyTextPresent Personal Preferences
select wysiwyg_editor Epoz
click listed
click form.button.Save
verifyTextPresent Your personal settings have been saved
verifyValue wysiwyg_editor Epoz
verifyValue listed off

This is a pretty long test that involves navigating through 6 or 7 pages. In a real-life testing situation this table should probably be split into several smaller tables, each one testing a specific piece of functionality. For the purpose of this tutorial I wanted as many various Selenese commands as I could fit into a still reasonably-sized table.

As it stands, the TestNewUser table tests the following functionality of a default, out-of-the-box Plone installation:
  • log out the existing user, if any
  • click the "New User?" button
  • fill in the registration form and save it
  • click the "log in" button
  • go to the My Preferences page
  • go to the Personal Preferences page
  • edit some of the preferences and save the form
  • check that the edited preferences were correctly saved
Instead of discussing the test table row by row, I'll discuss types of commands such as clicking on links, entering text, selecting value, clicking buttons, validating elements, etc. and I'll refer to the rows which use these commands.

Using variables

A brand-new feature of Selenium is the ability to use variables directly in the test tables. In the official documentation, the only way to deal with variables is via separate HTML pages where these variables are set. In the TesNewUser table, I'm using the setVariable command to set 3 variables: base_url, join_url and logout_url. The syntax for setting a variable is:

setVariablevar_namevar_value

(if the value is a string, it needs to be enclosed in quotes)

The syntax for getting the value of a variable is: ${var_name}

Note that join_url and logout_url use the value of base_url via interpolation, by enclosing the expression containing the variable in quotes:

setVariablelogout_url'${base_url}/logout'

An important use of a variable, especially when testing Web sites that provide log in functionality, is to set a random user ID that will be used in the test. TestNewUser does this via:

setVariablerandom_user'user'+(new Date()).getTime()

Note that in this case the value for the random_user variable is obtained by concatenating a string ('user') with the value returned by a JavaScript function (the getTime() method for a Date object). So you can use JavaScript code in your variable assignments.

Opening Web pages by URL

Web pages can be opened by their URL via the open command. An example is:

open${logout_url}

This particular "log out" command is the first real action in the TestNewUser table. It is necessary because after creating a user and logging in, that user will still be logged in when running the test table again. In Plone, the home page for a logged-in user is different from the home page of a non-logged in user, and I wanted to test the former situation.

Clicking on links

This is one of the trickiest aspects of writing tests in Selenium. In general, HTML elements on the Web pages you're trying to test can be referred to in Selenium commands via "element locators", which can be one of the following:
  • identifiers: the id or name attribute of the element
  • DOM traversal syntax: document.forms['myForm'].myDropdown
  • XPath syntax: //img[@alt='The image alt text']
Life is easy when HTML elements such as links have id attributes such as id="the_link_id". In this case, the command you need to use for clicking on the link is simply:

clickthe_link_id

Life is also pretty easy when links have text that is on the same line with the starting a tag and the closing /a tag. In this case, you can use the following XPath syntax (a good XPath tutorial recommended by Ian Bicking is here):

click//a[text() = "the link text]

A recently introduced Selenium command for accessing links by their text is the link: command:

clicklink:the link text

Note that you need to follow link by a colon, then immediately by the text of the link with no quotes. I used this command in the TestNewUser table like this:

clicklink:set up your Preferences

In other cases, especially when the link text is on a line by itself or spans multiple lines, the text method will not work and you will need to identify the link by some other attributes. Here is an example from TestNewUser:

click//a[@href='${join_url}']

This represents the command for clicking on the "New user?" link at the bottom of the "log in" portlet on the Plone home page. I initially tried to identify the link by the "New user?" text, but that method didn't work, because the starting a tag, the link text and the closing /a tag were on different lines. The only solution I found was to identify the link by its href tag. The XPath syntax I used was //a[@href='url'] where url is identified by the value of the variable ${join_url}.

Here is another example from TestNewUser:

click//img[@src='user.gif']

This represents the command for clicking on the Personal Preferences link on the "My Preferences" page. Again the text method didn't work, so this time I identified the link via the src tag of its image.

Clicking on submit buttons

The command you need to use for clicking on form submit buttons is click. The target of the command is the element locator for the button. This can be either the button's name attribute or its value attribute. An example of clicking a submit button by its name in TestNewUser is:

clickform.button.Register

This represents the command for clicking on the "register" button at the bottom of the new user registration form.

Here is an example of clicking a submit button by its value, using an XPath expression:

click//input[@value='Log in']

This represents clicking on the "log in" button on the Welcome page that is shown immediately after a successful registration.

Clicking on check boxes

The click command again accomplishes this. You need to indicate the name of the check-box field as the target of the command. The click command, when applied to a check box, toggles the value of that box. Here is an example from TestNewUser, where the check box for "Listed in searches" on the Personal Preferences page is clicked. The name of that box is "listed":

clicklisted

Note that the click command can also be used for clicking on radio buttons.

Entering text in input fields

If you need to fill in input fields in forms, use the type command, which takes 2 arguments: the name of the input field (you'll need to figure what that name is by inspecting the HTML source) and the value you need to type in that field. Here are some examples from TestNewUser, for filling in the user name and email information on the registration form:

typeusername${random_user}
typeemail${random_user}@example.com

Selecting values in drop-down lists

This is accomplished via the select command, which also takes 2 arguments: the name of the drop-down list field and the value you need to select in that list. An example from TestNewUser shows how to select the Epoz editor in the Personal Preferences page:

selectwysiwyg_editorEpoz

Verifying the state of the application

So far I have shown how Selenium can drive an embedded browser via "Selenese" commands. This is only one aspect of testing a Web application, since it indirectly verifies that the HTML elements it expects to click or open are indeed present. For direct verifications, Selenium provides a variety of "verify" commands that check the values of the different elements under test. Perhaps the simplest check is verifyTextPresent, which makes sure that a given snippet of text is present in a Web page. This example from TestNewUser checks that the default Plone home page contains "Welcome to Plone":

verifyTextPresentWelcome to Plone

Values for elements of a form (input fields, drop-down lists, check boxes) can be verified via the verifyValue command:

verifyValuewysiwyg_editorEpoz

verifyValuelistedoff

(note that for a check box such as "listed", the value that we compare with is either off or on)

To verify that a Web page contains a specific URL, use either verifyAbsouteLocation (which checks that the URL of the page is identical with a given string) or verifyLocation (which checks that the URL of the page ends with a given string).

This example from TestNewUser checks that the URL of the new user registration page ends with /join_form:

verifyLocationjoin_form

There are many other types of check commands available on the Selenium TestRunner reference page.

To be continued...

As I mentioned before, new and exciting features are added to Selenium on a daily basis. If you're interested in the tool and want to see what is going on with its development, browse the selenium-devel mailing list archive and consider joining the list and contributing.

I intend to follow up this article with other Selenium-related posts that will cover things such as:
  • using wildcards and regular expressions in verification commands
  • embedding JavaScript code in Selenese commands
  • organizing and reusing tests
  • using Plone-specific features such as Setup, TearDown and PostResults pages

9 comments:

Anonymous said...

Just stumbled upon your post on using Selenium when I Googled for 'Plone' and 'XPath'.

I've chatted with Jason Huggins before too- he's done an excellent job with this tool. Thanks for illustrating it so well!

-sprken

Anonymous said...

thanx for the the help. I started doing patches to get ids in all the plone links i wanted to use... so the xpath notation was a great help...

the logout doesn't work for me though...

Anonymous said...

Selenium is teh r0x0r! Been using it for just 3 days and have been really impressed with its power and flexibility.

I was just wondering, how would one go about clicking on an area map? For example:

<MAP NAME="mytitle">
<AREA SHAPE="rect" ALT="Blah" COORDS="308,87,445,125" HREF="/blah.html">
</MAP>

Personally, I don't write web pages like this (ie. don't work with lynx, etc.), but unfortunately I don't have a say in the matter for the current project I am working on now.

Is it possible to clickAndWait on a map? I'm guessing DOM might be the way to go, but...

Anonymous said...

Why do not talk about PloneSelenium ?

Anonymous said...

I have been trying to try out Selenium Twisted, and this article is the best guide that I have found to setting it up.

However, I seem to be unable to check out the trunk, and all of the downloads available from the main Selenium site do not include the /code folder mentioned in the other portion of the email. If anyone knows another way to get a copy, or could post a zip file online somewhere, please let me know.

Thanks,

david.whitesell@gmail.com, david.whitesell@mac.com

Anonymous said...

I agree that this article is the best that I've found on the web until now.

I've used it several times, so tnx very much!!!

Now, a commands reference that I read often, mentions a 'css' locator that I can't use yet (don't know enough CSS...). Maybe you could add something about it...

Shannon said...

I just downloaded the Selenium IDE, and it says 'setVariable' is an "Unknown command". In addition, the ${varname} syntax doesn't seem to work as you specify. Is this a feature of vanilla Selenium, or some modified version you're using? Thanks :)

Grig Gheorghiu said...

rehevkor5 -- I wrote this blog post a long time ago, when the Selenium IDE was not even on the radar screen. The syntax has also changed in the mean time. Selenium IDE is kept largely in sync with the Selenium syntax, so just use that, and click on the elements you need for the pages under test. Then you can add assertions/verifications to the HTML code that it generates for you.

Anonymous said...

hey, thx 4 the help. Now that setvariable command doesnt work. I wanna use a randomly generated text. How can that be done

Modifying EC2 security groups via AWS Lambda functions

One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...