Using Selenium to test a Plone site (part 2)

In this post I'll talk about some Plone-specific features available in Selenium, such as setup, tearDown and postResults methods. See part 1 for more general Selenium features that can be used to test any Web site.

Jason Huggins recently released a new version (selenium-0.3rc2-plone.zip) of the Plone product implementation of Selenium. If you already have an old version of the Selenium Plone product installed, you need to uninstall it and install the new version. Alternatively, you can check out the latest Selenium source code via subversion, then follow the instructions in Installing Selenium as a Plone product in my previous post.

The most important addition in the Selenium Plone product is a "Plone tool" called FunctionalTestTool.py (on my test system, it is in /var/lib/plone2/main/Products/Selenium). A Plone tool is Python code that adds functionality to a Plone-based Web site. You can use a tool for example wherever you would use a CGI script. You can find a good intro on Plone tools here.

Every Plone tool has an id which is used as part of the URL that actually invokes the tool. The id for FunctionalTestTool is selenium_ft_tool. The 3 methods that are "public" (i.e. callable from other pages) in FunctionalTestTool.py are setup, tearDown and postResults. If you wanted to invoke the setup method for example, you would need to open the URL http://www.example.com:8080/Plone/selenium_ft_tool/setup followed by optional arguments.

The purpose of the setup method is to facilitate the creation of Plone users at the beginning of each Selenium test. In my previous post on Selenium and Plone, I used the example of testing the "New user" functionality in Plone. I used a Selenium test table to fill in the necessary values for user name, password and email in the Plone registration form, then submitted that form so that the user can be created. If you want to test specific functionality for a new user, having to go through the process of filling the registration form at the beginning of each of your tests can quickly become tedious. Enter the setup method, which allows you to create a new user in one line of your test table.

In its current incarnation, the setup method accepts a user_type argument which can be either Member of Manager. The method uses the Plone API (via utils.getToolByName(self, 'portal_membership').addMember) to create a new user of the specified type, with a random user name (prefixed by FTuser or FTmanageruser, depending on the user type), with a password set to the user name, and with an email set to username@example.org. I modified the FunctionalTestTool.py code and added a second parameter called user, which, if specified, sets both the user name and the password to the specified value. This is so that I can create my own random user name at the very beginning of my test table, then use that value as the user name throughout my test. You can find my modified version of FunctionalTestTool.py here.

The purpose of the tearDown method is to delete the user created via setup at the end of your test, so that the Plone user database doesn't get cluttered with a new user for each test you run. The tearDown method takes a user name in the user parameter and uses the Plone API (via utils.getToolByName(self, 'portal_membership').pruneMemberDataContents()) to delete the specified user.

The purpose of the postResults method is to save the HTML output (the test table(s) with rows colored green or red) produced by the current test run into its own file, so that you can keep a record of all your test runs. The method also saves the Pass/Fail/Total test count for the current test run in a plain text file which can be archived for future inspection. The 2 files generated by postResults can be found in /var/lib/plone2/main/Products/Selenium/skins/selenium_test_results. The HTML file is called selenium-results-USER_AGENT.html, where USER_AGENT is set to the name of the browser you're using. In my case, the file is called selenium-results-Firefox.html. The test count file is called selenium-results-metadata-USER_AGENT.txt.dtml. In my case, the file is called selenium-results-metadata-Firefox.txt.dtml.

Currently, you can't call postResults directly. It gets called implicitly if you run TestRunner.html with auto=true. In my case, if I wanted to post the results at the end of the test run for the test suite shipped with Selenium, I would open this URL: http://www.example.com:8080/Plone/TestRunner.html?test=PloneTestSuite.html&auto=true
(replace example with your domain name)

Here is an example of a test table that uses the methods I described so far. Its purpose is to test the capability for brand new Plone users to edit their home page. Here is the test table I created:

testEditHomePage
setVariable	username	'user'+(new Date()).getTime()
open	./selenium_ft_tool/setup?user=${username}
setVariable	base_url	'http://www.example.com:8080/Plone'
open	${base_url}
type	__ac_name	${username}
type	__ac_password	${username}
click	submit
verifyTextPresent	Welcome! You are now logged in
setVariable	myfolder_url	'${base_url}/Members/${username}/folder_contents'
click	//a[@href='${myfolder_url}']
verifyTextPresent	Home page area that contains the items created and collected by ${username}
setVariable	homepage_url	'${base_url}/Members/${username}/index_html/document_view'
click	//a[@href='${homepage_url}']
verifyTextPresent	This item does not have any body text, click the edit tab to change it.
setVariable	edit_homepage_url	'${base_url}/Members/${username}/index_html/document_edit_form'
click	//a[@href='${edit_homepage_url}']
verifyTextPresent	Fill in the details of this document.
type	text	Hello World!
click	form.button.Save
verifyTextPresent	Document changes saved
verifyTextPresent	Hello World!
setVariable	member_url	'${base_url}/Members/${username}'
open	${member_url}
verifyTextPresent	Home page for ${username}
verifyTextPresent	Hello World!
open	./selenium_ft_tool/tearDown?user=${username}

The test table starts by assigning a random value to the username, then it calls the setup method with the user argument to create that user. It continues by logging in into Plone via the "log in" form on the main Plone page, then it opens the "my folder" page, it goes to the user's home page and fills in "Hello, World!" as the text of the page. It finally saves the home page, then opens the Members/username Plone page to verify that the user's home page is there. Along the way, the test uses the verifyTextPresent assertion to check that expected text values are indeed present on the various pages that it opens. The last line in the test table uses the tearDown method to delete the newly-created user.

The Plone product version of Selenium ships with a test suite that contains one test table called testJoinPortal. The test suite file (called PloneTestSuite.html.dtml) and the testJoinPortal file (called testJoinPortal.html.dtml) are both in /var/lib/plone2/main/Products/Selenium/skins/ftests_browser_driven. I called my custom test table testEditHomePage.html.dtml and I edited PloneTestSuite.html.dtml to add a new row corresponding to testEditHomePage.html. I then restarted Plone and I went to http://www.example.com:8080/Plone/TestRunner.html?test=PloneTestSuite.html&auto=true.
The test suite got executed automatically (because auto was set to true), and postResults got called automatically at the end of the test suite run (also because auto was set to true).

The frame where the emebedded browser runs shows this at the end of the test suite run:

Results have been successfully posted to the server here:
/var/lib/plone2/main/Products/Selenium/skins/selenium_test_results

selenium-results-metadata-Firefox.txt
selenium-results-Firefox

Clicking on the selenium-results-Firefox link shows the saved output of the test suite run:

Test Suite

testJoinPortal

testEditHomePage

testJoinPortal
open	./join_form
verifyTextPresent	Registration Form
type	fullname	Jason Huggins
type	username	jrhuggins12345
type	email	jrhuggins@example.com
type	password	12345
type	confirm	12345
clickAndWait	form.button.Register
verifyTextPresent	You have been registered as a member.
clickAndWait	document.forms[1].elements[3]
verifyTextPresent	Welcome! You are now logged in.
open	./selenium_ft_tool/tearDown?user=jrhuggins12345

testEditHomePage
setVariable	username	'user'+(new Date()).getTime()
open	./selenium_ft_tool/setup?user=${username}
setVariable	base_url	'http://www.example.com:8080/Plone'
open	${base_url}
type	__ac_name	${username}
type	__ac_password	${username}
click	submit
verifyTextPresent	Welcome! You are now logged in
setVariable	myfolder_url	'${base_url}/Members/${username}/folder_contents'
click	//a[@href='${myfolder_url}']
verifyTextPresent	Home page area that contains the items created and collected by ${username}
setVariable	homepage_url	'${base_url}/Members/${username}/index_html/document_view'
click	//a[@href='${homepage_url}']
verifyTextPresent	This item does not have any body text, click the edit tab to change it.
setVariable	edit_homepage_url	'${base_url}/Members/${username}/index_html/document_edit_form'
click	//a[@href='${edit_homepage_url}']
verifyTextPresent	Fill in the details of this document.
type	text	Hello World!
click	form.button.Save
verifyTextPresent	Document changes saved
verifyTextPresent	Hello World!
setVariable	member_url	'${base_url}/Members/${username}'
open	${member_url}
verifyTextPresent	Home page for ${username}
verifyTextPresent	Hello World!
open	./selenium_ft_tool/tearDown?user=${username}

Clicking on the selenium-results-metadata-Firefox.txt link shows the test total/pass/fail count
for the current test suite run:

totalTime: 7
numTestFailures: 0
numCommandPasses: 11
numCommandFailures: 0
numCommandErrors: 0
result: passed
numTestPasses: 2

Conclusion

The new Plone test tool added in the latest version of the Plone product implementation of Selenium provides valuable functionality for test management via the setup/tearDown/postResults methods. Even though these methods are Plone-specific, the ideas behind them should be easily adaptable to other Web applications. The only requirement is for the application to provide API hooks for user management functionality. This requirement is easily met by Plone, and I have to say I found it very easy to extend the Plone tool, even though I had no prior experience in writing Plone-specific code.

As a future Selenium enhancement, I'd like to see a way to specify 'SetUp' and 'TearDown' test tables at the test suite level that would automatically run at the beginning and at the end of each test suite run. In my example above, the log in process could be encapsulated in a SetUp table, and all other tables in the suite would then implicitly use the functionality in the SetUp table.

The alternative of course would be to extend the FunctionalTestTool and create a log_in method that would use the Plone API to log in the newly-created user. The current setup method in FunctionalTestTool could then get an extra parameter (such as login=true) that would call the log_in method if specified.