Running headless Selenium WebDriver tests in Docker containers

In my previous post, I showed how to install firefox in headless mode on an Ubuntu box and how to use Xvfb to allow Selenium WebDriver scripts to run against firefox in headless mode.

Here I want to show how run each Selenium test suite in a Docker container, so that the suite gets access to its own firefox browser. This makes it easy to parallelize the test runs, and thus allows you to load test your Web infrastructure with real-life test cases.

Install docker-engine on Ubuntu 14.04

We import the dockerproject.org signing key and apt repo into our apt repositories, then we install the linux-image-extra and docker-engine packages.

# apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
# echo “deb https://apt.dockerproject.org/repo ubuntu-trusty main” > /etc/apt/sources.list.d/docker.list
# apt-get update
# apt-get install linux-image-extra-$(uname -r)
# apt-get install docker-engine

Start the docker service and verify that it is operational

Installing docker-engine actually starts up docker as well, but to start the service you do:

# service docker start

To verify that the docker service is operational, run a container based on the public “hello-world” Docker image:

# docker run hello-world
Unable to find image ‘hello-world:latest’ locally
latest: Pulling from library/hello-world
b901d36b6f2f: Pull complete
0a6ba66e537a: Pull complete
Digest: sha256:8be990ef2aeb16dbcb9271ddfe2610fa6658d13f6dfb8bc72074cc1ca36966a7
Status: Downloaded newer image for hello-world:latest
Hello from Docker.

This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the “hello-world” image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker Hub account:
https://hub.docker.com
For more examples and ideas, visit:
https://docs.docker.com/userguide/

Pull the ubuntu:trusty public Docker image

# docker pull ubuntu:trusty
trusty: Pulling from library/ubuntu
fcee8bcfe180: Pull complete
4cdc0cbc1936: Pull complete
d9e545b90db8: Pull complete
c4bea91afef3: Pull complete
Digest: sha256:3a7f4c0573b303f7a36b20ead6190bd81cabf323fc62c77d52fb8fa3e9f7edfe
Status: Downloaded newer image for ubuntu:trusty

# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
ubuntu trusty c4bea91afef3 3 days ago 187.9 MB
hello-world latest 0a6ba66e537a 12 weeks ago 960 B

Build custom Docker image for headless Selenium WebDriver testing

I created a directory called selwd on my host Ubuntu 14.04 box, and in that directory I created this Dockerfile:

FROM ubuntu:trusty
RUN echo “deb http://ppa.launchpad.net/mozillateam/firefox-next/ubuntu trusty main” > /etc/apt/sources.list.d//mozillateam-firefox-next-trusty.list 

RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CE49EC21 

RUN apt-get update 

RUN apt-get install -y firefox xvfb python-pip 

RUN pip install selenium 

RUN mkdir -p /root/selenium_wd_tests 

ADD sel_wd_new_user.py /root/selenium_wd_tests 

ADD xvfb.init /etc/init.d/xvfb 

RUN chmod +x /etc/init.d/xvfb 

RUN update-rc.d xvfb defaults
CMD (service xvfb start; export DISPLAY=:10; python /root/selenium_wd_tests/sel_wd_new_user.py)

This Dockerfile tells docker, via the FROM instruction, to create an image based on the ubuntu:trusty image that we pulled before (if we hadn’t pulled it, it would be pulled the first time our image was built).

The various RUN instructions specify commands to be run at build time. The above instructions add the Firefox Beta repository and key to the apt repositories inside the image, then install firefox, xvfb and python-pip. Then they install the selenium Python package via pip and create a directory structure for the Selenium tests.

The ADD instructions copy local files to the image. In my case, I copy one Selenium WebDriver Python script, and an init.d-type file for starting Xvfb as a service (by default it starts in the foreground, which is not something I want inside a Docker container).

The last two RUN instructions make the /etc/init.d/xvfb script executable and run update-rc.d to install it as a service. The xvfb script is the usual init.d wrapper around a command, in my case this command:

PROG=”/usr/bin/Xvfb” 

PROG_OPTIONS=”:10 -ac”

Here is a gist for the xvfb.init script for reference.

Finally, the CMD instruction specifies what gets executed when a container based on this image starts up (assuming no other commands are given in the ‘docker run’ command-line for this container). The CMD instruction in the Dockerfile above starts up the xvfb service (which connects to DISPLAY 10 as specified in the xvfb init script), sets the DISPLAY environment variable to 10, then runs the Selenium WebDriver script sel_wd_new_user.py, which will launch firefox in headless mode and execute its commands against it.

Here’s the official documentation for Dockerfile instructions.

To build a Docker image based on this Dockerfile, run:

# docker build -t selwd:v1 .

selwd is the name of the image and v1 is a tag associated with this name. The dot . tells docker to look for a Dockerfile in the current directory.

The build process will take a while intially because it will install all the dependencies necessary for the packages we are installing with apt. Every time you make a modification to the Dockerfile, you need to run ‘docker build’ again, but subsequent runs will be much faster.

Run Docker containers based on the custom image

At this point, we are ready to run Docker containers based on the selwd image we created above.

Here’s how to run a single container:

# docker run --rm selwd:v1

In this format, the command specified in the CMD instruction inside the Dockerfile will get executed, then the container will stop. This is exactly what we need: we run our Selenium WebDriver tests against headless firefox, inside their own container isolated from any other container.

The output of the ‘docker run’ command above is:


Starting : X Virtual Frame Buffer 

. — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Ran 1 test in 40.441s

OK
(or a traceback if the Selenium test encountered an error)

Note that we also specified the rm flag to ‘docker run’ so that the container gets removed once it stops — otherwise these short-lived containers will be kept around and will pile up, as you can see for yourself if you run:

# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 

6c9673e59585 selwd:v1 “/bin/bash” 5 minutes ago Exited (130) 5 seconds ago modest_mccarthy 980651e1b167 selwd:v1 “/bin/sh -c ‘(service” 9 minutes ago Exited (0) 8 minutes ago stupefied_turing 

4a9b2f4c8c28 selwd:v1 “/bin/sh -c ‘(service” 13 minutes ago Exited (0) 12 minutes ago nostalgic_ride 

9f1fa953c83b selwd:v1 “/bin/sh -c ‘(service” 13 minutes ago Exited (0) 12 minutes ago admiring_ride 

c15b180832f6 selwd:v1 “/bin/sh -c ‘(service” 13 minutes ago Exited (0) 12 minutes ago jovial_booth .....etc

If you do have large numbers of containers that you want to remove in one go, use this command:

# docker rm `docker ps -aq`

For troubleshooting purposes, we can run a container in interactive mode (with the -i and -t flags) and specify a shell command to be executed on startup, which will override the CMD instruction in the Dockerfile:

# docker run -it selwd:v1 /bin/bash 

root@6c9673e59585:/#
At the bash prompt, you can run the shell commands specified by the Dockerfile CMD instruction in order to see interactively what is going on. The official ‘docker run’ documentation has lots of details.

One other thing I found useful for troubleshooting Selenium WebDriver scripts running against headless firefox was to have the scripts take screenshots during their execution with the save_screenshot command:

driver.save_screenshot(“before_place_order.png”)
# Click Place Order driver.find_element_by_xpath("//*[@id='order_submit_button']").click()
driver.save_screenshot(“after_place_order.png”)

I then inspected the PNG files to see what was going on.

Running multiple Docker containers for load testing

Because our Selenium WebDriver tests run isolated in their own Docker container, it enables us to run N containers in parallel to do a poor man’s load testing of our site.

We’ll use the -d option to ‘docker run’ to run each container in ‘detached’ mode. Here is a bash script that launches COUNT Docker containers, where COUNT is the 1st command line argument, or 2 by default:


#!/bin/bash
COUNT=$1 

if [ -z “$COUNT” ]; then 

 COUNT=2 

fi
for i in `seq 1 $COUNT`; do 

 docker run -d selwd:v1 

done

The output of the script consists in a list of container IDs, one for each container that was launched.

Note that if you launch a container in detached mode with -d, you can’t specify the rm flag to have the container removed automatically when it stops. You will need to periodically clean up your containers with the command I referenced above (docker rm `docker ps -aq`).

To inspect the output of the Selenium scripts in the containers that were launched, first get the container IDs:

# docker ps -a 

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 

6fb931689c03 selwd:v1 “/bin/sh -c ‘(service” About an hour ago Exited (0) About an hour ago grave_northcutt 

1b82ef59ad46 selwd:v1 “/bin/sh -c ‘(service” About an hour ago Exited (0) About an hour ago admiring_fermat

Then run ‘docker logs <container_id>’ to see the output for a specific container:

# docker logs 6fb931689c03 

. — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 

Ran 1 test in 68.436s

OK 

Starting : X Virtual Frame Buffer
Have fun load testing your site!

Running Selenium WebDriver tests using Firefox headless mode on Ubuntu

Selenium IDE is a very good tool for recording and troubleshooting Selenium tests, but you are limited to clicking around in a GUI. For a better testing workflow, including load testing, you need to use Selenium WebDriver, which can programatically drive a browser and run Selenium test cases.

In its default mode, WebDriver will launch a browser and run the test scripts in the browser, then exit. If you like to work exclusively from the command line, then you need to look into running the browser in headless mode. Fortunately, this is easy to do with Firefox on Ubuntu. Here’s what you need to do:

Install the official Firefox Beta PPA:

$ sudo apt-add-repository ppa:mozillateam/firefox-next

(this will add the file /etc/apt/sources.list.d/mozillateam-firefox-next-trusty.list and also fetch the PPA’s key, which enables your Ubuntu system to verify that the packages in the PPA have not been interfered with since they were built)

Run apt-get update:

$ sudo apt-get update

Install firefox and xvfb (the X windows virtual framebuffer) packages:

$ sudo apt-get install firefox xvfb

Run Xvfb in the background and specify a display number (10 in my example):

$ Xvfb :10 -ac &

Set the DISPLAY variable to the number you chose:

$ export DISPLAY=:10

Test that you can run firefox in the foreground with no errors:

$ firefox
(kill it with Ctrl-C)

Now run your regular Selenium WebDriver scripts (no modifications required if they already use Firefox as their browser).

Here is an example of a script I have written in Python, which clicks on a category link in an e-commerce store, adds an item to the cart, that starts filling out the user’s information in the cart:

# -*- coding: utf-8 -*-
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import NoAlertPresentException
import unittest, time, re, random

class SelWebdriverNewUser(unittest.TestCase):
  def setUp(self):
    self.driver = webdriver.Firefox()
    self.driver.implicitly_wait(20)
    self.base_url = “http://myhost.mycompany.com/"
    self.verificationErrors = []
    self.accept_next_alert = True
  

  def test_sel_webdriver_new_user(self):
    driver = self.driver
    HOST = “myhost.mycompany.com”
    RANDINT = random.random()*10000
    driver.get(“https://” + HOST)

    # Click on category link
    driver.find_element_by_xpath(“//*[@id=’nav’]/ol/li[3]/a”).click()
    # Click on sub-category link
    driver.find_element_by_xpath(“//*[@id=’top’]/body/div/div[2]/div[2]/div/div[2]/ul/li[4]/a/span”).click()
    # Click on product image
    driver.find_element_by_xpath(“//*[@id=’product-collection-image-374']”).click()
    # Click Checkout button
    driver.find_element_by_xpath(“//*[@id=’checkout-button’]/span/span”).click()
    driver.find_element_by_id(“billing:firstname”).clear()
driver.find_element_by_id(“billing:firstname”).send_keys(“selenium”, RANDINT, “_fname”)
    driver.find_element_by_id(“billing:lastname”).clear()
driver.find_element_by_id(“billing:lastname”).send_keys(“selenium”, RANDINT, “_lname”)
    # Click Place Order
    driver.find_element_by_xpath(“//*[@id=’order_submit_button’]”).click()

  def is_element_present(self, how, what):
    try: self.driver.find_element(by=how, value=what)
    except NoSuchElementException as e: return False
    return True

  def is_alert_present(self):
    try: self.driver.switch_to_alert()
    except NoAlertPresentException as e: return False
    return True

  def close_alert_and_get_its_text(self):
    try:
      alert = self.driver.switch_to_alert()
      alert_text = alert.text
      if self.accept_next_alert:
        alert.accept()
      else:
        alert.dismiss()
      return alert_text
    finally: self.accept_next_alert = True

def tearDown(self):
    self.driver.quit()
    self.assertEqual([], self.verificationErrors)

if __name__ == “__main__”:
unittest.main()

To run this script, you first need to install the selenium Python package:

$ sudo pip install selenium

Then run the script (called selenium_webdriver_new_user.py in my case):

$ python selenium_webdriver_new_user.py

After hopefully not so long of a wait, you should see a successful test run:

$ python selenium_webdriver_new_user.py
.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
Ran 1 test in 29.317s

A few notes regarding Selenium WebDriver scripts.

I was stumped for a while when I was trying to use the “find_element_by_id” form of finding an HTML element on a Web page. It was working fine in Selenium IDE, but Selenium WebDriver couldn’t find that element. I had to resort to finding all elements via their XPath id using “find_element_by_xpath”. Fortunately, Chrome for example makes it easy to right click an element on a page, choose Inspect, then righ click the HTML code for the element and choose Copy->Copy XPath to get their id which can then be pasted in the Selenium WebDriver script.

I also had to use time.sleep(N) (where N is in seconds at least for Python) to wait for certain elements of the page to load asynchronously. I know it’s not best practices, but it works.

Distributing a beta version of an iOS app

I am not an iOS expert by any means, but recently I’ve had to maintain an iOS app and distribute it to beta testers. I had to jump through a few hoops, so I am documenting here the steps I had to take.

First of all, I am using Xcode 6.4 with the Fabric 2.1.1 plugin. I assume you are already signed up for the Fabric/Crashlytics service and that you also have an Apple developer account.

1. Ask each beta tester to send you the UUID of the devices they want to run your app on.
2. Go to developer.apple.com -> “Certificates, Identifiers and Profiles” -> “Devices” and add each device with its associated UUID. Let’s say you add a device called “Tom’s iPhone 6s” with its UUID.
3. Go to Xcode -> Preferences -> Accounts. If you already have an account set up, remove it by selecting it and clicking the minus icon on the lower left side. Add an account: click the plus icon, choose “Add Apple ID” and enter your Apple ID and password. This will import your Apple developer provisioning profile into Xcode, with the newly added device UUIDs (note: there may be a better way of adding/modifying the provisioning profile within Xcode but this worked for me)
4. Make sure the Fabric plugin is running on your Mac.
5. Go to Xcode and choose the iOS application you want to distribute. Choose iOS Device as the target for the build.
6. Go to Xcode -> Product -> Archive. This will build the app, then the Fabric plugin will pop up a message box asking you if you want to distribute the archive build. Click Distribute.
7. The Fabric plugin will pop up a dialog box asking you for the email of the tester(s) you want to invite. Enter one or more email addresses. Enter release notes. At this point the Fabric plugin will upload your app build to the Fabric site and notify the tester(s) that they are invited to test the app.