Running headless Selenium WebDriver tests in Docker containers

In my previous post, I showed how to install firefox in headless mode on an Ubuntu box and how to use Xvfb to allow Selenium WebDriver scripts to run against firefox in headless mode.

Here I want to show how run each Selenium test suite in a Docker container, so that the suite gets access to its own firefox browser. This makes it easy to parallelize the test runs, and thus allows you to load test your Web infrastructure with real-life test cases.

Install docker-engine on Ubuntu 14.04

We import the dockerproject.org signing key and apt repo into our apt repositories, then we install the linux-image-extra and docker-engine packages.

# apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
# echo “deb https://apt.dockerproject.org/repo ubuntu-trusty main” > /etc/apt/sources.list.d/docker.list
# apt-get update
# apt-get install linux-image-extra-$(uname -r)
# apt-get install docker-engine

Start the docker service and verify that it is operational

Installing docker-engine actually starts up docker as well, but to start the service you do:

# service docker start

To verify that the docker service is operational, run a container based on the public “hello-world” Docker image:

# docker run hello-world
Unable to find image ‘hello-world:latest’ locally
latest: Pulling from library/hello-world
b901d36b6f2f: Pull complete
0a6ba66e537a: Pull complete
Digest: sha256:8be990ef2aeb16dbcb9271ddfe2610fa6658d13f6dfb8bc72074cc1ca36966a7
Status: Downloaded newer image for hello-world:latest
Hello from Docker.

This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the “hello-world” image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker Hub account:
https://hub.docker.com
For more examples and ideas, visit:
https://docs.docker.com/userguide/

Pull the ubuntu:trusty public Docker image

# docker pull ubuntu:trusty
trusty: Pulling from library/ubuntu
fcee8bcfe180: Pull complete
4cdc0cbc1936: Pull complete
d9e545b90db8: Pull complete
c4bea91afef3: Pull complete
Digest: sha256:3a7f4c0573b303f7a36b20ead6190bd81cabf323fc62c77d52fb8fa3e9f7edfe
Status: Downloaded newer image for ubuntu:trusty

# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
ubuntu trusty c4bea91afef3 3 days ago 187.9 MB
hello-world latest 0a6ba66e537a 12 weeks ago 960 B

Build custom Docker image for headless Selenium WebDriver testing

I created a directory called selwd on my host Ubuntu 14.04 box, and in that directory I created this Dockerfile:

FROM ubuntu:trusty
RUN echo “deb http://ppa.launchpad.net/mozillateam/firefox-next/ubuntu trusty main” > /etc/apt/sources.list.d//mozillateam-firefox-next-trusty.list 

RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CE49EC21 

RUN apt-get update 

RUN apt-get install -y firefox xvfb python-pip 

RUN pip install selenium 

RUN mkdir -p /root/selenium_wd_tests 

ADD sel_wd_new_user.py /root/selenium_wd_tests 

ADD xvfb.init /etc/init.d/xvfb 

RUN chmod +x /etc/init.d/xvfb 

RUN update-rc.d xvfb defaults
CMD (service xvfb start; export DISPLAY=:10; python /root/selenium_wd_tests/sel_wd_new_user.py)

This Dockerfile tells docker, via the FROM instruction, to create an image based on the ubuntu:trusty image that we pulled before (if we hadn’t pulled it, it would be pulled the first time our image was built).

The various RUN instructions specify commands to be run at build time. The above instructions add the Firefox Beta repository and key to the apt repositories inside the image, then install firefox, xvfb and python-pip. Then they install the selenium Python package via pip and create a directory structure for the Selenium tests.

The ADD instructions copy local files to the image. In my case, I copy one Selenium WebDriver Python script, and an init.d-type file for starting Xvfb as a service (by default it starts in the foreground, which is not something I want inside a Docker container).

The last two RUN instructions make the /etc/init.d/xvfb script executable and run update-rc.d to install it as a service. The xvfb script is the usual init.d wrapper around a command, in my case this command:

PROG=”/usr/bin/Xvfb” 

PROG_OPTIONS=”:10 -ac”

Here is a gist for the xvfb.init script for reference.

Finally, the CMD instruction specifies what gets executed when a container based on this image starts up (assuming no other commands are given in the ‘docker run’ command-line for this container). The CMD instruction in the Dockerfile above starts up the xvfb service (which connects to DISPLAY 10 as specified in the xvfb init script), sets the DISPLAY environment variable to 10, then runs the Selenium WebDriver script sel_wd_new_user.py, which will launch firefox in headless mode and execute its commands against it.

Here’s the official documentation for Dockerfile instructions.

To build a Docker image based on this Dockerfile, run:

# docker build -t selwd:v1 .

selwd is the name of the image and v1 is a tag associated with this name. The dot . tells docker to look for a Dockerfile in the current directory.

The build process will take a while intially because it will install all the dependencies necessary for the packages we are installing with apt. Every time you make a modification to the Dockerfile, you need to run ‘docker build’ again, but subsequent runs will be much faster.

Run Docker containers based on the custom image

At this point, we are ready to run Docker containers based on the selwd image we created above.

Here’s how to run a single container:

# docker run --rm selwd:v1

In this format, the command specified in the CMD instruction inside the Dockerfile will get executed, then the container will stop. This is exactly what we need: we run our Selenium WebDriver tests against headless firefox, inside their own container isolated from any other container.

The output of the ‘docker run’ command above is:


Starting : X Virtual Frame Buffer 

. — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Ran 1 test in 40.441s

OK
(or a traceback if the Selenium test encountered an error)

Note that we also specified the rm flag to ‘docker run’ so that the container gets removed once it stops — otherwise these short-lived containers will be kept around and will pile up, as you can see for yourself if you run:

# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 

6c9673e59585 selwd:v1 “/bin/bash” 5 minutes ago Exited (130) 5 seconds ago modest_mccarthy 980651e1b167 selwd:v1 “/bin/sh -c ‘(service” 9 minutes ago Exited (0) 8 minutes ago stupefied_turing 

4a9b2f4c8c28 selwd:v1 “/bin/sh -c ‘(service” 13 minutes ago Exited (0) 12 minutes ago nostalgic_ride 

9f1fa953c83b selwd:v1 “/bin/sh -c ‘(service” 13 minutes ago Exited (0) 12 minutes ago admiring_ride 

c15b180832f6 selwd:v1 “/bin/sh -c ‘(service” 13 minutes ago Exited (0) 12 minutes ago jovial_booth .....etc

If you do have large numbers of containers that you want to remove in one go, use this command:

# docker rm `docker ps -aq`

For troubleshooting purposes, we can run a container in interactive mode (with the -i and -t flags) and specify a shell command to be executed on startup, which will override the CMD instruction in the Dockerfile:

# docker run -it selwd:v1 /bin/bash 

root@6c9673e59585:/#
At the bash prompt, you can run the shell commands specified by the Dockerfile CMD instruction in order to see interactively what is going on. The official ‘docker run’ documentation has lots of details.

One other thing I found useful for troubleshooting Selenium WebDriver scripts running against headless firefox was to have the scripts take screenshots during their execution with the save_screenshot command:

driver.save_screenshot(“before_place_order.png”)
# Click Place Order driver.find_element_by_xpath("//*[@id='order_submit_button']").click()
driver.save_screenshot(“after_place_order.png”)

I then inspected the PNG files to see what was going on.

Running multiple Docker containers for load testing

Because our Selenium WebDriver tests run isolated in their own Docker container, it enables us to run N containers in parallel to do a poor man’s load testing of our site.

We’ll use the -d option to ‘docker run’ to run each container in ‘detached’ mode. Here is a bash script that launches COUNT Docker containers, where COUNT is the 1st command line argument, or 2 by default:


#!/bin/bash
COUNT=$1 

if [ -z “$COUNT” ]; then 

 COUNT=2 

fi
for i in `seq 1 $COUNT`; do 

 docker run -d selwd:v1 

done

The output of the script consists in a list of container IDs, one for each container that was launched.

Note that if you launch a container in detached mode with -d, you can’t specify the rm flag to have the container removed automatically when it stops. You will need to periodically clean up your containers with the command I referenced above (docker rm `docker ps -aq`).

To inspect the output of the Selenium scripts in the containers that were launched, first get the container IDs:

# docker ps -a 

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 

6fb931689c03 selwd:v1 “/bin/sh -c ‘(service” About an hour ago Exited (0) About an hour ago grave_northcutt 

1b82ef59ad46 selwd:v1 “/bin/sh -c ‘(service” About an hour ago Exited (0) About an hour ago admiring_fermat

Then run ‘docker logs <container_id>’ to see the output for a specific container:

# docker logs 6fb931689c03 

. — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 

Ran 1 test in 68.436s

OK 

Starting : X Virtual Frame Buffer
Have fun load testing your site!