Friday, February 05, 2016

Setting up Jenkins to run headless Selenium tests in Docker containers

This is the third post in a series on running headless Selenium WebDriver tests. Here are the first two posts:
  1. Running Selenium WebDriver tests using Firefox headless mode on Ubuntu
  2. Running headless Selenium WebDriver tests in Docker containers
In this post I will show how to add the final piece to this workflow, namely how to fully automate the execution of Selenium-based WebDriver tests running Firefox in headless mode in Docker containers. I will use Jenkins for this example, but the same applies to other continuous integration systems.

1) Install docker-engine on the server running Jenkins (I covered this in my post #2 above)

2) Add the jenkins user to the docker group so that Jenkins can run the docker command-line tool in order to communicate with the docker daemon. Remember to restart Jenkins after doing this.

3) Go through the rest of the workflow in my post above ("Running headless Selenium WebDriver tests in Docker containers") and make sure you can run all the commands in that post from the command line of the server running Jenkins.

4) Create a directory structure for your Selenium WebDriver tests (mine are written in Python). 

I have a directory called selenium-docker which contains a directory called tests, under which I put all my Python WebDriver tests named sel_wd_*.py. I also  have a simple shell script I named which does the following:


TARGET=$1 # e.g. (if not specified, the default is

for f in `ls tests/sel_wd_*.py`; do
    echo Running $f against $TARGET
    python $f $TARGET

My selenium-docker directory also contains the xvfb.init file I need for starting up Xvfb in the container, and finally it contains this Dockerfile:

FROM ubuntu:trusty

RUN echo "deb trusty main" > /etc/apt/sources.list.d//mozillateam-firefox-next-trusty.list
RUN apt-key adv --keyserver --recv-keys CE49EC21
RUN apt-get update
RUN apt-get install -y firefox xvfb python-pip
RUN pip install selenium
RUN mkdir -p /root/selenium/tests

ADD tests /root/selenium/tests
ADD /root/selenium

ADD xvfb.init /etc/init.d/xvfb
RUN chmod +x /etc/init.d/xvfb
RUN update-rc.d xvfb defaults


CMD (service xvfb start; export DISPLAY=:10; cd /root/selenium; ./ $TARGET)

I explained what this Dockerfile achieves in the 2nd post referenced above. The ADD instructions will copy all the files in the tests directory to the directory called /root/selenium/tests, and will copy to /root/selenium. The ENV variable TARGET represents the URL against which we want to run our Selenium tests. It is set by default to, and is used as the first argument when running in the CMD instruction.

At this point, I checked in the selenium-docker directory and all files and directories under it into a Github repository I will call 'devops'.

5) Create a new Jenkins project (I usually create a New Item and copy it from an existing project).

I specified that the build is parameterized and I indicated a choice parameter called TARGET_HOST with a few host/domain names that I want to test. I also specified Git as the Source Code Management type, and I indicated the URL of the devops repository on Github. Most of the action of course happens in the Jenkins build step, which in my case is of type "Execute shell". Here it is:


set +e


cd $WORKSPACE/selenium-docker

# build the image out of the Dockerfile in the current directory
/usr/bin/docker build -t $IMAGE_NAME .

# run a container based on the image
CONTAINER_ID=`/usr/bin/docker run -d -e "TARGET=$TARGET_HOST" $IMAGE_NAME`

# while the container is still running, sleep and check logs; repeat every 40 sec
while [ $? -eq 0 ];
  sleep 40
  /usr/bin/docker logs $CONTAINER_ID
  /usr/bin/docker ps | grep $IMAGE_NAME

# docker logs sends errors to stderr so we need to save its output to a file first
/usr/bin/docker logs $CONTAINER_ID > d.out 2>&1

# remove the container so they don't keep accumulating
docker rm $CONTAINER_ID

# mark jenkins build as failed if log output contains FAILED
grep "FAILED" d.out

if [[ $? -eq 0 ]]; then  
  rm d.out 
  exit 1
  rm d.out
  exit 0

Some notes:
  • it is recommended that you specify #!/bin/bash as the 1st line of your script, to make sure that bash is the shell that is being used
  • use set +e if you want the Jenkins shell script to continue after hitting a non-zero return code (the default behavior is for the script to stop on the first line it encounters an error and for the build to be marked as failed; subsequent lines won't get executed, resulting in much pulling of hair)
  • the Jenkins script will build a new image every time it runs, so that we make sure we have updated Selenium scripts in place
  • when running the container via docker run, we specify -e "TARGET=$TARGET_HOST" as an extra command line argument. This will override the ENV variable named TARGET in the Dockerfile with the value received from the Jenkins multiple choice dropdown for TARGET_HOST
  • the main part of the shell script stays in a while loop that checks for the return code of "/usr/bin/docker ps | grep $IMAGE_NAME". This is so we wait for all the Selenium tests to finish, at which point docker ps will not show the container running anymore (you can still see the container by running docker ps -a)
  • once the tests finish, we save the stdout and stderr of the docker logs command for our container to a file (this is so we capture both stdout and stderr; at first I tried something like docker logs $CONTAINER_ID | grep FAILED but this was never successful, because it was grep-ing against stdout, and errors are sent to stderr)
  • we grep the file (d.out) for the string FAILED and if we find it, we exit with code 1, i.e. unsuccessful as far as Jenkins is concerned. If we don't find it, we exit successfully with code 0.

No comments:

Modifying EC2 security groups via AWS Lambda functions

One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...