Some notes on Ansible playbooks and roles

Some quick notes I jotted down while documenting our Ansible setup. Maybe they will be helpful for people new to Ansible.

Ansible playbooks and roles

Playbooks are YAML files that specify which roles are applied to hosts of certain type.

Example: api-servers.yml

$ cat api-servers.yml

---

- hosts: api

 sudo: yes

 roles:

   - base

   - tuning

   - postfix

   - monitoring

   - nginx

   - api

   - logstash-forwarder

This says that for each host in the api group we will run tasks defined in the roles listed above.

Example of a role: the base role is one that (in our case) is applied to all hosts. Here is its directory/file structure:

roles/base

roles/base/defaults

roles/base/defaults/main.yml

roles/base/files

roles/base/files/newrelic

roles/base/files/newrelic/newrelic-sysmond_2.0.2.111_amd64.deb

roles/base/files/pubkeys

roles/base/files/pubkeys/id_rsa.pub.jenkins

roles/base/files/rsyslog

roles/base/files/rsyslog/50-default.conf

roles/base/files/rsyslog/60-papertrail.conf

roles/base/files/rsyslog/papertrail-bundle.pem

roles/base/files/sudoers.d

roles/base/files/sudoers.d/10-admin-users

roles/base/handlers

roles/base/handlers/main.yml

roles/base/meta

roles/base/meta/main.yml

roles/base/README.md

roles/base/tasks

roles/base/tasks/install.yml

roles/base/tasks/main.yml

roles/base/tasks/newrelic.yml

roles/base/tasks/papertrail.yml

roles/base/tasks/users.yml

roles/base/templates

roles/base/templates/hostname.j2

roles/base/templates/nrsysmond.cfg.j2

roles/base/vars

roles/base/vars/main.yml

An Ansible role has the following important sub-directories:

defaults - contains the main.yml file which defines default values for variables used throughout other role files; note that the role’s files are checked in to GitHub, so these values shouldn’t contain secrets such as passwords, API keys etc. For those types of variables, use group_vars or host_vars files which will be discussed below.

files - contains static files that are copied over by ansible tasks to remote hosts

handlers - contains the main.yml file which defines actions such as stopping/starting/restarting services such as nginx, rsyslog etc.

meta - metadata about the role; things like author, description etc.

tasks - the meat and potatoes of ansible, contains one or more files that specify the actions to be taken on the host that is being configured; the main.yml file contains all the other files that get executed

Here are 2 examples of task files, one for configuring rsyslog to send logs to Papertrail and the other for installing the newrelic agent:

$ cat tasks/papertrail.yml

- name: copy papertrail pem certificate file to /etc

 copy: >

   src=rsyslog/{{item}}

   dest=/etc/{{item}}

 with_items:

   - papertrail-bundle.pem

- name: copy rsyslog config files for papertrail integration

 copy: >

   src=rsyslog/{{item}}

   dest=/etc/rsyslog.d/{{item}}

 with_items:

   - 50-default.conf

   - 60-papertrail.conf

 notify:

    - restart rsyslog

$ cat tasks/newrelic.yml

- name: copy newrelic debian package

 copy: >

   src=newrelic/{{newrelic_deb_pkg}}

   dest=/opt/{{newrelic_deb_pkg}}

- name: install newrelic debian package

 apt: deb=/opt/{{newrelic_deb_pkg}}

- name: configure newrelic with proper license key

 template: >

   src=nrsysmond.cfg.j2

   dest=/etc/newrelic/nrsysmond.cfg

   owner=newrelic

   group=newrelic

   mode=0640

 notify:

    - restart newrelic

templates - contains Jinja2 templates with variables that get their values from defaults/main.yml or from group_vars or host_vars files. One special variable that we use (and is not defined in these files, but instead is predefined by Ansible) is inventory_hostname which points to the hostname of the target being configured. For example, here is the template for a hostname file which will be dropped into /etc/hostname on the target:

$ cat roles/base/templates/hostname.j2

{{ inventory_hostname }}

Once you have a playbook and a role, there are a few more files you need to take care of:

• hosts/myhosts - this is an INI-type file which defines groups of hosts. For example the following snippet of this file defines 2 groups called api and magento.

[api]

api01 ansible_ssh_host=api01.mydomain.co

api02 ansible_ssh_host=api02.mydomain.co

[magento]

mgto ansible_ssh_host=mgto.mydomain.co

The api-servers.yml playbook file referenced at the beginning of this document sets the hosts variable to the api group, so all Ansible tasks will get run against the hosts included in that group. In the hosts/myhosts file above, these hosts are api01 and api02.

• group_vars/somegroupname - this is where variables with ‘secret’ values get defined for a specific group called somegroupname. The group_vars directory is not checked into GitHub. somegroupname needs to exactly correspond to the group defined in hosts/myhosts.

Example:

$ cat group_vars/api

ses_smtp_endpoint: email-smtp.us-west-2.amazonaws.com

ses_smtp_port: 587

ses_smtp_username: some_username

ses_smtp_password: some_password

datadog_api_key: some_api_key

. . . other variables (DB credentials etc)

• host_vars/somehostname - this is where variables with ‘secret’ values get defined for a specific host called somehostname. The host_vars directory is not checked into GitHub. somehostname needs to exactly correspond to a host defined in hosts/myhosts.

Example:

$ cat host_vars/api02

insert_sample_data: false

This overrides the insert_sample_data variable and sets it to false only for the host called api02. This could also be used for differentiating between a DB master and slave for example.

Tying it all together

First you need to have ansible installed on your local machine. I used:

$ pip install ansible

To execute a playbook for a given hosts file against all api server, you would run:

$ ansible-playbook -i hosts/myhosts api-servers.yml

The name that ties together the hosts/myhosts file, the api-servers.yml file and the group_vars/groupname file is in this case api.

You need to make sure you have the desired values for that group in these 3 files:

• hosts/myhosts: make sure you have the desired hosts under the [api] group
• api-server.yml: make sure you have the desired roles for hosts in the api group
• group_vars/api: make sure you have the desired values for variables that will be applied to the hosts in the api group

Launching a new api instance in EC2

I blogged about this here.

Updating an existing api instance

Make sure the instance hostname is the only hostname in the [api] group in the hosts/myhosts file. Then run:

$ ansible-playbook -i hosts/myhosts api-servers.yml

Setting up a mailinator-like test mail server with postfix and MailHog

The purpose of this exercise is to set up a mailinator-style mail server under our control. If you haven't used mailinator, it's a free service which provides an easy way to test sending email to random recipients. If you send mail to somebody189@mailinator.com and then go to
https://mailinator.com/inbox.jsp?to=somebody189, you will see a mailbox associated with that user, and any incoming email messages destined for that user. It's a handy way to also sign up for services that send confirmation emails.

I have been playing with MailHog, which is a mail server written in Golang for exactly the same purpose as mailinator. In fact, MailHog can happily intercept ANY recipient at ANY mail domain, provided it is set up properly. In my case, I didn't want to expose MailHog on port 25 externally, because that is a recipe for spam. Instead, I wanted to set up a regular postfix server for mydomain.com, then set up a catch-all user which will receive mail destined for anyuser@maildomain.com, and finally send all that mail to MailHog via procmail. Quite a few moving parts, but I got it to work and I am hastening to jot down my notes before I forget how I did it.

The nice thing about MailHog is that it provides a Web UI where you can eyeball the email messages you sent, including in raw format, and it also provides a JSON API which allows you to list messages and search for specific terms within messages. This last feature is very useful for end-to-end testing of your application's email sending capabilities.

I set up everything on a Google Cloud Engine instance running Ubuntu 14.04.

• instance name: mailhog-mydomain-com
• DNS/IP: mailhog.mydomain.com / 1.2.3.4

Install go 1.5.3 from source

First install the binaries for go 1.4, then compile go 1.5.

# apt-get update
# apt-get install build-essential git mercurial bazaar unzip
# cd /root
# wget https://storage.googleapis.com/golang/go1.4.3.linux-amd64.tar.gz
# tar xvfz go1.4.3.linux-amd64.tar.gz
# mv go go1.4
# git clone https://go.googlesource.com/go
# cd go
# git checkout go1.5.3
# cd src
# ./all.bash

# mkdir /opt/gocode

Edit /root/.bashrc and add:

export GOPATH=/opt/gocode
export PATH=$PATH:/root/go/bin:$GOPATH/bin

then source /root/.bashrc.

# go version
go version go1.5.3 linux/amd64


Set up postfix

Install postfix and mailutils

# apt-get install postfix mailutils

- specified System mail name as mydomain.com

Set up catch-all mail user (local Unix user)

# adduser catchall

Edit /etc/aliases and replace content with the lines below.

# cat /etc/aliases

# See man 5 aliases for format
mailer-daemon: postmaster
postmaster: root
nobody: root
hostmaster: root
usenet: root
news: root
webmaster: root
www: root
ftp: root
abuse: root
root: catchall

Run:

# newaliases

Edit /etc/postfix/main.cf and add lines:

luser_relay = catchall
local_recipient_maps =

Restart postfix:

# service postfix restart

Use Google Cloud Platform Web UI to add firewall rule called allow-smtp for the “default” network associated with the mailhog-mydomain-com instance. The rule allows incoming traffic from everywhere to port tcp:25.

Set up DNS

Add A record for mailhog.mydomain.com pointing to 1.2.3.4.

Add MX record for catchallpayments.com pointing to mailhog.mydomain.com.

Test the incoming mail setup

Send mail to catchall@mydomain.com from gmail.

Run mail utility on GCE instance as user catchall:

catchall@mailhog-mydomain-com:~$ mail

"/var/mail/catchall": 1 message 1 new

>N 1 Some Body     Tue Feb 9 00:23 52/2595 test from gmail

? 1

Return-Path: <some.body@gmail.com>
X-Original-To: catchall@mydomain.com
Delivered-To: catchall@mydomain.com


Send mail to random user which doesn’t exist locally catchall333@mydomain.com and verify that user catchall receives it:

catchall@mailhog-mydomain-com:~$ mail

"/var/mail/catchall": 1 message 1 new

>N 1 Some Body     Tue Feb 9 18:32 52/2702 test 3

? 1

Return-Path: <some.body@gmail.com>
X-Original-To: catchall333@mydomain.com
Delivered-To: catchall333@mydomain.com

Install and configure MailHog

Get MailHog

# go get github.com/mailhog/MailHog

- this will drop several binaries in /opt/gocode/bin, including mhsendmail and MailHog (for reference, the code for mhsendmail is here)

# which MailHog
/opt/gocode/bin/MailHog

# which mhsendmail
/opt/gocode/bin/mhsendmail
Configure HTTP basic authentication

MailHog supports HTTP basic authentication via a file similar to .htpasswd. It uses bcrypt for password (see more details here). The MailHog binary can also generate passwords with bcrypt.

I created a password with MailHog:

# MailHog bcrypt somepassword
somebcryptoutput

Then I created a file called .mailhogrc in /root and specified a user called mailhogapi with the password generated above:

# cat /root/.mailhogrc
mailhogapi:somebcryptoutput

Create upstart init file for MailHog

I specified the port MailHog listens on (I chose the same port as its default which is 1025) and the filed used for HTTP basic auth.

# cat /etc/init/mailhog.conf
# MailHog Test SMTP Server (Upstart unit)
description "MailHog Test SMTP Server"
start on (local-filesystems and net-device-up IFACE!=lo)
stop on runlevel [06]

exec /opt/gocode/bin/MailHog -smtp-bind-addr 0.0.0.0:1025 -auth-file /root/.mailhogrc
respawn
respawn limit 10 10
kill timeout 10
See more command line options for MailHog in this doc.

Start mailhog service

# start mailhog
mailhog start/running, process 25458

# ps -efd|grep Mail
root 7782 1 0 22:04 ? 00:00:00 /opt/gocode/bin/MailHog -smtp-bind-addr 0.0.0.0:1025 -auth-file /root/.mailhogrc

At this point MailHog is listening for SMTP messages on port 1025. It also provides a Web UI on default UI port 8025 and a JSON API also on port 8025.

Install procmail and configure it for user catchall

This is so messages addressed to user catchall (which again is our catch-all user) can get processed by a script via procmail.

# apt-get install procmail

Add this line to /etc/postfix/main.cf:

mailbox_command = /usr/bin/procmail -a "$EXTENSION" DEFAULT=$HOME/Maildir/ MAILDIR=$HOME/Maildir

(this will send all messages to procmail instead of individual user mailboxes)

Then su as user catchall and create .procmailrc file in its home directory:

catchall@mailhog-mydomain-com:~$ cat .procmailrc
:0
| /opt/gocode/bin/mhsendmail --smtp-addr="localhost:1025"

This tells procmail to pipe the incoming mail message to mhsendmail, which will format it properly and pass it to port 1025, where MailHog is listening.

Test end-to-end

Use Google Cloud Platform Web UI to add firewall rule called allow-mailhog-ui for the “default” network associated with the mailhog-mydomain-com instance. The rule allows incoming traffic from everywhere to tcp:8025 (where the MailHog UI server listens). It’s OK to allow traffic to port 8025 from everywhere because it is protected via HTTP basic auth.

The MailHog UI is at http://mailhog.mydomain.com:8025

Any email sent to xyz@mydomain.com should appear in the MailHog Inbox.

By default, MailHog stores incoming messages in memory. Restarting MailHog (via ‘restart mailhog’ at the cmdline) will remove all messages.

MailHog also supports MongoDB as a persistent storage backend for incoming messages (exercise left to the reader.)

Use the MailHog JSON API to verify messages

List all messages:

$ curl --user 'mailhogapi:somepassword' -H "Content-Type: application/json" "http://mailhog.mydomain.com:8025/api/v2/messages"

Search messages for specific terms (for example for the recipient’s email):

$ curl -i --user 'mailhogapi:somepassword' -H "Content-Type: application/json" "http://mailhog.mydomain.com:8025/api/v2/search?kind=containing&query=test1%40mydomain.com"

See the MailHog API v2 docs here.

That's it, hope it makes your email sending testing more fun!

Setting up Jenkins to run headless Selenium tests in Docker containers

This is the third post in a series on running headless Selenium WebDriver tests. Here are the first two posts:

1. Running Selenium WebDriver tests using Firefox headless mode on Ubuntu
2. Running headless Selenium WebDriver tests in Docker containers

In this post I will show how to add the final piece to this workflow, namely how to fully automate the execution of Selenium-based WebDriver tests running Firefox in headless mode in Docker containers. I will use Jenkins for this example, but the same applies to other continuous integration systems.

1) Install docker-engine on the server running Jenkins (I covered this in my post #2 above)

2) Add the jenkins user to the docker group so that Jenkins can run the docker command-line tool in order to communicate with the docker daemon. Remember to restart Jenkins after doing this.

3) Go through the rest of the workflow in my post above ("Running headless Selenium WebDriver tests in Docker containers") and make sure you can run all the commands in that post from the command line of the server running Jenkins.

4) Create a directory structure for your Selenium WebDriver tests (mine are written in Python). 

I have a directory called selenium-docker which contains a directory called tests, under which I put all my Python WebDriver tests named sel_wd_*.py. I also  have a simple shell script I named run_selenium_tests.sh which does the following:

#!/bin/bash

TARGET=$1 # e.g. someotherdomain.example.com (if not specified, the default is somedomain.example.com)

for f in `ls tests/sel_wd_*.py`; do

    echo Running $f against $TARGET

    python $f $TARGET

done

My selenium-docker directory also contains the xvfb.init file I need for starting up Xvfb in the container, and finally it contains this Dockerfile:

FROM ubuntu:trusty

RUN echo "deb http://ppa.launchpad.net/mozillateam/firefox-next/ubuntu trusty main" > /etc/apt/sources.list.d//mozillateam-firefox-next-trusty.list

RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CE49EC21

RUN apt-get update

RUN apt-get install -y firefox xvfb python-pip

RUN pip install selenium

RUN mkdir -p /root/selenium/tests

ADD tests /root/selenium/tests

ADD run_all_selenium_tests.sh /root/selenium

ADD xvfb.init /etc/init.d/xvfb

RUN chmod +x /etc/init.d/xvfb

RUN update-rc.d xvfb defaults

ENV TARGET=somedomain.example.com

CMD (service xvfb start; export DISPLAY=:10; cd /root/selenium; ./run_all_selenium_tests.sh $TARGET)

I explained what this Dockerfile achieves in the 2nd post referenced above. The ADD instructions will copy all the files in the tests directory to the directory called /root/selenium/tests, and will copy run_all_selenium_tests.sh to /root/selenium. The ENV variable TARGET represents the URL against which we want to run our Selenium tests. It is set by default to somedomain.example.com, and is used as the first argument when running run_all_selenium_tests.sh in the CMD instruction.

At this point, I checked in the selenium-docker directory and all files and directories under it into a Github repository I will call 'devops'.

5) Create a new Jenkins project (I usually create a New Item and copy it from an existing project).

I specified that the build is parameterized and I indicated a choice parameter called TARGET_HOST with a few host/domain names that I want to test. I also specified Git as the Source Code Management type, and I indicated the URL of the devops repository on Github. Most of the action of course happens in the Jenkins build step, which in my case is of type "Execute shell". Here it is:

#!/bin/bash

set +e

IMAGE_NAME=selenium-wd:v1

cd $WORKSPACE/selenium-docker

# build the image out of the Dockerfile in the current directory

/usr/bin/docker build -t $IMAGE_NAME .

# run a container based on the image

CONTAINER_ID=`/usr/bin/docker run -d -e "TARGET=$TARGET_HOST" $IMAGE_NAME`

echo CONTAINER_ID=$CONTAINER_ID

  

# while the container is still running, sleep and check logs; repeat every 40 sec

while [ $? -eq 0 ];

do

  sleep 40

  /usr/bin/docker logs $CONTAINER_ID

  /usr/bin/docker ps | grep $IMAGE_NAME

done

# docker logs sends errors to stderr so we need to save its output to a file first

/usr/bin/docker logs $CONTAINER_ID > d.out 2>&1

# remove the container so they don't keep accumulating

docker rm $CONTAINER_ID

# mark jenkins build as failed if log output contains FAILED

grep "FAILED" d.out

if [[ $? -eq 0 ]]; then  

  rm d.out 

  exit 1

else

  rm d.out

  exit 0

fi

Some notes:

• it is recommended that you specify #!/bin/bash as the 1st line of your script, to make sure that bash is the shell that is being used
• use set +e if you want the Jenkins shell script to continue after hitting a non-zero return code (the default behavior is for the script to stop on the first line it encounters an error and for the build to be marked as failed; subsequent lines won't get executed, resulting in much pulling of hair)
• the Jenkins script will build a new image every time it runs, so that we make sure we have updated Selenium scripts in place
• when running the container via docker run, we specify -e "TARGET=$TARGET_HOST" as an extra command line argument. This will override the ENV variable named TARGET in the Dockerfile with the value received from the Jenkins multiple choice dropdown for TARGET_HOST
• the main part of the shell script stays in a while loop that checks for the return code of "/usr/bin/docker ps | grep $IMAGE_NAME". This is so we wait for all the Selenium tests to finish, at which point docker ps will not show the container running anymore (you can still see the container by running docker ps -a)
• once the tests finish, we save the stdout and stderr of the docker logs command for our container to a file (this is so we capture both stdout and stderr; at first I tried something like docker logs $CONTAINER_ID | grep FAILED but this was never successful, because it was grep-ing against stdout, and errors are sent to stderr)
• we grep the file (d.out) for the string FAILED and if we find it, we exit with code 1, i.e. unsuccessful as far as Jenkins is concerned. If we don't find it, we exit successfully with code 0.

Running headless Selenium WebDriver tests in Docker containers

In my previous post, I showed how to install firefox in headless mode on an Ubuntu box and how to use Xvfb to allow Selenium WebDriver scripts to run against firefox in headless mode.

Here I want to show how run each Selenium test suite in a Docker container, so that the suite gets access to its own firefox browser. This makes it easy to parallelize the test runs, and thus allows you to load test your Web infrastructure with real-life test cases.

Install docker-engine on Ubuntu 14.04

We import the dockerproject.org signing key and apt repo into our apt repositories, then we install the linux-image-extra and docker-engine packages.

# apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
# echo “deb https://apt.dockerproject.org/repo ubuntu-trusty main” > /etc/apt/sources.list.d/docker.list
# apt-get update
# apt-get install linux-image-extra-$(uname -r)
# apt-get install docker-engine

Start the docker service and verify that it is operational

Installing docker-engine actually starts up docker as well, but to start the service you do:

# service docker start

To verify that the docker service is operational, run a container based on the public “hello-world” Docker image:

# docker run hello-world
Unable to find image ‘hello-world:latest’ locally
latest: Pulling from library/hello-world
b901d36b6f2f: Pull complete
0a6ba66e537a: Pull complete
Digest: sha256:8be990ef2aeb16dbcb9271ddfe2610fa6658d13f6dfb8bc72074cc1ca36966a7
Status: Downloaded newer image for hello-world:latest
Hello from Docker.

This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the “hello-world” image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker Hub account:
https://hub.docker.com
For more examples and ideas, visit:
https://docs.docker.com/userguide/

Pull the ubuntu:trusty public Docker image

# docker pull ubuntu:trusty
trusty: Pulling from library/ubuntu
fcee8bcfe180: Pull complete
4cdc0cbc1936: Pull complete
d9e545b90db8: Pull complete
c4bea91afef3: Pull complete
Digest: sha256:3a7f4c0573b303f7a36b20ead6190bd81cabf323fc62c77d52fb8fa3e9f7edfe
Status: Downloaded newer image for ubuntu:trusty

# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
ubuntu trusty c4bea91afef3 3 days ago 187.9 MB
hello-world latest 0a6ba66e537a 12 weeks ago 960 B

Build custom Docker image for headless Selenium WebDriver testing

I created a directory called selwd on my host Ubuntu 14.04 box, and in that directory I created this Dockerfile:

FROM ubuntu:trusty
RUN echo “deb http://ppa.launchpad.net/mozillateam/firefox-next/ubuntu trusty main” > /etc/apt/sources.list.d//mozillateam-firefox-next-trusty.list 

RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CE49EC21 

RUN apt-get update 

RUN apt-get install -y firefox xvfb python-pip 

RUN pip install selenium 

RUN mkdir -p /root/selenium_wd_tests 

ADD sel_wd_new_user.py /root/selenium_wd_tests 

ADD xvfb.init /etc/init.d/xvfb 

RUN chmod +x /etc/init.d/xvfb 

RUN update-rc.d xvfb defaults
CMD (service xvfb start; export DISPLAY=:10; python /root/selenium_wd_tests/sel_wd_new_user.py)

This Dockerfile tells docker, via the FROM instruction, to create an image based on the ubuntu:trusty image that we pulled before (if we hadn’t pulled it, it would be pulled the first time our image was built).

The various RUN instructions specify commands to be run at build time. The above instructions add the Firefox Beta repository and key to the apt repositories inside the image, then install firefox, xvfb and python-pip. Then they install the selenium Python package via pip and create a directory structure for the Selenium tests.

The ADD instructions copy local files to the image. In my case, I copy one Selenium WebDriver Python script, and an init.d-type file for starting Xvfb as a service (by default it starts in the foreground, which is not something I want inside a Docker container).

The last two RUN instructions make the /etc/init.d/xvfb script executable and run update-rc.d to install it as a service. The xvfb script is the usual init.d wrapper around a command, in my case this command:

PROG=”/usr/bin/Xvfb” 

PROG_OPTIONS=”:10 -ac”

Here is a gist for the xvfb.init script for reference.

Finally, the CMD instruction specifies what gets executed when a container based on this image starts up (assuming no other commands are given in the ‘docker run’ command-line for this container). The CMD instruction in the Dockerfile above starts up the xvfb service (which connects to DISPLAY 10 as specified in the xvfb init script), sets the DISPLAY environment variable to 10, then runs the Selenium WebDriver script sel_wd_new_user.py, which will launch firefox in headless mode and execute its commands against it.

Here’s the official documentation for Dockerfile instructions.

To build a Docker image based on this Dockerfile, run:

# docker build -t selwd:v1 .

selwd is the name of the image and v1 is a tag associated with this name. The dot . tells docker to look for a Dockerfile in the current directory.

The build process will take a while intially because it will install all the dependencies necessary for the packages we are installing with apt. Every time you make a modification to the Dockerfile, you need to run ‘docker build’ again, but subsequent runs will be much faster.

Run Docker containers based on the custom image

At this point, we are ready to run Docker containers based on the selwd image we created above.

Here’s how to run a single container:

# docker run --rm selwd:v1

In this format, the command specified in the CMD instruction inside the Dockerfile will get executed, then the container will stop. This is exactly what we need: we run our Selenium WebDriver tests against headless firefox, inside their own container isolated from any other container.

The output of the ‘docker run’ command above is:


Starting : X Virtual Frame Buffer 

. — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Ran 1 test in 40.441s

OK
(or a traceback if the Selenium test encountered an error)

Note that we also specified the rm flag to ‘docker run’ so that the container gets removed once it stops — otherwise these short-lived containers will be kept around and will pile up, as you can see for yourself if you run:

# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 

6c9673e59585 selwd:v1 “/bin/bash” 5 minutes ago Exited (130) 5 seconds ago modest_mccarthy 980651e1b167 selwd:v1 “/bin/sh -c ‘(service” 9 minutes ago Exited (0) 8 minutes ago stupefied_turing 

4a9b2f4c8c28 selwd:v1 “/bin/sh -c ‘(service” 13 minutes ago Exited (0) 12 minutes ago nostalgic_ride 

9f1fa953c83b selwd:v1 “/bin/sh -c ‘(service” 13 minutes ago Exited (0) 12 minutes ago admiring_ride 

c15b180832f6 selwd:v1 “/bin/sh -c ‘(service” 13 minutes ago Exited (0) 12 minutes ago jovial_booth .....etc

If you do have large numbers of containers that you want to remove in one go, use this command:

# docker rm `docker ps -aq`

For troubleshooting purposes, we can run a container in interactive mode (with the -i and -t flags) and specify a shell command to be executed on startup, which will override the CMD instruction in the Dockerfile:

# docker run -it selwd:v1 /bin/bash 

root@6c9673e59585:/#
At the bash prompt, you can run the shell commands specified by the Dockerfile CMD instruction in order to see interactively what is going on. The official ‘docker run’ documentation has lots of details.

One other thing I found useful for troubleshooting Selenium WebDriver scripts running against headless firefox was to have the scripts take screenshots during their execution with the save_screenshot command:

driver.save_screenshot(“before_place_order.png”)
# Click Place Order driver.find_element_by_xpath("//*[@id='order_submit_button']").click()
driver.save_screenshot(“after_place_order.png”)

I then inspected the PNG files to see what was going on.

Running multiple Docker containers for load testing

Because our Selenium WebDriver tests run isolated in their own Docker container, it enables us to run N containers in parallel to do a poor man’s load testing of our site.

We’ll use the -d option to ‘docker run’ to run each container in ‘detached’ mode. Here is a bash script that launches COUNT Docker containers, where COUNT is the 1st command line argument, or 2 by default:


#!/bin/bash
COUNT=$1 

if [ -z “$COUNT” ]; then 

 COUNT=2 

fi
for i in `seq 1 $COUNT`; do 

 docker run -d selwd:v1 

done

The output of the script consists in a list of container IDs, one for each container that was launched.

Note that if you launch a container in detached mode with -d, you can’t specify the rm flag to have the container removed automatically when it stops. You will need to periodically clean up your containers with the command I referenced above (docker rm `docker ps -aq`).

To inspect the output of the Selenium scripts in the containers that were launched, first get the container IDs:

# docker ps -a 

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 

6fb931689c03 selwd:v1 “/bin/sh -c ‘(service” About an hour ago Exited (0) About an hour ago grave_northcutt 

1b82ef59ad46 selwd:v1 “/bin/sh -c ‘(service” About an hour ago Exited (0) About an hour ago admiring_fermat

Then run ‘docker logs <container_id>’ to see the output for a specific container:

# docker logs 6fb931689c03 

. — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 

Ran 1 test in 68.436s

OK 

Starting : X Virtual Frame Buffer
Have fun load testing your site!

Running Selenium WebDriver tests using Firefox headless mode on Ubuntu

Selenium IDE is a very good tool for recording and troubleshooting Selenium tests, but you are limited to clicking around in a GUI. For a better testing workflow, including load testing, you need to use Selenium WebDriver, which can programatically drive a browser and run Selenium test cases.

In its default mode, WebDriver will launch a browser and run the test scripts in the browser, then exit. If you like to work exclusively from the command line, then you need to look into running the browser in headless mode. Fortunately, this is easy to do with Firefox on Ubuntu. Here’s what you need to do:

Install the official Firefox Beta PPA:

$ sudo apt-add-repository ppa:mozillateam/firefox-next

(this will add the file /etc/apt/sources.list.d/mozillateam-firefox-next-trusty.list and also fetch the PPA’s key, which enables your Ubuntu system to verify that the packages in the PPA have not been interfered with since they were built)

Run apt-get update:

$ sudo apt-get update

Install firefox and xvfb (the X windows virtual framebuffer) packages:

$ sudo apt-get install firefox xvfb

Run Xvfb in the background and specify a display number (10 in my example):

$ Xvfb :10 -ac &

Set the DISPLAY variable to the number you chose:

$ export DISPLAY=:10

Test that you can run firefox in the foreground with no errors:

$ firefox
(kill it with Ctrl-C)

Now run your regular Selenium WebDriver scripts (no modifications required if they already use Firefox as their browser).

Here is an example of a script I have written in Python, which clicks on a category link in an e-commerce store, adds an item to the cart, that starts filling out the user’s information in the cart:

# -*- coding: utf-8 -*-
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import NoAlertPresentException
import unittest, time, re, random

class SelWebdriverNewUser(unittest.TestCase):
  def setUp(self):
    self.driver = webdriver.Firefox()
    self.driver.implicitly_wait(20)
    self.base_url = “http://myhost.mycompany.com/"
    self.verificationErrors = []
    self.accept_next_alert = True
  

  def test_sel_webdriver_new_user(self):
    driver = self.driver
    HOST = “myhost.mycompany.com”
    RANDINT = random.random()*10000
    driver.get(“https://” + HOST)

    # Click on category link
    driver.find_element_by_xpath(“//*[@id=’nav’]/ol/li[3]/a”).click()
    # Click on sub-category link
    driver.find_element_by_xpath(“//*[@id=’top’]/body/div/div[2]/div[2]/div/div[2]/ul/li[4]/a/span”).click()
    # Click on product image
    driver.find_element_by_xpath(“//*[@id=’product-collection-image-374']”).click()
    # Click Checkout button
    driver.find_element_by_xpath(“//*[@id=’checkout-button’]/span/span”).click()
    driver.find_element_by_id(“billing:firstname”).clear()
driver.find_element_by_id(“billing:firstname”).send_keys(“selenium”, RANDINT, “_fname”)
    driver.find_element_by_id(“billing:lastname”).clear()
driver.find_element_by_id(“billing:lastname”).send_keys(“selenium”, RANDINT, “_lname”)
    # Click Place Order
    driver.find_element_by_xpath(“//*[@id=’order_submit_button’]”).click()

  def is_element_present(self, how, what):
    try: self.driver.find_element(by=how, value=what)
    except NoSuchElementException as e: return False
    return True

  def is_alert_present(self):
    try: self.driver.switch_to_alert()
    except NoAlertPresentException as e: return False
    return True

  def close_alert_and_get_its_text(self):
    try:
      alert = self.driver.switch_to_alert()
      alert_text = alert.text
      if self.accept_next_alert:
        alert.accept()
      else:
        alert.dismiss()
      return alert_text
    finally: self.accept_next_alert = True

def tearDown(self):
    self.driver.quit()
    self.assertEqual([], self.verificationErrors)

if __name__ == “__main__”:
unittest.main()

To run this script, you first need to install the selenium Python package:

$ sudo pip install selenium

Then run the script (called selenium_webdriver_new_user.py in my case):

$ python selenium_webdriver_new_user.py

After hopefully not so long of a wait, you should see a successful test run:

$ python selenium_webdriver_new_user.py
.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
Ran 1 test in 29.317s

A few notes regarding Selenium WebDriver scripts.

I was stumped for a while when I was trying to use the “find_element_by_id” form of finding an HTML element on a Web page. It was working fine in Selenium IDE, but Selenium WebDriver couldn’t find that element. I had to resort to finding all elements via their XPath id using “find_element_by_xpath”. Fortunately, Chrome for example makes it easy to right click an element on a page, choose Inspect, then righ click the HTML code for the element and choose Copy->Copy XPath to get their id which can then be pasted in the Selenium WebDriver script.

I also had to use time.sleep(N) (where N is in seconds at least for Python) to wait for certain elements of the page to load asynchronously. I know it’s not best practices, but it works.

Distributing a beta version of an iOS app

I am not an iOS expert by any means, but recently I’ve had to maintain an iOS app and distribute it to beta testers. I had to jump through a few hoops, so I am documenting here the steps I had to take.

First of all, I am using Xcode 6.4 with the Fabric 2.1.1 plugin. I assume you are already signed up for the Fabric/Crashlytics service and that you also have an Apple developer account.

1. Ask each beta tester to send you the UUID of the devices they want to run your app on.
2. Go to developer.apple.com -> “Certificates, Identifiers and Profiles” -> “Devices” and add each device with its associated UUID. Let’s say you add a device called “Tom’s iPhone 6s” with its UUID.
3. Go to Xcode -> Preferences -> Accounts. If you already have an account set up, remove it by selecting it and clicking the minus icon on the lower left side. Add an account: click the plus icon, choose “Add Apple ID” and enter your Apple ID and password. This will import your Apple developer provisioning profile into Xcode, with the newly added device UUIDs (note: there may be a better way of adding/modifying the provisioning profile within Xcode but this worked for me)
4. Make sure the Fabric plugin is running on your Mac.
5. Go to Xcode and choose the iOS application you want to distribute. Choose iOS Device as the target for the build.
6. Go to Xcode -> Product -> Archive. This will build the app, then the Fabric plugin will pop up a message box asking you if you want to distribute the archive build. Click Distribute.
7. The Fabric plugin will pop up a dialog box asking you for the email of the tester(s) you want to invite. Enter one or more email addresses. Enter release notes. At this point the Fabric plugin will upload your app build to the Fabric site and notify the tester(s) that they are invited to test the app.

Installing and configuring Raspbian Jessie on a Raspberry Pi B+

I blogged before about configuring a Raspberry Pi B+ with Raspbian Wheezy. Here are some notes I took today while going through the whole process again, but this time with the latest Raspbian version, Jessie, from 2015-11-21. Many steps are the same, but I will add instructions for configuring a wireless connection.

1) Bought micro SD card. Note: DO NOT get a regular SD card for the B+ because it will not fit in the SD card slot. You need a micro SD card.

2) Inserted the SD card via an SD USB adaptor in my MacBook Pro.

3) Went to the command line and ran df to see which volume the SD card was mounted as. In my case, it was /dev/disk2s1.

4) Unmounted the SD card from my Mac. I initially tried 'sudo umount /dev/disk2s1' but the system told me to use 'diskutil unmount', so the command that worked for me was:

$ diskutil unmount /dev/disk2s1

5) Downloaded 2015-11-21-raspbian-jessie.zip from  https://downloads.raspberrypi.org/raspbian/images. Unzipped it to obtain the image file 2015-11-21-raspbian-jessie.img

6) Used dd to copy the image from my Mac to the SD card. Thanks to an anonymous commenter on my previous blog post, I specified the target of the dd command as the raw device /dev/rdisk2. Note: DO NOT specify the target as /dev/disk2s1 or /dev/rdisk2s1. Either /dev/disk2 or /dev/rdisk2 will work, but copying to the raw device is faster. Here is the dd command I used:

$ sudo dd if=2015-11-21-raspbian-jessie.img of=/dev/rdisk2 bs=1m

3752+0 records in

3752+0 records out

3934257152 bytes transferred in 233.218961 secs (16869371 bytes/sec)

7) I unmounted the SD card from my Mac one more time:

$ diskutil unmount /dev/disk2s1

8) I inserted the SD card into my Raspberry Pi. I also inserted a USB WiFi adapter (I used the Wi-Pi 802.11n adapter). My Pi was also connected to a USB keyboard, to a USB mouse and to a monitor via HDMI. 

9) I powered up the Pi. It went through the Raspbian Jessie boot process uneventfully, and it brought up the X Windows GUI interface (which is the default in Jessie, as opposed to the console in Wheezy). At this point, I configured the Pi to boot back into console mode by going to Menu -> Preferences -> Raspberry Pi Configuration and changing the Boot option from "To Desktop" to "To CLI". While in the configuration dialog, I also changed the default password for user pi, and unchecked the autologin option.

10) I rebooted the Pi and this time it booted up in console mode and stopped at the login prompt. I logged in as user pi.

11) I spent the next 30 minutes googling around to find out how to make the wireless interface work. It's always been a chore for me to get wlan to work on a Pi, hence the following instructions (based on this really good blog post).

12) Edit /etc/network/interfaces:

(i)  change "auto l0" to "auto wlan0"

(ii) change "iface wlan0 inet manual" to "iface wlan0 inet dhcp"

13) Edit /etc/wpa_supplicant/wpa_supplicant.conf and add this at the end:

network={

  ssid="your_ssid"

  psk="your_ssid_password"

}

14) Rebooted the Pi and ran ifconfig. At this point I could see wlan0 configured properly with an IP address.

Hope these instructions work for you. Merry Christmas!

Protecting your site for free with Let's Encrypt SSL certificates and acmetool

The buzz level around Let's Encrypt has been more elevated lately, due to their opening up their service as a public beta. If you don't know what Let's Encrypt is, it's a Certificate Authority which provides SSL certificates free of charge. The twist is that they implement a protocol called ACME ("Automated Certificate Management Environment") for automating the management of domain-validation certificates, based on a simple JSON-over-HTTPS interface. Read more technical details about Let's Encrypt here.

The certificates from Let's Encrypt have a short life of 90 days, and this is done on purpose so that they encourage web site administrators to renew them programatically and automatically. In what follows, I'll walk you through how to obtain and install Let's Encrypt certificates for nginx on Ubuntu. I will use a tool called acmetool, and not the official Let's Encrypt client tools, because acmetool generates standalone SSL keys and certs and doesn't try to reconfigure a given web server automatically in order to use them (like the letsencrypt client tools do). I like this separation of concerns. Plus acmetool is written in Go, so you just deploy it as a binary and you're off to the races.

1) Configure nginx to serve your domain name

I will assume you want to protect www.mydomain.com with SSL certificates from Let's Encrypt. The very first step, which I assume you have already taken, is to configure nginx to serve www.mydomain.com on port 80. I also assume the document root is /var/www/mydomain.

2) Install acmetool

$ sudo apt-get install libcap-dev

$ git clone https://github.com/hlandau/acme 

$ cd acme

$ make && sudo make install

3) Run "acmetool quickstart" to configure ACME

The ACME protocol requires a verification of your ownership of mydomain.com. There are multiple ways to prove that ownership and the one I chose below was to let the ACME agent (in this case acmetool) to drop a file under the nginx document root. As part of the verification, the ACME agent will also generate a keypair under the covers, and sign a nonce sent from the ACME server with the private key, in order to prove possession of the keypair.

# acmetool quickstart

------------------------- Select ACME Server -----------------------

Please choose an ACME server from which to request certificates. Your principal choices are the Let's Encrypt Live Server, and the Let's Encrypt Staging Server.

You can use the Let's Encrypt Live Server to get real certificates.

The Let's Encrypt Staging Server does not issue publically trusted certificates. It is useful for development purposes, as it has far higher rate limits than the live server.

  1) Let's Encrypt Live Server - I want live certificates

  2) Let's Encrypt Staging Server - I want test certificates

  3) Enter an ACME server URL

I chose option 1 (Let's Encrypt Live Server).

----------------- Select Challenge Conveyance Method ---------------

acmetool needs to be able to convey challenge responses to the ACME server in order to prove its control of the domains for which you issue certificates. These authorizations expire rapidly, as do ACME-issued certificates (Let's Encrypt certificates have a 90 day lifetime), thus it is essential that the completion of these challenges is a) automated and b) functioning properly. There are several options by which challenges can be facilitated:

WEBROOT: The webroot option installs challenge files to a given directory. You must configure your web server so that the files will be available at <http://[HOST]/.well-known/acme-challenge/>. For example, if your webroot is "/var/www", specifying a webroot of "/var/www/.well-known/acme-challenge" is likely to work well. The directory will be created automatically if it does not already exist.

PROXY: The proxy option requires you to configure your web server to proxy requests for paths under /.well-known/acme-challenge/ to a special web server running on port 402, which will serve challenges appropriately.

REDIRECTOR: The redirector option runs a special web server daemon on port 80. This means that you cannot run your own web server on port 80. The redirector redirects all HTTP requests to the equivalent HTTPS URL, so this is useful if you want to enforce use of HTTPS. You will need to configure your web server to not listen on port 80, and you will need to configure your system to run "acmetool redirector" as a daemon. If your system uses systemd, an appropriate unit file can automatically be installed.

LISTEN: Directly listen on port 80 or 443, whichever is available, in order to complete challenges. This is useful only for development purposes.

  1) WEBROOT - Place challenges in a directory

  2) PROXY - I'll proxy challenge requests to an HTTP server

  3) REDIRECTOR - I want to use acmetool's redirect-to-HTTPS functionality

  4) LISTEN - Listen on port 80 or 443 (only useful for development purposes)

I chose option 1 (WEBROOT).

------------------------- Enter Webroot Path -----------------------

Please enter the path at which challenges should be stored.

If your webroot path is /var/www, you would enter /var/www/.well-known/acme-challenge here.

The directory will be created if it does not exist.

Webroot paths vary by OS; please consult your web server configuration.

I indicated /var/www/mydomain/.well-known/acme-challenge as the directory where the challenge will be stored.

------------------------- Quickstart Complete ----------------------

The quickstart process is complete.

Ensure your chosen challenge conveyance method is configured properly before attempting to request certificates. You can find more information about how to configure your system for each method in the acmetool documentation: https://github.com/hlandau/acme.t/blob/master/doc/WSCONFIG.md

To request a certificate, run:

$ sudo acmetool want example.com www.example.com

If the certificate is successfully obtained, it will be placed in /var/lib/acme/live/example.com/{cert,chain,fullchain,privkey}.

Press Return to continue.

4) Obtain the Let's Encrypt SSL key and certificates for www.mydomain.com

As the quickstart output indicates above, we need to run:

# acmetool want www.mydomain.com

This should run with no errors and drop the following files in /var/lib/acme/live/www.mydomain.com: cert, chain, fullchain, privkey and url.

5) Configure nginx to use the Let's Encrypt SSL key and certificate chain

I found a good resource for specifying secure (as of Dec. 2015) SSL configurations for a variety of software, including nginx: cipherli.st.

Here is the nginx configuration pertaining to SSL that I used, pointing to the SSL key and certificate chain retrieved by acmetool from Let's Encrypt:

        listen 443 ssl default_server;

        listen [::]:443 ssl default_server;

        ssl_certificate     /var/lib/acme/live/www.mydomain.com/fullchain;

        ssl_certificate_key /var/lib/acme/live/www.mydomain.com/privkey;

        ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

        ssl_prefer_server_ciphers on;

        ssl_session_cache shared:SSL:10m;

        add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";

        add_header X-Frame-Options DENY;

        add_header X-Content-Type-Options nosniff;

        ssl_session_tickets off; # Requires nginx >= 1.5.9

        ssl_stapling on; # Requires nginx >= 1.3.7

        ssl_stapling_verify on; # Requires nginx => 1.3.7

At this point, if you hit www.mydomain.com over SSL, you should be able to inspect the SSL certificate and see that it's considered valid by your browser (I tested it in Chrome, Firefox and Safari). The Issuer Name has Organization Name "Let's Encrypt" and Common Name "Let's Encrypt Authority X1".

6) Configure cron job for SSL certificate renewal

Let's Encrypt certificates expire in 90 days after the issue date, so you need to renew them more often than you are used to with regular SSL certificates. I added this line to my crontab on the server that handles www.mydomain.com:

# m h  dom mon dow   command

0 0 1 * * /usr/local/bin/acmetool reconcile --batch; service nginx restart

This runs the acmetool "reconcile" command in batch mode (with no input required from the user) at midnight on the 1st day of every month, then restarts nginx just in case the certificate has changed. If the Let's Encrypt SSL certificate is 30 days away from expiring, acmetool reconcile will renew it.

I think Let's Encrypt is a great service, and you should start using it if you're not already!