Two lessons on haproxy checks and swap space

Let's assume you want to host a Wordpress site which is not going to get a lot of traffic. You want to use EC2 for this. You still want as much fault tolerance as you can get at a decent price, so you create an Elastic Load Balancer endpoint which points to 2 (smallish) EC2 instances running haproxy, with each haproxy instance pointing in turn to 2 (not-so-smallish) EC2 instances running Wordpress (Apache + MySQL). 

You choose to run haproxy behind the ELB because it gives you more flexibitity in terms of load balancing algorithms, health checks, redirections etc. Within haproxy, one of the Wordpress servers is marked as a backup for the other, so it only gets hit by haproxy when the primary one goes down. On this secondary Wordpress instance you set up MySQL to be a slave of the primary instance's MySQL. 

Here are two things (at least) that you need to make sure you have in this scenario:

1) Make sure you specify the httpchk option in haproxy.cfg, otherwise the primary server will not be marked as down even if Apache goes down. So you should have something like:

backend servers-http

  server s1 10.0.1.1:80 weight 1 maxconn 5000 check port 80

  server s2 10.0.1.2:80 backup weight 1 maxconn 5000 check port 80

  option httpchk GET /

2) Make sure you have swap space in case the memory on the Wordpress instances gets exhausted, in which case random processes will be killed by the oom process (and one of those processes can be mysqld). By default, there is no swap space when you spin up an Ubuntu EC2 instance. Here's how to set up a 2 GB swapfile:

dd if=/dev/zero of=/swapfile1 bs=1024 count=2097152

mkswap /swapfile1

chmod 0600 /swapfile1

swapon /swapfile1

echo "/swapfile1 swap swap defaults 0 0" >> /etc/fstab

I hope these two things will help you if you're not already doing them ;-)

Managing OpenStack security groups from the command line

I had an issue today where I couldn't connect to a particular OpenStack instance on port 443. I decided to inspect the security group it belongs (let's call it myapp) to from the command line:

# nova secgroup-list-rules myapp
+-------------+-----------+---------+------------+--------------+
| IP Protocol | From Port | To Port | IP Range   | Source Group |
+-------------+-----------+---------+------------+--------------+
| tcp         | 80        | 80      | 0.0.0.0/0  |              |
| tcp         | 443       | 443     | 0.0.0.0/24 |              |
+-------------+-----------+---------+------------+--------------+

Note that the IP range for port 443 is wrong. It should be all IPs and not a /24 network.

I proceeded to delete the wrong rule:

# nova secgroup-delete-rule myapp tcp 443 443 0.0.0.0/24                                                               
+-------------+-----------+---------+------------+--------------+
| IP Protocol | From Port | To Port | IP Range   | Source Group |
+-------------+-----------+---------+------------+--------------+
| tcp         | 443       | 443     | 0.0.0.0/24 |              |
+-------------+-----------+---------+------------+--------------+

Then I added back the correct rule:

 # nova secgroup-add-rule myapp tcp 443 443 0.0.0.0/0                                                                   

+-------------+-----------+---------+-----------+--------------+

| IP Protocol | From Port | To Port | IP Range  | Source Group |

+-------------+-----------+---------+-----------+--------------+

| tcp         | 443       | 443     | 0.0.0.0/0 |              |

+-------------+-----------+---------+------------+--------------+

Finally, I verified that the rules are now correct:

# nova secgroup-list-rules myapp                                                                                       

+-------------+-----------+---------+-----------+--------------+

| IP Protocol | From Port | To Port | IP Range  | Source Group |

+-------------+-----------+---------+-----------+--------------+

| tcp         | 443       | 443     | 0.0.0.0/0 |              |

| tcp         | 80        | 80      | 0.0.0.0/0 |              |

+-------------+-----------+---------+-----------+--------------+

Of course, the real test was to see if I could now hit port 443 on my instance, and indeed I was able to.