I was using the latest version of HAProxy 1.3 and was load balancing backend MySQL servers while also checking their ports, so if one server went down it would be taken out of the load balancing pool. However, since the port checks in HAProxy happen at the TCP level, the MySQL instance which was being hit by the port checks wasn't happy, because it wasn't a proper MySQL connection. As a result, after a number of some checks, MySQL refused to allow clients to connect, with a message like this:
OperationalError: (1129, "Host 'myhost' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'")
Solution: upgrade HAProxy to the newly released version 1.4 (at the date of this writing, the exact version is 1.4.0). Documentation is here.
For MySQL specific checks, you can specify 'option mysql-check' in a backend or 'listen' section of the configuration file. For example, I have something similar to this in my HAProxy configuration file:
listen mysql 0.0.0.0:3306
server mysql1 10.0.1.1:3306 check port 3306
server mysql2 10.0.1.2:3306 check port 3306 backup
Saturday, February 27, 2010
Use HAProxy 1.4 if you need MySQL health checks
Subscribe to: Post Comments (Atom)
Modifying EC2 security groups via AWS Lambda functions
One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...
This post is a continuation of my previous one on " Running Gatling tests in Docker containers via Jenkins ". As I continued to se...
For the last month or so I've been experimenting with Rancher as the orchestration layer for Docker-based deployments. I've been pr...
Here's a good interview question for a tester: how do you define performance/load/stress testing? Many times people use these terms inte...
Using HAProxy is not a measure of MySQL health. Querying MySQL is a measure of MySQL health. Extract useful metrics and correlate them with related activity and latency to determine the true "health" of your storage engine.
Hey Jason -- thanks for the feedback. I am not relying on HAProxy to do any kind of deep MySQL monitoring, I have many nagios and munin checks for that purpose. I use HAProxy as a load balancer in front of multiple MySQL instances, and I merely want HAProxy to flag a backend instance as 'down' when it doesn't respond anymore on a particular port number. That's what I mean by 'health check' in this particular scenario.
You know that... and I know that... but there are too many people that still equate availability with a healthy system. Doing my part to debunk the myth. ;-)
HA-Proxy version 1.4.3 2010/03/30.
listen mysql1 127.0.0.1:3340
server mysql1 mysql_host:3306 check port 3306
[WARNING] 091/134441 (5601) : Server mysql1/mysql1 is DOWN, reason: Layer7 wrong status, code: 0, info: "Host 'client_host' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'", check duration: 0ms.
[ALERT] 091/134441 (5601) : proxy 'mysql1' has no server available!
azalio -- it turns out MySQL health checks are still not working right in 1.4. I ended up applying this solution:
Post a Comment