We had a situation recently where our web application started to behave strangely. First nginx (which sits in front of the application) started to error out with messages of this type:
upstream sent too big header while reading response header from upstream
A quick Google search revealed that a fix for this is to bump up proxy_buffer_size in nginx.conf, for both http and https traffic, along these lines:
proxy_buffers 4 256k;
Now nginx was happy when hit directly. However, haproxy was still erroring out with a 502 'bad gateway' return code, followed by PH. Here is a snippet from the haproxy log file:
Jul 22 21:27:13 127.0.0.1 haproxy: 172.16.38.57:53408 [22/Jul/2014:21:27:12.776] www-frontend www-backend/www2:80 1/0/1/-1/898 502 8396 - - PH-- 0/0/0/0/0 0/0 "GET /someurl HTTP/1.1"
Another Google search revealed that PH means that haproxy rejected the header from the backend because it was malformed.
At this point, an investigation into the web app did discover a loop in the code that kept adding elements to a cookie included in the response header.
Anyway, I leave this here in the hope that somebody will stumble on it and benefit from it.
Tuesday, July 22, 2014
Troubleshooting haproxy 502 errors related to malformed/large HTTP headers
Subscribe to: Post Comments (Atom)
Modifying EC2 security groups via AWS Lambda functions
One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...
This post is a continuation of my previous one on " Running Gatling tests in Docker containers via Jenkins ". As I continued to se...
For the last month or so I've been experimenting with Rancher as the orchestration layer for Docker-based deployments. I've been pr...
Here's a good interview question for a tester: how do you define performance/load/stress testing? Many times people use these terms inte...
Came across something similar with the header size being too large for HAProxy. Added this to the config:
# Descrease the size of the rewrite buffer so that
# there is more room to handle large (>8K) headers. See the HAProxy
# manual around tune.bufsize and tune.maxrewrite.
I was facing 502 bad gateway issue. This one line - "tune.maxrewrite 4096" resolved my issue :)
Post a Comment