We had a situation recently where our web application started to behave strangely. First nginx (which sits in front of the application) started to error out with messages of this type:
upstream sent too big header while reading response header from upstream
A quick Google search revealed that a fix for this is to bump up proxy_buffer_size in nginx.conf, for both http and https traffic, along these lines:
proxy_buffers 4 256k;
Now nginx was happy when hit directly. However, haproxy was still erroring out with a 502 'bad gateway' return code, followed by PH. Here is a snippet from the haproxy log file:
Jul 22 21:27:13 127.0.0.1 haproxy: 172.16.38.57:53408 [22/Jul/2014:21:27:12.776] www-frontend www-backend/www2:80 1/0/1/-1/898 502 8396 - - PH-- 0/0/0/0/0 0/0 "GET /someurl HTTP/1.1"
Another Google search revealed that PH means that haproxy rejected the header from the backend because it was malformed.
At this point, an investigation into the web app did discover a loop in the code that kept adding elements to a cookie included in the response header.
Anyway, I leave this here in the hope that somebody will stumble on it and benefit from it.
Tuesday, July 22, 2014
One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...
I first saw nsupdate mentioned on the devops-toolchain mailing list as a tool for dynamically updating DNS zone files from the command line....
Here's a good interview question for a tester: how do you define performance/load/stress testing? Many times people use these terms inte...
A common requirement when setting up a development or staging server is to try to mimic production as much as possible. One scenario I'...