The answer is that Glassfish in the latest versions splits the response into multiple packets.
I posted on the haproxy mailing list and had a remarkably quick response.
Krzysztof Oledzki confirmed that haproxy assumes that the response will all be contained within the the first packet as that is the behavior of most known web servers. He built a patch with a quick and dirty fix which is available in the mailing list archives if you search for Glassfish and can be applied to the beta or latest stable version 1.3.22
I also tried to find out why Glassfish has started to behave this way but without paid support I got nowhere. If anyone can answer that, the bounty is still open.
Your timeouts are too low. Increase them.
timeout connect 30s
timeout client 30s
The absolute minimum is 5 seconds for traffic between two servers in the same rack. A TCP connection takes 3 seconds to open if there is any packet loss, which invariably happens from time to time.
The minimum timeout is 15 seconds to support international traffic, like a client from Australia connecting to a server in North America. There is quite a high latency and low bandwidth in some locations in the world, much worse than one would expect. Being reasonable on timeouts is a prerequisite to do business worldwide.
The minimum timeout is 30 seconds to support mobile connections and poor reception WiFi. It's unreliable connectivity that can and do experience short periods of blackout.
Keep in mind. Timeouts are meant to handle the worst case scenario of connectivity and they should only catch truly failed connections. They could be set somewhat shorter but this has no benefit except generating errors on clients and servers, which is not a benefit.
Consider that a periodic request made every 5 seconds, something as simple as a healthcheck or a polling API, is actually as much as 17280 requests per day. Thus a good timeout setting should cause less than 0.01% of false positives or it's creating errors every day for no reason.
88500 Sessions and 4500 errors in the last 20 minutes.
That's 5% of errors. It's a very high error rate.
Considering that the average webpage takes more than 20 sub requests to load, it means that every single page on your site is failing to load partially.
Best Answer
You need to have health checks in place https://www.haproxy.com/doc/aloha/7.0/haproxy/healthchecks.html. You specifically need check_timeouts "timeout check : time let to the server to answer the check. If both inter and timeout check are set, then the smallest value of both of them is used, after the TCP connection is established"