Linux – Issues with Haproxy slowing down after time

high-availabilitylinuxreverse-proxytraffic

We recently migrated from Perlbal to HAProxy due to Perlbal's memory usage. It's been a fairly flawless migration though lately we've run into an issue that I can't seem to figure out (it doesn't help I'm not familiar with HAproxy's umpteen configuration options).

We're a media website serving a fair amount of traffic – we use HAProxy to spread the load across our media servers. Upon starting HAProxy everything is snappy and works accordingly – though after about 10-20 minutes things start to slow down and media is served very slowly. I'm fairly certain it's HAProxy since a simple restart of the service rectifies the issue.

I'm using a fairly vanilla HAProxy configuration with no bells or whistles:

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        retries 3
        option redispatch
        maxconn 2000
        contimeout      5000
        clitimeout      50000
        srvtimeout      50000

listen  media 0.0.0.0:80
        balance roundrobin
        server  media_1 x.x.x.x:8080 weight 1 maxconn 1024 check
       server  media_2 x.x.x.x:8080 weight 1 maxconn 1024 check

Could anyone shed some light onto what the issue may be?

Best Answer

Your config is very simple indeed. I see that you don't have "option httpclose" and you have a very low "maxconn", maybe you're simply running out of connections under high load ? You should definitely enable this option. You should also enable the stats page, by adding "stats uri /haproxy-stats" in the "listen" section. Then you connect your browser to that URL on your site and you'll see all the stats with concurrent connections, connections per second, errors, ... That will help you a lot.

Oh, what version is this ? 1.3.16 has a nasty bug which makes it eat a lot of CPU on some traffic patterns. You should be using either 1.3.18 or 1.3.15.X.

Willy

Related Topic