Freebsd – BIND 9.10 constantly killed on FreeBSD 10.0 with out of swap space

binddomain-name-systemfreebsdkill-processswap

In one of our slave DNS servers BIND, version bind910-9.10.0P2_3, constantly get killed with the following message in /var/log/messages:

Jul 30 01:00:10 cinnabar kernel: pid 602 (named), uid 53, was killed: out of swap space

This service runs on a FreeBSD 10.0 VM in XenServer 6.2, it has 512MB of system memory.

At this moment pstat -m -s return this:

Device          1M-blocks     Used    Avail Capacity
/dev/ada0p3           512        9      502     2%

I don't think it's a swap problem, it appears to be memory leakage, but I'm unsure.

EDIT: Access information.

This is one of two slave DNS servers, they only store the zones from the authoritative server and act as a recursive server for the internal users to the outside world. The number of clients is something between 700-1500 simultaneous users. Since we have a /21 internal space and a /23 public IPv4 space and there's no queries from the outside world, the port 53 is even blocked on the firewall to those machines.

Best Answer

If you have any kind of monitoring on this server, it would be nice to check if there are any peaks on memory usage right around the time processes get killed. Then you could try to find a correlation with number of requests, etc.

That being said, it could either mean there is indeed no memory left on the system but most likely Bind is requesting a contiguous area of memory, fragmentation is getting in the way and FreeBSD is trying to swap out some processes to make room for that. It probably can't swap out many pages, fails to allocate and triggers the out of memory killer.

If you have disk space, the easiest solution is to add more swap through a swap file (not need for a partition). Ideally, you should limit the cache size (Bind defaults to no unlimited), as suggested by Håkan, but that could have a performance impact. Without more statistics is really hard to tell. Even domestic routers have 512MB of RAM nowadays so you should consider increasing that (and limiting the cache) for a production server serving 700-1500 simultaneous users (which could translate in much more req/sec, again, without more information it's hard to tell).

You could also try tweaking the malloc implementation via the MALLOC_PRODUCTION knob, but I think that is too extreme in the face of easier solutions available.

Related Solutions

Swap Space – How to Reclaim Increasing Swap Size in Linux

If info is swapped out to disc and later read back into memory, it will often be left allocated in the swap area until swap space runs low. That means that if the same info needs to be swapped out again later and hasn't changed, the OS can just drop the pages from allocated RAM without needing to write anything to disc saving time.

Swap allocated to stuff that has been read back into memory will be freed either

when the relevant pages are no longer needed at all (i.e. are freed by the application)
when the relevant pages are changed (so the copy on disc is no longer up to date)
the machine runs low on swap space so clears some things that are already in RAM to make room

Look in /proc/meminfo for a line called "SwapCached". This entry counts pages that are found both in RAM and in swap partitions. For instance, picking a small VM at random, the /proc/meminfo virtual file one of my VMs shows:

SwapTotal:        698816 kB
SwapFree:         624520 kB
SwapCached:        17232 kB

indicating that 74268K of swap space is allocated, but that 17232K worth of those pages are also currently mapped into RAM too (so could be deallocated from swap at a moment's notice if the space is needed by something else).

Also there will no doubt be pages sat there that was swapped out ages ago and have never been used again since. The kernel will not reload pages from swap just because there is some free RAM to read it back into as that free RAM might be better used for cache or buffers - pages written to swap are generally only reread when they are next needed.

If you want to clear out what is in swap, as long as you have enough free and/or freeable (i.e. free+cache+buffers (less those parts of the c+b counts that are not freeable RightThisInstant)), just turn it off and back on again with swapoff -a && swapon -a.

Of course you could also have a memory leak somewhere too, but that is not the only explanation for the behaviour you are seeing.

Httpd – Running out of swap space on web servers, what to do

You're running out of swap because you're using all your RAM and then some. You have a serious problem that you need to rectify right now.

You have two choices: ignore cause and just add more RAM, or target the problem of what's actually munching on your memory.

Adding RAM is fairly cheap and fairly easy if it's your server but it's a temporary fix and if it's a VPS or a rented server, it's not so cheap. Let's have a go at fixing the root problem instead. What's sucking in that much memory? Here are a few tips:

Turn off InnoDB (unless you need it) in MySQL
Beat Apache (and MySQL) with the Stick of Configuration +5
Consider a smaller httpd like cherokee, lighttpd or nginx (they're really fast and eat almost no RAM). Main downside is you can't use .htaccess files but you can hard-code in their functionality.
Are you using an OP-code cache for PHP? Try turning it off or switching to another, more efficient one.

In terms of just getting swap items back into real RAM, you can do that by:

swapoff -a && swapon -a

But don't try then when you've got less free memory than you have things in swap. It'll crash your server.

Best Answer

Related Solutions

Swap Space – How to Reclaim Increasing Swap Size in Linux

Httpd – Running out of swap space on web servers, what to do

Related Topic