Redhat – OOM-Killer, jboss and kernel panic

32-bitjbossoomredhatvmware-esx

We are running RedHat 3.4.6(x32) on VMWareEsx3.5(x64) with 6GB RAM.
A few java processes(including jboss) are running in the background.

The problem is that the java processes consume lots of memory, and sometimes they are killed by the OOM-killer.
When OOM-killer is about to act, the free physical memory is very low 100MB-200MB, but the swap is not used (99% free).
Sometimes this causes a kernel panic too.

So why isn't the swap used?
How to investigate this kernel panic?
Is using 6GB memory on 32bit Redhat wise?

Thanks

Best Answer

Personally I would never use PAE (over 4G of RAM on a 32-bit system). You'll get much better mileage running an actual 64-bit kernel and system.

OOM should only trigger when a malloc might fail. (not when you have lots of swap available)

The 32-bit kernel is likely to be part of the cause. PAE uses different memory zones and it may be that one zone isn't allowed to malloc from another.

Have you modified your swappiness? (How readily the kernel will use swap.) cat /proc/sys/vm/swappiness ?

You might also investigate tuning vm.dirty_ratio or vm.lower_zone_protection = 100.

Have you captured the kernel panic? (a serial console is often a good way to do this)

You can also try to preempt the OOM-Killer with your own process monitoring software. (take a look at Monit)

Best of luck

Related Solutions

Linux – Prevent Linux’s OOM from killing apache on our webserver

It doesn't sound like you are addressing the root cause of the issue by actually debugging why this cron job is using up so much memory.

You can try setting this option

echo 1 > /proc/sys/vm/oom_kill_allocating_task

which will tell the OOM killer to kill the process that triggered the OOM condition, but this is not guaranteed to be your cron job. You can also use "ulimit -m" in your script to set the max amount of resident memory to use. I think your best bet would be to evaluate why the cronjob is using up so much memory and if its perhaps best suited for another host or to be rewritten to consume less memory.

Linux – OOM killer goes insane

It gets inherited from the process that spawned it. If SSH is set to -17 then Bash will be. If you restart via Bash, you'll spawn it even further.

[i-180ae177] root@migrantgeek ~ # pgrep mysqld_safe
11395
[i-180ae177] root@migrantgeek ~ # cat /proc/11395/oom_adj 
0
[i-180ae177] root@migrantgeek ~ # for pid in `pgrep bash`; do echo -17 >  /proc/$pid/oom_adj; done
[i-180ae177] root@migrantgeek ~ # /etc/init.d/mysqld  restart
Stopping MySQL:                                            [  OK  ]
Starting MySQL:                                            [  OK  ]
[i-180ae177] root@migrantgeek ~ # pgrep mysqld_safe
11523
[i-180ae177] root@migrantgeek ~ # cat /proc/11523/oom_adj 
-17

Editing the init script to change the value at the end of the startup process should fix this.

Best Answer

Related Solutions

Linux – Prevent Linux’s OOM from killing apache on our webserver

Linux – OOM killer goes insane

Related Topic