Amazon EC2 Memory Issues – Troubleshooting Tomcat Out of Memory on Amazon Linux

amazon ec2amazon-web-servicesmemory leakmemory usagetomcat

I use AWS Elastic Beanstalk to run my application on Tomcat 8.5 with Java 8 running on 64bit Amazon Linux. I use t3.small which according to specification has 2vCPI and 2.0 GB of memory.
My configuration looks like this:

enter image description here

After some time (2 days) my application runs out of memory. System logs show:

[ 4627.110117] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name

[ 4627.204880] [ 8096]     0  8096   191163    10071     105       4        0             0 aws

[ 4627.209125] [ 8466]    91  8466  1132802   432642     998       8        0             0 java

[ 4627.217630] [ 8540]    48  8540   328251      999      98       4        0             0 httpd

[ 4627.221861] [ 8541]    48  8541   328285     1158      98       4        0             0 httpd

[ 4627.226172] [ 8542]    48  8542   328280     1265      98       4        0             0 httpd

[ 4627.234663] [ 8655]     0  8655   134228     5330      81       3        0             0 cfn-hup

[ 4627.273722] [ 8738]    48  8738   328280     1297      98       4        0             0 httpd
.......

[ 4627.299082] Out of memory: Kill process 8466 (java) score 865 or sacrifice child
[ 4627.303727] Killed process 8466 (java) total-vm:4531208kB, anon-rss:1730568kB, file-rss:0kB, shmem-rss:0kB

Cloud Watch log shows:
enter image description here

Does it mean t3.small is not enough to run my server without memory issues or I have memory leak in my java app? Can I fix it somehow by changing configuration?

Best Answer

You want to run an app that needs 1.5GB on a 2GB machine - that's quite a stretch as there won't be much left for the system, disk caches, other processes, etc. I wouldn't expect a great performance even if it didn't crash.

Run it on t3.medium and see if it still crashes. If it does you may have a memory leak. If it doesn't it means you app simply needs more RAM than t3.small can provide.

If then you still insist on using t3.small even if it doesn't provide an adequate memory size you can try adding a swap space (e.g. 2GB) - that will increase the amount of memory the app will be able to allocate but some of will be very slow, because it's on a disk. It may or may not matter, depends on how the app uses the allocated memory.

Also be aware of the CPU Credits that T2/T3 instances use - you may experience a slow-down after a period of fast processing. It's described here: On clarifying t2 and t3 working conditions?

The bottom line is: use a right-sized instance. You don't want to run out of memory of CPU power, especially not in production.

Hope that helps :)

Related Topic