I have a standalone application where I am trying to create 5 threads for each thread I am creating a mongoclient and persisting the details in MongoDB and then closing the mongoclient.
When I start my mongo service it is running but as soon as I start my java application after some time my mongo service is getting killed and I can see the below error in syslog.
Out of memory: Kill process 12715 (mongod) score 433 or sacrifice child\\
kernel: [2946780.340246] Killed process 12715 (mongod) total-vm:6646800kB, anon-rss:6411432kB, file-rss:0kB
I am using Linux server it has 10GB RAM. As far as I know we can do thousands of transactions in MongoDB. And before I start I can see free memory after some time I am getting the above error.
Not sure exactly what to do, Do we need to set anything in linux server to handle multiple mongo requests? or do I need to increase the limit of mongo service any where?
Can any one please suggest me on this?
Thanks in Advance.
o/p of free -m before and after execution of my java application are as follows:
Before:
total used free shared buffers cached
Mem: 14032 7646 6385 54 4 153
-/+ buffers/cache: 7489 6542
Swap: 0 0 0
After:
total used free shared buffers cached
Mem: 14032 13845 186 54 106 4131
-/+ buffers/cache: 9607 4425
Swap: 0 0 0
The below are the details from mongod.conf
# mongod.conf
storage:
dbPath: /xxx/xxx
journal:
enabled: true
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
net:
port: 27017
bindIp: 127.0.0.1,xx.xx.xx.xx
processManagement:
fork: true
security:
authorization: enabled
Output of cat /proc/<PId>/limits
:
cat /proc/17694/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 112088 112088 processes
Max open files 1024 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 112088 112088 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
ps xua | grep java results :::
root 17142 0.4 3.1 5645648 453180 pts/0 Sl Sep04 2:27 java -d64 -Xms1024m -Xmx2048m -server -cp bin:lib/* javaclassname -resourceDir /xxx/bin -dataDir /xxx/data
root 17491 22.1 11.2 5653432 1610716 pts/0 Sl Sep04 115:14 java -d64 -Xms1024m -Xmx2048m -server -cp bin:lib_new/* javaclassname -resourceDir /xxx/bin -dataDir /xxx/data -outConfigPath /xxx/xxx.xml -outCatBrandPath /xxx/xxx.xml -outCatAndSunCatPath /xxx/xxx.xml
xxx 19223 3.5 22.1 20396560 3184668 ? Sl Aug28 398:11 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/elasticsearch-5.4.0 -cp /elasticsearch-5.4.0/lib/* org.elasticsearch.bootstrap.Elasticsearch
root 21617 2.2 21.7 18549248 3124416 ? Sl Aug28 247:34 /usr/bin/java -Djava.util.logging.config.file=/opt/apache-tomcat-8.0.30/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/opt/apache-tomcat-8.0.30/endorsed -classpath /opt/apache-tomcat-8.0.30/bin/bootstrap.jar:/opt/apache-tomcat-8.0.30/bin/tomcat-juli.jar -Dcatalina.base=/opt/apache-tomcat-8.0.30 -Dcatalina.home=/opt/apache-tomcat-8.0.30 -Djava.io.tmpdir=/opt/apache-tomcat-8.0.30/temp org.apache.catalina.startup.Bootstrap start
swapon -s output :
Filename Type Size Used Priority
Best Answer
It seems that the OS is running out of physical memory, and since your server doesn't seem to have any swap configured, the kernel's out-of-memory killer (OOM-killer) kicks in and terminates processes to free up some RAM (in this case it's mongodb, but could be any other).
You should create a swap partition on that server.