Okay - there are a few things to keep in mind. First, in modern Linux kernels, threads are simply processes that share a few resources and flags, most notably, address space. So yep, threads in the output of something like 'top' can make things pretty confusing. Example: Our Oracle server has about 30 threads running simply appearing as 'oracle' - they each show up as consuming 38 GB of ram. Pretty amusing on first glance.
Ok - the suggestion in the comment above is spot on the money. Use 'pmap' to see exactly not just the memory consumption of the process, but the breakdown of that memory usage. The shared library entries (.so) entries aren't much to worry about. Your real concern is the [ anon ] entries and the [ stack ] entry. See if you can get a total of those. That will give you the honest picture of how much non-shared memory the process is consuming.
A really good reference to read on GC and performance is Oracle's GC Tuning whit
http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html
In general, the more memory you can allocate to your JVM, the better. The larger you set the young generation (to half or so the total VM size) the better. I've played with JVMs of up to 28 GB in size.
For monitoring, you want to use jconsole (if you're a sysadmin, it's horribly interfaced, instrumented, and limiting, but it's what you've got) and watch in particular your GC activity and total sweeps. This will also give you a sense of what your memory utilization patterns look like. This is a GUI monitor instrumented in Java. You'll need to enable the JMX extensions in your JVM to use it.
For seeing what's actually using memory, the 'jmap' utility will show you both heap size ('jmap -heap ') and an inventory ("histogram") of objects in heap ('jmap -F -histo '). The first usually runs quickly, I've seen the second run for 20-30 minutes, during which the JVM is nonresponsive to other tasks, so this isn't something to use lightly on production instances.
Somewhat counterintuitively, the larger you make your ParNew / young generation, the less often, faster, and more effectively, GC runs. GC works by tracking live objects in heap, and runs as ParNew fills up. With a bigger ParNew, more time elapses, and more objects have expired (died), so GC runs less often, and more of the objects have died. allowing more memory to be GCd. We had a configuration in which ParNew was set absurdly small (~50 MB or so), and bumping it to 1-2 GB dropped GC overhead to about 1-5% of what it had been previously (0.01% - 2% of CPU depending on host/workload now, had been 10-50%).
You'll also want to size PermGen such that you don't run out of space. This is where persistent objects (mostly classes) are kept, and when you run out of space it's usually a Bad Thing.
Best Answer
I'm pretty sure that the answer you got from the ElasticSearch community relates to the ZFS ARC (Adaptive Replacement Cache). This of course assumes that your file system is ZFS?
On ZFS the ARC will potentially take up all of the available RAM on the host less 1 Gb. So on a ZFS host tools like
topwill sometimes show that your physical RAM is close to the limit even if it isn't. This is by design. The ARC will automatically release memory to processes that need memory. What memory the ARC uses counts as kernel memory so you can't really see it in a process output.On most of the Solaris systems that I look at daily the physical RAM consumption is about 90%. This is not because they are very utilized, it is the ZFS that grabs unused RAM for its own purpose. Don't be alarmed by this. As the ARC is part of the kernel it can release memory to processes that need it at the speed of the light so to speak. Hence - although you can - I typically do not see a point in limiting the size of the ZFS ARC. Better to let ZFS do its job.
So, if we're talking about ZFS, then yes, file system caching doesn't show up as memory consumption on an individual process. You'll need to execute something like:
to reveal how your memory is actually used. The line "Anon" covers all the user land processes that you see in e.g.
prstatoutput.The other thing for you to know is how the JVM works in terms of memory allocation and release of memory. The JVM grabs memory from the OS as it needs it only restricted by the JVM
-Xmxcommand line parameter. The open question is how (if ever) the JVM will release memory back to the OS if it no longer needs it ? You'll find that is is very difficult to find information on this subject. It seems to depend on which garbage collector is used. As it is difficult to get precise information on this subject (don't know why really) your best option is to assume that the JVM is extremely reluctant to release memory back to the OS. In other words : if you allow a JVM process to grab say 50 GB memory then you better be in a position where you can afford this permanently rather than assuming that this is just a burst.So if you want to limit how much memory the ElasticSearch process can consume then you need to look into the JVM command line parameters, in particular the
-Xmxoption.