Based on the snapshot here, the memory utilization seems fine and an upgrade wouldn't be necessary (and probably wouldn't help).
Free & top both state that there is just ~50 MB or so free, but when accounting for buffers, that number is significantly higher. Linux is going to use any unallocated/unused physical RAM as cache & buffer space. If your applications need more memory, brk & sbrk will yank it from cache. There are performance implications as that happens more frequently, but that's probably not a concern here at all.
Now, it's very possible that you're indeed spiking on occasion during times of high traffic or maintenance script runs. Do you have any cron jobs scheduled that execute large queries as part of a cleanup process or anything? Is there anything that might swallow up RAM temporarily and then release it? Account freeze batch jobs or anything?
All that said, can you describe your problem a bit more? If you're not able to ping the system, I highly, highly doubt it's a memory problem at all. Does your server just go away for a while and then come back? Check your web server logs during that period, is there a period of inactivity for everyone, or just you? If your machine is just falling off of the planet for a period of time, I'd vote that it's something network related.
What about dmesg output? Net device errors?
Well, you have practically all the time over 8G either free (yellow, "unused") or as a cache (dark blue). In my opinion, your upper limit to allocate for memcached is maybe 6GB (give or take), which will still allow quite a lot of headroom for other processes and some disk cache without real risk of swapping. One could also argue you could allocate more. It's a matter of taste and real performance benefits will depend on your specific application.
Keep in mind that if you tell memcached to use up to 6GB memory, it won't use that much unless you store 6G in it. It will only consume 1GB if you only store 1GB in it, even if the limit is 6GB. I'd recommend you add memcached to your monitoring, and keep an eye on memcached's bytes statistic and limit the memory usage accordingly to make sure the system will not do anything unexpected if there's surprise surge of users.
From shell you can use free to determine memory usage, and it'll tell something like this:
total used free shared buffers cached
Mem: 4060096 3775976 284120 0 270828 1899720
-/+ buffers/cache: 1605428 2454668
Swap: 577528 0 577528
Free is unused memory, cached is disk cache. Free memory will get used as a disk cache over time, so it's quite safe to consider memory used as a cache as free, even if it's being used for something useful at the moment.
Best Answer
Well, try allowing more processes to apache. Here are some helpful links: