I am answering to the linux tag. My answer is specific only to Linux.
Yes, huge pages are more prone to fragmentation. There are two views of memory, the one your process gets (virtual) and the one the kernel manages (real). The larger any page, the more difficult it's going to be to group (and keep it with) its neighbors, especially when your service is running on a system that also has to support others that by default allocate and write to way more memory than they actually end up using.
The kernel's mapping of (real) granted addresses is private. There's a very good reason why userspace sees them as the kernel presents them, because the kernel needs to be able to overcommit without confusing userspace. Your process gets a nice, contiguous "Disneyfied" address space in which to work, oblivious of what the kernel is actually doing with that memory behind the scenes.
The reason you see degraded performance on long running servers is most likely because allocated blocks that have not been explicitly locked (e.g. mlock()/mlockall() or posix_madvise()) and not modified in a while have been paged out, which means your service skids to disk when it has to read them. Modifying this behavior makes your process a bad neighbor, which is why many people put their RDBMS on a completely different server than web/php/python/ruby/whatever. The only way to fix that, sanely, is to reduce the competition for contiguous blocks.
Fragmentation is only really noticeable (in most cases) when page A is in memory and page B has moved to swap. Naturally, re-starting your service would seem to 'cure' this, but only because the kernel has not yet had an opportunity to page out the process' (now) newly allocated blocks within the confines of its overcommit ratio.
In fact, re-starting (lets say) 'apache' under a high load is likely going to send blocks owned by other services straight to disk. So yes, 'apache' would improve for a short time, but 'mysql' might suffer .. at least until the kernel makes them suffer equally when there is simply a lack of ample physical memory.
Add more memory, or split up demanding malloc() consumers :) Its not just fragmentation that you need to be looking at.
Try vmstat to get an overview of what's actually being stored where.
Best Answer
Page: A fixed length of contiguous block of virtual memory.
Segment: A segment is your interface into the shared memory. A segment is made up of one or more pages. If you (or your process) haven't created a segment, you're not using shared memory.
ceil: AKA 'ceiling'. A well-defined math function that returns the next highest integer (aka rounding up). See Wikipedia: Floor and ceiling functions
PAGE_SIZE is the number of bytes the OS is using to split up its' chunks of memory. You can find the size with getconf:
shmmax is the maximum size of any individual segment in bytes (not pages).
shmall needs to be at least 'ceil(shmmax/PAGE_SIZE)' because if it was any less you could not create a segment that's shmmax in size. You would run out of pages to use.
Let's say you want to use no more than 8MiB (MB being base 10, MiB - mebibytes being base 2, what your computer actually uses when calculating the sizes) for shared memory on the system.
To find the number of pages 8MiB is you simply divide by PAGE_SIZE.
8MiB / 4096 bytes = 2048.
Set shmall to 2048.
Now let's say that you know you only need a single segment that's 512K (KiB, not KB) in size for postgres. You have all the data to calculate the minimum number you should set shmall to.
ceil(512KiB / 4096 bytes) = 128
The smallest you should set shmall to would be 128 pages. There's nothing stopping you from setting it higher. Shmall is simply a limit specifying that you will not use more than that amount of memory for shared memory whether there's one segment or ten.