MongoDB Capacity Planning – Handling Datasets That Don’t Fit in RAM

On the Nexenta side, there's a volume-check script that's setup to run hourly by default. It will:
Check volume health and capacity, clear correctable device errors, validate mountpoints.
It also sends a weekly summary report via email.

However, there are some things you should consider when planning a Nexenta storage solution for the purposes you've listed.

• You may want to consider having multiple pools for flexibility. A single pool works, but sometimes it's necessary to move data around or just have the option of a second pool on local storage.
• ZFS zvols can be expanded/contracted on the fly. For instance, if you allocate 20TB to a thin-provisioned zvol, you can change it to 30TB or 100TB very easily. You don't need to over-provision 100TB for the future if you don't have it at present.
• With thin-provisioned zvols, once the space is used, you can't reclaim it. If you thin provision a 2TB zvol in a 10TB pool, fill the zvol up, then delete the VMs on that zvol, your pool will still only show 8TB free. That 2TB is going to remain.
• Will you be using ZFS compression or deduplication or both? One situation where it DOES make sense to over-provision is if you're using inline compression and highly-compressible data. Same for data that is deduped. In my case, the data sets I work with compress 60%-80%, so I present larger zvols than the amount of storage I actually have.
• Using mirrors versus raidz1/2/3 makes it easier to expand underlying storage. You can add mirrored disk pairs to a zpool, but you can't expand raidz1/2/3 unless you add another vdev (group of raidz(x) disks). You'd also want to rebalance the data within to redistribute across the disks.
• Which virtualization technology will you be using? If VMWare, you can thin-provision. You will see datastore warnings near 80% utilization, I believe. VMware also complains if you're in a dangerous situation with snapshot size growth.
• If you are doing a lot of VM testing, or have VMs that fluctuate in size, I'd suggest using iSCSI and zvols for the relatively static VMs and NFS for the test VMs (if that's an option for your preferred virtualization solution). With NFS, you can make more efficient use of your storage space since you see the zpool's full available size and don't have any size ceiling to worry about.

In short... I wouldn't overprovision to account for future growth. It's not necessary. There are hourly checks in Nexenta to alert to space utilization. Also think about whether you will use compression or not (deduplication requires a bit more planning). test things out and see what the VM footprint will look like before going into production. It will be more difficult to change afterwards.

The short answer is: Nobody can answer this question except you.

The long answer is that benchmarking your specific workload is something that you need to undertake yourself, because it's a bit like asking "How long is a piece of string?".

A simple one-page static website could be hosted on a Pentium Pro 150 and still serve thousands of impressions every day.

The basic approach you need to take to answer this question is to try it and see what happens. There are plenty of tools that you can use to artificially put your system under pressure to see where it buckles.

A brief overview of this is:

• Put your scenario in place
• Add monitoring
• Add traffic
• Evaluate results
• Remediate based on results
• Rinse, repeat until reasonably happy

Put your scenario in place

Basically, in order to test some load, you need something to test against. Set up an environment to test against. This should be a fairly close guess to your production hardware if possible, otherwise you will be left extrapolating your data.

Set up your servers, accounts, websites, bandwidth, etc. Even if you do this on VMs that's OK just as long as you're prepared to scale your results.

So, I'm going to set up a mid-powered virtual machine (two cores, 512 MB RAM, 4 GB HDD) and install my favourite load balancer, haproxy inside Red Hat Linux on the VM.

I'm also going to have two web servers behind the load balancer that I'm going to use to stress test the load balancer. These two web servers are set up identically to my live systems.

Add Monitoring

You'll need some metrics to monitor, so I'm going to measure how many requests get through to my web servers, and how many requests I can squeeze through per second before users start getting a response time of over two seconds.

I'm also going to monitor RAM, CPU and disk usage on the haproxy instance to make sure that the load balancer can handle the connections.

How to do this depends a lot on your platforms and is outside of the scope of this answer. You might need to review web server log files, start performance counters, or rely on the reporting ability of your stress test tool.

A few things you always want to monitor:

• CPU usage
• RAM usage
• Disk usage
• Disk latency
• Network utilisation

You might also choose to look at SQL deadlocks, seek times, etc depending on what you're specifically testing.

Add traffic

This is where things get fun. Now you need to simulate a test load. There are plenty of tools that can do this, with configurable options:

• JMeter (Web, LDAP)
• Apache Benchmark (Web)
• Grinder (Web)
• httperf (Web)
• WCAT (Web)
• Visual Studio Load Test (Web)
• SQLIO (SQL Server)

Choose a number, any number. Let's say you're going to see how the system responds with 10,000 hits a minute. It doesn't matter what number you choose because you're going to repeat this step many times, adjusting that number up or down to see how the system responds.

Ideally, you should distribute these 10,000 requests over multiple load testing clients/nodes so that a single client does not become a bottleneck of requests. For example, JMeter's Remote Testing provides a central interface from which to launch several clients from a controlling Jmeter machine.

Press the magic Go button and watch your web servers melt down and crash.

Evaluate results

So, now you need to go back to your metrics you collected in step 2. You see that with 10,000 concurrent connections, your haproxy box is barely breaking a sweat, but the response time with two web servers is a touch over five seconds. That's not cool - remember, your response time is aiming for two seconds. So, we need to make some changes.

Remediate

Now, you need to speed up your website by more than twice. So you know that you need to either scale up, or scale out.

To scale up, get bigger web servers, more RAM, faster disks.

To scale out, get more servers.

Use your metrics from step 2, and testing, to make this decision. For example, if you saw that the disk latency was massive during the testing, you know you need to scale up and get faster hard drives.

If you saw that the processor was sitting at 100% during the test, perhaps you need to scale out to add additional web servers to reduce the pressure on the existing servers.

There's no generic right or wrong answer, there's only what's right for you. Try scaling up, and if that doesn't work, scale out instead. Or not, it's up to you and some thinking outside the box.

Let's say we're going to scale out. So I decide to clone my two web servers (they're VMs) and now I have four web servers.

Rinse, repeat

Start again from Step 3. If you find that things aren't going as you expected (for example, we doubled the web servers, but the reponse times are still more than two seconds), then look into other bottlenecks. For example, you doubled the web servers, but still have a crappy database server. Or, you cloned more VMs, but because they're on the same physical host, you only achieved higher contention for the servers resources.

You can then use this procedure to test other parts of the system. Instead of hitting the load balancer, try hitting the web server directly, or the SQL server using an SQL benchmarking tool.