MongoDB WiredTiger storage not using two CPU cores

mongodb

As claimed in WiredTiger storage engine description, it provides an ability of better concurrency due to document-level locking. From this post:

WiredTiger scales on modern, multi-CPU architectures. Using a variety of programming techniques such as hazard pointers, lock-free algorithms, fast latching and message passing, WiredTiger performs more work per CPU core than alternative engines.

For some reason, my use case doesn't seem to benefit from this. I have a database with many concurrent writes (mostly updates), and this kind of load can't seem to overcome the limit of 2000 updates per sec. Here is the mongostat 10 output:

insert query update delete getmore command % dirty % used flushes  vsize    res qr|qw ar|aw netIn netOut conn     time
     3   780   1936    141      42     3|0     0.3    1.0       0 717.0M 289.0M   0|0   1|0  433k     6m  141 17:16:32

Disk throughput is not saturated, iostat -x 10 output:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          20.56    0.00   20.31    0.10    0.10   58.93

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvda              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
xvdb              0.00     5.80    0.00  102.50     0.00  6188.80    60.38     3.46   33.79   0.46   4.72

Given all of this, I assume that the bottleneck is in CPU usage, which is always stable at 100% all the time for mongod process in the top (from the total of 200%, which means that only one CPU from two is used). 58% idle time from iostat also confirms that.

Are there any methods to determine whether the WiredTiger storage is using document-level locking and two CPU cores simultaneously as it should be? Or could this happen due to any reasons other than CPU capacity limit?

Best Answer

Write operations are being done serial on single core instance. Read operations have concurency. This is what i have understood from my readings so far.

Related Solutions

Linux – Enabling WiredTiger engine in MongoDB 3

If I remember correctly, you have to have this in your config:

storage:
    engine: wiredTiger

Though I can't test it right now...

Edit: Found the same suggestion in How to migrate MongoDB 2.6 to 3.0 with WiredTiger

Does the MongoDB 3.2 WiredTiger compression include stuff stored in RAM

WiredTiger has different representations of data on disk versus in memory, and uses different compression approaches for indexes vs collection data.

The answer on what is compressed in memory is somewhat nuanced, but the high level summary is:

collection data is compressed in the filesystem cache
collection data is uncompressed in the WiredTiger internal cache
indexes are compressed on disk and in memory

Compression Approaches

By default WiredTiger uses Snappy block compression for collection data, but there are other options available including zlib compression or no compression. Block compression can provide significant storage savings, but data must be uncompressed to be manipulated by the server. Irrespective of compression options, data is still written to the disk in a block format that differs from the in-memory representation in the WiredTiger cache.

Indexes are compressed using index prefix compression, which effectively deduplicates common prefixes from indexed fields. This can be especially effective for compound indexes, since leading field values will be repeated with unique values for additional fields in the index. Prefix compression also allows queries to operate directly on compressed indexes.

What's compressed in RAM?

As at MongoDB 3.4 (and including prior MongoDB versions with WiredTiger), there are two significant caches for data in RAM:

The WiredTiger internal cache, which is controlled by the cacheSizeGB configuration setting.

The default cache size in MongoDB 3.4 is the larger of 50% of RAM less 1GB, or 256MB. Collection data in the internal cache is uncompressed, however index data still uses prefix compression. The data in the internal WiredTiger cache is effectively the current working set.
The O/S filesystem cache, which is generally the remainder of free RAM that is not used by the WiredTiger cache or other processes.

The filesystem cache is identical to on-disk representation.

Cache Tuning

Note that the WiredTiger cache doesn't represent MongoDB's total memory usage: mongod will still need to allocate memory outside the cache for other uses such as connections and data processing (eg. aggregation, map/reduce, in-memory sort).

The WiredTiger internal cache should generally be left at the default size or potentially reduced. If your data compresses well and the uncompressed data is much larger than RAM, you will be able to fit more data in RAM overall by reducing the WiredTiger cache size to free memory for the filesystem cache. The MongoDB manual has an FAQ with more information: To what size should I set the WiredTiger internal cache?.

For more background, I'd recommend reviewing the New Compression Options in MongoDB 3.0 blog post and A Technical Introduction to WiredTiger presentation.

Related Topic