Outgrew MongoDB … now what?

mongodb

We dump debug and transaction logs into MongoDB.

We really like MongoDB because:

  • Blazing insert perf
  • document oriented
  • Ability to let the engine drop inserts when needed for performance

But there is this big problem with MongoDB: The index must fit in physical RAM. In practice, this limits us to 80-150 GB of raw data (we currently run on a system with 16 GB RAM).

So, for us to have 500 GB or a tb of data, we would need 50 gb or 80gb of RAM.

Yes, I know this is possible. We can add servers and use MongoDB sharding. We can buy a special server box that can take 100 or 200 GB of RAM, but this is the tail wagging the dog! We could spend boucoup $$$ on hardware to run FOSS, when SQL Server Express can handle WAY more data on WAY less hardware than Mongo (SQL Server does not meet our architectural desires, or we would use it!) We are not going to spend huge $ on hardware here, because it is necessary only because of the Mongo architecture, not because of the inherent processing/storage needs. (And sharding? Cost aside, who needs the ongoing complexity of three, five, or more servers to manage a relatively small load?)

Bottom line: MongoDB is FOSS, but we gotta spend $$$$$$$ on hardware to run it? We should rather buy commercial software!

I'm sure we are not the first to hit this issue, so we ask the community:

Where do we go next?

(We already run Mongo v2)

Best Answer

If you are at a point where the current performance is too slow or the limits are reached then you have three options. And they are true for any problem.

  1. Scale vertically: Meaning increase your machine power. More CPU or more RAM.
  2. Scale horizontally: Meaning increase the amount of workers. More processes, more threads, more machines.
  3. Change design: Do it differently. Other software, other algorithms, other storage system, other whatever.

As you excluded 1) and 2) from your options, there is only solution 3) left. So go for it...

Related Topic