We dump debug and transaction logs into MongoDB
.
We really like MongoDB
because:
- Blazing insert perf
- document oriented
- Ability to let the engine drop inserts when needed for performance
But there is this big problem with MongoDB
: The index must fit in physical RAM. In practice, this limits us to 80-150 GB of raw data (we currently run on a system with 16 GB RAM).
So, for us to have 500 GB or a tb of data, we would need 50 gb or 80gb of RAM.
Yes, I know this is possible. We can add servers and use MongoDB sharding
. We can buy a special server box that can take 100 or 200 GB of RAM, but this is the tail wagging the dog! We could spend boucoup $$$ on hardware to run FOSS
, when SQL Server Express can handle WAY more data on WAY less hardware than Mongo
(SQL Server does not meet our architectural desires, or we would use it!) We are not going to spend huge $ on hardware here, because it is necessary only because of the Mongo
architecture, not because of the inherent processing/storage needs. (And sharding? Cost aside, who needs the ongoing complexity of three, five, or more servers to manage a relatively small load?)
Bottom line: MongoDB
is FOSS
, but we gotta spend $$$$$$$ on hardware to run it? We should rather buy commercial software!
I'm sure we are not the first to hit this issue, so we ask the community:
Where do we go next?
(We already run Mongo v2)
Best Answer
If you are at a point where the current performance is too slow or the limits are reached then you have three options. And they are true for any problem.
As you excluded 1) and 2) from your options, there is only solution 3) left. So go for it...