Postgresql – 100 terabytes database on PostgreSQL without sharding

databasepostgresql

Is it realistic to setup a 100 TB database (about 90 TB actually) on PostgreSQL without data sharding between a number of nodes? Are there any success stories / examples about similar setups?

Best Answer

50K writes per second that need to be absorbed is more than a challenge usually. Even in synthetic benchmarks with quite simple inserts, PostgreSQL's limits tend to max out around roughly 10 K/s - and there you don't even have such a large beast in terms of database size.

Also the I/O system for that single PostgreSQL node is going to be interesting as even with RAID 10 and assuming that 50K inserts are going to be equal to just 50K IOPS (which is probably wrong, but it depends on your database scheme and indices), you are going to need roughly a hundred disks paired with a very good array that saves you from buying several hundred disks to service those writes in a timely manner.

If sharding is easy and you expect such a huge write load then go for sharding. Writes can be very difficult to scale.

Related Topic