Optimal number of partition for kafka topic on 5 brokers with replication factor=3 in 1 cluster

apache-kafka

What is optimal no. of partition for a topic that has 5 brokers and replication factor=3?
Total there are only 6 topics in this cluster.

Machine configuration that each broker is running as follows.

Memory=16gb
Porcessor= Octacore, Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz

Any challenges other than open file for 1000 partitions per topic??

Best Answer

Starting since Kafka 1.1.0 it's recommended to have up to 4'000 partitions per broker and up to 200'000 partitions per cluster. Read more here.

Pls, note that even if you can create a large number of partitions it doesn't mean that your hardware will be able to support it. Make sure you scale your cluster hardware(CPU, RAM, HDD etc) accordingly to your load.

Related Topic