What is optimal no. of partition for a topic that has 5 brokers and replication factor=3?
Total there are only 6 topics in this cluster.
Machine configuration that each broker is running as follows.
Memory=16gb
Porcessor= Octacore, Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
Any challenges other than open file for 1000 partitions per topic??
Best Answer
Starting since Kafka 1.1.0 it's recommended to have up to 4'000 partitions per broker and up to 200'000 partitions per cluster. Read more here.
Pls, note that even if you can create a large number of partitions it doesn't mean that your hardware will be able to support it. Make sure you scale your cluster hardware(CPU, RAM, HDD etc) accordingly to your load.