I am looking for some clarification on the subject.
In Kafka documentations I found the following:
Kafka only provides a total order over messages within a partition, not between different partitions in a topic. Per-partition ordering combined with the ability to partition data by key is sufficient for most applications. However, if you require a total order over messages this can be achieved with a topic that has only one partition, though this will mean only one consumer process per consumer group.
So here are my questions:
-
Does it mean if i want to have more than 1 consumer (from the same group) reading from one topic I need to have more than 1 partition?
-
Does it mean I need same amount of partitions as amount of consumers for the same group?
-
How many consumers can read from one partition?
Also have some questions regarding relationship between keys and partitions with regard to API. I only looked at .net APIs (especially one from MS) but looks like the mimic Java API.
I see when using a producer to send a message to a topic there is a key parameter. But when consumer reads from a topic there is a partition number.
- How are partitions numbered? Starting from 0 or 1?
- What exactly relationship between a key and partition?
As I understand some function on key will determine a partition. is that correct? - If I have 2 partitions in a topic and want some particular messages go to one partition and other messages go to another I should use a specific key for one specific partition, and the rest for another?
- What if I have 3 partitions and one type of messages to one particular partition and the rest to other 2?
- How in general I send messages to a particular partition in order to know for a consumer from where to read?
Or I better off with multiple topics?
Thanks in advance.
Best Answer
Let's see the following properties of kafka:
With these properties, kafka is smartly able to provide both
ordering guarantees
andload balancing
over a pool of consumer processes.To answer your question, yes, in the context of the same group, if you want to have
N consumers
, you have to haveat least N partitions
.I think this has been explained in the first answer.
The
number of consumers
that can read from one partition is always equal to thenumber of consumer groups
subscribing to that topic.First, we must understand that the
producer
is responsible for choosing which record to assign to which partition within the topic.Now, lets see how producer does so. First, lets see the class definition of
ProducerRecord.java
:Here, the field that we have to understand from the class is
partition
.From the ProducerRecord docs,
partition number
is specified, thatpartition
will be used when sending the record.key
is present a partition will be chosen using ahash of the key
.key
norpartition
is present a partition will be assigned in around-robin fashion
.