Number of consumer threads for a kafka source

[hartfordfive]

Based on the kafka source documentation, it doesn't appear you can directly specify the number of consumer threads. Does it end up being a single thread per defined kafka source block or does it automatically detect the number of partitions and set the number of consumers accordingly?

I'm looking at running multiple Vector instances in Kubernetes to consumer from various topics, although I need each instance to have an optimal consumer count in order to avoid having too many Vector pods deployed, each using up an IP. For example, if I have a topic with 10 partitions, I could choose to deploy two Vector pods, each with a kafka source, configured with the same topic and consumer group but each only having a total of 5 consumer threads. This would ensure that the consumption of all 10 partitions is evenly distributed amongst both Vector instances. For example, with a tool such as Logstash, I can easily specify the number of consumer threads:

input {
  kafka {
    bootstrap_servers => "kafka01:9092,kafka02:9092,kafka03:9092"
    topics => ["service-logs"]
    group_id => "service-logs-consumer"
    consumer_threads => 5
    codec => json
  }
}

I appreciate any information/help relating to this!

[jszwedko]

Hey! You can see a description of librdkafka's (the library we use to interact with Kafka) thread model here: https://github.com/confluentinc/librdkafka/wiki/FAQ#number-of-internal-threads