Kafka disable autocommit

[Wuerike]

I've been losing messages using the kafka input, I put a log as the first processor and the messages I lost dont even get logged.

I noticed that the lost messages are always close to Stopped consuming messages logs, I'm not sure what is happening, maybe the kafka cluster is rebalancing the partitions.

I think a solution would be disable the offest auto commit, but it seems there isn't this option.

Can somebody give me some help?

[mihaitodor]

Hey @Wuerike 👋 What do you mean by "losing messages" and can you provide some means to reproduce this issue you're seeing? Also, can you reproduce it with both the kafka and kafka_franz inputs? Benthos does not ack any message in Kafka until it's successfully written to some output (including drop / drop_on), or deleted() via bloblang. There might be some bug in the underlying library (either Sarama or franz-go, depending on which input you're using), but we'd need a way to reproduce it before raising an issue with them.

[Wuerike]

@mihaitodor dont know if it is possible to reproduce it locally because it was a problem I only saw when working with high throughput, but I've changed to kafka_franz and it got solved

The weird is that I had a log as the first processor, and if I redeployed my app or if a pod restarted, some messages being processed at that time were lost whitout being logged by the first processor.

It seems like the messages were fetched by sarama but before be sent to the processors a restart ocorred and the messages were commited without being processed

[mihaitodor]

The only recent issue in Sarama (which is what the kafka components use) that I'm aware of is this one which was fixed in Benthos v4.24.0 (it wasn't mentioned in the changelog unfortunately) and there's this open PR for a recently-discovered bug which might lead to some issues, but I don't know if it can really produce the behaviour you're seeing.

[Wuerike]

The issue IBM/sarama#2677 is about consuming twice the same offsets, the problem I had was not consuming any time (or consuming without processing it). But I do have something in common, I'm consuming from Azure Event Hubs with kafka version 1.0.0

As kafka_franz solved for my use case, and I'm leaving on vacation, I'll not be able to keep this investigation for a while, but I can try find a way to reproduce it in the next months

[mihaitodor]

Like I said, it might be a related issue if you're not running v4.24.0+. Also, the PR I linked might be another cause. Really hard to tell without being able to reproduce it. Enjoy your time off!