I have a Kafka DLQ (dead-letter queue) topic where I'm sending the failed records, but I don't want to keep reprocessing them until a patch is deployed. Thus, the idea would be to have the consumer stopped until the fix is applied.
The Spring Kafka gives has the KafkaListenerEndpointRegistry
, which allows managing the pause/resume of the Kafka consumer, and from the small local tests I have done, it seems to be working. However, I feel that pausing a Kafka Consumer is not a good practice, and I wonder if this could be done with a different approach.
Another possible approach could be having the DLQ messages constantly being retried (which would keep failing until the fix is applied), but I believe that it would only put a load on the cluster and wouldn't bring any benefit.
Has anyone had experience with a similar situation, and how have you dealt with it? Does pausing a consumer is/isn't a good practice? Any suggestions for this problem are welcome.