We are doing RND to use Kafka in our company. While learning about Kafka, I have the following questions.
I know that Kafka can assign 1 consumer per 1 partition and can only process 1 message.
So, under the following assumptions:
- Consumers can be horizontally expanded according to the number of partitions, and are always configured to have a 1-1 structure.
- It takes 1 second to process 1 message, and this cannot be optimized or separated any more.
When the processing target of our service is 200,000 TPS,
- Do we need to create 200,000 partitions to implement this? 1-1. What problems occur when partitions are created in large quantities like the above?
- Is Kafka horizontally scalable? In other words, if we add more servers, will there be any problems when creating and managing more partitions?
This question was asked to see if it is possible to flexibly handle the desired throughput by adding additional servers when the target is fluid.