Kafka is a powerful streaming platform that can be used to build real-time data pipelines and streaming applications. One of the key features of Kafka is its ability to handle large volumes of data and ensure that messages are delivered reliably and efficiently. To achieve this, Kafka provides several mechanisms to control message loss and transmission speed.
One of the primary ways to control message loss and transmission speed in Kafka is through batch transmission. When a message is received, the Kafka producer stores it in the Record Accumulator in a batch format, rather than sending it immediately to the broker. The batch size can be adjusted using the buffer.memory and buffer.size options. This approach can help optimize network traffic and reduce the number of requests being sent to the broker, thereby improving the overall efficiency of message transmission.
Another way to control message loss in Kafka is through the use of acknowledgement (acks) options. The acks option is used to specify the number of acknowledgments a producer requires the broker to receive before considering a message as sent. For example, if the acks option is set to "all," the producer will not wait for all followers to replicate the message, but instead waits for the message to be replicated to at least the number of brokers specified by the min.insync.replica configuration. This approach ensures that messages are not lost during transmission and that the producer has a confirmation once the message has been successfully replicated across the cluster.
In addition to batch transmission and acks options, Kafka provides a retry mechanism that allows producers to resend messages that failed to be sent to the broker. The retry mechanism applies to both synchronous and asynchronous producers. While synchronous producers guarantee message order, they can have poor performance due to the need to wait for confirmation from the broker before sending the next message. Conversely, asynchronous producers may not maintain message order, as messages may be sent out of order due to the nature of the asynchronous callback mechanism.
Kafka also provides three types of message delivery guarantees: at most once, at least once, and exactly once. The at most once guarantee means that messages are sent once and not retried if a failure occurs. The at least once guarantee means that messages are sent at least once, and duplicates may occur due to retries. Finally, the exactly once guarantee means that messages are sent exactly once, and no duplicates will occur, even if retries are performed. To achieve the exactly once guarantee, the producer must use the idempotence feature, which ensures that messages are sent once and only once, even if the producer retries sending the message.
In summary, Kafka provides several mechanisms to control message loss and transmission speed, including batch transmission, acks options, and retry mechanisms. Kafka also provides three types of message delivery guarantees: at most once, at least once, and exactly once. The exact once guarantee can be achieved using the idempotence feature, which ensures that messages are sent once and only once, even if the producer retries sending the message.
하나의 kafka broker가 죽는다면?
kafka의 모든 명령어의 --bootstrap-server 옵션은 필수이다.
(토픽 생성) kafka-topics —bootstrrap-server localhost:9092 —create test_topic_01
(토픽 목록 출력 ) kafka-topics —bootstrap-server localhost:9092 —list
(파티션이 3개인 토픽 생성) kafka-topics —bootstrap-server localhost:9092 —create test_topic_02 —partitions 3
실제 파티션 파일 시스템 구조
