Kafka Design

While developing Kafka, the main focus was to provide the following:   An API for producers and consumers to support custom implementation   Low overheads for network and storage with message persistence on disk   A high throughput supporting millions of messages for both publishing and subscribing—for example, real-time log aggregation or data feeds   Distributed and highly scalable architecture to handle low-latency delivery   Auto-balancing multiple consumers in the […]

Kafka Installation 1

There are number of ways in which Kafka can be used in any architecture. This section discusses some of the popular use cases for Apache Kafka and the well-known companies that have adopted Kafka. The following are the popular Kafka use cases: Log aggregation This is the process of collecting physical log files from servers and putting them in a central place (a file server or HDFS) for processing. Using […]