Kafka


Apache Storm Integration With Apache Kafka

Installing Apache Storm The prerequisite for storm to work on the machine. a. Download and installation commands for ZeroMQ 2.1.7: Run the following commands on terminals

b. Download and installation commands for JZMQ: 

  2. Download latest storm from http://storm.apache.org/downloads.html 

Second start Storm Cluster by starting master and worker nodes. Start master node i.e. nimbus. To start master i.e. nimbus go to the ‘bin’ directory of the […]


Kafka Design

While developing Kafka, the main focus was to provide the following:   An API for producers and consumers to support custom implementation   Low overheads for network and storage with message persistence on disk   A high throughput supporting millions of messages for both publishing and subscribing—for example, real-time log aggregation or data feeds   Distributed and highly scalable architecture to handle low-latency delivery   Auto-balancing multiple consumers in the […]


Kafka Installation 1

There are number of ways in which Kafka can be used in any architecture. This section discusses some of the popular use cases for Apache Kafka and the well-known companies that have adopted Kafka. The following are the popular Kafka use cases: Log aggregation This is the process of collecting physical log files from servers and putting them in a central place (a file server or HDFS) for processing. Using […]


Kafka Installation and Test Broker Setup

In this post we will discuss about Kafka Installation and Test Broker Setup in Ubuntu machine in a stand-alone zookeeper mode. Apache Kafka: Open Source Message Broker from Apache Software Foundation.  Initially developed at LinkedIn and later contributed to Open-source community. It is written in Scala. Kafka Provides a unified, high-throughput, low-latency platform for handling real-time data feeds. Types of data being transported through Kafka? Metrics: operational telemetry data Tracking: […]