Flume to Phoenix as Sink


Hadoop Eco System Forums Hadoop Discussion Forum Flume to Phoenix as Sink

This topic contains 2 replies, has 2 voices, and was last updated by  Mahesh 2 years, 9 months ago.

Viewing 3 posts - 1 through 3 (of 3 total)
  • Author
    Posts
  • #2629

    Mahesh
    Participant

    Hi,

    AS we are working in flume to store apache log into phoenix using phoenix-plugin & access log properties configuration .we could run flume and see the events get process,when quering hbase through phoenix there is no record stored.Can you provide tutorial for flume-phoenix connection.

    14/12/18 12:53:09 INFO client.HConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
    14/12/18 12:53:09 INFO client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14a5b888d18003e
    14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Session: 0x14a5b888d18003e closed
    14/12/18 12:53:09 INFO zookeeper.ClientCnxn: EventThread shut down
    14/12/18 12:53:09 INFO query.ConnectionQueryServicesImpl: Found quorum: nn01:2181
    14/12/18 12:53:09 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x843735 connecting to ZooKeeper ensemble=nn01:2181
    14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=nn01:2181 sessionTimeout=180000 watcher=hconnection-0x843735, quorum=nn01:2181, baseZNode=/hbase
    14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Opening socket connection to server nn01/10.10.10.25:2181. Will not attempt to authenticate using SASL (unknown error)
    14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Socket connection established to nn01/10.10.10.25:2181, initiating session
    14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Session establishment complete on server nn01/10.10.10.25:2181, sessionid = 0x14a5b888d18003f, negotiated timeout = 40000
    14/12/18 12:53:09 INFO client.HConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
    14/12/18 12:53:09 INFO client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14a5b888d18003f
    14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Session: 0x14a5b888d18003f closed
    14/12/18 12:53:09 INFO zookeeper.ClientCnxn: EventThread shut down
    14/12/18 12:53:15 INFO serializer.BaseEventSerializer: the upsert statement is UPSERT INTO APACHE_LOG4 (“HOST”, “IDENTITY”, “USER”, “TIME”, “METHOD”, “REQUEST”, “PROTOCOL”, “STATUS”, “SIZE”, “REFERER”, “AGENT”, “F_HOST”, “UID”) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
    14/12/18 12:53:18 INFO sink.PhoenixSink: Time taken to process [10] events was [3] seconds
    14/12/18 12:53:22 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:53:27 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:53:33 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:53:40 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:53:48 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:53:56 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:54:04 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:54:12 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:54:20 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:54:28 INFO sink.PhoenixSink: Time taken to process [1] events was [3] seconds
    14/12/18 12:54:36 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds

    Thanks
    Mahesh.S

    #2634

    Siva
    Keymaster

    Hi Mahesh

    There is some issue with the BaseEventSerializer in the flume sink, so it is not able to parse the records properly and values are updated in the Phoenix/Hbase Table. Even I tried it multiple ways but facing the same issue again and again. So lets wait for some more time so that Phoenix will release stable version of flume plugin.

    Please let everybody know if this issue is solved to you by any chance.

    Regards,
    Siva

    #2645

    Mahesh
    Participant

    Hi Siva,
    Thanks for your valuable reply.Lets wait for next stable version.

    Thanks,
    Mahesh

Viewing 3 posts - 1 through 3 (of 3 total)

The topic ‘Flume to Phoenix as Sink’ is closed to new replies.

Review Comments
default image

I have attended Siva’s Spark and Scala training. He is good in presentation skills and explaining technical concepts easily to everyone in the group. He is having excellent real time experience and provided enough use cases to understand each concepts. Duration of the course and time management is awesome. Happy that I found a right person on time to learn Spark. Thanks Siva!!!

Dharmeswaran ETL / Hadoop Developer Spark Nov 2016 September 21, 2017

.