channel has been removed due to an error


Error Scenario:

channel has been removed due to an error during configuration.

This error message is received in ~/logs/flume.log file when starting a flume agent with missing or wrong configuration with JDBC channel setup.

Root Cause:

DerbySchemaHandler.schemaExists error message is received when a flume agent’s sink is not able to receive the events from this JDBC channel and for some reason agent process stopped and when we try to restart the same agent with JDBC channel, then as the database schema already exists from previous then this error message is received.

This is due to the default parameters of JDBC channel:

Property Name Default Description
create.schema true If true, then creates db schema.
create.index true Create indexes to speed up lookups
create.foreignkey true  

As per create.schema = true property, it tries to create the jdbc channel once again with the same schema. That’s why we are receiving error message

Resolution:

We can either remove the JDBC channel’s db contents from /home/siva/.flume/jdbc-channel/db directory or change the configuration parameters for JDBC channel as shown below in flume.conf properties file.

Once the properties are changed or existing db is deleted, we need to start the agent again. Then we will not receive this error message in flume.log file.


About Siva

Senior Hadoop developer with 4 years of experience in designing and architecture solutions for the Big Data domain and has been involved with several complex engagements. Technical strengths include Hadoop, YARN, Mapreduce, Hive, Sqoop, Flume, Pig, HBase, Phoenix, Oozie, Falcon, Kafka, Storm, Spark, MySQL and Java.

Leave a comment

Your email address will not be published. Required fields are marked *


Review Comments
default image

I have attended Siva’s Spark and Scala training. He is good in presentation skills and explaining technical concepts easily to everyone in the group. He is having excellent real time experience and provided enough use cases to understand each concepts. Duration of the course and time management is awesome. Happy that I found a right person on time to learn Spark. Thanks Siva!!!

Dharmeswaran ETL / Hadoop Developer Spark Nov 2016 September 21, 2017

.