Error Repository

Oozie Share Lib does not exist error 1

This entry was posted in Error Repository on by

In this post we will discuss about one of common/frequent error message faced by many oozie beginners due to incorrect/incomplete oozie setup. Oozie Share Lib does not exist error is received if oozie sharelib is not installed properly. Error Scenario: java.io.FileNotFoundException: File /user/user/share/lib does not exist This error message we will get when we submit any oozie job with below property set to true in job.properties file

and if you […]

Exhausted available authentication methods

This entry was posted in Error Repository on by

In this post we will discuss about common problem in installing cloudera manager 5.3.2 on Ubuntu 14.04 machine and solution for the root cause. Problem: When installing Cloudera Manager 5.3.2 on Ubuntu 14.04.2 Machine getting below error messages after giving root privileges on ssh configuration page. Exhausted available authentication methods or Installation failed on all hosts. Installation failed. Failed to authenticate In this case, ssh to root might be a […]

Cannot create an instance of InputFormat

This entry was posted in Error Repository and tagged cannot create an instance of inputformat class org.apache.hadoop.mapred.textinputformat hive cannot create an instance of inputformat class on by

Error Scenario: java.io.IOException: Cannot create an instance of InputFormat class We will get this error message when we try to execute simple hadoop fs commands or running any hive queries. Below is the complete error message.

Root Cause: This error message will be received when there are any spaces or spelling mistakes in any of the site.xml configuration file. Suppose in below core-site.xml file value for io.compression.codecs has spaces which […]

JDOFatalDataStoreException: Communications link failure

This entry was posted in Error Repository and tagged Error in metadata: javax.jdo.JDOFatalDataStoreException Error in starting hive metastore Hive MetaStore not starting Starting Hive Metastore Server error on by

Error Scenario: javax.jdo.JDOFatalDataStoreException: Communications link failure We get this error message when starting hive metastore service with MySQL as local metastore.  This error message is received when MySQL database is not created properly or it is not updated with the tables correctly while configuring it as hive metastore database. While configuring hive metastore with MySQL, some times, metastore database (say hive_metastore_db) will not be created properly with all of the […]

NoClassDefFoundError or ClassNotFoundException

This entry was posted in Error Repository and tagged ClassNotFoundException Container exited with a non-zero exit code 1 Error starting MRAppMaster fatal org.apache.hadoop.mapreduce.v2.app.mrappmaster error starting mrappmaster hadoop mapreduce container exited with a non zero exit code 1 hadoop mapreduce Error starting MRAppMaster NoClassDefFoundError on by

In this post we will discuss about most frequent error messages NoClassDefFoundError or ClassNotFoundException in hadoop mapreduce jobs execution and possible solutions for them. Error Scenario: java.lang.ClassNotFoundException       or java.lang.NoClassDefFoundError Error starting MRAppMaster   &  Container exited with a non-zero exit code 1 When we encounter a situation of NoClassDefFoundError or ClassNotFoundException even though we have required jar files added to our build path in eclipse, then we have […]

Hadoop Integration – Avro Errors 1

This entry was posted in Avro Error Repository and tagged Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addDeprecations hadoop Bad connection to FS. command aborted. exception: org.apache.hadoop.ipc.RemoteException:Server IPC version 9 cannot communicate with client version 4 Server IPC version 7 cannot communicate with client version 4 Server IPC version 9 cannot communicate with client version 4 on by

In this post we will discuss about some of the errors or exceptions that can occur when there is mismatch in the integration of Avro and Hadoop distributions. When we do not use the correct version of avro release then we will run into so many errors or exceptions. In this post, we will consider the version compatibility for Hadoop-2.3.0 release. As it is Hadoop2  we need to use the […]

Expected timestamp in the Flume event headers, but it was null 1

This entry was posted in Error Repository and tagged flume hdfs.useLocalTimeStamp flume TimestampInterceptor HDFS sink with format escape sequences on by

Error Scenario: Expected timestamp in the Flume event headers, but it was null – NullPointerException This error message is received in ~/logs/flume.log file when starting a flume agent with HDFS sink with format escape sequences.(%Y, %M, %D, %H, %M, %S).

If a sink expects a header but does not find it, events will become stuck in the channel and Flume will log NullPointer and EventDelivery exceptions. Root Cause: We receive […]

channel has been removed due to an error

This entry was posted in Error Repository and tagged flume jdbc channel create schema jdbc-channel has been removed due to an error during configuration java.lang.NullPointerException org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.schemaExists on by

Error Scenario: channel has been removed due to an error during configuration. This error message is received in ~/logs/flume.log file when starting a flume agent with missing or wrong configuration with JDBC channel setup.

Root Cause: DerbySchemaHandler.schemaExists error message is received when a flume agent’s sink is not able to receive the events from this JDBC channel and for some reason agent process stopped and when we try to […]

Mapreduce job stuck at map 0% reduce 0% 1

This entry was posted in Error Repository and tagged hadoop ssh could not resolve hostname hadoop ssh name or service not known Problem with hadoop start-dfs.sh on by

Error Scenario: Mapreduce job stuck at map 0% reduce 0%. : Could not resolve hostname

Mapreduce jobs are not running. And the below error message is coming when starting HDFS daemons.

Resolution: These above error messages are received when there is some issue with ssh communication. So, in these situations, we need to remove openssh-client and openssh-server and re-install them again.

Once the ssh is re-installed, we will […]

org.apache.flume.EventDeliveryException: Failed to open file

This entry was posted in Error Repository and tagged Flume agent unable to deliver event Flume errors org.apache.flume.eventdeliveryexception failed to send events on by

Error Scenario: org.apache.flume.EventDeliveryException: Failed to open file  We will receive this error message when a flume agent is started and it is trying to start FILE_ROLL sink with a given target sink directory. Below are the error messages sequence from ~/logs/flume.log file.

Root Cause: When a flume agent is started with FILE_ROLL sink type and without creating <sink.directory> folder prior to starting agent or  Flume user doesn’t have enough […]

UnsatisfiedLinkError

This entry was posted in Error Repository Snappy and tagged ERROR snappy.SnappyCompressor: failed to load SnappyCompressor Exception in thread "main" java.lang.UnsatisfiedLinkError java.lang.unsatisfiedlinkerror snappy java.lang.UnsatisfiedLinkError: Cannot load libsnappy.so org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z on by

Error Scenario: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z We will receive this error message if we try to read snappy files without having snappy compression/decompression codec installed properly. 

Or

Resolution: We should install snappy compression/decompression codec to resolve this error message and its corresponding .so files should be available (copied from /usr/local/bin) in $HADOOP_HOME/lib/native directory location. For Snappy Installation instructions please read through this post.

Unable to open a test connection to the given database 2

This entry was posted in Error Repository and tagged Communications link failure communications link failure due to underlying exception communications link failure hive metastore communications link failure in mysql communications link failure mysql hive metastore on by

Error Scenario 1: In this post, we will discuss about a few varieties of error message Unable to open a test connection to the given database and their resolutions. Unable to open a test connection to the given database / Communications link failure Below is the error message for this communications link failure:

Root Cause: We will get the above error message, if we specify incorrect hostname: port values in property javax.jdo.option.ConnectionURL in […]

Access denied for user ‘APP’@’localhost’ (using password: YES)

This entry was posted in Error Repository and tagged Access denied for user 'APP'@'localhost Access denied for user 'root'@'localhost' (using password: YES) caused by java.sql.sqlexception unable to open a test connection to the given database hive local metastore Unable to open a test connection to the given database on by

Error Scenario: Unable to open a test connection to the given database  or Access denied for user ‘APP’@’localhost’ (using password: YES) This error message (Access denied for user ‘APP’@’localhost’) will be received while trying to connect hive service through MySQL local metastore.

Root Cause: When we don’t provide username (javax.jdo.option.ConnectionUserName) and password (javax.jdo.option.ConnectionPassword) configurations in hive-site.xml for accessing MySQL local metastore database we will receive this error message. By default these two […]

datastore driver was not found in the CLASSPATH 1

This entry was posted in Error Repository and tagged HiveMetastore: Unable to Make JDBC connection Trouble using mysql metastore Trouble with Hive mysql installation on by

Error Scenario: The specified datastore driver (“com.mysql.jdbc.Driver”) was not found in the CLASSPATH We usually get this error message while configuring Hive Local Metastore through MySQL database.

Root Cause: If the JDBC driver JAR file for MySQL (Connector/J) is not on Hive’s classpath, ( in Hive’s lib directory – $HIVE_HOME/lib), we will get this error message. Resolution: We have to download this MySQL Connector/J jar file from MySQL downloads pages . For […]

Datanode denied communication with namenode

This entry was posted in Error Repository and tagged cannot Add New Datanodes to HDFS datanode denied communication with namenode because hostname cannot be resolved datanode denied communication with namenode datanoderegistration DisallowedDatanodeException: Datanode denied communication with namenode error: Datanode denied communication with namenode How to solve a DisallowedDatanodeException on by

Error Scenario: Datanode denied communication with namenode: We usually get this error message while adding a data node to an existing HDFS cluster or at the time of cluster setup. Below is the snapshot of this error message from log file in data node.

Root Cause: This error message is receive if the hadoop user (for example hduser) on namenode machine doesn’t have full (write) permissions to the directory location […]

Review Comments
default image

I have attended Siva’s Spark and Scala training. He is good in presentation skills and explaining technical concepts easily to everyone in the group. He is having excellent real time experience and provided enough use cases to understand each concepts. Duration of the course and time management is awesome. Happy that I found a right person on time to learn Spark. Thanks Siva!!!

Dharmeswaran ETL / Hadoop Developer Spark Nov 2016 September 21, 2017

.