storage directory does not exist 1


Error Scenario:  

storage directory does not exist or is not accessible / Exception in namenode join

When we receive the error messages in name node log or data node logs similar to below then it belongs to this error scenario.

Root Cause:

This kind of error messages are received, when we are configuring hadoop without creating source directories for saving namenode or datanode meta data files in local file system.

These are the directories defined in <dfs.namenode.name.dir> & <dfs.datanode.data.dir> environment variables in hdfs-site.xml file.

For example in the below settings,

We should have already created below two directories

/usr/lib/hadoop/hadoop-2.3.0/hdfs/namenode   

/usr/lib/hadoop/hadoop-2.3.0/hdfs/datanode

though it is not mandatory to create name or data sub directories under these.

In our error scenario case,

We have directory level created only up to /usr/lib/hadoop/hadoop-2.3.0 but hdfs/namenode or hdfs/datanode directories are not created.

So, we have received error messages, stating that hdfs/namenode or hdfs/datanode storage directories doesn’t exist.

Solution:

To resolve this kind of error messages,
  1. We need to make sure that entire source directory level is created before starting namenode or datanode daemons.
  2. Also we need to give full (r+w+x) permissions on these storage directories to hadoop user account (say hduser for example).

About Siva

Senior Hadoop developer with 4 years of experience in designing and architecture solutions for the Big Data domain and has been involved with several complex engagements. Technical strengths include Hadoop, YARN, Mapreduce, Hive, Sqoop, Flume, Pig, HBase, Phoenix, Oozie, Falcon, Kafka, Storm, Spark, MySQL and Java.

Leave a comment

Your email address will not be published. Required fields are marked *


Review Comments
default image

I have attended Siva’s Spark and Scala training. He is good in presentation skills and explaining technical concepts easily to everyone in the group. He is having excellent real time experience and provided enough use cases to understand each concepts. Duration of the course and time management is awesome. Happy that I found a right person on time to learn Spark. Thanks Siva!!!

Dharmeswaran ETL / Hadoop Developer Spark Nov 2016 September 21, 2017

.