Fake Multi Node Cluster Creation on Single Machine with hadoop-2.6.0


Hadoop Eco System Forums Hadoop Discussion Forum Fake Multi Node Cluster Creation on Single Machine with hadoop-2.6.0

This topic contains 0 replies, has 1 voice, and was last updated by  Siva 2 years, 7 months ago.

Viewing 1 post (of 1 total)
  • Author
    Posts
  • #3021

    Siva
    Keymaster

    Hi All,

    Below is the high level procedure for creation of 3 node hadoop cluster on a single Ubuntu machine.

    storage folder is for storing all the nodes’ files. This folder is just for convenience, as it’s easier to restart from scratch, just be deleting the storage folder content.

    Node1 will hosts:

    • HDFS namenode
    • HDFS datanode
    • Resource Manager
    • NodeManager

    So, the node1/etc/hadoop/core-site.xml file contains the location of the namenode. Following are core-site.xml, hdfs-site.xml, mapred-site.xml and yarn-site.xml files respectively.

    Node2 will hosts:

    • HDFS datanode
    • Yarn Nodemanager

    Below are its configuration properties that needs to be edited respectively for all the above *-site.xml files

    Node3 will hosts:

    • HDFS datanode
    • Yarn Nodemanager

    Similar to node2, node3 has its own below configuration properties that needs to be edited respectively for all the above *-site.xml files

    After making all the above changes, run the below commands from respective directories.

    Format the Namenode:

     

     

    We are now ready to start the namenode, datanode, resourcemanager and nodemanager on node1:

    We start a datanode and a nodemanager on node2:

    We start a datanode and a nodemanager on node3:

    We access to the HDFS web console (http://localhost:50070) to verify that the namenode is able to see the 3 live datanodes and resource manager at http://localhost:8088/cluster .

    we can also verify the services running on single machine.

Viewing 1 post (of 1 total)

The topic ‘Fake Multi Node Cluster Creation on Single Machine with hadoop-2.6.0’ is closed to new replies.

Review Comments
default image

I have attended Siva’s Spark and Scala training. He is good in presentation skills and explaining technical concepts easily to everyone in the group. He is having excellent real time experience and provided enough use cases to understand each concepts. Duration of the course and time management is awesome. Happy that I found a right person on time to learn Spark. Thanks Siva!!!

Dharmeswaran ETL / Hadoop Developer Spark Nov 2016 September 21, 2017

.