Apache Oozie Installation on Ubuntu-14.04

In this post we will discuss about the Apache Oozie Installation on Ubuntu machine and we will run some sample mapreduce jobs on oozie scheduler.

Table of Contents

Apache Oozie Installation on Ubuntu
Run Examples in Oozie
- - Extract Examples Tarball file
  - Edit job.properties files
- Copy these examples into HDFS

Apache Oozie Installation on Ubuntu

We are building the oozie distribution tar ball by downloading the source code from apache and building the tar ball with the help of Maven.

Prerequisite

If we plan to install Oozie-4.0.1 or prior version Jdk-1.6 is required on our machine. If our jdk version on Ubuntu is greater than or equal to jdk-1.7 then we need to make some changes in pom.xml file.
But if we are going to install oozie-4.1.0 or later versions then Jdk-1.7 on Ubuntu will not cause any issues.
Hadoop-2 installed on our machine.
Mapreduce Job History Server should be configured and started successfully and remaining hadoop & yarn daemons should be running fine.

Procedure

Create oozie installation directory under preferred location, usually in /usr/lib/oozie and provide full permissions and download the oozie source code into that folder.

Downloading Oozie Sourc and building distribution
Shell
user@ubuntu-1:~$ cd /usr/lib
user@ubuntu-1:/usr/lib$ sudo mkdir oozie
user@ubuntu-1:/usr/lib$ sudo chmod -R 777 oozie
user@ubuntu-1:/usr/lib$ cd oozie
user@ubuntu-1:/usr/lib/oozie$ curl http://apache.cs.utah.edu/oozie/4.1.0/oozie-4.1.0.tar.gz | tar -zx
user@ubuntu-1:/usr/lib/oozie$ cd oozie-4.1.0/
user@ubuntu-1:/usr/lib/oozie/oozie-4.1.0$ bin/mkdistro.sh -DskipTests -Dhadoopversion=2.6.0

				1
2
3
4
5
6
7
8

						 
user@ubuntu-1:~$ cd /usr/lib
user@ubuntu-1:/usr/lib$ sudo mkdir oozie
user@ubuntu-1:/usr/lib$ sudo chmod -R 777 oozie
user@ubuntu-1:/usr/lib$ cd oozie
user@ubuntu-1:/usr/lib/oozie$ curl http://apache.cs.utah.edu/oozie/4.1.0/oozie-4.1.0.tar.gz | tar -zx
user@ubuntu-1:/usr/lib/oozie$ cd oozie-4.1.0/
user@ubuntu-1:/usr/lib/oozie/oozie-4.1.0$ bin/mkdistro.sh -DskipTests -Dhadoopversion=2.6.0

					

Once the build is successful, copy the binary distribution, from /usr/lib/oozie/oozie-4.1.0/distro/target/oozie-4.1.0-distro/oozie-4.1.0 to our convenient location so that no need to traverse longer directory structure to access oozie, suppose into /usr/lib/oozie/oozie-4.1/.

Shell
user@ubuntu-1:/usr/lib/oozie/oozie-4.1.0$ sudo mkdir ../oozie-bin/
user@ubuntu-1:/usr/lib/oozie/oozie-4.1.0$ cp -R distro/target/oozie-4.1.0-distro/oozie-4.1.0/* ../oozie-bin/

				1
2
3

						 
user@ubuntu-1:/usr/lib/oozie/oozie-4.1.0$ sudo mkdir ../oozie-bin/
user@ubuntu-1:/usr/lib/oozie/oozie-4.1.0$ cp -R distro/target/oozie-4.1.0-distro/oozie-4.1.0/* ../oozie-bin/

					

Add the bin directory of oozie-bin into .bashrc file as shown below. $ gedit ~/.bashrc

.bashrc file changes
Shell
### OOZIE -4.1.0 ####
export OOZIE_VERSION=4.1.0
export OOZIE_HOME="/usr/lib/oozie/oozie-bin/"
export PATH="$OOZIE_HOME/bin:$PATH"

				1
2
3
4
5

						 
### OOZIE -4.1.0 ####
export OOZIE_VERSION=4.1.0
export OOZIE_HOME="/usr/lib/oozie/oozie-bin/"
export PATH="$OOZIE_HOME/bin:$PATH"

					

Oozie Web Console:

For enabling Web Console for Oozie we need ext-*.*.zip library present in Oozie distribution. By default this will not be available with Oozie distribution, we have to download it separately from extjs site.
Now lets create library directory under Oozie binary distribution and add required jars to it and later lets download required extjs zip files into it.

Shell
user@ubuntu-1:/usr/lib/oozie/oozie-4.1.0$ cd ../oozie-bin/
user@ubuntu-1:/usr/lib/oozie/oozie-bin$ mkdir libext
user@ubuntu-1:/usr/lib/oozie/oozie-bin$ cp /usr/lib/oozie/oozie-4.1.0/hadooplibs/target/oozie-4.1.0-hadooplibs.tar.gz .
user@ubuntu-1:/usr/lib/oozie/oozie-bin$ tar xzvf oozie-4.1.0-hadooplibs.tar.gz
user@ubuntu-1:/usr/lib/oozie/oozie-bin$ cp oozie-4.1.0/hadooplibs/hadooplib-2.3.0.oozie-4.1.0/* libext/
user@ubuntu-1:/usr/lib/oozie/oozie-bin$ cd libext/
user@ubuntu-1:/usr/lib/oozie/oozie-bin/libext$ wget http://extjs.com/deploy/ext-2.2.zip

				1
2
3
4
5
6
7
8

						 
user@ubuntu-1:/usr/lib/oozie/oozie-4.1.0$ cd ../oozie-bin/
user@ubuntu-1:/usr/lib/oozie/oozie-bin$ mkdir libext
user@ubuntu-1:/usr/lib/oozie/oozie-bin$ cp /usr/lib/oozie/oozie-4.1.0/hadooplibs/target/oozie-4.1.0-hadooplibs.tar.gz .
user@ubuntu-1:/usr/lib/oozie/oozie-bin$ tar xzvf oozie-4.1.0-hadooplibs.tar.gz
user@ubuntu-1:/usr/lib/oozie/oozie-bin$ cp oozie-4.1.0/hadooplibs/hadooplib-2.3.0.oozie-4.1.0/* libext/
user@ubuntu-1:/usr/lib/oozie/oozie-bin$ cd libext/
user@ubuntu-1:/usr/lib/oozie/oozie-bin/libext$ wget http://extjs.com/deploy/ext-2.2.zip

					

Now we are almost ready to start Oozie web console but we need to create user and groups specific to Oozie as shown below

Add these two lines to “core-site.xml” under “$HADOOP_CONF_DIR” or $HADOOP_HOME/etc/hadoop. Here, USERNAME should be replaced with appropriate value, in our case it is user only (user@ubuntu-1:)