Brief Notes on Unix Shell Scripting Concepts


This post provides a very brief notes on Unix Shell Scripting. As this topic is very well described in many text books,we are not going much deep into the details of each point. This post is for quick review/revision/reference of common Unix commands or Unix Shell Scripting.

Unix Shell Scripting

Kernel

The kernel is the heart of the UNIX system. It provides utilities with a means of accessing a machine’s hardware. It also handles the scheduling and execution of commands.

Note: When the computer is booted, the kernel is loaded from disk into memory. The kernel remains in memory until the machine is turned off. Utilities, on the other hand, are stored on disk and loaded into memory only when they are executed.

Shell

shell is an interface to the UNIX system. It collects input from user and executes programs based on that input. Once program finishes executing, shell displays that program’s output.

The different Bourne-type shells follow:

  • Bourne shell ( sh)
  • Korn shell ( ksh)
  • Bourne Again shell ( bash)
  • POSIX shell ( sh)

The #!/bin/sh must be the first line of a shell script in order for sh to be used to run the script. If this appears on any other line, it is treated as a comment and ignored by all shells.

PATH

The PATH specifies the locations in which the shell will look for commands. Usually it is set as follows:PATH=/bin:/usr/bin

Each of the individual entries separated by the colon character, :, are directories.

Compound Command

A compound command consists of a list of simple and complex commands separated by the semicolon character ( ;). An example of a complex command is

Here hostname and date are simple commands and who am i is complex command.

Comments

In shell scripts, comments start with the # character. Everything between the # and end of the line are considered part of the comment and are ignored by the shell.

Counting Words in a file

wc command can be used to get a count of the total number of lines, words, and characters contained in a file. The syntax of this command is

  • -l Counts the number of lines
  • -w Counts the number of words
  • -c Counts the number of characters
File Types in Unix

-rwxr-xr-x 1 siva users 2368 Jul 11 15:57 /home/siva/test.txt

Here first character is a hyphen (-). This indicates that the file is a regular file. For special files, the first character will be one of the letters given in below table.

Character File Type

  •  Regular file
  • l  Symbolic link
  • c  Character special
  • b  Block special
  • p  Named pipe
  • s  Socket
  • d  Directory file

A symbolic link is a special file that points to another file on the system.A symbolic link is similar to a shortcut or an alias.

Creating Symbolic Links

We can create symbolic links using the ln command with the -s option. The syntax is as follows:

Here, source is either the absolute or relative path to the original version of the file, and destination is the name we want the link to have.

Changing File and Directory Permissions

We can change permissions of a file/directory with the chmod command. Chmod options are below

Here (user) options are:

Symbol Represents

  • u Owner
  • g Group
  • o Other
  • a All

Actions

Symbol Represents

  • + Adding permissions to the file
  • Removing permission from the file
  • = Explicitly set the file permissions

Permissions:

Symbol Represents

  • r Read
  • w Write
  • x Execute
  • s SUID or SGID

Octal Method:

By changing permissions with an octal expression, we can only explicitly set file permissions.

The values of the individual permissions are the following:

  • Read permission has a value of 4
  • Write permission has a value of 2
  • Execute permission has a value of 1
Background Process

The simplest way to start a process in background is to add an ampersand (&) at the end of the command.

For example:

Moving a Foreground Process to the Background

While a foreground process is running, the shell does not process any new commands. Before we can enter any commands, we have to suspend the foreground process to get a command prompt. The suspend key is Ctrl+Z.

When a foreground process is suspended, a command prompt enables us to enter more commands; the original process is still in memory but is not getting any CPU time. To resume the foreground process, we have two choices–background and foreground. The bg command enables us to resume the suspended

process in the background; the fg command returns it to the foreground.

Some useful unix tips:

COPY & PASTE (WITHIN A TERMINAL):

COPY: CONTROL + SHIFT + C
PASTE: CONTROL + SHIFT + V

SETTING UP A VARIABLE GLOBALLY:

/etc/profile (one time per session on logon)
/etc/bash.bashrc (every time you close and open a terminal)

COMMAND TO REFRESH PROFILE CHANGES:
./etc/profile

SUDO EDIT:
[cloudera@localhost ~]$ sudo gedit /etc/profile
[cloudera@localhost ~]$ sudo gedit /etc/bashrc

 COMMAND COMPLETION:
[cloudera@localhost ~]$ cd s + TAB key –> will take you into scripts directory
[cloudera@localhost scripts]$

 CLEAR THE SCREEN:
CONTROL + L

 CUSTOMIZE COMMAND PROMPT:
[cloudera@localhost ~]$ export PS1=’$ ‘
$
$

 TO SPAN A COMMAND INTO MULTIPLE LINES: (use \)

TO LIST ALL JAVAE PROGRAMS (demons) RUNNING:

 REPLICATION FACTOR

Note: here 3 is the replication factor for _SUCCESS and part-r00000

TO LIST FILES IN THE LOCAL FILE SYSTEM (using hadoop fs command):
[cloudera@localhost ~]$ hadoop fs -ls file:///

RUN A MAPREDUCE PROGRAM:
[cloudera@localhost ~]$ hadoop jar <jar_file_name> <class_name> <input_dir> <output_dir>

To find all files modified in the last 24 hours (last full day) in a particular specific directory and its sub-directories:

files created, modified or accessed in the last hour


Profile photo of Siva

About Siva

Senior Hadoop developer with 4 years of experience in designing and architecture solutions for the Big Data domain and has been involved with several complex engagements. Technical strengths include Hadoop, YARN, Mapreduce, Hive, Sqoop, Flume, Pig, HBase, Phoenix, Oozie, Falcon, Kafka, Storm, Spark, MySQL and Java.

Leave a comment

Your email address will not be published. Required fields are marked *


Review Comments
default image

I am a plsql developer. Intrested to move into bigdata.

Neetika Singh ITA Hadoop in Dec/2016 December 22, 2016

.