HDFS Offline Edits Viewer Tool – oev 1


HDFS OEV Tool

Similar to Offline Image Viewer (oiv), Hadoop also provides viewer tool for edits log files since these are also not in human-readable format. This is called as HDFS Offline Edits Viewer (oev) tool.

This tool doesn’t require HDFS to be running and it can run in offline entirely.

oev supports both binary (native binary format that Hadoop uses internally) and XML input formats for parsing and provides output in binary or XML formats.

Advantages:
  • The output formats of the tool can be converted back to original edits files.
  • Partial recovery of edits file is possible in case, the edits file is corrupted at any time. if there is some problem with HDFS cluster and edits file is corrupted, then edit file can be converted to human-readable XML format and delete/correct the bad record from XML file. Finally the corrected XML file can be converted back to original edits file so that at least a part of edit file can be recovered.
Basic Syntax:

hdfs oevAs shown in the above screen, binary input format is assumed by default unless until input file contains .xml or .XML file extension.

If the output file already exists, then it will be overridden silently.

Example oev usage is described below with XML output format.

hdfs oev cmd

XML Output created is as shown below:

hdfs oev xml op

Other useful command options available are:
  • -p <arg> : Processor options. Currently supported options are:

binary (native binary format that Hadoop uses),

xml (default, XML format),

stats (prints statistics about edits file).

  • -f,–fix-txids : This option can be used to renumber the transaction IDs in the input edits file, so that there are no gaps or invalid transaction IDs.
  • -r,–recover : This option can be used to skip corrupt parts of the edit log when reading binary edit logs.

The statistics of edits file can be created with stats processor option into text files with the command as shown below.

hdfs oev statsstats output looks as shown below:

hdfs oev stats output

Note: One of the common problem with corrupted edits file is that missing of closing record (record with opCode -1). This can be recognized by viewing the edits log file with oev tool in XML format and if there is no closing record in the XML file then it can be added after last correct record.

Anything after the record with opCode -1 is ignored.

Example of a closing record (with opCode -1):


Profile photo of Siva

About Siva

Senior Hadoop developer with 4 years of experience in designing and architecture solutions for the Big Data domain and has been involved with several complex engagements. Technical strengths include Hadoop, YARN, Mapreduce, Hive, Sqoop, Flume, Pig, HBase, Phoenix, Oozie, Falcon, Kafka, Storm, Spark, MySQL and Java.


Leave a comment

Your email address will not be published. Required fields are marked *

One thought on “HDFS Offline Edits Viewer Tool – oev


Review Comments
default gravatar

I am a plsql developer. Intrested to move into bigdata.

Neetika Singh ITA

.