HDFS OEV Tool
Similar to Offline Image Viewer (oiv), Hadoop also provides viewer tool for edits log files since these are also not in human-readable format. This is called as HDFS Offline Edits Viewer (oev) tool.
oev supports both binary (native binary format that Hadoop uses internally) and XML input formats for parsing and provides output in binary or XML formats.
- The output formats of the tool can be converted back to original edits files.
- Partial recovery of edits file is possible in case, the edits file is corrupted at any time. if there is some problem with HDFS cluster and edits file is corrupted, then edit file can be converted to human-readable XML format and delete/correct the bad record from XML file. Finally the corrected XML file can be converted back to original edits file so that at least a part of edit file can be recovered.
If the output file already exists, then it will be overridden silently.
Example oev usage is described below with XML output format.
XML Output created is as shown below:
Other useful command options available are:
- -p <arg> : Processor options. Currently supported options are:
binary (native binary format that Hadoop uses),
xml (default, XML format),
stats (prints statistics about edits file).
- -f,–fix-txids : This option can be used to renumber the transaction IDs in the input edits file, so that there are no gaps or invalid transaction IDs.
- -r,–recover : This option can be used to skip corrupt parts of the edit log when reading binary edit logs.
The statistics of edits file can be created with stats processor option into text files with the command as shown below.
Note: One of the common problem with corrupted edits file is that missing of closing record (record with opCode -1). This can be recognized by viewing the edits log file with oev tool in XML format and if there is no closing record in the XML file then it can be added after last correct record.
Anything after the record with opCode -1 is ignored.
Example of a closing record (with opCode -1):