HBase Integration with Hive 21


In this post, we will discuss about the setup needed for HBase Integration with Hive and we will test this integration with the creation of some test hbase tables from hive shell and populate the contents of it from another hive table and finally verify these contents in hbase table.

Reasons to use Hive on HBase is that a lot of data sitting in HBase due to its usage in a real-time environment, but never used for analysis as there are less connectivity tools to HBase directly.

We will use storage handler mechanism to create hbase tables via hive. HBaseStorageHandler allows Hive DDL for managing table definitions in both Hive metastore and HBase’s catalog simultaneously and consistently.

Setup for HBase Integration with Hive:

For setting up of HBase Integration with Hive, we mainly require a few jar files to be present in $HIVE_HOME/lib or $HBASE_HOME/lib directory. The required jar files are:

Here $HBASE_HOME/lib directory will contain many hbase-*.jar files and below are the list for Hadoop 2 API.

We need to add the paths for above jar files to value of hive.aux.jars.path property in hive-site.xml configuration file.

Verify HBase Integration with Hive:

Lets create a new hbase table via hive shell. To Test the hbase table creations we need Hadoop, Yarn and Hbase daemons to be running.

Below is a sample hbase table creation DDL statements. In this we are creating hbase_table_emp table in Hive and emp table in HBase. This table will contain 3 columns in Hive, key int, name string and role string. These are mapped to two columns name and role belonging to cf1 column family. Here “:key” is specified at the beginning of “hbase.columns.mapping” property which automatically maps to first column (id int) in Hive table.

HBase is a special case here, it has a unique row key map with :key but not all the columns in the table need to be mapped.

Hbase Table Creation

Lets verify this table emp in HBase shell and view its metadata.

Hbase table

We can not directly load data into hbase table “emp” with load data inpath hive command. We have to copy data into it from another Hive table. Lets create another test hive table with the same schema as hbase_table_emp and we will insert records into it with hive load data input command.

Hive table creation

Lets copy contents into hbase_table_emp table from testemp and verify its contents.

Hive Table Insert

Lets see the contents of emp table from Hbase shell.

Hbase Table Scan

So we have successfully integrated Hbase with Hive and Created & populated new HBase tables from Hive shell.

Note:

If our table size is very big, in order to save the storage space on HDFS, we can delete the testemp table from Hive, after inserting its records into hbase_table_emp table instead of maintaining two copies of same table data.

Mapping Existing HBase Tables to Hive:

Similar to creating new HBase tables, we can also map HBase existing tables to Hive. To give Hive access to an existing HBase table with multiple columns and families, we need to use CREATE EXTERNAL TABLE. But, hbase.columns.mapping is required and it will be validated against the existing HBase table’s column families, whereas hbase.table.name is optional.

For testing this, we will create ‘user’ table in HBase as shown below and map this to Hive table.

HBase Table Operations

Lets create corresponding Hive table for the above ‘user’ Hbase table. Below is the DDL for creation of external table ‘hbase_table_user’.

Verify the contents of hbase_table_user table.

Hive Table Mapping with HBase

So, we have successfully mapped HBase table with Hive External Table.


Profile photo of Siva

About Siva

Senior Hadoop developer with 4 years of experience in designing and architecture solutions for the Big Data domain and has been involved with several complex engagements. Technical strengths include Hadoop, YARN, Mapreduce, Hive, Sqoop, Flume, Pig, HBase, Phoenix, Oozie, Falcon, Kafka, Storm, Spark, MySQL and Java.


Leave a comment

Your email address will not be published. Required fields are marked *

21 thoughts on “HBase Integration with Hive

  • Bharathi

    Dear sir,

    Please help me with my problem. when am executing the “insert overwrite table” hive is looking for the jar files in “hdfs” and says file not found and stops executing it. how to solve this. am a new bee in hive. Please Help me.

     

    Thanks

    Bharathi

    • Profile photo of Siva
      Siva Post author

      Are u trying to perform this query after Hbase integration with Hive or in just Hive ? Can you post your complete error message details in Hadoop Discussion Forum, So, that I can understand the exact the problem and can suggest a possible solution. You may need to register for posting a topic in Hadoop discussion forum.

  • neuser

    I have hive 0.13.1 hbase 0.98.8 and hadoop 2.5.1
    I am trying to integrate hive with hbase. steps Followed are as follows:

    1. $ hive –auxpath /home/hduser/hive/lib/hive-hbase-handler-0.13.1.jar,/home/hduser/hive/lib/hbase-common-0.98.8-hadoop2.jar,/home/hduser/hive/lib/zookeeper-3.4.6.jar,/home/hduser/hive/lib/guava-11.0.2.jar -hiveconf hbase.master=masternode:60000

    2. CREATE TABLE sahi23(rowkey int, x int, y string)
    STORED BY ‘org.apache.hadoop.hive.hbase.HBaseStorageHandler’
    WITH SERDEPROPERTIES (“hbase.columns.mapping” = “:key,cf:age, cf:name”)
    TBLPROPERTIES(“hbase.table.name” = “sahi23”);

    This Command works fine. I am able to create table in hive and hbase

    3. INSERT OVERWRITE TABLE sahi23 SELECT * FROM hello where no=1;

    But I am facing issue here:
    Total jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there’s no reduce operator
    Starting Job = job_1420005248631_0011, Tracking URL = http://example.co.in:8088/proxy/application_1420005248631_0011/
    Kill Command = /usr/local/hadoop//bin/hadoop job -kill job_1420005248631_0011
    Hadoop job information for Stage-0: number of mappers: 1; number of reducers: 0
    2014-12-31 15:27:34,524 Stage-0 map = 0%, reduce = 0%
    2014-12-31 15:28:09,206 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
    2014-12-31 15:29:09,264 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
    2014-12-31 15:30:10,185 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
    2014-12-31 15:31:11,069 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
    2014-12-31 15:32:11,861 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
    2014-12-31 15:33:12,628 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
    2014-12-31 15:34:13,396 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
    2014-12-31 15:35:14,126 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
    2014-12-31 15:36:14,837 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
    2014-12-31 15:37:15,546 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
    2014-12-31 15:38:16,216 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
    2014-12-31 15:38:33,714 Stage-0 map = 0%, reduce = 0%
    2014-12-31 15:38:44,972 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.27 sec
    2014-12-31 15:39:45,855 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.27 sec
    2014-12-31 15:40:46,532 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.27 sec
    2014-12-31 15:41:47,245 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.27 sec
    2014-12-31 15:42:47,906 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.27 sec
    2014-12-31 15:43:48,561 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.27 sec
    2014-12-31 15:44:49,370 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.27 sec
    2014-12-31 15:45:50,130 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.27 sec
    2014-12-31 15:46:50,763 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.27 sec
    2014-12-31 15:47:51,521 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.27 sec
    2014-12-31 15:48:52,101 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.27 sec
    2014-12-31 15:49:04,438 Stage-0 map = 0%, reduce = 0%
    2014-12-31 15:49:15,702 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
    2014-12-31 15:50:16,321 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
    2014-12-31 15:51:16,897 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
    2014-12-31 15:52:17,468 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
    2014-12-31 15:53:18,077 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
    2014-12-31 15:54:18,808 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
    2014-12-31 15:55:19,430 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
    2014-12-31 15:56:20,011 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
    2014-12-31 15:57:20,623 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
    2014-12-31 15:58:21,173 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
    2014-12-31 15:59:21,761 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
    2014-12-31 15:59:34,084 Stage-0 map = 0%, reduce = 0%
    2014-12-31 16:00:00,733 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 1.35 sec

    MapReduce Total cumulative CPU time: 1 seconds 350 msec
    Ended Job = job_1420005248631_0011 with errors
    Error during job, obtaining debugging information…
    Examining task ID: task_1420005248631_0011_m_000000 (and more) from job job_1420005248631_0011

    Task with the most failures(4):
    —–
    Task ID:
    task_1420005248631_0011_m_000000

    URL:
    http://masternode:8088/taskdetails.jsp?jobid=job_1420005248631_0011&tipid=task_1420005248631_0011_m_000000
    —–
    Diagnostic Messages for this Task:
    AttemptID:attempt_1420005248631_0011_m_000000_3 Timed out after 600 secs

    FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
    MapReduce Jobs Launched:
    Job 0: Map: 1 Cumulative CPU: 1.35 sec HDFS Read: 0 HDFS Write: 0 FAIL
    Total MapReduce CPU Time Spent: 1 seconds 350 msec

    I am able to create table in both hive and Hbase but not able to insert data into that.

      • newuser

        This is contents of hive.log
        2015-01-01 08:49:25,233 INFO [main]: SessionState (SessionState.java:printInfo(536)) –
        Logging initialized using configuration in jar:file:/home/hduser/hive/lib/hive-common-0.13.1.jar!/hive-log4j.properties
        2015-01-01 08:49:26,012 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(493)) – 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
        2015-01-01 08:49:26,058 INFO [main]: metastore.ObjectStore (ObjectStore.java:initialize(246)) – ObjectStore, initialize called
        2015-01-01 08:49:29,296 INFO [main]: metastore.ObjectStore (ObjectStore.java:getPMF(315)) – Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes=”Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order”
        2015-01-01 08:49:29,424 INFO [main]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:(110)) – MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: “@” (64), after : “”.
        2015-01-01 08:49:31,885 INFO [main]: metastore.ObjectStore (ObjectStore.java:setConf(229)) – Initialized ObjectStore
        2015-01-01 08:49:32,256 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:createDefaultRoles(551)) – Added admin role in metastore
        2015-01-01 08:49:32,269 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:createDefaultRoles(560)) – Added public role in metastore
        2015-01-01 08:49:32,436 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:addAdminUsers(588)) – No user is added in admin role, since config is empty
        2015-01-01 08:49:33,334 INFO [main]: session.SessionState (SessionState.java:start(360)) – No Tez session required at this point. hive.execution.engine=mr.
        2015-01-01 08:49:33,735 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(623)) – 0: get_all_databases
        2015-01-01 08:49:33,743 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) – ugi=hduser ip=unknown-ip-addr cmd=get_all_databases
        2015-01-01 08:49:33,779 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(623)) – 0: get_functions: db=default pat=*
        2015-01-01 08:49:33,779 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) – ugi=hduser ip=unknown-ip-addr cmd=get_functions: db=default pat=*
        2015-01-01 08:50:00,583 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) –
        2015-01-01 08:50:00,584 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) –
        2015-01-01 08:50:00,585 INFO [main]: ql.Driver (Driver.java:checkConcurrency(158)) – Concurrency mode is disabled, not creating a lock manager
        2015-01-01 08:50:00,587 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) –
        2015-01-01 08:50:00,606 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) –
        2015-01-01 08:50:00,608 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(185)) – Parsing command: CREATE TABLE san(rowkey int, x int, y string)
        STORED BY ‘org.apache.hadoop.hive.hbase.HBaseStorageHandler’
        WITH SERDEPROPERTIES (“hbase.columns.mapping” = “:key,cf:age, cf:name”)
        TBLPROPERTIES(“hbase.table.name” = “san”)
        2015-01-01 08:50:00,870 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(206)) – Parse Completed
        2015-01-01 08:50:00,876 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) –
        2015-01-01 08:50:00,876 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) –
        2015-01-01 08:50:00,931 INFO [main]: parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(9172)) – Starting Semantic Analysis
        2015-01-01 08:50:00,932 INFO [main]: parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeCreateTable(9801)) – Creating table san position=13
        2015-01-01 08:50:00,936 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(623)) – 0: get_database: default
        2015-01-01 08:50:00,936 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) – ugi=hduser ip=unknown-ip-addr cmd=get_database: default
        2015-01-01 08:50:00,947 INFO [main]: ql.Driver (Driver.java:compile(431)) – Semantic Analysis Completed
        2015-01-01 08:50:00,948 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) –
        2015-01-01 08:50:00,954 INFO [main]: ql.Driver (Driver.java:getSchema(238)) – Returning Hive schema: Schema(fieldSchemas:null, properties:null)
        2015-01-01 08:50:00,954 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) –

        • Profile photo of Siva
          Siva Post author

          It seems like the log is incomplete, as I don’t see ERROR/WARN/DEBUG messages in the above can you try to find out on any WARN or ERROR messages on hive.log file. As per the above log the table creation under default database is successful. I doubt whether you are running your HBase daemons are not.

          Also check the HBase version once again whether you are using hadoop2 version or not. Are you able to see the table description under HBase?

  • user

    hi ! could you help me please I get an error while executing this :

    hive> insert overwrite table hbase_table_emp select * from testemp;

    Error:

    Query ID = hduser_20150129032929_1fce8beb-f4a0-42d1-99e8-a7c799cf72b0
    Total jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there’s no reduce operator
    java.lang.NoClassDefFoundError: org/cliffc/high_scale_lib/Counter
    at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureJobConf(HBaseStorageHandler.java:470)
    at org.apache.hadoop.hive.ql.plan.PlanUtils.configureJobConf(PlanUtils.java:856)
    at org.apache.hadoop.hive.ql.plan.MapWork.configureJobConf(MapWork.java:544)
    at org.apache.hadoop.hive.ql.plan.MapredWork.configureJobConf(MapredWork.java:68)
    at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:370)
    at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1604)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1364)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1177)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:994)
    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:783)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
    Caused by: java.lang.ClassNotFoundException: org.cliffc.high_scale_lib.Counter
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    … 24 more
    FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org/cliffc/high_scale_lib/Counter

    • Profile photo of Siva
      Siva Post author

      Hi Yosser,

      Please download the jar (high-scale-lib-*.jar) file containing the class org.cliffc.high_scale_lib.Counter and add the jar to HIVE_CLASSPATH or $HIVE_HOME/lib and $HBASE_HOME/lib directories and then retry the integration procedure.

      Make sure that there are no version mismatches with Hive and HBase while integrating.

  • sanjeeb

    Hi ,

    I am getting below error after loading data into hbase using org.apache.hadoop.hbase.mapreduce.ImportTsv.

    hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=HBASE_ROW_KEY,basic_info:empname,basic_info:age employee /user/spanda20/hbase_sample/20150102_emp.dat

     

    => Hbase::Table – employee
    hbase(main):008:0> list
    TABLE
    customers
    employee
    transactions_hbase
    3 row(s) in 0.0230 seconds

    => [“customers”, “employee”, “transactions_hbase”]
    hbase(main):009:0> scan employee
    NameError: undefined local variable or method `employee’ for #<Object:0xb52dc3>

    hbase(main):010:0>

    • Profile photo of Siva
      Siva Post author

      Hi Sanjeeb,

      It is just a syntax error. You have to put your table names in single quotes like hbase(main)>scan ’employee’ instead of just hbase(main)> scan employee

  • sanjeeb

    Hello Siva, Thanks a lot. I got that after i sent.. I have one project where I have to write data in Hbase from webserver. May I know how to write using Java or thrift client.?

    I really need this..

  • Anjali

    I am facing some count mismatch issue while testing HIve-Hbase integration…

    select count(*) from table_name gives count of all rows in hbase irrespective of which column family is associated with hive table created.

    is there any issue with serde ?

  • Vikram Chandel

    Hi Siva

    I want to set a ttl of one day in my hive hbase table is there any way to do that.
    We basically want to expire those records which have not been refreshed in 24 hours.

  • Sudarshan kumar

    Hi ,

    I have added all the jar as explained by you like

    <property>
    <name>hive.aux.jars.path</name>
    <value>
    file:///usr/lib/hive/lib/zookeeper.jar,
    file:///usr/lib/hive/lib/hive-hbase-handler-1.1.0-cdh5.7.0.jar,
    file:///usr/lib/hive/lib/guava-14.0.1.jar,
    file:///usr/lib/hbase/lib/hbase-client-1.2.0-cdh5.8.0.jar,
    file:///usr/lib/hbase/lib/hbase-common-1.2.0-cdh5.8.0.jar,
    file:///usr/lib/hbase/lib/hbase-protocol-1.2.0-cdh5.8.0.jar,
    file:///usr/lib/hbase/lib/hbase-server-1.2.0-cdh5.8.0.jar,
    file:///usr/lib/hbase/lib/hbase-rest-1.2.0-cdh5.8.0.jar,
    file:///usr/lib/hbase/lib/hbase-thrift-1.2.0-cdh5.8.0.jar
    </value>
    </property>

     

    but getting following exception

    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. /usr/lib/hive/lib/hbase-0.92.0.jar (No such file or directory)

     

    I am using HBase Version 1.2.0-cdh5.8.0  and Hadoop Version 2.6.0-cdh5.7.0

     

  • Nabayoti

    HiSiva,
    Thanks for the above blog..It’s really amazing.I have an issue.After loading the jars into the hive shell.while creating hive table I get the error from hive shell:::

    Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat

    My hive query is:
    CREATE EXTERNAL TABLE hive_table (Row_key string, First_name string,last_name string,age int,City string,Team string) STORED BY ‘org.apache.hadoop.hive.hbase.HBaseStorageHandler’ WITH SERDEPROPERTIES(“hbase.columns.mapping”=”:key,name:first_name,name:last_name,details:age,details:city,details:team”) TBLPROPERTIES (“hbase.table.name” = “hbase_table”);

    plz suggest.

  • Rajesh

    Hi Shiva,

    I am getting following error while running query,Please help me out.I tried with all combination of hadoop-hbase-hive version.

    hive> insert overwrite table hbase_table_emp33 select * from testemp33;
    Total jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there’s no reduce operator
    java.lang.IllegalArgumentException: Must specify table name
    at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:193)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.hive.ql.io.HivePassThroughOutputFormat.createActualOF(HivePassThroughOutputFormat.java:75)
    at org.apache.hadoop.hive.ql.io.HivePassThroughOutputFormat.checkOutputSpecs(HivePassThroughOutputFormat.java:83)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.checkOutputSpecs(FileSinkOperator.java:951)
    at org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl.checkOutputSpecs(HiveOutputFormatImpl.java:67)
    at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
    at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420)
    at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1503)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1270)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
    Job Submission failed with exception ‘java.lang.IllegalArgumentException(Must specify table name)’
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask


Review Comments
default image

I am a plsql developer. Intrested to move into bigdata.

Neetika Singh ITA Hadoop in Dec/2016 December 22, 2016

.