Hive环境搭建

发表: 2016-12-09 浏览: 5804

Hive

大数据系列课程之Hive环境搭建——https://edu.hellobi.com/course/93/play/lesson/1690

1. 下载解压

http://hive.apache.org/

视频中所采用的是Hive-2.1版本

2. 配置环境变量

在/etc/profile文件中加入HIVE_HOME变量，并在PATH中加入$HIVE_HOME/bin

3. 配置Hive

cp hive-env.sh.template hive-env.sh

加入HADOOP_HOME变量

拷贝hive-site配置文件

cp hive-default.xml.template hive-site.xml

3.1 单用户模式

即元数据使用Derby存储的模式

直接进入Hive，在命令行敲hive按回车，会出现以下错误信息：

Exception in thread "main"java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative pathin absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D

配置以下两个参数的路径解决此问题：

hive.exec.local.scratchdir

hive.downloaded.resources.dir

这两个参数是Hive的缓存目录，如果不配置，使用Hive时将报错，配置之后即单用户模式

3.2 多用户模式

即元数据使用非Derby，而使用其他的关系型数据库存储（例如MySQL、Oracle等）

Hive的元数据：表信息，表属性，分区，列等等信息，Owner

Hive的实际数据：在HDFS上

Hive-site.xml的配置

添加4个配置项即可

1. javax.jdo.option.ConnectionURL(jdbc:mysql://centos:3306/hive?createDatabaseIfNotExist=true)，指定的是Hive连接数据库的连接字符串；

2. javax.jdo.option.ConnectionDriver(com.mysql.jdbc.Driver)，指定的是MySQL驱动类的入口；

3. javax.jdo.option.ConnectionUserName(hive)，指定了数据库的用户名；

4. javax.jdo.option.ConnectionPassword(hive)，指定了数据库的密码；

可能遇到的ERROR2：

Caused by:org.datanucleus.exceptions.NucleusException: Attempt to invoke the"BONECP" plugin to create a ConnectionPool gave an error : Thespecified datastore driver ("com.mysql.jdbc.Driver") was not found inthe CLASSPATH. Please check your CLASSPATH specification, and the name of thedriver.

此问题是因为缺少数据库对应的驱动包造成的，将MySQL数据库的驱动包拷贝到$HIVE_HOME/lib/下

cp mysql-connector….jar $HIVE_HOME/lib/

可能遇到的ERROR3：

Caused by:org.apache.hadoop.hive.ql.metadata.HiveException:org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Hivemetastore database is not initialized. Please use schematool (e.g. ./schematool-initSchema -dbType ...) to create the schema. If needed, don't forget toinclude the option to auto-create the underlying database in your JDBCconnection string (e.g. ?createDatabaseIfNotExist=true for mysql))

此问题是因为没有初始化元数据库

schematool -initSchema –dbType mysql

可能遇到的ERROR4：

MySQL远程访问受限，需要授权

grant all privileges on *.* to hive@"%" identified by "hive" with grant option;

flush privileges;

3.3 远程模式

启动metastore server

Hive –service metastore &

10 个评论

haobiql

谢谢老师

ahhbgxy

老师我在安装hive时报这个错，是不是哪里没有配置好
[root@bigdata opt]# hive
which: no hbase in (/usr/java/default/bin:/opt/hadoop-2.7.2/bin:/opt/hive-2.1.1/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/java/default/bin:/opt/hadoop-2.7.2/bin:/opt/hive-2.1.1/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/opt/hive-2.1.1/lib/hive-common-2.1.1.jar!/hive-log4j2.properties Async: true
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:591)
at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:366)
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:310)
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:290)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:266)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:558)
... 9 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3367)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3406)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3386)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3640)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:236)
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:221)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1652)
... 23 more
Caused by: MetaException(message:Version information not found in metastore. )
at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7753)
at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7731)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
at com.sun.proxy.$Proxy21.verifySchema(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:565)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:626)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:416)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6490)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:238)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
... 28 more

MarsJ 回复 ahhbgxy

注意错误日志中的关键字：Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Metastore没有做好初始化。

双鱼回复 MarsJ

这个在哪里配置啊？如何配置？2.1.1才出现的问题。

MarsJ 回复双鱼

视频都有的。配置文件文章也提到了hive-site.xml，hive-env.sh

xl1358783261 回复双鱼

在启动hive前，初始化metastore，单用户模式用schematool -initSchema –dbType derby,多用户模式用schematool -initSchema –dbType mysql

Cinderella 回复 MarsJ

请问是报错第一句which: no hbase in 的关系么，我也遇到这个问题，Hadoop版本2.7.2 hive2.1.0，也遇到这个问题

〆゛Big Hua丨灬

[root@master conf]# hive
which: no hbase in (/usr/java/jdk1.7.0_67/bin:/hadoop/hadoop-2.6.5/bin:/apache-hive-2.1.1/bin:/usr/java/jdk1.7.0_67/bin:/hadoop/hadoop-2.6.5/bin:/usr/java/jdk1.7.0_67/bin:/hadoop/hadoop-2.6.5/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/bighua/zookeeper-3.4.6/bin:/sqoop/sqoop-1.4.6/bin:/home/bighua/.local/bin:/home/bighua/bin:/home/bighua/zookeeper-3.4.6/bin:/sqoop/sqoop-1.4.6/bin:/home/bighua/zookeeper-3.4.6/bin:/sqoop/sqoop-1.4.6/bin:/apache-hive-2.1.1/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/apache-hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/hadoop-2.6.5/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/apache-hive-2.1.1/lib/hive-common-2.1.1.jar!/hive-log4j2.properties Async: true
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:591)
at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:366)
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:310)
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:290)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:266)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:558)

娜样の星光

[root@hadoop-master conf]# which: no hbase in (/usr/java/default//bin:/opt/hive-2.3.0/bin:/opt/hadoop-2.7.4/bin:/usr/java/default//bin:/opt/hadoop-2.7.4/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive-2.3.0/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "main" java.lang.ClassCastException: java.base/jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to java.base/java.net.URLClassLoader
at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:394)
at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:370)
at org.apache.hadoop.hive.cli.CliSessionState.<init>(CliSessionState.java:60)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:708)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

ivan.yang

老师你好，我配置hive时，初始化元数据库遇到问题如下，但是我用hive是可以登录mysql的；
which: no hbase in (/usr/java/jdk1.8.0_131/bin:/usr/java/jdk1.8.0_131/jre/bin:/usr/hadoop/bin:/usr/java/jdk1.8.0_131/bin:/usr/java/jdk1.8.0_131/jre/bin:/usr/hadoop/bin:/home/hadoop/hive/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/hadoop/.local/bin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:mysql://192.168.1.101:3306/hive?createDatabaseIfNotExist=true&useSSL=false
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: hive
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version.
Underlying cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException : Access denied for user 'hive'@'master' to database 'hive'
SQL Error code: 1044
Use --verbose for detailed stacktrace.
*** schemaTool failed ***

要回复文章请先登录或注册

文章目录