hadoop学习日记四 hive环境搭建
搭建hive环境
?
从官网http://hive.apache.org/下载hive-0.8.1-bin.tar.gz，下载完成后copy?hive-0.8.1-bin.tar.gz到服务器目录下
?
执行tar -zxvf?hive-0.8.1-bin.tar.gz 解压hive。
?
将hive加入环境变量，主要是方便hive命令的执行，命令如下
?
?
exprot HIVE_HOME=/home/hive-0.8.1exprot PATH=$HIVE_HOME:$PATH
?
将conf下面的template文件copy一份，便于个性化配置
?
?
cp hive-default.xml.template hive-default.xml cp hive-default.xml.template hive-site.xmlcp hive-env.sh.template hive-env.shcp hive-log4j.properties.template hive-log4j.properties
?
在hive-env.sh中添加HADOOP_HOME的安装目录地址
在hive-log4j.properties中将log4j.appender.EventCounter的值修改为org.apache.hadoop.log.metrics.EventCounter，这样就不会报WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.的警告了。
?
以上一切完成后，执行
?
?
root@wenbo00:/home/hive-0.8.1-bin/conf# hive?
进入hive命令行模式，然后执行
?
?
hive> show tables;OKTime taken: 6.909 seconds?
出现以上结果表示安装成功。
?
执行建表语句
?
?
hive> create table invites(foo INT, bar STRING) partitioned by (ds STRING);OKTime taken: 5.918 seconds
?
查看结果
?
hive> show tables;OKinvitesTime taken: 0.246 seconds?
?
加载数据
?
?
hive> LOAD DATA LOCAL INPATH '/home/wenbo/kv2.txt' OVERWRITE INTO TABLE invites PARTITION (ds='2012-03-16 17:56:25'); hive> LOAD DATA LOCAL INPATH '/home/wenbo/kv3.txt' OVERWRITE INTO TABLE invites PARTITION (ds='2012-03-16 17:57:25');
?
查询数据
?
hive> select count(*) from invites;Total MapReduce jobs = 1Launching Job 1 out of 1Number of reduce tasks determined at compile time: 1In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number>In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number>In order to set a constant number of reducers: set mapred.reduce.tasks=<number>Starting Job = job_201203160053_0004, Tracking URL = http://wenbo00:50030/jobdetails.jsp?jobid=job_201203160053_0004Kill Command = /home/hadoop-1.0.1/libexec/../bin/hadoop job -Dmapred.job.tracker=wenbo00:9001 -kill job_201203160053_0004Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 12012-03-16 03:03:25,739 Stage-1 map = 0%, reduce = 0%2012-03-16 03:03:37,819 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec2012-03-16 03:03:38,825 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec2012-03-16 03:03:39,837 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec2012-03-16 03:03:40,852 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec2012-03-16 03:03:41,870 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec2012-03-16 03:03:42,879 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec2012-03-16 03:03:43,885 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec2012-03-16 03:03:44,898 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec2012-03-16 03:03:45,907 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec2012-03-16 03:03:46,914 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec2012-03-16 03:03:47,926 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec2012-03-16 03:03:48,933 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec2012-03-16 03:03:49,949 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 9.74 sec2012-03-16 03:03:50,958 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 9.74 sec2012-03-16 03:03:51,964 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 9.74 sec2012-03-16 03:03:52,978 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 9.74 sec2012-03-16 03:03:53,997 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 9.74 sec2012-03-16 03:03:55,016 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 9.74 sec2012-03-16 03:03:56,029 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 9.74 secMapReduce Total cumulative CPU time: 9 seconds 740 msecEnded Job = job_201203160053_0004MapReduce Jobs Launched:Job 0: Map: 1 Reduce: 1 Accumulative CPU: 9.74 sec HDFS Read: 6342 HDFS Write: 4 SUCESSTotal MapReduce CPU Time Spent: 9 seconds 740 msecOK525Time taken: 42.688 seconds
?
查询的时候利用mapreduce创建任务执行，可惜我这里的环境是三台虚拟机运行在一台windows7上面，无法发挥集群优势，才导致一个简单的查询就耗费了将近43秒。
?