windows配置hadoop-1.1.0的伪分布式环境
?
1. 安装cygwin
http://cygwin.com/install.html
?
2. cygwin中配置sshd
http://docs.oracle.com/cd/E24628_01/install.121/e22624/preinstall_req_cygwin_ssh.htm#CBHIAFGI
?
3. 伪分布式配置
hadoop-1.1.0/docs/single_node_setup.html
?
bin/hadoop namenode -formatbin/start-all.shbin/stop-all.shhttp://localhost:50030http://localhost:50070
?
Q 在真正运行的时刻会遇到几个问题:
1 设置的路径并非使用cygwin linux的路径。
?hadoop.tmp.dir在/tmp目录下面，理论上应该在C:\cygwin\tmp，但实际的路径确实C:\tmp
2 启动datanode和jobtracker，以及tasktacker时会有路径权限的问题
?
2012-11-25 13:53:05,031 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:Winseliu cause:java.io.IOException: Failed to set permissions of path: C:\cygwin\home\Winseliu\hadoop-1.1.0\logs\history to 07552012-11-25 13:53:05,032 FATAL org.apache.hadoop.mapred.JobTracker: java.io.IOException: Failed to set permissions of path: C:\cygwin\home\Winseliu\hadoop-1.1.0\logs\history to 0755 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:670) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189) at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:510)
?
2012-11-25 13:53:04,389 INFO org.apache.hadoop.mapred.TaskTracker: Good mapred local directories are: /cygwin/home/Winseliu/cloud/mapred/local2012-11-25 13:53:04,396 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because java.io.IOException: Failed to set permissions of path: \cygwin\home\Winseliu\cloud\mapred\local\taskTracker to 0755 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:670)
?
3 使用jps查不全真正执行的java进程，不知道那个进程启动或未启动
Winseliu@WINSE ~/hadoop-1.1.0$ jps6364 NameNode7168 JobTracker2692 JpsWinseliu@WINSE ~/hadoop-1.1.0$ ps aux | grep java 7880 1 5544 7028 ? 1001 13:20:40 /cygdrive/c/Java/jdk1.7.0_02/bin/java 5968 1 7500 4592 ? 1001 13:20:36 /cygdrive/c/Java/jdk1.7.0_02/bin/java 5784 1 484 6364 pty0 1001 13:20:31 /cygdrive/c/Java/jdk1.7.0_02/bin/java 6732 1 484 7168 pty0 1001 13:20:38 /cygdrive/c/Java/jdk1.7.0_02/bin/java 7976 1 5716 5628 ? 1001 13:20:34 /cygdrive/c/Java/jdk1.7.0_02/bin/java 4492 0 0 4492 pty0 1001 Jan 1 /cygdrive/c/Java/jdk1.7.0_02/bin/java
?
?
R 处理办法：
1 路径不同意，我们就设置自己的目录就可以了
<property> <name>hadoop.tmp.dir</name> <value>/cygwin/home/Winseliu/cloud</value> </property>
?
2 权限问题，直接修改FileUtils的checkReturnValue()方法，替换hadoop-core-1.1.0.jar中的FileUtils.class文件
private static void checkReturnValue(boolean rv, File p, FsPermission permission ) throws IOException { if (!rv) { // FIXMEtry {throw new IOException("Failed to set permissions of path: " + p+ " to " + String.format("%04o", permission.toShort()));} catch (Exception e) {e.printStackTrace();} } }?
3 查看已经启动的hadoop相关的进程
直接再执行一次start-all.sh，如果会提示让你先stop就说明该进程已经启动了。
Winseliu@WINSE ~$ cd hadoop-1.1.0/Winseliu@WINSE ~/hadoop-1.1.0$ bin/start-all.shstarting namenode, logging to /home/Winseliu/hadoop-1.1.0/libexec/../logs/hadoop-Winseliu-namenode-WINSE.outlocalhost: starting datanode, logging to /home/Winseliu/hadoop-1.1.0/libexec/../logs/hadoop-Winseliu-datanode-WINSE.outlocalhost: starting secondarynamenode, logging to /home/Winseliu/hadoop-1.1.0/libexec/../logs/hadoop-Winseliu-secondarynamenode-WINSE.outstarting jobtracker, logging to /home/Winseliu/hadoop-1.1.0/libexec/../logs/hadoop-Winseliu-jobtracker-WINSE.outlocalhost: starting tasktracker, logging to /home/Winseliu/hadoop-1.1.0/libexec/../logs/hadoop-Winseliu-tasktracker-WINSE.outWinseliu@WINSE ~/hadoop-1.1.0$ bin/start-all.shnamenode running as process 2648. Stop it first.localhost: datanode running as process 3512. Stop it first.localhost: secondarynamenode running as process 2468. Stop it first.jobtracker running as process 2388. Stop it first.localhost: tasktracker running as process 860. Stop it first.Winseliu@WINSE ~/hadoop-1.1.0$
?