读书人

单机测试NN时遇到的有关问题

发布时间: 2012-08-11 20:50:31 作者: rapoo

单机测试NN时遇到的问题

在Hadoop的测试框架上写了一个小测试,用于比对NN使用不同策略分配数据块位置的速度:

??? Configuration conf = new Configuration();
??? FileSystem.setDefaultUri(conf, "hdfs://localhost:" + 0);
??? conf.set("dfs.http.address", "0.0.0.0:0");

然后用该conf去NameNode.createNameNode()

? ->new Namenode()

??? ->new FSNamesystem()

????? ->this.replicator = BlockPlacementPolicy.getInstance(conf,? this,? clusterMap);//这里我用了Raid的Policy

??????? ->BlockPlacementPolicyRaid.initialize()

????????? ->为了得到this.xorPrefix调用RaidNode.xorDestinationPath(conf)

??????????? ->FileSystem.get(p.toUri(), conf)

????????????? ->CACHE.get(uri, conf)

??????????????? ->createFileSystem(uri, conf)

????????????????? ->FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);得到DistributedFileSystem

??????????????????? ->DistributedFileSystem.initialize()

????????????????????? ->new DFSClient(namenode, conf, statistics)

???????????????????????? ->createRPCNamenode(nameNodeAddr, conf, ugi);

OK这时问题来了,在NN的建立过程中建立到NN的RPC,所以程序程序会停住不动。。。这时用jstack就可以查看得详细的调用过程

"main" prio=10 tid=0x0875ac00 nid=0xd70 in Object.wait() [0xb6d0d000]
?? java.lang.Thread.State: WAITING (on object monitor)
??? at java.lang.Object.wait(Native Method)
??? - waiting on <0x506cdcc8> (a org.apache.hadoop.ipc.Client$Call)
??? at java.lang.Object.wait(Object.java:485)
??? at org.apache.hadoop.ipc.Client.call(Client.java:701)
??? - locked <0x506cdcc8> (a org.apache.hadoop.ipc.Client$Call)
??? at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
??? at $Proxy4.getProtocolVersion(Unknown Source)
??? at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)
??? at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:103)
??? at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:176)
??? at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:78)
??? at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1443)
??? at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:56)
??? at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1456)
??? - locked <0x50960460> (a org.apache.hadoop.fs.FileSystem$Cache)
??? at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:218)
??? at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:120)
??? at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:207)
??? at org.apache.hadoop.raid.RaidNode.xorDestinationPath(RaidNode.java:1072)
??? at org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicyRaid.initialize(BlockPlacementPolicyRaid.java:96)
??? at org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicy.getInstance(BlockPlacementPolicy.java:193)
??? at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setConfigurationParameters(FSNamesystem.java:446)
??? at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:325)
??? at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:309)
??? at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:173)
??? at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:256)
??? at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:998)
??? at org.apache.hadoop.hdfs.NNThroughputBenchmark.<init>(NNThroughputBenchmark.java:112)
??? at org.apache.hadoop.hdfs.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1250)
??? at org.apache.hadoop.hdfs.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1360)
??? at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
??? at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
??? at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
??? at java.lang.reflect.Method.invoke(Method.java:597)
??? at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
??? at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
??? at org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:90)
??? at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
??? at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
??? at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
??? at java.lang.reflect.Method.invoke(Method.java:597)
??? at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
??? at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
??? at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
??? at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
??? at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)

?

解决:由于测试不需要用到xorPrefix,rsPrefix,所以直接把xorPrefix=/raid,rsPrefix=/raidrs。

读书人网 >开源软件

热点推荐