HDFS常用代码片段备忘

FileSystem fs;

DistributedFileSystem dfs;

?

得到FileSystem: ?? fs = new Path("/").getFileSystem(conf);

RPC调用Namenode方法：??? fs.getClient().namenode.XXXXX();

得到某个文件corrupt的数据块：

????? LocatedBlocks locatedBlocks =

?????????????????? fs.getClient().namenode.getBlockLocations(uriPath, 0, stat.getLen());
????? for (LocatedBlock b: locatedBlocks.getLocatedBlocks()) {
??????? if (b.isCorrupt() ||
??????????? (b.getLocations().length == 0 && b.getBlockSize() > 0)) {
??????????? // the block is corrupt;
??????? }
????? }

?

????? final BlockLocation[] fileBlocks =
?????? ? ? ? ? ?? dfs.getFileBlockLocations(fileStatus, 0, fileLength);

????? for (BlockLocation fileBlock: fileBlocks) {
????????? if (fileBlock.isCorrupt() ||
??????????? (fileBlock.getNames().length == 0 && fileBlock.getLength() > 0)) {

????????????? // the block is corrupt;
????????? }
????? }

?

Path->String:? String uri = path.toUri().getPath();

String->Path:? Path path = new Path(uri);

?

java.net.URI dfsUri = DistributedFileSystem.getUri();

System.out.println(dfsUri)? ------>? hdfs://namenodehostname:9000

?

DistributedFileSystem.makeQualified(new Path("/foo")));?? --------->? hdfs://namenodehostname:9000/foo

????????????????????????????????????? ? ? ? ? ? ? ? ? ? ? ?? ? ? ?? new Path(scheme+":"+"//"+authority + pathUri.getPath();

?

?

Namenode上记录了那些LOG：

static void turnOffNameNodeLogging() {
??? // change log level to ERROR: NameNode.LOG & NameNode.stateChangeLog
??? ((Log4JLogger)NameNode.LOG).getLogger().setLevel(Level.ERROR);
??? ((Log4JLogger)NameNode.stateChangeLog).getLogger().setLevel(Level.ERROR);
??? ((Log4JLogger)NetworkTopology.LOG).getLogger().setLevel(Level.ERROR);
??? ((Log4JLogger)FSNamesystem.LOG).getLogger().setLevel(Level.ERROR);
??? ((Log4JLogger)FSNamesystem.auditLog).getLogger().setLevel(Level.ERROR);
??? ((Log4JLogger)LeaseManager.LOG).getLogger().setLevel(Level.ERROR);
? }

?

?

切换HDFS的数据块位置选择策略：conf的dfs.block.replicator.classname

?conf.set("dfs.block.replicator.classname", "org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicyRaid");

?

?

Path p = new Path("/raid"); LOG.info(p) --->? /raid

String xorPrefix = p.toUri().getPath();? LOG.info(xorPrefix);? ----->/raid

?

Path p = new Path("hdfs://0.0.0.0:3000" + "/raid");? --->hdfs://0.0.0.0:3000/raid

String xorPrefix = p.toUri().getPath();? LOG.info(xorPrefix);? ----->/raid

?