读书人

HBase Split 进程

发布时间: 2012-07-27 11:03:00 作者: rapoo

HBase Split 过程
看了下hbase split的代码 记录一下学习心得

private void split(final HRegion parent, final byte [] midKey)throws IOException {    final long startTime = System.currentTimeMillis();    SplitTransaction st = new SplitTransaction(parent, midKey);    // If prepare does not return true, for some reason -- logged inside in    // the prepare call -- we are not ready to split just now.  Just return.    if (!st.prepare()) return;    try {      st.execute(this.server, this.server);


SplitTransation表示处理split是一个类似事务性质的行为,接下来我们会经常和它打交道
初始化了一个path:tablename/parent/split/

st.prepare():构造了两个HRegionInfo:hri_a和hri_b,a是top,其startkey为parent的startkey,endkey是midkey;hri_b是bottom,startkey为midkey,endkey是parent的endkey。

然后st.execute(this.server, this.server);

1.创建splitdir:tablename/parent/split/,在状态机中加入CREATE_SPLIT_DIR
createSplitDir(this.parent.getFilesystem(), this.splitdir);    this.journal.add(JournalEntry.CREATE_SPLIT_DIR);

2.close parentregion,并返回所有的storefile,状态机加入CLOSED_PARENT_REGION
close的过程会刷一次磁盘memstore的数据写入磁盘;
    List<StoreFile> hstoreFilesToSplit;    hstoreFilesToSplit = this.parent.close(false);


3.从rs的onlineregion列表中移除parent region,状态机添OFFLINED_PARENT
     services.removeFromOnlineRegions(this.parent.getRegionInfo().getEncodedName());     this.journal.add(JournalEntry.OFFLINED_PARENT);


4.splitStoreFiles(this.splitdir, hstoreFilesToSplit);
多线程处理所有的storefiles.创建文件/table/parent/split/hri_a
(hri_b)/family/storefilename. parent,在该文件中写入boolean top和midkey
    splitStoreFiles(this.splitdir, hstoreFilesToSplit);    this.journal.add(JournalEntry.STARTED_REGION_A_CREATION);

5.状态机添STARTED_REGION_A_CREATION,创建子Region A。将上面的hri_a下的文件move到table/hri_a下即变为/hbase/hri_a/family/storefilename.parent

6.同理状态机添STARTED_REGION_B_CREATION而后创建子Region B
    this.journal.add(JournalEntry.STARTED_REGION_A_CREATION);    HRegion a = createDaughterRegion(this.hri_a,this.parent.flushRequester, this.parent.rsServices);    this.journal.add(JournalEntry.STARTED_REGION_B_CREATION);    HRegion b = createDaughterRegion(this.hri_b, this.parent.flushRequester, this.parent.rsServices);


7.meta表中下线parent;将parent在meta表中offline和split标志置为true,parent
Region添加两列SPLITA和SPLITB,值为HRegioninfo
8.状态机添PONR
 try{      if (!testing) {        MetaEditor.offlineParentInMeta(server.getCatalogTracker(),            this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());      }    } catch(IOException e){      throw e;    } finally{      this.journal.add(JournalEntry.PONR);    }


9.开两个线程open REGION A和B。

1>调用openRegion函数进行initilize,主要步骤如下

a)向hdfs上写入.regionInfo文件以便meta挂掉以便恢复
b)初始化其下的HStore,主要是LoadStoreFiles函数:
对于该store函数会构造storefile对象,从hdfs上获取路径和文件,每个文件一个
storefile对象,对每个storefile对象会读取文件上的内容创建一个
HalfStoreFileReader读对象来操作该region的父region上的相应的文件,及该
region上目前存储的是引用文件,其指向的是其父region上的相应的文件,对该
region的所有读或写都将关联到父region上
2>将子Region添加到rs的online region列表上,并添加到meta表上
 DaughterOpener aOpener = new DaughterOpener(server, services, a);    DaughterOpener bOpener = new DaughterOpener(server, services, b);    aOpener.start();    bOpener.start();    try {      aOpener.join();      bOpener.join();    } catch (InterruptedException e) {      Thread.currentThread().interrupt();      throw new IOException("Interrupted " + e.getMessage());    }


整个过程如上所述,如果碰到异常的话,会进行rollback检查状态机上的状态并依次进行rollback

读书人网 >其他数据库

热点推荐