HBase什么时候做minor major compact
我们都知道compact分为两类,一类叫Minor compact ,一类叫Major compact,
两者有什么区别呢?
两者的区别在于:Minor compact只是进行文件merge操作,而Major compact除了做文件Merge操作,还会将其中的delete项删除。
// TODO: Use better method for determining stamp of last major (HBASE-2990) long lowTimestamp = getLowestTimestamp(filesToCompact); long now = System.currentTimeMillis(); if (lowTimestamp > 0l && lowTimestamp < (now - mcTime)) { // Major compaction time has elapsed. if (filesToCompact.size() == 1) { // Single file StoreFile sf = filesToCompact.get(0); long oldest = (sf.getReader().timeRangeTracker == null) ? Long.MIN_VALUE : now - sf.getReader().timeRangeTracker.minimumTimestamp; if (sf.isMajorCompaction() && (this.ttl == HConstants.FOREVER || oldest < this.ttl)) { if (LOG.isDebugEnabled()) { LOG.debug("Skipping major compaction of " + this.storeNameStr + " because one (major) compacted file only and oldestTime " + oldest + "ms is < ttl=" + this.ttl); } } else if (this.ttl != HConstants.FOREVER && oldest > this.ttl) { LOG.debug("Major compaction triggered on store " + this.storeNameStr + ", because keyvalues outdated; time since last major compaction " + (now - lowTimestamp) + "ms"); result = true; } } else { if (LOG.isDebugEnabled()) { LOG.debug("Major compaction triggered on store " + this.storeNameStr + "; time since last major compaction " + (now - lowTimestamp) + "ms"); } result = true; } } return result; }
long getNextMajorCompactTime() { // default = 24hrs long ret = conf.getLong(HConstants.MAJOR_COMPACTION_PERIOD, 1000*60*60*24); if (family.getValue(HConstants.MAJOR_COMPACTION_PERIOD) != null) { String strCompactionTime = family.getValue(HConstants.MAJOR_COMPACTION_PERIOD); ret = (new Long(strCompactionTime)).longValue(); } if (ret > 0) { // default = 20% = +/- 4.8 hrs double jitterPct = conf.getFloat("hbase.hregion.majorcompaction.jitter", 0.20F); if (jitterPct > 0) { long jitter = Math.round(ret * jitterPct); // deterministic jitter avoids a major compaction storm on restart ImmutableList<StoreFile> snapshot = storefiles; if (snapshot != null && !snapshot.isEmpty()) { String seed = snapshot.get(0).getPath().getName(); double curRand = new Random(seed.hashCode()).nextDouble(); ret += jitter - Math.round(2L * jitter * curRand); } else { ret = 0; // no storefiles == no major compaction } } } return ret; }
2. 在compact时,当选中的文件数小于这个store所有文件数*ratio(hbase.hstore.compaction.ratio)时,进行Minor compact.
当
boolean isMajor = (filesToCompact.getFilesToCompact().size() == this.storefiles.size());时,也就是当选中的文件数等于该store数时,会进行Major compact.