[ https://issues.apache.org/jira/browse/HBASE-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337686#comment-15337686 ]

Yu Li commented on HBASE-16032:

Yes, this is really something I neglected, thanks for pointing this out [~enis]. Will update the patch soon, and will wait for your +1 before commit
Possible memory leak in StoreScanner

Key: HBASE-16032
URL: https://issues.apache.org/jira/browse/HBASE-16032
Project: HBase
Issue Type: Bug
Affects Versions: 1.2.1, 1.1.5, 0.98.20
Reporter: Yu Li
Assignee: Yu Li
Fix For: 2.0.0, 1.1.6, 1.3.1, 0.98.21, 1.2.3

Attachments: HBASE-16032.patch, HBASE-16032_v2.patch, HBASE-16032_v3.patch

We observed frequent fullGC of RS in our production environment, and after analyzing the heapdump, we found large memory occupancy by HStore#changedReaderObservers, the map is surprisingly containing 7500w objects...
After some debugging, I located some possible memory leak in StoreScanner constructor:
public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final NavigableSet<byte[]> columns,
long readPt)
throws IOException {
this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
if (columns != null && scan.isRaw()) {
throw new DoNotRetryIOException("Cannot specify any column for a raw scan");
matcher = new ScanQueryMatcher(scan, scanInfo, columns,
oldestUnexpiredTS, now, store.getCoprocessorHost());
// Pass columns to try to filter out unnecessary StoreFiles.
List<KeyValueScanner> scanners = getScannersNoCompaction();
seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
&& lazySeekEnabledGlobally, parallelSeekEnabled);
resetKVHeap(scanners, store.getComparator());
If there's any Exception thrown after {{this.store.addChangedReaderObserver(this)}}, the returned scanner might be null and there's no chance to remove the scanner from changedReaderObservers, like in {{HRegion#get}}
RegionScanner scanner = null;
try {
scanner = getScanner(scan);
} finally {
if (scanner != null)
What's more, all exception thrown in the {{HRegion#getScanner}} path will cause scanner==null then memory leak, so we also need to handle this part.

This message was sent by Atlassian JIRA

Search Discussions

Discussion Posts


Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 34 of 36 | next ›
Discussion Overview
groupissues @
categorieshbase, hadoop
postedJun 15, '16 at 2:51p
activeJun 18, '16 at 11:54a

1 user in discussion

Hadoop QA (JIRA): 36 posts



site design / logo © 2021 Grokbase