 |
Jean-Daniel Cryans (JIRA) |
at May 19, 2009 at 4:49 pm
|
⇧ | 
| |
[
https://issues.apache.org/jira/browse/HBASE-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710789#action_12710789 ]
Jean-Daniel Cryans edited comment on HBASE-1430 at 5/19/09 9:48 AM:
--------------------------------------------------------------------
Comment from JK in HBASE-1008:
{quote}
I'm not a big fan of having to read all the logs into memory.
My suggestion would be for each unique region in the HLog(s), create a blocking queue and a thread that will
dequeue entries and write them directly to the log file. Then you have one thread doing the reading and multiple
threads writing, and the memory footprint is reduced significantly.
Make sense?
{quote}
was (Author: jdcryans):
Comment from JK in HBASE-1008:
Read the logs in batches during log splitting to avoid OOME
-----------------------------------------------------------
Key: HBASE-1430
URL:
https://issues.apache.org/jira/browse/HBASE-1430Project: Hadoop HBase
Issue Type: Improvement
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Priority: Blocker
Fix For: 0.20.0, 0.19.3
During log splitting, in the worst case we can read 64 logs of 64MB which will surely bust any heap. Instead we should read the logs in batches of 5-10 that we write right away.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
