Grokbase Groups HBase dev May 2009
FAQ
Read the logs in batches during log splitting to avoid OOME
-----------------------------------------------------------

Key: HBASE-1430
URL: https://issues.apache.org/jira/browse/HBASE-1430
Project: Hadoop HBase
Issue Type: Improvement
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Priority: Blocker
Fix For: 0.20.0, 0.19.3


During log splitting, in the worst case we can read 64 logs of 64MB which will surely bust any heap. Instead we should read the logs in batches of 5-10 that we write right away.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Jean-Daniel Cryans (JIRA) at May 19, 2009 at 4:49 pm
    [ https://issues.apache.org/jira/browse/HBASE-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710789#action_12710789 ]

    Jean-Daniel Cryans edited comment on HBASE-1430 at 5/19/09 9:48 AM:
    --------------------------------------------------------------------

    Comment from JK in HBASE-1008:
    {quote}
    I'm not a big fan of having to read all the logs into memory.

    My suggestion would be for each unique region in the HLog(s), create a blocking queue and a thread that will
    dequeue entries and write them directly to the log file. Then you have one thread doing the reading and multiple
    threads writing, and the memory footprint is reduced significantly.

    Make sense?
    {quote}

    was (Author: jdcryans):
    Comment from JK in HBASE-1008:

    Read the logs in batches during log splitting to avoid OOME
    -----------------------------------------------------------

    Key: HBASE-1430
    URL: https://issues.apache.org/jira/browse/HBASE-1430
    Project: Hadoop HBase
    Issue Type: Improvement
    Reporter: Jean-Daniel Cryans
    Assignee: Jean-Daniel Cryans
    Priority: Blocker
    Fix For: 0.20.0, 0.19.3


    During log splitting, in the worst case we can read 64 logs of 64MB which will surely bust any heap. Instead we should read the logs in batches of 5-10 that we write right away.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Jean-Daniel Cryans (JIRA) at May 19, 2009 at 4:49 pm
    [ https://issues.apache.org/jira/browse/HBASE-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710789#action_12710789 ]

    Jean-Daniel Cryans commented on HBASE-1430:
    -------------------------------------------

    Comment from JK in HBASE-1008:

    Read the logs in batches during log splitting to avoid OOME
    -----------------------------------------------------------

    Key: HBASE-1430
    URL: https://issues.apache.org/jira/browse/HBASE-1430
    Project: Hadoop HBase
    Issue Type: Improvement
    Reporter: Jean-Daniel Cryans
    Assignee: Jean-Daniel Cryans
    Priority: Blocker
    Fix For: 0.20.0, 0.19.3


    During log splitting, in the worst case we can read 64 logs of 64MB which will surely bust any heap. Instead we should read the logs in batches of 5-10 that we write right away.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Jean-Daniel Cryans (JIRA) at May 19, 2009 at 9:59 pm
    [ https://issues.apache.org/jira/browse/HBASE-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jean-Daniel Cryans updated HBASE-1430:
    --------------------------------------

    Attachment: hbase-1430-trunk-v1.patch

    This patch makes it that we read, by default, in batches of 10 logs that we then write right away. Was tested with 12 logs to split.
    Read the logs in batches during log splitting to avoid OOME
    -----------------------------------------------------------

    Key: HBASE-1430
    URL: https://issues.apache.org/jira/browse/HBASE-1430
    Project: Hadoop HBase
    Issue Type: Improvement
    Reporter: Jean-Daniel Cryans
    Assignee: Jean-Daniel Cryans
    Priority: Blocker
    Fix For: 0.20.0, 0.19.3

    Attachments: hbase-1430-trunk-v1.patch


    During log splitting, in the worst case we can read 64 logs of 64MB which will surely bust any heap. Instead we should read the logs in batches of 5-10 that we write right away.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at May 19, 2009 at 11:58 pm
    [ https://issues.apache.org/jira/browse/HBASE-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710947#action_12710947 ]

    stack commented on HBASE-1430:
    ------------------------------

    I tested it on TRUNK. Worked. I applied it to TRUNK. Want to make one for the branch j-d?
    Read the logs in batches during log splitting to avoid OOME
    -----------------------------------------------------------

    Key: HBASE-1430
    URL: https://issues.apache.org/jira/browse/HBASE-1430
    Project: Hadoop HBase
    Issue Type: Improvement
    Reporter: Jean-Daniel Cryans
    Assignee: Jean-Daniel Cryans
    Priority: Blocker
    Fix For: 0.20.0, 0.19.3

    Attachments: hbase-1430-trunk-v1.patch


    During log splitting, in the worst case we can read 64 logs of 64MB which will surely bust any heap. Instead we should read the logs in batches of 5-10 that we write right away.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at May 19, 2009 at 11:58 pm
    [ https://issues.apache.org/jira/browse/HBASE-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710949#action_12710949 ]

    stack commented on HBASE-1430:
    ------------------------------

    Maybe I can have a go at it this evening... would like to get 0.19.3 RC out soon.
    Read the logs in batches during log splitting to avoid OOME
    -----------------------------------------------------------

    Key: HBASE-1430
    URL: https://issues.apache.org/jira/browse/HBASE-1430
    Project: Hadoop HBase
    Issue Type: Improvement
    Reporter: Jean-Daniel Cryans
    Assignee: Jean-Daniel Cryans
    Priority: Blocker
    Fix For: 0.20.0, 0.19.3

    Attachments: hbase-1430-trunk-v1.patch


    During log splitting, in the worst case we can read 64 logs of 64MB which will surely bust any heap. Instead we should read the logs in batches of 5-10 that we write right away.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at May 20, 2009 at 2:49 am
    [ https://issues.apache.org/jira/browse/HBASE-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710976#action_12710976 ]

    stack commented on HBASE-1430:
    ------------------------------

    Answering j-d question up on #hbase, I tested a few times with from 2 to 6 or so log files.. never > 10.

    Read the logs in batches during log splitting to avoid OOME
    -----------------------------------------------------------

    Key: HBASE-1430
    URL: https://issues.apache.org/jira/browse/HBASE-1430
    Project: Hadoop HBase
    Issue Type: Improvement
    Reporter: Jean-Daniel Cryans
    Assignee: Jean-Daniel Cryans
    Priority: Blocker
    Fix For: 0.20.0, 0.19.3

    Attachments: hbase-1430-trunk-v1.patch


    During log splitting, in the worst case we can read 64 logs of 64MB which will surely bust any heap. Instead we should read the logs in batches of 5-10 that we write right away.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Jean-Daniel Cryans (JIRA) at May 20, 2009 at 5:07 pm
    [ https://issues.apache.org/jira/browse/HBASE-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jean-Daniel Cryans resolved HBASE-1430.
    ---------------------------------------

    Resolution: Fixed
    Hadoop Flags: [Reviewed]

    Committed to branch.
    Read the logs in batches during log splitting to avoid OOME
    -----------------------------------------------------------

    Key: HBASE-1430
    URL: https://issues.apache.org/jira/browse/HBASE-1430
    Project: Hadoop HBase
    Issue Type: Improvement
    Reporter: Jean-Daniel Cryans
    Assignee: Jean-Daniel Cryans
    Priority: Blocker
    Fix For: 0.20.0, 0.19.3

    Attachments: hbase-1430-trunk-v1.patch


    During log splitting, in the worst case we can read 64 logs of 64MB which will surely bust any heap. Instead we should read the logs in batches of 5-10 that we write right away.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshbase, hadoop
postedMay 16, '09 at 2:43p
activeMay 20, '09 at 5:07p
posts8
users1
websitehbase.apache.org

1 user in discussion

Jean-Daniel Cryans (JIRA): 8 posts

People

Translate

site design / logo © 2022 Grokbase