FAQ
[ https://issues.apache.org/jira/browse/HBASE-69?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bryan Duxbury updated HBASE-69:
-------------------------------

Component/s: regionserver
[hbase] Make cache flush triggering less simplistic
---------------------------------------------------

Key: HBASE-69
URL: https://issues.apache.org/jira/browse/HBASE-69
Project: Hadoop HBase
Issue Type: Improvement
Components: regionserver
Reporter: stack
Assignee: Jim Kellerman
Attachments: patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt


When flusher runs -- its triggered when the sum of all Stores in a Region > a configurable max size -- we flush all Stores though a Store memcache might have but a few bytes.
I would think Stores should only dump their memcache disk if they have some substance.
The problem becomes more acute, the more families you have in a Region.
Possible behaviors would be to dump the biggest Store only, or only those Stores > 50% of max memcache size. Behavior would vary dependent on the prompt that provoked the flush. Would also log why the flush is running: optional or > max size.
This issue comes out of HADOOP-2621.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Bryan Duxbury (JIRA) at Feb 6, 2008 at 12:37 am
    [ https://issues.apache.org/jira/browse/HBASE-69?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Bryan Duxbury updated HBASE-69:
    -------------------------------

    Fix Version/s: 0.2.0
    [hbase] Make cache flush triggering less simplistic
    ---------------------------------------------------

    Key: HBASE-69
    URL: https://issues.apache.org/jira/browse/HBASE-69
    Project: Hadoop HBase
    Issue Type: Improvement
    Components: regionserver
    Reporter: stack
    Assignee: Jim Kellerman
    Fix For: 0.2.0

    Attachments: patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt


    When flusher runs -- its triggered when the sum of all Stores in a Region > a configurable max size -- we flush all Stores though a Store memcache might have but a few bytes.
    I would think Stores should only dump their memcache disk if they have some substance.
    The problem becomes more acute, the more families you have in a Region.
    Possible behaviors would be to dump the biggest Store only, or only those Stores > 50% of max memcache size. Behavior would vary dependent on the prompt that provoked the flush. Would also log why the flush is running: optional or > max size.
    This issue comes out of HADOOP-2621.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Jim Kellerman (JIRA) at Feb 10, 2008 at 9:59 pm
    [ https://issues.apache.org/jira/browse/HBASE-69?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jim Kellerman updated HBASE-69:
    -------------------------------

    Attachment: patch.txt

    A new version of this patch that builds on, rather than ignores HBASE-138.

    What I'd like to see is:
    - a code review from stack
    - Billy Pearson, Bryan Duxbury and Stack run this patch through their bulk load tests to ensure that this patch is better than the current code.


    [hbase] Make cache flush triggering less simplistic
    ---------------------------------------------------

    Key: HBASE-69
    URL: https://issues.apache.org/jira/browse/HBASE-69
    Project: Hadoop HBase
    Issue Type: Improvement
    Components: regionserver
    Reporter: stack
    Assignee: Jim Kellerman
    Fix For: 0.2.0

    Attachments: patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt


    When flusher runs -- its triggered when the sum of all Stores in a Region > a configurable max size -- we flush all Stores though a Store memcache might have but a few bytes.
    I would think Stores should only dump their memcache disk if they have some substance.
    The problem becomes more acute, the more families you have in a Region.
    Possible behaviors would be to dump the biggest Store only, or only those Stores > 50% of max memcache size. Behavior would vary dependent on the prompt that provoked the flush. Would also log why the flush is running: optional or > max size.
    This issue comes out of HADOOP-2621.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Jim Kellerman (JIRA) at Feb 11, 2008 at 8:20 pm
    [ https://issues.apache.org/jira/browse/HBASE-69?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jim Kellerman updated HBASE-69:
    -------------------------------

    Attachment: patch.txt

    This patch should address the issue with only store being flushed.

    It contains fixes for TTMR and TTI

    It speeds up TestLogRolling by a factor of 3

    Cache sizes bubble up from HStore to HRegion, but aside from blocking updates, no other memory accounting is done.

    Please review and test.
    [hbase] Make cache flush triggering less simplistic
    ---------------------------------------------------

    Key: HBASE-69
    URL: https://issues.apache.org/jira/browse/HBASE-69
    Project: Hadoop HBase
    Issue Type: Improvement
    Components: regionserver
    Reporter: stack
    Assignee: Jim Kellerman
    Fix For: 0.2.0

    Attachments: patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt


    When flusher runs -- its triggered when the sum of all Stores in a Region > a configurable max size -- we flush all Stores though a Store memcache might have but a few bytes.
    I would think Stores should only dump their memcache disk if they have some substance.
    The problem becomes more acute, the more families you have in a Region.
    Possible behaviors would be to dump the biggest Store only, or only those Stores > 50% of max memcache size. Behavior would vary dependent on the prompt that provoked the flush. Would also log why the flush is running: optional or > max size.
    This issue comes out of HADOOP-2621.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Jim Kellerman (JIRA) at Feb 12, 2008 at 6:08 am
    [ https://issues.apache.org/jira/browse/HBASE-69?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jim Kellerman updated HBASE-69:
    -------------------------------

    Attachment: patch.txt

    - Update migration tool and change file system version number
    - FSUtils.checkVersion now returns the version string
    - Less simplistic update blocking mechanism is based on the number of stores and how many have requested cache flushes. Block if the number of stores that have requested flushes >= number of stores - 1, unless there is only one store. If only one store no blocking is done. A more sophisticated memory management system will be part of HBASE-70
    - HRegion now implements HStoreListener so it can intercept and count cache flush requests

    [hbase] Make cache flush triggering less simplistic
    ---------------------------------------------------

    Key: HBASE-69
    URL: https://issues.apache.org/jira/browse/HBASE-69
    Project: Hadoop HBase
    Issue Type: Improvement
    Components: regionserver
    Reporter: stack
    Assignee: Jim Kellerman
    Fix For: 0.2.0

    Attachments: patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt


    When flusher runs -- its triggered when the sum of all Stores in a Region > a configurable max size -- we flush all Stores though a Store memcache might have but a few bytes.
    I would think Stores should only dump their memcache disk if they have some substance.
    The problem becomes more acute, the more families you have in a Region.
    Possible behaviors would be to dump the biggest Store only, or only those Stores > 50% of max memcache size. Behavior would vary dependent on the prompt that provoked the flush. Would also log why the flush is running: optional or > max size.
    This issue comes out of HADOOP-2621.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Jim Kellerman (JIRA) at Feb 21, 2008 at 3:32 am
    [ https://issues.apache.org/jira/browse/HBASE-69?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jim Kellerman updated HBASE-69:
    -------------------------------

    Attachment: patch.txt

    Seems to actually work. Your milage may vary.
    [hbase] Make cache flush triggering less simplistic
    ---------------------------------------------------

    Key: HBASE-69
    URL: https://issues.apache.org/jira/browse/HBASE-69
    Project: Hadoop HBase
    Issue Type: Improvement
    Components: regionserver
    Reporter: stack
    Assignee: Jim Kellerman
    Fix For: 0.2.0

    Attachments: patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt


    When flusher runs -- its triggered when the sum of all Stores in a Region > a configurable max size -- we flush all Stores though a Store memcache might have but a few bytes.
    I would think Stores should only dump their memcache disk if they have some substance.
    The problem becomes more acute, the more families you have in a Region.
    Possible behaviors would be to dump the biggest Store only, or only those Stores > 50% of max memcache size. Behavior would vary dependent on the prompt that provoked the flush. Would also log why the flush is running: optional or > max size.
    This issue comes out of HADOOP-2621.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Jim Kellerman (JIRA) at Feb 22, 2008 at 5:12 pm
    [ https://issues.apache.org/jira/browse/HBASE-69?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jim Kellerman updated HBASE-69:
    -------------------------------

    Attachment: patch.txt

    Latest patch. OOMEs during PerformanceEvaluation sequential write test, which doesn't make sense since sequential write should have at most two regions open and cached ~128MB of data in memory. Since this patch cannot handle this simple case,I don't think it should be committed.

    We can salvage the changes to HMaster and Migrate since those are needed, but otherwise, I think this effort should be shut down and tossed.

    I have spent too much time on it and it does not bring enough to the table to be considered for inclusion.
    [hbase] Make cache flush triggering less simplistic
    ---------------------------------------------------

    Key: HBASE-69
    URL: https://issues.apache.org/jira/browse/HBASE-69
    Project: Hadoop HBase
    Issue Type: Improvement
    Components: regionserver
    Reporter: stack
    Assignee: Jim Kellerman
    Fix For: 0.2.0

    Attachments: patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt


    When flusher runs -- its triggered when the sum of all Stores in a Region > a configurable max size -- we flush all Stores though a Store memcache might have but a few bytes.
    I would think Stores should only dump their memcache disk if they have some substance.
    The problem becomes more acute, the more families you have in a Region.
    Possible behaviors would be to dump the biggest Store only, or only those Stores > 50% of max memcache size. Behavior would vary dependent on the prompt that provoked the flush. Would also log why the flush is running: optional or > max size.
    This issue comes out of HADOOP-2621.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshbase, hadoop
postedFeb 4, '08 at 9:27p
activeFeb 22, '08 at 5:12p
posts7
users1
websitehbase.apache.org

1 user in discussion

Jim Kellerman (JIRA): 7 posts

People

Translate

site design / logo © 2022 Grokbase