FAQ
Reevaluate HBASE-288 block caching work; should it be enabled always?
---------------------------------------------------------------------

Key: HBASE-953
URL: https://issues.apache.org/jira/browse/HBASE-953
Project: Hadoop HBase
Issue Type: Task
Reporter: stack
Priority: Critical
Fix For: 0.19.0


Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • stack (JIRA) at Oct 24, 2008 at 11:24 pm
    [ https://issues.apache.org/jira/browse/HBASE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642568#action_12642568 ]

    stack commented on HBASE-953:
    -----------------------------

    Enabling cache, in PE tests I see 4X improvement in sequential read test. My guess is that scan speed will improve again too and that compactions should go the faster also. Downside is that I'm OOME'ing running random read test. Digging in some, we're creating lots of instances of the BlockFSDataStream. Each instance has its own ReferenceMap. Each ReferenceMap has 1-3 entries when blocksize is 64MB. References are cleared only when we put or get. I'm thinking this is not enough; that caches are not being cleared fast enough. Trying to dig in more.
    Reevaluate HBASE-288 block caching work; should it be enabled always?
    ---------------------------------------------------------------------

    Key: HBASE-953
    URL: https://issues.apache.org/jira/browse/HBASE-953
    Project: Hadoop HBase
    Issue Type: Task
    Reporter: stack
    Priority: Critical
    Fix For: 0.19.0


    Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Oct 24, 2008 at 11:28 pm
    [ https://issues.apache.org/jira/browse/HBASE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642569#action_12642569 ]

    stack commented on HBASE-953:
    -----------------------------

    PE random read also ran a good big slower.
    Reevaluate HBASE-288 block caching work; should it be enabled always?
    ---------------------------------------------------------------------

    Key: HBASE-953
    URL: https://issues.apache.org/jira/browse/HBASE-953
    Project: Hadoop HBase
    Issue Type: Task
    Reporter: stack
    Priority: Critical
    Fix For: 0.19.0


    Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Oct 25, 2008 at 2:20 am
    [ https://issues.apache.org/jira/browse/HBASE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642587#action_12642587 ]

    stack commented on HBASE-953:
    -----------------------------

    Using SoftSorteMap instead of commons ReferenceMap seems to do better. Still not good enough though; I get "OOME: GC Time and OutOfMemoryError" which doesn't exactly kill it.. it hobbles along. I see random reading we're getting 90% cache hits if run after a sequential read which means cache has been warmed but the random reads should be going faster than they are. They're slowed by all the GCing I'm seeing (I have gc logging enabled). Trying w/ 1.6.0_10 JVM. Was using 1.7. If random reads were just as fast as without block reads, it would be worth enabling by default because of the faster sequential reads, scans and compactions.
    Reevaluate HBASE-288 block caching work; should it be enabled always?
    ---------------------------------------------------------------------

    Key: HBASE-953
    URL: https://issues.apache.org/jira/browse/HBASE-953
    Project: Hadoop HBase
    Issue Type: Task
    Reporter: stack
    Priority: Critical
    Fix For: 0.19.0


    Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rong-En Fan (JIRA) at Oct 25, 2008 at 3:52 am
    [ https://issues.apache.org/jira/browse/HBASE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642590#action_12642590 ]

    Rong-En Fan commented on HBASE-953:
    -----------------------------------

    I have been using 1.6.0_04 with concurrent sweep GC. Under modest random read workload, w/ block caching, it's faster.
    Reevaluate HBASE-288 block caching work; should it be enabled always?
    ---------------------------------------------------------------------

    Key: HBASE-953
    URL: https://issues.apache.org/jira/browse/HBASE-953
    Project: Hadoop HBase
    Issue Type: Task
    Reporter: stack
    Priority: Critical
    Fix For: 0.19.0


    Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Oct 25, 2008 at 4:30 am
    [ https://issues.apache.org/jira/browse/HBASE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642593#action_12642593 ]

    stack commented on HBASE-953:
    -----------------------------

    Rong-en: Thanks for the tip. I'll try some loading tests using it. It might be better with soft references.

    With 1.6.0_10, with 8 concurrent clients, random reads completed. Trying again with single client.. but also thinking on it on way home, smaller blocks make more sense. I cut it down to 16k as suggested by jgray
    Reevaluate HBASE-288 block caching work; should it be enabled always?
    ---------------------------------------------------------------------

    Key: HBASE-953
    URL: https://issues.apache.org/jira/browse/HBASE-953
    Project: Hadoop HBase
    Issue Type: Task
    Reporter: stack
    Priority: Critical
    Fix For: 0.19.0


    Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rong-En Fan (JIRA) at Oct 25, 2008 at 7:04 am
    [ https://issues.apache.org/jira/browse/HBASE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642599#action_12642599 ]

    Rong-En Fan commented on HBASE-953:
    -----------------------------------

    So, you are talking about using smaller block size in block cache or hdfs block size?
    Reevaluate HBASE-288 block caching work; should it be enabled always?
    ---------------------------------------------------------------------

    Key: HBASE-953
    URL: https://issues.apache.org/jira/browse/HBASE-953
    Project: Hadoop HBase
    Issue Type: Task
    Reporter: stack
    Priority: Critical
    Fix For: 0.19.0


    Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Oct 25, 2008 at 3:48 pm
    [ https://issues.apache.org/jira/browse/HBASE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642626#action_12642626 ]

    stack commented on HBASE-953:
    -----------------------------

    Rong-en: I'm looking to enable BLOCKCACHE in HColumnDescriptor as the default. Currently its off. The size I'm playing with is this one: hbase.hstore.blockCache.blockSize

    I ran with 1.6.0_10 JDK and 16k blocksize and all single-client runs ran as well as an overnight 16 concurrent client test. All passed. Running some more tests to see if I can make an OOME happen.
    Reevaluate HBASE-288 block caching work; should it be enabled always?
    ---------------------------------------------------------------------

    Key: HBASE-953
    URL: https://issues.apache.org/jira/browse/HBASE-953
    Project: Hadoop HBase
    Issue Type: Task
    Reporter: stack
    Priority: Critical
    Fix For: 0.19.0


    Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Oct 25, 2008 at 5:15 pm
    [ https://issues.apache.org/jira/browse/HBASE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642641#action_12642641 ]

    stack commented on HBASE-953:
    -----------------------------

    1.6.0_10 works fine.

    java version "1.7.0-ea"
    Java(TM) SE Runtime Environment (build 1.7.0-ea-b31)
    Java HotSpot(TM) Server VM (build 14.0-b01, mixed mode)

    Will OOME. Stuck doing full GCs. Works if I enable the JVM flag which says don't OOME if too many full GCs.
    Reevaluate HBASE-288 block caching work; should it be enabled always?
    ---------------------------------------------------------------------

    Key: HBASE-953
    URL: https://issues.apache.org/jira/browse/HBASE-953
    Project: Hadoop HBase
    Issue Type: Task
    Reporter: stack
    Priority: Critical
    Fix For: 0.19.0


    Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Oct 25, 2008 at 9:29 pm
    [ https://issues.apache.org/jira/browse/HBASE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    stack updated HBASE-953:
    ------------------------

    Summary: Enable BLOCKCACHE by default [WAS -> Reevaluate HBASE-288 block caching work....?] (was: Reevaluate HBASE-288 block caching work; should it be enabled always?)
    Enable BLOCKCACHE by default [WAS -> Reevaluate HBASE-288 block caching work....?]
    ----------------------------------------------------------------------------------

    Key: HBASE-953
    URL: https://issues.apache.org/jira/browse/HBASE-953
    Project: Hadoop HBase
    Issue Type: Task
    Reporter: stack
    Priority: Critical
    Fix For: 0.19.0


    Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Oct 25, 2008 at 9:39 pm
    [ https://issues.apache.org/jira/browse/HBASE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    stack updated HBASE-953:
    ------------------------

    Attachment: blockcache.patch

    Commit comment below
    {code}
    HBASE-953 Enable BLOCKCACHE by default [WAS -> Reevaluate HBASE-288 block caching work....?]
    Change that makes BLOCKCACHE on by default but caching blocks of 16k rather than 64k.
    Means that upgrading, users must update their hbase-default.xml to pick up the smaller
    value else OOME issues. This patch may introduce OOME issues anyways. Need to watch
    out for it (Was find on 1.6.0_10 but on an early 1.7.0 was easy to OOME).

    This patch removes commons-collections.jar. Was only added for its ReferenceMap which
    is replaced in this patch by a version of what used to be called SoftSortedMap (renamed
    as SoftValueSortedMap); the ReferenceMap replacement is SoftValueMap. Use our own
    because I want to be able to watch evictions if issues with OOMEs and RM is all private
    when it runs the ReferenceQueue clean up (RM also had features we don't need).

    M conf/hbase-default.xml
    Update hbase.hstore.blockCache.blockSize; make 16k instaed of 64k.
    D lib/commons-collections-3.2.jar
    Removed. Not used.
    A src/test/org/apache/hadoop/hbase/util/SoftValueSortedMapTest.java
    Rename of SoftSortedMapTest.
    D src/test/org/apache/hadoop/hbase/util/SoftSortedMapTest.java
    Removed
    M src/java/org/apache/hadoop/hbase/HColumnDescriptor.java
    Enable blockcaching as default.
    M src/java/org/apache/hadoop/hbase/io/BlockFSInputStream.java
    Use SoftValueMap instead of ReferenceMap.
    A src/java/org/apache/hadoop/hbase/util/ReferenceQueueUtil.java
    ReferenceQueue utility class.
    A src/java/org/apache/hadoop/hbase/util/SoftValueMap.java
    Like SoftValueSortedMap but not sorted (based on it).
    A src/java/org/apache/hadoop/hbase/util/SoftValue.java
    Added. Was internal class of SoftSortedMap.
    A src/java/org/apache/hadoop/hbase/util/SoftValueSortedMap.java
    Rename of SoftSortedMap.
    A src/java/org/apache/hadoop/hbase/util/SoftSortedMap.java
    Like SoftValueSortedMap but a plain Map.
    M src/java/org/apache/hadoop/hbase/client/HConnectionManager.java
    SoftSortedMap was renamed SoftValueSortedMap.
    {code}
    Enable BLOCKCACHE by default [WAS -> Reevaluate HBASE-288 block caching work....?]
    ----------------------------------------------------------------------------------

    Key: HBASE-953
    URL: https://issues.apache.org/jira/browse/HBASE-953
    Project: Hadoop HBase
    Issue Type: Task
    Reporter: stack
    Priority: Critical
    Fix For: 0.19.0

    Attachments: blockcache.patch


    Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Oct 25, 2008 at 11:01 pm
    [ https://issues.apache.org/jira/browse/HBASE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642664#action_12642664 ]

    stack commented on HBASE-953:
    -----------------------------

    Passes all tests locally. I'll keep running loading tests over the weekend but I'd like to commit this. Any chance of a review?
    Enable BLOCKCACHE by default [WAS -> Reevaluate HBASE-288 block caching work....?]
    ----------------------------------------------------------------------------------

    Key: HBASE-953
    URL: https://issues.apache.org/jira/browse/HBASE-953
    Project: Hadoop HBase
    Issue Type: Task
    Reporter: stack
    Priority: Critical
    Fix For: 0.19.0

    Attachments: blockcache.patch


    Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Jean-Daniel Cryans (JIRA) at Oct 27, 2008 at 5:53 pm
    [ https://issues.apache.org/jira/browse/HBASE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642978#action_12642978 ]

    Jean-Daniel Cryans commented on HBASE-953:
    ------------------------------------------

    Review: there are some long lines in HCM but +1 if corrected.
    Enable BLOCKCACHE by default [WAS -> Reevaluate HBASE-288 block caching work....?]
    ----------------------------------------------------------------------------------

    Key: HBASE-953
    URL: https://issues.apache.org/jira/browse/HBASE-953
    Project: Hadoop HBase
    Issue Type: Task
    Reporter: stack
    Priority: Critical
    Fix For: 0.19.0

    Attachments: blockcache.patch


    Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Oct 27, 2008 at 6:01 pm
    [ https://issues.apache.org/jira/browse/HBASE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642979#action_12642979 ]

    stack commented on HBASE-953:
    -----------------------------

    Tried with jdk1.7.0b38 (as opposed to the b31 used above) and it held up. Lots of full GC's but weathered the PE tests w/o OOME. J-D gave my patch a review and said fine except for some line-lengths which I've fixed.
    Enable BLOCKCACHE by default [WAS -> Reevaluate HBASE-288 block caching work....?]
    ----------------------------------------------------------------------------------

    Key: HBASE-953
    URL: https://issues.apache.org/jira/browse/HBASE-953
    Project: Hadoop HBase
    Issue Type: Task
    Reporter: stack
    Priority: Critical
    Fix For: 0.19.0

    Attachments: blockcache.patch


    Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Oct 27, 2008 at 6:39 pm
    [ https://issues.apache.org/jira/browse/HBASE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    stack resolved HBASE-953.
    -------------------------

    Resolution: Fixed

    Committed.
    Enable BLOCKCACHE by default [WAS -> Reevaluate HBASE-288 block caching work....?]
    ----------------------------------------------------------------------------------

    Key: HBASE-953
    URL: https://issues.apache.org/jira/browse/HBASE-953
    Project: Hadoop HBase
    Issue Type: Task
    Reporter: stack
    Priority: Critical
    Fix For: 0.19.0

    Attachments: blockcache.patch


    Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Dec 13, 2008 at 10:07 pm
    [ https://issues.apache.org/jira/browse/HBASE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    stack reassigned HBASE-953:
    ---------------------------

    Assignee: stack
    Enable BLOCKCACHE by default [WAS -> Reevaluate HBASE-288 block caching work....?]
    ----------------------------------------------------------------------------------

    Key: HBASE-953
    URL: https://issues.apache.org/jira/browse/HBASE-953
    Project: Hadoop HBase
    Issue Type: Task
    Reporter: stack
    Assignee: stack
    Priority: Critical
    Fix For: 0.19.0

    Attachments: blockcache.patch


    Go back and take another look at the Tom White work. We've gotten boost in writing and scanning because of J-D work. HBASE-288 looks like it boosts sequential reads and perhaps random read a little. Take another look.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshbase, hadoop
postedOct 24, '08 at 4:28a
activeDec 13, '08 at 10:07p
posts16
users1
websitehbase.apache.org

1 user in discussion

stack (JIRA): 16 posts

People

Translate

site design / logo © 2022 Grokbase