Grokbase Groups HBase user May 2011
FAQ
Afternoon everyone,

I am researching what the best practice is for using HBase in user facing
applications. I do not know all of the applications that will be ported to
use HBase, but they do share common characteristics such as

- simple key / value data. Not serving large files ATM. Perhaps a couple
columns in a single column family
- very tall tables
- hundreds of millions of rows
- need millisecond access times for a single row
- random access
- maintain very, very good query times while loading in new data


The quick choice would be to use something like memcache or Redis, but the
data is growing faster than the memory of a single box or even few boxes.
We also have a significant investment in Hadoop technologies so keeping
HBase prime seems to make a lot of sense.

So, some questions:

1. do you find that having a single HBase cluster to serve all applications
vs smaller clusters to serve application specific data is better?
2. In the real world do people hook API's directly to HBase or is there some
caching layer that is used?
3. I remember hearing people like StumbleUpon use different clusters for
analytics vs customer apps. Is this still best practice?
4. Anyone using MSLAB's to reduce GC pauses in production? Experiences /
landmines?
5. What other considerations have you found when hooking HBase up for
user-facing applications?

Thanks in advance and I'd love to hear some bragging!

-Matt

Search Discussions

  • Xavier Stevens at May 5, 2011 at 10:00 pm
    1.) We use a different cluster for each app. I don't know if this is
    best practice or not to be honest. We just wanted to separate downtime
    and potential damage for each application.
    2.) We usually use the HBase APIs directly. Having said that, we
    recently started working on a new service. We were looking at Redis and
    Hazelcast. We chose Hazelcast because we could do a MapStore
    implementation easily that would persist the data to HBase.

    Here's the implementation that we have so far:
    https://github.com/xstevens/bagheera/blob/master/src/java/com/mozilla/bagheera/hazelcast/persistence/HBaseMapStore.java

    We are really only using this as a memory staging area before we do
    batch puts into HBase.

    3.) The StumbleUpon guys have helped us a lot and we have a setup
    similar to them. As of last night we're now using HBase replication to a
    secondary "analysis" cluster.
    4.) Can't really speak for MSLAB yet. We're using it on our analysis
    cluster as a testing ground before using it in production. Todd's blog
    posts were pretty thorough on the topic though.

    Cheers,

    -Xavier
    On 5/5/11 2:26 PM, Matt Davies wrote:
    Afternoon everyone,

    I am researching what the best practice is for using HBase in user facing
    applications. I do not know all of the applications that will be ported to
    use HBase, but they do share common characteristics such as

    - simple key / value data. Not serving large files ATM. Perhaps a couple
    columns in a single column family
    - very tall tables
    - hundreds of millions of rows
    - need millisecond access times for a single row
    - random access
    - maintain very, very good query times while loading in new data


    The quick choice would be to use something like memcache or Redis, but the
    data is growing faster than the memory of a single box or even few boxes.
    We also have a significant investment in Hadoop technologies so keeping
    HBase prime seems to make a lot of sense.

    So, some questions:

    1. do you find that having a single HBase cluster to serve all applications
    vs smaller clusters to serve application specific data is better?
    2. In the real world do people hook API's directly to HBase or is there some
    caching layer that is used?
    3. I remember hearing people like StumbleUpon use different clusters for
    analytics vs customer apps. Is this still best practice?
    4. Anyone using MSLAB's to reduce GC pauses in production? Experiences /
    landmines?
    5. What other considerations have you found when hooking HBase up for
    user-facing applications?

    Thanks in advance and I'd love to hear some bragging!

    -Matt
  • Patrick Angeles at May 6, 2011 at 2:25 am
    Inline...
    On Thu, May 5, 2011 at 5:26 PM, Matt Davies wrote:

    Afternoon everyone,

    I am researching what the best practice is for using HBase in user facing
    applications. I do not know all of the applications that will be ported to
    use HBase, but they do share common characteristics such as

    - simple key / value data. Not serving large files ATM. Perhaps a couple
    columns in a single column family
    - very tall tables
    - hundreds of millions of rows
    - need millisecond access times for a single row
    - random access
    - maintain very, very good query times while loading in new data


    The quick choice would be to use something like memcache or Redis, but the
    data is growing faster than the memory of a single box or even few boxes.
    We also have a significant investment in Hadoop technologies so keeping
    HBase prime seems to make a lot of sense.

    So, some questions:

    1. do you find that having a single HBase cluster to serve all applications
    vs smaller clusters to serve application specific data is better?
    In general, you get higher utilization with the former, but you can better
    tune for a specific application with the latter. Also consider that a single
    20-node cluster running 2 apps will balance load and tolerate node failures
    better than two 10-node clusters running one app each. Another consideration
    is upgrades and maintenance. You might have one app out of 10 that needs the
    shiny new HBase v1.3.37 hotness, but you'll have to bring them all down if
    they're served out of the same cluster.

    2. In the real world do people hook API's directly to HBase or is there
    some
    caching layer that is used?
    We see both in the real world. I think unless you a priori have a good idea
    of what to cache, it's better to rely on HBase's block cache. One severe
    limitation on block cache, though, that is that you can only make it so big,
    because large heaps will make you prone to long GC pauses (even with MSLAB,
    you will GC eventually). However, the OS does a pretty good job
    compensating, because it will use up available memory to cache the
    filesystem.

    3. I remember hearing people like StumbleUpon use different clusters for
    analytics vs customer apps. Is this still best practice?
    This is good practice in RDBMS-land, and I think it still holds, assuming
    you can afford 2 clusters. The former is read-mostly and the latter is
    transactional, and some tuning parameters like memstore/blockcache ratios
    are cluster wide.

    4. Anyone using MSLAB's to reduce GC pauses in production? Experiences /
    landmines?
    5. What other considerations have you found when hooking HBase up for
    user-facing applications?

    Thanks in advance and I'd love to hear some bragging!

    -Matt
  • Gaojinchao at May 6, 2011 at 2:47 am
    A test case was timeout.

    Who has the experience for this.
    Thanks

    <message priority="info"><![CDATA[[exec] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.101 sec]]></message>
    <message priority="info"><![CDATA[[exec] Running org.apache.hadoop.hbase.TestInfoServers]]></message>
    <message priority="info"><![CDATA[[exec] killed.]]></message>
    <message priority="info"><![CDATA[[exec] [INFO] ------------------------------------------------------------------------]]></message>
    <message priority="info"><![CDATA[[exec] [INFO] BUILD FAILURE]]></message>
    <message priority="info"><![CDATA[[exec] [INFO] ------------------------------------------------------------------------]]></message>
    <message priority="info"><![CDATA[[exec] [INFO] Total time: 1:15:10.277s]]></message>
    <message priority="info"><![CDATA[[exec] [INFO] Finished at: Fri May 06 04:52:47 GMT+08:00 2011]]></message>
    <message priority="info"><![CDATA[[exec] [INFO] Final Memory: 53M/949M]]></message>
    <message priority="info"><![CDATA[[exec] [INFO] ------------------------------------------------------------------------]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.4.3:test (default-test) on project hbase: Error while executing forked tests.; nested exception is org.apache.maven.surefire.booter.shade.org.codehaus.plexus.util.cli.CommandLineException: Error while executing external command, process killed. Process timeout out after 900 seconds -> [Help 1]]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR]]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR] Re-run Maven using the -X switch to enable full debug logging.]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR]]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR] For more information about the errors and possible solutions, please read the following articles:]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException]]></message>
    <message priority="info"><![CDATA[[echo] llt success.]]></message>
    <message priority="info"><![CDATA[llt.report:]]></message>

    The log is half

    2011-05-06 04:38:15,466 DEBUG [main-EventThread] zookeeper.ZKUtil(1107): master:56634-0x12fc1e1c2fa0000 Retrieved 54 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, server=linux1.site,48768,1304627889113, state=RS_ZK_REGION_OPENING
    2011-05-06 04:38:15,466 DEBUG [main-EventThread] master.AssignmentManager(371): Handling transition=RS_ZK_REGION_OPENING, server=linux1.site,48768,13046278891 // The log is half
  • Ted Yu at May 6, 2011 at 2:59 am
    Can you enable -X switch ?
    Also, 1 minute into running the test, you can use jstack to see what the
    main thread is doing.
    On Thu, May 5, 2011 at 7:47 PM, Gaojinchao wrote:

    A test case was timeout.

    Who has the experience for this.
    Thanks

    <message priority="info"><![CDATA[[exec] Tests run: 3, Failures: 0,
    Errors: 0, Skipped: 0, Time elapsed: 0.101 sec]]></message>
    <message priority="info"><![CDATA[[exec] Running
    org.apache.hadoop.hbase.TestInfoServers]]></message>
    <message priority="info"><![CDATA[[exec]
    killed.]]></message>
    <message priority="info"><![CDATA[[exec] [INFO]
    ------------------------------------------------------------------------]]></message>
    <message priority="info"><![CDATA[[exec] [INFO] BUILD
    FAILURE]]></message>
    <message priority="info"><![CDATA[[exec] [INFO]
    ------------------------------------------------------------------------]]></message>
    <message priority="info"><![CDATA[[exec] [INFO] Total time:
    1:15:10.277s]]></message>
    <message priority="info"><![CDATA[[exec] [INFO] Finished at:
    Fri May 06 04:52:47 GMT+08:00 2011]]></message>
    <message priority="info"><![CDATA[[exec] [INFO] Final
    Memory: 53M/949M]]></message>
    <message priority="info"><![CDATA[[exec] [INFO]
    ------------------------------------------------------------------------]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR] Failed to
    execute goal org.apache.maven.plugins:maven-surefire-plugin:2.4.3:test
    (default-test) on project hbase: Error while executing forked tests.; nested
    exception is
    org.apache.maven.surefire.booter.shade.org.codehaus.plexus.util.cli.CommandLineException:
    Error while executing external command, process killed. Process timeout out
    after 900 seconds -> [Help 1]]]></message>
    <message priority="info"><![CDATA[[exec]
    [ERROR]]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR] To see the
    full stack trace of the errors, re-run Maven with the -e
    switch.]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR] Re-run
    Maven using the -X switch to enable full debug logging.]]></message>
    <message priority="info"><![CDATA[[exec]
    [ERROR]]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR] For more
    information about the errors and possible solutions, please read the
    following articles:]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR] [Help 1]
    http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException]
    ]></message>
    <message priority="info"><![CDATA[[echo] llt
    success.]]></message>
    <message priority="info"><![CDATA[llt.report:]]></message>

    The log is half

    2011-05-06 04:38:15,466 DEBUG [main-EventThread] zookeeper.ZKUtil(1107):
    master:56634-0x12fc1e1c2fa0000 Retrieved 54 byte(s) of data from znode
    /hbase/unassigned/1028785192 and set watcher; region=.META.,,1,
    server=linux1.site,48768,1304627889113, state=RS_ZK_REGION_OPENING
    2011-05-06 04:38:15,466 DEBUG [main-EventThread]
    master.AssignmentManager(371): Handling transition=RS_ZK_REGION_OPENING,
    server=linux1.site,48768,13046278891 // The log is half
  • Gaojinchao at May 6, 2011 at 5:32 am
    Unlucky. It can't reproduce. It happened last night.

    It seems some threads for freeze or locking.



    -----邮件原件-----
    发件人: Ted Yu
    发送时间: 2011年5月6日 10:59
    收件人: user@hbase.apache.org
    抄送: Chenjian
    主题: Re: A question about unit test

    Can you enable -X switch ?
    Also, 1 minute into running the test, you can use jstack to see what the
    main thread is doing.
    On Thu, May 5, 2011 at 7:47 PM, Gaojinchao wrote:

    A test case was timeout.

    Who has the experience for this.
    Thanks

    <message priority="info"><![CDATA[[exec] Tests run: 3, Failures: 0,
    Errors: 0, Skipped: 0, Time elapsed: 0.101 sec]]></message>
    <message priority="info"><![CDATA[[exec] Running
    org.apache.hadoop.hbase.TestInfoServers]]></message>
    <message priority="info"><![CDATA[[exec]
    killed.]]></message>
    <message priority="info"><![CDATA[[exec] [INFO]
    ------------------------------------------------------------------------]]></message>
    <message priority="info"><![CDATA[[exec] [INFO] BUILD
    FAILURE]]></message>
    <message priority="info"><![CDATA[[exec] [INFO]
    ------------------------------------------------------------------------]]></message>
    <message priority="info"><![CDATA[[exec] [INFO] Total time:
    1:15:10.277s]]></message>
    <message priority="info"><![CDATA[[exec] [INFO] Finished at:
    Fri May 06 04:52:47 GMT+08:00 2011]]></message>
    <message priority="info"><![CDATA[[exec] [INFO] Final
    Memory: 53M/949M]]></message>
    <message priority="info"><![CDATA[[exec] [INFO]
    ------------------------------------------------------------------------]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR] Failed to
    execute goal org.apache.maven.plugins:maven-surefire-plugin:2.4.3:test
    (default-test) on project hbase: Error while executing forked tests.; nested
    exception is
    org.apache.maven.surefire.booter.shade.org.codehaus.plexus.util.cli.CommandLineException:
    Error while executing external command, process killed. Process timeout out
    after 900 seconds -> [Help 1]]]></message>
    <message priority="info"><![CDATA[[exec]
    [ERROR]]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR] To see the
    full stack trace of the errors, re-run Maven with the -e
    switch.]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR] Re-run
    Maven using the -X switch to enable full debug logging.]]></message>
    <message priority="info"><![CDATA[[exec]
    [ERROR]]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR] For more
    information about the errors and possible solutions, please read the
    following articles:]]></message>
    <message priority="info"><![CDATA[[exec] [ERROR] [Help 1]
    http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException]
    ]></message>
    <message priority="info"><![CDATA[[echo] llt
    success.]]></message>
    <message priority="info"><![CDATA[llt.report:]]></message>

    The log is half

    2011-05-06 04:38:15,466 DEBUG [main-EventThread] zookeeper.ZKUtil(1107):
    master:56634-0x12fc1e1c2fa0000 Retrieved 54 byte(s) of data from znode
    /hbase/unassigned/1028785192 and set watcher; region=.META.,,1,
    server=linux1.site,48768,1304627889113, state=RS_ZK_REGION_OPENING
    2011-05-06 04:38:15,466 DEBUG [main-EventThread]
    master.AssignmentManager(371): Handling transition=RS_ZK_REGION_OPENING,
    server=linux1.site,48768,13046278891 // The log is half

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshbase, hadoop
postedMay 5, '11 at 9:27p
activeMay 6, '11 at 5:32a
posts6
users5
websitehbase.apache.org

People

Translate

site design / logo © 2022 Grokbase