FAQ
Hi,

We have 20 1U servers (4 core, 12G ram) as a cluster,
3 zookeepers, 10 region servers.
My program is to read hbase table data one by one.
Read -> hbaseTable1
hbaseTable2
hbaseTable3
.
.
.
hbaseTable30
---------------------------------------------------
I collected some performance data while I started different number of
client to read data
from hbase concurrently.

10 client duration : 53 (sec)
50 181
200 839
400 1801

I think the multiple reading performance is not good.
How can I start to fine tune it?
Any suggestion would be appreciated!


Fleming Chiu(邱宏明)
707-6128
y_823910@tsmc.com
週一無肉日吃素救地球(Meat Free Monday Taiwan)


---------------------------------------------------------------------------
TSMC PROPERTY
This email communication (and any attachments) is proprietary information
for the sole use of its
intended recipient. Any unauthorized review, use or distribution by anyone
other than the intended
recipient is strictly prohibited. If you are not the intended recipient,
please notify the sender by
replying to this email, and then delete this email and any copies of it
immediately. Thank you.
---------------------------------------------------------------------------

Search Discussions

  • Stack at Feb 25, 2010 at 12:52 pm
    What kind of reading are you doing ? Random reads or scans? What
    version of hbase?
    Thanks,
    St.Ack

    2010/2/25 <y_823910@tsmc.com>:
    Hi,

    We have 20 1U servers (4 core, 12G ram) as a cluster,
    3 zookeepers, 10 region servers.
    My program is to read hbase table data one by one.
    Read -> hbaseTable1
    hbaseTable2
    hbaseTable3
    .
    .
    .
    hbaseTable30
    ---------------------------------------------------
    I collected some performance data while I started different number of
    client to read data
    from hbase concurrently.

    10 client duration : 53 (sec)
    50 181
    200 839
    400 1801

    I think the multiple reading performance is not good.
    How can I start to fine tune it?
    Any suggestion would be appreciated!


    Fleming Chiu(邱宏明)
    707-6128
    y_823910@tsmc.com
    週一無肉日吃素救地球(Meat Free Monday Taiwan)


    ---------------------------------------------------------------------------
    TSMC PROPERTY
    This email communication (and any attachments) is proprietary information
    for the sole use of its
    intended recipient. Any unauthorized review, use or distribution by anyone
    other than the intended
    recipient is strictly prohibited. If you are not the intended recipient,
    please notify the sender by
    replying to this email, and then delete this email and any copies of it
    immediately. Thank you.
    ---------------------------------------------------------------------------


  • Michael Segel at Feb 25, 2010 at 2:11 pm
    Also how are you reading from HBase?
    Are you using the native APIs? Stargate? something else?

    Date: Thu, 25 Feb 2010 04:51:39 -0800
    Subject: Re: HBase reading performance
    From: stack@duboce.net
    To: hbase-user@hadoop.apache.org

    What kind of reading are you doing ? Random reads or scans? What
    version of hbase?
    Thanks,
    St.Ack

    2010/2/25 <y_823910@tsmc.com>:
    Hi,

    We have 20 1U servers (4 core, 12G ram) as a cluster,
    3 zookeepers, 10 region servers.
    My program is to read hbase table data one by one.
    Read -> hbaseTable1
    hbaseTable2
    hbaseTable3
    .
    .
    .
    hbaseTable30
    ---------------------------------------------------
    I collected some performance data while I started different number of
    client to read data
    from hbase concurrently.

    10 client duration : 53 (sec)
    50 181
    200 839
    400 1801

    I think the multiple reading performance is not good.
    How can I start to fine tune it?
    Any suggestion would be appreciated!


    Fleming Chiu(邱宏明)
    707-6128
    y_823910@tsmc.com
    週一無肉日吃素救地球(Meat Free Monday Taiwan)


    ---------------------------------------------------------------------------
    TSMC PROPERTY
    This email communication (and any attachments) is proprietary information
    for the sole use of its
    intended recipient. Any unauthorized review, use or distribution by anyone
    other than the intended
    recipient is strictly prohibited. If you are not the intended recipient,
    please notify the sender by
    replying to this email, and then delete this email and any copies of it
    immediately. Thank you.
    ---------------------------------------------------------------------------


    _________________________________________________________________
    Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
    http://clk.atdmt.com/GBL/go/201469229/direct/01/
  • Y_823910 at Feb 26, 2010 at 12:43 am
    Sorry for my ignoring their basic informations.

    What kind of reading are you doing ? Random reads or scans? What
    version of hbase?
    ---------------------------------------------------------------------------------------

    HBase Version 0.20.2, r834515

    ResultScanner scanner = table2.getIndexedScanner(idx_column,
    Bytes.toBytes(idx_value), Bytes.toBytes(idx_value+"~"),null,
    filter,
    new byte[][] { Bytes.toBytes(columnfamily+":")});


    Are you using the native APIs? Stargate? something else?
    ----------------------------------------------------------------------------------------
    Yes, I am using the native APIS.
    I dispatched these java code to different machines and run it concurrently
    through GridGain.

    ----------------------------------------------------------------------------------------

    My zookeeper servers are not in the dedicated machine.
    <name>hbase.zookeeper.property.maxClientCnxns</name>
    <value>1000</value>
    ----------------------------------------------------------------------------------------

    Region Count
    f12p4cc06.tsmc:60030 1266903209803 requests=0, regions=143, usedHeap=1396,
    maxHeap=1991
    f12p4cc07.tsmc:60030 1266903219641 requests=193, regions=147, usedHeap=897,
    maxHeap=1991
    f12p4cc08.tsmc:60030 1266903209410 requests=0, regions=151, usedHeap=955,
    maxHeap=1991
    f12p4cc09.tsmc:60030 1266903210645 requests=0, regions=153, usedHeap=1162,
    maxHeap=1991
    f12p4cc10.tsmc:60030 1266903208231 requests=0, regions=155, usedHeap=1102,
    maxHeap=1991
    f12p4cc11.tsmc:60030 1266903208718 requests=0, regions=146, usedHeap=1321,
    maxHeap=1991
    f12p4cc12.tsmc:60030 1266903205539 requests=0, regions=142, usedHeap=659,
    maxHeap=1991
    f12p4cc13.tsmc:60030 1266903214999 requests=0, regions=157, usedHeap=1528,
    maxHeap=1991
    f12p4cc14.tsmc:60030 1266903215714 requests=0, regions=153, usedHeap=939,
    maxHeap=1991
    f12p4cc15.tsmc:60030 1266903207766 requests=0, regions=153, usedHeap=1004,
    maxHeap=1991
    Total: servers: 10 requests=193, regions=1500



    Fleming Chiu(邱宏明)
    707-6128
    y_823910@tsmc.com
    週一無肉日吃素救地球(Meat Free Monday Taiwan)





    Michael Segel
    <michael_segel@ho To: <hbase-user@hadoop.apache.org>
    tmail.com> cc: (bcc: Y_823910/TSMC)
    Subject: RE: HBase reading performance
    2010/02/25 10:10
    PM
    Please respond to
    hbase-user







    Also how are you reading from HBase?
    Are you using the native APIs? Stargate? something else?

    Date: Thu, 25 Feb 2010 04:51:39 -0800
    Subject: Re: HBase reading performance
    From: stack@duboce.net
    To: hbase-user@hadoop.apache.org

    What kind of reading are you doing ? Random reads or scans? What
    version of hbase?
    Thanks,
    St.Ack

    2010/2/25 <y_823910@tsmc.com>:
    Hi,

    We have 20 1U servers (4 core, 12G ram) as a cluster,
    3 zookeepers, 10 region servers.
    My program is to read hbase table data one by one.
    Read -> hbaseTable1
    hbaseTable2
    hbaseTable3
    .
    .
    .
    hbaseTable30
    ---------------------------------------------------
    I collected some performance data while I started different number of
    client to read data
    from hbase concurrently.

    10 client duration : 53 (sec)
    50 181
    200 839
    400 1801

    I think the multiple reading performance is not good.
    How can I start to fine tune it?
    Any suggestion would be appreciated!


    Fleming Chiu(邱宏明)
    707-6128
    y_823910@tsmc.com
    週一無肉日吃素救地球(Meat Free Monday Taiwan)

    ---------------------------------------------------------------------------
    TSMC PROPERTY
    This email communication (and any attachments) is proprietary
    information
    for the sole use of its
    intended recipient. Any unauthorized review, use or distribution by
    anyone
    other than the intended
    recipient is strictly prohibited. If you are not the intended
    recipient,
    please notify the sender by
    replying to this email, and then delete this email and any copies of
    it
    immediately. Thank you.
    ---------------------------------------------------------------------------

    _________________________________________________________________
    Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
    http://clk.atdmt.com/GBL/go/201469229/direct/01/



    ---------------------------------------------------------------------------
    TSMC PROPERTY
    This email communication (and any attachments) is proprietary information
    for the sole use of its
    intended recipient. Any unauthorized review, use or distribution by anyone
    other than the intended
    recipient is strictly prohibited. If you are not the intended recipient,
    please notify the sender by
    replying to this email, and then delete this email and any copies of it
    immediately. Thank you.
    ---------------------------------------------------------------------------
  • Y_823910 at Feb 26, 2010 at 9:55 am
    My META table just be splitted by one region in one machine.
    Name Region Server Encoded Name Start Key End Key
    .META.,,1 f12p4cc11.tsmc:60030 -
    That machine has high CPU and network usage.
    I guess that's the bottleneck of multiple client reading performance.
    Any ideas ?
    Thanks

    Fleming Chiu(邱宏明)
    707-6128
    y_823910@tsmc.com
    週一無肉日吃素救地球(Meat Free Monday Taiwan)





    y_823910@tsmc.com
    To: hbase-user@hadoop.apache.org
    2010/02/26 08:42 cc: (bcc: Y_823910/TSMC)
    AM Subject: RE: HBase reading performance
    Please respond to
    hbase-user






    Sorry for my ignoring their basic informations.

    What kind of reading are you doing ? Random reads or scans? What
    version of hbase?
    ---------------------------------------------------------------------------------------


    HBase Version 0.20.2, r834515

    ResultScanner scanner = table2.getIndexedScanner(idx_column,
    Bytes.toBytes(idx_value), Bytes.toBytes(idx_value+"~"),null,
    filter,
    new byte[][] { Bytes.toBytes(columnfamily+":")});


    Are you using the native APIs? Stargate? something else?
    ----------------------------------------------------------------------------------------

    Yes, I am using the native APIS.
    I dispatched these java code to different machines and run it concurrently
    through GridGain.

    ----------------------------------------------------------------------------------------


    My zookeeper servers are not in the dedicated machine.
    <name>hbase.zookeeper.property.maxClientCnxns</name>
    <value>1000</value>
    ----------------------------------------------------------------------------------------


    Region Count
    f12p4cc06.tsmc:60030 1266903209803 requests=0, regions=143, usedHeap=1396,
    maxHeap=1991
    f12p4cc07.tsmc:60030 1266903219641 requests=193, regions=147, usedHeap=897,
    maxHeap=1991
    f12p4cc08.tsmc:60030 1266903209410 requests=0, regions=151, usedHeap=955,
    maxHeap=1991
    f12p4cc09.tsmc:60030 1266903210645 requests=0, regions=153, usedHeap=1162,
    maxHeap=1991
    f12p4cc10.tsmc:60030 1266903208231 requests=0, regions=155, usedHeap=1102,
    maxHeap=1991
    f12p4cc11.tsmc:60030 1266903208718 requests=0, regions=146, usedHeap=1321,
    maxHeap=1991
    f12p4cc12.tsmc:60030 1266903205539 requests=0, regions=142, usedHeap=659,
    maxHeap=1991
    f12p4cc13.tsmc:60030 1266903214999 requests=0, regions=157, usedHeap=1528,
    maxHeap=1991
    f12p4cc14.tsmc:60030 1266903215714 requests=0, regions=153, usedHeap=939,
    maxHeap=1991
    f12p4cc15.tsmc:60030 1266903207766 requests=0, regions=153, usedHeap=1004,
    maxHeap=1991
    Total: servers: 10 requests=193, regions=1500



    Fleming Chiu(邱宏明)
    707-6128
    y_823910@tsmc.com
    週一無肉日吃素救地球(Meat Free Monday Taiwan)





    Michael Segel

    <michael_segel@ho To:
    <hbase-user@hadoop.apache.org>

    tmail.com> cc: (bcc:
    Y_823910/TSMC)

    Subject: RE: HBase reading
    performance

    2010/02/25 10:10

    PM

    Please respond to

    hbase-user








    Also how are you reading from HBase?
    Are you using the native APIs? Stargate? something else?

    Date: Thu, 25 Feb 2010 04:51:39 -0800
    Subject: Re: HBase reading performance
    From: stack@duboce.net
    To: hbase-user@hadoop.apache.org

    What kind of reading are you doing ? Random reads or scans? What
    version of hbase?
    Thanks,
    St.Ack

    2010/2/25 <y_823910@tsmc.com>:
    Hi,

    We have 20 1U servers (4 core, 12G ram) as a cluster,
    3 zookeepers, 10 region servers.
    My program is to read hbase table data one by one.
    Read -> hbaseTable1
    hbaseTable2
    hbaseTable3
    .
    .
    .
    hbaseTable30
    ---------------------------------------------------
    I collected some performance data while I started different number of
    client to read data
    from hbase concurrently.

    10 client duration : 53 (sec)
    50 181
    200 839
    400 1801

    I think the multiple reading performance is not good.
    How can I start to fine tune it?
    Any suggestion would be appreciated!


    Fleming Chiu(邱宏明)
    707-6128
    y_823910@tsmc.com
    週一無肉日吃素救地球(Meat Free Monday Taiwan)

    ---------------------------------------------------------------------------
    TSMC PROPERTY
    This email communication (and any attachments) is proprietary
    information
    for the sole use of its
    intended recipient. Any unauthorized review, use or distribution by
    anyone
    other than the intended
    recipient is strictly prohibited. If you are not the intended
    recipient,
    please notify the sender by
    replying to this email, and then delete this email and any copies of
    it
    immediately. Thank you.
    ---------------------------------------------------------------------------

    _________________________________________________________________
    Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
    http://clk.atdmt.com/GBL/go/201469229/direct/01/




    ---------------------------------------------------------------------------

    TSMC PROPERTY

    This email communication (and any attachments) is proprietary information

    for the sole use of its

    intended recipient. Any unauthorized review, use or distribution by anyone

    other than the intended
    recipient is strictly prohibited. If you are not the intended recipient,

    please notify the sender by

    replying to this email, and then delete this email and any copies of it

    immediately. Thank you.


    ---------------------------------------------------------------------------








    ---------------------------------------------------------------------------
    TSMC PROPERTY
    This email communication (and any attachments) is proprietary information
    for the sole use of its
    intended recipient. Any unauthorized review, use or distribution by anyone
    other than the intended
    recipient is strictly prohibited. If you are not the intended recipient,
    please notify the sender by
    replying to this email, and then delete this email and any copies of it
    immediately. Thank you.
    ---------------------------------------------------------------------------

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshbase, hadoop
postedFeb 25, '10 at 10:00a
activeFeb 26, '10 at 9:55a
posts5
users3
websitehbase.apache.org

3 users in discussion

Y_823910: 3 posts Michael Segel: 1 post Stack: 1 post

People

Translate

site design / logo © 2022 Grokbase