FAQ

[HBase-issues] [jira] [Commented] (HBASE-1364) [performance] Distributed splitting of regionserver commit logs

stack (JIRA)
Apr 6, 2011 at 9:18 pm
[ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016539#comment-13016539 ]

stack commented on HBASE-1364:
------------------------------

I did the top 50% or so again over on rb. Over there I write "...I'd be up for committing this sooner rather than later, especially if its bascially working for you. My thought is this is a big patch and its critical functionality so commit and get the rest of the community helping iron out bugs. Let us know when you think its good to commit."

FYI Prakash, you need to update here when you post a new patch, at least for the moment, because email of notifications is not working (We'll be moving to Apache's review board instance some time soon).
[performance] Distributed splitting of regionserver commit logs
---------------------------------------------------------------

Key: HBASE-1364
URL: https://issues.apache.org/jira/browse/HBASE-1364
Project: HBase
Issue Type: Improvement
Components: coprocessors
Reporter: stack
Assignee: Prakash Khemani
Priority: Critical
Fix For: 0.92.0

Attachments: HBASE-1364.patch

Time Spent: 8h
Remaining Estimate: 0h

HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
(Below is from HBASE-1008)
In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
reply

Search Discussions

18 responses

  • Ted Yu (JIRA) at Apr 7, 2011 at 4:34 am
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016681#comment-13016681 ]

    Ted Yu commented on HBASE-1364:
    -------------------------------

    I got the following from TestDistributedLogSplitting:
    {code}
    testOrphanLogCreation(org.apache.hadoop.hbase.master.TestDistributedLogSplitting) Time elapsed: 33.203 sec <<< ERROR!
    java.lang.Exception: Unexpected exception, expected<org.apache.hadoop.hbase.regionserver.wal.OrphanHLogAfterSplitException> but was(ExpectException.java:28)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
    at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:62)
    at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:140)
    at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:165)
    at org.apache.maven.surefire.Surefire.run(Surefire.java:107)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:289)
    at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1005)
    Caused by: java.lang.Error: Unresolved compilation problem:
    The method appendNoSync(HRegionInfo, byte[], WALEdit, long) is undefined for the type HLog
    {code}

    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: HBASE-1364.patch

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Ted Yu (JIRA) at Apr 8, 2011 at 5:24 pm
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017553#comment-13017553 ]

    Ted Yu commented on HBASE-1364:
    -------------------------------

    TestDistributedLogSplitting.testThreeRSAbort() consistently hangs on my laptop.
    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: HBASE-1364.patch

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • dhruba borthakur (JIRA) at Apr 8, 2011 at 5:28 pm
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017557#comment-13017557 ]

    dhruba borthakur commented on HBASE-1364:
    -----------------------------------------





    --
    Connect to me at http://www.facebook.com/dhruba

    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: HBASE-1364.patch

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Prakash Khemani (JIRA) at Apr 12, 2011 at 6:58 am
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018744#comment-13018744 ]

    Prakash Khemani commented on HBASE-1364:
    ----------------------------------------

    posted a revised patch at https://review.cloudera.org/r/1655/
    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: HBASE-1364.patch

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Prakash Khemani (JIRA) at Apr 12, 2011 at 6:30 pm
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018973#comment-13018973 ]

    Prakash Khemani commented on HBASE-1364:
    ----------------------------------------

    updated patch at https://review.cloudera.org/r/1655/
    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: HBASE-1364.patch

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • stack (JIRA) at Apr 14, 2011 at 11:41 pm
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020105#comment-13020105 ]

    stack commented on HBASE-1364:
    ------------------------------

    org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testWorkerAbort fails for me w/ latest patch:

    {code}
    -------------------------------------------------------------------------------
    Test set: org.apache.hadoop.hbase.master.TestDistributedLogSplitting
    -------------------------------------------------------------------------------
    Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 182.303 sec <<< FAILURE!
    testWorkerAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting) Time elapsed: 60.731 sec <<< FAILURE!
    java.lang.AssertionError: expected:<1> but was:(Assert.java:91)
    at org.junit.Assert.failNotEquals(Assert.java:645)
    at org.junit.Assert.assertEquals(Assert.java:126)
    at org.junit.Assert.assertEquals(Assert.java:470)
    at org.junit.Assert.assertEquals(Assert.java:454)
    at org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testWorkerAbort(TestDistributedLogSplitting.java:288)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
    at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:62)
    at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:140)
    at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:165)
    at org.apache.maven.surefire.Surefire.run(Surefire.java:107)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:289)
    at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1005)
    {code}
    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: HBASE-1364.patch

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • stack (JIRA) at Apr 14, 2011 at 11:58 pm
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020116#comment-13020116 ]

    stack commented on HBASE-1364:
    ------------------------------

    Did some review over on rb but not done yet. Will finish this evening.
    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: 1364-v5.txt, HBASE-1364.patch

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Prakash Khemani (JIRA) at Apr 15, 2011 at 7:16 am
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020216#comment-13020216 ]

    Prakash Khemani commented on HBASE-1364:
    ----------------------------------------

    TestDistributedLogSplitting.testWorkerAbort test failed because the SplitLogWorker was incrementing the tot_wkr_task_resigned twice. My test runs were passing because of a race - if the test happens to look at the counter between the 2 increments then the test will pass. Fixing this in the latest patch.


    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: 1364-v5.txt, HBASE-1364.patch

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • stack (JIRA) at Apr 15, 2011 at 7:58 am
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020229#comment-13020229 ]

    stack commented on HBASE-1364:
    ------------------------------

    @Prakash I uploaded more review. See if it helps.
    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: 1364-v5.txt, HBASE-1364.patch

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Prakash Khemani (JIRA) at Apr 15, 2011 at 11:52 pm
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020488#comment-13020488 ]

    Prakash Khemani commented on HBASE-1364:
    ----------------------------------------

    I uploaded a new diff at the review board https://review.cloudera.org/r/1655/

    I think it takes care of all of Stack's comments.

    added a new test in TestHLogSplit to test that when skip-errors is set to true then corrupted log files are ignored and correctly moved to the .corrupted directory.

    Some of the tests - especially in TestDistributedLogSplitting - are somewhat timing dependent. For example I will abort a few region servers and wait at most few seconds for all those servers to go down. Sometimes it takes longer and the test fails. Last night I had to bump up the time-limit in one such test (testThreeRSAbort()). I am sure these tests can be made more robust ....
    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: 1364-v5.txt, HBASE-1364.patch

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • stack (JIRA) at Apr 16, 2011 at 8:46 pm
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020657#comment-13020657 ]

    stack commented on HBASE-1364:
    ------------------------------

    Latest patch looks good but still fails TestDistributedLogSplitting it seems. I'm on apache hbase trunk and apply your r6 patch from review board. I get:

    {code}
    -------------------------------------------------------------------------------
    Test set: org.apache.hadoop.hbase.master.TestDistributedLogSplitting
    -------------------------------------------------------------------------------
    Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 164.652 sec <<< FAILURE!
    testWorkerAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting) Time elapsed: 48.77 sec <<< FAILURE!
    java.lang.AssertionError: region server completed the split before aborting
    at org.junit.Assert.fail(Assert.java:91)
    at org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testWorkerAbort(TestDistributedLogSplitting.java:291)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
    at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:62)
    at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:140)
    at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:165)
    at org.apache.maven.surefire.Surefire.run(Surefire.java:107)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:289)
    at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1005)
    {code}

    I'll attach the log.

    If you want me to dig in too Prakash, just say and I'll have a go. Otherwise defer to you since you know best whats in motion here.

    Meantime I'm going to try it out on a cluster. Good stuff Prakash!
    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: 1364-v5.txt, HBASE-1364.patch, org.apache.hadoop.hbase.master.TestDistributedLogSplitting-output.txt

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • stack (JIRA) at Apr 17, 2011 at 2:27 am
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020712#comment-13020712 ]

    stack commented on HBASE-1364:
    ------------------------------

    I tried it on a small cluster w/ loads of regions under load > 1k regions each. It worked for me. Its great. Fast.
    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: 1364-v5.txt, HBASE-1364.patch, org.apache.hadoop.hbase.master.TestDistributedLogSplitting-output.txt

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Prakash Khemani (JIRA) at Apr 17, 2011 at 4:33 am
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020721#comment-13020721 ]

    Prakash Khemani commented on HBASE-1364:
    ----------------------------------------

    This is a problem with the test-case which I will fix.

    You got this error because of (1) not being able to interrupt the split-log-worker thread when it is doing dfs operations (I think the interrupt is swallowed somewhere) (2) timing issues where in the aborting region server the filesystem and the zk session don't close before the split-log-worker thread completes its splitting task ...

    I will fix this by removing fail("region server completed the split before aborting") from the test case.




    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: 1364-v5.txt, HBASE-1364.patch, org.apache.hadoop.hbase.master.TestDistributedLogSplitting-output.txt

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Prakash Khemani (JIRA) at Apr 17, 2011 at 6:46 am
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020730#comment-13020730 ]

    Prakash Khemani commented on HBASE-1364:
    ----------------------------------------

    'fixed' TestDistriButedLogSplitting.testWorkerAbort() by not letting the test fail if the aborting region server completes the split before it closes dfs or zk session.

    uploaded a new patch in rb
    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: 1364-v5.txt, HBASE-1364.patch, org.apache.hadoop.hbase.master.TestDistributedLogSplitting-output.txt

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • stack (JIRA) at Apr 17, 2011 at 6:41 pm
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020842#comment-13020842 ]

    stack commented on HBASE-1364:
    ------------------------------

    This fails for me Prakash. Does it pass for you?

    {code}
    Running org.apache.hadoop.hbase.regionserver.TestSplitLogWorker
    Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.107 sec <<< FAILURE!
    {code}

    Here is the fail:

    {code}
    -------------------------------------------------------------------------------
    Test set: org.apache.hadoop.hbase.regionserver.TestSplitLogWorker
    -------------------------------------------------------------------------------
    Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.107 sec <<< FAILURE!
    testRaceForTask(org.apache.hadoop.hbase.regionserver.TestSplitLogWorker) Time elapsed: 0.182 sec <<< FAILURE!
    java.lang.AssertionError:
    at org.junit.Assert.fail(Assert.java:91)
    at org.junit.Assert.assertTrue(Assert.java:43)
    at org.junit.Assert.assertTrue(Assert.java:54)
    at org.apache.hadoop.hbase.regionserver.TestSplitLogWorker.waitForCounter(TestSplitLogWorker.java:75)
    at org.apache.hadoop.hbase.regionserver.TestSplitLogWorker.testRaceForTask(TestSplitLogWorker.java:165)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
    at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:62)
    at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:140)
    at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:165)
    at org.apache.maven.surefire.Surefire.run(Surefire.java:107)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:289)
    at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1005)
    {code}

    Let me know if you want me to post the log.

    Good stuff.
    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: 1364-v5.txt, HBASE-1364.patch, org.apache.hadoop.hbase.master.TestDistributedLogSplitting-output.txt

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Prakash Khemani (JIRA) at Apr 17, 2011 at 10:03 pm
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020874#comment-13020874 ]

    Prakash Khemani commented on HBASE-1364:
    ----------------------------------------

    Yes, it passes for me - just ran it again.

    This is another of timing related errors.


    164 waitForCounter(tot_wkr_task_acquired, 0, 1, 100);
    165 waitForCounter(tot_wkr_failed_to_grab_task_lost_race, 0, 1, 100);


    In your case the failure occurred when in line 165 the counter tot_wkr_failed_to_grab_task_lost_race did not change value from 0 to 1 in 100ms.

    Can you please increase the timeout in both these lines from 100ms to 1000ms and retry ...

    I will go over all my tests and try to improve them but I won't be able to get to that before the end of this week.





    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: 1364-v5.txt, HBASE-1364.patch, org.apache.hadoop.hbase.master.TestDistributedLogSplitting-output.txt

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • stack (JIRA) at Apr 18, 2011 at 5:16 pm
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021119#comment-13021119 ]

    stack commented on HBASE-1364:
    ------------------------------

    Hmm... i ran test on mac and linux and now it passes. I'll up the timer on commit anyways.
    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: 1364-v5.txt, HBASE-1364.patch, org.apache.hadoop.hbase.master.TestDistributedLogSplitting-output.txt

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Jonathan Gray (JIRA) at Apr 18, 2011 at 6:22 pm
    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021160#comment-13021160 ]

    Jonathan Gray commented on HBASE-1364:
    --------------------------------------

    Great work Prakash!
    [performance] Distributed splitting of regionserver commit logs
    ---------------------------------------------------------------

    Key: HBASE-1364
    URL: https://issues.apache.org/jira/browse/HBASE-1364
    Project: HBase
    Issue Type: Improvement
    Components: coprocessors
    Reporter: stack
    Assignee: Prakash Khemani
    Priority: Critical
    Fix For: 0.92.0

    Attachments: 1364-v5.txt, HBASE-1364.patch, org.apache.hadoop.hbase.master.TestDistributedLogSplitting-output.txt

    Time Spent: 8h
    Remaining Estimate: 0h

    HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs to run even faster.
    (Below is from HBASE-1008)
    In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need to distribute or at least multithread the splitting.
    1. As is, regions starting up expect to find one reconstruction log only. Need to make it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in an output directory written by all split participants whether multithreaded or a mapreduce-like distributed process (Lets write our distributed sort first as a MR so we learn whats involved; distributed sort, as much as possible should use MR framework pieces). On startup, regions go to this directory and pick up the files written by split participants deleting and clearing the dir when all have been read in. Making it so can take multiple logs for input, can also make the split process more robust rather than current tenuous process which loses all edits if it doesn't make it to the end without error.
    2. Each column family rereads the reconstruction log to find its edits. Need to fix that. Split can sort the edits by column family so store only reads its edits.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post

1 user in discussion

Jonathan Gray (JIRA): 19 posts