FAQ
stack created HBASE-14559:
-----------------------------

              Summary: branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
                  Key: HBASE-14559
                  URL: https://issues.apache.org/jira/browse/HBASE-14559
              Project: HBase
           Issue Type: Sub-task
             Reporter: stack
             Assignee: stack


Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Search Discussions

  • stack (JIRA) at Oct 6, 2015 at 5:39 am
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    stack updated HBASE-14559:
    --------------------------
         Attachment: 14559.branch-1.txt
                     14559.master.txt

    What I pushed to branch-1, branch-1.2 and to master (master is a subset)
    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • stack (JIRA) at Oct 6, 2015 at 5:39 am
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    stack resolved HBASE-14559.
    ---------------------------
            Resolution: Fixed
         Fix Version/s: 1.3.0
                        1.2.0
                        2.0.0
    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mikhail Antonov (JIRA) at Oct 6, 2015 at 5:49 am
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944542#comment-14944542 ]

    Mikhail Antonov commented on HBASE-14559:
    -----------------------------------------

    Is that regression from jira when we made superuser's requests go thru high-priority rpc thread pool?
    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Elliott Clark (JIRA) at Oct 6, 2015 at 6:09 am
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944553#comment-14944553 ]

    Elliott Clark commented on HBASE-14559:
    ---------------------------------------

    bq.Is that regression from jira when we made superuser's requests go thru high-priority rpc thread pool?
    That already had to be backed out a while ago for master. I wonder if we should just back it out completely ?
    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mikhail Antonov (JIRA) at Oct 6, 2015 at 6:16 am
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944558#comment-14944558 ]

    Mikhail Antonov commented on HBASE-14559:
    -----------------------------------------

    Hm? I didn't know it was reverted on master. Perhaps I missed that jira?

    HBASE-13375 / HBASE-13351? These 2 have to do with rpc priority.
    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Elliott Clark (JIRA) at Oct 6, 2015 at 6:25 am
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944564#comment-14944564 ]

    Elliott Clark commented on HBASE-14559:
    ---------------------------------------

    HBASE-13635
    HBASE-14322
    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mikhail Antonov (JIRA) at Oct 6, 2015 at 6:27 am
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944566#comment-14944566 ]

    Mikhail Antonov commented on HBASE-14559:
    -----------------------------------------

    Oh i see, thanks for pointer.
    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mikhail Antonov (JIRA) at Oct 6, 2015 at 7:06 am
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944604#comment-14944604 ]

    Mikhail Antonov commented on HBASE-14559:
    -----------------------------------------

    I think there're several things here (or maybe I'm missing something).

    Conceptually it looks logical to me that on large cluster overall responsiveness to admin commands would be improved by serving admin commands from separate threadpool (is that actually right? if not, we can revert it completely?). There were some deadlock-type bugs (these 2 referenced by [~eclark]), but that should be fixed? In theory having yet one more threadpool to separate admin commands from things like meta lookups might help prevent it, but looks overkill.

    For the tests, these features are revealed in corner case, as in minicluster everything is running as admin, so admin threadpool is overloaded. On current master as I'm seeing in HConstants:

    bq. public static final int DEFAULT_REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT = 20;

    as was set in HBASE-13351.

    [~stack] so in these 2 oneliners, you're dropping down the number of high priority handlers (to tighten up thread usage?) right?



    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Hudson (JIRA) at Oct 6, 2015 at 8:09 am
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944668#comment-14944668 ]

    Hudson commented on HBASE-14559:
    --------------------------------

    FAILURE: Integrated in HBase-1.2 #229 (See [https://builds.apache.org/job/HBase-1.2/229/])
    HBASE-14559 branch-1 test tweeks; disable assert explicit region lands (stack: rev e568cda7f05880577b27a8e693f8788cae372596)
    * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java
    * hbase-server/src/test/java/org/apache/hadoop/hbase/master/handler/TestEnableTableHandler.java
    * hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithACL.java
    * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java

    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Hudson (JIRA) at Oct 6, 2015 at 8:38 am
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944702#comment-14944702 ]

    Hudson commented on HBASE-14559:
    --------------------------------

    SUCCESS: Integrated in HBase-1.2-IT #192 (See [https://builds.apache.org/job/HBase-1.2-IT/192/])
    HBASE-14559 branch-1 test tweeks; disable assert explicit region lands (stack: rev e568cda7f05880577b27a8e693f8788cae372596)
    * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
    * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java
    * hbase-server/src/test/java/org/apache/hadoop/hbase/master/handler/TestEnableTableHandler.java
    * hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithACL.java

    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Hudson (JIRA) at Oct 6, 2015 at 8:54 am
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944733#comment-14944733 ]

    Hudson commented on HBASE-14559:
    --------------------------------

    SUCCESS: Integrated in HBase-1.3-IT #212 (See [https://builds.apache.org/job/HBase-1.3-IT/212/])
    HBASE-14559 branch-1 test tweeks; disable assert explicit region lands (stack: rev 80961187aa7053d886c88be56311b88a4e02d28f)
    * hbase-server/src/test/java/org/apache/hadoop/hbase/master/handler/TestEnableTableHandler.java
    * hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithACL.java
    * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java
    * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java

    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Hudson (JIRA) at Oct 6, 2015 at 8:55 am
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944737#comment-14944737 ]

    Hudson commented on HBASE-14559:
    --------------------------------

    FAILURE: Integrated in HBase-1.3 #237 (See [https://builds.apache.org/job/HBase-1.3/237/])
    HBASE-14559 branch-1 test tweeks; disable assert explicit region lands (stack: rev 80961187aa7053d886c88be56311b88a4e02d28f)
    * hbase-server/src/test/java/org/apache/hadoop/hbase/master/handler/TestEnableTableHandler.java
    * hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithACL.java
    * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java
    * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java

    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • stack (JIRA) at Oct 6, 2015 at 3:29 pm
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945189#comment-14945189 ]

    stack commented on HBASE-14559:
    -------------------------------

    Thanks for noticing lads.

    I'm running a little rig here w/ branch-1. The tests in the patch pass with the one-liner (upped priorities) where w/o they failed near 100% of the time. At this stage in the zombie stomping session, I'm -- ahem -- a little less concerned about root cause of a failure and more about just getting stuff going again so did not spend much time on why these fixes are needed in branch-1 and not on master.

    bq. stack so in these 2 oneliners, you're dropping down the number of high priority handlers (to tighten up thread usage?) right?

    No sir. HBASE-14290 set the number of handlers down when I noticed tests with 500 threads running... that failed to run on my local machine because OOME, could not create thread. So, here, I'm upping the handlers on a few tests. I'd already done a pass on master -- a few tests there needed more handlers or they hung (we need to fix!) -- so was a bit surprised this necessary in branch-1 but it looks like you and [~eclark] have identified why.

    We should revert HBASE-13635 and HBASE-14322 from branch-1?

    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Elliott Clark (JIRA) at Oct 6, 2015 at 4:44 pm
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945329#comment-14945329 ]

    Elliott Clark commented on HBASE-14559:
    ---------------------------------------

    bq.We should revert HBASE-13635 and HBASE-14322 from branch-1?
    Nope those two jiras are partial reverts of HBASE-13375. I'm asking if we should remove HBASE-13375 completely. Since we've had to remove it when requests are going to master and we are upping the number of threads that a regionserver needs.
    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • stack (JIRA) at Oct 6, 2015 at 7:14 pm
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945602#comment-14945602 ]

    stack commented on HBASE-14559:
    -------------------------------

    These fixes here made it so branch-1 now passes on my internal rig.

    I can't speak to whether we should remove HBASE-13375 completely. It would seem to explain why some of the tweaks here were necessary in branch-1 but not in master -- that helps.
    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mikhail Antonov (JIRA) at Oct 6, 2015 at 7:30 pm
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945623#comment-14945623 ]

    Mikhail Antonov commented on HBASE-14559:
    -----------------------------------------

    Thanks for clarifying [~stack]

    regarding tests - I remember we upped the number of high priority handler used in all tests (and I admit to have suggested/supported that :), as after finding several tests where default number of thread wasn't enough after changes made in HBase-13375 bumping the default number of threads looked easy thing to do). Should we also lower DEFAULT_REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT back to 10 as default (it's 20 now on master, as I see, and set higher only in selected tests?)

    Regarding removing completely - I guess that's up to the judgement of folks running big clusters. The way we treat admin user requests _seems_ logical to me, but I definitely don't want to argue with production observations using logical conclusions :)
    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • stack (JIRA) at Oct 6, 2015 at 7:39 pm
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945634#comment-14945634 ]

    stack commented on HBASE-14559:
    -------------------------------

    bq. Should we also lower DEFAULT_REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT back to 10 as default (it's 20 now on master, as I see, and set higher only in selected tests?)

    I'm just looking at tests at the moment. I saw priority handlers set to 40 in a few instances which seemed excessive. Other tests with many regions had reams of handlers just sitting there doing nothing clouding thread dumps where i was trying to figure why the test was hung...

    bq. The way we treat admin user requests seems logical to me, but I definitely don't want to argue with production observations using logical conclusions

    I'm with you. Lets get other input.
    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mikhail Antonov (JIRA) at Oct 6, 2015 at 9:02 pm
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945789#comment-14945789 ]

    Mikhail Antonov commented on HBASE-14559:
    -----------------------------------------

    [~stack] when you traced the tests using excessive number of threads, did they timeout because they run slower with lower number of threads, or did they deadlock?

    I think there're still bugs lurking around in the implementation :( If we have number of thread handlers 3 rather than 40, I might expect things running noticeably slower, but not the deadlock?
    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • stack (JIRA) at Oct 6, 2015 at 9:34 pm
    [ https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945864#comment-14945864 ]

    stack commented on HBASE-14559:
    -------------------------------

    bq. stack when you traced the tests using excessive number of threads, did they timeout because they run slower with lower number of threads, or did they deadlock?

    At the extreme, the test would not run (OOME could not create native thread). I cut the thread count down and then they would not complete (smile).

    bq. I think there're still bugs lurking around in the implementation If we have number of thread handlers 3 rather than 40, I might expect things running noticeably slower, but not the deadlock?

    The deadlock was handlers all occupied.. not enough for the test to complete.
    branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers
    ---------------------------------------------------------------------------------------------

    Key: HBASE-14559
    URL: https://issues.apache.org/jira/browse/HBASE-14559
    Project: HBase
    Issue Type: Sub-task
    Components: test
    Reporter: stack
    Assignee: stack
    Fix For: 2.0.0, 1.2.0, 1.3.0

    Attachments: 14559.branch-1.txt, 14559.master.txt


    Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). Small tweaks get tests to pass. Small one liners that up priority handler count and disable assert that seems wrong -- that we'll always get an explcit region to land on a newly started server.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupissues @
categorieshbase, hadoop
postedOct 6, '15 at 5:33a
activeOct 6, '15 at 9:34p
posts20
users1
websitehbase.apache.org

1 user in discussion

stack (JIRA): 20 posts

People

Translate

site design / logo © 2021 Grokbase