FAQ
Reduce often attempts in memory merge with no work
--------------------------------------------------

Key: HADOOP-3940
URL: https://issues.apache.org/jira/browse/HADOOP-3940
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Chris Douglas
Priority: Minor


ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests < numRequiredMapOutputs}}, is false until the first map output is fetched and false again after the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Chris Douglas (JIRA) at Aug 12, 2008 at 9:18 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Chris Douglas updated HADOOP-3940:
    ----------------------------------

    Attachment: 3940-0.patch

    Simplest fix, waiting on numRequiredMapOutputs == 0.

    Initializing numRequiredMapOutputs to Integer.MAX_VALUE would require a different check at the end; this should handle both.
    Reduce often attempts in memory merge with no work
    --------------------------------------------------

    Key: HADOOP-3940
    URL: https://issues.apache.org/jira/browse/HADOOP-3940
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Chris Douglas
    Priority: Minor
    Attachments: 3940-0.patch


    ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests < numRequiredMapOutputs}}, is false until the first map output is fetched and false again after the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Chris Douglas (JIRA) at Aug 12, 2008 at 10:00 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Chris Douglas updated HADOOP-3940:
    ----------------------------------

    Status: Patch Available (was: Open)
    Reduce often attempts in memory merge with no work
    --------------------------------------------------

    Key: HADOOP-3940
    URL: https://issues.apache.org/jira/browse/HADOOP-3940
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Chris Douglas
    Priority: Minor
    Attachments: 3940-0.patch


    ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests < numRequiredMapOutputs}}, is false until the first map output is fetched and false again after the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Chris Douglas (JIRA) at Aug 12, 2008 at 10:08 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622004#action_12622004 ]

    Chris Douglas commented on HADOOP-3940:
    ---------------------------------------

    Alternatively, I think this could equivalently wait on {{numPendingRequests <= numRequiredMapOutputs}}, relying on close to signal the last merge
    Reduce often attempts in memory merge with no work
    --------------------------------------------------

    Key: HADOOP-3940
    URL: https://issues.apache.org/jira/browse/HADOOP-3940
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Chris Douglas
    Priority: Minor
    Attachments: 3940-0.patch


    ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests < numRequiredMapOutputs}}, is false until the first map output is fetched and false again after the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Chris Douglas (JIRA) at Aug 12, 2008 at 10:20 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622004#action_12622004 ]

    chris.douglas edited comment on HADOOP-3940 at 8/12/08 3:19 PM:
    ----------------------------------------------------------------

    [ edit - nevermind ]

    was (Author: chris.douglas):
    Alternatively, I think this could equivalently wait on {{numPendingRequests <= numRequiredMapOutputs}}, relying on close to signal the last merge
    Reduce often attempts in memory merge with no work
    --------------------------------------------------

    Key: HADOOP-3940
    URL: https://issues.apache.org/jira/browse/HADOOP-3940
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Chris Douglas
    Priority: Minor
    Attachments: 3940-0.patch


    ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests < numRequiredMapOutputs}}, is false until the first map output is fetched and false again after the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Aug 13, 2008 at 3:45 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622256#action_12622256 ]

    Hadoop QA commented on HADOOP-3940:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12388084/3940-0.patch
    against trunk revision 685425.

    +1 @author. The patch does not contain any @author tags.

    -1 tests included. The patch doesn't appear to include any new or modified tests.
    Please justify why no tests are needed for this patch.

    -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

    +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs. The patch does not introduce any new Findbugs warnings.

    +1 release audit. The applied patch does not increase the total number of release audit warnings.

    -1 core tests. The patch failed core unit tests.

    -1 contrib tests. The patch failed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3058/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3058/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3058/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3058/console

    This message is automatically generated.
    Reduce often attempts in memory merge with no work
    --------------------------------------------------

    Key: HADOOP-3940
    URL: https://issues.apache.org/jira/browse/HADOOP-3940
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Chris Douglas
    Priority: Minor
    Attachments: 3940-0.patch


    ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests < numRequiredMapOutputs}}, is false until the first map output is fetched and false again after the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Chris Douglas (JIRA) at Aug 13, 2008 at 9:45 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Chris Douglas updated HADOOP-3940:
    ----------------------------------

    Attachment: 3940-1.patch

    Added some comments to waitForDataToMerge, particularly explaining why numRequiredMapOutputs == 0 is a special case, per Arun's suggestion.

    Test failures are due to HADOOP-3946.
    Reduce often attempts in memory merge with no work
    --------------------------------------------------

    Key: HADOOP-3940
    URL: https://issues.apache.org/jira/browse/HADOOP-3940
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Chris Douglas
    Priority: Minor
    Attachments: 3940-0.patch, 3940-1.patch


    ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests < numRequiredMapOutputs}}, is false until the first map output is fetched and false again after the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Arun C Murthy (JIRA) at Aug 13, 2008 at 10:15 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Arun C Murthy reassigned HADOOP-3940:
    -------------------------------------

    Assignee: Chris Douglas

    +1
    Reduce often attempts in memory merge with no work
    --------------------------------------------------

    Key: HADOOP-3940
    URL: https://issues.apache.org/jira/browse/HADOOP-3940
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Chris Douglas
    Assignee: Chris Douglas
    Priority: Minor
    Attachments: 3940-0.patch, 3940-1.patch


    ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests < numRequiredMapOutputs}}, is false until the first map output is fetched and false again after the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Chris Douglas (JIRA) at Aug 13, 2008 at 10:41 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Chris Douglas updated HADOOP-3940:
    ----------------------------------

    Resolution: Fixed
    Fix Version/s: 0.19.0
    Hadoop Flags: [Reviewed]
    Status: Resolved (was: Patch Available)

    I just committed this
    Reduce often attempts in memory merge with no work
    --------------------------------------------------

    Key: HADOOP-3940
    URL: https://issues.apache.org/jira/browse/HADOOP-3940
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Chris Douglas
    Assignee: Chris Douglas
    Priority: Minor
    Fix For: 0.19.0

    Attachments: 3940-0.patch, 3940-1.patch


    ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests < numRequiredMapOutputs}}, is false until the first map output is fetched and false again after the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Chris Douglas (JIRA) at Aug 13, 2008 at 10:51 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Chris Douglas updated HADOOP-3940:
    ----------------------------------

    Affects Version/s: 0.18.0
    Reduce often attempts in memory merge with no work
    --------------------------------------------------

    Key: HADOOP-3940
    URL: https://issues.apache.org/jira/browse/HADOOP-3940
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.18.0
    Reporter: Chris Douglas
    Assignee: Chris Douglas
    Priority: Minor
    Fix For: 0.19.0

    Attachments: 3940-0.patch, 3940-1.patch


    ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests < numRequiredMapOutputs}}, is false until the first map output is fetched and false again after the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hudson (JIRA) at Aug 22, 2008 at 12:37 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624712#action_12624712 ]

    Hudson commented on HADOOP-3940:
    --------------------------------

    Integrated in Hadoop-trunk #581 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/])
    Reduce often attempts in memory merge with no work
    --------------------------------------------------

    Key: HADOOP-3940
    URL: https://issues.apache.org/jira/browse/HADOOP-3940
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.18.0
    Reporter: Chris Douglas
    Assignee: Chris Douglas
    Priority: Minor
    Fix For: 0.19.0

    Attachments: 3940-0.patch, 3940-1.patch


    ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests < numRequiredMapOutputs}}, is false until the first map output is fetched and false again after the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Owen O'Malley (JIRA) at Sep 5, 2008 at 4:01 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628660#action_12628660 ]

    Owen O'Malley commented on HADOOP-3940:
    ---------------------------------------

    There was a suggestion on the list to include this in 0.18.1. That sounds right to me. Thoughts?
    Reduce often attempts in memory merge with no work
    --------------------------------------------------

    Key: HADOOP-3940
    URL: https://issues.apache.org/jira/browse/HADOOP-3940
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.18.0
    Reporter: Chris Douglas
    Assignee: Chris Douglas
    Priority: Minor
    Fix For: 0.19.0

    Attachments: 3940-0.patch, 3940-1.patch


    ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests < numRequiredMapOutputs}}, is false until the first map output is fetched and false again after the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Chris Douglas (JIRA) at Sep 5, 2008 at 10:57 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Chris Douglas updated HADOOP-3940:
    ----------------------------------

    Fix Version/s: 0.18.1

    +1

    Merged into 0.18 branch
    Reduce often attempts in memory merge with no work
    --------------------------------------------------

    Key: HADOOP-3940
    URL: https://issues.apache.org/jira/browse/HADOOP-3940
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.18.0
    Reporter: Chris Douglas
    Assignee: Chris Douglas
    Priority: Minor
    Fix For: 0.18.1, 0.19.0

    Attachments: 3940-0.patch, 3940-1.patch


    ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests < numRequiredMapOutputs}}, is false until the first map output is fetched and false again after the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hudson (JIRA) at Sep 6, 2008 at 1:26 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628865#action_12628865 ]

    Hudson commented on HADOOP-3940:
    --------------------------------

    Integrated in Hadoop-trunk #595 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/595/])
    Reduce often attempts in memory merge with no work
    --------------------------------------------------

    Key: HADOOP-3940
    URL: https://issues.apache.org/jira/browse/HADOOP-3940
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.18.0
    Reporter: Chris Douglas
    Assignee: Chris Douglas
    Priority: Minor
    Fix For: 0.18.1, 0.19.0

    Attachments: 3940-0.patch, 3940-1.patch


    ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests < numRequiredMapOutputs}}, is false until the first map output is fetched and false again after the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedAug 12, '08 at 9:00p
activeSep 6, '08 at 1:26p
posts14
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Hudson (JIRA): 14 posts

People

Translate

site design / logo © 2022 Grokbase