[Hadoop-common-dev] [jira] Created: (HADOOP-5895) Log message shows -ve number of bytes to be merged in the final merge pass when there are no intermediate merges and merge factor is < number of segments

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-5895:
---------------------------------

Attachment: HADOOP-5895.patch

In computeBytesInMerges, getPassFactor() can return factor < number of segments and even though n++ is done based on condition includeFinalMerge, that is not good enough for the while loop to be executed.

Attaching patch with the fix. Please review and provide your comments.

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-5895:
---------------------------------

Description: Log message shows -ve number of bytes to be merged in the final merge pass when there are no intermediate merges and the mergeFactor is > total number of segments to be merged. This issue is because of code changes done in HADOOP-5572. (was: Log message shows -ve number of bytes to be merged in the final merge pass when there are no intermediate merges and the mergeFactor is < total number of segments to be merged. This issue is because of code changes done in HADOOP-5572.)
Summary: Log message shows -ve number of bytes to be merged in the final merge pass when there are no intermediate merges and merge factor is > number of segments (was: Log message shows -ve number of bytes to be merged in the final merge pass when there are no intermediate merges and merge factor is < number of segments)

Issue is when mergeFactor > number of segments (not less than).

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712629#action_12712629 ]

Jothi Padmanabhan commented on HADOOP-5895:
-------------------------------------------

Using a flag to control the number of iterations in includeFinalMerge would be more readable than modifying the variable n itself, no?

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-5895:
---------------------------------

Attachment: HADOOP-5895.v1.patch

Agreed Jothi.
Here is the patch with a flag doing the work of considering final merge as part of calculation of estimated input bytes of merges.

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712684#action_12712684 ]

Jothi Padmanabhan commented on HADOOP-5895:
-------------------------------------------

+1

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-5895:
---------------------------------

Release Note: This patch fixes the issue of final merge not getting included in the calculation of estimated bytes in merges in map task when the merge factor is > total number of segments.
Status: Patch Available (was: Open)

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-5895:
---------------------------------

Status: Open (was: Patch Available)

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-5895:
---------------------------------

Attachment: HADOOP-5895.v1.1.patch

Attaching a new patch fixing a minor issue for the same log message.

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-5895:
---------------------------------

Status: Patch Available (was: Open)

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711990#action_12711990 ]

Ravi Gummadi edited comment on HADOOP-5895 at 5/25/09 9:01 PM:
---------------------------------------------------------------

In computeBytesInMerges, getPassFactor() can return factor > number of segments and even though n++ is done based on condition includeFinalMerge, that is not good enough for the while loop to be executed.

Attaching patch with the fix. Please review and provide your comments.

was (Author: ravidotg):
In computeBytesInMerges, getPassFactor() can return factor < number of segments and even though n++ is done based on condition includeFinalMerge, that is not good enough for the while loop to be executed.

Attaching patch with the fix. Please review and provide your comments.

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712852#action_12712852 ]

Ravi Gummadi commented on HADOOP-5895:
--------------------------------------

Unit tests passed on my machine.

ant test-patch gave:

[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713397#action_12713397 ]

Jothi Padmanabhan commented on HADOOP-5895:
-------------------------------------------

+1 for the last patch

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713566#action_12713566 ]

Hadoop QA commented on HADOOP-5895:
-----------------------------------

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12408983/HADOOP-5895.v1.1.patch
against trunk revision 779066.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/411/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/411/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/411/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/411/console

This message is automatically generated.

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713631#action_12713631 ]

Ravi Gummadi commented on HADOOP-5895:
--------------------------------------

Unit tests failures shown above are not related to this patch.

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-5895:
--------------------------------

Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)

I just committed this. Thanks, Ravi!

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715376#action_12715376 ]

Ravi Gummadi commented on HADOOP-5895:
--------------------------------------

I dont think we need a new testcase for this as the issue is seen in the log message itself just before final merge in map tasks. So I haven't added new testcase.

[ https://issues.apache.org/jira/browse/HADOOP-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715377#action_12715377 ]

Ravi Gummadi commented on HADOOP-5895:
--------------------------------------

Also, tested manually with many jobs like loadgen, sort, wordcount(in addition to unit tests) to check if -ve number can be seen in the log message in Merger.