[Hadoop-common-dev] [jira] Created: (HADOOP-1932) Test dfs.TestFileCreation.testFileCreation failed on Windows

[ https://issues.apache.org/jira/browse/HADOOP-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur reassigned HADOOP-1932:
----------------------------------------

Assignee: dhruba borthakur

[ https://issues.apache.org/jira/browse/HADOOP-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-1932:
-------------------------------------

Affects Version/s: (was: 0.14.2)
0.15.0

[ https://issues.apache.org/jira/browse/HADOOP-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-1932:
-------------------------------------

Priority: Blocker (was: Major)

[ https://issues.apache.org/jira/browse/HADOOP-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-1932:
-------------------------------------

Attachment: fileCloseTest.patch

The test was creating a file with two blocks. After writing to the two blocks (and before closing the file), it was verfying that the file size was 2 blocks. This wasn't always true because the client might not have flushed/closed the last block.

The fix is to make the test create a file that is slightly larger than 2 blocks. When the client finished writing to the whole file, it is guaranteed that the first two blocks have been flushed to the datanodes.

[ https://issues.apache.org/jira/browse/HADOOP-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529935 ]

Raghu Angadi commented on HADOOP-1932:
--------------------------------------

Isn't it true that no blocks are guaranteed to be present unless the file is closed? Adding one more byte, only makes the issue less probable, right?

Flusing to datanodes is guaranteed (by implementation, not by contract) even while writing exactly two blocks. What is not guaranteed is datanodes reporting the block to Namenode.

[ https://issues.apache.org/jira/browse/HADOOP-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529941 ]

dhruba borthakur commented on HADOOP-1932:
------------------------------------------

It is *guaranteed* that the namenode will allocate a new block for the file only if the penultimate block has been confirmed by at least the min number of datanodes. That shud take care of the issue that you mention.

[ https://issues.apache.org/jira/browse/HADOOP-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529956 ]

Raghu Angadi commented on HADOOP-1932:
--------------------------------------

Thats right. Thanks.

[ https://issues.apache.org/jira/browse/HADOOP-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530009 ]

Nigel Daley commented on HADOOP-1932:
-------------------------------------

Also occurs on Solaris:
http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/817/testReport/org.apache.hadoop.dfs/TestFileCreation/testFileCreation/

[ https://issues.apache.org/jira/browse/HADOOP-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-1932:
-------------------------------------

Status: Patch Available (was: Open)

[ https://issues.apache.org/jira/browse/HADOOP-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530037 ]

Hadoop QA commented on HADOOP-1932:
-----------------------------------

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12366478/fileCloseTest.patch
against trunk revision r579005.

@author +1. The patch does not contain any @author tags.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new compiler warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests -1. The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/820/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/820/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/820/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/820/console

This message is automatically generated.

[ https://issues.apache.org/jira/browse/HADOOP-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-1932:
-------------------------------------

Resolution: Fixed
Fix Version/s: 0.15.0
Status: Resolved (was: Patch Available)

I just committed this.

[ https://issues.apache.org/jira/browse/HADOOP-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530241 ]

Hudson commented on HADOOP-1932:
--------------------------------

Integrated in Hadoop-Nightly #250 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/250/])