[Hadoop-common-dev] [jira] Created: (HADOOP-6048) Need to handle access token expiration when re-establishing the pipeline for dfs write

[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6048:
------------------------------

Attachment: 6048-09.patch

this patch does the following.
1. when the primary datanode recovers a block, it sends back a newly generated access token along a new generation stamp if any.
2. add a namenode method so that a DFSClient can call to get a new access token when access token becomes expired during datanode error recovery.
3. To differentiate access token error from other errors, datanode now needs to send back a status code together with firstBadLink during pipeline setup for dfs write (only for DFSClients, not for other datanodes).

[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6048:
------------------------------

Status: Patch Available (was: Open)

[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6048:
------------------------------

Attachment: 6048-11.patch

updated ClientProtocol version.

[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6048:
------------------------------

Attachment: 6048-12.patch

attaching a new patch, which relies on block recovery to always return a newly generated access token, so that there is no need for adding a new method to ClientProtocol to get access token.

[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6048:
------------------------------

Attachment: 6048-14.patch

minor updates

[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720494#action_12720494 ]

Kan Zhang commented on HADOOP-6048:
-----------------------------------

To be clear to the reader, the last patch removed null checking on the return value from recoverBlock() since it always returns a LocatedBlock object or throw an exception.

[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6048:
------------------------------

Attachment: 6048-15.patch

one cosmetic change.

[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6048:
------------------------------

Attachment: 6048-16.patch

attached a new patch. removed the testing for InvalidAccessTokenException in processDatanodeError() since it can never happen. processDatanodeError() is called in 2 places. 1) when an error happens while writing. since the pipeline has already been set up in this case, the error can't be access token error. 2) when initializing for append, in which case, there is no previous error yet.

[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6048:
------------------------------

Status: Patch Available (was: Open)

[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6048:
------------------------------

Status: Open (was: Patch Available)

[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6048:
------------------------------

Attachment: 6048-17.patch

[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6048:
------------------------------

Attachment: 6048-18.patch

[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721606#action_12721606 ]

Kan Zhang commented on HADOOP-6048:
-----------------------------------

attached a new patch, which bumps up the ClientDatanodeProtocol version to signal the fact that now recoverBlock() never returns null and always returns a newly generated access token as part of the return value. Also, reverted the changes made in previous patches that changed the way how lastException is set by sync() and createBlockOutputStream().

[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721724#action_12721724 ]

Hadoop QA commented on HADOOP-6048:
-----------------------------------

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12411172/6048-18.patch
against trunk revision 786278.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/console

This message is automatically generated.