[Hadoop-common-dev] [jira] Created: (HADOOP-6001) Incorporate storage directories into EditLogFileInput/Output streams

[ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717748#action_12717748 ]

Luca Telloli commented on HADOOP-6001:
--------------------------------------

StorageDirectory is currently a class used to manage both FSIMAGE and edits.
When used for edits, it could be incorporated into the EditLogFileInput/Output classes.
This should help the development of HADOOP-5188

[ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luca Telloli updated HADOOP-6001:
---------------------------------

Attachment: HADOOP-6001.patch

Initial patch.
- Now EditLogFileOutputStream handles StorageDirectory directly.
- Previous methods to revert and divert file streams have been moved inside this class, since they're file related
- The header of some methods in FSEditLog has been replaced by one which handles storage directories instead of files
- Method FSEditLog.revertFileStreams() has been removed.

Tested for core-tests matching namenode/Test* and it passed.

[ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luca Telloli updated HADOOP-6001:
---------------------------------

Status: Patch Available (was: Open)

[ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719911#action_12719911 ]

Hadoop QA commented on HADOOP-6001:
-----------------------------------

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12410418/HADOOP-6001.patch
against trunk revision 785025.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

-1 javadoc. The javadoc tool appears to have generated 1 warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/501/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/501/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/501/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/501/console

This message is automatically generated.

[ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720107#action_12720107 ]

Luca Telloli commented on HADOOP-6001:
--------------------------------------

As far as I understand from the console output of Hudson

- the javadoc warning is not related directly to the content of the patch
{noformat}
[exec] [javadoc] javadoc: warning - Error fetching URL: http://java.sun.com/javase/6/docs/api/package-list
{noformat}

- the contrib test failing is the one related to HADOOP-6007

so this patch to me seems ready to be reviewed and eventually committed

[ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720107#action_12720107 ]

Luca Telloli edited comment on HADOOP-6001 at 6/16/09 6:04 AM:
---------------------------------------------------------------

As far as I understand from the console output of Hudson

- _if I'm not mistaken_, the javadoc warning is not related directly to the content of the patch
{noformat}
[exec] [javadoc] javadoc: warning - Error fetching URL: http://java.sun.com/javase/6/docs/api/package-list
{noformat}

- the contrib test failing is the one related to HADOOP-6007

so this patch to me seems ready to be reviewed and eventually committed

was (Author: lucat):
As far as I understand from the console output of Hudson

- the javadoc warning is not related directly to the content of the patch
{noformat}
[exec] [javadoc] javadoc: warning - Error fetching URL: http://java.sun.com/javase/6/docs/api/package-list
{noformat}

- the contrib test failing is the one related to HADOOP-6007

so this patch to me seems ready to be reviewed and eventually committed

[ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luca Telloli updated HADOOP-6001:
---------------------------------

Attachment: HADOOP-6001.patch

Added some Javadoc documentation. No real code has been modified.

[ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721842#action_12721842 ]

Benjamin Reed commented on HADOOP-6001:
---------------------------------------

the patch looks fine, but you haven't motivated it. it looks like you are just doing code cleanup. can you explain why you must do this to plug in bookkeeper?

[ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721855#action_12721855 ]

Luca Telloli commented on HADOOP-6001:
--------------------------------------

I would like to avoid in HADOOP-5189 situations like:

if (stream is of type Bookkeeper)
do this
else
do that

Code in FSEditlog is mostly file related and many methods (rollEditLog, purgeEditLog) manipulate files. Storage directories, for instance, are an example of file-based logging.

This patch should help to isolate possibly all situations where the file-dependent calls are made in FSEditLog.

I was also hoping that dividing the steps in multiple independent sub-patches for 5188/5189 would speed up commit

[ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721887#action_12721887 ]

Luca Telloli commented on HADOOP-6001:
--------------------------------------

I guess we need some additional clarification on the purpose of this patch and on the interrelationship between this one and the other ones we're working on (5188, 5189, 5832, etc).

The overall goal of 5188 is simple: to have the EditLogInput/OuputStream to properly incapsulate everything is needed for log of NameNode operations, and FSEditLog to call the same methods independently of the type of stream it's working on.

Then, with 5832 I moved a step towards this goal: to treat the properties in the .xml file which specify the log devices and their parameters as a URI. While deploying that patch I realized that I couldn't pass a URI as argument to EditLogFileInput/Output Stream, because the constructor was dependent on the role of the file. By role of the file, I mean edits or edits.new, two different files used in file-based logging to take care of changes. You can see that if you look at the original code for FSEditLog, when EditLogFileOutputStream is instantiated at times with the EDITS file, and some other times with the EDITS_NEW file. This is bad, because it breaks generality.

With some simple modifications, contained in this patch, I can now pass to EditLogFileOutputStream the container of these files, which is a StorageDirectory and, depending on the call from FSEditLog, I can switch between edits and edits.new inside EditLogFileOutputStream, making it transparent to FSEditLog which doesn't have to instantiate this class based on a specific file and its role.
At the same time, a StorageDirectory is exactly what is passed inside the URI of a file-based logger, which connects this patch to 5832.

I hope this clarifies some of the doubt on this patch, but I'm happy to answer more questions if they come up.