FAQ
Incorporate storage directories into EditLogFileInput/Output streams
--------------------------------------------------------------------

Key: HADOOP-6001
URL: https://issues.apache.org/jira/browse/HADOOP-6001
Project: Hadoop Core
Issue Type: Sub-task
Reporter: Luca Telloli




--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Luca Telloli (JIRA) at Jun 9, 2009 at 5:03 pm
    [ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717748#action_12717748 ]

    Luca Telloli commented on HADOOP-6001:
    --------------------------------------

    StorageDirectory is currently a class used to manage both FSIMAGE and edits.
    When used for edits, it could be incorporated into the EditLogFileInput/Output classes.
    This should help the development of HADOOP-5188
    Incorporate storage directories into EditLogFileInput/Output streams
    --------------------------------------------------------------------

    Key: HADOOP-6001
    URL: https://issues.apache.org/jira/browse/HADOOP-6001
    Project: Hadoop Core
    Issue Type: Sub-task
    Reporter: Luca Telloli

    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 11, 2009 at 5:37 pm
    [ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Luca Telloli updated HADOOP-6001:
    ---------------------------------

    Attachment: HADOOP-6001.patch

    Initial patch.
    - Now EditLogFileOutputStream handles StorageDirectory directly.
    - Previous methods to revert and divert file streams have been moved inside this class, since they're file related
    - The header of some methods in FSEditLog has been replaced by one which handles storage directories instead of files
    - Method FSEditLog.revertFileStreams() has been removed.

    Tested for core-tests matching namenode/Test* and it passed.
    Incorporate storage directories into EditLogFileInput/Output streams
    --------------------------------------------------------------------

    Key: HADOOP-6001
    URL: https://issues.apache.org/jira/browse/HADOOP-6001
    Project: Hadoop Core
    Issue Type: Sub-task
    Reporter: Luca Telloli
    Attachments: HADOOP-6001.patch

    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 11, 2009 at 5:39 pm
    [ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Luca Telloli updated HADOOP-6001:
    ---------------------------------

    Status: Patch Available (was: Open)
    Incorporate storage directories into EditLogFileInput/Output streams
    --------------------------------------------------------------------

    Key: HADOOP-6001
    URL: https://issues.apache.org/jira/browse/HADOOP-6001
    Project: Hadoop Core
    Issue Type: Sub-task
    Reporter: Luca Telloli
    Attachments: HADOOP-6001.patch

    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Jun 16, 2009 at 3:54 am
    [ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719911#action_12719911 ]

    Hadoop QA commented on HADOOP-6001:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12410418/HADOOP-6001.patch
    against trunk revision 785025.

    +1 @author. The patch does not contain any @author tags.

    +1 tests included. The patch appears to include 3 new or modified tests.

    -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

    +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs. The patch does not introduce any new Findbugs warnings.

    +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

    +1 release audit. The applied patch does not increase the total number of release audit warnings.

    +1 core tests. The patch passed core unit tests.

    -1 contrib tests. The patch failed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/501/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/501/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/501/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/501/console

    This message is automatically generated.
    Incorporate storage directories into EditLogFileInput/Output streams
    --------------------------------------------------------------------

    Key: HADOOP-6001
    URL: https://issues.apache.org/jira/browse/HADOOP-6001
    Project: Hadoop Core
    Issue Type: Sub-task
    Reporter: Luca Telloli
    Attachments: HADOOP-6001.patch

    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 16, 2009 at 1:05 pm
    [ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720107#action_12720107 ]

    Luca Telloli commented on HADOOP-6001:
    --------------------------------------

    As far as I understand from the console output of Hudson

    - the javadoc warning is not related directly to the content of the patch
    {noformat}
    [exec] [javadoc] javadoc: warning - Error fetching URL: http://java.sun.com/javase/6/docs/api/package-list
    {noformat}

    - the contrib test failing is the one related to HADOOP-6007

    so this patch to me seems ready to be reviewed and eventually committed
    Incorporate storage directories into EditLogFileInput/Output streams
    --------------------------------------------------------------------

    Key: HADOOP-6001
    URL: https://issues.apache.org/jira/browse/HADOOP-6001
    Project: Hadoop Core
    Issue Type: Sub-task
    Reporter: Luca Telloli
    Attachments: HADOOP-6001.patch

    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 16, 2009 at 1:05 pm
    [ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720107#action_12720107 ]

    Luca Telloli edited comment on HADOOP-6001 at 6/16/09 6:04 AM:
    ---------------------------------------------------------------

    As far as I understand from the console output of Hudson

    - _if I'm not mistaken_, the javadoc warning is not related directly to the content of the patch
    {noformat}
    [exec] [javadoc] javadoc: warning - Error fetching URL: http://java.sun.com/javase/6/docs/api/package-list
    {noformat}

    - the contrib test failing is the one related to HADOOP-6007

    so this patch to me seems ready to be reviewed and eventually committed

    was (Author: lucat):
    As far as I understand from the console output of Hudson

    - the javadoc warning is not related directly to the content of the patch
    {noformat}
    [exec] [javadoc] javadoc: warning - Error fetching URL: http://java.sun.com/javase/6/docs/api/package-list
    {noformat}

    - the contrib test failing is the one related to HADOOP-6007

    so this patch to me seems ready to be reviewed and eventually committed
    Incorporate storage directories into EditLogFileInput/Output streams
    --------------------------------------------------------------------

    Key: HADOOP-6001
    URL: https://issues.apache.org/jira/browse/HADOOP-6001
    Project: Hadoop Core
    Issue Type: Sub-task
    Reporter: Luca Telloli
    Attachments: HADOOP-6001.patch

    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 18, 2009 at 6:28 pm
    [ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Luca Telloli updated HADOOP-6001:
    ---------------------------------

    Attachment: HADOOP-6001.patch

    Added some Javadoc documentation. No real code has been modified.
    Incorporate storage directories into EditLogFileInput/Output streams
    --------------------------------------------------------------------

    Key: HADOOP-6001
    URL: https://issues.apache.org/jira/browse/HADOOP-6001
    Project: Hadoop Core
    Issue Type: Sub-task
    Reporter: Luca Telloli
    Attachments: HADOOP-6001.patch, HADOOP-6001.patch

    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Benjamin Reed (JIRA) at Jun 19, 2009 at 4:02 pm
    [ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721842#action_12721842 ]

    Benjamin Reed commented on HADOOP-6001:
    ---------------------------------------

    the patch looks fine, but you haven't motivated it. it looks like you are just doing code cleanup. can you explain why you must do this to plug in bookkeeper?
    Incorporate storage directories into EditLogFileInput/Output streams
    --------------------------------------------------------------------

    Key: HADOOP-6001
    URL: https://issues.apache.org/jira/browse/HADOOP-6001
    Project: Hadoop Core
    Issue Type: Sub-task
    Reporter: Luca Telloli
    Attachments: HADOOP-6001.patch, HADOOP-6001.patch

    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 19, 2009 at 4:14 pm
    [ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721855#action_12721855 ]

    Luca Telloli commented on HADOOP-6001:
    --------------------------------------

    I would like to avoid in HADOOP-5189 situations like:

    if (stream is of type Bookkeeper)
    do this
    else
    do that

    Code in FSEditlog is mostly file related and many methods (rollEditLog, purgeEditLog) manipulate files. Storage directories, for instance, are an example of file-based logging.

    This patch should help to isolate possibly all situations where the file-dependent calls are made in FSEditLog.

    I was also hoping that dividing the steps in multiple independent sub-patches for 5188/5189 would speed up commit
    Incorporate storage directories into EditLogFileInput/Output streams
    --------------------------------------------------------------------

    Key: HADOOP-6001
    URL: https://issues.apache.org/jira/browse/HADOOP-6001
    Project: Hadoop Core
    Issue Type: Sub-task
    Reporter: Luca Telloli
    Attachments: HADOOP-6001.patch, HADOOP-6001.patch

    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 19, 2009 at 5:20 pm
    [ https://issues.apache.org/jira/browse/HADOOP-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721887#action_12721887 ]

    Luca Telloli commented on HADOOP-6001:
    --------------------------------------

    I guess we need some additional clarification on the purpose of this patch and on the interrelationship between this one and the other ones we're working on (5188, 5189, 5832, etc).

    The overall goal of 5188 is simple: to have the EditLogInput/OuputStream to properly incapsulate everything is needed for log of NameNode operations, and FSEditLog to call the same methods independently of the type of stream it's working on.

    Then, with 5832 I moved a step towards this goal: to treat the properties in the .xml file which specify the log devices and their parameters as a URI. While deploying that patch I realized that I couldn't pass a URI as argument to EditLogFileInput/Output Stream, because the constructor was dependent on the role of the file. By role of the file, I mean edits or edits.new, two different files used in file-based logging to take care of changes. You can see that if you look at the original code for FSEditLog, when EditLogFileOutputStream is instantiated at times with the EDITS file, and some other times with the EDITS_NEW file. This is bad, because it breaks generality.

    With some simple modifications, contained in this patch, I can now pass to EditLogFileOutputStream the container of these files, which is a StorageDirectory and, depending on the call from FSEditLog, I can switch between edits and edits.new inside EditLogFileOutputStream, making it transparent to FSEditLog which doesn't have to instantiate this class based on a specific file and its role.
    At the same time, a StorageDirectory is exactly what is passed inside the URI of a file-based logger, which connects this patch to 5832.

    I hope this clarifies some of the doubt on this patch, but I'm happy to answer more questions if they come up.
    Incorporate storage directories into EditLogFileInput/Output streams
    --------------------------------------------------------------------

    Key: HADOOP-6001
    URL: https://issues.apache.org/jira/browse/HADOOP-6001
    Project: Hadoop Core
    Issue Type: Sub-task
    Reporter: Luca Telloli
    Attachments: HADOOP-6001.patch, HADOOP-6001.patch

    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedJun 9, '09 at 5:01p
activeJun 19, '09 at 5:20p
posts11
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Luca Telloli (JIRA): 11 posts

People

Translate

site design / logo © 2022 Grokbase