FAQ
Process dfs.name.edits.dirs as URI
-----------------------------------

Key: HADOOP-5832
URL: https://issues.apache.org/jira/browse/HADOOP-5832
Project: Hadoop Core
Issue Type: Sub-task
Reporter: Luca Telloli


Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • dhruba borthakur (JIRA) at May 18, 2009 at 4:58 am
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-5832:
    -------------------------------------

    Component/s: dfs
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Reporter: Luca Telloli

    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at May 18, 2009 at 4:58 am
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710270#action_12710270 ]

    dhruba borthakur commented on HADOOP-5832:
    ------------------------------------------

    This looks like a good one to have !
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Reporter: Luca Telloli

    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at May 22, 2009 at 1:31 am
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711916#action_12711916 ]

    Konstantin Shvachko commented on HADOOP-5832:
    ---------------------------------------------

    This comment is relevant to this issue.
    [http://issues.apache.org/jira/browse/HADOOP-5188#action_12711913]
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Reporter: Luca Telloli

    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 5, 2009 at 2:38 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Luca Telloli updated HADOOP-5832:
    ---------------------------------

    Attachment: HADOOP-5832.patch

    Attaching a specific patch for this issue, since HADOOP-5188 has not been fully agreed so far. This should also help the review process

    This patch allows passing arguments of the properties dfs.name.dir and dfs.name.edits.dir as URI, along with fs.checkpoint.dir and fs.checkpoint.edits.dir.
    The patch supports current (plain) directories as values, assuming that they refer to a file:// type of URI.


    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 5, 2009 at 2:38 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Luca Telloli updated HADOOP-5832:
    ---------------------------------

    Hadoop Flags: (was: [Incompatible change])
    Status: Patch Available (was: Open)
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Reporter: Luca Telloli
    Attachments: HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 5, 2009 at 2:38 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Luca Telloli updated HADOOP-5832:
    ---------------------------------

    Affects Version/s: 0.20.0
    Fix Version/s: 0.21.0
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 5, 2009 at 2:40 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716625#action_12716625 ]

    Luca Telloli edited comment on HADOOP-5832 at 6/5/09 7:38 AM:
    --------------------------------------------------------------

    Attaching a specific patch for this issue, since HADOOP-5188 has not been fully agreed so far. This should also help the review process

    This patch allows passing arguments of the properties dfs.name.dir and dfs.name.edits.dir as URI, along with fs.checkpoint.dir and fs.checkpoint.edits.dir.
    The patch supports current (plain) directories as values, assuming that they refer to a file:// type of URI.

    This patch should also help HADOOP-5188, for the deployment of different types of logging than files.

    was (Author: lucat):
    Attaching a specific patch for this issue, since HADOOP-5188 has not been fully agreed so far. This should also help the review process

    This patch allows passing arguments of the properties dfs.name.dir and dfs.name.edits.dir as URI, along with fs.checkpoint.dir and fs.checkpoint.edits.dir.
    The patch supports current (plain) directories as values, assuming that they refer to a file:// type of URI.


    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Benjamin Reed (JIRA) at Jun 5, 2009 at 2:54 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716628#action_12716628 ]

    Benjamin Reed commented on HADOOP-5832:
    ---------------------------------------

    +1 looks good
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Jun 5, 2009 at 6:59 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716724#action_12716724 ]

    Raghu Angadi commented on HADOOP-5832:
    --------------------------------------

    This is a pretty spread out patch. Could you fix the indentation in multiple places? There are patch guidelines on the wiki, but two most important ones are :
    # use spaces instead of tabs, and
    # use indentation of 2.

    It would matter less if the patch was a small isolated change.
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Flavio Paiva Junqueira (JIRA) at Jun 6, 2009 at 2:34 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716882#action_12716882 ]

    Flavio Paiva Junqueira commented on HADOOP-5832:
    ------------------------------------------------

    On the comments from Raghu:

    * -1 on indentation. I'm seeing tabs in some included lines, and some statements seem misplaced, like the "finally" declaration in TestBackupNode.java;
    * It is true that the patch is spread out, but most of it replaces the use of File with URI, and touching multiple files seems necessary to add the proposed functionality. I don't see a good way around it, so it is fine with me.

    I have run the hdfs tests, and it passes all but one: Test org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer

    It gives me the following error:

    {noformat}
    Testcase: testOIV took 7.667 sec
    Caused an ERROR
    null
    java.lang.ArrayStoreException
    at java.lang.System.arraycopy(Native Method)
    at java.util.Arrays.copyOf(Arrays.java:2763)
    at java.util.ArrayList.toArray(ArrayList.java:305)
    at org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer.initFsimage(TestOfflineImageViewer.java:127)
    at org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer.testOIV(TestOfflineImageViewer.java:77)
    {noformat}

    Inspecting the code, we see that line 127 of the test is:

    {code:title=TestOfflineImageViewer.java|borderStyle=solid}
    File [] files = cluster.getNameDirs().toArray(new File[0]);
    {code}

    which looks like it could be caused by the changes of this patch. In fact, without the patch, this test runs fine for me.
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 7, 2009 at 1:02 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Luca Telloli updated HADOOP-5832:
    ---------------------------------

    Status: Open (was: Patch Available)

    cancelling for later resubmission due to latest comments
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 7, 2009 at 3:10 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Luca Telloli updated HADOOP-5832:
    ---------------------------------

    Status: Patch Available (was: Open)
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 7, 2009 at 3:10 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Luca Telloli updated HADOOP-5832:
    ---------------------------------

    Attachment: HADOOP-5832.patch

    Addressed the issue of previous comments; in particular:
    - fixed the failing test
    - fixed the indentation issue; since most of the work have been done manually, there's a couple of places where the code doesn't change, but only the indentation; I apologize in advance for those lines but at the same time I don't think I should revert them, since the bad indentation was there before the patch, and it's resolved afterwards. Additionally, as I said, the number of lines where this happens is very limited.
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Flavio Paiva Junqueira (JIRA) at Jun 7, 2009 at 8:52 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717097#action_12717097 ]

    Flavio Paiva Junqueira commented on HADOOP-5832:
    ------------------------------------------------

    +1, it looks good to me. I've only been able to find 3 instances in which this patch replaces an existing tab with the correct indentation. HDFS tests pass fine for me. Thanks, Luca!


    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 8, 2009 at 9:58 am
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Luca Telloli updated HADOOP-5832:
    ---------------------------------

    Status: Open (was: Patch Available)
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 8, 2009 at 9:58 am
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Luca Telloli updated HADOOP-5832:
    ---------------------------------

    Status: Patch Available (was: Open)

    resubmitting since it looks there's issues with hudson
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Jun 8, 2009 at 10:14 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717460#action_12717460 ]

    Hadoop QA commented on HADOOP-5832:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12410084/HADOOP-5832.patch
    against trunk revision 782708.

    +1 @author. The patch does not contain any @author tags.

    +1 tests included. The patch appears to include 23 new or modified tests.

    +1 javadoc. The javadoc tool did not generate any warning messages.

    +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs. The patch does not introduce any new Findbugs warnings.

    +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

    +1 release audit. The applied patch does not increase the total number of release audit warnings.

    -1 core tests. The patch failed core unit tests.

    -1 contrib tests. The patch failed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/481/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/481/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/481/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/481/console

    This message is automatically generated.
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Jun 8, 2009 at 10:26 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717467#action_12717467 ]

    Raghu Angadi commented on HADOOP-5832:
    --------------------------------------
    It is true that the patch is spread out, but most of it replaces the use of File with URI, and touching multiple files seems necessary to add the proposed functionality. I don't see a good way around it, so it is fine with me.
    Right. I meant since it is a spread out patch, indentation is more important.
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 9, 2009 at 1:16 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717668#action_12717668 ]

    Luca Telloli commented on HADOOP-5832:
    --------------------------------------

    I checked the two tests which failed in hudson:

    - mapred/TestReduceFetch failed on my node with trunk *both* patched and unpatched, so the problem seems related to some previous issue. Btw, on Hudson, this test was timing out, on my box it's failing due to an error.

    - contrib/TestHdfsProxy passes without issues on my node; as far a I can see this test has been failing on other patches previous to this one so again I'm not sure if its outcome is related to this patch at all



    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Flavio Paiva Junqueira (JIRA) at Jun 10, 2009 at 4:30 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718130#action_12718130 ]

    Flavio Paiva Junqueira commented on HADOOP-5832:
    ------------------------------------------------

    There is an issue open about TestHdfsProxy failing (HADOOP-6007), so it sounds like it is a problem with the test.

    If no one else has any other comment on the patch, it would be great to have it committed.
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Erik Eldridge (JIRA) at Jun 10, 2009 at 4:42 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718134#action_12718134 ]

    Erik Eldridge commented on HADOOP-5832:
    ---------------------------------------

    Hi,

    I will be out of the office at the Hadoop Summit :) on 6/10. I'll be
    checking email occassionaly. For urgent matters, please try my cell
    phone (it's on backyard).

    Thanks!

    On Jun 10, 2009, at 9:30 AM, "Flavio Paiva Junqueira (JIRA)" <jira@apache.org


    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Jun 12, 2009 at 6:40 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Konstantin Shvachko updated HADOOP-5832:
    ----------------------------------------

    Comment: was deleted

    (was: Hi,

    I will be out of the office at the Hadoop Summit :) on 6/10. I'll be
    checking email occassionaly. For urgent matters, please try my cell
    phone (it's on backyard).

    Thanks!

    On Jun 10, 2009, at 9:30 AM, "Flavio Paiva Junqueira (JIRA)" <jira@apache.org

    )
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Jun 12, 2009 at 9:07 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718979#action_12718979 ]

    Raghu Angadi commented on HADOOP-5832:
    --------------------------------------

    What happens if someone specifies a non-file URI with this patch? I would say any unsported URI should result in a hard error.

    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Flavio Paiva Junqueira (JIRA) at Jun 12, 2009 at 9:31 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718988#action_12718988 ]

    Flavio Paiva Junqueira commented on HADOOP-5832:
    ------------------------------------------------

    Raghu, being able to specify non-file URIs is part of the point of this patch. As Luca is working on it towards the integration of BookKeeper (HADOOP-5189), we would like to eventually be able to specify a URI for non-file types of logging such as BookKeeper.
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Jun 12, 2009 at 10:15 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718999#action_12718999 ]

    Raghu Angadi commented on HADOOP-5832:
    --------------------------------------

    Luca,

    Could you confirm if any of the error handling is different compared current trunk? If there are any differences could you list them here? It is harder reviewers alone to do this. Could you confirm that things that caused a hard error still does and vice versa.
    For e.g: {noformat}
    try {
    URI u = new URI(name);
    // If the scheme was not declared, default to file://
    // and use the absolute path of the file
    if(u.getScheme() == null)
    u = new URI("file://" + new File(name).getAbsolutePath());
    dirs.add(u);
    } catch (Exception e) {
    LOG.error("Error while processing URI: " + name +
    ". The error message was: " + e.getMessage());
    }
    {noformat}
    Edits and FSImage directories are very critical part of NN operation. Ignoring error like this might not be good.

    Other comments :
    # Please update hdfs-default.xml for {{dfs.name.dir}}. Mainly the description needs to be updated for new syntax.
    # May be a warning when there is no schema would help get configs updated.
    # Indentation "fixes" : I don't see any reason do indentation fixes far away from actual changes for this patch. Some of these are not even fixes. A good rule of thumb I follow is that if I have a hunk in the patch that has only these changes, I remove it from my patch. That is easy to do. On the plus side, you won't get 'svn blame'd for bugs around there :).




    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 14, 2009 at 7:17 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Luca Telloli updated HADOOP-5832:
    ---------------------------------

    Attachment: HADOOP-5832.patch
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 14, 2009 at 7:29 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719323#action_12719323 ]

    Luca Telloli commented on HADOOP-5832:
    --------------------------------------
    Could you confirm if any of the error handling is different compared current trunk? If there are any differences could you list them here?
    My approach has been to catch any new exception and throw a new IOException with its content
    Please update hdfs-default.xml for dfs.name.dir. Mainly the description needs to be updated for new syntax. Done
    May be a warning when there is no schema would help get configs updated. Done
    Indentation "fixes" : I don't see any reason do indentation fixes far away from actual changes for this patch. Some of these are not even fixes. A good rule of thumb I follow is that if I have a hunk in the patch that has only these changes, I remove it from my patch. That is easy to do. On the plus side, you won't get 'svn blame'd for bugs around there .
    It's a consequence of using Eclipse in a different configuration. Just setting the values of tabstop to 2 and whitespace instead of tab proved not to be sufficient to avoid these situations. Although I tried to fix most of them manually, there might be a couple of leftovers.

    Finally, I think the approach should be:
    - non URI: consider it valid, use as file:// and warn the user, as you suggested
    - unknown uri scheme: the usi should be specified inside the JournalType enumeration, otherwise hard error

    I'm posting a new patch that fixes the issues you stated above, please let me know your impressions on it
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 15, 2009 at 9:07 am
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719323#action_12719323 ]

    Luca Telloli edited comment on HADOOP-5832 at 6/15/09 2:06 AM:
    ---------------------------------------------------------------
    Could you confirm if any of the error handling is different compared current trunk? If there are any differences could you list them here?
    My approach has been to catch any new exception and throw a new IOException with its content
    Please update hdfs-default.xml for dfs.name.dir. Mainly the description needs to be updated for new syntax. Done
    May be a warning when there is no schema would help get configs updated. Done
    Indentation "fixes" : I don't see any reason do indentation fixes far away from actual changes for this patch. Some of these are not even fixes. A good rule of thumb I follow is that if I have a hunk in the patch that has only these changes, I remove it from my patch. That is easy to do. On the plus side, you won't get 'svn blame'd for bugs around there .
    It's a consequence of using Eclipse in a different configuration. Just setting the values of tabstop to 2 and whitespace instead of tab proved not to be sufficient to avoid these situations. Although I tried to fix most of them manually, there might be a couple of leftovers.

    Finally, I think the approach should be:
    - non URI: consider it valid, use as file:// and warn the user, as you suggested
    - unknown uri scheme: the scheme should be among the ones specified inside the JournalType enumeration, otherwise hard error

    I'm posting a new patch that fixes the issues you stated above, please let me know your impressions on it

    was (Author: lucat):
    Could you confirm if any of the error handling is different compared current trunk? If there are any differences could you list them here?
    My approach has been to catch any new exception and throw a new IOException with its content
    Please update hdfs-default.xml for dfs.name.dir. Mainly the description needs to be updated for new syntax. Done
    May be a warning when there is no schema would help get configs updated. Done
    Indentation "fixes" : I don't see any reason do indentation fixes far away from actual changes for this patch. Some of these are not even fixes. A good rule of thumb I follow is that if I have a hunk in the patch that has only these changes, I remove it from my patch. That is easy to do. On the plus side, you won't get 'svn blame'd for bugs around there .
    It's a consequence of using Eclipse in a different configuration. Just setting the values of tabstop to 2 and whitespace instead of tab proved not to be sufficient to avoid these situations. Although I tried to fix most of them manually, there might be a couple of leftovers.

    Finally, I think the approach should be:
    - non URI: consider it valid, use as file:// and warn the user, as you suggested
    - unknown uri scheme: the usi should be specified inside the JournalType enumeration, otherwise hard error

    I'm posting a new patch that fixes the issues you stated above, please let me know your impressions on it
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 16, 2009 at 2:07 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Luca Telloli updated HADOOP-5832:
    ---------------------------------

    Status: Open (was: Patch Available)
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 16, 2009 at 2:07 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Luca Telloli updated HADOOP-5832:
    ---------------------------------

    Status: Patch Available (was: Open)
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Jun 16, 2009 at 10:26 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720388#action_12720388 ]

    Raghu Angadi commented on HADOOP-5832:
    --------------------------------------

    Thanks for the changes. Looks better.

    nit: Could you move the block of code that checks for consistency (including the check to see if it file://) a method? It is repeated for dirs and edits (it around 20 lines).

    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Luca Telloli (JIRA) at Jun 17, 2009 at 3:56 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Luca Telloli updated HADOOP-5832:
    ---------------------------------

    Attachment: HADOOP-5832.patch

    Raghu, thanks for the comments they're helpful to me for making good progress!

    - I factored out the repeated code
    - I added a consistency check also for properties fs.checkpoint.dir and fs.checkpoint.edits.dir that wasn't currently in place
    Process dfs.name.edits.dirs as URI
    -----------------------------------

    Key: HADOOP-5832
    URL: https://issues.apache.org/jira/browse/HADOOP-5832
    Project: Hadoop Core
    Issue Type: Sub-task
    Components: dfs
    Affects Versions: 0.20.0
    Reporter: Luca Telloli
    Fix For: 0.21.0

    Attachments: HADOOP-5832.patch, HADOOP-5832.patch, HADOOP-5832.patch, HADOOP-5832.patch


    Process the value of property dfs.name.edits.dirs as URI, to allow different schemes than just file. As an advantage, Java supports the constructor File(URI) so the transition is straightforward for files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMay 14, '09 at 9:23a
activeJun 17, '09 at 3:56p
posts33
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Luca Telloli (JIRA): 33 posts

People

Translate

site design / logo © 2022 Grokbase