FAQ
"du" fails on Cygwin
--------------------

Key: HADOOP-5496
URL: https://issues.apache.org/jira/browse/HADOOP-5496
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Affects Versions: 0.19.1
Environment: Windows + Cygwin
Reporter: Kohsuke Kawaguchi


When I try to run a datanode on Windows, I get the following exception:

{noformat}
java.io.IOException: Expecting a line not the end of stream
at org.apache.hadoop.fs.DU.parseExecResult(DU.java:181)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
at org.apache.hadoop.util.Shell.run(Shell.java:134)
at org.apache.hadoop.fs.DU.(DU.java:63)
at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.(FSDataset.java:681)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:291)
at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:1238)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1193)
{noformat}

This is because Hadoop execs "du -sk C:\tmp\hadoop-SYSTEM\dfs\data" with a Windows path representation, which cygwin du doesn't understand.

{noformat}
C:\hudson>du -sk C:\tmp\hadoop-SYSTEM\dfs\data
du -sk C:\tmp\hadoop-SYSTEM\dfs\data
du: cannot access `C:\\tmp\\hadoop-SYSTEM\\dfs\\data': No such file or directory
{noformat}

For this to work correctly, Hadoop would have to run cygpath first to get a Unix path representation, then to call DU.

Also, I had to use the debugger to get this information. Shell.runCommand should catch IOException from parseExecResult and add the buffered stderr to simplify the error diagnostics.


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Kohsuke Kawaguchi (JIRA) at Mar 16, 2009 at 5:10 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682375#action_12682375 ]

    Kohsuke Kawaguchi commented on HADOOP-5496:
    -------------------------------------------

    The patch for this can be found on http://github.com/kohsuke/hadoop/commit/27273cd0515e598b51eed48a1d0c87fc16ca995f
    "du" fails on Cygwin
    --------------------

    Key: HADOOP-5496
    URL: https://issues.apache.org/jira/browse/HADOOP-5496
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.19.1
    Environment: Windows + Cygwin
    Reporter: Kohsuke Kawaguchi

    When I try to run a datanode on Windows, I get the following exception:
    {noformat}
    java.io.IOException: Expecting a line not the end of stream
    at org.apache.hadoop.fs.DU.parseExecResult(DU.java:181)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
    at org.apache.hadoop.util.Shell.run(Shell.java:134)
    at org.apache.hadoop.fs.DU.<init>(DU.java:53)
    at org.apache.hadoop.fs.DU.<init>(DU.java:63)
    at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDataset.java:325)
    at org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:681)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:291)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:205)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1238)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1193)
    {noformat}
    This is because Hadoop execs "du -sk C:\tmp\hadoop-SYSTEM\dfs\data" with a Windows path representation, which cygwin du doesn't understand.
    {noformat}
    C:\hudson>du -sk C:\tmp\hadoop-SYSTEM\dfs\data
    du -sk C:\tmp\hadoop-SYSTEM\dfs\data
    du: cannot access `C:\\tmp\\hadoop-SYSTEM\\dfs\\data': No such file or directory
    {noformat}
    For this to work correctly, Hadoop would have to run cygpath first to get a Unix path representation, then to call DU.
    Also, I had to use the debugger to get this information. Shell.runCommand should catch IOException from parseExecResult and add the buffered stderr to simplify the error diagnostics.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Mar 16, 2009 at 6:30 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Doug Cutting reassigned HADOOP-5496:
    ------------------------------------

    Assignee: Kohsuke Kawaguchi
    The patch for this can be found on http://github.com/kohsuke [ ...]
    Can you please attach your patch directly to this issue in Jira, checking the license checkbox if you wish your work to be includable in an Apache release. Thanks!
    "du" fails on Cygwin
    --------------------

    Key: HADOOP-5496
    URL: https://issues.apache.org/jira/browse/HADOOP-5496
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.19.1
    Environment: Windows + Cygwin
    Reporter: Kohsuke Kawaguchi
    Assignee: Kohsuke Kawaguchi

    When I try to run a datanode on Windows, I get the following exception:
    {noformat}
    java.io.IOException: Expecting a line not the end of stream
    at org.apache.hadoop.fs.DU.parseExecResult(DU.java:181)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
    at org.apache.hadoop.util.Shell.run(Shell.java:134)
    at org.apache.hadoop.fs.DU.<init>(DU.java:53)
    at org.apache.hadoop.fs.DU.<init>(DU.java:63)
    at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDataset.java:325)
    at org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:681)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:291)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:205)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1238)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1193)
    {noformat}
    This is because Hadoop execs "du -sk C:\tmp\hadoop-SYSTEM\dfs\data" with a Windows path representation, which cygwin du doesn't understand.
    {noformat}
    C:\hudson>du -sk C:\tmp\hadoop-SYSTEM\dfs\data
    du -sk C:\tmp\hadoop-SYSTEM\dfs\data
    du: cannot access `C:\\tmp\\hadoop-SYSTEM\\dfs\\data': No such file or directory
    {noformat}
    For this to work correctly, Hadoop would have to run cygpath first to get a Unix path representation, then to call DU.
    Also, I had to use the debugger to get this information. Shell.runCommand should catch IOException from parseExecResult and add the buffered stderr to simplify the error diagnostics.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Kohsuke Kawaguchi (JIRA) at Mar 16, 2009 at 7:34 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Kohsuke Kawaguchi updated HADOOP-5496:
    --------------------------------------

    Attachment: HADOOP-5486

    Patch
    "du" fails on Cygwin
    --------------------

    Key: HADOOP-5496
    URL: https://issues.apache.org/jira/browse/HADOOP-5496
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.19.1
    Environment: Windows + Cygwin
    Reporter: Kohsuke Kawaguchi
    Assignee: Kohsuke Kawaguchi
    Attachments: HADOOP-5486


    When I try to run a datanode on Windows, I get the following exception:
    {noformat}
    java.io.IOException: Expecting a line not the end of stream
    at org.apache.hadoop.fs.DU.parseExecResult(DU.java:181)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
    at org.apache.hadoop.util.Shell.run(Shell.java:134)
    at org.apache.hadoop.fs.DU.<init>(DU.java:53)
    at org.apache.hadoop.fs.DU.<init>(DU.java:63)
    at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDataset.java:325)
    at org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:681)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:291)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:205)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1238)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1193)
    {noformat}
    This is because Hadoop execs "du -sk C:\tmp\hadoop-SYSTEM\dfs\data" with a Windows path representation, which cygwin du doesn't understand.
    {noformat}
    C:\hudson>du -sk C:\tmp\hadoop-SYSTEM\dfs\data
    du -sk C:\tmp\hadoop-SYSTEM\dfs\data
    du: cannot access `C:\\tmp\\hadoop-SYSTEM\\dfs\\data': No such file or directory
    {noformat}
    For this to work correctly, Hadoop would have to run cygpath first to get a Unix path representation, then to call DU.
    Also, I had to use the debugger to get this information. Shell.runCommand should catch IOException from parseExecResult and add the buffered stderr to simplify the error diagnostics.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Mar 16, 2009 at 7:48 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682427#action_12682427 ]

    Doug Cutting commented on HADOOP-5496:
    --------------------------------------

    Does the TestDU unit test fail on Windows without this? If not, can we easily modify it so that it does?

    "du" fails on Cygwin
    --------------------

    Key: HADOOP-5496
    URL: https://issues.apache.org/jira/browse/HADOOP-5496
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.19.1
    Environment: Windows + Cygwin
    Reporter: Kohsuke Kawaguchi
    Assignee: Kohsuke Kawaguchi
    Attachments: HADOOP-5486


    When I try to run a datanode on Windows, I get the following exception:
    {noformat}
    java.io.IOException: Expecting a line not the end of stream
    at org.apache.hadoop.fs.DU.parseExecResult(DU.java:181)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
    at org.apache.hadoop.util.Shell.run(Shell.java:134)
    at org.apache.hadoop.fs.DU.<init>(DU.java:53)
    at org.apache.hadoop.fs.DU.<init>(DU.java:63)
    at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDataset.java:325)
    at org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:681)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:291)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:205)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1238)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1193)
    {noformat}
    This is because Hadoop execs "du -sk C:\tmp\hadoop-SYSTEM\dfs\data" with a Windows path representation, which cygwin du doesn't understand.
    {noformat}
    C:\hudson>du -sk C:\tmp\hadoop-SYSTEM\dfs\data
    du -sk C:\tmp\hadoop-SYSTEM\dfs\data
    du: cannot access `C:\\tmp\\hadoop-SYSTEM\\dfs\\data': No such file or directory
    {noformat}
    For this to work correctly, Hadoop would have to run cygpath first to get a Unix path representation, then to call DU.
    Also, I had to use the debugger to get this information. Shell.runCommand should catch IOException from parseExecResult and add the buffered stderr to simplify the error diagnostics.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMar 15, '09 at 5:23p
activeMar 16, '09 at 7:48p
posts5
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Doug Cutting (JIRA): 5 posts

People

Translate

site design / logo © 2022 Grokbase