FAQ
I've got a smallish cluster of 12 nodes up from 6, that we're using to dip
our feet into hadoop. One of my users has a few directories in his HDFS
home which he was using to test, and which exist, according to

hadoop fs -ls <home directory>

ie:

...
/user/ken/testoutput4
/user/ken/testoutput5
...

but if you do:

hadoop fs -ls /user/ken/testoutput5

you get:

ls: Cannot access /user/ken/testoutput5: No such file or directory.


I can even hadoop fs -mkdir /user/ken/testoutput5, and then -rmr it, and it
works fine, but then -ls still shows it.

same for -rmr. There's nothing important in the directories, so I'd just
remove them, but it won't let me. I've tried fsck'ing, no luck there.
Anyone have an idea how I might clean this up, and how it might have
happened in the first place?

I'm pretty sure the directories were copied over from another cluster in
their current state, if that helps shine a light.


Any help is much appreciated.

Thanks,
Seth

Search Discussions

  • Allen Wittenauer at Dec 13, 2010 at 8:58 pm

    On Dec 13, 2010, at 8:51 AM, Seth Lepzelter wrote:

    I've got a smallish cluster of 12 nodes up from 6, that we're using to dip
    our feet into hadoop. One of my users has a few directories in his HDFS
    home which he was using to test, and which exist, according to

    hadoop fs -ls <home directory>

    ie:

    ...
    /user/ken/testoutput4
    /user/ken/testoutput5
    ...

    but if you do:

    hadoop fs -ls /user/ken/testoutput5

    you get:

    ls: Cannot access /user/ken/testoutput5: No such file or directory.

    There is likely one or more spaces after the testoutput5 . Try using hadoop fs -ls /user/ken/*/* .
  • Seth Lepzelter at Dec 13, 2010 at 11:15 pm
    Alright, a little further investigation along that line (thanks for the
    hint, can't believe I didn't think of that), shows that there's actually a
    carriage return character (%0D, aka \r) at the end of the filename.

    using the hdfs web ui, I browsed to his directory, and the ui wants to send
    me to /user/ken/testoutput5%0D

    which also kind of matches the output of hadoop fs -lsr:

    drwx------ - ken users 0 2010-08-26 19:48 /user/ken/testoutput5
    /_logs---- - ken users 0 2010-08-26 19:48 /user/ken/testoutput5
    /_logs/history ken users 0 2010-08-26 19:48 /user/ken/testoutput5


    hadoop fs -ls /user/ken/test*

    results in:
    ls: Cannot access /user/ken/test*: No such file or directory.


    I guess *, in hadoop's parlance, doesn't include a \r.

    got a \r into the command line, -rmr'ed that, it's now fixed.


    Thanks!
    -Seth
    On Mon, Dec 13, 2010 at 08:57:35PM +0000, Allen Wittenauer wrote:
    On Dec 13, 2010, at 8:51 AM, Seth Lepzelter wrote:

    I've got a smallish cluster of 12 nodes up from 6, that we're using to dip
    our feet into hadoop. One of my users has a few directories in his HDFS
    home which he was using to test, and which exist, according to

    hadoop fs -ls <home directory>

    ie:

    ...
    /user/ken/testoutput4
    /user/ken/testoutput5
    ...

    but if you do:

    hadoop fs -ls /user/ken/testoutput5

    you get:

    ls: Cannot access /user/ken/testoutput5: No such file or directory.

    There is likely one or more spaces after the testoutput5 . Try using hadoop fs -ls /user/ken/*/* .
  • Allen Wittenauer at Dec 14, 2010 at 5:33 pm

    On Dec 13, 2010, at 3:14 PM, Seth Lepzelter wrote:

    Alright, a little further investigation along that line (thanks for the
    hint, can't believe I didn't think of that), shows that there's actually a
    carriage return character (%0D, aka \r) at the end of the filename.
    This falls into that "you never forget your first time" area. ;)
    I guess *, in hadoop's parlance, doesn't include a \r.
    That's a bug in HDFS, IMO. Please file a JIRA.
    got a \r into the command line, -rmr'ed that, it's now fixed.
    Awesome. :)

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedDec 13, '10 at 4:52p
activeDec 14, '10 at 5:33p
posts4
users2
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase