FAQ
Hi,

I'm trying to get Hadoop running on Windows, and I've found that it
isn't exactly a simple process (yes, I'm using Cygwin).

I've had a quick look though the mailing list, and I see the occasional
question about it, generally with the response "use Cygwin", and no
further details.

Is anyone actually successfully running Hadoop on Windows/Cygwin? If so,
have you had similar problems to those outlined below? I think there's a
fair chance the first two of these are environmental local to my
machine, but it would be nice to be sure that if I keep working on this
it will work in the end...

1) Running bin/start-all.sh gives:
: no address associated with name

That turns out to be a Unix vs DOS line endings thing (!).

Running the following commands fixed that:

dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/masters
dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/slaves

2) Any script using hadoop-daemon.sh failed. This turned out to be
another Unix vs DOS line endings thing. The critical error looked like
this:

<machine-name>: head: cannot open
`/cygdrive/c/dev/prog/hadoop-0.14.1/bin/../logs/hadoop-nlothian-namenode
-<machine-name>\r.out' for reading: No such file or directory

(note that "\r.out" in the logfile name...)

The fix for that was to alter the hadoop-daemon.sh script. Replace:

export
HADOOP_LOGFILE=hadoop-$HADOOP_IDENT_STRING-$command-`hostname`.log
with
export
HADOOP_LOGFILE=hadoop-$HADOOP_IDENT_STRING-$command-`'hostname' | sed
s/\r//`.log

and


log=$HADOOP_LOG_DIR/hadoop-$HADOOP_IDENT_STRING-$command-`hostname`.out
with

log=$HADOOP_LOG_DIR/hadoop-$HADOOP_IDENT_STRING-$command-`'hostname' |
sed s/\r//`.out

3) Now the start-all.sh script seems to run ok, but the Java processes
which are started on the slave machine (in this case the same machine as
the master) die soon after launching.

If I manually run:
bin/hadoop jobtracker
bin/hadoop tasktracker
bin/hadoop namenode
bin/hadoop datanode

I can succesfully submit jobs. However, when using the start-all.sh
script the submitted jobs fail to connect (because the jobtracker dies).
Has anyone else seen this? Is there a workaround?

Regards
Nick Lothian

IMPORTANT: This e-mail, including any attachments, may contain private or confidential information.
If you think you may not be the intended recipient, or if you have received this e-mail in error, please contact the sender immediately and delete all copies of this e-mail.
If you are not the intended recipient, you must not reproduce any part of this e-mail or disclose its contents to any other party.
This email represents the views of the individual sender, which do not necessarily reflect those of education.au limited except where the sender expressly states otherwise.
It is your responsibility to scan this email and any files transmitted with it for viruses or any other defects. education.au limited will not be liable for any loss, damage or consequence caused directly or indirectly by this email.

Search Discussions

  • Doug Cutting at Oct 10, 2007 at 6:11 pm

    Nick Lothian wrote:
    That turns out to be a Unix vs DOS line endings thing (!).

    Running the following commands fixed that:

    dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/masters
    dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/slaves
    That should not be required. When you install Cygwin, it has a setting
    for line endings, which I think defaults and should be left as
    unix-style, not windows style. Could that be the issue?

    Doug
  • Nick Lothian at Oct 10, 2007 at 11:20 pm
    Yes, mine are set for the Windows style (they weren't, but at some stage
    I upgraded my Cywin install and they switched).

    However if I switch back to Unix style endings all my bash scripts fail,
    again with complaints about line endings (I can't seem to win!):

    /cygdrive/c/dev/prog/hadoop-0.14.1/bin/hadoop-config.sh: line 4: $'\r':
    command not found
    /cygdrive/c/dev/prog/hadoop-0.14.1/bin/hadoop-config.sh: line 6: $'\r':
    command not found
    /cygdrive/c/dev/prog/hadoop-0.14.1/bin/hadoop-config.sh: line 54: syntax
    error: unexpected end of file
    etc etc

    Nick

    -----Original Message-----
    From: Doug Cutting
    Sent: Thursday, October 11, 2007 3:37 AM
    To: hadoop-user@lucene.apache.org
    Subject: Re: Hadoop on Windows

    Nick Lothian wrote:
    That turns out to be a Unix vs DOS line endings thing (!).

    Running the following commands fixed that:

    dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/masters
    dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/slaves
    That should not be required. When you install Cygwin, it has
    a setting for line endings, which I think defaults and should
    be left as unix-style, not windows style. Could that be the issue?

    Doug
    IMPORTANT: This e-mail, including any attachments, may contain private or confidential information.
    If you think you may not be the intended recipient, or if you have received this e-mail in error, please contact the sender immediately and delete all copies of this e-mail.
    If you are not the intended recipient, you must not reproduce any part of this e-mail or disclose its contents to any other party.
    This email represents the views of the individual sender, which do not necessarily reflect those of education.au limited except where the sender expressly states otherwise.
    It is your responsibility to scan this email and any files transmitted with it for viruses or any other defects. education.au limited will not be liable for any loss, damage or consequence caused directly or indirectly by this email.
  • Doug Cutting at Oct 10, 2007 at 11:27 pm

    Nick Lothian wrote:
    Yes, mine are set for the Windows style (they weren't, but at some stage
    I upgraded my Cywin install and they switched).

    However if I switch back to Unix style endings all my bash scripts fail,
    again with complaints about line endings (I can't seem to win!):
    This is in a fresh cygwin install with a fresh hadoop install?

    Doug
  • Nick Lothian at Oct 11, 2007 at 5:28 am

    Nick Lothian wrote:
    Yes, mine are set for the Windows style (they weren't, but at some
    stage I upgraded my Cywin install and they switched).

    However if I switch back to Unix style endings all my bash scripts
    fail, again with complaints about line endings (I can't
    seem to win!):

    This is in a fresh cygwin install with a fresh hadoop install?

    Doug
    Yes that's right.

    I suspect it's more of a Cygwin bug with switching line endings than a
    Hadoop bug, though

    IMPORTANT: This e-mail, including any attachments, may contain private or confidential information.
    If you think you may not be the intended recipient, or if you have received this e-mail in error, please contact the sender immediately and delete all copies of this e-mail.
    If you are not the intended recipient, you must not reproduce any part of this e-mail or disclose its contents to any other party.
    This email represents the views of the individual sender, which do not necessarily reflect those of education.au limited except where the sender expressly states otherwise.
    It is your responsibility to scan this email and any files transmitted with it for viruses or any other defects. education.au limited will not be liable for any loss, damage or consequence caused directly or indirectly by this email.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 8, '07 at 5:44a
activeOct 11, '07 at 5:28a
posts5
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Nick Lothian: 3 posts Doug Cutting: 2 posts

People

Translate

site design / logo © 2022 Grokbase