________________________________
From: Uma Maheswara Rao G 72686 <maheswara@huawei.com>
To: common-user@hadoop.apache.org; A Df <abbey_dragonforest@yahoo.com>
Cc: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Sent: Wednesday, 27 July 2011, 17:31
Subject: Re: cygwin not connecting to Hadoop server
Hi A Df,
Did you format the NameNode first?
** I had formatted it already but then I had reinstalled Java and upgraded the plugins in cygwin so I reformatted it again. :D yes it worked!! I am not sure all the steps that got it to finally work but I will have to document it to prevent this headache in the future. Although I typed ssh localhost too , so question is, do I need to type ssh localhost each time I need to run hadoop?? Also, since I need to work with Eclipse maybe you can have a look at my post about the plugin cause I can get the patch to work. The subject is "Re: Cygwin not working with Hadoop and Eclipse Plugin". I plan to read up on how to write programs for Hadoop. I am using the tutorial at Yahoo but if you know of any really good about coding with Hadoop or just about understanding Hadoop then please let me know.
Can you check the NN logs whether NN is started or not?
** I checked and the previous runs had some logs missing but now the last one have all 5 logs and I got two conf files in xml. I also copied out the other output files which I plan to examine. Where do I specify the output extension format that I want for my output file? I was hoping for an txt file it shows the output in a file with no extension even though I can read it in Notepad++. I also got to view the web interface at:
NameNode -
http://localhost:50070/JobTracker -
http://localhost:50030/** See below for the working version, finally!! Thanks
<CMD>
Williams@TWilliams-LTPC ~/hadoop-0.20.2
$ bin/hadoop jar hadoop-0.20.2-examples.jar grep input
11/07/27 17:42:20 INFO mapred.FileInputFormat: Total in
11/07/27 17:42:20 INFO mapred.JobClient: Running job: j
11/07/27 17:42:21 INFO mapred.JobClient: map 0% reduce
11/07/27 17:42:33 INFO mapred.JobClient: map 15% reduc
11/07/27 17:42:36 INFO mapred.JobClient: map 23% reduc
11/07/27 17:42:39 INFO mapred.JobClient: map 38% reduc
11/07/27 17:42:42 INFO mapred.JobClient: map 38% reduc
11/07/27 17:42:45 INFO mapred.JobClient: map 53% reduc
11/07/27 17:42:48 INFO mapred.JobClient: map 69% reduc
11/07/27 17:42:51 INFO mapred.JobClient: map 76% reduc
11/07/27 17:42:54 INFO mapred.JobClient: map 92% reduc
11/07/27 17:42:57 INFO mapred.JobClient: map 100% redu
11/07/27 17:43:06 INFO mapred.JobClient: map 100% redu
11/07/27 17:43:09 INFO mapred.JobClient: Job complete:
11/07/27 17:43:09 INFO mapred.JobClient: Counters: 18
11/07/27 17:43:09 INFO mapred.JobClient: Job Counters
11/07/27 17:43:09 INFO mapred.JobClient: Launched r
11/07/27 17:43:09 INFO mapred.JobClient: Launched m
11/07/27 17:43:09 INFO mapred.JobClient: Data-local
11/07/27 17:43:09 INFO mapred.JobClient: FileSystemCo
11/07/27 17:43:09 INFO mapred.JobClient: FILE_BYTES
11/07/27 17:43:09 INFO mapred.JobClient: HDFS_BYTES
11/07/27 17:43:09 INFO mapred.JobClient: FILE_BYTES
11/07/27 17:43:09 INFO mapred.JobClient: HDFS_BYTES
11/07/27 17:43:09 INFO mapred.JobClient: Map-Reduce F
11/07/27 17:43:09 INFO mapred.JobClient: Reduce inp
11/07/27 17:43:09 INFO mapred.JobClient: Combine ou
11/07/27 17:43:09 INFO mapred.JobClient: Map input
11/07/27 17:43:09 INFO mapred.JobClient: Reduce shu
11/07/27 17:43:09 INFO mapred.JobClient: Reduce out
11/07/27 17:43:09 INFO mapred.JobClient: Spilled Re
11/07/27 17:43:09 INFO mapred.JobClient: Map output
11/07/27 17:43:09 INFO mapred.JobClient: Map input
11/07/27 17:43:09 INFO mapred.JobClient: Combine in
11/07/27 17:43:09 INFO mapred.JobClient: Map output
11/07/27 17:43:09 INFO mapred.JobClient: Reduce inp
11/07/27 17:43:09 WARN mapred.JobClient: Use GenericOpt
e arguments. Applications should implement Tool for the
11/07/27 17:43:09 INFO mapred.FileInputFormat: Total in
11/07/27 17:43:09 INFO mapred.JobClient: Running job: j
11/07/27 17:43:10 INFO mapred.JobClient: map 0% reduce
11/07/27 17:43:22 INFO mapred.JobClient: map 100% redu
11/07/27 17:43:31 INFO mapred.JobClient: map 100% redu
11/07/27 17:43:36 INFO mapred.JobClient: map 100% redu
11/07/27 17:43:38 INFO mapred.JobClient: Job complete:
11/07/27 17:43:39 INFO mapred.JobClient: Counters: 18
11/07/27 17:43:39 INFO mapred.JobClient: Job Counters
11/07/27 17:43:39 INFO mapred.JobClient: Launched r
11/07/27 17:43:39 INFO mapred.JobClient: Launched m
11/07/27 17:43:39 INFO mapred.JobClient: Data-local
11/07/27 17:43:39 INFO mapred.JobClient: FileSystemCo
11/07/27 17:43:39 INFO mapred.JobClient: FILE_BYTES
11/07/27 17:43:39 INFO mapred.JobClient: HDFS_BYTES
11/07/27 17:43:39 INFO mapred.JobClient: FILE_BYTES
11/07/27 17:43:39 INFO mapred.JobClient: HDFS_BYTES
11/07/27 17:43:39 INFO mapred.JobClient: Map-Reduce F
11/07/27 17:43:39 INFO mapred.JobClient: Reduce inp
11/07/27 17:43:39 INFO mapred.JobClient: Combine ou
11/07/27 17:43:39 INFO mapred.JobClient: Map input
11/07/27 17:43:39 INFO mapred.JobClient: Reduce shu
11/07/27 17:43:39 INFO mapred.JobClient: Reduce out
11/07/27 17:43:39 INFO mapred.JobClient: Spilled Re
11/07/27 17:43:39 INFO mapred.JobClient: Map output
11/07/27 17:43:39 INFO mapred.JobClient: Map input
11/07/27 17:43:39 INFO mapred.JobClient: Combine in
11/07/27 17:43:39 INFO mapred.JobClient: Map output
11/07/27 17:43:39 INFO mapred.JobClient: Reduce inp
Williams@TWilliams-LTPC ~/hadoop-0.20.2
$ bin/hadoop fs -get output output
Williams@TWilliams-LTPC ~/hadoop-0.20.2
$ cat output/*
cat: output/_logs: Is a directory
3 dfs.class
2 dfs.period
1 dfs.file
1 dfs.replication
1 dfs.servers
1 dfsadmin
1 dfsmetrics.log
</CMD>
Regards,
Uma
******************************************************************************************
This email and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained here in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it!
*****************************************************************************************
----- Original Message -----
From: A Df <abbey_dragonforest@yahoo.com>
Date: Wednesday, July 27, 2011 9:55 pm
Subject: cygwin not connecting to Hadoop server
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Hi All:
I am have Hadoop 0.20.2 and I am using cygwin on Windows 7. I
modified the files as shown below for the Hadoop configuration.
conf/core-site.xml:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9100</value>
</property>
</configuration>
conf/hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
conf/mapred-site.xml:
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9101</value>
</property>
</configuration>
Then I have the PATH variable with
$PATH:/cygdrive/c/cygwin/bin:/cygdrive/c/cygwin/usr/bin
I added JAVA_HOME to the file in cygwin\home\Williams\hadoop-
0.20.2\conf\hadoop-env.sh.
My Java home is now at C:\Java\jdk1.6.0_26 so there is not space. I
also turned off my firewall.
However, I get the error from the command line:
<CODE>
Williams@TWilliams-LTPC ~
$ pwd
/home/Williams
Williams@TWilliams-LTPC ~
$ cd hadoop-0.20.2
Williams@TWilliams-LTPC ~/hadoop-0.20.2
$ bin/start-all.sh
starting namenode, logging to /home/Williams/hadoop-
0.20.2/bin/../logs/hadoop-Wi
lliams-namenode-TWilliams-LTPC.out
localhost: starting datanode, logging to /home/Williams/hadoop-
0.20.2/bin/../logs/hadoop-Williams-datanode-TWilliams-LTPC.out
localhost: starting secondarynamenode, logging to
/home/Williams/hadoop-0.20.2/b
in/../logs/hadoop-Williams-secondarynamenode-TWilliams-LTPC.out
starting jobtracker, logging to /home/Williams/hadoop-
0.20.2/bin/../logs/hadoop-
Williams-jobtracker-TWilliams-LTPC.out
localhost: starting tasktracker, logging to /home/Williams/hadoop-
0.20.2/bin/../logs/hadoop-Williams-tasktracker-TWilliams-LTPC.out
Williams@TWilliams-LTPC ~/hadoop-0.20.2
$ bin/hadoop fs -put conf input
11/07/27 17:11:28 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 0 time(s).
11/07/27 17:11:30 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 1 time(s).
11/07/27 17:11:32 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 2 time(s).
11/07/27 17:11:34 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 3 time(s).
11/07/27 17:11:36 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 4 time(s).
11/07/27 17:11:38 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 5 time(s).
11/07/27 17:11:40 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 6 time(s).
11/07/27 17:11:43 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 7 time(s).
11/07/27 17:11:45 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 8 time(s).
11/07/27 17:11:47 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 9 time(s).
Bad connection to FS. command aborted.
Williams@TWilliams-LTPC ~/hadoop-0.20.2
$ bin/hadoop fs -put conf input
11/07/27 17:17:29 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 0 time(s).
11/07/27 17:17:31 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 1 time(s).
11/07/27 17:17:33 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 2 time(s).
11/07/27 17:17:35 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 3 time(s).
11/07/27 17:17:37 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 4 time(s).
11/07/27 17:17:39 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 5 time(s).
11/07/27 17:17:41 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 6 time(s).
11/07/27 17:17:44 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 7 time(s).
11/07/27 17:17:46 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 8 time(s).
11/07/27 17:17:48 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9100. Already tried 9 time(s).
Bad connection to FS. command aborted.
Williams@TWilliams-LTPC ~/hadoop-0.20.2
$ ping 127.0.0.1:9100
Ping request could not find host 127.0.0.1:9100. Please check the
name and try a
gain.
</CODE>
I am not sure why the ip address seem to have localhost/127.0.0.1
which seems to be repeating itself. The conf files are fine. I also
know that when Hadoop is running there is a web interface to check
but do the default ones work from cygwin which are:
* NameNode -
http://localhost:50070/* JobTracker -
http://localhost:50030/I wanted to give the cygwin a try once more before just switching
to a cloudera hadoop vmware. I was hoping that it would not have so
many problems just to get it working on Windows! Thanks again.
Cheers,
A Df
