[Hadoop-common-user] How to apply Patch

Sorry, Just check the attachment now.

Adarsh Sharma wrote:

There is a utility available for Unix called 'patch'. You can use that
with a suitable -p(num) argument (man patch, for more info).

Thanks Harsh,

I am trying the patch command but below error exists :

[root@ws-test project]# patch hadoop-0.20.2 fix-test-pipes.patch
patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
[root@ws-test project]# patch -R hadoop-0.20.2 fix-test-pipes.patch
patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
[root@ws-test project]# patch -p hadoop-0.20.2 fix-test-pipes.patch
patch: **** strip count hadoop-0.20.2 is not a number
[root@ws-test project]#


Best regards,
Adarsh

Harsh J wrote:

Hi Adarsh,
see if the information at http://wiki.apache.org/hadoop/HowToContribute is helpful to you. I'll walk you thru the typical process, but first a couple questions:

Did you get a tar file for the whole source tree of Hadoop, or only the binary distribution? To apply patches you must get the source tree (usually from the Apache svn server), apply the patch, then re-build.

Also, if you're just starting out, you might consider using a newer version like 0.22 instead of 0.20.

And finally, is the patch you are looking at intended to work with v0.20.2? A patch targeted for 0.22 is only maybe 50% likely to work as intended with a v0.20 version.

So, here's a sketch of how to apply a patch (to v0.20.2) and rebuild:

Per the instructions at http://wiki.apache.org/hadoop/HowToContribute, create a directory for your hadoop projects, and from within that directory do:
svn checkout http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.2/ hadoop-0.20.2

In the same projects directory create a "patches" subdirectory (as a sibling to the "hadoop-0.20.2" subdirectory) and save the patch into it (perhaps from your browser while viewing the Jira)

Now cd into the HADOOP_HOME directory:
cd hadoop-0.20.2
Observe that it has subdirectories like build/, conf/, ivy/, and src/.

Now apply the patch:
patch -p 0 < ../patches/fix-test-pipes.patch

Let's take a look at that last line:
patch -p 0 < ../patches/fix-test-pipes.patch
^^^^ ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Now, re-build hadoop:
ant veryclean compile bin-package

Of course, that assumes you have installed ant and ivy. Each of those utilities have home pages that are very informative, along with the info on the above HowToContribute page. You can also look at the docs at http://hadoop.apache.org/common/docs/current/

You should now have all the pieces necessary to install and run the patched version of Hadoop. (The jars are in the "build" subdirectory in hadoop-0.20.2) In my experience it is always easier to get a vanilla system installed and running, and then overlay that installation with the new jars, rather than try to install from your build directory, but that's just my personal preference.

If you want to work in v21 or higher, you'll have to deal with three hadoop subdirectories instead of one, and use the "ant -Dresolvers=internal mvn-install" invocation to get them to build the patch right across components. But in v20 the above should work fine.

If you are going to work with Hadoop long term, you'll want an IDE like Eclipse. So web pages like http://wiki.apache.org/hadoop/EclipseEnvironment may be helpful.

Hope this helps,
--Matt


On Mar 30, 2011, at 9:24 PM, Adarsh Sharma wrote:

Thanks Harsh,

I am trying the patch command but below error exists :

[root@ws-test project]# patch hadoop-0.20.2 fix-test-pipes.patch
patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
[root@ws-test project]# patch -R hadoop-0.20.2 fix-test-pipes.patch
patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
[root@ws-test project]# patch -p hadoop-0.20.2 fix-test-pipes.patch
patch: **** strip count hadoop-0.20.2 is not a number
[root@ws-test project]#


Best regards,
Adarsh

Harsh J wrote:
There is a utility available for Unix called 'patch'. You can use that
with a suitable -p(num) argument (man patch, for more info).

On Thu, Mar 31, 2011 at 9:41 AM, Adarsh Sharma wrote:

Dear all,

Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.

I attached the patch.

Please find the attachment. I just follow below steps for Hadoop :
1. Download Hadoop-0.20.2.tar.gz
2. Extract the file.
3. Set Configurations in site.xml files

Thanks & best Regards,
Adarsh Sharma

Thanks a lot for such deep explanation :

I have done it now, but it doesn't help me in my original problem for
which I'm doing this.

Please if you have some idea comment on it. I attached the problem.


Thanks & best Regards,
Adarsh Sharma




Matthew Foley wrote:

Sadly. Matt's deep explanation is what you need, low-level that it is

-patches are designed to be applied to source, so you need the apache
source tree, not any binary installations.

-you need to be sure that the source version you have matches that the
patch is designed to be applied against, unless you want to get into the
problem of understanding the source enough to fix inconsistencies.

-you need to rebuild hadoop afterwards.

Because Apache code is open source, patches and the like are all bits of
source. This is not windows where the source is secret and all os
updates are bits of binary code. The view is that if you want to apply
patches, then yes, you do have to play at the source level.

The good news is once you can do that for one patch, you can apply
others, and you will be in a position to find and fix bugs yourself.

-steve

Thanks Steve , U helped me to clear my doubts several times.

I explain U What my Problem is :

I am trying to run *wordcount-nopipe.cc* program in
*/home/hadoop/project/hadoop-0.20.2/src/examples/pipes/impl* directory.
I am able to run a simple wordcount.cpp program in Hadoop Cluster but
whebn I am going to run this program, ifaced below exception :

*bash-3.2$ bin/hadoop pipes -D hadoop.pipes.java.recordreader=true -D
hadoop.pipes.java.recordwriter=true -input gutenberg -output
gutenberg-out1101 -program bin/wordcount-nopipe2*
11/03/31 14:59:07 WARN mapred.JobClient: No job jar file set. User
classes may not be found. See JobConf(Class) or JobConf#setJar(String).
11/03/31 14:59:07 INFO mapred.FileInputFormat: Total input paths to
process : 3
11/03/31 14:59:08 INFO mapred.JobClient: Running job: job_201103310903_0007
11/03/31 14:59:09 INFO mapred.JobClient: map 0% reduce 0%
11/03/31 14:59:18 INFO mapred.JobClient: Task Id :
attempt_201103310903_0007_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.flush(BinaryProtocol.java:316)
at
org.apache.hadoop.mapred.pipes.Application.waitForFinish(Application.java:129)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:99)
... 3 more

attempt_201103310903_0007_m_000000_0: Hadoop Pipes Exception: failed to
open at wordcount-nopipe2.cc:86 in
WordCountReader::WordCountReader(HadoopPipes::MapContext&)
11/03/31 14:59:18 INFO mapred.JobClient: Task Id :
attempt_201103310903_0007_m_000001_0, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.flush(BinaryProtocol.java:316)
at
org.apache.hadoop.mapred.pipes.Application.waitForFinish(Application.java:129)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:99)
... 3 more

After some R&D , i find the below links quite useful :

http://lucene.472066.n3.nabble.com/pipe-application-error-td650185.html
http://stackoverflow.com/questions/4395140/eofexception-thrown-by-a-hadoop-pipes-program

But don't know how to resolve this. I think my program try to open the
file as file://gutenberg but it requires as hdfs://.....

Here is the contents of my Makefile :

CC = g++
HADOOP_INSTALL =/home/hadoop/project/hadoop-0.20.2
PLATFORM = Linux-amd64-64
CPPFLAGS = -m64
-I/home/hadoop/project/hadoop-0.20.2/c++/Linux-amd64-64/include
-I/usr/local/cuda/include

wordcount-nopipe2 : wordcount-nopipe2.cc
$(CC) $(CPPFLAGS) $< -Wall
-L/home/hadoop/project/hadoop-0.20.2/c++/Linux-amd64-64/lib
-L/usr/local/cuda/lib64 -lhadooppipes \
-lhadooputils -lpthread -g -O2 -o $@

Would it be a bug in hadoop-0.20.2 and if not Please guide me how to
debug it.



Thanks & best Regards,
Adarsh Sharma












Steve Loughran wrote:

Adarsh,
Your command should be :
patch -p0 < fix-test-pipes.patch

See http://wiki.apache.org/hadoop/HowToContribute for details on how to contribute.

Thanks
Amareshwari

On 3/31/11 9:54 AM, "Adarsh Sharma" wrote:

Thanks Harsh,

I am trying the patch command but below error exists :

[root@ws-test project]# patch hadoop-0.20.2 fix-test-pipes.patch
patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
[root@ws-test project]# patch -R hadoop-0.20.2 fix-test-pipes.patch
patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
[root@ws-test project]# patch -p hadoop-0.20.2 fix-test-pipes.patch
patch: **** strip count hadoop-0.20.2 is not a number
[root@ws-test project]#


Best regards,
Adarsh

Harsh J wrote: