FAQ
Dear all,

Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.

I attached the patch.

Please find the attachment. I just follow below steps for Hadoop :
1. Download Hadoop-0.20.2.tar.gz
2. Extract the file.
3. Set Configurations in site.xml files

Thanks & best Regards,
Adarsh Sharma

Search Discussions

  • Adarsh Sharma at Mar 31, 2011 at 4:12 am
    Sorry, Just check the attachment now.

    Adarsh Sharma wrote:
    Dear all,

    Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.

    I attached the patch.

    Please find the attachment. I just follow below steps for Hadoop :
    1. Download Hadoop-0.20.2.tar.gz
    2. Extract the file.
    3. Set Configurations in site.xml files

    Thanks & best Regards,
    Adarsh Sharma
  • Harsh J at Mar 31, 2011 at 4:19 am
    There is a utility available for Unix called 'patch'. You can use that
    with a suitable -p(num) argument (man patch, for more info).
    On Thu, Mar 31, 2011 at 9:41 AM, Adarsh Sharma wrote:
    Dear all,

    Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.

    I attached the patch.

    Please find the attachment. I just follow below steps for Hadoop :
    1. Download Hadoop-0.20.2.tar.gz
    2. Extract the file.
    3. Set Configurations in site.xml files

    Thanks & best Regards,
    Adarsh Sharma


    --
    Harsh J
    http://harshj.com
  • Adarsh Sharma at Mar 31, 2011 at 4:24 am
    Thanks Harsh,

    I am trying the patch command but below error exists :

    [root@ws-test project]# patch hadoop-0.20.2 fix-test-pipes.patch
    patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
    [root@ws-test project]# patch -R hadoop-0.20.2 fix-test-pipes.patch
    patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
    [root@ws-test project]# patch -p hadoop-0.20.2 fix-test-pipes.patch
    patch: **** strip count hadoop-0.20.2 is not a number
    [root@ws-test project]#


    Best regards,
    Adarsh

    Harsh J wrote:
    There is a utility available for Unix called 'patch'. You can use that
    with a suitable -p(num) argument (man patch, for more info).
    On Thu, Mar 31, 2011 at 9:41 AM, Adarsh Sharma wrote:

    Dear all,

    Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.

    I attached the patch.

    Please find the attachment. I just follow below steps for Hadoop :
    1. Download Hadoop-0.20.2.tar.gz
    2. Extract the file.
    3. Set Configurations in site.xml files

    Thanks & best Regards,
    Adarsh Sharma

  • Matthew Foley at Mar 31, 2011 at 6:28 am
    Hi Adarsh,
    see if the information at http://wiki.apache.org/hadoop/HowToContribute is helpful to you. I'll walk you thru the typical process, but first a couple questions:

    Did you get a tar file for the whole source tree of Hadoop, or only the binary distribution? To apply patches you must get the source tree (usually from the Apache svn server), apply the patch, then re-build.

    Also, if you're just starting out, you might consider using a newer version like 0.22 instead of 0.20.

    And finally, is the patch you are looking at intended to work with v0.20.2? A patch targeted for 0.22 is only maybe 50% likely to work as intended with a v0.20 version.

    So, here's a sketch of how to apply a patch (to v0.20.2) and rebuild:

    Per the instructions at http://wiki.apache.org/hadoop/HowToContribute, create a directory for your hadoop projects, and from within that directory do:
    svn checkout http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.2/ hadoop-0.20.2

    In the same projects directory create a "patches" subdirectory (as a sibling to the "hadoop-0.20.2" subdirectory) and save the patch into it (perhaps from your browser while viewing the Jira)

    Now cd into the HADOOP_HOME directory:
    cd hadoop-0.20.2
    Observe that it has subdirectories like build/, conf/, ivy/, and src/.

    Now apply the patch:
    patch -p 0 < ../patches/fix-test-pipes.patch

    Let's take a look at that last line:
    patch -p 0 < ../patches/fix-test-pipes.patch
    ^^^^ ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    _where you stored the patch file
    _the patch command is funny, you locate yourself at the root of
    the source tree, then pipe the patch file contents into the "patch"
    command. The target source directory name is not an argument.

    _this "-p 0" argument is generally applicable for patches generated the way we like to do patches in the hadoop dev community. Read the unix man page for "patch", and http://wiki.apache.org/hadoop/HowToContribute if you're interested.
    Now, re-build hadoop:
    ant veryclean compile bin-package

    Of course, that assumes you have installed ant and ivy. Each of those utilities have home pages that are very informative, along with the info on the above HowToContribute page. You can also look at the docs at http://hadoop.apache.org/common/docs/current/

    You should now have all the pieces necessary to install and run the patched version of Hadoop. (The jars are in the "build" subdirectory in hadoop-0.20.2) In my experience it is always easier to get a vanilla system installed and running, and then overlay that installation with the new jars, rather than try to install from your build directory, but that's just my personal preference.

    If you want to work in v21 or higher, you'll have to deal with three hadoop subdirectories instead of one, and use the "ant -Dresolvers=internal mvn-install" invocation to get them to build the patch right across components. But in v20 the above should work fine.

    If you are going to work with Hadoop long term, you'll want an IDE like Eclipse. So web pages like http://wiki.apache.org/hadoop/EclipseEnvironment may be helpful.

    Hope this helps,
    --Matt


    On Mar 30, 2011, at 9:24 PM, Adarsh Sharma wrote:

    Thanks Harsh,

    I am trying the patch command but below error exists :

    [root@ws-test project]# patch hadoop-0.20.2 fix-test-pipes.patch
    patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
    [root@ws-test project]# patch -R hadoop-0.20.2 fix-test-pipes.patch
    patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
    [root@ws-test project]# patch -p hadoop-0.20.2 fix-test-pipes.patch
    patch: **** strip count hadoop-0.20.2 is not a number
    [root@ws-test project]#


    Best regards,
    Adarsh

    Harsh J wrote:
    There is a utility available for Unix called 'patch'. You can use that
    with a suitable -p(num) argument (man patch, for more info).

    On Thu, Mar 31, 2011 at 9:41 AM, Adarsh Sharma wrote:

    Dear all,

    Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.

    I attached the patch.

    Please find the attachment. I just follow below steps for Hadoop :
    1. Download Hadoop-0.20.2.tar.gz
    2. Extract the file.
    3. Set Configurations in site.xml files

    Thanks & best Regards,
    Adarsh Sharma
  • Adarsh Sharma at Mar 31, 2011 at 6:38 am
    Thanks a lot for such deep explanation :

    I have done it now, but it doesn't help me in my original problem for
    which I'm doing this.

    Please if you have some idea comment on it. I attached the problem.


    Thanks & best Regards,
    Adarsh Sharma




    Matthew Foley wrote:
    Hi Adarsh,
    see if the information at http://wiki.apache.org/hadoop/HowToContribute is helpful to you. I'll walk you thru the typical process, but first a couple questions:

    Did you get a tar file for the whole source tree of Hadoop, or only the binary distribution? To apply patches you must get the source tree (usually from the Apache svn server), apply the patch, then re-build.

    Also, if you're just starting out, you might consider using a newer version like 0.22 instead of 0.20.

    And finally, is the patch you are looking at intended to work with v0.20.2? A patch targeted for 0.22 is only maybe 50% likely to work as intended with a v0.20 version.

    So, here's a sketch of how to apply a patch (to v0.20.2) and rebuild:

    Per the instructions at http://wiki.apache.org/hadoop/HowToContribute, create a directory for your hadoop projects, and from within that directory do:
    svn checkout http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.2/ hadoop-0.20.2

    In the same projects directory create a "patches" subdirectory (as a sibling to the "hadoop-0.20.2" subdirectory) and save the patch into it (perhaps from your browser while viewing the Jira)

    Now cd into the HADOOP_HOME directory:
    cd hadoop-0.20.2
    Observe that it has subdirectories like build/, conf/, ivy/, and src/.

    Now apply the patch:
    patch -p 0 < ../patches/fix-test-pipes.patch

    Let's take a look at that last line:
    patch -p 0 < ../patches/fix-test-pipes.patch
    ^^^^ ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    _where you stored the patch file
    _the patch command is funny, you locate yourself at the root of
    the source tree, then pipe the patch file contents into the "patch"
    command. The target source directory name is not an argument.

    _this "-p 0" argument is generally applicable for patches generated the way we like to do patches in the hadoop dev community. Read the unix man page for "patch", and http://wiki.apache.org/hadoop/HowToContribute if you're interested.
    Now, re-build hadoop:
    ant veryclean compile bin-package

    Of course, that assumes you have installed ant and ivy. Each of those utilities have home pages that are very informative, along with the info on the above HowToContribute page. You can also look at the docs at http://hadoop.apache.org/common/docs/current/

    You should now have all the pieces necessary to install and run the patched version of Hadoop. (The jars are in the "build" subdirectory in hadoop-0.20.2) In my experience it is always easier to get a vanilla system installed and running, and then overlay that installation with the new jars, rather than try to install from your build directory, but that's just my personal preference.

    If you want to work in v21 or higher, you'll have to deal with three hadoop subdirectories instead of one, and use the "ant -Dresolvers=internal mvn-install" invocation to get them to build the patch right across components. But in v20 the above should work fine.

    If you are going to work with Hadoop long term, you'll want an IDE like Eclipse. So web pages like http://wiki.apache.org/hadoop/EclipseEnvironment may be helpful.

    Hope this helps,
    --Matt
  • Steve Loughran at Mar 31, 2011 at 9:42 am

    On 31/03/11 07:37, Adarsh Sharma wrote:
    Thanks a lot for such deep explanation :

    I have done it now, but it doesn't help me in my original problem for
    which I'm doing this.

    Please if you have some idea comment on it. I attached the problem.
    Sadly. Matt's deep explanation is what you need, low-level that it is

    -patches are designed to be applied to source, so you need the apache
    source tree, not any binary installations.

    -you need to be sure that the source version you have matches that the
    patch is designed to be applied against, unless you want to get into the
    problem of understanding the source enough to fix inconsistencies.

    -you need to rebuild hadoop afterwards.

    Because Apache code is open source, patches and the like are all bits of
    source. This is not windows where the source is secret and all os
    updates are bits of binary code. The view is that if you want to apply
    patches, then yes, you do have to play at the source level.

    The good news is once you can do that for one patch, you can apply
    others, and you will be in a position to find and fix bugs yourself.

    -steve
  • Adarsh Sharma at Mar 31, 2011 at 9:54 am
    Thanks Steve , U helped me to clear my doubts several times.

    I explain U What my Problem is :

    I am trying to run *wordcount-nopipe.cc* program in
    */home/hadoop/project/hadoop-0.20.2/src/examples/pipes/impl* directory.
    I am able to run a simple wordcount.cpp program in Hadoop Cluster but
    whebn I am going to run this program, ifaced below exception :

    *bash-3.2$ bin/hadoop pipes -D hadoop.pipes.java.recordreader=true -D
    hadoop.pipes.java.recordwriter=true -input gutenberg -output
    gutenberg-out1101 -program bin/wordcount-nopipe2*
    11/03/31 14:59:07 WARN mapred.JobClient: No job jar file set. User
    classes may not be found. See JobConf(Class) or JobConf#setJar(String).
    11/03/31 14:59:07 INFO mapred.FileInputFormat: Total input paths to
    process : 3
    11/03/31 14:59:08 INFO mapred.JobClient: Running job: job_201103310903_0007
    11/03/31 14:59:09 INFO mapred.JobClient: map 0% reduce 0%
    11/03/31 14:59:18 INFO mapred.JobClient: Task Id :
    attempt_201103310903_0007_m_000000_0, Status : FAILED
    java.io.IOException: pipe child exception
    at
    org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
    at
    org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.net.SocketException: Broken pipe
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at
    java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at
    java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
    at java.io.DataOutputStream.flush(DataOutputStream.java:106)
    at
    org.apache.hadoop.mapred.pipes.BinaryProtocol.flush(BinaryProtocol.java:316)
    at
    org.apache.hadoop.mapred.pipes.Application.waitForFinish(Application.java:129)
    at
    org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:99)
    ... 3 more

    attempt_201103310903_0007_m_000000_0: Hadoop Pipes Exception: failed to
    open at wordcount-nopipe2.cc:86 in
    WordCountReader::WordCountReader(HadoopPipes::MapContext&)
    11/03/31 14:59:18 INFO mapred.JobClient: Task Id :
    attempt_201103310903_0007_m_000001_0, Status : FAILED
    java.io.IOException: pipe child exception
    at
    org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
    at
    org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.net.SocketException: Broken pipe
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at
    java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at
    java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
    at java.io.DataOutputStream.flush(DataOutputStream.java:106)
    at
    org.apache.hadoop.mapred.pipes.BinaryProtocol.flush(BinaryProtocol.java:316)
    at
    org.apache.hadoop.mapred.pipes.Application.waitForFinish(Application.java:129)
    at
    org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:99)
    ... 3 more

    After some R&D , i find the below links quite useful :

    http://lucene.472066.n3.nabble.com/pipe-application-error-td650185.html
    http://stackoverflow.com/questions/4395140/eofexception-thrown-by-a-hadoop-pipes-program

    But don't know how to resolve this. I think my program try to open the
    file as file://gutenberg but it requires as hdfs://.....

    Here is the contents of my Makefile :

    CC = g++
    HADOOP_INSTALL =/home/hadoop/project/hadoop-0.20.2
    PLATFORM = Linux-amd64-64
    CPPFLAGS = -m64
    -I/home/hadoop/project/hadoop-0.20.2/c++/Linux-amd64-64/include
    -I/usr/local/cuda/include

    wordcount-nopipe2 : wordcount-nopipe2.cc
    $(CC) $(CPPFLAGS) $< -Wall
    -L/home/hadoop/project/hadoop-0.20.2/c++/Linux-amd64-64/lib
    -L/usr/local/cuda/lib64 -lhadooppipes \
    -lhadooputils -lpthread -g -O2 -o $@

    Would it be a bug in hadoop-0.20.2 and if not Please guide me how to
    debug it.



    Thanks & best Regards,
    Adarsh Sharma












    Steve Loughran wrote:
    On 31/03/11 07:37, Adarsh Sharma wrote:
    Thanks a lot for such deep explanation :

    I have done it now, but it doesn't help me in my original problem for
    which I'm doing this.

    Please if you have some idea comment on it. I attached the problem.
    Sadly. Matt's deep explanation is what you need, low-level that it is

    -patches are designed to be applied to source, so you need the apache
    source tree, not any binary installations.

    -you need to be sure that the source version you have matches that the
    patch is designed to be applied against, unless you want to get into
    the problem of understanding the source enough to fix inconsistencies.

    -you need to rebuild hadoop afterwards.

    Because Apache code is open source, patches and the like are all bits
    of source. This is not windows where the source is secret and all os
    updates are bits of binary code. The view is that if you want to apply
    patches, then yes, you do have to play at the source level.

    The good news is once you can do that for one patch, you can apply
    others, and you will be in a position to find and fix bugs yourself.

    -steve
  • Amareshwari Sri Ramadasu at Mar 31, 2011 at 6:29 am
    Adarsh,
    Your command should be :
    patch -p0 < fix-test-pipes.patch

    See http://wiki.apache.org/hadoop/HowToContribute for details on how to contribute.

    Thanks
    Amareshwari

    On 3/31/11 9:54 AM, "Adarsh Sharma" wrote:

    Thanks Harsh,

    I am trying the patch command but below error exists :

    [root@ws-test project]# patch hadoop-0.20.2 fix-test-pipes.patch
    patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
    [root@ws-test project]# patch -R hadoop-0.20.2 fix-test-pipes.patch
    patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
    [root@ws-test project]# patch -p hadoop-0.20.2 fix-test-pipes.patch
    patch: **** strip count hadoop-0.20.2 is not a number
    [root@ws-test project]#


    Best regards,
    Adarsh

    Harsh J wrote:
    There is a utility available for Unix called 'patch'. You can use that
    with a suitable -p(num) argument (man patch, for more info).
    On Thu, Mar 31, 2011 at 9:41 AM, Adarsh Sharma wrote:

    Dear all,

    Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.

    I attached the patch.

    Please find the attachment. I just follow below steps for Hadoop :
    1. Download Hadoop-0.20.2.tar.gz
    2. Extract the file.
    3. Set Configurations in site.xml files

    Thanks & best Regards,
    Adarsh Sharma

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMar 31, '11 at 4:11a
activeMar 31, '11 at 9:54a
posts9
users5
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase