FAQ
I am starting to learn Hadoop, using the Yahoo virtual machine with version
0.18.

My question is rather simple. I would like to execute a map/reduce job. In
addition to getting the results from the reduce, I would also like to save
the intermediate results from the map in another HDFS file. Is this
possible?

--gordon

Search Discussions

  • Jason Venner at Nov 23, 2009 at 2:28 am
    You can manually write the map output to a new file, there are a number of
    examples of opening a sequence file and writing to it on the web or in the
    example code for various hadoop books.

    You can also disable the removal of intermediate data, which will result in
    potentially large amounts of data being left in the mapred.local.dir.

    On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff wrote:

    I am starting to learn Hadoop, using the Yahoo virtual machine with version
    0.18.

    My question is rather simple. I would like to execute a map/reduce job.
    In
    addition to getting the results from the reduce, I would also like to save
    the intermediate results from the map in another HDFS file. Is this
    possible?

    --gordon


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Jeff Zhang at Nov 23, 2009 at 2:39 am
    Hi Jason,

    which option is for setting disable the removal of intermediate data ?

    Thank you

    Jeff Zhang

    On Mon, Nov 23, 2009 at 10:27 AM, Jason Venner wrote:

    You can manually write the map output to a new file, there are a number of
    examples of opening a sequence file and writing to it on the web or in the
    example code for various hadoop books.

    You can also disable the removal of intermediate data, which will result in
    potentially large amounts of data being left in the mapred.local.dir.

    On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff wrote:

    I am starting to learn Hadoop, using the Yahoo virtual machine with version
    0.18.

    My question is rather simple. I would like to execute a map/reduce job.
    In
    addition to getting the results from the reduce, I would also like to save
    the intermediate results from the map in another HDFS file. Is this
    possible?

    --gordon


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Amogh Vasekar at Nov 23, 2009 at 7:51 am
    Hi,
    keep.tasks.files.pattern is what you need, as the name suggests its a pattern match on intermediate outputs generated.

    Wrt to copying map data to hdfs, your mappers close() method should help you achieve this, but might slow up your tasks.

    Amogh


    On 11/23/09 8:08 AM, "Jeff Zhang" wrote:

    Hi Jason,

    which option is for setting disable the removal of intermediate data ?

    Thank you

    Jeff Zhang

    On Mon, Nov 23, 2009 at 10:27 AM, Jason Venner wrote:

    You can manually write the map output to a new file, there are a number of
    examples of opening a sequence file and writing to it on the web or in the
    example code for various hadoop books.

    You can also disable the removal of intermediate data, which will result in
    potentially large amounts of data being left in the mapred.local.dir.

    On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff wrote:

    I am starting to learn Hadoop, using the Yahoo virtual machine with version
    0.18.

    My question is rather simple. I would like to execute a map/reduce job.
    In
    addition to getting the results from the reduce, I would also like to save
    the intermediate results from the map in another HDFS file. Is this
    possible?

    --gordon


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Jason Venner at Nov 23, 2009 at 2:00 pm
    <property>
    <name>keep.task.files.pattern</name>
    <value>.*_m_123456_0</value>
    <description>Keep all files from tasks whose task names match the given
    regular expression. Defaults to none.</description>
    </property>
    On Sun, Nov 22, 2009 at 11:46 PM, Amogh Vasekar wrote:

    Hi,
    keep.tasks.files.pattern is what you need, as the name suggests its a
    pattern match on intermediate outputs generated.

    Wrt to copying map data to hdfs, your mappers close() method should help
    you achieve this, but might slow up your tasks.

    Amogh


    On 11/23/09 8:08 AM, "Jeff Zhang" wrote:

    Hi Jason,

    which option is for setting disable the removal of intermediate data ?

    Thank you

    Jeff Zhang


    On Mon, Nov 23, 2009 at 10:27 AM, Jason Venner <jason.hadoop@gmail.com
    wrote:
    You can manually write the map output to a new file, there are a number of
    examples of opening a sequence file and writing to it on the web or in the
    example code for various hadoop books.

    You can also disable the removal of intermediate data, which will result in
    potentially large amounts of data being left in the mapred.local.dir.

    On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff wrote:

    I am starting to learn Hadoop, using the Yahoo virtual machine with version
    0.18.

    My question is rather simple. I would like to execute a map/reduce
    job.
    In
    addition to getting the results from the reduce, I would also like to save
    the intermediate results from the map in another HDFS file. Is this
    possible?

    --gordon


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals

    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Gordon Linoff at Nov 25, 2009 at 4:05 am
    Does anyone have a pointer to code that allows the map to save data in
    intermediate files, for use in a later map/reduce job? I have been looking
    for an example and cannot find one.

    I have investigated MultipleOutputFormat and MultipleOutputs. Because I am
    using version 0.18.3, I don't have MultipleOutputs. The problem with
    MultipleOutputFormat is that the data I want to save is a different format
    from the data I want to pass to the Reducer. I have also tried opening a
    sequence file directly from the mapper, but I am concerned that this is not
    fault tolerant.

    The process currently is:

    Job1: Mapper: reads complicated data, saves out data structure.
    Job2: Mapper: reads saved data, processes and sends data to Reducer 2.
    Job3: Mapper: reads saved data, processes and sends data to Reducer 3.

    I would like to combine the first two steps, so the process is:

    Job1: Mapper: reads complicated data, saves out data structure, and passes
    processed data to Reducer 2.
    Job2: Mapper: reads saved data, processes and sends to Reducer 3.

    --gordon


    On Sun, Nov 22, 2009 at 9:27 PM, Jason Venner wrote:

    You can manually write the map output to a new file, there are a number of
    examples of opening a sequence file and writing to it on the web or in the
    example code for various hadoop books.

    You can also disable the removal of intermediate data, which will result in
    potentially large amounts of data being left in the mapred.local.dir.


    On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff wrote:

    I am starting to learn Hadoop, using the Yahoo virtual machine with
    version
    0.18.

    My question is rather simple. I would like to execute a map/reduce job.
    In
    addition to getting the results from the reduce, I would also like to save
    the intermediate results from the map in another HDFS file. Is this
    possible?

    --gordon


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Amogh Vasekar at Nov 25, 2009 at 6:50 am
    Hi,
    I'm not sure if this will apply to your case since i'm not aware of the common part of job2:mapper and job3:mapper but would like to give it a shot.
    The whole process can be combined into a single mapred job. The mapper will read a record and process till the "saved data part" , then for each record will output 2 records , one each for the job2 and job3 mappers. The keys of records will be tagged ( <tag,key> ) depending on what reducer processing you want to do. In reduce() you can use this tag to make processing decision. A custom partitioner might be needed depending on the key types to ensure unique sets for reducer.
    Ignore if this doesn't fit your bill :)

    Amogh


    On 11/25/09 9:35 AM, "Gordon Linoff" wrote:

    Does anyone have a pointer to code that allows the map to save data in
    intermediate files, for use in a later map/reduce job? I have been looking
    for an example and cannot find one.

    I have investigated MultipleOutputFormat and MultipleOutputs. Because I am
    using version 0.18.3, I don't have MultipleOutputs. The problem with
    MultipleOutputFormat is that the data I want to save is a different format
    from the data I want to pass to the Reducer. I have also tried opening a
    sequence file directly from the mapper, but I am concerned that this is not
    fault tolerant.

    The process currently is:

    Job1: Mapper: reads complicated data, saves out data structure.
    Job2: Mapper: reads saved data, processes and sends data to Reducer 2.
    Job3: Mapper: reads saved data, processes and sends data to Reducer 3.

    I would like to combine the first two steps, so the process is:

    Job1: Mapper: reads complicated data, saves out data structure, and passes
    processed data to Reducer 2.
    Job2: Mapper: reads saved data, processes and sends to Reducer 3.

    --gordon


    On Sun, Nov 22, 2009 at 9:27 PM, Jason Venner wrote:

    You can manually write the map output to a new file, there are a number of
    examples of opening a sequence file and writing to it on the web or in the
    example code for various hadoop books.

    You can also disable the removal of intermediate data, which will result in
    potentially large amounts of data being left in the mapred.local.dir.


    On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff wrote:

    I am starting to learn Hadoop, using the Yahoo virtual machine with
    version
    0.18.

    My question is rather simple. I would like to execute a map/reduce job.
    In
    addition to getting the results from the reduce, I would also like to save
    the intermediate results from the map in another HDFS file. Is this
    possible?

    --gordon


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Rekha Joshi at Nov 25, 2009 at 7:09 am
    https://issues.apache.org/jira/browse/HADOOP-372 has valuable information on InputFormat/MapInput/RecordReader, you may try using the pseudo code. Thanks!

    On 11/25/09 9:35 AM, "Gordon Linoff" wrote:

    Does anyone have a pointer to code that allows the map to save data in
    intermediate files, for use in a later map/reduce job? I have been looking
    for an example and cannot find one.

    I have investigated MultipleOutputFormat and MultipleOutputs. Because I am
    using version 0.18.3, I don't have MultipleOutputs. The problem with
    MultipleOutputFormat is that the data I want to save is a different format
    from the data I want to pass to the Reducer. I have also tried opening a
    sequence file directly from the mapper, but I am concerned that this is not
    fault tolerant.

    The process currently is:

    Job1: Mapper: reads complicated data, saves out data structure.
    Job2: Mapper: reads saved data, processes and sends data to Reducer 2.
    Job3: Mapper: reads saved data, processes and sends data to Reducer 3.

    I would like to combine the first two steps, so the process is:

    Job1: Mapper: reads complicated data, saves out data structure, and passes
    processed data to Reducer 2.
    Job2: Mapper: reads saved data, processes and sends to Reducer 3.

    --gordon


    On Sun, Nov 22, 2009 at 9:27 PM, Jason Venner wrote:

    You can manually write the map output to a new file, there are a number of
    examples of opening a sequence file and writing to it on the web or in the
    example code for various hadoop books.

    You can also disable the removal of intermediate data, which will result in
    potentially large amounts of data being left in the mapred.local.dir.


    On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff wrote:

    I am starting to learn Hadoop, using the Yahoo virtual machine with
    version
    0.18.

    My question is rather simple. I would like to execute a map/reduce job.
    In
    addition to getting the results from the reduce, I would also like to save
    the intermediate results from the map in another HDFS file. Is this
    possible?

    --gordon


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Raymond Jennings III at Nov 23, 2009 at 4:46 pm
    I am running ant on windows from the src/contrib/eclipse-plugin directory
    Using jdk1.6.0_18:

    Anyone have any experience with this? Thank you.



    [ivy:retrieve] 0 artifacts copied, 2 already retrieved (0kB/0ms)
    No ivy:settings found for the default reference 'ivy.instance'. A default insta
    nce will be used
    DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
    :: loading settings :: file = E:\hadoop\hadoop-0.20.1\ivy\ivysettings.xml

    compile:
    [echo] contrib: eclipse-plugin
    [javac] Compiling 45 source files to E:\hadoop\hadoop-0.20.1\build\contrib\eclipse-plugin\classes
    [javac] E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:35: cannot find symbol
    [javac] symbol : class JavaApplicationLaunchShortcut
    [javac] location: package org.eclipse.jdt.internal.debug.ui.launcher
    [javac] import org.eclipse.jdt.internal.debug.ui.launcher.JavaApplicationLaunchShortcut;
    [javac] ^
    [javac] E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:49: cannot find symbol
    [javac] symbol: class JavaApplicationLaunchShortcut
    [javac] JavaApplicationLaunchShortcut {
    [javac] ^
    [javac] E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:66: cannot find symbol
    [javac] symbol : variable super
    [javac] location: class org.apache.hadoop.eclipse.launch.HadoopApplicationLa
    unchShortcut
    [javac] super.findLaunchConfiguration(type, configType);
    [javac] ^
    [javac] E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:60: method does not override or implement a method from a supertype
    [javac] @Override
    [javac] ^
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 4 errors

    BUILD FAILED
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\build.xml:63: Compile failed;
    see the compiler error output for details.
  • Stephen Watt at Nov 23, 2009 at 5:02 pm
    You'll need to be more specific about which version of Hadoop and Eclipse
    you're using. There are known issues building the plugin on Hadoop 0.20.1.
    I am in the process of providing a patch for this, in the interim, you
    can try using the plugin jar attached at
    http://issues.apache.org/jira/browse/HADOOP-6360

    Kind regards
    Steve Watt



    From:
    Raymond Jennings III <raymondjiii@yahoo.com>
    To:
    common-user@hadoop.apache.org
    Date:
    11/23/2009 10:46 AM
    Subject:
    Error trying to build hadoop eclipse plugin



    I am running ant on windows from the src/contrib/eclipse-plugin directory
    Using jdk1.6.0_18:

    Anyone have any experience with this? Thank you.



    [ivy:retrieve] 0 artifacts copied, 2 already retrieved (0kB/0ms)
    No ivy:settings found for the default reference 'ivy.instance'. A default
    insta
    nce will be used
    DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
    :: loading settings :: file = E:\hadoop\hadoop-0.20.1\ivy\ivysettings.xml

    compile:
    [echo] contrib: eclipse-plugin
    [javac] Compiling 45 source files to
    E:\hadoop\hadoop-0.20.1\build\contrib\eclipse-plugin\classes
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:35:
    cannot find symbol
    [javac] symbol : class JavaApplicationLaunchShortcut
    [javac] location: package org.eclipse.jdt.internal.debug.ui.launcher
    [javac] import
    org.eclipse.jdt.internal.debug.ui.launcher.JavaApplicationLaunchShortcut;
    [javac] ^
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:49:
    cannot find symbol
    [javac] symbol: class JavaApplicationLaunchShortcut
    [javac] JavaApplicationLaunchShortcut {
    [javac] ^
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:66:
    cannot find symbol
    [javac] symbol : variable super
    [javac] location: class
    org.apache.hadoop.eclipse.launch.HadoopApplicationLa
    unchShortcut
    [javac] super.findLaunchConfiguration(type, configType);
    [javac] ^
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:60:
    method does not override or implement a method from a supertype
    [javac] @Override
    [javac] ^
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 4 errors

    BUILD FAILED
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\build.xml:63: Compile
    failed;
    see the compiler error output for details.
  • Raymond Jennings III at Nov 23, 2009 at 5:17 pm
    Hi, it's hadoop-0.20.1 and eclipse Ganymede. I did try that prebuilt plugin over the weekend but no luck with that on Windows. Thank you.


    --- On Mon, 11/23/09, Stephen Watt wrote:
    From: Stephen Watt <swatt@us.ibm.com>
    Subject: Re: Error trying to build hadoop eclipse plugin
    To: common-user@hadoop.apache.org
    Cc: common-user@hadoop.apache.org
    Date: Monday, November 23, 2009, 12:01 PM
    You'll need to be more specific about
    which version of Hadoop and Eclipse
    you're using. There are known issues building the plugin on
    Hadoop 0.20.1.
    I am in the process of providing a patch for this, in the
    interim, you
    can try using the plugin jar attached at
    http://issues.apache.org/jira/browse/HADOOP-6360

    Kind regards
    Steve Watt



    From:
    Raymond Jennings III <raymondjiii@yahoo.com>
    To:
    common-user@hadoop.apache.org
    Date:
    11/23/2009 10:46 AM
    Subject:
    Error trying to build hadoop eclipse plugin



    I am running ant on windows from the
    src/contrib/eclipse-plugin directory
    Using jdk1.6.0_18:

    Anyone have any experience with this?  Thank you.



    [ivy:retrieve]  0 artifacts copied, 2 already
    retrieved (0kB/0ms)
    No ivy:settings found for the default reference
    'ivy.instance'.  A default
    insta
    nce will be used
    DEPRECATED: 'ivy.conf.file' is deprecated, use
    'ivy.settings.file' instead
    :: loading settings :: file =
    E:\hadoop\hadoop-0.20.1\ivy\ivysettings.xml

    compile:
    [echo] contrib: eclipse-plugin
    [javac] Compiling 45 source files to
    E:\hadoop\hadoop-0.20.1\build\contrib\eclipse-plugin\classes
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:35:

    cannot find symbol
    [javac] symbol  : class
    JavaApplicationLaunchShortcut
    [javac] location: package
    org.eclipse.jdt.internal.debug.ui.launcher
    [javac] import
    org.eclipse.jdt.internal.debug.ui.launcher.JavaApplicationLaunchShortcut;
    [javac]


    ^
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:49:

    cannot find symbol
    [javac] symbol: class
    JavaApplicationLaunchShortcut
    [javac]
    JavaApplicationLaunchShortcut {
    [javac]     ^
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:66:

    cannot find symbol
    [javac] symbol  : variable super
    [javac] location: class
    org.apache.hadoop.eclipse.launch.HadoopApplicationLa
    unchShortcut
    [javac]
    super.findLaunchConfiguration(type,
    configType);
    [javac]
    ^
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:60:

    method does not override or implement a method from a
    supertype
    [javac]   @Override
    [javac]   ^
    [javac] Note: Some input files use or
    override a deprecated API.
    [javac] Note: Recompile with
    -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked
    or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked
    for details.
    [javac] 4 errors

    BUILD FAILED
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\build.xml:63:
    Compile
    failed;
    see the compiler error output for details.




  • Stephen Watt at Nov 23, 2009 at 5:26 pm
    Can you go into detail regarding your scenario. What exactly is failing
    and how is it failing ?

    Kind regards
    Steve Watt



    From:
    Raymond Jennings III <raymondjiii@yahoo.com>
    To:
    common-user@hadoop.apache.org
    Date:
    11/23/2009 11:17 AM
    Subject:
    Re: Error trying to build hadoop eclipse plugin



    Hi, it's hadoop-0.20.1 and eclipse Ganymede. I did try that prebuilt
    plugin over the weekend but no luck with that on Windows. Thank you.


    --- On Mon, 11/23/09, Stephen Watt wrote:
    From: Stephen Watt <swatt@us.ibm.com>
    Subject: Re: Error trying to build hadoop eclipse plugin
    To: common-user@hadoop.apache.org
    Cc: common-user@hadoop.apache.org
    Date: Monday, November 23, 2009, 12:01 PM
    You'll need to be more specific about
    which version of Hadoop and Eclipse
    you're using. There are known issues building the plugin on
    Hadoop 0.20.1.
    I am in the process of providing a patch for this, in the
    interim, you
    can try using the plugin jar attached at
    http://issues.apache.org/jira/browse/HADOOP-6360

    Kind regards
    Steve Watt



    From:
    Raymond Jennings III <raymondjiii@yahoo.com>
    To:
    common-user@hadoop.apache.org
    Date:
    11/23/2009 10:46 AM
    Subject:
    Error trying to build hadoop eclipse plugin



    I am running ant on windows from the
    src/contrib/eclipse-plugin directory
    Using jdk1.6.0_18:

    Anyone have any experience with this? Thank you.



    [ivy:retrieve] 0 artifacts copied, 2 already
    retrieved (0kB/0ms)
    No ivy:settings found for the default reference
    'ivy.instance'. A default
    insta
    nce will be used
    DEPRECATED: 'ivy.conf.file' is deprecated, use
    'ivy.settings.file' instead
    :: loading settings :: file =
    E:\hadoop\hadoop-0.20.1\ivy\ivysettings.xml

    compile:
    [echo] contrib: eclipse-plugin
    [javac] Compiling 45 source files to
    E:\hadoop\hadoop-0.20.1\build\contrib\eclipse-plugin\classes
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:35:
    cannot find symbol
    [javac] symbol : class
    JavaApplicationLaunchShortcut
    [javac] location: package
    org.eclipse.jdt.internal.debug.ui.launcher
    [javac] import
    org.eclipse.jdt.internal.debug.ui.launcher.JavaApplicationLaunchShortcut;
    [javac]


    ^
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:49:
    cannot find symbol
    [javac] symbol: class
    JavaApplicationLaunchShortcut
    [javac]
    JavaApplicationLaunchShortcut {
    [javac] ^
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:66:
    cannot find symbol
    [javac] symbol : variable super
    [javac] location: class
    org.apache.hadoop.eclipse.launch.HadoopApplicationLa
    unchShortcut
    [javac]
    super.findLaunchConfiguration(type,
    configType);
    [javac]
    ^
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:60:
    method does not override or implement a method from a
    supertype
    [javac] @Override
    [javac] ^
    [javac] Note: Some input files use or
    override a deprecated API.
    [javac] Note: Recompile with
    -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked
    or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked
    for details.
    [javac] 4 errors

    BUILD FAILED
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\build.xml:63:
    Compile
    failed;
    see the compiler error output for details.




  • Edmund Kohlwey at Nov 24, 2009 at 3:03 am
    The eclipse plug-in doesn't build using Hadoop 0.20.1 and Eclipse 3.5
    (Galileo). I haven't tried to build it on 3.4, but the build errors
    you're getting are the same as what you get for 3.5.

    Even if you eliminate the compiler errors, the ant script doesn't
    properly package the various plug-in resources. I'm not sure if the JIRA
    listed below takes all this into account or not, as I haven't tried
    their build, but I suspect it doesn't given your issues. You can,
    however, easily fix and export the plug-in by importing the source into
    your eclipse workspace, opening up any offending files, pressing
    cmd+shift+o (which will import the correct packages -- the classes were
    just moved, but the api is the same), and then going through the normal
    PDE export process (which you should be able to find via Google).

    The eclipse build issues (and maintaining the eclipse plugin) are
    complicated by the fact that there's no unit tests for the plug-in, and
    Eclipse doesn't publish up-to-date artifacts in any public maven or ant
    repository, so its a pain to get Hudson, or whatever build server to
    check it. I don't know weather or not this plug-in will ever be well
    maintained unless a commercial vendor starts putting effort into it,
    simply because Eclipse's APIs shift so frequently as to make maintaining
    plug-ins for it quite prohibitive without some sort of commercial interest.

    On 11/23/09 12:16 PM, Raymond Jennings III wrote:
    Hi, it's hadoop-0.20.1 and eclipse Ganymede. I did try that prebuilt plugin over the weekend but no luck with that on Windows. Thank you.


    --- On Mon, 11/23/09, Stephen Wattwrote:

    From: Stephen Watt<swatt@us.ibm.com>
    Subject: Re: Error trying to build hadoop eclipse plugin
    To: common-user@hadoop.apache.org
    Cc: common-user@hadoop.apache.org
    Date: Monday, November 23, 2009, 12:01 PM
    You'll need to be more specific about
    which version of Hadoop and Eclipse
    you're using. There are known issues building the plugin on
    Hadoop 0.20.1.
    I am in the process of providing a patch for this, in the
    interim, you
    can try using the plugin jar attached at
    http://issues.apache.org/jira/browse/HADOOP-6360

    Kind regards
    Steve Watt



    From:
    Raymond Jennings III<raymondjiii@yahoo.com>
    To:
    common-user@hadoop.apache.org
    Date:
    11/23/2009 10:46 AM
    Subject:
    Error trying to build hadoop eclipse plugin



    I am running ant on windows from the
    src/contrib/eclipse-plugin directory
    Using jdk1.6.0_18:

    Anyone have any experience with this? Thank you.



    [ivy:retrieve] 0 artifacts copied, 2 already
    retrieved (0kB/0ms)
    No ivy:settings found for the default reference
    'ivy.instance'. A default
    insta
    nce will be used
    DEPRECATED: 'ivy.conf.file' is deprecated, use
    'ivy.settings.file' instead
    :: loading settings :: file =
    E:\hadoop\hadoop-0.20.1\ivy\ivysettings.xml

    compile:
    [echo] contrib: eclipse-plugin
    [javac] Compiling 45 source files to
    E:\hadoop\hadoop-0.20.1\build\contrib\eclipse-plugin\classes
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:35:

    cannot find symbol
    [javac] symbol : class
    JavaApplicationLaunchShortcut
    [javac] location: package
    org.eclipse.jdt.internal.debug.ui.launcher
    [javac] import
    org.eclipse.jdt.internal.debug.ui.launcher.JavaApplicationLaunchShortcut;
    [javac]


    ^
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:49:

    cannot find symbol
    [javac] symbol: class
    JavaApplicationLaunchShortcut
    [javac]
    JavaApplicationLaunchShortcut {
    [javac] ^
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:66:

    cannot find symbol
    [javac] symbol : variable super
    [javac] location: class
    org.apache.hadoop.eclipse.launch.HadoopApplicationLa
    unchShortcut
    [javac]
    super.findLaunchConfiguration(type,
    configType);
    [javac]
    ^
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:60:

    method does not override or implement a method from a
    supertype
    [javac] @Override
    [javac] ^
    [javac] Note: Some input files use or
    override a deprecated API.
    [javac] Note: Recompile with
    -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked
    or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked
    for details.
    [javac] 4 errors

    BUILD FAILED
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\build.xml:63:
    Compile
    failed;
    see the compiler error output for details.





  • Raymond Jennings III at Nov 24, 2009 at 8:01 am
    From what I understand IBM created the first mapreduce plugin (which they still make available for download.) But once it was integrated into the Hadoop the build the developers here (I also work for IBM) were no longer involved. They told me that the primary person who was maintaining it now works for Google. The original IBM plugin installs okay on Europa but has limited functionality and the DFS browse option does not work (for me with hadoop 0.20.1) I just created a new Ganymede install and used Stephen's prebuilt plugin - it seems okay now at least initially. Not sure if I had some invalid plugin data from my attempted build within eclipse or what. But I am looking forward to playing with it now. I'll post back here if I find anything of value that might help get it working properly. Thanks all.


    --- On Mon, 11/23/09, Edmund Kohlwey wrote:
    From: Edmund Kohlwey <ekohlwey@gmail.com>
    Subject: Re: Error trying to build hadoop eclipse plugin
    To: common-user@hadoop.apache.org
    Date: Monday, November 23, 2009, 10:02 PM
    The eclipse plug-in doesn't build
    using Hadoop 0.20.1 and Eclipse 3.5 (Galileo). I haven't
    tried to build it on 3.4, but the build errors you're
    getting are the same as what you get for 3.5.

    Even if you eliminate the compiler errors, the ant script
    doesn't properly package the various plug-in resources. I'm
    not sure if the JIRA listed below takes all this into
    account or not, as I haven't tried their build, but I
    suspect it doesn't given your issues. You can, however,
    easily fix and export the plug-in by importing the source
    into your eclipse workspace, opening up any offending files,
    pressing cmd+shift+o (which will import the correct packages
    -- the classes were just moved, but the api is the same),
    and then going through the normal PDE export process (which
    you should be able to find via Google).

    The eclipse build issues (and maintaining the eclipse
    plugin) are complicated by the fact that there's no unit
    tests for the plug-in, and Eclipse doesn't publish
    up-to-date artifacts in any public maven or ant repository,
    so its a pain to get Hudson, or whatever build server to
    check it. I don't know weather or not this plug-in will ever
    be well maintained unless a commercial vendor starts putting
    effort into it, simply because Eclipse's APIs shift so
    frequently as to make maintaining plug-ins for it quite
    prohibitive without some sort of commercial interest.

    On 11/23/09 12:16 PM, Raymond Jennings III wrote:
    Hi, it's hadoop-0.20.1 and eclipse Ganymede.  I
    did try that prebuilt plugin over the weekend but no luck
    with that on Windows.  Thank you.

    --- On Mon, 11/23/09, Stephen Wattwrote:

    From: Stephen Watt<swatt@us.ibm.com>
    Subject: Re: Error trying to build hadoop eclipse
    plugin
    To: common-user@hadoop.apache.org
    Cc: common-user@hadoop.apache.org
    Date: Monday, November 23, 2009, 12:01 PM
    You'll need to be more specific about
    which version of Hadoop and Eclipse
    you're using. There are known issues building the
    plugin on
    Hadoop 0.20.1.
    I am in the process of providing
    a patch for this, in the
    interim, you
    can try using the plugin jar attached at
    http://issues.apache.org/jira/browse/HADOOP-6360

    Kind regards
    Steve Watt



    From:
    Raymond Jennings III<raymondjiii@yahoo.com>
    To:
    common-user@hadoop.apache.org
    Date:
    11/23/2009 10:46 AM
    Subject:
    Error trying to build hadoop eclipse plugin



    I am running ant on windows from the
    src/contrib/eclipse-plugin directory
    Using jdk1.6.0_18:

    Anyone have any experience with this?  Thank
    you.


    [ivy:retrieve]  0 artifacts copied, 2
    already
    retrieved (0kB/0ms)
    No ivy:settings found for the default reference
    'ivy.instance'.  A default
    insta
    nce will be used
    DEPRECATED: 'ivy.conf.file' is deprecated, use
    'ivy.settings.file' instead
    :: loading settings :: file =
    E:\hadoop\hadoop-0.20.1\ivy\ivysettings.xml

    compile:
    [echo] contrib:
    eclipse-plugin
    [javac] Compiling 45 source
    files to
    E:\hadoop\hadoop-0.20.1\build\contrib\eclipse-plugin\classes
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:35:
    cannot find symbol
    [javac] symbol  : class
    JavaApplicationLaunchShortcut
    [javac] location: package
    org.eclipse.jdt.internal.debug.ui.launcher
    [javac] import
    org.eclipse.jdt.internal.debug.ui.launcher.JavaApplicationLaunchShortcut;
    [javac]

    ^
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:49:
    cannot find symbol
    [javac] symbol: class
    JavaApplicationLaunchShortcut
    [javac]
    JavaApplicationLaunchShortcut {
    [javac]
    ^
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:66:
    cannot find symbol
    [javac] symbol  :
    variable super
    [javac] location: class
    org.apache.hadoop.eclipse.launch.HadoopApplicationLa
    unchShortcut
    [javac]
    super.findLaunchConfiguration(type,
    configType);
    [javac]
    ^
    [javac]
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:60:
    method does not override or implement a method
    from a
    supertype
    [javac]   @Override
    [javac]   ^
    [javac] Note: Some input files
    use or
    override a deprecated API.
    [javac] Note: Recompile with
    -Xlint:deprecation for details.
    [javac] Note: Some input files
    use unchecked
    or unsafe operations.
    [javac] Note: Recompile with
    -Xlint:unchecked
    for details.
    [javac] 4 errors

    BUILD FAILED
    E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\build.xml:63:
    Compile
    failed;
    see the compiler error output for
    details.




Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedNov 23, '09 at 2:04a
activeNov 25, '09 at 7:09a
posts14
users8
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase