FAQ
Hello,

I'm very sorry to trouble you, I'm developing a MapReduce
Application, And I can get Log.INFO in InputFormat ,but In Mapper or Reducer
, I can't get anything

. And Now an error occured in the reduce stage. Because the code is a little
complicated, I can't find where is the mistake just form the thrown
exception. I want to use

log4j to log the intermediate info. But I have tried a whole day , Now I
can't still get anything. who can help me? Thanks very much.

ddream.

Search Discussions

  • Alex Loddengaard at Nov 12, 2008 at 2:54 am
    Have you seen this:
    <http://wiki.apache.org/hadoop/HowToDebugMapReducePrograms>

    Alex
    On Tue, Nov 11, 2008 at 6:03 PM, ZhiHong Fu wrote:

    Hello,

    I'm very sorry to trouble you, I'm developing a MapReduce
    Application, And I can get Log.INFO in InputFormat ,but In Mapper or
    Reducer
    , I can't get anything

    . And Now an error occured in the reduce stage. Because the code is a
    little
    complicated, I can't find where is the mistake just form the thrown
    exception. I want to use

    log4j to log the intermediate info. But I have tried a whole day , Now I
    can't still get anything. who can help me? Thanks very much.

    ddream.
  • Scott Whitecross at Nov 24, 2008 at 1:59 am
    I've looked around for a while, but it seems there isn't a way to log
    from Hadoop, without going through the logs/userlogs/ and the
    'attempt' directories? That would mean that for logging I'm
    restricted to writing to System.out and System.err, then collecting
    via scripts?

    Thanks.




    On Nov 11, 2008, at 9:53 PM, Alex Loddengaard wrote:

    Have you seen this:
    <http://wiki.apache.org/hadoop/HowToDebugMapReducePrograms>

    Alex
    On Tue, Nov 11, 2008 at 6:03 PM, ZhiHong Fu wrote:

    Hello,

    I'm very sorry to trouble you, I'm developing a MapReduce
    Application, And I can get Log.INFO in InputFormat ,but In Mapper or
    Reducer
    , I can't get anything

    . And Now an error occured in the reduce stage. Because the code is a
    little
    complicated, I can't find where is the mistake just form the thrown
    exception. I want to use

    log4j to log the intermediate info. But I have tried a whole day ,
    Now I
    can't still get anything. who can help me? Thanks very much.

    ddream.
  • Brian Bockelman at Nov 24, 2008 at 2:19 am
    Hey Scott,

    Have you tried configuring things from

    $HADOOP_HOME/conf/log4j.properties

    ?

    I'd just use my own logger and set up a separate syslog server. It's
    not an extremely elaborate setup (certainly, would quickly become a
    headache on a large cluster...), but it should be pretty easy to set up.

    Brian
    On Nov 23, 2008, at 7:58 PM, Scott Whitecross wrote:

    I've looked around for a while, but it seems there isn't a way to
    log from Hadoop, without going through the logs/userlogs/ and the
    'attempt' directories? That would mean that for logging I'm
    restricted to writing to System.out and System.err, then collecting
    via scripts?

    Thanks.




    On Nov 11, 2008, at 9:53 PM, Alex Loddengaard wrote:

    Have you seen this:
    <http://wiki.apache.org/hadoop/HowToDebugMapReducePrograms>

    Alex

    On Tue, Nov 11, 2008 at 6:03 PM, ZhiHong Fu <ddream84@gmail.com>
    wrote:
    Hello,

    I'm very sorry to trouble you, I'm developing a MapReduce
    Application, And I can get Log.INFO in InputFormat ,but In Mapper or
    Reducer
    , I can't get anything

    . And Now an error occured in the reduce stage. Because the code
    is a
    little
    complicated, I can't find where is the mistake just form the thrown
    exception. I want to use

    log4j to log the intermediate info. But I have tried a whole day ,
    Now I
    can't still get anything. who can help me? Thanks very much.

    ddream.
  • Scott Whitecross at Nov 24, 2008 at 3:02 am
    Thanks Brian. I've played w/ the log4j.properties a bit, and haven't
    had any luck. Can you share how youve setup log4j? I am probably
    missing the obvious, but here is what I setup:

    log4j.logger.com.mycompany.hadoop=DEBUG,DX,console
    log4j.appender.DX=org.apache.log4j.DailyRollingFileAppender
    log4j.appender.DX.File=/opt/hadoop/myLogs/test.log
    log4j.appender.DX.layout=org.apache.log4j.PatternLayout
    log4j.appender.DX.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

    Within my map class:
    private static final Logger logger = Logger.getLogger(MyMap.class);

    and to log:

    logger.info("entering map");

    I also tried creating a logger in the code and attaching appenders w/
    out luck.

    Thanks.


    On Nov 23, 2008, at 9:15 PM, Brian Bockelman wrote:

    Hey Scott,

    Have you tried configuring things from

    $HADOOP_HOME/conf/log4j.properties

    ?

    I'd just use my own logger and set up a separate syslog server.
    It's not an extremely elaborate setup (certainly, would quickly
    become a headache on a large cluster...), but it should be pretty
    easy to set up.

    Brian
    On Nov 23, 2008, at 7:58 PM, Scott Whitecross wrote:

    I've looked around for a while, but it seems there isn't a way to
    log from Hadoop, without going through the logs/userlogs/ and the
    'attempt' directories? That would mean that for logging I'm
    restricted to writing to System.out and System.err, then collecting
    via scripts?

    Thanks.




    On Nov 11, 2008, at 9:53 PM, Alex Loddengaard wrote:

    Have you seen this:
    <http://wiki.apache.org/hadoop/HowToDebugMapReducePrograms>

    Alex

    On Tue, Nov 11, 2008 at 6:03 PM, ZhiHong Fu <ddream84@gmail.com>
    wrote:
    Hello,

    I'm very sorry to trouble you, I'm developing a MapReduce
    Application, And I can get Log.INFO in InputFormat ,but In Mapper
    or
    Reducer
    , I can't get anything

    . And Now an error occured in the reduce stage. Because the code
    is a
    little
    complicated, I can't find where is the mistake just form the thrown
    exception. I want to use

    log4j to log the intermediate info. But I have tried a whole
    day , Now I
    can't still get anything. who can help me? Thanks very much.

    ddream.
  • Brian Bockelman at Nov 24, 2008 at 3:18 am
    Hey Scott,

    I see nothing wrong offhand; have you tried to run in "local" mode?
    It'd be quicker to debug logging problems that way, as any bad
    misconfigurations (I think) should get printed out to stderr.

    Brian
    On Nov 23, 2008, at 9:01 PM, Scott Whitecross wrote:

    Thanks Brian. I've played w/ the log4j.properties a bit, and
    haven't had any luck. Can you share how youve setup log4j? I am
    probably missing the obvious, but here is what I setup:

    log4j.logger.com.mycompany.hadoop=DEBUG,DX,console
    log4j.appender.DX=org.apache.log4j.DailyRollingFileAppender
    log4j.appender.DX.File=/opt/hadoop/myLogs/test.log
    log4j.appender.DX.layout=org.apache.log4j.PatternLayout
    log4j.appender.DX.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

    Within my map class:
    private static final Logger logger = Logger.getLogger(MyMap.class);

    and to log:

    logger.info("entering map");

    I also tried creating a logger in the code and attaching appenders w/
    out luck.

    Thanks.


    On Nov 23, 2008, at 9:15 PM, Brian Bockelman wrote:

    Hey Scott,

    Have you tried configuring things from

    $HADOOP_HOME/conf/log4j.properties

    ?

    I'd just use my own logger and set up a separate syslog server.
    It's not an extremely elaborate setup (certainly, would quickly
    become a headache on a large cluster...), but it should be pretty
    easy to set up.

    Brian
    On Nov 23, 2008, at 7:58 PM, Scott Whitecross wrote:

    I've looked around for a while, but it seems there isn't a way to
    log from Hadoop, without going through the logs/userlogs/ and the
    'attempt' directories? That would mean that for logging I'm
    restricted to writing to System.out and System.err, then
    collecting via scripts?

    Thanks.




    On Nov 11, 2008, at 9:53 PM, Alex Loddengaard wrote:

    Have you seen this:
    <http://wiki.apache.org/hadoop/HowToDebugMapReducePrograms>

    Alex

    On Tue, Nov 11, 2008 at 6:03 PM, ZhiHong Fu <ddream84@gmail.com>
    wrote:
    Hello,

    I'm very sorry to trouble you, I'm developing a MapReduce
    Application, And I can get Log.INFO in InputFormat ,but In
    Mapper or
    Reducer
    , I can't get anything

    . And Now an error occured in the reduce stage. Because the code
    is a
    little
    complicated, I can't find where is the mistake just form the
    thrown
    exception. I want to use

    log4j to log the intermediate info. But I have tried a whole
    day , Now I
    can't still get anything. who can help me? Thanks very much.

    ddream.
  • Scott Whitecross at Nov 24, 2008 at 3:25 am
    Thanks Brian. So you have had luck w/ log4j?

    I haven't tried local mode. I will try it tonight and see how it goes
    for quick debugging. More so, I wanted to be able to easily log and
    watch events on a cluster, rather then digging through all the hadoop
    logging levels. I've also read that you can attach scripts as well?

    On Nov 23, 2008, at 10:18 PM, Brian Bockelman wrote:

    Hey Scott,

    I see nothing wrong offhand; have you tried to run in "local" mode?
    It'd be quicker to debug logging problems that way, as any bad
    misconfigurations (I think) should get printed out to stderr.

    Brian
    On Nov 23, 2008, at 9:01 PM, Scott Whitecross wrote:

    Thanks Brian. I've played w/ the log4j.properties a bit, and
    haven't had any luck. Can you share how youve setup log4j? I am
    probably missing the obvious, but here is what I setup:

    log4j.logger.com.mycompany.hadoop=DEBUG,DX,console
    log4j.appender.DX=org.apache.log4j.DailyRollingFileAppender
    log4j.appender.DX.File=/opt/hadoop/myLogs/test.log
    log4j.appender.DX.layout=org.apache.log4j.PatternLayout
    log4j.appender.DX.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

    Within my map class:
    private static final Logger logger = Logger.getLogger(MyMap.class);

    and to log:

    logger.info("entering map");

    I also tried creating a logger in the code and attaching appenders
    w/out luck.

    Thanks.


    On Nov 23, 2008, at 9:15 PM, Brian Bockelman wrote:

    Hey Scott,

    Have you tried configuring things from

    $HADOOP_HOME/conf/log4j.properties

    ?

    I'd just use my own logger and set up a separate syslog server.
    It's not an extremely elaborate setup (certainly, would quickly
    become a headache on a large cluster...), but it should be pretty
    easy to set up.

    Brian
    On Nov 23, 2008, at 7:58 PM, Scott Whitecross wrote:

    I've looked around for a while, but it seems there isn't a way to
    log from Hadoop, without going through the logs/userlogs/ and the
    'attempt' directories? That would mean that for logging I'm
    restricted to writing to System.out and System.err, then
    collecting via scripts?

    Thanks.




    On Nov 11, 2008, at 9:53 PM, Alex Loddengaard wrote:

    Have you seen this:
    <http://wiki.apache.org/hadoop/HowToDebugMapReducePrograms>

    Alex

    On Tue, Nov 11, 2008 at 6:03 PM, ZhiHong Fu <ddream84@gmail.com>
    wrote:
    Hello,

    I'm very sorry to trouble you, I'm developing a MapReduce
    Application, And I can get Log.INFO in InputFormat ,but In
    Mapper or
    Reducer
    , I can't get anything

    . And Now an error occured in the reduce stage. Because the
    code is a
    little
    complicated, I can't find where is the mistake just form the
    thrown
    exception. I want to use

    log4j to log the intermediate info. But I have tried a whole
    day , Now I
    can't still get anything. who can help me? Thanks very much.

    ddream.
  • Steve Loughran at Nov 24, 2008 at 3:50 pm

    Scott Whitecross wrote:
    Thanks Brian. So you have had luck w/ log4j?
    We grab logs off machines by not using lo4j and routing to our own
    logging infrastructure that can feed events to other boxes via RMI and
    queues. This stuff slots in behind commons-logging, with a custom
    commons-logging bridge specified on the command line. To get this into
    Hadoop I had to patch hadoop.jar and remove the properties file that
    bound it only to log4j. The central receiver/SPOF logs events by sent
    time and received time and can store all results into text files
    intermixed for post-processing. It's good for testing, but on a big
    production cluster you'd want something more robust and scaleable
  • Brian Bockelman at Nov 24, 2008 at 3:56 pm

    On Nov 24, 2008, at 9:49 AM, Steve Loughran wrote:

    Scott Whitecross wrote:
    Thanks Brian. So you have had luck w/ log4j?
    We grab logs off machines by not using lo4j and routing to our own
    logging infrastructure that can feed events to other boxes via RMI
    and queues. This stuff slots in behind commons-logging, with a
    custom commons-logging bridge specified on the command line. To get
    this into Hadoop I had to patch hadoop.jar and remove the properties
    file that bound it only to log4j. The central receiver/SPOF logs
    events by sent time and received time and can store all results into
    text files intermixed for post-processing. It's good for testing,
    but on a big production cluster you'd want something more robust and
    scaleable
    Hey Steve,

    Sounds like a cool setup, but might be a little much for Scott's
    purposes (trying to debug a single Map phase...). Scott, I have been
    able to successfully add new log4j loggers, but in Hadoop code, not in
    a M-R task. If you try things in local mode, you'll be guaranteed to
    have the same JVM, so the configuration should be loaded the same way.

    Then again, I might be putting words into Scott's mouth: maybe he does
    indeed want to scale this way up and turn it into a "logging
    infrastructure".

    Scott, did you have any luck debugging the job through the wiki
    document on debugging mapreduce? I'd make sure to start there before
    you take too much of a detour into log4j-land.

    Brian

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedNov 12, '08 at 2:04a
activeNov 24, '08 at 3:56p
posts9
users5
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase