Hi,

We use Hadoop 0.20.2 version.The log4j.properties file has a
property *hadoop.tasklog.logsRetainHours *(mentioned as 24 hours by default)
* *which we have set to 12.Despite this property being set we still have
old userlogs in the system.Any idea why the attempt directories (that have
the syserr/sysout files) have not been cleaned up? Are we missing any other
configuration?

Any help on this topic would be great.

Regards,
Sahana

Search Discussions

  • Harsh J at Dec 7, 2011 at 8:59 am
    Sahana,

    IIRC, "mapred.userlog.retain.hours" (24h default) controls this in my
    environment and it seems to work fine on my cluster. Are you sure you
    have tasklogs older than 24h lying around? It might even be a bug that
    may have been fixed in the subsequent 0.20 releases that went out
    recently.
    On Wed, Dec 7, 2011 at 1:53 PM, Sahana Bhat wrote:
    Hi,

            We use Hadoop 0.20.2 version.The log4j.properties file has a
    property hadoop.tasklog.logsRetainHours (mentioned as 24 hours by
    default) which we have set to 12.Despite this property being set we still
    have old userlogs in the system.Any idea why the attempt directories (that
    have the syserr/sysout files) have not been cleaned up? Are we missing any
    other configuration?

    Any help on this topic would be great.

    Regards,
    Sahana


    --
    Harsh J
  • Sahana Bhat at Dec 7, 2011 at 10:49 am
    Hi Harsh,

    Thanks for the reply. I realized that the property you mentioned
    was missing in my mapred-site.xml.
    I added the entry and it works just fine.

    Was my assumption that "*hadoop.tasklog.logsRetainHours " *in
    log4j.properties will do the same wrong? What is this property for in that
    case?

    Regards,
    Sahana



    On Wed, Dec 7, 2011 at 2:28 PM, Harsh J wrote:

    Sahana,

    IIRC, "mapred.userlog.retain.hours" (24h default) controls this in my
    environment and it seems to work fine on my cluster. Are you sure you
    have tasklogs older than 24h lying around? It might even be a bug that
    may have been fixed in the subsequent 0.20 releases that went out
    recently.
    On Wed, Dec 7, 2011 at 1:53 PM, Sahana Bhat wrote:
    Hi,

    We use Hadoop 0.20.2 version.The log4j.properties file has a
    property hadoop.tasklog.logsRetainHours (mentioned as 24 hours by
    default) which we have set to 12.Despite this property being set we still
    have old userlogs in the system.Any idea why the attempt directories (that
    have the syserr/sysout files) have not been cleaned up? Are we missing any
    other configuration?

    Any help on this topic would be great.

    Regards,
    Sahana


    --
    Harsh J
  • Harsh J at Dec 7, 2011 at 11:38 am
    I'm using CDH3 here and I do not see that option in my
    log4j.properties. Probably something remaining from an older
    mechanism?

    Good to know the actual property worked for you.
    On Wed, Dec 7, 2011 at 4:18 PM, Sahana Bhat wrote:
    Hi Harsh,

              Thanks for the reply. I realized that the property you mentioned
    was missing in my mapred-site.xml.
    I added the entry and it works just fine.

    Was my assumption that "hadoop.tasklog.logsRetainHours " in log4j.properties
    will do the same wrong? What is this property for in that case?

    Regards,
    Sahana



    On Wed, Dec 7, 2011 at 2:28 PM, Harsh J wrote:

    Sahana,

    IIRC, "mapred.userlog.retain.hours" (24h default) controls this in my
    environment and it seems to work fine on my cluster. Are you sure you
    have tasklogs older than 24h lying around? It might even be a bug that
    may have been fixed in the subsequent 0.20 releases that went out
    recently.
    On Wed, Dec 7, 2011 at 1:53 PM, Sahana Bhat wrote:
    Hi,

            We use Hadoop 0.20.2 version.The log4j.properties file has a
    property hadoop.tasklog.logsRetainHours (mentioned as 24 hours by
    default) which we have set to 12.Despite this property being set we
    still
    have old userlogs in the system.Any idea why the attempt directories
    (that
    have the syserr/sysout files) have not been cleaned up? Are we missing
    any
    other configuration?

    Any help on this topic would be great.

    Regards,
    Sahana


    --
    Harsh J


    --
    Harsh J
  • Vinod Kumar Vavilapalli at Dec 7, 2011 at 10:00 pm
    That is a stale property in log4j.properties that isn't used for anything. There are other such things like noKeepSplits, purgeLogSplits etc which seem to be stale too.

    mapred.userlog.retain.hours should work just fine as Harsh correctly mentioned.

    HTH,
    +Vinod

    On Dec 7, 2011, at 2:48 AM, Sahana Bhat wrote:

    Was my assumption that "hadoop.tasklog.logsRetainHours " in log4j.properties will do the same wrong? What is this property for in that case?

    Regards,
    Sahana
  • Sahana Bhat at Dec 8, 2011 at 3:57 am
    Hi Vinod,

    Thanks for that info :).

    Regards,
    Sahana
    On Thu, Dec 8, 2011 at 3:30 AM, Vinod Kumar Vavilapalli wrote:


    That is a stale property in log4j.properties that isn't used for anything.
    There are other such things like noKeepSplits, purgeLogSplits etc which
    seem to be stale too.

    mapred.userlog.retain.hours should work just fine as Harsh correctly
    mentioned.

    HTH,
    +Vinod


    On Dec 7, 2011, at 2:48 AM, Sahana Bhat wrote:

    Was my assumption that "*hadoop.tasklog.logsRetainHours " *in
    log4j.properties will do the same wrong? What is this property for in that
    case?

    Regards,
    Sahana

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedDec 7, '11 at 8:24a
activeDec 8, '11 at 3:57a
posts6
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase