Grokbase › Groups › Hadoop › common-user › October 2010
FAQ
hadoop logs keep increasing and not removed automatically according to the logging property. I didn't change the default logging property but it never works as it should. The default logging property is as follows, anyone know why or it is a bug not fixed ?


<property>
<name>hadoop.logfile.size</name>
<value>10000000</value>
<description>The max size of each log file</description>
</property>
<property>
<name>hadoop.logfile.count</name>
<value>10</value>
<description>The max number of log files</description>
</property>


2010-10-18



shangan

Search Discussions

  • Allen Wittenauer at Oct 18, 2010 at 5:23 am ⇧

    On Oct 17, 2010, at 10:05 PM, shangan wrote:

    hadoop logs keep increasing and not removed automatically according to the logging property. I didn't change the default logging property but it never works as it should. The default logging property is as follows, anyone know why or it is a bug not fixed ?
    I don't think hadoop.logfile.count is actually tied to anything in the source. Likely a property from a bygone era.
  • Shangan at Oct 18, 2010 at 5:29 am ⇧
    then you just write a shell to remove the logs periodically as a workaround? or better ideas ?


    2010-10-18



    shangan




    发件人： Allen Wittenauer
    发送时间： 2010-10-18 13:23:34
    收件人：
    抄送：
    主题： Re: hadoop can't remove logs automatically
    On Oct 17, 2010, at 10:05 PM, shangan wrote:
    hadoop logs keep increasing and not removed automatically according to the logging property. I didn't change the default logging property but it never works as it should. The default logging property is as follows, anyone know why or it is a bug not fixed ?
    I don't think hadoop.logfile.count is actually tied to anything in the source. Likely a property from a bygone era.
    __________ Information from ESET NOD32 Antivirus, version of virus signature database 5540 (20101017) __________
    The message was checked by ESET NOD32 Antivirus.
    http://www.eset.com
  • Allen Wittenauer at Oct 18, 2010 at 5:32 am ⇧

    On Oct 17, 2010, at 10:29 PM, shangan wrote:

    then you just write a shell to remove the logs periodically as a workaround? or better ideas ?
    We basically have a cron job that does a few things as part of our maintenance. We have it rigged up such that it runs on the namenode and then, over ssh, runs each of these on the slave nodes:

    - purge old logs
    - purge old files out of mapred temp space
    - kill stale/stuck tasks

    Hadoop really should manage this stuff on its own, but well, Hadoop should do a lot of things to be more operable. ;)
  • Allen Wittenauer at Oct 18, 2010 at 5:36 am ⇧

    On Oct 17, 2010, at 10:32 PM, Allen Wittenauer wrote:

    On Oct 17, 2010, at 10:29 PM, shangan wrote:

    then you just write a shell to remove the logs periodically as a workaround? or better ideas ?
    We basically have a cron job that does a few things as part of our maintenance. We have it rigged up such that it runs on the namenode and then, over ssh, runs each of these on the slave nodes:

    - purge old logs
    - purge old files out of mapred temp space
    - kill stale/stuck tasks

    Hadoop really should manage this stuff on its own, but well, Hadoop should do a lot of things to be more operable. ;)

    Of course, it is worth mentioning you could also tie these logs to logadm/logwatch/rotatelogs/etc. But I like to have them centrally managed so that they are consistent across the grid.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 18, '10 at 5:06a
activeOct 18, '10 at 5:36a
posts5
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Allen Wittenauer: 3 posts Shangan: 2 posts

Content

People

Support

Translate

site design / logo © 2023 Grokbase