FAQ
Hi Folks,

In the hadoop-env.sh, we find, ...

# Where log files are stored. $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

is there any reason this location could not be a separate filesystem on the name node?

Thx, jackc...
------------------------------------
Jack Craig, Operations
CarrierIQ.com<http://CarrierIQ.com>
1200 Villa Ct, Suite 200
Mountain View, CA. 94041
650-625-5456

Search Discussions

  • Madhu Ramanna at Jun 22, 2011 at 8:06 pm
    Looks like you missed the '#' in line beginning

    Feel free to set HADOOP_LOG_DIR in that script or elsewhere
    On 6/22/11 1:02 PM, "Jack Craig" wrote:

    Hi Folks,

    In the hadoop-env.sh, we find, ...

    # Where log files are stored. $HADOOP_HOME/logs by default.
    # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

    is there any reason this location could not be a separate filesystem on
    the name node?

    Thx, jackc...
    ------------------------------------
    Jack Craig, Operations
    CarrierIQ.com<http://CarrierIQ.com>
    1200 Villa Ct, Suite 200
    Mountain View, CA. 94041
    650-625-5456
  • Harsh J at Jun 22, 2011 at 8:10 pm
    Jack,

    I believe the location can definitely be set to any desired path.
    Could you tell us the issues you face when you change it?

    P.s. The env var is used to set the config property hadoop.log.dir
    internally. So as long as you use the regular scripts (bin/ or init.d/
    ones) to start daemons, it would apply fine.
    On Thu, Jun 23, 2011 at 1:32 AM, Jack Craig wrote:
    Hi Folks,

    In the hadoop-env.sh, we find, ...

    # Where log files are stored.  $HADOOP_HOME/logs by default.
    # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

    is there any reason this location could not be a separate filesystem on the name node?

    Thx, jackc...
    ------------------------------------
    Jack Craig, Operations
    CarrierIQ.com<http://CarrierIQ.com>
    1200 Villa Ct, Suite 200
    Mountain View, CA. 94041
    650-625-5456


    --
    Harsh J
  • Jack Craig at Jun 22, 2011 at 9:00 pm
    Thx to both respondents.

    Note i've not tried this redirection as I have only production grids available.

    Our grids are growing and with them, log volume.

    As until now that log location has been in the same fs as the grid data,
    so running out of space due log bloat is a growing problem.
    From your replies, sounds like I can relocate my logs, Cool!
    But now the tough question, if i set up a too small partition and it runs out of space,
    will my grid become unstable if hadoop can no longer write to its logs?

    Thx again, jackc...

    ------------------------------------
    Jack Craig, Operations
    CarrierIQ.com<http://CarrierIQ.com>
    1200 Villa Ct, Suite 200
    Mountain View, CA. 94041
    650-625-5456

    On Jun 22, 2011, at 1:09 PM, Harsh J wrote:

    Jack,

    I believe the location can definitely be set to any desired path.
    Could you tell us the issues you face when you change it?

    P.s. The env var is used to set the config property hadoop.log.dir
    internally. So as long as you use the regular scripts (bin/ or init.d/
    ones) to start daemons, it would apply fine.

    On Thu, Jun 23, 2011 at 1:32 AM, Jack Craig wrote:
    Hi Folks,

    In the hadoop-env.sh, we find, ...

    # Where log files are stored. $HADOOP_HOME/logs by default.
    # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

    is there any reason this location could not be a separate filesystem on the name node?

    Thx, jackc...
    ------------------------------------
    Jack Craig, Operations
    CarrierIQ.com<http://CarrierIQ.com>
    1200 Villa Ct, Suite 200
    Mountain View, CA. 94041
    650-625-5456





    --
    Harsh J
  • Jagaran das at Jun 22, 2011 at 9:15 pm
    Hi,

    Can I limit the log file duration ?
    I want to keep files for last 15 days only.

    Regards,
    Jagaran



    ________________________________
    From: Jack Craig <JCraig@carrieriq.com>
    To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
    Sent: Wed, 22 June, 2011 2:00:23 PM
    Subject: Re: Any reason Hadoop logs cant be directed to a separate filesystem?

    Thx to both respondents.

    Note i've not tried this redirection as I have only production grids available.

    Our grids are growing and with them, log volume.

    As until now that log location has been in the same fs as the grid data,
    so running out of space due log bloat is a growing problem.

    From your replies, sounds like I can relocate my logs, Cool!

    But now the tough question, if i set up a too small partition and it runs out of
    space,
    will my grid become unstable if hadoop can no longer write to its logs?

    Thx again, jackc...

    ------------------------------------
    Jack Craig, Operations
    CarrierIQ.com<http://CarrierIQ.com>
    1200 Villa Ct, Suite 200
    Mountain View, CA. 94041
    650-625-5456

    On Jun 22, 2011, at 1:09 PM, Harsh J wrote:

    Jack,

    I believe the location can definitely be set to any desired path.
    Could you tell us the issues you face when you change it?

    P.s. The env var is used to set the config property hadoop.log.dir
    internally. So as long as you use the regular scripts (bin/ or init.d/
    ones) to start daemons, it would apply fine.

    On Thu, Jun 23, 2011 at 1:32 AM, Jack Craig
    wrote:
    Hi Folks,

    In the hadoop-env.sh, we find, ...

    # Where log files are stored. $HADOOP_HOME/logs by default.
    # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

    is there any reason this location could not be a separate filesystem on the name
    node?

    Thx, jackc...
    ------------------------------------
    Jack Craig, Operations
    CarrierIQ.com<http://CarrierIQ.com>
    1200 Villa Ct, Suite 200
    Mountain View, CA. 94041
    650-625-5456





    --
    Harsh J
  • Michael Segel at Jun 25, 2011 at 10:58 am
    Yes, and its called using cron and writing a simple ksh script to clear out any files that are older than 15 days.
    There may be another way, but that's really the easiest.

    Date: Thu, 23 Jun 2011 02:44:48 +0530
    From: jagaran_das@yahoo.co.in
    Subject: Re: Any reason Hadoop logs cant be directed to a separate filesystem?
    To: common-user@hadoop.apache.org

    Hi,

    Can I limit the log file duration ?
    I want to keep files for last 15 days only.

    Regards,
    Jagaran



    ________________________________
    From: Jack Craig <JCraig@carrieriq.com>
    To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
    Sent: Wed, 22 June, 2011 2:00:23 PM
    Subject: Re: Any reason Hadoop logs cant be directed to a separate filesystem?

    Thx to both respondents.

    Note i've not tried this redirection as I have only production grids available.

    Our grids are growing and with them, log volume.

    As until now that log location has been in the same fs as the grid data,
    so running out of space due log bloat is a growing problem.

    From your replies, sounds like I can relocate my logs, Cool!

    But now the tough question, if i set up a too small partition and it runs out of
    space,
    will my grid become unstable if hadoop can no longer write to its logs?

    Thx again, jackc...

    ------------------------------------
    Jack Craig, Operations
    CarrierIQ.com<http://CarrierIQ.com>
    1200 Villa Ct, Suite 200
    Mountain View, CA. 94041
    650-625-5456

    On Jun 22, 2011, at 1:09 PM, Harsh J wrote:

    Jack,

    I believe the location can definitely be set to any desired path.
    Could you tell us the issues you face when you change it?

    P.s. The env var is used to set the config property hadoop.log.dir
    internally. So as long as you use the regular scripts (bin/ or init.d/
    ones) to start daemons, it would apply fine.

    On Thu, Jun 23, 2011 at 1:32 AM, Jack Craig
    wrote:
    Hi Folks,

    In the hadoop-env.sh, we find, ...

    # Where log files are stored. $HADOOP_HOME/logs by default.
    # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

    is there any reason this location could not be a separate filesystem on the name
    node?

    Thx, jackc...
    ------------------------------------
    Jack Craig, Operations
    CarrierIQ.com<http://CarrierIQ.com>
    1200 Villa Ct, Suite 200
    Mountain View, CA. 94041
    650-625-5456





    --
    Harsh J
  • Jagaran das at Jun 25, 2011 at 6:07 pm
    yeah, tats what we do.
    But its again an extra process, if hadoop had an ability, then it would be
    great.
    it uses log4j, i tired to tweak it, but it is throwing error.

    Regards,
    Jagaran



    ________________________________
    From: Michael Segel <michael_segel@hotmail.com>
    To: common-user@hadoop.apache.org
    Sent: Sat, 25 June, 2011 3:58:19 AM
    Subject: RE: Any reason Hadoop logs cant be directed to a separate filesystem?


    Yes, and its called using cron and writing a simple ksh script to clear out any
    files that are older than 15 days.

    There may be another way, but that's really the easiest.

    Date: Thu, 23 Jun 2011 02:44:48 +0530
    From: jagaran_das@yahoo.co.in
    Subject: Re: Any reason Hadoop logs cant be directed to a separate filesystem?
    To: common-user@hadoop.apache.org

    Hi,

    Can I limit the log file duration ?
    I want to keep files for last 15 days only.

    Regards,
    Jagaran



    ________________________________
    From: Jack Craig <JCraig@carrieriq.com>
    To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
    Sent: Wed, 22 June, 2011 2:00:23 PM
    Subject: Re: Any reason Hadoop logs cant be directed to a separate filesystem?

    Thx to both respondents.

    Note i've not tried this redirection as I have only production grids
    available.
    Our grids are growing and with them, log volume.

    As until now that log location has been in the same fs as the grid data,
    so running out of space due log bloat is a growing problem.

    From your replies, sounds like I can relocate my logs, Cool!

    But now the tough question, if i set up a too small partition and it runs out
    of

    space,
    will my grid become unstable if hadoop can no longer write to its logs?

    Thx again, jackc...

    ------------------------------------
    Jack Craig, Operations
    CarrierIQ.com<http://CarrierIQ.com>
    1200 Villa Ct, Suite 200
    Mountain View, CA. 94041
    650-625-5456

    On Jun 22, 2011, at 1:09 PM, Harsh J wrote:

    Jack,

    I believe the location can definitely be set to any desired path.
    Could you tell us the issues you face when you change it?

    P.s. The env var is used to set the config property hadoop.log.dir
    internally. So as long as you use the regular scripts (bin/ or init.d/
    ones) to start daemons, it would apply fine.

    On Thu, Jun 23, 2011 at 1:32 AM, Jack Craig
    wrote:
    Hi Folks,

    In the hadoop-env.sh, we find, ...

    # Where log files are stored. $HADOOP_HOME/logs by default.
    # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

    is there any reason this location could not be a separate filesystem on the
    name

    node?

    Thx, jackc...
    ------------------------------------
    Jack Craig, Operations
    CarrierIQ.com<http://CarrierIQ.com>
    1200 Villa Ct, Suite 200
    Mountain View, CA. 94041
    650-625-5456





    --
    Harsh J

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJun 22, '11 at 8:02p
activeJun 25, '11 at 6:07p
posts7
users5
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase