FAQ
Hi,

Does anyone know how I can reduce the level of logging of the Hive History
file which are created per Hive statement executed in the (default) location
of /tmp/${user.name}/ hive_job_log_hdfs_201109122109_675419678.txt. The
level of detail is very high by default but I can't find a way to tune it
down.


It looks like this:
-------------------------------
SessionStart SESSION_ID="hdfs_201109122110" TIME="1315854630381"
QueryStart QUERY_STRING="show partitions dart_facts"
QUERY_ID="hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4"
TIME="1315854630789"
Counters
plan="{"queryId":"hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4","queryType":null,"queryAttributes":{"queryString":"show
partitions
dart_facts"},"queryCounters":"null","stageGraph":{"nodeType":"STAGE","roots":"null","adjacencyList":"]"},"stageList":[{"stageId":"Stage-0","stageType":"DDL","stageAttributes":"null","stageCounters":"}","taskList":[{"taskId":"Stage-0_OTHER","taskType":"OTHER","taskAttributes":"null","taskCounters":"null","operatorGraph":"null","operatorList":"]","done":"false","started":"false"}],"done":"false","started":"false"}],"done":"false","started":"true"}"
TIME="1315854630797"
TaskStart TASK_NAME="org.apache.hadoop.hive.ql.exec.DDLTask"
TASK_ID="Stage-0"
QUERY_ID="hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4"
TIME="1315854630805"
Counters
plan="{"queryId":"hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4","queryType":null,"queryAttributes":{"queryString":"show
partitions
dart_facts"},"queryCounters":"null","stageGraph":{"nodeType":"STAGE","roots":"null","adjacencyList":"]"},"stageList":[{"stageId":"Stage-0","stageType":"DDL","stageAttributes":"null","stageCounters":"}","taskList":[{"taskId":"Stage-0_OTHER","taskType":"OTHER","taskAttributes":"null","taskCounters":"null","operatorGraph":"null","operatorList":"]","done":"false","started":"true"}],"done":"false","started":"true"}],"done":"false","started":"true"}"
TIME="1315854630806"
-----------------------------------------

It doesn't look like standard log4j logging to me and this logging does not
respond to the log4j properties in :

-/usr/lib/hive/conf/hive-log4j (these settings influence
the detail in HIVE_LOG_DIR/hive.log)
-/usr/lib/hive/conf/hive-exec-log4j (don't know where these
settings are meant for. Anyone ? )

I have to find a way to turn the logging in /tmp/${user.name}/ down. I ran a
large Hive job once that generated 8GB of logging this way which stalled the
server.

Thanks

--
Kind Regards,

Search Discussions

  • Adriaan Tijsseling at Sep 12, 2011 at 7:43 pm
    Should be here:
    ./conf/hive-exec-log4j.properties
    ./conf/hive-log4j.properties

    Cheers,

    Adriaan
    On 2011/09/12, at 21:32, Jasper Knulst wrote:

    Hi,

    Does anyone know how I can reduce the level of logging of the Hive History
    file which are created per Hive statement executed in the (default) location
    of /tmp/${user.name}/ hive_job_log_hdfs_201109122109_675419678.txt. The
    level of detail is very high by default but I can't find a way to tune it
    down.


    It looks like this:
    -------------------------------
    SessionStart SESSION_ID="hdfs_201109122110" TIME="1315854630381"
    QueryStart QUERY_STRING="show partitions dart_facts"
    QUERY_ID="hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4"
    TIME="1315854630789"
    Counters
    plan="{"queryId":"hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4","queryType":null,"queryAttributes":{"queryString":"show
    partitions
    dart_facts"},"queryCounters":"null","stageGraph":{"nodeType":"STAGE","roots":"null","adjacencyList":"]"},"stageList":[{"stageId":"Stage-0","stageType":"DDL","stageAttributes":"null","stageCounters":"}","taskList":[{"taskId":"Stage-0_OTHER","taskType":"OTHER","taskAttributes":"null","taskCounters":"null","operatorGraph":"null","operatorList":"]","done":"false","started":"false"}],"done":"false","started":"false"}],"done":"false","started":"true"}"
    TIME="1315854630797"
    TaskStart TASK_NAME="org.apache.hadoop.hive.ql.exec.DDLTask"
    TASK_ID="Stage-0"
    QUERY_ID="hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4"
    TIME="1315854630805"
    Counters
    plan="{"queryId":"hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4","queryType":null,"queryAttributes":{"queryString":"show
    partitions
    dart_facts"},"queryCounters":"null","stageGraph":{"nodeType":"STAGE","roots":"null","adjacencyList":"]"},"stageList":[{"stageId":"Stage-0","stageType":"DDL","stageAttributes":"null","stageCounters":"}","taskList":[{"taskId":"Stage-0_OTHER","taskType":"OTHER","taskAttributes":"null","taskCounters":"null","operatorGraph":"null","operatorList":"]","done":"false","started":"true"}],"done":"false","started":"true"}],"done":"false","started":"true"}"
    TIME="1315854630806"
    -----------------------------------------

    It doesn't look like standard log4j logging to me and this logging does not
    respond to the log4j properties in :

    -/usr/lib/hive/conf/hive-log4j (these settings influence
    the detail in HIVE_LOG_DIR/hive.log)
    -/usr/lib/hive/conf/hive-exec-log4j (don't know where these
    settings are meant for. Anyone ? )

    I have to find a way to turn the logging in /tmp/${user.name}/ down. I ran a
    large Hive job once that generated 8GB of logging this way which stalled the
    server.

    Thanks

    --
    Kind Regards,

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedSep 12, '11 at 7:32p
activeSep 12, '11 at 7:43p
posts2
users2
websitehive.apache.org

People

Translate

site design / logo © 2021 Grokbase