Grokbase Groups Hive user May 2011
FAQ
My Hive metastore uses MySQL. I'd like to see Hive CLI log all SQL queries that are issued to MySQL. What config/property should I set to accomplish this?

Thanks.
Steven

Search Discussions

  • Ashish Thusoo at May 23, 2011 at 10:07 pm
    you will have to write a pre execute or post execute hook to do this. The Hook api is at

    http://hive.apache.org/docs/r0.7.0/api/org/apache/hadoop/hive/ql/hooks/package-summary.html

    <http://hive.apache.org/docs/r0.7.0/api/org/apache/hadoop/hive/ql/hooks/package-summary.html>and then specify your hook in the site.xml for hive as follows...

    <property>
    <name>hive.exec.post.hooks</name>
    <value>[name of the class that implements the interface]</value>
    <description>Post execution hooks</description>
    </property>

    Ashish

    On May 23, 2011, at 2:45 PM, Steven Wong wrote:

    My Hive metastore uses MySQL. I’d like to see Hive CLI log all SQL queries that are issued to MySQL. What config/property should I set to accomplish this?

    Thanks.
    Steven
  • Carl Steinbach at May 23, 2011 at 10:51 pm
    Hi Steven,

    I think you can tell Datanucleus to log SQL using some of the configuration
    properties described here:
    http://www.datanucleus.org/products/accessplatform/logging.html

    It looks like the Datanucleus.Datastore.* categories are the ones you are
    interested in.

    Another option which may work is better is to use the log4jdbc proxy
    driver: http://code.google.com/p/log4jdbc/

    Hope this helps.

    Carl
    On Mon, May 23, 2011 at 2:45 PM, Steven Wong wrote:

    My Hive metastore uses MySQL. I’d like to see Hive CLI log all SQL queries
    that are issued to MySQL. What config/property should I set to accomplish
    this?



    Thanks.

    Steven
  • Steven Wong at May 23, 2011 at 11:13 pm
    After posting my question, I did some digging and also found the log4j categories. Unfortunately, setting them to DEBUG in hive-log4j.properties has no effect.

    I will look into log4jdbc. Thanks for pointing it out.

    Steven


    From: Carl Steinbach
    Sent: Monday, May 23, 2011 3:51 PM
    To: user@hive.apache.org
    Subject: Re: Logging MySQL queries

    Hi Steven,

    I think you can tell Datanucleus to log SQL using some of the configuration properties described here: http://www.datanucleus.org/products/accessplatform/logging.html

    It looks like the Datanucleus.Datastore.* categories are the ones you are interested in.

    Another option which may work is better is to use the log4jdbc proxy driver: http://code.google.com/p/log4jdbc/

    Hope this helps.

    Carl
    On Mon, May 23, 2011 at 2:45 PM, Steven Wong wrote:
    My Hive metastore uses MySQL. I'd like to see Hive CLI log all SQL queries that are issued to MySQL. What config/property should I set to accomplish this?

    Thanks.
    Steven
  • Steven Wong at May 25, 2011 at 12:01 am
    I am using Amazon EMR. I have discovered that setting those log4j categories has the desired effect in EMR Hive 0.5, but has no effect in EMR Hive 0.7 beta version.

    Setting them in Apache Hive 0.7 release version (on my computer, with Derby instead of MySQL) has the desired effect also.


    From: Steven Wong
    Sent: Monday, May 23, 2011 4:13 PM
    To: user@hive.apache.org
    Subject: RE: Logging MySQL queries

    After posting my question, I did some digging and also found the log4j categories. Unfortunately, setting them to DEBUG in hive-log4j.properties has no effect.

    I will look into log4jdbc. Thanks for pointing it out.

    Steven


    From: Carl Steinbach
    Sent: Monday, May 23, 2011 3:51 PM
    To: user@hive.apache.org
    Subject: Re: Logging MySQL queries

    Hi Steven,

    I think you can tell Datanucleus to log SQL using some of the configuration properties described here: http://www.datanucleus.org/products/accessplatform/logging.html

    It looks like the Datanucleus.Datastore.* categories are the ones you are interested in.

    Another option which may work is better is to use the log4jdbc proxy driver: http://code.google.com/p/log4jdbc/

    Hope this helps.

    Carl
    On Mon, May 23, 2011 at 2:45 PM, Steven Wong wrote:
    My Hive metastore uses MySQL. I'd like to see Hive CLI log all SQL queries that are issued to MySQL. What config/property should I set to accomplish this?

    Thanks.
    Steven
  • Edward Capriolo at May 25, 2011 at 2:24 am
    MySQL has query logging, also mysql-proxy exists.

    Edward
    On Tue, May 24, 2011 at 8:01 PM, Steven Wong wrote:

    I am using Amazon EMR. I have discovered that setting those log4j
    categories has the desired effect in EMR Hive 0.5, but has no effect in EMR
    Hive 0.7 beta version.



    Setting them in Apache Hive 0.7 release version (on my computer, with Derby
    instead of MySQL) has the desired effect also.





    *From:* Steven Wong
    *Sent:* Monday, May 23, 2011 4:13 PM
    *To:* user@hive.apache.org
    *Subject:* RE: Logging MySQL queries



    After posting my question, I did some digging and also found the log4j
    categories. Unfortunately, setting them to DEBUG in hive-log4j.properties
    has no effect.



    I will look into log4jdbc. Thanks for pointing it out.



    Steven





    *From:* Carl Steinbach
    *Sent:* Monday, May 23, 2011 3:51 PM
    *To:* user@hive.apache.org
    *Subject:* Re: Logging MySQL queries



    Hi Steven,



    I think you can tell Datanucleus to log SQL using some of the configuration
    properties described here:
    http://www.datanucleus.org/products/accessplatform/logging.html



    It looks like the Datanucleus.Datastore.* categories are the ones you are
    interested in.



    Another option which may work is better is to use the log4jdbc proxy
    driver: http://code.google.com/p/log4jdbc/



    Hope this helps.



    Carl

    On Mon, May 23, 2011 at 2:45 PM, Steven Wong wrote:

    My Hive metastore uses MySQL. I’d like to see Hive CLI log all SQL queries
    that are issued to MySQL. What config/property should I set to accomplish
    this?



    Thanks.

    Steven

  • Steven Wong at Jun 2, 2011 at 5:39 pm
    It's my bad. I was editing hive/conf/hive-log4j.properties, but turns out hive is actually a symlink to the 0.5 tree, not 0.7. So, the properties have an effect in 0.7 for me now.


    From: Steven Wong
    Sent: Tuesday, May 24, 2011 5:01 PM
    To: user@hive.apache.org
    Subject: RE: Logging MySQL queries

    I am using Amazon EMR. I have discovered that setting those log4j categories has the desired effect in EMR Hive 0.5, but has no effect in EMR Hive 0.7 beta version.

    Setting them in Apache Hive 0.7 release version (on my computer, with Derby instead of MySQL) has the desired effect also.


    From: Steven Wong
    Sent: Monday, May 23, 2011 4:13 PM
    To: user@hive.apache.org
    Subject: RE: Logging MySQL queries

    After posting my question, I did some digging and also found the log4j categories. Unfortunately, setting them to DEBUG in hive-log4j.properties has no effect.

    I will look into log4jdbc. Thanks for pointing it out.

    Steven


    From: Carl Steinbach
    Sent: Monday, May 23, 2011 3:51 PM
    To: user@hive.apache.org
    Subject: Re: Logging MySQL queries

    Hi Steven,

    I think you can tell Datanucleus to log SQL using some of the configuration properties described here: http://www.datanucleus.org/products/accessplatform/logging.html

    It looks like the Datanucleus.Datastore.* categories are the ones you are interested in.

    Another option which may work is better is to use the log4jdbc proxy driver: http://code.google.com/p/log4jdbc/

    Hope this helps.

    Carl
    On Mon, May 23, 2011 at 2:45 PM, Steven Wong wrote:
    My Hive metastore uses MySQL. I'd like to see Hive CLI log all SQL queries that are issued to MySQL. What config/property should I set to accomplish this?

    Thanks.
    Steven
  • Dilip Joseph at May 24, 2011 at 12:26 am
    If you just want to temporarily look at the queries while debugging some
    problem, one option I have found useful is to enable logging of all queries
    on your mysqld (assuming mysqld instance is used only for hive, and is not
    under heavy load).

    Dilip
    On Mon, May 23, 2011 at 2:45 PM, Steven Wong wrote:

    My Hive metastore uses MySQL. I’d like to see Hive CLI log all SQL queries
    that are issued to MySQL. What config/property should I set to accomplish
    this?



    Thanks.

    Steven


    --
    _________________________________________
    Dilip Antony Joseph
    http://csgrad.blogspot.com
    http://www.marydilip.info

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedMay 23, '11 at 9:46p
activeJun 2, '11 at 5:39p
posts8
users5
websitehive.apache.org

People

Translate

site design / logo © 2021 Grokbase