FAQ
Are you open to Jython or are you set on CPython? I have found a lot of
luck by using Jython and JDBC for both Impala and Hive.

I'd be happy to share some sample code if you'd like.
On Tue, Apr 9, 2013 at 1:31 PM, infalco wrote:

Hi Everyone,

We're using python and SQL Alchemy expression language for our databases,
mainly MySQL.
I think it would very handy if we could use SQL Alchemy for Impala as well.
Have anyone tried to do so?

And another question: in general, what is the best way to access Hive or
Impala from Python?
Is there any specific libraries for this?

Thanks in advance,
Ed


--
Alex Breshears
e: apbresh@gmail.com
l: www.linkedin.com/in/alexbreshears/ <http://p-k.co/BxbZcI>

Search Discussions

  • Ricky Saltzer at Apr 9, 2013 at 6:44 pm
    Impala's shell is written in Python, it makes use of Thrift for
    communication to Impala daemons. I have experience using SQLAlchemy, but
    I'm not familiar with implementing a custom DB, although, I'd
    be interested in seeing this happen! Perhaps taking a look at the code for
    Impala's shell might help.

    https://github.com/cloudera/impala/tree/master/shell



    On Tue, Apr 9, 2013 at 2:34 PM, Alex Breshears wrote:

    Are you open to Jython or are you set on CPython? I have found a lot of
    luck by using Jython and JDBC for both Impala and Hive.

    I'd be happy to share some sample code if you'd like.
    On Tue, Apr 9, 2013 at 1:31 PM, infalco wrote:

    Hi Everyone,

    We're using python and SQL Alchemy expression language for our databases,
    mainly MySQL.
    I think it would very handy if we could use SQL Alchemy for Impala as
    well.
    Have anyone tried to do so?

    And another question: in general, what is the best way to access Hive or
    Impala from Python?
    Is there any specific libraries for this?

    Thanks in advance,
    Ed


    --
    Alex Breshears
    e: apbresh@gmail.com
    l: www.linkedin.com/in/alexbreshears/ <http://p-k.co/BxbZcI>


    --
    Ricky Saltzer
    Customer Operations Engineer
    http://www.cloudera.com
  • Stephen Boesch at Apr 9, 2013 at 7:28 pm
    Hi Alex,
       I would be most interested in your jython code and
    configuration/setup/build stuff. The latter can b challenging sometimes in
    jvm langugages


    2013/4/9 Alex Breshears <apbresh@gmail.com>
    Are you open to Jython or are you set on CPython? I have found a lot of
    luck by using Jython and JDBC for both Impala and Hive.

    I'd be happy to share some sample code if you'd like.
    On Tue, Apr 9, 2013 at 1:31 PM, infalco wrote:

    Hi Everyone,

    We're using python and SQL Alchemy expression language for our databases,
    mainly MySQL.
    I think it would very handy if we could use SQL Alchemy for Impala as
    well.
    Have anyone tried to do so?

    And another question: in general, what is the best way to access Hive or
    Impala from Python?
    Is there any specific libraries for this?

    Thanks in advance,
    Ed


    --
    Alex Breshears
    e: apbresh@gmail.com
    l: www.linkedin.com/in/alexbreshears/ <http://p-k.co/BxbZcI>
  • Alex Breshears at Apr 9, 2013 at 7:50 pm
    I'm running Django on top of Jython specifically for building applications
    that interact with Hadoop. Pretty much the entire load is on the Hadoop
    cluster, so I'm trading the performance loss (if it's even really there)
    for speed of development.

    Here's some specifics:

        - Django 1.3.1
        - Jython 2.5.2
        - Django-jython for ORM that works with Postgres &
        Jython https://code.google.com/p/django-jython/
        - Tomcat

    There are some tools in the Django-jython package that you can deploy a WAR
    for Tomcat.

    However, one really annoying thing about Jython is that there isn't a clean
    way of deploying a jar for your "regular" applications. But there are
    plenty of tools for Django to help out.

    Any other specifics you wanna know about? I'll work on getting a few files
    that I'm ok with open sourcing for a hive connector that works on Jython.

    Alex
    On Tuesday, April 9, 2013 2:28:51 PM UTC-5, Stephen Boesch wrote:

    Hi Alex,
    I would be most interested in your jython code and
    configuration/setup/build stuff. The latter can b challenging sometimes in
    jvm langugages


    2013/4/9 Alex Breshears <apb...@gmail.com <javascript:>>
    Are you open to Jython or are you set on CPython? I have found a lot of
    luck by using Jython and JDBC for both Impala and Hive.

    I'd be happy to share some sample code if you'd like.

    On Tue, Apr 9, 2013 at 1:31 PM, infalco <inf...@gmail.com <javascript:>>wrote:
    Hi Everyone,

    We're using python and SQL Alchemy expression language for our
    databases, mainly MySQL.
    I think it would very handy if we could use SQL Alchemy for Impala as
    well.
    Have anyone tried to do so?

    And another question: in general, what is the best way to access Hive or
    Impala from Python?
    Is there any specific libraries for this?

    Thanks in advance,
    Ed


    --
    Alex Breshears
    e: apb...@gmail.com
    <javascript:>
    l: www.linkedin.com/in/alexbreshears/ <http://p-k.co/BxbZcI>
  • Alex Breshears at Apr 9, 2013 at 8:50 pm
    Here's a github repo with the connector I'm using. It's not pretty or
    clean, but it does work.

    https://github.com/t3hi3x/hive-connector-jython
    On Tuesday, April 9, 2013 2:50:11 PM UTC-5, Alex Breshears wrote:

    I'm running Django on top of Jython specifically for building applications
    that interact with Hadoop. Pretty much the entire load is on the Hadoop
    cluster, so I'm trading the performance loss (if it's even really there)
    for speed of development.

    Here's some specifics:

    - Django 1.3.1
    - Jython 2.5.2
    - Django-jython for ORM that works with Postgres & Jython
    https://code.google.com/p/django-jython/
    - Tomcat

    There are some tools in the Django-jython package that you can deploy a
    WAR for Tomcat.

    However, one really annoying thing about Jython is that there isn't a
    clean way of deploying a jar for your "regular" applications. But there are
    plenty of tools for Django to help out.

    Any other specifics you wanna know about? I'll work on getting a few files
    that I'm ok with open sourcing for a hive connector that works on Jython.

    Alex
    On Tuesday, April 9, 2013 2:28:51 PM UTC-5, Stephen Boesch wrote:

    Hi Alex,
    I would be most interested in your jython code and
    configuration/setup/build stuff. The latter can b challenging sometimes in
    jvm langugages


    2013/4/9 Alex Breshears <apb...@gmail.com>
    Are you open to Jython or are you set on CPython? I have found a lot of
    luck by using Jython and JDBC for both Impala and Hive.

    I'd be happy to share some sample code if you'd like.
    On Tue, Apr 9, 2013 at 1:31 PM, infalco wrote:

    Hi Everyone,

    We're using python and SQL Alchemy expression language for our
    databases, mainly MySQL.
    I think it would very handy if we could use SQL Alchemy for Impala as
    well.
    Have anyone tried to do so?

    And another question: in general, what is the best way to access Hive
    or Impala from Python?
    Is there any specific libraries for this?

    Thanks in advance,
    Ed


    --
    Alex Breshears
    e: apb...@gmail.com
    l: www.linkedin.com/in/alexbreshears/ <http://p-k.co/BxbZcI>
  • Stephen Boesch at Apr 9, 2013 at 8:53 pm
    thx Alex! my company uses jruby and python presently, appreciate the input
    on jython (as a potential addtional option )


    2013/4/9 Alex Breshears <apbresh@gmail.com>
    Here's a github repo with the connector I'm using. It's not pretty or
    clean, but it does work.

    https://github.com/t3hi3x/hive-connector-jython

    On Tuesday, April 9, 2013 2:50:11 PM UTC-5, Alex Breshears wrote:

    I'm running Django on top of Jython specifically for building
    applications that interact with Hadoop. Pretty much the entire load is on
    the Hadoop cluster, so I'm trading the performance loss (if it's even
    really there) for speed of development.

    Here's some specifics:

    - Django 1.3.1
    - Jython 2.5.2
    - Django-jython for ORM that works with Postgres & Jython
    https://code.google.**com/p/django-jython/<https://code.google.com/p/django-jython/>
    - Tomcat

    There are some tools in the Django-jython package that you can deploy a
    WAR for Tomcat.

    However, one really annoying thing about Jython is that there isn't a
    clean way of deploying a jar for your "regular" applications. But there are
    plenty of tools for Django to help out.

    Any other specifics you wanna know about? I'll work on getting a few
    files that I'm ok with open sourcing for a hive connector that works on
    Jython.

    Alex
    On Tuesday, April 9, 2013 2:28:51 PM UTC-5, Stephen Boesch wrote:

    Hi Alex,
    I would be most interested in your jython code and
    configuration/setup/build stuff. The latter can b challenging sometimes in
    jvm langugages


    2013/4/9 Alex Breshears <apb...@gmail.com>
    Are you open to Jython or are you set on CPython? I have found a lot of
    luck by using Jython and JDBC for both Impala and Hive.

    I'd be happy to share some sample code if you'd like.
    On Tue, Apr 9, 2013 at 1:31 PM, infalco wrote:

    Hi Everyone,

    We're using python and SQL Alchemy expression language for our
    databases, mainly MySQL.
    I think it would very handy if we could use SQL Alchemy for Impala as
    well.
    Have anyone tried to do so?

    And another question: in general, what is the best way to access Hive
    or Impala from Python?
    Is there any specific libraries for this?

    Thanks in advance,
    Ed


    --
    Alex Breshears
    e: apb...@gmail.com
    l: www.linkedin.com/in/**alexbreshears/ <http://p-k.co/BxbZcI>
  • Philip Zeyliger at Apr 10, 2013 at 3:41 pm
    I've been using Impala from python, and I tend to simply call impala-shell
    as a subprocess. I posted a little gist (
    https://gist.github.com/philz/5355729) to call the Thrift APIs directly
    from Python, but it's not really any better than what impala-shell does at
    the moment and doesn't handle query cancellation.

    Cheers,

    -- Philip
    On Tuesday, April 9, 2013 1:53:43 PM UTC-7, Stephen Boesch wrote:

    thx Alex! my company uses jruby and python presently, appreciate the
    input on jython (as a potential addtional option )


    2013/4/9 Alex Breshears <apb...@gmail.com <javascript:>>
    Here's a github repo with the connector I'm using. It's not pretty or
    clean, but it does work.

    https://github.com/t3hi3x/hive-connector-jython

    On Tuesday, April 9, 2013 2:50:11 PM UTC-5, Alex Breshears wrote:

    I'm running Django on top of Jython specifically for building
    applications that interact with Hadoop. Pretty much the entire load is on
    the Hadoop cluster, so I'm trading the performance loss (if it's even
    really there) for speed of development.

    Here's some specifics:

    - Django 1.3.1
    - Jython 2.5.2
    - Django-jython for ORM that works with Postgres & Jython
    https://code.google.**com/p/django-jython/<https://code.google.com/p/django-jython/>
    - Tomcat

    There are some tools in the Django-jython package that you can deploy a
    WAR for Tomcat.

    However, one really annoying thing about Jython is that there isn't a
    clean way of deploying a jar for your "regular" applications. But there are
    plenty of tools for Django to help out.

    Any other specifics you wanna know about? I'll work on getting a few
    files that I'm ok with open sourcing for a hive connector that works on
    Jython.

    Alex
    On Tuesday, April 9, 2013 2:28:51 PM UTC-5, Stephen Boesch wrote:

    Hi Alex,
    I would be most interested in your jython code and
    configuration/setup/build stuff. The latter can b challenging sometimes in
    jvm langugages


    2013/4/9 Alex Breshears <apb...@gmail.com>
    Are you open to Jython or are you set on CPython? I have found a lot
    of luck by using Jython and JDBC for both Impala and Hive.

    I'd be happy to share some sample code if you'd like.
    On Tue, Apr 9, 2013 at 1:31 PM, infalco wrote:

    Hi Everyone,

    We're using python and SQL Alchemy expression language for our
    databases, mainly MySQL.
    I think it would very handy if we could use SQL Alchemy for Impala as
    well.
    Have anyone tried to do so?

    And another question: in general, what is the best way to access Hive
    or Impala from Python?
    Is there any specific libraries for this?

    Thanks in advance,
    Ed


    --
    Alex Breshears
    e: apb...@gmail.com
    l: www.linkedin.com/in/**alexbreshears/ <http://p-k.co/BxbZcI>
  • Uri Laserson at May 13, 2013 at 11:44 pm
    Also this:

    https://github.com/laserson/impyla

    This currently works with impala(-shell) 0.7. Working on fixing it up to
    1.0...

    Uri

    On Wed, Apr 10, 2013 at 8:41 AM, Philip Zeyliger wrote:

    I've been using Impala from python, and I tend to simply call impala-shell
    as a subprocess. I posted a little gist (
    https://gist.github.com/philz/5355729) to call the Thrift APIs directly
    from Python, but it's not really any better than what impala-shell does at
    the moment and doesn't handle query cancellation.

    Cheers,

    -- Philip
    On Tuesday, April 9, 2013 1:53:43 PM UTC-7, Stephen Boesch wrote:

    thx Alex! my company uses jruby and python presently, appreciate the
    input on jython (as a potential addtional option )


    2013/4/9 Alex Breshears <apb...@gmail.com>

    Here's a github repo with the connector I'm using. It's not pretty or
    clean, but it does work.

    https://github.com/t3hi3x/**hive-connector-jython<https://github.com/t3hi3x/hive-connector-jython>

    On Tuesday, April 9, 2013 2:50:11 PM UTC-5, Alex Breshears wrote:

    I'm running Django on top of Jython specifically for building
    applications that interact with Hadoop. Pretty much the entire load is on
    the Hadoop cluster, so I'm trading the performance loss (if it's even
    really there) for speed of development.

    Here's some specifics:

    - Django 1.3.1
    - Jython 2.5.2
    - Django-jython for ORM that works with Postgres & Jython
    https://code.google.**com**/p/django-jython/<https://code.google.com/p/django-jython/>
    - Tomcat

    There are some tools in the Django-jython package that you can deploy a
    WAR for Tomcat.

    However, one really annoying thing about Jython is that there isn't a
    clean way of deploying a jar for your "regular" applications. But there are
    plenty of tools for Django to help out.

    Any other specifics you wanna know about? I'll work on getting a few
    files that I'm ok with open sourcing for a hive connector that works on
    Jython.

    Alex
    On Tuesday, April 9, 2013 2:28:51 PM UTC-5, Stephen Boesch wrote:

    Hi Alex,
    I would be most interested in your jython code and
    configuration/setup/build stuff. The latter can b challenging sometimes in
    jvm langugages


    2013/4/9 Alex Breshears <apb...@gmail.com>
    Are you open to Jython or are you set on CPython? I have found a lot
    of luck by using Jython and JDBC for both Impala and Hive.

    I'd be happy to share some sample code if you'd like.
    On Tue, Apr 9, 2013 at 1:31 PM, infalco wrote:

    Hi Everyone,

    We're using python and SQL Alchemy expression language for our
    databases, mainly MySQL.
    I think it would very handy if we could use SQL Alchemy for Impala
    as well.
    Have anyone tried to do so?

    And another question: in general, what is the best way to access
    Hive or Impala from Python?
    Is there any specific libraries for this?

    Thanks in advance,
    Ed


    --
    Alex Breshears
    e: apb...@gmail.com
    l: www.linkedin.com/in/**alexbre**shears/ <http://p-k.co/BxbZcI>

    --
    Uri Laserson, PhD
    Data Scientist, Cloudera
    Twitter/GitHub: @laserson
    +1 617 910 0447
    laserson@cloudera.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimpala-user @
categorieshadoop
postedApr 9, '13 at 6:34p
activeMay 13, '13 at 11:44p
posts8
users5
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase