Re: Query Impala with SQL Alchemy or Python in general

Ricky Saltzer

at Apr 9, 2013 at 6:44 pm

Impala's shell is written in Python, it makes use of Thrift for
communication to Impala daemons. I have experience using SQLAlchemy, but
I'm not familiar with implementing a custom DB, although, I'd
be interested in seeing this happen! Perhaps taking a look at the code for
Impala's shell might help.

https://github.com/cloudera/impala/tree/master/shell

On Tue, Apr 9, 2013 at 2:34 PM, Alex Breshears wrote:

Are you open to Jython or are you set on CPython? I have found a lot of
luck by using Jython and JDBC for both Impala and Hive.

I'd be happy to share some sample code if you'd like.

On Tue, Apr 9, 2013 at 1:31 PM, infalco wrote:

Hi Everyone,

We're using python and SQL Alchemy expression language for our databases,
mainly MySQL.
I think it would very handy if we could use SQL Alchemy for Impala as
well.
Have anyone tried to do so?

And another question: in general, what is the best way to access Hive or
Impala from Python?
Is there any specific libraries for this?

Thanks in advance,
Ed

--
Alex Breshears
e: apbresh@gmail.com
l: www.linkedin.com/in/alexbreshears/ <http://p-k.co/BxbZcI>

--
Ricky Saltzer
Customer Operations Engineer
http://www.cloudera.com

reply | permalink

Stephen Boesch

at Apr 9, 2013 at 7:28 pm

⇧

Hi Alex,
I would be most interested in your jython code and
configuration/setup/build stuff. The latter can b challenging sometimes in
jvm langugages

2013/4/9 Alex Breshears <apbresh@gmail.com>

Are you open to Jython or are you set on CPython? I have found a lot of
luck by using Jython and JDBC for both Impala and Hive.

I'd be happy to share some sample code if you'd like.

On Tue, Apr 9, 2013 at 1:31 PM, infalco wrote:

Hi Everyone,

We're using python and SQL Alchemy expression language for our databases,
mainly MySQL.
I think it would very handy if we could use SQL Alchemy for Impala as
well.
Have anyone tried to do so?

And another question: in general, what is the best way to access Hive or
Impala from Python?
Is there any specific libraries for this?

Thanks in advance,
Ed

--
Alex Breshears
e: apbresh@gmail.com
l: www.linkedin.com/in/alexbreshears/ <http://p-k.co/BxbZcI>

reply | permalink

Alex Breshears

at Apr 9, 2013 at 7:50 pm

⇧

I'm running Django on top of Jython specifically for building applications
that interact with Hadoop. Pretty much the entire load is on the Hadoop
cluster, so I'm trading the performance loss (if it's even really there)
for speed of development.

Here's some specifics:

    - Django 1.3.1
    - Jython 2.5.2
    - Django-jython for ORM that works with Postgres &
    Jython https://code.google.com/p/django-jython/
    - Tomcat

There are some tools in the Django-jython package that you can deploy a WAR
for Tomcat.

However, one really annoying thing about Jython is that there isn't a clean
way of deploying a jar for your "regular" applications. But there are
plenty of tools for Django to help out.

Any other specifics you wanna know about? I'll work on getting a few files
that I'm ok with open sourcing for a hive connector that works on Jython.

Alex

On Tuesday, April 9, 2013 2:28:51 PM UTC-5, Stephen Boesch wrote:

Hi Alex,
I would be most interested in your jython code and
configuration/setup/build stuff. The latter can b challenging sometimes in
jvm langugages

2013/4/9 Alex Breshears <apb...@gmail.com <javascript:>>

Are you open to Jython or are you set on CPython? I have found a lot of
luck by using Jython and JDBC for both Impala and Hive.

I'd be happy to share some sample code if you'd like.

On Tue, Apr 9, 2013 at 1:31 PM, infalco <inf...@gmail.com <javascript:>>wrote:

Hi Everyone,

We're using python and SQL Alchemy expression language for our
databases, mainly MySQL.
I think it would very handy if we could use SQL Alchemy for Impala as
well.
Have anyone tried to do so?

And another question: in general, what is the best way to access Hive or
Impala from Python?
Is there any specific libraries for this?

Thanks in advance,
Ed

--
Alex Breshears
e: apb...@gmail.com
<javascript:>
l: www.linkedin.com/in/alexbreshears/ <http://p-k.co/BxbZcI>

reply | permalink

Alex Breshears

at Apr 9, 2013 at 8:50 pm

⇧

Here's a github repo with the connector I'm using. It's not pretty or
clean, but it does work.

https://github.com/t3hi3x/hive-connector-jython

On Tuesday, April 9, 2013 2:50:11 PM UTC-5, Alex Breshears wrote:

I'm running Django on top of Jython specifically for building applications
that interact with Hadoop. Pretty much the entire load is on the Hadoop
cluster, so I'm trading the performance loss (if it's even really there)
for speed of development.

Here's some specifics:

- Django 1.3.1
- Jython 2.5.2
- Django-jython for ORM that works with Postgres & Jython
https://code.google.com/p/django-jython/
- Tomcat

There are some tools in the Django-jython package that you can deploy a
WAR for Tomcat.

However, one really annoying thing about Jython is that there isn't a
clean way of deploying a jar for your "regular" applications. But there are
plenty of tools for Django to help out.

Any other specifics you wanna know about? I'll work on getting a few files
that I'm ok with open sourcing for a hive connector that works on Jython.

Alex

On Tuesday, April 9, 2013 2:28:51 PM UTC-5, Stephen Boesch wrote:

Hi Alex,
I would be most interested in your jython code and
configuration/setup/build stuff. The latter can b challenging sometimes in
jvm langugages

2013/4/9 Alex Breshears <apb...@gmail.com>

Are you open to Jython or are you set on CPython? I have found a lot of
luck by using Jython and JDBC for both Impala and Hive.

I'd be happy to share some sample code if you'd like.

On Tue, Apr 9, 2013 at 1:31 PM, infalco wrote:

Hi Everyone,

We're using python and SQL Alchemy expression language for our
databases, mainly MySQL.
I think it would very handy if we could use SQL Alchemy for Impala as
well.
Have anyone tried to do so?

And another question: in general, what is the best way to access Hive
or Impala from Python?
Is there any specific libraries for this?

Thanks in advance,
Ed

--
Alex Breshears
e: apb...@gmail.com
l: www.linkedin.com/in/alexbreshears/ <http://p-k.co/BxbZcI>

reply | permalink

Stephen Boesch

at Apr 9, 2013 at 8:53 pm

⇧

thx Alex! my company uses jruby and python presently, appreciate the input
on jython (as a potential addtional option )

2013/4/9 Alex Breshears <apbresh@gmail.com>

Here's a github repo with the connector I'm using. It's not pretty or
clean, but it does work.

https://github.com/t3hi3x/hive-connector-jython

On Tuesday, April 9, 2013 2:50:11 PM UTC-5, Alex Breshears wrote:

I'm running Django on top of Jython specifically for building
applications that interact with Hadoop. Pretty much the entire load is on
the Hadoop cluster, so I'm trading the performance loss (if it's even
really there) for speed of development.

Here's some specifics:

- Django 1.3.1
- Jython 2.5.2
- Django-jython for ORM that works with Postgres & Jython
https://code.google.**com/p/django-jython/<https://code.google.com/p/django-jython/>
- Tomcat

There are some tools in the Django-jython package that you can deploy a
WAR for Tomcat.

However, one really annoying thing about Jython is that there isn't a
clean way of deploying a jar for your "regular" applications. But there are
plenty of tools for Django to help out.

Any other specifics you wanna know about? I'll work on getting a few
files that I'm ok with open sourcing for a hive connector that works on
Jython.

Alex

On Tuesday, April 9, 2013 2:28:51 PM UTC-5, Stephen Boesch wrote:

Hi Alex,
I would be most interested in your jython code and
configuration/setup/build stuff. The latter can b challenging sometimes in
jvm langugages

2013/4/9 Alex Breshears <apb...@gmail.com>

Are you open to Jython or are you set on CPython? I have found a lot of
luck by using Jython and JDBC for both Impala and Hive.

I'd be happy to share some sample code if you'd like.

On Tue, Apr 9, 2013 at 1:31 PM, infalco wrote:

Hi Everyone,

We're using python and SQL Alchemy expression language for our
databases, mainly MySQL.
I think it would very handy if we could use SQL Alchemy for Impala as
well.
Have anyone tried to do so?

And another question: in general, what is the best way to access Hive
or Impala from Python?
Is there any specific libraries for this?

Thanks in advance,
Ed

--
Alex Breshears
e: apb...@gmail.com
l: www.linkedin.com/in/**alexbreshears/ <http://p-k.co/BxbZcI>

reply | permalink

Philip Zeyliger

at Apr 10, 2013 at 3:41 pm

⇧

I've been using Impala from python, and I tend to simply call impala-shell
as a subprocess. I posted a little gist (
https://gist.github.com/philz/5355729) to call the Thrift APIs directly
from Python, but it's not really any better than what impala-shell does at
the moment and doesn't handle query cancellation.

Cheers,

-- Philip

On Tuesday, April 9, 2013 1:53:43 PM UTC-7, Stephen Boesch wrote:

thx Alex! my company uses jruby and python presently, appreciate the
input on jython (as a potential addtional option )

2013/4/9 Alex Breshears <apb...@gmail.com <javascript:>>

Here's a github repo with the connector I'm using. It's not pretty or
clean, but it does work.

https://github.com/t3hi3x/hive-connector-jython

On Tuesday, April 9, 2013 2:50:11 PM UTC-5, Alex Breshears wrote:

I'm running Django on top of Jython specifically for building
applications that interact with Hadoop. Pretty much the entire load is on
the Hadoop cluster, so I'm trading the performance loss (if it's even
really there) for speed of development.

Here's some specifics:

- Django 1.3.1
- Jython 2.5.2
- Django-jython for ORM that works with Postgres & Jython
https://code.google.**com/p/django-jython/<https://code.google.com/p/django-jython/>
- Tomcat

There are some tools in the Django-jython package that you can deploy a
WAR for Tomcat.

However, one really annoying thing about Jython is that there isn't a
clean way of deploying a jar for your "regular" applications. But there are
plenty of tools for Django to help out.

Any other specifics you wanna know about? I'll work on getting a few
files that I'm ok with open sourcing for a hive connector that works on
Jython.

Alex

On Tuesday, April 9, 2013 2:28:51 PM UTC-5, Stephen Boesch wrote:

Hi Alex,
I would be most interested in your jython code and
configuration/setup/build stuff. The latter can b challenging sometimes in
jvm langugages

2013/4/9 Alex Breshears <apb...@gmail.com>

Are you open to Jython or are you set on CPython? I have found a lot
of luck by using Jython and JDBC for both Impala and Hive.

I'd be happy to share some sample code if you'd like.

On Tue, Apr 9, 2013 at 1:31 PM, infalco wrote:

Hi Everyone,

We're using python and SQL Alchemy expression language for our
databases, mainly MySQL.
I think it would very handy if we could use SQL Alchemy for Impala as
well.
Have anyone tried to do so?

And another question: in general, what is the best way to access Hive
or Impala from Python?
Is there any specific libraries for this?

Thanks in advance,
Ed

--
Alex Breshears
e: apb...@gmail.com
l: www.linkedin.com/in/**alexbreshears/ <http://p-k.co/BxbZcI>

reply | permalink

Uri Laserson

at May 13, 2013 at 11:44 pm

⇧

Also this:

https://github.com/laserson/impyla

This currently works with impala(-shell) 0.7. Working on fixing it up to
1.0...

Uri

On Wed, Apr 10, 2013 at 8:41 AM, Philip Zeyliger wrote:

I've been using Impala from python, and I tend to simply call impala-shell
as a subprocess. I posted a little gist (
https://gist.github.com/philz/5355729) to call the Thrift APIs directly
from Python, but it's not really any better than what impala-shell does at
the moment and doesn't handle query cancellation.

Cheers,

-- Philip

On Tuesday, April 9, 2013 1:53:43 PM UTC-7, Stephen Boesch wrote:

thx Alex! my company uses jruby and python presently, appreciate the
input on jython (as a potential addtional option )

2013/4/9 Alex Breshears <apb...@gmail.com>

Here's a github repo with the connector I'm using. It's not pretty or

clean, but it does work.

https://github.com/t3hi3x/**hive-connector-jython<https://github.com/t3hi3x/hive-connector-jython>

On Tuesday, April 9, 2013 2:50:11 PM UTC-5, Alex Breshears wrote:

I'm running Django on top of Jython specifically for building
applications that interact with Hadoop. Pretty much the entire load is on
the Hadoop cluster, so I'm trading the performance loss (if it's even
really there) for speed of development.

Here's some specifics:

- Django 1.3.1
- Jython 2.5.2
- Django-jython for ORM that works with Postgres & Jython
https://code.google.**com**/p/django-jython/<https://code.google.com/p/django-jython/>
- Tomcat

There are some tools in the Django-jython package that you can deploy a
WAR for Tomcat.

However, one really annoying thing about Jython is that there isn't a
clean way of deploying a jar for your "regular" applications. But there are
plenty of tools for Django to help out.

Any other specifics you wanna know about? I'll work on getting a few
files that I'm ok with open sourcing for a hive connector that works on
Jython.

Alex

On Tuesday, April 9, 2013 2:28:51 PM UTC-5, Stephen Boesch wrote:

Hi Alex,
I would be most interested in your jython code and
configuration/setup/build stuff. The latter can b challenging sometimes in
jvm langugages

2013/4/9 Alex Breshears <apb...@gmail.com>

Are you open to Jython or are you set on CPython? I have found a lot
of luck by using Jython and JDBC for both Impala and Hive.

I'd be happy to share some sample code if you'd like.

On Tue, Apr 9, 2013 at 1:31 PM, infalco wrote:

Hi Everyone,

We're using python and SQL Alchemy expression language for our
databases, mainly MySQL.
I think it would very handy if we could use SQL Alchemy for Impala
as well.
Have anyone tried to do so?

And another question: in general, what is the best way to access
Hive or Impala from Python?
Is there any specific libraries for this?

Thanks in advance,
Ed

--
Alex Breshears
e: apb...@gmail.com
l: www.linkedin.com/in/**alexbre**shears/ <http://p-k.co/BxbZcI>

--
Uri Laserson, PhD
Data Scientist, Cloudera
Twitter/GitHub: @laserson
+1 617 910 0447
laserson@cloudera.com

reply | permalink

group	impala-user @
categories	hadoop
posted	Apr 9, '13 at 6:34p
active	May 13, '13 at 11:44p
posts	8
users	5
website	cloudera.com
irc	#hadoop

responses	expanded	Hotkey: s
font	variable	Hotkey: f
user style	avatars	Hotkey: a

Re: Query Impala with SQL Alchemy or Python in general

Search Discussions

Related Discussions

5 users in discussion