Re: CDH 4.3 HUE 2.3 Can't make it work with HiveServer2

Hi.
I'm doing my best to remove beeswax from HUE UI. I want to use
HiveServer2

I've changed service conf using CM 4.6.3, here it is:

hive_conf_dir

/var/run/cloudera-scm-agent/**process/5965-hue-HUE_SERVER/**hive-conf

*Hive configuration directory, where hive-site.xml is located.*

Default: */var/run/cloudera-scm-agent/process/5965-hue-
HUE_SERVER/hive-conf*
share_saved_queries

True

*Share saved queries with all users. If set to false, saved queries
are visible only to the owner and administrators.*

Default: *True*
metastore_conn_timeout

10

*Timeouts in seconds for Thrift calls to the Hive metastore. This
timeout should take into account that the metastore could talk to an
external database.*

Default: *10*
beeswax_server_port

8002

*Configure the port the Beeswax Thrift server runs on.*

Default: *8002*
beeswax_running_query_lifetime

604800000

*Time in milliseconds for Beeswax to persist queries in its cache.*

Default: *604800000*
hive_home_dir

/opt/cloudera/parcels/CDH-4.3.**0-1.cdh4.3.0.p0.22/lib/hive

*Path to the root of the Hive installation; defaults to environment
variable when not set.*

Default: */opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/hive*
browse_partitioned_table_limit

250

*Set a LIMIT clause when browsing a partitioned table. A positive
value will be set as the LIMIT. If 0 or negative, do not set any limit.
*

Default: *250*
beeswax_server_heapsize

1024

*Maximum Java heap size (in megabytes) used by Beeswax Server. Note
that the setting of HADOOP_HEAPSIZE in $HADOOP_CONF_DIR/hadoop-env.sh may
override this setting.*

Default: *1000*
beeswax_server_conn_timeout

120

*Timeout in seconds for Thrift calls to Beeswax service.*

Default: *120*
beeswax_meta_server_port

8003

*Configure the port the internal metastore daemon runs on. Used only
if hive.metastore.local is true.*

Default: *8003*
local_examples_data_dir

/opt/cloudera/parcels/CDH-4.3.**0-1.cdh4.3.0.p0.22/share/hue/**
apps/beeswax/src/beeswax/../..**/data

*The local filesystem path containing the Beeswax examples.*

Default: */opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.
p0.22/share/hue/apps/beeswax/src/beeswax/../../data*
server_interface

hiveserver2

*Beeswax or Hive Server 2 Thrift API used. Choices are: 'beeswax' or
'hiveserver2'.*

Default: *beeswax*

I got an error:


Invalid method name: 'OpenSession'

Here is a log from HUE. There is nothing special in Beeswax log
strerr/stdout/role log. It says, CM says that service is started and status
is OK
[06/Sep/2013 02:39:15 +0000] thrift_util ERROR Thrift saw
exception (this may be expected).
Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-4.**3.0-1.cdh4.3.0.p0.22/share/**
hue/desktop/core/src/desktop/**lib/thrift_util.py", line 338, in
wrapper
ret = res(*args, **kwargs)
File "/opt/cloudera/parcels/CDH-4.**3.0-1.cdh4.3.0.p0.22/share/**
hue/apps/beeswax/src/beeswax/.**./../gen-py/TCLIService/**TCLIService.py",
line 154, in OpenSession
return self.recv_OpenSession()
File "/opt/cloudera/parcels/CDH-4.**3.0-1.cdh4.3.0.p0.22/share/**
hue/apps/beeswax/src/beeswax/.**./../gen-py/TCLIService/**TCLIService.py",
line 170, in recv_OpenSession
raise x
TApplicationException: Invalid method name: 'OpenSession'
[06/Sep/2013 02:39:15 +0000] thrift_util INFO Thrift saw an
application exception: Invalid method name: 'OpenSession'
[06/Sep/2013 02:39:15 +0000] middleware INFO Processing
exception: Invalid method name: 'OpenSession' (code THRIFTAPPLICATION):
None: Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-4.**3.0-1.cdh4.3.0.p0.22/share/**
hue/build/env/lib/python2.6/**site-packages/Django-1.2.3-**
py2.6.egg/django/core/**handlers/base.py", line 100, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/opt/cloudera/parcels/CDH-4.**3.0-1.cdh4.3.0.p0.22/share/**
hue/apps/beeswax/src/beeswax/**views.py", line 60, in index
return execute_query(request)
File "/opt/cloudera/parcels/CDH-4.**3.0-1.cdh4.3.0.p0.22/share/**
hue/apps/beeswax/src/beeswax/**views.py", line 382, in execute_query
dbs = db.get_databases()
File "/opt/cloudera/parcels/CDH-4.**3.0-1.cdh4.3.0.p0.22/share/**
hue/apps/beeswax/src/beeswax/**server/dbms.py", line 119, in
get_databases
return self.client.get_databases()
File "/opt/cloudera/parcels/CDH-4.**3.0-1.cdh4.3.0.p0.22/share/**
hue/apps/beeswax/src/beeswax/**server/hive_server2_lib.py", line
581, in get_databases
return [table[col] for table in self._client.get_databases()]
File "/opt/cloudera/parcels/CDH-4.**3.0-1.cdh4.3.0.p0.22/share/**
hue/apps/beeswax/src/beeswax/**server/hive_server2_lib.py", line
324, in get_databases
res = self.call(self._client.**GetSchemas, req)
File "/opt/cloudera/parcels/CDH-4.**3.0-1.cdh4.3.0.p0.22/share/**
hue/apps/beeswax/src/beeswax/**server/hive_server2_lib.py", line
289, in call
session = self.open_session(self.user)
File "/opt/cloudera/parcels/CDH-4.**3.0-1.cdh4.3.0.p0.22/share/**
hue/apps/beeswax/src/beeswax/**server/hive_server2_lib.py", line
270, in open_session
res = self._client.OpenSession(req)
File "/opt/cloudera/parcels/CDH-4.**3.0-1.cdh4.3.0.p0.22/share/**
hue/desktop/core/src/desktop/**lib/thrift_util.py", line 289, in
wrapper
raise StructuredException('**THRIFTAPPLICATION', str(e),
data=None, error_code=502)
StructuredException: Invalid method name: 'OpenSession' (code
THRIFTAPPLICATION): None

[06/Sep/2013 02:39:15 +0000] access INFO 10.66.48.122 hdfs
- "GET /oozie/list_oozie_coordinator/**0001067-130903120020812-oozie-
**oozi-C/ HTTP/1.1"


--

I've used
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_hue_configuration.html
as a guide

пятница, 6 сентября 2013 г., 2:53:58 UTC+4 пользователь Serega
Sheypak написал:
---
You received this message because you are subscribed to the Google
Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to cdh-user+unsubscribe@cloudera.org.

For more options, visit
https://groups.google.com/a/cloudera.org/groups/opt_out.

I would recommend to wait for CDH4.4 (end of this week) and test
HiveServer2 before switching to it. This is a pretty big change.

In the meantime:
beeswax_server_port

8002 --> 10000

Is beeswax_server_host pointing to a HiveServer2?


Romain


On Thu, Sep 5, 2013 at 3:54 PM, Serega Sheypak <
serega.sheypak@gmail.com> wrote:
--

---
You received this message because you are subscribed to a topic in
the Google Groups "CDH Users" group.
To unsubscribe from this topic, visit
https://groups.google.com/a/cloudera.org/d/topic/cdh-user/n7wKRtcz4Tw/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
cdh-user+unsubscribe@cloudera.org.

For more options, visit
https://groups.google.com/a/cloudera.org/groups/opt_out.

Hi. I've added
[beeswax]
server_interface=hiveserver2 #was before
beeswax_server_host=prod-beeswax.lol.ru #specified host

And restarted hue service.
I don't see beeswax_server_host property on the
hue->configuration->beeswax tab.
I see only port.




2013/9/6 Romain Rigaux <romain@cloudera.com>
--

---
You received this message because you are subscribed to the Google
Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to cdh-user+unsubscribe@cloudera.org.
For more options, visit
https://groups.google.com/a/cloudera.org/groups/opt_out.

You would need to add the port too, or change 8002 to 10000:

[beeswax]
...
beeswax_server_port=10000

Romain




On Fri, Sep 6, 2013 at 1:51 AM, Serega Sheypak <
serega.sheypak@gmail.com> wrote:
--

---
You received this message because you are subscribed to a topic in
the Google Groups "CDH Users" group.
To unsubscribe from this topic, visit
https://groups.google.com/a/cloudera.org/d/topic/cdh-user/n7wKRtcz4Tw/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
cdh-user+unsubscribe@cloudera.org.
For more options, visit
https://groups.google.com/a/cloudera.org/groups/opt_out.

Romain, I have a mess in my head.
Right now we have:

Hive Service
- HiveServer2 Role
- Metastore role (lives in embedded postgre. We still didn't move it
to separate DB installation instance)

Hue service
- Hue server
- Beeswax server. Beeswax did see Hive Service-> Metastore Role data.
So we hve single metastore for HiveService->HiveServer2 and
HueService->Beeswax

Now I want to disable "beeswax interface" and try to use HiveServer2

As I understood, beeswax it's a "jar" created by Cloudera long time
ago with concurrncy problem fix.
If I choose HiveServer2 interface for beeswax, beeswax role would
start HiveServer2 jar instead of "cloudera-beeswax.jar"

Am I right?

I have prod-beeswax.lol.ru host
There do live:
HiveService
-HiveServer2 (10000 port)
-HiveMetastore(9083 port, postgre DB is on the other host, named
prod-scm.lol.ru)

HueService
-Hue Server (8888 port)
-Beeswax service (8002 port, mestastore port: 8003)

What do I have to do If I want to run HiveServer2 insterface instead
of beeswax interface?



2013/9/6 Romain Rigaux <romain@cloudera.com>
--

---
You received this message because you are subscribed to the Google
Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to cdh-user+unsubscribe@cloudera.org.
For more options, visit
https://groups.google.com/a/cloudera.org/groups/opt_out.

start HiveServer2 jar instead of "cloudera-beeswax.jar"

No, when Hue is configured to use the HiveServer2 interface, Beeswax
won't be started. HiveServer2 belongs to Hive and should be started
independently (e.g. like Oozie, HBase...).

This will be cleaner in CDH5/CM5 as Beeswax will be gone. For now you
need to start a HiveServer2 in the Hive Service and add this in the Hue
safety valve:

[beeswax]
server_interface=hiveserver2
beeswax_server_port=10000
beeswax_server_host=prod-beeswax.lol.ru #HiveServer2 host


We have a blog post about this coming up soon too.

Romain



On Sun, Sep 8, 2013 at 11:38 AM, Serega Sheypak <
serega.sheypak@gmail.com> wrote:
--

---
You received this message because you are subscribed to a topic in
the Google Groups "CDH Users" group.
To unsubscribe from this topic, visit
https://groups.google.com/a/cloudera.org/d/topic/cdh-user/n7wKRtcz4Tw/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
cdh-user+unsubscribe@cloudera.org.
For more options, visit
https://groups.google.com/a/cloudera.org/groups/opt_out.

Ok. I did it and got the same error with open session. The other
problem is that i dont understand how to tell hue location of metastore?
Does it use CM configs to get metastore host and thrift port?
09.09.2013 20:26 пользователь "Romain Rigaux" <romain@cloudera.com>
написал:

"If I choose HiveServer2 interface for beeswax, beeswax role would
To unsubscribe from this group and stop receiving emails from it,
send an email to hue-user+unsubscribe@cloudera.org.

For the metastore, you don't need to worry. The property is not used
anymore as all the calls go to HiveServer2 afterwards.

Romain


On Mon, Sep 9, 2013 at 10:52 AM, Serega Sheypak <
serega.sheypak@gmail.com> wrote:

Cool, thanks.
I'll try tomorrow morning again and report!


2013/9/9 Romain Rigaux <romain@cloudera.com>
--

---
You received this message because you are subscribed to the Google
Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to cdh-user+unsubscribe@cloudera.org.
For more options, visit
https://groups.google.com/a/cloudera.org/groups/opt_out.