Creating Hive metastore tables always fail.

wrote:
Hi,

I try to build a LAB for Cloudera 5.0

There are 2 hosts in the same domain
describe as below,

Cloudera51»Manager, Name Node
2GB RAM
50GB Space

Cloudera52»Data Node
2GB RAM
50GB Space

I followed the "Cloudera-Manager-Installation-Guide.pdf", and used
"Installation Path A - Automated Installation by Cloudera Manager "

first, I installed the cloudera manager 5 on Cloudera51
Second, Choosed the cluster hosts
Third, Manager automatically assigned Cloudera51 is Name Node, and
Cloudera52 is Data Node.
fourth, Using Manager Web UI to install CDH5 on both Cloudera51 and
Cloudera52 (with Parcel)
I used embedded PostgreSQL Database

Everything is fine until Manager try to install Hive service.
It's always fail when creating megastore tables. (I had tried at least 3
times)

Here is error log:

Failed to Create Hive Metastore Database Tables.

*Program:* hive/hive.sh ["create_metastore_tables"]

+ [[ create_metastore_tables == updatelocation ]]
+ [[ create_metastore_tables == create_metastore_tables ]]
+ [[ -z '' ]]
+ JDBC_JARS_CLASSPATH='/usr/share/cmf/lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar'
+ [[ create_metastore_tables == create_metastore_tables ]]
+ '[' 5 -ge 5 ']'
+ exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool -dbType postgres -initSchema -dbOpts postgres.filter.81
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/04/23 17:21:50 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
*** schemaTool failed ***


Can anyone help me to solved this problem?
Thanks a lot.

To unsubscribe from this group and stop receiving emails from it, send an
email to [email protected] <javascript:>.

Hi Ivan,

You've configured Hive to talk to a schema that already has tables, as
indicated by this message:
Error: ERROR: relation "BUCKETING_COLS" already exists
(state=42P07,code=0)

Configure Hive with a clean schema and this problem should go away.

Thanks,
Darren

To unsubscribe from this group and stop receiving emails from it, send

wrote:
Thanks Darren,

It's weird, I installed CDH 5 from clean system.
There shouldn't be an existing table.

Another problem is I can't use schematool to initialise the DB.
Would you please describe more specifically how to clean schema.

Thanks again..


Darren Lo於 2014年4月24日星期四UTC+8上午12時50分04秒寫道：
an email to [email protected] <javascript:>.

Hi Ivan,

You're right that the schema should have been clean when you first
started.

Can you confirm your Cloudera Manager version? Click on Support -> About
in the upper right. You may need to exit the wizard to see this link (exit
the wizard by clicking on the Cloudera Manager logo).

Did you use parcels for CDH binaries? (this is the default choice)

What version of postgresql was installed? ssh in to the machine and see
what the package manager says the version of postgresql is.

I suspect that the first time you ran the create tables command, the
error log had a different message. SSH in to the machine where the Hive
Metastore Server is running, then look in /var/run/cloudera-scm-agent/process
and find the directories with a name like "52-hive-metastore-create-tables".
The "52" part will be different on your system. Find the earliest attempt,
and look in the logs directory for the stderr log. Please share the end of
that log.

Thanks,
Darren

To unsubscribe from this group and stop receiving emails from it, send

wrote:
Dear Darren,

1.My Cloudera version is "Cloudera Express 5.0.0 (#215 built by jenkins
on 20140331-1424 git"

2.Yes, I used parcels for CDH binaries.

3.I used embedded PostgreSQL Database (default option). Sorry I can't
find the version info.

4.You are right too. First time when I ran the create tables command
seems no error. The log is as below

+ exec
/opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool
-dbType postgres -initSchema -dbOpts postgres.filter.81
14/04/23 16:56:10 INFO Configuration.deprecation:
mapred.input.dir.recursive is deprecated. Instead, use
mapreduce.input.fileinputformat.input.dir.recursive
14/04/23 16:56:10 INFO Configuration.deprecation: mapred.max.split.size
is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
14/04/23 16:56:10 INFO Configuration.deprecation: mapred.min.split.size
is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/04/23 16:56:10 INFO Configuration.deprecation:
mapred.min.split.size.per.rack is deprecated. Instead, use
mapreduce.input.fileinputformat.split.minsize.per.rack
14/04/23 16:56:10 INFO Configuration.deprecation:
mapred.min.split.size.per.node is deprecated. Instead, use
mapreduce.input.fileinputformat.split.minsize.per.node
14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks is
deprecated. Instead, use mapreduce.job.reduces
14/04/23 16:56:10 INFO Configuration.deprecation:
mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
mapreduce.reduce.speculative
14/04/23 16:56:17 WARN conf.HiveConf: DEPRECATED: Configuration property
hive.metastore.local no longer has any effect. Make sure to provide a valid
value for hive.metastore.uris if you are connecting to a remote megastore.

So I try to check next step process (26-hdfs-NAMENODE-createdir) log.
There are some error messages.

+ echo 'Created directory /user/hive.'
+
/opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
--config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir dfs
-chown hive:hive /user/hive
log4j:ERROR Could not find value for key log4j.appender.EventCounter
log4j:ERROR Could not instantiate appender named "EventCounter".
+ '[' 0 -eq 0 ']'
+ echo 'Changed ownership of /user/hive to hive:hive.'
+
/opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
--config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir dfs
-chmod 1775 /user/hive
log4j:ERROR Could not find value for key log4j.appender.EventCounter
log4j:ERROR Could not instantiate appender named "EventCounter".




Darren Lo於 2014年4月24日星期四UTC+8上午1時59分25秒寫道：
an email to [email protected] <javascript:>.

Hi Ivan,

It looks like create table actually succeeded the first time. What
happens if you try to start the metastore (via the CM UI)?

Also, what operating system are you using?

Thanks,
Darren

To unsubscribe from this group and stop receiving emails from it, send

wrote:
Dear Darren,

It's seems work. I can start the metastore via UI.

Thu Apr 24 03:26:29 CST 2014
JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
using /usr/java/jdk1.7.0_45-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive as HIVE_HOME
using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE as HIVE_CONF_DIR
using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop as HADOOP_HOME
using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE/yarn-conf as HADOOP_CONF_DIR
Starting Hive Metastore Server


Both OS are CentOS 6.4.

If it does really work, can I just pass this process and keep running the
rest of process via CM UI?


<https://lh3.googleusercontent.com/-7Mkz3nBNeJs/U1gWaWKBePI/AAAAAAAAAhI/YgDpz6I1F-8/s1600/Screen+Shot+2014-04-24+at+3.31.40+AM.png>

Many thanks.


Darren Lo於 2014年4月24日星期四UTC+8上午3時23分24秒寫道：
an email to [email protected] <javascript:>.

Creating Hive Metastore Database Tables

You should run the Create Hive User Directory and Create Hive Warehouse
Directory commands (available if you click on Hive and look in the actions
menu on the right), then the Start command for hive.

Then you should be able to re-try the first run command (find it in your
command history at http://<cm_host>:7180/cmf/commands/commands, click on
your first run command, and look for a retry button), or you can run each
of the listed commands manually, one by one.

On Wednesday, April 23, 2014 1:29:00 PM UTC-7, Ivan Hsueh wrote:

Dear Darren,

Finally, all services are start.
Thank you for helping me!


Darren Lo於 2014年4月24日星期四UTC+8上午3時47分36秒寫道：