To create the tables in the hive metastore relational database, click on
your Hive service, then find the appropriate command in the Actions
dropdown menu on the right. These are the tables that Hive itself uses to
keep track of logical user data.
To create logical hive tables where you would actually run sql queries on,
use any hive client. Beeline is the command-line tool used to talk to
HiveServer2. Hue has a Beeswax app that can talk to Hive as well, if you
prefer a GUI interface. If not already configured, you can add a Hue
service through the Cloudera Manager UI. If it's not already present, use
click the dropdown next to your cluster name and click Add Service.
Thanks,
Darren
On Fri, May 16, 2014 at 1:40 AM, Kevin Huang wrote:
Ivan,
How did you resolve it finally?
Where I can launch Hive Metastore database tables in UI?
Thanks a lot!
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].Ivan,
How did you resolve it finally?
Creating Hive Metastore Database Tables
Thanks a lot!
On Wednesday, April 23, 2014 1:29:00 PM UTC-7, Ivan Hsueh wrote:
Dear Darren,
Finally, all services are start.
Thank you for helping me!
Darren Lo於 2014年4月24日星期四UTC+8上午3時47分36秒寫道：
an email to [email protected].Dear Darren,
Finally, all services are start.
Thank you for helping me!
Darren Lo於 2014年4月24日星期四UTC+8上午3時47分36秒寫道：
You should run the Create Hive User Directory and Create Hive Warehouse
Directory commands (available if you click on Hive and look in the actions
menu on the right), then the Start command for hive.
Then you should be able to re-try the first run command (find it in your
command history at http://<cm_host>:7180/cmf/commands/commands, click
on your first run command, and look for a retry button), or you can run
each of the listed commands manually, one by one.
Directory commands (available if you click on Hive and look in the actions
menu on the right), then the Start command for hive.
Then you should be able to re-try the first run command (find it in your
command history at http://<cm_host>:7180/cmf/commands/commands, click
on your first run command, and look for a retry button), or you can run
each of the listed commands manually, one by one.
On Wed, Apr 23, 2014 at 12:41 PM, Ivan Hsueh wrote:
Dear Darren,
It's seems work. I can start the metastore via UI.
Thu Apr 24 03:26:29 CST 2014
JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
using /usr/java/jdk1.7.0_45-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive as HIVE_HOME
using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE as HIVE_CONF_DIR
using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop as HADOOP_HOME
using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE/yarn-conf as HADOOP_CONF_DIR
Starting Hive Metastore Server
Both OS are CentOS 6.4.
If it does really work, can I just pass this process and keep running
the rest of process via CM UI?
<https://lh3.googleusercontent.com/-7Mkz3nBNeJs/U1gWaWKBePI/AAAAAAAAAhI/YgDpz6I1F-8/s1600/Screen+Shot+2014-04-24+at+3.31.40+AM.png>
Many thanks.
Darren Lo於 2014年4月24日星期四UTC+8上午3時23分24秒寫道：
To unsubscribe from this group and stop receiving emails from it, sendDear Darren,
It's seems work. I can start the metastore via UI.
Thu Apr 24 03:26:29 CST 2014
JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
using /usr/java/jdk1.7.0_45-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive as HIVE_HOME
using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE as HIVE_CONF_DIR
using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop as HADOOP_HOME
using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE/yarn-conf as HADOOP_CONF_DIR
Starting Hive Metastore Server
Both OS are CentOS 6.4.
If it does really work, can I just pass this process and keep running
the rest of process via CM UI?
<https://lh3.googleusercontent.com/-7Mkz3nBNeJs/U1gWaWKBePI/AAAAAAAAAhI/YgDpz6I1F-8/s1600/Screen+Shot+2014-04-24+at+3.31.40+AM.png>
Many thanks.
Darren Lo於 2014年4月24日星期四UTC+8上午3時23分24秒寫道：
Hi Ivan,
It looks like create table actually succeeded the first time. What
happens if you try to start the metastore (via the CM UI)?
Also, what operating system are you using?
Thanks,
Darren
send an email to [email protected].It looks like create table actually succeeded the first time. What
happens if you try to start the metastore (via the CM UI)?
Also, what operating system are you using?
Thanks,
Darren
On Wed, Apr 23, 2014 at 11:50 AM, Ivan Hsueh wrote:
Dear Darren,
1.My Cloudera version is "Cloudera Express 5.0.0 (#215 built by
jenkins on 20140331-1424 git"
2.Yes, I used parcels for CDH binaries.
3.I used embedded PostgreSQL Database (default option). Sorry I can't
find the version info.
4.You are right too. First time when I ran the create tables command
seems no error. The log is as below
+ exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool
-dbType postgres -initSchema -dbOpts postgres.filter.81
14/04/23 16:56:10 INFO Configuration.deprecation:
mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.
fileinputformat.input.dir.recursive
14/04/23 16:56:10 INFO Configuration.deprecation:
mapred.max.split.size is deprecated. Instead, use mapreduce.input.
fileinputformat.split.maxsize
14/04/23 16:56:10 INFO Configuration.deprecation:
mapred.min.split.size is deprecated. Instead, use mapreduce.input.
fileinputformat.split.minsize
14/04/23 16:56:10 INFO Configuration.deprecation:
mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.
fileinputformat.split.minsize.per.rack
14/04/23 16:56:10 INFO Configuration.deprecation:
mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.
fileinputformat.split.minsize.per.node
14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks
is deprecated. Instead, use mapreduce.job.reduces
14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks.
speculative.execution is deprecated. Instead, use
mapreduce.reduce.speculative
14/04/23 16:56:17 WARN conf.HiveConf: DEPRECATED: Configuration
property hive.metastore.local no longer has any effect. Make sure to
provide a valid value for hive.metastore.uris if you are connecting to a
remote megastore.
So I try to check next step process (26-hdfs-NAMENODE-createdir) log.
There are some error messages.
+ echo 'Created directory /user/hive.'
+ /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
--config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir
dfs -chown hive:hive /user/hive
log4j:ERROR Could not find value for key log4j.appender.EventCounter
log4j:ERROR Could not instantiate appender named "EventCounter".
+ '[' 0 -eq 0 ']'
+ echo 'Changed ownership of /user/hive to hive:hive.'
+ /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
--config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir
dfs -chmod 1775 /user/hive
log4j:ERROR Could not find value for key log4j.appender.EventCounter
log4j:ERROR Could not instantiate appender named "EventCounter".
Darren Lo於 2014年4月24日星期四UTC+8上午1時59分25秒寫道：
To unsubscribe from this group and stop receiving emails from it,Dear Darren,
1.My Cloudera version is "Cloudera Express 5.0.0 (#215 built by
jenkins on 20140331-1424 git"
2.Yes, I used parcels for CDH binaries.
3.I used embedded PostgreSQL Database (default option). Sorry I can't
find the version info.
4.You are right too. First time when I ran the create tables command
seems no error. The log is as below
+ exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool
-dbType postgres -initSchema -dbOpts postgres.filter.81
14/04/23 16:56:10 INFO Configuration.deprecation:
mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.
fileinputformat.input.dir.recursive
14/04/23 16:56:10 INFO Configuration.deprecation:
mapred.max.split.size is deprecated. Instead, use mapreduce.input.
fileinputformat.split.maxsize
14/04/23 16:56:10 INFO Configuration.deprecation:
mapred.min.split.size is deprecated. Instead, use mapreduce.input.
fileinputformat.split.minsize
14/04/23 16:56:10 INFO Configuration.deprecation:
mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.
fileinputformat.split.minsize.per.rack
14/04/23 16:56:10 INFO Configuration.deprecation:
mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.
fileinputformat.split.minsize.per.node
14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks
is deprecated. Instead, use mapreduce.job.reduces
14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks.
speculative.execution is deprecated. Instead, use
mapreduce.reduce.speculative
14/04/23 16:56:17 WARN conf.HiveConf: DEPRECATED: Configuration
property hive.metastore.local no longer has any effect. Make sure to
provide a valid value for hive.metastore.uris if you are connecting to a
remote megastore.
So I try to check next step process (26-hdfs-NAMENODE-createdir) log.
There are some error messages.
+ echo 'Created directory /user/hive.'
+ /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
--config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir
dfs -chown hive:hive /user/hive
log4j:ERROR Could not find value for key log4j.appender.EventCounter
log4j:ERROR Could not instantiate appender named "EventCounter".
+ '[' 0 -eq 0 ']'
+ echo 'Changed ownership of /user/hive to hive:hive.'
+ /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
--config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir
dfs -chmod 1775 /user/hive
log4j:ERROR Could not find value for key log4j.appender.EventCounter
log4j:ERROR Could not instantiate appender named "EventCounter".
Darren Lo於 2014年4月24日星期四UTC+8上午1時59分25秒寫道：
Hi Ivan,
You're right that the schema should have been clean when you first
started.
Can you confirm your Cloudera Manager version? Click on Support ->
About in the upper right. You may need to exit the wizard to see this link
(exit the wizard by clicking on the Cloudera Manager logo).
Did you use parcels for CDH binaries? (this is the default choice)
What version of postgresql was installed? ssh in to the machine and
see what the package manager says the version of postgresql is.
I suspect that the first time you ran the create tables command, the
error log had a different message. SSH in to the machine where the Hive
Metastore Server is running, then look in /var/run/cloudera-scm-agent/
process and find the directories with a name like
"52-hive-metastore-create-tables". The "52" part will be different
on your system. Find the earliest attempt, and look in the logs directory
for the stderr log. Please share the end of that log.
Thanks,
Darren
send an email to [email protected].You're right that the schema should have been clean when you first
started.
Can you confirm your Cloudera Manager version? Click on Support ->
About in the upper right. You may need to exit the wizard to see this link
(exit the wizard by clicking on the Cloudera Manager logo).
Did you use parcels for CDH binaries? (this is the default choice)
What version of postgresql was installed? ssh in to the machine and
see what the package manager says the version of postgresql is.
I suspect that the first time you ran the create tables command, the
error log had a different message. SSH in to the machine where the Hive
Metastore Server is running, then look in /var/run/cloudera-scm-agent/
process and find the directories with a name like
"52-hive-metastore-create-tables". The "52" part will be different
on your system. Find the earliest attempt, and look in the logs directory
for the stderr log. Please share the end of that log.
Thanks,
Darren
On Wed, Apr 23, 2014 at 10:48 AM, Ivan Hsueh wrote:
Thanks Darren,
It's weird, I installed CDH 5 from clean system.
There shouldn't be an existing table.
Another problem is I can't use schematool to initialise the DB.
Would you please describe more specifically how to clean schema.
Thanks again..
Darren Lo於 2014年4月24日星期四UTC+8上午12時50分04秒寫道：
To unsubscribe from this group and stop receiving emails from it,Thanks Darren,
It's weird, I installed CDH 5 from clean system.
There shouldn't be an existing table.
Another problem is I can't use schematool to initialise the DB.
Would you please describe more specifically how to clean schema.
Thanks again..
Darren Lo於 2014年4月24日星期四UTC+8上午12時50分04秒寫道：
Hi Ivan,
You've configured Hive to talk to a schema that already has
tables, as indicated by this message:
Error: ERROR: relation "BUCKETING_COLS" already exists
(state=42P07,code=0)
Configure Hive with a clean schema and this problem should go away.
Thanks,
Darren
it, send an email to [email protected].You've configured Hive to talk to a schema that already has
tables, as indicated by this message:
Error: ERROR: relation "BUCKETING_COLS" already exists
(state=42P07,code=0)
Configure Hive with a clean schema and this problem should go away.
Thanks,
Darren
On Wed, Apr 23, 2014 at 7:31 AM, Ivan Hsueh wrote:
Hi,
I try to build a LAB for Cloudera 5.0
There are 2 hosts in the same domain
describe as below,
Cloudera51»Manager, Name Node
2GB RAM
50GB Space
Cloudera52»Data Node
2GB RAM
50GB Space
I followed the "Cloudera-Manager-Installation-Guide.pdf", and
used "Installation Path A - Automated Installation by Cloudera Manager "
first, I installed the cloudera manager 5 on Cloudera51
Second, Choosed the cluster hosts
Third, Manager automatically assigned Cloudera51 is Name Node,
and Cloudera52 is Data Node.
fourth, Using Manager Web UI to install CDH5 on both Cloudera51
and Cloudera52 (with Parcel)
I used embedded PostgreSQL Database
Everything is fine until Manager try to install Hive service.
It's always fail when creating megastore tables. (I had tried at
least 3 times)
Here is error log:
Failed to Create Hive Metastore Database Tables.
*Program:* hive/hive.sh ["create_metastore_tables"]
+ [[ create_metastore_tables == updatelocation ]]
+ [[ create_metastore_tables == create_metastore_tables ]]
+ [[ -z '' ]]
+ JDBC_JARS_CLASSPATH='/usr/share/cmf/lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar'
+ [[ create_metastore_tables == create_metastore_tables ]]
+ '[' 5 -ge 5 ']'
+ exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool -dbType postgres -initSchema -dbOpts postgres.filter.81
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/04/23 17:21:50 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
*** schemaTool failed ***
Can anyone help me to solved this problem?
Thanks a lot.
To unsubscribe from this group and stop receiving emails from it,
send an email to [email protected].
To unsubscribe from this group and stop receiving emails fromHi,
I try to build a LAB for Cloudera 5.0
There are 2 hosts in the same domain
describe as below,
Cloudera51»Manager, Name Node
2GB RAM
50GB Space
Cloudera52»Data Node
2GB RAM
50GB Space
I followed the "Cloudera-Manager-Installation-Guide.pdf", and
used "Installation Path A - Automated Installation by Cloudera Manager "
first, I installed the cloudera manager 5 on Cloudera51
Second, Choosed the cluster hosts
Third, Manager automatically assigned Cloudera51 is Name Node,
and Cloudera52 is Data Node.
fourth, Using Manager Web UI to install CDH5 on both Cloudera51
and Cloudera52 (with Parcel)
I used embedded PostgreSQL Database
Everything is fine until Manager try to install Hive service.
It's always fail when creating megastore tables. (I had tried at
least 3 times)
Here is error log:
Failed to Create Hive Metastore Database Tables.
*Program:* hive/hive.sh ["create_metastore_tables"]
+ [[ create_metastore_tables == updatelocation ]]
+ [[ create_metastore_tables == create_metastore_tables ]]
+ [[ -z '' ]]
+ JDBC_JARS_CLASSPATH='/usr/share/cmf/lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar'
+ [[ create_metastore_tables == create_metastore_tables ]]
+ '[' 5 -ge 5 ']'
+ exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool -dbType postgres -initSchema -dbOpts postgres.filter.81
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/04/23 17:21:50 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
*** schemaTool failed ***
Can anyone help me to solved this problem?
Thanks a lot.
To unsubscribe from this group and stop receiving emails from it,
send an email to [email protected].
