FAQ
Hi,

I try to build a LAB for Cloudera 5.0

There are 2 hosts in the same domain
describe as below,

Cloudera51»Manager, Name Node
2GB RAM
50GB Space

Cloudera52»Data Node
2GB RAM
50GB Space

I followed the "Cloudera-Manager-Installation-Guide.pdf", and used
"Installation Path A - Automated Installation by Cloudera Manager "

first, I installed the cloudera manager 5 on Cloudera51
Second, Choosed the cluster hosts
Third, Manager automatically assigned Cloudera51 is Name Node, and
Cloudera52 is Data Node.
fourth, Using Manager Web UI to install CDH5 on both Cloudera51 and
Cloudera52 (with Parcel)
            I used embedded PostgreSQL Database

Everything is fine until Manager try to install Hive service.
It's always fail when creating megastore tables. (I had tried at least 3
times)

Here is error log:

Failed to Create Hive Metastore Database Tables.

*Program:* hive/hive.sh ["create_metastore_tables"]

+ [[ create_metastore_tables == updatelocation ]]
+ [[ create_metastore_tables == create_metastore_tables ]]
+ [[ -z '' ]]
+ JDBC_JARS_CLASSPATH='/usr/share/cmf/lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar'
+ [[ create_metastore_tables == create_metastore_tables ]]
+ '[' 5 -ge 5 ']'
+ exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool -dbType postgres -initSchema -dbOpts postgres.filter.81
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/04/23 17:21:50 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
*** schemaTool failed ***


Can anyone help me to solved this problem?
Thanks a lot.

To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Search Discussions

  • Darren Lo at Apr 23, 2014 at 4:50 pm
    Hi Ivan,

    You've configured Hive to talk to a schema that already has tables, as
    indicated by this message:
    Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)

    Configure Hive with a clean schema and this problem should go away.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 7:31 AM, Ivan Hsueh wrote:

    Hi,

    I try to build a LAB for Cloudera 5.0

    There are 2 hosts in the same domain
    describe as below,

    Cloudera51»Manager, Name Node
    2GB RAM
    50GB Space

    Cloudera52»Data Node
    2GB RAM
    50GB Space

    I followed the "Cloudera-Manager-Installation-Guide.pdf", and used
    "Installation Path A - Automated Installation by Cloudera Manager "

    first, I installed the cloudera manager 5 on Cloudera51
    Second, Choosed the cluster hosts
    Third, Manager automatically assigned Cloudera51 is Name Node, and
    Cloudera52 is Data Node.
    fourth, Using Manager Web UI to install CDH5 on both Cloudera51 and
    Cloudera52 (with Parcel)
    I used embedded PostgreSQL Database

    Everything is fine until Manager try to install Hive service.
    It's always fail when creating megastore tables. (I had tried at least 3
    times)

    Here is error log:

    Failed to Create Hive Metastore Database Tables.

    *Program:* hive/hive.sh ["create_metastore_tables"]

    + [[ create_metastore_tables == updatelocation ]]
    + [[ create_metastore_tables == create_metastore_tables ]]
    + [[ -z '' ]]
    + JDBC_JARS_CLASSPATH='/usr/share/cmf/lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar'
    + [[ create_metastore_tables == create_metastore_tables ]]
    + '[' 5 -ge 5 ']'
    + exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
    14/04/23 17:21:50 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
    Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)
    org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
    *** schemaTool failed ***


    Can anyone help me to solved this problem?
    Thanks a lot.

    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Ivan Hsueh at Apr 23, 2014 at 5:48 pm
    Thanks Darren,

    It's weird, I installed CDH 5 from clean system.
    There shouldn't be an existing table.

    Another problem is I can't use schematool to initialise the DB.
    Would you please describe more specifically how to clean schema.

    Thanks again..


    Darren Lo於 2014年4月24日星期四UTC+8上午12時50分04秒寫道:
    Hi Ivan,

    You've configured Hive to talk to a schema that already has tables, as
    indicated by this message:
    Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)

    Configure Hive with a clean schema and this problem should go away.

    Thanks,
    Darren


    On Wed, Apr 23, 2014 at 7:31 AM, Ivan Hsueh <ivan....@gmail.com<javascript:>
    wrote:
    Hi,

    I try to build a LAB for Cloudera 5.0

    There are 2 hosts in the same domain
    describe as below,

    Cloudera51»Manager, Name Node
    2GB RAM
    50GB Space

    Cloudera52»Data Node
    2GB RAM
    50GB Space

    I followed the "Cloudera-Manager-Installation-Guide.pdf", and used
    "Installation Path A - Automated Installation by Cloudera Manager "

    first, I installed the cloudera manager 5 on Cloudera51
    Second, Choosed the cluster hosts
    Third, Manager automatically assigned Cloudera51 is Name Node, and
    Cloudera52 is Data Node.
    fourth, Using Manager Web UI to install CDH5 on both Cloudera51 and
    Cloudera52 (with Parcel)
    I used embedded PostgreSQL Database

    Everything is fine until Manager try to install Hive service.
    It's always fail when creating megastore tables. (I had tried at least 3
    times)

    Here is error log:

    Failed to Create Hive Metastore Database Tables.

    *Program:* hive/hive.sh ["create_metastore_tables"]

    + [[ create_metastore_tables == updatelocation ]]
    + [[ create_metastore_tables == create_metastore_tables ]]
    + [[ -z '' ]]
    + JDBC_JARS_CLASSPATH='/usr/share/cmf/lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar'
    + [[ create_metastore_tables == create_metastore_tables ]]
    + '[' 5 -ge 5 ']'
    + exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
    14/04/23 17:21:50 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
    Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)
    org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
    *** schemaTool failed ***


    Can anyone help me to solved this problem?
    Thanks a lot.

    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Darren Lo at Apr 23, 2014 at 5:59 pm
    Hi Ivan,

    You're right that the schema should have been clean when you first started.

    Can you confirm your Cloudera Manager version? Click on Support -> About in
    the upper right. You may need to exit the wizard to see this link (exit the
    wizard by clicking on the Cloudera Manager logo).

    Did you use parcels for CDH binaries? (this is the default choice)

    What version of postgresql was installed? ssh in to the machine and see
    what the package manager says the version of postgresql is.

    I suspect that the first time you ran the create tables command, the error
    log had a different message. SSH in to the machine where the Hive Metastore
    Server is running, then look in /var/run/cloudera-scm-agent/process and
    find the directories with a name like "52-hive-metastore-create-tables".
    The "52" part will be different on your system. Find the earliest attempt,
    and look in the logs directory for the stderr log. Please share the end of
    that log.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 10:48 AM, Ivan Hsueh wrote:

    Thanks Darren,

    It's weird, I installed CDH 5 from clean system.
    There shouldn't be an existing table.

    Another problem is I can't use schematool to initialise the DB.
    Would you please describe more specifically how to clean schema.

    Thanks again..


    Darren Lo於 2014年4月24日星期四UTC+8上午12時50分04秒寫道:
    Hi Ivan,

    You've configured Hive to talk to a schema that already has tables, as
    indicated by this message:
    Error: ERROR: relation "BUCKETING_COLS" already exists
    (state=42P07,code=0)

    Configure Hive with a clean schema and this problem should go away.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 7:31 AM, Ivan Hsueh wrote:

    Hi,

    I try to build a LAB for Cloudera 5.0

    There are 2 hosts in the same domain
    describe as below,

    Cloudera51>>Manager, Name Node
    2GB RAM
    50GB Space

    Cloudera52>>Data Node
    2GB RAM
    50GB Space

    I followed the "Cloudera-Manager-Installation-Guide.pdf", and used
    "Installation Path A - Automated Installation by Cloudera Manager "

    first, I installed the cloudera manager 5 on Cloudera51
    Second, Choosed the cluster hosts
    Third, Manager automatically assigned Cloudera51 is Name Node, and
    Cloudera52 is Data Node.
    fourth, Using Manager Web UI to install CDH5 on both Cloudera51 and
    Cloudera52 (with Parcel)
    I used embedded PostgreSQL Database

    Everything is fine until Manager try to install Hive service.
    It's always fail when creating megastore tables. (I had tried at least 3
    times)

    Here is error log:

    Failed to Create Hive Metastore Database Tables.

    *Program:* hive/hive.sh ["create_metastore_tables"]

    + [[ create_metastore_tables == updatelocation ]]
    + [[ create_metastore_tables == create_metastore_tables ]]
    + [[ -z '' ]]
    + JDBC_JARS_CLASSPATH='/usr/share/cmf/lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar'
    + [[ create_metastore_tables == create_metastore_tables ]]
    + '[' 5 -ge 5 ']'
    + exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
    14/04/23 17:21:50 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
    Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)
    org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
    *** schemaTool failed ***


    Can anyone help me to solved this problem?
    Thanks a lot.

    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Ivan Hsueh at Apr 23, 2014 at 6:50 pm
    Dear Darren,

    1.My Cloudera version is "Cloudera Express 5.0.0 (#215 built by jenkins on
    20140331-1424 git"

    2.Yes, I used parcels for CDH binaries.

    3.I used embedded PostgreSQL Database (default option). Sorry I can't find
    the version info.

    4.You are right too. First time when I ran the create tables command seems
    no error. The log is as below

    + exec
    /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool
    -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.input.dir.recursive is deprecated. Instead, use
    mapreduce.input.fileinputformat.input.dir.recursive
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.max.split.size is
    deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.min.split.size is
    deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size.per.rack is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize.per.rack
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size.per.node is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize.per.node
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks is
    deprecated. Instead, use mapreduce.job.reduces
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
    mapreduce.reduce.speculative
    14/04/23 16:56:17 WARN conf.HiveConf: DEPRECATED: Configuration property
    hive.metastore.local no longer has any effect. Make sure to provide a valid
    value for hive.metastore.uris if you are connecting to a remote megastore.

    So I try to check next step process (26-hdfs-NAMENODE-createdir) log.
    There are some error messages.

    + echo 'Created directory /user/hive.'
    + /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
    --config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir dfs
    -chown hive:hive /user/hive
    log4j:ERROR Could not find value for key log4j.appender.EventCounter
    log4j:ERROR Could not instantiate appender named "EventCounter".
    + '[' 0 -eq 0 ']'
    + echo 'Changed ownership of /user/hive to hive:hive.'
    + /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
    --config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir dfs
    -chmod 1775 /user/hive
    log4j:ERROR Could not find value for key log4j.appender.EventCounter
    log4j:ERROR Could not instantiate appender named "EventCounter".




    Darren Lo於 2014年4月24日星期四UTC+8上午1時59分25秒寫道:
    Hi Ivan,

    You're right that the schema should have been clean when you first started.

    Can you confirm your Cloudera Manager version? Click on Support -> About
    in the upper right. You may need to exit the wizard to see this link (exit
    the wizard by clicking on the Cloudera Manager logo).

    Did you use parcels for CDH binaries? (this is the default choice)

    What version of postgresql was installed? ssh in to the machine and see
    what the package manager says the version of postgresql is.

    I suspect that the first time you ran the create tables command, the error
    log had a different message. SSH in to the machine where the Hive Metastore
    Server is running, then look in /var/run/cloudera-scm-agent/process and
    find the directories with a name like "52-hive-metastore-create-tables".
    The "52" part will be different on your system. Find the earliest attempt,
    and look in the logs directory for the stderr log. Please share the end of
    that log.

    Thanks,
    Darren


    On Wed, Apr 23, 2014 at 10:48 AM, Ivan Hsueh <ivan....@gmail.com<javascript:>
    wrote:
    Thanks Darren,

    It's weird, I installed CDH 5 from clean system.
    There shouldn't be an existing table.

    Another problem is I can't use schematool to initialise the DB.
    Would you please describe more specifically how to clean schema.

    Thanks again..


    Darren Lo於 2014年4月24日星期四UTC+8上午12時50分04秒寫道:
    Hi Ivan,

    You've configured Hive to talk to a schema that already has tables, as
    indicated by this message:
    Error: ERROR: relation "BUCKETING_COLS" already exists
    (state=42P07,code=0)

    Configure Hive with a clean schema and this problem should go away.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 7:31 AM, Ivan Hsueh wrote:

    Hi,

    I try to build a LAB for Cloudera 5.0

    There are 2 hosts in the same domain
    describe as below,

    Cloudera51»Manager, Name Node
    2GB RAM
    50GB Space

    Cloudera52»Data Node
    2GB RAM
    50GB Space

    I followed the "Cloudera-Manager-Installation-Guide.pdf", and used
    "Installation Path A - Automated Installation by Cloudera Manager "

    first, I installed the cloudera manager 5 on Cloudera51
    Second, Choosed the cluster hosts
    Third, Manager automatically assigned Cloudera51 is Name Node, and
    Cloudera52 is Data Node.
    fourth, Using Manager Web UI to install CDH5 on both Cloudera51 and
    Cloudera52 (with Parcel)
    I used embedded PostgreSQL Database

    Everything is fine until Manager try to install Hive service.
    It's always fail when creating megastore tables. (I had tried at least
    3 times)

    Here is error log:

    Failed to Create Hive Metastore Database Tables.

    *Program:* hive/hive.sh ["create_metastore_tables"]

    + [[ create_metastore_tables == updatelocation ]]
    + [[ create_metastore_tables == create_metastore_tables ]]
    + [[ -z '' ]]
    + JDBC_JARS_CLASSPATH='/usr/share/cmf/lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar'
    + [[ create_metastore_tables == create_metastore_tables ]]
    + '[' 5 -ge 5 ']'
    + exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
    14/04/23 17:21:50 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
    Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)
    org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
    *** schemaTool failed ***


    Can anyone help me to solved this problem?
    Thanks a lot.

    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Darren Lo at Apr 23, 2014 at 7:23 pm
    Hi Ivan,

    It looks like create table actually succeeded the first time. What happens
    if you try to start the metastore (via the CM UI)?

    Also, what operating system are you using?

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 11:50 AM, Ivan Hsueh wrote:

    Dear Darren,

    1.My Cloudera version is "Cloudera Express 5.0.0 (#215 built by jenkins on
    20140331-1424 git"

    2.Yes, I used parcels for CDH binaries.

    3.I used embedded PostgreSQL Database (default option). Sorry I can't find
    the version info.

    4.You are right too. First time when I ran the create tables command seems
    no error. The log is as below

    + exec
    /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool
    -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.input.dir.recursive is deprecated. Instead, use
    mapreduce.input.fileinputformat.input.dir.recursive
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.max.split.size is
    deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.min.split.size is
    deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size.per.rack is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize.per.rack
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size.per.node is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize.per.node
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks is
    deprecated. Instead, use mapreduce.job.reduces
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
    mapreduce.reduce.speculative
    14/04/23 16:56:17 WARN conf.HiveConf: DEPRECATED: Configuration property
    hive.metastore.local no longer has any effect. Make sure to provide a valid
    value for hive.metastore.uris if you are connecting to a remote megastore.

    So I try to check next step process (26-hdfs-NAMENODE-createdir) log.
    There are some error messages.

    + echo 'Created directory /user/hive.'
    +
    /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
    --config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir dfs
    -chown hive:hive /user/hive
    log4j:ERROR Could not find value for key log4j.appender.EventCounter
    log4j:ERROR Could not instantiate appender named "EventCounter".
    + '[' 0 -eq 0 ']'
    + echo 'Changed ownership of /user/hive to hive:hive.'
    +
    /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
    --config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir dfs
    -chmod 1775 /user/hive
    log4j:ERROR Could not find value for key log4j.appender.EventCounter
    log4j:ERROR Could not instantiate appender named "EventCounter".




    Darren Lo於 2014年4月24日星期四UTC+8上午1時59分25秒寫道:
    Hi Ivan,

    You're right that the schema should have been clean when you first
    started.

    Can you confirm your Cloudera Manager version? Click on Support -> About
    in the upper right. You may need to exit the wizard to see this link (exit
    the wizard by clicking on the Cloudera Manager logo).

    Did you use parcels for CDH binaries? (this is the default choice)

    What version of postgresql was installed? ssh in to the machine and see
    what the package manager says the version of postgresql is.

    I suspect that the first time you ran the create tables command, the
    error log had a different message. SSH in to the machine where the Hive
    Metastore Server is running, then look in /var/run/cloudera-scm-agent/process
    and find the directories with a name like "52-hive-metastore-create-tables".
    The "52" part will be different on your system. Find the earliest attempt,
    and look in the logs directory for the stderr log. Please share the end of
    that log.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 10:48 AM, Ivan Hsueh wrote:

    Thanks Darren,

    It's weird, I installed CDH 5 from clean system.
    There shouldn't be an existing table.

    Another problem is I can't use schematool to initialise the DB.
    Would you please describe more specifically how to clean schema.

    Thanks again..


    Darren Lo於 2014年4月24日星期四UTC+8上午12時50分04秒寫道:
    Hi Ivan,

    You've configured Hive to talk to a schema that already has tables, as
    indicated by this message:
    Error: ERROR: relation "BUCKETING_COLS" already exists
    (state=42P07,code=0)

    Configure Hive with a clean schema and this problem should go away.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 7:31 AM, Ivan Hsueh wrote:

    Hi,

    I try to build a LAB for Cloudera 5.0

    There are 2 hosts in the same domain
    describe as below,

    Cloudera51»Manager, Name Node
    2GB RAM
    50GB Space

    Cloudera52»Data Node
    2GB RAM
    50GB Space

    I followed the "Cloudera-Manager-Installation-Guide.pdf", and used
    "Installation Path A - Automated Installation by Cloudera Manager "

    first, I installed the cloudera manager 5 on Cloudera51
    Second, Choosed the cluster hosts
    Third, Manager automatically assigned Cloudera51 is Name Node, and
    Cloudera52 is Data Node.
    fourth, Using Manager Web UI to install CDH5 on both Cloudera51 and
    Cloudera52 (with Parcel)
    I used embedded PostgreSQL Database

    Everything is fine until Manager try to install Hive service.
    It's always fail when creating megastore tables. (I had tried at least
    3 times)

    Here is error log:

    Failed to Create Hive Metastore Database Tables.

    *Program:* hive/hive.sh ["create_metastore_tables"]

    + [[ create_metastore_tables == updatelocation ]]
    + [[ create_metastore_tables == create_metastore_tables ]]
    + [[ -z '' ]]
    + JDBC_JARS_CLASSPATH='/usr/share/cmf/lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar'
    + [[ create_metastore_tables == create_metastore_tables ]]
    + '[' 5 -ge 5 ']'
    + exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
    14/04/23 17:21:50 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
    Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)
    org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
    *** schemaTool failed ***


    Can anyone help me to solved this problem?
    Thanks a lot.

    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Ivan Hsueh at Apr 23, 2014 at 7:41 pm
    Dear Darren,

    It's seems work. I can start the metastore via UI.

    Thu Apr 24 03:26:29 CST 2014
    JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
    using /usr/java/jdk1.7.0_45-cloudera as JAVA_HOME
    using 5 as CDH_VERSION
    using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive as HIVE_HOME
    using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE as HIVE_CONF_DIR
    using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop as HADOOP_HOME
    using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE/yarn-conf as HADOOP_CONF_DIR
    Starting Hive Metastore Server


    Both OS are CentOS 6.4.

    If it does really work, can I just pass this process and keep running the
    rest of process via CM UI?

    <https://lh3.googleusercontent.com/-7Mkz3nBNeJs/U1gWaWKBePI/AAAAAAAAAhI/YgDpz6I1F-8/s1600/Screen+Shot+2014-04-24+at+3.31.40+AM.png>

    Many thanks.


    Darren Lo於 2014年4月24日星期四UTC+8上午3時23分24秒寫道:
    Hi Ivan,

    It looks like create table actually succeeded the first time. What happens
    if you try to start the metastore (via the CM UI)?

    Also, what operating system are you using?

    Thanks,
    Darren


    On Wed, Apr 23, 2014 at 11:50 AM, Ivan Hsueh <ivan....@gmail.com<javascript:>
    wrote:
    Dear Darren,

    1.My Cloudera version is "Cloudera Express 5.0.0 (#215 built by jenkins
    on 20140331-1424 git"

    2.Yes, I used parcels for CDH binaries.

    3.I used embedded PostgreSQL Database (default option). Sorry I can't
    find the version info.

    4.You are right too. First time when I ran the create tables command
    seems no error. The log is as below

    + exec
    /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool
    -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.input.dir.recursive is deprecated. Instead, use
    mapreduce.input.fileinputformat.input.dir.recursive
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.max.split.size
    is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.min.split.size
    is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size.per.rack is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize.per.rack
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size.per.node is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize.per.node
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks is
    deprecated. Instead, use mapreduce.job.reduces
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
    mapreduce.reduce.speculative
    14/04/23 16:56:17 WARN conf.HiveConf: DEPRECATED: Configuration property
    hive.metastore.local no longer has any effect. Make sure to provide a valid
    value for hive.metastore.uris if you are connecting to a remote megastore.

    So I try to check next step process (26-hdfs-NAMENODE-createdir) log.
    There are some error messages.

    + echo 'Created directory /user/hive.'
    +
    /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
    --config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir dfs
    -chown hive:hive /user/hive
    log4j:ERROR Could not find value for key log4j.appender.EventCounter
    log4j:ERROR Could not instantiate appender named "EventCounter".
    + '[' 0 -eq 0 ']'
    + echo 'Changed ownership of /user/hive to hive:hive.'
    +
    /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
    --config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir dfs
    -chmod 1775 /user/hive
    log4j:ERROR Could not find value for key log4j.appender.EventCounter
    log4j:ERROR Could not instantiate appender named "EventCounter".




    Darren Lo於 2014年4月24日星期四UTC+8上午1時59分25秒寫道:
    Hi Ivan,

    You're right that the schema should have been clean when you first
    started.

    Can you confirm your Cloudera Manager version? Click on Support -> About
    in the upper right. You may need to exit the wizard to see this link (exit
    the wizard by clicking on the Cloudera Manager logo).

    Did you use parcels for CDH binaries? (this is the default choice)

    What version of postgresql was installed? ssh in to the machine and see
    what the package manager says the version of postgresql is.

    I suspect that the first time you ran the create tables command, the
    error log had a different message. SSH in to the machine where the Hive
    Metastore Server is running, then look in /var/run/cloudera-scm-agent/process
    and find the directories with a name like "52-hive-metastore-create-tables".
    The "52" part will be different on your system. Find the earliest attempt,
    and look in the logs directory for the stderr log. Please share the end of
    that log.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 10:48 AM, Ivan Hsueh wrote:

    Thanks Darren,

    It's weird, I installed CDH 5 from clean system.
    There shouldn't be an existing table.

    Another problem is I can't use schematool to initialise the DB.
    Would you please describe more specifically how to clean schema.

    Thanks again..


    Darren Lo於 2014年4月24日星期四UTC+8上午12時50分04秒寫道:
    Hi Ivan,

    You've configured Hive to talk to a schema that already has tables, as
    indicated by this message:
    Error: ERROR: relation "BUCKETING_COLS" already exists
    (state=42P07,code=0)

    Configure Hive with a clean schema and this problem should go away.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 7:31 AM, Ivan Hsueh wrote:

    Hi,

    I try to build a LAB for Cloudera 5.0

    There are 2 hosts in the same domain
    describe as below,

    Cloudera51»Manager, Name Node
    2GB RAM
    50GB Space

    Cloudera52»Data Node
    2GB RAM
    50GB Space

    I followed the "Cloudera-Manager-Installation-Guide.pdf", and used
    "Installation Path A - Automated Installation by Cloudera Manager "

    first, I installed the cloudera manager 5 on Cloudera51
    Second, Choosed the cluster hosts
    Third, Manager automatically assigned Cloudera51 is Name Node, and
    Cloudera52 is Data Node.
    fourth, Using Manager Web UI to install CDH5 on both Cloudera51 and
    Cloudera52 (with Parcel)
    I used embedded PostgreSQL Database

    Everything is fine until Manager try to install Hive service.
    It's always fail when creating megastore tables. (I had tried at
    least 3 times)

    Here is error log:

    Failed to Create Hive Metastore Database Tables.

    *Program:* hive/hive.sh ["create_metastore_tables"]

    + [[ create_metastore_tables == updatelocation ]]
    + [[ create_metastore_tables == create_metastore_tables ]]
    + [[ -z '' ]]
    + JDBC_JARS_CLASSPATH='/usr/share/cmf/lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar'
    + [[ create_metastore_tables == create_metastore_tables ]]
    + '[' 5 -ge 5 ']'
    + exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
    14/04/23 17:21:50 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
    Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)
    org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
    *** schemaTool failed ***


    Can anyone help me to solved this problem?
    Thanks a lot.

    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Darren Lo at Apr 23, 2014 at 7:47 pm
    You should run the Create Hive User Directory and Create Hive Warehouse
    Directory commands (available if you click on Hive and look in the actions
    menu on the right), then the Start command for hive.

    Then you should be able to re-try the first run command (find it in your
    command history at http://<cm_host>:7180/cmf/commands/commands, click on
    your first run command, and look for a retry button), or you can run each
    of the listed commands manually, one by one.

    On Wed, Apr 23, 2014 at 12:41 PM, Ivan Hsueh wrote:

    Dear Darren,

    It's seems work. I can start the metastore via UI.

    Thu Apr 24 03:26:29 CST 2014
    JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
    using /usr/java/jdk1.7.0_45-cloudera as JAVA_HOME
    using 5 as CDH_VERSION
    using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive as HIVE_HOME
    using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE as HIVE_CONF_DIR
    using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop as HADOOP_HOME
    using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE/yarn-conf as HADOOP_CONF_DIR
    Starting Hive Metastore Server


    Both OS are CentOS 6.4.

    If it does really work, can I just pass this process and keep running the
    rest of process via CM UI?


    <https://lh3.googleusercontent.com/-7Mkz3nBNeJs/U1gWaWKBePI/AAAAAAAAAhI/YgDpz6I1F-8/s1600/Screen+Shot+2014-04-24+at+3.31.40+AM.png>

    Many thanks.


    Darren Lo於 2014年4月24日星期四UTC+8上午3時23分24秒寫道:
    Hi Ivan,

    It looks like create table actually succeeded the first time. What
    happens if you try to start the metastore (via the CM UI)?

    Also, what operating system are you using?

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 11:50 AM, Ivan Hsueh wrote:

    Dear Darren,

    1.My Cloudera version is "Cloudera Express 5.0.0 (#215 built by jenkins
    on 20140331-1424 git"

    2.Yes, I used parcels for CDH binaries.

    3.I used embedded PostgreSQL Database (default option). Sorry I can't
    find the version info.

    4.You are right too. First time when I ran the create tables command
    seems no error. The log is as below

    + exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool
    -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.
    fileinputformat.input.dir.recursive
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.max.split.size
    is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.maxsize
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.min.split.size
    is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.minsize
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.minsize.per.rack
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.minsize.per.node
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks is
    deprecated. Instead, use mapreduce.job.reduces
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution
    is deprecated. Instead, use mapreduce.reduce.speculative
    14/04/23 16:56:17 WARN conf.HiveConf: DEPRECATED: Configuration property
    hive.metastore.local no longer has any effect. Make sure to provide a valid
    value for hive.metastore.uris if you are connecting to a remote megastore.

    So I try to check next step process (26-hdfs-NAMENODE-createdir) log.
    There are some error messages.

    + echo 'Created directory /user/hive.'
    + /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
    --config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir
    dfs -chown hive:hive /user/hive
    log4j:ERROR Could not find value for key log4j.appender.EventCounter
    log4j:ERROR Could not instantiate appender named "EventCounter".
    + '[' 0 -eq 0 ']'
    + echo 'Changed ownership of /user/hive to hive:hive.'
    + /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
    --config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir
    dfs -chmod 1775 /user/hive
    log4j:ERROR Could not find value for key log4j.appender.EventCounter
    log4j:ERROR Could not instantiate appender named "EventCounter".




    Darren Lo於 2014年4月24日星期四UTC+8上午1時59分25秒寫道:
    Hi Ivan,

    You're right that the schema should have been clean when you first
    started.

    Can you confirm your Cloudera Manager version? Click on Support ->
    About in the upper right. You may need to exit the wizard to see this link
    (exit the wizard by clicking on the Cloudera Manager logo).

    Did you use parcels for CDH binaries? (this is the default choice)

    What version of postgresql was installed? ssh in to the machine and see
    what the package manager says the version of postgresql is.

    I suspect that the first time you ran the create tables command, the
    error log had a different message. SSH in to the machine where the Hive
    Metastore Server is running, then look in /var/run/cloudera-scm-agent/
    process and find the directories with a name like
    "52-hive-metastore-create-tables". The "52" part will be different on
    your system. Find the earliest attempt, and look in the logs directory for
    the stderr log. Please share the end of that log.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 10:48 AM, Ivan Hsueh wrote:

    Thanks Darren,

    It's weird, I installed CDH 5 from clean system.
    There shouldn't be an existing table.

    Another problem is I can't use schematool to initialise the DB.
    Would you please describe more specifically how to clean schema.

    Thanks again..


    Darren Lo於 2014年4月24日星期四UTC+8上午12時50分04秒寫道:
    Hi Ivan,

    You've configured Hive to talk to a schema that already has tables,
    as indicated by this message:
    Error: ERROR: relation "BUCKETING_COLS" already exists
    (state=42P07,code=0)

    Configure Hive with a clean schema and this problem should go away.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 7:31 AM, Ivan Hsueh wrote:

    Hi,

    I try to build a LAB for Cloudera 5.0

    There are 2 hosts in the same domain
    describe as below,

    Cloudera51>>Manager, Name Node
    2GB RAM
    50GB Space

    Cloudera52>>Data Node
    2GB RAM
    50GB Space

    I followed the "Cloudera-Manager-Installation-Guide.pdf", and used
    "Installation Path A - Automated Installation by Cloudera Manager "

    first, I installed the cloudera manager 5 on Cloudera51
    Second, Choosed the cluster hosts
    Third, Manager automatically assigned Cloudera51 is Name Node, and
    Cloudera52 is Data Node.
    fourth, Using Manager Web UI to install CDH5 on both Cloudera51 and
    Cloudera52 (with Parcel)
    I used embedded PostgreSQL Database

    Everything is fine until Manager try to install Hive service.
    It's always fail when creating megastore tables. (I had tried at
    least 3 times)

    Here is error log:

    Failed to Create Hive Metastore Database Tables.

    *Program:* hive/hive.sh ["create_metastore_tables"]

    + [[ create_metastore_tables == updatelocation ]]
    + [[ create_metastore_tables == create_metastore_tables ]]
    + [[ -z '' ]]
    + JDBC_JARS_CLASSPATH='/usr/share/cmf/lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar'
    + [[ create_metastore_tables == create_metastore_tables ]]
    + '[' 5 -ge 5 ']'
    + exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
    14/04/23 17:21:50 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
    Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)
    org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
    *** schemaTool failed ***


    Can anyone help me to solved this problem?
    Thanks a lot.

    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Ivan Hsueh at Apr 23, 2014 at 8:29 pm
    Dear Darren,

    Finally, all services are start.
    Thank you for helping me!


    Darren Lo於 2014年4月24日星期四UTC+8上午3時47分36秒寫道:
    You should run the Create Hive User Directory and Create Hive Warehouse
    Directory commands (available if you click on Hive and look in the actions
    menu on the right), then the Start command for hive.

    Then you should be able to re-try the first run command (find it in your
    command history at http://<cm_host>:7180/cmf/commands/commands, click on
    your first run command, and look for a retry button), or you can run each
    of the listed commands manually, one by one.


    On Wed, Apr 23, 2014 at 12:41 PM, Ivan Hsueh <ivan....@gmail.com<javascript:>
    wrote:
    Dear Darren,

    It's seems work. I can start the metastore via UI.

    Thu Apr 24 03:26:29 CST 2014
    JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
    using /usr/java/jdk1.7.0_45-cloudera as JAVA_HOME
    using 5 as CDH_VERSION
    using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive as HIVE_HOME
    using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE as HIVE_CONF_DIR
    using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop as HADOOP_HOME
    using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE/yarn-conf as HADOOP_CONF_DIR
    Starting Hive Metastore Server


    Both OS are CentOS 6.4.

    If it does really work, can I just pass this process and keep running the
    rest of process via CM UI?


    <https://lh3.googleusercontent.com/-7Mkz3nBNeJs/U1gWaWKBePI/AAAAAAAAAhI/YgDpz6I1F-8/s1600/Screen+Shot+2014-04-24+at+3.31.40+AM.png>

    Many thanks.


    Darren Lo於 2014年4月24日星期四UTC+8上午3時23分24秒寫道:
    Hi Ivan,

    It looks like create table actually succeeded the first time. What
    happens if you try to start the metastore (via the CM UI)?

    Also, what operating system are you using?

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 11:50 AM, Ivan Hsueh wrote:

    Dear Darren,

    1.My Cloudera version is "Cloudera Express 5.0.0 (#215 built by jenkins
    on 20140331-1424 git"

    2.Yes, I used parcels for CDH binaries.

    3.I used embedded PostgreSQL Database (default option). Sorry I can't
    find the version info.

    4.You are right too. First time when I ran the create tables command
    seems no error. The log is as below

    + exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool
    -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.
    fileinputformat.input.dir.recursive
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.max.split.size
    is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.maxsize
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.min.split.size
    is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.minsize
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.minsize.per.rack
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.minsize.per.node
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks
    is deprecated. Instead, use mapreduce.job.reduces
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution
    is deprecated. Instead, use mapreduce.reduce.speculative
    14/04/23 16:56:17 WARN conf.HiveConf: DEPRECATED: Configuration
    property hive.metastore.local no longer has any effect. Make sure to
    provide a valid value for hive.metastore.uris if you are connecting to a
    remote megastore.

    So I try to check next step process (26-hdfs-NAMENODE-createdir) log.
    There are some error messages.

    + echo 'Created directory /user/hive.'
    + /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
    --config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir
    dfs -chown hive:hive /user/hive
    log4j:ERROR Could not find value for key log4j.appender.EventCounter
    log4j:ERROR Could not instantiate appender named "EventCounter".
    + '[' 0 -eq 0 ']'
    + echo 'Changed ownership of /user/hive to hive:hive.'
    + /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
    --config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir
    dfs -chmod 1775 /user/hive
    log4j:ERROR Could not find value for key log4j.appender.EventCounter
    log4j:ERROR Could not instantiate appender named "EventCounter".




    Darren Lo於 2014年4月24日星期四UTC+8上午1時59分25秒寫道:
    Hi Ivan,

    You're right that the schema should have been clean when you first
    started.

    Can you confirm your Cloudera Manager version? Click on Support ->
    About in the upper right. You may need to exit the wizard to see this link
    (exit the wizard by clicking on the Cloudera Manager logo).

    Did you use parcels for CDH binaries? (this is the default choice)

    What version of postgresql was installed? ssh in to the machine and
    see what the package manager says the version of postgresql is.

    I suspect that the first time you ran the create tables command, the
    error log had a different message. SSH in to the machine where the Hive
    Metastore Server is running, then look in /var/run/cloudera-scm-agent/
    process and find the directories with a name like
    "52-hive-metastore-create-tables". The "52" part will be different on
    your system. Find the earliest attempt, and look in the logs directory for
    the stderr log. Please share the end of that log.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 10:48 AM, Ivan Hsueh wrote:

    Thanks Darren,

    It's weird, I installed CDH 5 from clean system.
    There shouldn't be an existing table.

    Another problem is I can't use schematool to initialise the DB.
    Would you please describe more specifically how to clean schema.

    Thanks again..


    Darren Lo於 2014年4月24日星期四UTC+8上午12時50分04秒寫道:
    Hi Ivan,

    You've configured Hive to talk to a schema that already has tables,
    as indicated by this message:
    Error: ERROR: relation "BUCKETING_COLS" already exists
    (state=42P07,code=0)

    Configure Hive with a clean schema and this problem should go away.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 7:31 AM, Ivan Hsueh wrote:

    Hi,

    I try to build a LAB for Cloudera 5.0

    There are 2 hosts in the same domain
    describe as below,

    Cloudera51»Manager, Name Node
    2GB RAM
    50GB Space

    Cloudera52»Data Node
    2GB RAM
    50GB Space

    I followed the "Cloudera-Manager-Installation-Guide.pdf", and used
    "Installation Path A - Automated Installation by Cloudera Manager "

    first, I installed the cloudera manager 5 on Cloudera51
    Second, Choosed the cluster hosts
    Third, Manager automatically assigned Cloudera51 is Name Node, and
    Cloudera52 is Data Node.
    fourth, Using Manager Web UI to install CDH5 on both Cloudera51 and
    Cloudera52 (with Parcel)
    I used embedded PostgreSQL Database

    Everything is fine until Manager try to install Hive service.
    It's always fail when creating megastore tables. (I had tried at
    least 3 times)

    Here is error log:

    Failed to Create Hive Metastore Database Tables.

    *Program:* hive/hive.sh ["create_metastore_tables"]

    + [[ create_metastore_tables == updatelocation ]]
    + [[ create_metastore_tables == create_metastore_tables ]]
    + [[ -z '' ]]
    + JDBC_JARS_CLASSPATH='/usr/share/cmf/lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar'
    + [[ create_metastore_tables == create_metastore_tables ]]
    + '[' 5 -ge 5 ']'
    + exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
    14/04/23 17:21:50 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
    Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)
    org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
    *** schemaTool failed ***


    Can anyone help me to solved this problem?
    Thanks a lot.

    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Kevin Huang at May 16, 2014 at 8:40 am
    Ivan,

    How did you resolve it finally?
    Creating Hive Metastore Database Tables
    Where I can launch Hive Metastore database tables in UI?

    Thanks a lot!
    On Wednesday, April 23, 2014 1:29:00 PM UTC-7, Ivan Hsueh wrote:

    Dear Darren,

    Finally, all services are start.
    Thank you for helping me!


    Darren Lo於 2014年4月24日星期四UTC+8上午3時47分36秒寫道:
    You should run the Create Hive User Directory and Create Hive Warehouse
    Directory commands (available if you click on Hive and look in the actions
    menu on the right), then the Start command for hive.

    Then you should be able to re-try the first run command (find it in your
    command history at http://<cm_host>:7180/cmf/commands/commands, click on
    your first run command, and look for a retry button), or you can run each
    of the listed commands manually, one by one.

    On Wed, Apr 23, 2014 at 12:41 PM, Ivan Hsueh wrote:

    Dear Darren,

    It's seems work. I can start the metastore via UI.

    Thu Apr 24 03:26:29 CST 2014
    JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
    using /usr/java/jdk1.7.0_45-cloudera as JAVA_HOME
    using 5 as CDH_VERSION
    using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive as HIVE_HOME
    using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE as HIVE_CONF_DIR
    using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop as HADOOP_HOME
    using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE/yarn-conf as HADOOP_CONF_DIR
    Starting Hive Metastore Server


    Both OS are CentOS 6.4.

    If it does really work, can I just pass this process and keep running
    the rest of process via CM UI?


    <https://lh3.googleusercontent.com/-7Mkz3nBNeJs/U1gWaWKBePI/AAAAAAAAAhI/YgDpz6I1F-8/s1600/Screen+Shot+2014-04-24+at+3.31.40+AM.png>

    Many thanks.


    Darren Lo於 2014年4月24日星期四UTC+8上午3時23分24秒寫道:
    Hi Ivan,

    It looks like create table actually succeeded the first time. What
    happens if you try to start the metastore (via the CM UI)?

    Also, what operating system are you using?

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 11:50 AM, Ivan Hsueh wrote:

    Dear Darren,

    1.My Cloudera version is "Cloudera Express 5.0.0 (#215 built by
    jenkins on 20140331-1424 git"

    2.Yes, I used parcels for CDH binaries.

    3.I used embedded PostgreSQL Database (default option). Sorry I can't
    find the version info.

    4.You are right too. First time when I ran the create tables command
    seems no error. The log is as below

    + exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool
    -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.
    fileinputformat.input.dir.recursive
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.max.split.size is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.maxsize
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.minsize
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.minsize.per.rack
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.minsize.per.node
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks
    is deprecated. Instead, use mapreduce.job.reduces
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution
    is deprecated. Instead, use mapreduce.reduce.speculative
    14/04/23 16:56:17 WARN conf.HiveConf: DEPRECATED: Configuration
    property hive.metastore.local no longer has any effect. Make sure to
    provide a valid value for hive.metastore.uris if you are connecting to a
    remote megastore.

    So I try to check next step process (26-hdfs-NAMENODE-createdir) log.
    There are some error messages.

    + echo 'Created directory /user/hive.'
    + /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
    --config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir
    dfs -chown hive:hive /user/hive
    log4j:ERROR Could not find value for key log4j.appender.EventCounter
    log4j:ERROR Could not instantiate appender named "EventCounter".
    + '[' 0 -eq 0 ']'
    + echo 'Changed ownership of /user/hive to hive:hive.'
    + /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
    --config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir
    dfs -chmod 1775 /user/hive
    log4j:ERROR Could not find value for key log4j.appender.EventCounter
    log4j:ERROR Could not instantiate appender named "EventCounter".




    Darren Lo於 2014年4月24日星期四UTC+8上午1時59分25秒寫道:
    Hi Ivan,

    You're right that the schema should have been clean when you first
    started.

    Can you confirm your Cloudera Manager version? Click on Support ->
    About in the upper right. You may need to exit the wizard to see this link
    (exit the wizard by clicking on the Cloudera Manager logo).

    Did you use parcels for CDH binaries? (this is the default choice)

    What version of postgresql was installed? ssh in to the machine and
    see what the package manager says the version of postgresql is.

    I suspect that the first time you ran the create tables command, the
    error log had a different message. SSH in to the machine where the Hive
    Metastore Server is running, then look in /var/run/cloudera-scm-agent/
    process and find the directories with a name like
    "52-hive-metastore-create-tables". The "52" part will be different
    on your system. Find the earliest attempt, and look in the logs directory
    for the stderr log. Please share the end of that log.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 10:48 AM, Ivan Hsueh wrote:

    Thanks Darren,

    It's weird, I installed CDH 5 from clean system.
    There shouldn't be an existing table.

    Another problem is I can't use schematool to initialise the DB.
    Would you please describe more specifically how to clean schema.

    Thanks again..


    Darren Lo於 2014年4月24日星期四UTC+8上午12時50分04秒寫道:
    Hi Ivan,

    You've configured Hive to talk to a schema that already has tables,
    as indicated by this message:
    Error: ERROR: relation "BUCKETING_COLS" already exists
    (state=42P07,code=0)

    Configure Hive with a clean schema and this problem should go away.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 7:31 AM, Ivan Hsueh wrote:

    Hi,

    I try to build a LAB for Cloudera 5.0

    There are 2 hosts in the same domain
    describe as below,

    Cloudera51»Manager, Name Node
    2GB RAM
    50GB Space

    Cloudera52»Data Node
    2GB RAM
    50GB Space

    I followed the "Cloudera-Manager-Installation-Guide.pdf", and
    used "Installation Path A - Automated Installation by Cloudera Manager "

    first, I installed the cloudera manager 5 on Cloudera51
    Second, Choosed the cluster hosts
    Third, Manager automatically assigned Cloudera51 is Name Node,
    and Cloudera52 is Data Node.
    fourth, Using Manager Web UI to install CDH5 on both Cloudera51
    and Cloudera52 (with Parcel)
    I used embedded PostgreSQL Database

    Everything is fine until Manager try to install Hive service.
    It's always fail when creating megastore tables. (I had tried at
    least 3 times)

    Here is error log:

    Failed to Create Hive Metastore Database Tables.

    *Program:* hive/hive.sh ["create_metastore_tables"]

    + [[ create_metastore_tables == updatelocation ]]
    + [[ create_metastore_tables == create_metastore_tables ]]
    + [[ -z '' ]]
    + JDBC_JARS_CLASSPATH='/usr/share/cmf/lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar'
    + [[ create_metastore_tables == create_metastore_tables ]]
    + '[' 5 -ge 5 ']'
    + exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
    14/04/23 17:21:50 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
    Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)
    org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
    *** schemaTool failed ***


    Can anyone help me to solved this problem?
    Thanks a lot.

    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Darren Lo at May 16, 2014 at 4:00 pm
    Hi Kevin,

    To create the tables in the hive metastore relational database, click on
    your Hive service, then find the appropriate command in the Actions
    dropdown menu on the right. These are the tables that Hive itself uses to
    keep track of logical user data.

    To create logical hive tables where you would actually run sql queries on,
    use any hive client. Beeline is the command-line tool used to talk to
    HiveServer2. Hue has a Beeswax app that can talk to Hive as well, if you
    prefer a GUI interface. If not already configured, you can add a Hue
    service through the Cloudera Manager UI. If it's not already present, use
    click the dropdown next to your cluster name and click Add Service.

    Thanks,
    Darren

    On Fri, May 16, 2014 at 1:40 AM, Kevin Huang wrote:

    Ivan,

    How did you resolve it finally?
    Creating Hive Metastore Database Tables
    Where I can launch Hive Metastore database tables in UI?

    Thanks a lot!

    On Wednesday, April 23, 2014 1:29:00 PM UTC-7, Ivan Hsueh wrote:

    Dear Darren,

    Finally, all services are start.
    Thank you for helping me!


    Darren Lo於 2014年4月24日星期四UTC+8上午3時47分36秒寫道:
    You should run the Create Hive User Directory and Create Hive Warehouse
    Directory commands (available if you click on Hive and look in the actions
    menu on the right), then the Start command for hive.

    Then you should be able to re-try the first run command (find it in your
    command history at http://<cm_host>:7180/cmf/commands/commands, click
    on your first run command, and look for a retry button), or you can run
    each of the listed commands manually, one by one.

    On Wed, Apr 23, 2014 at 12:41 PM, Ivan Hsueh wrote:

    Dear Darren,

    It's seems work. I can start the metastore via UI.

    Thu Apr 24 03:26:29 CST 2014
    JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
    using /usr/java/jdk1.7.0_45-cloudera as JAVA_HOME
    using 5 as CDH_VERSION
    using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive as HIVE_HOME
    using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE as HIVE_CONF_DIR
    using /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop as HADOOP_HOME
    using /var/run/cloudera-scm-agent/process/33-hive-HIVEMETASTORE/yarn-conf as HADOOP_CONF_DIR
    Starting Hive Metastore Server


    Both OS are CentOS 6.4.

    If it does really work, can I just pass this process and keep running
    the rest of process via CM UI?


    <https://lh3.googleusercontent.com/-7Mkz3nBNeJs/U1gWaWKBePI/AAAAAAAAAhI/YgDpz6I1F-8/s1600/Screen+Shot+2014-04-24+at+3.31.40+AM.png>

    Many thanks.


    Darren Lo於 2014年4月24日星期四UTC+8上午3時23分24秒寫道:
    Hi Ivan,

    It looks like create table actually succeeded the first time. What
    happens if you try to start the metastore (via the CM UI)?

    Also, what operating system are you using?

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 11:50 AM, Ivan Hsueh wrote:

    Dear Darren,

    1.My Cloudera version is "Cloudera Express 5.0.0 (#215 built by
    jenkins on 20140331-1424 git"

    2.Yes, I used parcels for CDH binaries.

    3.I used embedded PostgreSQL Database (default option). Sorry I can't
    find the version info.

    4.You are right too. First time when I ran the create tables command
    seems no error. The log is as below

    + exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool
    -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.
    fileinputformat.input.dir.recursive
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.max.split.size is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.maxsize
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.minsize
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.minsize.per.rack
    14/04/23 16:56:10 INFO Configuration.deprecation:
    mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.
    fileinputformat.split.minsize.per.node
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks
    is deprecated. Instead, use mapreduce.job.reduces
    14/04/23 16:56:10 INFO Configuration.deprecation: mapred.reduce.tasks.
    speculative.execution is deprecated. Instead, use
    mapreduce.reduce.speculative
    14/04/23 16:56:17 WARN conf.HiveConf: DEPRECATED: Configuration
    property hive.metastore.local no longer has any effect. Make sure to
    provide a valid value for hive.metastore.uris if you are connecting to a
    remote megastore.

    So I try to check next step process (26-hdfs-NAMENODE-createdir) log.
    There are some error messages.

    + echo 'Created directory /user/hive.'
    + /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
    --config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir
    dfs -chown hive:hive /user/hive
    log4j:ERROR Could not find value for key log4j.appender.EventCounter
    log4j:ERROR Could not instantiate appender named "EventCounter".
    + '[' 0 -eq 0 ']'
    + echo 'Changed ownership of /user/hive to hive:hive.'
    + /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop-hdfs/bin/hdfs
    --config /var/run/cloudera-scm-agent/process/26-hdfs-NAMENODE-createdir
    dfs -chmod 1775 /user/hive
    log4j:ERROR Could not find value for key log4j.appender.EventCounter
    log4j:ERROR Could not instantiate appender named "EventCounter".




    Darren Lo於 2014年4月24日星期四UTC+8上午1時59分25秒寫道:
    Hi Ivan,

    You're right that the schema should have been clean when you first
    started.

    Can you confirm your Cloudera Manager version? Click on Support ->
    About in the upper right. You may need to exit the wizard to see this link
    (exit the wizard by clicking on the Cloudera Manager logo).

    Did you use parcels for CDH binaries? (this is the default choice)

    What version of postgresql was installed? ssh in to the machine and
    see what the package manager says the version of postgresql is.

    I suspect that the first time you ran the create tables command, the
    error log had a different message. SSH in to the machine where the Hive
    Metastore Server is running, then look in /var/run/cloudera-scm-agent/
    process and find the directories with a name like
    "52-hive-metastore-create-tables". The "52" part will be different
    on your system. Find the earliest attempt, and look in the logs directory
    for the stderr log. Please share the end of that log.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 10:48 AM, Ivan Hsueh wrote:

    Thanks Darren,

    It's weird, I installed CDH 5 from clean system.
    There shouldn't be an existing table.

    Another problem is I can't use schematool to initialise the DB.
    Would you please describe more specifically how to clean schema.

    Thanks again..


    Darren Lo於 2014年4月24日星期四UTC+8上午12時50分04秒寫道:
    Hi Ivan,

    You've configured Hive to talk to a schema that already has
    tables, as indicated by this message:
    Error: ERROR: relation "BUCKETING_COLS" already exists
    (state=42P07,code=0)

    Configure Hive with a clean schema and this problem should go away.

    Thanks,
    Darren

    On Wed, Apr 23, 2014 at 7:31 AM, Ivan Hsueh wrote:

    Hi,

    I try to build a LAB for Cloudera 5.0

    There are 2 hosts in the same domain
    describe as below,

    Cloudera51»Manager, Name Node
    2GB RAM
    50GB Space

    Cloudera52»Data Node
    2GB RAM
    50GB Space

    I followed the "Cloudera-Manager-Installation-Guide.pdf", and
    used "Installation Path A - Automated Installation by Cloudera Manager "

    first, I installed the cloudera manager 5 on Cloudera51
    Second, Choosed the cluster hosts
    Third, Manager automatically assigned Cloudera51 is Name Node,
    and Cloudera52 is Data Node.
    fourth, Using Manager Web UI to install CDH5 on both Cloudera51
    and Cloudera52 (with Parcel)
    I used embedded PostgreSQL Database

    Everything is fine until Manager try to install Hive service.
    It's always fail when creating megastore tables. (I had tried at
    least 3 times)

    Here is error log:

    Failed to Create Hive Metastore Database Tables.

    *Program:* hive/hive.sh ["create_metastore_tables"]

    + [[ create_metastore_tables == updatelocation ]]
    + [[ create_metastore_tables == create_metastore_tables ]]
    + [[ -z '' ]]
    + JDBC_JARS_CLASSPATH='/usr/share/cmf/lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar'
    + [[ create_metastore_tables == create_metastore_tables ]]
    + '[' 5 -ge 5 ']'
    + exec /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hive/bin/schematool -dbType postgres -initSchema -dbOpts postgres.filter.81
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    14/04/23 17:21:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
    14/04/23 17:21:50 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
    Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)
    org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
    *** schemaTool failed ***


    Can anyone help me to solved this problem?
    Thanks a lot.

    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from
    it, send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedApr 23, '14 at 2:31p
activeMay 16, '14 at 4:00p
posts11
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase