FAQ
Hi all,

I install Impala on Ubuntu server 12.04 using Cloudera
Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.

HDFS, Hive, MapReduce services work fine, the status is Good health. But I
can't start Impala, the error is:

*Supervisor returned FATAL. Please check the role log file, stderr, or
stdout.*
*
*
The error log is:

*+
HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMPALAD/hadoop-conf
*
*+ export
HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMPALAD/hive-conf*
*+
HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMPALAD/hive-conf*
*+ export
HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMPALAD/hbase-conf
*
*+
HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMPALAD/hbase-conf
*
*+
JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
*
*+ [[ -z '' ]]*
*+ export
AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
*
*+
AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
*
*+ [[ -z '' ]]*
*+ export
CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
*
*+
CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
*
*+
FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-conf/impalad_flags
*
*+ USE_DEBUG_BUILD=false*
*+ perl -pi -e
's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/208-impala-IMPALAD#g'
/run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-conf/impalad_flags
*
*+ '[' -f
/run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-conf/.htpasswd ']'
*
*+ chmod 600
/run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-conf/.htpasswd*
*+ false*
*+ export
IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail*
*+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail
*
*+ '[' impalad = impalad ']'*
*+ exec
/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/impalad
--flagfile=/run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-conf/impalad_flags
*
*
*
Are there anyone know how to fix this problem? Thank you.
*
*

Search Discussions

  • Vikas Singh at Aug 8, 2013 at 3:44 pm
    Hi Andy,

    Can you get the impalad and statestore log and provide it to us. The
    stderr/stdout log is from the bash file that launches impalad/statestore.
    The log from impalad/statestore will tell us more on why we can't start
    these daemons.

    Thanks,
    Vikas

    On Thu, Aug 8, 2013 at 8:17 AM, Andy wrote:

    Hi all,

    I install Impala on Ubuntu server 12.04 using Cloudera
    Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.

    HDFS, Hive, MapReduce services work fine, the status is Good health. But I
    can't start Impala, the error is:

    *Supervisor returned FATAL. Please check the role log file, stderr, or
    stdout.*
    *
    *
    The error log is:

    *+
    HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMPALAD/hadoop-conf
    *
    *+ export
    HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMPALAD/hive-conf
    *
    *+
    HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMPALAD/hive-conf
    *
    *+ export
    HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMPALAD/hbase-conf
    *
    *+
    HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMPALAD/hbase-conf
    *
    *+
    JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
    *
    *+ [[ -z '' ]]*
    *+ export
    AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
    *
    *+
    AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
    *
    *+ [[ -z '' ]]*
    *+ export
    CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
    *
    *+
    CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
    *
    *+
    FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-conf/impalad_flags
    *
    *+ USE_DEBUG_BUILD=false*
    *+ perl -pi -e
    's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/208-impala-IMPALAD#g'
    /run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-conf/impalad_flags
    *
    *+ '[' -f
    /run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-conf/.htpasswd ']'
    *
    *+ chmod 600
    /run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-conf/.htpasswd*
    *+ false*
    *+ export
    IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail*
    *+
    IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail*
    *+ '[' impalad = impalad ']'*
    *+ exec
    /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/impalad
    --flagfile=/run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-conf/impalad_flags
    *
    *
    *
    Are there anyone know how to fix this problem? Thank you.
    *
    *
  • Vikas Singh at Aug 8, 2013 at 5:39 pm
    Hi Andy,

    So either the binaries are terminating very early (even before opening a
    log), or we are not able to exec them (some dependent libraries are
    missing). What happens if you run them command directly from the
    commandline:

    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/statestored*
    *
    *
    *Try using ldd on
    **/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail/statestored
    to see if it shows up something.*
    *
    *
    *- Vikas*

    On Thu, Aug 8, 2013 at 10:18 AM, Andy wrote:

    Hi Vikas,

    There's nothing in */var/log/impalad *and */var/log/statestore *directories.
    This is their permission:
    *
    *
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:05 impalad/*
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:16 statestore/*

    Thanks,
    Andy
    On Thursday, August 8, 2013 11:44:57 PM UTC+8, vikas wrote:

    Hi Andy,

    Can you get the impalad and statestore log and provide it to us. The
    stderr/stdout log is from the bash file that launches impalad/statestore.
    The log from impalad/statestore will tell us more on why we can't start
    these daemons.

    Thanks,
    Vikas

    On Thu, Aug 8, 2013 at 8:17 AM, Andy wrote:

    Hi all,

    I install Impala on Ubuntu server 12.04 using Cloudera
    Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.

    HDFS, Hive, MapReduce services work fine, the status is Good health. But
    I can't start Impala, the error is:

    *Supervisor returned FATAL. Please check the role log file, stderr, or
    stdout.*
    *
    *
    The error log is:

    *+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hadoop-conf*
    *+ export HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hive-conf*
    *+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hive-conf*
    *+ export HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hbase-conf*
    *+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hbase-conf*
    *+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/impala-conf/impalad_flags*
    *+ USE_DEBUG_BUILD=false*
    *+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/208-impala-IMPALAD#g'
    /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/impalad_flags*
    *+ '[' -f /run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-conf/.htpasswd
    ']'*
    *+ chmod 600 /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/.htpasswd*
    *+ false*
    *+ export IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
    impala/sbin-retail*
    *+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
    impala/sbin-retail*
    *+ '[' impalad = impalad ']'*
    *+ exec /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/impalad
    --flagfile=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/impala-conf/impalad_flags*
    *
    *
    Are there anyone know how to fix this problem? Thank you.
    *
    *
  • Nguyen Dang Minh at Aug 8, 2013 at 5:45 pm
    Hi Vikas,

    The output is:
    *$ /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/statestored*
    *Illegal instruction (core dumped)*

    *$ ldd
    /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail/statestored*
    * linux-vdso.so.1 => (0x00007fff7413c000)*
    * libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
    (0x00007f5c3a8e6000)*
    * libhdfs.so.0.0.0 => not found*
    * librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5c3a6dd000)*
    * libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c3a4d9000)*
    * libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
    (0x00007f5c3a27d000)*
    * libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    (0x00007f5c39f7c000)*
    * libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c39c80000)*
    * libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5c39a6a000)*
    * libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c396aa000)*
    * /lib64/ld-linux-x86-64.so.2 (0x00007f5c3ab0a000)*
    * libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    (0x00007f5c392e2000)*
    * libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c390ca000)*
    *
    *
    *
    $ locate libhdfs.so.0.0.0
    /opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/libhdfs.so.0.0.0
    /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/lib/libhdfs.so.0.0.0

    *
    Thanks,
    Andy

    On Fri, Aug 9, 2013 at 1:39 AM, Vikas Singh wrote:

    Hi Andy,

    So either the binaries are terminating very early (even before opening a
    log), or we are not able to exec them (some dependent libraries are
    missing). What happens if you run them command directly from the
    commandline:

    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/statestored*
    *
    *
    *Try using ldd on **/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail/statestored
    to see if it shows up something.*
    *
    *
    *- Vikas*

    On Thu, Aug 8, 2013 at 10:18 AM, Andy wrote:

    Hi Vikas,

    There's nothing in */var/log/impalad *and */var/log/statestore *directories.
    This is their permission:
    *
    *
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:05 impalad/*
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:16 statestore/*

    Thanks,
    Andy
    On Thursday, August 8, 2013 11:44:57 PM UTC+8, vikas wrote:

    Hi Andy,

    Can you get the impalad and statestore log and provide it to us. The
    stderr/stdout log is from the bash file that launches impalad/statestore.
    The log from impalad/statestore will tell us more on why we can't start
    these daemons.

    Thanks,
    Vikas

    On Thu, Aug 8, 2013 at 8:17 AM, Andy wrote:

    Hi all,

    I install Impala on Ubuntu server 12.04 using Cloudera
    Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.

    HDFS, Hive, MapReduce services work fine, the status is Good health.
    But I can't start Impala, the error is:

    *Supervisor returned FATAL. Please check the role log file, stderr, or
    stdout.*
    *
    *
    The error log is:

    *+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hadoop-conf*
    *+ export HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hive-conf*
    *+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hive-conf*
    *+ export HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hbase-conf*
    *+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hbase-conf*
    *+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/impala-conf/impalad_flags*
    *+ USE_DEBUG_BUILD=false*
    *+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/208-impala-IMPALAD#g'
    /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/impalad_flags*
    *+ '[' -f /run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-conf/.htpasswd
    ']'*
    *+ chmod 600 /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/.htpasswd*
    *+ false*
    *+ export IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
    impala/sbin-retail*
    *+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
    impala/sbin-retail*
    *+ '[' impalad = impalad ']'*
    *+ exec /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/impalad
    --flagfile=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/impala-conf/impalad_flags*
    *
    *
    Are there anyone know how to fix this problem? Thank you.
    *
    *

    --
    Nguyen Dang Minh - 阮登明
    http://www.minhnd.com
  • Andy at Aug 8, 2013 at 6:13 pm
    Hi Vikas,

    The output is:
    *$ /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/statestored*
    *Illegal instruction (core dumped)*

    *$ ldd
    /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail/statestored*
    * linux-vdso.so.1 => (0x00007fff7413c000)*
    * libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
    (0x00007f5c3a8e6000)*
    * libhdfs.so.0.0.0 => not found*
    * librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5c3a6dd000)*
    * libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c3a4d9000)*
    * libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
    (0x00007f5c3a27d000)*
    * libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    (0x00007f5c39f7c000)*
    * libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c39c80000)*
    * libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5c39a6a000)*
    * libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c396aa000)*
    * /lib64/ld-linux-x86-64.so.2 (0x00007f5c3ab0a000)*
    * libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    (0x00007f5c392e2000)*
    * libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c390ca000)*
    *
    *
    *$ locate libhdfs.so.0.0.0*
    */opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/libhdfs.so.0.0.0*
    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/lib/libhdfs.so.0.0.0*

    Thanks,
    Andy
    On Friday, August 9, 2013 1:39:06 AM UTC+8, vikas wrote:

    Hi Andy,

    So either the binaries are terminating very early (even before opening a
    log), or we are not able to exec them (some dependent libraries are
    missing). What happens if you run them command directly from the
    commandline:

    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/statestored*
    *
    *
    *Try using ldd on **/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail/statestored
    to see if it shows up something.*
    *
    *
    *- Vikas*


    On Thu, Aug 8, 2013 at 10:18 AM, Andy <nguyend...@gmail.com <javascript:>>wrote:
    Hi Vikas,

    There's nothing in */var/log/impalad *and */var/log/statestore *directories.
    This is their permission:
    *
    *
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:05 impalad/*
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:16 statestore/*

    Thanks,
    Andy
    On Thursday, August 8, 2013 11:44:57 PM UTC+8, vikas wrote:

    Hi Andy,

    Can you get the impalad and statestore log and provide it to us. The
    stderr/stdout log is from the bash file that launches impalad/statestore.
    The log from impalad/statestore will tell us more on why we can't start
    these daemons.

    Thanks,
    Vikas

    On Thu, Aug 8, 2013 at 8:17 AM, Andy wrote:

    Hi all,

    I install Impala on Ubuntu server 12.04 using Cloudera
    Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.

    HDFS, Hive, MapReduce services work fine, the status is Good health.
    But I can't start Impala, the error is:

    *Supervisor returned FATAL. Please check the role log file, stderr, or
    stdout.*
    *
    *
    The error log is:

    *+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hadoop-conf*
    *+ export HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hive-conf*
    *+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hive-conf*
    *+ export HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hbase-conf*
    *+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hbase-conf*
    *+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/impala-conf/impalad_flags*
    *+ USE_DEBUG_BUILD=false*
    *+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/208-impala-IMPALAD#g'
    /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/impalad_flags*
    *+ '[' -f /run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-conf/.htpasswd
    ']'*
    *+ chmod 600 /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/.htpasswd*
    *+ false*
    *+ export IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
    impala/sbin-retail*
    *+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
    impala/sbin-retail*
    *+ '[' impalad = impalad ']'*
    *+ exec /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/impalad
    --flagfile=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/impala-conf/impalad_flags*
    *
    *
    Are there anyone know how to fix this problem? Thank you.
    *
    *
  • Andy at Aug 8, 2013 at 6:36 pm
    Hi Vikas,

    1. I already did as you guided. The directory is *210-impala-STATESTORE. *Its
    content is in attach file
    *$ sudo ls -l /run/cloudera-scm-agent/process/*
    *total 0*
    *drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28
    188-hdfs-SECONDARYNAMENODE*
    *drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28 191-hdfs-DATANODE*
    *drwxr-x--x 3 hdfs hdfs 360 Aug 8 06:28 192-hdfs-NAMENODE*
    *drwxr-x--x 3 mapred hadoop 380 Aug 8 06:28 193-mapreduce-JOBTRACKER*
    *drwxr-x--x 3 mapred hadoop 240 Aug 8 06:28 196-mapreduce-TASKTRACKER
    *
    *drwxr-x--x 3 zookeeper zookeeper 220 Aug 8 06:28 197-zookeeper-server*
    *drwxr-x--x 4 hive hive 160 Aug 8 06:28 198-hive-HIVEMETASTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 07:56 199-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 07:57 202-impala-STATESTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 07:59 203-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 07:59 206-impala-STATESTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 08:13 207-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 08:13 210-impala-STATESTORE*

    2. It works! How can I run without using Debug Build?

    Thanks,
    Andy
    On Friday, August 9, 2013 2:15:14 AM UTC+8, vikas wrote:

    Thanks Andy, the libraries seem to be ok. CM adds the libhdfs to the
    LD_LIBRARY_PATH so that is fine.

    The core dump part is interesting. There are few things we can try:

    1. Collect the core dump for debugging. In the CM UI, got to Impala
    configuration -> Service Wide -> Enable core dump. Check the check-box and
    save the configuration. Then restart the statestore and this time it will
    core dump, the dump should go to its cwd, which will be under '/run/cloudera-scm-agent/process/'
    directory (the latest created directory with STATESTORE in its name).
    Please see if you have a hs_err_<pid>.log there too. Please send us these.

    2. To get things working, try running debug build. You can go to CM
    console and select the Impala configuration option -> Sevice Wide -> Use
    Debug Build. Check this option and save configuration. Try restarting
    service now and see if it helps.

    You may be hitting this: https://issues.cloudera.org/browse/IMPALA-160,
    please look at output of "stdout" and see if the error matches.

    Vikas


    On Thu, Aug 8, 2013 at 11:13 AM, Andy <nguyend...@gmail.com <javascript:>>wrote:
    Hi Vikas,

    The output is:
    *$
    /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/statestored*
    *Illegal instruction (core dumped)*

    *$ ldd
    /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail/statestored
    *
    * linux-vdso.so.1 => (0x00007fff7413c000)*
    * libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
    (0x00007f5c3a8e6000)*
    * libhdfs.so.0.0.0 => not found*
    * librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5c3a6dd000)*
    * libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c3a4d9000)*
    * libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
    (0x00007f5c3a27d000)*
    * libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    (0x00007f5c39f7c000)*
    * libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c39c80000)*
    * libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
    (0x00007f5c39a6a000)*
    * libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c396aa000)*
    * /lib64/ld-linux-x86-64.so.2 (0x00007f5c3ab0a000)*
    * libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    (0x00007f5c392e2000)*
    * libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c390ca000)*
    *
    *
    *$ locate libhdfs.so.0.0.0*
    */opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/libhdfs.so.0.0.0*
    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/lib/libhdfs.so.0.0.0*

    Thanks,
    Andy
    On Friday, August 9, 2013 1:39:06 AM UTC+8, vikas wrote:

    Hi Andy,

    So either the binaries are terminating very early (even before opening a
    log), or we are not able to exec them (some dependent libraries are
    missing). What happens if you run them command directly from the
    commandline:

    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../
    bin/statestored*
    *
    *
    *Try using ldd on **/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail/statestored
    to see if it shows up something.*
    *
    *
    *- Vikas*

    On Thu, Aug 8, 2013 at 10:18 AM, Andy wrote:

    Hi Vikas,

    There's nothing in */var/log/impalad *and */var/log/statestore *directories.
    This is their permission:
    *
    *
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:05 impalad/*
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:16
    statestore/*

    Thanks,
    Andy
    On Thursday, August 8, 2013 11:44:57 PM UTC+8, vikas wrote:

    Hi Andy,

    Can you get the impalad and statestore log and provide it to us. The
    stderr/stdout log is from the bash file that launches impalad/statestore.
    The log from impalad/statestore will tell us more on why we can't start
    these daemons.

    Thanks,
    Vikas

    On Thu, Aug 8, 2013 at 8:17 AM, Andy wrote:

    Hi all,

    I install Impala on Ubuntu server 12.04 using Cloudera
    Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.

    HDFS, Hive, MapReduce services work fine, the status is Good health.
    But I can't start Impala, the error is:

    *Supervisor returned FATAL. Please check the role log file, stderr,
    or stdout.*
    *
    *
    The error log is:

    *+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-I
    MPALAD/hadoop-conf*
    *+ export HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hive-conf*
    *+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMP
    ALAD/hive-conf*
    *+ export HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hbase-conf*
    *+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IM
    PALAD/hbase-conf*
    *+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/shar
    e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/orac
    le-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export AUX_CLASSPATH=/usr/share/java/
    mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-8
    01.jdbc4.jar:/usr/share/java/oracle-connector-java.jar*
    *+ AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/o
    racle-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/orac
    le-connector-java.jar*
    *+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/shar
    e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/orac
    le-connector-java.jar*
    *+ FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-IMPALAD
    /impala-conf/impalad_flags*
    *+ USE_DEBUG_BUILD=false*
    *+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/
    208-impala-IMPALAD#g' /run/cloudera-scm-agent/proces
    s/208-impala-IMPALAD/impala-conf/impalad_flags*
    *+ '[' -f /run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-conf/.htpasswd
    ']'*
    *+ chmod 600 /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/.htpasswd*
    *+ false*
    *+ export IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
    impala/sbin-retail*
    *+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impal
    a/sbin-retail*
    *+ '[' impalad = impalad ']'*
    *+ exec /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/impalad
    --flagfile=/run/cloudera-scm-agent/process/208-impala-IMPALA
    D/impala-conf/impalad_flags*
    *
    *
    Are there anyone know how to fix this problem? Thank you.
    *
    *
  • Vikas Singh at Aug 8, 2013 at 7:16 pm
    Hi Andy,

    1. Unfortunately there is no core present. By default the core get
    generated in the current working directory of the process. What is the
    value of /proc/sys/kernel/core_pattern on your setup?
    2. To run retail build, if possible try using some other os than Ubuntu
    12.04 till we get this issue sorted out.

    - Vikas

    On Thu, Aug 8, 2013 at 11:36 AM, Andy wrote:

    Hi Vikas,

    1. I already did as you guided. The directory is *210-impala-STATESTORE. *Its
    content is in attach file
    *$ sudo ls -l /run/cloudera-scm-agent/process/*
    *total 0*
    *drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28
    188-hdfs-SECONDARYNAMENODE*
    *drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28 191-hdfs-DATANODE*
    *drwxr-x--x 3 hdfs hdfs 360 Aug 8 06:28 192-hdfs-NAMENODE*
    *drwxr-x--x 3 mapred hadoop 380 Aug 8 06:28
    193-mapreduce-JOBTRACKER*
    *drwxr-x--x 3 mapred hadoop 240 Aug 8 06:28
    196-mapreduce-TASKTRACKER*
    *drwxr-x--x 3 zookeeper zookeeper 220 Aug 8 06:28 197-zookeeper-server*
    *drwxr-x--x 4 hive hive 160 Aug 8 06:28 198-hive-HIVEMETASTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 07:56 199-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 07:57 202-impala-STATESTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 07:59 203-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 07:59 206-impala-STATESTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 08:13 207-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 08:13 210-impala-STATESTORE*

    2. It works! How can I run without using Debug Build?

    Thanks,
    Andy
    On Friday, August 9, 2013 2:15:14 AM UTC+8, vikas wrote:

    Thanks Andy, the libraries seem to be ok. CM adds the libhdfs to the
    LD_LIBRARY_PATH so that is fine.

    The core dump part is interesting. There are few things we can try:

    1. Collect the core dump for debugging. In the CM UI, got to Impala
    configuration -> Service Wide -> Enable core dump. Check the check-box and
    save the configuration. Then restart the statestore and this time it will
    core dump, the dump should go to its cwd, which will be under '
    /run/cloudera-scm-agent/**process/' directory (the latest created
    directory with STATESTORE in its name). Please see if you have a
    hs_err_<pid>.log there too. Please send us these.

    2. To get things working, try running debug build. You can go to CM
    console and select the Impala configuration option -> Sevice Wide -> Use
    Debug Build. Check this option and save configuration. Try restarting
    service now and see if it helps.

    You may be hitting this: https://issues.cloudera.**org/browse/IMPALA-160<https://issues.cloudera.org/browse/IMPALA-160>,
    please look at output of "stdout" and see if the error matches.

    Vikas

    On Thu, Aug 8, 2013 at 11:13 AM, Andy wrote:

    Hi Vikas,

    The output is:
    *$ /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../
    bin/statestored*
    *Illegal instruction (core dumped)*

    *$ ldd /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-
    retail/statestored*
    * linux-vdso.so.1 => (0x00007fff7413c000)*
    * libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
    (0x00007f5c3a8e6000)*
    * libhdfs.so.0.0.0 => not found*
    * librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5c3a6dd000)*
    * libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c3a4d9000)*
    * libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
    (0x00007f5c3a27d000)*
    * libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    (0x00007f5c39f7c000)*
    * libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c39c80000)*
    * libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
    (0x00007f5c39a6a000)*
    * libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c396aa000)*
    * /lib64/ld-linux-x86-64.so.2 (0x00007f5c3ab0a000)*
    * libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    (0x00007f5c392e2000)*
    * libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c390ca000)*
    *
    *
    *$ locate libhdfs.so.0.0.0*
    */opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/libhdfs.so.0.0.0*
    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/lib/libhdfs.so.0.0.0
    *

    Thanks,
    Andy
    On Friday, August 9, 2013 1:39:06 AM UTC+8, vikas wrote:

    Hi Andy,

    So either the binaries are terminating very early (even before opening
    a log), or we are not able to exec them (some dependent libraries are
    missing). What happens if you run them command directly from the
    commandline:

    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
    /statestored*
    *
    *
    *Try using ldd on **/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/
    sbin-retail/statestored to see if it shows up something.*
    *
    *
    *- Vikas*

    On Thu, Aug 8, 2013 at 10:18 AM, Andy wrote:

    Hi Vikas,

    There's nothing in */var/log/impalad *and */var/log/statestore *directories.
    This is their permission:
    *
    *
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:05 impalad/*
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:16
    statestore/*

    Thanks,
    Andy
    On Thursday, August 8, 2013 11:44:57 PM UTC+8, vikas wrote:

    Hi Andy,

    Can you get the impalad and statestore log and provide it to us. The
    stderr/stdout log is from the bash file that launches impalad/statestore.
    The log from impalad/statestore will tell us more on why we can't start
    these daemons.

    Thanks,
    Vikas

    On Thu, Aug 8, 2013 at 8:17 AM, Andy wrote:

    Hi all,

    I install Impala on Ubuntu server 12.04 using Cloudera
    Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.

    HDFS, Hive, MapReduce services work fine, the status is Good health.
    But I can't start Impala, the error is:

    *Supervisor returned FATAL. Please check the role log file, stderr,
    or stdout.*
    *
    *
    The error log is:

    *+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-I
    MPALAD/hadoop-conf*
    *+ export HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hive-conf*
    *+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMP
    ALAD/hive-conf*
    *+ export HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hbase-conf*
    *+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IM
    PALAD/hbase-conf*
    *+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/shar
    e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
    e-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export AUX_CLASSPATH=/usr/share/java/
    mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-8
    01.jdbc4.jar:/usr/share/java/oracle-connector-java.jar*
    *+ AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/o
    racle-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
    e-connector-java.jar*
    *+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/shar
    e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
    e-connector-java.jar*
    *+ FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-IMPALAD
    /impala-conf/impalad_flags*
    *+ USE_DEBUG_BUILD=false*
    *+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/
    208-impala-IMPALAD#g' /run/cloudera-scm-agent/proces
    s/208-impala-IMPALAD/impala-conf/impalad_flags*
    *+ '[' -f /run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-
    conf/.htpasswd ']'*
    *+ chmod 600 /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/.htpasswd*
    *+ false*
    *+ export IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
    impala/sbin-retail*
    *+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impal
    a/sbin-retail*
    *+ '[' impalad = impalad ']'*
    *+ exec /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/impalad
    --flagfile=/run/cloudera-scm-agent/process/208-impala-IMPALA
    D/impala-conf/impalad_flags*
    *
    *
    Are there anyone know how to fix this problem? Thank you.
    *
    *
  • Andy at Aug 8, 2013 at 7:59 pm
    Hi Vikas,

    1. The value of */proc/sys/kernel/core_pattern *is:
    *|/usr/share/apport/apport %p %s %c*
    *
    *
    2. Thanks, I'll try CentOS.

    Andy.
    On Friday, August 9, 2013 3:16:46 AM UTC+8, vikas wrote:

    Hi Andy,

    1. Unfortunately there is no core present. By default the core get
    generated in the current working directory of the process. What is the
    value of /proc/sys/kernel/core_pattern on your setup?
    2. To run retail build, if possible try using some other os than Ubuntu
    12.04 till we get this issue sorted out.

    - Vikas


    On Thu, Aug 8, 2013 at 11:36 AM, Andy <nguyend...@gmail.com <javascript:>>wrote:
    Hi Vikas,

    1. I already did as you guided. The directory is *210-impala-STATESTORE.
    *Its content is in attach file
    *$ sudo ls -l /run/cloudera-scm-agent/process/*
    *total 0*
    *drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28
    188-hdfs-SECONDARYNAMENODE*
    *drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28 191-hdfs-DATANODE*
    *drwxr-x--x 3 hdfs hdfs 360 Aug 8 06:28 192-hdfs-NAMENODE*
    *drwxr-x--x 3 mapred hadoop 380 Aug 8 06:28
    193-mapreduce-JOBTRACKER*
    *drwxr-x--x 3 mapred hadoop 240 Aug 8 06:28
    196-mapreduce-TASKTRACKER*
    *drwxr-x--x 3 zookeeper zookeeper 220 Aug 8 06:28 197-zookeeper-server*
    *drwxr-x--x 4 hive hive 160 Aug 8 06:28 198-hive-HIVEMETASTORE
    *
    *drwxr-x--x 6 impala impala 180 Aug 8 07:56 199-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 07:57 202-impala-STATESTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 07:59 203-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 07:59 206-impala-STATESTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 08:13 207-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 08:13 210-impala-STATESTORE*

    2. It works! How can I run without using Debug Build?

    Thanks,
    Andy
    On Friday, August 9, 2013 2:15:14 AM UTC+8, vikas wrote:

    Thanks Andy, the libraries seem to be ok. CM adds the libhdfs to the
    LD_LIBRARY_PATH so that is fine.

    The core dump part is interesting. There are few things we can try:

    1. Collect the core dump for debugging. In the CM UI, got to Impala
    configuration -> Service Wide -> Enable core dump. Check the check-box and
    save the configuration. Then restart the statestore and this time it will
    core dump, the dump should go to its cwd, which will be under '
    /run/cloudera-scm-agent/**process/' directory (the latest created
    directory with STATESTORE in its name). Please see if you have a
    hs_err_<pid>.log there too. Please send us these.

    2. To get things working, try running debug build. You can go to CM
    console and select the Impala configuration option -> Sevice Wide -> Use
    Debug Build. Check this option and save configuration. Try restarting
    service now and see if it helps.

    You may be hitting this: https://issues.cloudera.**org/browse/IMPALA-160<https://issues.cloudera.org/browse/IMPALA-160>,
    please look at output of "stdout" and see if the error matches.

    Vikas

    On Thu, Aug 8, 2013 at 11:13 AM, Andy wrote:

    Hi Vikas,

    The output is:
    *$ /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../
    bin/statestored*
    *Illegal instruction (core dumped)*

    *$ ldd /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-
    retail/statestored*
    * linux-vdso.so.1 => (0x00007fff7413c000)*
    * libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
    (0x00007f5c3a8e6000)*
    * libhdfs.so.0.0.0 => not found*
    * librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5c3a6dd000)*
    * libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c3a4d9000)*
    * libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
    (0x00007f5c3a27d000)*
    * libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    (0x00007f5c39f7c000)*
    * libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c39c80000)*
    * libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
    (0x00007f5c39a6a000)*
    * libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c396aa000)*
    * /lib64/ld-linux-x86-64.so.2 (0x00007f5c3ab0a000)*
    * libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    (0x00007f5c392e2000)*
    * libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c390ca000)*
    *
    *
    *$ locate libhdfs.so.0.0.0*
    */opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/libhdfs.so.0.0.0*
    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/lib/
    libhdfs.so.0.0.0*

    Thanks,
    Andy
    On Friday, August 9, 2013 1:39:06 AM UTC+8, vikas wrote:

    Hi Andy,

    So either the binaries are terminating very early (even before opening
    a log), or we are not able to exec them (some dependent libraries are
    missing). What happens if you run them command directly from the
    commandline:

    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
    /statestored*
    *
    *
    *Try using ldd on **/opt/cloudera/parcels/IMPAL
    A-1.1-1.p0.8/lib/impala/sbin-retail/statestored to see if it shows up
    something.*
    *
    *
    *- Vikas*

    On Thu, Aug 8, 2013 at 10:18 AM, Andy wrote:

    Hi Vikas,

    There's nothing in */var/log/impalad *and */var/log/statestore *directories.
    This is their permission:
    *
    *
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:05 impalad/
    *
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:16
    statestore/*

    Thanks,
    Andy
    On Thursday, August 8, 2013 11:44:57 PM UTC+8, vikas wrote:

    Hi Andy,

    Can you get the impalad and statestore log and provide it to us. The
    stderr/stdout log is from the bash file that launches impalad/statestore.
    The log from impalad/statestore will tell us more on why we can't start
    these daemons.

    Thanks,
    Vikas

    On Thu, Aug 8, 2013 at 8:17 AM, Andy wrote:

    Hi all,

    I install Impala on Ubuntu server 12.04 using Cloudera
    Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.

    HDFS, Hive, MapReduce services work fine, the status is Good
    health. But I can't start Impala, the error is:

    *Supervisor returned FATAL. Please check the role log file,
    stderr, or stdout.*
    *
    *
    The error log is:

    *+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-I
    MPALAD/hadoop-conf*
    *+ export HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hive-conf*
    *+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMP
    ALAD/hive-conf*
    *+ export HBASE_CONF_DIR=/run/cloudera-s
    cm-agent/process/208-impala-IMPALAD/hbase-conf*
    *+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IM
    PALAD/hbase-conf*
    *+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/shar
    e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
    e-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export AUX_CLASSPATH=/usr/share/java/
    mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-8
    01.jdbc4.jar:/usr/share/java/oracle-connector-java.jar*
    *+ AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/o
    racle-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
    e-connector-java.jar*
    *+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/shar
    e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
    e-connector-java.jar*
    *+ FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-IMPALAD
    /impala-conf/impalad_flags*
    *+ USE_DEBUG_BUILD=false*
    *+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/
    208-impala-IMPALAD#g' /run/cloudera-scm-agent/proces
    s/208-impala-IMPALAD/impala-conf/impalad_flags*
    *+ '[' -f /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/.htpasswd ']'*
    *+ chmod 600 /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/.htpasswd*
    *+ false*
    *+ export IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
    impala/sbin-retail*
    *+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impal
    a/sbin-retail*
    *+ '[' impalad = impalad ']'*
    *+ exec /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../
    bin/impalad --flagfile=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/impala-conf/impalad_flags*
    *
    *
    Are there anyone know how to fix this problem? Thank you.
    *
    *
  • Vikas Singh at Aug 8, 2013 at 8:14 pm
    Hi Andy,

    So you have Ubuntu apport enabled to capture dumps. The crash file may be
    in '/var/crash'.

    If you can't find the core there, try setting core_pattern to simple 'core'
    (echo core > /proc/sys/kernel/core_pattern) and then it should dump the
    core in the current working directory.

    Vikas

    On Thu, Aug 8, 2013 at 12:59 PM, Andy wrote:

    Hi Vikas,

    1. The value of */proc/sys/kernel/core_pattern *is:
    *|/usr/share/apport/apport %p %s %c*
    *
    *
    2. Thanks, I'll try CentOS.

    Andy.
    On Friday, August 9, 2013 3:16:46 AM UTC+8, vikas wrote:

    Hi Andy,

    1. Unfortunately there is no core present. By default the core get
    generated in the current working directory of the process. What is the
    value of /proc/sys/kernel/core_**pattern on your setup?
    2. To run retail build, if possible try using some other os than Ubuntu
    12.04 till we get this issue sorted out.

    - Vikas

    On Thu, Aug 8, 2013 at 11:36 AM, Andy wrote:

    Hi Vikas,

    1. I already did as you guided. The directory is *210-impala-STATESTORE.
    *Its content is in attach file
    *$ sudo ls -l /run/cloudera-scm-agent/process/*
    *total 0*
    *drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28
    188-hdfs-SECONDARYNAMENODE*
    *drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28 191-hdfs-DATANODE*
    *drwxr-x--x 3 hdfs hdfs 360 Aug 8 06:28 192-hdfs-NAMENODE*
    *drwxr-x--x 3 mapred hadoop 380 Aug 8 06:28
    193-mapreduce-JOBTRACKER*
    *drwxr-x--x 3 mapred hadoop 240 Aug 8 06:28
    196-mapreduce-TASKTRACKER*
    *drwxr-x--x 3 zookeeper zookeeper 220 Aug 8 06:28 197-zookeeper-server*
    *drwxr-x--x 4 hive hive 160 Aug 8 06:28
    198-hive-HIVEMETASTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 07:56 199-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 07:57 202-impala-STATESTORE
    *
    *drwxr-x--x 6 impala impala 180 Aug 8 07:59 203-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 07:59 206-impala-STATESTORE
    *
    *drwxr-x--x 6 impala impala 180 Aug 8 08:13 207-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 08:13 210-impala-STATESTORE
    *

    2. It works! How can I run without using Debug Build?

    Thanks,
    Andy
    On Friday, August 9, 2013 2:15:14 AM UTC+8, vikas wrote:

    Thanks Andy, the libraries seem to be ok. CM adds the libhdfs to the
    LD_LIBRARY_PATH so that is fine.

    The core dump part is interesting. There are few things we can try:

    1. Collect the core dump for debugging. In the CM UI, got to Impala
    configuration -> Service Wide -> Enable core dump. Check the check-box and
    save the configuration. Then restart the statestore and this time it will
    core dump, the dump should go to its cwd, which will be under '
    /run/cloudera-scm-agent/**proce**ss/' directory (the latest created
    directory with STATESTORE in its name). Please see if you have a
    hs_err_<pid>.log there too. Please send us these.

    2. To get things working, try running debug build. You can go to CM
    console and select the Impala configuration option -> Sevice Wide -> Use
    Debug Build. Check this option and save configuration. Try restarting
    service now and see if it helps.

    You may be hitting this: https://issues.cloudera.****
    org/browse/IMPALA-160 <https://issues.cloudera.org/browse/IMPALA-160>,
    please look at output of "stdout" and see if the error matches.

    Vikas

    On Thu, Aug 8, 2013 at 11:13 AM, Andy wrote:

    Hi Vikas,

    The output is:
    *$ /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
    /statestored*
    *Illegal instruction (core dumped)*

    *$ ldd /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-reta
    il/statestored*
    * linux-vdso.so.1 => (0x00007fff7413c000)*
    * libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
    (0x00007f5c3a8e6000)*
    * libhdfs.so.0.0.0 => not found*
    * librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5c3a6dd000)*
    * libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c3a4d9000)*
    * libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
    (0x00007f5c3a27d000)*
    * libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    (0x00007f5c39f7c000)*
    * libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c39c80000)*
    * libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
    (0x00007f5c39a6a000)*
    * libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c396aa000)*
    * /lib64/ld-linux-x86-64.so.2 (0x00007f5c3ab0a000)*
    * libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    (0x00007f5c392e2000)*
    * libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c390ca000)*
    *
    *
    *$ locate libhdfs.so.0.0.0*
    */opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/libhdfs.so.0.0.0
    *
    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/lib/libhd
    fs.so.0.0.0*

    Thanks,
    Andy
    On Friday, August 9, 2013 1:39:06 AM UTC+8, vikas wrote:

    Hi Andy,

    So either the binaries are terminating very early (even before
    opening a log), or we are not able to exec them (some dependent libraries
    are missing). What happens if you run them command directly from the
    commandline:

    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
    /statestored*
    *
    *
    *Try using ldd on **/opt/cloudera/parcels/IMPAL
    A-1.1-1.p0.8/lib/impala/sbin-retail/statestored to see if it shows
    up something.*
    *
    *
    *- Vikas*

    On Thu, Aug 8, 2013 at 10:18 AM, Andy wrote:

    Hi Vikas,

    There's nothing in */var/log/impalad *and */var/log/statestore *directories.
    This is their permission:
    *
    *
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:05
    impalad/*
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:16
    statestore/*

    Thanks,
    Andy
    On Thursday, August 8, 2013 11:44:57 PM UTC+8, vikas wrote:

    Hi Andy,

    Can you get the impalad and statestore log and provide it to us.
    The stderr/stdout log is from the bash file that launches
    impalad/statestore. The log from impalad/statestore will tell us more on
    why we can't start these daemons.

    Thanks,
    Vikas

    On Thu, Aug 8, 2013 at 8:17 AM, Andy wrote:

    Hi all,

    I install Impala on Ubuntu server 12.04 using Cloudera
    Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.

    HDFS, Hive, MapReduce services work fine, the status is Good
    health. But I can't start Impala, the error is:

    *Supervisor returned FATAL. Please check the role log file,
    stderr, or stdout.*
    *
    *
    The error log is:

    *+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-I
    MPALAD/hadoop-conf*
    *+ export HIVE_CONF_DIR=/run/cloudera-sc
    m-agent/process/208-impala-IMPALAD/hive-conf*
    *+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMP
    ALAD/hive-conf*
    *+ export HBASE_CONF_DIR=/run/cloudera-s
    cm-agent/process/208-impala-IMPALAD/hbase-conf*
    *+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IM
    PALAD/hbase-conf*
    *+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/shar
    e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
    e-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export AUX_CLASSPATH=/usr/share/java/
    mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-8
    01.jdbc4.jar:/usr/share/java/oracle-connector-java.jar*
    *+ AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/o
    racle-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
    e-connector-java.jar*
    *+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/shar
    e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
    e-connector-java.jar*
    *+ FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-IMPALAD
    /impala-conf/impalad_flags*
    *+ USE_DEBUG_BUILD=false*
    *+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloud
    era-scm-agent/process/208-impala-IMPALAD#g'
    /run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-co
    nf/impalad_flags*
    *+ '[' -f /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/.htpasswd ']'*
    *+ chmod 600 /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/.htpasswd*
    *+ false*
    *+ export IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
    impala/sbin-retail*
    *+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impal
    a/sbin-retail*
    *+ '[' impalad = impalad ']'*
    *+ exec /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../
    bin/impalad --flagfile=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/impala-conf/impalad_flags*
    *
    *
    Are there anyone know how to fix this problem? Thank you.
    *
    *
  • Andy at Aug 8, 2013 at 8:30 pm
    Hi Vikas,

    There's nothing in '*/var/crash'* directory:
    *$ /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/statestored*
    *Illegal instruction (core dumped)*
    *$ ls /var/crash/*
    *$ *
    *
    *
    I can't change the value of '*/proc/sys/kernel/core_pattern*':
    *$ sudo echo core > /proc/sys/kernel/core_pattern *
    *-bash: /proc/sys/kernel/core_pattern: Permission denied*

    Andy.
    On Friday, August 9, 2013 4:14:25 AM UTC+8, vikas wrote:

    Hi Andy,

    So you have Ubuntu apport enabled to capture dumps. The crash file may be
    in '/var/crash'.

    If you can't find the core there, try setting core_pattern to simple
    'core' (echo core > /proc/sys/kernel/core_pattern) and then it should dump
    the core in the current working directory.

    Vikas


    On Thu, Aug 8, 2013 at 12:59 PM, Andy <nguyend...@gmail.com <javascript:>>wrote:
    Hi Vikas,

    1. The value of */proc/sys/kernel/core_pattern *is:
    *|/usr/share/apport/apport %p %s %c*
    *
    *
    2. Thanks, I'll try CentOS.

    Andy.
    On Friday, August 9, 2013 3:16:46 AM UTC+8, vikas wrote:

    Hi Andy,

    1. Unfortunately there is no core present. By default the core get
    generated in the current working directory of the process. What is the
    value of /proc/sys/kernel/core_**pattern on your setup?
    2. To run retail build, if possible try using some other os than Ubuntu
    12.04 till we get this issue sorted out.

    - Vikas

    On Thu, Aug 8, 2013 at 11:36 AM, Andy wrote:

    Hi Vikas,

    1. I already did as you guided. The directory is *210-impala-STATESTORE.
    *Its content is in attach file
    *$ sudo ls -l /run/cloudera-scm-agent/process/*
    *total 0*
    *drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28
    188-hdfs-SECONDARYNAMENODE*
    *drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28 191-hdfs-DATANODE*
    *drwxr-x--x 3 hdfs hdfs 360 Aug 8 06:28 192-hdfs-NAMENODE*
    *drwxr-x--x 3 mapred hadoop 380 Aug 8 06:28
    193-mapreduce-JOBTRACKER*
    *drwxr-x--x 3 mapred hadoop 240 Aug 8 06:28
    196-mapreduce-TASKTRACKER*
    *drwxr-x--x 3 zookeeper zookeeper 220 Aug 8 06:28 197-zookeeper-server
    *
    *drwxr-x--x 4 hive hive 160 Aug 8 06:28
    198-hive-HIVEMETASTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 07:56 199-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 07:57
    202-impala-STATESTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 07:59 203-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 07:59
    206-impala-STATESTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 08:13 207-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 08:13
    210-impala-STATESTORE*

    2. It works! How can I run without using Debug Build?

    Thanks,
    Andy
    On Friday, August 9, 2013 2:15:14 AM UTC+8, vikas wrote:

    Thanks Andy, the libraries seem to be ok. CM adds the libhdfs to the
    LD_LIBRARY_PATH so that is fine.

    The core dump part is interesting. There are few things we can try:

    1. Collect the core dump for debugging. In the CM UI, got to Impala
    configuration -> Service Wide -> Enable core dump. Check the check-box and
    save the configuration. Then restart the statestore and this time it will
    core dump, the dump should go to its cwd, which will be under '
    /run/cloudera-scm-agent/**proce**ss/' directory (the latest created
    directory with STATESTORE in its name). Please see if you have a
    hs_err_<pid>.log there too. Please send us these.

    2. To get things working, try running debug build. You can go to CM
    console and select the Impala configuration option -> Sevice Wide -> Use
    Debug Build. Check this option and save configuration. Try restarting
    service now and see if it helps.

    You may be hitting this: https://issues.cloudera.****
    org/browse/IMPALA-160 <https://issues.cloudera.org/browse/IMPALA-160>,
    please look at output of "stdout" and see if the error matches.

    Vikas

    On Thu, Aug 8, 2013 at 11:13 AM, Andy wrote:

    Hi Vikas,

    The output is:
    *$ /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
    /statestored*
    *Illegal instruction (core dumped)*

    *$ ldd /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-reta
    il/statestored*
    * linux-vdso.so.1 => (0x00007fff7413c000)*
    * libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
    (0x00007f5c3a8e6000)*
    * libhdfs.so.0.0.0 => not found*
    * librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5c3a6dd000)
    *
    * libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c3a4d9000)
    *
    * libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
    (0x00007f5c3a27d000)*
    * libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    (0x00007f5c39f7c000)*
    * libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c39c80000)*
    * libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
    (0x00007f5c39a6a000)*
    * libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c396aa000)*
    * /lib64/ld-linux-x86-64.so.2 (0x00007f5c3ab0a000)*
    * libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    (0x00007f5c392e2000)*
    * libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c390ca000)*
    *
    *
    *$ locate libhdfs.so.0.0.0*
    */opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/libhdfs
    .so.0.0.0*
    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/lib/libhd
    fs.so.0.0.0*

    Thanks,
    Andy
    On Friday, August 9, 2013 1:39:06 AM UTC+8, vikas wrote:

    Hi Andy,

    So either the binaries are terminating very early (even before
    opening a log), or we are not able to exec them (some dependent libraries
    are missing). What happens if you run them command directly from the
    commandline:

    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
    /statestored*
    *
    *
    *Try using ldd on **/opt/cloudera/parcels/IMPAL
    A-1.1-1.p0.8/lib/impala/sbin-retail/statestored to see if it shows
    up something.*
    *
    *
    *- Vikas*

    On Thu, Aug 8, 2013 at 10:18 AM, Andy wrote:

    Hi Vikas,

    There's nothing in */var/log/impalad *and */var/log/statestore *directories.
    This is their permission:
    *
    *
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:05
    impalad/*
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:16
    statestore/*

    Thanks,
    Andy
    On Thursday, August 8, 2013 11:44:57 PM UTC+8, vikas wrote:

    Hi Andy,

    Can you get the impalad and statestore log and provide it to us.
    The stderr/stdout log is from the bash file that launches
    impalad/statestore. The log from impalad/statestore will tell us more on
    why we can't start these daemons.

    Thanks,
    Vikas

    On Thu, Aug 8, 2013 at 8:17 AM, Andy wrote:

    Hi all,

    I install Impala on Ubuntu server 12.04 using Cloudera
    Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.

    HDFS, Hive, MapReduce services work fine, the status is Good
    health. But I can't start Impala, the error is:

    *Supervisor returned FATAL. Please check the role log file,
    stderr, or stdout.*
    *
    *
    The error log is:

    *+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-I
    MPALAD/hadoop-conf*
    *+ export HIVE_CONF_DIR=/run/cloudera-sc
    m-agent/process/208-impala-IMPALAD/hive-conf*
    *+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMP
    ALAD/hive-conf*
    *+ export HBASE_CONF_DIR=/run/cloudera-s
    cm-agent/process/208-impala-IMPALAD/hbase-conf*
    *+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IM
    PALAD/hbase-conf*
    *+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/shar
    e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
    e-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export AUX_CLASSPATH=/usr/share/java/
    mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-8
    01.jdbc4.jar:/usr/share/java/oracle-connector-java.jar*
    *+ AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/o
    racle-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export CLASSPATH=/usr/share/java/mysq
    l-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.j
    dbc4.jar:/usr/share/java/oracle-connector-java.jar*
    *+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/shar
    e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
    e-connector-java.jar*
    *+ FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-IMPALAD
    /impala-conf/impalad_flags*
    *+ USE_DEBUG_BUILD=false*
    *+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloud
    era-scm-agent/process/208-impala-IMPALAD#g'
    /run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-co
    nf/impalad_flags*
    *+ '[' -f /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/.htpasswd ']'*
    *+ chmod 600 /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/.htpasswd*
    *+ false*
    *+ export IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
    impala/sbin-retail*
    *+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impal
    a/sbin-retail*
    *+ '[' impalad = impalad ']'*
    *+ exec /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../
    bin/impalad --flagfile=/run/cloudera-scm-a
    gent/process/208-impala-IMPALAD/impala-conf/impalad_flags*
    *
    *
    Are there anyone know how to fix this problem? Thank you.
    *
    *
  • Vikas Singh at Aug 9, 2013 at 4:16 pm
    Hi Andy,

    Try logging in as root and edit the file (sudo su -l root followed by
    editing it using vi or echo core > */proc/sys/kernel/core_pattern*). Then
    try to dump core of a process to confirm the setting. As an example
    following dumps core of the 'tail' process I start:

    $ ulimit -c unlimited # this enables core dump
    $ tail -f <any_file_name> & # start tail in background, pid of process
    will be printed on first line
    $ kill -s SIGABRT <tail_pid>

    Now the current directory contains the core. Once this work, please give it
    a try with Impala process (You may have to stop the apport service as it
    may override the core_pattern setting that you modify).

    - Vikas


    On Thu, Aug 8, 2013 at 1:30 PM, Andy wrote:

    Hi Vikas,

    There's nothing in '*/var/crash'* directory:
    *$
    /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/statestored*
    *Illegal instruction (core dumped)*
    *$ ls /var/crash/*
    *$ *
    *
    *
    I can't change the value of '*/proc/sys/kernel/core_pattern*':
    *$ sudo echo core > /proc/sys/kernel/core_pattern *
    *-bash: /proc/sys/kernel/core_pattern: Permission denied*

    Andy.
    On Friday, August 9, 2013 4:14:25 AM UTC+8, vikas wrote:

    Hi Andy,

    So you have Ubuntu apport enabled to capture dumps. The crash file may be
    in '/var/crash'.

    If you can't find the core there, try setting core_pattern to simple
    'core' (echo core > /proc/sys/kernel/core_pattern) and then it should dump
    the core in the current working directory.

    Vikas

    On Thu, Aug 8, 2013 at 12:59 PM, Andy wrote:

    Hi Vikas,

    1. The value of */proc/sys/kernel/core_pattern *is:
    *|/usr/share/apport/apport %p %s %c*
    *
    *
    2. Thanks, I'll try CentOS.

    Andy.
    On Friday, August 9, 2013 3:16:46 AM UTC+8, vikas wrote:

    Hi Andy,

    1. Unfortunately there is no core present. By default the core get
    generated in the current working directory of the process. What is the
    value of /proc/sys/kernel/core_**patte**rn on your setup?
    2. To run retail build, if possible try using some other os than Ubuntu
    12.04 till we get this issue sorted out.

    - Vikas

    On Thu, Aug 8, 2013 at 11:36 AM, Andy wrote:

    Hi Vikas,

    1. I already did as you guided. The directory is *210-impala-STATESTORE.
    *Its content is in attach file
    *$ sudo ls -l /run/cloudera-scm-agent/process/*
    *total 0*
    *drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28
    188-hdfs-SECONDARYNAMENODE*
    *drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28 191-hdfs-DATANODE*
    *drwxr-x--x 3 hdfs hdfs 360 Aug 8 06:28 192-hdfs-NAMENODE*
    *drwxr-x--x 3 mapred hadoop 380 Aug 8 06:28
    193-mapreduce-JOBTRACKER*
    *drwxr-x--x 3 mapred hadoop 240 Aug 8 06:28
    196-mapreduce-TASKTRACKER*
    *drwxr-x--x 3 zookeeper zookeeper 220 Aug 8 06:28
    197-zookeeper-server*
    *drwxr-x--x 4 hive hive 160 Aug 8 06:28
    198-hive-HIVEMETASTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 07:56 199-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 07:57
    202-impala-STATESTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 07:59 203-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 07:59
    206-impala-STATESTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 08:13 207-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 08:13
    210-impala-STATESTORE*

    2. It works! How can I run without using Debug Build?

    Thanks,
    Andy
    On Friday, August 9, 2013 2:15:14 AM UTC+8, vikas wrote:

    Thanks Andy, the libraries seem to be ok. CM adds the libhdfs to the
    LD_LIBRARY_PATH so that is fine.

    The core dump part is interesting. There are few things we can try:

    1. Collect the core dump for debugging. In the CM UI, got to Impala
    configuration -> Service Wide -> Enable core dump. Check the check-box and
    save the configuration. Then restart the statestore and this time it will
    core dump, the dump should go to its cwd, which will be under '
    /run/cloudera-scm-agent/**proce****ss/' directory (the latest
    created directory with STATESTORE in its name). Please see if you have a
    hs_err_<pid>.log there too. Please send us these.

    2. To get things working, try running debug build. You can go to CM
    console and select the Impala configuration option -> Sevice Wide -> Use
    Debug Build. Check this option and save configuration. Try restarting
    service now and see if it helps.

    You may be hitting this: https://issues.cloudera.******
    org/browse/IMPALA-160 <https://issues.cloudera.org/browse/IMPALA-160>,
    please look at output of "stdout" and see if the error matches.

    Vikas

    On Thu, Aug 8, 2013 at 11:13 AM, Andy wrote:

    Hi Vikas,

    The output is:
    *$ /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
    /statestored*
    *Illegal instruction (core dumped)*

    *$ ldd /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-reta
    il/statestored*
    * linux-vdso.so.1 => (0x00007fff7413c000)*
    * libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
    (0x00007f5c3a8e6000)*
    * libhdfs.so.0.0.0 => not found*
    * librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
    (0x00007f5c3a6dd000)*
    * libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
    (0x00007f5c3a4d9000)*
    * libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
    (0x00007f5c3a27d000)*
    * libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    (0x00007f5c39f7c000)*
    * libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c39c80000)*
    * libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
    (0x00007f5c39a6a000)*
    * libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c396aa000)*
    * /lib64/ld-linux-x86-64.so.2 (0x00007f5c3ab0a000)*
    * libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    (0x00007f5c392e2000)*
    * libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c390ca000)*
    *
    *
    *$ locate libhdfs.so.0.0.0*
    */opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/libhdfs
    .so.0.0.0*
    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/lib/libhd
    fs.so.0.0.0*

    Thanks,
    Andy
    On Friday, August 9, 2013 1:39:06 AM UTC+8, vikas wrote:

    Hi Andy,

    So either the binaries are terminating very early (even before
    opening a log), or we are not able to exec them (some dependent libraries
    are missing). What happens if you run them command directly from the
    commandline:

    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
    /statestored*
    *
    *
    *Try using ldd on **/opt/cloudera/parcels/IMPAL
    A-1.1-1.p0.8/lib/impala/sbin-retail/statestored to see if it shows
    up something.*
    *
    *
    *- Vikas*

    On Thu, Aug 8, 2013 at 10:18 AM, Andy wrote:

    Hi Vikas,

    There's nothing in */var/log/impalad *and */var/log/statestore *directories.
    This is their permission:
    *
    *
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:05
    impalad/*
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:16
    statestore/*

    Thanks,
    Andy
    On Thursday, August 8, 2013 11:44:57 PM UTC+8, vikas wrote:

    Hi Andy,

    Can you get the impalad and statestore log and provide it to us.
    The stderr/stdout log is from the bash file that launches
    impalad/statestore. The log from impalad/statestore will tell us more on
    why we can't start these daemons.

    Thanks,
    Vikas

    On Thu, Aug 8, 2013 at 8:17 AM, Andy wrote:

    Hi all,

    I install Impala on Ubuntu server 12.04 using Cloudera
    Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.

    HDFS, Hive, MapReduce services work fine, the status is Good
    health. But I can't start Impala, the error is:

    *Supervisor returned FATAL. Please check the role log file,
    stderr, or stdout.*
    *
    *
    The error log is:

    *+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-I
    MPALAD/hadoop-conf*
    *+ export HIVE_CONF_DIR=/run/cloudera-sc
    m-agent/process/208-impala-IMPALAD/hive-conf*
    *+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMP
    ALAD/hive-conf*
    *+ export HBASE_CONF_DIR=/run/cloudera-s
    cm-agent/process/208-impala-IMPALAD/hbase-conf*
    *+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IM
    PALAD/hbase-conf*
    *+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/shar
    e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
    e-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export AUX_CLASSPATH=/usr/share/java/
    mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-8
    01.jdbc4.jar:/usr/share/java/oracle-connector-java.jar*
    *+ AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/o
    racle-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export CLASSPATH=/usr/share/java/mysq
    l-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.j
    dbc4.jar:/usr/share/java/oracle-connector-java.jar*
    *+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/shar
    e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
    e-connector-java.jar*
    *+ FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-IMPALAD
    /impala-conf/impalad_flags*
    *+ USE_DEBUG_BUILD=false*
    *+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloud
    era-scm-agent/process/208-impala-IMPALAD#g'
    /run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-co
    nf/impalad_flags*
    *+ '[' -f /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/.htpasswd ']'*
    *+ chmod 600 /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/.htpasswd*
    *+ false*
    *+ export IMPALA_BIN=/opt/cloudera/parce
    ls/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail*
    *+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impal
    a/sbin-retail*
    *+ '[' impalad = impalad ']'*
    *+ exec /opt/cloudera/parcels/IMPALA-1
    .1-1.p0.8/lib/impala/../../bin/impalad
    --flagfile=/run/cloudera-scm-agent/process/208-impala-IMPALA
    D/impala-conf/impalad_flags*
    *
    *
    Are there anyone know how to fix this problem? Thank you.
    *
    *
  • Kieran Evans at Sep 4, 2013 at 4:45 pm
    I'm also seeing this issue, with IMPALA-1.1.1-1.p0.17.

    It works fine with debugging enabled, but not without.

    Attached is the folder containing the core dump.

    /Kieran

    On Sunday, 11 August 2013 04:09:20 UTC+1, Andy wrote:

    Hi Vikas,

    I did it. The attached file is the directory *
    /run/cloudera-scm-agent/process/211-impala-STATESTORE/*
    *
    *
    Andy.
    On Saturday, August 10, 2013 12:16:35 AM UTC+8, vikas wrote:

    Hi Andy,

    Try logging in as root and edit the file (sudo su -l root followed by
    editing it using vi or echo core > */proc/sys/kernel/core_pattern*).
    Then try to dump core of a process to confirm the setting. As an example
    following dumps core of the 'tail' process I start:

    $ ulimit -c unlimited # this enables core dump
    $ tail -f <any_file_name> & # start tail in background, pid of process
    will be printed on first line
    $ kill -s SIGABRT <tail_pid>

    Now the current directory contains the core. Once this work, please give
    it a try with Impala process (You may have to stop the apport service as it
    may override the core_pattern setting that you modify).

    - Vikas


    On Thu, Aug 8, 2013 at 1:30 PM, Andy wrote:

    Hi Vikas,

    There's nothing in '*/var/crash'* directory:
    *$
    /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/statestored
    *
    *Illegal instruction (core dumped)*
    *$ ls /var/crash/*
    *$ *
    *
    *
    I can't change the value of '*/proc/sys/kernel/core_pattern*':
    *$ sudo echo core > /proc/sys/kernel/core_pattern *
    *-bash: /proc/sys/kernel/core_pattern: Permission denied*

    Andy.
    On Friday, August 9, 2013 4:14:25 AM UTC+8, vikas wrote:

    Hi Andy,

    So you have Ubuntu apport enabled to capture dumps. The crash file may
    be in '/var/crash'.

    If you can't find the core there, try setting core_pattern to simple
    'core' (echo core > /proc/sys/kernel/core_pattern) and then it should dump
    the core in the current working directory.

    Vikas

    On Thu, Aug 8, 2013 at 12:59 PM, Andy wrote:

    Hi Vikas,

    1. The value of */proc/sys/kernel/core_pattern *is:
    *|/usr/share/apport/apport %p %s %c*
    *
    *
    2. Thanks, I'll try CentOS.

    Andy.
    On Friday, August 9, 2013 3:16:46 AM UTC+8, vikas wrote:

    Hi Andy,

    1. Unfortunately there is no core present. By default the core get
    generated in the current working directory of the process. What is the
    value of /proc/sys/kernel/core_**patte**rn on your setup?
    2. To run retail build, if possible try using some other os than
    Ubuntu 12.04 till we get this issue sorted out.

    - Vikas

    On Thu, Aug 8, 2013 at 11:36 AM, Andy wrote:

    Hi Vikas,

    1. I already did as you guided. The directory is *210-impala-STATESTORE.
    *Its content is in attach file
    *$ sudo ls -l /run/cloudera-scm-agent/process/*
    *total 0*
    *drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28
    188-hdfs-SECONDARYNAMENODE*
    *drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28 191-hdfs-DATANODE
    *
    *drwxr-x--x 3 hdfs hdfs 360 Aug 8 06:28 192-hdfs-NAMENODE
    *
    *drwxr-x--x 3 mapred hadoop 380 Aug 8 06:28
    193-mapreduce-JOBTRACKER*
    *drwxr-x--x 3 mapred hadoop 240 Aug 8 06:28
    196-mapreduce-TASKTRACKER*
    *drwxr-x--x 3 zookeeper zookeeper 220 Aug 8 06:28
    197-zookeeper-server*
    *drwxr-x--x 4 hive hive 160 Aug 8 06:28
    198-hive-HIVEMETASTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 07:56
    199-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 07:57
    202-impala-STATESTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 07:59
    203-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 07:59
    206-impala-STATESTORE*
    *drwxr-x--x 6 impala impala 180 Aug 8 08:13
    207-impala-IMPALAD*
    *drwxr-x--x 4 impala impala 140 Aug 8 08:13
    210-impala-STATESTORE*

    2. It works! How can I run without using Debug Build?

    Thanks,
    Andy
    On Friday, August 9, 2013 2:15:14 AM UTC+8, vikas wrote:

    Thanks Andy, the libraries seem to be ok. CM adds the libhdfs to
    the LD_LIBRARY_PATH so that is fine.

    The core dump part is interesting. There are few things we can try:

    1. Collect the core dump for debugging. In the CM UI, got to Impala
    configuration -> Service Wide -> Enable core dump. Check the check-box and
    save the configuration. Then restart the statestore and this time it will
    core dump, the dump should go to its cwd, which will be under '
    /run/cloudera-scm-agent/**proce****ss/' directory (the latest
    created directory with STATESTORE in its name). Please see if you have a
    hs_err_<pid>.log there too. Please send us these.

    2. To get things working, try running debug build. You can go to CM
    console and select the Impala configuration option -> Sevice Wide -> Use
    Debug Build. Check this option and save configuration. Try restarting
    service now and see if it helps.

    You may be hitting this: https://issues.cloudera.******
    org/browse/IMPALA-160<https://issues.cloudera.org/browse/IMPALA-160>,
    please look at output of "stdout" and see if the error matches.

    Vikas

    On Thu, Aug 8, 2013 at 11:13 AM, Andy wrote:

    Hi Vikas,

    The output is:
    *$ /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
    /statestored*
    *Illegal instruction (core dumped)*

    *$ ldd /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-
    retail/statestored*
    * linux-vdso.so.1 => (0x00007fff7413c000)*
    * libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
    (0x00007f5c3a8e6000)*
    * libhdfs.so.0.0.0 => not found*
    * librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
    (0x00007f5c3a6dd000)*
    * libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
    (0x00007f5c3a4d9000)*
    * libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
    (0x00007f5c3a27d000)*
    * libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    (0x00007f5c39f7c000)*
    * libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
    (0x00007f5c39c80000)*
    * libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
    (0x00007f5c39a6a000)*
    * libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
    (0x00007f5c396aa000)*
    * /lib64/ld-linux-x86-64.so.2 (0x00007f5c3ab0a000)*
    * libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    (0x00007f5c392e2000)*
    * libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1
    (0x00007f5c390ca000)*
    *
    *
    *$ locate libhdfs.so.0.0.0*
    */opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/libhdfs
    .so.0.0.0*
    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/lib/libhd
    fs.so.0.0.0*

    Thanks,
    Andy
    On Friday, August 9, 2013 1:39:06 AM UTC+8, vikas wrote:

    Hi Andy,

    So either the binaries are terminating very early (even before
    opening a log), or we are not able to exec them (some dependent libraries
    are missing). What happens if you run them command directly from the
    commandline:

    */opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
    /statestored*
    *
    *
    *Try using ldd on **/opt/cloudera/parcels/IMPAL
    A-1.1-1.p0.8/lib/impala/sbin-retail/statestored to see if it
    shows up something.*
    *
    *
    *- Vikas*

    On Thu, Aug 8, 2013 at 10:18 AM, Andy wrote:

    Hi Vikas,

    There's nothing in */var/log/impalad *and */var/log/statestore *directories.
    This is their permission:
    *
    *
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:05
    impalad/*
    *drwxr-xr-x 2 impala impala 4096 Aug 4 12:16
    statestore/*

    Thanks,
    Andy
    On Thursday, August 8, 2013 11:44:57 PM UTC+8, vikas wrote:

    Hi Andy,

    Can you get the impalad and statestore log and provide it to
    us. The stderr/stdout log is from the bash file that launches
    impalad/statestore. The log from impalad/statestore will tell us more on
    why we can't start these daemons.

    Thanks,
    Vikas

    On Thu, Aug 8, 2013 at 8:17 AM, Andy wrote:

    Hi all,

    I install Impala on Ubuntu server 12.04 using Cloudera
    Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.

    HDFS, Hive, MapReduce services work fine, the status is Good
    health. But I can't start Impala, the error is:

    *Supervisor returned FATAL. Please check the role log file,
    stderr, or stdout.*
    *
    *
    The error log is:

    *+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hadoop-conf*
    *+ export HIVE_CONF_DIR=/run/cloudera-sc
    m-agent/process/208-impala-IMPALAD/hive-conf*
    *+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hive-conf*
    *+ export HBASE_CONF_DIR=/run/cloudera-s
    cm-agent/process/208-impala-IMPALAD/hbase-conf*
    *+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/hbase-conf*
    *+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export AUX_CLASSPATH=/usr/share/java/
    mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-8
    01.jdbc4.jar:/usr/share/java/oracle-connector-java.jar*
    *+ AUX_CLASSPATH=/usr/share/java/
    mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-8
    01.jdbc4.jar:/usr/share/java/oracle-connector-java.jar*
    *+ [[ -z '' ]]*
    *+ export CLASSPATH=/usr/share/java/mysq
    l-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.j
    dbc4.jar:/usr/share/java/oracle-connector-java.jar*
    *+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
    share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
    oracle-connector-java.jar*
    *+ FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-
    IMPALAD/impala-conf/impalad_flags*
    *+ USE_DEBUG_BUILD=false*
    *+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloud
    era-scm-agent/process/208-impala-IMPALAD#g'
    /run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-co
    nf/impalad_flags*
    *+ '[' -f /run/cloudera-scm-agent/process/208-impala-IMPALAD/
    impala-conf/.htpasswd ']'*
    *+ chmod 600 /run/cloudera-scm-agent/proces
    s/208-impala-IMPALAD/impala-conf/.htpasswd*
    *+ false*
    *+ export IMPALA_BIN=/opt/cloudera/parce
    ls/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail*
    *+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
    impala/sbin-retail*
    *+ '[' impalad = impalad ']'*
    *+ exec /opt/cloudera/parcels/IMPALA-1
    .1-1.p0.8/lib/impala/../../bin/impalad
    --flagfile=/run/cloudera-scm-agent/process/208-impala-IMPALA
    D/impala-conf/impalad_flags*
    *
    *
    Are there anyone know how to fix this problem? Thank you.
    *
    *
    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimpala-user @
categorieshadoop
postedAug 8, '13 at 3:17p
activeSep 4, '13 at 4:45p
posts12
users3
websitecloudera.com
irc#hadoop

3 users in discussion

Andy: 6 posts Vikas Singh: 5 posts Kieran Evans: 1 post

People

Translate

site design / logo © 2022 Grokbase