1. Unfortunately there is no core present. By default the core get
generated in the current working directory of the process. What is the
value of /proc/sys/kernel/core_pattern on your setup?
2. To run retail build, if possible try using some other os than Ubuntu
12.04 till we get this issue sorted out.
- Vikas
On Thu, Aug 8, 2013 at 11:36 AM, Andy wrote:
Hi Vikas,
1. I already did as you guided. The directory is *210-impala-STATESTORE. *Its
content is in attach file
*$ sudo ls -l /run/cloudera-scm-agent/process/*
*total 0*
*drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28
188-hdfs-SECONDARYNAMENODE*
*drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28 191-hdfs-DATANODE*
*drwxr-x--x 3 hdfs hdfs 360 Aug 8 06:28 192-hdfs-NAMENODE*
*drwxr-x--x 3 mapred hadoop 380 Aug 8 06:28
193-mapreduce-JOBTRACKER*
*drwxr-x--x 3 mapred hadoop 240 Aug 8 06:28
196-mapreduce-TASKTRACKER*
*drwxr-x--x 3 zookeeper zookeeper 220 Aug 8 06:28 197-zookeeper-server*
*drwxr-x--x 4 hive hive 160 Aug 8 06:28 198-hive-HIVEMETASTORE*
*drwxr-x--x 6 impala impala 180 Aug 8 07:56 199-impala-IMPALAD*
*drwxr-x--x 4 impala impala 140 Aug 8 07:57 202-impala-STATESTORE*
*drwxr-x--x 6 impala impala 180 Aug 8 07:59 203-impala-IMPALAD*
*drwxr-x--x 4 impala impala 140 Aug 8 07:59 206-impala-STATESTORE*
*drwxr-x--x 6 impala impala 180 Aug 8 08:13 207-impala-IMPALAD*
*drwxr-x--x 4 impala impala 140 Aug 8 08:13 210-impala-STATESTORE*
2. It works! How can I run without using Debug Build?
Thanks,
Andy
Hi Vikas,
1. I already did as you guided. The directory is *210-impala-STATESTORE. *Its
content is in attach file
*$ sudo ls -l /run/cloudera-scm-agent/process/*
*total 0*
*drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28
188-hdfs-SECONDARYNAMENODE*
*drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28 191-hdfs-DATANODE*
*drwxr-x--x 3 hdfs hdfs 360 Aug 8 06:28 192-hdfs-NAMENODE*
*drwxr-x--x 3 mapred hadoop 380 Aug 8 06:28
193-mapreduce-JOBTRACKER*
*drwxr-x--x 3 mapred hadoop 240 Aug 8 06:28
196-mapreduce-TASKTRACKER*
*drwxr-x--x 3 zookeeper zookeeper 220 Aug 8 06:28 197-zookeeper-server*
*drwxr-x--x 4 hive hive 160 Aug 8 06:28 198-hive-HIVEMETASTORE*
*drwxr-x--x 6 impala impala 180 Aug 8 07:56 199-impala-IMPALAD*
*drwxr-x--x 4 impala impala 140 Aug 8 07:57 202-impala-STATESTORE*
*drwxr-x--x 6 impala impala 180 Aug 8 07:59 203-impala-IMPALAD*
*drwxr-x--x 4 impala impala 140 Aug 8 07:59 206-impala-STATESTORE*
*drwxr-x--x 6 impala impala 180 Aug 8 08:13 207-impala-IMPALAD*
*drwxr-x--x 4 impala impala 140 Aug 8 08:13 210-impala-STATESTORE*
2. It works! How can I run without using Debug Build?
Thanks,
Andy
On Friday, August 9, 2013 2:15:14 AM UTC+8, vikas wrote:
Thanks Andy, the libraries seem to be ok. CM adds the libhdfs to the
LD_LIBRARY_PATH so that is fine.
The core dump part is interesting. There are few things we can try:
1. Collect the core dump for debugging. In the CM UI, got to Impala
configuration -> Service Wide -> Enable core dump. Check the check-box and
save the configuration. Then restart the statestore and this time it will
core dump, the dump should go to its cwd, which will be under '
/run/cloudera-scm-agent/**process/' directory (the latest created
directory with STATESTORE in its name). Please see if you have a
hs_err_<pid>.log there too. Please send us these.
2. To get things working, try running debug build. You can go to CM
console and select the Impala configuration option -> Sevice Wide -> Use
Debug Build. Check this option and save configuration. Try restarting
service now and see if it helps.
You may be hitting this: https://issues.cloudera.**org/browse/IMPALA-160<https://issues.cloudera.org/browse/IMPALA-160>,
please look at output of "stdout" and see if the error matches.
Vikas
Thanks Andy, the libraries seem to be ok. CM adds the libhdfs to the
LD_LIBRARY_PATH so that is fine.
The core dump part is interesting. There are few things we can try:
1. Collect the core dump for debugging. In the CM UI, got to Impala
configuration -> Service Wide -> Enable core dump. Check the check-box and
save the configuration. Then restart the statestore and this time it will
core dump, the dump should go to its cwd, which will be under '
/run/cloudera-scm-agent/**process/' directory (the latest created
directory with STATESTORE in its name). Please see if you have a
hs_err_<pid>.log there too. Please send us these.
2. To get things working, try running debug build. You can go to CM
console and select the Impala configuration option -> Sevice Wide -> Use
Debug Build. Check this option and save configuration. Try restarting
service now and see if it helps.
You may be hitting this: https://issues.cloudera.**org/browse/IMPALA-160<https://issues.cloudera.org/browse/IMPALA-160>,
please look at output of "stdout" and see if the error matches.
Vikas
On Thu, Aug 8, 2013 at 11:13 AM, Andy wrote:
Hi Vikas,
The output is:
*$ /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../
bin/statestored*
*Illegal instruction (core dumped)*
*$ ldd /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-
retail/statestored*
* linux-vdso.so.1 => (0x00007fff7413c000)*
* libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f5c3a8e6000)*
* libhdfs.so.0.0.0 => not found*
* librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5c3a6dd000)*
* libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c3a4d9000)*
* libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
(0x00007f5c3a27d000)*
* libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(0x00007f5c39f7c000)*
* libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c39c80000)*
* libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x00007f5c39a6a000)*
* libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c396aa000)*
* /lib64/ld-linux-x86-64.so.2 (0x00007f5c3ab0a000)*
* libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(0x00007f5c392e2000)*
* libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c390ca000)*
*
*
*$ locate libhdfs.so.0.0.0*
*/opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/libhdfs.so.0.0.0*
*/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/lib/libhdfs.so.0.0.0
*
Thanks,
Andy
Hi Vikas,
The output is:
*$ /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../
bin/statestored*
*Illegal instruction (core dumped)*
*$ ldd /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-
retail/statestored*
* linux-vdso.so.1 => (0x00007fff7413c000)*
* libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f5c3a8e6000)*
* libhdfs.so.0.0.0 => not found*
* librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5c3a6dd000)*
* libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c3a4d9000)*
* libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
(0x00007f5c3a27d000)*
* libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(0x00007f5c39f7c000)*
* libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c39c80000)*
* libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x00007f5c39a6a000)*
* libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c396aa000)*
* /lib64/ld-linux-x86-64.so.2 (0x00007f5c3ab0a000)*
* libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(0x00007f5c392e2000)*
* libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c390ca000)*
*
*
*$ locate libhdfs.so.0.0.0*
*/opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/libhdfs.so.0.0.0*
*/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/lib/libhdfs.so.0.0.0
*
Thanks,
Andy
On Friday, August 9, 2013 1:39:06 AM UTC+8, vikas wrote:
Hi Andy,
So either the binaries are terminating very early (even before opening
a log), or we are not able to exec them (some dependent libraries are
missing). What happens if you run them command directly from the
commandline:
*/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
/statestored*
*
*
*Try using ldd on **/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/
sbin-retail/statestored to see if it shows up something.*
*
*
*- Vikas*
Hi Andy,
So either the binaries are terminating very early (even before opening
a log), or we are not able to exec them (some dependent libraries are
missing). What happens if you run them command directly from the
commandline:
*/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
/statestored*
*
*
*Try using ldd on **/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/
sbin-retail/statestored to see if it shows up something.*
*
*
*- Vikas*
On Thu, Aug 8, 2013 at 10:18 AM, Andy wrote:
Hi Vikas,
There's nothing in */var/log/impalad *and */var/log/statestore *directories.
This is their permission:
*
*
*drwxr-xr-x 2 impala impala 4096 Aug 4 12:05 impalad/*
*drwxr-xr-x 2 impala impala 4096 Aug 4 12:16
statestore/*
Thanks,
Andy
Hi Vikas,
There's nothing in */var/log/impalad *and */var/log/statestore *directories.
This is their permission:
*
*
*drwxr-xr-x 2 impala impala 4096 Aug 4 12:05 impalad/*
*drwxr-xr-x 2 impala impala 4096 Aug 4 12:16
statestore/*
Thanks,
Andy
On Thursday, August 8, 2013 11:44:57 PM UTC+8, vikas wrote:
Hi Andy,
Can you get the impalad and statestore log and provide it to us. The
stderr/stdout log is from the bash file that launches impalad/statestore.
The log from impalad/statestore will tell us more on why we can't start
these daemons.
Thanks,
Vikas
Hi Andy,
Can you get the impalad and statestore log and provide it to us. The
stderr/stdout log is from the bash file that launches impalad/statestore.
The log from impalad/statestore will tell us more on why we can't start
these daemons.
Thanks,
Vikas
On Thu, Aug 8, 2013 at 8:17 AM, Andy wrote:
Hi all,
I install Impala on Ubuntu server 12.04 using Cloudera
Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.
HDFS, Hive, MapReduce services work fine, the status is Good health.
But I can't start Impala, the error is:
*Supervisor returned FATAL. Please check the role log file, stderr,
or stdout.*
*
*
The error log is:
*+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-I
MPALAD/hadoop-conf*
*+ export HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
IMPALAD/hive-conf*
*+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMP
ALAD/hive-conf*
*+ export HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
IMPALAD/hbase-conf*
*+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IM
PALAD/hbase-conf*
*+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/shar
e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
e-connector-java.jar*
*+ [[ -z '' ]]*
*+ export AUX_CLASSPATH=/usr/share/java/
mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-8
01.jdbc4.jar:/usr/share/java/oracle-connector-java.jar*
*+ AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/o
racle-connector-java.jar*
*+ [[ -z '' ]]*
*+ export CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
e-connector-java.jar*
*+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/shar
e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
e-connector-java.jar*
*+ FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-IMPALAD
/impala-conf/impalad_flags*
*+ USE_DEBUG_BUILD=false*
*+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/
208-impala-IMPALAD#g' /run/cloudera-scm-agent/proces
s/208-impala-IMPALAD/impala-conf/impalad_flags*
*+ '[' -f /run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-
conf/.htpasswd ']'*
*+ chmod 600 /run/cloudera-scm-agent/process/208-impala-IMPALAD/
impala-conf/.htpasswd*
*+ false*
*+ export IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
impala/sbin-retail*
*+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impal
a/sbin-retail*
*+ '[' impalad = impalad ']'*
*+ exec /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/impalad
--flagfile=/run/cloudera-scm-agent/process/208-impala-IMPALA
D/impala-conf/impalad_flags*
*
*
Are there anyone know how to fix this problem? Thank you.
*
*
Hi all,
I install Impala on Ubuntu server 12.04 using Cloudera
Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.
HDFS, Hive, MapReduce services work fine, the status is Good health.
But I can't start Impala, the error is:
*Supervisor returned FATAL. Please check the role log file, stderr,
or stdout.*
*
*
The error log is:
*+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-I
MPALAD/hadoop-conf*
*+ export HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
IMPALAD/hive-conf*
*+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMP
ALAD/hive-conf*
*+ export HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
IMPALAD/hbase-conf*
*+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IM
PALAD/hbase-conf*
*+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/shar
e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
e-connector-java.jar*
*+ [[ -z '' ]]*
*+ export AUX_CLASSPATH=/usr/share/java/
mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-8
01.jdbc4.jar:/usr/share/java/oracle-connector-java.jar*
*+ AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/o
racle-connector-java.jar*
*+ [[ -z '' ]]*
*+ export CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
e-connector-java.jar*
*+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/shar
e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
e-connector-java.jar*
*+ FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-IMPALAD
/impala-conf/impalad_flags*
*+ USE_DEBUG_BUILD=false*
*+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/
208-impala-IMPALAD#g' /run/cloudera-scm-agent/proces
s/208-impala-IMPALAD/impala-conf/impalad_flags*
*+ '[' -f /run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-
conf/.htpasswd ']'*
*+ chmod 600 /run/cloudera-scm-agent/process/208-impala-IMPALAD/
impala-conf/.htpasswd*
*+ false*
*+ export IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
impala/sbin-retail*
*+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impal
a/sbin-retail*
*+ '[' impalad = impalad ']'*
*+ exec /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/impalad
--flagfile=/run/cloudera-scm-agent/process/208-impala-IMPALA
D/impala-conf/impalad_flags*
*
*
Are there anyone know how to fix this problem? Thank you.
*
*
