There's nothing in '*/var/crash'* directory:
*$ /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/statestored*
*Illegal instruction (core dumped)*
*$ ls /var/crash/*
*$ *
*
*
I can't change the value of '*/proc/sys/kernel/core_pattern*':
*$ sudo echo core > /proc/sys/kernel/core_pattern *
*-bash: /proc/sys/kernel/core_pattern: Permission denied*
Andy.
On Friday, August 9, 2013 4:14:25 AM UTC+8, vikas wrote:
Hi Andy,
So you have Ubuntu apport enabled to capture dumps. The crash file may be
in '/var/crash'.
If you can't find the core there, try setting core_pattern to simple
'core' (echo core > /proc/sys/kernel/core_pattern) and then it should dump
the core in the current working directory.
Vikas
On Thu, Aug 8, 2013 at 12:59 PM, Andy <[email protected] <javascript:>>wrote:
Hi Andy,
So you have Ubuntu apport enabled to capture dumps. The crash file may be
in '/var/crash'.
If you can't find the core there, try setting core_pattern to simple
'core' (echo core > /proc/sys/kernel/core_pattern) and then it should dump
the core in the current working directory.
Vikas
On Thu, Aug 8, 2013 at 12:59 PM, Andy <[email protected] <javascript:>>wrote:
Hi Vikas,
1. The value of */proc/sys/kernel/core_pattern *is:
*|/usr/share/apport/apport %p %s %c*
*
*
2. Thanks, I'll try CentOS.
Andy.
1. The value of */proc/sys/kernel/core_pattern *is:
*|/usr/share/apport/apport %p %s %c*
*
*
2. Thanks, I'll try CentOS.
Andy.
On Friday, August 9, 2013 3:16:46 AM UTC+8, vikas wrote:
Hi Andy,
1. Unfortunately there is no core present. By default the core get
generated in the current working directory of the process. What is the
value of /proc/sys/kernel/core_**pattern on your setup?
2. To run retail build, if possible try using some other os than Ubuntu
12.04 till we get this issue sorted out.
- Vikas
Hi Andy,
1. Unfortunately there is no core present. By default the core get
generated in the current working directory of the process. What is the
value of /proc/sys/kernel/core_**pattern on your setup?
2. To run retail build, if possible try using some other os than Ubuntu
12.04 till we get this issue sorted out.
- Vikas
On Thu, Aug 8, 2013 at 11:36 AM, Andy wrote:
Hi Vikas,
1. I already did as you guided. The directory is *210-impala-STATESTORE.
*Its content is in attach file
*$ sudo ls -l /run/cloudera-scm-agent/process/*
*total 0*
*drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28
188-hdfs-SECONDARYNAMENODE*
*drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28 191-hdfs-DATANODE*
*drwxr-x--x 3 hdfs hdfs 360 Aug 8 06:28 192-hdfs-NAMENODE*
*drwxr-x--x 3 mapred hadoop 380 Aug 8 06:28
193-mapreduce-JOBTRACKER*
*drwxr-x--x 3 mapred hadoop 240 Aug 8 06:28
196-mapreduce-TASKTRACKER*
*drwxr-x--x 3 zookeeper zookeeper 220 Aug 8 06:28 197-zookeeper-server
*
*drwxr-x--x 4 hive hive 160 Aug 8 06:28
198-hive-HIVEMETASTORE*
*drwxr-x--x 6 impala impala 180 Aug 8 07:56 199-impala-IMPALAD*
*drwxr-x--x 4 impala impala 140 Aug 8 07:57
202-impala-STATESTORE*
*drwxr-x--x 6 impala impala 180 Aug 8 07:59 203-impala-IMPALAD*
*drwxr-x--x 4 impala impala 140 Aug 8 07:59
206-impala-STATESTORE*
*drwxr-x--x 6 impala impala 180 Aug 8 08:13 207-impala-IMPALAD*
*drwxr-x--x 4 impala impala 140 Aug 8 08:13
210-impala-STATESTORE*
2. It works! How can I run without using Debug Build?
Thanks,
Andy
Hi Vikas,
1. I already did as you guided. The directory is *210-impala-STATESTORE.
*Its content is in attach file
*$ sudo ls -l /run/cloudera-scm-agent/process/*
*total 0*
*drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28
188-hdfs-SECONDARYNAMENODE*
*drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28 191-hdfs-DATANODE*
*drwxr-x--x 3 hdfs hdfs 360 Aug 8 06:28 192-hdfs-NAMENODE*
*drwxr-x--x 3 mapred hadoop 380 Aug 8 06:28
193-mapreduce-JOBTRACKER*
*drwxr-x--x 3 mapred hadoop 240 Aug 8 06:28
196-mapreduce-TASKTRACKER*
*drwxr-x--x 3 zookeeper zookeeper 220 Aug 8 06:28 197-zookeeper-server
*
*drwxr-x--x 4 hive hive 160 Aug 8 06:28
198-hive-HIVEMETASTORE*
*drwxr-x--x 6 impala impala 180 Aug 8 07:56 199-impala-IMPALAD*
*drwxr-x--x 4 impala impala 140 Aug 8 07:57
202-impala-STATESTORE*
*drwxr-x--x 6 impala impala 180 Aug 8 07:59 203-impala-IMPALAD*
*drwxr-x--x 4 impala impala 140 Aug 8 07:59
206-impala-STATESTORE*
*drwxr-x--x 6 impala impala 180 Aug 8 08:13 207-impala-IMPALAD*
*drwxr-x--x 4 impala impala 140 Aug 8 08:13
210-impala-STATESTORE*
2. It works! How can I run without using Debug Build?
Thanks,
Andy
On Friday, August 9, 2013 2:15:14 AM UTC+8, vikas wrote:
Thanks Andy, the libraries seem to be ok. CM adds the libhdfs to the
LD_LIBRARY_PATH so that is fine.
The core dump part is interesting. There are few things we can try:
1. Collect the core dump for debugging. In the CM UI, got to Impala
configuration -> Service Wide -> Enable core dump. Check the check-box and
save the configuration. Then restart the statestore and this time it will
core dump, the dump should go to its cwd, which will be under '
/run/cloudera-scm-agent/**proce**ss/' directory (the latest created
directory with STATESTORE in its name). Please see if you have a
hs_err_<pid>.log there too. Please send us these.
2. To get things working, try running debug build. You can go to CM
console and select the Impala configuration option -> Sevice Wide -> Use
Debug Build. Check this option and save configuration. Try restarting
service now and see if it helps.
You may be hitting this: https://issues.cloudera.****
org/browse/IMPALA-160 <https://issues.cloudera.org/browse/IMPALA-160>,
please look at output of "stdout" and see if the error matches.
Vikas
Thanks Andy, the libraries seem to be ok. CM adds the libhdfs to the
LD_LIBRARY_PATH so that is fine.
The core dump part is interesting. There are few things we can try:
1. Collect the core dump for debugging. In the CM UI, got to Impala
configuration -> Service Wide -> Enable core dump. Check the check-box and
save the configuration. Then restart the statestore and this time it will
core dump, the dump should go to its cwd, which will be under '
/run/cloudera-scm-agent/**proce**ss/' directory (the latest created
directory with STATESTORE in its name). Please see if you have a
hs_err_<pid>.log there too. Please send us these.
2. To get things working, try running debug build. You can go to CM
console and select the Impala configuration option -> Sevice Wide -> Use
Debug Build. Check this option and save configuration. Try restarting
service now and see if it helps.
You may be hitting this: https://issues.cloudera.****
org/browse/IMPALA-160 <https://issues.cloudera.org/browse/IMPALA-160>,
please look at output of "stdout" and see if the error matches.
Vikas
On Thu, Aug 8, 2013 at 11:13 AM, Andy wrote:
Hi Vikas,
The output is:
*$ /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
/statestored*
*Illegal instruction (core dumped)*
*$ ldd /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-reta
il/statestored*
* linux-vdso.so.1 => (0x00007fff7413c000)*
* libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f5c3a8e6000)*
* libhdfs.so.0.0.0 => not found*
* librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5c3a6dd000)
*
* libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c3a4d9000)
*
* libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
(0x00007f5c3a27d000)*
* libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(0x00007f5c39f7c000)*
* libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c39c80000)*
* libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x00007f5c39a6a000)*
* libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c396aa000)*
* /lib64/ld-linux-x86-64.so.2 (0x00007f5c3ab0a000)*
* libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(0x00007f5c392e2000)*
* libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c390ca000)*
*
*
*$ locate libhdfs.so.0.0.0*
*/opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/libhdfs
.so.0.0.0*
*/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/lib/libhd
fs.so.0.0.0*
Thanks,
Andy
Hi Vikas,
The output is:
*$ /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
/statestored*
*Illegal instruction (core dumped)*
*$ ldd /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-reta
il/statestored*
* linux-vdso.so.1 => (0x00007fff7413c000)*
* libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f5c3a8e6000)*
* libhdfs.so.0.0.0 => not found*
* librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5c3a6dd000)
*
* libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c3a4d9000)
*
* libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
(0x00007f5c3a27d000)*
* libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(0x00007f5c39f7c000)*
* libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c39c80000)*
* libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x00007f5c39a6a000)*
* libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c396aa000)*
* /lib64/ld-linux-x86-64.so.2 (0x00007f5c3ab0a000)*
* libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(0x00007f5c392e2000)*
* libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c390ca000)*
*
*
*$ locate libhdfs.so.0.0.0*
*/opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/libhdfs
.so.0.0.0*
*/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/lib/libhd
fs.so.0.0.0*
Thanks,
Andy
On Friday, August 9, 2013 1:39:06 AM UTC+8, vikas wrote:
Hi Andy,
So either the binaries are terminating very early (even before
opening a log), or we are not able to exec them (some dependent libraries
are missing). What happens if you run them command directly from the
commandline:
*/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
/statestored*
*
*
*Try using ldd on **/opt/cloudera/parcels/IMPAL
A-1.1-1.p0.8/lib/impala/sbin-retail/statestored to see if it shows
up something.*
*
*
*- Vikas*
Hi Andy,
So either the binaries are terminating very early (even before
opening a log), or we are not able to exec them (some dependent libraries
are missing). What happens if you run them command directly from the
commandline:
*/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
/statestored*
*
*
*Try using ldd on **/opt/cloudera/parcels/IMPAL
A-1.1-1.p0.8/lib/impala/sbin-retail/statestored to see if it shows
up something.*
*
*
*- Vikas*
On Thu, Aug 8, 2013 at 10:18 AM, Andy wrote:
Hi Vikas,
There's nothing in */var/log/impalad *and */var/log/statestore *directories.
This is their permission:
*
*
*drwxr-xr-x 2 impala impala 4096 Aug 4 12:05
impalad/*
*drwxr-xr-x 2 impala impala 4096 Aug 4 12:16
statestore/*
Thanks,
Andy
Hi Vikas,
There's nothing in */var/log/impalad *and */var/log/statestore *directories.
This is their permission:
*
*
*drwxr-xr-x 2 impala impala 4096 Aug 4 12:05
impalad/*
*drwxr-xr-x 2 impala impala 4096 Aug 4 12:16
statestore/*
Thanks,
Andy
On Thursday, August 8, 2013 11:44:57 PM UTC+8, vikas wrote:
Hi Andy,
Can you get the impalad and statestore log and provide it to us.
The stderr/stdout log is from the bash file that launches
impalad/statestore. The log from impalad/statestore will tell us more on
why we can't start these daemons.
Thanks,
Vikas
Hi Andy,
Can you get the impalad and statestore log and provide it to us.
The stderr/stdout log is from the bash file that launches
impalad/statestore. The log from impalad/statestore will tell us more on
why we can't start these daemons.
Thanks,
Vikas
On Thu, Aug 8, 2013 at 8:17 AM, Andy wrote:
Hi all,
I install Impala on Ubuntu server 12.04 using Cloudera
Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.
HDFS, Hive, MapReduce services work fine, the status is Good
health. But I can't start Impala, the error is:
*Supervisor returned FATAL. Please check the role log file,
stderr, or stdout.*
*
*
The error log is:
*+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-I
MPALAD/hadoop-conf*
*+ export HIVE_CONF_DIR=/run/cloudera-sc
m-agent/process/208-impala-IMPALAD/hive-conf*
*+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMP
ALAD/hive-conf*
*+ export HBASE_CONF_DIR=/run/cloudera-s
cm-agent/process/208-impala-IMPALAD/hbase-conf*
*+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IM
PALAD/hbase-conf*
*+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/shar
e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
e-connector-java.jar*
*+ [[ -z '' ]]*
*+ export AUX_CLASSPATH=/usr/share/java/
mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-8
01.jdbc4.jar:/usr/share/java/oracle-connector-java.jar*
*+ AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/o
racle-connector-java.jar*
*+ [[ -z '' ]]*
*+ export CLASSPATH=/usr/share/java/mysq
l-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.j
dbc4.jar:/usr/share/java/oracle-connector-java.jar*
*+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/shar
e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
e-connector-java.jar*
*+ FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-IMPALAD
/impala-conf/impalad_flags*
*+ USE_DEBUG_BUILD=false*
*+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloud
era-scm-agent/process/208-impala-IMPALAD#g'
/run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-co
nf/impalad_flags*
*+ '[' -f /run/cloudera-scm-agent/process/208-impala-IMPALAD/
impala-conf/.htpasswd ']'*
*+ chmod 600 /run/cloudera-scm-agent/process/208-impala-IMPALAD/
impala-conf/.htpasswd*
*+ false*
*+ export IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
impala/sbin-retail*
*+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impal
a/sbin-retail*
*+ '[' impalad = impalad ']'*
*+ exec /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../
bin/impalad --flagfile=/run/cloudera-scm-a
gent/process/208-impala-IMPALAD/impala-conf/impalad_flags*
*
*
Are there anyone know how to fix this problem? Thank you.
*
*
Hi all,
I install Impala on Ubuntu server 12.04 using Cloudera
Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.
HDFS, Hive, MapReduce services work fine, the status is Good
health. But I can't start Impala, the error is:
*Supervisor returned FATAL. Please check the role log file,
stderr, or stdout.*
*
*
The error log is:
*+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-I
MPALAD/hadoop-conf*
*+ export HIVE_CONF_DIR=/run/cloudera-sc
m-agent/process/208-impala-IMPALAD/hive-conf*
*+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IMP
ALAD/hive-conf*
*+ export HBASE_CONF_DIR=/run/cloudera-s
cm-agent/process/208-impala-IMPALAD/hbase-conf*
*+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-IM
PALAD/hbase-conf*
*+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/shar
e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
e-connector-java.jar*
*+ [[ -z '' ]]*
*+ export AUX_CLASSPATH=/usr/share/java/
mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-8
01.jdbc4.jar:/usr/share/java/oracle-connector-java.jar*
*+ AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/o
racle-connector-java.jar*
*+ [[ -z '' ]]*
*+ export CLASSPATH=/usr/share/java/mysq
l-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.j
dbc4.jar:/usr/share/java/oracle-connector-java.jar*
*+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/shar
e/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracl
e-connector-java.jar*
*+ FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-IMPALAD
/impala-conf/impalad_flags*
*+ USE_DEBUG_BUILD=false*
*+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloud
era-scm-agent/process/208-impala-IMPALAD#g'
/run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-co
nf/impalad_flags*
*+ '[' -f /run/cloudera-scm-agent/process/208-impala-IMPALAD/
impala-conf/.htpasswd ']'*
*+ chmod 600 /run/cloudera-scm-agent/process/208-impala-IMPALAD/
impala-conf/.htpasswd*
*+ false*
*+ export IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
impala/sbin-retail*
*+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impal
a/sbin-retail*
*+ '[' impalad = impalad ']'*
*+ exec /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../
bin/impalad --flagfile=/run/cloudera-scm-a
gent/process/208-impala-IMPALAD/impala-conf/impalad_flags*
*
*
Are there anyone know how to fix this problem? Thank you.
*
*
