FAQ
I'm also seeing this issue, with IMPALA-1.1.1-1.p0.17.

It works fine with debugging enabled, but not without.

Attached is the folder containing the core dump.

/Kieran

On Sunday, 11 August 2013 04:09:20 UTC+1, Andy wrote:

Hi Vikas,

I did it. The attached file is the directory *
/run/cloudera-scm-agent/process/211-impala-STATESTORE/*
*
*
Andy.
On Saturday, August 10, 2013 12:16:35 AM UTC+8, vikas wrote:

Hi Andy,

Try logging in as root and edit the file (sudo su -l root followed by
editing it using vi or echo core > */proc/sys/kernel/core_pattern*).
Then try to dump core of a process to confirm the setting. As an example
following dumps core of the 'tail' process I start:

$ ulimit -c unlimited # this enables core dump
$ tail -f <any_file_name> & # start tail in background, pid of process
will be printed on first line
$ kill -s SIGABRT <tail_pid>

Now the current directory contains the core. Once this work, please give
it a try with Impala process (You may have to stop the apport service as it
may override the core_pattern setting that you modify).

- Vikas


On Thu, Aug 8, 2013 at 1:30 PM, Andy wrote:

Hi Vikas,

There's nothing in '*/var/crash'* directory:
*$
/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/statestored
*
*Illegal instruction (core dumped)*
*$ ls /var/crash/*
*$ *
*
*
I can't change the value of '*/proc/sys/kernel/core_pattern*':
*$ sudo echo core > /proc/sys/kernel/core_pattern *
*-bash: /proc/sys/kernel/core_pattern: Permission denied*

Andy.
On Friday, August 9, 2013 4:14:25 AM UTC+8, vikas wrote:

Hi Andy,

So you have Ubuntu apport enabled to capture dumps. The crash file may
be in '/var/crash'.

If you can't find the core there, try setting core_pattern to simple
'core' (echo core > /proc/sys/kernel/core_pattern) and then it should dump
the core in the current working directory.

Vikas

On Thu, Aug 8, 2013 at 12:59 PM, Andy wrote:

Hi Vikas,

1. The value of */proc/sys/kernel/core_pattern *is:
*|/usr/share/apport/apport %p %s %c*
*
*
2. Thanks, I'll try CentOS.

Andy.
On Friday, August 9, 2013 3:16:46 AM UTC+8, vikas wrote:

Hi Andy,

1. Unfortunately there is no core present. By default the core get
generated in the current working directory of the process. What is the
value of /proc/sys/kernel/core_**patte**rn on your setup?
2. To run retail build, if possible try using some other os than
Ubuntu 12.04 till we get this issue sorted out.

- Vikas

On Thu, Aug 8, 2013 at 11:36 AM, Andy wrote:

Hi Vikas,

1. I already did as you guided. The directory is *210-impala-STATESTORE.
*Its content is in attach file
*$ sudo ls -l /run/cloudera-scm-agent/process/*
*total 0*
*drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28
188-hdfs-SECONDARYNAMENODE*
*drwxr-x--x 3 hdfs hdfs 220 Aug 8 06:28 191-hdfs-DATANODE
*
*drwxr-x--x 3 hdfs hdfs 360 Aug 8 06:28 192-hdfs-NAMENODE
*
*drwxr-x--x 3 mapred hadoop 380 Aug 8 06:28
193-mapreduce-JOBTRACKER*
*drwxr-x--x 3 mapred hadoop 240 Aug 8 06:28
196-mapreduce-TASKTRACKER*
*drwxr-x--x 3 zookeeper zookeeper 220 Aug 8 06:28
197-zookeeper-server*
*drwxr-x--x 4 hive hive 160 Aug 8 06:28
198-hive-HIVEMETASTORE*
*drwxr-x--x 6 impala impala 180 Aug 8 07:56
199-impala-IMPALAD*
*drwxr-x--x 4 impala impala 140 Aug 8 07:57
202-impala-STATESTORE*
*drwxr-x--x 6 impala impala 180 Aug 8 07:59
203-impala-IMPALAD*
*drwxr-x--x 4 impala impala 140 Aug 8 07:59
206-impala-STATESTORE*
*drwxr-x--x 6 impala impala 180 Aug 8 08:13
207-impala-IMPALAD*
*drwxr-x--x 4 impala impala 140 Aug 8 08:13
210-impala-STATESTORE*

2. It works! How can I run without using Debug Build?

Thanks,
Andy
On Friday, August 9, 2013 2:15:14 AM UTC+8, vikas wrote:

Thanks Andy, the libraries seem to be ok. CM adds the libhdfs to
the LD_LIBRARY_PATH so that is fine.

The core dump part is interesting. There are few things we can try:

1. Collect the core dump for debugging. In the CM UI, got to Impala
configuration -> Service Wide -> Enable core dump. Check the check-box and
save the configuration. Then restart the statestore and this time it will
core dump, the dump should go to its cwd, which will be under '
/run/cloudera-scm-agent/**proce****ss/' directory (the latest
created directory with STATESTORE in its name). Please see if you have a
hs_err_<pid>.log there too. Please send us these.

2. To get things working, try running debug build. You can go to CM
console and select the Impala configuration option -> Sevice Wide -> Use
Debug Build. Check this option and save configuration. Try restarting
service now and see if it helps.

You may be hitting this: https://issues.cloudera.******
org/browse/IMPALA-160<https://issues.cloudera.org/browse/IMPALA-160>,
please look at output of "stdout" and see if the error matches.

Vikas

On Thu, Aug 8, 2013 at 11:13 AM, Andy wrote:

Hi Vikas,

The output is:
*$ /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
/statestored*
*Illegal instruction (core dumped)*

*$ ldd /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-
retail/statestored*
* linux-vdso.so.1 => (0x00007fff7413c000)*
* libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f5c3a8e6000)*
* libhdfs.so.0.0.0 => not found*
* librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
(0x00007f5c3a6dd000)*
* libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
(0x00007f5c3a4d9000)*
* libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
(0x00007f5c3a27d000)*
* libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(0x00007f5c39f7c000)*
* libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
(0x00007f5c39c80000)*
* libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x00007f5c39a6a000)*
* libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
(0x00007f5c396aa000)*
* /lib64/ld-linux-x86-64.so.2 (0x00007f5c3ab0a000)*
* libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(0x00007f5c392e2000)*
* libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1
(0x00007f5c390ca000)*
*
*
*$ locate libhdfs.so.0.0.0*
*/opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/libhdfs
.so.0.0.0*
*/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/lib/libhd
fs.so.0.0.0*

Thanks,
Andy
On Friday, August 9, 2013 1:39:06 AM UTC+8, vikas wrote:

Hi Andy,

So either the binaries are terminating very early (even before
opening a log), or we are not able to exec them (some dependent libraries
are missing). What happens if you run them command directly from the
commandline:

*/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin
/statestored*
*
*
*Try using ldd on **/opt/cloudera/parcels/IMPAL
A-1.1-1.p0.8/lib/impala/sbin-retail/statestored to see if it
shows up something.*
*
*
*- Vikas*

On Thu, Aug 8, 2013 at 10:18 AM, Andy wrote:

Hi Vikas,

There's nothing in */var/log/impalad *and */var/log/statestore *directories.
This is their permission:
*
*
*drwxr-xr-x 2 impala impala 4096 Aug 4 12:05
impalad/*
*drwxr-xr-x 2 impala impala 4096 Aug 4 12:16
statestore/*

Thanks,
Andy
On Thursday, August 8, 2013 11:44:57 PM UTC+8, vikas wrote:

Hi Andy,

Can you get the impalad and statestore log and provide it to
us. The stderr/stdout log is from the bash file that launches
impalad/statestore. The log from impalad/statestore will tell us more on
why we can't start these daemons.

Thanks,
Vikas

On Thu, Aug 8, 2013 at 8:17 AM, Andy wrote:

Hi all,

I install Impala on Ubuntu server 12.04 using Cloudera
Manager 4.6.2, Parcels. Then I add services: HDFS, Hive, MapReduce, Impala.

HDFS, Hive, MapReduce services work fine, the status is Good
health. But I can't start Impala, the error is:

*Supervisor returned FATAL. Please check the role log file,
stderr, or stdout.*
*
*
The error log is:

*+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
IMPALAD/hadoop-conf*
*+ export HIVE_CONF_DIR=/run/cloudera-sc
m-agent/process/208-impala-IMPALAD/hive-conf*
*+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
IMPALAD/hive-conf*
*+ export HBASE_CONF_DIR=/run/cloudera-s
cm-agent/process/208-impala-IMPALAD/hbase-conf*
*+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/208-impala-
IMPALAD/hbase-conf*
*+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/
share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
oracle-connector-java.jar*
*+ [[ -z '' ]]*
*+ export AUX_CLASSPATH=/usr/share/java/
mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-8
01.jdbc4.jar:/usr/share/java/oracle-connector-java.jar*
*+ AUX_CLASSPATH=/usr/share/java/
mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-8
01.jdbc4.jar:/usr/share/java/oracle-connector-java.jar*
*+ [[ -z '' ]]*
*+ export CLASSPATH=/usr/share/java/mysq
l-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.j
dbc4.jar:/usr/share/java/oracle-connector-java.jar*
*+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/
share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/
oracle-connector-java.jar*
*+ FLAG_FILE=/run/cloudera-scm-agent/process/208-impala-
IMPALAD/impala-conf/impalad_flags*
*+ USE_DEBUG_BUILD=false*
*+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloud
era-scm-agent/process/208-impala-IMPALAD#g'
/run/cloudera-scm-agent/process/208-impala-IMPALAD/impala-co
nf/impalad_flags*
*+ '[' -f /run/cloudera-scm-agent/process/208-impala-IMPALAD/
impala-conf/.htpasswd ']'*
*+ chmod 600 /run/cloudera-scm-agent/proces
s/208-impala-IMPALAD/impala-conf/.htpasswd*
*+ false*
*+ export IMPALA_BIN=/opt/cloudera/parce
ls/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail*
*+ IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/
impala/sbin-retail*
*+ '[' impalad = impalad ']'*
*+ exec /opt/cloudera/parcels/IMPALA-1
.1-1.p0.8/lib/impala/../../bin/impalad
--flagfile=/run/cloudera-scm-agent/process/208-impala-IMPALA
D/impala-conf/impalad_flags*
*
*
Are there anyone know how to fix this problem? Thank you.
*
*
To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.

Search Discussions

Discussion Posts

Previous

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 12 of 12 | next ›
Discussion Overview
groupimpala-user @
categorieshadoop
postedAug 8, '13 at 3:17p
activeSep 4, '13 at 4:45p
posts12
users3
websitecloudera.com
irc#hadoop

3 users in discussion

Andy: 6 posts Vikas Singh: 5 posts Kieran Evans: 1 post

People

Translate

site design / logo © 2022 Grokbase