FAQ
I'm new to CDH4 and Impala, but thus far, I've been successful in
installing CDH4 on SLES 11 SP2 x64, with 8 nodes in the cluster. I've been
able to configure mapreduce, hdfs, zookeeper, hbase, and hive services on
the cluster (and start them,) without issue. However, when I attempt to
start impala, the Impala statestore fails to start, as well as all of the
Daemon processes. Stderr logs seem to be of little help, but I've attached
them to this entry regardless. Anyone have any thoughts on this? I've
reached the limit of my knowledge.

Thanks!

Search Discussions

  • Ek778 at May 20, 2013 at 3:21 pm
    Unfortunately, it seems I can't edit my original post with more
    information. I should have also mentioned that I'm running Java 1.6.0_31
    on all nodes
    On Monday, May 20, 2013 11:12:04 AM UTC-4, ek778 wrote:

    I'm new to CDH4 and Impala, but thus far, I've been successful in
    installing CDH4 on SLES 11 SP2 x64, with 8 nodes in the cluster. I've been
    able to configure mapreduce, hdfs, zookeeper, hbase, and hive services on
    the cluster (and start them,) without issue. However, when I attempt to
    start impala, the Impala statestore fails to start, as well as all of the
    Daemon processes. Stderr logs seem to be of little help, but I've attached
    them to this entry regardless. Anyone have any thoughts on this? I've
    reached the limit of my knowledge.

    Thanks!
  • Ek778 at May 20, 2013 at 6:25 pm
    Hi Vikas,
       Thanks for the reply. I've attached the .INFO log from my node that is
    supposed to have the statestore on it (there are no .ERROR or .WARNING
    logs.) However, for my first node, the one that is supposed to host the
    statestore, it looks like the log is being truncated (I've denoted this in
    the attached file's name.) I've also attached the .INFO, .ERROR, and
    .WARNING logs from another node that just hosts the impala daemon. These
    do not appear to be truncated. From the latter logs, it looks to me as
    though the statestore is not starting on my first node, and as a result,
    all other impalad instances on other nodes are failing to start because of
    it. Beyond that, I don't see much other useful information.

    Thanks,
    Eric

    On Monday, May 20, 2013 12:30:00 PM UTC-4, vikas wrote:

    Hi,

    The bash was able to "exec" the daemons. Can you look in the daemon logs
    under /var/log/impala and see if that contains something useful.

    Vikas
  • Vikas Singh at May 20, 2013 at 7:57 pm
    Hi Eric,

    Did you forget to attach the logs? I don't see it in the reply.

    - Vikas
  • Ek778 at May 20, 2013 at 8:01 pm
    Looks like they didn't stick. Here they are now.
    On Monday, May 20, 2013 3:57:12 PM UTC-4, vikas wrote:

    Hi Eric,

    Did you forget to attach the logs? I don't see it in the reply.

    - Vikas
  • Vikas Singh at May 20, 2013 at 8:08 pm
    Hi Eric,

    Can you please also attach the statestore logs? As you mentioned, Impalad
    isn't able to start because it can't connect to statestore. The statestore
    logs should be in /var/log/statestore.

    Vikas

    On Mon, May 20, 2013 at 1:01 PM, ek778 wrote:

    Looks like they didn't stick. Here they are now.

    On Monday, May 20, 2013 3:57:12 PM UTC-4, vikas wrote:

    Hi Eric,

    Did you forget to attach the logs? I don't see it in the reply.

    - Vikas
  • Vikas Singh at May 20, 2013 at 8:29 pm
    Hi Eric,

    The statestore log is truncated too. I think the logs are buffered and are
    not getting flushed. So we don't see the rest of the logs, and there may
    already be a statestore process running on this node.

    Can you check if there is an existing statestore process running on this
    node. (ps -aef | grep -i statestore)? And if there is already such process,
    can you please kill it (and any impalad process too) and try restarting
    again.

    - Vikas
  • Ek778 at May 21, 2013 at 1:00 pm
    Hi Vikas,
       Unfortunately, neither the statestored or impalad processes were running
    on that node:

    cloudera-160:~ # ps -aef | grep -i statestore
    root 18731 18661 0 08:56 pts/0 00:00:00 grep -i statestore
    cloudera-160:~ # ps -aef | grep -i impalad
    root 18733 18661 0 08:56 pts/0 00:00:00 grep -i impalad
    cloudera-160:~ #

    I also checked the other nodes, impalad is not running on those either.

    Anything else I might check?

    Thanks,
    Eric

    On Monday, May 20, 2013 4:29:48 PM UTC-4, vikas wrote:

    Hi Eric,

    The statestore log is truncated too. I think the logs are buffered and are
    not getting flushed. So we don't see the rest of the logs, and there may
    already be a statestore process running on this node.

    Can you check if there is an existing statestore process running on this
    node. (ps -aef | grep -i statestore)? And if there is already such process,
    can you please kill it (and any impalad process too) and try restarting
    again.

    - Vikas
  • Vikas Singh at May 21, 2013 at 4:42 pm
    Hi Eric,

    Can you please enable following Impala settings in CM and start the
    "statestore" and "impalad" on the node where we see truncated logs:

    1. Use Debug Build
    2. Enable Core Dump

    Now if the process terminates abruptly we will have a core dump for it. The
    core dump goes in the "current working directory" which will be in
    /var/run/cloudera-scm-agent/process/<number>-impala-<statestore or
    impalad>. Please look at the last created directory, as there may be more
    than one. Along with the "core" file you may also find a JVM crash log file
    named like hs_err_pid.log. Content of that may provide a clue on what is
    going on.

    Which Linux distro are you using?

    - Vikas
  • Ek778 at May 21, 2013 at 8:13 pm
    Hi Vikas,
       So I've been running on SLES 11 SP2. I had to make a change to the
    virtual machines, so I shut down and restarted today. On startup, I
    noticed that hbase was not in good shape. I restarted hbase, made the
    debug build / core dump changes to impala, and started impala. Oddly
    enough, impala started and is now running without issue. I don't know
    whether it was the reboot, the hbase restart, or the debug build / core
    dump changes that made the difference, but it looks like we're running
    smoothly for now. I'm going to get an impala job going, and see if that
    works ok and go from there.

    Thanks for your help.

    -Eric

    On Tuesday, May 21, 2013 12:42:12 PM UTC-4, vikas wrote:

    Hi Eric,

    Can you please enable following Impala settings in CM and start the
    "statestore" and "impalad" on the node where we see truncated logs:
    1. Use Debug Build
    2. Enable Core Dump

    Now if the process terminates abruptly we will have a core dump for it.
    The core dump goes in the "current working directory" which will be in
    /var/run/cloudera-scm-agent/process/<number>-impala-<statestore or
    impalad>. Please look at the last created directory, as there may be more
    than one. Along with the "core" file you may also find a JVM crash log file
    named like hs_err_pid.log. Content of that may provide a clue on what is
    going on.

    Which Linux distro are you using?

    - Vikas
  • Vikas Singh at May 21, 2013 at 9:18 pm
    Hi Eric,

    You might want to turn off the "debug build" option for production run. The
    Debug builds are not optimized by compiler and perform poorly as compared
    to normal/fully optimized "retail" build.

    Vikas

    On Tue, May 21, 2013 at 1:12 PM, ek778 wrote:

    Hi Vikas,
    So I've been running on SLES 11 SP2. I had to make a change to the
    virtual machines, so I shut down and restarted today. On startup, I
    noticed that hbase was not in good shape. I restarted hbase, made the
    debug build / core dump changes to impala, and started impala. Oddly
    enough, impala started and is now running without issue. I don't know
    whether it was the reboot, the hbase restart, or the debug build / core
    dump changes that made the difference, but it looks like we're running
    smoothly for now. I'm going to get an impala job going, and see if that
    works ok and go from there.

    Thanks for your help.

    -Eric

    On Tuesday, May 21, 2013 12:42:12 PM UTC-4, vikas wrote:

    Hi Eric,

    Can you please enable following Impala settings in CM and start the
    "statestore" and "impalad" on the node where we see truncated logs:
    1. Use Debug Build
    2. Enable Core Dump

    Now if the process terminates abruptly we will have a core dump for it.
    The core dump goes in the "current working directory" which will be in
    /var/run/cloudera-scm-agent/**process/<number>-impala-<**statestore or
    impalad>. Please look at the last created directory, as there may be more
    than one. Along with the "core" file you may also find a JVM crash log file
    named like hs_err_pid.log. Content of that may provide a clue on what is
    going on.

    Which Linux distro are you using?

    - Vikas
  • Ek778 at May 22, 2013 at 7:40 pm
    Hi Vikas,
       So I tried turning debug back off, and on next restart, impala failed to
    start again. I happened to leave the core option enabled, so I grabbed the
    contents of the process directories for both impala and statestore in the
    hopes you might be able to identify what's going on. I've attached them
    here as tar.gz files, one each for statestore and impalad. Again, these
    are with debug off, but core on. Prior to restarting impala, I noticed
    hbase was "bad" again, restarted, and it was healthy. Perhaps there is an
    underlying problem in hbase? Are there any hbase logs you'd like to see?

    Thanks,
    Eric


    On Tuesday, May 21, 2013 5:18:43 PM UTC-4, vikas wrote:

    Hi Eric,

    You might want to turn off the "debug build" option for production run.
    The Debug builds are not optimized by compiler and perform poorly as
    compared to normal/fully optimized "retail" build.

    Vikas


    On Tue, May 21, 2013 at 1:12 PM, ek778 <eric....@gmail.com <javascript:>>wrote:
    Hi Vikas,
    So I've been running on SLES 11 SP2. I had to make a change to the
    virtual machines, so I shut down and restarted today. On startup, I
    noticed that hbase was not in good shape. I restarted hbase, made the
    debug build / core dump changes to impala, and started impala. Oddly
    enough, impala started and is now running without issue. I don't know
    whether it was the reboot, the hbase restart, or the debug build / core
    dump changes that made the difference, but it looks like we're running
    smoothly for now. I'm going to get an impala job going, and see if that
    works ok and go from there.

    Thanks for your help.

    -Eric

    On Tuesday, May 21, 2013 12:42:12 PM UTC-4, vikas wrote:

    Hi Eric,

    Can you please enable following Impala settings in CM and start the
    "statestore" and "impalad" on the node where we see truncated logs:
    1. Use Debug Build
    2. Enable Core Dump

    Now if the process terminates abruptly we will have a core dump for it.
    The core dump goes in the "current working directory" which will be in
    /var/run/cloudera-scm-agent/**process/<number>-impala-<**statestore or
    impalad>. Please look at the last created directory, as there may be more
    than one. Along with the "core" file you may also find a JVM crash log file
    named like hs_err_pid.log. Content of that may provide a clue on what is
    going on.

    Which Linux distro are you using?

    - Vikas
  • Ishaan Joshi at May 23, 2013 at 4:12 pm
    Eric,

       Hbase should not cause Impala to crash and not start. Can you get impala
    to start in debug mode, or this is problem persistent ? Did impala manage
    to generate any logs at all in /var/log/impalad?

    Thanks,

    -- Ishaan

    On Wed, May 22, 2013 at 12:40 PM, ek778 wrote:

    Hi Vikas,
    So I tried turning debug back off, and on next restart, impala failed to
    start again. I happened to leave the core option enabled, so I grabbed the
    contents of the process directories for both impala and statestore in the
    hopes you might be able to identify what's going on. I've attached them
    here as tar.gz files, one each for statestore and impalad. Again, these
    are with debug off, but core on. Prior to restarting impala, I noticed
    hbase was "bad" again, restarted, and it was healthy. Perhaps there is an
    underlying problem in hbase? Are there any hbase logs you'd like to see?

    Thanks,
    Eric


    On Tuesday, May 21, 2013 5:18:43 PM UTC-4, vikas wrote:

    Hi Eric,

    You might want to turn off the "debug build" option for production run.
    The Debug builds are not optimized by compiler and perform poorly as
    compared to normal/fully optimized "retail" build.

    Vikas

    On Tue, May 21, 2013 at 1:12 PM, ek778 wrote:

    Hi Vikas,
    So I've been running on SLES 11 SP2. I had to make a change to the
    virtual machines, so I shut down and restarted today. On startup, I
    noticed that hbase was not in good shape. I restarted hbase, made the
    debug build / core dump changes to impala, and started impala. Oddly
    enough, impala started and is now running without issue. I don't know
    whether it was the reboot, the hbase restart, or the debug build / core
    dump changes that made the difference, but it looks like we're running
    smoothly for now. I'm going to get an impala job going, and see if that
    works ok and go from there.

    Thanks for your help.

    -Eric

    On Tuesday, May 21, 2013 12:42:12 PM UTC-4, vikas wrote:

    Hi Eric,

    Can you please enable following Impala settings in CM and start the
    "statestore" and "impalad" on the node where we see truncated logs:
    1. Use Debug Build
    2. Enable Core Dump

    Now if the process terminates abruptly we will have a core dump for it.
    The core dump goes in the "current working directory" which will be in
    /var/run/cloudera-scm-agent/**pr**ocess/<number>-impala-<**statest**ore
    or impalad>. Please look at the last created directory, as there may be
    more than one. Along with the "core" file you may also find a JVM crash log
    file named like hs_err_pid.log. Content of that may provide a clue on what
    is going on.

    Which Linux distro are you using?

    - Vikas

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimpala-user @
categorieshadoop
postedMay 20, '13 at 3:12p
activeMay 23, '13 at 4:12p
posts13
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase