FAQ
Impala is working for me with simple select queries but it is not working
with queries involving joins. When I execute the join query in the impala
shell, I get the following result.
Query finished, fetching results ...
Error communicating with impalad: TSocket read 0 bytes

And then I am no longer connected to impala. How do I fix this?

Alex

Search Discussions

  • Ricky Saltzer at Jun 17, 2013 at 2:39 pm
    Hi Alex -

    Could you set the GLOG level to
    2<http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_logging.html>
    (you
    can use Cloudera manager to do it, too). Then, re-run your query and attach
    the impalad.INFO from the coordinator (the node you submit your query to).

    Ricky

    On Mon, Jun 17, 2013 at 6:23 AM, Alex Minnaar wrote:

    Impala is working for me with simple select queries but it is not working
    with queries involving joins. When I execute the join query in the impala
    shell, I get the following result.
    Query finished, fetching results ...
    Error communicating with impalad: TSocket read 0 bytes

    And then I am no longer connected to impala. How do I fix this?

    Alex


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com
  • Alex Minnaar at Jun 17, 2013 at 3:24 pm
    I'm not sure how to set GLOG level to 2 within the Cloudera Manager. How
    do I do this?

    Thanks,

    Alex
    On Monday, June 17, 2013 10:39:21 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Could you set the GLOG level to 2<http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_logging.html> (you
    can use Cloudera manager to do it, too). Then, re-run your query and attach
    the impalad.INFO from the coordinator (the node you submit your query to).

    Ricky


    On Mon, Jun 17, 2013 at 6:23 AM, Alex Minnaar <minna...@gmail.com<javascript:>
    wrote:
    Impala is working for me with simple select queries but it is not working
    with queries involving joins. When I execute the join query in the impala
    shell, I get the following result.
    Query finished, fetching results ...
    Error communicating with impalad: TSocket read 0 bytes

    And then I am no longer connected to impala. How do I fix this?

    Alex


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com

  • Alex Minnaar at Jun 17, 2013 at 3:39 pm
    OK I figured out how to set the GLOG level, but when I try to download the
    impalad.INFO file I get

    Problem accessing /cmf/process/all/logs/download. Reason:

         Connection timed out

    Could not connect to host.
    On Monday, June 17, 2013 10:39:21 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Could you set the GLOG level to 2<http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_logging.html> (you
    can use Cloudera manager to do it, too). Then, re-run your query and attach
    the impalad.INFO from the coordinator (the node you submit your query to).

    Ricky


    On Mon, Jun 17, 2013 at 6:23 AM, Alex Minnaar <minna...@gmail.com<javascript:>
    wrote:
    Impala is working for me with simple select queries but it is not working
    with queries involving joins. When I execute the join query in the impala
    shell, I get the following result.
    Query finished, fetching results ...
    Error communicating with impalad: TSocket read 0 bytes

    And then I am no longer connected to impala. How do I fix this?

    Alex


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com

  • Ricky Saltzer at Jun 17, 2013 at 3:44 pm
    Hi Alex -

    Try connecting to the node via SSH and grabbing the log directly from *
    /var/log/impalad *
    *
    *
    Ricky

    On Mon, Jun 17, 2013 at 8:39 AM, Alex Minnaar wrote:

    OK I figured out how to set the GLOG level, but when I try to download the
    impalad.INFO file I get

    Problem accessing /cmf/process/all/logs/download. Reason:

    Connection timed out

    Could not connect to host.

    On Monday, June 17, 2013 10:39:21 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Could you set the GLOG level to 2<http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_logging.html> (you
    can use Cloudera manager to do it, too). Then, re-run your query and attach
    the impalad.INFO from the coordinator (the node you submit your query to).

    Ricky

    On Mon, Jun 17, 2013 at 6:23 AM, Alex Minnaar wrote:

    Impala is working for me with simple select queries but it is not
    working with queries involving joins. When I execute the join query in the
    impala shell, I get the following result.
    Query finished, fetching results ...
    Error communicating with impalad: TSocket read 0 bytes

    And then I am no longer connected to impala. How do I fix this?

    Alex


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com
  • Alex Minnaar at Jun 17, 2013 at 4:03 pm
    I have attached what was in the impalad.INFO file

    Alex
    On Monday, June 17, 2013 11:44:10 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Try connecting to the node via SSH and grabbing the log directly from *
    /var/log/impalad *
    *
    *
    Ricky


    On Mon, Jun 17, 2013 at 8:39 AM, Alex Minnaar <minna...@gmail.com<javascript:>
    wrote:
    OK I figured out how to set the GLOG level, but when I try to download
    the impalad.INFO file I get

    Problem accessing /cmf/process/all/logs/download. Reason:

    Connection timed out

    Could not connect to host.

    On Monday, June 17, 2013 10:39:21 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Could you set the GLOG level to 2<http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_logging.html> (you
    can use Cloudera manager to do it, too). Then, re-run your query and attach
    the impalad.INFO from the coordinator (the node you submit your query to).

    Ricky

    On Mon, Jun 17, 2013 at 6:23 AM, Alex Minnaar wrote:

    Impala is working for me with simple select queries but it is not
    working with queries involving joins. When I execute the join query in the
    impala shell, I get the following result.
    Query finished, fetching results ...
    Error communicating with impalad: TSocket read 0 bytes

    And then I am no longer connected to impala. How do I fix this?

    Alex


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com

  • Alex Minnaar at Jun 17, 2013 at 4:04 pm
    I have attached what was in the impalad.INFO file

    Alex
    On Monday, June 17, 2013 11:44:10 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Try connecting to the node via SSH and grabbing the log directly from *
    /var/log/impalad *
    *
    *
    Ricky


    On Mon, Jun 17, 2013 at 8:39 AM, Alex Minnaar <minna...@gmail.com<javascript:>
    wrote:
    OK I figured out how to set the GLOG level, but when I try to download
    the impalad.INFO file I get

    Problem accessing /cmf/process/all/logs/download. Reason:

    Connection timed out

    Could not connect to host.

    On Monday, June 17, 2013 10:39:21 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Could you set the GLOG level to 2<http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_logging.html> (you
    can use Cloudera manager to do it, too). Then, re-run your query and attach
    the impalad.INFO from the coordinator (the node you submit your query to).

    Ricky

    On Mon, Jun 17, 2013 at 6:23 AM, Alex Minnaar wrote:

    Impala is working for me with simple select queries but it is not
    working with queries involving joins. When I execute the join query in the
    impala shell, I get the following result.
    Query finished, fetching results ...
    Error communicating with impalad: TSocket read 0 bytes

    And then I am no longer connected to impala. How do I fix this?

    Alex


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com

  • Ricky Saltzer at Jun 17, 2013 at 4:08 pm
    Hey Alex -

    Sorry about that, I forgot that the log get's rolled over after a crash.
    Inside of */var/log/impalad*, you should be able to find the last log file
    written to before the crash using an "*ls -lthr*", take a peek inside..it
    should contain a lot more data.

    Thanks,
    Ricky

    On Mon, Jun 17, 2013 at 9:04 AM, Alex Minnaar wrote:

    I have attached what was in the impalad.INFO file

    Alex

    On Monday, June 17, 2013 11:44:10 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Try connecting to the node via SSH and grabbing the log directly from *
    /var/log/impalad *
    *
    *
    Ricky

    On Mon, Jun 17, 2013 at 8:39 AM, Alex Minnaar wrote:

    OK I figured out how to set the GLOG level, but when I try to download
    the impalad.INFO file I get

    Problem accessing /cmf/process/all/logs/**download. Reason:

    Connection timed out

    Could not connect to host.

    On Monday, June 17, 2013 10:39:21 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Could you set the GLOG level to 2<http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_logging.html> (you
    can use Cloudera manager to do it, too). Then, re-run your query and attach
    the impalad.INFO from the coordinator (the node you submit your query to).

    Ricky

    On Mon, Jun 17, 2013 at 6:23 AM, Alex Minnaar wrote:

    Impala is working for me with simple select queries but it is not
    working with queries involving joins. When I execute the join query in the
    impala shell, I get the following result.
    Query finished, fetching results ...
    Error communicating with impalad: TSocket read 0 bytes

    And then I am no longer connected to impala. How do I fix this?

    Alex


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com
  • Alex Minnaar at Jun 17, 2013 at 4:14 pm
    It looks like the last log file was this one. But I'm not sure it tells us
    anything either.

    Log file created at: 2013/06/17 16:10:41
    Running on machine: ip-10-245-112-238
    Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
    I0617 16:10:41.500978 15390 daemon.cc:34] impalad version 1.0 RELEASE
    (build d1bf0d1dac339af3692ffa17a5e3fdae0aed751f)
    Built on Sun, 28 Apr 2013 15:58:48 PST
    I0617 16:10:41.521280 15390 daemon.cc:35] Using hostname:
    ip-10-245-112-238.us-west-2.compute.internal
    I0617 16:10:41.522235 15390 logging.cc:76] Flags (see also /varz are on
    debug webserver):
    --dump_ir=false
    --module_output=
    --be_port=22000
    --hostname=ip-10-245-112-238.us-west-2.compute.internal
    --keytab_file=
    --mem_limit=80%
    --planservice_host=localhost
    --planservice_port=20000
    --principal=
    --exchg_node_buffer_size_bytes=10485760
    --max_row_batches=0
    --randomize_splits=false
    --num_disks=0
    --num_threads_per_disk=1
    --read_size=8388608
    --enable_webserver=true
    --state_store_host=ip-10-245-112-238.us-west-2.compute.internal
    --state_store_subscriber_port=23000
    --use_statestore=true
    --nn=ip-10-245-112-238.us-west-2.compute.internal
    --nn_port=8020
    --serialize_batch=false
    --status_report_interval=5
    --compress_rowbatches=true
    --num_threads_per_core=3
    --abort_on_config_error=true
    --be_service_threads=64
    --beeswax_port=21000
    --default_query_options=
    --fe_service_threads=64
    --heap_profile_dir=
    --hs2_port=21050
    --load_catalog_at_startup=false
    --log_mem_usage_interval=0
    --log_query_to_file=true
    --query_log_size=25
    --use_planservice=false
    --statestore_subscriber_timeout_seconds=10
    --state_store_port=24000
    --statestore_max_missed_heartbeats=5
    --statestore_num_heartbeat_threads=10
    --statestore_suspect_heartbeats=2
    --kerberos_reinit_interval=60
    --sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
    --web_log_bytes=1048576
    --log_filename=impalad
    --periodic_counter_update_period_ms=500
    --rpc_cnxn_attempts=10
    --rpc_cnxn_retry_interval_ms=2000
    --enable_webserver_doc_root=true
    --webserver_doc_root=/usr/lib/impala
    --webserver_interface=
    --webserver_port=25000
    --flagfile=/run/cloudera-scm-agent/process/40-impala-IMPALAD/impala-conf/impalad_flags
    --fromenv=
    --tryfromenv=
    --undefok=
    --tab_completion_columns=80
    --tab_completion_word=
    --help=false
    --helpfull=false
    --helpmatch=
    --helpon=
    --helppackage=false
    --helpshort=false
    --helpxml=false
    --version=false
    --alsologtoemail=
    --alsologtostderr=false
    --drop_log_memory=true
    --log_backtrace_at=
    --log_dir=/var/log/impalad
    --log_link=
    --log_prefix=true
    --logbuflevel=-1
    --logbufsecs=30
    --logemaillevel=999
    --logmailer=/bin/mail
    --logtostderr=false
    --max_log_size=200
    --minloglevel=0
    --stderrthreshold=2
    --stop_logging_if_full_disk=false
    --symbolize_stacktrace=true
    --v=1
    --vmodule=
    I0617 16:10:41.533797 15390 mem-info.cc:66] Physical Memory: 3.66 GB
    I0617 16:10:41.539779 15390 daemon.cc:43] Cpu Info:
       Model: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
       Cores: 1
       L1 Cache: 32.00 KB
       L2 Cache: 256.00 KB
       L3 Cache: 20.00 MB
       Hardware Supports:
         ssse3
         sse4_1
         sse4_2
         popcnt
    I0617 16:10:41.539821 15390 daemon.cc:44] Disk Info:
       Num disks 2: xvda, xvdb
    I0617 16:10:41.539855 15390 daemon.cc:45] Mem Info: 3.66 GB
    I0617 16:10:50.755198 15390 impala-server.cc:1809] Default query
    options:TQueryOptions {
       01: abort_on_error (bool) = false,
       02: max_errors (i32) = 0,
       03: disable_codegen (bool) = false,
       04: batch_size (i32) = 0,
       05: num_nodes (i32) = 0,
       06: max_scan_range_length (i64) = 0,
       07: num_scanner_threads (i32) = 0,
       08: max_io_buffers (i32) = 0,
       09: allow_unsupported_formats (bool) = false,
       10: default_order_by_limit (i64) = -1,
       11: debug_action (string) = "",
       12: mem_limit (i64) = 0,
       13: abort_on_default_limit_exceeded (bool) = false,
    }
    WARN0617 16:10:51.236000 main org.apache.hadoop.conf.Configuration]
    mapred.max.split.size is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.maxsize
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.Configuration]
    mapred.min.split.size is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.Configuration]
    mapred.min.split.size.per.rack is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize.per.rack
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.Configuration]
    mapred.min.split.size.per.node is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize.per.node
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.Configuration]
    mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.Configuration]
    mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
    mapreduce.reduce.speculative
    WARN0617 16:10:51.892000 main org.apache.hadoop.conf.Configuration]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@66b51404:an attempt
    to override final parameter:
    mapreduce.job.end-notification.max.retry.interval; Ignoring.
    WARN0617 16:10:51.954000 main org.apache.hadoop.conf.Configuration]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@66b51404:an attempt
    to override final parameter: mapreduce.job.end-notification.max.attempts;
      Ignoring.
    WARN0617 16:10:51.972000 main org.apache.hadoop.hive.conf.HiveConf]
    DEPRECATED: Configuration property hive.metastore.local no longer has any
    effect. Make sure to provide a valid value for hive.metastore.uris if you
    are connecting to a remote metastore.
    INFO0617 16:10:52.240000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:10:52.494000 main hive.metastore] Waiting 1 seconds before next
    connection attempt.
    INFO0617 16:10:53.495000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:53.495000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:10:53.497000 main hive.metastore] Waiting 1 seconds before next
    connection attempt.
    INFO0617 16:10:54.498000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:54.498000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:10:54.505000 main hive.metastore] Waiting 1 seconds before next
    connection attempt.
    INFO0617 16:10:55.506000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:55.507000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:10:55.515000 main hive.metastore] Waiting 1 seconds before next
    connection attempt.
    INFO0617 16:10:56.515000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:56.515000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:10:56.519000 main hive.metastore] Waiting 1 seconds before next
    connection attempt.
    INFO0617 16:10:57.519000 main hive.metastore] Connected to metastore.
    I0617 16:10:59.891072 15390 impala-server.cc:2074] Impala Beeswax Service
    listening on 21000
    I0617 16:10:59.893288 15390 impala-server.cc:2085] Impala HiveServer2
    Service listening on 21050
    I0617 16:10:59.893335 15390 impala-server.cc:2093] ImpalaInternalService
    listening on 22000
    I0617 16:10:59.905498 15390 thrift-server.cc:365] ThriftServer 'backend'
    started on port: 22000
    I0617 16:10:59.905547 15390 exec-env.cc:146] Starting global services
    I0617 16:10:59.905670 15390 exec-env.cc:167] Using global memory limit:
    2.93 GB
    I0617 16:10:59.909502 15390 webserver.cc:129] Starting webserver on all
    interfaces, port 25000
    I0617 16:10:59.909559 15390 webserver.cc:139] Document root: /usr/lib/impala
    I0617 16:10:59.921815 15390 webserver.cc:178] Webserver started
    I0617 16:10:59.921897 15390 simple-scheduler.cc:99] Starting simple
    scheduler
    I0617 16:10:59.926578 15390 simple-scheduler.cc:133] Simple-scheduler using
    10.245.112.238 as IP address
    I0617 16:10:59.926632 15390 state-store-subscriber.cc:137] Starting
    subscriber
    I0617 16:10:59.937484 15390 thrift-server.cc:365] ThriftServer
    'StateStoreSubscriber' started on port: 23000
    I0617 16:10:59.949283 15390 client-cache.cc:98] CreateClient(): adding new
    client for ip-10-245-112-238.us-west-2.compute.internal:24000
    I0617 16:10:59.982250 15390 thrift-server.cc:365] ThriftServer
    'beeswax-frontend' started on port: 21000
    I0617 16:11:00.017025 15390 thrift-server.cc:365] ThriftServer
    'hiveServer2-frontend' started on port: 21050
    I0617 16:11:00.017144 15390 impalad-main.cc:101] Impala has started.

    On Monday, June 17, 2013 12:08:01 PM UTC-4, Ricky Saltzer wrote:

    Hey Alex -

    Sorry about that, I forgot that the log get's rolled over after a crash.
    Inside of */var/log/impalad*, you should be able to find the last log
    file written to before the crash using an "*ls -lthr*", take a peek
    inside..it should contain a lot more data.

    Thanks,
    Ricky


    On Mon, Jun 17, 2013 at 9:04 AM, Alex Minnaar <minna...@gmail.com<javascript:>
    wrote:
    I have attached what was in the impalad.INFO file

    Alex

    On Monday, June 17, 2013 11:44:10 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Try connecting to the node via SSH and grabbing the log directly from *
    /var/log/impalad *
    *
    *
    Ricky

    On Mon, Jun 17, 2013 at 8:39 AM, Alex Minnaar wrote:

    OK I figured out how to set the GLOG level, but when I try to download
    the impalad.INFO file I get

    Problem accessing /cmf/process/all/logs/**download. Reason:

    Connection timed out

    Could not connect to host.

    On Monday, June 17, 2013 10:39:21 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Could you set the GLOG level to 2<http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_logging.html> (you
    can use Cloudera manager to do it, too). Then, re-run your query and attach
    the impalad.INFO from the coordinator (the node you submit your query to).

    Ricky

    On Mon, Jun 17, 2013 at 6:23 AM, Alex Minnaar wrote:

    Impala is working for me with simple select queries but it is not
    working with queries involving joins. When I execute the join query in the
    impala shell, I get the following result.
    Query finished, fetching results ...
    Error communicating with impalad: TSocket read 0 bytes

    And then I am no longer connected to impala. How do I fix this?

    Alex


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com

  • Ricky Saltzer at Jun 17, 2013 at 4:30 pm
    It would be the log file before this one
    On Jun 17, 2013 12:14 PM, "Alex Minnaar" wrote:

    It looks like the last log file was this one. But I'm not sure it tells us
    anything either.

    Log file created at: 2013/06/17 16:10:41
    Running on machine: ip-10-245-112-238
    Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
    I0617 16:10:41.500978 15390 daemon.cc:34] impalad version 1.0 RELEASE
    (build d1bf0d1dac339af3692ffa17a5e3fdae0aed751f)
    Built on Sun, 28 Apr 2013 15:58:48 PST
    I0617 16:10:41.521280 15390 daemon.cc:35] Using hostname:
    ip-10-245-112-238.us-west-2.compute.internal
    I0617 16:10:41.522235 15390 logging.cc:76] Flags (see also /varz are on
    debug webserver):
    --dump_ir=false
    --module_output=
    --be_port=22000
    --hostname=ip-10-245-112-238.us-west-2.compute.internal
    --keytab_file=
    --mem_limit=80%
    --planservice_host=localhost
    --planservice_port=20000
    --principal=
    --exchg_node_buffer_size_bytes=10485760
    --max_row_batches=0
    --randomize_splits=false
    --num_disks=0
    --num_threads_per_disk=1
    --read_size=8388608
    --enable_webserver=true
    --state_store_host=ip-10-245-112-238.us-west-2.compute.internal
    --state_store_subscriber_port=23000
    --use_statestore=true
    --nn=ip-10-245-112-238.us-west-2.compute.internal
    --nn_port=8020
    --serialize_batch=false
    --status_report_interval=5
    --compress_rowbatches=true
    --num_threads_per_core=3
    --abort_on_config_error=true
    --be_service_threads=64
    --beeswax_port=21000
    --default_query_options=
    --fe_service_threads=64
    --heap_profile_dir=
    --hs2_port=21050
    --load_catalog_at_startup=false
    --log_mem_usage_interval=0
    --log_query_to_file=true
    --query_log_size=25
    --use_planservice=false
    --statestore_subscriber_timeout_seconds=10
    --state_store_port=24000
    --statestore_max_missed_heartbeats=5
    --statestore_num_heartbeat_threads=10
    --statestore_suspect_heartbeats=2
    --kerberos_reinit_interval=60

    --sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
    --web_log_bytes=1048576
    --log_filename=impalad
    --periodic_counter_update_period_ms=500
    --rpc_cnxn_attempts=10
    --rpc_cnxn_retry_interval_ms=2000
    --enable_webserver_doc_root=true
    --webserver_doc_root=/usr/lib/impala
    --webserver_interface=
    --webserver_port=25000

    --flagfile=/run/cloudera-scm-agent/process/40-impala-IMPALAD/impala-conf/impalad_flags
    --fromenv=
    --tryfromenv=
    --undefok=
    --tab_completion_columns=80
    --tab_completion_word=
    --help=false
    --helpfull=false
    --helpmatch=
    --helpon=
    --helppackage=false
    --helpshort=false
    --helpxml=false
    --version=false
    --alsologtoemail=
    --alsologtostderr=false
    --drop_log_memory=true
    --log_backtrace_at=
    --log_dir=/var/log/impalad
    --log_link=
    --log_prefix=true
    --logbuflevel=-1
    --logbufsecs=30
    --logemaillevel=999
    --logmailer=/bin/mail
    --logtostderr=false
    --max_log_size=200
    --minloglevel=0
    --stderrthreshold=2
    --stop_logging_if_full_disk=false
    --symbolize_stacktrace=true
    --v=1
    --vmodule=
    I0617 16:10:41.533797 15390 mem-info.cc:66] Physical Memory: 3.66 GB
    I0617 16:10:41.539779 15390 daemon.cc:43] Cpu Info:
    Model: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
    Cores: 1
    L1 Cache: 32.00 KB
    L2 Cache: 256.00 KB
    L3 Cache: 20.00 MB
    Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    I0617 16:10:41.539821 15390 daemon.cc:44] Disk Info:
    Num disks 2: xvda, xvdb
    I0617 16:10:41.539855 15390 daemon.cc:45] Mem Info: 3.66 GB
    I0617 16:10:50.755198 15390 impala-server.cc:1809] Default query
    options:TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_exceeded (bool) = false,
    }
    WARN0617 16:10:51.236000 main org.apache.hadoop.conf.Configuration]
    mapred.max.split.size is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.maxsize
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.Configuration]
    mapred.min.split.size is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.Configuration]
    mapred.min.split.size.per.rack is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize.per.rack
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.Configuration]
    mapred.min.split.size.per.node is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize.per.node
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.Configuration]
    mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.Configuration]
    mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
    mapreduce.reduce.speculative
    WARN0617 16:10:51.892000 main org.apache.hadoop.conf.Configuration]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@66b51404:an
    attempt to override final parameter:
    mapreduce.job.end-notification.max.retry.interval; Ignoring.
    WARN0617 16:10:51.954000 main org.apache.hadoop.conf.Configuration]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@66b51404:an
    attempt to override final parameter:
    mapreduce.job.end-notification.max.attempts; Ignoring.
    WARN0617 16:10:51.972000 main org.apache.hadoop.hive.conf.HiveConf]
    DEPRECATED: Configuration property hive.metastore.local no longer has any
    effect. Make sure to provide a valid value for hive.metastore.uris if you
    are connecting to a remote metastore.
    INFO0617 16:10:52.240000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:10:52.494000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:10:53.495000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:53.495000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:10:53.497000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:10:54.498000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:54.498000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:10:54.505000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:10:55.506000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:55.507000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:10:55.515000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:10:56.515000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:56.515000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:10:56.519000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:10:57.519000 main hive.metastore] Connected to metastore.
    I0617 16:10:59.891072 15390 impala-server.cc:2074] Impala Beeswax Service
    listening on 21000
    I0617 16:10:59.893288 15390 impala-server.cc:2085] Impala HiveServer2
    Service listening on 21050
    I0617 16:10:59.893335 15390 impala-server.cc:2093] ImpalaInternalService
    listening on 22000
    I0617 16:10:59.905498 15390 thrift-server.cc:365] ThriftServer 'backend'
    started on port: 22000
    I0617 16:10:59.905547 15390 exec-env.cc:146] Starting global services
    I0617 16:10:59.905670 15390 exec-env.cc:167] Using global memory limit:
    2.93 GB
    I0617 16:10:59.909502 15390 webserver.cc:129] Starting webserver on all
    interfaces, port 25000
    I0617 16:10:59.909559 15390 webserver.cc:139] Document root:
    /usr/lib/impala
    I0617 16:10:59.921815 15390 webserver.cc:178] Webserver started
    I0617 16:10:59.921897 15390 simple-scheduler.cc:99] Starting simple
    scheduler
    I0617 16:10:59.926578 15390 simple-scheduler.cc:133] Simple-scheduler
    using 10.245.112.238 as IP address
    I0617 16:10:59.926632 15390 state-store-subscriber.cc:137] Starting
    subscriber
    I0617 16:10:59.937484 15390 thrift-server.cc:365] ThriftServer
    'StateStoreSubscriber' started on port: 23000
    I0617 16:10:59.949283 15390 client-cache.cc:98] CreateClient(): adding new
    client for ip-10-245-112-238.us-west-2.compute.internal:24000
    I0617 16:10:59.982250 15390 thrift-server.cc:365] ThriftServer
    'beeswax-frontend' started on port: 21000
    I0617 16:11:00.017025 15390 thrift-server.cc:365] ThriftServer
    'hiveServer2-frontend' started on port: 21050
    I0617 16:11:00.017144 15390 impalad-main.cc:101] Impala has started.

    On Monday, June 17, 2013 12:08:01 PM UTC-4, Ricky Saltzer wrote:

    Hey Alex -

    Sorry about that, I forgot that the log get's rolled over after a crash.
    Inside of */var/log/impalad*, you should be able to find the last log
    file written to before the crash using an "*ls -lthr*", take a peek
    inside..it should contain a lot more data.

    Thanks,
    Ricky

    On Mon, Jun 17, 2013 at 9:04 AM, Alex Minnaar wrote:

    I have attached what was in the impalad.INFO file

    Alex

    On Monday, June 17, 2013 11:44:10 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Try connecting to the node via SSH and grabbing the log directly from *
    /var/log/impalad *
    *
    *
    Ricky

    On Mon, Jun 17, 2013 at 8:39 AM, Alex Minnaar wrote:

    OK I figured out how to set the GLOG level, but when I try to download
    the impalad.INFO file I get

    Problem accessing /cmf/process/all/logs/**download**. Reason:

    Connection timed out

    Could not connect to host.

    On Monday, June 17, 2013 10:39:21 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Could you set the GLOG level to 2<http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_logging.html> (you
    can use Cloudera manager to do it, too). Then, re-run your query and attach
    the impalad.INFO from the coordinator (the node you submit your query to).

    Ricky

    On Mon, Jun 17, 2013 at 6:23 AM, Alex Minnaar wrote:

    Impala is working for me with simple select queries but it is not
    working with queries involving joins. When I execute the join query in the
    impala shell, I get the following result.
    Query finished, fetching results ...
    Error communicating with impalad: TSocket read 0 bytes

    And then I am no longer connected to impala. How do I fix this?

    Alex


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com

  • Ricky Saltzer at Jun 17, 2013 at 5:18 pm
    Hey Alex -

    So, the following query is the one crashing?

    select A.search_parameters,reseller_count,total from (select
    search_parameters,count(distinct(reseller)) as reseller_count from
    all_lookups group by search_parameters)A join (select
    search_parameters,count(*) as total from all_lookups group by
    search_parameters)B on A.search_parameters=B.search_parameters limit 50

    Ricky

    On Mon, Jun 17, 2013 at 10:05 AM, Alex Minnaar wrote:

    OK the previous log file was

    Log file created at: 2013/06/17 15:59:58
    Running on machine: ip-10-245-112-238
    Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
    I0617 15:59:58.518903 9628 daemon.cc:34] impalad version 1.0 RELEASE
    (build d1bf0d1dac339af3692ffa17a5e3fdae0aed751f)
    Built on Sun, 28 Apr 2013 15:58:48 PST
    I0617 15:59:58.547598 9628 daemon.cc:35] Using hostname:
    ip-10-245-112-238.us-west-2.compute.internal
    I0617 15:59:58.548459 9628 logging.cc:76] Flags (see also /varz are on
    debug webserver):
    --dump_ir=false
    --module_output=
    --be_port=22000
    --hostname=ip-10-245-112-238.us-west-2.compute.internal
    --keytab_file=
    --mem_limit=80%
    --planservice_host=localhost
    --planservice_port=20000
    --principal=
    --exchg_node_buffer_size_bytes=10485760
    --max_row_batches=0
    --randomize_splits=false
    --num_disks=0
    --num_threads_per_disk=1
    --read_size=8388608
    --enable_webserver=true
    --state_store_host=ip-10-245-112-238.us-west-2.compute.internal
    --state_store_subscriber_port=23000
    --use_statestore=true
    --nn=ip-10-245-112-238.us-west-2.compute.internal
    --nn_port=8020
    --serialize_batch=false
    --status_report_interval=5
    --compress_rowbatches=true
    --num_threads_per_core=3
    --abort_on_config_error=true
    --be_service_threads=64
    --beeswax_port=21000
    --default_query_options=
    --fe_service_threads=64
    --heap_profile_dir=
    --hs2_port=21050
    --load_catalog_at_startup=false
    --log_mem_usage_interval=0
    --log_query_to_file=true
    --query_log_size=25
    --use_planservice=false
    --statestore_subscriber_timeout_seconds=10
    --state_store_port=24000
    --statestore_max_missed_heartbeats=5
    --statestore_num_heartbeat_threads=10
    --statestore_suspect_heartbeats=2
    --kerberos_reinit_interval=60

    --sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
    --web_log_bytes=1048576
    --log_filename=impalad
    --periodic_counter_update_period_ms=500
    --rpc_cnxn_attempts=10
    --rpc_cnxn_retry_interval_ms=2000
    --enable_webserver_doc_root=true
    --webserver_doc_root=/usr/lib/impala
    --webserver_interface=
    --webserver_port=25000

    --flagfile=/run/cloudera-scm-agent/process/40-impala-IMPALAD/impala-conf/impalad_flags
    --fromenv=
    --tryfromenv=
    --undefok=
    --tab_completion_columns=80
    --tab_completion_word=
    --help=false
    --helpfull=false
    --helpmatch=
    --helpon=
    --helppackage=false
    --helpshort=false
    --helpxml=false
    --version=false
    --alsologtoemail=
    --alsologtostderr=false
    --drop_log_memory=true
    --log_backtrace_at=
    --log_dir=/var/log/impalad
    --log_link=
    --log_prefix=true
    --logbuflevel=-1
    --logbufsecs=30
    --logemaillevel=999
    --logmailer=/bin/mail
    --logtostderr=false
    --max_log_size=200
    --minloglevel=0
    --stderrthreshold=2
    --stop_logging_if_full_disk=false
    --symbolize_stacktrace=true
    --v=1
    --vmodule=
    I0617 15:59:58.551378 9628 mem-info.cc:66] Physical Memory: 3.66 GB
    I0617 15:59:58.552459 9628 daemon.cc:43] Cpu Info:
    Model: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
    Cores: 1
    L1 Cache: 32.00 KB
    L2 Cache: 256.00 KB
    L3 Cache: 20.00 MB
    Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    I0617 15:59:58.552492 9628 daemon.cc:44] Disk Info:
    Num disks 2: xvda, xvdb
    I0617 15:59:58.552525 9628 daemon.cc:45] Mem Info: 3.66 GB
    I0617 16:00:07.111661 9628 impala-server.cc:1809] Default query
    options:TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_exceeded (bool) = false,
    }
    WARN0617 16:00:07.622000 main org.apache.hadoop.conf.Configuration]
    mapred.max.split.size is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.maxsize
    WARN0617 16:00:07.629000 main org.apache.hadoop.conf.Configuration]
    mapred.min.split.size is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.Configuration]
    mapred.min.split.size.per.rack is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize.per.rack
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.Configuration]
    mapred.min.split.size.per.node is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize.per.node
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.Configuration]
    mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.Configuration]
    mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
    mapreduce.reduce.speculative
    WARN0617 16:00:08.620000 main org.apache.hadoop.conf.Configuration]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@24c68a98:an
    attempt to override final parameter:
    mapreduce.job.end-notification.max.retry.interval; Ignoring.
    WARN0617 16:00:08.689000 main org.apache.hadoop.conf.Configuration]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@24c68a98:an
    attempt to override final parameter:
    mapreduce.job.end-notification.max.attempts; Ignoring.
    WARN0617 16:00:08.713000 main org.apache.hadoop.hive.conf.HiveConf]
    DEPRECATED: Configuration property hive.metastore.local no longer has any
    effect. Make sure to provide a valid value for hive.metastore.uris if you
    are connecting to a remote metastore.
    INFO0617 16:00:08.882000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:00:09.081000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:10.081000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:10.082000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:00:10.084000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:11.084000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:11.085000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:00:11.087000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:12.092000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:12.093000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:00:12.095000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:13.095000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:13.096000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:00:13.098000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:14.098000 main hive.metastore] Connected to metastore.
    I0617 16:00:16.692451 9628 impala-server.cc:2074] Impala Beeswax Service
    listening on 21000
    I0617 16:00:16.692525 9628 impala-server.cc:2085] Impala HiveServer2
    Service listening on 21050
    I0617 16:00:16.692553 9628 impala-server.cc:2093] ImpalaInternalService
    listening on 22000
    I0617 16:00:16.699118 9628 thrift-server.cc:365] ThriftServer 'backend'
    started on port: 22000
    I0617 16:00:16.699157 9628 exec-env.cc:146] Starting global services
    I0617 16:00:16.699275 9628 exec-env.cc:167] Using global memory limit:
    2.93 GB
    I0617 16:00:16.699409 9628 webserver.cc:129] Starting webserver on all
    interfaces, port 25000
    I0617 16:00:16.699470 9628 webserver.cc:139] Document root:
    /usr/lib/impala
    I0617 16:00:16.711299 9628 webserver.cc:178] Webserver started
    I0617 16:00:16.711383 9628 simple-scheduler.cc:99] Starting simple
    scheduler
    I0617 16:00:16.712973 9628 simple-scheduler.cc:133] Simple-scheduler
    using 10.245.112.238 as IP address
    I0617 16:00:16.713028 9628 state-store-subscriber.cc:137] Starting
    subscriber
    I0617 16:00:16.719905 9628 thrift-server.cc:365] ThriftServer
    'StateStoreSubscriber' started on port: 23000
    I0617 16:00:16.722825 9628 client-cache.cc:98] CreateClient(): adding new
    client for ip-10-245-112-238.us-west-2.compute.internal:24000
    I0617 16:00:16.745887 9628 thrift-server.cc:365] ThriftServer
    'beeswax-frontend' started on port: 21000
    I0617 16:00:16.761379 9628 thrift-server.cc:365] ThriftServer
    'hiveServer2-frontend' started on port: 21050
    I0617 16:00:16.761443 9628 impalad-main.cc:101] Impala has started.
    I0617 16:09:51.630908 9737 impala-beeswax-server.cc:137] query():
    query=select A.search_parameters,reseller_count,total from (select
    search_parameters,count(distinct(reseller)) as reseller_count from
    all_lookups group by search_parameters)A join (select
    search_parameters,count(*) as total from all_lookups group by
    search_parameters)B on A.search_parameters=B.search_parameters limit 50
    I0617 16:09:51.631084 9737 impala-beeswax-server.cc:477] query: Query {
    01: query (string) = "select A.search_[...](315)",
    03: configuration (list) = list<string>[0] {
    },
    04: hadoop_user (string) = "ubuntu",
    }
    I0617 16:09:51.639899 9737 impala-beeswax-server.cc:489]
    TClientRequest.queryOptions: TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_exceeded (bool) = false,
    }
    INFO0617 16:09:51.703000 Thread-3 com.cloudera.impala.service.Frontend]
    analyze query select A.search_parameters,reseller_count,total from (select
    search_parameters,count(distinct(reseller)) as reseller_count from
    all_lookups group by search_parameters)A join (select
    search_parameters,count(*) as total from all_lookups group by
    search_parameters)B on A.search_parameters=B.search_parameters limit 50
    INFO0617 16:09:52.332000 Thread-3 com.cloudera.impala.catalog.HdfsTable]
    load table all_lookups
    INFO0617 16:09:52.697000 Thread-3 com.cloudera.impala.catalog.HdfsTable]
    load partition block md for all_lookups
    INFO0617 16:09:52.716000 Thread-3 com.cloudera.impala.catalog.HdfsTable]
    loaded partition PartitionBlockMetadata{#blocks=0, #filenames=0,
    totalStringLen=0}
    INFO0617 16:09:52.949000 Thread-3 com.cloudera.impala.catalog.HdfsTable]
    loaded partition PartitionBlockMetadata{#blocks=8, #filenames=8,
    totalStringLen=1197}
    INFO0617 16:09:53.073000 Thread-3 com.cloudera.impala.catalog.HdfsTable]
    loaded disk ids for table default.all_lookups
    INFO0617 16:09:53.074000 Thread-3 com.cloudera.impala.catalog.HdfsTable] 1
    INFO0617 16:09:53.074000 Thread-3 com.cloudera.impala.catalog.HdfsTable]
    table #rows=0
    INFO0617 16:09:53.103000 Thread-3
    com.cloudera.impala.analysis.AggregateInfo] agg info:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=search_parameters,
    id=0} SlotRef{tblName=null, col=reseller, id=1}), aggregate_exprs=,
    agg_tuple=TupleDescriptor{id=1, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=2, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=3, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=null,
    col=search_parameters, id=0}:SlotRef{tblName=null, col=null, id=2}
    SlotRef{tblName=null, col=reseller, id=1}:SlotRef{tblName=null, col=null,
    id=3})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=null, id=2}
    SlotRef{tblName=null, col=null, id=3}), aggregate_exprs=,
    agg_tuple=TupleDescriptor{id=1, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=2, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=3, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=null,
    col=search_parameters, id=0}:SlotRef{tblName=null, col=null, id=2}
    SlotRef{tblName=null, col=reseller, id=1}:SlotRef{tblName=null, col=null,
    id=3})}
    secondPhaseDistinctAggInfo:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=null, id=2}),
    aggregate_exprs=(AggregateExpr{op=COUNT, isStar=false, isDistinct=false,
    (SlotRef{tblName=null, col=null, id=3})}), agg_tuple=TupleDescriptor{id=2,
    tbl=null, byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=4,
    col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=5,
    col=null, type=BIGINT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=null, col=search_parameters,
    id=0}:SlotRef{tblName=null, col=null, id=4} AggregateExpr{op=COUNT,
    isStar=false, isDistinct=true, (SlotRef{tblName=null, col=reseller,
    id=1})}:SlotRef{tblName=null, col=null, id=5})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=null, id=4}),
    aggregate_exprs=(AggregateExpr{op=SUM, isStar=false, isDistinct=false,
    (SlotRef{tblName=null, col=null, id=5})}), agg_tuple=TupleDescriptor{id=2,
    tbl=null, byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=4,
    col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=5,
    col=null, type=BIGINT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=null, col=search_parameters,
    id=0}:SlotRef{tblName=null, col=null, id=4} AggregateExpr{op=COUNT,
    isStar=false, isDistinct=true, (SlotRef{tblName=null, col=reseller,
    id=1})}:SlotRef{tblName=null, col=null, id=5})}
    INFO0617 16:09:53.110000 Thread-3
    com.cloudera.impala.analysis.AggregateInfo] agg info:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=search_parameters,
    id=8}), aggregate_exprs=(AggregateExpr{op=COUNT, isStar=true,
    isDistinct=false, }), agg_tuple=TupleDescriptor{id=5, tbl=null,
    byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=9, col=null,
    type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=10,
    col=null, type=BIGINT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=null, col=search_parameters,
    id=8}:SlotRef{tblName=null, col=null, id=9} AggregateExpr{op=COUNT,
    isStar=true, isDistinct=false, }:SlotRef{tblName=null, col=null, id=10})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=null, id=9}),
    aggregate_exprs=(AggregateExpr{op=SUM, isStar=false, isDistinct=false,
    (SlotRef{tblName=null, col=null, id=10})}), agg_tuple=TupleDescriptor{id=5,
    tbl=null, byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=9,
    col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=10,
    col=null, type=BIGINT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=null, col=search_parameters,
    id=8}:SlotRef{tblName=null, col=null, id=9} AggregateExpr{op=COUNT,
    isStar=true, isDistinct=false, }:SlotRef{tblName=null, col=null, id=10})}
    INFO0617 16:09:53.140000 Thread-3
    com.cloudera.impala.analysis.BinaryPredicate] A.search_parameters =
    B.search_parameters selectivity: 0.1
    INFO0617 16:09:53.257000 Thread-3 com.cloudera.impala.service.Frontend]
    create plan
    INFO0617 16:09:53.282000 Thread-3 com.cloudera.impala.planner.Planner]
    create single-node plan
    INFO0617 16:09:53.290000 Thread-3
    com.cloudera.impala.planner.AggregationNode] grouping expr:
    search_parameters #distinct=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] stats Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] grouping expr: <slot 2>
    #distinct=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] stats Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] grouping expr:
    search_parameters #distinct=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] stats Agg: cardinality=-1
    INFO0617 16:09:53.297000 Thread-3
    com.cloudera.impala.planner.HdfsScanNode] collecting partitions for table
    all_lookups
    INFO0617 16:09:53.297000 Thread-3
    com.cloudera.impala.planner.HdfsScanNode] finalize HdfsScan: cardinality=0
    INFO0617 16:09:53.297000 Thread-3
    com.cloudera.impala.planner.HdfsScanNode] finalize HdfsScan: #nodes=1
    INFO0617 16:09:53.297000 Thread-3
    com.cloudera.impala.planner.AggregationNode] grouping expr:
    search_parameters #distinct=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] stats Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] grouping expr: <slot 2>
    #distinct=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] stats Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.HdfsScanNode] collecting partitions for table
    all_lookups
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.HdfsScanNode] finalize HdfsScan: cardinality=0
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.HdfsScanNode] finalize HdfsScan: #nodes=1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] grouping expr:
    search_parameters #distinct=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] stats Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.HashJoinNode] stats HashJoin: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.Planner]
    create plan fragments
    INFO0617 16:09:53.299000 Thread-3 com.cloudera.impala.planner.Planner]
    memlimit=0
    INFO0617 16:09:53.310000 Thread-3 com.cloudera.impala.planner.Planner]
    broadcast: cost=0
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.Planner]
    card=-1 row_size=0.0 #nodes=1
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.Planner]
    partition: cost=0
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.Planner] lhs
    card=-1 row_size=24.0
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.Planner] rhs
    card=-1 row_size=0.0
    INFO0617 16:09:53.317000 Thread-3 com.cloudera.impala.planner.Planner]
    9:AGGREGATE
    output: SUM(<slot 10>)
    group by: <slot 9>
    8:EXCHANGE

    INFO0617 16:09:53.317000 Thread-3 com.cloudera.impala.planner.Planner]
    finalize plan fragments
    INFO0617 16:09:53.641000 Thread-3 com.cloudera.impala.service.Frontend]
    get scan range locations
    INFO0617 16:09:53.678000 Thread-3 com.cloudera.impala.service.Frontend]
    create result set metadata
    INFO0617 16:09:53.690000 Thread-3 com.cloudera.impala.service.JniFrontend]
    PLAN FRAGMENT 0
    PARTITION: UNPARTITIONED

    11:EXCHANGE
    limit: 50
    tuple ids: 2 5

    PLAN FRAGMENT 1
    PARTITION: HASH_PARTITIONED: <slot 2>

    STREAM DATA SINK
    EXCHANGE ID: 11
    UNPARTITIONED

    5:HASH JOIN
    join op: INNER JOIN (BROADCAST)
    hash predicates:
    <slot 4> = <slot 9>
    limit: 50
    tuple ids: 2 5

    ----10:EXCHANGE
    tuple ids: 5
    2:AGGREGATE
    output: COUNT(<slot 3>)
    group by: <slot 2>
    tuple ids: 2
    7:AGGREGATE
    group by: <slot 2>, <slot 3>
    tuple ids: 1
    6:EXCHANGE
    tuple ids: 1

    PLAN FRAGMENT 2
    PARTITION: HASH_PARTITIONED: <slot 9>

    STREAM DATA SINK
    EXCHANGE ID: 10
    UNPARTITIONED

    9:AGGREGATE
    output: SUM(<slot 10>)
    group by: <slot 9>
    tuple ids: 5
    8:EXCHANGE
    tuple ids: 5

    PLAN FRAGMENT 3
    PARTITION: RANDOM

    STREAM DATA SINK
    EXCHANGE ID: 8
    HASH_PARTITIONED: <slot 9>

    4:AGGREGATE
    output: COUNT(*)
    group by: search_parameters
    tuple ids: 5
    3:SCAN HDFS
    table=default.all_lookups #partitions=1 size=397.15MB
    tuple ids: 4

    PLAN FRAGMENT 4
    PARTITION: RANDOM

    STREAM DATA SINK
    EXCHANGE ID: 6
    HASH_PARTITIONED: <slot 2>

    1:AGGREGATE
    group by: search_parameters, reseller
    tuple ids: 1
    0:SCAN HDFS
    table=default.all_lookups #partitions=1 size=397.15MB
    tuple ids: 0

    I0617 16:09:53.755270 9737 coordinator.cc:290] Exec()
    query_id=8eb364939bbb41b0:a0a6981049ec71f0
    I0617 16:09:53.755636 9737 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:a0a6981049ec71f0
    instance_id=8eb364939bbb41b0:a0a6981049ec71f1
    I0617 16:09:53.884407 9737 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:a0a6981049ec71f1
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:53.888123 9737 exchange-node.cc:50] Exch id=11
    input_desc=Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])

    output_desc=Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.021406 9737 coordinator.cc:377] starting 4 backends for
    query 8eb364939bbb41b0:a0a6981049ec71f0
    I0617 16:09:54.021667 9900 client-cache.cc:98] CreateClient(): adding new
    client for ip-10-245-112-238.us-west-2.compute.internal:22000
    I0617 16:09:54.022987 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:a0a6981049ec71f2
    coord=ip-10-245-112-238.us-west-2.compute.internal:22000 backend#=0
    I0617 16:09:54.023074 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:a0a6981049ec71f0
    instance_id=8eb364939bbb41b0:a0a6981049ec71f2
    I0617 16:09:54.032918 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:a0a6981049ec71f2
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.034090 9901 exchange-node.cc:50] Exch id=6
    input_desc=Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])

    output_desc=Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1
    offset=8 null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    I0617 16:09:54.044239 9901 exchange-node.cc:50] Exch id=10
    input_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])

    output_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.618634 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:a0a6981049ec71f3
    coord=ip-10-245-112-238.us-west-2.compute.internal:22000 backend#=1
    I0617 16:09:54.618748 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:a0a6981049ec71f0
    instance_id=8eb364939bbb41b0:a0a6981049ec71f3
    I0617 16:09:54.623918 9902 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:a0a6981049ec71f2
    I0617 16:09:54.631587 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:a0a6981049ec71f3
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.631726 9901 exchange-node.cc:50] Exch id=8
    input_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])

    output_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.974669 9905 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:a0a6981049ec71f3
    I0617 16:09:54.977622 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:a0a6981049ec71f4
    coord=ip-10-245-112-238.us-west-2.compute.internal:22000 backend#=2
    I0617 16:09:54.977705 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:a0a6981049ec71f0
    instance_id=8eb364939bbb41b0:a0a6981049ec71f4
    I0617 16:09:55.005800 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:a0a6981049ec71f4
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:55.522994 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:a0a6981049ec71f5
    coord=ip-10-245-112-238.us-west-2.compute.internal:22000 backend#=3
    I0617 16:09:55.523129 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:a0a6981049ec71f0
    instance_id=8eb364939bbb41b0:a0a6981049ec71f5
    I0617 16:09:55.529832 9908 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:a0a6981049ec71f4
    I0617 16:09:55.550024 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:a0a6981049ec71f5
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:56.376770 9917 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:a0a6981049ec71f1
    I0617 16:09:56.379158 9916 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:a0a6981049ec71f5
    I0617 16:09:56.474069 9737 impala-beeswax-server.cc:290]
    get_results_metadata(): query_id=8eb364939bbb41b0:a0a6981049ec71f0
    I0617 16:10:01.533941 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 12% Complete (2 out of 16)
    I0617 16:10:02.386214 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 25% Complete (4 out of 16)
    I0617 16:10:06.535214 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 31% Complete (5 out of 16)
    I0617 16:10:07.386981 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 37% Complete (6 out of 16)
    I0617 16:10:11.535930 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 43% Complete (7 out of 16)
    I0617 16:10:12.387742 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 50% Complete (8 out of 16)
    I0617 16:10:16.536659 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 62% Complete (10 out of 16)
    I0617 16:10:17.388424 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 75% Complete (12 out of 16)
    I0617 16:10:21.556238 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 87% Complete (14 out of 16)
    I0617 16:10:21.556130 10155 client-cache.cc:98] CreateClient(): adding new
    client for ip-10-245-112-238.us-west-2.compute.internal:22000
    I0617 16:10:22.389418 10156 progress-updater.cc:45] Query
    8eb364939bbb41b0:a0a6981049ec71f0 100% Complete (16 out of 16)
    I0617 16:10:25.982318 9906 client-cache.cc:98] CreateClient(): adding new
    client for ip-10-245-112-238.us-west-2.compute.internal:22000

    On Monday, June 17, 2013 12:30:34 PM UTC-4, Ricky Saltzer wrote:

    It would be the log file before this one
    On Jun 17, 2013 12:14 PM, "Alex Minnaar" wrote:

    It looks like the last log file was this one. But I'm not sure it tells
    us anything either.

    Log file created at: 2013/06/17 16:10:41
    Running on machine: ip-10-245-112-238
    Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
    I0617 16:10:41.500978 15390 daemon.cc:34] impalad version 1.0 RELEASE
    (build d1bf0d1dac339af3692ffa17a5e3fd**ae0aed751f)
    Built on Sun, 28 Apr 2013 15:58:48 PST
    I0617 16:10:41.521280 15390 daemon.cc:35] Using hostname:
    ip-10-245-112-238.us-west-2.**compute.internal
    I0617 16:10:41.522235 15390 logging.cc:76] Flags (see also /varz are on
    debug webserver):
    --dump_ir=false
    --module_output=
    --be_port=22000
    --hostname=ip-10-245-112-238.**us-west-2.compute.internal
    --keytab_file=
    --mem_limit=80%
    --planservice_host=localhost
    --planservice_port=20000
    --principal=
    --exchg_node_buffer_size_**bytes=10485760
    --max_row_batches=0
    --randomize_splits=false
    --num_disks=0
    --num_threads_per_disk=1
    --read_size=8388608
    --enable_webserver=true
    --state_store_host=ip-10-245-**112-238.us-west-2.compute.**internal
    --state_store_subscriber_port=**23000
    --use_statestore=true
    --nn=ip-10-245-112-238.us-**west-2.compute.internal
    --nn_port=8020
    --serialize_batch=false
    --status_report_interval=5
    --compress_rowbatches=true
    --num_threads_per_core=3
    --abort_on_config_error=true
    --be_service_threads=64
    --beeswax_port=21000
    --default_query_options=
    --fe_service_threads=64
    --heap_profile_dir=
    --hs2_port=21050
    --load_catalog_at_startup=**false
    --log_mem_usage_interval=0
    --log_query_to_file=true
    --query_log_size=25
    --use_planservice=false
    --statestore_subscriber_**timeout_seconds=10
    --state_store_port=24000
    --statestore_max_missed_**heartbeats=5
    --statestore_num_heartbeat_**threads=10
    --statestore_suspect_**heartbeats=2
    --kerberos_reinit_interval=60
    --sasl_path=/usr/lib/sasl2:/**usr/lib64/sasl2:/usr/local/**
    lib/sasl2:/usr/lib/x86_64-**linux-gnu/sasl2
    --web_log_bytes=1048576
    --log_filename=impalad
    --periodic_counter_update_**period_ms=500
    --rpc_cnxn_attempts=10
    --rpc_cnxn_retry_interval_ms=**2000
    --enable_webserver_doc_root=**true
    --webserver_doc_root=/usr/lib/**impala
    --webserver_interface=
    --webserver_port=25000
    --flagfile=/run/cloudera-scm-**agent/process/40-impala-**
    IMPALAD/impala-conf/impalad_**flags
    --fromenv=
    --tryfromenv=
    --undefok=
    --tab_completion_columns=80
    --tab_completion_word=
    --help=false
    --helpfull=false
    --helpmatch=
    --helpon=
    --helppackage=false
    --helpshort=false
    --helpxml=false
    --version=false
    --alsologtoemail=
    --alsologtostderr=false
    --drop_log_memory=true
    --log_backtrace_at=
    --log_dir=/var/log/impalad
    --log_link=
    --log_prefix=true
    --logbuflevel=-1
    --logbufsecs=30
    --logemaillevel=999
    --logmailer=/bin/mail
    --logtostderr=false
    --max_log_size=200
    --minloglevel=0
    --stderrthreshold=2
    --stop_logging_if_full_disk=**false
    --symbolize_stacktrace=true
    --v=1
    --vmodule=
    I0617 16:10:41.533797 15390 mem-info.cc:66] Physical Memory: 3.66 GB
    I0617 16:10:41.539779 15390 daemon.cc:43] Cpu Info:
    Model: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
    Cores: 1
    L1 Cache: 32.00 KB
    L2 Cache: 256.00 KB
    L3 Cache: 20.00 MB
    Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    I0617 16:10:41.539821 15390 daemon.cc:44] Disk Info:
    Num disks 2: xvda, xvdb
    I0617 16:10:41.539855 15390 daemon.cc:45] Mem Info: 3.66 GB
    I0617 16:10:50.755198 15390 impala-server.cc:1809] Default query
    options:TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_**exceeded (bool) = false,
    }
    WARN0617 16:10:51.236000 main org.apache.hadoop.conf.**Configuration]
    mapred.max.split.size is deprecated. Instead, use mapreduce.input.**
    fileinputformat.split.maxsize
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configuration]
    mapred.min.split.size is deprecated. Instead, use mapreduce.input.**
    fileinputformat.split.minsize
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configuration]
    mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.
    **fileinputformat.split.minsize.**per.rack
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configuration]
    mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.
    **fileinputformat.split.minsize.**per.node
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configuration]
    mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configuration]
    mapred.reduce.tasks.**speculative.execution is deprecated. Instead, use
    mapreduce.reduce.speculative
    WARN0617 16:10:51.892000 main org.apache.hadoop.conf.**Configuration]
    org.apache.hadoop.hive.conf.**LoopingByteArrayInputStream@**66b51404:an
    attempt to override final parameter: mapreduce.job.end-**
    notification.max.retry.**interval; Ignoring.
    WARN0617 16:10:51.954000 main org.apache.hadoop.conf.**Configuration]
    org.apache.hadoop.hive.conf.**LoopingByteArrayInputStream@**66b51404:an
    attempt to override final parameter: mapreduce.job.end-**notification.max.attempts;
    Ignoring.
    WARN0617 16:10:51.972000 main org.apache.hadoop.hive.conf.**HiveConf]
    DEPRECATED: Configuration property hive.metastore.local no longer has any
    effect. Make sure to provide a valid value for hive.metastore.uris if you
    are connecting to a remote metastore.
    INFO0617 16:10:52.240000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-**
    west-2.compute.internal:9083
    INFO0617 16:10:52.494000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:10:53.495000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:53.495000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-**
    west-2.compute.internal:9083
    INFO0617 16:10:53.497000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:10:54.498000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:54.498000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-**
    west-2.compute.internal:9083
    INFO0617 16:10:54.505000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:10:55.506000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:55.507000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-**
    west-2.compute.internal:9083
    INFO0617 16:10:55.515000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:10:56.515000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:56.515000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-**
    west-2.compute.internal:9083
    INFO0617 16:10:56.519000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:10:57.519000 main hive.metastore] Connected to metastore.
    I0617 16:10:59.891072 15390 impala-server.cc:2074] Impala Beeswax
    Service listening on 21000
    I0617 16:10:59.893288 15390 impala-server.cc:2085] Impala HiveServer2
    Service listening on 21050
    I0617 16:10:59.893335 15390 impala-server.cc:2093] ImpalaInternalService
    listening on 22000
    I0617 16:10:59.905498 15390 thrift-server.cc:365] ThriftServer 'backend'
    started on port: 22000
    I0617 16:10:59.905547 15390 exec-env.cc:146] Starting global services
    I0617 16:10:59.905670 15390 exec-env.cc:167] Using global memory limit:
    2.93 GB
    I0617 16:10:59.909502 15390 webserver.cc:129] Starting webserver on all
    interfaces, port 25000
    I0617 16:10:59.909559 15390 webserver.cc:139] Document root:
    /usr/lib/impala
    I0617 16:10:59.921815 15390 webserver.cc:178] Webserver started
    I0617 16:10:59.921897 15390 simple-scheduler.cc:99] Starting simple
    scheduler
    I0617 16:10:59.926578 15390 simple-scheduler.cc:133] Simple-scheduler
    using 10.245.112.238 as IP address
    I0617 16:10:59.926632 15390 state-store-subscriber.cc:137] Starting
    subscriber
    I0617 16:10:59.937484 15390 thrift-server.cc:365] ThriftServer
    'StateStoreSubscriber' started on port: 23000
    I0617 16:10:59.949283 15390 client-cache.cc:98] CreateClient(): adding
    new client for ip-10-245-112-238.us-west-2.**compute.internal:24000
    I0617 16:10:59.982250 15390 thrift-server.cc:365] ThriftServer
    'beeswax-frontend' started on port: 21000
    I0617 16:11:00.017025 15390 thrift-server.cc:365] ThriftServer
    'hiveServer2-frontend' started on port: 21050
    I0617 16:11:00.017144 15390 impalad-main.cc:101] Impala has started.

    On Monday, June 17, 2013 12:08:01 PM UTC-4, Ricky Saltzer wrote:

    Hey Alex -

    Sorry about that, I forgot that the log get's rolled over after a
    crash. Inside of */var/log/impalad*, you should be able to find the
    last log file written to before the crash using an "*ls -lthr*", take
    a peek inside..it should contain a lot more data.

    Thanks,
    Ricky

    On Mon, Jun 17, 2013 at 9:04 AM, Alex Minnaar wrote:

    I have attached what was in the impalad.INFO file

    Alex

    On Monday, June 17, 2013 11:44:10 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Try connecting to the node via SSH and grabbing the log directly
    from */var/log/impalad *
    *
    *
    Ricky

    On Mon, Jun 17, 2013 at 8:39 AM, Alex Minnaar wrote:

    OK I figured out how to set the GLOG level, but when I try to
    download the impalad.INFO file I get

    Problem accessing /cmf/process/all/logs/**download****. Reason:

    Connection timed out

    Could not connect to host.

    On Monday, June 17, 2013 10:39:21 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Could you set the GLOG level to 2<http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_logging.html> (you
    can use Cloudera manager to do it, too). Then, re-run your query and attach
    the impalad.INFO from the coordinator (the node you submit your query to).

    Ricky

    On Mon, Jun 17, 2013 at 6:23 AM, Alex Minnaar wrote:

    Impala is working for me with simple select queries but it is not
    working with queries involving joins. When I execute the join query in the
    impala shell, I get the following result.
    Query finished, fetching results ...
    Error communicating with impalad: TSocket read 0 bytes

    And then I am no longer connected to impala. How do I fix this?

    Alex


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com
  • Alex Minnaar at Jun 17, 2013 at 5:20 pm
    Yes that's the one
    On Monday, June 17, 2013 1:18:50 PM UTC-4, Ricky Saltzer wrote:

    Hey Alex -

    So, the following query is the one crashing?

    select A.search_parameters,reseller_count,total from (select
    search_parameters,count(distinct(reseller)) as reseller_count from
    all_lookups group by search_parameters)A join (select
    search_parameters,count(*) as total from all_lookups group by
    search_parameters)B on A.search_parameters=B.search_parameters limit 50

    Ricky


    On Mon, Jun 17, 2013 at 10:05 AM, Alex Minnaar <minna...@gmail.com<javascript:>
    wrote:
    OK the previous log file was

    Log file created at: 2013/06/17 15:59:58
    Running on machine: ip-10-245-112-238
    Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
    I0617 15:59:58.518903 9628 daemon.cc:34] impalad version 1.0 RELEASE
    (build d1bf0d1dac339af3692ffa17a5e3fdae0aed751f)
    Built on Sun, 28 Apr 2013 15:58:48 PST
    I0617 15:59:58.547598 9628 daemon.cc:35] Using hostname:
    ip-10-245-112-238.us-west-2.compute.internal
    I0617 15:59:58.548459 9628 logging.cc:76] Flags (see also /varz are on
    debug webserver):
    --dump_ir=false
    --module_output=
    --be_port=22000
    --hostname=ip-10-245-112-238.us-west-2.compute.internal
    --keytab_file=
    --mem_limit=80%
    --planservice_host=localhost
    --planservice_port=20000
    --principal=
    --exchg_node_buffer_size_bytes=10485760
    --max_row_batches=0
    --randomize_splits=false
    --num_disks=0
    --num_threads_per_disk=1
    --read_size=8388608
    --enable_webserver=true
    --state_store_host=ip-10-245-112-238.us-west-2.compute.internal
    --state_store_subscriber_port=23000
    --use_statestore=true
    --nn=ip-10-245-112-238.us-west-2.compute.internal
    --nn_port=8020
    --serialize_batch=false
    --status_report_interval=5
    --compress_rowbatches=true
    --num_threads_per_core=3
    --abort_on_config_error=true
    --be_service_threads=64
    --beeswax_port=21000
    --default_query_options=
    --fe_service_threads=64
    --heap_profile_dir=
    --hs2_port=21050
    --load_catalog_at_startup=false
    --log_mem_usage_interval=0
    --log_query_to_file=true
    --query_log_size=25
    --use_planservice=false
    --statestore_subscriber_timeout_seconds=10
    --state_store_port=24000
    --statestore_max_missed_heartbeats=5
    --statestore_num_heartbeat_threads=10
    --statestore_suspect_heartbeats=2
    --kerberos_reinit_interval=60

    --sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
    --web_log_bytes=1048576
    --log_filename=impalad
    --periodic_counter_update_period_ms=500
    --rpc_cnxn_attempts=10
    --rpc_cnxn_retry_interval_ms=2000
    --enable_webserver_doc_root=true
    --webserver_doc_root=/usr/lib/impala
    --webserver_interface=
    --webserver_port=25000

    --flagfile=/run/cloudera-scm-agent/process/40-impala-IMPALAD/impala-conf/impalad_flags
    --fromenv=
    --tryfromenv=
    --undefok=
    --tab_completion_columns=80
    --tab_completion_word=
    --help=false
    --helpfull=false
    --helpmatch=
    --helpon=
    --helppackage=false
    --helpshort=false
    --helpxml=false
    --version=false
    --alsologtoemail=
    --alsologtostderr=false
    --drop_log_memory=true
    --log_backtrace_at=
    --log_dir=/var/log/impalad
    --log_link=
    --log_prefix=true
    --logbuflevel=-1
    --logbufsecs=30
    --logemaillevel=999
    --logmailer=/bin/mail
    --logtostderr=false
    --max_log_size=200
    --minloglevel=0
    --stderrthreshold=2
    --stop_logging_if_full_disk=false
    --symbolize_stacktrace=true
    --v=1
    --vmodule=
    I0617 15:59:58.551378 9628 mem-info.cc:66] Physical Memory: 3.66 GB
    I0617 15:59:58.552459 9628 daemon.cc:43] Cpu Info:
    Model: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
    Cores: 1
    L1 Cache: 32.00 KB
    L2 Cache: 256.00 KB
    L3 Cache: 20.00 MB
    Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    I0617 15:59:58.552492 9628 daemon.cc:44] Disk Info:
    Num disks 2: xvda, xvdb
    I0617 15:59:58.552525 9628 daemon.cc:45] Mem Info: 3.66 GB
    I0617 16:00:07.111661 9628 impala-server.cc:1809] Default query
    options:TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_exceeded (bool) = false,
    }
    WARN0617 16:00:07.622000 main org.apache.hadoop.conf.Configuration]
    mapred.max.split.size is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.maxsize
    WARN0617 16:00:07.629000 main org.apache.hadoop.conf.Configuration]
    mapred.min.split.size is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.Configuration]
    mapred.min.split.size.per.rack is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize.per.rack
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.Configuration]
    mapred.min.split.size.per.node is deprecated. Instead, use
    mapreduce.input.fileinputformat.split.minsize.per.node
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.Configuration]
    mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.Configuration]
    mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
    mapreduce.reduce.speculative
    WARN0617 16:00:08.620000 main org.apache.hadoop.conf.Configuration]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@24c68a98:an attempt
    to override final parameter:
    mapreduce.job.end-notification.max.retry.interval; Ignoring.
    WARN0617 16:00:08.689000 main org.apache.hadoop.conf.Configuration]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@24c68a98:an attempt
    to override final parameter: mapreduce.job.end-notification.max.attempts;
    Ignoring.
    WARN0617 16:00:08.713000 main org.apache.hadoop.hive.conf.HiveConf]
    DEPRECATED: Configuration property hive.metastore.local no longer has any
    effect. Make sure to provide a valid value for hive.metastore.uris if you
    are connecting to a remote metastore.
    INFO0617 16:00:08.882000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:00:09.081000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:10.081000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:10.082000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:00:10.084000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:11.084000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:11.085000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:00:11.087000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:12.092000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:12.093000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:00:12.095000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:13.095000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:13.096000 main hive.metastore] Trying to connect to
    metastore with URI
    thrift://ip-10-245-112-238.us-west-2.compute.internal:9083
    INFO0617 16:00:13.098000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:14.098000 main hive.metastore] Connected to metastore.
    I0617 16:00:16.692451 9628 impala-server.cc:2074] Impala Beeswax Service
    listening on 21000
    I0617 16:00:16.692525 9628 impala-server.cc:2085] Impala HiveServer2
    Service listening on 21050
    I0617 16:00:16.692553 9628 impala-server.cc:2093] ImpalaInternalService
    listening on 22000
    I0617 16:00:16.699118 9628 thrift-server.cc:365] ThriftServer 'backend'
    started on port: 22000
    I0617 16:00:16.699157 9628 exec-env.cc:146] Starting global services
    I0617 16:00:16.699275 9628 exec-env.cc:167] Using global memory limit:
    2.93 GB
    I0617 16:00:16.699409 9628 webserver.cc:129] Starting webserver on all
    interfaces, port 25000
    I0617 16:00:16.699470 9628 webserver.cc:139] Document root:
    /usr/lib/impala
    I0617 16:00:16.711299 9628 webserver.cc:178] Webserver started
    I0617 16:00:16.711383 9628 simple-scheduler.cc:99] Starting simple
    scheduler
    I0617 16:00:16.712973 9628 simple-scheduler.cc:133] Simple-scheduler
    using 10.245.112.238 as IP address
    I0617 16:00:16.713028 9628 state-store-subscriber.cc:137] Starting
    subscriber
    I0617 16:00:16.719905 9628 thrift-server.cc:365] ThriftServer
    'StateStoreSubscriber' started on port: 23000
    I0617 16:00:16.722825 9628 client-cache.cc:98] CreateClient(): adding
    new client for ip-10-245-112-238.us-west-2.compute.internal:24000
    I0617 16:00:16.745887 9628 thrift-server.cc:365] ThriftServer
    'beeswax-frontend' started on port: 21000
    I0617 16:00:16.761379 9628 thrift-server.cc:365] ThriftServer
    'hiveServer2-frontend' started on port: 21050
    I0617 16:00:16.761443 9628 impalad-main.cc:101] Impala has started.
    I0617 16:09:51.630908 9737 impala-beeswax-server.cc:137] query():
    query=select A.search_parameters,reseller_count,total from (select
    search_parameters,count(distinct(reseller)) as reseller_count from
    all_lookups group by search_parameters)A join (select
    search_parameters,count(*) as total from all_lookups group by
    search_parameters)B on A.search_parameters=B.search_parameters limit 50
    I0617 16:09:51.631084 9737 impala-beeswax-server.cc:477] query: Query {
    01: query (string) = "select A.search_[...](315)",
    03: configuration (list) = list<string>[0] {
    },
    04: hadoop_user (string) = "ubuntu",
    }
    I0617 16:09:51.639899 9737 impala-beeswax-server.cc:489]
    TClientRequest.queryOptions: TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_exceeded (bool) = false,
    }
    INFO0617 16:09:51.703000 Thread-3 com.cloudera.impala.service.Frontend]
    analyze query select A.search_parameters,reseller_count,total from (select
    search_parameters,count(distinct(reseller)) as reseller_count from
    all_lookups group by search_parameters)A join (select
    search_parameters,count(*) as total from all_lookups group by
    search_parameters)B on A.search_parameters=B.search_parameters limit 50
    INFO0617 16:09:52.332000 Thread-3 com.cloudera.impala.catalog.HdfsTable]
    load table all_lookups
    INFO0617 16:09:52.697000 Thread-3 com.cloudera.impala.catalog.HdfsTable]
    load partition block md for all_lookups
    INFO0617 16:09:52.716000 Thread-3 com.cloudera.impala.catalog.HdfsTable]
    loaded partition PartitionBlockMetadata{#blocks=0, #filenames=0,
    totalStringLen=0}
    INFO0617 16:09:52.949000 Thread-3 com.cloudera.impala.catalog.HdfsTable]
    loaded partition PartitionBlockMetadata{#blocks=8, #filenames=8,
    totalStringLen=1197}
    INFO0617 16:09:53.073000 Thread-3 com.cloudera.impala.catalog.HdfsTable]
    loaded disk ids for table default.all_lookups
    INFO0617 16:09:53.074000 Thread-3 com.cloudera.impala.catalog.HdfsTable] 1
    INFO0617 16:09:53.074000 Thread-3 com.cloudera.impala.catalog.HdfsTable]
    table #rows=0
    INFO0617 16:09:53.103000 Thread-3
    com.cloudera.impala.analysis.AggregateInfo] agg info:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null,
    col=search_parameters, id=0} SlotRef{tblName=null, col=reseller, id=1}),
    aggregate_exprs=, agg_tuple=TupleDescriptor{id=1, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=2, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=3, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=null,
    col=search_parameters, id=0}:SlotRef{tblName=null, col=null, id=2}
    SlotRef{tblName=null, col=reseller, id=1}:SlotRef{tblName=null, col=null,
    id=3})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=null, id=2}
    SlotRef{tblName=null, col=null, id=3}), aggregate_exprs=,
    agg_tuple=TupleDescriptor{id=1, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=2, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=3, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=null,
    col=search_parameters, id=0}:SlotRef{tblName=null, col=null, id=2}
    SlotRef{tblName=null, col=reseller, id=1}:SlotRef{tblName=null, col=null,
    id=3})}
    secondPhaseDistinctAggInfo:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=null, id=2}),
    aggregate_exprs=(AggregateExpr{op=COUNT, isStar=false, isDistinct=false,
    (SlotRef{tblName=null, col=null, id=3})}), agg_tuple=TupleDescriptor{id=2,
    tbl=null, byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=4,
    col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=5,
    col=null, type=BIGINT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=null, col=search_parameters,
    id=0}:SlotRef{tblName=null, col=null, id=4} AggregateExpr{op=COUNT,
    isStar=false, isDistinct=true, (SlotRef{tblName=null, col=reseller,
    id=1})}:SlotRef{tblName=null, col=null, id=5})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=null, id=4}),
    aggregate_exprs=(AggregateExpr{op=SUM, isStar=false, isDistinct=false,
    (SlotRef{tblName=null, col=null, id=5})}), agg_tuple=TupleDescriptor{id=2,
    tbl=null, byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=4,
    col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=5,
    col=null, type=BIGINT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=null, col=search_parameters,
    id=0}:SlotRef{tblName=null, col=null, id=4} AggregateExpr{op=COUNT,
    isStar=false, isDistinct=true, (SlotRef{tblName=null, col=reseller,
    id=1})}:SlotRef{tblName=null, col=null, id=5})}
    INFO0617 16:09:53.110000 Thread-3
    com.cloudera.impala.analysis.AggregateInfo] agg info:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null,
    col=search_parameters, id=8}), aggregate_exprs=(AggregateExpr{op=COUNT,
    isStar=true, isDistinct=false, }), agg_tuple=TupleDescriptor{id=5,
    tbl=null, byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=9,
    col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=10,
    col=null, type=BIGINT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=null, col=search_parameters,
    id=8}:SlotRef{tblName=null, col=null, id=9} AggregateExpr{op=COUNT,
    isStar=true, isDistinct=false, }:SlotRef{tblName=null, col=null, id=10})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=null, id=9}),
    aggregate_exprs=(AggregateExpr{op=SUM, isStar=false, isDistinct=false,
    (SlotRef{tblName=null, col=null, id=10})}), agg_tuple=TupleDescriptor{id=5,
    tbl=null, byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=9,
    col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=10,
    col=null, type=BIGINT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=null, col=search_parameters,
    id=8}:SlotRef{tblName=null, col=null, id=9} AggregateExpr{op=COUNT,
    isStar=true, isDistinct=false, }:SlotRef{tblName=null, col=null, id=10})}
    INFO0617 16:09:53.140000 Thread-3
    com.cloudera.impala.analysis.BinaryPredicate] A.search_parameters =
    B.search_parameters selectivity: 0.1
    INFO0617 16:09:53.257000 Thread-3 com.cloudera.impala.service.Frontend]
    create plan
    INFO0617 16:09:53.282000 Thread-3 com.cloudera.impala.planner.Planner]
    create single-node plan
    INFO0617 16:09:53.290000 Thread-3
    com.cloudera.impala.planner.AggregationNode] grouping expr:
    search_parameters #distinct=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] stats Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] grouping expr: <slot 2>
    #distinct=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] stats Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] grouping expr:
    search_parameters #distinct=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3
    com.cloudera.impala.planner.AggregationNode] stats Agg: cardinality=-1
    INFO0617 16:09:53.297000 Thread-3
    com.cloudera.impala.planner.HdfsScanNode] collecting partitions for table
    all_lookups
    INFO0617 16:09:53.297000 Thread-3
    com.cloudera.impala.planner.HdfsScanNode] finalize HdfsScan: cardinality=0
    INFO0617 16:09:53.297000 Thread-3
    com.cloudera.impala.planner.HdfsScanNode] finalize HdfsScan: #nodes=1
    INFO0617 16:09:53.297000 Thread-3
    com.cloudera.impala.planner.AggregationNode] grouping expr:
    search_parameters #distinct=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] stats Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] grouping expr: <slot 2>
    #distinct=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] stats Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.HdfsScanNode] collecting partitions for table
    all_lookups
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.HdfsScanNode] finalize HdfsScan: cardinality=0
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.HdfsScanNode] finalize HdfsScan: #nodes=1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] grouping expr:
    search_parameters #distinct=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.AggregationNode] stats Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3
    com.cloudera.impala.planner.HashJoinNode] stats HashJoin: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.Planner]
    create plan fragments
    INFO0617 16:09:53.299000 Thread-3 com.cloudera.impala.planner.Planner]
    memlimit=0
    INFO0617 16:09:53.310000 Thread-3 com.cloudera.impala.planner.Planner]
    broadcast: cost=0
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.Planner]
    card=-1 row_size=0.0 #nodes=1
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.Planner]
    partition: cost=0
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.Planner]
    lhs card=-1 row_size=24.0
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.Planner]
    rhs card=-1 row_size=0.0
    INFO0617 16:09:53.317000 Thread-3 com.cloudera.impala.planner.Planner]
    9:AGGREGATE
    output: SUM(<slot 10>)
    group by: <slot 9>
    8:EXCHANGE

    INFO0617 16:09:53.317000 Thread-3 com.cloudera.impala.planner.Planner]
    finalize plan fragments
    INFO0617 16:09:53.641000 Thread-3 com.cloudera.impala.service.Frontend]
    get scan range locations
    INFO0617 16:09:53.678000 Thread-3 com.cloudera.impala.service.Frontend]
    create result set metadata
    INFO0617 16:09:53.690000 Thread-3
    com.cloudera.impala.service.JniFrontend] PLAN FRAGMENT 0
    PARTITION: UNPARTITIONED

    11:EXCHANGE
    limit: 50
    tuple ids: 2 5

    PLAN FRAGMENT 1
    PARTITION: HASH_PARTITIONED: <slot 2>

    STREAM DATA SINK
    EXCHANGE ID: 11
    UNPARTITIONED

    5:HASH JOIN
    join op: INNER JOIN (BROADCAST)
    hash predicates:
    <slot 4> = <slot 9>
    limit: 50
    tuple ids: 2 5

    ----10:EXCHANGE
    tuple ids: 5
    2:AGGREGATE
    output: COUNT(<slot 3>)
    group by: <slot 2>
    tuple ids: 2
    7:AGGREGATE
    group by: <slot 2>, <slot 3>
    tuple ids: 1
    6:EXCHANGE
    tuple ids: 1

    PLAN FRAGMENT 2
    PARTITION: HASH_PARTITIONED: <slot 9>

    STREAM DATA SINK
    EXCHANGE ID: 10
    UNPARTITIONED

    9:AGGREGATE
    output: SUM(<slot 10>)
    group by: <slot 9>
    tuple ids: 5
    8:EXCHANGE
    tuple ids: 5

    PLAN FRAGMENT 3
    PARTITION: RANDOM

    STREAM DATA SINK
    EXCHANGE ID: 8
    HASH_PARTITIONED: <slot 9>

    4:AGGREGATE
    output: COUNT(*)
    group by: search_parameters
    tuple ids: 5
    3:SCAN HDFS
    table=default.all_lookups #partitions=1 size=397.15MB
    tuple ids: 4

    PLAN FRAGMENT 4
    PARTITION: RANDOM

    STREAM DATA SINK
    EXCHANGE ID: 6
    HASH_PARTITIONED: <slot 2>

    1:AGGREGATE
    group by: search_parameters, reseller
    tuple ids: 1
    0:SCAN HDFS
    table=default.all_lookups #partitions=1 size=397.15MB
    tuple ids: 0

    I0617 16:09:53.755270 9737 coordinator.cc:290] Exec()
    query_id=8eb364939bbb41b0:a0a6981049ec71f0
    I0617 16:09:53.755636 9737 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:a0a6981049ec71f0
    instance_id=8eb364939bbb41b0:a0a6981049ec71f1
    I0617 16:09:53.884407 9737 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:a0a6981049ec71f1
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:53.888123 9737 exchange-node.cc:50] Exch id=11
    input_desc=Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])

    output_desc=Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.021406 9737 coordinator.cc:377] starting 4 backends for
    query 8eb364939bbb41b0:a0a6981049ec71f0
    I0617 16:09:54.021667 9900 client-cache.cc:98] CreateClient(): adding
    new client for ip-10-245-112-238.us-west-2.compute.internal:22000
    I0617 16:09:54.022987 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:a0a6981049ec71f2
    coord=ip-10-245-112-238.us-west-2.compute.internal:22000 backend#=0
    I0617 16:09:54.023074 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:a0a6981049ec71f0
    instance_id=8eb364939bbb41b0:a0a6981049ec71f2
    I0617 16:09:54.032918 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:a0a6981049ec71f2
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.034090 9901 exchange-node.cc:50] Exch id=6
    input_desc=Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1
    offset=8 null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])

    output_desc=Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1
    offset=8 null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    I0617 16:09:54.044239 9901 exchange-node.cc:50] Exch id=10
    input_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])

    output_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.618634 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:a0a6981049ec71f3
    coord=ip-10-245-112-238.us-west-2.compute.internal:22000 backend#=1
    I0617 16:09:54.618748 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:a0a6981049ec71f0
    instance_id=8eb364939bbb41b0:a0a6981049ec71f3
    I0617 16:09:54.623918 9902 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:a0a6981049ec71f2
    I0617 16:09:54.631587 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:a0a6981049ec71f3
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.631726 9901 exchange-node.cc:50] Exch id=8
    input_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])

    output_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.974669 9905 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:a0a6981049ec71f3
    I0617 16:09:54.977622 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:a0a6981049ec71f4
    coord=ip-10-245-112-238.us-west-2.compute.internal:22000 backend#=2
    I0617 16:09:54.977705 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:a0a6981049ec71f0
    instance_id=8eb364939bbb41b0:a0a6981049ec71f4
    I0617 16:09:55.005800 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:a0a6981049ec71f4
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:55.522994 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:a0a6981049ec71f5
    coord=ip-10-245-112-238.us-west-2.compute.internal:22000 backend#=3
    I0617 16:09:55.523129 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:a0a6981049ec71f0
    instance_id=8eb364939bbb41b0:a0a6981049ec71f5
    I0617 16:09:55.529832 9908 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:a0a6981049ec71f4
    I0617 16:09:55.550024 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:a0a6981049ec71f5
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:56.376770 9917 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:a0a6981049ec71f1
    I0617 16:09:56.379158 9916 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:a0a6981049ec71f5
    I0617 16:09:56.474069 9737 impala-beeswax-server.cc:290]
    get_results_metadata(): query_id=8eb364939bbb41b0:a0a6981049ec71f0
    I0617 16:10:01.533941 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 12% Complete (2 out of 16)
    I0617 16:10:02.386214 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 25% Complete (4 out of 16)
    I0617 16:10:06.535214 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 31% Complete (5 out of 16)
    I0617 16:10:07.386981 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 37% Complete (6 out of 16)
    I0617 16:10:11.535930 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 43% Complete (7 out of 16)
    I0617 16:10:12.387742 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 50% Complete (8 out of 16)
    I0617 16:10:16.536659 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 62% Complete (10 out of 16)
    I0617 16:10:17.388424 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 75% Complete (12 out of 16)
    I0617 16:10:21.556238 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:a0a6981049ec71f0: 87% Complete (14 out of 16)
    I0617 16:10:21.556130 10155 client-cache.cc:98] CreateClient(): adding
    new client for ip-10-245-112-238.us-west-2.compute.internal:22000
    I0617 16:10:22.389418 10156 progress-updater.cc:45] Query
    8eb364939bbb41b0:a0a6981049ec71f0 100% Complete (16 out of 16)
    I0617 16:10:25.982318 9906 client-cache.cc:98] CreateClient(): adding
    new client for ip-10-245-112-238.us-west-2.compute.internal:22000

    On Monday, June 17, 2013 12:30:34 PM UTC-4, Ricky Saltzer wrote:

    It would be the log file before this one
    On Jun 17, 2013 12:14 PM, "Alex Minnaar" wrote:

    It looks like the last log file was this one. But I'm not sure it tells
    us anything either.

    Log file created at: 2013/06/17 16:10:41
    Running on machine: ip-10-245-112-238
    Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
    I0617 16:10:41.500978 15390 daemon.cc:34] impalad version 1.0 RELEASE
    (build d1bf0d1dac339af3692ffa17a5e3fd**ae0aed751f)
    Built on Sun, 28 Apr 2013 15:58:48 PST
    I0617 16:10:41.521280 15390 daemon.cc:35] Using hostname:
    ip-10-245-112-238.us-west-2.**compute.internal
    I0617 16:10:41.522235 15390 logging.cc:76] Flags (see also /varz are on
    debug webserver):
    --dump_ir=false
    --module_output=
    --be_port=22000
    --hostname=ip-10-245-112-238.**us-west-2.compute.internal
    --keytab_file=
    --mem_limit=80%
    --planservice_host=localhost
    --planservice_port=20000
    --principal=
    --exchg_node_buffer_size_**bytes=10485760
    --max_row_batches=0
    --randomize_splits=false
    --num_disks=0
    --num_threads_per_disk=1
    --read_size=8388608
    --enable_webserver=true
    --state_store_host=ip-10-245-**112-238.us-west-2.compute.**internal
    --state_store_subscriber_port=**23000
    --use_statestore=true
    --nn=ip-10-245-112-238.us-**west-2.compute.internal
    --nn_port=8020
    --serialize_batch=false
    --status_report_interval=5
    --compress_rowbatches=true
    --num_threads_per_core=3
    --abort_on_config_error=true
    --be_service_threads=64
    --beeswax_port=21000
    --default_query_options=
    --fe_service_threads=64
    --heap_profile_dir=
    --hs2_port=21050
    --load_catalog_at_startup=**false
    --log_mem_usage_interval=0
    --log_query_to_file=true
    --query_log_size=25
    --use_planservice=false
    --statestore_subscriber_**timeout_seconds=10
    --state_store_port=24000
    --statestore_max_missed_**heartbeats=5
    --statestore_num_heartbeat_**threads=10
    --statestore_suspect_**heartbeats=2
    --kerberos_reinit_interval=60
    --sasl_path=/usr/lib/sasl2:/**usr/lib64/sasl2:/usr/local/**
    lib/sasl2:/usr/lib/x86_64-**linux-gnu/sasl2
    --web_log_bytes=1048576
    --log_filename=impalad
    --periodic_counter_update_**period_ms=500
    --rpc_cnxn_attempts=10
    --rpc_cnxn_retry_interval_ms=**2000
    --enable_webserver_doc_root=**true
    --webserver_doc_root=/usr/lib/**impala
    --webserver_interface=
    --webserver_port=25000
    --flagfile=/run/cloudera-scm-**agent/process/40-impala-**
    IMPALAD/impala-conf/impalad_**flags
    --fromenv=
    --tryfromenv=
    --undefok=
    --tab_completion_columns=80
    --tab_completion_word=
    --help=false
    --helpfull=false
    --helpmatch=
    --helpon=
    --helppackage=false
    --helpshort=false
    --helpxml=false
    --version=false
    --alsologtoemail=
    --alsologtostderr=false
    --drop_log_memory=true
    --log_backtrace_at=
    --log_dir=/var/log/impalad
    --log_link=
    --log_prefix=true
    --logbuflevel=-1
    --logbufsecs=30
    --logemaillevel=999
    --logmailer=/bin/mail
    --logtostderr=false
    --max_log_size=200
    --minloglevel=0
    --stderrthreshold=2
    --stop_logging_if_full_disk=**false
    --symbolize_stacktrace=true
    --v=1
    --vmodule=
    I0617 16:10:41.533797 15390 mem-info.cc:66] Physical Memory: 3.66 GB
    I0617 16:10:41.539779 15390 daemon.cc:43] Cpu Info:
    Model: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
    Cores: 1
    L1 Cache: 32.00 KB
    L2 Cache: 256.00 KB
    L3 Cache: 20.00 MB
    Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    I0617 16:10:41.539821 15390 daemon.cc:44] Disk Info:
    Num disks 2: xvda, xvdb
    I0617 16:10:41.539855 15390 daemon.cc:45] Mem Info: 3.66 GB
    I0617 16:10:50.755198 15390 impala-server.cc:1809] Default query
    options:TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_**exceeded (bool) = false,
    }
    WARN0617 16:10:51.236000 main org.apache.hadoop.conf.**Configuration]
    mapred.max.split.size is deprecated. Instead, use mapreduce.input.**
    fileinputformat.split.maxsize
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configuration]
    mapred.min.split.size is deprecated. Instead, use mapreduce.input.**
    fileinputformat.split.minsize
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configuration]
    mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.
    **fileinputformat.split.minsize.**per.rack
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configuration]
    mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.
    **fileinputformat.split.minsize.**per.node
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configuration]
    mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configuration]
    mapred.reduce.tasks.**speculative.execution is deprecated. Instead,
    use mapreduce.reduce.speculative
    WARN0617 16:10:51.892000 main org.apache.hadoop.conf.**Configuration]
    org.apache.hadoop.hive.conf.**LoopingByteArrayInputStream@**66b51404:an
    attempt to override final parameter: mapreduce.job.end-**
    notification.max.retry.**interval; Ignoring.
    WARN0617 16:10:51.954000 main org.apache.hadoop.conf.**Configuration]
    org.apache.hadoop.hive.conf.**LoopingByteArrayInputStream@**66b51404:an
    attempt to override final parameter: mapreduce.job.end-**notification.max.attempts;
    Ignoring.
    WARN0617 16:10:51.972000 main org.apache.hadoop.hive.conf.**HiveConf]
    DEPRECATED: Configuration property hive.metastore.local no longer has any
    effect. Make sure to provide a valid value for hive.metastore.uris if you
    are connecting to a remote metastore.
    INFO0617 16:10:52.240000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-**
    west-2.compute.internal:9083
    INFO0617 16:10:52.494000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:10:53.495000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:53.495000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-**
    west-2.compute.internal:9083
    INFO0617 16:10:53.497000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:10:54.498000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:54.498000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-**
    west-2.compute.internal:9083
    INFO0617 16:10:54.505000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:10:55.506000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:55.507000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-**
    west-2.compute.internal:9083
    INFO0617 16:10:55.515000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:10:56.515000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:56.515000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-**
    west-2.compute.internal:9083
    INFO0617 16:10:56.519000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:10:57.519000 main hive.metastore] Connected to metastore.
    I0617 16:10:59.891072 15390 impala-server.cc:2074] Impala Beeswax
    Service listening on 21000
    I0617 16:10:59.893288 15390 impala-server.cc:2085] Impala HiveServer2
    Service listening on 21050
    I0617 16:10:59.893335 15390 impala-server.cc:2093]
    ImpalaInternalService listening on 22000
    I0617 16:10:59.905498 15390 thrift-server.cc:365] ThriftServer
    'backend' started on port: 22000
    I0617 16:10:59.905547 15390 exec-env.cc:146] Starting global services
    I0617 16:10:59.905670 15390 exec-env.cc:167] Using global memory limit:
    2.93 GB
    I0617 16:10:59.909502 15390 webserver.cc:129] Starting webserver on all
    interfaces, port 25000
    I0617 16:10:59.909559 15390 webserver.cc:139] Document root:
    /usr/lib/impala
    I0617 16:10:59.921815 15390 webserver.cc:178] Webserver started
    I0617 16:10:59.921897 15390 simple-scheduler.cc:99] Starting simple
    scheduler
    I0617 16:10:59.926578 15390 simple-scheduler.cc:133] Simple-scheduler
    using 10.245.112.238 as IP address
    I0617 16:10:59.926632 15390 state-store-subscriber.cc:137] Starting
    subscriber
    I0617 16:10:59.937484 15390 thrift-server.cc:365] ThriftServer
    'StateStoreSubscriber' started on port: 23000
    I0617 16:10:59.949283 15390 client-cache.cc:98] CreateClient(): adding
    new client for ip-10-245-112-238.us-west-2.**compute.internal:24000
    I0617 16:10:59.982250 15390 thrift-server.cc:365] ThriftServer
    'beeswax-frontend' started on port: 21000
    I0617 16:11:00.017025 15390 thrift-server.cc:365] ThriftServer
    'hiveServer2-frontend' started on port: 21050
    I0617 16:11:00.017144 15390 impalad-main.cc:101] Impala has started.

    On Monday, June 17, 2013 12:08:01 PM UTC-4, Ricky Saltzer wrote:

    Hey Alex -

    Sorry about that, I forgot that the log get's rolled over after a
    crash. Inside of */var/log/impalad*, you should be able to find the
    last log file written to before the crash using an "*ls -lthr*", take
    a peek inside..it should contain a lot more data.

    Thanks,
    Ricky

    On Mon, Jun 17, 2013 at 9:04 AM, Alex Minnaar wrote:

    I have attached what was in the impalad.INFO file

    Alex

    On Monday, June 17, 2013 11:44:10 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Try connecting to the node via SSH and grabbing the log directly
    from */var/log/impalad *
    *
    *
    Ricky

    On Mon, Jun 17, 2013 at 8:39 AM, Alex Minnaar wrote:

    OK I figured out how to set the GLOG level, but when I try to
    download the impalad.INFO file I get

    Problem accessing /cmf/process/all/logs/**download****. Reason:

    Connection timed out

    Could not connect to host.

    On Monday, June 17, 2013 10:39:21 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Could you set the GLOG level to 2<http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_logging.html> (you
    can use Cloudera manager to do it, too). Then, re-run your query and attach
    the impalad.INFO from the coordinator (the node you submit your query to).

    Ricky

    On Mon, Jun 17, 2013 at 6:23 AM, Alex Minnaar wrote:

    Impala is working for me with simple select queries but it is not
    working with queries involving joins. When I execute the join query in the
    impala shell, I get the following result.
    Query finished, fetching results ...
    Error communicating with impalad: TSocket read 0 bytes

    And then I am no longer connected to impala. How do I fix this?

    Alex


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com

  • Alex Minnaar at Jun 17, 2013 at 6:00 pm
    I just tried the query again and it froze and now the health of that host
    has turned to bad, so I guess I need to fix that first
    On Monday, June 17, 2013 1:29:12 PM UTC-4, Ricky Saltzer wrote:

    Hey Alex -

    Are you sure GLOG was on level 2 when you issued this query? Maybe you
    could issue it once more now that GLOG is on level 2, and get then pull the
    log which contains that query. If you could, please attach the log instead
    of pasting (easier to read).

    Thanks
    Ricky


    On Mon, Jun 17, 2013 at 10:20 AM, Alex Minnaar <minna...@gmail.com<javascript:>
    wrote:
    Yes that's the one

    On Monday, June 17, 2013 1:18:50 PM UTC-4, Ricky Saltzer wrote:

    Hey Alex -

    So, the following query is the one crashing?

    select A.search_parameters,reseller_c**ount,total from (select
    search_parameters,count(distin**ct(reseller)) as reseller_count from
    all_lookups group by search_parameters)A join (select
    search_parameters,count(*) as total from all_lookups group by
    search_parameters)B on A.search_parameters=B.search_p**arameters limit
    50

    Ricky

    On Mon, Jun 17, 2013 at 10:05 AM, Alex Minnaar wrote:

    OK the previous log file was

    Log file created at: 2013/06/17 15:59:58
    Running on machine: ip-10-245-112-238
    Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
    I0617 15:59:58.518903 9628 daemon.cc:34] impalad version 1.0 RELEASE
    (build d1bf0d1dac339af3692ffa17a5e3fd**ae0aed751f)
    Built on Sun, 28 Apr 2013 15:58:48 PST
    I0617 15:59:58.547598 9628 daemon.cc:35] Using hostname:
    ip-10-245-112-238.us-west-2.**compute.internal
    I0617 15:59:58.548459 9628 logging.cc:76] Flags (see also /varz are on
    debug webserver):
    --dump_ir=false
    --module_output=
    --be_port=22000
    --hostname=ip-10-245-112-238.**us-west-2.compute.internal
    --keytab_file=
    --mem_limit=80%
    --planservice_host=localhost
    --planservice_port=20000
    --principal=
    --exchg_node_buffer_size_**bytes=10485760
    --max_row_batches=0
    --randomize_splits=false
    --num_disks=0
    --num_threads_per_disk=1
    --read_size=8388608
    --enable_webserver=true
    --state_store_host=ip-10-245-**112-238.us-west-2.compute.**internal
    --state_store_subscriber_port=**23000
    --use_statestore=true
    --nn=ip-10-245-112-238.us-**west-2.compute.internal
    --nn_port=8020
    --serialize_batch=false
    --status_report_interval=5
    --compress_rowbatches=true
    --num_threads_per_core=3
    --abort_on_config_error=true
    --be_service_threads=64
    --beeswax_port=21000
    --default_query_options=
    --fe_service_threads=64
    --heap_profile_dir=
    --hs2_port=21050
    --load_catalog_at_startup=**false
    --log_mem_usage_interval=0
    --log_query_to_file=true
    --query_log_size=25
    --use_planservice=false
    --statestore_subscriber_**timeout_seconds=10
    --state_store_port=24000
    --statestore_max_missed_**heartbeats=5
    --statestore_num_heartbeat_**threads=10
    --statestore_suspect_**heartbeats=2
    --kerberos_reinit_interval=60
    --sasl_path=/usr/lib/sasl2:/**usr/lib64/sasl2:/usr/local/**
    lib/sasl2:/usr/lib/x86_64-**linux-gnu/sasl2
    --web_log_bytes=1048576
    --log_filename=impalad
    --periodic_counter_update_**period_ms=500
    --rpc_cnxn_attempts=10
    --rpc_cnxn_retry_interval_ms=**2000
    --enable_webserver_doc_root=**true
    --webserver_doc_root=/usr/lib/**impala
    --webserver_interface=
    --webserver_port=25000
    --flagfile=/run/cloudera-scm-**agent/process/40-impala-**
    IMPALAD/impala-conf/impalad_**flags
    --fromenv=
    --tryfromenv=
    --undefok=
    --tab_completion_columns=80
    --tab_completion_word=
    --help=false
    --helpfull=false
    --helpmatch=
    --helpon=
    --helppackage=false
    --helpshort=false
    --helpxml=false
    --version=false
    --alsologtoemail=
    --alsologtostderr=false
    --drop_log_memory=true
    --log_backtrace_at=
    --log_dir=/var/log/impalad
    --log_link=
    --log_prefix=true
    --logbuflevel=-1
    --logbufsecs=30
    --logemaillevel=999
    --logmailer=/bin/mail
    --logtostderr=false
    --max_log_size=200
    --minloglevel=0
    --stderrthreshold=2
    --stop_logging_if_full_disk=**false
    --symbolize_stacktrace=true
    --v=1
    --vmodule=
    I0617 15:59:58.551378 9628 mem-info.cc:66] Physical Memory: 3.66 GB
    I0617 15:59:58.552459 9628 daemon.cc:43] Cpu Info:
    Model: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
    Cores: 1
    L1 Cache: 32.00 KB
    L2 Cache: 256.00 KB
    L3 Cache: 20.00 MB
    Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    I0617 15:59:58.552492 9628 daemon.cc:44] Disk Info:
    Num disks 2: xvda, xvdb
    I0617 15:59:58.552525 9628 daemon.cc:45] Mem Info: 3.66 GB
    I0617 16:00:07.111661 9628 impala-server.cc:1809] Default query
    options:TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_**exceeded (bool) = false,
    }
    WARN0617 16:00:07.622000 main org.apache.hadoop.conf.**Configuration]
    mapred.max.split.size is deprecated. Instead, use mapreduce.input.**
    fileinputformat.split.maxsize
    WARN0617 16:00:07.629000 main org.apache.hadoop.conf.**Configuration]
    mapred.min.split.size is deprecated. Instead, use mapreduce.input.**
    fileinputformat.split.minsize
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.**Configuration]
    mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.
    **fileinputformat.split.minsize.**per.rack
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.**Configuration]
    mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.
    **fileinputformat.split.minsize.**per.node
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.**Configuration]
    mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.**Configuration]
    mapred.reduce.tasks.**speculative.execution is deprecated. Instead,
    use mapreduce.reduce.speculative
    WARN0617 16:00:08.620000 main org.apache.hadoop.conf.**Configuration]
    org.apache.hadoop.hive.conf.**LoopingByteArrayInputStream@**24c68a98:an
    attempt to override final parameter: mapreduce.job.end-**
    notification.max.retry.**interval; Ignoring.
    WARN0617 16:00:08.689000 main org.apache.hadoop.conf.**Configuration]
    org.apache.hadoop.hive.conf.**LoopingByteArrayInputStream@**24c68a98:an
    attempt to override final parameter: mapreduce.job.end-**notification.max.attempts;
    Ignoring.
    WARN0617 16:00:08.713000 main org.apache.hadoop.hive.conf.**HiveConf]
    DEPRECATED: Configuration property hive.metastore.local no longer has any
    effect. Make sure to provide a valid value for hive.metastore.uris if you
    are connecting to a remote metastore.
    INFO0617 16:00:08.882000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-**
    west-2.compute.internal:9083
    INFO0617 16:00:09.081000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:10.081000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:10.082000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-**
    west-2.compute.internal:9083
    INFO0617 16:00:10.084000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:11.084000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:11.085000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-**
    west-2.compute.internal:9083
    INFO0617 16:00:11.087000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:12.092000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:12.093000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-**
    west-2.compute.internal:9083
    INFO0617 16:00:12.095000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:13.095000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:13.096000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-**
    west-2.compute.internal:9083
    INFO0617 16:00:13.098000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:14.098000 main hive.metastore] Connected to metastore.
    I0617 16:00:16.692451 9628 impala-server.cc:2074] Impala Beeswax
    Service listening on 21000
    I0617 16:00:16.692525 9628 impala-server.cc:2085] Impala HiveServer2
    Service listening on 21050
    I0617 16:00:16.692553 9628 impala-server.cc:2093]
    ImpalaInternalService listening on 22000
    I0617 16:00:16.699118 9628 thrift-server.cc:365] ThriftServer
    'backend' started on port: 22000
    I0617 16:00:16.699157 9628 exec-env.cc:146] Starting global services
    I0617 16:00:16.699275 9628 exec-env.cc:167] Using global memory limit:
    2.93 GB
    I0617 16:00:16.699409 9628 webserver.cc:129] Starting webserver on all
    interfaces, port 25000
    I0617 16:00:16.699470 9628 webserver.cc:139] Document root:
    /usr/lib/impala
    I0617 16:00:16.711299 9628 webserver.cc:178] Webserver started
    I0617 16:00:16.711383 9628 simple-scheduler.cc:99] Starting simple
    scheduler
    I0617 16:00:16.712973 9628 simple-scheduler.cc:133] Simple-scheduler
    using 10.245.112.238 as IP address
    I0617 16:00:16.713028 9628 state-store-subscriber.cc:137] Starting
    subscriber
    I0617 16:00:16.719905 9628 thrift-server.cc:365] ThriftServer
    'StateStoreSubscriber' started on port: 23000
    I0617 16:00:16.722825 9628 client-cache.cc:98] CreateClient(): adding
    new client for ip-10-245-112-238.us-west-2.**compute.internal:24000
    I0617 16:00:16.745887 9628 thrift-server.cc:365] ThriftServer
    'beeswax-frontend' started on port: 21000
    I0617 16:00:16.761379 9628 thrift-server.cc:365] ThriftServer
    'hiveServer2-frontend' started on port: 21050
    I0617 16:00:16.761443 9628 impalad-main.cc:101] Impala has started.
    I0617 16:09:51.630908 9737 impala-beeswax-server.cc:137] query():
    query=select A.search_parameters,reseller_**count,total from (select
    search_parameters,count(**distinct(reseller)) as reseller_count from
    all_lookups group by search_parameters)A join (select
    search_parameters,count(*) as total from all_lookups group by
    search_parameters)B on A.search_parameters=B.search_**parameters limit
    50
    I0617 16:09:51.631084 9737 impala-beeswax-server.cc:477] query: Query {
    01: query (string) = "select A.search_[...](315)",
    03: configuration (list) = list<string>[0] {
    },
    04: hadoop_user (string) = "ubuntu",
    }
    I0617 16:09:51.639899 9737 impala-beeswax-server.cc:489]
    TClientRequest.queryOptions: TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_**exceeded (bool) = false,
    }
    INFO0617 16:09:51.703000 Thread-3 com.cloudera.impala.service.**Frontend]
    analyze query select A.search_parameters,reseller_**count,total from
    (select search_parameters,count(**distinct(reseller)) as
    reseller_count from all_lookups group by search_parameters)A join (select
    search_parameters,count(*) as total from all_lookups group by
    search_parameters)B on A.search_parameters=B.search_**parameters limit
    50
    INFO0617 16:09:52.332000 Thread-3 com.cloudera.impala.catalog.**HdfsTable]
    load table all_lookups
    INFO0617 16:09:52.697000 Thread-3 com.cloudera.impala.catalog.**HdfsTable]
    load partition block md for all_lookups
    INFO0617 16:09:52.716000 Thread-3 com.cloudera.impala.catalog.**HdfsTable]
    loaded partition PartitionBlockMetadata{#**blocks=0, #filenames=0,
    totalStringLen=0}
    INFO0617 16:09:52.949000 Thread-3 com.cloudera.impala.catalog.**HdfsTable]
    loaded partition PartitionBlockMetadata{#**blocks=8, #filenames=8,
    totalStringLen=1197}
    INFO0617 16:09:53.073000 Thread-3 com.cloudera.impala.catalog.**HdfsTable]
    loaded disk ids for table default.all_lookups
    INFO0617 16:09:53.074000 Thread-3 com.cloudera.impala.catalog.**HdfsTable]
    1
    INFO0617 16:09:53.074000 Thread-3 com.cloudera.impala.catalog.**HdfsTable]
    table #rows=0
    INFO0617 16:09:53.103000 Thread-3 com.cloudera.impala.analysis.**AggregateInfo]
    agg info:
    AggregateInfo{grouping_exprs=(**SlotRef{tblName=null,
    col=search_parameters, id=0} SlotRef{tblName=null, col=reseller, id=1}),
    aggregate_exprs=, agg_tuple=TupleDescriptor{id=**1, tbl=null,
    byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=2, col=null,
    type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=3,
    col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=**null, col=search_parameters,
    id=0}:SlotRef{tblName=null, col=null, id=2} SlotRef{tblName=null,
    col=reseller, id=1}:SlotRef{tblName=null, col=null, id=3})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(**SlotRef{tblName=null, col=null, id=2}
    SlotRef{tblName=null, col=null, id=3}), aggregate_exprs=,
    agg_tuple=TupleDescriptor{id=**1, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=2, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=3, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=**null,
    col=search_parameters, id=0}:SlotRef{tblName=null, col=null, id=2}
    SlotRef{tblName=null, col=reseller, id=1}:SlotRef{tblName=null, col=null,
    id=3})}
    secondPhaseDistinctAggInfo:
    AggregateInfo{grouping_exprs=(**SlotRef{tblName=null, col=null,
    id=2}), aggregate_exprs=(**AggregateExpr{op=COUNT, isStar=false,
    isDistinct=false, (SlotRef{tblName=null, col=null, id=3})}),
    agg_tuple=TupleDescriptor{id=**2, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=4, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=5, col=null, type=BIGINT,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=**null,
    col=search_parameters, id=0}:SlotRef{tblName=null, col=null, id=4}
    AggregateExpr{op=COUNT, isStar=false, isDistinct=true,
    (SlotRef{tblName=null, col=reseller, id=1})}:SlotRef{tblName=null,
    col=null, id=5})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(**SlotRef{tblName=null, col=null,
    id=4}), aggregate_exprs=(**AggregateExpr{op=SUM, isStar=false,
    isDistinct=false, (SlotRef{tblName=null, col=null, id=5})}),
    agg_tuple=TupleDescriptor{id=**2, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=4, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=5, col=null, type=BIGINT,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=**null,
    col=search_parameters, id=0}:SlotRef{tblName=null, col=null, id=4}
    AggregateExpr{op=COUNT, isStar=false, isDistinct=true,
    (SlotRef{tblName=null, col=reseller, id=1})}:SlotRef{tblName=null,
    col=null, id=5})}
    INFO0617 16:09:53.110000 Thread-3 com.cloudera.impala.analysis.**AggregateInfo]
    agg info:
    AggregateInfo{grouping_exprs=(**SlotRef{tblName=null,
    col=search_parameters, id=8}), aggregate_exprs=(**AggregateExpr{op=COUNT,
    isStar=true, isDistinct=false, }), agg_tuple=TupleDescriptor{id=**5,
    tbl=null, byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=9,
    col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=10,
    col=null, type=BIGINT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=**null, col=search_parameters,
    id=8}:SlotRef{tblName=null, col=null, id=9} AggregateExpr{op=COUNT,
    isStar=true, isDistinct=false, }:SlotRef{tblName=null, col=null, id=10})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(**SlotRef{tblName=null, col=null,
    id=9}), aggregate_exprs=(**AggregateExpr{op=SUM, isStar=false,
    isDistinct=false, (SlotRef{tblName=null, col=null, id=10})}),
    agg_tuple=TupleDescriptor{id=**5, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=9, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=10, col=null,
    type=BIGINT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=**null, col=search_parameters,
    id=8}:SlotRef{tblName=null, col=null, id=9} AggregateExpr{op=COUNT,
    isStar=true, isDistinct=false, }:SlotRef{tblName=null, col=null, id=10})}
    INFO0617 16:09:53.140000 Thread-3 com.cloudera.impala.analysis.**BinaryPredicate]
    A.search_parameters = B.search_parameters selectivity: 0.1
    INFO0617 16:09:53.257000 Thread-3 com.cloudera.impala.service.**Frontend]
    create plan
    INFO0617 16:09:53.282000 Thread-3 com.cloudera.impala.planner.**Planner]
    create single-node plan
    INFO0617 16:09:53.290000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    grouping expr: search_parameters #distinct=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    grouping expr: <slot 2> #distinct=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    grouping expr: search_parameters #distinct=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.297000 Thread-3 com.cloudera.impala.planner.**HdfsScanNode]
    collecting partitions for table all_lookups
    INFO0617 16:09:53.297000 Thread-3 com.cloudera.impala.planner.**HdfsScanNode]
    finalize HdfsScan: cardinality=0
    INFO0617 16:09:53.297000 Thread-3 com.cloudera.impala.planner.**HdfsScanNode]
    finalize HdfsScan: #nodes=1
    INFO0617 16:09:53.297000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    grouping expr: search_parameters #distinct=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    grouping expr: <slot 2> #distinct=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**HdfsScanNode]
    collecting partitions for table all_lookups
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**HdfsScanNode]
    finalize HdfsScan: cardinality=0
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**HdfsScanNode]
    finalize HdfsScan: #nodes=1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    grouping expr: search_parameters #distinct=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**AggregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**HashJoinNode]
    stats HashJoin: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Planner]
    create plan fragments
    INFO0617 16:09:53.299000 Thread-3 com.cloudera.impala.planner.**Planner]
    memlimit=0
    INFO0617 16:09:53.310000 Thread-3 com.cloudera.impala.planner.**Planner]
    broadcast: cost=0
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.**Planner]
    card=-1 row_size=0.0 #nodes=1
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.**Planner]
    partition: cost=0
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.**Planner]
    lhs card=-1 row_size=24.0
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.**Planner]
    rhs card=-1 row_size=0.0
    INFO0617 16:09:53.317000 Thread-3 com.cloudera.impala.planner.**Planner]
    9:AGGREGATE
    output: SUM(<slot 10>)
    group by: <slot 9>
    8:EXCHANGE

    INFO0617 16:09:53.317000 Thread-3 com.cloudera.impala.planner.**Planner]
    finalize plan fragments
    INFO0617 16:09:53.641000 Thread-3 com.cloudera.impala.service.**Frontend]
    get scan range locations
    INFO0617 16:09:53.678000 Thread-3 com.cloudera.impala.service.**Frontend]
    create result set metadata
    INFO0617 16:09:53.690000 Thread-3 com.cloudera.impala.service.**JniFrontend]
    PLAN FRAGMENT 0
    PARTITION: UNPARTITIONED

    11:EXCHANGE
    limit: 50
    tuple ids: 2 5

    PLAN FRAGMENT 1
    PARTITION: HASH_PARTITIONED: <slot 2>

    STREAM DATA SINK
    EXCHANGE ID: 11
    UNPARTITIONED

    5:HASH JOIN
    join op: INNER JOIN (BROADCAST)
    hash predicates:
    <slot 4> = <slot 9>
    limit: 50
    tuple ids: 2 5

    ----10:EXCHANGE
    tuple ids: 5
    2:AGGREGATE
    output: COUNT(<slot 3>)
    group by: <slot 2>
    tuple ids: 2
    7:AGGREGATE
    group by: <slot 2>, <slot 3>
    tuple ids: 1
    6:EXCHANGE
    tuple ids: 1

    PLAN FRAGMENT 2
    PARTITION: HASH_PARTITIONED: <slot 9>

    STREAM DATA SINK
    EXCHANGE ID: 10
    UNPARTITIONED

    9:AGGREGATE
    output: SUM(<slot 10>)
    group by: <slot 9>
    tuple ids: 5
    8:EXCHANGE
    tuple ids: 5

    PLAN FRAGMENT 3
    PARTITION: RANDOM

    STREAM DATA SINK
    EXCHANGE ID: 8
    HASH_PARTITIONED: <slot 9>

    4:AGGREGATE
    output: COUNT(*)
    group by: search_parameters
    tuple ids: 5
    3:SCAN HDFS
    table=default.all_lookups #partitions=1 size=397.15MB
    tuple ids: 4

    PLAN FRAGMENT 4
    PARTITION: RANDOM

    STREAM DATA SINK
    EXCHANGE ID: 6
    HASH_PARTITIONED: <slot 2>

    1:AGGREGATE
    group by: search_parameters, reseller
    tuple ids: 1
    0:SCAN HDFS
    table=default.all_lookups #partitions=1 size=397.15MB
    tuple ids: 0

    I0617 16:09:53.755270 9737 coordinator.cc:290] Exec()
    query_id=8eb364939bbb41b0:**a0a6981049ec71f0
    I0617 16:09:53.755636 9737 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:**a0a6981049ec71f0
    instance_id=8eb364939bbb41b0:**a0a6981049ec71f1
    I0617 16:09:53.884407 9737 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:**a0a6981049ec71f1
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:53.888123 9737 exchange-node.cc:50] Exch id=11
    input_desc=Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])

    output_desc=Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.021406 9737 coordinator.cc:377] starting 4 backends for
    query 8eb364939bbb41b0:**a0a6981049ec71f0
    I0617 16:09:54.021667 9900 client-cache.cc:98] CreateClient(): adding
    new client for ip-10-245-112-238.us-west-2.**compute.internal:22000
    I0617 16:09:54.022987 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:**a0a6981049ec71f2
    coord=ip-10-245-112-238.us-**west-2.compute.internal:22000 backend#=0
    I0617 16:09:54.023074 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:**a0a6981049ec71f0
    instance_id=8eb364939bbb41b0:**a0a6981049ec71f2
    I0617 16:09:54.032918 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:**a0a6981049ec71f2
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.034090 9901 exchange-node.cc:50] Exch id=6
    input_desc=Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1
    offset=8 null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])

    output_desc=Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1
    offset=8 null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    I0617 16:09:54.044239 9901 exchange-node.cc:50] Exch id=10
    input_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])

    output_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.618634 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:**a0a6981049ec71f3
    coord=ip-10-245-112-238.us-**west-2.compute.internal:22000 backend#=1
    I0617 16:09:54.618748 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:**a0a6981049ec71f0
    instance_id=8eb364939bbb41b0:**a0a6981049ec71f3
    I0617 16:09:54.623918 9902 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:**a0a6981049ec71f2
    I0617 16:09:54.631587 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:**a0a6981049ec71f3
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.631726 9901 exchange-node.cc:50] Exch id=8
    input_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])

    output_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.974669 9905 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:**a0a6981049ec71f3
    I0617 16:09:54.977622 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:**a0a6981049ec71f4
    coord=ip-10-245-112-238.us-**west-2.compute.internal:22000 backend#=2
    I0617 16:09:54.977705 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:**a0a6981049ec71f0
    instance_id=8eb364939bbb41b0:**a0a6981049ec71f4
    I0617 16:09:55.005800 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:**a0a6981049ec71f4
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:55.522994 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:**a0a6981049ec71f5
    coord=ip-10-245-112-238.us-**west-2.compute.internal:22000 backend#=3
    I0617 16:09:55.523129 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:**a0a6981049ec71f0
    instance_id=8eb364939bbb41b0:**a0a6981049ec71f5
    I0617 16:09:55.529832 9908 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:**a0a6981049ec71f4
    I0617 16:09:55.550024 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:**a0a6981049ec71f5
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:56.376770 9917 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:**a0a6981049ec71f1
    I0617 16:09:56.379158 9916 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:**a0a6981049ec71f5
    I0617 16:09:56.474069 9737 impala-beeswax-server.cc:290]
    get_results_metadata(): query_id=8eb364939bbb41b0:**a0a6981049ec71f0
    I0617 16:10:01.533941 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec71f0: 12% Complete (2 out of 16)
    I0617 16:10:02.386214 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec71f0: 25% Complete (4 out of 16)
    I0617 16:10:06.535214 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec71f0: 31% Complete (5 out of 16)
    I0617 16:10:07.386981 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec71f0: 37% Complete (6 out of 16)
    I0617 16:10:11.535930 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec71f0: 43% Complete (7 out of 16)
    I0617 16:10:12.387742 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec71f0: 50% Complete (8 out of 16)
    I0617 16:10:16.536659 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec71f0: 62% Complete (10 out of 16)
    I0617 16:10:17.388424 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec71f0: 75% Complete (12 out of 16)
    I0617 16:10:21.556238 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec71f0: 87% Complete (14 out of 16)
    I0617 16:10:21.556130 10155 client-cache.cc:98] CreateClient(): adding
    new client for ip-10-245-112-238.us-west-2.**compute.internal:22000
    I0617 16:10:22.389418 10156 progress-updater.cc:45] Query
    8eb364939bbb41b0:**a0a6981049ec71f0 100% Complete (16 out of 16)
    I0617 16:10:25.982318 9906 client-cache.cc:98] CreateClient(): adding
    new client for ip-10-245-112-238.us-west-2.**compute.internal:22000

    On Monday, June 17, 2013 12:30:34 PM UTC-4, Ricky Saltzer wrote:

    It would be the log file before this one
    On Jun 17, 2013 12:14 PM, "Alex Minnaar" wrote:

    It looks like the last log file was this one. But I'm not sure it
    tells us anything either.

    Log file created at: 2013/06/17 16:10:41
    Running on machine: ip-10-245-112-238
    Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
    I0617 16:10:41.500978 15390 daemon.cc:34] impalad version 1.0 RELEASE
    (build d1bf0d1dac339af3692ffa17a5e3fd****ae0aed751f)
    Built on Sun, 28 Apr 2013 15:58:48 PST
    I0617 16:10:41.521280 15390 daemon.cc:35] Using hostname:
    ip-10-245-112-238.us-west-2.**co**mpute.internal
    I0617 16:10:41.522235 15390 logging.cc:76] Flags (see also /varz are
    on debug webserver):
    --dump_ir=false
    --module_output=
    --be_port=22000
    --hostname=ip-10-245-112-238.**u**s-west-2.compute.internal
    --keytab_file=
    --mem_limit=80%
    --planservice_host=localhost
    --planservice_port=20000
    --principal=
    --exchg_node_buffer_size_**bytes**=10485760
    --max_row_batches=0
    --randomize_splits=false
    --num_disks=0
    --num_threads_per_disk=1
    --read_size=8388608
    --enable_webserver=true
    --state_store_host=ip-10-245-**1**12-238.us-west-2.compute.**inter**
    nal
    --state_store_subscriber_port=****23000
    --use_statestore=true
    --nn=ip-10-245-112-238.us-**west**-2.compute.internal
    --nn_port=8020
    --serialize_batch=false
    --status_report_interval=5
    --compress_rowbatches=true
    --num_threads_per_core=3
    --abort_on_config_error=true
    --be_service_threads=64
    --beeswax_port=21000
    --default_query_options=
    --fe_service_threads=64
    --heap_profile_dir=
    --hs2_port=21050
    --load_catalog_at_startup=**fals**e
    --log_mem_usage_interval=0
    --log_query_to_file=true
    --query_log_size=25
    --use_planservice=false
    --statestore_subscriber_**timeou**t_seconds=10
    --state_store_port=24000
    --statestore_max_missed_**heartb**eats=5
    --statestore_num_heartbeat_**thr**eads=10
    --statestore_suspect_**heartbeat**s=2
    --kerberos_reinit_interval=60
    --sasl_path=/usr/lib/sasl2:/**us**r/lib64/sasl2:/usr/local/**lib/**
    sasl2:/usr/lib/x86_64-**linux-**gnu/sasl2
    --web_log_bytes=1048576
    --log_filename=impalad
    --periodic_counter_update_**peri**od_ms=500
    --rpc_cnxn_attempts=10
    --rpc_cnxn_retry_interval_ms=**2**000
    --enable_webserver_doc_root=**tr**ue
    --webserver_doc_root=/usr/lib/****impala
    --webserver_interface=
    --webserver_port=25000
    --flagfile=/run/cloudera-scm-**a**gent/process/40-impala-**IMPALAD**
    /impala-conf/impalad_**flags
    --fromenv=
    --tryfromenv=
    --undefok=
    --tab_completion_columns=80
    --tab_completion_word=
    --help=false
    --helpfull=false
    --helpmatch=
    --helpon=
    --helppackage=false
    --helpshort=false
    --helpxml=false
    --version=false
    --alsologtoemail=
    --alsologtostderr=false
    --drop_log_memory=true
    --log_backtrace_at=
    --log_dir=/var/log/impalad
    --log_link=
    --log_prefix=true
    --logbuflevel=-1
    --logbufsecs=30
    --logemaillevel=999
    --logmailer=/bin/mail
    --logtostderr=false
    --max_log_size=200
    --minloglevel=0
    --stderrthreshold=2
    --stop_logging_if_full_disk=**fa**lse
    --symbolize_stacktrace=true
    --v=1
    --vmodule=
    I0617 16:10:41.533797 15390 mem-info.cc:66] Physical Memory: 3.66 GB
    I0617 16:10:41.539779 15390 daemon.cc:43] Cpu Info:
    Model: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
    Cores: 1
    L1 Cache: 32.00 KB
    L2 Cache: 256.00 KB
    L3 Cache: 20.00 MB
    Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    I0617 16:10:41.539821 15390 daemon.cc:44] Disk Info:
    Num disks 2: xvda, xvdb
    I0617 16:10:41.539855 15390 daemon.cc:45] Mem Info: 3.66 GB
    I0617 16:10:50.755198 15390 impala-server.cc:1809] Default query
    options:TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_**exceede**d (bool) = false,
    }
    WARN0617 16:10:51.236000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.max.split.size is deprecated. Instead, use mapreduce.input.**
    fileinputforma**t.split.maxsize
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.min.split.size is deprecated. Instead, use mapreduce.input.**
    fileinputforma**t.split.minsize
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.
    **fileinputforma**t.split.minsize.**per.rack
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.
    **fileinputforma**t.split.minsize.**per.node
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.reduce.tasks.**speculativ**e.execution is deprecated.
    Instead, use mapreduce.reduce.speculative
    WARN0617 16:10:51.892000 main org.apache.hadoop.conf.**Configu**ration]
    org.apache.hadoop.hive.conf.**Lo**opingByteArrayInputStream@**66b5**1404:an
    attempt to override final parameter: mapreduce.job.end-**notification
    **.max.retry.**interval; Ignoring.
    WARN0617 16:10:51.954000 main org.apache.hadoop.conf.**Configu**ration]
    org.apache.hadoop.hive.conf.**Lo**opingByteArrayInputStream@**66b5**1404:an
    attempt to override final parameter: mapreduce.job.end-**notification
    **.max.attempts; Ignoring.
    WARN0617 16:10:51.972000 main org.apache.hadoop.hive.conf.**Hi**veConf]
    DEPRECATED: Configuration property hive.metastore.local no longer has any
    effect. Make sure to provide a valid value for hive.metastore.uris if you
    are connecting to a remote metastore.
    INFO0617 16:10:52.240000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-****
    west-2.compute.internal:9083
    INFO0617 16:10:52.494000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:10:53.495000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:53.495000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-****
    west-2.compute.internal:9083
    INFO0617 16:10:53.497000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:10:54.498000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:54.498000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-****
    west-2.compute.internal:9083
    INFO0617 16:10:54.505000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:10:55.506000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:55.507000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-****
    west-2.compute.internal:9083
    INFO0617 16:10:55.515000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:10:56.515000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:56.515000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-****
    west-2.compute.internal:9083
    INFO0617 16:10:56.519000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:10:57.519000 main hive.metastore] Connected to metastore.
    I0617 16:10:59.891072 15390 impala-server.cc:2074] Impala Beeswax
    Service listening on 21000
    I0617 16:10:59.893288 15390 impala-server.cc:2085] Impala HiveServer2
    Service listening on 21050
    I0617 16:10:59.893335 15390 impala-server.cc:2093]
    ImpalaInternalService listening on 22000
    I0617 16:10:59.905498 15390 thrift-server.cc:365] ThriftServer
    'backend' started on port: 22000
    I0617 16:10:59.905547 15390 exec-env.cc:146] Starting global services
    I0617 16:10:59.905670 15390 exec-env.cc:167] Using global memory
    limit: 2.93 GB
    I0617 16:10:59.909502 15390 webserver.cc:129] Starting webserver on
    all interfaces, port 25000
    I0617 16:10:59.909559 15390 webserver.cc:139] Document root:
    /usr/lib/impala
    I0617 16:10:59.921815 15390 webserver.cc:178] Webserver started
    I0617 16:10:59.921897 15390 simple-scheduler.cc:99] Starting simple
    scheduler
    I0617 16:10:59.926578 15390 simple-scheduler.cc:133] Simple-scheduler
    using 10.245.112.238 as IP address
    I0617 16:10:59.926632 15390 state-store-subscriber.cc:137] Starting
    subscriber
    I0617 16:10:59.937484 15390 thrift-server.cc:365] ThriftServer
    'StateStoreSubscriber' started on port: 23000
    I0617 16:10:59.949283 15390 client-cache.cc:98] CreateClient():
    adding new client for ip-10-245-112-238.us-west-2.**co**
    mpute.internal:24000
    I0617 16:10:59.982250 15390 thrift-server.cc:365] ThriftServer
    'beeswax-frontend' started on port: 21000
    I0617 16:11:00.017025 15390 thrift-server.cc:365] ThriftServer
    'hiveServer2-frontend' started on port: 21050
    I0617 16:11:00.017144 15390 impalad-main.cc:101] Impala has started.

    On Monday, June 17, 2013 12:08:01 PM UTC-4, Ricky Saltzer wrote:

    Hey Alex -

    Sorry about that, I forgot that the log get's rolled over after a
    crash. Inside of */var/log/impalad*, you should be able to find the
    last log file written to before the crash using an "*ls -lthr*",
    take a peek inside..it should contain a lot more data.

    Thanks,
    Ricky

    On Mon, Jun 17, 2013 at 9:04 AM, Alex Minnaar wrote:

    I have attached what was in the impalad.INFO file

    Alex

    On Monday, June 17, 2013 11:44:10 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Try connecting to the node via SSH and grabbing the log directly
    from */var/log/impalad *
    *
    *
    Ricky

    On Mon, Jun 17, 2013 at 8:39 AM, Alex Minnaar wrote:

    OK I figured out how to set the GLOG level, but when I try to
    download the impalad.INFO file I get

    Problem accessing /cmf/process/all/logs/**download******. Reason:

    Connection timed out

    Could not connect to host.

    On Monday, June 17, 2013 10:39:21 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Could you set the GLOG level to 2<http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_logging.html> (you
    can use Cloudera manager to do it, too). Then, re-run your query and attach
    the impalad.INFO from the coordinator (the node you submit your query to).

    Ricky


    On Mon, Jun 17, 2013 at 6:23 AM, Alex Minnaar <
    minna...@gmail.com> wrote:
    Impala is working for me with simple select queries but it is
    not working with queries involving joins. When I execute the join query in
    the impala shell, I get the following result.
    Query finished, fetching results ...
    Error communicating with impalad: TSocket read 0 bytes

    And then I am no longer connected to impala. How do I fix this?

    Alex


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com

  • Ricky Saltzer at Jun 17, 2013 at 6:07 pm
    Hey Alex -

    Alright, you might want to check that the node isn't running out of memory
    during the query...

    You can monitor with this command

    $ watch -d -n 1 "free -m"


    On Mon, Jun 17, 2013 at 11:00 AM, Alex Minnaar wrote:

    I just tried the query again and it froze and now the health of that host
    has turned to bad, so I guess I need to fix that first

    On Monday, June 17, 2013 1:29:12 PM UTC-4, Ricky Saltzer wrote:

    Hey Alex -

    Are you sure GLOG was on level 2 when you issued this query? Maybe you
    could issue it once more now that GLOG is on level 2, and get then pull the
    log which contains that query. If you could, please attach the log instead
    of pasting (easier to read).

    Thanks
    Ricky

    On Mon, Jun 17, 2013 at 10:20 AM, Alex Minnaar wrote:

    Yes that's the one

    On Monday, June 17, 2013 1:18:50 PM UTC-4, Ricky Saltzer wrote:

    Hey Alex -

    So, the following query is the one crashing?

    select A.search_parameters,reseller_c****ount,total from (select
    search_parameters,count(distin****ct(reseller)) as reseller_count from
    all_lookups group by search_parameters)A join (select
    search_parameters,count(*) as total from all_lookups group by
    search_parameters)B on A.search_parameters=B.search_p****arameters
    limit 50

    Ricky

    On Mon, Jun 17, 2013 at 10:05 AM, Alex Minnaar wrote:

    OK the previous log file was

    Log file created at: 2013/06/17 15:59:58
    Running on machine: ip-10-245-112-238
    Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
    I0617 15:59:58.518903 9628 daemon.cc:34] impalad version 1.0 RELEASE
    (build d1bf0d1dac339af3692ffa17a5e3fd****ae0aed751f)
    Built on Sun, 28 Apr 2013 15:58:48 PST
    I0617 15:59:58.547598 9628 daemon.cc:35] Using hostname:
    ip-10-245-112-238.us-west-2.**co**mpute.internal
    I0617 15:59:58.548459 9628 logging.cc:76] Flags (see also /varz are
    on debug webserver):
    --dump_ir=false
    --module_output=
    --be_port=22000
    --hostname=ip-10-245-112-238.**u**s-west-2.compute.internal
    --keytab_file=
    --mem_limit=80%
    --planservice_host=localhost
    --planservice_port=20000
    --principal=
    --exchg_node_buffer_size_**bytes**=10485760
    --max_row_batches=0
    --randomize_splits=false
    --num_disks=0
    --num_threads_per_disk=1
    --read_size=8388608
    --enable_webserver=true
    --state_store_host=ip-10-245-**1**12-238.us-west-2.compute.**inter**
    nal
    --state_store_subscriber_port=****23000
    --use_statestore=true
    --nn=ip-10-245-112-238.us-**west**-2.compute.internal
    --nn_port=8020
    --serialize_batch=false
    --status_report_interval=5
    --compress_rowbatches=true
    --num_threads_per_core=3
    --abort_on_config_error=true
    --be_service_threads=64
    --beeswax_port=21000
    --default_query_options=
    --fe_service_threads=64
    --heap_profile_dir=
    --hs2_port=21050
    --load_catalog_at_startup=**fals**e
    --log_mem_usage_interval=0
    --log_query_to_file=true
    --query_log_size=25
    --use_planservice=false
    --statestore_subscriber_**timeou**t_seconds=10
    --state_store_port=24000
    --statestore_max_missed_**heartb**eats=5
    --statestore_num_heartbeat_**thr**eads=10
    --statestore_suspect_**heartbeat**s=2
    --kerberos_reinit_interval=60
    --sasl_path=/usr/lib/sasl2:/**us**r/lib64/sasl2:/usr/local/**lib/**
    sasl2:/usr/lib/x86_64-**linux-**gnu/sasl2
    --web_log_bytes=1048576
    --log_filename=impalad
    --periodic_counter_update_**peri**od_ms=500
    --rpc_cnxn_attempts=10
    --rpc_cnxn_retry_interval_ms=**2**000
    --enable_webserver_doc_root=**tr**ue
    --webserver_doc_root=/usr/lib/****impala
    --webserver_interface=
    --webserver_port=25000
    --flagfile=/run/cloudera-scm-**a**gent/process/40-impala-**IMPALAD**
    /impala-conf/impalad_**flags
    --fromenv=
    --tryfromenv=
    --undefok=
    --tab_completion_columns=80
    --tab_completion_word=
    --help=false
    --helpfull=false
    --helpmatch=
    --helpon=
    --helppackage=false
    --helpshort=false
    --helpxml=false
    --version=false
    --alsologtoemail=
    --alsologtostderr=false
    --drop_log_memory=true
    --log_backtrace_at=
    --log_dir=/var/log/impalad
    --log_link=
    --log_prefix=true
    --logbuflevel=-1
    --logbufsecs=30
    --logemaillevel=999
    --logmailer=/bin/mail
    --logtostderr=false
    --max_log_size=200
    --minloglevel=0
    --stderrthreshold=2
    --stop_logging_if_full_disk=**fa**lse
    --symbolize_stacktrace=true
    --v=1
    --vmodule=
    I0617 15:59:58.551378 9628 mem-info.cc:66] Physical Memory: 3.66 GB
    I0617 15:59:58.552459 9628 daemon.cc:43] Cpu Info:
    Model: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
    Cores: 1
    L1 Cache: 32.00 KB
    L2 Cache: 256.00 KB
    L3 Cache: 20.00 MB
    Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    I0617 15:59:58.552492 9628 daemon.cc:44] Disk Info:
    Num disks 2: xvda, xvdb
    I0617 15:59:58.552525 9628 daemon.cc:45] Mem Info: 3.66 GB
    I0617 16:00:07.111661 9628 impala-server.cc:1809] Default query
    options:TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_**exceede**d (bool) = false,
    }
    WARN0617 16:00:07.622000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.max.split.size is deprecated. Instead, use mapreduce.input.**
    fileinputforma**t.split.maxsize
    WARN0617 16:00:07.629000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.min.split.size is deprecated. Instead, use mapreduce.input.**
    fileinputforma**t.split.minsize
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.
    **fileinputforma**t.split.minsize.**per.rack
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.
    **fileinputforma**t.split.minsize.**per.node
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.reduce.tasks.**speculativ**e.execution is deprecated. Instead,
    use mapreduce.reduce.speculative
    WARN0617 16:00:08.620000 main org.apache.hadoop.conf.**Configu**ration]
    org.apache.hadoop.hive.conf.**Lo**opingByteArrayInputStream@**24c6**8a98:an
    attempt to override final parameter: mapreduce.job.end-**notification*
    *.max.retry.**interval; Ignoring.
    WARN0617 16:00:08.689000 main org.apache.hadoop.conf.**Configu**ration]
    org.apache.hadoop.hive.conf.**Lo**opingByteArrayInputStream@**24c6**8a98:an
    attempt to override final parameter: mapreduce.job.end-**notification*
    *.max.attempts; Ignoring.
    WARN0617 16:00:08.713000 main org.apache.hadoop.hive.conf.**Hi**veConf]
    DEPRECATED: Configuration property hive.metastore.local no longer has any
    effect. Make sure to provide a valid value for hive.metastore.uris if you
    are connecting to a remote metastore.
    INFO0617 16:00:08.882000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-****
    west-2.compute.internal:9083
    INFO0617 16:00:09.081000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:10.081000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:10.082000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-****
    west-2.compute.internal:9083
    INFO0617 16:00:10.084000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:11.084000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:11.085000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-****
    west-2.compute.internal:9083
    INFO0617 16:00:11.087000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:12.092000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:12.093000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-****
    west-2.compute.internal:9083
    INFO0617 16:00:12.095000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:13.095000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:13.096000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-****
    west-2.compute.internal:9083
    INFO0617 16:00:13.098000 main hive.metastore] Waiting 1 seconds before
    next connection attempt.
    INFO0617 16:00:14.098000 main hive.metastore] Connected to metastore.
    I0617 16:00:16.692451 9628 impala-server.cc:2074] Impala Beeswax
    Service listening on 21000
    I0617 16:00:16.692525 9628 impala-server.cc:2085] Impala HiveServer2
    Service listening on 21050
    I0617 16:00:16.692553 9628 impala-server.cc:2093]
    ImpalaInternalService listening on 22000
    I0617 16:00:16.699118 9628 thrift-server.cc:365] ThriftServer
    'backend' started on port: 22000
    I0617 16:00:16.699157 9628 exec-env.cc:146] Starting global services
    I0617 16:00:16.699275 9628 exec-env.cc:167] Using global memory
    limit: 2.93 GB
    I0617 16:00:16.699409 9628 webserver.cc:129] Starting webserver on
    all interfaces, port 25000
    I0617 16:00:16.699470 9628 webserver.cc:139] Document root:
    /usr/lib/impala
    I0617 16:00:16.711299 9628 webserver.cc:178] Webserver started
    I0617 16:00:16.711383 9628 simple-scheduler.cc:99] Starting simple
    scheduler
    I0617 16:00:16.712973 9628 simple-scheduler.cc:133] Simple-scheduler
    using 10.245.112.238 as IP address
    I0617 16:00:16.713028 9628 state-store-subscriber.cc:137] Starting
    subscriber
    I0617 16:00:16.719905 9628 thrift-server.cc:365] ThriftServer
    'StateStoreSubscriber' started on port: 23000
    I0617 16:00:16.722825 9628 client-cache.cc:98] CreateClient(): adding
    new client for ip-10-245-112-238.us-west-2.**co**mpute.internal:24000
    I0617 16:00:16.745887 9628 thrift-server.cc:365] ThriftServer
    'beeswax-frontend' started on port: 21000
    I0617 16:00:16.761379 9628 thrift-server.cc:365] ThriftServer
    'hiveServer2-frontend' started on port: 21050
    I0617 16:00:16.761443 9628 impalad-main.cc:101] Impala has started.
    I0617 16:09:51.630908 9737 impala-beeswax-server.cc:137] query():
    query=select A.search_parameters,reseller_**c**ount,total from
    (select search_parameters,count(**distin**ct(reseller)) as
    reseller_count from all_lookups group by search_parameters)A join (select
    search_parameters,count(*) as total from all_lookups group by
    search_parameters)B on A.search_parameters=B.search_**p**arameters
    limit 50
    I0617 16:09:51.631084 9737 impala-beeswax-server.cc:477] query: Query
    {
    01: query (string) = "select A.search_[...](315)",
    03: configuration (list) = list<string>[0] {
    },
    04: hadoop_user (string) = "ubuntu",
    }
    I0617 16:09:51.639899 9737 impala-beeswax-server.cc:489]
    TClientRequest.queryOptions: TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_**exceede**d (bool) = false,
    }
    INFO0617 16:09:51.703000 Thread-3 com.cloudera.impala.service.**Fr**ontend]
    analyze query select A.search_parameters,reseller_**c**ount,total
    from (select search_parameters,count(**distin**ct(reseller)) as
    reseller_count from all_lookups group by search_parameters)A join (select
    search_parameters,count(*) as total from all_lookups group by
    search_parameters)B on A.search_parameters=B.search_**p**arameters
    limit 50
    INFO0617 16:09:52.332000 Thread-3 com.cloudera.impala.catalog.**Hd**fsTable]
    load table all_lookups
    INFO0617 16:09:52.697000 Thread-3 com.cloudera.impala.catalog.**Hd**fsTable]
    load partition block md for all_lookups
    INFO0617 16:09:52.716000 Thread-3 com.cloudera.impala.catalog.**Hd**fsTable]
    loaded partition PartitionBlockMetadata{#**blocks**=0, #filenames=0,
    totalStringLen=0}
    INFO0617 16:09:52.949000 Thread-3 com.cloudera.impala.catalog.**Hd**fsTable]
    loaded partition PartitionBlockMetadata{#**blocks**=8, #filenames=8,
    totalStringLen=1197}
    INFO0617 16:09:53.073000 Thread-3 com.cloudera.impala.catalog.**Hd**fsTable]
    loaded disk ids for table default.all_lookups
    INFO0617 16:09:53.074000 Thread-3 com.cloudera.impala.catalog.**Hd**fsTable]
    1
    INFO0617 16:09:53.074000 Thread-3 com.cloudera.impala.catalog.**Hd**fsTable]
    table #rows=0
    INFO0617 16:09:53.103000 Thread-3 com.cloudera.impala.analysis.**A**ggregateInfo]
    agg info:
    AggregateInfo{grouping_exprs=(****SlotRef{tblName=null,
    col=search_parameters, id=0} SlotRef{tblName=null, col=reseller, id=1}),
    aggregate_exprs=, agg_tuple=TupleDescriptor{id=**1**, tbl=null,
    byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=2, col=null,
    type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=3,
    col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=**null**, col=search_parameters,
    id=0}:SlotRef{tblName=null, col=null, id=2} SlotRef{tblName=null,
    col=reseller, id=1}:SlotRef{tblName=null, col=null, id=3})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(****SlotRef{tblName=null, col=null,
    id=2} SlotRef{tblName=null, col=null, id=3}), aggregate_exprs=,
    agg_tuple=TupleDescriptor{id=**1**, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=2, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=3, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=**null**,
    col=search_parameters, id=0}:SlotRef{tblName=null, col=null, id=2}
    SlotRef{tblName=null, col=reseller, id=1}:SlotRef{tblName=null, col=null,
    id=3})}
    secondPhaseDistinctAggInfo:
    AggregateInfo{grouping_exprs=(****SlotRef{tblName=null, col=null,
    id=2}), aggregate_exprs=(**AggregateExpr**{op=COUNT, isStar=false,
    isDistinct=false, (SlotRef{tblName=null, col=null, id=3})}),
    agg_tuple=TupleDescriptor{id=**2**, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=4, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=5, col=null, type=BIGINT,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=**null**,
    col=search_parameters, id=0}:SlotRef{tblName=null, col=null, id=4}
    AggregateExpr{op=COUNT, isStar=false, isDistinct=true,
    (SlotRef{tblName=null, col=reseller, id=1})}:SlotRef{tblName=null,
    col=null, id=5})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(****SlotRef{tblName=null, col=null,
    id=4}), aggregate_exprs=(**AggregateExpr**{op=SUM, isStar=false,
    isDistinct=false, (SlotRef{tblName=null, col=null, id=5})}),
    agg_tuple=TupleDescriptor{id=**2**, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=4, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=5, col=null, type=BIGINT,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=**null**,
    col=search_parameters, id=0}:SlotRef{tblName=null, col=null, id=4}
    AggregateExpr{op=COUNT, isStar=false, isDistinct=true,
    (SlotRef{tblName=null, col=reseller, id=1})}:SlotRef{tblName=null,
    col=null, id=5})}
    INFO0617 16:09:53.110000 Thread-3 com.cloudera.impala.analysis.**A**ggregateInfo]
    agg info:
    AggregateInfo{grouping_exprs=(****SlotRef{tblName=null,
    col=search_parameters, id=8}), aggregate_exprs=(**AggregateExpr**{op=COUNT,
    isStar=true, isDistinct=false, }), agg_tuple=TupleDescriptor{id=**5**,
    tbl=null, byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=9,
    col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=10,
    col=null, type=BIGINT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=**null**, col=search_parameters,
    id=8}:SlotRef{tblName=null, col=null, id=9} AggregateExpr{op=COUNT,
    isStar=true, isDistinct=false, }:SlotRef{tblName=null, col=null, id=10})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(****SlotRef{tblName=null, col=null,
    id=9}), aggregate_exprs=(**AggregateExpr**{op=SUM, isStar=false,
    isDistinct=false, (SlotRef{tblName=null, col=null, id=10})}),
    agg_tuple=TupleDescriptor{id=**5**, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=9, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=10, col=null,
    type=BIGINT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=**null**, col=search_parameters,
    id=8}:SlotRef{tblName=null, col=null, id=9} AggregateExpr{op=COUNT,
    isStar=true, isDistinct=false, }:SlotRef{tblName=null, col=null, id=10})}
    INFO0617 16:09:53.140000 Thread-3 com.cloudera.impala.analysis.**B**inaryPredicate]
    A.search_parameters = B.search_parameters selectivity: 0.1
    INFO0617 16:09:53.257000 Thread-3 com.cloudera.impala.service.**Fr**ontend]
    create plan
    INFO0617 16:09:53.282000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    create single-node plan
    INFO0617 16:09:53.290000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    grouping expr: search_parameters #distinct=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    grouping expr: <slot 2> #distinct=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    grouping expr: search_parameters #distinct=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.297000 Thread-3 com.cloudera.impala.planner.**Hd**fsScanNode]
    collecting partitions for table all_lookups
    INFO0617 16:09:53.297000 Thread-3 com.cloudera.impala.planner.**Hd**fsScanNode]
    finalize HdfsScan: cardinality=0
    INFO0617 16:09:53.297000 Thread-3 com.cloudera.impala.planner.**Hd**fsScanNode]
    finalize HdfsScan: #nodes=1
    INFO0617 16:09:53.297000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    grouping expr: search_parameters #distinct=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    grouping expr: <slot 2> #distinct=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Hd**fsScanNode]
    collecting partitions for table all_lookups
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Hd**fsScanNode]
    finalize HdfsScan: cardinality=0
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Hd**fsScanNode]
    finalize HdfsScan: #nodes=1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    grouping expr: search_parameters #distinct=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ha**shJoinNode]
    stats HashJoin: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    create plan fragments
    INFO0617 16:09:53.299000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    memlimit=0
    INFO0617 16:09:53.310000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    broadcast: cost=0
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    card=-1 row_size=0.0 #nodes=1
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    partition: cost=0
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    lhs card=-1 row_size=24.0
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    rhs card=-1 row_size=0.0
    INFO0617 16:09:53.317000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    9:AGGREGATE
    output: SUM(<slot 10>)
    group by: <slot 9>
    8:EXCHANGE

    INFO0617 16:09:53.317000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    finalize plan fragments
    INFO0617 16:09:53.641000 Thread-3 com.cloudera.impala.service.**Fr**ontend]
    get scan range locations
    INFO0617 16:09:53.678000 Thread-3 com.cloudera.impala.service.**Fr**ontend]
    create result set metadata
    INFO0617 16:09:53.690000 Thread-3 com.cloudera.impala.service.**Jn**iFrontend]
    PLAN FRAGMENT 0
    PARTITION: UNPARTITIONED

    11:EXCHANGE
    limit: 50
    tuple ids: 2 5

    PLAN FRAGMENT 1
    PARTITION: HASH_PARTITIONED: <slot 2>

    STREAM DATA SINK
    EXCHANGE ID: 11
    UNPARTITIONED

    5:HASH JOIN
    join op: INNER JOIN (BROADCAST)
    hash predicates:
    <slot 4> = <slot 9>
    limit: 50
    tuple ids: 2 5

    ----10:EXCHANGE
    tuple ids: 5
    2:AGGREGATE
    output: COUNT(<slot 3>)
    group by: <slot 2>
    tuple ids: 2
    7:AGGREGATE
    group by: <slot 2>, <slot 3>
    tuple ids: 1
    6:EXCHANGE
    tuple ids: 1

    PLAN FRAGMENT 2
    PARTITION: HASH_PARTITIONED: <slot 9>

    STREAM DATA SINK
    EXCHANGE ID: 10
    UNPARTITIONED

    9:AGGREGATE
    output: SUM(<slot 10>)
    group by: <slot 9>
    tuple ids: 5
    8:EXCHANGE
    tuple ids: 5

    PLAN FRAGMENT 3
    PARTITION: RANDOM

    STREAM DATA SINK
    EXCHANGE ID: 8
    HASH_PARTITIONED: <slot 9>

    4:AGGREGATE
    output: COUNT(*)
    group by: search_parameters
    tuple ids: 5
    3:SCAN HDFS
    table=default.all_lookups #partitions=1 size=397.15MB
    tuple ids: 4

    PLAN FRAGMENT 4
    PARTITION: RANDOM

    STREAM DATA SINK
    EXCHANGE ID: 6
    HASH_PARTITIONED: <slot 2>

    1:AGGREGATE
    group by: search_parameters, reseller
    tuple ids: 1
    0:SCAN HDFS
    table=default.all_lookups #partitions=1 size=397.15MB
    tuple ids: 0

    I0617 16:09:53.755270 9737 coordinator.cc:290] Exec()
    query_id=8eb364939bbb41b0:**a0a6**981049ec71f0
    I0617 16:09:53.755636 9737 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:**a0a6**981049ec71f0
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f1
    I0617 16:09:53.884407 9737 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:**a0a6**981049ec71f1
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:53.888123 9737 exchange-node.cc:50] Exch id=11
    input_desc=Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])

    output_desc=Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.021406 9737 coordinator.cc:377] starting 4 backends
    for query 8eb364939bbb41b0:**a0a6981049ec7**1f0
    I0617 16:09:54.021667 9900 client-cache.cc:98] CreateClient(): adding
    new client for ip-10-245-112-238.us-west-2.**co**mpute.internal:22000
    I0617 16:09:54.022987 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f2
    coord=ip-10-245-112-238.us-**wes**t-2.compute.internal:22000
    backend#=0
    I0617 16:09:54.023074 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:**a0a6**981049ec71f0
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f2
    I0617 16:09:54.032918 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:**a0a6**981049ec71f2
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.034090 9901 exchange-node.cc:50] Exch id=6
    input_desc=Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1
    offset=8 null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])

    output_desc=Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1
    offset=8 null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    I0617 16:09:54.044239 9901 exchange-node.cc:50] Exch id=10
    input_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])

    output_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.618634 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f3
    coord=ip-10-245-112-238.us-**wes**t-2.compute.internal:22000
    backend#=1
    I0617 16:09:54.618748 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:**a0a6**981049ec71f0
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f3
    I0617 16:09:54.623918 9902 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f2
    I0617 16:09:54.631587 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:**a0a6**981049ec71f3
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.631726 9901 exchange-node.cc:50] Exch id=8
    input_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])

    output_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.974669 9905 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f3
    I0617 16:09:54.977622 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f4
    coord=ip-10-245-112-238.us-**wes**t-2.compute.internal:22000
    backend#=2
    I0617 16:09:54.977705 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:**a0a6**981049ec71f0
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f4
    I0617 16:09:55.005800 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:**a0a6**981049ec71f4
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:55.522994 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f5
    coord=ip-10-245-112-238.us-**wes**t-2.compute.internal:22000
    backend#=3
    I0617 16:09:55.523129 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:**a0a6**981049ec71f0
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f5
    I0617 16:09:55.529832 9908 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f4
    I0617 16:09:55.550024 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:**a0a6**981049ec71f5
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:56.376770 9917 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f1
    I0617 16:09:56.379158 9916 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f5
    I0617 16:09:56.474069 9737 impala-beeswax-server.cc:290]
    get_results_metadata(): query_id=8eb364939bbb41b0:**a0a6**981049ec71f0
    I0617 16:10:01.533941 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 12% Complete (2 out of 16)
    I0617 16:10:02.386214 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 25% Complete (4 out of 16)
    I0617 16:10:06.535214 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 31% Complete (5 out of 16)
    I0617 16:10:07.386981 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 37% Complete (6 out of 16)
    I0617 16:10:11.535930 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 43% Complete (7 out of 16)
    I0617 16:10:12.387742 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 50% Complete (8 out of 16)
    I0617 16:10:16.536659 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 62% Complete (10 out of 16)
    I0617 16:10:17.388424 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 75% Complete (12 out of 16)
    I0617 16:10:21.556238 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 87% Complete (14 out of 16)
    I0617 16:10:21.556130 10155 client-cache.cc:98] CreateClient(): adding
    new client for ip-10-245-112-238.us-west-2.**co**mpute.internal:22000
    I0617 16:10:22.389418 10156 progress-updater.cc:45] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0 100% Complete (16 out of 16)
    I0617 16:10:25.982318 9906 client-cache.cc:98] CreateClient(): adding
    new client for ip-10-245-112-238.us-west-2.**co**mpute.internal:22000

    On Monday, June 17, 2013 12:30:34 PM UTC-4, Ricky Saltzer wrote:

    It would be the log file before this one
    On Jun 17, 2013 12:14 PM, "Alex Minnaar" wrote:

    It looks like the last log file was this one. But I'm not sure it
    tells us anything either.

    Log file created at: 2013/06/17 16:10:41
    Running on machine: ip-10-245-112-238
    Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
    I0617 16:10:41.500978 15390 daemon.cc:34] impalad version 1.0
    RELEASE (build d1bf0d1dac339af3692ffa17a5e3fd******ae0aed751f)
    Built on Sun, 28 Apr 2013 15:58:48 PST
    I0617 16:10:41.521280 15390 daemon.cc:35] Using hostname:
    ip-10-245-112-238.us-west-2.**co****mpute.internal
    I0617 16:10:41.522235 15390 logging.cc:76] Flags (see also /varz are
    on debug webserver):
    --dump_ir=false
    --module_output=
    --be_port=22000
    --hostname=ip-10-245-112-238.**u****s-west-2.compute.internal
    --keytab_file=
    --mem_limit=80%
    --planservice_host=localhost
    --planservice_port=20000
    --principal=
    --exchg_node_buffer_size_**bytes****=10485760
    --max_row_batches=0
    --randomize_splits=false
    --num_disks=0
    --num_threads_per_disk=1
    --read_size=8388608
    --enable_webserver=true
    --state_store_host=ip-10-245-**1****12-238.us-west-2.compute.**inter
    ****nal
    --state_store_subscriber_port=******23000
    --use_statestore=true
    --nn=ip-10-245-112-238.us-**west****-2.compute.internal
    --nn_port=8020
    --serialize_batch=false
    --status_report_interval=5
    --compress_rowbatches=true
    --num_threads_per_core=3
    --abort_on_config_error=true
    --be_service_threads=64
    --beeswax_port=21000
    --default_query_options=
    --fe_service_threads=64
    --heap_profile_dir=
    --hs2_port=21050
    --load_catalog_at_startup=**fals****e
    --log_mem_usage_interval=0
    --log_query_to_file=true
    --query_log_size=25
    --use_planservice=false
    --statestore_subscriber_**timeou****t_seconds=10
    --state_store_port=24000
    --statestore_max_missed_**heartb****eats=5
    --statestore_num_heartbeat_**thr****eads=10
    --statestore_suspect_**heartbeat****s=2
    --kerberos_reinit_interval=60
    --sasl_path=/usr/lib/sasl2:/**us****r/lib64/sasl2:/usr/local/**lib/*
    *s**asl2:/usr/lib/x86_64-**linux-**gnu**/sasl2
    --web_log_bytes=1048576
    --log_filename=impalad
    --periodic_counter_update_**peri****od_ms=500
    --rpc_cnxn_attempts=10
    --rpc_cnxn_retry_interval_ms=**2****000
    --enable_webserver_doc_root=**tr****ue
    --webserver_doc_root=/usr/lib/******impala
    --webserver_interface=
    --webserver_port=25000
    --flagfile=/run/cloudera-scm-**a****gent/process/40-impala-**IMPALAD
    ****/impala-conf/impalad_**flags
    --fromenv=
    --tryfromenv=
    --undefok=
    --tab_completion_columns=80
    --tab_completion_word=
    --help=false
    --helpfull=false
    --helpmatch=
    --helpon=
    --helppackage=false
    --helpshort=false
    --helpxml=false
    --version=false
    --alsologtoemail=
    --alsologtostderr=false
    --drop_log_memory=true
    --log_backtrace_at=
    --log_dir=/var/log/impalad
    --log_link=
    --log_prefix=true
    --logbuflevel=-1
    --logbufsecs=30
    --logemaillevel=999
    --logmailer=/bin/mail
    --logtostderr=false
    --max_log_size=200
    --minloglevel=0
    --stderrthreshold=2
    --stop_logging_if_full_disk=**fa****lse
    --symbolize_stacktrace=true
    --v=1
    --vmodule=
    I0617 16:10:41.533797 15390 mem-info.cc:66] Physical Memory: 3.66 GB
    I0617 16:10:41.539779 15390 daemon.cc:43] Cpu Info:
    Model: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
    Cores: 1
    L1 Cache: 32.00 KB
    L2 Cache: 256.00 KB
    L3 Cache: 20.00 MB
    Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    I0617 16:10:41.539821 15390 daemon.cc:44] Disk Info:
    Num disks 2: xvda, xvdb
    I0617 16:10:41.539855 15390 daemon.cc:45] Mem Info: 3.66 GB
    I0617 16:10:50.755198 15390 impala-server.cc:1809] Default query
    options:TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_**exceede****d (bool) = false,
    }
    WARN0617 16:10:51.236000 main org.apache.hadoop.conf.**Configu****ration]
    mapred.max.split.size is deprecated. Instead, use mapreduce.input.**
    fileinputforma****t.split.maxsize
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configu****ration]
    mapred.min.split.size is deprecated. Instead, use mapreduce.input.**
    fileinputforma****t.split.minsize
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configu****ration]
    mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.
    **fileinputforma****t.split.minsize.**per.rack
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configu****ration]
    mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.
    **fileinputforma****t.split.minsize.**per.node
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configu****ration]
    mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configu****ration]
    mapred.reduce.tasks.**speculativ****e.execution is deprecated.
    Instead, use mapreduce.reduce.speculative
    WARN0617 16:10:51.892000 main org.apache.hadoop.conf.**Configu****ration]
    org.apache.hadoop.hive.conf.**Lo****opingByteArrayInputStream@**66b5
    ****1404:an attempt to override final parameter: mapreduce.job.end-*
    *notification****.max.retry.**interval; Ignoring.
    WARN0617 16:10:51.954000 main org.apache.hadoop.conf.**Configu****ration]
    org.apache.hadoop.hive.conf.**Lo****opingByteArrayInputStream@**66b5
    ****1404:an attempt to override final parameter: mapreduce.job.end-*
    *notification****.max.attempts; Ignoring.
    WARN0617 16:10:51.972000 main org.apache.hadoop.hive.conf.**Hi****veConf]
    DEPRECATED: Configuration property hive.metastore.local no longer has any
    effect. Make sure to provide a valid value for hive.metastore.uris if you
    are connecting to a remote metastore.
    INFO0617 16:10:52.240000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-******
    west-2.compute.internal:9083
    INFO0617 16:10:52.494000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:10:53.495000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:53.495000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-******
    west-2.compute.internal:9083
    INFO0617 16:10:53.497000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:10:54.498000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:54.498000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-******
    west-2.compute.internal:9083
    INFO0617 16:10:54.505000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:10:55.506000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:55.507000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-******
    west-2.compute.internal:9083
    INFO0617 16:10:55.515000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:10:56.515000 main hive.metastore] Connected to metastore.
    INFO0617 16:10:56.515000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-******
    west-2.compute.internal:9083
    INFO0617 16:10:56.519000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:10:57.519000 main hive.metastore] Connected to metastore.
    I0617 16:10:59.891072 15390 impala-server.cc:2074] Impala Beeswax
    Service listening on 21000
    I0617 16:10:59.893288 15390 impala-server.cc:2085] Impala
    HiveServer2 Service listening on 21050
    I0617 16:10:59.893335 15390 impala-server.cc:2093]
    ImpalaInternalService listening on 22000
    I0617 16:10:59.905498 15390 thrift-server.cc:365] ThriftServer
    'backend' started on port: 22000
    I0617 16:10:59.905547 15390 exec-env.cc:146] Starting global services
    I0617 16:10:59.905670 15390 exec-env.cc:167] Using global memory
    limit: 2.93 GB
    I0617 16:10:59.909502 15390 webserver.cc:129] Starting webserver on
    all interfaces, port 25000
    I0617 16:10:59.909559 15390 webserver.cc:139] Document root:
    /usr/lib/impala
    I0617 16:10:59.921815 15390 webserver.cc:178] Webserver started
    I0617 16:10:59.921897 15390 simple-scheduler.cc:99] Starting simple
    scheduler
    I0617 16:10:59.926578 15390 simple-scheduler.cc:133]
    Simple-scheduler using 10.245.112.238 as IP address
    I0617 16:10:59.926632 15390 state-store-subscriber.cc:137] Starting
    subscriber
    I0617 16:10:59.937484 15390 thrift-server.cc:365] ThriftServer
    'StateStoreSubscriber' started on port: 23000
    I0617 16:10:59.949283 15390 client-cache.cc:98] CreateClient():
    adding new client for ip-10-245-112-238.us-west-2.**co****
    mpute.internal:24000
    I0617 16:10:59.982250 15390 thrift-server.cc:365] ThriftServer
    'beeswax-frontend' started on port: 21000
    I0617 16:11:00.017025 15390 thrift-server.cc:365] ThriftServer
    'hiveServer2-frontend' started on port: 21050
    I0617 16:11:00.017144 15390 impalad-main.cc:101] Impala has started.

    On Monday, June 17, 2013 12:08:01 PM UTC-4, Ricky Saltzer wrote:

    Hey Alex -

    Sorry about that, I forgot that the log get's rolled over after a
    crash. Inside of */var/log/impalad*, you should be able to find
    the last log file written to before the crash using an "*ls -lthr*",
    take a peek inside..it should contain a lot more data.

    Thanks,
    Ricky

    On Mon, Jun 17, 2013 at 9:04 AM, Alex Minnaar wrote:

    I have attached what was in the impalad.INFO file

    Alex

    On Monday, June 17, 2013 11:44:10 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Try connecting to the node via SSH and grabbing the log
    directly from */var/log/impalad *
    *
    *
    Ricky


    On Mon, Jun 17, 2013 at 8:39 AM, Alex Minnaar <minna...@gmail.com
    wrote:
    OK I figured out how to set the GLOG level, but when I try to
    download the impalad.INFO file I get

    Problem accessing /cmf/process/all/logs/**download********.
    Reason:

    Connection timed out

    Could not connect to host.

    On Monday, June 17, 2013 10:39:21 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Could you set the GLOG level to 2<http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_logging.html> (you
    can use Cloudera manager to do it, too). Then, re-run your query and attach
    the impalad.INFO from the coordinator (the node you submit your query to).

    Ricky


    On Mon, Jun 17, 2013 at 6:23 AM, Alex Minnaar <
    minna...@gmail.com> wrote:
    Impala is working for me with simple select queries but it is
    not working with queries involving joins. When I execute the join query in
    the impala shell, I get the following result.
    Query finished, fetching results ...
    Error communicating with impalad: TSocket read 0 bytes

    And then I am no longer connected to impala. How do I fix
    this?

    Alex


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com
  • Alex Minnaar at Jun 17, 2013 at 6:28 pm
    So do I run this command inside the impala shell?
    On Monday, June 17, 2013 2:07:49 PM UTC-4, Ricky Saltzer wrote:

    Hey Alex -

    Alright, you might want to check that the node isn't running out of memory
    during the query...

    You can monitor with this command

    $ watch -d -n 1 "free -m"



    On Mon, Jun 17, 2013 at 11:00 AM, Alex Minnaar <minna...@gmail.com<javascript:>
    wrote:
    I just tried the query again and it froze and now the health of that host
    has turned to bad, so I guess I need to fix that first

    On Monday, June 17, 2013 1:29:12 PM UTC-4, Ricky Saltzer wrote:

    Hey Alex -

    Are you sure GLOG was on level 2 when you issued this query? Maybe you
    could issue it once more now that GLOG is on level 2, and get then pull the
    log which contains that query. If you could, please attach the log instead
    of pasting (easier to read).

    Thanks
    Ricky

    On Mon, Jun 17, 2013 at 10:20 AM, Alex Minnaar wrote:

    Yes that's the one

    On Monday, June 17, 2013 1:18:50 PM UTC-4, Ricky Saltzer wrote:

    Hey Alex -

    So, the following query is the one crashing?

    select A.search_parameters,reseller_c****ount,total from (select
    search_parameters,count(distin****ct(reseller)) as reseller_count
    from all_lookups group by search_parameters)A join (select
    search_parameters,count(*) as total from all_lookups group by
    search_parameters)B on A.search_parameters=B.search_p****arameters
    limit 50

    Ricky

    On Mon, Jun 17, 2013 at 10:05 AM, Alex Minnaar wrote:

    OK the previous log file was

    Log file created at: 2013/06/17 15:59:58
    Running on machine: ip-10-245-112-238
    Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
    I0617 15:59:58.518903 9628 daemon.cc:34] impalad version 1.0 RELEASE
    (build d1bf0d1dac339af3692ffa17a5e3fd****ae0aed751f)
    Built on Sun, 28 Apr 2013 15:58:48 PST
    I0617 15:59:58.547598 9628 daemon.cc:35] Using hostname:
    ip-10-245-112-238.us-west-2.**co**mpute.internal
    I0617 15:59:58.548459 9628 logging.cc:76] Flags (see also /varz are
    on debug webserver):
    --dump_ir=false
    --module_output=
    --be_port=22000
    --hostname=ip-10-245-112-238.**u**s-west-2.compute.internal
    --keytab_file=
    --mem_limit=80%
    --planservice_host=localhost
    --planservice_port=20000
    --principal=
    --exchg_node_buffer_size_**bytes**=10485760
    --max_row_batches=0
    --randomize_splits=false
    --num_disks=0
    --num_threads_per_disk=1
    --read_size=8388608
    --enable_webserver=true
    --state_store_host=ip-10-245-**1**12-238.us-west-2.compute.**inter**
    nal
    --state_store_subscriber_port=****23000
    --use_statestore=true
    --nn=ip-10-245-112-238.us-**west**-2.compute.internal
    --nn_port=8020
    --serialize_batch=false
    --status_report_interval=5
    --compress_rowbatches=true
    --num_threads_per_core=3
    --abort_on_config_error=true
    --be_service_threads=64
    --beeswax_port=21000
    --default_query_options=
    --fe_service_threads=64
    --heap_profile_dir=
    --hs2_port=21050
    --load_catalog_at_startup=**fals**e
    --log_mem_usage_interval=0
    --log_query_to_file=true
    --query_log_size=25
    --use_planservice=false
    --statestore_subscriber_**timeou**t_seconds=10
    --state_store_port=24000
    --statestore_max_missed_**heartb**eats=5
    --statestore_num_heartbeat_**thr**eads=10
    --statestore_suspect_**heartbeat**s=2
    --kerberos_reinit_interval=60
    --sasl_path=/usr/lib/sasl2:/**us**r/lib64/sasl2:/usr/local/**lib/**
    sasl2:/usr/lib/x86_64-**linux-**gnu/sasl2
    --web_log_bytes=1048576
    --log_filename=impalad
    --periodic_counter_update_**peri**od_ms=500
    --rpc_cnxn_attempts=10
    --rpc_cnxn_retry_interval_ms=**2**000
    --enable_webserver_doc_root=**tr**ue
    --webserver_doc_root=/usr/lib/****impala
    --webserver_interface=
    --webserver_port=25000
    --flagfile=/run/cloudera-scm-**a**gent/process/40-impala-**IMPALAD**
    /impala-conf/impalad_**flags
    --fromenv=
    --tryfromenv=
    --undefok=
    --tab_completion_columns=80
    --tab_completion_word=
    --help=false
    --helpfull=false
    --helpmatch=
    --helpon=
    --helppackage=false
    --helpshort=false
    --helpxml=false
    --version=false
    --alsologtoemail=
    --alsologtostderr=false
    --drop_log_memory=true
    --log_backtrace_at=
    --log_dir=/var/log/impalad
    --log_link=
    --log_prefix=true
    --logbuflevel=-1
    --logbufsecs=30
    --logemaillevel=999
    --logmailer=/bin/mail
    --logtostderr=false
    --max_log_size=200
    --minloglevel=0
    --stderrthreshold=2
    --stop_logging_if_full_disk=**fa**lse
    --symbolize_stacktrace=true
    --v=1
    --vmodule=
    I0617 15:59:58.551378 9628 mem-info.cc:66] Physical Memory: 3.66 GB
    I0617 15:59:58.552459 9628 daemon.cc:43] Cpu Info:
    Model: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
    Cores: 1
    L1 Cache: 32.00 KB
    L2 Cache: 256.00 KB
    L3 Cache: 20.00 MB
    Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    I0617 15:59:58.552492 9628 daemon.cc:44] Disk Info:
    Num disks 2: xvda, xvdb
    I0617 15:59:58.552525 9628 daemon.cc:45] Mem Info: 3.66 GB
    I0617 16:00:07.111661 9628 impala-server.cc:1809] Default query
    options:TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_**exceede**d (bool) = false,
    }
    WARN0617 16:00:07.622000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.max.split.size is deprecated. Instead, use mapreduce.input.**
    fileinputforma**t.split.maxsize
    WARN0617 16:00:07.629000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.min.split.size is deprecated. Instead, use mapreduce.input.**
    fileinputforma**t.split.minsize
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.
    **fileinputforma**t.split.minsize.**per.rack
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.
    **fileinputforma**t.split.minsize.**per.node
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    WARN0617 16:00:07.630000 main org.apache.hadoop.conf.**Configu**ration]
    mapred.reduce.tasks.**speculativ**e.execution is deprecated.
    Instead, use mapreduce.reduce.speculative
    WARN0617 16:00:08.620000 main org.apache.hadoop.conf.**Configu**ration]
    org.apache.hadoop.hive.conf.**Lo**opingByteArrayInputStream@**24c6**8a98:an
    attempt to override final parameter: mapreduce.job.end-**notification
    **.max.retry.**interval; Ignoring.
    WARN0617 16:00:08.689000 main org.apache.hadoop.conf.**Configu**ration]
    org.apache.hadoop.hive.conf.**Lo**opingByteArrayInputStream@**24c6**8a98:an
    attempt to override final parameter: mapreduce.job.end-**notification
    **.max.attempts; Ignoring.
    WARN0617 16:00:08.713000 main org.apache.hadoop.hive.conf.**Hi**veConf]
    DEPRECATED: Configuration property hive.metastore.local no longer has any
    effect. Make sure to provide a valid value for hive.metastore.uris if you
    are connecting to a remote metastore.
    INFO0617 16:00:08.882000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-****
    west-2.compute.internal:9083
    INFO0617 16:00:09.081000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:00:10.081000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:10.082000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-****
    west-2.compute.internal:9083
    INFO0617 16:00:10.084000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:00:11.084000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:11.085000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-****
    west-2.compute.internal:9083
    INFO0617 16:00:11.087000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:00:12.092000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:12.093000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-****
    west-2.compute.internal:9083
    INFO0617 16:00:12.095000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:00:13.095000 main hive.metastore] Connected to metastore.
    INFO0617 16:00:13.096000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-****
    west-2.compute.internal:9083
    INFO0617 16:00:13.098000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:00:14.098000 main hive.metastore] Connected to metastore.
    I0617 16:00:16.692451 9628 impala-server.cc:2074] Impala Beeswax
    Service listening on 21000
    I0617 16:00:16.692525 9628 impala-server.cc:2085] Impala HiveServer2
    Service listening on 21050
    I0617 16:00:16.692553 9628 impala-server.cc:2093]
    ImpalaInternalService listening on 22000
    I0617 16:00:16.699118 9628 thrift-server.cc:365] ThriftServer
    'backend' started on port: 22000
    I0617 16:00:16.699157 9628 exec-env.cc:146] Starting global services
    I0617 16:00:16.699275 9628 exec-env.cc:167] Using global memory
    limit: 2.93 GB
    I0617 16:00:16.699409 9628 webserver.cc:129] Starting webserver on
    all interfaces, port 25000
    I0617 16:00:16.699470 9628 webserver.cc:139] Document root:
    /usr/lib/impala
    I0617 16:00:16.711299 9628 webserver.cc:178] Webserver started
    I0617 16:00:16.711383 9628 simple-scheduler.cc:99] Starting simple
    scheduler
    I0617 16:00:16.712973 9628 simple-scheduler.cc:133] Simple-scheduler
    using 10.245.112.238 as IP address
    I0617 16:00:16.713028 9628 state-store-subscriber.cc:137] Starting
    subscriber
    I0617 16:00:16.719905 9628 thrift-server.cc:365] ThriftServer
    'StateStoreSubscriber' started on port: 23000
    I0617 16:00:16.722825 9628 client-cache.cc:98] CreateClient():
    adding new client for ip-10-245-112-238.us-west-2.**co**
    mpute.internal:24000
    I0617 16:00:16.745887 9628 thrift-server.cc:365] ThriftServer
    'beeswax-frontend' started on port: 21000
    I0617 16:00:16.761379 9628 thrift-server.cc:365] ThriftServer
    'hiveServer2-frontend' started on port: 21050
    I0617 16:00:16.761443 9628 impalad-main.cc:101] Impala has started.
    I0617 16:09:51.630908 9737 impala-beeswax-server.cc:137] query():
    query=select A.search_parameters,reseller_**c**ount,total from
    (select search_parameters,count(**distin**ct(reseller)) as
    reseller_count from all_lookups group by search_parameters)A join (select
    search_parameters,count(*) as total from all_lookups group by
    search_parameters)B on A.search_parameters=B.search_**p**arameters
    limit 50
    I0617 16:09:51.631084 9737 impala-beeswax-server.cc:477] query:
    Query {
    01: query (string) = "select A.search_[...](315)",
    03: configuration (list) = list<string>[0] {
    },
    04: hadoop_user (string) = "ubuntu",
    }
    I0617 16:09:51.639899 9737 impala-beeswax-server.cc:489]
    TClientRequest.queryOptions: TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_**exceede**d (bool) = false,
    }
    INFO0617 16:09:51.703000 Thread-3 com.cloudera.impala.service.**Fr**ontend]
    analyze query select A.search_parameters,reseller_**c**ount,total
    from (select search_parameters,count(**distin**ct(reseller)) as
    reseller_count from all_lookups group by search_parameters)A join (select
    search_parameters,count(*) as total from all_lookups group by
    search_parameters)B on A.search_parameters=B.search_**p**arameters
    limit 50
    INFO0617 16:09:52.332000 Thread-3 com.cloudera.impala.catalog.**Hd**fsTable]
    load table all_lookups
    INFO0617 16:09:52.697000 Thread-3 com.cloudera.impala.catalog.**Hd**fsTable]
    load partition block md for all_lookups
    INFO0617 16:09:52.716000 Thread-3 com.cloudera.impala.catalog.**Hd**fsTable]
    loaded partition PartitionBlockMetadata{#**blocks**=0, #filenames=0,
    totalStringLen=0}
    INFO0617 16:09:52.949000 Thread-3 com.cloudera.impala.catalog.**Hd**fsTable]
    loaded partition PartitionBlockMetadata{#**blocks**=8, #filenames=8,
    totalStringLen=1197}
    INFO0617 16:09:53.073000 Thread-3 com.cloudera.impala.catalog.**Hd**fsTable]
    loaded disk ids for table default.all_lookups
    INFO0617 16:09:53.074000 Thread-3 com.cloudera.impala.catalog.**Hd**fsTable]
    1
    INFO0617 16:09:53.074000 Thread-3 com.cloudera.impala.catalog.**Hd**fsTable]
    table #rows=0
    INFO0617 16:09:53.103000 Thread-3 com.cloudera.impala.analysis.**A**ggregateInfo]
    agg info:
    AggregateInfo{grouping_exprs=(****SlotRef{tblName=null,
    col=search_parameters, id=0} SlotRef{tblName=null, col=reseller, id=1}),
    aggregate_exprs=, agg_tuple=TupleDescriptor{id=**1**, tbl=null,
    byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=2, col=null,
    type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=3,
    col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=**null**, col=search_parameters,
    id=0}:SlotRef{tblName=null, col=null, id=2} SlotRef{tblName=null,
    col=reseller, id=1}:SlotRef{tblName=null, col=null, id=3})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(****SlotRef{tblName=null, col=null,
    id=2} SlotRef{tblName=null, col=null, id=3}), aggregate_exprs=,
    agg_tuple=TupleDescriptor{id=**1**, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=2, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=3, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=**null**,
    col=search_parameters, id=0}:SlotRef{tblName=null, col=null, id=2}
    SlotRef{tblName=null, col=reseller, id=1}:SlotRef{tblName=null, col=null,
    id=3})}
    secondPhaseDistinctAggInfo:
    AggregateInfo{grouping_exprs=(****SlotRef{tblName=null, col=null,
    id=2}), aggregate_exprs=(**AggregateExpr**{op=COUNT, isStar=false,
    isDistinct=false, (SlotRef{tblName=null, col=null, id=3})}),
    agg_tuple=TupleDescriptor{id=**2**, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=4, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=5, col=null, type=BIGINT,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=**null**,
    col=search_parameters, id=0}:SlotRef{tblName=null, col=null, id=4}
    AggregateExpr{op=COUNT, isStar=false, isDistinct=true,
    (SlotRef{tblName=null, col=reseller, id=1})}:SlotRef{tblName=null,
    col=null, id=5})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(****SlotRef{tblName=null, col=null,
    id=4}), aggregate_exprs=(**AggregateExpr**{op=SUM, isStar=false,
    isDistinct=false, (SlotRef{tblName=null, col=null, id=5})}),
    agg_tuple=TupleDescriptor{id=**2**, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=4, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=5, col=null, type=BIGINT,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=**null**,
    col=search_parameters, id=0}:SlotRef{tblName=null, col=null, id=4}
    AggregateExpr{op=COUNT, isStar=false, isDistinct=true,
    (SlotRef{tblName=null, col=reseller, id=1})}:SlotRef{tblName=null,
    col=null, id=5})}
    INFO0617 16:09:53.110000 Thread-3 com.cloudera.impala.analysis.**A**ggregateInfo]
    agg info:
    AggregateInfo{grouping_exprs=(****SlotRef{tblName=null,
    col=search_parameters, id=8}), aggregate_exprs=(**AggregateExpr**{op=COUNT,
    isStar=true, isDistinct=false, }), agg_tuple=TupleDescriptor{id=**5**,
    tbl=null, byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=9,
    col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=10,
    col=null, type=BIGINT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=**null**, col=search_parameters,
    id=8}:SlotRef{tblName=null, col=null, id=9} AggregateExpr{op=COUNT,
    isStar=true, isDistinct=false, }:SlotRef{tblName=null, col=null, id=10})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(****SlotRef{tblName=null, col=null,
    id=9}), aggregate_exprs=(**AggregateExpr**{op=SUM, isStar=false,
    isDistinct=false, (SlotRef{tblName=null, col=null, id=10})}),
    agg_tuple=TupleDescriptor{id=**5**, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=9, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=10, col=null,
    type=BIGINT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=**null**, col=search_parameters,
    id=8}:SlotRef{tblName=null, col=null, id=9} AggregateExpr{op=COUNT,
    isStar=true, isDistinct=false, }:SlotRef{tblName=null, col=null, id=10})}
    INFO0617 16:09:53.140000 Thread-3 com.cloudera.impala.analysis.**B**inaryPredicate]
    A.search_parameters = B.search_parameters selectivity: 0.1
    INFO0617 16:09:53.257000 Thread-3 com.cloudera.impala.service.**Fr**ontend]
    create plan
    INFO0617 16:09:53.282000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    create single-node plan
    INFO0617 16:09:53.290000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    grouping expr: search_parameters #distinct=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    grouping expr: <slot 2> #distinct=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    grouping expr: search_parameters #distinct=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.291000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.297000 Thread-3 com.cloudera.impala.planner.**Hd**fsScanNode]
    collecting partitions for table all_lookups
    INFO0617 16:09:53.297000 Thread-3 com.cloudera.impala.planner.**Hd**fsScanNode]
    finalize HdfsScan: cardinality=0
    INFO0617 16:09:53.297000 Thread-3 com.cloudera.impala.planner.**Hd**fsScanNode]
    finalize HdfsScan: #nodes=1
    INFO0617 16:09:53.297000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    grouping expr: search_parameters #distinct=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    grouping expr: <slot 2> #distinct=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Hd**fsScanNode]
    collecting partitions for table all_lookups
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Hd**fsScanNode]
    finalize HdfsScan: cardinality=0
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Hd**fsScanNode]
    finalize HdfsScan: #nodes=1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    grouping expr: search_parameters #distinct=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ag**gregationNode]
    stats Agg: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Ha**shJoinNode]
    stats HashJoin: cardinality=-1
    INFO0617 16:09:53.298000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    create plan fragments
    INFO0617 16:09:53.299000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    memlimit=0
    INFO0617 16:09:53.310000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    broadcast: cost=0
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    card=-1 row_size=0.0 #nodes=1
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    partition: cost=0
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    lhs card=-1 row_size=24.0
    INFO0617 16:09:53.311000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    rhs card=-1 row_size=0.0
    INFO0617 16:09:53.317000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    9:AGGREGATE
    output: SUM(<slot 10>)
    group by: <slot 9>
    8:EXCHANGE

    INFO0617 16:09:53.317000 Thread-3 com.cloudera.impala.planner.**Pl**anner]
    finalize plan fragments
    INFO0617 16:09:53.641000 Thread-3 com.cloudera.impala.service.**Fr**ontend]
    get scan range locations
    INFO0617 16:09:53.678000 Thread-3 com.cloudera.impala.service.**Fr**ontend]
    create result set metadata
    INFO0617 16:09:53.690000 Thread-3 com.cloudera.impala.service.**Jn**iFrontend]
    PLAN FRAGMENT 0
    PARTITION: UNPARTITIONED

    11:EXCHANGE
    limit: 50
    tuple ids: 2 5

    PLAN FRAGMENT 1
    PARTITION: HASH_PARTITIONED: <slot 2>

    STREAM DATA SINK
    EXCHANGE ID: 11
    UNPARTITIONED

    5:HASH JOIN
    join op: INNER JOIN (BROADCAST)
    hash predicates:
    <slot 4> = <slot 9>
    limit: 50
    tuple ids: 2 5

    ----10:EXCHANGE
    tuple ids: 5
    2:AGGREGATE
    output: COUNT(<slot 3>)
    group by: <slot 2>
    tuple ids: 2
    7:AGGREGATE
    group by: <slot 2>, <slot 3>
    tuple ids: 1
    6:EXCHANGE
    tuple ids: 1

    PLAN FRAGMENT 2
    PARTITION: HASH_PARTITIONED: <slot 9>

    STREAM DATA SINK
    EXCHANGE ID: 10
    UNPARTITIONED

    9:AGGREGATE
    output: SUM(<slot 10>)
    group by: <slot 9>
    tuple ids: 5
    8:EXCHANGE
    tuple ids: 5

    PLAN FRAGMENT 3
    PARTITION: RANDOM

    STREAM DATA SINK
    EXCHANGE ID: 8
    HASH_PARTITIONED: <slot 9>

    4:AGGREGATE
    output: COUNT(*)
    group by: search_parameters
    tuple ids: 5
    3:SCAN HDFS
    table=default.all_lookups #partitions=1 size=397.15MB
    tuple ids: 4

    PLAN FRAGMENT 4
    PARTITION: RANDOM

    STREAM DATA SINK
    EXCHANGE ID: 6
    HASH_PARTITIONED: <slot 2>

    1:AGGREGATE
    group by: search_parameters, reseller
    tuple ids: 1
    0:SCAN HDFS
    table=default.all_lookups #partitions=1 size=397.15MB
    tuple ids: 0

    I0617 16:09:53.755270 9737 coordinator.cc:290] Exec()
    query_id=8eb364939bbb41b0:**a0a6**981049ec71f0
    I0617 16:09:53.755636 9737 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:**a0a6**981049ec71f0
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f1
    I0617 16:09:53.884407 9737 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:**a0a6**981049ec71f1
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:53.888123 9737 exchange-node.cc:50] Exch id=11
    input_desc=Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])

    output_desc=Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.021406 9737 coordinator.cc:377] starting 4 backends
    for query 8eb364939bbb41b0:**a0a6981049ec7**1f0
    I0617 16:09:54.021667 9900 client-cache.cc:98] CreateClient():
    adding new client for ip-10-245-112-238.us-west-2.**co**
    mpute.internal:22000
    I0617 16:09:54.022987 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f2
    coord=ip-10-245-112-238.us-**wes**t-2.compute.internal:22000
    backend#=0
    I0617 16:09:54.023074 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:**a0a6**981049ec71f0
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f2
    I0617 16:09:54.032918 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:**a0a6**981049ec71f2
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.034090 9901 exchange-node.cc:50] Exch id=6
    input_desc=Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1
    offset=8 null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])

    output_desc=Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1
    offset=8 null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    I0617 16:09:54.044239 9901 exchange-node.cc:50] Exch id=10
    input_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])

    output_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.618634 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f3
    coord=ip-10-245-112-238.us-**wes**t-2.compute.internal:22000
    backend#=1
    I0617 16:09:54.618748 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:**a0a6**981049ec71f0
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f3
    I0617 16:09:54.623918 9902 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f2
    I0617 16:09:54.631587 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:**a0a6**981049ec71f3
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.631726 9901 exchange-node.cc:50] Exch id=8
    input_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])

    output_desc=Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1
    offset=16 null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:54.974669 9905 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f3
    I0617 16:09:54.977622 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f4
    coord=ip-10-245-112-238.us-**wes**t-2.compute.internal:22000
    backend#=2
    I0617 16:09:54.977705 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:**a0a6**981049ec71f0
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f4
    I0617 16:09:55.005800 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:**a0a6**981049ec71f4
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:55.522994 9901 impala-server.cc:1660] ExecPlanFragment()
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f5
    coord=ip-10-245-112-238.us-**wes**t-2.compute.internal:22000
    backend#=3
    I0617 16:09:55.523129 9901 plan-fragment-executor.cc:75] Prepare():
    query_id=8eb364939bbb41b0:**a0a6**981049ec71f0
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f5
    I0617 16:09:55.529832 9908 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f4
    I0617 16:09:55.550024 9901 plan-fragment-executor.cc:115] descriptor
    table for fragment=8eb364939bbb41b0:**a0a6**981049ec71f5
    tuples:
    Tuple(id=0 size=40 slots=[Slot(id=0 type=STRING col=2 offset=8
    null=(offset=0 mask=1)), Slot(id=1 type=STRING col=1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=1 size=40 slots=[Slot(id=2 type=STRING col=-1 offset=8
    null=(offset=0 mask=1)), Slot(id=3 type=STRING col=-1 offset=24
    null=(offset=0 mask=2))])
    Tuple(id=2 size=32 slots=[Slot(id=4 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=5 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    Tuple(id=4 size=24 slots=[Slot(id=8 type=STRING col=2 offset=8
    null=(offset=0 mask=1))])
    Tuple(id=5 size=32 slots=[Slot(id=9 type=STRING col=-1 offset=16
    null=(offset=0 mask=1)), Slot(id=10 type=BIGINT col=-1 offset=8
    null=(offset=0 mask=0))])
    I0617 16:09:56.376770 9917 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f1
    I0617 16:09:56.379158 9916 plan-fragment-executor.cc:213] Open():
    instance_id=8eb364939bbb41b0:**a**0a6981049ec71f5
    I0617 16:09:56.474069 9737 impala-beeswax-server.cc:290]
    get_results_metadata(): query_id=8eb364939bbb41b0:**a0a6**
    981049ec71f0
    I0617 16:10:01.533941 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 12% Complete (2 out of 16)
    I0617 16:10:02.386214 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 25% Complete (4 out of 16)
    I0617 16:10:06.535214 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 31% Complete (5 out of 16)
    I0617 16:10:07.386981 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 37% Complete (6 out of 16)
    I0617 16:10:11.535930 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 43% Complete (7 out of 16)
    I0617 16:10:12.387742 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 50% Complete (8 out of 16)
    I0617 16:10:16.536659 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 62% Complete (10 out of 16)
    I0617 16:10:17.388424 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 75% Complete (12 out of 16)
    I0617 16:10:21.556238 9901 progress-updater.cc:55] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0: 87% Complete (14 out of 16)
    I0617 16:10:21.556130 10155 client-cache.cc:98] CreateClient():
    adding new client for ip-10-245-112-238.us-west-2.**co**
    mpute.internal:22000
    I0617 16:10:22.389418 10156 progress-updater.cc:45] Query
    8eb364939bbb41b0:**a0a6981049ec7**1f0 100% Complete (16 out of 16)
    I0617 16:10:25.982318 9906 client-cache.cc:98] CreateClient():
    adding new client for ip-10-245-112-238.us-west-2.**co**
    mpute.internal:22000

    On Monday, June 17, 2013 12:30:34 PM UTC-4, Ricky Saltzer wrote:

    It would be the log file before this one
    On Jun 17, 2013 12:14 PM, "Alex Minnaar" wrote:

    It looks like the last log file was this one. But I'm not sure it
    tells us anything either.

    Log file created at: 2013/06/17 16:10:41
    Running on machine: ip-10-245-112-238
    Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
    I0617 16:10:41.500978 15390 daemon.cc:34] impalad version 1.0
    RELEASE (build d1bf0d1dac339af3692ffa17a5e3fd******ae0aed751f)
    Built on Sun, 28 Apr 2013 15:58:48 PST
    I0617 16:10:41.521280 15390 daemon.cc:35] Using hostname:
    ip-10-245-112-238.us-west-2.**co****mpute.internal
    I0617 16:10:41.522235 15390 logging.cc:76] Flags (see also /varz
    are on debug webserver):
    --dump_ir=false
    --module_output=
    --be_port=22000
    --hostname=ip-10-245-112-238.**u****s-west-2.compute.internal
    --keytab_file=
    --mem_limit=80%
    --planservice_host=localhost
    --planservice_port=20000
    --principal=
    --exchg_node_buffer_size_**bytes****=10485760
    --max_row_batches=0
    --randomize_splits=false
    --num_disks=0
    --num_threads_per_disk=1
    --read_size=8388608
    --enable_webserver=true
    --state_store_host=ip-10-245-**1****12-238.us-west-2.compute.**
    inter****nal
    --state_store_subscriber_port=******23000
    --use_statestore=true
    --nn=ip-10-245-112-238.us-**west****-2.compute.internal
    --nn_port=8020
    --serialize_batch=false
    --status_report_interval=5
    --compress_rowbatches=true
    --num_threads_per_core=3
    --abort_on_config_error=true
    --be_service_threads=64
    --beeswax_port=21000
    --default_query_options=
    --fe_service_threads=64
    --heap_profile_dir=
    --hs2_port=21050
    --load_catalog_at_startup=**fals****e
    --log_mem_usage_interval=0
    --log_query_to_file=true
    --query_log_size=25
    --use_planservice=false
    --statestore_subscriber_**timeou****t_seconds=10
    --state_store_port=24000
    --statestore_max_missed_**heartb****eats=5
    --statestore_num_heartbeat_**thr****eads=10
    --statestore_suspect_**heartbeat****s=2
    --kerberos_reinit_interval=60
    --sasl_path=/usr/lib/sasl2:/**us****r/lib64/sasl2:/usr/local/**lib/
    **s**asl2:/usr/lib/x86_64-**linux-**gnu**/sasl2
    --web_log_bytes=1048576
    --log_filename=impalad
    --periodic_counter_update_**peri****od_ms=500
    --rpc_cnxn_attempts=10
    --rpc_cnxn_retry_interval_ms=**2****000
    --enable_webserver_doc_root=**tr****ue
    --webserver_doc_root=/usr/lib/******impala
    --webserver_interface=
    --webserver_port=25000
    --flagfile=/run/cloudera-scm-**a****gent/process/40-impala-**
    IMPALAD****/impala-conf/impalad_**flags
    --fromenv=
    --tryfromenv=
    --undefok=
    --tab_completion_columns=80
    --tab_completion_word=
    --help=false
    --helpfull=false
    --helpmatch=
    --helpon=
    --helppackage=false
    --helpshort=false
    --helpxml=false
    --version=false
    --alsologtoemail=
    --alsologtostderr=false
    --drop_log_memory=true
    --log_backtrace_at=
    --log_dir=/var/log/impalad
    --log_link=
    --log_prefix=true
    --logbuflevel=-1
    --logbufsecs=30
    --logemaillevel=999
    --logmailer=/bin/mail
    --logtostderr=false
    --max_log_size=200
    --minloglevel=0
    --stderrthreshold=2
    --stop_logging_if_full_disk=**fa****lse
    --symbolize_stacktrace=true
    --v=1
    --vmodule=
    I0617 16:10:41.533797 15390 mem-info.cc:66] Physical Memory: 3.66 GB
    I0617 16:10:41.539779 15390 daemon.cc:43] Cpu Info:
    Model: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
    Cores: 1
    L1 Cache: 32.00 KB
    L2 Cache: 256.00 KB
    L3 Cache: 20.00 MB
    Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    I0617 16:10:41.539821 15390 daemon.cc:44] Disk Info:
    Num disks 2: xvda, xvdb
    I0617 16:10:41.539855 15390 daemon.cc:45] Mem Info: 3.66 GB
    I0617 16:10:50.755198 15390 impala-server.cc:1809] Default query
    options:TQueryOptions {
    01: abort_on_error (bool) = false,
    02: max_errors (i32) = 0,
    03: disable_codegen (bool) = false,
    04: batch_size (i32) = 0,
    05: num_nodes (i32) = 0,
    06: max_scan_range_length (i64) = 0,
    07: num_scanner_threads (i32) = 0,
    08: max_io_buffers (i32) = 0,
    09: allow_unsupported_formats (bool) = false,
    10: default_order_by_limit (i64) = -1,
    11: debug_action (string) = "",
    12: mem_limit (i64) = 0,
    13: abort_on_default_limit_**exceede****d (bool) = false,
    }
    WARN0617 16:10:51.236000 main org.apache.hadoop.conf.**Configu****ration]
    mapred.max.split.size is deprecated. Instead, use mapreduce.input.*
    *fileinputforma****t.split.maxsize
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configu****ration]
    mapred.min.split.size is deprecated. Instead, use mapreduce.input.*
    *fileinputforma****t.split.minsize
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configu****ration]
    mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.
    **fileinputforma****t.split.minsize.**per.rack
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configu****ration]
    mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.
    **fileinputforma****t.split.minsize.**per.node
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configu****ration]
    mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
    WARN0617 16:10:51.245000 main org.apache.hadoop.conf.**Configu****ration]
    mapred.reduce.tasks.**speculativ****e.execution is deprecated.
    Instead, use mapreduce.reduce.speculative
    WARN0617 16:10:51.892000 main org.apache.hadoop.conf.**Configu****ration]
    org.apache.hadoop.hive.conf.**Lo****opingByteArrayInputStream@**
    66b5****1404:an attempt to override final parameter:
    mapreduce.job.end-**notification****.max.retry.**interval;
    Ignoring.
    WARN0617 16:10:51.954000 main org.apache.hadoop.conf.**Configu****ration]
    org.apache.hadoop.hive.conf.**Lo****opingByteArrayInputStream@**
    66b5****1404:an attempt to override final parameter:
    mapreduce.job.end-**notification****.max.attempts; Ignoring.
    WARN0617 16:10:51.972000 main org.apache.hadoop.hive.conf.**Hi****veConf]
    DEPRECATED: Configuration property hive.metastore.local no longer has any
    effect. Make sure to provide a valid value for hive.metastore.uris if you
    are connecting to a remote metastore.
    INFO0617 16:10:52.240000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-******
    west-2.compute.internal:9083
    INFO0617 16:10:52.494000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:10:53.495000 main hive.metastore] Connected to
    metastore.
    INFO0617 16:10:53.495000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-******
    west-2.compute.internal:9083
    INFO0617 16:10:53.497000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:10:54.498000 main hive.metastore] Connected to
    metastore.
    INFO0617 16:10:54.498000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-******
    west-2.compute.internal:9083
    INFO0617 16:10:54.505000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:10:55.506000 main hive.metastore] Connected to
    metastore.
    INFO0617 16:10:55.507000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-******
    west-2.compute.internal:9083
    INFO0617 16:10:55.515000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:10:56.515000 main hive.metastore] Connected to
    metastore.
    INFO0617 16:10:56.515000 main hive.metastore] Trying to connect to
    metastore with URI thrift://ip-10-245-112-238.us-******
    west-2.compute.internal:9083
    INFO0617 16:10:56.519000 main hive.metastore] Waiting 1 seconds
    before next connection attempt.
    INFO0617 16:10:57.519000 main hive.metastore] Connected to
    metastore.
    I0617 16:10:59.891072 15390 impala-server.cc:2074] Impala Beeswax
    Service listening on 21000
    I0617 16:10:59.893288 15390 impala-server.cc:2085] Impala
    HiveServer2 Service listening on 21050
    I0617 16:10:59.893335 15390 impala-server.cc:2093]
    ImpalaInternalService listening on 22000
    I0617 16:10:59.905498 15390 thrift-server.cc:365] ThriftServer
    'backend' started on port: 22000
    I0617 16:10:59.905547 15390 exec-env.cc:146] Starting global
    services
    I0617 16:10:59.905670 15390 exec-env.cc:167] Using global memory
    limit: 2.93 GB
    I0617 16:10:59.909502 15390 webserver.cc:129] Starting webserver on
    all interfaces, port 25000
    I0617 16:10:59.909559 15390 webserver.cc:139] Document root:
    /usr/lib/impala
    I0617 16:10:59.921815 15390 webserver.cc:178] Webserver started
    I0617 16:10:59.921897 15390 simple-scheduler.cc:99] Starting simple
    scheduler
    I0617 16:10:59.926578 15390 simple-scheduler.cc:133]
    Simple-scheduler using 10.245.112.238 as IP address
    I0617 16:10:59.926632 15390 state-store-subscriber.cc:137] Starting
    subscriber
    I0617 16:10:59.937484 15390 thrift-server.cc:365] ThriftServer
    'StateStoreSubscriber' started on port: 23000
    I0617 16:10:59.949283 15390 client-cache.cc:98] CreateClient():
    adding new client for ip-10-245-112-238.us-west-2.**co****
    mpute.internal:24000
    I0617 16:10:59.982250 15390 thrift-server.cc:365] ThriftServer
    'beeswax-frontend' started on port: 21000
    I0617 16:11:00.017025 15390 thrift-server.cc:365] ThriftServer
    'hiveServer2-frontend' started on port: 21050
    I0617 16:11:00.017144 15390 impalad-main.cc:101] Impala has started.

    On Monday, June 17, 2013 12:08:01 PM UTC-4, Ricky Saltzer wrote:

    Hey Alex -

    Sorry about that, I forgot that the log get's rolled over after a
    crash. Inside of */var/log/impalad*, you should be able to find
    the last log file written to before the crash using an "*ls -lthr*",
    take a peek inside..it should contain a lot more data.

    Thanks,
    Ricky

    On Mon, Jun 17, 2013 at 9:04 AM, Alex Minnaar wrote:

    I have attached what was in the impalad.INFO file

    Alex

    On Monday, June 17, 2013 11:44:10 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Try connecting to the node via SSH and grabbing the log
    directly from */var/log/impalad *
    *
    *
    Ricky


    On Mon, Jun 17, 2013 at 8:39 AM, Alex Minnaar <
    minna...@gmail.com> wrote:
    OK I figured out how to set the GLOG level, but when I try to
    download the impalad.INFO file I get

    Problem accessing /cmf/process/all/logs/**download********.
    Reason:

    Connection timed out

    Could not connect to host.

    On Monday, June 17, 2013 10:39:21 AM UTC-4, Ricky Saltzer wrote:

    Hi Alex -

    Could you set the GLOG level to 2<http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_logging.html> (you
    can use Cloudera manager to do it, too). Then, re-run your query and attach
    the impalad.INFO from the coordinator (the node you submit your query to).

    Ricky


    On Mon, Jun 17, 2013 at 6:23 AM, Alex Minnaar <
    minna...@gmail.com> wrote:
    Impala is working for me with simple select queries but it is
    not working with queries involving joins. When I execute the join query in
    the impala shell, I get the following result.
    Query finished, fetching results ...
    Error communicating with impalad: TSocket read 0 bytes

    And then I am no longer connected to impala. How do I fix
    this?

    Alex


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimpala-user @
categorieshadoop
postedJun 17, '13 at 1:23p
activeJun 17, '13 at 6:28p
posts15
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Alex Minnaar: 9 posts Ricky Saltzer: 6 posts

People

Translate

site design / logo © 2021 Grokbase