Re: Error communicating with impalad: TSocket read 0 bytes - impala-0.7-1

I meet the same problem.
  Ubuntu 12.04.2 LTS




2013/4/26 Lenni Kuff <lskuff@cloudera.com>

Impala download from
http://beta.cloudera.com/impala/ubuntu/precise/amd64/impala/pool/contrib/i/impala/



2013/4/26 宾莉金 <binlijin@gmail.com>

The core dump is


warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by
`/home/tianzhao.blj/cloudera/impala/be/build/debug/service/impalad
-state_store_'.
Program terminated with signal 6, Aborted.
#0 0x00007f35fd6fa425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007f35fd6fa425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f35fd6fdb8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f35fca8fa37 in os::abort(bool) () from
/usr/lib/jvm/java-6-oracle/jre/lib/amd64/server/libjvm.so
#3 0x00007f35fcbe3f28 in VMError::report_and_die() () from
/usr/lib/jvm/java-6-oracle/jre/lib/amd64/server/libjvm.so
#4 0x00007f35fca963f5 in JVM_handle_linux_signal () from
/usr/lib/jvm/java-6-oracle/jre/lib/amd64/server/libjvm.so
#5 0x00007f35fca926fe in signalHandler(int, siginfo*, void*) () from
/usr/lib/jvm/java-6-oracle/jre/lib/amd64/server/libjvm.so
#6 <signal handler called>
#7 impala::HdfsScanNode::UpdateCounters (this=0x37e6800) at
/home/tianzhao.blj/cloudera/impala/be/src/exec/hdfs-scan-node.cc:842
#8 0x0000000000e5b9cc in impala::HdfsScanNode::GetNext (this=0x37e6800,
state=0x51cca80, row_batch=0x45f4a00, eos=0x51c8171)
     at /home/tianzhao.blj/cloudera/impala/be/src/exec/hdfs-scan-node.cc:162
#9 0x0000000000d0af2d in impala::PlanFragmentExecutor::GetNextInternal
(this=0x51c8038, batch=0x7f35de6203b8)
     at
/home/tianzhao.blj/cloudera/impala/be/src/runtime/plan-fragment-executor.cc:377
#10 0x0000000000d09b0a in impala::PlanFragmentExecutor::OpenInternal
(this=0x51c8038)
     at
/home/tianzhao.blj/cloudera/impala/be/src/runtime/plan-fragment-executor.cc:249
#11 0x0000000000d09885 in impala::PlanFragmentExecutor::Open
(this=0x51c8038)
     at
/home/tianzhao.blj/cloudera/impala/be/src/runtime/plan-fragment-executor.cc:226
#12 0x0000000000b323a3 in impala::ImpalaServer::FragmentExecState::Exec
(this=0x51c8000)
     at
/home/tianzhao.blj/cloudera/impala/be/src/service/impala-server.cc:470
#13 0x0000000000b45c67 in impala::ImpalaServer::RunExecPlanFragment
(this=0x3cddc00, exec_state=0x51c8000)
     at
/home/tianzhao.blj/cloudera/impala/be/src/service/impala-server.cc:1692
#14 0x0000000000bad05f in boost::_mfi::mf1<void, impala::ImpalaServer,
impala::ImpalaServer::FragmentExecState*>::operator() (
     this=0x2b1f308, p=0x3cddc00, a1=0x51c8000) at
/usr/include/boost/bind/mem_fn_template.hpp:165
#15 0x0000000000bac741 in
boost::_bi::list2<boost::_bi::value<impala::ImpalaServer*>,
boost::_bi::value<impala::ImpalaServer::FragmentExecState*>
impala::ImpalaServer::FragmentExecState*>, boost::_bi::list0>
(this=0x2b1f318, f=..., a=...) at /usr/include/boost/bind/bind.hpp:313
#16 0x0000000000bab717 in boost::_bi::bind_t<void, boost::_mfi::mf1<void,
impala::ImpalaServer, impala::ImpalaServer::FragmentExecState*>,
boost::_bi::list2<boost::_bi::value<impala::ImpalaServer*>,
boost::_bi::value<impala::ImpalaServer::FragmentExecState*> > >::operator()
(this=0x2b1f308) at /usr/include/boost/bind/bind_template.hpp:20
#17 0x0000000000baa48a in
boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf1<void,
impala::ImpalaServer, impala::ImpalaServer::FragmentExecState*>,
boost::_bi::list2<boost::_bi::value<impala::ImpalaServer*>,
boost::_bi::value<impala::ImpalaServer::FragmentExecState*> > > >::run
(this=0x2b1f180) at /usr/include/boost/thread/detail/thread.hpp:61
#18 0x00007f35ff62ace9 in thread_proxy () from
/usr/lib/libboost_thread.so.1.46.1
#19 0x00007f35ff408e9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#20 0x00007f35fd7b7ccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#21 0x0000000000000000 in ?? ()



java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)





2013/4/26 宾莉金 <binlijin@gmail.com>

The code not in https://github.com/cloudera/impala?
I use the code from https://github.com/cloudera/impala.git has the same
problem.



2013/4/27 Skye Wanderman-Milne <skye@cloudera.com>

@Nong Li : thanx for sharing the issue, that is really helpful to know what
was causing that error.
@Lenni : Its hard for me to look into the C code as it is not my forte,
Java is. Only option left with me is to wait for new release :).

Thanks,
*Rahul Mathur*

So, started a brand new installed using CDH 4.5.2

*Version*: 4.5.2 (#327 built by jenkins on 20130429-1453 git:
16cab2c7b76194b7877d64a4215494daa387a266)
Impala Shell v1.0 (d1bf0d1) built on Sun Apr 28 15:33:52 PDT 2013

Using IMPALA 1.0-1.p0.371 parcel

    - MR jobs work
    - HiveQL works (for same queries stated below)
    - Impala
       - Simple select works (in hive too)
       - Queries with group fails

Query: select search_word, count(*) from cs group by search_word

Error communicating with impalad: TSocket read 0 bytes



Anybody spot what I may be doing wrong? Missing any config? I couldn't find
anything useful in /var/logs/impala/impalad.INFO
What else can I provide?

thanks, nitin

Hi Nitin,
Could you please provide the impalad logs so we can help investigate this
problem? It would also be useful to have "describe extended" output of your
target table. You can get this by running:
Thanks,
Lenni
Software Engineer - Cloudera

Hive output

Logging initialized using configuration in
Table creation:
Logs I found in /var/logs/impalad are here - http://pastebin.com/BgtnRq7Y

thanks, nitin

Thanks for the information. I don't see anything immediately obvious from
the logs of the table definition. In your browser could you go to:
http://<hostname>:25000 on one of the machines running Impalad? You will
see a "Hardware Info" box, could you send that output back?

Thanks,
Lenni

Here is the output of :25000; however I did realize that the port is not
open externally. Although, I don't think the "debug web server" would have
to be open for all, right?

impalad version 1.0 RELEASE (build d1bf0d1dac339af3692ffa17a5e3fdae0aed751f)
Built on Sun, 28 Apr 2013 15:59:46 PST

Hardware Info

Cpu Info:
   Model: Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
   Cores: 32
   L1 Cache: 0.00
   L2 Cache: 0.00
   L3 Cache: 0.00
   Hardware Supports:
     ssse3
     sse4_1
     sse4_2
     popcnt
Mem Info: 252.01 GB
Disk Info:
   Num disks 12: cciss/c0d, cciss/c0d0p, sda, sdb, sdc, sdd, dm-, VxDMP, VxDMP2p, VxDMP3p, VxDMP4p, VxVM

Hi lskuff,

     I face the same issue on cloudera impala 1.1. I 'm runnning "set
disable_codegen=true" impalad did not died.

I have generaged core from impalad , If you need to find out root cause.


[core file ]
http://www.boyr.com/getfile.php?id=1402049&key=51fb6cc71b178 (plase
rename to impala.tar.gz)

or

http://www.speedyshare.com/a5UdG/impala.tar.gz

hive> describe extended proxy_snappy_seq;
OK
date_time string
category string
msisdn string
client_ip string
http_code string
content_length int
method string
domain string
dest_port string
uri string
content_type string
uaprofile string
dest_ip string
server_ip string

Detailed Table Information Table(tableName:proxy_snappy_seq,
dbName:ais, owner:root, createTime:1375430008, lastAccessTime:0,
retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:date_time,
type:string, comment:null), FieldSchema(name:category, type:string,
comment:null), FieldSchema(name:msisdn, type:string, comment:null),
FieldSchema(name:client_ip, type:string, comment:null),
FieldSchema(name:http_code, type:string, comment:null),
FieldSchema(name:content_length, type:int, comment:null),
FieldSchema(name:method, type:string, comment:null),
FieldSchema(name:domain, type:string, comment:null),
FieldSchema(name:dest_port, type:string, comment:null),
FieldSchema(name:uri, type:string, comment:null),
FieldSchema(name:content_type, type:string, comment:null),
FieldSchema(name:uaprofile, type:string, comment:null),
FieldSchema(name:dest_ip, type:string, comment:null),
FieldSchema(name:server_ip, type:string, comment:null)],
location:hdfs://hadoop.g-able.com:8020/user/hive/warehouse/ais.db/proxy_snappy_seq,
inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
parameters:{serialization.format=|, line.delim=
, field.delim=|}), bucketCols:[], sortCols:[], parameters:{},
skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
skewedColValueLocationMaps:{}), storedAsSubDirectories:false),
partitionKeys:[], parameters:{numPartitions=0, numFiles=18,
transient_lastDdlTime=1375430448, totalSize=5101757765, numRows=0,
rawDataSize=0}, viewOriginalText:null, viewExpandedText:null,
tableType:MANAGED_TABLE)

impala-shell> select msisdn from proxy_snappy_seq group by msisdn limit 10;
Query: select msisdn from proxy_snappy_seq group by msisdn limit 10
Query finished, fetching results ...
Error communicating with impalad: TSocket read 0 bytes
Cancelling query ...
Not connected (use CONNECT to establish a connection)
[Not connected] >


Regards,
Screen

More information,

   I use RHEL 5.8. I found some case like this use RHEL 5.9. May be RH
version has to concern.

Hi,
We have identified and fixed the problem (see
https://issues.cloudera.org/browse/IMPALA-477) and the fix will be
available in the next Impala release. It is a problem related to codegen
and RHEL5, so until the next release goes out the workarounds are:

1) Disable codegen
2) Upgrade to RHEL6

We understand that neither of these options are ideal and will prioritize
getting a fix available as soon as possible.

Thanks,
Lenni