FAQ
Hi,

I Have one cluster running on impala 1.0 with 10 nodes.Queries without
joins run successfuly instead queries with joins failed at starting time,
impala-shell returns :
Backend 20:Couldn't open transport for NODE05:22000(Could not resolve host
for client socket.)
Backend 23:Couldn't open transport for NODE08:22000(Could not resolve host
for client socket.)
Backend 26:Couldn't open transport for NODE09:22000(Could not resolve host
for client socket.)

The same query is OK on Hive. It's very strange considering queries without
joins are OK so network config also.

But I have added recently 6 nodes to the 4 legacy nodes. Tables joined are
on the hdfs 4 legacy nodes and my large table is on all hdfs cluster.
strangely, error message refers only to the 6 new nodes. Is that a
metastore or metadata problem ? every deamons (mysql, hive, impala) and
refresh are OK.


Thanks for you help,

Franck

Search Discussions

  • Franckgallos at May 16, 2013 at 10:30 am
    I have re-created tables join on entire cluster (4 legacy nodes + 6 new
    nodes) in other db typically for debug, this issue seems (workaround)
    solved. Query is running, I'm waiting for results.
    In this case, if one node crashes, when replacing it, we'll have to
    re-write all data with regard to the node ?? It's big metadata problem, I'm
    concerned about that issue in production stage.

    Thanks for your response.
    (sorry for my poor english)


    Franck


    Le jeudi 16 mai 2013 11:03:55 UTC+2, Franck Gallos a écrit :
    Hi,

    I Have one cluster running on impala 1.0 with 10 nodes.Queries without
    joins run successfuly instead queries with joins failed at starting time,
    impala-shell returns :
    Backend 20:Couldn't open transport for NODE05:22000(Could not resolve host
    for client socket.)
    Backend 23:Couldn't open transport for NODE08:22000(Could not resolve host
    for client socket.)
    Backend 26:Couldn't open transport for NODE09:22000(Could not resolve host
    for client socket.)

    The same query is OK on Hive. It's very strange considering queries
    without joins are OK so network config also.

    But I have added recently 6 nodes to the 4 legacy nodes. Tables joined are
    on the hdfs 4 legacy nodes and my large table is on all hdfs cluster.
    strangely, error message refers only to the 6 new nodes. Is that a
    metastore or metadata problem ? every deamons (mysql, hive, impala) and
    refresh are OK.


    Thanks for you help,

    Franck
  • Henry Robinson at May 16, 2013 at 3:32 pm
    Hi Franck -

    If an Impala node crashes, the state-store should detect it, and cancel any
    queries that are running on that node. The next time you issue a query, it
    should ignore the crashed node and work correctly (although most likely a
    little slower).

    The error that you are seeing ("Could not resolve host for client socket")
    suggests that your new Impala daemons are advertising themselves using a
    hostname that the legacy daemons cannot resolve. What command line are you
    using to start the 6 new nodes? Are they really on machines whose hostname
    is NODE05, NODE08 etc?

    Best,
    Henry

    On 16 May 2013 03:30, wrote:

    I have re-created tables join on entire cluster (4 legacy nodes + 6 new
    nodes) in other db typically for debug, this issue seems (workaround)
    solved. Query is running, I'm waiting for results.
    In this case, if one node crashes, when replacing it, we'll have to
    re-write all data with regard to the node ?? It's big metadata problem, I'm
    concerned about that issue in production stage.

    Thanks for your response.
    (sorry for my poor english)


    Franck


    Le jeudi 16 mai 2013 11:03:55 UTC+2, Franck Gallos a écrit :
    Hi,

    I Have one cluster running on impala 1.0 with 10 nodes.Queries without
    joins run successfuly instead queries with joins failed at starting time,
    impala-shell returns :
    Backend 20:Couldn't open transport for NODE05:22000(Could not resolve
    host for client socket.)
    Backend 23:Couldn't open transport for NODE08:22000(Could not resolve
    host for client socket.)
    Backend 26:Couldn't open transport for NODE09:22000(Could not resolve
    host for client socket.)

    The same query is OK on Hive. It's very strange considering queries
    without joins are OK so network config also.

    But I have added recently 6 nodes to the 4 legacy nodes. Tables joined
    are on the hdfs 4 legacy nodes and my large table is on all hdfs cluster.
    strangely, error message refers only to the 6 new nodes. Is that a
    metastore or metadata problem ? every deamons (mysql, hive, impala) and
    refresh are OK.


    Thanks for you help,

    Franck

    --
    Henry Robinson
    Software Engineer
    Cloudera
    415-994-6679
  • Franck Gallos at May 17, 2013 at 1:51 pm
    Hi Henry,

    As described in my previous mail,I found a workaround without solving the
    issue.

    The steps are described below :

    My cluster is made of legacy nodes (called NODE01 to 04) to which I added 6
    new nodes (called NODE05 to 10).


    ** First,* 3 tables were was deployed only on the 4 legacy nodes and a
    large data table is deployed on all the 10 nodes, because I loaded the
    large data table after having added the 6 new nodes. When I query the large
    data table with jointing the 3 tables, the query is quickly aborted at the
    beginning of the process and not executed, making impossible to fetch
    results.

    Query aborted,unable to fetch results

    Backend 20:Couldn't open transport for NODE05:22000(Could not resolve host
    for client socket.)
    Backend 23:Couldn't open transport for NODE08:22000(Could not resolve host
    for client socket.)
    Backend 26:Couldn't open transport for NODE09:22000(Could not resolve host
    for client socket.)



      ** Next,* I recreated the 3 tables on the whole cluster (4 legacy nodes +
    6 new nodes) in other schema typically for debug . this issue seems
    workarounded. This time, the query is executed, but fails with the same
    error a few minutes later. In both cases, the error is the same.

    Query finished, fetching results ...
    ERROR: Couldn't open transport for USR-P8HDS07:22000(Could not resolve host
    for client socket.)
    ERROR: Invalid query handle

    In both cases, the logs show :
    E0516 15:16:45.759006 7069 impala-server.cc:1681] unknown query id:
    51e46af512fc4d63:878eaac359809d8b
    E0516 15:16:45.760674 925 impala-server.cc:1681] unknown query id:
    51e46af512fc4d63:878eaac359809d8b
    E0516 15:16:45.762706 917 impala-server.cc:1681] unknown query id:
    51e46af512fc4d63:878eaac359809d8b

    → Two questions :

    1) why do the metadata look seem wrong in the first case, failing by
    adressing nodes which haven't the tables ?

    2) In the second case, it seems to me that it's not a network problem : the
    query runs fine on hive, and a query without any joint also runs on impala.
    I have read in similar issues that maybe a memory tuning problem or other ?

    Best regards

    Franck



    2013/5/16 Henry Robinson <henry@cloudera.com>
    Hi Franck -

    If an Impala node crashes, the state-store should detect it, and cancel
    any queries that are running on that node. The next time you issue a query,
    it should ignore the crashed node and work correctly (although most likely
    a little slower).

    The error that you are seeing ("Could not resolve host for client socket")
    suggests that your new Impala daemons are advertising themselves using a
    hostname that the legacy daemons cannot resolve. What command line are you
    using to start the 6 new nodes? Are they really on machines whose hostname
    is NODE05, NODE08 etc?

    Best,
    Henry

    On 16 May 2013 03:30, wrote:

    I have re-created tables join on entire cluster (4 legacy nodes + 6 new
    nodes) in other db typically for debug, this issue seems (workaround)
    solved. Query is running, I'm waiting for results.
    In this case, if one node crashes, when replacing it, we'll have to
    re-write all data with regard to the node ?? It's big metadata problem, I'm
    concerned about that issue in production stage.

    Thanks for your response.
    (sorry for my poor english)


    Franck


    Le jeudi 16 mai 2013 11:03:55 UTC+2, Franck Gallos a écrit :
    Hi,

    I Have one cluster running on impala 1.0 with 10 nodes.Queries without
    joins run successfuly instead queries with joins failed at starting time,
    impala-shell returns :
    Backend 20:Couldn't open transport for NODE05:22000(Could not resolve
    host for client socket.)
    Backend 23:Couldn't open transport for NODE08:22000(Could not resolve
    host for client socket.)
    Backend 26:Couldn't open transport for NODE09:22000(Could not resolve
    host for client socket.)

    The same query is OK on Hive. It's very strange considering queries
    without joins are OK so network config also.

    But I have added recently 6 nodes to the 4 legacy nodes. Tables joined
    are on the hdfs 4 legacy nodes and my large table is on all hdfs cluster.
    strangely, error message refers only to the 6 new nodes. Is that a
    metastore or metadata problem ? every deamons (mysql, hive, impala) and
    refresh are OK.


    Thanks for you help,

    Franck

    --
    Henry Robinson
    Software Engineer
    Cloudera
    415-994-6679
  • Henry Robinson at May 20, 2013 at 7:16 pm
    Hi Franck -

    This error "Could not resolve host for client socket." means that Impala is
    unable to resolve the hostname for a remote daemon. That means there's some
    kind of configuration issue - either Impala is seeing the wrong hostnames
    for the backends, or the machines that Impala is running on are not
    configured to resolve the hostnames correctly.

    Is USR-P8HDS07 supposed to be in your cluster? Can you ping it from the
    Impala machine that you sent your query to?

    Henry

    On 17 May 2013 06:51, Franck Gallos wrote:



    Hi Henry,

    As described in my previous mail,I found a workaround without solving the
    issue.

    The steps are described below :

    My cluster is made of legacy nodes (called NODE01 to 04) to which I added
    6 new nodes (called NODE05 to 10).


    ** First,* 3 tables were was deployed only on the 4 legacy nodes and a
    large data table is deployed on all the 10 nodes, because I loaded the
    large data table after having added the 6 new nodes. When I query the large
    data table with jointing the 3 tables, the query is quickly aborted at the
    beginning of the process and not executed, making impossible to fetch
    results.

    Query aborted,unable to fetch results

    Backend 20:Couldn't open transport for NODE05:22000(Could not resolve host
    for client socket.)
    Backend 23:Couldn't open transport for NODE08:22000(Could not resolve host
    for client socket.)
    Backend 26:Couldn't open transport for NODE09:22000(Could not resolve host
    for client socket.)



    ** Next,* I recreated the 3 tables on the whole cluster (4 legacy nodes
    + 6 new nodes) in other schema typically for debug . this issue seems
    workarounded. This time, the query is executed, but fails with the same
    error a few minutes later. In both cases, the error is the same.

    Query finished, fetching results ...
    ERROR: Couldn't open transport for USR-P8HDS07:22000(Could not resolve
    host for client socket.)
    ERROR: Invalid query handle

    In both cases, the logs show :
    E0516 15:16:45.759006 7069 impala-server.cc:1681] unknown query id:
    51e46af512fc4d63:878eaac359809d8b
    E0516 15:16:45.760674 925 impala-server.cc:1681] unknown query id:
    51e46af512fc4d63:878eaac359809d8b
    E0516 15:16:45.762706 917 impala-server.cc:1681] unknown query id:
    51e46af512fc4d63:878eaac359809d8b

    → Two questions :

    1) why do the metadata look seem wrong in the first case, failing by
    adressing nodes which haven't the tables ?

    2) In the second case, it seems to me that it's not a network problem :
    the query runs fine on hive, and a query without any joint also runs on
    impala. I have read in similar issues that maybe a memory tuning problem or
    other ?

    Best regards

    Franck



    2013/5/16 Henry Robinson <henry@cloudera.com>
    Hi Franck -

    If an Impala node crashes, the state-store should detect it, and cancel
    any queries that are running on that node. The next time you issue a query,
    it should ignore the crashed node and work correctly (although most likely
    a little slower).

    The error that you are seeing ("Could not resolve host for client
    socket") suggests that your new Impala daemons are advertising themselves
    using a hostname that the legacy daemons cannot resolve. What command line
    are you using to start the 6 new nodes? Are they really on machines whose
    hostname is NODE05, NODE08 etc?

    Best,
    Henry

    On 16 May 2013 03:30, wrote:

    I have re-created tables join on entire cluster (4 legacy nodes + 6 new
    nodes) in other db typically for debug, this issue seems (workaround)
    solved. Query is running, I'm waiting for results.
    In this case, if one node crashes, when replacing it, we'll have to
    re-write all data with regard to the node ?? It's big metadata problem, I'm
    concerned about that issue in production stage.

    Thanks for your response.
    (sorry for my poor english)


    Franck


    Le jeudi 16 mai 2013 11:03:55 UTC+2, Franck Gallos a écrit :
    Hi,

    I Have one cluster running on impala 1.0 with 10 nodes.Queries without
    joins run successfuly instead queries with joins failed at starting time,
    impala-shell returns :
    Backend 20:Couldn't open transport for NODE05:22000(Could not resolve
    host for client socket.)
    Backend 23:Couldn't open transport for NODE08:22000(Could not resolve
    host for client socket.)
    Backend 26:Couldn't open transport for NODE09:22000(Could not resolve
    host for client socket.)

    The same query is OK on Hive. It's very strange considering queries
    without joins are OK so network config also.

    But I have added recently 6 nodes to the 4 legacy nodes. Tables joined
    are on the hdfs 4 legacy nodes and my large table is on all hdfs cluster.
    strangely, error message refers only to the 6 new nodes. Is that a
    metastore or metadata problem ? every deamons (mysql, hive, impala) and
    refresh are OK.


    Thanks for you help,

    Franck

    --
    Henry Robinson
    Software Engineer
    Cloudera
    415-994-6679

    --
    Henry Robinson
    Software Engineer
    Cloudera
    415-994-6679
  • Franck Gallos at May 23, 2013 at 11:28 am
    Hi Henry,

    You're right. The problem was a bad configuring /etc/hosts on 1 node.
    Now, impala works fine.

    Thanks for your insistence and assistance

    Franck


    2013/5/20 Henry Robinson <henry@cloudera.com>
    Hi Franck -

    This error "Could not resolve host for client socket." means that Impala
    is unable to resolve the hostname for a remote daemon. That means there's
    some kind of configuration issue - either Impala is seeing the wrong
    hostnames for the backends, or the machines that Impala is running on are
    not configured to resolve the hostnames correctly.

    Is USR-P8HDS07 supposed to be in your cluster? Can you ping it from the
    Impala machine that you sent your query to?

    Henry

    On 17 May 2013 06:51, Franck Gallos wrote:



    Hi Henry,

    As described in my previous mail,I found a workaround without solving the
    issue.

    The steps are described below :

    My cluster is made of legacy nodes (called NODE01 to 04) to which I added
    6 new nodes (called NODE05 to 10).


    ** First,* 3 tables were was deployed only on the 4 legacy nodes and a
    large data table is deployed on all the 10 nodes, because I loaded the
    large data table after having added the 6 new nodes. When I query the large
    data table with jointing the 3 tables, the query is quickly aborted at the
    beginning of the process and not executed, making impossible to fetch
    results.

    Query aborted,unable to fetch results

    Backend 20:Couldn't open transport for NODE05:22000(Could not resolve
    host for client socket.)
    Backend 23:Couldn't open transport for NODE08:22000(Could not resolve
    host for client socket.)
    Backend 26:Couldn't open transport for NODE09:22000(Could not resolve
    host for client socket.)



    ** Next,* I recreated the 3 tables on the whole cluster (4 legacy nodes
    + 6 new nodes) in other schema typically for debug . this issue seems
    workarounded. This time, the query is executed, but fails with the same
    error a few minutes later. In both cases, the error is the same.

    Query finished, fetching results ...
    ERROR: Couldn't open transport for USR-P8HDS07:22000(Could not resolve
    host for client socket.)
    ERROR: Invalid query handle

    In both cases, the logs show :
    E0516 15:16:45.759006 7069 impala-server.cc:1681] unknown query id:
    51e46af512fc4d63:878eaac359809d8b
    E0516 15:16:45.760674 925 impala-server.cc:1681] unknown query id:
    51e46af512fc4d63:878eaac359809d8b
    E0516 15:16:45.762706 917 impala-server.cc:1681] unknown query id:
    51e46af512fc4d63:878eaac359809d8b

    → Two questions :

    1) why do the metadata look seem wrong in the first case, failing by
    adressing nodes which haven't the tables ?

    2) In the second case, it seems to me that it's not a network problem :
    the query runs fine on hive, and a query without any joint also runs on
    impala. I have read in similar issues that maybe a memory tuning problem or
    other ?

    Best regards

    Franck



    2013/5/16 Henry Robinson <henry@cloudera.com>
    Hi Franck -

    If an Impala node crashes, the state-store should detect it, and cancel
    any queries that are running on that node. The next time you issue a query,
    it should ignore the crashed node and work correctly (although most likely
    a little slower).

    The error that you are seeing ("Could not resolve host for client
    socket") suggests that your new Impala daemons are advertising themselves
    using a hostname that the legacy daemons cannot resolve. What command line
    are you using to start the 6 new nodes? Are they really on machines whose
    hostname is NODE05, NODE08 etc?

    Best,
    Henry

    On 16 May 2013 03:30, wrote:

    I have re-created tables join on entire cluster (4 legacy nodes + 6 new
    nodes) in other db typically for debug, this issue seems (workaround)
    solved. Query is running, I'm waiting for results.
    In this case, if one node crashes, when replacing it, we'll have to
    re-write all data with regard to the node ?? It's big metadata problem, I'm
    concerned about that issue in production stage.

    Thanks for your response.
    (sorry for my poor english)


    Franck


    Le jeudi 16 mai 2013 11:03:55 UTC+2, Franck Gallos a écrit :
    Hi,

    I Have one cluster running on impala 1.0 with 10 nodes.Queries without
    joins run successfuly instead queries with joins failed at starting time,
    impala-shell returns :
    Backend 20:Couldn't open transport for NODE05:22000(Could not resolve
    host for client socket.)
    Backend 23:Couldn't open transport for NODE08:22000(Could not resolve
    host for client socket.)
    Backend 26:Couldn't open transport for NODE09:22000(Could not resolve
    host for client socket.)

    The same query is OK on Hive. It's very strange considering queries
    without joins are OK so network config also.

    But I have added recently 6 nodes to the 4 legacy nodes. Tables joined
    are on the hdfs 4 legacy nodes and my large table is on all hdfs cluster.
    strangely, error message refers only to the 6 new nodes. Is that a
    metastore or metadata problem ? every deamons (mysql, hive, impala) and
    refresh are OK.


    Thanks for you help,

    Franck

    --
    Henry Robinson
    Software Engineer
    Cloudera
    415-994-6679

    --
    Henry Robinson
    Software Engineer
    Cloudera
    415-994-6679
  • Henry Robinson at May 23, 2013 at 4:16 pm
    Glad to hear you've got it working - let us know if we can help any further.

    Best,
    Henry

    On 23 May 2013 04:28, Franck Gallos wrote:

    Hi Henry,

    You're right. The problem was a bad configuring /etc/hosts on 1 node.
    Now, impala works fine.

    Thanks for your insistence and assistance

    Franck


    2013/5/20 Henry Robinson <henry@cloudera.com>
    Hi Franck -

    This error "Could not resolve host for client socket." means that Impala
    is unable to resolve the hostname for a remote daemon. That means there's
    some kind of configuration issue - either Impala is seeing the wrong
    hostnames for the backends, or the machines that Impala is running on are
    not configured to resolve the hostnames correctly.

    Is USR-P8HDS07 supposed to be in your cluster? Can you ping it from the
    Impala machine that you sent your query to?

    Henry

    On 17 May 2013 06:51, Franck Gallos wrote:



    Hi Henry,

    As described in my previous mail,I found a workaround without solving
    the issue.

    The steps are described below :

    My cluster is made of legacy nodes (called NODE01 to 04) to which I
    added 6 new nodes (called NODE05 to 10).


    ** First,* 3 tables were was deployed only on the 4 legacy nodes and a
    large data table is deployed on all the 10 nodes, because I loaded the
    large data table after having added the 6 new nodes. When I query the large
    data table with jointing the 3 tables, the query is quickly aborted at the
    beginning of the process and not executed, making impossible to fetch
    results.

    Query aborted,unable to fetch results

    Backend 20:Couldn't open transport for NODE05:22000(Could not resolve
    host for client socket.)
    Backend 23:Couldn't open transport for NODE08:22000(Could not resolve
    host for client socket.)
    Backend 26:Couldn't open transport for NODE09:22000(Could not resolve
    host for client socket.)



    ** Next,* I recreated the 3 tables on the whole cluster (4 legacy
    nodes + 6 new nodes) in other schema typically for debug . this issue seems
    workarounded. This time, the query is executed, but fails with the same
    error a few minutes later. In both cases, the error is the same.

    Query finished, fetching results ...
    ERROR: Couldn't open transport for USR-P8HDS07:22000(Could not resolve
    host for client socket.)
    ERROR: Invalid query handle

    In both cases, the logs show :
    E0516 15:16:45.759006 7069 impala-server.cc:1681] unknown query id:
    51e46af512fc4d63:878eaac359809d8b
    E0516 15:16:45.760674 925 impala-server.cc:1681] unknown query id:
    51e46af512fc4d63:878eaac359809d8b
    E0516 15:16:45.762706 917 impala-server.cc:1681] unknown query id:
    51e46af512fc4d63:878eaac359809d8b

    → Two questions :

    1) why do the metadata look seem wrong in the first case, failing by
    adressing nodes which haven't the tables ?

    2) In the second case, it seems to me that it's not a network problem :
    the query runs fine on hive, and a query without any joint also runs on
    impala. I have read in similar issues that maybe a memory tuning problem or
    other ?

    Best regards

    Franck



    2013/5/16 Henry Robinson <henry@cloudera.com>
    Hi Franck -

    If an Impala node crashes, the state-store should detect it, and cancel
    any queries that are running on that node. The next time you issue a query,
    it should ignore the crashed node and work correctly (although most likely
    a little slower).

    The error that you are seeing ("Could not resolve host for client
    socket") suggests that your new Impala daemons are advertising themselves
    using a hostname that the legacy daemons cannot resolve. What command line
    are you using to start the 6 new nodes? Are they really on machines whose
    hostname is NODE05, NODE08 etc?

    Best,
    Henry

    On 16 May 2013 03:30, wrote:

    I have re-created tables join on entire cluster (4 legacy nodes + 6
    new nodes) in other db typically for debug, this issue seems (workaround)
    solved. Query is running, I'm waiting for results.
    In this case, if one node crashes, when replacing it, we'll have to
    re-write all data with regard to the node ?? It's big metadata problem, I'm
    concerned about that issue in production stage.

    Thanks for your response.
    (sorry for my poor english)


    Franck


    Le jeudi 16 mai 2013 11:03:55 UTC+2, Franck Gallos a écrit :
    Hi,

    I Have one cluster running on impala 1.0 with 10 nodes.Queries
    without joins run successfuly instead queries with joins failed at starting
    time, impala-shell returns :
    Backend 20:Couldn't open transport for NODE05:22000(Could not resolve
    host for client socket.)
    Backend 23:Couldn't open transport for NODE08:22000(Could not resolve
    host for client socket.)
    Backend 26:Couldn't open transport for NODE09:22000(Could not resolve
    host for client socket.)

    The same query is OK on Hive. It's very strange considering queries
    without joins are OK so network config also.

    But I have added recently 6 nodes to the 4 legacy nodes. Tables
    joined are on the hdfs 4 legacy nodes and my large table is on all hdfs
    cluster. strangely, error message refers only to the 6 new nodes. Is that a
    metastore or metadata problem ? every deamons (mysql, hive, impala) and
    refresh are OK.


    Thanks for you help,

    Franck

    --
    Henry Robinson
    Software Engineer
    Cloudera
    415-994-6679

    --
    Henry Robinson
    Software Engineer
    Cloudera
    415-994-6679

    --
    Henry Robinson
    Software Engineer
    Cloudera
    415-994-6679

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimpala-user @
categorieshadoop
postedMay 16, '13 at 9:04a
activeMay 23, '13 at 4:16p
posts7
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Franck Gallos: 4 posts Henry Robinson: 3 posts

People

Translate

site design / logo © 2021 Grokbase