FAQ
Hi all,

I am loading data into hive tables by connecting to hiveserver through
thrift api using "load data local inpath ... " query .
Hive server is running as a background process for days . After some 4-5
days hiveserver throws this error :

11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error occurred
during acceptance of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException:
Too many open files
at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at
org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184)
at
org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
... 9 more

I checked network connections using netstat and it show about 1000 active
network connections, most of them to hadoop datanode . Output is attached as
netstat.out

after i restarted hiveserver network connections falls to about 20 . output
is attached as netstat_after_HS_restart.out

Is this a problem with hive or hadoop configuration ?

Vaibhav Negi

Search Discussions

  • Ajo Fod at Feb 11, 2011 at 3:07 pm
    Are you using hive 0.6? ... may be fixed in the latest version.

    Also I wonder why these thrift libraries are being used ... is this normal
    hive operation, or can you do something to avoid using thrift?

    -Ajo
    On Fri, Feb 11, 2011 at 12:05 AM, vaibhav negi wrote:


    Hi all,

    I am loading data into hive tables by connecting to hiveserver through
    thrift api using "load data local inpath ... " query .
    Hive server is running as a background process for days . After some 4-5
    days hiveserver throws this error :

    11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error occurred
    during acceptance of message.
    org.apache.thrift.transport.TTransportException: java.net.SocketException:
    Too many open files
    at
    org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
    at
    org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
    at
    org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
    at
    org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184)
    at
    org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: java.net.SocketException: Too many open files
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    at java.net.ServerSocket.implAccept(ServerSocket.java:453)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at
    org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
    ... 9 more

    I checked network connections using netstat and it show about 1000 active
    network connections, most of them to hadoop datanode . Output is attached as
    netstat.out

    after i restarted hiveserver network connections falls to about 20 . output
    is attached as netstat_after_HS_restart.out

    Is this a problem with hive or hadoop configuration ?

    Vaibhav Negi
  • Viral Bajaria at Feb 12, 2011 at 12:37 am
    We use hive through the thrift api too. I have seen the above error when
    someone accidentally connects to the thrift server port through multiple
    instances of our hive-wrapper code. We open a bunch of hive thrift ports to
    achieve the automation of running multiple hive queries in parallel i.e.
    continuous aggregations.

    you need to make sure you are not facing a similar issue because the hive
    thrift server as far as i know is single threaded up until hive 0.5.0 (the
    version that i use). Not too sure if that's been changed in 0.6.0 or higher.

    -Viral
    On Fri, Feb 11, 2011 at 7:06 AM, Ajo Fod wrote:

    Are you using hive 0.6? ... may be fixed in the latest version.

    Also I wonder why these thrift libraries are being used ... is this normal
    hive operation, or can you do something to avoid using thrift?

    -Ajo

    On Fri, Feb 11, 2011 at 12:05 AM, vaibhav negi wrote:


    Hi all,

    I am loading data into hive tables by connecting to hiveserver through
    thrift api using "load data local inpath ... " query .
    Hive server is running as a background process for days . After some 4-5
    days hiveserver throws this error :

    11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error occurred
    during acceptance of message.
    org.apache.thrift.transport.TTransportException: java.net.SocketException:
    Too many open files
    at
    org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
    at
    org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
    at
    org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
    at
    org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184)
    at
    org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: java.net.SocketException: Too many open files
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    at java.net.ServerSocket.implAccept(ServerSocket.java:453)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at
    org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
    ... 9 more

    I checked network connections using netstat and it show about 1000 active
    network connections, most of them to hadoop datanode . Output is attached as
    netstat.out

    after i restarted hiveserver network connections falls to about 20 .
    output is attached as netstat_after_HS_restart.out

    Is this a problem with hive or hadoop configuration ?

    Vaibhav Negi
  • Vaibhav negi at Feb 15, 2011 at 9:24 am
    Hi ,

    I am using hive version 0.7 ( as suggested by lib/hive-hwi-0.7.0.war) . I
    checked out code from svn two weeks back . I use thrift api's to load data
    into hive programmatically .

    Vaibhav Negi

    On Sat, Feb 12, 2011 at 6:06 AM, Viral Bajaria wrote:

    We use hive through the thrift api too. I have seen the above error when
    someone accidentally connects to the thrift server port through multiple
    instances of our hive-wrapper code. We open a bunch of hive thrift ports to
    achieve the automation of running multiple hive queries in parallel i.e.
    continuous aggregations.

    you need to make sure you are not facing a similar issue because the hive
    thrift server as far as i know is single threaded up until hive 0.5.0 (the
    version that i use). Not too sure if that's been changed in 0.6.0 or higher.

    -Viral
    On Fri, Feb 11, 2011 at 7:06 AM, Ajo Fod wrote:

    Are you using hive 0.6? ... may be fixed in the latest version.

    Also I wonder why these thrift libraries are being used ... is this normal
    hive operation, or can you do something to avoid using thrift?

    -Ajo

    On Fri, Feb 11, 2011 at 12:05 AM, vaibhav negi wrote:


    Hi all,

    I am loading data into hive tables by connecting to hiveserver through
    thrift api using "load data local inpath ... " query .
    Hive server is running as a background process for days . After some 4-5
    days hiveserver throws this error :

    11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error occurred
    during acceptance of message.
    org.apache.thrift.transport.TTransportException:
    java.net.SocketException: Too many open files
    at
    org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
    at
    org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
    at
    org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
    at
    org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184)
    at
    org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: java.net.SocketException: Too many open files
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    at java.net.ServerSocket.implAccept(ServerSocket.java:453)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at
    org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
    ... 9 more

    I checked network connections using netstat and it show about 1000 active
    network connections, most of them to hadoop datanode . Output is attached as
    netstat.out

    after i restarted hiveserver network connections falls to about 20 .
    output is attached as netstat_after_HS_restart.out

    Is this a problem with hive or hadoop configuration ?

    Vaibhav Negi
  • Ajo Fod at Feb 15, 2011 at 1:06 pm
    You could try to save the information in a tab separated file and try to
    load that into hive.

    -Ajo
    On Tue, Feb 15, 2011 at 1:24 AM, vaibhav negi wrote:

    Hi ,

    I am using hive version 0.7 ( as suggested by lib/hive-hwi-0.7.0.war) . I
    checked out code from svn two weeks back . I use thrift api's to load data
    into hive programmatically .

    Vaibhav Negi


    On Sat, Feb 12, 2011 at 6:06 AM, Viral Bajaria wrote:

    We use hive through the thrift api too. I have seen the above error when
    someone accidentally connects to the thrift server port through multiple
    instances of our hive-wrapper code. We open a bunch of hive thrift ports to
    achieve the automation of running multiple hive queries in parallel i.e.
    continuous aggregations.

    you need to make sure you are not facing a similar issue because the hive
    thrift server as far as i know is single threaded up until hive 0.5.0 (the
    version that i use). Not too sure if that's been changed in 0.6.0 or higher.

    -Viral
    On Fri, Feb 11, 2011 at 7:06 AM, Ajo Fod wrote:

    Are you using hive 0.6? ... may be fixed in the latest version.

    Also I wonder why these thrift libraries are being used ... is this
    normal hive operation, or can you do something to avoid using thrift?

    -Ajo

    On Fri, Feb 11, 2011 at 12:05 AM, vaibhav negi wrote:


    Hi all,

    I am loading data into hive tables by connecting to hiveserver through
    thrift api using "load data local inpath ... " query .
    Hive server is running as a background process for days . After some 4-5
    days hiveserver throws this error :

    11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error
    occurred during acceptance of message.
    org.apache.thrift.transport.TTransportException:
    java.net.SocketException: Too many open files
    at
    org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
    at
    org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
    at
    org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
    at
    org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184)
    at
    org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: java.net.SocketException: Too many open files
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    at java.net.ServerSocket.implAccept(ServerSocket.java:453)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at
    org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
    ... 9 more

    I checked network connections using netstat and it show about 1000
    active network connections, most of them to hadoop datanode . Output is
    attached as netstat.out

    after i restarted hiveserver network connections falls to about 20 .
    output is attached as netstat_after_HS_restart.out

    Is this a problem with hive or hadoop configuration ?

    Vaibhav Negi

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedFeb 11, '11 at 8:05a
activeFeb 15, '11 at 1:06p
posts5
users3
websitehive.apache.org

People

Translate

site design / logo © 2021 Grokbase