FAQ
Connection with RemoteException is not removed from cached HashTable and cause memory leak
------------------------------------------------------------------------------------------

Key: HADOOP-7456
URL: https://issues.apache.org/jira/browse/HADOOP-7456
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 0.20.2
Reporter: Angelo K. Huang


In a long running system like Oozie, we use hadoop client APIs, such as FileSystem.exists(), to check files exist on hdfs or not to kick off a user job. But in a production environment, user sometimes gives wrong or invalid format of file/directory paths. In that case, after server was up for couple days, we found around 80% of memory were taken away by hadoop ipc client connections. In one of the connections, there was a hashtable contains 200k entries. We cross-checked Hadoop code and found out that in org.apache.hadoop.ipc.Client.receiveResponse(), if state if fatal, the call object does not remove from the hashtable (calls) and keeps in the memory until system throws OutOfMemory error or crash. The code in question is here :

# org.apache.hadoop.ipc.Client.receiveResponse()
} else if (state == Status.FATAL.state) {
// Close the connection
markClosed(new RemoteException(WritableUtils.readString(in),
WritableUtils.readString(in)));
}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Search Discussions

  • Angelo K. Huang (JIRA) at Jul 12, 2011 at 3:01 am
    [ https://issues.apache.org/jira/browse/HADOOP-7456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Angelo K. Huang resolved HADOOP-7456.
    -------------------------------------

    Resolution: Duplicate
    Connection with RemoteException is not removed from cached HashTable and cause memory leak
    ------------------------------------------------------------------------------------------

    Key: HADOOP-7456
    URL: https://issues.apache.org/jira/browse/HADOOP-7456
    Project: Hadoop Common
    Issue Type: Bug
    Components: fs
    Affects Versions: 0.20.2
    Reporter: Angelo K. Huang

    In a long running system like Oozie, we use hadoop client APIs, such as FileSystem.exists(), to check files exist on hdfs or not to kick off a user job. But in a production environment, user sometimes gives wrong or invalid format of file/directory paths. In that case, after server was up for couple days, we found around 80% of memory were taken away by hadoop ipc client connections. In one of the connections, there was a hashtable contains 200k entries. We cross-checked Hadoop code and found out that in org.apache.hadoop.ipc.Client.receiveResponse(), if state if fatal, the call object does not remove from the hashtable (calls) and keeps in the memory until system throws OutOfMemory error or crash. The code in question is here :
    * org.apache.hadoop.ipc.Client.receiveResponse()
    } else if (state == Status.FATAL.state) {
    // Close the connection
    markClosed(new RemoteException(WritableUtils.readString(in),
    WritableUtils.readString(in)));
    }
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedJul 9, '11 at 1:10a
activeJul 12, '11 at 3:01a
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Angelo K. Huang (JIRA): 2 posts

People

Translate

site design / logo © 2022 Grokbase