FAQ
Reduce RPC packet size for homogeneous arrays, such as the array responses to listStatus() and getBlockLocations()
------------------------------------------------------------------------------------------------------------------

Key: HADOOP-7158
URL: https://issues.apache.org/jira/browse/HADOOP-7158
Project: Hadoop Common
Issue Type: Improvement
Components: io
Affects Versions: 0.22.0
Reporter: Matt Foley
Assignee: Matt Foley


While commenting on HADOOP-6949, which proposes a big improvement in the RPC wire format for arrays of primitives, Konstantin Shvachko said:
"Can/should we extend this to arrays of non-primitive types? This should benefit return types for calls like listStatus() and getBlockLocations() on a large directory."

The improvement for primitive arrays is based on not type-labeling every element in the array, so the array in question must be strictly homogenous; it cannot have subtypes of the assignable type. For instance, it could not be applied to heartbeat responses of DatanodeCommand[], whose array elements carry subtypes of DatanodeCommand, each of which must be type-labeled independently. However, as Konstantin points out, it could really help lengthy response arrays for things like listStatus() and getBlockLocations().

I will attach a prototype implementation to this Jira, for discussion. However, since it can't be automatically applied to all arrays passing through RPC, I'll just providing the wrapper type. By using it, a caller is asserting that the array is strictly homogeneous in the above sense.


--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedFeb 28, '11 at 9:24a
activeFeb 28, '11 at 9:24a
posts1
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Matt Foley (JIRA): 1 post

People

Translate

site design / logo © 2022 Grokbase