FAQ
Speed up DFS read path
----------------------

Key: HDFS-2080
URL: https://issues.apache.org/jira/browse/HDFS-2080
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs client
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: 0.23.0


I've developed a series of patches that speeds up the HDFS read path by a factor of about 2.5x (~300M/sec to ~800M/sec for localhost reading from buffer cache) and also will make it easier to allow for advanced users (eg hbase) to skip a buffer copy.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Search Discussions

  • Todd Lipcon (Resolved) (JIRA) at Nov 3, 2011 at 7:02 am
    [ https://issues.apache.org/jira/browse/HDFS-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Todd Lipcon resolved HDFS-2080.
    -------------------------------

    Resolution: Fixed
    Fix Version/s: 0.23.1
    Hadoop Flags: Reviewed

    All of the subtasks have now been completed and committed for 0.23.1 and 0.24. Thanks to those that helped, especially Nathan, Kihwal, Eli, and Nicholas for the many reviews.
    Speed up DFS read path by lessening checksum overhead
    -----------------------------------------------------

    Key: HDFS-2080
    URL: https://issues.apache.org/jira/browse/HDFS-2080
    Project: Hadoop HDFS
    Issue Type: Improvement
    Components: hdfs client, performance
    Affects Versions: 0.23.0
    Reporter: Todd Lipcon
    Assignee: Todd Lipcon
    Fix For: 0.24.0, 0.23.1

    Attachments: hdfs-2080.txt, hdfs-2080.txt


    I've developed a series of patches that speeds up the HDFS read path by a factor of about 2.5x (~300M/sec to ~800M/sec for localhost reading from buffer cache) and also will make it easier to allow for advanced users (eg hbase) to skip a buffer copy.
    --
    This message is automatically generated by JIRA.
    If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouphdfs-dev @
categorieshadoop
postedJun 17, '11 at 8:53a
activeNov 3, '11 at 7:02a
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Todd Lipcon (Resolved) (JIRA): 2 posts

People

Translate

site design / logo © 2022 Grokbase