FAQ
bufferSize argument is ignored in FileSystem.create(File, boolean, int)
-----------------------------------------------------------------------

Key: HADOOP-40
URL: http://issues.apache.org/jira/browse/HADOOP-40
Project: Hadoop
Type: Bug
Components: fs
Reporter: Konstantin Shvachko
Priority: Minor


org.apache.hadoop.fs.FileSystem.create(File f, boolean overwrite, int bufferSize)

ignores the input parameter bufferSize.
It passes further down the internal configuration, which includes the buffer size, but not the parameter value.
This works fine within the file system, since everything that calls create extracts buffer size from the same config.
MapReduce although is probably affected by that, see

org.apache.hadoop.io.SequenceFile.Sorter.MergeQueue.MergeQueue(int size, String outName, boolean done)

The attached patch would fix it.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira

Search Discussions

  • Konstantin Shvachko (JIRA) at Feb 17, 2006 at 10:45 pm
    [ http://issues.apache.org/jira/browse/HADOOP-40?page=all ]

    Konstantin Shvachko updated HADOOP-40:
    --------------------------------------

    Attachment: BufferSize.patch
    bufferSize argument is ignored in FileSystem.create(File, boolean, int)
    -----------------------------------------------------------------------

    Key: HADOOP-40
    URL: http://issues.apache.org/jira/browse/HADOOP-40
    Project: Hadoop
    Type: Bug
    Components: fs
    Reporter: Konstantin Shvachko
    Priority: Minor
    Attachments: BufferSize.patch

    org.apache.hadoop.fs.FileSystem.create(File f, boolean overwrite, int bufferSize)
    ignores the input parameter bufferSize.
    It passes further down the internal configuration, which includes the buffer size, but not the parameter value.
    This works fine within the file system, since everything that calls create extracts buffer size from the same config.
    MapReduce although is probably affected by that, see
    org.apache.hadoop.io.SequenceFile.Sorter.MergeQueue.MergeQueue(int size, String outName, boolean done)
    The attached patch would fix it.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Doug Cutting (JIRA) at Feb 23, 2006 at 5:24 pm
    [ http://issues.apache.org/jira/browse/HADOOP-40?page=all ]

    Doug Cutting updated HADOOP-40:
    -------------------------------

    Attachment: (was: bufferSize.patch)
    bufferSize argument is ignored in FileSystem.create(File, boolean, int)
    -----------------------------------------------------------------------

    Key: HADOOP-40
    URL: http://issues.apache.org/jira/browse/HADOOP-40
    Project: Hadoop
    Type: Bug
    Components: fs
    Reporter: Konstantin Shvachko
    Priority: Minor
    Attachments: BufferSize.patch

    org.apache.hadoop.fs.FileSystem.create(File f, boolean overwrite, int bufferSize)
    ignores the input parameter bufferSize.
    It passes further down the internal configuration, which includes the buffer size, but not the parameter value.
    This works fine within the file system, since everything that calls create extracts buffer size from the same config.
    MapReduce although is probably affected by that, see
    org.apache.hadoop.io.SequenceFile.Sorter.MergeQueue.MergeQueue(int size, String outName, boolean done)
    The attached patch would fix it.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Doug Cutting (JIRA) at Feb 23, 2006 at 5:24 pm
    [ http://issues.apache.org/jira/browse/HADOOP-40?page=all ]

    Doug Cutting updated HADOOP-40:
    -------------------------------

    Attachment: bufferSize.patch

    I don't think we should modify the configuration in this case, since that will affect code which uses this configuration that runs later. SequenceFile uses very large buffers when sorting and merging, in order to minimize disk seeks, and we don't want everything to start using such large buffers.

    So why not just pass the missing parameter down? I've attached a patch that does this. Does this look good to you?
    bufferSize argument is ignored in FileSystem.create(File, boolean, int)
    -----------------------------------------------------------------------

    Key: HADOOP-40
    URL: http://issues.apache.org/jira/browse/HADOOP-40
    Project: Hadoop
    Type: Bug
    Components: fs
    Reporter: Konstantin Shvachko
    Priority: Minor
    Attachments: BufferSize.patch

    org.apache.hadoop.fs.FileSystem.create(File f, boolean overwrite, int bufferSize)
    ignores the input parameter bufferSize.
    It passes further down the internal configuration, which includes the buffer size, but not the parameter value.
    This works fine within the file system, since everything that calls create extracts buffer size from the same config.
    MapReduce although is probably affected by that, see
    org.apache.hadoop.io.SequenceFile.Sorter.MergeQueue.MergeQueue(int size, String outName, boolean done)
    The attached patch would fix it.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Doug Cutting (JIRA) at Feb 23, 2006 at 5:26 pm
    [ http://issues.apache.org/jira/browse/HADOOP-40?page=all ]

    Doug Cutting updated HADOOP-40:
    -------------------------------

    Attachment: bufferSize.patch
    bufferSize argument is ignored in FileSystem.create(File, boolean, int)
    -----------------------------------------------------------------------

    Key: HADOOP-40
    URL: http://issues.apache.org/jira/browse/HADOOP-40
    Project: Hadoop
    Type: Bug
    Components: fs
    Reporter: Konstantin Shvachko
    Priority: Minor
    Attachments: BufferSize.patch, bufferSize.patch

    org.apache.hadoop.fs.FileSystem.create(File f, boolean overwrite, int bufferSize)
    ignores the input parameter bufferSize.
    It passes further down the internal configuration, which includes the buffer size, but not the parameter value.
    This works fine within the file system, since everything that calls create extracts buffer size from the same config.
    MapReduce although is probably affected by that, see
    org.apache.hadoop.io.SequenceFile.Sorter.MergeQueue.MergeQueue(int size, String outName, boolean done)
    The attached patch would fix it.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Doug Cutting (JIRA) at Feb 24, 2006 at 9:39 pm
    [ http://issues.apache.org/jira/browse/HADOOP-40?page=all ]

    Doug Cutting resolved HADOOP-40:
    --------------------------------

    Fix Version: 0.1
    Resolution: Fixed

    I just committed my patch for this.
    bufferSize argument is ignored in FileSystem.create(File, boolean, int)
    -----------------------------------------------------------------------

    Key: HADOOP-40
    URL: http://issues.apache.org/jira/browse/HADOOP-40
    Project: Hadoop
    Type: Bug
    Components: fs
    Reporter: Konstantin Shvachko
    Priority: Minor
    Fix For: 0.1
    Attachments: BufferSize.patch, bufferSize.patch

    org.apache.hadoop.fs.FileSystem.create(File f, boolean overwrite, int bufferSize)
    ignores the input parameter bufferSize.
    It passes further down the internal configuration, which includes the buffer size, but not the parameter value.
    This works fine within the file system, since everything that calls create extracts buffer size from the same config.
    MapReduce although is probably affected by that, see
    org.apache.hadoop.io.SequenceFile.Sorter.MergeQueue.MergeQueue(int size, String outName, boolean done)
    The attached patch would fix it.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Konstantin Shvachko (JIRA) at Feb 24, 2006 at 11:26 pm
    [ http://issues.apache.org/jira/browse/HADOOP-40?page=comments#action_12367738 ]

    Konstantin Shvachko commented on HADOOP-40:
    -------------------------------------------

    Yes. This is the right way of doing it.
    It particularly makes sense, since the main file and the checksum file
    do not necessarily need to share the same buffer size.
    With the file buffer large the checksum buffer doesn't need to be large at all.
    bufferSize argument is ignored in FileSystem.create(File, boolean, int)
    -----------------------------------------------------------------------

    Key: HADOOP-40
    URL: http://issues.apache.org/jira/browse/HADOOP-40
    Project: Hadoop
    Type: Bug
    Components: fs
    Reporter: Konstantin Shvachko
    Priority: Minor
    Fix For: 0.1
    Attachments: BufferSize.patch, bufferSize.patch

    org.apache.hadoop.fs.FileSystem.create(File f, boolean overwrite, int bufferSize)
    ignores the input parameter bufferSize.
    It passes further down the internal configuration, which includes the buffer size, but not the parameter value.
    This works fine within the file system, since everything that calls create extracts buffer size from the same config.
    MapReduce although is probably affected by that, see
    org.apache.hadoop.io.SequenceFile.Sorter.MergeQueue.MergeQueue(int size, String outName, boolean done)
    The attached patch would fix it.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedFeb 17, '06 at 10:44p
activeFeb 24, '06 at 11:26p
posts7
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Konstantin Shvachko (JIRA): 7 posts

People

Translate

site design / logo © 2022 Grokbase