FAQ
SequenceFile.MergeQueue.merge inadvertently creates merge-outputs in the wrong FileSystem, at times in the InMemoryFileSystem
-----------------------------------------------------------------------------------------------------------------------------

Key: HADOOP-3356
URL: https://issues.apache.org/jira/browse/HADOOP-3356
Project: Hadoop Core
Issue Type: Bug
Components: io
Affects Versions: 0.16.3
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Fix For: 0.18.0


The offending code is:

{code:title=SequenceFile.java}
Path outputFile = lDirAlloc.getLocalPathForWrite(
tmpFilename.toString(),
approxOutputSize, conf);
LOG.debug("writing intermediate results to " + outputFile);
Writer writer = cloneFileAttributes(
fs.makeQualified(segmentsToMerge.get(0).segmentPathName),
fs.makeQualified(outputFile), null);
{code}

*fs* is InMemoryFileSystem when ReduceTask.ReduceCopier constructs it... so the wrong FileSystem is used during intermediate merges.



--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Arun C Murthy (JIRA) at May 7, 2008 at 5:46 am
    [ https://issues.apache.org/jira/browse/HADOOP-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Arun C Murthy updated HADOOP-3356:
    ----------------------------------

    Priority: Critical (was: Major)
    SequenceFile.MergeQueue.merge inadvertently creates merge-outputs in the wrong FileSystem, at times in the InMemoryFileSystem
    -----------------------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3356
    URL: https://issues.apache.org/jira/browse/HADOOP-3356
    Project: Hadoop Core
    Issue Type: Bug
    Components: io
    Affects Versions: 0.16.3
    Reporter: Arun C Murthy
    Assignee: Arun C Murthy
    Priority: Critical
    Fix For: 0.18.0


    The offending code is:
    {code:title=SequenceFile.java}
    Path outputFile = lDirAlloc.getLocalPathForWrite(
    tmpFilename.toString(),
    approxOutputSize, conf);
    LOG.debug("writing intermediate results to " + outputFile);
    Writer writer = cloneFileAttributes(
    fs.makeQualified(segmentsToMerge.get(0).segmentPathName),
    fs.makeQualified(outputFile), null);
    {code}
    *fs* is InMemoryFileSystem when ReduceTask.ReduceCopier constructs it... so the wrong FileSystem is used during intermediate merges.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Devaraj Das (JIRA) at May 7, 2008 at 11:33 am
    [ https://issues.apache.org/jira/browse/HADOOP-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Devaraj Das updated HADOOP-3356:
    --------------------------------

    Priority: Minor (was: Critical)

    This part of the code must never be hit under normal circumstances for intermediate merges (during shuffle). I chatted with Arun offline and he agreed on this.
    SequenceFile.MergeQueue.merge inadvertently creates merge-outputs in the wrong FileSystem, at times in the InMemoryFileSystem
    -----------------------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3356
    URL: https://issues.apache.org/jira/browse/HADOOP-3356
    Project: Hadoop Core
    Issue Type: Bug
    Components: io
    Affects Versions: 0.16.3
    Reporter: Arun C Murthy
    Assignee: Arun C Murthy
    Priority: Minor
    Fix For: 0.18.0


    The offending code is:
    {code:title=SequenceFile.java}
    Path outputFile = lDirAlloc.getLocalPathForWrite(
    tmpFilename.toString(),
    approxOutputSize, conf);
    LOG.debug("writing intermediate results to " + outputFile);
    Writer writer = cloneFileAttributes(
    fs.makeQualified(segmentsToMerge.get(0).segmentPathName),
    fs.makeQualified(outputFile), null);
    {code}
    *fs* is InMemoryFileSystem when ReduceTask.ReduceCopier constructs it... so the wrong FileSystem is used during intermediate merges.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Devaraj Das (JIRA) at May 7, 2008 at 11:41 am
    [ https://issues.apache.org/jira/browse/HADOOP-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594858#action_12594858 ]

    devaraj edited comment on HADOOP-3356 at 5/7/08 4:39 AM:
    -------------------------------------------------------------

    This part of the code must never be hit under normal circumstances for intermediate merges (during shuffle). We should only do single-level merges for the intermediate merges. I chatted with Arun offline and he agreed on this.
    Note that this part works as expected when it is supposed to be executed - for multi-level merges and that happens only at the end of the shuffle (when the fs is the localfs).
    We probably should fix this for completeness sake but it is definitely not a critical/major issue.

    was (Author: devaraj):
    This part of the code must never be hit under normal circumstances for intermediate merges (during shuffle). I chatted with Arun offline and he agreed on this.
    SequenceFile.MergeQueue.merge inadvertently creates merge-outputs in the wrong FileSystem, at times in the InMemoryFileSystem
    -----------------------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3356
    URL: https://issues.apache.org/jira/browse/HADOOP-3356
    Project: Hadoop Core
    Issue Type: Bug
    Components: io
    Affects Versions: 0.16.3
    Reporter: Arun C Murthy
    Assignee: Arun C Murthy
    Priority: Minor
    Fix For: 0.18.0


    The offending code is:
    {code:title=SequenceFile.java}
    Path outputFile = lDirAlloc.getLocalPathForWrite(
    tmpFilename.toString(),
    approxOutputSize, conf);
    LOG.debug("writing intermediate results to " + outputFile);
    Writer writer = cloneFileAttributes(
    fs.makeQualified(segmentsToMerge.get(0).segmentPathName),
    fs.makeQualified(outputFile), null);
    {code}
    *fs* is InMemoryFileSystem when ReduceTask.ReduceCopier constructs it... so the wrong FileSystem is used during intermediate merges.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Mukund Madhugiri (JIRA) at Jun 7, 2008 at 1:27 am
    [ https://issues.apache.org/jira/browse/HADOOP-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Mukund Madhugiri updated HADOOP-3356:
    -------------------------------------

    Fix Version/s: (was: 0.18.0)
    SequenceFile.MergeQueue.merge inadvertently creates merge-outputs in the wrong FileSystem, at times in the InMemoryFileSystem
    -----------------------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3356
    URL: https://issues.apache.org/jira/browse/HADOOP-3356
    Project: Hadoop Core
    Issue Type: Bug
    Components: io
    Affects Versions: 0.16.3
    Reporter: Arun C Murthy
    Assignee: Arun C Murthy
    Priority: Minor

    The offending code is:
    {code:title=SequenceFile.java}
    Path outputFile = lDirAlloc.getLocalPathForWrite(
    tmpFilename.toString(),
    approxOutputSize, conf);
    LOG.debug("writing intermediate results to " + outputFile);
    Writer writer = cloneFileAttributes(
    fs.makeQualified(segmentsToMerge.get(0).segmentPathName),
    fs.makeQualified(outputFile), null);
    {code}
    *fs* is InMemoryFileSystem when ReduceTask.ReduceCopier constructs it... so the wrong FileSystem is used during intermediate merges.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMay 7, '08 at 5:46a
activeJun 7, '08 at 1:27a
posts5
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Mukund Madhugiri (JIRA): 5 posts

People

Translate

site design / logo © 2021 Grokbase