FAQ
output format classes that can write to different files depending on keys and/or config variable
-------------------------------------------------------------------------------------------------

Key: HADOOP-2906
URL: https://issues.apache.org/jira/browse/HADOOP-2906
Project: Hadoop Core
Issue Type: New Feature
Reporter: Runping Qi



I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Runping Qi (JIRA) at Feb 26, 2008 at 11:31 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Component/s: mapred
    Description:
    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.


    was:

    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.


    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi

    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 28, 2008 at 6:32 am
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi reassigned HADOOP-2906:
    ----------------------------------

    Assignee: Runping Qi
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi

    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 28, 2008 at 6:34 am
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Attachment: patch.2096.txt
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 28, 2008 at 6:44 am
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Status: Patch Available (was: Open)


    The attached patch include a common abstract base class (MultipleOutputFormat) and two concrete classes:
    MultipleTextOutputFormat and MultipleSequenceFileOutputFormat. These classes implement the default behaviors,
    which are the same as TextOutputFormat class and SequenceFileOutputFormat class, respectively.
    The users can subclass these classes and overwrite one of the protected method to implement a specific logic
    of writing data to different output files.
    The patch also contains a test case, which also illustrates two special ways of using these classes.


    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Feb 28, 2008 at 8:14 am
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573177#action_12573177 ]

    Hadoop QA commented on HADOOP-2906:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12376696/patch.2096.txt
    against trunk revision 619744.

    @author +1. The patch does not contain any @author tags.

    tests included +1. The patch appears to include 3 new or modified tests.

    javadoc +1. The javadoc tool did not generate any warning messages.

    javac -1. The applied patch generated 620 javac compiler warnings (more than the trunk's current 619 warnings).

    release audit +1. The applied patch does not generate any new release audit warnings.

    findbugs +1. The patch does not introduce any new Findbugs warnings.

    core tests +1. The patch passed core unit tests.

    contrib tests +1. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1859/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1859/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1859/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1859/console

    This message is automatically generated.
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 28, 2008 at 2:48 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573306#action_12573306 ]

    Runping Qi commented on HADOOP-2906:
    ------------------------------------


    I think the extra javac warning is due to the
    @SuppressWarnings("unchecked") directive in the following code

    {code}
    @SuppressWarnings("unchecked")
    public void write(WritableComparable key, Writable value) throws IOException {

    // get the file name based on the key
    String keyBasedPath = generateFileNameForKey(key, myName);

    // get the file name based on the input file name
    String finalPath = getInputFileBasedOutputFileName(myJob, keyBasedPath);

    // get the actual key
    WritableComparable actualKey = generateActualKey(key);

    RecordWriter rw = this.recordWriters.get(finalPath);
    if (rw == null) {
    // if we don't have the record writer yet for the final path, create one
    // and add it to the cache
    rw = getRecordWriter_inner(myFS, myJob, finalPath, myProgressable);
    this.recordWriters.put(finalPath, rw);
    }
    rw.write(actualKey, value);
    };
    {code}
    Since javac warns about
    rw.write(actualKey, value)
    The reason for that is rw is RecordWriter type, not the parameterized one.
    The reason for that is that rw may be a record writer generated by SequenceFileOutputFormat
    which does not generate object of parameterized RecordWriter. Tried a few ways to get rid of the warning, but all failed.

    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 28, 2008 at 5:30 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Attachment: patch.2096.1.txt
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.1.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 28, 2008 at 5:30 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Status: Patch Available (was: Open)


    Finally managed to get rid of the javac warning

    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.1.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 28, 2008 at 5:30 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Status: Open (was: Patch Available)
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.1.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 28, 2008 at 5:30 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Attachment: (was: patch.2096.txt)
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.1.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Feb 28, 2008 at 6:50 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573398#action_12573398 ]

    Hadoop QA commented on HADOOP-2906:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12376745/patch.2096.1.txt
    against trunk revision 619744.

    @author +1. The patch does not contain any @author tags.

    tests included +1. The patch appears to include 3 new or modified tests.

    javadoc +1. The javadoc tool did not generate any warning messages.

    javac -1. The applied patch generated 616 javac compiler warnings (more than the trunk's current 615 warnings).

    release audit +1. The applied patch does not generate any new release audit warnings.

    findbugs +1. The patch does not introduce any new Findbugs warnings.

    core tests -1. The patch failed core unit tests.

    contrib tests +1. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1862/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1862/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1862/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1862/console

    This message is automatically generated.
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.1.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 28, 2008 at 11:08 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Status: Open (was: Patch Available)
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.2.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 28, 2008 at 11:09 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Attachment: patch.2096.2.txt
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.2.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 28, 2008 at 11:09 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Attachment: (was: patch.2096.1.txt)
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.2.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 28, 2008 at 11:10 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Status: Patch Available (was: Open)


    There was a javac warning in the test class.
    The new patch fixes it.

    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.2.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 29, 2008 at 1:53 am
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Attachment: patch.2096.3.txt

    Incorporate some feedback comments
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.2.txt, patch.2096.3.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 29, 2008 at 1:53 am
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Status: Open (was: Patch Available)
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.2.txt, patch.2096.3.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 29, 2008 at 1:55 am
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Status: Patch Available (was: Open)
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.3.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Feb 29, 2008 at 1:55 am
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Attachment: (was: patch.2096.2.txt)
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.3.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Feb 29, 2008 at 2:49 am
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573573#action_12573573 ]

    Hadoop QA commented on HADOOP-2906:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12376775/patch.2096.2.txt
    against trunk revision 619744.

    @author +1. The patch does not contain any @author tags.

    tests included +1. The patch appears to include 3 new or modified tests.

    javadoc +1. The javadoc tool did not generate any warning messages.

    javac +1. The applied patch does not generate any new javac compiler warnings.

    release audit +1. The applied patch does not generate any new release audit warnings.

    findbugs +1. The patch does not introduce any new Findbugs warnings.

    core tests -1. The patch failed core unit tests.

    contrib tests +1. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1869/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1869/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1869/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1869/console

    This message is automatically generated.
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.3.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Feb 29, 2008 at 5:15 am
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573595#action_12573595 ]

    Hadoop QA commented on HADOOP-2906:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12376785/patch.2096.3.txt
    against trunk revision 619744.

    @author +1. The patch does not contain any @author tags.

    tests included +1. The patch appears to include 3 new or modified tests.

    javadoc +1. The javadoc tool did not generate any warning messages.

    javac +1. The applied patch does not generate any new javac compiler warnings.

    release audit +1. The applied patch does not generate any new release audit warnings.

    findbugs +1. The patch does not introduce any new Findbugs warnings.

    core tests -1. The patch failed core unit tests.

    contrib tests +1. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1872/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1872/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1872/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1872/console

    This message is automatically generated.
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.3.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Mar 4, 2008 at 1:48 am
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Attachment: (was: patch.2096.3.txt)
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.4


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Mar 4, 2008 at 1:48 am
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Attachment: patch.2096.4
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.4


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Mar 4, 2008 at 7:26 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Attachment: patch.2096.5.txt


    Previously attached patch was wrong.
    Attach the correct version now.

    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.5.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Mar 4, 2008 at 7:26 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Attachment: (was: patch.2096.4)
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.5.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Chris Douglas (JIRA) at Mar 4, 2008 at 9:38 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12575154#action_12575154 ]

    Chris Douglas commented on HADOOP-2906:
    ---------------------------------------

    A couple suggestions:
    * If "num.of.trailing.legs.to.use" exceeds the number of segments in the input file path string, then this will throw an IllegalArgumentException from Path. A more helpful message should probably accompany this condition.
    * It might be worth calling out in the javadocs that {{generateActualKey}} and {{generateActualValue}} should be aware of side-effects, since {{write}} typically doesn't modify its args and the framework will reuse them. The code is clear enough that users can educate themselves, but this is deserving of a footnote.

    Otherwise, +1
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.5.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Martin Traverso (JIRA) at Mar 4, 2008 at 10:09 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12575161#action_12575161 ]

    Martin Traverso commented on HADOOP-2906:
    -----------------------------------------

    I would suggest changing the name of the property from "num.of.trailing.legs.to.use" to something that reflects the hierarchy in which the property lives. Maybe something like mapred.output.format.multi.trailingLegs or similar.
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.5.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Mar 5, 2008 at 3:52 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Attachment: (was: patch.2096.5.txt)
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi

    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Runping Qi (JIRA) at Mar 5, 2008 at 3:54 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Runping Qi updated HADOOP-2906:
    -------------------------------

    Attachment: patch.2096.6.txt


    replaced the attribute name "num.of.trailing.legs.to/use' with "mapred.outputformat.numOfTrailingLegs"

    address the case where the number specified by the above variable is larger than the number of legs
    in the input file.

    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.6.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Chris Douglas (JIRA) at Mar 5, 2008 at 7:18 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Chris Douglas updated HADOOP-2906:
    ----------------------------------

    Status: Open (was: Patch Available)
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.6.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Chris Douglas (JIRA) at Mar 5, 2008 at 7:19 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Chris Douglas updated HADOOP-2906:
    ----------------------------------

    Status: Patch Available (was: Open)
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.6.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Mar 5, 2008 at 9:51 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12575503#action_12575503 ]

    Hadoop QA commented on HADOOP-2906:
    -----------------------------------

    +1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12377175/patch.2096.6.txt
    against trunk revision 619744.

    @author +1. The patch does not contain any @author tags.

    tests included +1. The patch appears to include 3 new or modified tests.

    javadoc +1. The javadoc tool did not generate any warning messages.

    javac +1. The applied patch does not generate any new javac compiler warnings.

    release audit +1. The applied patch does not generate any new release audit warnings.

    findbugs +1. The patch does not introduce any new Findbugs warnings.

    core tests +1. The patch passed core unit tests.

    contrib tests +1. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1897/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1897/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1897/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1897/console

    This message is automatically generated.
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Attachments: patch.2096.6.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Chris Douglas (JIRA) at Mar 6, 2008 at 3:02 am
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Chris Douglas updated HADOOP-2906:
    ----------------------------------

    Resolution: Fixed
    Fix Version/s: 0.17.0
    Status: Resolved (was: Patch Available)

    I just committed this. Thanks, Runping!
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Fix For: 0.17.0

    Attachments: patch.2096.6.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hudson (JIRA) at Mar 6, 2008 at 12:30 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12575671#action_12575671 ]

    Hudson commented on HADOOP-2906:
    --------------------------------

    Integrated in Hadoop-trunk #421 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/421/])
    output format classes that can write to different files depending on keys and/or config variable
    -------------------------------------------------------------------------------------------------

    Key: HADOOP-2906
    URL: https://issues.apache.org/jira/browse/HADOOP-2906
    Project: Hadoop Core
    Issue Type: New Feature
    Components: mapred
    Reporter: Runping Qi
    Assignee: Runping Qi
    Fix For: 0.17.0

    Attachments: patch.2096.6.txt


    I've a few apps that require to write out data into different files/directories depending on keys and/or configuration variables.
    I've implemented such classes for those apps. I noticed that many other users have similar need from time to time.
    So I think it may be a good idea to contribute to Hadoop mapred.lib package so that other users can benefit from it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedFeb 26, '08 at 11:31p
activeMar 6, '08 at 12:30p
posts35
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Hudson (JIRA): 35 posts

People

Translate

site design / logo © 2022 Grokbase