Grokbase Groups Hive dev April 2011
FAQ
alter table concatenate fails and deletes data
----------------------------------------------

Key: HIVE-2125
URL: https://issues.apache.org/jira/browse/HIVE-2125
Project: Hive
Issue Type: Bug
Reporter: Joydeep Sen Sarma
Priority: Critical


the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)

hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
Hadoop job information for null: number of mappers: 0; number of reducers: 0
2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
Ended Job = job_201103101203_453180 with errors
Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
after the job fails - the partition is deleted

thankfully it's still in trash

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Search Discussions

  • Joydeep Sen Sarma (JIRA) at Apr 22, 2011 at 5:38 pm
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Joydeep Sen Sarma reassigned HIVE-2125:
    ---------------------------------------

    Assignee: He Yongqiang
    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical

    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • He Yongqiang (JIRA) at Apr 25, 2011 at 10:46 pm
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    He Yongqiang updated HIVE-2125:
    -------------------------------

    Status: Patch Available (was: Open)
    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • He Yongqiang (JIRA) at Apr 25, 2011 at 10:46 pm
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    He Yongqiang updated HIVE-2125:
    -------------------------------

    Attachment: HIVE-2125.1.patch
    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • He Yongqiang (JIRA) at Apr 25, 2011 at 10:48 pm
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13024993#comment-13024993 ]

    He Yongqiang commented on HIVE-2125:
    ------------------------------------

    It is because the number of reducers is set to -1.
    In this patch, it is set to zero.
    Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job.
    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • jiraposter@reviews.apache.org (JIRA) at Apr 25, 2011 at 10:50 pm
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13024994#comment-13024994 ]

    jiraposter@reviews.apache.org commented on HIVE-2125:
    -----------------------------------------------------


    -----------------------------------------------------------
    This is an automatically generated e-mail. To reply, visit:
    https://reviews.apache.org/r/665/
    -----------------------------------------------------------

    Review request for hive.


    Summary
    -------

    alter table concatenate fails and deletes data

    It is because the number of reducers is set to -1.

    In this patch, it is set to zero.

    Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job.


    This addresses bug HIVE-2125.
    https://issues.apache.org/jira/browse/HIVE-2125


    Diffs
    -----

    trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096599
    trunk/conf/hive-default.xml 1096599
    trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096599
    trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096599
    trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1096599
    trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1096599
    trunk/ql/src/test/queries/clientnegative/alter_merge_index.q PRE-CREATION
    trunk/ql/src/test/queries/clientpositive/alter_merge_index.q PRE-CREATION
    trunk/ql/src/test/results/clientnegative/alter_merge_index.q.out PRE-CREATION
    trunk/ql/src/test/results/clientpositive/alter_merge_index.q.out PRE-CREATION

    Diff: https://reviews.apache.org/r/665/diff


    Testing
    -------


    Thanks,

    Yongqiang


    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • jiraposter@reviews.apache.org (JIRA) at Apr 25, 2011 at 11:24 pm
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025005#comment-13025005 ]

    jiraposter@reviews.apache.org commented on HIVE-2125:
    -----------------------------------------------------


    -----------------------------------------------------------
    This is an automatically generated e-mail. To reply, visit:
    https://reviews.apache.org/r/665/#review554
    -----------------------------------------------------------



    trunk/conf/hive-default.xml
    <https://reviews.apache.org/r/665/#comment1169>

    can you make the indentation consistent with the other property elements?



    trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
    <https://reviews.apache.org/r/665/#comment1170>

    So after adding this, does the block-level merge after INSERT OVERWRITE be automatically supported?


    - Ning


    On 2011-04-25 22:47:17, Yongqiang He wrote:
    bq.
    bq. -----------------------------------------------------------
    bq. This is an automatically generated e-mail. To reply, visit:
    bq. https://reviews.apache.org/r/665/
    bq. -----------------------------------------------------------
    bq.
    bq. (Updated 2011-04-25 22:47:17)
    bq.
    bq.
    bq. Review request for hive.
    bq.
    bq.
    bq. Summary
    bq. -------
    bq.
    bq. alter table concatenate fails and deletes data
    bq.
    bq. It is because the number of reducers is set to -1.
    bq.
    bq. In this patch, it is set to zero.
    bq.
    bq. Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job.
    bq.
    bq.
    bq. This addresses bug HIVE-2125.
    bq. https://issues.apache.org/jira/browse/HIVE-2125
    bq.
    bq.
    bq. Diffs
    bq. -----
    bq.
    bq. trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096599
    bq. trunk/conf/hive-default.xml 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1096599
    bq. trunk/ql/src/test/queries/clientnegative/alter_merge_index.q PRE-CREATION
    bq. trunk/ql/src/test/queries/clientpositive/alter_merge_index.q PRE-CREATION
    bq. trunk/ql/src/test/results/clientnegative/alter_merge_index.q.out PRE-CREATION
    bq. trunk/ql/src/test/results/clientpositive/alter_merge_index.q.out PRE-CREATION
    bq.
    bq. Diff: https://reviews.apache.org/r/665/diff
    bq.
    bq.
    bq. Testing
    bq. -------
    bq.
    bq.
    bq. Thanks,
    bq.
    bq. Yongqiang
    bq.
    bq.


    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • jiraposter@reviews.apache.org (JIRA) at Apr 26, 2011 at 12:18 am
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025014#comment-13025014 ]

    jiraposter@reviews.apache.org commented on HIVE-2125:
    -----------------------------------------------------



    bq. On 2011-04-25 23:20:59, Ning Zhang wrote:
    bq. > trunk/conf/hive-default.xml, line 1042
    bq. > <https://reviews.apache.org/r/665/diff/1/?file=17317#file17317line1042>
    bq. >
    bq. > can you make the indentation consistent with the other property elements?

    It shows the same indentation on my local. So it might just be an review board display issue.


    bq. On 2011-04-25 23:20:59, Ning Zhang wrote:
    bq. > trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java, line 1203
    bq. > <https://reviews.apache.org/r/665/diff/1/?file=17321#file17321line1203>
    bq. >
    bq. > So after adding this, does the block-level merge after INSERT OVERWRITE be automatically supported?

    No. Not automatically supported. We still need to do some work there. but it is a separate issue.


    - Yongqiang


    -----------------------------------------------------------
    This is an automatically generated e-mail. To reply, visit:
    https://reviews.apache.org/r/665/#review554
    -----------------------------------------------------------


    On 2011-04-25 22:47:17, Yongqiang He wrote:
    bq.
    bq. -----------------------------------------------------------
    bq. This is an automatically generated e-mail. To reply, visit:
    bq. https://reviews.apache.org/r/665/
    bq. -----------------------------------------------------------
    bq.
    bq. (Updated 2011-04-25 22:47:17)
    bq.
    bq.
    bq. Review request for hive.
    bq.
    bq.
    bq. Summary
    bq. -------
    bq.
    bq. alter table concatenate fails and deletes data
    bq.
    bq. It is because the number of reducers is set to -1.
    bq.
    bq. In this patch, it is set to zero.
    bq.
    bq. Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job.
    bq.
    bq.
    bq. This addresses bug HIVE-2125.
    bq. https://issues.apache.org/jira/browse/HIVE-2125
    bq.
    bq.
    bq. Diffs
    bq. -----
    bq.
    bq. trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096599
    bq. trunk/conf/hive-default.xml 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1096599
    bq. trunk/ql/src/test/queries/clientnegative/alter_merge_index.q PRE-CREATION
    bq. trunk/ql/src/test/queries/clientpositive/alter_merge_index.q PRE-CREATION
    bq. trunk/ql/src/test/results/clientnegative/alter_merge_index.q.out PRE-CREATION
    bq. trunk/ql/src/test/results/clientpositive/alter_merge_index.q.out PRE-CREATION
    bq.
    bq. Diff: https://reviews.apache.org/r/665/diff
    bq.
    bq.
    bq. Testing
    bq. -------
    bq.
    bq.
    bq. Thanks,
    bq.
    bq. Yongqiang
    bq.
    bq.


    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • jiraposter@reviews.apache.org (JIRA) at Apr 26, 2011 at 5:35 am
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025078#comment-13025078 ]

    jiraposter@reviews.apache.org commented on HIVE-2125:
    -----------------------------------------------------


    -----------------------------------------------------------
    This is an automatically generated e-mail. To reply, visit:
    https://reviews.apache.org/r/665/#review557
    -----------------------------------------------------------



    trunk/conf/hive-default.xml
    <https://reviews.apache.org/r/665/#comment1177>

    The indentation looks the same in your editor because these are TABs. Please remove, and then configure your editor to disallow TABs.



    trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
    <https://reviews.apache.org/r/665/#comment1176>

    Spacing.



    trunk/ql/src/test/queries/clientnegative/alter_merge_index.q
    <https://reviews.apache.org/r/665/#comment1174>

    Please add some comments explaining what it is that you're testing, e.g. why is this a negative test? Why is expected to fail?



    trunk/ql/src/test/queries/clientpositive/alter_merge_index.q
    <https://reviews.apache.org/r/665/#comment1175>

    Please add some comments here too.


    - Carl


    On 2011-04-25 22:47:17, Yongqiang He wrote:
    bq.
    bq. -----------------------------------------------------------
    bq. This is an automatically generated e-mail. To reply, visit:
    bq. https://reviews.apache.org/r/665/
    bq. -----------------------------------------------------------
    bq.
    bq. (Updated 2011-04-25 22:47:17)
    bq.
    bq.
    bq. Review request for hive.
    bq.
    bq.
    bq. Summary
    bq. -------
    bq.
    bq. alter table concatenate fails and deletes data
    bq.
    bq. It is because the number of reducers is set to -1.
    bq.
    bq. In this patch, it is set to zero.
    bq.
    bq. Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job.
    bq.
    bq.
    bq. This addresses bug HIVE-2125.
    bq. https://issues.apache.org/jira/browse/HIVE-2125
    bq.
    bq.
    bq. Diffs
    bq. -----
    bq.
    bq. trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096599
    bq. trunk/conf/hive-default.xml 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1096599
    bq. trunk/ql/src/test/queries/clientnegative/alter_merge_index.q PRE-CREATION
    bq. trunk/ql/src/test/queries/clientpositive/alter_merge_index.q PRE-CREATION
    bq. trunk/ql/src/test/results/clientnegative/alter_merge_index.q.out PRE-CREATION
    bq. trunk/ql/src/test/results/clientpositive/alter_merge_index.q.out PRE-CREATION
    bq.
    bq. Diff: https://reviews.apache.org/r/665/diff
    bq.
    bq.
    bq. Testing
    bq. -------
    bq.
    bq.
    bq. Thanks,
    bq.
    bq. Yongqiang
    bq.
    bq.


    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Carl Steinbach (JIRA) at Apr 26, 2011 at 5:37 am
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Carl Steinbach updated HIVE-2125:
    ---------------------------------

    Status: Open (was: Patch Available)
    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • jiraposter@reviews.apache.org (JIRA) at Apr 26, 2011 at 6:05 am
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025083#comment-13025083 ]

    jiraposter@reviews.apache.org commented on HIVE-2125:
    -----------------------------------------------------



    bq. On 2011-04-26 05:33:19, Carl Steinbach wrote:
    bq. > trunk/ql/src/test/queries/clientpositive/alter_merge_index.q, line 1
    bq. > <https://reviews.apache.org/r/665/diff/1/?file=17323#file17323line1>
    bq. >
    bq. > Please add some comments here too.

    Carl, what kind of comments do you want here?


    bq. On 2011-04-26 05:33:19, Carl Steinbach wrote:
    bq. > trunk/ql/src/test/queries/clientnegative/alter_merge_index.q, line 1
    bq. > <https://reviews.apache.org/r/665/diff/1/?file=17322#file17322line1>
    bq. >
    bq. > Please add some comments explaining what it is that you're testing, e.g. why is this a negative test? Why is expected to fail?

    bq. >why is this a negative test? Why is expected to fail?
    should i put a positive test on a negative test dir? Should we explain all test cases in client negative dir?
    Come on, please put some positive suggestions.


    - Yongqiang


    -----------------------------------------------------------
    This is an automatically generated e-mail. To reply, visit:
    https://reviews.apache.org/r/665/#review557
    -----------------------------------------------------------


    On 2011-04-25 22:47:17, Yongqiang He wrote:
    bq.
    bq. -----------------------------------------------------------
    bq. This is an automatically generated e-mail. To reply, visit:
    bq. https://reviews.apache.org/r/665/
    bq. -----------------------------------------------------------
    bq.
    bq. (Updated 2011-04-25 22:47:17)
    bq.
    bq.
    bq. Review request for hive.
    bq.
    bq.
    bq. Summary
    bq. -------
    bq.
    bq. alter table concatenate fails and deletes data
    bq.
    bq. It is because the number of reducers is set to -1.
    bq.
    bq. In this patch, it is set to zero.
    bq.
    bq. Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job.
    bq.
    bq.
    bq. This addresses bug HIVE-2125.
    bq. https://issues.apache.org/jira/browse/HIVE-2125
    bq.
    bq.
    bq. Diffs
    bq. -----
    bq.
    bq. trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096599
    bq. trunk/conf/hive-default.xml 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1096599
    bq. trunk/ql/src/test/queries/clientnegative/alter_merge_index.q PRE-CREATION
    bq. trunk/ql/src/test/queries/clientpositive/alter_merge_index.q PRE-CREATION
    bq. trunk/ql/src/test/results/clientnegative/alter_merge_index.q.out PRE-CREATION
    bq. trunk/ql/src/test/results/clientpositive/alter_merge_index.q.out PRE-CREATION
    bq.
    bq. Diff: https://reviews.apache.org/r/665/diff
    bq.
    bq.
    bq. Testing
    bq. -------
    bq.
    bq.
    bq. Thanks,
    bq.
    bq. Yongqiang
    bq.
    bq.


    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • jiraposter@reviews.apache.org (JIRA) at Apr 26, 2011 at 6:55 am
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025098#comment-13025098 ]

    jiraposter@reviews.apache.org commented on HIVE-2125:
    -----------------------------------------------------


    -----------------------------------------------------------
    This is an automatically generated e-mail. To reply, visit:
    https://reviews.apache.org/r/665/#review560
    -----------------------------------------------------------



    trunk/conf/hive-default.xml
    <https://reviews.apache.org/r/665/#comment1187>

    Besides describing the effect of this configuration property, can you also please explain the motivation? Why would the user want to set this to true? Is there a valid reason for setting it to false?



    trunk/ql/src/test/queries/clientnegative/alter_merge_index.q
    <https://reviews.apache.org/r/665/#comment1183>
    Should we explain all test cases in client negative dir?
    Emphatically yes!

    And since I'm a helpful person I'll even write the comment for you:

    "Verify that an error is thrown if the user attempts to concatenate an indexed table when hive.exec.concatenate.check.index=true"

    Also, please explicitly set hive.exec.concatenate.check.index=true at the beginning of this test.

    Also, please change the name of this file and the name of the table so that it matches the semantics of the ALTER TABLE command, i.e. alter_concatenate_indexed_table.q and src_rc_concatenate_test.



    trunk/ql/src/test/queries/clientpositive/alter_merge_index.q
    <https://reviews.apache.org/r/665/#comment1186>

    How about this:

    * Verify that an indexed table can be concatenated when hive.exec.concatenate.check.index=false.

    * Verify that a partitioned table can be concatenated when hive.exec.concatenate.check.index=false.

    The point of the comments is help the reader distinguish between details that are relevant to the intended purpose of the test from those that are not.




    trunk/ql/src/test/queries/clientpositive/alter_merge_index.q
    <https://reviews.apache.org/r/665/#comment1188>

    Please replace "merge" with "concatenate". A better name for this test would be "alter_concatenate_indexed_table.q"



    trunk/ql/src/test/queries/clientpositive/alter_merge_index.q
    <https://reviews.apache.org/r/665/#comment1185>

    These tables get cleaned up automatically. Please remove the drop statements.


    - Carl


    On 2011-04-25 22:47:17, Yongqiang He wrote:
    bq.
    bq. -----------------------------------------------------------
    bq. This is an automatically generated e-mail. To reply, visit:
    bq. https://reviews.apache.org/r/665/
    bq. -----------------------------------------------------------
    bq.
    bq. (Updated 2011-04-25 22:47:17)
    bq.
    bq.
    bq. Review request for hive.
    bq.
    bq.
    bq. Summary
    bq. -------
    bq.
    bq. alter table concatenate fails and deletes data
    bq.
    bq. It is because the number of reducers is set to -1.
    bq.
    bq. In this patch, it is set to zero.
    bq.
    bq. Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job.
    bq.
    bq.
    bq. This addresses bug HIVE-2125.
    bq. https://issues.apache.org/jira/browse/HIVE-2125
    bq.
    bq.
    bq. Diffs
    bq. -----
    bq.
    bq. trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096599
    bq. trunk/conf/hive-default.xml 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1096599
    bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1096599
    bq. trunk/ql/src/test/queries/clientnegative/alter_merge_index.q PRE-CREATION
    bq. trunk/ql/src/test/queries/clientpositive/alter_merge_index.q PRE-CREATION
    bq. trunk/ql/src/test/results/clientnegative/alter_merge_index.q.out PRE-CREATION
    bq. trunk/ql/src/test/results/clientpositive/alter_merge_index.q.out PRE-CREATION
    bq.
    bq. Diff: https://reviews.apache.org/r/665/diff
    bq.
    bq.
    bq. Testing
    bq. -------
    bq.
    bq.
    bq. Thanks,
    bq.
    bq. Yongqiang
    bq.
    bq.


    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • He Yongqiang (JIRA) at Apr 26, 2011 at 10:27 pm
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    He Yongqiang updated HIVE-2125:
    -------------------------------

    Attachment: HIVE-2125.2.patch
    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch, HIVE-2125.2.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • jiraposter@reviews.apache.org (JIRA) at Apr 26, 2011 at 10:29 pm
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025484#comment-13025484 ]

    jiraposter@reviews.apache.org commented on HIVE-2125:
    -----------------------------------------------------


    -----------------------------------------------------------
    This is an automatically generated e-mail. To reply, visit:
    https://reviews.apache.org/r/665/
    -----------------------------------------------------------

    (Updated 2011-04-26 22:27:30.586885)


    Review request for hive.


    Changes
    -------

    address Carl's comments about .q files


    Summary
    -------

    alter table concatenate fails and deletes data

    It is because the number of reducers is set to -1.

    In this patch, it is set to zero.

    Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job.


    This addresses bug HIVE-2125.
    https://issues.apache.org/jira/browse/HIVE-2125


    Diffs (updated)
    -----

    trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096925
    trunk/conf/hive-default.xml 1096925
    trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096925
    trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096925
    trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1096925
    trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1096925
    trunk/ql/src/test/queries/clientnegative/alter_concatenate_indexed_table.q PRE-CREATION
    trunk/ql/src/test/queries/clientpositive/alter_concatenate_indexed_table.q PRE-CREATION
    trunk/ql/src/test/results/clientnegative/alter_concatenate_indexed_table.q.out PRE-CREATION
    trunk/ql/src/test/results/clientpositive/alter_concatenate_indexed_table.q.out PRE-CREATION

    Diff: https://reviews.apache.org/r/665/diff


    Testing
    -------


    Thanks,

    Yongqiang


    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch, HIVE-2125.2.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • He Yongqiang (JIRA) at Apr 26, 2011 at 10:29 pm
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    He Yongqiang updated HIVE-2125:
    -------------------------------

    Status: Patch Available (was: Open)
    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch, HIVE-2125.2.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • He Yongqiang (JIRA) at Apr 26, 2011 at 10:31 pm
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025485#comment-13025485 ]

    He Yongqiang commented on HIVE-2125:
    ------------------------------------

    Carl, the comments for the config is the hive-site.xml. so i think it should not be added to .q files. And also putting comments before 'set' command can lead to some random errors. And all other comments are addressed in this new patch.
    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch, HIVE-2125.2.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Ning Zhang (JIRA) at Apr 27, 2011 at 6:52 am
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025639#comment-13025639 ]

    Ning Zhang commented on HIVE-2125:
    ----------------------------------

    The new patch looks good to me and addressed most of Carl's comments.

    Also about comments in the test .q files, I'm aware of the bug Yongqiang mentioned (test failed in some cases if comments are present, but passed otherwise). We should fix that bug in QTestsUtil and make a group effort to add comments to .q files as much as possible. Carl, what do you think?
    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch, HIVE-2125.2.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Ning Zhang (JIRA) at Apr 27, 2011 at 9:00 pm
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026007#comment-13026007 ]

    Ning Zhang commented on HIVE-2125:
    ----------------------------------

    All unit tests passed.

    @Carl, do you have further comments? If you are +1 we can open a separate JIRA to fix the comment bug in .q files.
    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch, HIVE-2125.2.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Ning Zhang (JIRA) at Apr 28, 2011 at 5:54 pm
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Ning Zhang updated HIVE-2125:
    -----------------------------

    Resolution: Fixed
    Status: Resolved (was: Patch Available)

    Just committed. Thanks Yongqiang!

    Also Yongqiang, can you file a JIRA to fix the comments in .q files (if it is not filed already)?
    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Attachments: HIVE-2125.1.patch, HIVE-2125.2.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Carl Steinbach (JIRA) at May 2, 2011 at 6:14 am
    [ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Carl Steinbach updated HIVE-2125:
    ---------------------------------

    Component/s: Query Processor
    Fix Version/s: 0.8.0
    Hadoop Flags: [Reviewed]
    alter table concatenate fails and deletes data
    ----------------------------------------------

    Key: HIVE-2125
    URL: https://issues.apache.org/jira/browse/HIVE-2125
    Project: Hive
    Issue Type: Bug
    Components: Query Processor
    Reporter: Joydeep Sen Sarma
    Assignee: He Yongqiang
    Priority: Critical
    Fix For: 0.8.0

    Attachments: HIVE-2125.1.patch, HIVE-2125.2.patch


    the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative)
    hive> alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
    Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180
    Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180
    Hadoop job information for null: number of mappers: 0; number of reducers: 0
    2011-04-22 10:21:24,046 null map = 100%, reduce = 100%
    Ended Job = job_201103101203_453180 with errors
    Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16
    after the job fails - the partition is deleted
    thankfully it's still in trash
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshive, hadoop
postedApr 22, '11 at 5:38p
activeMay 2, '11 at 6:14a
posts20
users1
websitehive.apache.org

1 user in discussion

Carl Steinbach (JIRA): 20 posts

People

Translate

site design / logo © 2021 Grokbase