FAQ
Mohit Sabharwal created HIVE-8707:
-------------------------------------

              Summary: Fix ordering differences due to Java 8 HashMap function
                  Key: HIVE-8707
                  URL: https://issues.apache.org/jira/browse/HIVE-8707
              Project: Hive
           Issue Type: Sub-task
             Reporter: Mohit Sabharwal
             Assignee: Mohit Sabharwal


Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])

For example:
   - In Explain output: Stages are numbered differently
   - In PostExecutePrinter hook, table order is different in lineage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Search Discussions

  • Mohit Sabharwal (JIRA) at Nov 3, 2014 at 6:42 am
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Mohit Sabharwal updated HIVE-8707:
    ----------------------------------
         Attachment: HIVE-8707.patch
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mohit Sabharwal (JIRA) at Nov 3, 2014 at 6:42 am
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Mohit Sabharwal updated HIVE-8707:
    ----------------------------------
         Status: Patch Available (was: Open)
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mohit Sabharwal (JIRA) at Nov 3, 2014 at 6:47 am
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14194304#comment-14194304 ]

    Mohit Sabharwal commented on HIVE-8707:
    ---------------------------------------

    Initial patch that uses insertion order hashmap to address some of the test failures under Java 8.

    Let's see how many failures we get for Java 7.
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Hive QA (JIRA) at Nov 3, 2014 at 8:29 am
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14194353#comment-14194353 ]

    Hive QA commented on HIVE-8707:
    -------------------------------



    {color:red}Overall{color}: -1 at least one tests failed

    Here are the results of testing the latest attachment:
    https://issues.apache.org/jira/secure/attachment/12678889/HIVE-8707.patch

    {color:red}ERROR:{color} -1 due to 313 failed/errored test(s), 6669 tests executed
    *Failed tests:*
    {noformat}
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join_pkfk
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18_multi_distinct
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join30
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join31
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_smb_mapjoin_14
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_6
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_4
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_5
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_8
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cluster
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_complex_alias
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer10
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer11
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer12
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer13
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer14
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer15
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer4
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer5
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer7
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer9
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_join
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_1
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_dependency
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_rearrange
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppd
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_self_join
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_innerjoin
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input39
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join11
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join12
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join13
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join14
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join15
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join17
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join18
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join18_multi_distinct
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join19
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join20
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join21
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join22
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join23
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join25
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join27
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join29
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join3
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join30
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join31
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join36
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join37
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join39
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join40
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join41
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join9
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_alt_syntax
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_1
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_3
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_4
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual1
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual3
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual4
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merge_multi_expressions
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merging
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_rc
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder3
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_star
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_thrift
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_vc
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown_negative
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lineage1
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part13
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part14
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_louter_join_ppr
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_distinct
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mergejoins
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mergejoins_mixed
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin1
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_join_union
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_no_hooks
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonblock_op_deduplicate
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_optimize_nullscan
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_optional_outer
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_outer_join_ppr
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_join
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_clusterby
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby_join
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join3
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join5
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_multi_insert
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join1
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join3
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join4
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join5
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_random
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_repeated_alias
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_udf_case
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_quotedid_skew
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_reduce_deduplicate_exclude_join
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_reduce_deduplicate_extended
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regex_col
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_router_join_ppr
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_transform_hint
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin1
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin10
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin11
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin3
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin4
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin6
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin7
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin9
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_noskew
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_union_remove_1
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_union_remove_2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt1
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt10
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt11
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt12
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt13
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt14
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt15
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt16
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt17
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt18
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt19
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt20
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt3
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt4
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt5
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt6
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt7
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt8
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt9
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin9
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_11
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_12
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_14
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_17
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_6
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_7
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subq_where_serialization
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_having
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_unqualcolumnrefs
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table_join1
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_case_column_pruning
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union20
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union22
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union23
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union24
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union25
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union26
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union3
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union30
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union31
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union32
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union34
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_lateralview
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_11
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_15
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_16
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_3
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_left_outer_join
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_mapjoin_reduce
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_context
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_nested_mapjoin
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_ptf
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_shufflejoin
    org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join1
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_1
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_11
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_12
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_14
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_15
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_2
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_3
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_4
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_5
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_7
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_8
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_9
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez1
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez2
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_correlationoptimizer1
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cross_join
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cross_product_check_1
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cross_product_check_2
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning_2
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_join1
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapjoin_decimal
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapjoin_mapjoin
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_skewjoin
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_temp_table
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_bmj_schema_evolution
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_tests
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_joins_explain
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_main
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_group_by
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union3
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_mapjoin
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_left_outer_join
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_bucketmapjoin1
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_context
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_mapjoin
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_nested_mapjoin
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_ptf
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_shufflejoin
    org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_map_operators
    org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
    org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_join1
    org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
    org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_bucket_mapjoin_mismatch1
    org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_join_nonexistent_part
    org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
    org.apache.hadoop.hive.ql.io.parquet.serde.TestParquetTimestampUtils.testTimezone
    org.apache.hadoop.hive.ql.parse.TestParse.testParse_join1
    org.apache.hadoop.hive.ql.parse.TestParse.testParse_join2
    org.apache.hadoop.hive.ql.parse.TestParse.testParse_join3
    org.apache.hadoop.hive.ql.parse.TestParse.testParse_join4
    org.apache.hadoop.hive.ql.parse.TestParse.testParse_join5
    org.apache.hadoop.hive.ql.parse.TestParse.testParse_join6
    org.apache.hadoop.hive.ql.parse.TestParse.testParse_join7
    org.apache.hadoop.hive.ql.parse.TestParse.testParse_join8
    org.apache.hive.hcatalog.listener.TestNotificationListener.testAMQListener
    org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
    {noformat}

    Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1611/testReport
    Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1611/console
    Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1611/

    Messages:
    {noformat}
    Executing org.apache.hive.ptest.execution.PrepPhase
    Executing org.apache.hive.ptest.execution.ExecutionPhase
    Executing org.apache.hive.ptest.execution.ReportingPhase
    Tests exited with: TestsFailedException: 313 tests failed
    {noformat}

    This message is automatically generated.

    ATTACHMENT ID: 12678889 - PreCommit-HIVE-TRUNK-Build
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mohit Sabharwal (JIRA) at Nov 14, 2014 at 1:39 am
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Mohit Sabharwal updated HIVE-8707:
    ----------------------------------
         Attachment: HIVE-8707.1.patch
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.1.patch, HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mohit Sabharwal (JIRA) at Nov 14, 2014 at 1:41 am
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211678#comment-14211678 ]

    Mohit Sabharwal commented on HIVE-8707:
    ---------------------------------------

    Regeneration q-test output files with "insert order" changes. The code changes are same as in HIVE-8707.patch.
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.1.patch, HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Hive QA (JIRA) at Nov 14, 2014 at 1:39 pm
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14212247#comment-14212247 ]

    Hive QA commented on HIVE-8707:
    -------------------------------



    {color:red}Overall{color}: -1 at least one tests failed

    Here are the results of testing the latest attachment:
    https://issues.apache.org/jira/secure/attachment/12681467/HIVE-8707.1.patch

    {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 6687 tests executed
    *Failed tests:*
    {noformat}
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_union
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_join2
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_union
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_join_nullsafe
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
    org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
    org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
    {noformat}

    Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1787/testReport
    Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1787/console
    Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1787/

    Messages:
    {noformat}
    Executing org.apache.hive.ptest.execution.PrepPhase
    Executing org.apache.hive.ptest.execution.ExecutionPhase
    Executing org.apache.hive.ptest.execution.ReportingPhase
    Tests exited with: TestsFailedException: 7 tests failed
    {noformat}

    This message is automatically generated.

    ATTACHMENT ID: 12681467 - PreCommit-HIVE-TRUNK-Build
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.1.patch, HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Brock Noland (JIRA) at Nov 16, 2014 at 1:38 am
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14213830#comment-14213830 ]

    Brock Noland commented on HIVE-8707:
    ------------------------------------

    Thank you Mohit! The change looks great. I am wondering should we do something to document that these maps should be a deterministic ordered map? For example we could add a javadoc to their declaration, create a tiny wrapper {{LinkedHashMap}} say called {{DeterminsiticOrderedMap}} or just put a comment where we create the Map saying must be deterministic referencing this jIRA?
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.1.patch, HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mohit Sabharwal (JIRA) at Nov 20, 2014 at 6:33 pm
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Mohit Sabharwal updated HIVE-8707:
    ----------------------------------
         Attachment: HIVE-8707.2.patch
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.1.patch, HIVE-8707.2.patch, HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mohit Sabharwal (JIRA) at Nov 20, 2014 at 6:35 pm
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219740#comment-14219740 ]

    Mohit Sabharwal commented on HIVE-8707:
    ---------------------------------------

    Updated patch:
       - Added comment where we create the Map saying must be deterministic and referencing this JIRA.
       - Regenerated q-test output files
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.1.patch, HIVE-8707.2.patch, HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Brock Noland (JIRA) at Nov 20, 2014 at 9:43 pm
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14220061#comment-14220061 ]

    Brock Noland commented on HIVE-8707:
    ------------------------------------

    +1 on the approach
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.1.patch, HIVE-8707.2.patch, HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Hive QA (JIRA) at Nov 21, 2014 at 9:25 am
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14220718#comment-14220718 ]

    Hive QA commented on HIVE-8707:
    -------------------------------



    {color:red}Overall{color}: -1 at least one tests failed

    Here are the results of testing the latest attachment:
    https://issues.apache.org/jira/secure/attachment/12682704/HIVE-8707.2.patch

    {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6649 tests executed
    *Failed tests:*
    {noformat}
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
    {noformat}

    Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1854/testReport
    Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1854/console
    Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1854/

    Messages:
    {noformat}
    Executing org.apache.hive.ptest.execution.PrepPhase
    Executing org.apache.hive.ptest.execution.ExecutionPhase
    Executing org.apache.hive.ptest.execution.ReportingPhase
    Tests exited with: TestsFailedException: 1 tests failed
    {noformat}

    This message is automatically generated.

    ATTACHMENT ID: 12682704 - PreCommit-HIVE-TRUNK-Build
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.1.patch, HIVE-8707.2.patch, HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mohit Sabharwal (JIRA) at Nov 21, 2014 at 3:40 pm
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Mohit Sabharwal updated HIVE-8707:
    ----------------------------------
         Attachment: HIVE-8707.3.patch
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.1.patch, HIVE-8707.2.patch, HIVE-8707.3.patch, HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mohit Sabharwal (JIRA) at Nov 21, 2014 at 3:41 pm
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221047#comment-14221047 ]

    Mohit Sabharwal commented on HIVE-8707:
    ---------------------------------------

    Updated patch with regenerated TestMiniTezCliDriver.testCliDriver_optimize_nullscan output to address ordering issue.
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.1.patch, HIVE-8707.2.patch, HIVE-8707.3.patch, HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Brock Noland (JIRA) at Nov 21, 2014 at 8:17 pm
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221369#comment-14221369 ]

    Brock Noland commented on HIVE-8707:
    ------------------------------------

    +1 pending tests
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.1.patch, HIVE-8707.2.patch, HIVE-8707.3.patch, HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Hive QA (JIRA) at Nov 23, 2014 at 5:14 am
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14222313#comment-14222313 ]

    Hive QA commented on HIVE-8707:
    -------------------------------



    {color:red}Overall{color}: -1 at least one tests failed

    Here are the results of testing the latest attachment:
    https://issues.apache.org/jira/secure/attachment/12682900/HIVE-8707.3.patch

    {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6651 tests executed
    *Failed tests:*
    {noformat}
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_acid_dynamic_partition
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
    org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
    {noformat}

    Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1872/testReport
    Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1872/console
    Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1872/

    Messages:
    {noformat}
    Executing org.apache.hive.ptest.execution.PrepPhase
    Executing org.apache.hive.ptest.execution.ExecutionPhase
    Executing org.apache.hive.ptest.execution.ReportingPhase
    Tests exited with: TestsFailedException: 3 tests failed
    {noformat}

    This message is automatically generated.

    ATTACHMENT ID: 12682900 - PreCommit-HIVE-TRUNK-Build
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.1.patch, HIVE-8707.2.patch, HIVE-8707.3.patch, HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Brock Noland (JIRA) at Nov 23, 2014 at 5:57 am
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14222324#comment-14222324 ]

    Brock Noland commented on HIVE-8707:
    ------------------------------------

    All tests pass locally for me.
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-8707.1.patch, HIVE-8707.2.patch, HIVE-8707.3.patch, HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Brock Noland (JIRA) at Nov 23, 2014 at 6:00 am
    [ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Brock Noland updated HIVE-8707:
    -------------------------------
            Resolution: Fixed
         Fix Version/s: 0.15.0
                Status: Resolved (was: Patch Available)

    Thank you so much Mohit!! I have committed this to trunk!
    Fix ordering differences due to Java 8 HashMap function
    -------------------------------------------------------

    Key: HIVE-8707
    URL: https://issues.apache.org/jira/browse/HIVE-8707
    Project: Hive
    Issue Type: Sub-task
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Fix For: 0.15.0

    Attachments: HIVE-8707.1.patch, HIVE-8707.2.patch, HIVE-8707.3.patch, HIVE-8707.patch


    Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29])
    For example:
    - In Explain output: Stages are numbered differently
    - In PostExecutePrinter hook, table order is different in lineage.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshive, hadoop
postedNov 3, '14 at 6:37a
activeNov 23, '14 at 6:00a
posts19
users1
websitehive.apache.org

1 user in discussion

Brock Noland (JIRA): 19 posts

People

Translate

site design / logo © 2021 Grokbase