FAQ
Mohit Sabharwal created HIVE-9181:
-------------------------------------

              Summary: Fix SkewJoinOptimizer related Java 8 ordering differences
                  Key: HIVE-9181
                  URL: https://issues.apache.org/jira/browse/HIVE-9181
              Project: Hive
           Issue Type: Sub-task
           Components: Tests
             Reporter: Mohit Sabharwal
             Assignee: Mohit Sabharwal


(1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different order in Java 8 vs Java 7. This leads to differences in EXPLAIN output related to {{MapWork}}.

(2) Dbproperties order for a "desc database" query is non-deterministic at the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are un-ordered).
{code}
create database test_db with dbproperties ('key1' = 'value1', 'key2' = 'value2');
desc database extended test_db;
{code}
At client side, we serialize the dbproperties map returned from HMS as string and write it to a temp file. The {{FetchTask}} then reads the temp file and returns the string back to the user. In test environment, for consistent ordering, we should sort the map before it is serialized to the temp file.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Search Discussions

  • Mohit Sabharwal (JIRA) at Dec 20, 2014 at 2:18 am
    [ https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Mohit Sabharwal updated HIVE-9181:
    ----------------------------------
         Attachment: HIVE-9181.patch
    Fix SkewJoinOptimizer related Java 8 ordering differences
    ---------------------------------------------------------

    Key: HIVE-9181
    URL: https://issues.apache.org/jira/browse/HIVE-9181
    Project: Hive
    Issue Type: Sub-task
    Components: Tests
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-9181.patch


    (1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different order in Java 8 vs Java 7. This leads to differences in EXPLAIN output related to {{MapWork}}.
    (2) Dbproperties order for a "desc database" query is non-deterministic at the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are un-ordered).
    {code}
    create database test_db with dbproperties ('key1' = 'value1', 'key2' = 'value2');
    desc database extended test_db;
    {code}
    At client side, we serialize the dbproperties map returned from HMS as string and write it to a temp file. The {{FetchTask}} then reads the temp file and returns the string back to the user. In test environment, for consistent ordering, we should sort the map before it is serialized to the temp file.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mohit Sabharwal (JIRA) at Dec 20, 2014 at 2:19 am
    [ https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Mohit Sabharwal updated HIVE-9181:
    ----------------------------------
         Status: Patch Available (was: Open)
    Fix SkewJoinOptimizer related Java 8 ordering differences
    ---------------------------------------------------------

    Key: HIVE-9181
    URL: https://issues.apache.org/jira/browse/HIVE-9181
    Project: Hive
    Issue Type: Sub-task
    Components: Tests
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-9181.patch


    (1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different order in Java 8 vs Java 7. This leads to differences in EXPLAIN output related to {{MapWork}}.
    (2) Dbproperties order for a "desc database" query is non-deterministic at the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are un-ordered).
    {code}
    create database test_db with dbproperties ('key1' = 'value1', 'key2' = 'value2');
    desc database extended test_db;
    {code}
    At client side, we serialize the dbproperties map returned from HMS as string and write it to a temp file. The {{FetchTask}} then reads the temp file and returns the string back to the user. In test environment, for consistent ordering, we should sort the map before it is serialized to the temp file.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Hive QA (JIRA) at Dec 20, 2014 at 7:25 am
    [ https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254565#comment-14254565 ]

    Hive QA commented on HIVE-9181:
    -------------------------------



    {color:red}Overall{color}: -1 at least one tests failed

    Here are the results of testing the latest attachment:
    https://issues.apache.org/jira/secure/attachment/12688485/HIVE-9181.patch

    {color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 6718 tests executed
    *Failed tests:*
    {noformat}
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_db_owner
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_owner_actions_db
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_database_properties
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_describe_database_json
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_virtual_column
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
    org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
    org.apache.hive.hcatalog.cli.TestSemanticAnalysis.testDescDB
    org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
    {noformat}

    Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2152/testReport
    Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2152/console
    Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2152/

    Messages:
    {noformat}
    Executing org.apache.hive.ptest.execution.PrepPhase
    Executing org.apache.hive.ptest.execution.ExecutionPhase
    Executing org.apache.hive.ptest.execution.ReportingPhase
    Tests exited with: TestsFailedException: 10 tests failed
    {noformat}

    This message is automatically generated.

    ATTACHMENT ID: 12688485 - PreCommit-HIVE-TRUNK-Build
    Fix SkewJoinOptimizer related Java 8 ordering differences
    ---------------------------------------------------------

    Key: HIVE-9181
    URL: https://issues.apache.org/jira/browse/HIVE-9181
    Project: Hive
    Issue Type: Sub-task
    Components: Tests
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-9181.patch


    (1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different order in Java 8 vs Java 7. This leads to differences in EXPLAIN output related to {{MapWork}}.
    (2) Dbproperties order for a "desc database" query is non-deterministic at the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are un-ordered).
    {code}
    create database test_db with dbproperties ('key1' = 'value1', 'key2' = 'value2');
    desc database extended test_db;
    {code}
    At client side, we serialize the dbproperties map returned from HMS as string and write it to a temp file. The {{FetchTask}} then reads the temp file and returns the string back to the user. In test environment, for consistent ordering, we should sort the map before it is serialized to the temp file.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Brock Noland (JIRA) at Dec 20, 2014 at 5:18 pm
    [ https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Brock Noland updated HIVE-9181:
    -------------------------------
         Attachment: HIVE-9181.patch
    Fix SkewJoinOptimizer related Java 8 ordering differences
    ---------------------------------------------------------

    Key: HIVE-9181
    URL: https://issues.apache.org/jira/browse/HIVE-9181
    Project: Hive
    Issue Type: Sub-task
    Components: Tests
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-9181.patch, HIVE-9181.patch


    (1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different order in Java 8 vs Java 7. This leads to differences in EXPLAIN output related to {{MapWork}}.
    (2) Dbproperties order for a "desc database" query is non-deterministic at the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are un-ordered).
    {code}
    create database test_db with dbproperties ('key1' = 'value1', 'key2' = 'value2');
    desc database extended test_db;
    {code}
    At client side, we serialize the dbproperties map returned from HMS as string and write it to a temp file. The {{FetchTask}} then reads the temp file and returns the string back to the user. In test environment, for consistent ordering, we should sort the map before it is serialized to the temp file.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Hive QA (JIRA) at Dec 20, 2014 at 8:15 pm
    [ https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254946#comment-14254946 ]

    Hive QA commented on HIVE-9181:
    -------------------------------



    {color:red}Overall{color}: -1 at least one tests failed

    Here are the results of testing the latest attachment:
    https://issues.apache.org/jira/secure/attachment/12688539/HIVE-9181.patch

    {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 6718 tests executed
    *Failed tests:*
    {noformat}
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_db_owner
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_owner_actions_db
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_database_properties
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_describe_database_json
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_virtual_column
    org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
    org.apache.hive.hcatalog.cli.TestSemanticAnalysis.testDescDB
    {noformat}

    Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2155/testReport
    Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2155/console
    Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2155/

    Messages:
    {noformat}
    Executing org.apache.hive.ptest.execution.PrepPhase
    Executing org.apache.hive.ptest.execution.ExecutionPhase
    Executing org.apache.hive.ptest.execution.ReportingPhase
    Tests exited with: TestsFailedException: 8 tests failed
    {noformat}

    This message is automatically generated.

    ATTACHMENT ID: 12688539 - PreCommit-HIVE-TRUNK-Build
    Fix SkewJoinOptimizer related Java 8 ordering differences
    ---------------------------------------------------------

    Key: HIVE-9181
    URL: https://issues.apache.org/jira/browse/HIVE-9181
    Project: Hive
    Issue Type: Sub-task
    Components: Tests
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-9181.patch, HIVE-9181.patch


    (1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different order in Java 8 vs Java 7. This leads to differences in EXPLAIN output related to {{MapWork}}.
    (2) Dbproperties order for a "desc database" query is non-deterministic at the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are un-ordered).
    {code}
    create database test_db with dbproperties ('key1' = 'value1', 'key2' = 'value2');
    desc database extended test_db;
    {code}
    At client side, we serialize the dbproperties map returned from HMS as string and write it to a temp file. The {{FetchTask}} then reads the temp file and returns the string back to the user. In test environment, for consistent ordering, we should sort the map before it is serialized to the temp file.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mohit Sabharwal (JIRA) at Dec 21, 2014 at 3:37 am
    [ https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255030#comment-14255030 ]

    Mohit Sabharwal commented on HIVE-9181:
    ---------------------------------------

    Attaching patch with added check for when database params is null.

    The nonmr_fetch.q and virtual_column.q failures are unrelated (these failing for other patches as well).

    Fix SkewJoinOptimizer related Java 8 ordering differences
    ---------------------------------------------------------

    Key: HIVE-9181
    URL: https://issues.apache.org/jira/browse/HIVE-9181
    Project: Hive
    Issue Type: Sub-task
    Components: Tests
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-9181.patch, HIVE-9181.patch


    (1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different order in Java 8 vs Java 7. This leads to differences in EXPLAIN output related to {{MapWork}}.
    (2) Dbproperties order for a "desc database" query is non-deterministic at the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are un-ordered).
    {code}
    create database test_db with dbproperties ('key1' = 'value1', 'key2' = 'value2');
    desc database extended test_db;
    {code}
    At client side, we serialize the dbproperties map returned from HMS as string and write it to a temp file. The {{FetchTask}} then reads the temp file and returns the string back to the user. In test environment, for consistent ordering, we should sort the map before it is serialized to the temp file.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mohit Sabharwal (JIRA) at Dec 21, 2014 at 3:38 am
    [ https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Mohit Sabharwal updated HIVE-9181:
    ----------------------------------
         Attachment: HIVE-9181.1.patch
    Fix SkewJoinOptimizer related Java 8 ordering differences
    ---------------------------------------------------------

    Key: HIVE-9181
    URL: https://issues.apache.org/jira/browse/HIVE-9181
    Project: Hive
    Issue Type: Sub-task
    Components: Tests
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-9181.1.patch, HIVE-9181.patch, HIVE-9181.patch


    (1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different order in Java 8 vs Java 7. This leads to differences in EXPLAIN output related to {{MapWork}}.
    (2) Dbproperties order for a "desc database" query is non-deterministic at the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are un-ordered).
    {code}
    create database test_db with dbproperties ('key1' = 'value1', 'key2' = 'value2');
    desc database extended test_db;
    {code}
    At client side, we serialize the dbproperties map returned from HMS as string and write it to a temp file. The {{FetchTask}} then reads the temp file and returns the string back to the user. In test environment, for consistent ordering, we should sort the map before it is serialized to the temp file.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Hive QA (JIRA) at Dec 21, 2014 at 5:42 am
    [ https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255050#comment-14255050 ]

    Hive QA commented on HIVE-9181:
    -------------------------------



    {color:red}Overall{color}: -1 at least one tests failed

    Here are the results of testing the latest attachment:
    https://issues.apache.org/jira/secure/attachment/12688563/HIVE-9181.1.patch

    {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6718 tests executed
    *Failed tests:*
    {noformat}
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
    org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_virtual_column
    org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
    {noformat}

    Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2157/testReport
    Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2157/console
    Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2157/

    Messages:
    {noformat}
    Executing org.apache.hive.ptest.execution.PrepPhase
    Executing org.apache.hive.ptest.execution.ExecutionPhase
    Executing org.apache.hive.ptest.execution.ReportingPhase
    Tests exited with: TestsFailedException: 3 tests failed
    {noformat}

    This message is automatically generated.

    ATTACHMENT ID: 12688563 - PreCommit-HIVE-TRUNK-Build
    Fix SkewJoinOptimizer related Java 8 ordering differences
    ---------------------------------------------------------

    Key: HIVE-9181
    URL: https://issues.apache.org/jira/browse/HIVE-9181
    Project: Hive
    Issue Type: Sub-task
    Components: Tests
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-9181.1.patch, HIVE-9181.patch, HIVE-9181.patch


    (1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different order in Java 8 vs Java 7. This leads to differences in EXPLAIN output related to {{MapWork}}.
    (2) Dbproperties order for a "desc database" query is non-deterministic at the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are un-ordered).
    {code}
    create database test_db with dbproperties ('key1' = 'value1', 'key2' = 'value2');
    desc database extended test_db;
    {code}
    At client side, we serialize the dbproperties map returned from HMS as string and write it to a temp file. The {{FetchTask}} then reads the temp file and returns the string back to the user. In test environment, for consistent ordering, we should sort the map before it is serialized to the temp file.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Mohit Sabharwal (JIRA) at Dec 21, 2014 at 7:02 pm
    [ https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255244#comment-14255244 ]

    Mohit Sabharwal commented on HIVE-9181:
    ---------------------------------------

    Test failures are present in other recent patch runs as well (like HIVE-8938), so these are unrelated to patch.
    Fix SkewJoinOptimizer related Java 8 ordering differences
    ---------------------------------------------------------

    Key: HIVE-9181
    URL: https://issues.apache.org/jira/browse/HIVE-9181
    Project: Hive
    Issue Type: Sub-task
    Components: Tests
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-9181.1.patch, HIVE-9181.patch, HIVE-9181.patch


    (1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different order in Java 8 vs Java 7. This leads to differences in EXPLAIN output related to {{MapWork}}.
    (2) Dbproperties order for a "desc database" query is non-deterministic at the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are un-ordered).
    {code}
    create database test_db with dbproperties ('key1' = 'value1', 'key2' = 'value2');
    desc database extended test_db;
    {code}
    At client side, we serialize the dbproperties map returned from HMS as string and write it to a temp file. The {{FetchTask}} then reads the temp file and returns the string back to the user. In test environment, for consistent ordering, we should sort the map before it is serialized to the temp file.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Szehon Ho (JIRA) at Dec 22, 2014 at 7:18 pm
    [ https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256065#comment-14256065 ]

    Szehon Ho commented on HIVE-9181:
    ---------------------------------

    +1 Looks good to me, thanks for the tough work.
    Fix SkewJoinOptimizer related Java 8 ordering differences
    ---------------------------------------------------------

    Key: HIVE-9181
    URL: https://issues.apache.org/jira/browse/HIVE-9181
    Project: Hive
    Issue Type: Sub-task
    Components: Tests
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-9181.1.patch, HIVE-9181.patch, HIVE-9181.patch


    (1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different order in Java 8 vs Java 7. This leads to differences in EXPLAIN output related to {{MapWork}}.
    (2) Dbproperties order for a "desc database" query is non-deterministic at the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are un-ordered).
    {code}
    create database test_db with dbproperties ('key1' = 'value1', 'key2' = 'value2');
    desc database extended test_db;
    {code}
    At client side, we serialize the dbproperties map returned from HMS as string and write it to a temp file. The {{FetchTask}} then reads the temp file and returns the string back to the user. In test environment, for consistent ordering, we should sort the map before it is serialized to the temp file.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Brock Noland (JIRA) at Dec 22, 2014 at 7:25 pm
    [ https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256076#comment-14256076 ]

    Brock Noland commented on HIVE-9181:
    ------------------------------------

    You beat me to it :)
    Fix SkewJoinOptimizer related Java 8 ordering differences
    ---------------------------------------------------------

    Key: HIVE-9181
    URL: https://issues.apache.org/jira/browse/HIVE-9181
    Project: Hive
    Issue Type: Sub-task
    Components: Tests
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Attachments: HIVE-9181.1.patch, HIVE-9181.patch, HIVE-9181.patch


    (1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different order in Java 8 vs Java 7. This leads to differences in EXPLAIN output related to {{MapWork}}.
    (2) Dbproperties order for a "desc database" query is non-deterministic at the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are un-ordered).
    {code}
    create database test_db with dbproperties ('key1' = 'value1', 'key2' = 'value2');
    desc database extended test_db;
    {code}
    At client side, we serialize the dbproperties map returned from HMS as string and write it to a temp file. The {{FetchTask}} then reads the temp file and returns the string back to the user. In test environment, for consistent ordering, we should sort the map before it is serialized to the temp file.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Brock Noland (JIRA) at Dec 24, 2014 at 2:35 pm
    [ https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Brock Noland updated HIVE-9181:
    -------------------------------
            Resolution: Fixed
         Fix Version/s: 0.15.0
                Status: Resolved (was: Patch Available)

    Thank you [~mohitsabharwal] for the patch and [~szehon] for the review! I have committed this to trunk!
    Fix SkewJoinOptimizer related Java 8 ordering differences
    ---------------------------------------------------------

    Key: HIVE-9181
    URL: https://issues.apache.org/jira/browse/HIVE-9181
    Project: Hive
    Issue Type: Sub-task
    Components: Tests
    Reporter: Mohit Sabharwal
    Assignee: Mohit Sabharwal
    Fix For: 0.15.0

    Attachments: HIVE-9181.1.patch, HIVE-9181.patch, HIVE-9181.patch


    (1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different order in Java 8 vs Java 7. This leads to differences in EXPLAIN output related to {{MapWork}}.
    (2) Dbproperties order for a "desc database" query is non-deterministic at the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are un-ordered).
    {code}
    create database test_db with dbproperties ('key1' = 'value1', 'key2' = 'value2');
    desc database extended test_db;
    {code}
    At client side, we serialize the dbproperties map returned from HMS as string and write it to a temp file. The {{FetchTask}} then reads the temp file and returns the string back to the user. In test environment, for consistent ordering, we should sort the map before it is serialized to the temp file.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshive, hadoop
postedDec 20, '14 at 2:13a
activeDec 24, '14 at 2:35p
posts13
users1
websitehive.apache.org

1 user in discussion

Brock Noland (JIRA): 13 posts

People

Translate

site design / logo © 2021 Grokbase