Grokbase Groups Hive dev April 2016
FAQ
Update on moving Hive2 tests to JDK8: I've addressed almost all test
failures in HIVE-13547 on java8 branch. There is one remaining
open item (HIVE-13834) which is currently assigned. Given current
state of flaky test runs, there might be few more.

I will work with Sergio to merge the test fixes to master and switch
the Hive2 pre-commit job to use JDK8, hopefully sometime tomorrow.

After Hive2 tests switch, if your patch sees ordering related test
failures in pre-commit run, it's likely JDK version related and you'll
need to build & re-run the test using JDK8. Number of such tests
should be relatively small.
On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal wrote:

Created HIVE-13547 to track switching 2x tests to JDK8.
On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena wrote:

I agree with such change as JDK7 is not longer supported.

Changes on Jenkins and Hive PTest shouldn't be hard. We just need to
replace the path from java7 to java8. But I think we should fix all JDK8
issues or most of them before doing the change or we will end up having a
lot of failures on all JIRAs running pre-commit tests.

+1 with the change.
On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth wrote:

Option 3 sounds good. I'd ideally like to make JDK8 the minimum
requirement
soon as well.
On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho wrote:

Sounds like a good plan, +1

On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <mohit@cloudera.com>
wrote:
Thanks, Ashutosh. Makes sense to keep the source and target as 1.7
since
we're not using any JDK8 specific features yet. So, option (3)
essentially
just means using JDK8 exclusively to build & test Hive2.

On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
hashutosh@apache.org>
wrote:
Hi Mohit,

I also think option 3 makes sense. We should strive to keep test
matrix
small so that we can do fast QA runs for dev patches.
We can just use jdk7 to build & test Hive1 and jdk8 to build &
test
Hive2.
However, I am not sure of explicitly dropping support altogether
for
jdk7
on Hive2. We should make sure that in pom.xml java source & target
compatibility is still 1.7 (which already is the case currently)
so
that
Hive2 is still compatible with jdk7. Unit tests as I said we can
run
on
jdk8.

Thanks,
Ashutosh

On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
mohit@cloudera.com>
wrote:
Hi folks,

Oracle EOL'ed (ended public updates) for Java 7 in April 2015.
In order to support Java 8, we fixed quite a bunch of tests in
HIVE-8607 (*) early last year. However, since our pre-commit
tests
run
on JDK7 only, any JDK8 test failures are getting ignored. As a
result,
the count has crept up
<
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
back
from zero to 125.

Some options to address this:

(1) Run pre-commit tests on both JDK7 and JDK8, in both 1.x
and 2.x: This will further slow down the pre-commit run.

(2) Alternate pre-commit test runs between JDK7 and JDK8 (in
both 1.x and 2.x): It's a cheap hack. More messy, though
failures
won't get ignored.

(3) Drop support for JDK7 in Hive 2.x, i.e. build and test Hive
2.x
on
JDK8 only. For Hive 1.x, continue running JDK7 pre-commit tests.
The pros are:
- Simple test matrix.
- No need to slow down test run or maintain version specific
golden
files
(**).
- 2.x looks like the logical place to move to JDK8.
- Users transitioning to JDK8 for all other services do not
have to
maintain
multiple java versions on the cluster.

Option (3) looks most attractive to me.

Moving to JDK8 also lines us up better for Java 9, which is on
the
horizon (Oracle will end public updates for Java 8 in Sep 2017)

Around 100 of the latest crop of failures are due to one cause
(HIVE-13409).
I can take a pass at triaging the rest if there is consensus
around
what
to do overall.

Thanks,
Mohit

(*) Most test failures are due to JDK8 using a different hash
function
for
HashMap compared to JDK7. This results in (mostly benign, but
hard
to debug) ordering differences in q-file output related to
serialization
order of map entries, numbering of stages in query plan, etc.

(**) In some cases, hash function related ordering differences
originate
in external libraries like Avro, antlr, json ,Thrift's map, etc.
for
which
code
changes are either more involved or led to more test failures.
To
address
this,
we added support for version specific golden files (HIVE-9109).
Currently,
there
are ~40 golden files with ".java1.8.out" extension.

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 8 of 18 | next ›
Discussion Overview
groupdev @
categorieshive, hadoop
postedApr 8, '16 at 11:34p
activeJun 5, '16 at 9:07a
posts18
users9
websitehive.apache.org

People

Translate

site design / logo © 2021 Grokbase