[Pig-dev] [jira] [Created] (PIG-2125) Make Pig work with hadoop .NEXT

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Description: We need to make Pig work with hadoop .NEXT, the svn branch currently is: https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-1.patch

Attach initial patch PIG-2125-1.patch solving some issues we find. There are still areas under working, such as illustrator, MiniCluster, MergedJoin.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049942#comment-13049942 ]

Dmitriy V. Ryaboy commented on PIG-2125:
----------------------------------------

are you guys planning on skipping 22 altogether?

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049970#comment-13049970 ]

Olga Natkovich commented on PIG-2125:
-------------------------------------

We don't have any current plans to use 22 at Yahoo or for Yahoo engineers to do the port. But of course a patch is welcome from anybody interested in taking this project on.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-2.patch

PIG-2125-2.patch tested against MR-279 branch. All Pig end-to-end test pass. Unit test still fail. Zebra test still have a few failures.

The patch borrow some ideas from PIG-924. However, a dynamic shims layer is not possible due to some static dependencies. Here we use a static shims layer, so for different hadoop version, we need to recompile.

Since hadoop.Next maven artifacts is not available, so we need to simulate it by copying all libraries to lib.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-3.patch

PIG-2125-3.patch is additional patch to make Zebra work.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064053#comment-13064053 ]

Alan Gates commented on PIG-2125:
---------------------------------

In general the patch looks good. I have a few questions/comments.

# How does one build a 0.23 compatible version of Pig? Do you have to download the 0.23 jar yourself? It looked like if you did this, build.xml would manage the rest properly. The answer to this question will need to be added to the release notes field of this JIRA.
# Javadocs explaining what the classes do would be very helpful. PigMapReduce, PigMapBase, HadoopShims all could use class level comments. HadoopShims in particular should have extensive comments on how the shims layer works and what needs to be shimmed.
# I wonder if HadoopShims should go in a separate package. I know it's just one class, but it performs a unique function. This would also allow many of the design and build comments I referenced in the last point to be covered in a package level doc. I don't feel strongly about this, so feel free to keep it where it is if you think that's better.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064268#comment-13064268 ]

Daniel Dai commented on PIG-2125:
---------------------------------

bq. 1. How does one build a 0.23 compatible version of Pig? Do you have to download the 0.23 jar yourself? It looked like if you did this, build.xml would manage the rest properly. The answer to this question will need to be added to the release notes field of this JIRA.
They need to copy all hadoop 23 jars and dependencies into lib. This step should go away once hadoop 23 publish maven artifacts
bq. 2. Javadocs explaining what the classes do would be very helpful. PigMapReduce, PigMapBase, HadoopShims all could use class level comments. HadoopShims in particular should have extensive comments on how the shims layer works and what needs to be shimmed.
Sure. I will add
bq. 3. I wonder if HadoopShims should go in a separate package. I know it's just one class, but it performs a unique function. This would also allow many of the design and build comments I referenced in the last point to be covered in a package level doc. I don't feel strongly about this, so feel free to keep it where it is if you think that's better.
How about org.apache.pig.backend.hadoop.shims?

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-4.patch

PIG-2125-4.patch merge PIG-2125-2.patch and PIG-2125-3.PATCH, address comments from Alan, fix unit test failures and test-patch failures.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067428#comment-13067428 ]

Daniel Dai commented on PIG-2125:
---------------------------------

PIG-2125-4.patch committed to both trunk and 0.9 branch. This is the first phase work (mapreduce and local mode end-to-end), I will continue work on the next phase (unit test).

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-5.patch

There are some piggybank and zebra test failures, attach PIG-2125-5.patch for the fix.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068739#comment-13068739 ]

Richard Ding commented on PIG-2125:
-----------------------------------

+1

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068744#comment-13068744 ]

Daniel Dai commented on PIG-2125:
---------------------------------

PIG-2125-5.patch committed to both trunk and 0.9 branch.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069124#comment-13069124 ]

Daniel Dai commented on PIG-2125:
---------------------------------

Steps to build pig.jar for hadoop 23:
1. change "build.xml", modify "hadoopversion" to "23"
2. change "ivy.xml", remove the lines:
<dependency org="org.apache.hadoop" name="hadoop-core" rev="${hadoop-core.version}"
conf="compile->default;test->default"/>
<dependency org="org.apache.hadoop" name="hadoop-test" rev="${hadoop-test.version}"
conf="compile->default;test->default"/>
3. Build hadoop 23, copy hadoop 23 libraries into lib, here is a list:
aopalliance-1.0.jar
commons-cli-1.2.jar
commons-codec-1.4.jar
commons-collections-3.2.1.jar
commons-configuration-1.6.jar
commons-httpclient-3.1.jar
guice-2.0.jar
guice-servlet-2.0.jar
hadoop-common-0.22.0-SNAPSHOT.jar
hadoop-common-test-0.22.0-SNAPSHOT.jar
hadoop-hdfs-0.22.0-SNAPSHOT.jar
hadoop-hdfs-test-0.22.0-SNAPSHOT.jar
hadoop-mapreduce-client-app-1.0-SNAPSHOT.jar
hadoop-mapreduce-client-common-1.0-SNAPSHOT.jar
hadoop-mapreduce-client-core-1.0-SNAPSHOT.jar
hadoop-mapreduce-client-hs-1.0-SNAPSHOT.jar
hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT-tests.jar
hadoop-mapreduce-client-shuffle-1.0-SNAPSHOT.jar
hadoop-metrics2-0.22.0-SNAPSHOT.jar
jetty-6.1.26.jar
log4j-1.2.16.jar
protobuf-java-2.4.0a.jar
servlet-api-2.5-20081211.jar
slf4j-api-1.6.1.jar
slf4j-log4j12-1.6.1.jar
yarn-api-1.0-SNAPSHOT.jar
yarn-common-1.0-SNAPSHOT.jar
yarn-server-common-1.0-SNAPSHOT.jar
yarn-server-nodemanager-1.0-SNAPSHOT.jar
yarn-server-resourcemanager-1.0-SNAPSHOT.jar
yarn-server-tests-1.0-SNAPSHOT-tests.jar

4. Then, do an "ant clean jar"
5. You may also need to recompile your udf with the new hadoop libraries.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: ContextFactory.java

Attach ContextFactory.java which eventually will be in hadoop's codebase. But for now, we need to put it in Pig's source tree in order to compile.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13078313#comment-13078313 ]

Daniel Dai commented on PIG-2125:
---------------------------------

Put ContextFactory.java into $PIG_HOME/src/org/apache/hadoop/mapreduce, then compile.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-6.patch

PIG-2125-6.patch fixed many of core unit test failures for hadoop 23. There are still 20 failures left. I am working with Hadoop 23 team on these issues:
I am running Pig unit tests with hadoop 23. Here is some issues I current find:

1. Stability issue in MiniMRCluster. Some are intermittent, some fails consistently. Here is the stack I saw:
java.io.FileNotFoundException:
file:/home/daniel/pig/target/PigMiniCluster/PigMiniCluster-localDir/usercache/daniel/appcache/application_1313021517794_0001/container_1313021517794_0002_000001.tokens
at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:187)
at org.apache.hadoop.fs.DelegateToFileSystem.open(DelegateToFileSystem.java:150)
at org.apache.hadoop.fs.AbstractFileSystem.open(AbstractFileSystem.java:595)
at org.apache.hadoop.fs.FilterFs.open(FilterFs.java:188)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:736)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:733)
at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2278)
at org.apache.hadoop.fs.FileContext.open(FileContext.java:733)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:137)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:85)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:779)

If success, I didn't see it in the log

2. Couter issue. Seems most counter tests fail.
JobClient.getJob(job.getAssignedJobID()) always give me null.

3. MiniMRYarnCluster.getConfig() give me this entry:
"mapreduce.job.hdfs-servers=${fs.default.name}", which should have done the variable substitution.

4. Failure to start MiniHBaseCluster, all HBase tests fail.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-7.patch

PIG-2125-7.patch catch up with Pig trunk, and compilable against Hadoop 0.23 brach.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: e2e-hadoop23.patch

e2e-hadoop23.patch is a separable patch from PIG-2125-7.patch which enable e2e test only.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: (was: PIG-2125-7.patch)

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-7.patch

Missing one file in PIG-2125-7.patch, reattach.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-7_0.9.patch

The same patch for 0.9 branch.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119812#comment-13119812 ]

Arun C Murthy commented on PIG-2125:
------------------------------------

I've committed MAPREDUCE-3138 i.e. the ContextFactory patch to Hadoop MapReduce. Thanks.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Weise updated PIG-2125:
------------------------------

Attachment: PIG-2125-ivy-0.9.patch

This patch is to support hadoopversion=23 to resolve dependencies through maven.

Note that currently mapreduce does not have the jar files in maven, so you will still need to copy those into lib/

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Weise updated PIG-2125:
------------------------------

Attachment: PIG-2125-buildxml-0.9.patch

PIG-2125-buildxml-0.9.patch - Changes to build.xml for buildJar target. Verified changes for .20 build via diff of jar tv ... before and after.

File list for .23 fat jar needs to be tuned once all maven dependencies are in place and tests compile/pass.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-commitJob.patch

PIG-2125-commitJob.patch is to address incompatibility change introduced in MAPREDUCE-2702

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-commitJob.patch

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: (was: PIG-2125-commitJob.patch)

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126756#comment-13126756 ]

Daniel Dai commented on PIG-2125:
---------------------------------

Tests pass for PIG-2125-commitJob.patch. No test included since it solve the problem only in hadoop 23, which is not released.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126785#comment-13126785 ]

Olga Natkovich commented on PIG-2125:
-------------------------------------

Did the tests pass for both .Next and 20.x version?

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126928#comment-13126928 ]

Daniel Dai commented on PIG-2125:
---------------------------------

All tests done on 20.X. I only did sanity tests on .Next, since I don't have the facility to run the whole thing currently.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126945#comment-13126945 ]

Olga Natkovich commented on PIG-2125:
-------------------------------------

I think that is good enough. As long as we do not regress on the current Hadoop I think we are good. Just need a review then.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126955#comment-13126955 ]

Thejas M Nair commented on PIG-2125:
------------------------------------

+1 for PIG-2125-commitJob.patch

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-zebra.patch

PIG-2125-zebra.patch fix zebra test failures introduced by PIG-2125-commitJob.patch

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Weise updated PIG-2125:
------------------------------

Attachment: PIG-2125-ivy-0.9-2.patch

Update to the mapreduce dependencies (replaces previous patch).

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Weise updated PIG-2125:
------------------------------

Attachment: PIG-2125-ivy-0.9-3.patch

Modified ivy patch to pull compile-test dependencies correctly. Currently the snapshots are outdated, compile-test should pass once they are refreshed.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Weise updated PIG-2125:
------------------------------

Attachment: (was: PIG-2125-ivy-0.9.patch)

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Weise updated PIG-2125:
------------------------------

Attachment: (was: PIG-2125-ivy-0.9-2.patch)

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-8.patch

PIG-2125-8.patch makes Pig compile and run against both 23 and 20 using maven artifacts. The only exception is hadoop-mapreduce.jar, which hadoop still fail to publish, and we need to copy it into lib. PIG-2125-8.patch include all changes in PIG-2125-7.patch, PIG-2125-buildxml-0.9.patch and PIG-2125-ivy-0.9-3.patch.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-9_0.9.patch
PIG-2125-9.patch

PIG-2125-9.patch & PIG-2125-9_0.9.patch compile and run Pig without any manual intervention.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-9_0.9.patch

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: (was: PIG-2125-9_0.9.patch)

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141787#comment-13141787 ]

Thomas Weise commented on PIG-2125:
-----------------------------------

+1 ant compile-test is good on branch-0.9 now

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Attachment: PIG-2125-10_0.9.patch
PIG-2125-10.patch

PIG-2125-10.patch, PIG-2125-10_0.9.patch fix test-patch warnings:
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 46 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] -1 release audit. The applied patch generated 462 release audit warnings (more than the trunk's current 447 warnings).

All new files has proper header.

Unit test pass.

Committed to trunk, 0.10 branch and 0.9 branch.

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142483#comment-13142483 ]

Daniel Dai commented on PIG-2125:
---------------------------------

After PIG-2125-10.patch, we can compile and run end-to-end tests on 23. There are still ~20 unit tests fail on 23 and we are working on that.

Since all patches attached to the Jira is committed, and this thread is becoming big and confusing, I would suggest to close this ticket and create a separate ticket to track unit test fixes for hadoop 23, agree?

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Affects Version/s: 0.9.1
Fix Version/s: 0.9.2

[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2125:
----------------------------

Resolution: Fixed
Status: Resolved (was: Patch Available)

Move the remaining work to PIG-2347.