FAQ
Author: jssarma
Date: Fri Dec 10 19:07:27 2010
New Revision: 1044481

URL: http://svn.apache.org/viewvc?rev=1044481&view=rev
Log:
HIVE-1846 Hive should not assume that local mode mappers run in same JVM (rvadali via jssarma)

Modified:
hive/trunk/CHANGES.txt
hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java

Modified: hive/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/hive/trunk/CHANGES.txt?rev=1044481&r1=1044480&r2=1044481&view=diff
==============================================================================
--- hive/trunk/CHANGES.txt (original)
+++ hive/trunk/CHANGES.txt Fri Dec 10 19:07:27 2010
@@ -602,6 +602,9 @@ Trunk - Unreleased
HIVE-1843 Add an option in dynamic partition inserts to throw an error if
0 partitions are created (Ning Zhang via namit)

+ HIVE-1846 Hive should not assume that local mode mappers run in same JVM
+ (rvadali via jssarma)
+
TESTS

HIVE-1464. improve test query performance

Modified: hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
URL: http://svn.apache.org/viewvc/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java?rev=1044481&r1=1044480&r2=1044481&view=diff
==============================================================================
--- hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java (original)
+++ hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java Fri Dec 10 19:07:27 2010
@@ -169,7 +169,15 @@ public final class Utilities {
assert jobID != null;
gWork = gWorkMap.get(jobID);
if (gWork == null) {
- InputStream in = new FileInputStream("HIVE_PLAN" + jobID);
+ String jtConf = HiveConf.getVar(job, HiveConf.ConfVars.HADOOPJT);
+ String path;
+ if (jtConf == "local") {
+ String planPath = HiveConf.getVar(job, HiveConf.ConfVars.PLAN);
+ path = new Path(planPath).toUri().getPath();
+ } else {
+ path = "HIVE_PLAN" + jobID;
+ }
+ InputStream in = new FileInputStream(path);
MapredWork ret = deserializeMapRedWork(in, job);
gWork = ret;
gWork.initialize();
@@ -302,15 +310,15 @@ public final class Utilities {
Path planPath = new Path(hiveScratchDir, jobID);
HiveConf.setVar(job, HiveConf.ConfVars.PLAN, planPath.toUri().toString());

+ // use the default file system of the job
+ FileSystem fs = planPath.getFileSystem(job);
+ FSDataOutputStream out = fs.create(planPath);
+ serializeMapRedWork(w, out);
+
// Serialize the plan to the default hdfs instance
// Except for hadoop local mode execution where we should be
// able to get the plan directly from the cache
if (!HiveConf.getVar(job, HiveConf.ConfVars.HADOOPJT).equals("local")) {
- // use the default file system of the job
- FileSystem fs = planPath.getFileSystem(job);
- FSDataOutputStream out = fs.create(planPath);
- serializeMapRedWork(w, out);
-
// Set up distributed cache
DistributedCache.createSymlink(job);
String uriWithLink = planPath.toUri().toString() + "#HIVE_PLAN" + jobID;

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 1 | next ›
Discussion Overview
groupcommits @
categorieshive, hadoop
postedDec 10, '10 at 7:07p
activeDec 10, '10 at 7:07p
posts1
users1
websitehive.apache.org

1 user in discussion

Jssarma: 1 post

People

Translate

site design / logo © 2021 Grokbase