[Hive-user] hive errors

[Hive-user] hive errors

	Yekesa Kosuru
	Mar 18, 2010 at 3:46 am

Hi,

I am a newbie to hiveql. I love what I see but I am having issues with simple hive queries. Below is the transcript. Can someone tell me why reducers going backwards (goes to 67% and then drops to 57%) and everytime I see this, the query finally errors out.

hive> from

(
select S.v k, sum (0.5 * e.val * S.r) Val from es e join journal_ratio S
on (e.k=S.k)
where e.ca = 1 group by S.v
) t
insert overwrite table temp select t.*;

Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Defaulting to jobconf value of: 7
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201003132100_0003, Tracking URL = http://dacn0:50030/jobdetails.jsp?jobid=job_201003132100_0003
Kill Command = /usr/local/hadoop/0.20.1/bin/../bin/hadoop job -Dmapred.job.tracker=dacn0:54311 -kill job_201003132100_0003
2010-03-18 05:21:38,618 Stage-1 map = 0%, reduce = 0%
2010-03-18 05:21:56,709 Stage-1 map = 6%, reduce = 0%
2010-03-18 05:21:57,717 Stage-1 map = 9%, reduce = 0%
2010-03-18 05:21:59,731 Stage-1 map = 13%, reduce = 0%
2010-03-18 05:22:00,737 Stage-1 map = 15%, reduce = 0%
2010-03-18 05:22:02,748 Stage-1 map = 19%, reduce = 0%
2010-03-18 05:22:03,755 Stage-1 map = 21%, reduce = 0%
2010-03-18 05:22:05,768 Stage-1 map = 23%, reduce = 0%
2010-03-18 05:22:06,777 Stage-1 map = 28%, reduce = 0%
2010-03-18 05:22:07,785 Stage-1 map = 32%, reduce = 0%
2010-03-18 05:22:09,798 Stage-1 map = 34%, reduce = 0%
2010-03-18 05:22:10,805 Stage-1 map = 37%, reduce = 0%
2010-03-18 05:22:12,820 Stage-1 map = 39%, reduce = 0%
2010-03-18 05:22:13,842 Stage-1 map = 41%, reduce = 0%
2010-03-18 05:22:14,850 Stage-1 map = 43%, reduce = 0%
2010-03-18 05:22:15,858 Stage-1 map = 45%, reduce = 1%
2010-03-18 05:22:17,872 Stage-1 map = 51%, reduce = 2%
2010-03-18 05:22:18,879 Stage-1 map = 53%, reduce = 5%
2010-03-18 05:22:20,893 Stage-1 map = 57%, reduce = 6%
2010-03-18 05:22:21,900 Stage-1 map = 60%, reduce = 8%
2010-03-18 05:22:22,907 Stage-1 map = 60%, reduce = 10%
2010-03-18 05:22:23,915 Stage-1 map = 64%, reduce = 11%
2010-03-18 05:22:24,922 Stage-1 map = 68%, reduce = 12%
2010-03-18 05:22:25,930 Stage-1 map = 70%, reduce = 12%
2010-03-18 05:22:26,937 Stage-1 map = 71%, reduce = 13%
2010-03-18 05:22:27,944 Stage-1 map = 74%, reduce = 13%
2010-03-18 05:22:28,952 Stage-1 map = 77%, reduce = 13%
2010-03-18 05:22:29,960 Stage-1 map = 79%, reduce = 14%
2010-03-18 05:22:30,968 Stage-1 map = 81%, reduce = 14%
2010-03-18 05:22:31,975 Stage-1 map = 83%, reduce = 14%
2010-03-18 05:22:32,983 Stage-1 map = 88%, reduce = 14%
2010-03-18 05:22:33,991 Stage-1 map = 91%, reduce = 17%
2010-03-18 05:22:35,001 Stage-1 map = 92%, reduce = 18%
2010-03-18 05:22:36,009 Stage-1 map = 96%, reduce = 19%
2010-03-18 05:22:37,017 Stage-1 map = 97%, reduce = 19%
2010-03-18 05:22:39,029 Stage-1 map = 99%, reduce = 19%
2010-03-18 05:22:40,037 Stage-1 map = 100%, reduce = 21%
2010-03-18 05:22:42,050 Stage-1 map = 100%, reduce = 23%
2010-03-18 05:22:43,058 Stage-1 map = 100%, reduce = 26%
2010-03-18 05:22:46,076 Stage-1 map = 100%, reduce = 28%
2010-03-18 05:22:48,091 Stage-1 map = 100%, reduce = 30%
2010-03-18 05:22:49,098 Stage-1 map = 100%, reduce = 36%
2010-03-18 05:22:52,117 Stage-1 map = 100%, reduce = 46%
2010-03-18 05:22:54,129 Stage-1 map = 100%, reduce = 52%
2010-03-18 05:22:55,136 Stage-1 map = 100%, reduce = 62%
2010-03-18 05:22:57,148 Stage-1 map = 100%, reduce = 67%
2010-03-18 05:23:51,425 Stage-1 map = 100%, reduce = 57%

after this, it continues

2010-03-18 05:24:03,487 Stage-1 map = 100%, reduce = 61%
2010-03-18 05:24:09,520 Stage-1 map = 100%, reduce = 62%
2010-03-18 05:24:12,539 Stage-1 map = 100%, reduce = 67%
2010-03-18 05:24:21,592 Stage-1 map = 100%, reduce = 68%
2010-03-18 05:25:08,823 Stage-1 map = 100%, reduce = 58%
2010-03-18 05:26:09,126 Stage-1 map = 100%, reduce = 58%

jumps around until it errors out as follows:

2010-03-18 05:38:23,638 Stage-1 map = 100%, reduce = 50%
2010-03-18 05:38:43,786 Stage-1 map = 100%, reduce = 40%
2010-03-18 05:38:46,802 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201003132100_0003 with errors

Failed tasks with most(4) failures :
Task URL: http://dacn0:50030/taskdetails.jsp?jobid=job_201003132100_0003&tipid=task_201003132100_0003_r_000003

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver

Hive version = 0.20.1, Java = 1.6

Is this a user error? as in should I add group by, or sort by or cluster by clauses to the query ? Query was run on the master name node, and there are 4 data nodes. With other queries I have seen the map % progresses and drops and eventually errors out.

Any help is appreciated.

Thanks

Vissu

Search Discussions

3 responses
Oldest
Nested

Mafish Liu

at Mar 18, 2010 at 4:02 am

⇧

Yekesa:

You'r reducer meet some errors while procesing.

You should check
http://dacn0:50030/taskdetails.jsp?jobid=job_201003132100_0003&tipid=task_201003132100_0003_r_000003
for more details of the failed reduce task.

Please post the error log here for further analyzing.

Mafish

2010/3/18 Yekesa Kosuru <vissuyk@yahoo.com>:

(
select S.v k, sum (0.5 * e.val * S.r) Val from es e join journal_ratio S
on (e.k=S.k)
where e.ca = 1 group by S.v
) t
insert overwrite table temp select t.*;

--
Mafish@gmail.com

reply | permalink

Yekesa Kosuru

at Mar 18, 2010 at 1:03 pm

⇧

I have this in my error log:

vi taskdetails.jsp?jobid=job_201003132100_0003
<head>
<link rel="stylesheet" type="text/css" href="/static/hadoop.css">
<title>Hadoop Task Details</title>
</head>
<body>
<h1>Job <a href="/jobdetails.jsp?jobid=job_201003132100_0003">job_201003132100_0003</a></h1>

<hr>

<h2>All Task Attempts</h2>
<center>

<h3>No Task Attempts found</h3>

<hr>
<a href="jobdetails.jsp?jobid=job_201003132100_0003">Go back to the job</a><br>
<a href="jobtracker.jsp">Go back to JobTracker</a><br>
<hr />
<a href='http://hadoop.apache.org/core'>Hadoop</a>, 2010.
</body></html>

--- On Thu, 3/18/10, Mafish Liu wrote:

From: Mafish Liu <mafish@gmail.com>
Subject: Re: hive errors
To: hive-user@hadoop.apache.org
Date: Thursday, March 18, 2010, 12:01 AM
Yekesa:

You'r reducer meet some errors while procesing.

You should check
http://dacn0:50030/taskdetails.jsp?jobid=job_201003132100_0003&tipid=task_201003132100_0003_r_000003
for more details of the failed reduce task.

Please post the error log here for further analyzing.

Mafish

2010/3/18 Yekesa Kosuru <vissuyk@yahoo.com>:

Hi,

I am a newbie to hiveql. I love what I see but I am

having issues with simple hive queries. Below is the
transcript. Can someone tell me why reducers going backwards
(goes to 67% and then drops to 57%) and everytime I see
this, the query finally errors out.

hive> from

(
select S.v k, sum (0.5 * e.val * S.r) Val

from es e join journal_ratio S

on (e.k=S.k)
where e.ca = 1 group by S.v
) t
insert overwrite table temp select t.*;

/usr/local/hadoop/0.20.1/bin/../bin/hadoop job
-Dmapred.job.tracker=dacn0:54311 -kill
job_201003132100_0003

2010-03-18 05:21:38,618 Stage-1 map = 0%, reduce = 0%
2010-03-18 05:21:56,709 Stage-1 map = 6%, reduce = 0%
2010-03-18 05:21:57,717 Stage-1 map = 9%, reduce = 0%
2010-03-18 05:21:59,731 Stage-1 map = 13%, reduce = 0%
2010-03-18 05:22:00,737 Stage-1 map = 15%, reduce = 0%
2010-03-18 05:22:02,748 Stage-1 map = 19%, reduce = 0%
2010-03-18 05:22:03,755 Stage-1 map = 21%, reduce = 0%
2010-03-18 05:22:05,768 Stage-1 map = 23%, reduce = 0%
2010-03-18 05:22:06,777 Stage-1 map = 28%, reduce = 0%
2010-03-18 05:22:07,785 Stage-1 map = 32%, reduce = 0%
2010-03-18 05:22:09,798 Stage-1 map = 34%, reduce = 0%
2010-03-18 05:22:10,805 Stage-1 map = 37%, reduce = 0%
2010-03-18 05:22:12,820 Stage-1 map = 39%, reduce = 0%
2010-03-18 05:22:13,842 Stage-1 map = 41%, reduce = 0%
2010-03-18 05:22:14,850 Stage-1 map = 43%, reduce = 0%
2010-03-18 05:22:15,858 Stage-1 map = 45%, reduce = 1%
2010-03-18 05:22:17,872 Stage-1 map = 51%, reduce = 2%
2010-03-18 05:22:18,879 Stage-1 map = 53%, reduce = 5%
2010-03-18 05:22:20,893 Stage-1 map = 57%, reduce = 6%
2010-03-18 05:22:21,900 Stage-1 map = 60%, reduce = 8%
2010-03-18 05:22:22,907 Stage-1 map = 60%, reduce = 10%
2010-03-18 05:22:23,915 Stage-1 map = 64%, reduce = 11%
2010-03-18 05:22:24,922 Stage-1 map = 68%, reduce = 12%
2010-03-18 05:22:25,930 Stage-1 map = 70%, reduce = 12%
2010-03-18 05:22:26,937 Stage-1 map = 71%, reduce = 13%
2010-03-18 05:22:27,944 Stage-1 map = 74%, reduce = 13%
2010-03-18 05:22:28,952 Stage-1 map = 77%, reduce = 13%
2010-03-18 05:22:29,960 Stage-1 map = 79%, reduce = 14%
2010-03-18 05:22:30,968 Stage-1 map = 81%, reduce = 14%
2010-03-18 05:22:31,975 Stage-1 map = 83%, reduce = 14%
2010-03-18 05:22:32,983 Stage-1 map = 88%, reduce = 14%
2010-03-18 05:22:33,991 Stage-1 map = 91%, reduce = 17%
2010-03-18 05:22:35,001 Stage-1 map = 92%, reduce = 18%
2010-03-18 05:22:36,009 Stage-1 map = 96%, reduce = 19%
2010-03-18 05:22:37,017 Stage-1 map = 97%, reduce = 19%
2010-03-18 05:22:39,029 Stage-1 map = 99%, reduce = 19%
2010-03-18 05:22:40,037 Stage-1 map = 100%, reduce = 21%
2010-03-18 05:22:42,050 Stage-1 map = 100%, reduce = 23%
2010-03-18 05:22:43,058 Stage-1 map = 100%, reduce = 26%
2010-03-18 05:22:46,076 Stage-1 map = 100%, reduce = 28%
2010-03-18 05:22:48,091 Stage-1 map = 100%, reduce = 30%
2010-03-18 05:22:49,098 Stage-1 map = 100%, reduce = 36%
2010-03-18 05:22:52,117 Stage-1 map = 100%, reduce = 46%
2010-03-18 05:22:54,129 Stage-1 map = 100%, reduce = 52%
2010-03-18 05:22:55,136 Stage-1 map = 100%, reduce = 62%
2010-03-18 05:22:57,148 Stage-1 map = 100%, reduce = 67%
2010-03-18 05:23:51,425 Stage-1 map = 100%, reduce = 57%
after this, it continues

2010-03-18 05:24:03,487 Stage-1 map = 100%, reduce = 61%
2010-03-18 05:24:09,520 Stage-1 map = 100%, reduce = 62%
2010-03-18 05:24:12,539 Stage-1 map = 100%, reduce = 67%
2010-03-18 05:24:21,592 Stage-1 map = 100%, reduce = 68%
2010-03-18 05:25:08,823 Stage-1 map = 100%, reduce = 58%
2010-03-18 05:26:09,126 Stage-1 map = 100%, reduce = 58%
jumps around until it errors out as follows:

2010-03-18 05:38:23,638 Stage-1 map = 100%, reduce = 50%
2010-03-18 05:38:43,786 Stage-1 map = 100%, reduce = 40%
2010-03-18 05:38:46,802 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201003132100_0003 with errors

Failed tasks with most(4) failures :
Task URL: http://dacn0:50030/taskdetails.jsp?jobid=job_201003132100_0003&tipid=task_201003132100_0003_r_000003

FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.ExecDriver

Hive version = 0.20.1, Java = 1.6

Is this a user error? as in should I add group by, or

sort by or cluster by clauses to the query ? Query was run
on the master name node, and there are 4 data nodes. With
other queries I have seen the map % progresses and drops and
eventually errors out.

Any help is appreciated.

Thanks

Vissu

--
Mafish@gmail.com

reply | permalink

Mafish Liu You should access the URL with browser, instead of vi. 2010/3/18 Yekesa Kosuru : -- Mafish@gmail.com

Mafish Liu at Mar 19, 2010 at 1:33 am ⇧

You should access the URL with browser, instead of vi.

2010/3/18 Yekesa Kosuru <vissuyk@yahoo.com>:
I have this in my error log:

vi taskdetails.jsp?jobid=job_201003132100_0003
<head>
<link rel="stylesheet" type="text/css" href="/static/hadoop.css">
<title>Hadoop Task Details</title>
</head>
<body>
<h1>Job <a href="/jobdetails.jsp?jobid=job_201003132100_0003">job_201003132100_0003</a></h1>

<hr>

<h2>All Task Attempts</h2>
<center>

<h3>No Task Attempts found</h3>

<hr>
<a href="jobdetails.jsp?jobid=job_201003132100_0003">Go back to the job</a><br>
<a href="jobtracker.jsp">Go back to JobTracker</a><br>
<hr />
<a href='http://hadoop.apache.org/core'>Hadoop</a>, 2010.
</body></html>

--- On Thu, 3/18/10, Mafish Liu wrote:
From: Mafish Liu <mafish@gmail.com>
Subject: Re: hive errors
To: hive-user@hadoop.apache.org
Date: Thursday, March 18, 2010, 12:01 AM
Yekesa:

You'r reducer meet some errors while procesing.

You should check
http://dacn0:50030/taskdetails.jsp?jobid=job_201003132100_0003&tipid=task_201003132100_0003_r_000003
for more details of the failed reduce task.

Please post the error log here for further analyzing.

Mafish

2010/3/18 Yekesa Kosuru <vissuyk@yahoo.com>:
Hi,

I am a newbie to hiveql. I love what I see but I am
having issues with simple hive queries. Below is the
transcript. Can someone tell me why reducers going backwards
(goes to 67% and then drops to 57%) and everytime I see
this, the query finally errors out.

hive> from
(
select S.v k, sum (0.5 * e.val * S.r) Val
from es e join journal_ratio S
on (e.k=S.k)
where e.ca = 1 group by S.v
) t
insert overwrite table temp select t.*;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Defaulting to
jobconf value of: 7
In order to change the average load for a reducer (in bytes):
set
hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201003132100_0003, Tracking URL =
http://dacn0:50030/jobdetails.jsp?jobid=job_201003132100_0003
Kill Command =
/usr/local/hadoop/0.20.1/bin/../bin/hadoop job
-Dmapred.job.tracker=dacn0:54311 -kill
job_201003132100_0003
2010-03-18 05:21:38,618 Stage-1 map = 0%, reduce = 0%
2010-03-18 05:21:56,709 Stage-1 map = 6%, reduce = 0%
2010-03-18 05:21:57,717 Stage-1 map = 9%, reduce = 0%
2010-03-18 05:21:59,731 Stage-1 map = 13%, reduce = 0%
2010-03-18 05:22:00,737 Stage-1 map = 15%, reduce = 0%
2010-03-18 05:22:02,748 Stage-1 map = 19%, reduce = 0%
2010-03-18 05:22:03,755 Stage-1 map = 21%, reduce = 0%
2010-03-18 05:22:05,768 Stage-1 map = 23%, reduce = 0%
2010-03-18 05:22:06,777 Stage-1 map = 28%, reduce = 0%
2010-03-18 05:22:07,785 Stage-1 map = 32%, reduce = 0%
2010-03-18 05:22:09,798 Stage-1 map = 34%, reduce = 0%
2010-03-18 05:22:10,805 Stage-1 map = 37%, reduce = 0%
2010-03-18 05:22:12,820 Stage-1 map = 39%, reduce = 0%
2010-03-18 05:22:13,842 Stage-1 map = 41%, reduce = 0%
2010-03-18 05:22:14,850 Stage-1 map = 43%, reduce = 0%
2010-03-18 05:22:15,858 Stage-1 map = 45%, reduce = 1%
2010-03-18 05:22:17,872 Stage-1 map = 51%, reduce = 2%
2010-03-18 05:22:18,879 Stage-1 map = 53%, reduce = 5%
2010-03-18 05:22:20,893 Stage-1 map = 57%, reduce = 6%
2010-03-18 05:22:21,900 Stage-1 map = 60%, reduce = 8%
2010-03-18 05:22:22,907 Stage-1 map = 60%, reduce = 10%
2010-03-18 05:22:23,915 Stage-1 map = 64%, reduce = 11%
2010-03-18 05:22:24,922 Stage-1 map = 68%, reduce = 12%
2010-03-18 05:22:25,930 Stage-1 map = 70%, reduce = 12%
2010-03-18 05:22:26,937 Stage-1 map = 71%, reduce = 13%
2010-03-18 05:22:27,944 Stage-1 map = 74%, reduce = 13%
2010-03-18 05:22:28,952 Stage-1 map = 77%, reduce = 13%
2010-03-18 05:22:29,960 Stage-1 map = 79%, reduce = 14%
2010-03-18 05:22:30,968 Stage-1 map = 81%, reduce = 14%
2010-03-18 05:22:31,975 Stage-1 map = 83%, reduce = 14%
2010-03-18 05:22:32,983 Stage-1 map = 88%, reduce = 14%
2010-03-18 05:22:33,991 Stage-1 map = 91%, reduce = 17%
2010-03-18 05:22:35,001 Stage-1 map = 92%, reduce = 18%
2010-03-18 05:22:36,009 Stage-1 map = 96%, reduce = 19%
2010-03-18 05:22:37,017 Stage-1 map = 97%, reduce = 19%
2010-03-18 05:22:39,029 Stage-1 map = 99%, reduce = 19%
2010-03-18 05:22:40,037 Stage-1 map = 100%, reduce = 21%
2010-03-18 05:22:42,050 Stage-1 map = 100%, reduce = 23%
2010-03-18 05:22:43,058 Stage-1 map = 100%, reduce = 26%
2010-03-18 05:22:46,076 Stage-1 map = 100%, reduce = 28%
2010-03-18 05:22:48,091 Stage-1 map = 100%, reduce = 30%
2010-03-18 05:22:49,098 Stage-1 map = 100%, reduce = 36%
2010-03-18 05:22:52,117 Stage-1 map = 100%, reduce = 46%
2010-03-18 05:22:54,129 Stage-1 map = 100%, reduce = 52%
2010-03-18 05:22:55,136 Stage-1 map = 100%, reduce = 62%
2010-03-18 05:22:57,148 Stage-1 map = 100%, reduce = 67%
2010-03-18 05:23:51,425 Stage-1 map = 100%, reduce = 57%
after this, it continues

2010-03-18 05:24:03,487 Stage-1 map = 100%, reduce = 61%
2010-03-18 05:24:09,520 Stage-1 map = 100%, reduce = 62%
2010-03-18 05:24:12,539 Stage-1 map = 100%, reduce = 67%
2010-03-18 05:24:21,592 Stage-1 map = 100%, reduce = 68%
2010-03-18 05:25:08,823 Stage-1 map = 100%, reduce = 58%
2010-03-18 05:26:09,126 Stage-1 map = 100%, reduce = 58%
jumps around until it errors out as follows:

2010-03-18 05:38:23,638 Stage-1 map = 100%, reduce = 50%
2010-03-18 05:38:43,786 Stage-1 map = 100%, reduce = 40%
2010-03-18 05:38:46,802 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201003132100_0003 with errors

Failed tasks with most(4) failures :
Task URL: http://dacn0:50030/taskdetails.jsp?jobid=job_201003132100_0003&tipid=task_201003132100_0003_r_000003

FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.ExecDriver

Hive version = 0.20.1, Java = 1.6

Is this a user error? as in should I add group by, or
sort by or cluster by clauses to the query ? Query was run
on the master name node, and there are 4 data nodes. With
other queries I have seen the map % progresses and drops and
eventually errors out.
Any help is appreciated.

Thanks

Vissu

--
Mafish@gmail.com

--
Mafish@gmail.com

reply | permalink

Related Discussions

Discussion Navigation

view	thread \| post

Discussion Overview

group	user @
categories	hive, hadoop
posted	Mar 18, '10 at 3:46a
active	Mar 19, '10 at 1:33a
posts	4
users	2
website	hive.apache.org

responses	expanded	Hotkey: s
font	variable	Hotkey: f
user style	avatars	Hotkey: a

[Hive-user] hive errors

Search Discussions

Related Discussions

2 users in discussion