Amareshwari Sriramadasu edited comment on HADOOP-2141 at 6/12/09 3:42 AM:
--------------------------------------------------------------------------
one minor comment:
Initialization for stats can be done inline. And comments to the stats, describing what the stats are for, can be added.
Otherwise, Changes look fine to me.
was (Author: amareshwari):
one minor comment:
Initialization for stats can be done inline. And comments to the stats, describing what the stats are for, can be added.
speculative execution start up condition based on completion time
-----------------------------------------------------------------
Key: HADOOP-2141
URL: https://issues.apache.org/jira/browse/HADOOP-2141
Project: Hadoop Core
Issue Type: Improvement
Components: mapred
Affects Versions: 0.21.0
Reporter: Koji Noguchi
Assignee: Andy Konwinski
Fix For: 0.21.0
Attachments: 2141.4.patch, 2141.5.patch, 2141.6.patch, 2141.patch, HADOOP-2141-v2.patch, HADOOP-2141-v3.patch, HADOOP-2141-v4.patch, HADOOP-2141-v5.patch, HADOOP-2141-v6.patch, HADOOP-2141.patch, HADOOP-2141.v7.patch, HADOOP-2141.v8.patch
We had one job with speculative execution hang.
4 reduce tasks were stuck with 95% completion because of a bad disk.
Devaraj pointed out
bq . One of the conditions that must be met for launching a speculative instance of a task is that it must be at least 20% behind the average progress, and this is not true here.
It would be nice if speculative execution also starts up when tasks stop making progress.
Devaraj suggested
bq. Maybe, we should introduce a condition for average completion time for tasks in the speculative execution check.
-------------------------------------------------------------------
Key: HADOOP-2141
URL: https://issues.apache.org/jira/browse/HADOOP-2141
Project: Hadoop Core
Issue Type: Improvement
Components: mapred
Affects Versions: 0.21.0
Reporter: Koji Noguchi
Assignee: Andy Konwinski
Fix For: 0.21.0
Attachments: 2141.4.patch, 2141.5.patch, 2141.6.patch, 2141.patch, HADOOP-2141-v2.patch, HADOOP-2141-v3.patch, HADOOP-2141-v4.patch, HADOOP-2141-v5.patch, HADOOP-2141-v6.patch, HADOOP-2141.patch, HADOOP-2141.v7.patch, HADOOP-2141.v8.patch
We had one job with speculative execution hang.
4 reduce tasks were stuck with 95% completion because of a bad disk.
Devaraj pointed out
bq . One of the conditions that must be met for launching a speculative instance of a task is that it must be at least 20% behind the average progress, and this is not true here.
It would be nice if speculative execution also starts up when tasks stop making progress.
Devaraj suggested
bq. Maybe, we should introduce a condition for average completion time for tasks in the speculative execution check.
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.