FAQ
Hi,
Could someone throw some light as to how intuitively fixed-type slots in
Hadoop have a negative impact of cluster utilization as mentioned in Arun's
blog?
http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/

Thanks,
Bikash

Search Discussions

  • Greg Roelofs at Mar 2, 2011 at 10:08 pm

    Could someone throw some light as to how intuitively fixed-type slots in
    Hadoop have a negative impact of cluster utilization as mentioned in Arun's
    blog?
    It's pretty simple: in the course of time, a real cluster with a mix of
    job types and submission times will have periods of high map-slot usage
    (with reduce-slots available), and vice versa. Being able to assign machine
    resources to whatever kind of task needs them at any given time means you're
    not wasting those resources.

    In short: map- vs. reduce-slots is an artificial and unnecessary restriction.

    Greg
  • Bikash sharma at Mar 3, 2011 at 12:20 am
    Thanks Greg!
    On Wed, Mar 2, 2011 at 5:07 PM, Greg Roelofs wrote:

    Could someone throw some light as to how intuitively fixed-type slots in
    Hadoop have a negative impact of cluster utilization as mentioned in Arun's
    blog?
    It's pretty simple: in the course of time, a real cluster with a mix of
    job types and submission times will have periods of high map-slot usage
    (with reduce-slots available), and vice versa. Being able to assign
    machine
    resources to whatever kind of task needs them at any given time means
    you're
    not wasting those resources.

    In short: map- vs. reduce-slots is an artificial and unnecessary
    restriction.

    Greg

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMar 2, '11 at 1:34p
activeMar 3, '11 at 12:20a
posts3
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Bikash sharma: 2 posts Greg Roelofs: 1 post

People

Translate

site design / logo © 2022 Grokbase