FAQ
Hello,

I often see reduce > copy (at 0.52 MB/s) phase with such speed.
Despite in my cluster all 5 nodes are in same rack.
Does it mean any network or other IO problems, or other reasons can
cause such slow speed?

Thanks in Advance,
Vitaliy S

Search Discussions

  • Harsh J at Oct 5, 2010 at 4:12 pm
    The reduce begins copying map outputs as they complete (starting at 5% of
    them) and this transfer may be very meagre and thus the low rate of
    transfer.

    Observe once all maps finish or near completion at their last wave, if the
    network status shown is still slow then there is a problem, whose common
    side effect would be failing reducers or long time waits before the sort
    phase kicks in even if all mappers are already done.

    Otherwise this isn't an issue. You can also increase the parallel fetching
    factor of each reducer :)

    On Oct 5, 2010 6:49 PM, "Vitaliy Semochkin" wrote:

    Hello,

    I often see reduce > copy (at 0.52 MB/s) phase with such speed.
    Despite in my cluster all 5 nodes are in same rack.
    Does it mean any network or other IO problems, or other reasons can
    cause such slow speed?

    Thanks in Advance,
    Vitaliy S

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 5, '10 at 1:19p
activeOct 5, '10 at 4:12p
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Vitaliy Semochkin: 1 post Harsh J: 1 post

People

Translate

site design / logo © 2022 Grokbase