FAQ
I have a MR job running on approximately 150000 lines of data in a text
file. The reducer hangs at 66% and at that moment the cpu usage is at 100%.
After 600 seconds, it kills the job... What could be going wrong and where
should I look for the problem?


Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz

Search Discussions

  • Schubert zhang at Mar 28, 2009 at 1:58 am
    It seems there is a endless loop in you reduce() function.You may do not
    call values.next().
    On Sat, Mar 28, 2009 at 7:31 AM, Amandeep Khurana wrote:

    I have a MR job running on approximately 150000 lines of data in a text
    file. The reducer hangs at 66% and at that moment the cpu usage is at 100%.
    After 600 seconds, it kills the job... What could be going wrong and where
    should I look for the problem?


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz
  • Billy Pearson at Mar 29, 2009 at 7:05 am
    66% is the start of the reduce function so its likely a endless loop there
    burning the cpu cycles


    "Amandeep Khurana" <amansk@gmail.com> wrote in
    message news:35a22e220903271631i25ff749bx5814348e66ff4d60@mail.gmail.com...
    I have a MR job running on approximately 150000 lines of data in a text
    file. The reducer hangs at 66% and at that moment the cpu usage is at
    100%.
    After 600 seconds, it kills the job... What could be going wrong and where
    should I look for the problem?


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMar 27, '09 at 11:31p
activeMar 29, '09 at 7:05a
posts3
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase