FAQ
Hello -



I have been running into problems with one or two reduce tasks stalling
out. This was occuring regularly in 0.13, and remains an issue with
0.14.1



In both cases the fsimage file in the namenode's current image is empty,
with the it, as well as the edits file not having been touched in either
a day (with 0.14.1) or 3 weeks (in the case of the old 0.13
installation). The edits.new file is continuing to grow. Shutting down
the namenode when this occurred resulted in a state where the namenode
would not restart.



In the namenode log, there are multiple exceptions of the form listed
below.

********************

2007-09-08 16:58:04,182 INFO org.apache.hadoop.fs.FSNamesystem: Roll
Edit Log fr

om 10.33.0.56

2007-09-08 16:58:04,182 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 34

on 50001, call rollEditLog() from 10.33.0.56:41291: error:

java.io.IOException: Attempt to roll edit log but edits.new exists

java.io.IOException: Attempt to roll edit log but edits.new exists

at
org.apache.hadoop.dfs.FSEditLog.rollEditLog(FSEditLog.java:582)

at
org.apache.hadoop.dfs.FSNamesystem.rollEditLog(FSNamesystem.java:3510

)

at org.apache.hadoop.dfs.NameNode.rollEditLog(NameNode.java:553)

at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces

sorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)



**************



Has anyone run into this?



Thanks

-Sameer

Search Discussions

  • Dhruba Borthakur at Sep 10, 2007 at 5:35 pm
    Hi Sameer,

    Is there a way for you to let me look at your Namenode and
    SecondaryNameNode logs? It might help me in analyzing the problem.

    Thanks,
    dhruba

    -----Original Message-----
    From: Sameer Alsakran
    Sent: Saturday, September 08, 2007 5:05 PM
    To: hadoop-user@lucene.apache.org
    Subject: Problem with edits.new rolling over

    Hello -



    I have been running into problems with one or two reduce tasks stalling
    out. This was occuring regularly in 0.13, and remains an issue with
    0.14.1



    In both cases the fsimage file in the namenode's current image is empty,
    with the it, as well as the edits file not having been touched in either
    a day (with 0.14.1) or 3 weeks (in the case of the old 0.13
    installation). The edits.new file is continuing to grow. Shutting down
    the namenode when this occurred resulted in a state where the namenode
    would not restart.



    In the namenode log, there are multiple exceptions of the form listed
    below.

    ********************

    2007-09-08 16:58:04,182 INFO org.apache.hadoop.fs.FSNamesystem: Roll
    Edit Log fr

    om 10.33.0.56

    2007-09-08 16:58:04,182 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 34

    on 50001, call rollEditLog() from 10.33.0.56:41291: error:

    java.io.IOException: Attempt to roll edit log but edits.new exists

    java.io.IOException: Attempt to roll edit log but edits.new exists

    at
    org.apache.hadoop.dfs.FSEditLog.rollEditLog(FSEditLog.java:582)

    at
    org.apache.hadoop.dfs.FSNamesystem.rollEditLog(FSNamesystem.java:3510

    )

    at org.apache.hadoop.dfs.NameNode.rollEditLog(NameNode.java:553)

    at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces

    sorImpl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)

    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)



    **************



    Has anyone run into this?



    Thanks

    -Sameer

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedSep 9, '07 at 12:05a
activeSep 10, '07 at 5:35p
posts2
users2
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase