FAQ
Oops...

I executed the following command:

./hadoop dfs -rmr .

Everything on the DFS, including the trash seems to be deleted. Is there
a way to recover my data?

Thanks,
Mathijs

--
Knowlogy
Helperpark 290 C
9723 ZA Groningen

mathijs.homminga@knowlogy.nl
+31 (0)6 15312977
http://www.knowlogy.nl

Search Discussions

  • Andrzej Bialecki at Jul 17, 2007 at 2:46 pm

    Mathijs Homminga wrote:
    Oops...

    I executed the following command:

    ./hadoop dfs -rmr .

    Everything on the DFS, including the trash seems to be deleted. Is there
    a way to recover my data?
    Depending which version of Hadoop you are running ... In 0.12.x it would
    be enough to kill -9 the NameNode ASAP, and remove this deletion
    operation from the editlog, before it gets re-integrated with fsimage.

    DO MAKE A BACKUP OF THESE FILES FIRST !!!

    --
    Best regards,
    Andrzej Bialecki <><
    ___. ___ ___ ___ _ _ __________________________________
    [__ || __|__/|__||\/| Information Retrieval, Semantic Web
    ___|||__|| \| || | Embedded Unix, System Integration
    http://www.sigram.com Contact: info at sigram dot com
  • Mathijs Homminga at Jul 17, 2007 at 3:17 pm
    Thanks,

    I stopped the namenode. How can I remove an entry from the editlog?

    FYI: The following action caused the mistake:

    I first copied a directory from the DFS to local:
    ./hadoop dfs -get segments/20070622192310 .

    Then, I edited this line (from my command history) to delete the directory:
    ./hadoop dfs -rmr segments/20070622192310 .

    But forgot the dot at the end... :(

    Andrzej Bialecki wrote:
    Mathijs Homminga wrote:
    Oops...

    I executed the following command:

    ./hadoop dfs -rmr .

    Everything on the DFS, including the trash seems to be deleted. Is
    there a way to recover my data?
    Depending which version of Hadoop you are running ... In 0.12.x it
    would be enough to kill -9 the NameNode ASAP, and remove this deletion
    operation from the editlog, before it gets re-integrated with fsimage.

    DO MAKE A BACKUP OF THESE FILES FIRST !!!
    --
    Knowlogy
    Helperpark 290 C
    9723 ZA Groningen

    mathijs.homminga@knowlogy.nl
    +31 (0)6 15312977
    http://www.knowlogy.nl
  • Andrzej Bialecki at Jul 17, 2007 at 3:21 pm

    Mathijs Homminga wrote:
    Thanks,

    I stopped the namenode. How can I remove an entry from the editlog?
    I wrote a tool for specifically this purpose ;) but it's not up to date
    anymore - I'm not sure how much hacking is required to make it work
    again. See HADOOP-915.


    --
    Best regards,
    Andrzej Bialecki <><
    ___. ___ ___ ___ _ _ __________________________________
    [__ || __|__/|__||\/| Information Retrieval, Semantic Web
    ___|||__|| \| || | Embedded Unix, System Integration
    http://www.sigram.com Contact: info at sigram dot com
  • Mathijs Homminga at Jul 17, 2007 at 8:31 pm
    You saved the day again.

    The FSEditLogTool worked like a charm, without modifications
    (https://issues.apache.org/jira/browse/HADOOP-915).

    Here is what I did to perform an undelete of the root directory on my
    HDFS (hadoop 0.12.2.)

    - first, I ran a few tests on another dfs to make sure the FSEditLogTool
    worked (and I understood what is was doing). Then on the affected dfs:
    - kill the namenode (already did that asap, and to be sure, I stopped
    all other hadoop daemons)
    - run FSEditLogTool -dump ${dfs.name.dir}/edits edits.txt
    - remove the last line from edits.txt which contains the DELETE entry.
    - run FSEditLogTool -restore edits.txt ${dfs.name.dir}/edits
    - start the namenode and datanodes again
    - I had to turn off save-mode manually:
    $ bin/hadoop dfsadmin -safemode leave

    \o/

    Andrzej Bialecki wrote:
    Mathijs Homminga wrote:
    Thanks,

    I stopped the namenode. How can I remove an entry from the editlog?
    I wrote a tool for specifically this purpose ;) but it's not up to
    date anymore - I'm not sure how much hacking is required to make it
    work again. See HADOOP-915.
    --
    Knowlogy
    Helperpark 290 C
    9723 ZA Groningen

    mathijs.homminga@knowlogy.nl
    +31 (0)6 15312977
    http://www.knowlogy.nl
  • Andrzej Bialecki at Jul 17, 2007 at 9:21 pm

    Mathijs Homminga wrote:
    You saved the day again.

    The FSEditLogTool worked like a charm, without modifications
    (https://issues.apache.org/jira/browse/HADOOP-915).
    That's great, I was afraid it was out of sync with your version of Hadoop.
    Here is what I did to perform an undelete of the root directory on my
    HDFS (hadoop 0.12.2.)

    - first, I ran a few tests on another dfs to make sure the FSEditLogTool
    worked (and I understood what is was doing). Then on the affected dfs:
    - kill the namenode (already did that asap, and to be sure, I stopped
    all other hadoop daemons)
    - run FSEditLogTool -dump ${dfs.name.dir}/edits edits.txt
    - remove the last line from edits.txt which contains the DELETE entry.
    - run FSEditLogTool -restore edits.txt ${dfs.name.dir}/edits
    - start the namenode and datanodes again
    - I had to turn off save-mode manually:
    $ bin/hadoop dfsadmin -safemode leave
    Just for the record - this procedure should be your absolutely last
    resort, because it's tricky and error prone. You can easily corrupt your
    filesystem beyond any hope.

    --
    Best regards,
    Andrzej Bialecki <><
    ___. ___ ___ ___ _ _ __________________________________
    [__ || __|__/|__||\/| Information Retrieval, Semantic Web
    ___|||__|| \| || | Embedded Unix, System Integration
    http://www.sigram.com Contact: info at sigram dot com
  • Andrzej Bialecki at Jul 17, 2007 at 3:24 pm

    Mathijs Homminga wrote:
    Thanks,

    I stopped the namenode. How can I remove an entry from the editlog?
    .. I forgot to add: if you feel adventurous (or desperate) enough, you
    can use a binary editor and remove this DELETE record from the file. Be
    sure to carefully read FSEditLog.logDelete() and related methods to
    figure out where the record starts / ends.


    --
    Best regards,
    Andrzej Bialecki <><
    ___. ___ ___ ___ _ _ __________________________________
    [__ || __|__/|__||\/| Information Retrieval, Semantic Web
    ___|||__|| \| || | Embedded Unix, System Integration
    http://www.sigram.com Contact: info at sigram dot com
  • Doug Cutting at Jul 17, 2007 at 5:15 pm
    Since Hadoop 0.12, if you configure fs.trash.interval to a non-zero
    value then 'bin/hadoop dfs -rm' will move things to a trash directory
    instead of immediately removing them. The Trash is periodically emptied
    of older items. Perhaps we should change the default value for this to
    60 (one hour)?

    Doug

    Mathijs Homminga wrote:
    Oops...

    I executed the following command:

    ./hadoop dfs -rmr .

    Everything on the DFS, including the trash seems to be deleted. Is there
    a way to recover my data?

    Thanks,
    Mathijs
  • Mathijs Homminga at Jul 17, 2007 at 9:17 pm
    +1

    Setting this property to a non-zero value by default will give less
    experienced Hadoop users (like me :)) a safety net.


    Doug Cutting wrote:
    Since Hadoop 0.12, if you configure fs.trash.interval to a non-zero
    value then 'bin/hadoop dfs -rm' will move things to a trash directory
    instead of immediately removing them. The Trash is periodically
    emptied of older items. Perhaps we should change the default value
    for this to 60 (one hour)?

    Doug

    Mathijs Homminga wrote:
    Oops...

    I executed the following command:

    ./hadoop dfs -rmr .

    Everything on the DFS, including the trash seems to be deleted. Is
    there a way to recover my data?

    Thanks,
    Mathijs
    --
    Knowlogy
    Helperpark 290 C
    9723 ZA Groningen

    mathijs.homminga@knowlogy.nl
    +31 (0)6 15312977
    http://www.knowlogy.nl

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJul 17, '07 at 2:26p
activeJul 17, '07 at 9:21p
posts9
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase