FAQ
Hi.

I have an issue where my application, when shutting down (at ShutdownHook
level), is unable to copy files to HDFS.

Each copy throws the following exception:

java.lang.IllegalStateException: Shutdown in progress
at
java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:39)
at java.lang.Runtime.addShutdownHook(Runtime.java:192)
at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1353)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)
at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:189)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1185)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1161)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1133)
at app.util.FileUtils.copyToCluster(FileUtils.java:392)


I read some reports, including this one (
https://issues.apache.org/jira/browse/HADOOP-3818), but there was no
definite answer how to do it.

Is there any good solution to this issue?

Thanks in advance.

Search Discussions

  • Stas Oskin at May 19, 2009 at 7:44 am
    Hi.

    Does anyone has any idea on this issue?

    Thanks!

    2009/5/17 Stas Oskin <stas.oskin@gmail.com>
    Hi.

    I have an issue where my application, when shutting down (at ShutdownHook
    level), is unable to copy files to HDFS.

    Each copy throws the following exception:

    java.lang.IllegalStateException: Shutdown in progress
    at
    java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:39)
    at java.lang.Runtime.addShutdownHook(Runtime.java:192)
    at
    org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1353)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)
    at
    org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:189)
    at
    org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1185)
    at
    org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1161)
    at
    org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1133)
    at app.util.FileUtils.copyToCluster(FileUtils.java:392)


    I read some reports, including this one (
    https://issues.apache.org/jira/browse/HADOOP-3818), but there was no
    definite answer how to do it.

    Is there any good solution to this issue?

    Thanks in advance.
  • Tom White at May 20, 2009 at 12:59 pm
    Looks like you are trying to copy file to HDFS in a shutdown hook.
    Since you can't control the order in which shutdown hooks run, this is
    won't work. There is a patch to allow Hadoop's FileSystem shutdown
    hook to be disabled so it doesn't close filesystems on exit. See
    https://issues.apache.org/jira/browse/HADOOP-4829.

    Cheers,
    Tom
    On Tue, May 19, 2009 at 8:44 AM, Stas Oskin wrote:
    Hi.

    Does anyone has any idea on this issue?

    Thanks!

    2009/5/17 Stas Oskin <stas.oskin@gmail.com>
    Hi.

    I have an issue where my application, when shutting down (at ShutdownHook
    level), is unable to copy files to HDFS.

    Each copy throws the following exception:

    java.lang.IllegalStateException: Shutdown in progress
    at
    java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:39)
    at java.lang.Runtime.addShutdownHook(Runtime.java:192)
    at
    org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1353)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)
    at
    org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:189)
    at
    org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1185)
    at
    org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1161)
    at
    org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1133)
    at app.util.FileUtils.copyToCluster(FileUtils.java:392)


    I read some reports, including this one (
    https://issues.apache.org/jira/browse/HADOOP-3818), but there was no
    definite answer how to do it.

    Is there any good solution to this issue?

    Thanks in advance.
  • Stas Oskin at May 20, 2009 at 9:08 pm
    Hi.

    2009/5/20 Tom White <tom@cloudera.com>
    Looks like you are trying to copy file to HDFS in a shutdown hook.
    Since you can't control the order in which shutdown hooks run, this is
    won't work. There is a patch to allow Hadoop's FileSystem shutdown
    hook to be disabled so it doesn't close filesystems on exit. See
    https://issues.apache.org/jira/browse/HADOOP-4829.

    Thanks! Any downside to this?

    Also, I see that the patch doesn't work against tree- but the 0.18.3 is
    supported?

    Regards.
  • Todd Lipcon at May 20, 2009 at 9:16 pm

    On Wed, May 20, 2009 at 2:07 PM, Stas Oskin wrote:

    Hi.

    2009/5/20 Tom White <tom@cloudera.com>
    Looks like you are trying to copy file to HDFS in a shutdown hook.
    Since you can't control the order in which shutdown hooks run, this is
    won't work. There is a patch to allow Hadoop's FileSystem shutdown
    hook to be disabled so it doesn't close filesystems on exit. See
    https://issues.apache.org/jira/browse/HADOOP-4829.

    Thanks! Any downside to this?
    You should only use this if you plan on manually closing FileSystems
    yourself from within your own shutdown hook. It's somewhat of an advanced
    feature, and I wouldn't recommend using this patch unless you fully
    understand the ramifications of modifying the shutdown sequence.

    Also, I see that the patch doesn't work against tree- but the 0.18.3 is
    supported?
    Just uploaded a patch based on branch 18 for you to that JIRA.

    Thanks
    -Todd
  • Stas Oskin at May 20, 2009 at 9:22 pm

    You should only use this if you plan on manually closing FileSystems
    yourself from within your own shutdown hook. It's somewhat of an advanced
    feature, and I wouldn't recommend using this patch unless you fully
    understand the ramifications of modifying the shutdown sequence.

    Standard dfs.close() would do the trick, no?

    Just uploaded a patch based on branch 18 for you to that JIRA.
    Thanks a lot!
  • Tom White at May 21, 2009 at 8:05 am

    On Wed, May 20, 2009 at 10:22 PM, Stas Oskin wrote:

    You should only use this if you plan on manually closing FileSystems
    yourself from within your own shutdown hook. It's somewhat of an advanced
    feature, and I wouldn't recommend using this patch unless you fully
    understand the ramifications of modifying the shutdown sequence.

    Standard dfs.close() would do the trick, no?
    After you've performed your application shutdown actions you should
    call FileSystem's closeAll() method.
    Just uploaded a patch based on branch 18 for you to that JIRA.
    Thanks a lot!
  • Stas Oskin at May 21, 2009 at 8:18 am


    After you've performed your application shutdown actions you should
    call FileSystem's closeAll() method.

    Ahh, thanks for the tip.

    Regards.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMay 17, '09 at 12:45p
activeMay 21, '09 at 8:18a
posts8
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase