FAQ
job fails because of "No valid local directories in property: " exception
-------------------------------------------------------------------------

Key: HADOOP-242
URL: http://issues.apache.org/jira/browse/HADOOP-242
Project: Hadoop
Type: Bug

Components: mapred
Reporter: Yoram Arnon
Assigned to: Sameer Paranjpye


when running a fairly large job, of 70+K map tasks, I get many exceptions as shown below, and eventually the job failes when a task fails four times.
The exception doesn't really tell us enough information to debug this properly, so the first thing to do would be to add more information (path) to the exception.
The path indicated in the config file exists, is writable and valid, though 'path' may be anything.

the exception:
java.io.IOException: No valid local directories in property: mapred.local.dir at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:293) at org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:153) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.localizeTask(TaskTracker.java:523) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:572) at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:389) at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:303) at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:418) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:920)

the code:
public Path getLocalPath(String dirsProp, String path)
throws IOException {
String[] dirs = getStrings(dirsProp);
int hashCode = path.hashCode();
FileSystem fs = FileSystem.getNamed("local", this);
for (int i = 0; i < dirs.length; i++) { // try each local dir
int index = (hashCode+i & Integer.MAX_VALUE) % dirs.length;
Path file = new Path(dirs[index], path);
Path dir = file.getParent();
if (fs.exists(dir) || fs.mkdirs(dir)) {
return file;
}
}
throw new IOException("No valid local directories in property: "+dirsProp);
}


--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira

Search Discussions

  • eric baldeschwieler (JIRA) at May 23, 2006 at 8:31 pm
    [ http://issues.apache.org/jira/browse/HADOOP-242?page=comments#action_12412984 ]

    eric baldeschwieler commented on HADOOP-242:
    --------------------------------------------

    perhaps we should not record failure until a job has been tried on two nodes? Then we could collect stats on frequency that a job fails on a given node.

    Also 4 failures seems odd. Shouldn't we just have a configurable allowable % failure? That will scale evenly.
    job fails because of "No valid local directories in property: " exception
    -------------------------------------------------------------------------

    Key: HADOOP-242
    URL: http://issues.apache.org/jira/browse/HADOOP-242
    Project: Hadoop
    Type: Bug
    Components: mapred
    Reporter: Yoram Arnon
    Assignee: Sameer Paranjpye
    when running a fairly large job, of 70+K map tasks, I get many exceptions as shown below, and eventually the job failes when a task fails four times.
    The exception doesn't really tell us enough information to debug this properly, so the first thing to do would be to add more information (path) to the exception.
    The path indicated in the config file exists, is writable and valid, though 'path' may be anything.
    the exception:
    java.io.IOException: No valid local directories in property: mapred.local.dir at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:293) at org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:153) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.localizeTask(TaskTracker.java:523) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:572) at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:389) at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:303) at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:418) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:920)
    the code:
    public Path getLocalPath(String dirsProp, String path)
    throws IOException {
    String[] dirs = getStrings(dirsProp);
    int hashCode = path.hashCode();
    FileSystem fs = FileSystem.getNamed("local", this);
    for (int i = 0; i < dirs.length; i++) { // try each local dir
    int index = (hashCode+i & Integer.MAX_VALUE) % dirs.length;
    Path file = new Path(dirs[index], path);
    Path dir = file.getParent();
    if (fs.exists(dir) || fs.mkdirs(dir)) {
    return file;
    }
    }
    throw new IOException("No valid local directories in property: "+dirsProp);
    }
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Owen O'Malley (JIRA) at May 23, 2006 at 8:39 pm
    [ http://issues.apache.org/jira/browse/HADOOP-242?page=comments#action_12412986 ]

    Owen O'Malley commented on HADOOP-242:
    --------------------------------------

    It is 4 failures of the _same_ fragment of work. So it takes map-0000000 failing 4 times before the job is killed. The job tracker does know which machines that each task has failed on and prefers to run other tasks on that node.
    job fails because of "No valid local directories in property: " exception
    -------------------------------------------------------------------------

    Key: HADOOP-242
    URL: http://issues.apache.org/jira/browse/HADOOP-242
    Project: Hadoop
    Type: Bug
    Components: mapred
    Reporter: Yoram Arnon
    Assignee: Sameer Paranjpye
    when running a fairly large job, of 70+K map tasks, I get many exceptions as shown below, and eventually the job failes when a task fails four times.
    The exception doesn't really tell us enough information to debug this properly, so the first thing to do would be to add more information (path) to the exception.
    The path indicated in the config file exists, is writable and valid, though 'path' may be anything.
    the exception:
    java.io.IOException: No valid local directories in property: mapred.local.dir at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:293) at org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:153) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.localizeTask(TaskTracker.java:523) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:572) at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:389) at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:303) at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:418) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:920)
    the code:
    public Path getLocalPath(String dirsProp, String path)
    throws IOException {
    String[] dirs = getStrings(dirsProp);
    int hashCode = path.hashCode();
    FileSystem fs = FileSystem.getNamed("local", this);
    for (int i = 0; i < dirs.length; i++) { // try each local dir
    int index = (hashCode+i & Integer.MAX_VALUE) % dirs.length;
    Path file = new Path(dirs[index], path);
    Path dir = file.getParent();
    if (fs.exists(dir) || fs.mkdirs(dir)) {
    return file;
    }
    }
    throw new IOException("No valid local directories in property: "+dirsProp);
    }
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Owen O'Malley (JIRA) at May 30, 2006 at 5:37 pm
    [ http://issues.apache.org/jira/browse/HADOOP-242?page=all ]

    Owen O'Malley updated HADOOP-242:
    ---------------------------------

    Attachment: no-local.patch

    Here is a patch that increases the logging associated with this failure. It also tries to switch the condition around based on the possibility of multiple processes or threads. So it does:

    fs.mkdirs(dir) || fs.exists(dir)

    instead of:

    fs.exists(dir) || fs.mkdirs(dir)
    job fails because of "No valid local directories in property: " exception
    -------------------------------------------------------------------------

    Key: HADOOP-242
    URL: http://issues.apache.org/jira/browse/HADOOP-242
    Project: Hadoop
    Type: Bug
    Components: mapred
    Reporter: Yoram Arnon
    Assignee: Sameer Paranjpye
    Attachments: no-local.patch

    when running a fairly large job, of 70+K map tasks, I get many exceptions as shown below, and eventually the job failes when a task fails four times.
    The exception doesn't really tell us enough information to debug this properly, so the first thing to do would be to add more information (path) to the exception.
    The path indicated in the config file exists, is writable and valid, though 'path' may be anything.
    the exception:
    java.io.IOException: No valid local directories in property: mapred.local.dir at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:293) at org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:153) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.localizeTask(TaskTracker.java:523) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:572) at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:389) at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:303) at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:418) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:920)
    the code:
    public Path getLocalPath(String dirsProp, String path)
    throws IOException {
    String[] dirs = getStrings(dirsProp);
    int hashCode = path.hashCode();
    FileSystem fs = FileSystem.getNamed("local", this);
    for (int i = 0; i < dirs.length; i++) { // try each local dir
    int index = (hashCode+i & Integer.MAX_VALUE) % dirs.length;
    Path file = new Path(dirs[index], path);
    Path dir = file.getParent();
    if (fs.exists(dir) || fs.mkdirs(dir)) {
    return file;
    }
    }
    throw new IOException("No valid local directories in property: "+dirsProp);
    }
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Doug Cutting (JIRA) at May 30, 2006 at 10:57 pm
    [ http://issues.apache.org/jira/browse/HADOOP-242?page=all ]

    Doug Cutting resolved HADOOP-242:
    ---------------------------------

    Fix Version: 0.3
    Resolution: Fixed

    I just committed this. Thanks, Owen.
    job fails because of "No valid local directories in property: " exception
    -------------------------------------------------------------------------

    Key: HADOOP-242
    URL: http://issues.apache.org/jira/browse/HADOOP-242
    Project: Hadoop
    Type: Bug
    Components: mapred
    Reporter: Yoram Arnon
    Assignee: Sameer Paranjpye
    Fix For: 0.3
    Attachments: no-local.patch

    when running a fairly large job, of 70+K map tasks, I get many exceptions as shown below, and eventually the job failes when a task fails four times.
    The exception doesn't really tell us enough information to debug this properly, so the first thing to do would be to add more information (path) to the exception.
    The path indicated in the config file exists, is writable and valid, though 'path' may be anything.
    the exception:
    java.io.IOException: No valid local directories in property: mapred.local.dir at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:293) at org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:153) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.localizeTask(TaskTracker.java:523) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:572) at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:389) at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:303) at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:418) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:920)
    the code:
    public Path getLocalPath(String dirsProp, String path)
    throws IOException {
    String[] dirs = getStrings(dirsProp);
    int hashCode = path.hashCode();
    FileSystem fs = FileSystem.getNamed("local", this);
    for (int i = 0; i < dirs.length; i++) { // try each local dir
    int index = (hashCode+i & Integer.MAX_VALUE) % dirs.length;
    Path file = new Path(dirs[index], path);
    Path dir = file.getParent();
    if (fs.exists(dir) || fs.mkdirs(dir)) {
    return file;
    }
    }
    throw new IOException("No valid local directories in property: "+dirsProp);
    }
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Owen O'Malley (JIRA) at Jun 1, 2006 at 8:44 pm
    [ http://issues.apache.org/jira/browse/HADOOP-242?page=all ]

    Owen O'Malley reopened HADOOP-242:
    ----------------------------------

    Assign To: Owen O'Malley (was: Sameer Paranjpye)

    My patch didn't fix the problem, so I need to track this one down.
    job fails because of "No valid local directories in property: " exception
    -------------------------------------------------------------------------

    Key: HADOOP-242
    URL: http://issues.apache.org/jira/browse/HADOOP-242
    Project: Hadoop
    Type: Bug
    Components: mapred
    Reporter: Yoram Arnon
    Assignee: Owen O'Malley
    Fix For: 0.3
    Attachments: no-local.patch

    when running a fairly large job, of 70+K map tasks, I get many exceptions as shown below, and eventually the job failes when a task fails four times.
    The exception doesn't really tell us enough information to debug this properly, so the first thing to do would be to add more information (path) to the exception.
    The path indicated in the config file exists, is writable and valid, though 'path' may be anything.
    the exception:
    java.io.IOException: No valid local directories in property: mapred.local.dir at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:293) at org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:153) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.localizeTask(TaskTracker.java:523) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:572) at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:389) at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:303) at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:418) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:920)
    the code:
    public Path getLocalPath(String dirsProp, String path)
    throws IOException {
    String[] dirs = getStrings(dirsProp);
    int hashCode = path.hashCode();
    FileSystem fs = FileSystem.getNamed("local", this);
    for (int i = 0; i < dirs.length; i++) { // try each local dir
    int index = (hashCode+i & Integer.MAX_VALUE) % dirs.length;
    Path file = new Path(dirs[index], path);
    Path dir = file.getParent();
    if (fs.exists(dir) || fs.mkdirs(dir)) {
    return file;
    }
    }
    throw new IOException("No valid local directories in property: "+dirsProp);
    }
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Doug Cutting (JIRA) at Jun 5, 2006 at 9:14 pm
    [ http://issues.apache.org/jira/browse/HADOOP-242?page=all ]

    Doug Cutting updated HADOOP-242:
    --------------------------------

    Fix Version: 0.4
    (was: 0.3.0)
    job fails because of "No valid local directories in property: " exception
    -------------------------------------------------------------------------

    Key: HADOOP-242
    URL: http://issues.apache.org/jira/browse/HADOOP-242
    Project: Hadoop
    Type: Bug
    Components: mapred
    Reporter: Yoram Arnon
    Assignee: Owen O'Malley
    Fix For: 0.4
    Attachments: no-local.patch

    when running a fairly large job, of 70+K map tasks, I get many exceptions as shown below, and eventually the job failes when a task fails four times.
    The exception doesn't really tell us enough information to debug this properly, so the first thing to do would be to add more information (path) to the exception.
    The path indicated in the config file exists, is writable and valid, though 'path' may be anything.
    the exception:
    java.io.IOException: No valid local directories in property: mapred.local.dir at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:293) at org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:153) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.localizeTask(TaskTracker.java:523) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:572) at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:389) at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:303) at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:418) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:920)
    the code:
    public Path getLocalPath(String dirsProp, String path)
    throws IOException {
    String[] dirs = getStrings(dirsProp);
    int hashCode = path.hashCode();
    FileSystem fs = FileSystem.getNamed("local", this);
    for (int i = 0; i < dirs.length; i++) { // try each local dir
    int index = (hashCode+i & Integer.MAX_VALUE) % dirs.length;
    Path file = new Path(dirs[index], path);
    Path dir = file.getParent();
    if (fs.exists(dir) || fs.mkdirs(dir)) {
    return file;
    }
    }
    throw new IOException("No valid local directories in property: "+dirsProp);
    }
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Owen O'Malley (JIRA) at Jun 23, 2006 at 4:41 pm
    [ http://issues.apache.org/jira/browse/HADOOP-242?page=all ]

    Owen O'Malley resolved HADOOP-242:
    ----------------------------------

    Fix Version: 0.3.2
    (was: 0.4.0)
    Resolution: Fixed

    I think the fix for HADOOP-277 fixed this one. We've seen it since, but only when the local dir was running out of space.
    job fails because of "No valid local directories in property: " exception
    -------------------------------------------------------------------------

    Key: HADOOP-242
    URL: http://issues.apache.org/jira/browse/HADOOP-242
    Project: Hadoop
    Type: Bug
    Components: mapred
    Reporter: Yoram Arnon
    Assignee: Owen O'Malley
    Fix For: 0.3.2
    Attachments: no-local.patch

    when running a fairly large job, of 70+K map tasks, I get many exceptions as shown below, and eventually the job failes when a task fails four times.
    The exception doesn't really tell us enough information to debug this properly, so the first thing to do would be to add more information (path) to the exception.
    The path indicated in the config file exists, is writable and valid, though 'path' may be anything.
    the exception:
    java.io.IOException: No valid local directories in property: mapred.local.dir at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:293) at org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:153) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.localizeTask(TaskTracker.java:523) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:572) at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:389) at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:303) at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:418) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:920)
    the code:
    public Path getLocalPath(String dirsProp, String path)
    throws IOException {
    String[] dirs = getStrings(dirsProp);
    int hashCode = path.hashCode();
    FileSystem fs = FileSystem.getNamed("local", this);
    for (int i = 0; i < dirs.length; i++) { // try each local dir
    int index = (hashCode+i & Integer.MAX_VALUE) % dirs.length;
    Path file = new Path(dirs[index], path);
    Path dir = file.getParent();
    if (fs.exists(dir) || fs.mkdirs(dir)) {
    return file;
    }
    }
    throw new IOException("No valid local directories in property: "+dirsProp);
    }
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMay 23, '06 at 7:16p
activeJun 23, '06 at 4:41p
posts8
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Owen O'Malley (JIRA): 8 posts

People

Translate

site design / logo © 2022 Grokbase