FAQ
setting fs.default.name to an invalid URI format kills init thread in JobTracker
--------------------------------------------------------------------------------

Key: HADOOP-5140
URL: https://issues.apache.org/jira/browse/HADOOP-5140
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.19.0
Reporter: sam rash
Priority: Minor


If you set fs.default.name in a JobConf object to something that causes java.net.URI to throw an IllegalArgumentException, the job not only fails initalization, but kills the JobInitThread

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • sam rash (JIRA) at Jan 29, 2009 at 2:29 am
    [ https://issues.apache.org/jira/browse/HADOOP-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668293#action_12668293 ]

    sam rash commented on HADOOP-5140:
    ----------------------------------

    first stack trace:
    2009-01-28 18:15:41,531 ERROR org.apache.hadoop.mapred.EagerTaskInitializationListener: Job initialization failed:
    java.lang.IllegalArgumentException
    at java.net.URI.create(URI.java:842)
    at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:128)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:208)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
    at org.apache.hadoop.mapred.JobHistory$JobInfo.getJobHistoryFileName(JobHistory.java:630)
    at org.apache.hadoop.mapred.JobHistory$JobInfo.logSubmitted(JobHistory.java:803)
    at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:353)
    at org.apache.hadoop.mapred.EagerTaskInitializationListener$JobInitThread.run(EagerTaskInitializationListener.java:55)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: java.net.URISyntaxException: Illegal character in authority at index 7: hdfs://${name}
    at java.net.URI$Parser.fail(URI.java:2809)
    at java.net.URI$Parser.parseAuthority(URI.java:3147)
    at java.net.URI$Parser.parseHierarchical(URI.java:3058)
    at java.net.URI$Parser.parse(URI.java:3014)
    at java.net.URI.(URI.java:840)
    ... 8 more

    code with the problem:

    class JobInitThread implements Runnable {
    public void run() {
    JobInProgress job;
    while (true) {
    job = null;
    try {
    synchronized (jobInitQueue) {
    while (jobInitQueue.isEmpty()) {
    jobInitQueue.wait();
    }
    job = jobInitQueue.remove(0);
    }
    job.initTasks();
    } catch (InterruptedException t) {
    break;
    } catch (Throwable t) {
    LOG.error("Job initialization failed:\n" +
    StringUtils.stringifyException(t));
    if (job != null) {
    job.fail();
    }
    }
    }
    }
    }

    job.fail() eventually calls garbageCollect() which trys to get the filesystem from the jobconf. The same exceptoin as above will occur when it trys to cleanup the dfs tmp dir, but this time it's not caught (being thrown inside the catch() block) and kills the JobInitThread
    setting fs.default.name to an invalid URI format kills init thread in JobTracker
    --------------------------------------------------------------------------------

    Key: HADOOP-5140
    URL: https://issues.apache.org/jira/browse/HADOOP-5140
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.19.0
    Reporter: sam rash
    Priority: Minor

    If you set fs.default.name in a JobConf object to something that causes java.net.URI to throw an IllegalArgumentException, the job not only fails initalization, but kills the JobInitThread
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • sam rash (JIRA) at Jan 29, 2009 at 2:31 am
    [ https://issues.apache.org/jira/browse/HADOOP-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668294#action_12668294 ]

    sam rash commented on HADOOP-5140:
    ----------------------------------

    and this manifests by the JobTracker never launching any more jobs (it receives ok and the jobs submitted count increases)
    setting fs.default.name to an invalid URI format kills init thread in JobTracker
    --------------------------------------------------------------------------------

    Key: HADOOP-5140
    URL: https://issues.apache.org/jira/browse/HADOOP-5140
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.19.0
    Reporter: sam rash
    Priority: Minor

    If you set fs.default.name in a JobConf object to something that causes java.net.URI to throw an IllegalArgumentException, the job not only fails initalization, but kills the JobInitThread
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Steve Loughran (JIRA) at May 22, 2009 at 8:23 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712237#action_12712237 ]

    Steve Loughran commented on HADOOP-5140:
    ----------------------------------------

    This is only loosely related to HADOOP-5687. That was bad failing on invalid fs.default.name values in the NN. This is bad handling of the (correct) failure to parse bad fs.default.name values. The changes I'm doing for HADOOP-5687 will change the error message to be more helpful, but you still need that code in the JobInitThread to catch the bad value during the cleanup process, which is what is apparently happening here.
    setting fs.default.name to an invalid URI format kills init thread in JobTracker
    --------------------------------------------------------------------------------

    Key: HADOOP-5140
    URL: https://issues.apache.org/jira/browse/HADOOP-5140
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.19.0
    Reporter: sam rash
    Priority: Minor

    If you set fs.default.name in a JobConf object to something that causes java.net.URI to throw an IllegalArgumentException, the job not only fails initalization, but kills the JobInitThread
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedJan 29, '09 at 2:27a
activeMay 22, '09 at 8:23p
posts4
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Steve Loughran (JIRA): 4 posts

People

Translate

site design / logo © 2022 Grokbase