FAQ
if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
--------------------------------------------------------------------------------------------------------------

Key: HADOOP-3135
URL: https://issues.apache.org/jira/browse/HADOOP-3135
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.16.1
Environment: all
Reporter: Alejandro Abdelnur
Priority: Blocker
Fix For: 0.17.0


Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.

Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.

Line 696-7 in the JobClient:

String jobId = jobSubmitClient.getNewJobId();
Path submitJobDir = new Path(job.getSystemDir(), jobId);

This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.

A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.

----
For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Owen O'Malley (JIRA) at Apr 1, 2008 at 6:12 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584024#action_12584024 ]

    Owen O'Malley commented on HADOOP-3135:
    ---------------------------------------

    +1 for adding JobSubmissionProtocol.getSystemDirectoryPath.

    However, I think you should also replace InterTrackerProtocol.getFileSystemName with InterTrackerProtocol.getSystemDirectoryPath. Clearly the two getSystemDirectoryPath's are the same...

    That way, only the JobTracker needs to have the system directory defined in its configuration.

    The other piece that would be really good to pass back to the client is the default number of mappers and reducers. See HADOOP-1100.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Priority: Blocker
    Fix For: 0.17.0


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Devaraj Das (JIRA) at Apr 1, 2008 at 6:12 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Devaraj Das reassigned HADOOP-3135:
    -----------------------------------

    Assignee: Amareshwari Sriramadasu
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Amareshwari Sriramadasu
    Priority: Blocker
    Fix For: 0.17.0


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amareshwari Sriramadasu (JIRA) at Apr 2, 2008 at 9:32 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Amareshwari Sriramadasu reassigned HADOOP-3135:
    -----------------------------------------------

    Assignee: Alejandro Abdelnur (was: Amareshwari Sriramadasu)
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Alejandro Abdelnur
    Priority: Blocker
    Fix For: 0.17.0


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 2, 2008 at 12:31 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Status: Patch Available (was: Open)

    The patch has the below mentioned changes:

    1) Added 'getSystemDir()' in JobSubmissionProtocol that would return the jobtracker system dir. Job Client will use this dir for uploading all the files on submission and JobInProgress will use it for retrieval
    2) Replaced InterTrackerProtocol.getFileSystemName with InterTrackerProtocol.getSystemDir & updated TaskTracker to use this to get the FileSystem handle
    3) Added 'getDefaultMaps()' & 'getDefaultReduces' to JobClient that will return the default number of Maps and Reduces respectively.

    Also added a TestCase - TestJobSysDirWithDFS that checks if a Job will run successfully in spite of different system dir config between Job Client & Job Tracker.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Alejandro Abdelnur
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 2, 2008 at 12:31 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Attachment: patch-3135.txt
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Alejandro Abdelnur
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Devaraj Das (JIRA) at Apr 2, 2008 at 2:43 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Devaraj Das updated HADOOP-3135:
    --------------------------------

    Status: Open (was: Patch Available)

    This requires some indentation fixes. Other than that it looks good.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Alejandro Abdelnur
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Apr 2, 2008 at 2:53 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584605#action_12584605 ]

    Hadoop QA commented on HADOOP-3135:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12379126/patch-3135.txt
    against trunk revision 643282.

    @author +1. The patch does not contain any @author tags.

    tests included +1. The patch appears to include 3 new or modified tests.

    javadoc +1. The javadoc tool did not generate any warning messages.

    javac +1. The applied patch does not generate any new javac compiler warnings.

    release audit +1. The applied patch does not generate any new release audit warnings.

    findbugs +1. The patch does not introduce any new Findbugs warnings.

    core tests -1. The patch failed core unit tests.

    contrib tests +1. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2123/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2123/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2123/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2123/console

    This message is automatically generated.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Alejandro Abdelnur
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 3, 2008 at 6:54 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Attachment: patch-3135-v2.txt
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Alejandro Abdelnur
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v2.txt, patch-3135.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 3, 2008 at 6:56 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Status: Patch Available (was: Open)

    Fixed the indentation
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Alejandro Abdelnur
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v2.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 3, 2008 at 6:56 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Attachment: (was: patch-3135.txt)
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Alejandro Abdelnur
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v2.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Apr 3, 2008 at 10:37 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585326#action_12585326 ]

    Hadoop QA commented on HADOOP-3135:
    -----------------------------------

    +1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12379235/patch-3135-v2.txt
    against trunk revision 643282.

    @author +1. The patch does not contain any @author tags.

    tests included +1. The patch appears to include 3 new or modified tests.

    javadoc +1. The javadoc tool did not generate any warning messages.

    javac +1. The applied patch does not generate any new javac compiler warnings.

    release audit +1. The applied patch does not generate any new release audit warnings.

    findbugs +1. The patch does not introduce any new Findbugs warnings.

    core tests +1. The patch passed core unit tests.

    contrib tests +1. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2145/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2145/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2145/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2145/console

    This message is automatically generated.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Alejandro Abdelnur
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v2.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Owen O'Malley (JIRA) at Apr 3, 2008 at 11:27 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Owen O'Malley updated HADOOP-3135:
    ----------------------------------

    Status: Open (was: Patch Available)

    This is close, but the job client needs to get the file system from the system directory rather than getting the default file system.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Alejandro Abdelnur
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v2.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Owen O'Malley (JIRA) at Apr 4, 2008 at 4:50 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Owen O'Malley reassigned HADOOP-3135:
    -------------------------------------

    Assignee: Subramaniam Krishnan (was: Alejandro Abdelnur)
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v2.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 4, 2008 at 6:10 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585409#action_12585409 ]

    Subramaniam Krishnan commented on HADOOP-3135:
    ----------------------------------------------

    I thought I had covered that ...:-)

    public synchronized FileSystem getFs() throws IOException {
    if (this.fs == null) {
    - String fsName = jobSubmitClient.getFilesystemName();
    - this.fs = FileSystem.getNamed(fsName, getConf());
    + Path sysDir = getSystemDir();
    + this.fs = sysDir.getFileSystem(getConf());
    }
    return fs;
    }

    Am I missing something?
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v2.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Owen O'Malley (JIRA) at Apr 4, 2008 at 8:04 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585734#action_12585734 ]

    Owen O'Malley commented on HADOOP-3135:
    ---------------------------------------

    I should have been more explicit and it was actually in the JobInProgress. The offending code is:

    {code}
    @@ -171,7 +172,7 @@
    this.localJarFile = default_job_conf.getLocalPath(JobTracker.SUBDIR
    +"/"+ jobid + ".jar");
    FileSystem fs = FileSystem.get(default_conf);
    - Path jobFile = new Path(default_conf.getSystemDir(), jobid + "/job.xml");
    + Path jobFile = new Path(this.jobtracker.getSystemDir(), jobid + "/job.xml");
    fs.copyToLocalFile(jobFile, localJobFile);
    conf = new JobConf(localJobFile);
    this.priority = conf.getJobPriority();
    {code}
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v2.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 7, 2008 at 10:47 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Attachment: (was: patch-3135-v2.txt)
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v3.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 7, 2008 at 10:47 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Attachment: patch-3135-v3.txt

    Got it!

    Have update the JIP to also use the JT's system dir to get the file system handle
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v3.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 8, 2008 at 5:26 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Status: Patch Available (was: Open)
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v3.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Apr 8, 2008 at 9:18 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586719#action_12586719 ]

    Hadoop QA commented on HADOOP-3135:
    -----------------------------------

    +1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12379542/patch-3135-v3.txt
    against trunk revision 645773.

    @author +1. The patch does not contain any @author tags.

    tests included +1. The patch appears to include 3 new or modified tests.

    javadoc +1. The javadoc tool did not generate any warning messages.

    javac +1. The applied patch does not generate any new javac compiler warnings.

    release audit +1. The applied patch does not generate any new release audit warnings.

    findbugs +1. The patch does not introduce any new Findbugs warnings.

    core tests +1. The patch passed core unit tests.

    contrib tests +1. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2182/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2182/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2182/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2182/console

    This message is automatically generated.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v3.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Enis Soztutar (JIRA) at Apr 8, 2008 at 3:06 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586828#action_12586828 ]

    Enis Soztutar commented on HADOOP-3135:
    ---------------------------------------

    Two minor issues with the patch :
    * We should also change
    {code}
    Path jobDirectory = new Path(jobConf.getSystemDir(), NAME + "_" + randomId);
    ..
    FileSystem jobfs = jobDirectory.getFileSystem(jobConf);
    {code}
    in {{CopyFiles#setup(...)}}
    * We already have {{JobClient#getClusterStatus()}}, so {{JobClient#getDefaultMaps()}} and {{JobClient#getDefaultReduces()}} might not be necessary

    other than these patch looks good, +1.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v3.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Arun C Murthy (JIRA) at Apr 8, 2008 at 11:28 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Arun C Murthy updated HADOOP-3135:
    ----------------------------------

    Status: Open (was: Patch Available)

    Cancelling patch while Enis's concerns are addressed...
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v3.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amareshwari Sriramadasu (JIRA) at Apr 9, 2008 at 7:10 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587068#action_12587068 ]

    Amareshwari Sriramadasu commented on HADOOP-3135:
    -------------------------------------------------

    bq. We already have JobClient#getClusterStatus(), so JobClient#getDefaultMaps() and JobClient#getDefaultReduces() might not be necessary

    JobClient#getDefaultMaps() and JobClient#getDefaultReduces() are the wrappers around JobClient#getClusterStatus() as requested in HADOOP-1100
    So, I think these can still be added.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v3.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 9, 2008 at 7:14 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Attachment: patch-3135-v4.txt
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 9, 2008 at 7:18 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Status: Patch Available (was: Open)

    Updated with Enis comment on CopyFile.setup().
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Owen O'Malley (JIRA) at Apr 10, 2008 at 4:51 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587487#action_12587487 ]

    Owen O'Malley commented on HADOOP-3135:
    ---------------------------------------

    Please go through all of the callers to FileSystem.get(conf) and make sure they are correct. I'd also ask you to create a unit test that has the system directory on a non-default file system and submits a job to it, if you don't have one already.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Arun C Murthy (JIRA) at Apr 10, 2008 at 7:04 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Arun C Murthy updated HADOOP-3135:
    ----------------------------------

    Status: Open (was: Patch Available)

    I'm sorry to jump in late on this, but I thought this was really important to consider...

    I'd like to propose that we deprecate JobConf.getSystemDir - it really is a bogus value in many cases as this jira explains. We should remove all references to JobConf.getSystemDir in our code and change them to use JobClient.getSystemDir, which is guaranteed to get u the right value. Thoughts?
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Arun C Murthy (JIRA) at Apr 10, 2008 at 7:06 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587505#action_12587505 ]

    Arun C Murthy commented on HADOOP-3135:
    ---------------------------------------

    Since you are generating a new patch, I'd appreciate if you could weed out miscellaneous changes like these which aren't necessary:

    {noformat}
    Index: src/java/org/apache/hadoop/mapred/JobTracker.java
    ===================================================================
    --- src/java/org/apache/hadoop/mapred/JobTracker.java (revision 643322)
    +++ src/java/org/apache/hadoop/mapred/JobTracker.java (working copy)
    @@ -37,7 +37,6 @@
    import java.util.Properties;
    import java.util.Set;
    import java.util.TreeMap;
    -import java.util.HashMap;
    import java.util.TreeSet;
    import java.util.Vector;
    import java.util.concurrent.LinkedBlockingQueue;
    @@ -51,18 +50,18 @@
    import org.apache.hadoop.fs.permission.FsPermission;
    import org.apache.hadoop.ipc.RPC;
    import org.apache.hadoop.ipc.RemoteException;
    +import org.apache.hadoop.ipc.Server;
    import org.apache.hadoop.ipc.RPC.VersionMismatch;
    -import org.apache.hadoop.ipc.Server;
    import org.apache.hadoop.metrics.MetricsContext;
    import org.apache.hadoop.metrics.MetricsRecord;
    import org.apache.hadoop.metrics.MetricsUtil;
    import org.apache.hadoop.metrics.Updater;
    import org.apache.hadoop.metrics.jvm.JvmMetrics;
    import org.apache.hadoop.net.DNSToSwitchMapping;
    +import org.apache.hadoop.net.NetUtils;
    import org.apache.hadoop.net.NetworkTopology;
    +import org.apache.hadoop.net.Node;
    import org.apache.hadoop.net.NodeBase;
    -import org.apache.hadoop.net.Node;
    -import org.apache.hadoop.net.NetUtils;
    import org.apache.hadoop.net.ScriptBasedMapping;
    import org.apache.hadoop.util.HostsFileReader;
    import org.apache.hadoop.util.ReflectionUtils;
    {noformat}

    Oh, we probably should cache the value of the system.dir in JobClient.getSystemDir rather than do an RPC to get it each time...
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Blocker
    Fix For: 0.17.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Owen O'Malley (JIRA) at Apr 16, 2008 at 6:35 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Owen O'Malley updated HADOOP-3135:
    ----------------------------------

    Priority: Critical (was: Blocker)
    Fix Version/s: (was: 0.17.0)
    0.18.0

    This is a *very* good thing to fix, but it is neither a bug or an incompatibility with 0.16. Furthermore, this is very likely to be a very difficult patch to test and get right.

    I'm moving it to 0.18.

    When this is resolved, it will fix my comment on HADOOP-1100.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 16, 2008 at 6:41 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Attachment: patch-3135-v5.txt
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 16, 2008 at 6:53 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Status: Patch Available (was: Open)

    I have updated the patch with the following changes:

    1) Went through the call hierarchy of FileSystem.get(conf) with Amareshwari's help and updated a few instances to make sure they are correct(in my limited knowledge).
    2) Deprecated JobConf.getSystemDir. Replaced all references to JobConf.getSystemDir in our code and change them to use JobClient.getSystemDir.
    3) Add caching for the value of the system directory in JobClient.getSystemDir.

    I already have added a test case that checks if a Job run successfully in spite of different system directory configuration between Job Client & Job Tracker. I couldn't set the system directory to a non-default file system as the MiniMRCluster doesn't support it currently. Have raised a jira for the same - [HADOOP-3244][https://issues.apache.org/jira/browse/HADOOP-3244]

    I haven't changed the imports as they were giving me warnings & are much better organized now.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 16, 2008 at 6:55 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589438#action_12589438 ]

    subru edited comment on HADOOP-3135 at 4/15/08 11:50 PM:
    ------------------------------------------------------------------------

    I have updated the patch with the following changes:

    1) Went through the call hierarchy of FileSystem.get(conf) with Amareshwari's help and updated a few instances to make sure they are correct(in my limited knowledge).
    2) Deprecated JobConf.getSystemDir. Replaced all references to JobConf.getSystemDir in our code and change them to use JobClient.getSystemDir.
    3) Add caching for the value of the system directory in JobClient.getSystemDir.

    I already have added a test case that checks if a Job run successfully in spite of different system directory configuration between Job Client & Job Tracker. I couldn't set the system directory to a non-default file system as the MiniMRCluster doesn't support it currently. Have raised a jira for the same - [HADOOP-3244]

    I haven't changed the imports as they were giving me warnings & are much better organized now.

    was (Author: subru):
    I have updated the patch with the following changes:

    1) Went through the call hierarchy of FileSystem.get(conf) with Amareshwari's help and updated a few instances to make sure they are correct(in my limited knowledge).
    2) Deprecated JobConf.getSystemDir. Replaced all references to JobConf.getSystemDir in our code and change them to use JobClient.getSystemDir.
    3) Add caching for the value of the system directory in JobClient.getSystemDir.

    I already have added a test case that checks if a Job run successfully in spite of different system directory configuration between Job Client & Job Tracker. I couldn't set the system directory to a non-default file system as the MiniMRCluster doesn't support it currently. Have raised a jira for the same - [HADOOP-3244][https://issues.apache.org/jira/browse/HADOOP-3244]

    I haven't changed the imports as they were giving me warnings & are much better organized now.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Apr 16, 2008 at 2:10 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589567#action_12589567 ]

    Hadoop QA commented on HADOOP-3135:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12380251/patch-3135-v5.txt
    against trunk revision 645773.

    @author +1. The patch does not contain any @author tags.

    tests included +1. The patch appears to include 6 new or modified tests.

    javadoc +1. The javadoc tool did not generate any warning messages.

    javac +1. The applied patch does not generate any new javac compiler warnings.

    release audit +1. The applied patch does not generate any new release audit warnings.

    findbugs +1. The patch does not introduce any new Findbugs warnings.

    core tests -1. The patch failed core unit tests.

    contrib tests -1. The patch failed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2253/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2253/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2253/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2253/console

    This message is automatically generated.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 17, 2008 at 8:40 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Attachment: patch-3135-v6.txt
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 17, 2008 at 8:40 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Status: Open (was: Patch Available)

    I am canceling this patch as I am going to submit a new one with a fix in LocalJobRunner.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 17, 2008 at 8:44 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Status: Patch Available (was: Open)


    LocalJobRunner was not setting the local Job File for the Map/Reduce Tasks, so updated it to set the same.

    This patch also contains an update in the Javadoc of the deprecated JobConf.getSystemDir() to link to the JobClient.getSystemDir().
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Apr 17, 2008 at 9:54 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589913#action_12589913 ]

    Hadoop QA commented on HADOOP-3135:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12380374/patch-3135-v6.txt
    against trunk revision 645773.

    @author +1. The patch does not contain any @author tags.

    tests included +1. The patch appears to include 6 new or modified tests.

    patch -1. The patch command could not apply the patch.

    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2262/console

    This message is automatically generated.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 17, 2008 at 12:46 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Attachment: patch-3135-v6_2.txt
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt, patch-3135-v6_2.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 17, 2008 at 12:46 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Status: Open (was: Patch Available)
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt, patch-3135-v6_2.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 17, 2008 at 12:48 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Status: Patch Available (was: Open)


    Updated the patch after sync-ing with trunk
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt, patch-3135-v6_2.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Apr 17, 2008 at 12:58 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12590012#action_12590012 ]

    Hadoop QA commented on HADOOP-3135:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12380387/patch-3135-v6_2.txt
    against trunk revision 645773.

    @author +1. The patch does not contain any @author tags.

    tests included +1. The patch appears to include 6 new or modified tests.

    patch -1. The patch command could not apply the patch.

    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2263/console

    This message is automatically generated.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt, patch-3135-v6_2.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 18, 2008 at 5:48 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Status: Open (was: Patch Available)
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt, patch-3135-v6_2.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 18, 2008 at 5:50 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Attachment: patch-3135-v6_3.txt
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt, patch-3135-v6_2.txt, patch-3135-v6_3.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Apr 18, 2008 at 5:50 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Status: Patch Available (was: Open)

    Corrected the file paths in patch file
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt, patch-3135-v6_2.txt, patch-3135-v6_3.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Apr 20, 2008 at 2:37 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12590768#action_12590768 ]

    Hadoop QA commented on HADOOP-3135:
    -----------------------------------

    +1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12380470/patch-3135-v6_3.txt
    against trunk revision 645773.

    @author +1. The patch does not contain any @author tags.

    tests included +1. The patch appears to include 6 new or modified tests.

    javadoc +1. The javadoc tool did not generate any warning messages.

    javac +1. The applied patch does not generate any new javac compiler warnings.

    release audit +1. The applied patch does not generate any new release audit warnings.

    findbugs +1. The patch does not introduce any new Findbugs warnings.

    core tests +1. The patch passed core unit tests.

    contrib tests +1. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2280/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2280/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2280/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2280/console

    This message is automatically generated.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt, patch-3135-v6_2.txt, patch-3135-v6_3.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Owen O'Malley (JIRA) at May 8, 2008 at 2:32 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Owen O'Malley updated HADOOP-3135:
    ----------------------------------

    Status: Open (was: Patch Available)

    I'm sorry, but this patch is out of sync with respect to trunk. Can you fix it once more?


    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt, patch-3135-v6_2.txt, patch-3135-v6_3.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at May 12, 2008 at 9:32 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596015#action_12596015 ]

    Subramaniam Krishnan commented on HADOOP-3135:
    ----------------------------------------------


    I am just waiting for the patch for Hadoop-3244 to go in so that I update the test case to set the system directory to a non-default file system.
    Then I'll sync & submit once again.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt, patch-3135-v6_2.txt, patch-3135-v6_3.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Devaraj Das (JIRA) at May 22, 2008 at 4:00 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598900#action_12598900 ]

    Devaraj Das commented on HADOOP-3135:
    -------------------------------------

    Now that HADOOP-3296 has been committed, could you pls submit a new patch with the testcase.
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt, patch-3135-v6_2.txt, patch-3135-v6_3.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at May 26, 2008 at 2:12 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Attachment: patch-3135-v7.txt
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt, patch-3135-v6_2.txt, patch-3135-v6_3.txt, patch-3135-v7.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at May 26, 2008 at 2:14 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Status: Patch Available (was: Open)


    I have made the following changes:

    1) Sync-ed with the Trunk.
    2) Updated the Test Case to make the JobTracker use a non-default file system.
    3) Updated FileSystem.get(conf) call in Multi File Splitter(had missed it previously).
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt, patch-3135-v6_2.txt, patch-3135-v6_3.txt, patch-3135-v7.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Subramaniam Krishnan (JIRA) at Jun 2, 2008 at 9:04 am
    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Subramaniam Krishnan updated HADOOP-3135:
    -----------------------------------------

    Status: Patch Available (was: Open)


    Submitting same patch once again
    if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
    --------------------------------------------------------------------------------------------------------------

    Key: HADOOP-3135
    URL: https://issues.apache.org/jira/browse/HADOOP-3135
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Affects Versions: 0.16.1
    Environment: all
    Reporter: Alejandro Abdelnur
    Assignee: Subramaniam Krishnan
    Priority: Critical
    Fix For: 0.18.0

    Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt, patch-3135-v6.txt, patch-3135-v6_2.txt, patch-3135-v6_3.txt, patch-3135-v7.txt


    Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting files in DFS was given by the client to the jobtracker.
    Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from the jobtracker and creating the directory for all the supporting files using the a system-dir computed from the local jobconf.
    Line 696-7 in the JobClient:
    String jobId = jobSubmitClient.getNewJobId();
    Path submitJobDir = new Path(job.getSystemDir(), jobId);
    This makes submissions to fail when the value of the 'mapred.system.dir' on the client is different from the one in the JobTracker.
    A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading all the files on submission.
    ----
    For the future: A more comprehensive way of this doing would to obtain a base jobConf from the jobtracker, carrying final information for each element and the construct the job.xml on the client using the final semantics. And, in this case the 'mapred.system.dir' property should be set as final in the jobtracker. As there may be some configuration properties that are sensitive and for security reasons should not be exposed to the clients a new flag 'private' could be introduced and only properties that don't have the 'private' flag would be send over from the jobtracker to the jobclient for job.xml resolution.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMar 31, '08 at 10:54a
activeAug 11, '08 at 2:40p
posts70
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Steve Loughran (JIRA): 70 posts

People

Translate

site design / logo © 2022 Grokbase