FAQ
mapred.job.tracker default value/docs appear out of sync with code
------------------------------------------------------------------

Key: HADOOP-3437
URL: https://issues.apache.org/jira/browse/HADOOP-3437
Project: Hadoop Core
Issue Type: Bug
Components: conf
Affects Versions: 0.18.0
Reporter: Steve Loughran
Priority: Minor


This is the settings for mapred.job.tracker in SVN_HEAD. If you dont override these, you get a RuntimeException
<property>
<name>mapred.job.tracker</name>
<value>local</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Steve Loughran (JIRA) at May 23, 2008 at 11:20 am
    [ https://issues.apache.org/jira/browse/HADOOP-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599330#action_12599330 ]

    Steve Loughran commented on HADOOP-3437:
    ----------------------------------------

    stack trace when mapred.job.tracker=local

    Caused by: java.lang.RuntimeException: Not a host:port pair: local
    at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:132)
    at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:119)
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:755)
    at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:646)

    //my code from here
    at org.apache.hadoop.mapred.ExtJobTracker.(ExtJobTracker.java:41)
    at org.smartfrog.services.hadoop.components.tracker.JobTrackerImpl.sfStart(JobTrackerImpl.java:53)
    at org.smartfrog.sfcore.compound.CompoundImpl.sfStartChildren(CompoundImpl.java:659)
    at org.smartfrog.sfcore.compound.CompoundImpl.sfStart(CompoundImpl.java:632)
    at org.smartfrog.services.assertions.TestCompoundImpl.sfStart(TestCompoundImpl.java:249)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
    at sun.rmi.transport.Transport$1.run(Transport.java:159)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:619)
    mapred.job.tracker default value/docs appear out of sync with code
    ------------------------------------------------------------------

    Key: HADOOP-3437
    URL: https://issues.apache.org/jira/browse/HADOOP-3437
    Project: Hadoop Core
    Issue Type: Bug
    Components: conf
    Affects Versions: 0.18.0
    Reporter: Steve Loughran
    Priority: Minor

    This is the settings for mapred.job.tracker in SVN_HEAD. If you dont override these, you get a RuntimeException
    <property>
    <name>mapred.job.tracker</name>
    <value>local</value>
    <description>The host and port that the MapReduce job tracker runs
    at. If "local", then jobs are run in-process as a single map
    and reduce task.
    </description>
    </property>
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Steve Loughran (JIRA) at May 23, 2008 at 11:22 am
    [ https://issues.apache.org/jira/browse/HADOOP-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599331#action_12599331 ]

    Steve Loughran commented on HADOOP-3437:
    ----------------------------------------

    I should add that this is a change since 0.16.3; so "local" used to be valid. A new default value/comment string is probably appropriate
    mapred.job.tracker default value/docs appear out of sync with code
    ------------------------------------------------------------------

    Key: HADOOP-3437
    URL: https://issues.apache.org/jira/browse/HADOOP-3437
    Project: Hadoop Core
    Issue Type: Bug
    Components: conf
    Affects Versions: 0.18.0
    Reporter: Steve Loughran
    Priority: Minor

    This is the settings for mapred.job.tracker in SVN_HEAD. If you dont override these, you get a RuntimeException
    <property>
    <name>mapred.job.tracker</name>
    <value>local</value>
    <description>The host and port that the MapReduce job tracker runs
    at. If "local", then jobs are run in-process as a single map
    and reduce task.
    </description>
    </property>
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at May 23, 2008 at 5:26 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599441#action_12599441 ]

    Doug Cutting commented on HADOOP-3437:
    --------------------------------------

    If mapred.job.tracker=local, then the JobTracker class should not be used, but rather LocalRunner. This logic is in JobClient. There is no corresponding server-side logic, since, when mapred.job.tracker=local, it is assumed that you are not starting any server daemons.

    What do you expect to happen here?

    mapred.job.tracker default value/docs appear out of sync with code
    ------------------------------------------------------------------

    Key: HADOOP-3437
    URL: https://issues.apache.org/jira/browse/HADOOP-3437
    Project: Hadoop Core
    Issue Type: Bug
    Components: conf
    Affects Versions: 0.18.0
    Reporter: Steve Loughran
    Priority: Minor

    This is the settings for mapred.job.tracker in SVN_HEAD. If you dont override these, you get a RuntimeException
    <property>
    <name>mapred.job.tracker</name>
    <value>local</value>
    <description>The host and port that the MapReduce job tracker runs
    at. If "local", then jobs are run in-process as a single map
    and reduce task.
    </description>
    </property>
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Erik Forsberg (JIRA) at Jun 15, 2009 at 6:55 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719707#action_12719707 ]

    Erik Forsberg commented on HADOOP-3437:
    ---------------------------------------

    Could the problem be IPv6-related? I'm getting this error when trying out the configuration listed at http://hadoop.apache.org/core/docs/current/quickstart.html#PseudoDistributed

    Reading http://wiki.github.com/klbostee/dumbo/short-tutorial, I got the idea to explicitly set my real IPv4 address instead of 0.0.0.0 for values where this was the default.

    I.e., I replaced 0.0.0.0 with my real IP address for dfs.secondary.http.address, dfs.datanode.address, dfs.datanode.http.address, dfs.datanode.ipc.address, dfs.http.address, dfs.datanode.https.address, dfs.https.address, mapred.job.tracker.http.address, and mapred.task.tracker.http.address. Setting 127.0.0.1 would probably have worked as well.

    That solved the problem.
    mapred.job.tracker default value/docs appear out of sync with code
    ------------------------------------------------------------------

    Key: HADOOP-3437
    URL: https://issues.apache.org/jira/browse/HADOOP-3437
    Project: Hadoop Core
    Issue Type: Bug
    Components: conf
    Affects Versions: 0.18.0
    Reporter: Steve Loughran
    Priority: Minor

    This is the settings for mapred.job.tracker in SVN_HEAD. If you dont override these, you get a RuntimeException
    <property>
    <name>mapred.job.tracker</name>
    <value>local</value>
    <description>The host and port that the MapReduce job tracker runs
    at. If "local", then jobs are run in-process as a single map
    and reduce task.
    </description>
    </property>
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Erik Forsberg (JIRA) at Jun 15, 2009 at 7:33 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719707#action_12719707 ]

    Erik Forsberg edited comment on HADOOP-3437 at 6/15/09 12:31 PM:
    -----------------------------------------------------------------

    Could the problem be IPv6-related? I'm getting this error when trying out the configuration listed at http://hadoop.apache.org/core/docs/current/quickstart.html#PseudoDistributed

    Reading http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Single-Node_Cluster)#Disabling_IPv6, I got the idea to explicitly set 127.0.0.1 instead of 0.0.0.0 for values where this was the default.

    I.e., I replaced 0.0.0.0 with 127.0.0.1 for dfs.secondary.http.address, dfs.datanode.address, dfs.datanode.http.address, dfs.datanode.ipc.address, dfs.http.address, dfs.datanode.https.address, dfs.https.address, mapred.job.tracker.http.address, and mapred.task.tracker.http.address.

    That solved the problem.

    was (Author: forsberg):
    Could the problem be IPv6-related? I'm getting this error when trying out the configuration listed at http://hadoop.apache.org/core/docs/current/quickstart.html#PseudoDistributed

    Reading http://wiki.github.com/klbostee/dumbo/short-tutorial, I got the idea to explicitly set my real IPv4 address instead of 0.0.0.0 for values where this was the default.

    I.e., I replaced 0.0.0.0 with my real IP address for dfs.secondary.http.address, dfs.datanode.address, dfs.datanode.http.address, dfs.datanode.ipc.address, dfs.http.address, dfs.datanode.https.address, dfs.https.address, mapred.job.tracker.http.address, and mapred.task.tracker.http.address. Setting 127.0.0.1 would probably have worked as well.

    That solved the problem.
    mapred.job.tracker default value/docs appear out of sync with code
    ------------------------------------------------------------------

    Key: HADOOP-3437
    URL: https://issues.apache.org/jira/browse/HADOOP-3437
    Project: Hadoop Core
    Issue Type: Bug
    Components: conf
    Affects Versions: 0.18.0
    Reporter: Steve Loughran
    Priority: Minor

    This is the settings for mapred.job.tracker in SVN_HEAD. If you dont override these, you get a RuntimeException
    <property>
    <name>mapred.job.tracker</name>
    <value>local</value>
    <description>The host and port that the MapReduce job tracker runs
    at. If "local", then jobs are run in-process as a single map
    and reduce task.
    </description>
    </property>
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Erik Forsberg (JIRA) at Jun 15, 2009 at 7:46 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719736#action_12719736 ]

    Erik Forsberg commented on HADOOP-3437:
    ---------------------------------------

    After some fiddling I found that the solution I proposed in the previous comment does work halfway - you can submit jobs, but the reduce processes fails to connect to hdfs to fetch files.

    I found a better workaround:

    1) Leave the configuration as detailed at http://hadoop.apache.org/core/docs/current/quickstart.html#PseudoDistributed

    2) Disable IPv6 in Java by editing conf/hadoop-env.sh, as follows:

    HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
    mapred.job.tracker default value/docs appear out of sync with code
    ------------------------------------------------------------------

    Key: HADOOP-3437
    URL: https://issues.apache.org/jira/browse/HADOOP-3437
    Project: Hadoop Core
    Issue Type: Bug
    Components: conf
    Affects Versions: 0.18.0
    Reporter: Steve Loughran
    Priority: Minor

    This is the settings for mapred.job.tracker in SVN_HEAD. If you dont override these, you get a RuntimeException
    <property>
    <name>mapred.job.tracker</name>
    <value>local</value>
    <description>The host and port that the MapReduce job tracker runs
    at. If "local", then jobs are run in-process as a single map
    and reduce task.
    </description>
    </property>
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Steve Loughran (JIRA) at Jun 16, 2009 at 12:35 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720099#action_12720099 ]

    Steve Loughran commented on HADOOP-3437:
    ----------------------------------------

    1. JobTracker should clearly recognise that the {{mapred.job.tracker}} ref is local, and refuse to play, with a more useful message.

    2. Erik, I think your problem is separate, and perhaps should be its own bugrep.

    We should disable IPv6 anyway; created a bug report for that HADOOP-6056 -as Ubuntu 9.04 doesnt let you turn IPv6 off in the kernel, doing it in Java is the best way to kill it.
    mapred.job.tracker default value/docs appear out of sync with code
    ------------------------------------------------------------------

    Key: HADOOP-3437
    URL: https://issues.apache.org/jira/browse/HADOOP-3437
    Project: Hadoop Core
    Issue Type: Bug
    Components: conf
    Affects Versions: 0.18.0
    Reporter: Steve Loughran
    Priority: Minor

    This is the settings for mapred.job.tracker in SVN_HEAD. If you dont override these, you get a RuntimeException
    <property>
    <name>mapred.job.tracker</name>
    <value>local</value>
    <description>The host and port that the MapReduce job tracker runs
    at. If "local", then jobs are run in-process as a single map
    and reduce task.
    </description>
    </property>
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMay 23, '08 at 11:20a
activeJun 16, '09 at 12:35p
posts8
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Steve Loughran (JIRA): 8 posts

People

Translate

site design / logo © 2022 Grokbase