FAQ
Hello all,

I deployed hadoop to a small cluster. The HDFS is running as user A.
Now user B comes in and wants to run a simple Map-Reduce task. The
Map-Reduce client creates all shared files in correct location
'/tmp/hadoop-B/mapred/system/...'. However, JobTracker, running as
user A, always tries to read the job related files from
'/tmp/hadoop-A/mapred/system/...'.

The questions are following.
a) Does everybody who wants to run map-reduce job have to launch one's
own JobTracker?
b) Do I have to set HOD for this? I really don't want to do this extra
since B is currently the only user running Map-Reduce.

Thank you,
YongChul

Search Discussions

  • Haijun Cao at Jun 26, 2008 at 7:21 pm
    Is it because you leave the mapred.system.dir as default (see
    hadoop-default.xml)?

    Haijun

    -----Original Message-----
    From: YongChul Kwon
    Sent: Thursday, June 26, 2008 12:10 PM
    To: core-user@hadoop.apache.org
    Subject: Sharing Hadoop cluster among multiple users

    Hello all,

    I deployed hadoop to a small cluster. The HDFS is running as user A.
    Now user B comes in and wants to run a simple Map-Reduce task. The
    Map-Reduce client creates all shared files in correct location
    '/tmp/hadoop-B/mapred/system/...'. However, JobTracker, running as
    user A, always tries to read the job related files from
    '/tmp/hadoop-A/mapred/system/...'.

    The questions are following.
    a) Does everybody who wants to run map-reduce job have to launch one's
    own JobTracker?
    b) Do I have to set HOD for this? I really don't want to do this extra
    since B is currently the only user running Map-Reduce.

    Thank you,
    YongChul
  • YongChul Kwon at Jun 26, 2008 at 7:46 pm
    Hi Hiajun,

    Yes, they, path related to shared things for map reduce, set to
    default. I thought there is a reason for the default values containing
    'username' in them. Is it safe to use some global path, say dropping
    the username reference from the default values?

    Thank you,
    YongChul
    On Thu, Jun 26, 2008 at 12:19 PM, Haijun Cao wrote:
    Is it because you leave the mapred.system.dir as default (see
    hadoop-default.xml)?

    Haijun

    -----Original Message-----
    From: YongChul Kwon
    Sent: Thursday, June 26, 2008 12:10 PM
    To: core-user@hadoop.apache.org
    Subject: Sharing Hadoop cluster among multiple users

    Hello all,

    I deployed hadoop to a small cluster. The HDFS is running as user A.
    Now user B comes in and wants to run a simple Map-Reduce task. The
    Map-Reduce client creates all shared files in correct location
    '/tmp/hadoop-B/mapred/system/...'. However, JobTracker, running as
    user A, always tries to read the job related files from
    '/tmp/hadoop-A/mapred/system/...'.

    The questions are following.
    a) Does everybody who wants to run map-reduce job have to launch one's
    own JobTracker?
    b) Do I have to set HOD for this? I really don't want to do this extra
    since B is currently the only user running Map-Reduce.

    Thank you,
    YongChul


    --
    YongChul
  • Haijun Cao at Jun 26, 2008 at 10:42 pm
    I don't think having the username in the path provides safety either
    (other than it forces separate cluster).

    Hadoop file system has access control (but without authentication) that
    prevents users stepping on each other's toes.



    Haijun

    -----Original Message-----
    From: YongChul Kwon
    Sent: Thursday, June 26, 2008 12:45 PM
    To: core-user@hadoop.apache.org
    Subject: Re: Sharing Hadoop cluster among multiple users

    Hi Hiajun,

    Yes, they, path related to shared things for map reduce, set to
    default. I thought there is a reason for the default values containing
    'username' in them. Is it safe to use some global path, say dropping
    the username reference from the default values?

    Thank you,
    YongChul
    On Thu, Jun 26, 2008 at 12:19 PM, Haijun Cao wrote:
    Is it because you leave the mapred.system.dir as default (see
    hadoop-default.xml)?

    Haijun

    -----Original Message-----
    From: YongChul Kwon
    Sent: Thursday, June 26, 2008 12:10 PM
    To: core-user@hadoop.apache.org
    Subject: Sharing Hadoop cluster among multiple users

    Hello all,

    I deployed hadoop to a small cluster. The HDFS is running as user A.
    Now user B comes in and wants to run a simple Map-Reduce task. The
    Map-Reduce client creates all shared files in correct location
    '/tmp/hadoop-B/mapred/system/...'. However, JobTracker, running as
    user A, always tries to read the job related files from
    '/tmp/hadoop-A/mapred/system/...'.

    The questions are following.
    a) Does everybody who wants to run map-reduce job have to launch one's
    own JobTracker?
    b) Do I have to set HOD for this? I really don't want to do this extra
    since B is currently the only user running Map-Reduce.

    Thank you,
    YongChul


    --
    YongChul
  • YongChul Kwon at Jun 27, 2008 at 12:21 am
    Hi,

    I meant conflicts in file system name space not the permission.
    Anyway, I found a related JIRA entry for this refers the setting of
    mapred.system.dir.

    Thank you for your responses! :-)
    YongChul
    On Thu, Jun 26, 2008 at 3:40 PM, Haijun Cao wrote:
    I don't think having the username in the path provides safety either
    (other than it forces separate cluster).

    Hadoop file system has access control (but without authentication) that
    prevents users stepping on each other's toes.



    Haijun

    -----Original Message-----
    From: YongChul Kwon
    Sent: Thursday, June 26, 2008 12:45 PM
    To: core-user@hadoop.apache.org
    Subject: Re: Sharing Hadoop cluster among multiple users

    Hi Hiajun,

    Yes, they, path related to shared things for map reduce, set to
    default. I thought there is a reason for the default values containing
    'username' in them. Is it safe to use some global path, say dropping
    the username reference from the default values?

    Thank you,
    YongChul
    On Thu, Jun 26, 2008 at 12:19 PM, Haijun Cao wrote:
    Is it because you leave the mapred.system.dir as default (see
    hadoop-default.xml)?

    Haijun

    -----Original Message-----
    From: YongChul Kwon
    Sent: Thursday, June 26, 2008 12:10 PM
    To: core-user@hadoop.apache.org
    Subject: Sharing Hadoop cluster among multiple users

    Hello all,

    I deployed hadoop to a small cluster. The HDFS is running as user A.
    Now user B comes in and wants to run a simple Map-Reduce task. The
    Map-Reduce client creates all shared files in correct location
    '/tmp/hadoop-B/mapred/system/...'. However, JobTracker, running as
    user A, always tries to read the job related files from
    '/tmp/hadoop-A/mapred/system/...'.

    The questions are following.
    a) Does everybody who wants to run map-reduce job have to launch one's
    own JobTracker?
    b) Do I have to set HOD for this? I really don't want to do this extra
    since B is currently the only user running Map-Reduce.

    Thank you,
    YongChul


    --
    YongChul


    --
    YongChul

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJun 26, '08 at 7:10p
activeJun 27, '08 at 12:21a
posts5
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

YongChul Kwon: 3 posts Haijun Cao: 2 posts

People

Translate

site design / logo © 2022 Grokbase