FAQ
Hi,
I need to install hadoop on 16-node cluster. I have a couple of related
questions:
1. I have installed hadoop on a shared directory, i.e., there is just one
place where the whole hadoop installation files exist and all the 16 nodes
use the same installation.
Is that an issue or I need to install hadoop on each of these nodes in their
local directory separately?
2. I installed hadoop-0.21 and after following the installation
instructions, when i tried formatting, I get the following error:

************************************************************/
Re-format filesystem in /var/tmp/data/dfs/name ? (Y or N) Y
11/04/13 09:16:23 INFO namenode.FSNamesystem: defaultReplication = 3
11/04/13 09:16:23 INFO namenode.FSNamesystem: maxReplication = 512
11/04/13 09:16:23 INFO namenode.FSNamesystem: minReplication = 1
11/04/13 09:16:23 INFO namenode.FSNamesystem: maxReplicationStreams = 2
11/04/13 09:16:23 INFO namenode.FSNamesystem: shouldCheckForEnoughRacks =
false
11/04/13 09:16:23 INFO security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
cacheTimeout=300000
11/04/13 09:16:23 INFO namenode.FSNamesystem: fsOwner=bus145
11/04/13 09:16:23 INFO namenode.FSNamesystem: supergroup=supergroup
11/04/13 09:16:23 INFO namenode.FSNamesystem: isPermissionEnabled=true
11/04/13 09:16:23 INFO namenode.FSNamesystem: isAccessTokenEnabled=false
accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
11/04/13 09:16:24 INFO common.Storage: Cannot lock storage
/var/tmp/data/dfs/name. The directory is already locked.
11/04/13 09:16:24 ERROR namenode.NameNode: java.io.IOException: Cannot lock
storage /var/tmp/data/dfs/name. The directory is already locked.
at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:617)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1426)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1444)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1242)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1348)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)

11/04/13 09:16:24 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at inti79.cse.psu.edu/130.203.58.207
************************************************************/

3. I was using before hadoop-0.20, and formatting was working fine.
4. Also, when i do a bin/start-dfs.sh, am able to see the Namenode, Datanode
up, however on bin/start-mapred.sh, am not able to see Jobtracker up on the
master node, though Tasktracker seems to be up on slaves.

Before upgrading to Hadoop-0.21, everything was working fine with
hadoop-0.20 including running benchmarks and getting stats.

Any suggestions in this regard is highly appreciated.

Thanks,
Bikash

Search Discussions

  • Bikash sharma at Apr 13, 2011 at 1:46 pm
    p.s.
    Also, while starting dfs using bin/start-dfs.sh, I get the following error:

    2011-04-13 09:42:31,729 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = inti84.cse.psu.edu/130.203.58.212
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 0.20.2
    STARTUP_MSG: build =
    https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
    911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
    ************************************************************/
    2011-04-13 09:42:31,853 ERROR
    org.apache.hadoop.hdfs.server.namenode.NameNode:
    java.lang.NullPointerException
    at
    org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:156)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:160)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:175)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:956)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)

    2011-04-13 09:42:31,854 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at inti84.cse.psu.edu/130.203.58.212
    ************************************************************/
    2011-04-13 09:44:03,265 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = inti84.cse.psu.edu/130.203.58.212
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 0.20.2
    STARTUP_MSG: build =
    https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
    911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
    ************************************************************/
    2011-04-13 09:44:03,384 ERROR
    org.apache.hadoop.hdfs.server.namenode.NameNode:
    java.lang.NullPointerException
    at
    org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:156)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:160)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:175)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:956)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)

    On Wed, Apr 13, 2011 at 9:20 AM, bikash sharma wrote:

    Hi,
    I need to install hadoop on 16-node cluster. I have a couple of related
    questions:
    1. I have installed hadoop on a shared directory, i.e., there is just one
    place where the whole hadoop installation files exist and all the 16 nodes
    use the same installation.
    Is that an issue or I need to install hadoop on each of these nodes in
    their local directory separately?
    2. I installed hadoop-0.21 and after following the installation
    instructions, when i tried formatting, I get the following error:

    ************************************************************/
    Re-format filesystem in /var/tmp/data/dfs/name ? (Y or N) Y
    11/04/13 09:16:23 INFO namenode.FSNamesystem: defaultReplication = 3
    11/04/13 09:16:23 INFO namenode.FSNamesystem: maxReplication = 512
    11/04/13 09:16:23 INFO namenode.FSNamesystem: minReplication = 1
    11/04/13 09:16:23 INFO namenode.FSNamesystem: maxReplicationStreams = 2
    11/04/13 09:16:23 INFO namenode.FSNamesystem: shouldCheckForEnoughRacks =
    false
    11/04/13 09:16:23 INFO security.Groups: Group mapping
    impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
    cacheTimeout=300000
    11/04/13 09:16:23 INFO namenode.FSNamesystem: fsOwner=bus145
    11/04/13 09:16:23 INFO namenode.FSNamesystem: supergroup=supergroup
    11/04/13 09:16:23 INFO namenode.FSNamesystem: isPermissionEnabled=true
    11/04/13 09:16:23 INFO namenode.FSNamesystem: isAccessTokenEnabled=false
    accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
    11/04/13 09:16:24 INFO common.Storage: Cannot lock storage
    /var/tmp/data/dfs/name. The directory is already locked.
    11/04/13 09:16:24 ERROR namenode.NameNode: java.io.IOException: Cannot lock
    storage /var/tmp/data/dfs/name. The directory is already locked.
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:617)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1426)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1444)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1242)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1348)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)

    11/04/13 09:16:24 INFO namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at inti79.cse.psu.edu/130.203.58.207
    ************************************************************/

    3. I was using before hadoop-0.20, and formatting was working fine.
    4. Also, when i do a bin/start-dfs.sh, am able to see the Namenode,
    Datanode up, however on bin/start-mapred.sh, am not able to see Jobtracker
    up on the master node, though Tasktracker seems to be up on slaves.

    Before upgrading to Hadoop-0.21, everything was working fine with
    hadoop-0.20 including running benchmarks and getting stats.

    Any suggestions in this regard is highly appreciated.

    Thanks,
    Bikash

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedApr 13, '11 at 1:21p
activeApr 13, '11 at 1:46p
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Bikash sharma: 2 posts

People

Translate

site design / logo © 2022 Grokbase