FAQ
Hello,
I am doing my master.... my final year project is on Hadoop ...so I
would like to know some thing about Hadoop cluster i.e, Do new version of
Hadoop are able to handle heterogeneous hardware.If you have any
informantion regarding these please mail me as my project is in heterogenous
environment.


Thanks!

Reagrds,
Ashish Pareek

Search Discussions

  • Jason hadoop at Jun 18, 2009 at 3:01 pm
    Hadoop has always been reasonably agnostic wrt hardware and homogeneity.
    There are optimizations in configuration for near homogeneous machines.


    On Thu, Jun 18, 2009 at 7:46 AM, ashish pareek wrote:

    Hello,
    I am doing my master.... my final year project is on Hadoop ...so I
    would like to know some thing about Hadoop cluster i.e, Do new version of
    Hadoop are able to handle heterogeneous hardware.If you have any
    informantion regarding these please mail me as my project is in
    heterogenous
    environment.


    Thanks!

    Reagrds,
    Ashish Pareek


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Ashish pareek at Jun 18, 2009 at 3:18 pm
    Does that mean hadoop is not scalable wrt heterogeneous environment? and one
    more question is can we run different application on the same hadoop cluster
    .

    Thanks.
    Regards,
    Ashish
    On Thu, Jun 18, 2009 at 8:30 PM, jason hadoop wrote:

    Hadoop has always been reasonably agnostic wrt hardware and homogeneity.
    There are optimizations in configuration for near homogeneous machines.


    On Thu, Jun 18, 2009 at 7:46 AM, ashish pareek wrote:

    Hello,
    I am doing my master.... my final year project is on Hadoop ...so I
    would like to know some thing about Hadoop cluster i.e, Do new version of
    Hadoop are able to handle heterogeneous hardware.If you have any
    informantion regarding these please mail me as my project is in
    heterogenous
    environment.


    Thanks!

    Reagrds,
    Ashish Pareek


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Jason hadoop at Jun 18, 2009 at 3:24 pm
    I don't know anyone who has a completely homogeneous cluster.

    So hadoop is scalable across heterogeneous environments.

    I stated that configuration is simpler if the machines are similar (There
    are optimizations in configuration for near homogeneous machines.)
    On Thu, Jun 18, 2009 at 8:10 AM, ashish pareek wrote:

    Does that mean hadoop is not scalable wrt heterogeneous environment? and
    one
    more question is can we run different application on the same hadoop
    cluster
    .

    Thanks.
    Regards,
    Ashish

    On Thu, Jun 18, 2009 at 8:30 PM, jason hadoop <jason.hadoop@gmail.com
    wrote:
    Hadoop has always been reasonably agnostic wrt hardware and homogeneity.
    There are optimizations in configuration for near homogeneous machines.



    On Thu, Jun 18, 2009 at 7:46 AM, ashish pareek <pareekash@gmail.com>
    wrote:
    Hello,
    I am doing my master.... my final year project is on Hadoop
    ...so
    I
    would like to know some thing about Hadoop cluster i.e, Do new version
    of
    Hadoop are able to handle heterogeneous hardware.If you have any
    informantion regarding these please mail me as my project is in
    heterogenous
    environment.


    Thanks!

    Reagrds,
    Ashish Pareek


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Ashish pareek at Jun 18, 2009 at 3:33 pm
    Can you tell few of the challenges in configuring heterogeneous cluster...or
    can pass on some link where I would get some information regarding
    challenges in running Hadoop on heterogeneous hardware

    One more things is How about running different applications on the same
    Hadoop cluster?and what challenges are involved in it ?

    Thanks,
    Regards,
    Ashish

    On Thu, Jun 18, 2009 at 8:53 PM, jason hadoop wrote:

    I don't know anyone who has a completely homogeneous cluster.

    So hadoop is scalable across heterogeneous environments.

    I stated that configuration is simpler if the machines are similar (There
    are optimizations in configuration for near homogeneous machines.)
    On Thu, Jun 18, 2009 at 8:10 AM, ashish pareek wrote:

    Does that mean hadoop is not scalable wrt heterogeneous environment? and
    one
    more question is can we run different application on the same hadoop
    cluster
    .

    Thanks.
    Regards,
    Ashish

    On Thu, Jun 18, 2009 at 8:30 PM, jason hadoop <jason.hadoop@gmail.com
    wrote:
    Hadoop has always been reasonably agnostic wrt hardware and
    homogeneity.
    There are optimizations in configuration for near homogeneous machines.



    On Thu, Jun 18, 2009 at 7:46 AM, ashish pareek <pareekash@gmail.com>
    wrote:
    Hello,
    I am doing my master.... my final year project is on Hadoop
    ...so
    I
    would like to know some thing about Hadoop cluster i.e, Do new
    version
    of
    Hadoop are able to handle heterogeneous hardware.If you have any
    informantion regarding these please mail me as my project is in
    heterogenous
    environment.


    Thanks!

    Reagrds,
    Ashish Pareek


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Jason hadoop at Jun 18, 2009 at 4:06 pm
    For me, I like to have one configuration file that I distribute to all of
    the machines in my cluster via rsync.

    In there are things like the number of tasks per node to run, and where to
    store dfs data and local temporary data, and the limits to storage for the
    machines.

    If the machines are very different, it becomes important to tailor the
    configuration file per machine or type of machine.

    At this point, you are pretty much going to have to spend the time, reading
    through the details of configuring a hadoop cluster.

    On Thu, Jun 18, 2009 at 8:33 AM, ashish pareek wrote:

    Can you tell few of the challenges in configuring heterogeneous
    cluster...or
    can pass on some link where I would get some information regarding
    challenges in running Hadoop on heterogeneous hardware

    One more things is How about running different applications on the same
    Hadoop cluster?and what challenges are involved in it ?

    Thanks,
    Regards,
    Ashish


    On Thu, Jun 18, 2009 at 8:53 PM, jason hadoop <jason.hadoop@gmail.com
    wrote:
    I don't know anyone who has a completely homogeneous cluster.

    So hadoop is scalable across heterogeneous environments.

    I stated that configuration is simpler if the machines are similar (There
    are optimizations in configuration for near homogeneous machines.)

    On Thu, Jun 18, 2009 at 8:10 AM, ashish pareek <pareekash@gmail.com>
    wrote:
    Does that mean hadoop is not scalable wrt heterogeneous environment?
    and
    one
    more question is can we run different application on the same hadoop
    cluster
    .

    Thanks.
    Regards,
    Ashish

    On Thu, Jun 18, 2009 at 8:30 PM, jason hadoop <jason.hadoop@gmail.com
    wrote:
    Hadoop has always been reasonably agnostic wrt hardware and
    homogeneity.
    There are optimizations in configuration for near homogeneous
    machines.


    On Thu, Jun 18, 2009 at 7:46 AM, ashish pareek <pareekash@gmail.com>
    wrote:
    Hello,
    I am doing my master.... my final year project is on Hadoop
    ...so
    I
    would like to know some thing about Hadoop cluster i.e, Do new
    version
    of
    Hadoop are able to handle heterogeneous hardware.If you have any
    informantion regarding these please mail me as my project is in
    heterogenous
    environment.


    Thanks!

    Reagrds,
    Ashish Pareek


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Ashish pareek at Jun 18, 2009 at 4:26 pm
    Hello Everybody,

    How can we handle different applications having
    different requirement being run on the same hadoop cluster ? What are the
    various approaches to solve such problem.. if possible please mention some
    of those ideas.

    Does such implementation exists ?

    Thanks ,

    Regards,
    Ashish
    On Thu, Jun 18, 2009 at 9:36 PM, jason hadoop wrote:

    For me, I like to have one configuration file that I distribute to all of
    the machines in my cluster via rsync.

    In there are things like the number of tasks per node to run, and where to
    store dfs data and local temporary data, and the limits to storage for the
    machines.

    If the machines are very different, it becomes important to tailor the
    configuration file per machine or type of machine.

    At this point, you are pretty much going to have to spend the time, reading
    through the details of configuring a hadoop cluster.

    On Thu, Jun 18, 2009 at 8:33 AM, ashish pareek wrote:

    Can you tell few of the challenges in configuring heterogeneous
    cluster...or
    can pass on some link where I would get some information regarding
    challenges in running Hadoop on heterogeneous hardware

    One more things is How about running different applications on the same
    Hadoop cluster?and what challenges are involved in it ?

    Thanks,
    Regards,
    Ashish


    On Thu, Jun 18, 2009 at 8:53 PM, jason hadoop <jason.hadoop@gmail.com
    wrote:
    I don't know anyone who has a completely homogeneous cluster.

    So hadoop is scalable across heterogeneous environments.

    I stated that configuration is simpler if the machines are similar
    (There
    are optimizations in configuration for near homogeneous machines.)

    On Thu, Jun 18, 2009 at 8:10 AM, ashish pareek <pareekash@gmail.com>
    wrote:
    Does that mean hadoop is not scalable wrt heterogeneous environment?
    and
    one
    more question is can we run different application on the same hadoop
    cluster
    .

    Thanks.
    Regards,
    Ashish

    On Thu, Jun 18, 2009 at 8:30 PM, jason hadoop <
    jason.hadoop@gmail.com
    wrote:
    Hadoop has always been reasonably agnostic wrt hardware and
    homogeneity.
    There are optimizations in configuration for near homogeneous
    machines.


    On Thu, Jun 18, 2009 at 7:46 AM, ashish pareek <
    pareekash@gmail.com>
    wrote:
    Hello,
    I am doing my master.... my final year project is on
    Hadoop
    ...so
    I
    would like to know some thing about Hadoop cluster i.e, Do new
    version
    of
    Hadoop are able to handle heterogeneous hardware.If you have any
    informantion regarding these please mail me as my project is in
    heterogenous
    environment.


    Thanks!

    Reagrds,
    Ashish Pareek


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Matt Massie at Jun 18, 2009 at 5:46 pm
    Hadoop can be run on a hardware heterogeneous cluster. Currently,
    Hadoop clusters really only run well on Linux although you can run a
    Hadoop client on non-Linux machines.

    You will need to have a special configuration for each of the machine
    in your cluster based on their hardware profile. Ideally, you'll be
    able to group the machines in your cluster into "classes" of machines
    (e.g. machines with 1GB of RAM and 2 core versus 4GB of RAM and 4
    core) to reduce the burden of managing multiple configurations. If
    you are talking about a Hadoop cluster that is completely
    heterogeneous (each machine is completely different), the management
    overhead could be high.

    Configuration variables like "mapred.tasktracker.map.tasks.maximum"
    and "mapred.tasktracker.reduce.tasks.maximum" should be set based on
    the number of cores/memory in each machine. Variables like
    "mapred.child.java.opts" need to be set differently based on the
    amount of memory the machine has (e.g. "-Xmx250m"). You should have
    at least 250MB of memory dedicated to each task although more is
    better. It's also wise to make sure that each task has the same
    amount of memory regardless of the machine it's scheduled on;
    otherwise, tasks might succeed or fail based on which machine gets the
    task. This asymmetry will make debugging harder.

    You can use our online configurator (http://www.cloudera.com/configurator/
    ), to generate optimized configurations for each class of machines in
    your cluster. It will ask simple question about your configuration
    and then produce a hadoop-site.xml file.

    Good luck!
    -Matt
    On Jun 18, 2009, at 8:33 AM, ashish pareek wrote:

    Can you tell few of the challenges in configuring heterogeneous
    cluster...or
    can pass on some link where I would get some information regarding
    challenges in running Hadoop on heterogeneous hardware

    One more things is How about running different applications on the
    same
    Hadoop cluster?and what challenges are involved in it ?

    Thanks,
    Regards,
    Ashish


    On Thu, Jun 18, 2009 at 8:53 PM, jason hadoop
    wrote:
    I don't know anyone who has a completely homogeneous cluster.

    So hadoop is scalable across heterogeneous environments.

    I stated that configuration is simpler if the machines are similar
    (There
    are optimizations in configuration for near homogeneous machines.)

    On Thu, Jun 18, 2009 at 8:10 AM, ashish pareek <pareekash@gmail.com>
    wrote:
    Does that mean hadoop is not scalable wrt heterogeneous
    environment? and
    one
    more question is can we run different application on the same hadoop
    cluster
    .

    Thanks.
    Regards,
    Ashish

    On Thu, Jun 18, 2009 at 8:30 PM, jason hadoop
    <jason.hadoop@gmail.com
    wrote:
    Hadoop has always been reasonably agnostic wrt hardware and
    homogeneity.
    There are optimizations in configuration for near homogeneous
    machines.



    On Thu, Jun 18, 2009 at 7:46 AM, ashish pareek
    <pareekash@gmail.com>
    wrote:
    Hello,
    I am doing my master.... my final year project is on Hadoop
    ...so
    I
    would like to know some thing about Hadoop cluster i.e, Do new
    version
    of
    Hadoop are able to handle heterogeneous hardware.If you have any
    informantion regarding these please mail me as my project is in
    heterogenous
    environment.


    Thanks!

    Reagrds,
    Ashish Pareek


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedJun 18, '09 at 2:47p
activeJun 18, '09 at 5:46p
posts8
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase