FAQ
Hello,

I have a general query regarding usage of Hadoop with my cloud infrastructure. I am trying to achieve scaling up and scaling down in cloud using Hadoop.

I have set up a cloud infrastructure which creates images consists of OS and applications. To access user applications, instance of image has to launch. Now I want to make this running or launched instance scalable based on some condition like -

a) If no. of users who are accessing the application which is hosting in cloud (i.e. in instance) are more then it should run one more instance of image and if no. of users are less then instances should be terminated.
b) If CPU usage is more then one more instance of image should run or if CPU usage is less then it should terminate the instance.

Can I achieve these goals using Hadoop?

Please guide me and let me know the possible solution.

Thanks in advance.

--------------------------------------------------------
With best regards,
Mohit Goel

________________________________
Important notice: This e-mail and any attachment there to contains corporate proprietary information. If you have received it by mistake, please notify us immediately by reply e-mail and delete this e-mail and its attachments from your system.
Thank You.

Search Discussions

  • Steve Loughran at Apr 15, 2010 at 12:16 pm

    Goel, Mohit IN BLR SISL wrote:
    Hello,

    I have a general query regarding usage of Hadoop with my cloud infrastructure. I am trying to achieve scaling up and scaling down in cloud using Hadoop.

    I have set up a cloud infrastructure which creates images consists of OS and applications. To access user applications, instance of image has to launch. Now I want to make this running or launched instance scalable based on some condition like -

    a) If no. of users who are accessing the application which is hosting in cloud (i.e. in instance) are more then it should run one more instance of image and if no. of users are less then instances should be terminated.
    b) If CPU usage is more then one more instance of image should run or if CPU usage is less then it should terminate the instance.

    Can I achieve these goals using Hadoop?
    1. Hadoop on Demand, HOD, does some of this
    2. Hadoop on EC2 does some of this
    3. I've been doing some of this, too; I have some slides up where I
    discuss issues
    http://www.slideshare.net/steve_l/new-roles-for-the-cloud

    One funny for Hadoop is that it likes locality, and it likes machines
    with TB of physical storage, which doesn't fit in quite as well with the
    VM-on-demand story. If you look at my slides, you can see that
    everything expects stable hostnames, reacts to failure by blacklisting,
    not by killing the VM and creating a new one with the same HDFS volumes
    mounted. There is room for improvement!
  • Segel, Mike at Apr 15, 2010 at 2:30 pm
    Steve,
    Outside of a EC2 or a commercial site which sells time on a 'cloud', I would argue against trying to do HOD or build a dynamic cloud for a corporate environment.

    Corporate clouds tend to be static in terms of usage. Meaning that they are being built for a task and any changes are not dynamic enough to justify HOD.

    I sat through a presentation from Sun. A nice guy, but in the end, I and others thought it was a way to make Sun's hardware (Sorry err I mean Oracle) relevant in the Hadoop world. Its counter to the concept of developing 'white box' commodity hardware.

    I'm not sold on virtualization, but its just my opinion and not necessarily shared by anyone, which means I need to make the following statement:

    The opinions expressed in this post are mine and mine alone. They do not reflect the opinions or position of my client, or my employer. Any resemblance to a rational coherent thought is pure coincidence.

    -Mike

    -----Original Message-----
    From: Steve Loughran
    Sent: Thursday, April 15, 2010 7:15 AM
    To: general@hadoop.apache.org
    Subject: Re: Query regarding Hadoop and cloud infrastructure

    Goel, Mohit IN BLR SISL wrote:
    Hello,

    I have a general query regarding usage of Hadoop with my cloud infrastructure. I am trying to achieve scaling up and scaling down in cloud using Hadoop.

    I have set up a cloud infrastructure which creates images consists of OS and applications. To access user applications, instance of image has to launch. Now I want to make this running or launched instance scalable based on some condition like -

    a) If no. of users who are accessing the application which is hosting in cloud (i.e. in instance) are more then it should run one more instance of image and if no. of users are less then instances should be terminated.
    b) If CPU usage is more then one more instance of image should run or if CPU usage is less then it should terminate the instance.

    Can I achieve these goals using Hadoop?
    1. Hadoop on Demand, HOD, does some of this
    2. Hadoop on EC2 does some of this
    3. I've been doing some of this, too; I have some slides up where I
    discuss issues
    http://www.slideshare.net/steve_l/new-roles-for-the-cloud

    One funny for Hadoop is that it likes locality, and it likes machines
    with TB of physical storage, which doesn't fit in quite as well with the
    VM-on-demand story. If you look at my slides, you can see that
    everything expects stable hostnames, reacts to failure by blacklisting,
    not by killing the VM and creating a new one with the same HDFS volumes
    mounted. There is room for improvement!




    The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
  • Goel, Mohit IN BLR SISL at Apr 19, 2010 at 8:23 am
    Hello,

    Thanks for the reply.

    It means that we can achieve scaling up and provisioning using Hadoop. Now the question is How to do this?

    Can I get source code to achieve the same? Or is there any service available using which we can implement provisioning and scaling up feature in our cloud infrastructure?

    --------------------------------------------------------
    With best regards,
    Mohit Goel

    -----Original Message-----
    From: Steve Loughran
    Sent: Thursday, April 15, 2010 5:45 PM
    To: general@hadoop.apache.org
    Subject: Re: Query regarding Hadoop and cloud infrastructure

    Goel, Mohit IN BLR SISL wrote:
    Hello,

    I have a general query regarding usage of Hadoop with my cloud infrastructure. I am trying to achieve scaling up and scaling down in cloud using Hadoop.

    I have set up a cloud infrastructure which creates images consists of OS and applications. To access user applications, instance of image has to launch. Now I want to make this running or launched instance scalable based on some condition like -

    a) If no. of users who are accessing the application which is hosting in cloud (i.e. in instance) are more then it should run one more instance of image and if no. of users are less then instances should be terminated.
    b) If CPU usage is more then one more instance of image should run or if CPU usage is less then it should terminate the instance.

    Can I achieve these goals using Hadoop?
    1. Hadoop on Demand, HOD, does some of this
    2. Hadoop on EC2 does some of this
    3. I've been doing some of this, too; I have some slides up where I
    discuss issues
    http://www.slideshare.net/steve_l/new-roles-for-the-cloud

    One funny for Hadoop is that it likes locality, and it likes machines
    with TB of physical storage, which doesn't fit in quite as well with the
    VM-on-demand story. If you look at my slides, you can see that
    everything expects stable hostnames, reacts to failure by blacklisting,
    not by killing the VM and creating a new one with the same HDFS volumes
    mounted. There is room for improvement!



    Important notice: This e-mail and any attachment there to contains corporate proprietary information. If you have received it by mistake, please notify us immediately by reply e-mail and delete this e-mail and its attachments from your system.
    Thank You.
  • Steve Loughran at Apr 19, 2010 at 10:52 am

    Goel, Mohit IN BLR SISL wrote:
    Hello,

    Thanks for the reply.

    It means that we can achieve scaling up and provisioning using Hadoop. Now the question is How to do this?

    Can I get source code to achieve the same? Or is there any service available using which we can implement provisioning and scaling up feature in our cloud infrastructure?
    That depends on your infrastructure. HOD is probably your best starting
    point, if you have something running eucalyptus there are more options

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgeneral @
categorieshadoop
postedApr 15, '10 at 11:09a
activeApr 19, '10 at 10:52a
posts5
users3
websitehadoop.apache.org
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase