FAQ
Hi everyone. I am working on a project with hadoop and now I come across
some problem. How can I deploy 100 files, with each file have one block by
setting the blocksize and controling the file size, on to 10 datanode, and
make sure each datanode has 10 blocks. I know the file system can deploy the
blocks automaticly, but I want to make sure for the assigns files, the files
will be deployed well-proportioned. How can I make it by the hadoop tool or
api?

Huang Qian(黄骞)
Institute of Remote Sensing and GIS,Peking University
Phone: (86-10) 5276-3109
Mobile: (86) 1590-126-8883
Address:Rm.554,Building 1,ChangChunXinYuan,Peking
Univ.,Beijing(100871),CHINA

Search Discussions

  • Jason Venner at Oct 20, 2009 at 3:54 am
    If you set your replication count to one and on each datanode, create 10
    files, you will achieve the pattern you are trying for.

    By default when a file is created on a machine hosting a datanode, that
    datanode will receive 1 replica of the file, and will be responsible for
    sending the file data to the next replica if any.

    On Thu, Oct 15, 2009 at 1:39 PM, Huang Qian wrote:

    Hi everyone. I am working on a project with hadoop and now I come across
    some problem. How can I deploy 100 files, with each file have one block by
    setting the blocksize and controling the file size, on to 10 datanode, and
    make sure each datanode has 10 blocks. I know the file system can deploy
    the
    blocks automaticly, but I want to make sure for the assigns files, the
    files
    will be deployed well-proportioned. How can I make it by the hadoop tool or
    api?

    Huang Qian(黄骞)
    Institute of Remote Sensing and GIS,Peking University
    Phone: (86-10) 5276-3109
    Mobile: (86) 1590-126-8883
    Address:Rm.554,Building 1,ChangChunXinYuan,Peking
    Univ.,Beijing(100871),CHINA


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Huang Qian at Oct 20, 2009 at 4:14 am
    How can I creat 10 files on each datanode? I think I can only create files
    on filesystem,
    Is there any method to assign a file to a datanode?

    2009/10/19 Jason Venner <jason.hadoop@gmail.com>
    If you set your replication count to one and on each datanode, create 10
    files, you will achieve the pattern you are trying for.

    By default when a file is created on a machine hosting a datanode, that
    datanode will receive 1 replica of the file, and will be responsible for
    sending the file data to the next replica if any.

    On Thu, Oct 15, 2009 at 1:39 PM, Huang Qian wrote:

    Hi everyone. I am working on a project with hadoop and now I come across
    some problem. How can I deploy 100 files, with each file have one block by
    setting the blocksize and controling the file size, on to 10 datanode, and
    make sure each datanode has 10 blocks. I know the file system can deploy
    the
    blocks automaticly, but I want to make sure for the assigns files, the
    files
    will be deployed well-proportioned. How can I make it by the hadoop tool or
    api?

    Huang Qian(黄骞)
    Institute of Remote Sensing and GIS,Peking University
    Phone: (86-10) 5276-3109
    Mobile: (86) 1590-126-8883
    Address:Rm.554,Building 1,ChangChunXinYuan,Peking
    Univ.,Beijing(100871),CHINA


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Brian Bockelman at Oct 20, 2009 at 12:06 pm

    On Oct 19, 2009, at 11:13 PM, Huang Qian wrote:

    How can I creat 10 files on each datanode? I think I can only create
    files
    on filesystem,
    Is there any method to assign a file to a datanode?
    No, and you probably don't want to. At such a small scale, I can't
    think of a benefit to having the cluster *exactly* balanced. Hadoop
    is designed to scale large, and it's simply too inefficient to put
    such a mechanism in place.

    If you want *approximately* the same number of blocks on each node,
    you can use the rebalancer.

    Brian

    PS - if you really think this is mission-critical (and if you do, I'd
    advise to re-check your assumptions), you may want to look at the
    custom block placement plugins in the upcoming 0.21.0 release.
    2009/10/19 Jason Venner <jason.hadoop@gmail.com>
    If you set your replication count to one and on each datanode,
    create 10
    files, you will achieve the pattern you are trying for.

    By default when a file is created on a machine hosting a datanode,
    that
    datanode will receive 1 replica of the file, and will be
    responsible for
    sending the file data to the next replica if any.


    On Thu, Oct 15, 2009 at 1:39 PM, Huang Qian <skyswind@gmail.com>
    wrote:
    Hi everyone. I am working on a project with hadoop and now I come
    across
    some problem. How can I deploy 100 files, with each file have one
    block by
    setting the blocksize and controling the file size, on to 10
    datanode, and
    make sure each datanode has 10 blocks. I know the file system can
    deploy
    the
    blocks automaticly, but I want to make sure for the assigns files,
    the
    files
    will be deployed well-proportioned. How can I make it by the
    hadoop tool or
    api?

    Huang Qian(黄骞)
    Institute of Remote Sensing and GIS,Peking University
    Phone: (86-10) 5276-3109
    Mobile: (86) 1590-126-8883
    Address:Rm.554,Building 1,ChangChunXinYuan,Peking
    Univ.,Beijing(100871),CHINA


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 15, '09 at 6:40p
activeOct 20, '09 at 12:06p
posts4
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase