 |
Brian Bockelman |
at Oct 20, 2009 at 12:06 pm
|
⇧ | 
| |
On Oct 19, 2009, at 11:13 PM, Huang Qian wrote:
How can I creat 10 files on each datanode? I think I can only create
files
on filesystem,
Is there any method to assign a file to a datanode?
No, and you probably don't want to. At such a small scale, I can't
think of a benefit to having the cluster *exactly* balanced. Hadoop
is designed to scale large, and it's simply too inefficient to put
such a mechanism in place.
If you want *approximately* the same number of blocks on each node,
you can use the rebalancer.
Brian
PS - if you really think this is mission-critical (and if you do, I'd
advise to re-check your assumptions), you may want to look at the
custom block placement plugins in the upcoming 0.21.0 release.
2009/10/19 Jason Venner <jason.hadoop@gmail.com>
If you set your replication count to one and on each datanode,
create 10
files, you will achieve the pattern you are trying for.
By default when a file is created on a machine hosting a datanode,
that
datanode will receive 1 replica of the file, and will be
responsible for
sending the file data to the next replica if any.
On Thu, Oct 15, 2009 at 1:39 PM, Huang Qian <skyswind@gmail.com>
wrote:
Hi everyone. I am working on a project with hadoop and now I come
across
some problem. How can I deploy 100 files, with each file have one
block by
setting the blocksize and controling the file size, on to 10
datanode, and
make sure each datanode has 10 blocks. I know the file system can
deploy
the
blocks automaticly, but I want to make sure for the assigns files,
the
files
will be deployed well-proportioned. How can I make it by the
hadoop tool or
api?
Huang Qian（黄骞）
Institute of Remote Sensing and GIS,Peking University
Phone: (86-10) 5276-3109
Mobile: (86) 1590-126-8883
Address:Rm.554,Building 1,ChangChunXinYuan,Peking
Univ.,Beijing(100871),CHINA
--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymallwww.prohadoopbook.com a community for Hadoop Professionals
