FAQ
Hi all,

I am writing to HDFS with this simple code

File[] files = new File(fileDir).listFiles();
for (File file : files) {
key.set(file.getPath());
byte[] bytes = new FileUtil().readCompleteFile(file);
System.out.println(file.getName() + ", size=" +
bytes.length);
value = new BytesWritable(bytes);
writer.append(key, value); // apparently majority of time
taken by this line
}

and I notice that writing takes about 1 sec per Meg, so the whole directory
takes about a minute, while cp copies them all in under one second.

Am I doing something wrong? I am running on a single fast machine.

Thank you,
Mark

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedFeb 6, '09 at 3:29a
activeFeb 6, '09 at 3:29a
posts1
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Mark Kerzner: 1 post

People

Translate

site design / logo © 2022 Grokbase