FAQ
Hi Everyone-

I'm not sure if this is the best place for this but figured I'd try.

I'm looking for TestDFSIO read and write test results to compare to my
cluster's output.

Cluster setup 4 datanodes, 8cores each, 16GB ram each.

My Tests
Write 50 files 128 mb each

12/12/05 12:54:12 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write

12/12/05 12:54:12 INFO fs.TestDFSIO: Date & time: Wed Dec 05
12:54:12 EST 2012

12/12/05 12:54:12 INFO fs.TestDFSIO: Number of files: 50

12/12/05 12:54:12 INFO fs.TestDFSIO: Total MBytes processed: 6400.0

12/12/05 12:54:12 INFO fs.TestDFSIO: Throughput mb/sec:
0.8708652440810759

12/12/05 12:54:12 INFO fs.TestDFSIO: Average IO rate mb/sec:
0.9578074812889099

12/12/05 12:54:12 INFO fs.TestDFSIO: IO rate std deviation:
0.3024037240254335

12/12/05 12:54:12 INFO fs.TestDFSIO: Test exec time sec: 305.269


Read 50 files 128 mb each

12/12/05 17:36:18 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read

12/12/05 17:36:18 INFO fs.TestDFSIO: Date & time: Wed Dec 05
17:36:18 EST 2012

12/12/05 17:36:18 INFO fs.TestDFSIO: Number of files: 50

12/12/05 17:36:18 INFO fs.TestDFSIO: Total MBytes processed: 6400.0

12/12/05 17:36:18 INFO fs.TestDFSIO: Throughput mb/sec:
20.35442136196522

12/12/05 17:36:18 INFO fs.TestDFSIO: Average IO rate mb/sec:
197.49246215820312

12/12/05 17:36:18 INFO fs.TestDFSIO: IO rate std deviation:
111.96049826586476

12/12/05 17:36:18 INFO fs.TestDFSIO: Test exec time sec: 54.172


12/12/05 17:38:38 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read

12/12/05 17:38:38 INFO fs.TestDFSIO: Date & time: Wed Dec 05
17:38:38 EST 2012

12/12/05 17:38:38 INFO fs.TestDFSIO: Number of files: 50

12/12/05 17:38:38 INFO fs.TestDFSIO: Total MBytes processed: 6400.0

12/12/05 17:38:38 INFO fs.TestDFSIO: Throughput mb/sec:
12.805890709726475

12/12/05 17:38:38 INFO fs.TestDFSIO: Average IO rate mb/sec:
180.38516235351562

12/12/05 17:38:38 INFO fs.TestDFSIO: IO rate std deviation:
129.25890715032295

12/12/05 17:38:38 INFO fs.TestDFSIO: Test exec time sec: 64.741


12/12/05 17:40:30 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read

12/12/05 17:40:30 INFO fs.TestDFSIO: Date & time: Wed Dec 05
17:40:30 EST 2012

12/12/05 17:40:30 INFO fs.TestDFSIO: Number of files: 50

12/12/05 17:40:30 INFO fs.TestDFSIO: Total MBytes processed: 6400.0

12/12/05 17:40:30 INFO fs.TestDFSIO: Throughput mb/sec:
15.852374760047061

12/12/05 17:40:30 INFO fs.TestDFSIO: Average IO rate mb/sec:
181.2574005126953

12/12/05 17:40:30 INFO fs.TestDFSIO: IO rate std deviation:
104.20816041404993

12/12/05 17:40:30 INFO fs.TestDFSIO: Test exec time sec: 61.045

One thing I noticed that when running the read test over and over I got
fairly different results.

If you could please post your results and cluster size to allow me to
compare it would be greatly appreciated.

--

Search Discussions

Discussion Posts

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 2 | next ›
Discussion Overview
groupcdh-user @
categorieshadoop
postedDec 7, '12 at 8:26p
activeDec 7, '12 at 8:29p
posts2
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Harakiro: 1 post Joe Travaglini: 1 post

People

Translate

site design / logo © 2022 Grokbase