FAQ
Hi,

I installed a small Hadoop-Cluster with Cloudera CDH3.

2 Server are running. One is running the NameNode and DataNode. The other server is just a DataNode.
A dedicated Client exists too.

I want to run some performance tests with bonnie++ and iozone.
bonnie++ works but when I start iozone, it interrups.

Is this a known issue?

$ mount | grep hdfs
fuse_dfs on /hdfs type fuse.fuse_dfs (rw,nosuid,nodev,allow_other,default_permissions)

$ iozone -Rac -g 3584M -f /hdfs/testfile | tee -a /tmp/iozone_results.txt &
[1] 6283
hadoop@ip-10-243-13-191:/home/ubuntu$ Iozone: Performance Test of File I/O
Version $Revision: 3.308 $
Compiled for 32 bit mode.
Build: linux

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

Run began: Tue Jul 6 16:40:35 2010

Excel chart generation enabled
Auto Mode
Include close in write timing
Using maximum file size of 3670016 kilobytes.
Command line used: iozone -Rac -g 3584M -f /hdfs/testfile
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
64 4write: Invalid argument

Error writing block 0, fd= 3

iozone: interrupted


Best Regards,
Christian

Search Discussions

  • Christian Baun at Jul 6, 2010 at 6:18 pm
    Hi,

    After some time, bonnie++ exits too with this error message:

    Can't write block.: Invalid argument
    Bonnie: drastic I/O error (re write(2)): Invalid argument
    [1]+ Exit 1 bonnie++ -s 3584 -d /hdfs/ -x5 -u hadoop -q -b -m HDFS_test > /tmp/ausgabe.txt 2>&1

    This looks not so good. :-(

    Best Regards,
    Christian


    Am Dienstag, 6. Juli 2010 schrieb Christian Baun:
    Hi,

    I installed a small Hadoop-Cluster with Cloudera CDH3.

    2 Server are running. One is running the NameNode and DataNode. The other server is just a DataNode.
    A dedicated Client exists too.

    I want to run some performance tests with bonnie++ and iozone.
    bonnie++ works but when I start iozone, it interrups.

    Is this a known issue?

    $ mount | grep hdfs
    fuse_dfs on /hdfs type fuse.fuse_dfs (rw,nosuid,nodev,allow_other,default_permissions)

    $ iozone -Rac -g 3584M -f /hdfs/testfile | tee -a /tmp/iozone_results.txt &
    [1] 6283
    hadoop@ip-10-243-13-191:/home/ubuntu$ Iozone: Performance Test of File I/O
    Version $Revision: 3.308 $
    Compiled for 32 bit mode.
    Build: linux

    Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
    Al Slater, Scott Rhine, Mike Wisner, Ken Goss
    Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
    Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
    Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
    Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

    Run began: Tue Jul 6 16:40:35 2010

    Excel chart generation enabled
    Auto Mode
    Include close in write timing
    Using maximum file size of 3670016 kilobytes.
    Command line used: iozone -Rac -g 3584M -f /hdfs/testfile
    Output is in Kbytes/sec
    Time Resolution = 0.000001 seconds.
    Processor cache size set to 1024 Kbytes.
    Processor cache line size set to 32 bytes.
    File stride size set to 17 * record size.
    random random bkwd record stride
    KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
    64 4write: Invalid argument

    Error writing block 0, fd= 3

    iozone: interrupted


    Best Regards,
    Christian
  • Brian Bockelman at Jul 6, 2010 at 6:57 pm
    Hey Christian,

    HDFS does not support the full set of POSIX semantics. I suspect bonnie++ is exercising things which HDFS doesn't support (such as writes to a file which is already closed or seeks during writes).

    Additionally, I would warn that, as bonnie++ is designed to test the performance of harddrives / network file systems / cluster file systems, you will find it doesn't capture well the performance aspects of a distributed file system.

    In other words, you are performance testing an apple with a test suite designed for oranges.

    Brian
    On Jul 6, 2010, at 1:14 PM, Christian Baun wrote:

    Hi,

    After some time, bonnie++ exits too with this error message:

    Can't write block.: Invalid argument
    Bonnie: drastic I/O error (re write(2)): Invalid argument
    [1]+ Exit 1 bonnie++ -s 3584 -d /hdfs/ -x5 -u hadoop -q -b -m HDFS_test > /tmp/ausgabe.txt 2>&1

    This looks not so good. :-(

    Best Regards,
    Christian


    Am Dienstag, 6. Juli 2010 schrieb Christian Baun:
    Hi,

    I installed a small Hadoop-Cluster with Cloudera CDH3.

    2 Server are running. One is running the NameNode and DataNode. The other server is just a DataNode.
    A dedicated Client exists too.

    I want to run some performance tests with bonnie++ and iozone.
    bonnie++ works but when I start iozone, it interrups.

    Is this a known issue?

    $ mount | grep hdfs
    fuse_dfs on /hdfs type fuse.fuse_dfs (rw,nosuid,nodev,allow_other,default_permissions)

    $ iozone -Rac -g 3584M -f /hdfs/testfile | tee -a /tmp/iozone_results.txt &
    [1] 6283
    hadoop@ip-10-243-13-191:/home/ubuntu$ Iozone: Performance Test of File I/O
    Version $Revision: 3.308 $
    Compiled for 32 bit mode.
    Build: linux

    Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
    Al Slater, Scott Rhine, Mike Wisner, Ken Goss
    Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
    Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
    Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
    Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

    Run began: Tue Jul 6 16:40:35 2010

    Excel chart generation enabled
    Auto Mode
    Include close in write timing
    Using maximum file size of 3670016 kilobytes.
    Command line used: iozone -Rac -g 3584M -f /hdfs/testfile
    Output is in Kbytes/sec
    Time Resolution = 0.000001 seconds.
    Processor cache size set to 1024 Kbytes.
    Processor cache line size set to 32 bytes.
    File stride size set to 17 * record size.
    random random bkwd record stride
    KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
    64 4write: Invalid argument

    Error writing block 0, fd= 3

    iozone: interrupted


    Best Regards,
    Christian
  • Allen Wittenauer at Jul 6, 2010 at 7:09 pm

    On Jul 6, 2010, at 11:56 AM, Brian Bockelman wrote:
    HDFS does not support the full set of POSIX semantics. I suspect bonnie++ is exercising things which HDFS doesn't support (such as writes to a file which is already closed or seeks during writes).

    In fact, I think the last time this came up a year or two ago, the stack trace showed that iozone was blowing up on a call to truncate(), which is definitely not supported.
  • Christian Baun at Jul 7, 2010 at 8:04 am
    Hi Brian,

    I wanted to test HDFS against several distributed filesystems.

    Do you know any popular performance benchmarks that run with HDFS?

    Best Regards,
    Christian


    Am Dienstag, 6. Juli 2010 schrieb Allen Wittenauer:
    On Jul 6, 2010, at 11:56 AM, Brian Bockelman wrote:
    HDFS does not support the full set of POSIX semantics. I suspect bonnie++ is exercising things which HDFS doesn't support (such as writes to a file which is already closed or seeks during writes).

    In fact, I think the last time this came up a year or two ago, the stack trace showed that iozone was blowing up on a call to truncate(), which is definitely not supported.
  • Brian Bockelman at Jul 7, 2010 at 9:32 pm

    On Jul 7, 2010, at 2:56 AM, Christian Baun wrote:

    Hi Brian,

    I wanted to test HDFS against several distributed filesystems.

    Do you know any popular performance benchmarks that run with HDFS?
    I can't think of anything off the top of my head. Any ideas out there on the list?

    The issue is that HDFS is not very much like a distributed file system. Maybe a good way is to do one of the MapReduce-based Test IO benchmarks on top of HDFS versus Lustre versus GPFS, etc?

    Brian
    Best Regards,
    Christian


    Am Dienstag, 6. Juli 2010 schrieb Allen Wittenauer:
    On Jul 6, 2010, at 11:56 AM, Brian Bockelman wrote:
    HDFS does not support the full set of POSIX semantics. I suspect bonnie++ is exercising things which HDFS doesn't support (such as writes to a file which is already closed or seeks during writes).

    In fact, I think the last time this came up a year or two ago, the stack trace showed that iozone was blowing up on a call to truncate(), which is definitely not supported.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJul 6, '10 at 4:53p
activeJul 7, '10 at 9:32p
posts6
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase