Grokbase › Groups › Hadoop › common-user › October 2010
FAQ
Hi,

I'm seeing in my experiments that Fuse-HDFS is significantly slower (around
3x slower) than using the Java hdfs API directly.
Wanted to ask if this slowness the norm? Or is there something wrong with my
configuration.
Also is this purely JNI slowness or is there something deeper to it?


My experiment is basically opening a file in write mode and calling writes
multiple times (close to 5GB data) to write to that file.

Thanks for the help,
aniket ray

Search Discussions

  • Ted Yu at Oct 26, 2010 at 5:41 am ⇧
    https://issues.apache.org/jira/browse/HADOOP-3805 tried to mitigate this
    problem.
    On Mon, Oct 25, 2010 at 10:17 PM, aniket ray wrote:

    Hi,

    I'm seeing in my experiments that Fuse-HDFS is significantly slower (around
    3x slower) than using the Java hdfs API directly.
    Wanted to ask if this slowness the norm? Or is there something wrong with
    my
    configuration.
    Also is this purely JNI slowness or is there something deeper to it?


    My experiment is basically opening a file in write mode and calling writes
    multiple times (close to 5GB data) to write to that file.

    Thanks for the help,
    aniket ray
  • Brian Bockelman at Oct 26, 2010 at 12:44 pm ⇧
    In general, unless you run newer kernels and versions of FUSE as that ticket suggests, it is significantly slower in raw throughput.

    However, we generally don't have a day go by at my site where we don't push FUSE over 30Gbps, as the bandwidth is spread throughout nodes. Additionally, as we are limited by the latency of spinning disk and random reads, we don't particularly hurt by going "only" 60MB/s on our nodes. If we wanted to go faster, we use the native clients.

    Of course, if anyone wants to donate a lowly university 1.5PB of SSDs, I'm all ears :)

    Brian
    On Oct 26, 2010, at 12:40 AM, Ted Yu wrote:

    https://issues.apache.org/jira/browse/HADOOP-3805 tried to mitigate this
    problem.
    On Mon, Oct 25, 2010 at 10:17 PM, aniket ray wrote:

    Hi,

    I'm seeing in my experiments that Fuse-HDFS is significantly slower (around
    3x slower) than using the Java hdfs API directly.
    Wanted to ask if this slowness the norm? Or is there something wrong with
    my
    configuration.
    Also is this purely JNI slowness or is there something deeper to it?


    My experiment is basically opening a file in write mode and calling writes
    multiple times (close to 5GB data) to write to that file.

    Thanks for the help,
    aniket ray
  • Hazem Mahmoud at Oct 26, 2010 at 6:26 pm ⇧
    That raises a question that I am currently looking into and would appreciate any and all advice people have.

    We are replacing our current NetApp solution, which has served us well but we have outgrown it.

    I am looking at either upgrading to a bigger and meaner NetApp or possibly going with Hadoop (HDFS and Fuse ).

    I need to mount the "storage solution" (HDFS or SAN) to about 5 or 6 systems. I'm a little concerned about utilizing HDFS/Fuse for a couple of reasons:
    1. Performance of Fuse (how does it compare to an iSCSI SAN solution for example)...i know, it probably depends on a lot of things, but just generally-speaking or any experiences anyone has had
    2. Security/permissions (owner of all files show up as "nobody"

    Another question: Are there other options for mounting HDFS on these 5 or 6 systems for pure filesystem access ? (using NFS, etc)

    Thanks everyone!

    -Hazem
    On Oct 26, 2010, at 5:43 AM, Brian Bockelman wrote:

    In general, unless you run newer kernels and versions of FUSE as that ticket suggests, it is significantly slower in raw throughput.

    However, we generally don't have a day go by at my site where we don't push FUSE over 30Gbps, as the bandwidth is spread throughout nodes. Additionally, as we are limited by the latency of spinning disk and random reads, we don't particularly hurt by going "only" 60MB/s on our nodes. If we wanted to go faster, we use the native clients.

    Of course, if anyone wants to donate a lowly university 1.5PB of SSDs, I'm all ears :)

    Brian
    On Oct 26, 2010, at 12:40 AM, Ted Yu wrote:

    https://issues.apache.org/jira/browse/HADOOP-3805 tried to mitigate this
    problem.
    On Mon, Oct 25, 2010 at 10:17 PM, aniket ray wrote:

    Hi,

    I'm seeing in my experiments that Fuse-HDFS is significantly slower (around
    3x slower) than using the Java hdfs API directly.
    Wanted to ask if this slowness the norm? Or is there something wrong with
    my
    configuration.
    Also is this purely JNI slowness or is there something deeper to it?


    My experiment is basically opening a file in write mode and calling writes
    multiple times (close to 5GB data) to write to that file.

    Thanks for the help,
    aniket ray
  • Allen Wittenauer at Oct 26, 2010 at 6:37 pm ⇧

    On Oct 26, 2010, at 11:25 AM, Hazem Mahmoud wrote:

    That raises a question that I am currently looking into and would appreciate any and all advice people have.

    We are replacing our current NetApp solution, which has served us well but we have outgrown it.

    I am looking at either upgrading to a bigger and meaner NetApp or possibly going with Hadoop (HDFS and Fuse ).
    You'd probably better looking at something like Ceph or Lustre which are meant to be fully POSIX compliant.
    I need to mount the "storage solution" (HDFS or SAN) to about 5 or 6 systems. I'm a little concerned about utilizing HDFS/Fuse for a couple of reasons:
    1. Performance of Fuse (how does it compare to an iSCSI SAN solution for example)...i know, it probably depends on a lot of things, but just generally-speaking or any experiences anyone has had
    FUSE in general (regardless of what you're using with it) is going to be significantly slower vs. a kernel-level file system.

    2. Security/permissions (owner of all files show up as "nobody"
    I doubt anyone has spent any time adding security the HDFS FUSE port. So even though NetApp's Kerberos stack is pretty crappy (3DES only... seriously?) , you're going to get a better security model with it.
    Another question: Are there other options for mounting HDFS on these 5 or 6 systems for pure filesystem access ? (using NFS, etc)
    No. I keep hoping someone builds a pNFS/NFSv4.1 server on top of Hadoop, but alas not yet.
    Thanks everyone!

    -Hazem
    On Oct 26, 2010, at 5:43 AM, Brian Bockelman wrote:

    In general, unless you run newer kernels and versions of FUSE as that ticket suggests, it is significantly slower in raw throughput.

    However, we generally don't have a day go by at my site where we don't push FUSE over 30Gbps, as the bandwidth is spread throughout nodes. Additionally, as we are limited by the latency of spinning disk and random reads, we don't particularly hurt by going "only" 60MB/s on our nodes. If we wanted to go faster, we use the native clients.

    Of course, if anyone wants to donate a lowly university 1.5PB of SSDs, I'm all ears :)

    Brian
    On Oct 26, 2010, at 12:40 AM, Ted Yu wrote:

    https://issues.apache.org/jira/browse/HADOOP-3805 tried to mitigate this
    problem.
    On Mon, Oct 25, 2010 at 10:17 PM, aniket ray wrote:

    Hi,

    I'm seeing in my experiments that Fuse-HDFS is significantly slower (around
    3x slower) than using the Java hdfs API directly.
    Wanted to ask if this slowness the norm? Or is there something wrong with
    my
    configuration.
    Also is this purely JNI slowness or is there something deeper to it?


    My experiment is basically opening a file in write mode and calling writes
    multiple times (close to 5GB data) to write to that file.

    Thanks for the help,
    aniket ray
  • Brian Bockelman at Oct 26, 2010 at 8:43 pm ⇧

    On Oct 26, 2010, at 1:36 PM, Allen Wittenauer wrote:

    On Oct 26, 2010, at 11:25 AM, Hazem Mahmoud wrote:

    That raises a question that I am currently looking into and would appreciate any and all advice people have.

    We are replacing our current NetApp solution, which has served us well but we have outgrown it.

    I am looking at either upgrading to a bigger and meaner NetApp or possibly going with Hadoop (HDFS and Fuse ).
    You'd probably better looking at something like Ceph or Lustre which are meant to be fully POSIX compliant.
    It's the difference between a freight train and a race car. NetApp/Lustre are race cars; Hadoop is closer to a freight train. If you're moving data to 2000 nodes.

    They're two different categories. It's a mainframe versus a Linux cluster. If you can transition your mainframe to a Linux cluster, you probably shouldn't have bought a mainframe in the first place.
    I need to mount the "storage solution" (HDFS or SAN) to about 5 or 6 systems. I'm a little concerned about utilizing HDFS/Fuse for a couple of reasons:
    1. Performance of Fuse (how does it compare to an iSCSI SAN solution for example)...i know, it probably depends on a lot of things, but just generally-speaking or any experiences anyone has had
    FUSE in general (regardless of what you're using with it) is going to be significantly slower vs. a kernel-level file system.
    Slower *per node*, but you can still get impressive throughput when you multiply this to 2000 nodes.
    2. Security/permissions (owner of all files show up as "nobody"
    I doubt anyone has spent any time adding security the HDFS FUSE port. So even though NetApp's Kerberos stack is pretty crappy (3DES only... seriously?) , you're going to get a better security model with it.
    Actually, unix permissions are there. If all files show up as "nobody", something has gone wrong in your install.

    Strong security (i.e. Kerberos) is probably untested and I suspect it wouldn't work as-is.
    Another question: Are there other options for mounting HDFS on these 5 or 6 systems for pure filesystem access ? (using NFS, etc)
    No. I keep hoping someone builds a pNFS/NFSv4.1 server on top of Hadoop, but alas not yet.

    Not yet, but again, we're pretty happy because per-process we use <10MB/s. So, FUSE is more than sufficient for our needs.

    If you're just at the 5-6 node level, I would seriously think about buying a nice big RAID server from Oracle, run Solaris's nice NFS implementation, and saving some time.

    I've attached our Ganglia network graphs below; the data rates really can add up.

    Brian

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 26, '10 at 5:17a
activeOct 26, '10 at 8:43p
posts6
users5
websitehadoop.apache.org...
irc#hadoop

5 users in discussion

Brian Bockelman: 2 posts Hazem Mahmoud: 1 post Allen Wittenauer: 1 post Aniket ray: 1 post Ted Yu: 1 post

Content

People

Support

Translate

site design / logo © 2022 Grokbase