FAQ
Hi All,

Has performance testing and comparison of HDFS single node cluster vs. NTFS file systems been performed? Any sample results of HDFS single node vs. NTFS performance comparison available?

Your input/feedback regarding this would be very helpful.

Regards,
Harish Kashyap

Search Discussions

  • Aaron Kimball at Sep 22, 2009 at 8:19 pm
    To my knowledge, nobody's benchmarked this in a rigorous fashion. It's
    virtually certain, though, that on the same machine, NTFS would perform
    faster. HDFS does not directly write to the disk driver, it uses the local
    filesystem of the node on which it's installed. So any HDFS writes would
    themselves be channeled through NTFS and then down to the disk. The read
    path, of course, would go through NTFS first and then via HDFS out to the
    client.

    So, HDFS can only add overhead. How much overhead is probably not a
    published number.

    - Aaron
    On Tue, Sep 22, 2009 at 7:25 AM, HarishKashyap TS wrote:

    Hi All,



    Has performance testing and comparison of HDFS *single node* cluster vs.
    NTFS file systems been performed? Any sample results of HDFS single node vs.
    NTFS performance comparison available?



    Your input/feedback regarding this would be very helpful.



    Regards,

    Harish Kashyap

  • HarishKashyap TS at Sep 23, 2009 at 3:11 pm
    Hi All,

    I have completed a performance testing activity of HDFS single node vs. NTFS file systems. Modified versions of SLG tools provided by Hadoop has been utilized for this activity. Under similar environment conditions, performance of the two file systems has been compared across various file operations.
    From our tests, statistics related to the amount of overhead introduced by HDFS can be obtained.
    For E.g. If number of file created is considers as a metric, then, local file system (NTFS) performs 30% better when compared to HDFS.

    We are planning to publish an article on this. Suggestions about the technical forums, where the publication of this article would be appropriate, will be of great help.

    Aaron,
    Thanks a lot for your inputs and time.

    Regards,
    Harish Kashyap

    From: Aaron Kimball
    Sent: Wednesday, September 23, 2009 1:49 AM
    To: hdfs-user@hadoop.apache.org
    Subject: Re: HDFS single node cluster vs. NTFS performance comparison

    To my knowledge, nobody's benchmarked this in a rigorous fashion. It's virtually certain, though, that on the same machine, NTFS would perform faster. HDFS does not directly write to the disk driver, it uses the local filesystem of the node on which it's installed. So any HDFS writes would themselves be channeled through NTFS and then down to the disk. The read path, of course, would go through NTFS first and then via HDFS out to the client.

    So, HDFS can only add overhead. How much overhead is probably not a published number.

    - Aaron
    On Tue, Sep 22, 2009 at 7:25 AM, HarishKashyap TS wrote:

    Hi All,



    Has performance testing and comparison of HDFS single node cluster vs. NTFS file systems been performed? Any sample results of HDFS single node vs. NTFS performance comparison available?



    Your input/feedback regarding this would be very helpful.



    Regards,

    Harish Kashyap
  • Anthony Urso at Sep 24, 2009 at 3:40 am
    The Annals of Improbable Research may be interested. I believe they
    recently published a study comparing apples and oranges.

    Cheers,
    Anthony

    On Wed, Sep 23, 2009 at 8:11 AM, HarishKashyap TS
    wrote:
    Hi All,



    I have completed a performance testing activity of HDFS single node vs. NTFS
    file systems. Modified versions of SLG tools provided by Hadoop has been
    utilized for this activity. Under similar environment conditions,
    performance of the two file systems has been compared across various file
    operations.

    From our tests, statistics related to the amount of overhead introduced by
    HDFS can be obtained.

    For E.g. If number of file created is considers as a metric, then, local
    file system (NTFS) performs 30% better when compared to HDFS.



    We are planning to publish an article on this. Suggestions about the
    technical forums, where the publication of this article would be
    appropriate, will be of great help.



    Aaron,

    Thanks a lot for your inputs and time.



    Regards,

    Harish Kashyap



    From: Aaron Kimball
    Sent: Wednesday, September 23, 2009 1:49 AM
    To: hdfs-user@hadoop.apache.org
    Subject: Re: HDFS single node cluster vs. NTFS performance comparison



    To my knowledge, nobody's benchmarked this in a rigorous fashion. It's
    virtually certain, though, that on the same machine, NTFS would perform
    faster. HDFS does not directly write to the disk driver, it uses the local
    filesystem of the node on which it's installed. So any HDFS writes would
    themselves be channeled through NTFS and then down to the disk. The read
    path, of course, would go through NTFS first and then via HDFS out to the
    client.

    So, HDFS can only add overhead. How much overhead is probably not a
    published number.

    - Aaron

    On Tue, Sep 22, 2009 at 7:25 AM, HarishKashyap TS
    wrote:

    Hi All,



    Has performance testing and comparison of HDFS single node cluster vs. NTFS
    file systems been performed? Any sample results of HDFS single node vs. NTFS
    performance comparison available?



    Your input/feedback regarding this would be very helpful.



    Regards,

    Harish Kashyap


Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouphdfs-user @
categorieshadoop
postedSep 22, '09 at 2:25p
activeSep 24, '09 at 3:40a
posts4
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase