Grokbase › Groups › Hadoop › common-user › November 2009
FAQ
Hi, I am new to hadoop. I am planning to do matrix multiplication(of order
millions) using hadoop.

I have a few queries regarding the above.

i) Will using hadoop be a fix for this or should I try some other
approaches?
ii) I will be using it in NFS. Will using hadoop still be a good option?

If I can use hadoop for this problem, could you plz send links to configure
hadoop-site.xml file for a nfs system.

P.S. I tried a few setup instructions via search, but everything seems to
give "Unable to connect to ...." error.

--
View this message in context: http://old.nabble.com/Using-hadoop-for-Matrix-Multiplication-in-NFS--tp26332382p26332382.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Search Discussions

  • Zjffdu at Nov 13, 2009 at 2:40 pm ⇧
    See my comments


    -----Original Message-----
    From: Gimick
    Sent: 2009年11月12日 23:22
    To: core-user@hadoop.apache.org
    Subject: Using hadoop for Matrix Multiplication in NFS?


    Hi, I am new to hadoop. I am planning to do matrix multiplication(of order
    millions) using hadoop.

    I have a few queries regarding the above.

    i) Will using hadoop be a fix for this or should I try some other
    approaches?

    --- Hama maybe such a tool that fit for your requirement,
    http://incubator.apache.org/hama/

    ii) I will be using it in NFS. Will using hadoop still be a good option?
    --- If you want to use NFS, I guess you have to provide your own
    InputFormat. So you'd better put your data into hdfs, it will make your work
    easy and improve your program's performance



    If I can use hadoop for this problem, could you plz send links to configure
    hadoop-site.xml file for a nfs system.

    P.S. I tried a few setup instructions via search, but everything seems to
    give "Unable to connect to ...." error.

    --
    View this message in context:
    http://old.nabble.com/Using-hadoop-for-Matrix-Multiplication-in-NFS--tp26332
    382p26332382.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
  • Brian Bockelman at Nov 13, 2009 at 5:07 pm ⇧
    Hi,

    Assuming you're doing math...
    What you want is PETSc for sparse matrices: http://www.mcs.anl.gov/petsc/petsc-as/
    If you're doing dense matrices, probable scalapack: http://www.netlib.org/scalapack/

    You benefit from working with someone who has a background in
    numerical analysis.

    Brian
    On Nov 14, 2009, at 12:42 AM, zjffdu wrote:

    See my comments


    -----Original Message-----
    From: Gimick
    Sent: 2009年11月12日 23:22
    To: core-user@hadoop.apache.org
    Subject: Using hadoop for Matrix Multiplication in NFS?


    Hi, I am new to hadoop. I am planning to do matrix multiplication
    (of order
    millions) using hadoop.

    I have a few queries regarding the above.

    i) Will using hadoop be a fix for this or should I try some other
    approaches?

    --- Hama maybe such a tool that fit for your requirement,
    http://incubator.apache.org/hama/

    ii) I will be using it in NFS. Will using hadoop still be a good
    option?
    --- If you want to use NFS, I guess you have to provide your own
    InputFormat. So you'd better put your data into hdfs, it will make
    your work
    easy and improve your program's performance



    If I can use hadoop for this problem, could you plz send links to
    configure
    hadoop-site.xml file for a nfs system.

    P.S. I tried a few setup instructions via search, but everything
    seems to
    give "Unable to connect to ...." error.

    --
    View this message in context:
    http://old.nabble.com/Using-hadoop-for-Matrix-Multiplication-in-NFS--tp26332
    382p26332382.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
  • Otis Gospodnetic at Nov 14, 2009 at 3:11 am ⇧
    I think another thing to look at is Mahout - http://lucene.apache.org/mahout

    See http://mahout.markmail.org/search/matrix+multiplication

    Otis
    --
    Sematext is hiring -- http://sematext.com/about/jobs.html?mls
    Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR


    ----- Original Message ----
    From: Brian Bockelman <bbockelm@cse.unl.edu>
    To: common-user@hadoop.apache.org
    Cc: core-user@hadoop.apache.org
    Sent: Fri, November 13, 2009 12:06:37 PM
    Subject: Re: Using hadoop for Matrix Multiplication in NFS?

    Hi,

    Assuming you're doing math...
    What you want is PETSc for sparse matrices:
    http://www.mcs.anl.gov/petsc/petsc-as/
    If you're doing dense matrices, probable scalapack:
    http://www.netlib.org/scalapack/

    You benefit from working with someone who has a background in numerical
    analysis.

    Brian
    On Nov 14, 2009, at 12:42 AM, zjffdu wrote:

    See my comments


    -----Original Message-----
    From: Gimick
    Sent: 2009年11月12日 23:22
    To: core-user@hadoop.apache.org
    Subject: Using hadoop for Matrix Multiplication in NFS?


    Hi, I am new to hadoop. I am planning to do matrix multiplication(of order
    millions) using hadoop.

    I have a few queries regarding the above.

    i) Will using hadoop be a fix for this or should I try some other
    approaches?

    --- Hama maybe such a tool that fit for your requirement,
    http://incubator.apache.org/hama/

    ii) I will be using it in NFS. Will using hadoop still be a good option?
    --- If you want to use NFS, I guess you have to provide your own
    InputFormat. So you'd better put your data into hdfs, it will make your work
    easy and improve your program's performance



    If I can use hadoop for this problem, could you plz send links to configure
    hadoop-site.xml file for a nfs system.

    P.S. I tried a few setup instructions via search, but everything seems to
    give "Unable to connect to ...." error.

    --View this message in context:
    http://old.nabble.com/Using-hadoop-for-Matrix-Multiplication-in-NFS--tp26332
    382p26332382.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
  • Allen Wittenauer at Nov 13, 2009 at 3:04 pm ⇧

    On 11/12/09 11:21 PM, "Gimick" wrote:
    ii) I will be using it in NFS. Will using hadoop still be a good option?
    If you are using NFS, then no.

    You should be looking at something in the more traditional HPC space: Sun
    Grid Engine, Torque/Maui, etc.
  • Martin Mituzas at Nov 24, 2009 at 7:18 am ⇧
    ii) I once run the mapred program DistCp to copy data from NFS into HDFS. I
    mount the file system directory to each node. Thus it can be connected.


    Gimick wrote:
    Hi, I am new to hadoop. I am planning to do matrix multiplication(of
    order millions) using hadoop.

    I have a few queries regarding the above.

    i) Will using hadoop be a fix for this or should I try some other
    approaches?
    ii) I will be using it in NFS. Will using hadoop still be a good option?

    If I can use hadoop for this problem, could you plz send links to
    configure hadoop-site.xml file for a nfs system.

    P.S. I tried a few setup instructions via search, but everything seems to
    give "Unable to connect to ...." error.
    --
    View this message in context: http://old.nabble.com/Using-hadoop-for-Matrix-Multiplication-in-NFS--tp26332382p26491406.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
  • Edward J. Yoon at Nov 25, 2009 at 6:31 pm ⇧
    Just FYI, Hadoop and M/R is a distributed computing system. So, there
    is a problem of locality and location of sub-matrix blocks. Moreover,
    M/R iteration method is really slow.

    To perform the matrix multiplication (and also graph algorithm) on
    Hadoop, Apache Hama team is considering a BSP (bulk synchronous
    parallel) model using Hadoop RPC instead of M/R.
    On Fri, Nov 13, 2009 at 4:21 PM, Gimick wrote:

    Hi, I am new to hadoop.  I am planning to do matrix multiplication(of order
    millions) using hadoop.

    I have a few queries regarding the above.

    i) Will using hadoop be a fix for this or should I try some other
    approaches?
    ii) I will be using it in NFS.  Will using hadoop still be a good option?

    If I can use hadoop for this problem, could you plz send links to configure
    hadoop-site.xml file for a nfs system.

    P.S. I tried a few setup instructions via search, but everything seems to
    give "Unable to connect to ...." error.

    --
    View this message in context: http://old.nabble.com/Using-hadoop-for-Matrix-Multiplication-in-NFS--tp26332382p26332382.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.


    --
    Best Regards, Edward J. Yoon @ NHN, corp.
    edwardyoon@apache.org
    http://blog.udanax.org
  • Tsz Wo \(Nicholas\), Sze at Nov 25, 2009 at 7:20 pm ⇧
    Hi Gimich,

    Could you describe your matrix multiplication problem in more details? Are the matrices sparse or dense? How big is the on-disk-size of a matrix?

    Thanks.
    Nicholas Sze



    ----- Original Message ----
    From: Edward J. Yoon <edwardyoon@apache.org>
    To: common-user@hadoop.apache.org
    Sent: Tue, November 24, 2009 2:07:57 AM
    Subject: Re: Using hadoop for Matrix Multiplication in NFS?

    Just FYI, Hadoop and M/R is a distributed computing system. So, there
    is a problem of locality and location of sub-matrix blocks. Moreover,
    M/R iteration method is really slow.

    To perform the matrix multiplication (and also graph algorithm) on
    Hadoop, Apache Hama team is considering a BSP (bulk synchronous
    parallel) model using Hadoop RPC instead of M/R.
    On Fri, Nov 13, 2009 at 4:21 PM, Gimick wrote:

    Hi, I am new to hadoop. I am planning to do matrix multiplication(of order
    millions) using hadoop.

    I have a few queries regarding the above.

    i) Will using hadoop be a fix for this or should I try some other
    approaches?
    ii) I will be using it in NFS. Will using hadoop still be a good option?

    If I can use hadoop for this problem, could you plz send links to configure
    hadoop-site.xml file for a nfs system.

    P.S. I tried a few setup instructions via search, but everything seems to
    give "Unable to connect to ...." error.

    --
    View this message in context:
    http://old.nabble.com/Using-hadoop-for-Matrix-Multiplication-in-NFS--tp26332382p26332382.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.


    --
    Best Regards, Edward J. Yoon @ NHN, corp.
    edwardyoon@apache.org
    http://blog.udanax.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedNov 13, '09 at 7:22a
activeNov 25, '09 at 7:20p
posts8
users8
websitehadoop.apache.org...
irc#hadoop

8 users in discussion

Zjffdu: 1 post Edward J. Yoon: 1 post Allen Wittenauer: 1 post Martin Mituzas: 1 post Brian Bockelman: 1 post Otis Gospodnetic: 1 post Tsz Wo \(Nicholas\), Sze: 1 post Gimick: 1 post

Content

People

Support

Translate

site design / logo © 2022 Grokbase