FAQ
Hi,

I am doing a student Independent Study Project and Harvery Mudd has given
me 13 Sun Netra X1 I can use as a dedicated Hadoop cluster. Right now they
are without an OS.

If anyone with experience with Hadoop and Solaris can contact me off list,
even to just say I am doing it and it is OK it would be appreciated.

Thanks in advance,
Bruce Williams
-----------------------------------------------------------------------------------------------------------------------------

Search Discussions

  • Allen Wittenauer at Oct 16, 2010 at 9:34 pm

    On Oct 16, 2010, at 1:08 PM, Bruce Williams wrote:
    I am doing a student Independent Study Project and Harvery Mudd has given
    me 13 Sun Netra X1 I can use as a dedicated Hadoop cluster. Right now they
    are without an OS.

    If anyone with experience with Hadoop and Solaris can contact me off list,
    even to just say I am doing it and it is OK it would be appreciated.


    That's my cue! :)

    We have a few grids that are running Solaris. It mostly works out of the box as long as you are aware of three things:

    - There are some settings in hadoop-env.sh and in the path that need to be dealt with. Rather than re-quote, these were added to the Hadoop FAQ a week or two ago so definitely take a look at that.

    - The native compression libraries will need to be compiled. Depending upon what you are doing/how performant the machines are, this may or may not make a big difference. Compiling under Solaris with gcc will work fine (but it is gcc... ugh!). Only a few minor changes are required to compile it with SUNWspro. [I have patches laying around here somewhere if anyone wants to play with them.]

    - The Solaris JRE is a mixed-mode implementation. So keep in mind that -d32 and -d64 have meaning and do work as advertised. You'll likely want to pick a bitsize and use that for all your Hadoop daemons and tasks, especially if you plan on using any JNI like the compression libraries.
  • Gregor Willemsen at Oct 17, 2010 at 10:19 am
    I have no recent experiences with Solaris nor with Hadoop on Solaris,
    but there is an blog: http://blogs.sun.com/george/category/Hadoop with
    three postings dedicated to Hadoop (although maybe a bit outdated). In
    2007 or 2008 GNU and Solaris did not play together very well as things
    as tar, gcc are concerned. But these issues should have gone now.

    Sun Grid Engine might be a starting point to look for as Hadoop is built in.



    2010/10/16 Allen Wittenauer <awittenauer@linkedin.com>:
    On Oct 16, 2010, at 1:08 PM, Bruce Williams wrote:
    I am doing a student Independent Study  Project and Harvery Mudd has given
    me 13 Sun Netra X1 I can use as a dedicated Hadoop cluster. Right now they
    are without an OS.

    If anyone with experience with Hadoop and Solaris can contact me off list,
    even to just say I am doing it and it is OK it would be appreciated.


    That's my cue! :)

    We have a few grids that are running Solaris. It mostly works out of the box as long as you are aware of three things:

    - There are some settings in hadoop-env.sh and in the path that need to be dealt with.  Rather than re-quote, these were added to the Hadoop FAQ a week or two ago so definitely take a look at that.

    - The native compression libraries will need to be compiled.  Depending upon what you are doing/how performant the machines are, this may or may not make a big difference.  Compiling under Solaris with gcc will work fine (but it is gcc... ugh!).  Only a few minor changes are required to compile it with SUNWspro.  [I have patches laying around here somewhere if anyone wants to play with them.]

    - The Solaris JRE is a mixed-mode implementation.  So keep in mind that -d32 and -d64 have meaning and do work as advertised.  You'll likely want to pick a bitsize and use that for all your Hadoop daemons and tasks, especially if you plan on using any JNI like the compression libraries.
  • Allen Wittenauer at Oct 17, 2010 at 9:22 pm

    On Oct 17, 2010, at 3:18 AM, Gregor Willemsen wrote:

    In
    2007 or 2008 GNU and Solaris did not play together very well as things
    as tar, gcc are concerned.
    Only if you didn't know to look in /usr/sfw or were using such an ancient version of Solaris that one be embarrassed.
  • Gregor Willemsen at Oct 17, 2010 at 9:42 pm
    Of course it is simple if you know about it. From an Linux and Java
    point of view the Solaris way is not that obvious.

    My 2 cents

    2010/10/17 Allen Wittenauer <awittenauer@linkedin.com>:
    On Oct 17, 2010, at 3:18 AM, Gregor Willemsen wrote:

    In
    2007 or 2008 GNU and Solaris did not play together very well as things
    as tar, gcc are concerned.
    Only if you didn't know to look in /usr/sfw or were using such an ancient version of Solaris that one be embarrassed.

  • Allen Wittenauer at Oct 18, 2010 at 12:45 am

    On Oct 17, 2010, at 2:41 PM, Gregor Willemsen wrote:

    Of course it is simple if you know about it. From an Linux and Java
    point of view the Solaris way is not that obvious.
    Java has nothing to do with GNU tar's extensions or gcc.

    Forward compatibility is definitely not the Linux way, so it is easy to understand the confusion.
  • Bruce Williams at Oct 19, 2010 at 12:04 pm
    I didn't want to jump in and say thanks for each post but thanks for
    all. Owen, you probably don't remember me, but we met at that first
    Hadoop meetup in Sunnyvale a couple years ago.
    Bruce Williams
    -----------------------------------------------------------------------------------------------------------------------------





    On Sun, Oct 17, 2010 at 5:44 PM, Allen Wittenauer
    wrote:
    On Oct 17, 2010, at 2:41 PM, Gregor Willemsen wrote:

    Of course it is simple if you know about it. From an Linux and Java
    point of view the Solaris way is not that obvious.
    Java has nothing to do with GNU tar's extensions or gcc.

    Forward compatibility is definitely not the Linux way, so it is easy to understand the confusion.
  • Owen O'Malley at Oct 17, 2010 at 9:33 pm

    On Oct 16, 2010, at 1:08 PM, Bruce Williams wrote:

    If anyone with experience with Hadoop and Solaris can contact me off
    list,
    even to just say I am doing it and it is OK it would be appreciated.
    LinkedIn is currently running Hadoop on Solaris. Hopefully, Allen
    Wittenauer can get back to you on some hints.

    -- Owen

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 16, '10 at 8:09p
activeOct 19, '10 at 12:04p
posts8
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase