[Hadoop-common-user] Project ideas !

Some of the projects include:
1) Categorise URLS based on domains
2) Content based searching
3) P2P information retrieval
4) Performance enhancements in map-reduce.
5) Sort and shuffle optimisations in MR framework.
6) Enhancements of scheduling strategies in hadoop
7) Document classification
8) Document Ranking

Infact all batch applications that can be parallelised are suitable for hadoop.
G Sudha Sadasivam



--- On Wed, 10/14/09, Siddu wrote:


From: Siddu <siddu.sjce@gmail.com>
Subject: Project ideas !
To: common-user@hadoop.apache.org
Cc: core-user@hadoop.apache.org
Date: Wednesday, October 14, 2009, 3:38 PM


Hello Hadoop Users,

Me and another friend of mine are looking out for some of the project ideas
based on hadoop

as a part of our  curriculum .


Can you give us some pointers please


Thanks in advance !

Regards,
~Sid~

I am interested to see more spatial processing carried out on hadoop.
I have done basic spatial joins intersecting 100s millions of points
with 100s thousands of polygons but this is all. It's something I'd
like to spend time researching, but don't have that time... could be a
nice piece of research since everybody loves maps.

Cheers,
Tim






On Wed, Oct 14, 2009 at 1:20 PM, sudha sadhasivam
wrote:

I have done basic spatial joins intersecting 100s millions of points
with 100s thousands of polygons but this is all. It's something I'd
like to spend time researching, but don't have that time... could be a
nice piece of research since everybody loves maps.

yes tim that sounds interesting ... do u ave any link of urs detailing the
work ?
would be happy to go through it !

Cheers,
Tim






On Wed, Oct 14, 2009 at 1:20 PM, sudha sadhasivam
wrote:

Has any publications been done in this area?
G Sudha Sadasivam

--- On Sat, 10/17/09, Siddu wrote:


From: Siddu <siddu.sjce@gmail.com>
Subject: Re: Project ideas !
To: common-user@hadoop.apache.org
Date: Saturday, October 17, 2009, 3:29 PM

I have done basic spatial joins intersecting 100s millions of points
with 100s thousands of polygons but this is all.  It's something I'd
like to spend time researching, but don't have that time... could be a
nice piece of research since everybody loves maps.

yes tim that sounds interesting ... do u ave any link of urs detailing the
work ?
would be happy to go through it !

Cheers,
Tim






On Wed, Oct 14, 2009 at 1:20 PM, sudha sadhasivam
wrote:

I saw Chris who does Cascading tweet about someone building RTree
indexes recently in hadoop and wanted to follow up with him about who
was doing that.

For my part I very hastily wrote a blog
(http://biodivertido.blogspot.com/2008/11/reproducing-spatial-joins-using-hadoop.html)
but really I did not get far. I could imagine running some initial MR
job on each side of a huge join before actually doing the join, to
determine the best join order, and potentially a (spatial)
partitioning strategy (e.g. build some RTrees, or perhaps subset the
data for different areas of the world and do the join in multiple
jobs). Then using the output of this analysis stage to actually run
the process / implement the join. There are some nice species
observation and specimen data (100s millions of point based data) that
we are often looking to join with polygon datasets (e.g. protected
areas of the world etc) - if you wanted a real world dataset and
fancied doing something worthwhile (helping protect and understand
species) it could be arranged.

Cheers,
Tim

Siddu,
If this is for an undergraduate class, I would suggest something that
allows you to get some work in with basic data structures such as
building an inverted index over a few million documents (maybe Wikipedia
pages?). You will also need to get a general feel for Hadoop.

The University of Washington has some really nice project ideas for
their distributed systems class:

http://www.cs.washington.edu/education/courses/cse490h/09wi/projects/490
H.project.ideas.pdf

If you wanted to tackle something a little more advanced, then you could
take a look at Pete Skomoroch's article on finding trends with Hadoop
and Hive:

http://www.cloudera.com/blog/2009/07/31/tracking-trends-with-hadoop-and-
hive-on-ec2/

http://www.cloudera.com/blog/2009/09/28/grouping-related-trends-with-had
oop-and-hive/

Things to keep in mind:

1.) Hadoop wont be as simple as writing a single Java app
2.) There will be some overhead involved in re-writing algorithms in Map
Reduce
3.) There will also be some overhead involved in setup and maintenance
of the Hadoop Cluster

Take these three things into account when planning how to manage your
time for the project during the semester, semesters can seem a lot
shorter when you spend too much time on things not related to just
implementing and testing your algorithm.

Good luck!

Josh Patterson
TVA



-----Original Message-----
From: Siddu
Sent: Wednesday, October 14, 2009 6:09 AM
To: common-user@hadoop.apache.org
Cc: core-user@hadoop.apache.org
Subject: Project ideas !

Hello Hadoop Users,

Me and another friend of mine are looking out for some of the project
ideas
based on hadoop

as a part of our curriculum .


Can you give us some pointers please


Thanks in advance !

Regards,
~Sid~

2009/10/14 Patterson, Josh <jpatterson0@tva.gov>
Related: CUSUM charts are used for interpretation of (noisy) time series
(e.g. collected from sensor measurements), perhaps efficiently create those
with hadoop?
http://www.variation.com/cpa/help/hs108.htm
http://www.itl.nist.gov/div898/handbook/pmc/section3/pmc323.htm
note: R, Pig, Hive could be of relevance(/overlap) for this.

Amund
http://atbrox.com