Hi all,
I have some architectural question.
For my app I have persistent 50 GB data, which stored in HDFS, data is
simple CSV format file.
Also for my app which should be run over this (50 GB) data I have 10 GB
input data also CSV format.
Persistent data and input data don't have commons keys.

In my cluster I have 5 data nodes.
The app does simple match every line of input data with every line of
persistent data.

For solving this task I see two different approaches:
1. Destribute input file to every node using attribute -files, and run job.
But in this case every map will go through 10 GB input data.
2. Devide input file (10 GB) to 5 parts (for instance), run 5 independent
jobs (one per data node for instance), and for every job we will put 2 GB
data. In this case every map should go through 2 GB data. In other words
I'll give every map node it's own input data. But drawback of this approache
is work which I should do before start job and after job finished.

And may be there is more subtle way in hadoop to do this work?

View this message in context: http://old.nabble.com/Architectural-question-tp31365870p31365870.html
Sent from the Hadoop core-dev mailing list archive at Nabble.com.

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 1 | next ›
Discussion Overview
groupcommon-dev @
postedApr 10, '11 at 9:12p
activeApr 10, '11 at 9:12p

1 user in discussion

Oleksiy: 1 post



site design / logo © 2022 Grokbase