[Hadoop-common-user] Is hadoop good for the my job?

On 9/24/06, howard chen wrote:

Hello,

I have a larger XML file, over 10GB, has a simple format like

<book>
<title></title>
<author></author>
...
</book>

I used to parse the XML and convert into another format, i.e. CSV.
Currently, the parsing only performed on a single server and speed is
slow (a few hours)

Is hadoop is a good solution for spliting the XML files and spread the
XML parsing on serveral clusters?

Thanks for any comment.

On 9/25/06, Feng Jiang wrote:
mapreduce doesn't know anything about your application logic. as long as you
can split the big xml into a lot of small xml files, then hadoop could help
you.

1. split this big xml file into 10000 small xml files, for example.
2. each small xml file could be one pair in sequence file.
3. then use mapreduce to read the sequence file and parse them, for example
you have 10 map & reduce tasks.
4. finally you have 10 output files, which contain the format you want.

On 9/27/06, howard chen wrote:

Hello,

in my example, XML paring to CSV seems to be one-to-one mapping, e.g.

<book>
<title>hadoop</title>
<author>peter</author>
<ISBN>121332</ISBN>
</book>

would become (CSV)
hadoop,peter,121332

use mapreduce seems not suitable?

thanks.

On Wed, 27 Sep 2006 06:37:06 +0200, Feng Jiang wrote:

One principle is that the input file must be a sequence of pairs, and you
must have a input formatter for the input file. otherwise you cannot use
mapreduce directly.

On 9/27/06, Vetle Roeim wrote:
Technically, yes, but that pair could just as well consist of XML and some
insignificant value. The most obvious example is processing log files,
where data might be read using TextInputFormat, and the line number simply
discarded.


[...]
In this case, you'd might have to write your own implementation of
InputFormat that reads the entire XML fragment into some kind of data
structure.



--
Vetle Roeim
Team Manager, Information Systems
Opera Software ASA <URL: http://www.opera.com/ >

On 9/28/06, Bryan A. P. Pendleton wrote:
It can be done! :) I'll see if I can contribute the code that I use for
these sorts of things... I've been parsing through nearly a terabyte of XML
on a regular basis using mapreduce since early this year.

One way to do it would be to define a variant of TextInputFormat that,
instead of using end-of-line to delimit elements, using a custom string,
perhaps a regexp. Just define a regexp for the XML element that you can
cleanly treat independently, and away you go. Might not split work as
ideally across block boundaries as you'd like, but it should be possible to
make it perform much better than single-machine single-parser work would.

Of course, worst cases kill you. It might be a problem if one of your
segments is a couple of gigabytes long, for instance, as is the case with my
dataset if you use the highest-level XML container. The current mapreduce
code really requires that your Writable instances fit in memory.


--
Bryan A. P. Pendleton
Ph: (877) geek-1-bp