[Hadoop-common-user] Manually splitting files in blocks

Yuri,

Probably the easiest thing is to actually create distinct files and
configure the block size per file such that HDFS doesn't split it into
smaller blocks for you.

- P

Hi Yuri,

You can also check the source code of FileInputFormat and create your own
RecordReader implementation.
Thanks and Regards,
Sonal
www.meghsoft.com

You should create a CustomInputSplit and CustomRecordReader (should have
start and end tag )

ok so far so good. thanks for the reply. i'm trying to implement a custom
file input format. but i can set it only in the job configuration:
job.setInputFormatClass(CustomFileInputFormat.class);

how do i make hadoop implement the file format, or the custom file split
when i upload new files to the hdfs? do i need a custom upload interface for
that or is there a hadoop config option for that?

tnx


ANKITBHATNAGAR wrote:

So this is how it goes

1- customInputFormat or whatever name extends the TextFormat
2- this class has a method that returns the RecordReader Object
3- u have to create a customRecordReader as well that reads the blocks

Ankit



-----Original Message-----
From: Yuri K.
Sent: Friday, March 26, 2010 10:49 AM
To: core-user@hadoop.apache.org
Subject: Re: Manually splitting files in blocks


ok so far so good. thanks for the reply. i'm trying to implement a custom
file input format. but i can set it only in the job configuration:
job.setInputFormatClass(CustomFileInputFormat.class);

how do i make hadoop implement the file format, or the custom file split
when i upload new files to the hdfs? do i need a custom upload interface for
that or is there a hadoop config option for that?

tnx


ANKITBHATNAGAR wrote:

HDFS splits files in chunks regardless of their content.

This is what I understood so far:
an InputFormat object reads the file and returns a list of InputSplit
object (an object that usually contains the boundaries of the file
section your map task will read). Moreover InputFormat contains a method
to return a RecordReader, able to read and interpret your InputSplit.

Therefore, when you have a file for which none of the existent
InputSplit implementations work, you can:

- if you are able to start reading from an arbitrary offset in the file
(there are sync points in the file, such as \n o spaces in textfile),
you can just define your custom recordreader.
- if your file is splittable in fixed length chunks, you have to
override the getSplits method of InputFormat in order to supply to the
job a whole record, and of course, subclass InputFormat.

BR,
Antonio Barbuzzi


-------- Original Message --------
Subject: Re: Manually splitting files in blocks
From: Yuri K. <mr_greenshit@hotmail.com>
To: core-user@hadoop.apache.org
Date: Fri Mar 26 2010 15:49:09 GMT+0100 (CET)

Inline
This is exactly right. The custom input code ends up bundled in your job jar
and is available to the job at runtime just like any other dependency
library. Alternately, you could package your new input format into it's own
jar and "install" it onto the cluster by pushing it out to the
$HADOOP_HOME/lib on every machine. Unless you're building a common
infrastructure for a disparate set of users, I'd recommend the former
approach.

how do i make hadoop implement the file format, or the custom file split
My understanding (please correct me, list) is that hadoop will always spit
your files based on the block size setting. The InputSplit and RecordReaders
are used by jobs to retrieve chunks of files for processing - that is, there
are two separate splits happening here: one "physical" split for storage and
one "logical" split for processing.

Cheers,
-Nick

My understanding (please correct me, list) is that hadoop will always spit
That's right. The physical split are HDFS "blocks". An InputSplit is a
logical split, and represents a unit of work that is sent to a single
Mapper. A RecordReader provides a record-oriented view of the data. In most
cases, the last record in each InputSplit will span the input split's
boundaries, or even a block's boundaries. In the latter case, data is
transfered from a DN that holds the next contiguous block such that the
reader can construct a full record.