FAQ
I am newbie.

Most of example shows that,
job.setOutputKeyClass(Text.class);

is it possible job.setOutputKeyClass(MapWritable.class);

because my key is combination of values(src IP, src Port, dst Port,
dst IP), so I want to use MapWritable as a key.

example code is like:

MapWritable mkey = new MapWritable();
MapWritable mval = new MapWritable();

mkey.put(new Text("sip"), new Text(array[3]));
mkey.put(new Text("sport"), new Text(array[5]));
mkey.put(new Text("proto"), new Text(array[7]));
mkey.put(new Text("dport"), new Text(array[6]));
mkey.put(new Text("dip"), new Text(array[4]));


mval.put(new Text("bcount"), new IntWritable(bcount));
mval.put(new Text("pcount"), new IntWritable(pcount));

context.write(mkey, mval);

Search Discussions

  • Rajesh putta at Jul 20, 2011 at 5:25 am
    Hi,
    As per my knowledege is concerned
    MapWritable doesn't implement Comparable, so I think it cannot be used as a
    key. If u want that functionality, then u have to have a subclass that
    implements Comparable and there u will define your key comparison logic.or
    the other option would be to use SortedMapWritable as key.


    Thanks& Regards
    Rajesh Putta
    M Tech CSE
    IIIT-H
    On Wed, Jul 20, 2011 at 5:32 AM, Choonho Son wrote:

    I am newbie.

    Most of example shows that,
    job.setOutputKeyClass(Text.class);

    is it possible job.setOutputKeyClass(MapWritable.class);

    because my key is combination of values(src IP, src Port, dst Port,
    dst IP), so I want to use MapWritable as a key.

    example code is like:

    MapWritable mkey = new MapWritable();
    MapWritable mval = new MapWritable();

    mkey.put(new Text("sip"), new Text(array[3]));
    mkey.put(new Text("sport"), new Text(array[5]));
    mkey.put(new Text("proto"), new Text(array[7]));
    mkey.put(new Text("dport"), new Text(array[6]));
    mkey.put(new Text("dip"), new Text(array[4]));


    mval.put(new Text("bcount"), new IntWritable(bcount));
    mval.put(new Text("pcount"), new IntWritable(pcount));

    context.write(mkey, mval);
  • Harsh J at Jul 20, 2011 at 6:08 am
    If your key is a "fixed" one of four attributes, why not simply use an
    ArrayWritable of Text objects, over a MapWritable?
    On Wed, Jul 20, 2011 at 5:32 AM, Choonho Son wrote:
    I am newbie.

    Most of example shows that,
    job.setOutputKeyClass(Text.class);

    is it possible job.setOutputKeyClass(MapWritable.class);

    because my key is combination of values(src IP, src Port, dst Port,
    dst IP), so I want to use MapWritable as a key.

    example code is like:

    MapWritable mkey = new MapWritable();
    MapWritable mval = new MapWritable();

    mkey.put(new Text("sip"), new Text(array[3]));
    mkey.put(new Text("sport"), new Text(array[5]));
    mkey.put(new Text("proto"), new Text(array[7]));
    mkey.put(new Text("dport"), new Text(array[6]));
    mkey.put(new Text("dip"), new Text(array[4]));


    mval.put(new Text("bcount"), new IntWritable(bcount));
    mval.put(new Text("pcount"), new IntWritable(pcount));

    context.write(mkey, mval);


    --
    Harsh J
  • Harsh J at Jul 20, 2011 at 6:11 am
    Btw, also checkout Avro's MapReduce components. Its a much better
    serialization framework, and you'll have lesser issues figuring out
    datatypes to use + more performance from good use of codecs.
    On Wed, Jul 20, 2011 at 11:37 AM, Harsh J wrote:
    If your key is a "fixed" one of four attributes, why not simply use an
    ArrayWritable of Text objects, over a MapWritable?
    On Wed, Jul 20, 2011 at 5:32 AM, Choonho Son wrote:
    I am newbie.

    Most of example shows that,
    job.setOutputKeyClass(Text.class);

    is it possible job.setOutputKeyClass(MapWritable.class);

    because my key is combination of values(src IP, src Port, dst Port,
    dst IP), so I want to use MapWritable as a key.

    example code is like:

    MapWritable mkey = new MapWritable();
    MapWritable mval = new MapWritable();

    mkey.put(new Text("sip"), new Text(array[3]));
    mkey.put(new Text("sport"), new Text(array[5]));
    mkey.put(new Text("proto"), new Text(array[7]));
    mkey.put(new Text("dport"), new Text(array[6]));
    mkey.put(new Text("dip"), new Text(array[4]));


    mval.put(new Text("bcount"), new IntWritable(bcount));
    mval.put(new Text("pcount"), new IntWritable(pcount));

    context.write(mkey, mval);


    --
    Harsh J


    --
    Harsh J
  • John Armstrong at Jul 20, 2011 at 4:11 pm

    On Tue, 19 Jul 2011 17:02:32 -0700, Choonho Son wrote:
    is it possible job.setOutputKeyClass(MapWritable.class);
    As others have said, MapWritable doesn't implement Comparable, so it can't
    be used as a key. The ArrayWritable of Texts is one idea, but I'd suggest
    instead implementing your OWN WritableComparable class MyKey, which would
    contain the four strings as member fields.

    Just make a Plain Old Java Object, implement Writable by telling it how to
    write and read the fields one at a time (and in the same order!), and
    implement comparable as usual for POJOs (remember to override .equals() and
    .hashCode() too!). This way you get type-safety as well, and you know
    exactly what your keys will look like.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedJul 20, '11 at 12:03a
activeJul 20, '11 at 4:11p
posts5
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase