FAQ
Hello,

I'm stuck with this for two days now ...I found a previous post discussing this, but not with arrays.

I know how to write Writable class with primitive type elements but this time I'm using an ARRAY of primitive type element, here it is in a simplified version for easy readability :

public class twoIntegersArrayWritable implements Writable {

private int vectorSize;
private twoIntegers[] vector ;

/* CONSTRUCTORS */ ===================================================

twoIntegersArrayWritable () {}

twoIntegersArrayWritable (int size, twoIntegers[] vec) {
this.vectorSize = size;
this.vector = vec;
}

/* WRITE FUNCTION */ ===================================================

public void write(DataOutput out) {

out.writeInt(this.vectorSize);
for(int i=0;i<this.vectorSize;i++){
out.writeInt( vector[i]. getFirstIntger());
out.writeInt( vector[i]. getSecondIneger());
}
}

/* READ FUNCTION */ ===================================================

public void readFields(DataInput in) throws IOException {

this.vectorSize = in.readInt();
this.CreateNewVector (this.vectorSize);
for(int i=0;i< this.vectorSize;i++){
this.vector[i].setFirstInteger(in.readInt());
this.vector[i].setSecondInteger(in.readInt());
}
}
}

The errors I get:
java.lang.NullPointerException
at twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)
at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1751)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1879)
at SequenceFileReader_Integers.main(SequenceFileReader_Integers.java:49)

Or it would work fine but array size is always 0. So it won't read the integer fields.

It would be helpful if someone already has an example of using array in their Writable class. I appreciate any thought.

Maha

Search Discussions

  • David Rosenstrauch at Mar 17, 2011 at 9:52 pm
    I would try implementing this using an ArrayWritable, which contains an
    array of IntWritables.

    HTH,

    DR
    On 03/17/2011 05:04 PM, maha wrote:
    Hello,

    I'm stuck with this for two days now ...I found a previous post
    discussing this, but not with arrays.

    I know how to write Writable class with primitive type elements but
    this time I'm using an ARRAY of primitive type element, here it is in
    a simplified version for easy readability :

    public class twoIntegersArrayWritable implements Writable {

    private int vectorSize; private twoIntegers[] vector ;

    /* CONSTRUCTORS */
    ===================================================

    twoIntegersArrayWritable () {}

    twoIntegersArrayWritable (int size, twoIntegers[] vec) {
    this.vectorSize = size; this.vector = vec; }

    /* WRITE FUNCTION */
    ===================================================

    public void write(DataOutput out) {

    out.writeInt(this.vectorSize); for(int i=0;i<this.vectorSize;i++){
    out.writeInt( vector[i]. getFirstIntger()); out.writeInt( vector[i].
    getSecondIneger()); } }

    /* READ FUNCTION */
    ===================================================

    public void readFields(DataInput in) throws IOException {
    this.vectorSize = in.readInt(); this.CreateNewVector
    (this.vectorSize); for(int i=0;i< this.vectorSize;i++){
    this.vector[i].setFirstInteger(in.readInt());
    this.vector[i].setSecondInteger(in.readInt()); } } }

    The errors I get: java.lang.NullPointerException at
    twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)
    at
    org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1751)
    at
    org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1879)

    at SequenceFileReader_Integers.main(SequenceFileReader_Integers.java:49)
    Or it would work fine but array size is always 0. So it won't read
    the integer fields.

    It would be helpful if someone already has an example of using array
    in their Writable class. I appreciate any thought.

    Maha
  • Matthew Foley at Mar 17, 2011 at 9:58 pm
    Hi Maha,

    There are three things missing from your simplified code needed to give a definitive answer, so I'll infer them as follows:
    - the specification of "twoIntegers"
    "twoIntegers" seems to be a tuple of ints with get/set accessors, so it is an object (array of primitives?) not a primitive, right?
    - in the exception "twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)", what line 46 refers to
    I'm betting it refers to the line "this.vector[i].setFirstInteger(in.readInt());"
    - the implementation of CreateNewVector(size)
    It seems likely it just does a "new twoIntegers[size];"

    If the above is correct, then the problem is that standard Java behavior is to fill the new array with nulls rather than with empty twoInteger objects. So in readFields(), when you enter the loop to read values for vector, the code "this.vector[0]" fetches a null object, and the continuation "this.vector[0].setFirstInteger()" causes a NullPointerException.

    You can fix this in CreateNewVector(), by explicitly allocating a new twoInteger object for each location in the "vector" array, or in the readFields() loop, whichever suits your program semantics better.

    --Matt


    On Mar 17, 2011, at 2:04 PM, maha wrote:

    Hello,

    I'm stuck with this for two days now ...I found a previous post discussing this, but not with arrays.

    I know how to write Writable class with primitive type elements but this time I'm using an ARRAY of primitive type element, here it is in a simplified version for easy readability :

    public class twoIntegersArrayWritable implements Writable {

    private int vectorSize;
    private twoIntegers[] vector ;

    /* CONSTRUCTORS */ ===================================================

    twoIntegersArrayWritable () {}

    twoIntegersArrayWritable (int size, twoIntegers[] vec) {
    this.vectorSize = size;
    this.vector = vec;
    }

    /* WRITE FUNCTION */ ===================================================

    public void write(DataOutput out) {

    out.writeInt(this.vectorSize);
    for(int i=0;i<this.vectorSize;i++){
    out.writeInt( vector[i]. getFirstIntger());
    out.writeInt( vector[i]. getSecondIneger());
    }
    }

    /* READ FUNCTION */ ===================================================

    public void readFields(DataInput in) throws IOException {

    this.vectorSize = in.readInt();
    this.CreateNewVector (this.vectorSize);
    for(int i=0;i< this.vectorSize;i++){
    this.vector[i].setFirstInteger(in.readInt());
    this.vector[i].setSecondInteger(in.readInt());
    }
    }
    }

    The errors I get:
    java.lang.NullPointerException
    at twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)
    at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1751)
    at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1879)
    at SequenceFileReader_Integers.main(SequenceFileReader_Integers.java:49)

    Or it would work fine but array size is always 0. So it won't read the integer fields.

    It would be helpful if someone already has an example of using array in their Writable class. I appreciate any thought.

    Maha
  • Maha at Mar 17, 2011 at 11:42 pm
    Thanks for your time Matt. Below are my clarifications ...

    There are three things missing from your simplified code needed to give a definitive answer, so I'll infer them as follows:
    - the specification of "twoIntegers"
    "twoIntegers" seems to be a tuple of ints with get/set accessors, so it is an object (array of primitives?) not a primitive, right?
    public class twoIntegers () {
    private int x;
    private int y;
    .....
    }

    so it's not an array. That's why I didn't use the ArrayWritable (although I tried but I didn't succeed).
    - in the exception "twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)", what line 46 refers to
    I'm betting it refers to the line "this.vector[i].setFirstInteger(in.readInt());"
    Yes it is . After some investigation, it turns that 'in' is empty at this line, but it's because even though in the write function I have :

    for(int i=0;i<this.vectorSize;i++)
    out.writeInt( vector[i]. getFirstIntger());
    out.writeInt( vector[i]. getSecondIneger());

    but vectorSize is 0 ! so that's why nothing is written after vectorSize into DataOutputStream in the first place to be read later :( although the initialization of the class specifies clearly that the size is 10.

    - the implementation of CreateNewVector(size)
    It seems likely it just does a "new twoIntegers[size];"
    Yes this is what I meant to do , because I wanted to make sure that my array is not of size 0, but it's still of size zero.
    If the above is correct, then the problem is that standard Java behavior is to fill the new array with nulls rather than with empty twoInteger objects. So in readFields(), when you enter the loop to read values for vector, the code "this.vector[0]" fetches a null object, and the continuation "this.vector[0].setFirstInteger()" causes a NullPointerException.
    Do you mean I have to use Integer objects instead of primitive types ??
    You can fix this in CreateNewVector(), by explicitly allocating a new twoInteger object for each location in the "vector" array, or in the readFields() loop, whichever suits your program semantics better.

    --Matt
    On Mar 17, 2011, at 2:04 PM, maha wrote:

    Hello,

    I'm stuck with this for two days now ...I found a previous post discussing this, but not with arrays.

    I know how to write Writable class with primitive type elements but this time I'm using an ARRAY of primitive type element, here it is in a simplified version for easy readability :

    public class twoIntegersArrayWritable implements Writable {

    private int vectorSize;
    private twoIntegers[] vector ;

    /* CONSTRUCTORS */ ===================================================

    twoIntegersArrayWritable () {}

    twoIntegersArrayWritable (int size, twoIntegers[] vec) {
    this.vectorSize = size;
    this.vector = vec;
    }

    /* WRITE FUNCTION */ ===================================================

    public void write(DataOutput out) {

    out.writeInt(this.vectorSize);
    for(int i=0;i<this.vectorSize;i++){
    out.writeInt( vector[i]. getFirstIntger());
    out.writeInt( vector[i]. getSecondIneger());
    }
    }

    /* READ FUNCTION */ ===================================================

    public void readFields(DataInput in) throws IOException {

    this.vectorSize = in.readInt();
    this.CreateNewVector (this.vectorSize);
    for(int i=0;i< this.vectorSize;i++){
    this.vector[i].setFirstInteger(in.readInt());
    this.vector[i].setSecondInteger(in.readInt());
    }
    }
    }

    The errors I get:
    java.lang.NullPointerException
    at twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)
    at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1751)
    at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1879)
    at SequenceFileReader_Integers.main(SequenceFileReader_Integers.java:49)

    Or it would work fine but array size is always 0. So it won't read the integer fields.

    It would be helpful if someone already has an example of using array in their Writable class. I appreciate any thought.

    Maha
  • Matthew Foley at Mar 18, 2011 at 4:59 pm
    Maha and I went off-line and resolved this issue, which was just a Java programming error.
    Cheers,
    --Matt

    On Mar 17, 2011, at 4:41 PM, maha wrote:

    Thanks for your time Matt. Below are my clarifications ...

    There are three things missing from your simplified code needed to give a definitive answer, so I'll infer them as follows:
    - the specification of "twoIntegers"
    "twoIntegers" seems to be a tuple of ints with get/set accessors, so it is an object (array of primitives?) not a primitive, right?
    public class twoIntegers () {
    private int x;
    private int y;
    .....
    }

    so it's not an array. That's why I didn't use the ArrayWritable (although I tried but I didn't succeed).
    - in the exception "twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)", what line 46 refers to
    I'm betting it refers to the line "this.vector[i].setFirstInteger(in.readInt());"
    Yes it is . After some investigation, it turns that 'in' is empty at this line, but it's because even though in the write function I have :

    for(int i=0;i<this.vectorSize;i++)
    out.writeInt( vector[i]. getFirstIntger());
    out.writeInt( vector[i]. getSecondIneger());

    but vectorSize is 0 ! so that's why nothing is written after vectorSize into DataOutputStream in the first place to be read later :( although the initialization of the class specifies clearly that the size is 10.

    - the implementation of CreateNewVector(size)
    It seems likely it just does a "new twoIntegers[size];"
    Yes this is what I meant to do , because I wanted to make sure that my array is not of size 0, but it's still of size zero.
    If the above is correct, then the problem is that standard Java behavior is to fill the new array with nulls rather than with empty twoInteger objects. So in readFields(), when you enter the loop to read values for vector, the code "this.vector[0]" fetches a null object, and the continuation "this.vector[0].setFirstInteger()" causes a NullPointerException.
    Do you mean I have to use Integer objects instead of primitive types ??
    You can fix this in CreateNewVector(), by explicitly allocating a new twoInteger object for each location in the "vector" array, or in the readFields() loop, whichever suits your program semantics better.

    --Matt
    On Mar 17, 2011, at 2:04 PM, maha wrote:

    Hello,

    I'm stuck with this for two days now ...I found a previous post discussing this, but not with arrays.

    I know how to write Writable class with primitive type elements but this time I'm using an ARRAY of primitive type element, here it is in a simplified version for easy readability :

    public class twoIntegersArrayWritable implements Writable {

    private int vectorSize;
    private twoIntegers[] vector ;

    /* CONSTRUCTORS */ ===================================================

    twoIntegersArrayWritable () {}

    twoIntegersArrayWritable (int size, twoIntegers[] vec) {
    this.vectorSize = size;
    this.vector = vec;
    }

    /* WRITE FUNCTION */ ===================================================

    public void write(DataOutput out) {

    out.writeInt(this.vectorSize);
    for(int i=0;i<this.vectorSize;i++){
    out.writeInt( vector[i]. getFirstIntger());
    out.writeInt( vector[i]. getSecondIneger());
    }
    }

    /* READ FUNCTION */ ===================================================

    public void readFields(DataInput in) throws IOException {

    this.vectorSize = in.readInt();
    this.CreateNewVector (this.vectorSize);
    for(int i=0;i< this.vectorSize;i++){
    this.vector[i].setFirstInteger(in.readInt());
    this.vector[i].setSecondInteger(in.readInt());
    }
    }
    }

    The errors I get:
    java.lang.NullPointerException
    at twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)
    at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1751)
    at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1879)
    at SequenceFileReader_Integers.main(SequenceFileReader_Integers.java:49)

    Or it would work fine but array size is always 0. So it won't read the integer fields.

    It would be helpful if someone already has an example of using array in their Writable class. I appreciate any thought.

    Maha

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMar 17, '11 at 9:05p
activeMar 18, '11 at 4:59p
posts5
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase