FAQ
Hi,
How do I convert DataInput to array of String?
How do I convert ResultSet to array of String?
Thanks. Following is the code:

static class Record implements Writable, DBWritable {
String [] aSAssoc;

public void write(DataOutput arg0) throws IOException {
throw new UnsupportedOperationException("Not supported yet.");
}

public void readFields(DataInput in) throws IOException {
this.aSAssoc = // How to convert DataInput to String Array?
}

public void write(PreparedStatement arg0) throws SQLException {
throw new UnsupportedOperationException("Not supported yet.");
}

public void readFields(ResultSet rs) throws SQLException {
this.aSAssoc = // How to convert ResultSet to String Array?
}
}

--
View this message in context: http://www.nabble.com/How-do-I-convert-DataInput-and-ResultSet-to-array-of-String--tp23770747p23770747.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Search Discussions

  • Dealmaker at Jun 2, 2009 at 4:53 am
    bump. Does anyone know?

    I am using the following class of arraywritable:

    private static class StringArrayWritable extends ArrayWritable {
    private StringArrayWritable (String [] aSString) {
    super (aSString);
    }
    }


    dealmaker wrote:
    Hi,
    How do I convert DataInput to array of String?
    How do I convert ResultSet to array of String?
    Thanks. Following is the code:

    static class Record implements Writable, DBWritable {
    String [] aSAssoc;

    public void write(DataOutput arg0) throws IOException {
    throw new UnsupportedOperationException("Not supported yet.");
    }

    public void readFields(DataInput in) throws IOException {
    this.aSAssoc = // How to convert DataInput to String Array?
    }

    public void write(PreparedStatement arg0) throws SQLException {
    throw new UnsupportedOperationException("Not supported yet.");
    }

    public void readFields(ResultSet rs) throws SQLException {
    this.aSAssoc = // How to convert ResultSet to String Array?
    }
    }
    --
    View this message in context: http://www.nabble.com/How-do-I-convert-DataInput-and-ResultSet-to-array-of-String--tp23770747p23826464.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
  • Aaron Kimball at Jun 2, 2009 at 5:52 pm
    Hi,

    You can't just turn either of these two types into arrays of strings
    automatically, because they are interfaces to underlying streams of data.
    You are required to know what protocol you are implementing -- i.e., how
    many fields you are transmitting -- and manually read through that many
    fields yourself. For example, a DataInput object is effectively a pointer
    into a byte array. There may be many records in that byte array, but you
    only want to read the fields of the first record out.

    For DataInput / DataOutput, you can UTF8-decode the next field by calling
    Text.readString(dataInput) and Text.writeString(dataOutput).
    For ResultSet, you want resultSet.getString(fieldNum)

    As (yet another) shameless plug ( :smile: ), check out the tool we just
    released, which automates database import tasks. It auto-generates the
    classes necessary for your tables, too.
    http://www.cloudera.com/blog/2009/06/01/introducing-sqoop/

    At the very least, you might want to play with it a bit and read its source
    code so you have a better idea of how to implement your own class (since
    you're doing some more creative stuff like building up associative arrays
    for each field).

    Cheers,
    - Aaron
    On Mon, Jun 1, 2009 at 9:53 PM, dealmaker wrote:


    bump. Does anyone know?

    I am using the following class of arraywritable:

    private static class StringArrayWritable extends ArrayWritable {
    private StringArrayWritable (String [] aSString) {
    super (aSString);
    }
    }


    dealmaker wrote:
    Hi,
    How do I convert DataInput to array of String?
    How do I convert ResultSet to array of String?
    Thanks. Following is the code:

    static class Record implements Writable, DBWritable {
    String [] aSAssoc;

    public void write(DataOutput arg0) throws IOException {
    throw new UnsupportedOperationException("Not supported yet.");
    }

    public void readFields(DataInput in) throws IOException {
    this.aSAssoc = // How to convert DataInput to String Array?
    }

    public void write(PreparedStatement arg0) throws SQLException {
    throw new UnsupportedOperationException("Not supported yet.");
    }

    public void readFields(ResultSet rs) throws SQLException {
    this.aSAssoc = // How to convert ResultSet to String Array?
    }
    }
    --
    View this message in context:
    http://www.nabble.com/How-do-I-convert-DataInput-and-ResultSet-to-array-of-String--tp23770747p23826464.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
  • Dealmaker at Jun 3, 2009 at 1:02 am
    Thanks. The number of elements in this array of String is unknown until run
    time. If datainput treats it as a byte array, I still have to know the size
    of each String. How do I do that? Would you suggest some code samples or
    links that deal with similar situation like this? The only examples I got
    are the ones about counting number of words which deal with integers.
    Thanks.


    Aaron Kimball-3 wrote:
    Hi,

    You can't just turn either of these two types into arrays of strings
    automatically, because they are interfaces to underlying streams of data.
    You are required to know what protocol you are implementing -- i.e., how
    many fields you are transmitting -- and manually read through that many
    fields yourself. For example, a DataInput object is effectively a pointer
    into a byte array. There may be many records in that byte array, but you
    only want to read the fields of the first record out.

    For DataInput / DataOutput, you can UTF8-decode the next field by calling
    Text.readString(dataInput) and Text.writeString(dataOutput).
    For ResultSet, you want resultSet.getString(fieldNum)

    As (yet another) shameless plug ( :smile: ), check out the tool we just
    released, which automates database import tasks. It auto-generates the
    classes necessary for your tables, too.
    http://www.cloudera.com/blog/2009/06/01/introducing-sqoop/

    At the very least, you might want to play with it a bit and read its
    source
    code so you have a better idea of how to implement your own class (since
    you're doing some more creative stuff like building up associative arrays
    for each field).

    Cheers,
    - Aaron
    On Mon, Jun 1, 2009 at 9:53 PM, dealmaker wrote:


    bump. Does anyone know?

    I am using the following class of arraywritable:

    private static class StringArrayWritable extends ArrayWritable {
    private StringArrayWritable (String [] aSString) {
    super (aSString);
    }
    }


    dealmaker wrote:
    Hi,
    How do I convert DataInput to array of String?
    How do I convert ResultSet to array of String?
    Thanks. Following is the code:

    static class Record implements Writable, DBWritable {
    String [] aSAssoc;

    public void write(DataOutput arg0) throws IOException {
    throw new UnsupportedOperationException("Not supported yet.");
    }

    public void readFields(DataInput in) throws IOException {
    this.aSAssoc = // How to convert DataInput to String Array?
    }

    public void write(PreparedStatement arg0) throws SQLException {
    throw new UnsupportedOperationException("Not supported yet.");
    }

    public void readFields(ResultSet rs) throws SQLException {
    this.aSAssoc = // How to convert ResultSet to String Array?
    }
    }
    --
    View this message in context:
    http://www.nabble.com/How-do-I-convert-DataInput-and-ResultSet-to-array-of-String--tp23770747p23826464.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
    --
    View this message in context: http://www.nabble.com/How-do-I-convert-DataInput-and-ResultSet-to-array-of-String--tp23770747p23843679.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
  • Aaron Kimball at Jun 3, 2009 at 11:02 pm
    The text serializer will pull out an entire string by using a null
    terminator at the end.

    If you need to know the number of string objects, though, you'll have to
    serialize that before the strings, then use a for loop to decode the rest of
    them.
    - Aaron
    On Tue, Jun 2, 2009 at 6:01 PM, dealmaker wrote:


    Thanks. The number of elements in this array of String is unknown until
    run
    time. If datainput treats it as a byte array, I still have to know the
    size
    of each String. How do I do that? Would you suggest some code samples or
    links that deal with similar situation like this? The only examples I got
    are the ones about counting number of words which deal with integers.
    Thanks.


    Aaron Kimball-3 wrote:
    Hi,

    You can't just turn either of these two types into arrays of strings
    automatically, because they are interfaces to underlying streams of data.
    You are required to know what protocol you are implementing -- i.e., how
    many fields you are transmitting -- and manually read through that many
    fields yourself. For example, a DataInput object is effectively a pointer
    into a byte array. There may be many records in that byte array, but you
    only want to read the fields of the first record out.

    For DataInput / DataOutput, you can UTF8-decode the next field by calling
    Text.readString(dataInput) and Text.writeString(dataOutput).
    For ResultSet, you want resultSet.getString(fieldNum)

    As (yet another) shameless plug ( :smile: ), check out the tool we just
    released, which automates database import tasks. It auto-generates the
    classes necessary for your tables, too.
    http://www.cloudera.com/blog/2009/06/01/introducing-sqoop/

    At the very least, you might want to play with it a bit and read its
    source
    code so you have a better idea of how to implement your own class (since
    you're doing some more creative stuff like building up associative arrays
    for each field).

    Cheers,
    - Aaron
    On Mon, Jun 1, 2009 at 9:53 PM, dealmaker wrote:


    bump. Does anyone know?

    I am using the following class of arraywritable:

    private static class StringArrayWritable extends ArrayWritable {
    private StringArrayWritable (String [] aSString) {
    super (aSString);
    }
    }


    dealmaker wrote:
    Hi,
    How do I convert DataInput to array of String?
    How do I convert ResultSet to array of String?
    Thanks. Following is the code:

    static class Record implements Writable, DBWritable {
    String [] aSAssoc;

    public void write(DataOutput arg0) throws IOException {
    throw new UnsupportedOperationException("Not supported yet.");
    }

    public void readFields(DataInput in) throws IOException {
    this.aSAssoc = // How to convert DataInput to String Array?
    }

    public void write(PreparedStatement arg0) throws SQLException {
    throw new UnsupportedOperationException("Not supported yet.");
    }

    public void readFields(ResultSet rs) throws SQLException {
    this.aSAssoc = // How to convert ResultSet to String Array?
    }
    }
    --
    View this message in context:
    http://www.nabble.com/How-do-I-convert-DataInput-and-ResultSet-to-array-of-String--tp23770747p23826464.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
    --
    View this message in context:
    http://www.nabble.com/How-do-I-convert-DataInput-and-ResultSet-to-array-of-String--tp23770747p23843679.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
  • Dealmaker at Jun 3, 2009 at 11:20 pm
    Would you provide a code sample of it? I don't know how to do serializer in
    hadoop.

    I am using the following class as type of my "value" object:

    private static class StringArrayWritable extends ArrayWritable {
    private StringArrayWritable (String [] aSString) {
    super (aSString);
    }
    }

    Thanks.


    Aaron Kimball-3 wrote:
    The text serializer will pull out an entire string by using a null
    terminator at the end.

    If you need to know the number of string objects, though, you'll have to
    serialize that before the strings, then use a for loop to decode the rest
    of
    them.
    - Aaron
    On Tue, Jun 2, 2009 at 6:01 PM, dealmaker wrote:


    Thanks. The number of elements in this array of String is unknown until
    run
    time. If datainput treats it as a byte array, I still have to know the
    size
    of each String. How do I do that? Would you suggest some code samples
    or
    links that deal with similar situation like this? The only examples I
    got
    are the ones about counting number of words which deal with integers.
    Thanks.


    Aaron Kimball-3 wrote:
    Hi,

    You can't just turn either of these two types into arrays of strings
    automatically, because they are interfaces to underlying streams of data.
    You are required to know what protocol you are implementing -- i.e., how
    many fields you are transmitting -- and manually read through that many
    fields yourself. For example, a DataInput object is effectively a pointer
    into a byte array. There may be many records in that byte array, but you
    only want to read the fields of the first record out.

    For DataInput / DataOutput, you can UTF8-decode the next field by calling
    Text.readString(dataInput) and Text.writeString(dataOutput).
    For ResultSet, you want resultSet.getString(fieldNum)

    As (yet another) shameless plug ( :smile: ), check out the tool we just
    released, which automates database import tasks. It auto-generates the
    classes necessary for your tables, too.
    http://www.cloudera.com/blog/2009/06/01/introducing-sqoop/

    At the very least, you might want to play with it a bit and read its
    source
    code so you have a better idea of how to implement your own class (since
    you're doing some more creative stuff like building up associative arrays
    for each field).

    Cheers,
    - Aaron
    On Mon, Jun 1, 2009 at 9:53 PM, dealmaker wrote:


    bump. Does anyone know?

    I am using the following class of arraywritable:

    private static class StringArrayWritable extends ArrayWritable {
    private StringArrayWritable (String [] aSString) {
    super (aSString);
    }
    }


    dealmaker wrote:
    Hi,
    How do I convert DataInput to array of String?
    How do I convert ResultSet to array of String?
    Thanks. Following is the code:

    static class Record implements Writable, DBWritable {
    String [] aSAssoc;

    public void write(DataOutput arg0) throws IOException {
    throw new UnsupportedOperationException("Not supported yet.");
    }

    public void readFields(DataInput in) throws IOException {
    this.aSAssoc = // How to convert DataInput to String Array?
    }

    public void write(PreparedStatement arg0) throws SQLException {
    throw new UnsupportedOperationException("Not supported yet.");
    }

    public void readFields(ResultSet rs) throws SQLException {
    this.aSAssoc = // How to convert ResultSet to String Array?
    }
    }
    --
    View this message in context:
    http://www.nabble.com/How-do-I-convert-DataInput-and-ResultSet-to-array-of-String--tp23770747p23826464.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
    --
    View this message in context:
    http://www.nabble.com/How-do-I-convert-DataInput-and-ResultSet-to-array-of-String--tp23770747p23843679.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
    --
    View this message in context: http://www.nabble.com/How-do-I-convert-DataInput-and-ResultSet-to-array-of-String--tp23770747p23861270.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
  • Aaron Kimball at Jun 4, 2009 at 8:16 am
    e.g. for readFields(),

    myItems = new ArrayList<String>();
    int numItems = dataInput.readInt();
    for (i = 0; i < numItems; i++) {
    myItems.add(Text.readString(dataInput));
    }

    then on the serialization (write) side, send:

    dataOutput.writeInt(myItems.length());
    for (int i = 0; i < myItems.length(); i++) {
    new Text(myItems.get(i)).writeString(dataOutput);
    }

    You should look at the source code for ArrayWritable, IntWritable, and Text
    -- specifically their write() and readFields() methods -- to get a feel for
    how to write such methods for your own types.

    - Aaron

    On Wed, Jun 3, 2009 at 4:19 PM, dealmaker wrote:


    Would you provide a code sample of it? I don't know how to do serializer
    in
    hadoop.

    I am using the following class as type of my "value" object:

    private static class StringArrayWritable extends ArrayWritable {
    private StringArrayWritable (String [] aSString) {
    super (aSString);
    }
    }

    Thanks.


    Aaron Kimball-3 wrote:
    The text serializer will pull out an entire string by using a null
    terminator at the end.

    If you need to know the number of string objects, though, you'll have to
    serialize that before the strings, then use a for loop to decode the rest
    of
    them.
    - Aaron
    On Tue, Jun 2, 2009 at 6:01 PM, dealmaker wrote:


    Thanks. The number of elements in this array of String is unknown until
    run
    time. If datainput treats it as a byte array, I still have to know the
    size
    of each String. How do I do that? Would you suggest some code samples
    or
    links that deal with similar situation like this? The only examples I
    got
    are the ones about counting number of words which deal with integers.
    Thanks.


    Aaron Kimball-3 wrote:
    Hi,

    You can't just turn either of these two types into arrays of strings
    automatically, because they are interfaces to underlying streams of data.
    You are required to know what protocol you are implementing -- i.e., how
    many fields you are transmitting -- and manually read through that
    many
    fields yourself. For example, a DataInput object is effectively a pointer
    into a byte array. There may be many records in that byte array, but you
    only want to read the fields of the first record out.

    For DataInput / DataOutput, you can UTF8-decode the next field by calling
    Text.readString(dataInput) and Text.writeString(dataOutput).
    For ResultSet, you want resultSet.getString(fieldNum)

    As (yet another) shameless plug ( :smile: ), check out the tool we
    just
    released, which automates database import tasks. It auto-generates the
    classes necessary for your tables, too.
    http://www.cloudera.com/blog/2009/06/01/introducing-sqoop/

    At the very least, you might want to play with it a bit and read its
    source
    code so you have a better idea of how to implement your own class (since
    you're doing some more creative stuff like building up associative arrays
    for each field).

    Cheers,
    - Aaron
    On Mon, Jun 1, 2009 at 9:53 PM, dealmaker wrote:


    bump. Does anyone know?

    I am using the following class of arraywritable:

    private static class StringArrayWritable extends ArrayWritable {
    private StringArrayWritable (String [] aSString) {
    super (aSString);
    }
    }


    dealmaker wrote:
    Hi,
    How do I convert DataInput to array of String?
    How do I convert ResultSet to array of String?
    Thanks. Following is the code:

    static class Record implements Writable, DBWritable {
    String [] aSAssoc;

    public void write(DataOutput arg0) throws IOException {
    throw new UnsupportedOperationException("Not supported
    yet.");
    }

    public void readFields(DataInput in) throws IOException {
    this.aSAssoc = // How to convert DataInput to String Array?
    }

    public void write(PreparedStatement arg0) throws SQLException {
    throw new UnsupportedOperationException("Not supported
    yet.");
    }

    public void readFields(ResultSet rs) throws SQLException {
    this.aSAssoc = // How to convert ResultSet to String Array?
    }
    }
    --
    View this message in context:
    http://www.nabble.com/How-do-I-convert-DataInput-and-ResultSet-to-array-of-String--tp23770747p23826464.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
    --
    View this message in context:
    http://www.nabble.com/How-do-I-convert-DataInput-and-ResultSet-to-array-of-String--tp23770747p23843679.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
    --
    View this message in context:
    http://www.nabble.com/How-do-I-convert-DataInput-and-ResultSet-to-array-of-String--tp23770747p23861270.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMay 28, '09 at 9:48p
activeJun 4, '09 at 8:16a
posts7
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Dealmaker: 4 posts Aaron Kimball: 3 posts

People

Translate

site design / logo © 2022 Grokbase