Grokbase Groups Thrift user July 2011
FAQ

[Thrift-user] File Uploads in Thrift with Ruby

Srinivasan Anand
Jul 18, 2011 at 1:44 pm
Hi,

I am a newbie with Apache Thrift. Though I was able to setup a simple
Server/Client message exchange with the help of Thrift in Ruby
(ruby-thrift-0.6.0), I am actually facing some difficulties in setting up
the code for a simple file upload/transfer functionality from the Client to
the Server.

Since there is no File data type accepted by thrift, I actually had read the
file contents with the File.open("file_path",'rb'){|f| f.read} as string and
transported over to the server and then writing to a new file copy. This
worked for txt files. But in the case of PDF, DOC and probably all other
file types, this doesn't work out. I also tried using the base64 encoding
and decoding mechanisms for this, but no luck.

Following some of the posts in the users-mailing list archives, I thought
the TFileTransport or TSimpleFileTransport or TFDTransport should do the
magic. But I am not able to find either of them in the ruby-thrift gem apart
from the Buffered, Framed, HTTPClient, IOStream and MemoryBuffer Transports.
Am I correct in understanding that these TFileTransport or
TSimpleFileTransport or TFDTransport acually help in transfer/upload of the
files in Java/C++ ? or Am I seriously missing something?

Can we actually do a kind of File Upload (not big file, max 10Mb) with the
ruby-thrift gem from the client to the server. If not, can you suggest any
other possible alternatives for doing this?

Some links I had referred to in the archives:
http://mail-archives.apache.org/mod_mbox/thrift-user/201010.mbox/%3C8976D98D-994D-4539-AF9C-17771F1CDC34@manbert.com%3E


Anand
reply

Search Discussions

3 responses

  • Bryan Duxbury at Jul 18, 2011 at 4:16 pm
    This is totally possible, but you do have to read the entire file into a
    ruby string object before sending, or make a multi-RPC-call scheme on your
    own. There aren't any transports that will help you with this - those are
    all for Thrift's communication, not for reading in large binaries.
    On Mon, Jul 18, 2011 at 5:30 AM, Srinivasan Anand wrote:

    Hi,

    I am a newbie with Apache Thrift. Though I was able to setup a simple
    Server/Client message exchange with the help of Thrift in Ruby
    (ruby-thrift-0.6.0), I am actually facing some difficulties in setting up
    the code for a simple file upload/transfer functionality from the Client to
    the Server.

    Since there is no File data type accepted by thrift, I actually had read
    the
    file contents with the File.open("file_path",'rb'){|f| f.read} as string
    and
    transported over to the server and then writing to a new file copy. This
    worked for txt files. But in the case of PDF, DOC and probably all other
    file types, this doesn't work out. I also tried using the base64 encoding
    and decoding mechanisms for this, but no luck.

    Following some of the posts in the users-mailing list archives, I thought
    the TFileTransport or TSimpleFileTransport or TFDTransport should do the
    magic. But I am not able to find either of them in the ruby-thrift gem
    apart
    from the Buffered, Framed, HTTPClient, IOStream and MemoryBuffer
    Transports.
    Am I correct in understanding that these TFileTransport or
    TSimpleFileTransport or TFDTransport acually help in transfer/upload of the
    files in Java/C++ ? or Am I seriously missing something?

    Can we actually do a kind of File Upload (not big file, max 10Mb) with the
    ruby-thrift gem from the client to the server. If not, can you suggest any
    other possible alternatives for doing this?

    Some links I had referred to in the archives:

    http://mail-archives.apache.org/mod_mbox/thrift-user/201010.mbox/%3C8976D98D-994D-4539-AF9C-17771F1CDC34@manbert.com%3E


    Anand
  • Srinivasan Anand at Jul 18, 2011 at 6:34 pm
    Hi Bryan,

    Thanks for the immediate response. As I was explaining to you, on the
    client side, I was reading the file into a ruby string with one of the
    two schemes like
    1. file_contents = File.open('file_path','rb'){|f| f.read}
    2. file_contents = IOStream.open('file_path')

    Is this what you were suggesting by reading the entire file and
    storing in a string.

    Then, I was calling a method of the server and sending the
    file_contents over the thrift. There on the server, I am again writing
    the file_contents into a file with similar name and extension.
    new_file = File.open('file_path','w')
    new_file.write(file_contents)
    new_file.close

    The issue here is that this works for text files (.txt files), but not
    for other file types like PDF, .doc and others.

    Apart from these, I actually did not get this multi-RPC-call thing.
    What exactly is a multi-RPC-call? It would be great if you can guide
    me on this and point me to somewhere where in the galaxy for this.

    Regards,
    Anand
    This is totally possible, but you do have to read the entire file into a
    ruby string object before sending, or make a multi-RPC-call scheme on your
    own. There aren't any transports that will help you with this - those are
    all for Thrift's communication, not for reading in large binaries.
    On, Jul 18, 2011 at 5:30 AM, Srinivasan Anand wrote:
    Hi,

    I am a newbie with Apache Thrift. Though I was able to setup a simple
    Server/Client message exchange with the help of Thrift in Ruby
    (ruby-thrift-0.6.0), I am actually facing some difficulties in setting up
    the code for a simple file upload/transfer functionality from the Client to
    the Server.

    Since there is no File data type accepted by thrift, I actually had read
    the
    file contents with the File.open("file_path",'rb'){|f| f.read} as string
    and
    transported over to the server and then writing to a new file copy. This
    worked for txt files. But in the case of PDF, DOC and probably all other
    file types, this doesn't work out. I also tried using the base64 encoding
    and decoding mechanisms for this, but no luck.

    Following some of the posts in the users-mailing list archives, I thought
    the TFileTransport or TSimpleFileTransport or TFDTransport should do the
    magic. But I am not able to find either of them in the ruby-thrift gem
    apart
    from the Buffered, Framed, HTTPClient, IOStream and MemoryBuffer
    Transports.
    Am I correct in understanding that these TFileTransport or
    TSimpleFileTransport or TFDTransport acually help in transfer/upload of the
    files in Java/C++ ? or Am I seriously missing something?

    Can we actually do a kind of File Upload (not big file, max 10Mb) with the
    ruby-thrift gem from the client to the server. If not, can you suggest any
    other possible alternatives for doing this?

    Some links I had referred to in the archives:

    http://mail-archives.apache.org/mod_mbox/thrift-user/201010.mbox/%3C8976D98D-994D-4539-AF9C-17771F1CDC34@manbert.com%3E


    Anand
  • Bryan Duxbury at Jul 18, 2011 at 7:35 pm
    My ruby is a little rusty, but yes, you seem to be on the right path for
    reading in the file. There might be some issues with the way you're opening
    the file, though. You should google around for how to read in a binary file.

    Forget I said anything about the multi-rpc thing. I see now that this is
    irrelevant.
    On Mon, Jul 18, 2011 at 11:34 AM, Srinivasan Anand wrote:

    Hi Bryan,

    Thanks for the immediate response. As I was explaining to you, on the
    client side, I was reading the file into a ruby string with one of the
    two schemes like
    1. file_contents = File.open('file_path','rb'){|f| f.read}
    2. file_contents = IOStream.open('file_path')

    Is this what you were suggesting by reading the entire file and
    storing in a string.

    Then, I was calling a method of the server and sending the
    file_contents over the thrift. There on the server, I am again writing
    the file_contents into a file with similar name and extension.
    new_file = File.open('file_path','w')
    new_file.write(file_contents)
    new_file.close

    The issue here is that this works for text files (.txt files), but not
    for other file types like PDF, .doc and others.

    Apart from these, I actually did not get this multi-RPC-call thing.
    What exactly is a multi-RPC-call? It would be great if you can guide
    me on this and point me to somewhere where in the galaxy for this.

    Regards,
    Anand
    This is totally possible, but you do have to read the entire file into a
    ruby string object before sending, or make a multi-RPC-call scheme on your
    own. There aren't any transports that will help you with this - those are
    all for Thrift's communication, not for reading in large binaries.
    On, Jul 18, 2011 at 5:30 AM, Srinivasan Anand <mail2sanand@gmail.com
    wrote:
    Hi,

    I am a newbie with Apache Thrift. Though I was able to setup a simple
    Server/Client message exchange with the help of Thrift in Ruby
    (ruby-thrift-0.6.0), I am actually facing some difficulties in setting
    up
    the code for a simple file upload/transfer functionality from the
    Client to
    the Server.

    Since there is no File data type accepted by thrift, I actually had
    read
    the
    file contents with the File.open("file_path",'rb'){|f| f.read} as
    string
    and
    transported over to the server and then writing to a new file copy.
    This
    worked for txt files. But in the case of PDF, DOC and probably all
    other
    file types, this doesn't work out. I also tried using the base64
    encoding
    and decoding mechanisms for this, but no luck.

    Following some of the posts in the users-mailing list archives, I
    thought
    the TFileTransport or TSimpleFileTransport or TFDTransport should do
    the
    magic. But I am not able to find either of them in the ruby-thrift gem
    apart
    from the Buffered, Framed, HTTPClient, IOStream and MemoryBuffer
    Transports.
    Am I correct in understanding that these TFileTransport or
    TSimpleFileTransport or TFDTransport acually help in transfer/upload of
    the
    files in Java/C++ ? or Am I seriously missing something?

    Can we actually do a kind of File Upload (not big file, max 10Mb) with
    the
    ruby-thrift gem from the client to the server. If not, can you suggest
    any
    other possible alternatives for doing this?

    Some links I had referred to in the archives:
    http://mail-archives.apache.org/mod_mbox/thrift-user/201010.mbox/%3C8976D98D-994D-4539-AF9C-17771F1CDC34@manbert.com%3E

    Anand

Related Discussions

Discussion Navigation
viewthread | post

2 users in discussion

Srinivasan Anand: 2 posts Bryan Duxbury: 2 posts