FAQ
Hi all,

I am trying to synchronize multiple read Streams so that I can process
equivalent bytes from them. For example, if there are multiple input
streams created with fs.createReadStream(), I need to process bytes from
the same position in all the files into a new value that is written to an
output write Stream. I'm trying not to use Buffers, because I want to
handle a potentially large amount of data. I could use fs.read() for the
case of input files and just read from the same position in each file, but
I want this to be general and operate on generic streams that are not
necessarily formed from a file.

Is this possible? Are there any solutions out there that have the same or
similar functionality?

Thanks in advance for any help.


--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Search Discussions

  • Jake Verbaten at Dec 19, 2012 at 7:42 am
    Look at the implementation of pipe. Rewrite it so that it reads from
    multiple inputs.

    The key points here is calling read with a fixed n and then calling write
    on the output with a list of buffers.

    If any readable stream returns less then n keep the chunks around until
    that stream is readable then alter your read(n) calls to get up to your n
    again and then call write.

    Pipe's default logic is to end the writable when the first readable ends.
    You may want to change that.

    On Tue, Dec 18, 2012 at 7:10 PM, Alex wrote:

    Hi all,

    I am trying to synchronize multiple read Streams so that I can process
    equivalent bytes from them. For example, if there are multiple input
    streams created with fs.createReadStream(), I need to process bytes from
    the same position in all the files into a new value that is written to an
    output write Stream. I'm trying not to use Buffers, because I want to
    handle a potentially large amount of data. I could use fs.read() for the
    case of input files and just read from the same position in each file, but
    I want this to be general and operate on generic streams that are not
    necessarily formed from a file.

    Is this possible? Are there any solutions out there that have the same or
    similar functionality?

    Thanks in advance for any help.


    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Alex at Dec 19, 2012 at 10:51 am
    Hi Raynos,
    Thank you for your reply. I just want to confirm: did you mean the .pipe()
    from the new streams,
    i.e. https://github.com/joyent/node/blob/master/lib/_stream_readable.js#L296?
    On Wednesday, December 19, 2012 2:42:46 AM UTC-5, Raynos wrote:

    Look at the implementation of pipe. Rewrite it so that it reads from
    multiple inputs.

    The key points here is calling read with a fixed n and then calling write
    on the output with a list of buffers.

    If any readable stream returns less then n keep the chunks around until
    that stream is readable then alter your read(n) calls to get up to your n
    again and then call write.

    Pipe's default logic is to end the writable when the first readable ends.
    You may want to change that.


    On Tue, Dec 18, 2012 at 7:10 PM, Alex <alex...@gmail.com <javascript:>>wrote:
    Hi all,

    I am trying to synchronize multiple read Streams so that I can process
    equivalent bytes from them. For example, if there are multiple input
    streams created with fs.createReadStream(), I need to process bytes from
    the same position in all the files into a new value that is written to an
    output write Stream. I'm trying not to use Buffers, because I want to
    handle a potentially large amount of data. I could use fs.read() for the
    case of input files and just read from the same position in each file, but
    I want this to be general and operate on generic streams that are not
    necessarily formed from a file.

    Is this possible? Are there any solutions out there that have the same or
    similar functionality?

    Thanks in advance for any help.


    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nod...@googlegroups.com<javascript:>
    To unsubscribe from this group, send email to
    nodejs+un...@googlegroups.com <javascript:>
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Jake Verbaten at Dec 19, 2012 at 6:02 pm
    Yeah when I was talking about read(n) I mean new streams, you could do a
    similar dance for old streams but there would be more buffering if pause is
    not respected.

    On Wed, Dec 19, 2012 at 2:51 AM, Alex wrote:

    Hi Raynos,
    Thank you for your reply. I just want to confirm: did you mean the .pipe()
    from the new streams, i.e.
    https://github.com/joyent/node/blob/master/lib/_stream_readable.js#L296?
    On Wednesday, December 19, 2012 2:42:46 AM UTC-5, Raynos wrote:

    Look at the implementation of pipe. Rewrite it so that it reads from
    multiple inputs.

    The key points here is calling read with a fixed n and then calling write
    on the output with a list of buffers.

    If any readable stream returns less then n keep the chunks around until
    that stream is readable then alter your read(n) calls to get up to your n
    again and then call write.

    Pipe's default logic is to end the writable when the first readable ends.
    You may want to change that.

    On Tue, Dec 18, 2012 at 7:10 PM, Alex wrote:

    Hi all,

    I am trying to synchronize multiple read Streams so that I can process
    equivalent bytes from them. For example, if there are multiple input
    streams created with fs.createReadStream(), I need to process bytes from
    the same position in all the files into a new value that is written to an
    output write Stream. I'm trying not to use Buffers, because I want to
    handle a potentially large amount of data. I could use fs.read() for the
    case of input files and just read from the same position in each file, but
    I want this to be general and operate on generic streams that are not
    necessarily formed from a file.

    Is this possible? Are there any solutions out there that have the same
    or similar functionality?

    Thanks in advance for any help.


    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/**node/wiki/Mailing-List-*
    *Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines>
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nod...@googlegroups.com

    To unsubscribe from this group, send email to
    nodejs+un...@**googlegroups.com

    For more options, visit this group at
    http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en>
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupnodejs @
categoriesnodejs
postedDec 19, '12 at 3:10a
activeDec 19, '12 at 6:02p
posts4
users2
websitenodejs.org
irc#node.js

2 users in discussion

Alex: 2 posts Jake Verbaten: 2 posts

People

Translate

site design / logo © 2022 Grokbase